Rendered at 12:49:19 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tyleo 1 days ago [-]
The chart at the top is somewhat confusing. It says, “lower is better” but the y-axis is inverted! So visually higher in the chart is better but lower in terms of # value.
layer8 22 hours ago [-]
It isn’t inverted though? 32000 is at the bottom, 40000 is at the top. Maybe it changed since you commented.
couAUIA 1 days ago [-]
well thank you so much for this
Tenoke 24 hours ago [-]
Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.
jrockway 18 hours ago [-]
I put context size in my Claude Code status bar and ... it does get "tired" when it's at 50%-60% context. I figured this out when I'm like "now modify the testbed to run this and add a test to exercise blah blah blah" and it was just like "That would be a substantial infra lift." In my head I say "I am paying a substantial amount for you to do this!" but just type /compact and re-prompt and my "substantial infra lift" is done without complaint.
I guess everyone needs a nap after a long day of conversing and writing code. So like us!
/compact is prone to error and I wouldn't recommend it in the middle of work. But when you are switching to a related but not completely new task, it helps. ("Now write the integration tests." vs. "on foo.go line 476 that you just wrote, I think there is a deadlock with bar.go line 123 that you just added". It doesn't really need the context to write the tests, it can get that by reading the code. But for iterating on lines of code it just produced, /compact is going to throw away whatever "thought process" led to that code and it's usually not a great thing to do.)
aytigra 7 hours ago [-]
I've actually found that compacting as often as possible, after planning, then between each implementation step works the best, it both unloads unnecessary context of previous edits and test runs, makes thinking cheaper, and most importantly after each compaction it re-loads CLAUDE.md which make it much more enforcing (otherwise it just moves to the back of context and slips from model's attention).
fendy3002 7 hours ago [-]
Use untracked file as memory and instruct agent to put progress into that, it'll help managing context in current and new sessions
Tenoke 1 hours ago [-]
I do that but it ends up putting so much extra crap there that it has the same drawbacks.
esperent 23 hours ago [-]
This is one of the advantages of pi. I made a /protect command that protects the message from compaction. I also protect skills automatically.
So for long running tasks I'll do
/protect your goal is...
kaelwd 6 hours ago [-]
https://github.com/k0valik/pi-blackhole works pretty well for this, instead of a summary of a summary of a summary it gets a rolling log of important instructions and discoveries, and can /recall exact contents of a previous message or tool call if needed.
esperent 3 hours ago [-]
That looks interesting, have you found it useful in practice? I'm a bit burnt out on these wildly ambitious big plugins (headspace, RTK etc.) because in each case when I've done personal evals they come out worse than the default.
sdesol 18 hours ago [-]
I am curious as to how this protect works. I am currently working on a pi brain extensions and one of the commands that I will support is:
/brains inspect
that will let you inspect a session and one of the features is the ability to review a compacted message like so:
With /compact in Pi, it creates a message that you can easily review and I am curious as to how '/protect' works.
One of the features that I am working on is to make it easy for agents to retrieve the exact message/event before compaction and I am curious if /protect is a deterministic process or if it is just instructions.
throw10920 20 hours ago [-]
I'm also a fan of Pi - although, doesn't Claude code (and most other harnesses) already have "custom summarization instructions" for compaction/summarization that already allow you to do this? Or have you added on something extra?
lionkor 18 hours ago [-]
They way I read it, this is a workaround for pi, not for Claude.
Edit: removed off topic political spam
throw10920 18 hours ago [-]
Can you please keep the unsolicited ranting out of the replies? It breaks the guidelines and degrades the value of the site. Thanks!
It's part of the rationale for why one would choose Pi over Claude, even if Pi lacks the feature. I admit my tone is entirely inappropriate though.
throw10920 18 hours ago [-]
> It's part of the rationale for why one would choose Pi over Claude
It's unrelated to my direct comment and it's a generic tangent off the thread further up, which is about technical issues with Claude Code. As the guidelines say:
"On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity."
"Eschew flamebait. Avoid generic tangents."
"Please don't use Hacker News for political or ideological battle. It tramples curiosity."
Nobody wants to see you hijacking technical threads to turn them into political flamewars. It's against the guidelines, anti-intellectual, and profoundly boring.
lionkor 18 hours ago [-]
Fair point
jswny 24 hours ago [-]
Unfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction.
For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact
brookst 23 hours ago [-]
Compacting at all is a mistake. With 1m context window there is no reason for a single task to require compaction.
Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after.
It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude) doing the compaction.
wgjordan 19 hours ago [-]
> With 1m context window there is no reason for a single task to require compaction
Only if money is no object. Cache reads are cheap (10% of uncached input costs) but definitely not free, and cached reads dominate session costs at long context lengths. A prompt at 20k context with $0.01 in cached reads would cost $0.40 in cached reads at 800k context, that quickly adds up for long sessions.
brookst 16 hours ago [-]
I’m not following the implication that these economics argue for jsing compaction instead of just clearing context?
jeeeb 11 hours ago [-]
Agree. Another consideration is that input token costs are effectively quadratic with the number of conversation turns. The minute you get into a long conversation you can see the costs shoot up.
You can get far more gas out of even a $20 plan of you’re careful to break things up into relatively small discrete steps, clear context regularly and give the model plenty of information to work with.
My workflow for bigger features is to write out a plan document and then proceed in smaller implementation steps, reviewing as I go. If I find something odd, I ask the agent why and often that leads to discovering a new dimension to the problem, which in turn is an opportunity to adjust the approach.
kccqzy 19 hours ago [-]
You just haven’t worked on tasks that are complicated enough. Occasionally it took more than 1M tokens just to come up with a plausible plan.
Personally I find using /rewind judiciously is better than using /compact. The latter essentially gives you no control of what details to discard, but the former at least has coarse-grained control.
brookst 16 hours ago [-]
Oh my goodness. I’ve used over 1B tokens on a single feature. I’m running at about 25B tokens/month right now.
My whole point was that by planning in advance you can shard the work into manageable sections with clear beginnings and outputs with acceptance croteria, and never compact, or even use more than a few hundfed thousand tokens in context.
It’s all hierarchical. Looking at an eval feature building right now, it’s 20ish build plans, each with zero to five or so /clear moments.
But maybe that’s the key thing… I don’t iteratively prompt ad hoc software writing. I do iterate on requirements, but if those are solid enough there is no “now write this function, now write that module”.
irthomasthomas 16 hours ago [-]
Have you ever looked at how much performance drops as context grows? The difference in intelligence between 100k and 1M is huge, like opus drops to haiku level performance, or worse. For that reason I try to keep under 200k. That feels about the upper bound for tasks requiring accuracy.
mrtesthah 22 hours ago [-]
Most models’ reasoning abilities drops off significantly between the 256K-1M token ranges of the context window. There’s too much stuff to “pay attention to” at once.
hedgehog 20 hours ago [-]
I auto compact around 200k tokens both due to this and because the cached read cost really escalates when sessions have more tokens than that (too short and you pay a lot in per-compact re-reading of state)
tony_cannistra 21 hours ago [-]
This is the way.
weiliddat 22 hours ago [-]
One interesting harness thing I saw Cursor do is to give the model access to the entire thread.
Even if it doesn't fit in the context window, the model can search through past turns and sanity check if something doesn't seem to be going right, or be prompted to follow an early message, "when starting on a new item, review the first message for how we should approach this"
semiquaver 23 hours ago [-]
Unlike other model/harness pairs, codex+gpt also passes an opaque encrypted artifact speculated to be an embedding representing the conversation back to the successor generation which is “denser” or at least higher fidelity than summarized text.
what-the-grump 13 hours ago [-]
Pretty much, I stopped using Claude for that reason, I shouldn't have to keep track of the context of the model all the time. And god forbid I don't, the amount of time I've lost to Claude just sending it and then lying is not worth it when 5.5 will just do stuff, compact 2-3 times in the task and still not go insane.
This whole 1 million context window is a lie after 300k Claude degrades to unusable, and compact doesnt help. I've had Codex tabs open for weeks, I have to regularly restart Claude.
perching_aix 19 hours ago [-]
That's wild to read considering Codex's compaction to me feels like the model becoming demented. Super aggressive, throws away almost everything.
jedisct1 22 hours ago [-]
This is not an issue with Claude, but with Claude Code.
With swival.dev you can use long sessions without ever doing any manual compaction or reset, even with Claude models.
Androider 23 hours ago [-]
Quality degrades long before you hit compaction, compaction is the "Next gas station, 100 miles" sign but you're already in the middle of nowhere.
You don't need to go down the rabbit hole of crazy workflows, but to avoid slop:
- Break down the work into tasks
- New context. Create a plan for one task.
- New context. Implement the plan.
- New context. /code-review the implementation
- New context. Fix the review findings.
- Repeat for next task.
I do this with Fable 5, and the quality is consistently quite good. If the context goes over 50%, the quality will become crap and you end up with 4 duplicates of the same thing across the codebase. Letting a current session review its own work is like asking a student to grade their own paper.
beefsack 23 hours ago [-]
For the vast majority of models I've worked with, 150k tokens of context seems like some magic spot. If you can stay under it then the session will stay on track and the agent won't start getting forgetful and weird.
Just like with real work, it's valuable to break down big tasks into small tasks that you can knock over in a single session. When a session does start getting too big, you just need to ask the agent to reply with a comprehensive handover report and paste it into a new session.
robwwilliams 21 hours ago [-]
For my quirky workflows I need 100K even to prep Opus 4.8 or higher to begin good work. Not programming, but a complex synthetic architecture for implementing more attentional autonomy for machines.
23 hours ago [-]
michaellee8 12 hours ago [-]
i found that at 700k-ish context even fable becomes an idiot, maybe openai's decision to cap codex context at 400k is correct, 400k is really a sweet spot where most part of the context is reliable.
epolanski 4 hours ago [-]
It doesn't forget, it doesn't care.
It's a great vibecoding model, but a crap assistant, and it forgets in few turns.
It's trained on benchmarks.
IceDane 23 hours ago [-]
Stop using compaction. Stop treating sessions as something you keep alive for weeks. Set up a proper workflow for handing off work and start handing off well before maximum context is hit. It's less efficient, slower and more costly to constantly be sitting at high context and compaction is just not good.
robwwilliams 21 hours ago [-]
There is no one-size-fits-all, even for programming.
demosthanos 15 hours ago [-]
There isn't, but everything that they said above is absolutely true and people who try to fight it will suffer. There is lots of flexibility within those constraints, but the constraints aren't imaginary.
stevenhuang 13 hours ago [-]
If you want to use LLMs effectively, everything OP said is true. It has been this way since the beginning. There will be a time when long context and compaction will be effective, but now isn't it.
IceDane 20 hours ago [-]
There's a good way and a bad way to use tools. My suggestions were based on my extensive usage of the tools in question.
theptip 20 hours ago [-]
Great eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that.
Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum.
Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.
ttul 20 hours ago [-]
I think ultra mode needs to be more clearly documented (or perhaps cautioned against!). Most devs - myself included - who saw “ultra” mode figured that it’s just a magic bullet that makes the model work harder and achieve better results. But, for many tasks, ultra mode is possibly worse and certainly more expensive.
lucas_t_a 36 minutes ago [-]
i let it run once to just generally find any bugs using all my quota, them once i get back to it i ran on effort /max to continue with explicit don't run workflows fix all yourself, it was ultra effective.
ultra becomes a mode where your model enhances your prompt them it is execution multiplied by all other agents, which have them going in all sorts of directions.
theptip 17 hours ago [-]
Yeah I think having a detailed mental model of what it’s actually doing helps.
At least for Fable/Opus (didn’t confirm for Sol yet) Ultra means “write an ephemeral programmatic harness encoding this workflow”. There is actually a TS harness that gets run for the workflow.
If you have a task where the agent/sub-agent pattern works, Ultra just adds indirection.
I think it is possible to get an intuition for an individual model but really you need to eval to be sure. My heuristic though is if you need to treat each work item differently depending on the results, probably agent/subagent. If you want to do the same steps across some queue / tree / DAG of work items, ultra is a better bet. (Or actually write a durable scaffold if you are going to repeatedly run it over > thousands of items.)
embedding-shape 18 hours ago [-]
> that it’s just a magic bullet that makes the model work harder and achieve better results
"Ultra" is a harness feature, and has nothing to do with the model itself. If OpenAI wanted to, they could offer "Ultra" with any of the GPT models, although 5.6 supposedly been trained with this specific harness feature in mind. "max" is basically the top "work harder and achieve better results" parameter for the reasoning effort for the current models.
I had good success with it initially, until I discovered that OpenAI encrypts the prompts that the main model sends to the sub-agents (even in Codex, in local files you only see cipher text), then I completely dropped any experimentation of it as without introspection, it becomes basically useless for any real usage.
ttul 18 hours ago [-]
Oh yes, I know it’s a harness feature.
alexpotato 19 hours ago [-]
> But, for many tasks, ultra mode is possibly worse and certainly more expensive.
Went through something similar. Fable would just spends minutes thinking, processing, confabulating etc.
I dropped down to Haiku and got an answer in >30 seconds.
Huppie 19 hours ago [-]
Simon Willison recently had a useful tip instructing Claude to use its judgement to pick an appropriate model for tasks.
After a short discussion about the idea with Claude mostly on how it fits in my workflow and what models / effort I would like for certain tasks it placed a paragraph in my global Claude.md and it has worked wonders.
Ultra became a lot better (faster, cheaper for the same output) and the amount of time Fable gets stuck overthinking things are reduced to the places where I think that model makes sense, for the rest it started fanning out a lot to Opus, Sonnet and even Haiku.
kingkongjaffa 16 hours ago [-]
How does it change model mid conversation?
Does it spawn sub agents with different models or is it the same single conversation dynamically switching models?
Huppie 6 hours ago [-]
Subagents indeed. This [0] was the post I was referencing, and here's the original suggested prompt:
> For all coding tasks use your judgement to decide an appropriate lower power model and run that in a subagent
As said, I tweaked it a little bit to my preference (mostly because it started delegating a bit too much) and had it placed in my global claude.md but that's the gist.
Yes, sub-agents I believe. You need to pay to reload context if you switch the main conversation model effort.
sreekanth850 20 hours ago [-]
Anthropic is losing bigtime against Open AI in coding space. I was using Claude code upto last march. Not an enterprise customer, but a responsible AI user where i don't over spend and use basic plan to manage repository with 400Kloc all together. We sell to local government and a team of 3. Claude code was super slow, never able to fix issues properly.(Despite with proper test cases, observability, documentation and layered architecture). After moving to codex, life has been much easy and free form usage anxiety. Now managing entire things with 2 codex plus account per team member. Its high time anthropic should stop scare mongering and build efficient models. Everyone doesn't need Fable. People need models that solve problems efficiently.
gmac 20 hours ago [-]
I've found Opus 4.8 pretty amazing, and Codex a bit meh. YMMV?
cyral 18 hours ago [-]
I just downloaded and paid for Codex this week because I want to stay on top of the AI tools and understand their capabilities. I've had some good results using 5.6 Sol, although it tends to never want to write any comments (despite modifying the project rules to tell it it MUST), and also occasionally just does a bit of thinking and stops. It'll say "Working through the remaining work" and just ends the chat until I tell it "continue" or something.
This is very anecdotal evidence but I have to rant about it... I tried 5.6 Terra (high) earlier today to fix a bug with a slow page. It just... removed the part of the page that was slow, and made it a client side request (still slow, but not blocking SSR I guess). I tried with Sonnet 5 and it correctly found the issue where an unhandled case was continuing to retry and failing. I am always telling people how the frontier models are SO much more capable than what they may think but this one thing today had me scratching my head at why it would ever do that. It was the first time I experienced the "great, I removed the failing test case" kind of issue.
michaellee8 12 hours ago [-]
I got the complete opposite of what you get, sol ultra literally vibed the entire system out for me from one plan mode approval.
jrockway 18 hours ago [-]
I think the comment you're replying to is talking about cost and speed, and Opus 4.8 xhigh is pretty expensive and pretty slow. I work around the slowness by having multiple auto mode sessions going in parallel (review this, investigate that, plan this, etc.; and yes, I use a VM that can't touch prod for auto-mode work), and work around the cost by my employer pays for it and everyone else I work with uses more AI than I do :)
I really really like Fable for software engineering cases; add an alert and write a runbook, go pull metrics for this incident, write me a TUI that does blank, etc. It is astoundingly good and I am very picky. It costs A LOT of money though. I am not sure how I get away with my Fable usage.
sreekanth850 20 hours ago [-]
There is one thing that stay apart, I can use luna high with pretty less cost and do a comprehensive audit using sol high on every push from github.
throwaway314155 15 hours ago [-]
My only issue with Codex (Sol 5.6) is that it sometimes finds tasks on its own to solve that are definitely out of scope. like it’s always aiming for the 1.0 release when we’re still on 0.1.0a
sreekanth850 5 hours ago [-]
Luna respect AGENTS.md and even refuse to mark a task slice completed without testing because i wrote this as rule.
rubyn00bie 13 hours ago [-]
I think it depends on the problem space, and to a large extent the languages used. I have had the opposite experience with GPT.
I have found GPT to be pretty awful at writing Elixir, and dealing with more open ended tasks. Opus is (for the moment) better at writing Elixir, and Fable is much better than either at actually grokking a problem space. I had switched to GPT for about six weeks and eventually stopped using it altogether after it relentlessly gaslight me. Those six weeks using it were essentially completely wasted work.
Right now I’ve moved to a mix of Opus/Fable and Deepseek Pro. Deepseek is far and away the most cost effective, fast, and quite good for 90% of implementation work. Deepseek really only falls apart (in Elixir) when it tries to use runtime features at compile time; which, for the initial lift of a project I have was a bit painful. Fable was able to pretty quickly get things sorted and now things are progressing nicely.
With all that said, I think the various models have unique strengths that are sometime hard to uncover. I also don’t see a world, at least in the near future, where I’ll only use one model. I’m happy Anthropic continues to push the frontier forward. I’m more than happy to trade “efficiency” for quality when I need to.
o10449366 1 days ago [-]
/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now:
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
embedding-shape 1 days ago [-]
> 3. /goal Spend X minutes from $time writing a technical design doc on $feature.
Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took.
I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes.
So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.
lonelyasacloud 23 hours ago [-]
Leaving to run unhindered makes sense when the agents understand both the complete domain context and can be supplied with a strong exit criteria that they understand well.
Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements.
Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.
illliillll 1 days ago [-]
Well, I’ve been having 5.6 sol work on tasks like “find every OTA app on the internet”
I find explicit time bounds are useful for tasks like this, otherwise the LLM will almost certainly return too early.
drivebyhooting 16 hours ago [-]
The proof prompts do say to spend at least 8 hours.
o10449366 22 hours ago [-]
I think like the other commentator noted, I don't do this blindly, as I also wouldn't if I was instructing a junior engineer. I have a rough sense of how long it would take me to research all angles of the task. 5 minutes for a complicated feature is too short for a comprehensive plan. Letting it run for a few hours is also probably unproductive. But 10-60 minutes depending on complexity with followup review is a nice sweet spot. Just my $0.02.
bob1029 24 hours ago [-]
The hypothesis generation phase is looking like the most critical part of having an agent reliably hit your targets.
Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.
I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.
ai_fry_ur_brain 23 hours ago [-]
[dead]
SvenL 20 hours ago [-]
Isn’t it weird to think a LLM “fully understands” something? What does this mean?
It reminds me of this printing trick “until you are 95% confident”…
Would really be interested to know how this influences the work of the LLM. Also, how would it be if I write “until you certainly understand it”?
aszen 22 hours ago [-]
Won't surprise me if the llm just calls sleep after it's convinced it knows all
tantalor 1 days ago [-]
What is /goal?
lambda 20 hours ago [-]
The article describes it.
Both Codex and Claude Code have it, but they work slightly differently.
Claude Code uses Haiku to read through the transcript and decide if the goal has been completed. If not, Haiku injects a prompt back to the main model to indicate what still needs to be done.
In Codex, instead it's a tool available to the main model, plus some part of the surrounding harness that will re-prompt it if the tool calls haven't yet indicated that the goal is complete.
The issue that they are trying to solve is that sometimes models will stop before they have actually fully completed whatever task they were given; attention isn't perfect, and someitmes they'll complete part of it but not the whole task. Rather than making the user come back and re-prompt to keep going, they add a way to automatically do a bit more nudging to try to get the model to finish the task.
tpicks 14 hours ago [-]
> Claude Code uses Haiku to read through the transcript and decide if the goal has been completed.
feels kinda odd to use a less capable model to determine if the goal is fully complete. Especially if the user is expecting /goal to thoroughly complete the task. A less capable model would be more likely to misclassify `isComplete?`
mdp2021 8 hours ago [-]
> odd
There is nothing odd in the basic principle of Economics of using a "good enough" tool, optimal in other respects (e.g. cheap/er).
(You do not hire Nobel Prize winners as receptionists.)
The issue remains, whether the compromise to be adopted actually is "good enough".
swader999 24 hours ago [-]
On Claude if you start with that, it won't stop until it achieves or exhausts your prompt. It feels like "here's your mission, go do it". I use it a few times a week.
brap 22 hours ago [-]
> won't stop until it achieves or exhausts your prompt
This is overselling it. In all implementations of this that I’ve seen, a more correct phrasing is “won’t stop until it pinky-promises it achieved your goal”.
It usually relies on something like a hook that refuses to let the agent stop unless it includes a “promise sentinel”, which is basically the model having to include “I swear I’m done” in the response (usually as a Markdown comment so the user doesn’t see it). If that sentinel isn’t in the response, the hook rejects the completion and hands it back to the agent (sometimes it also restates the goal to keep it focused). It’s useful for sure, but it fails many times for obvious reasons.
More robust implementations use the hook to spawn another agent that verifies the goal is achieved, but that too isn’t prefect and sometimes performs worse.
Of course if you have a problem that can be deterministically verified, you could have the hook run this verification instead, but this is usually not the case (and even when it is, agents will often cheat the verification e.g. by deleting a test file).
hungryhobbit 21 hours ago [-]
The key thing is that normally Claude grades Claude's work, so you do get a"pinky promise, as all of Claude's training bias tells it to plunge ahead (even if that means "fixing" a test by commenting it out).
With /goal, a different Claude instance evaluates the goal. This is the robust version you described, and such an "adversarial Claude's" approach, with or without/goal, is exactly how you avoid pinky promises (from one Claude).
9dev 21 hours ago [-]
I'm way too afraid of a "paperclip problem"-style target overfitting to use it really; how could I even describe a goal so well it doesn't sacrifice other things that are important to me? Like, if my goal is to make an endpoint faster, will it create an over-engineered mess out of the clearly readable code I have to reach that goal?
mh- 17 hours ago [-]
/goal open a draft PR that makes <endpoint> ~200ms faster while still passing tests, and adhering to nearby style+pattern
It's obviously not perfect, but it's not like you're going to YOLO it into prod without evaluating it on whatever you care about.
If you want to be able to use it like that, you need to find a way to encode (most of) your concerns into something that can be programmatically verified.
9dev 15 hours ago [-]
That kind of emphasises the problem I was referring to; if I pick 200ms, will it stop before doing a proper optimisation because that would make it faster than that? Will it do something stupid to justify pushing from 195ms to >=200ms?
mh- 15 hours ago [-]
It might! In my prompt, I'm supposing you have some priors here about what's possible. Try saying something like "materially faster". But you need to also make sure it knows how to measure that endpoint's performance on its own, otherwise I wouldn't expect this to work well at all.
jascha_eng 23 hours ago [-]
Is this useful? I feel like the problem is usually not that the model isn't capable of achieving what I give it, but the way it does it. Especially if originally I didn't 100% know how I would do it myself the model often takes weird paths through the code base, takes shortcuts that end up in weird feature interactions or pulls in a dependency without weighting if it could've been done without that.
I haven't really found a good way to solve this other than:
1. Produce an initial PR fulfilling all the requirements I knew at the start
2. Chat with the model about any weird snippets I notice and talk through alternatives
3. Simplify anything that I think is overengineered or plain unncessary
Sometimes I restart all over with more precise requirements but then it sometimes makes different mistakes/takes different shortcuts.
In practice the earlier I review the better the end result imo, so /goal seems very unproductive to me?
ubermon 16 hours ago [-]
For some frontier models like Fable 5 it doesn't matter, but for models less trained on long horizon tasks it very useful.
swader999 22 hours ago [-]
It's useful for things where it just needs to get through to completion. Long running tasks. I walk away and expect it to be done without pausing for input.
jascha_eng 22 hours ago [-]
Can you give an example? And more curious about what you do with the resulting code afterwards I imagine its gonna be a big chunk then?
hungryhobbit 21 hours ago [-]
I feel like "do X untill tests are green" is sort of the prototypical /goal case.
madeofpalk 21 hours ago [-]
Adding a new lint rule that a lot of code violates.
jascha_eng 20 hours ago [-]
And that doesn't work with a simple prompt?
mh- 17 hours ago [-]
It's just a way of defining what "done" means, and passing off evaluation of whether "it's done" is true to a separate context and/or model. If false, it prompts the original context accordingly.
ryan_n 24 hours ago [-]
The llm runs in a loop until it meets a condition (the goal).
aetch 21 hours ago [-]
It loops until it thinks it has finished the goal
deepdarkforest 22 hours ago [-]
An agent is an llm running on a loop until it decides it should stop.
/goal is a gimmick where you run a "parent" agent on top that runs the agent on a loop until the it decides to stop, just prompting it "nope, not done yet, continue".
krackers 17 hours ago [-]
isn't this what they called a "ralph loop"?
ai_fry_ur_brain 23 hours ago [-]
[dead]
enraged_camel 20 hours ago [-]
I've used GPT 5.6 Sol Xhigh extensively since its launch, alongside Fable 5.
My impression is that it is about as intelligent as 5.5, but they dialed up the relentlessness meter to eleven. This makes it more likely that it will accomplish the task you give it, which I think is the primary reason it looks competitive in benchmarks. However, it also makes it more likely that it will resort to... unconventional, weird or outright unsafe methods to do it. So I have to watch it like a hawk.
The other day it tried to read env variables from prod using a CLI command. The task it was working on did not necessitate doing that even remotely. I have the SSH keys for that particular CLI tool tied to my 1Password. So when the agent failed (because I never authenticated the SSH key access), it wanted to take over the computer, for which I got an OS prompt. At that point I stopped the agent and asked it why it did that. It said it wanted to dig around 1Password itself to see if it could get the key. I asked it why it needed prod env variables, and it thought for a bit and admitted it actually shouldn't. So as of yesterday I stopped using the "approve for me" mode and now use it only for simpler tweaks and bug fixes.
Fable is not only more intelligent, but also way more insightful. It can sniff out my intent far more effectively, and its "real world" knowledge allows it to act as a seasoned product manager with domain expertise. It can also think outside the box and make suggestions that I would not have thought of. With GPT 5.6 I have to be way more literal.
ttul 19 hours ago [-]
Fable seems to be a larger model. It costs more to run and does not seem superior for _typical_ software engineering work. But for work requiring raw intelligence, perhaps its size is an advantage.
On the DeepSWE 1.1 benchmark (IMHO currently the most relevant and least gamed SWE benchmark), the cost-benefit is clear: 5.6-Sol on xhigh achieves a slightly higher score than Fable 5, but consuming half the tokens and at about 1/3rd the cost.
But, on the Artificial Analysis intelligence index, Fable 5 appears to slightly beat 5.6-Sol, albeit at 3x the cost.
When I am coding, I send tasks to each model to get multiple opinions and it can be hard to predict which model will “win” because the results can be subjective. OP’s task is at least quantifiable, which is great. But many SWE tasks cannot be quantified so easily.
osti 16 hours ago [-]
GPT should be better at these optimization problems given that they won the recent atcoder heuristics competition against top humans. And Anthropic is less focused on these types of things.
spikk 18 hours ago [-]
I would personally love to see the best score over time, not only the final. Then IMO it's also useful to know about effectiveness of /goal
varispeed 23 hours ago [-]
Both are entirely useless on complex problem, because they have a bias in training data they can only partially detect in their own output. The answers are getting worse and worse as you dive deeper into the topic you are working on. I thought I could sharpen one of the documents I worked on using Opus 4.8 and GPT-5.5 together. Fable 5 and GPT-5.6 completely destroyed it. Not only it is not human readable anymore, but also doesn't make sense.
andai 1 days ago [-]
Results seem mostly noise to me. One eval per model, in a large problem space (i.e. a problem which requires many attempts to solve well).
couAUIA 24 hours ago [-]
Yes I agree, but I actually did a lot more runs, with different prompts, different times ect... And each time /goal had a small or insignificant impact
smrtinsert 6 hours ago [-]
This resonated:
Initially these statements were so absurd on their face that I thought it was some cynical ploy to achieve thought leader status, and there are certainly some people doing this – I have had it admitted to me. But the broader reality is so much worse: people who have no background in the technology at all actually believe what they are saying.
codelion 22 hours ago [-]
Just use OpenEvolve for such problems.
matt123456789 18 hours ago [-]
An interesting project. I am curious about its cost per unit improvement over each model's best /goal on the author's benchmark.
akoboldfrying 23 hours ago [-]
If you're curious what the actual optimal Paris cost is, I suggest formulating the problem as an integer linear program and submitting it to Gurobi on NEOS [0]. Gurobi is arguably the strongest commercial ILP solver; big companies pay big dollars to use it to optimise schedules, industrial processes, etc. I'm not sure it could solve this problem to optimality in the 8 hours NEOS provides you, but it might -- KIRO has some similarities to the Vehicle Routing Problem, variants of which are very commercially important. In any case, Gurobi is a monster, and even if you don't get an exact solution, it will give you a lower bound (which may not be tight, but it's nonetheless interesting).
A deepdive on the /goal effect on a problem literally made for this.
techpression 1 days ago [-]
I love that we have this on one hand and me cleaning up catastrophic CSS made by Sol on the other.
Then again, maybe CSS is the ultimate benchmark.
thickclearglass 22 hours ago [-]
Codex has a very bad system prompt which no one at OpenAI is seemingly aware of.
Sol frontend is perfectly serviceable if you use it from something like Pi.
cwmoore 24 hours ago [-]
I do not know the whole picture, but if you are asking for blind one-shot CSS, you might benefit from wiring the model to take screenshots of various end-browsers and discuss them as you iterate.
Offering freelance estimates for CSS design changes before frameworks were around was a problem.
techpression 20 hours ago [-]
I actually have very detailed specifications around the CSS, the problem is that LLM's are trained mostly on codebases with high amounts of complexity so they like to create layer of abstractions that only make your codebase much harder to reason about, machine and man alike.
This is very visible in CSS, where almost every single thing automatically becomes a component wrapping the CSS (be it React or LiveView in my case), which is very often meaningless because the CSS itself is the component.
baq 24 hours ago [-]
CSS is the reason I refuse to do any frontend work except FE infra and I know I’m not alone here, soooo yes I guess?
MaxoumanBoss 22 hours ago [-]
Very interesting, will surely help my future projects
stevefan1999 21 hours ago [-]
...is this not a Travelling Salesman Problem?
hyperpape 21 hours ago [-]
Related, but not quite the same. The solution to this problem can have more than one loop, whereas TSP definitionally has only one. Not quite sure how that changes the difficulty or the tools that you’d use.
stevefan1999 20 hours ago [-]
Because my instinct immediately went into simulated annealing or ant colony.
Or I think my PTSD on advanced algorithm course kicked in
I guess everyone needs a nap after a long day of conversing and writing code. So like us!
/compact is prone to error and I wouldn't recommend it in the middle of work. But when you are switching to a related but not completely new task, it helps. ("Now write the integration tests." vs. "on foo.go line 476 that you just wrote, I think there is a deadlock with bar.go line 123 that you just added". It doesn't really need the context to write the tests, it can get that by reading the code. But for iterating on lines of code it just produced, /compact is going to throw away whatever "thought process" led to that code and it's usually not a great thing to do.)
So for long running tasks I'll do
/brains inspect
that will let you inspect a session and one of the features is the ability to review a compacted message like so:
https://gitsense.com/screenshots/inspect-pi-session-compacte...
https://gitsense.com/screenshots/inspect-pi-session-compacte...
https://gitsense.com/screenshots/inspect-pi-session-compacte...
With /compact in Pi, it creates a message that you can easily review and I am curious as to how '/protect' works.
One of the features that I am working on is to make it easy for agents to retrieve the exact message/event before compaction and I am curious if /protect is a deterministic process or if it is just instructions.
Edit: removed off topic political spam
https://news.ycombinator.com/newsguidelines.html
It's unrelated to my direct comment and it's a generic tangent off the thread further up, which is about technical issues with Claude Code. As the guidelines say:
"On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity."
"Eschew flamebait. Avoid generic tangents."
"Please don't use Hacker News for political or ideological battle. It tramples curiosity."
Nobody wants to see you hijacking technical threads to turn them into political flamewars. It's against the guidelines, anti-intellectual, and profoundly boring.
For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact
Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after.
It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude) doing the compaction.
Only if money is no object. Cache reads are cheap (10% of uncached input costs) but definitely not free, and cached reads dominate session costs at long context lengths. A prompt at 20k context with $0.01 in cached reads would cost $0.40 in cached reads at 800k context, that quickly adds up for long sessions.
You can get far more gas out of even a $20 plan of you’re careful to break things up into relatively small discrete steps, clear context regularly and give the model plenty of information to work with.
My workflow for bigger features is to write out a plan document and then proceed in smaller implementation steps, reviewing as I go. If I find something odd, I ask the agent why and often that leads to discovering a new dimension to the problem, which in turn is an opportunity to adjust the approach.
Personally I find using /rewind judiciously is better than using /compact. The latter essentially gives you no control of what details to discard, but the former at least has coarse-grained control.
My whole point was that by planning in advance you can shard the work into manageable sections with clear beginnings and outputs with acceptance croteria, and never compact, or even use more than a few hundfed thousand tokens in context.
It’s all hierarchical. Looking at an eval feature building right now, it’s 20ish build plans, each with zero to five or so /clear moments.
But maybe that’s the key thing… I don’t iteratively prompt ad hoc software writing. I do iterate on requirements, but if those are solid enough there is no “now write this function, now write that module”.
Even if it doesn't fit in the context window, the model can search through past turns and sanity check if something doesn't seem to be going right, or be prompted to follow an early message, "when starting on a new item, review the first message for how we should approach this"
This whole 1 million context window is a lie after 300k Claude degrades to unusable, and compact doesnt help. I've had Codex tabs open for weeks, I have to regularly restart Claude.
With swival.dev you can use long sessions without ever doing any manual compaction or reset, even with Claude models.
You don't need to go down the rabbit hole of crazy workflows, but to avoid slop:
- Break down the work into tasks
- New context. Create a plan for one task.
- New context. Implement the plan.
- New context. /code-review the implementation
- New context. Fix the review findings.
- Repeat for next task.
I do this with Fable 5, and the quality is consistently quite good. If the context goes over 50%, the quality will become crap and you end up with 4 duplicates of the same thing across the codebase. Letting a current session review its own work is like asking a student to grade their own paper.
Just like with real work, it's valuable to break down big tasks into small tasks that you can knock over in a single session. When a session does start getting too big, you just need to ask the agent to reply with a comprehensive handover report and paste it into a new session.
It's a great vibecoding model, but a crap assistant, and it forgets in few turns.
It's trained on benchmarks.
Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum.
Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.
ultra becomes a mode where your model enhances your prompt them it is execution multiplied by all other agents, which have them going in all sorts of directions.
At least for Fable/Opus (didn’t confirm for Sol yet) Ultra means “write an ephemeral programmatic harness encoding this workflow”. There is actually a TS harness that gets run for the workflow.
If you have a task where the agent/sub-agent pattern works, Ultra just adds indirection.
I think it is possible to get an intuition for an individual model but really you need to eval to be sure. My heuristic though is if you need to treat each work item differently depending on the results, probably agent/subagent. If you want to do the same steps across some queue / tree / DAG of work items, ultra is a better bet. (Or actually write a durable scaffold if you are going to repeatedly run it over > thousands of items.)
"Ultra" is a harness feature, and has nothing to do with the model itself. If OpenAI wanted to, they could offer "Ultra" with any of the GPT models, although 5.6 supposedly been trained with this specific harness feature in mind. "max" is basically the top "work harder and achieve better results" parameter for the reasoning effort for the current models.
I had good success with it initially, until I discovered that OpenAI encrypts the prompts that the main model sends to the sub-agents (even in Codex, in local files you only see cipher text), then I completely dropped any experimentation of it as without introspection, it becomes basically useless for any real usage.
Went through something similar. Fable would just spends minutes thinking, processing, confabulating etc.
I dropped down to Haiku and got an answer in >30 seconds.
After a short discussion about the idea with Claude mostly on how it fits in my workflow and what models / effort I would like for certain tasks it placed a paragraph in my global Claude.md and it has worked wonders. Ultra became a lot better (faster, cheaper for the same output) and the amount of time Fable gets stuck overthinking things are reduced to the places where I think that model makes sense, for the rest it started fanning out a lot to Opus, Sonnet and even Haiku.
Does it spawn sub agents with different models or is it the same single conversation dynamically switching models?
> For all coding tasks use your judgement to decide an appropriate lower power model and run that in a subagent
As said, I tweaked it a little bit to my preference (mostly because it started delegating a bit too much) and had it placed in my global claude.md but that's the gist.
[0] https://simonwillison.net/2026/Jul/3/judgement/
This is very anecdotal evidence but I have to rant about it... I tried 5.6 Terra (high) earlier today to fix a bug with a slow page. It just... removed the part of the page that was slow, and made it a client side request (still slow, but not blocking SSR I guess). I tried with Sonnet 5 and it correctly found the issue where an unhandled case was continuing to retry and failing. I am always telling people how the frontier models are SO much more capable than what they may think but this one thing today had me scratching my head at why it would ever do that. It was the first time I experienced the "great, I removed the failing test case" kind of issue.
I really really like Fable for software engineering cases; add an alert and write a runbook, go pull metrics for this incident, write me a TUI that does blank, etc. It is astoundingly good and I am very picky. It costs A LOT of money though. I am not sure how I get away with my Fable usage.
I have found GPT to be pretty awful at writing Elixir, and dealing with more open ended tasks. Opus is (for the moment) better at writing Elixir, and Fable is much better than either at actually grokking a problem space. I had switched to GPT for about six weeks and eventually stopped using it altogether after it relentlessly gaslight me. Those six weeks using it were essentially completely wasted work.
Right now I’ve moved to a mix of Opus/Fable and Deepseek Pro. Deepseek is far and away the most cost effective, fast, and quite good for 90% of implementation work. Deepseek really only falls apart (in Elixir) when it tries to use runtime features at compile time; which, for the initial lift of a project I have was a bit painful. Fable was able to pretty quickly get things sorted and now things are progressing nicely.
With all that said, I think the various models have unique strengths that are sometime hard to uncover. I also don’t see a world, at least in the near future, where I’ll only use one model. I’m happy Anthropic continues to push the frontier forward. I’m more than happy to trade “efficiency” for quality when I need to.
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took.
I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes.
So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.
Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements.
Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.
I find explicit time bounds are useful for tasks like this, otherwise the LLM will almost certainly return too early.
Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.
I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.
It reminds me of this printing trick “until you are 95% confident”…
Would really be interested to know how this influences the work of the LLM. Also, how would it be if I write “until you certainly understand it”?
Both Codex and Claude Code have it, but they work slightly differently.
Claude Code uses Haiku to read through the transcript and decide if the goal has been completed. If not, Haiku injects a prompt back to the main model to indicate what still needs to be done.
In Codex, instead it's a tool available to the main model, plus some part of the surrounding harness that will re-prompt it if the tool calls haven't yet indicated that the goal is complete.
The issue that they are trying to solve is that sometimes models will stop before they have actually fully completed whatever task they were given; attention isn't perfect, and someitmes they'll complete part of it but not the whole task. Rather than making the user come back and re-prompt to keep going, they add a way to automatically do a bit more nudging to try to get the model to finish the task.
feels kinda odd to use a less capable model to determine if the goal is fully complete. Especially if the user is expecting /goal to thoroughly complete the task. A less capable model would be more likely to misclassify `isComplete?`
There is nothing odd in the basic principle of Economics of using a "good enough" tool, optimal in other respects (e.g. cheap/er).
(You do not hire Nobel Prize winners as receptionists.)
The issue remains, whether the compromise to be adopted actually is "good enough".
This is overselling it. In all implementations of this that I’ve seen, a more correct phrasing is “won’t stop until it pinky-promises it achieved your goal”.
It usually relies on something like a hook that refuses to let the agent stop unless it includes a “promise sentinel”, which is basically the model having to include “I swear I’m done” in the response (usually as a Markdown comment so the user doesn’t see it). If that sentinel isn’t in the response, the hook rejects the completion and hands it back to the agent (sometimes it also restates the goal to keep it focused). It’s useful for sure, but it fails many times for obvious reasons.
More robust implementations use the hook to spawn another agent that verifies the goal is achieved, but that too isn’t prefect and sometimes performs worse.
Of course if you have a problem that can be deterministically verified, you could have the hook run this verification instead, but this is usually not the case (and even when it is, agents will often cheat the verification e.g. by deleting a test file).
With /goal, a different Claude instance evaluates the goal. This is the robust version you described, and such an "adversarial Claude's" approach, with or without/goal, is exactly how you avoid pinky promises (from one Claude).
If you want to be able to use it like that, you need to find a way to encode (most of) your concerns into something that can be programmatically verified.
I haven't really found a good way to solve this other than:
1. Produce an initial PR fulfilling all the requirements I knew at the start
2. Chat with the model about any weird snippets I notice and talk through alternatives
3. Simplify anything that I think is overengineered or plain unncessary
Sometimes I restart all over with more precise requirements but then it sometimes makes different mistakes/takes different shortcuts.
In practice the earlier I review the better the end result imo, so /goal seems very unproductive to me?
/goal is a gimmick where you run a "parent" agent on top that runs the agent on a loop until the it decides to stop, just prompting it "nope, not done yet, continue".
My impression is that it is about as intelligent as 5.5, but they dialed up the relentlessness meter to eleven. This makes it more likely that it will accomplish the task you give it, which I think is the primary reason it looks competitive in benchmarks. However, it also makes it more likely that it will resort to... unconventional, weird or outright unsafe methods to do it. So I have to watch it like a hawk.
The other day it tried to read env variables from prod using a CLI command. The task it was working on did not necessitate doing that even remotely. I have the SSH keys for that particular CLI tool tied to my 1Password. So when the agent failed (because I never authenticated the SSH key access), it wanted to take over the computer, for which I got an OS prompt. At that point I stopped the agent and asked it why it did that. It said it wanted to dig around 1Password itself to see if it could get the key. I asked it why it needed prod env variables, and it thought for a bit and admitted it actually shouldn't. So as of yesterday I stopped using the "approve for me" mode and now use it only for simpler tweaks and bug fixes.
Fable is not only more intelligent, but also way more insightful. It can sniff out my intent far more effectively, and its "real world" knowledge allows it to act as a seasoned product manager with domain expertise. It can also think outside the box and make suggestions that I would not have thought of. With GPT 5.6 I have to be way more literal.
On the DeepSWE 1.1 benchmark (IMHO currently the most relevant and least gamed SWE benchmark), the cost-benefit is clear: 5.6-Sol on xhigh achieves a slightly higher score than Fable 5, but consuming half the tokens and at about 1/3rd the cost.
But, on the Artificial Analysis intelligence index, Fable 5 appears to slightly beat 5.6-Sol, albeit at 3x the cost.
When I am coding, I send tasks to each model to get multiple opinions and it can be hard to predict which model will “win” because the results can be subjective. OP’s task is at least quantifiable, which is great. But many SWE tasks cannot be quantified so easily.
[0] https://neos-server.org/neos/
Sol frontend is perfectly serviceable if you use it from something like Pi.
Offering freelance estimates for CSS design changes before frameworks were around was a problem.
Or I think my PTSD on advanced algorithm course kicked in
Edit: looks like the closet reduction should be https://en.wikipedia.org/wiki/Ring_star_problem with bounded circuit length
https://chatgpt.com/c/6a5bbe6a-a760-83ea-931a-4e2bbe028486
Thoughts?