I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration.
> In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a single working day, achieving comparable accuracy nearly 7× faster.
This is actually insane.
Hopefully the release ultrafast of Terra and Luna too.
I'm finding Luna suprisingly adequate for my work. I slept on it due to the benchmarks, but it's very fast and even on low reasoning I'm finding it more than adequate for "menial" work. (The speed is crucial for "interactive" work -- if a model is fast enough it goes from "async" to "real time", subjectively, which is a huge difference.)
In fact, I'd say it's overqualified for the kind of work I'm doing, because it spends >half the time verifying trivial changes (and the verification isn't as helpful as you'd expect, even with bigger models).
Maybe I can prompt it to be less aggressive about that (the new GPT models do it even without prompting).
Anyway, Ultrafast Luna would be amazing, though I strongly doubt they can offer Cerebras at anything approaching the current prices. Now we wait for Moore's Law? :)
Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out. It would be more meaningful to know how much time was required for a single complete answer to a difficult HLE question.
The animation in the article shows GPT-5.6 Sol Ultrafast solving a question at 3 secs vs Fable's 27 secs. The "11 hour vs 78 hours" animation implied that they were solved in sequence and not in parallel
Amadahl’s law dictates it’s approximately always better (as in, more efficient computationally) to have one super fast thing than many slower things doing the same job in parallel.
The task the majority of people here care about (and which makes up a huge part of AI lab revenue these days) is collaborative coding with a single human which is not embarrassingly parallel.
Feels like the 90's again where single threaded speed is improving fast. ASICs and wafer scale rather than node shrinks, but end result to me the consumer feels the same.
ChatJimmy is a much smaller model and, AFAIK, has no reasoning capability. Absolutely insane raw speed, like a supercar, while Sol is more like a freight truck.
> At such output speed, I wouldn’t expect reasoning.
As the sibling comment to yours mentioned, if they had a reasoning model “hardware-ified” onto a custom chip (as is their plan for IIRC this or next year, a new ASIC), it’d output fast decode speeds for the regular output as well as reasoning sections. Both would be ≈equally fast.
People underestimate the importance of speed on quality of thought, because people underestimate just how much quality is a result of simple iteration.
When an LLM thinks, it typically just makes one pass. It outputs tokens from top to bottom, beginning to end, and then it's done. But when people think, especially strong thinkers, we typically iterate and revise our thoughts on the fly. We do numerous passes. We stop and restart, we reconsider, we review, we reevaluate. Sometimes we do this so quickly and automatically that we don't even realize we're doing it. I think a lot of what separates a highly intelligent or effective person from others has less to do with the quality of their first pass and more to do with just how many additional passes they're able to do in the same amount of time, and of course what kind of criteria they're habituated to consider during their review passes.
Introspecting about this is difficult, but experimenting with LLMs is easy. First, simply ask an LLM to do something complex. For example, to come up with a new business idea, or to plan the next month of your life, etc. After it finishes, tell it:
"Review what you just wrote, according to some appropriate list of evaluation criteria that you come up with first. And then, based on the results, iterate and generate a better response if warranted."
It's insane how much better the next answer will usually to be. Often it'll catch and erase tons of hallucinations, logical errors, and inefficiencies. And you can simply copy-paste this again and again until you begin to hit diminishing returns. Or, in a harness like Claude Code, for example, I might shortcut this whole process by saying, "Use sub-agents to iteratively review and iterate on your work until convergence."
The reason why most people don't prompt LLMs to do this (besides simply not thinking of it) is that it takes time.
But what if it didn't?
What if the LLM's response came back in milliseconds rather than minutes? Then there would be almost no reason NOT to do this. In fact, one could almost imagine it baked into the assistant/harness -- a massive step change in practical quality, enabled by nothing more than speed.
This is foundationally similar to a lesson I've found from years of pre-LLM software development: builds that turn around in 500ms instead of 5 minutes fundamentally change the way you can work as a software engineer. I think a lot of the same applies to working with LLMs. I'm not sure, though, what the path from where we are today to some future state of high speed token abundance actually looks like...I think there's plenty of chance that we see the bubble pop in the near term over token costs and complexities of today's infrastructure, then some totally different landscape of LLM use in 5-10 years that looks quite unlike what we have today, similar to how waiting 30 minutes for an MP3 of a single song to download on a 28k modem in 1999 seems quaint today.
Going back between two different company's AI tools when facing a tricky architecture question often surfaces holes in an approach I'd been building.
Similarly, if I ever get a bit too vibey and don't carefully review code changes myself, the blast radius is generally significantly resolved by a carefully tuned "did you consider x, y, and z" skill after a first draft partnered with a "deploy an adversarial review agent for the worktree".
Just putting more details in, and giving it ways to check itself brings massive improvements. I often ask model to set up test for the problem before actually trying to solve it and it improves it a lot, both in how much babysitting is required (if it can test it itself quickly it goes faster), and the fact the context now contains more detailed description of the problem that came up when making tests.
Turns out TDD is far better for robots than humans, who knew
Unless I have read over it, besides the animation in the intelligence vs speed graph which only mentions internal data and not whether they truly reran the AA suite, there is no actually solid statement on the important aspect of performance.
Neither the Cerebras or OpenAI post [0] outright state that this performs exactly the same as regular 5.6 Sol. I feel if this was 1:1 just Sol but much faster, they'd (rightfully) scream that off the rooftops. A line such as "this is the same performance, just faster, with no downsides" would go a long way in clarity and communication. Along with no pricing information, I'll hold out on further information.
No quality compromise/degradation is something I have had this industry, including especially OpenAI, claim multiple times in the past and I have more than once been able to verify that it was in fact not the case. Examples being gpt-3.5-turbo vs text-davinci-003, GPT-4-Turbo and all the other post training checkpoints they had under one name (which was a major bug bear for me back then witnessing degradations with no naming change, industry got better in transparent checkpoint naming since), Opus 4.6 Fast Mode (which just was faster by skipping much of the required work), etc.
Same for massive performance differences in the way providers like Cerebras, Groq, etc. have deployed models including K2.6 on Cereberas specifically. Massive deltas in tool call and overall quality despite there being far more clarity in open weight vs proprietary model deployment.
The AA suite graph with that animation is the only time in either post that absolute parity is being asserted and I'd be amazed if that was the case, but am doubtful why their phrasing is so cagey.
Why not assert full parity in writing? It "performs the same (within run-to-run variance) across all evals that Sol has been tested with" is very different to "no quality compromise/degradation", the later allowing for a lot more wiggle room and interpretation in what evals you use to assess that, what quality truly means, etc., the former meaning identical in all situations.
Could also be a language barrier here in fairness, maybe this phrasing is more iron clad than I give them credit, but especially with OpenAI, I have seen enough checkpoints asserted as unchanged in "quality" to where I am skeptical. Ironically, I never saw that with Anthropic (which has gotten far more heat for degradation accusations) while a model was deployed with one exception in mid-late April this year. Pure speculation, but believe it wasn't noticed much before "agentic coding" became more popular, because chat output is far more subjective without a rating framework vs code passing which can be an objective metric with more potential for frustration.
The problem is the compromise is probably based on their tests for key things, but for the rest of us, it means changing how we talk to the model or the flow we use to adjust for quirks, but then its fully about the same.
There's probably some quantization and other processing needed to get Sol to fit on a Celebras chip, so it's not an exact 1:1 copy of the Sol model running on Nvidia GPUs, because it's not running on GPUs doing abstract math calculations anymore, so they can run benchmarks at them and say it's similar enough, but no one believes in benchmarks in this new field of performance that's more computer astrology than computer science.
No one claimed gpt-3.5-turbo doesn't have any degradation over davinci-003. In fact it was quite obvious that gpt-3.5 had way less knowledge but more post trained to be helpful.
That quite strong "no one" surprised me so I checked and looking through a few blog posts from back then, they did advertise gpt-3.5-turbo as a straight up improvement and, once text-davinci-003 was to be deprecated, the instruct tuned variant as the drop in replacement [0]. If anything, they did not just promise similar performance but actually an improvement ("our best model") when compared to text-davinci-003:
> It’s also our best model for many non-chat use cases—we’ve seen early testers migrate from text-davinci-003 to gpt-3.5-turbo with only a small amount of adjustment needed to their prompts.
That's why I still remember this so well, they claimed one model to be their best and a straight up drop-in during deprecation when in my (back then even more amateurish then today) testing this was plainly not the case. A model cannot be "best" if it's measurably worse in many situations, then what was still available at the time.
They're nearly certainly going to use it internally to speed up research that is serially bottlenecked. I would bet this is why they're interested in the Cerebras partnership more than everything else
The stake in the side of cerebras has always been that the economics are pretty poor.
Who knows if they will subsidizes it to mitigate sticker shock, but it's a safe assumption that it will be scarily expensive. However if you are in a "cost is no obstacle, speed is god" position, it will likely be pure magic.
Can anyone explain why Cerberus needs to be _fast_ instead of _cheap_?
I don't think I understand why they aren't leveraging the increased speed to do batching to serve more customers at a "normal" tok/s.
Is the limitation, even on cerberus, still that the cache can only serve so many concurrent sessions over time? Is there no scaling advantage? I genuinely do not understand how any of this works.
There are many industries and situations that will pay absurdly high multiples on API rates for low latency access. Anything remotely related to finance, deal flow, or trading, including the development and checking of Excel financial models. Live analytics that can be delivered mid-meeting to C-suite executives, or ultra-luxe dealmakers who need answers while on a live call. Intelligence and defense, where milliseconds matter.
With the tool calls that can be done, you're not pricing this against an executive assistant or pocket analyst - you're pricing this against the ability to have an entire Bourne Identity style analysis room at your disposal. The limited inventory will go to the people for whom money is no object.
You'd need hundreds of GB alone for the KV cache of each user. For something like LLama 3 405B you need ~67GB at ~130k tokens.
A single CS-3 has 44GB on-chip sram.
So, afaik, Cerebras are optimizing for ultra-low latency batch=1 inference.
They're cache limited, almost certainly, so more slower sessions doesn't solve the problem - you still have to load and unload the whole cache hierarchy at some level and that's a network bandwidth and memory bandwidth problem between the external systems and the waferscale chip.
Also worth looking into how they do cooling for it, because that's kind of absurd and awesome as well.
> I don't think I understand why they aren't leveraging the increased speed to do batching to serve more customers at a "normal" tok/s.
There's some technical hypotheses about it that other people are offering.
But also from a business perspective, it totally makes sense not to go any sort of batching play. It's really valuable and very clear to consumers to make your pitch entirely about lower latency rather than higher bandwidth.
There are so many scenarios that are latency-constrained that will be difficult or even impossible for someone even with fleets of high-bandwidth compute to compete with you on.
Very easy pitch to sell a customer who asks what differentiates you from other companies: you pay us a premium for lower latency than anyone else.
To use a car analogy, they built. Ferrari and technically you could put a tow hitch on it and go slow and carry a bunch of cargo instead of 1 passenger and go real fast, that's just not what they've built. There's one really fast core to send tokens at and get tokens back out at insane speeds.
Cerebras is a large plate sized chip. It has 50GB of SRAM, and few hundred K simple cores that can access that SRAM really fast. I don't know semiconductors well, but I understand that the same manufacturing technique that makes this huge chip possible, on the flip-side limits inter-chip communcation bandwidth. In cerebras, it is 150 GB/s (compared to nvlink's 2TB/s or groq's similar).
One way large models are served on a bunch of cerebras chips is by essentially distributing layers' weights across chips. Few layers's weights per chip - as many as the KV cache + activations + weights will allow. You use pipelining to hide the latency of the inter-chip 150 GB/s link.
On GPUs, you amortize the cost of loading weights from HBM to SRAM across multiple users - thereby making it cheaper _per_ user. But here, there is no such amortization. The weights are already there. It is the activations that stream through.
You _could_ do batching/continuous batching, but that would just service more users at lower token/s each without any amortization of fixed cost, due to fixed cost (loading weights) being non-existent.
> Compared with output speeds reported by Artificial Analysis GPT-5.6 Sol on Ultrafast mode runs 11x faster than Fable 5, and 5x faster than Opus 4.8 on Fast mode.
Awesome work. I'm personally very excited for faster models/inference.
I think speed is underrated to some degree in the current conversation. For a while, I was using Cursor's Composer quite a lot, even over frontier models, just because of how darn fast it was.
What do you need speed for? That's a genuine question, I feel like the limiting factor already is my creativity, attention span and budget. And I'm not even yet optimizing cost by batching things like review to slow local models over night, or schedule tasks to take full advantage of my subscriptions.
IME waiting for an agent to work through a problem is a detriment to attention span; your mind drifts to other things while you wait. Maybe you can steer several agents in a round robin instead, but then there's a cognitive tax from context switching. Faster models mean fewer gaps in focus.
I just finished waiting almost four hours for Fable to write 700 lines of code, based on my three paragraph prompt. Some speed on these harder tasks would definitely be welcome!
Humans have human attention spans and there's a lot of study going back decades for UX design because of this. < 100 Ms is instantaneous, 100-300 ms is noticable but still responsive. At 1 second, flow.gets interrupted, 2-5 seconds, you're clearly waiting, 5-10 attention wanders and 10+ seconds, you've lost them. The 0.1 / 1 / 10 second rule comes from Jakob Nielsen's HCI work. Perceived latency matters almost as much as actual latency, which is why chat interfaces drip out/stream words instead of just dumping out the answer at the end. At 750/tok/s, for Sol grade inference, it can spend 3 seconds on thinking tokens before outputting something to the user for a better answer while still feeling usable.
This does look pretty incredible, but don't forget that incredible token thoroughput can only necessarily solve certain bottlenecks. If your e2e tests take an hour, they'll still take an hour after Ultracode. If the agent runs a 10 minute typecheck after a change, that will still take 10 minutes. grep over a massive codebase is still just as slow, etc. I say this not to take away from this accomplishment but just to ensure everyone here keeps a clear head about what it means - 14x faster tokens does not mean it completes every task 14x faster.
I suspect Humanity's Last Exam is without tool-calls, making it kind of the perfect benchmark to highlight how fast Ultrafast is, but not really the same as the everyday work you or I do.
Rught now on large scale codebases the bottleneck is both claude/codex inference, as well as time it takes to run tens of thousands of tests. We put those workloads on dedicated epyc 9005 build machines - but it still takes minutes per run. Those who can afford the fast tokens will be in the market for faster CPU that money can buy today.
i know people are joking about the sol ultrafast prices (its unlikely to be accessible for average joes) but this shows scaling wafer cores works for inference boost
which makes me very excited, sol ultrafast will be as slow as it will get if that makes sense. at these token speeds , we will see a much deeper economic impact.
Right now the bottle neck is not the CPU, so people aren‘t spending big $ on them. But with this ultra fast mode, CPU becomes a bigger part of the bottleneck and thus Intel and AMD can charge more $$$.
In light of recent news, it is hard not to think about the 4.5-day hack on Hugging Face's systems happening ~10 times faster and be slightly concerned.
ASIC makes it sound like it's a single chip, but in reality serving trillion-param models on Cerebras requires a full cluster (as in multiple racks, MW of power).
> top developers outside ai labs will be spending 50k USD+ on inference
I think it is more like top companies, not top developers, and the problem with developers in top companies was and is - absolute majority of them are not actually directly working on things that increase revenue, so companies can spend a ton of money and see barely if any changes in the product and the bottom line, so companies, at least legacy ones will be reluctant to sponsor that long term.
not sure how "top developers" are defined here, but there is huge diminishing return curve starts kicking in after $200/month price point for typical eng work.
I agree with the diminishing returns on spend, but worth noting that when on an Enterprise seat and paying API rates, I'd say that you can easily spend above 200/mo before seeing the curve begin to flatten
Obviously there are a ton of ways to spend money / tokens and people have different levels of experience that will put this ceiling at very different levels for different people.
I mean I don't think $50k is the ceiling, unless you're talking about actual cash out. Claude code subscriptions right now can easily clear you $25-35k a year in nominal value for $2400 out of pocket cost.
Given sufficient budget and scope, I could certainly productively burn a half million dollars in tokens a year or more. I think that's where we're headed anyway, buying a 2nd or 5th claude max subscription feels slightly excessive for personal usage, but at a corporate level...
And a moderately heavy user dev can easily spend a few $K a month, so yeah. Not impossible, but a high cost, and the diminishing returns definitely kick in
I'd just like to point out that the largest model Cerebras has ever served is Kimi K2.6 which is 1T parameters, so that either means that theyve had a breakthrough on the hardware engineering side of things, or GPT-5.6 Sol is likely a lot smaller than people think.
If it truly is only ~1-2T parameters, then this kinda kills 2 narratives for me.
1. all the handwringing about open source catching up via Kimi K3 (3T params) is complete nonsense. All that matters imo for determining which labs are leading is intelligence per parameter. Anyone with a enough compute can train a giant model, but being able to squeeze capabilities into smaller models gives you a massive inference and training edge.
2. Inference margins are clearly insane, and this explains why OpenAI was able to lower the price of Luna by 80%. Id guess that thing is probably 120b params based on the TPS they are serving it at.
no, because closed sourced model pricing has no relationship to its size. Thats what im saying. the inference margins are crazy, but people think the fonrtiner models must be 10T params or something because theyre expensive
I haven't wrapped my head around what level of reasoning this involves. Is it equivalent to max?
I didn't like Sol initially but it is growing on me the more I use it. Its personality is a bit flat and I caught it taking shortcuts a few times. But once I learned how to interact with it, I'm genuinely warming up to it. I find that it writes code that has fewer bugs even than Fable (although, to be fair I reach for Fable when the task is less well defined).
If this has similar performance to Sol at max reasoning level, this would be a compelling reason to shift even more of my work (maybe the majority) to this model.
This is really cool. Someone here commented about similarity between this and hardware advancements for AV encode/decode.
I think it's only a matter of time before miniaturization can have a thumbnail sized user-replaceable accessory that contains the LLM built onto the hardware. I admit I don't know how any of that works, but would be amazing to experience. Fully local, fully offline, ultra fast local inference better than any personal computing product.
In 5-10 years nice smartphones will be able to run ChatGPT (~gpt3-4) class models. A memory rich laptop (highend mac/framework) can run GPT-OSS:120b or full Gemma4 at very interactive speeds.
High end phones can already run the smaller models at enough speed to be probably useful, especially for background/overnight photo tagging and curation and things like that.
I don’t know if this is that useful for coding. In some autonomous world, where no one check the code and the agent can just spend 10X more time checking its work and leading to better results, yes maybe it is useful.
But if humans need to check its work, then 10X speed doesn’t really matter I guess.
I can imagine that all the tool calls that agents regularly make during regular software dev workloads would become the new bottleneck and so tasks overall won't be completed anywhere near 10x faster.
However, I wouldn't diss the "ultraspeed" options untill I try them. Having agent thinking become near instant could change the way I (or you) use agents.
Working with agents right now is like working on huge legacy C++ codebase (feel free to insert your nightmares). You change 10 lines of code and then wait 20 minutes before it compiles and runs tests (or write prompt and wait 20 minues for code). External distractions and huge waiting times breaks your flow. Working long time in the flow is what makes mythical 10x programmer. IMHO 10x programmers are not superior in skill but in the ability to stay focused. Making agents faster is more important for people who care about the code than for vibecoders.
>But if humans need to check its work, then 10X speed doesn’t really matter I guess.
A human could have an agent run 10x more correction checks. If even after that they still need to check manually for issues, then they really need to work on their specification skills.
This is something I'm ready to pay for. Not more per token, but I will be happy to burn through 20x Pro subscription as fast as I consume my Plus weekly limit now, with 10x more tokens per unit of time. I've learned how to deal with and steer Sol medium quite efficiently, but at the same time I realize it's so slow for the small tasks it can do well, and still so unreliable for open-ended tasks.
The example they give about debugging a service outage is good. For large companies, a very large burst expenditure on fast tokens to debug the outage as quickly as possible would be a good investment.
Fast mode is already 1.5 times faster and 2x more expensive in the Codex subscription plan. If this thing is 14 times faster, then I can imagine running out of my quota in one session.
Compilation time will be a genuine bottleneck for slop coding if this becomes the standard generation rate over the next few years. Go, Zig or even C99 with TCC for dev builds, any language that can get you systems-level performance (or close to it) in a dev environment where you can iterate in ms rather than minutes is going to be immensely more appealing than generating a potential prototype in 10 seconds and waiting 15 minutes for it to compile.
A great future for us all to look forward to. Human readability gone, any changes must go through a providers API so cost+restrictions are a constant and we'll need far more tokens for even simple operations. Call me old fashioned, still like to understand the code I merge, especially since I've seen Fable and Sol make sometimes utterly egregious mistakes in simple, well scoped requests that erode any trust I can muster...
The latter now takes up 10x as many tokens (= 10x the cost/time, + context penalties), and is now architecture-specific, impossible to apply non-brittle program-wide optimizations to, etc. There is absolutely zero reason to ever have the LLM act as a compiler no matter how fast it is. Even if you believe LLMs will reach a state where they can actually generate good code at this level, you would be better off having them generate the compiler they would use.
For what it's worth, this theoretical LLM-based compiler would probably generate LLVM or some other intermediate representation rather than platform-specific machine code.
I mean you're right, but what about a different tokenizer and some guardrails... trained on compiled code... I just learned that gpt-5.6 will talk to you in base64 without tool calls so maybe it's possible.
Still, what'd be the advantage? There is a lot, a massive delta, a chasm still to improve before there is even a theoretical potential re. performance. GPT-5.6-Sol still writes JS in inefficient and roundabout ways that leave a lot on the table and even if we got to a point were models provide the highest performance output possible, there are hundreds of languages far more performant before machine code should be considered. Rust, C, Assembly if need be (at which point performance vs binary is, unless I am mistaken, pretty much moot). All machine code would gain us is lock in and no auditability.
It's good enough at decoding hex from some packet dumps. And I was doing that even with 5.5. And it was good at decompiling some code (with tools) and searching for offsets of buffers and commands. Found viable exploit that allowed me to rescue broken update system in devices I was maintaining for my company (it was broken by chatgpt forgetting -v in hexdump, heh).
I think it's pretty obvious that, in that world, the AIs will simply be tasked with making the compilers faster. It's already happening with their own stack, after all.
Speed is independent of quality. LLMs are currently completely incapable of producing a good compiler. I'm not convinced that's going to change (and if it does change, we're all unemployed).
They don't have produce a good compiler. They just have to take the ones that already exist go much faster. That's likely much easier, and you can validate using existing test suites.
> In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a single working day, achieving comparable accuracy nearly 7× faster.
This is actually insane.
Hopefully the release ultrafast of Terra and Luna too.
In fact, I'd say it's overqualified for the kind of work I'm doing, because it spends >half the time verifying trivial changes (and the verification isn't as helpful as you'd expect, even with bigger models).
Maybe I can prompt it to be less aggressive about that (the new GPT models do it even without prompting).
Anyway, Ultrafast Luna would be amazing, though I strongly doubt they can offer Cerebras at anything approaching the current prices. Now we wait for Moore's Law? :)
So is calculating the total time required to answer all of the questions.
But why is that important if they're measuring time?
Don’t blink.
(Chatjimmy has 14,200 TPS.)
700 TPS with reasoning is awesome and it speeds things up.
Cerebras as public traded company is worth keeping an eye what they produce.
As the sibling comment to yours mentioned, if they had a reasoning model “hardware-ified” onto a custom chip (as is their plan for IIRC this or next year, a new ASIC), it’d output fast decode speeds for the regular output as well as reasoning sections. Both would be ≈equally fast.
I recognize that the former is the multiplication symbol, but I don't think it should be used that way.
[0] https://news.ycombinator.com/item?id=35490837
When an LLM thinks, it typically just makes one pass. It outputs tokens from top to bottom, beginning to end, and then it's done. But when people think, especially strong thinkers, we typically iterate and revise our thoughts on the fly. We do numerous passes. We stop and restart, we reconsider, we review, we reevaluate. Sometimes we do this so quickly and automatically that we don't even realize we're doing it. I think a lot of what separates a highly intelligent or effective person from others has less to do with the quality of their first pass and more to do with just how many additional passes they're able to do in the same amount of time, and of course what kind of criteria they're habituated to consider during their review passes.
Introspecting about this is difficult, but experimenting with LLMs is easy. First, simply ask an LLM to do something complex. For example, to come up with a new business idea, or to plan the next month of your life, etc. After it finishes, tell it:
"Review what you just wrote, according to some appropriate list of evaluation criteria that you come up with first. And then, based on the results, iterate and generate a better response if warranted."
It's insane how much better the next answer will usually to be. Often it'll catch and erase tons of hallucinations, logical errors, and inefficiencies. And you can simply copy-paste this again and again until you begin to hit diminishing returns. Or, in a harness like Claude Code, for example, I might shortcut this whole process by saying, "Use sub-agents to iteratively review and iterate on your work until convergence."
The reason why most people don't prompt LLMs to do this (besides simply not thinking of it) is that it takes time.
But what if it didn't?
What if the LLM's response came back in milliseconds rather than minutes? Then there would be almost no reason NOT to do this. In fact, one could almost imagine it baked into the assistant/harness -- a massive step change in practical quality, enabled by nothing more than speed.
Similarly, if I ever get a bit too vibey and don't carefully review code changes myself, the blast radius is generally significantly resolved by a carefully tuned "did you consider x, y, and z" skill after a first draft partnered with a "deploy an adversarial review agent for the worktree".
Turns out TDD is far better for robots than humans, who knew
Neither the Cerebras or OpenAI post [0] outright state that this performs exactly the same as regular 5.6 Sol. I feel if this was 1:1 just Sol but much faster, they'd (rightfully) scream that off the rooftops. A line such as "this is the same performance, just faster, with no downsides" would go a long way in clarity and communication. Along with no pricing information, I'll hold out on further information.
[0] https://openai.com/index/previewing-ultrafast/
Same for massive performance differences in the way providers like Cerebras, Groq, etc. have deployed models including K2.6 on Cereberas specifically. Massive deltas in tool call and overall quality despite there being far more clarity in open weight vs proprietary model deployment.
The AA suite graph with that animation is the only time in either post that absolute parity is being asserted and I'd be amazed if that was the case, but am doubtful why their phrasing is so cagey.
Why not assert full parity in writing? It "performs the same (within run-to-run variance) across all evals that Sol has been tested with" is very different to "no quality compromise/degradation", the later allowing for a lot more wiggle room and interpretation in what evals you use to assess that, what quality truly means, etc., the former meaning identical in all situations.
Could also be a language barrier here in fairness, maybe this phrasing is more iron clad than I give them credit, but especially with OpenAI, I have seen enough checkpoints asserted as unchanged in "quality" to where I am skeptical. Ironically, I never saw that with Anthropic (which has gotten far more heat for degradation accusations) while a model was deployed with one exception in mid-late April this year. Pure speculation, but believe it wasn't noticed much before "agentic coding" became more popular, because chat output is far more subjective without a rating framework vs code passing which can be an objective metric with more potential for frustration.
> It’s also our best model for many non-chat use cases—we’ve seen early testers migrate from text-davinci-003 to gpt-3.5-turbo with only a small amount of adjustment needed to their prompts.
That's why I still remember this so well, they claimed one model to be their best and a straight up drop-in during deprecation when in my (back then even more amateurish then today) testing this was plainly not the case. A model cannot be "best" if it's measurably worse in many situations, then what was still available at the time.
[0] https://openai.com/index/gpt-4-api-general-availability/
[1] https://openai.com/index/introducing-chatgpt-and-whisper-api...
There is no pricing info, which could mean it's "if you have to ask..." territory or they are simply gauging interest before deciding
Who knows if they will subsidizes it to mitigate sticker shock, but it's a safe assumption that it will be scarily expensive. However if you are in a "cost is no obstacle, speed is god" position, it will likely be pure magic.
I don't think I understand why they aren't leveraging the increased speed to do batching to serve more customers at a "normal" tok/s.
Is the limitation, even on cerberus, still that the cache can only serve so many concurrent sessions over time? Is there no scaling advantage? I genuinely do not understand how any of this works.
With the tool calls that can be done, you're not pricing this against an executive assistant or pocket analyst - you're pricing this against the ability to have an entire Bourne Identity style analysis room at your disposal. The limited inventory will go to the people for whom money is no object.
So, afaik, Cerebras are optimizing for ultra-low latency batch=1 inference.
https://newsletter.semianalysis.com/p/cerebras-faster-tokens... goes quite in-depth.
[1]: https://vllm.ai/blog/2026-04-24-deepseek-v4
Also worth looking into how they do cooling for it, because that's kind of absurd and awesome as well.
There's some technical hypotheses about it that other people are offering.
But also from a business perspective, it totally makes sense not to go any sort of batching play. It's really valuable and very clear to consumers to make your pitch entirely about lower latency rather than higher bandwidth.
There are so many scenarios that are latency-constrained that will be difficult or even impossible for someone even with fleets of high-bandwidth compute to compete with you on.
Very easy pitch to sell a customer who asks what differentiates you from other companies: you pay us a premium for lower latency than anyone else.
Batching works because of severe memory bottleneck, but Cerebras whole thing is serving models out of "L1 cache" (?).
One way large models are served on a bunch of cerebras chips is by essentially distributing layers' weights across chips. Few layers's weights per chip - as many as the KV cache + activations + weights will allow. You use pipelining to hide the latency of the inter-chip 150 GB/s link.
On GPUs, you amortize the cost of loading weights from HBM to SRAM across multiple users - thereby making it cheaper _per_ user. But here, there is no such amortization. The weights are already there. It is the activations that stream through.
You _could_ do batching/continuous batching, but that would just service more users at lower token/s each without any amortization of fixed cost, due to fixed cost (loading weights) being non-existent.
Awesome work. I'm personally very excited for faster models/inference.
I think speed is underrated to some degree in the current conversation. For a while, I was using Cursor's Composer quite a lot, even over frontier models, just because of how darn fast it was.
IME waiting for an agent to work through a problem is a detriment to attention span; your mind drifts to other things while you wait. Maybe you can steer several agents in a round robin instead, but then there's a cognitive tax from context switching. Faster models mean fewer gaps in focus.
It also spent almost 800k tokens on these lines…
I suspect Humanity's Last Exam is without tool-calls, making it kind of the perfect benchmark to highlight how fast Ultrafast is, but not really the same as the everyday work you or I do.
Rught now on large scale codebases the bottleneck is both claude/codex inference, as well as time it takes to run tens of thousands of tests. We put those workloads on dedicated epyc 9005 build machines - but it still takes minutes per run. Those who can afford the fast tokens will be in the market for faster CPU that money can buy today.
i know people are joking about the sol ultrafast prices (its unlikely to be accessible for average joes) but this shows scaling wafer cores works for inference boost
which makes me very excited, sol ultrafast will be as slow as it will get if that makes sense. at these token speeds , we will see a much deeper economic impact.
It is a bit outdated (scores ± 40% lower), but smart enough for a lot of coding tasks, and can cost under 1/10th of Sol.
https://mimo.mi.com/models/en-US/mimo-v2.5-pro-ultraspeed
https://taalas.com/products/
> delivering 17k tokens per second per user on Llama 3.1 8B model.
Obviously this is a much smaller model, but I really can't wait for ASICs to take over the LLM space.
Imagine running a model like Sol/Fable (even half the size with 60-70% of it's intelligence) on your own ASIC hardware.
Some interesting twitter analysis here:
https://x.com/bleysg/status/2073937651150029084
My prediction is that, this time next year, top developers outside ai labs will be spending 50k USD+ on inference.
Within labs, I've heard spend is already far beyond this per developer.
I think it is more like top companies, not top developers, and the problem with developers in top companies was and is - absolute majority of them are not actually directly working on things that increase revenue, so companies can spend a ton of money and see barely if any changes in the product and the bottom line, so companies, at least legacy ones will be reluctant to sponsor that long term.
If someone subsidize maybe, but if the companies need to pay no way, unless there is hard evidence of the return.
Obviously there are a ton of ways to spend money / tokens and people have different levels of experience that will put this ceiling at very different levels for different people.
Given sufficient budget and scope, I could certainly productively burn a half million dollars in tokens a year or more. I think that's where we're headed anyway, buying a 2nd or 5th claude max subscription feels slightly excessive for personal usage, but at a corporate level...
And a moderately heavy user dev can easily spend a few $K a month, so yeah. Not impossible, but a high cost, and the diminishing returns definitely kick in
If it truly is only ~1-2T parameters, then this kinda kills 2 narratives for me.
1. all the handwringing about open source catching up via Kimi K3 (3T params) is complete nonsense. All that matters imo for determining which labs are leading is intelligence per parameter. Anyone with a enough compute can train a giant model, but being able to squeeze capabilities into smaller models gives you a massive inference and training edge.
2. Inference margins are clearly insane, and this explains why OpenAI was able to lower the price of Luna by 80%. Id guess that thing is probably 120b params based on the TPS they are serving it at.
I didn't like Sol initially but it is growing on me the more I use it. Its personality is a bit flat and I caught it taking shortcuts a few times. But once I learned how to interact with it, I'm genuinely warming up to it. I find that it writes code that has fewer bugs even than Fable (although, to be fair I reach for Fable when the task is less well defined).
If this has similar performance to Sol at max reasoning level, this would be a compelling reason to shift even more of my work (maybe the majority) to this model.
I think it's only a matter of time before miniaturization can have a thumbnail sized user-replaceable accessory that contains the LLM built onto the hardware. I admit I don't know how any of that works, but would be amazing to experience. Fully local, fully offline, ultra fast local inference better than any personal computing product.
High end phones can already run the smaller models at enough speed to be probably useful, especially for background/overnight photo tagging and curation and things like that.
But if humans need to check its work, then 10X speed doesn’t really matter I guess.
However, I wouldn't diss the "ultraspeed" options untill I try them. Having agent thinking become near instant could change the way I (or you) use agents.
A human could have an agent run 10x more correction checks. If even after that they still need to check manually for issues, then they really need to work on their specification skills.
Curious, what are some of the use cases?
Compilation time will be a genuine bottleneck for slop coding if this becomes the standard generation rate over the next few years. Go, Zig or even C99 with TCC for dev builds, any language that can get you systems-level performance (or close to it) in a dev environment where you can iterate in ms rather than minutes is going to be immensely more appealing than generating a potential prototype in 10 seconds and waiting 15 minutes for it to compile.
Why? Because you are defining the implementation based on its observable behaviour rather than as a rule set to be followed.