Most models forget. Feed them a large codebase, a long transcript, or a year of tickets, and the important detail three hundred pages back quietly drops out of reach. For the work we care about, automating software engineering and research, that limit is the whole problem.
Today we are sharing an update on our long-context models, the evaluation we built to keep ourselves honest, and the compute that makes it possible.
Why context matters
A model that can hold a hundred million tokens in context does not need to be told what is relevant. It can read the entire repository, every prior design doc, and the full history of a bug before it writes a single line. Retrieval systems approximate this by guessing which fragments matter. Long context removes the guess.
The goal is not a bigger prompt. It is a model that reasons over an entire body of work at once, the way a senior engineer holds a system in their head.
The hard part was never storing the tokens. It was making attention over them cheap enough to run, and meaningful enough that the model actually uses what is far away rather than what is merely recent.
What we built
Our latest model sustains reasoning across context windows two orders of magnitude larger than a typical frontier model, without the sharp quality cliff that usually appears past the first few thousand tokens.
Reasoning, not retrieval
The distinction we hold ourselves to is whether the model can perform multi-hop reasoning across the window, not just locate a single fact. Finding a needle is easy. Following a chain of five references scattered across millions of tokens is the real test.
How we evaluate it
Standard long-context tests are too forgiving. A single distinctive fact in a wall of unrelated text is trivially findable, so a model can score well while still being useless on real work.
- Multi-hop retrieval across widely separated passages.
- Reasoning that requires combining many small facts, not one big one.
- Resistance to distractor text that looks superficially relevant.
The infrastructure
None of this runs without hardware. Through our partnership with Google Cloud we operate thousands of GB200s, with custom networking and kernels built to keep attention over very long sequences fed and fast.
The new funding lets us scale that cluster further and keep the training runs long enough to matter. We would rather solve a short list of fundamental problems well than a long list shallowly.
What comes next
Long context is a means, not the end. The end is a model that can take on real engineering and research autonomously, holding an entire problem in view. If that sounds like the work you want to do, we would love to hear from you.