The largest public experiment in autonomous AI research to date produced a striking negative result. Across 153 runs spanning 18 frontier models, not one invented a fundamentally new method for training neural networks. The best performer — Anthropic’s Fable 5 — closed 81.7% of the gap between a tuned baseline and the human record, but did so by rediscovering known optimizer tricks, not by breaking new ground.
Prime Intellect, a decentralised compute startup, released the results on 14 August 2026 after running the experiment on 8×H200 GPU nodes, with each run lasting up to eight days. The interactive leaderboard shows every trajectory, every experiment, and every validated result.
What the speedrun tests
The task is deceptively simple. Train a 124-million-parameter GPT model to a target validation loss of 3.28 in as few optimizer steps as possible. The model architecture, dataset, batch size, and sequence length are frozen. The agent can edit the optimizer, its hyperparameters, the learning rate schedule, and the weight initialisation. Everything else is locked.
The baseline — a carefully tuned starting configuration — passes at 3,290 steps. The human record, sitting in an open pull request on the modded-nanoGPT repository, claims 2,600 steps. That 690-step gap is the prize. Every model gets the same starting code and a single instruction: beat the current best.
This is not a toy benchmark. It is the same task that appears in Anthropic’s internal automated R&D evaluation and OpenAI’s GPT-5.6 Sol system card, where it serves as a measure of whether models can do useful research work. The difference is scale: Prime Intellect gave each model its own GPU node and up to eight days of unsupervised time.
The leaderboard
The results separate cleanly into tiers. Four models broke past 45% of the gap: Fable 5 at 81.7%, Opus 5 at 53.6%, and two configurations of Kimi K3 at 52.2% and 45.8%. Then a cliff drops to 39.4% for Opus 4.8, and a long tail of models that never crossed a quarter of the gap.
Fable 5’s winning run took 8.7 days of agent time and 800 million tokens. It ran 811 experiments and made roughly 3,000 tool calls. Opus 5, in second place, achieved 53.6% in just 2.9 days with 183 million tokens — less than a quarter of Fable 5’s compute budget. On a tokens-per-step-gained basis, Opus 5 is the most efficient model on the board.
The bottom of the table tells its own story. DeepSeek V4 Pro managed 12.3% in 1.1 days. Grok 4.6 closed 10.1% in 0.6 days. GPT-5.5 — an older OpenAI model — barely moved the needle at 8.1%. GLM 5.3, the newest model from Zhipu, failed to produce a validated result at all.
Nobody invented anything
The most important finding is buried in a single sentence from Prime Intellect’s report: “None of the runs produced a fundamentally new method; the winning ingredients are all similar to existing ones in the literature.”
The improvements that worked are all known optimizer techniques — better preconditioning, caps and floors on update magnitudes, keeping the learning rate hot for longer, weight averaging near the end of training. These are the kind of tweaks a graduate student with a laptop and a few afternoons might try. The models found them by hill-climbing through parameter space, not by having a research insight.
This matters because the discourse around autonomous AI research has been dominated by claims of recursive self-improvement — the idea that models will soon improve themselves in a feedback loop of escalating capability. The speedrun suggests a more modest reality. Frontier models can do useful optimization work, given enough compute and time. They cannot, or at least have not yet, demonstrated the kind of creative leaps that would distinguish research from search.
Notably, the agents had no internet access — a deliberate change from earlier experiments where models over-anchored on existing pull requests. Rediscovering the literature from parameters alone is a real capability. It just isn’t the capability that “recursive self-improvement” usually implies.
This connects to our earlier analysis of Anthropic’s own claims about recursive improvement, and to the broader question of whether AI agents can handle open-ended research — a Princeton study from earlier this year found they cannot.
The cost column reorders everything
Flip the leaderboard to tokens-per-step-gained and the ranking falls apart. Grok 4.5 bought its 24.6% improvement at 0.27 million tokens per step. GPT-5.6 Sol paid 11.7 million tokens per step for its 35.9% — a 43× spread that says nothing about model quality and everything about compute economics.
The model that looks best on both axes is Opus 5. Second place on raw performance, achieved in under three days, at 0.49 million tokens per step. Fable 5 wins outright but spends nearly three times that rate and takes three times as long. For anyone making decisions about which model to deploy for autonomous research work, the cost-adjusted ranking is the one that matters.
Why this benchmark matters
The NanoGPT speedrun is useful precisely because it is boring. It has a clear metric (steps to loss), a frozen architecture, and a tight feedback loop. There is no room for models to hide behind rhetorical flourishes or cherry-picked examples. A step is a step. Either you found a faster way to converge, or you didn’t.
For New Zealand’s growing AI research community, this kind of benchmark is relevant in a practical way. The Revved AI conference in Auckland earlier this month highlighted the gap between frontier labs and everyone else. The speedrun quantifies that gap with numbers, not vibes. The top four models are in a different population from the rest, and the rest includes some very expensive frontier models.
❓ FAQ
What is nanoGPT? A minimal implementation of OpenAI’s original GPT architecture, small enough to train quickly on a single GPU. It has become a standard benchmark for testing optimizer improvements because the task is well-defined and the compute requirements are modest.
Why no internet access for the models? Prime Intellect deliberately removed internet access after earlier experiments showed models copying existing solutions from the modded-nanoGPT repository rather than finding their own. Without internet, any improvement the model discovers comes from its own reasoning, not from reading someone else’s work.
Did any model beat the human record? No. The human record sits at 2,600 steps. Fable 5’s best run reached 2,726 steps — close, but not there. The remaining 18.3% of the gap is where a human already is.
What does “serial era” mean in the leaderboard? Some runs were conducted under a stricter rulebook that limited parallelism. These are marked separately because the constraint makes direct comparison unfair.
