The AI industry’s biggest promise right now is that AI will soon improve itself. A new study from Princeton says not so fast.
Researchers led by Peter Kirgis and Sayash Kapoor at Princeton gave Anthropic’s Claude Opus 4.8 six days, $3,000 in API credits, a GPU budget, virtual computers, and open web access. The task: produce research papers worthy of a top-tier AI conference. The agents reviewed literature, ran hundreds of experiments, compiled results. Both papers were rejected by the original authors who graded them.
The agents could do the engineering. They couldn’t do the research.
🔍 THE BOTTOM LINE
The gap between what AI companies promise and what AI agents can deliver is widening on the open-ended front. AI systems can write code, run benchmarks, and automate narrow tasks with checkable answers. But the creative judgement required to choose a hypothesis, decide what evidence would settle a question, and know when to abandon a failing approach — that remains human territory. If recursive self-improvement depends on that kind of thinking, the timelines running through venture capital pitch decks are ahead of the evidence.
What the Researchers Actually Tested
Most existing evaluations of AI research agents measure narrow tasks: solve this engineering problem, post-train this model against this benchmark. The Princeton team wanted something harder. They proposed a method called “shadow evaluation” — giving the AI a research question derived from a high-quality unpublished paper, so the answer couldn’t be memorised from training data.
The two questions came from papers submitted to NeurIPS 2026. One asked whether a language model’s behavioural “personas” can be controlled by editing its weights. The other asked how to design a detector that flags when a predictive model working from spreadsheet data has become unreliable.
The agents were given everything they needed. Six days. Real money. Real compute. The open web. They produced two papers. Both were rejected.
Where the Agents Fell Apart
The human graders found the agents capable of all the engineering required. They could review literature, run experiments, compile results. The failure was in the research itself.
The agents ran bizarre experiments — in some cases testing their hypotheses on tiny synthetic datasets that couldn’t possibly answer the question. They committed to unpromising approaches too quickly. They developed novel and ambitious hypotheses resembling what the original authors started with, then rejected them on the basis of very limited data. They could make small pivots but couldn’t fundamentally rethink an approach or start over from scratch.
When they got feedback, they didn’t revise their methodology. They narrowed their claims and added caveats. They couldn’t manage their own resources — tokens, compute, time — or follow instructions about how long to spend on different phases.
Notably, the agents didn’t cheat. They didn’t engage in what researchers call reward hacking — hiding or misrepresenting experiments. Subagents occasionally hallucinated results, but the orchestrator agent caught those. That’s the one encouraging finding.
Why This Matters for the Self-Improvement Timeline
The study, published on arXiv, lands at an awkward moment for the AI industry. In June, Anthropic published a blog post titled “When AI Builds Itself”, charting progress toward models that accelerate their own development. In July, OpenAI advertised that GPT-5.6 Sol had helped post-train a smaller model, saving researchers weeks of work. The message from both labs: the loop is closing.
Anthropic cofounder Jack Clark, writing in his Import AI newsletter, struck a different note. He described “a certain absence of valuable, intuitive creativity in today’s AI systems” and called their lack of creativity a “bearish signal on short recursive self-improvement timelines.” That’s not a pundit talking. That’s a cofounder of the company building the most capable agents saying the hype is running ahead of the evidence.
As we’ve covered, Anthropic already acknowledged that Claude writes 80 per cent of its code and called for a pause on the most aggressive self-improvement trajectories. But the Princeton study suggests the engineering wins — writing code, optimising chips, generating synthetic training data — may be the easy part. The hard part is knowing what to build next.
The Trillion-Dollar Question
The Princeton finding raises a question that splits the AI community. Can AI systems grind their way to recursive self-improvement by getting better at narrow tasks alone — making models train faster, boosting benchmark scores? Or does transformative progress require the creative leaps that the Princeton agents couldn’t muster?
Kapoor points to history. “If we look back to the biggest advances in the field, the invention of transformers or the invention of big new architectures that allowed us to make a lot of AI progress — all of those did require creative leaps.”
Others hold what Kapoor calls the hypothesis that everything needed for transformative AI is already there. Just make training faster and benchmarks higher.
“That’s frankly the trillion-dollar question right now,” he says.
The Princeton team is now running the same experiment with Mythos, Anthropic’s most advanced model, which launched in April and was subsequently restricted by the Trump administration to approved organisations only. Anthropic did not respond to a request for comment.
What This Means for New Zealand
NZ’s AI policy conversation, including the government’s approach to sovereign AI options, has been shaped by the assumption that recursive self-improvement is a near-term possibility. If the Princeton study is right that open-ended research judgement is the bottleneck — not engineering capacity — then the timeline for disruptive AI-driven automation stretches out, and the policy window for getting frameworks right widens. That’s not a reason to delay. It’s a reason to use the time.
❓ FAQ
What is recursive self-improvement? The idea that an AI system could improve its own capabilities — writing better code, generating better training data, designing better architectures — without human intervention, creating a feedback loop that accelerates progress beyond what human engineers could achieve alone.
Why were both papers rejected? The agents could execute engineering tasks but failed at the research itself: choosing promising hypotheses, designing experiments that could actually answer the question, knowing when to abandon a failing approach. The papers were “nowhere close to the quality of a top AI conference,” according to the researchers.
Did the AI agents cheat? No. The agents did not engage in reward hacking — they didn’t hide or misrepresent experiments. Subagents occasionally hallucinated results, but the main orchestrator agent caught those. That’s a genuine positive signal for safety.
What happens next? The Princeton team is repeating the experiment with Mythos, Anthropic’s most advanced model. If Mythos also fails at open-ended research, the bearish case on short RSI timelines strengthens considerably.
