Ornith AI released Ornith-1.5 on Tuesday, a family of open-source models that does something most frontier labs only talk about: it generates its own training data. The model proposes new tasks, constructs task-specific scaffolds, and produces solution rollouts for reinforcement learning — creating a self-improvement loop that does not depend on human-curated datasets.
The largest variant, Ornith-1.5-397B (a mixture-of-experts model), scores 86.1 on Terminal-Bench 2.1 and 56.0 on DeepSWE, matching Claude Opus 4.8 on both benchmarks. It outperforms GLM-5.2 (82.7 and 46.2) and DeepSeek-V4-Flash-0731 (82.7 and 54.4) on the same tests. The smallest variant, a 9B dense model, can run on a phone and still beats Gemma 4-31B and Qwen 3.6-35B on several coding and agentic tasks.
What is self-improving AI? Most language models are trained on fixed datasets — human-written text, curated question-answer pairs, human-graded reasoning traces. Self-improving AI flips this: the model itself generates the tasks it trains on, evaluates its own solutions, and uses reinforcement learning to improve from the results. The appeal is obvious — no human bottleneck, no dataset ceiling, continuous improvement. The risk is that a model training on its own output can reinforce its own blind spots.
🔍 THE BOTTOM LINE
The interesting thing about Ornith-1.5 is not the benchmark scores. It is the training method. If a model can reliably generate useful tasks, build working evaluation harnesses, and improve from its own solutions, the bottleneck shifts from data to compute. The question is whether the self-improvement loop actually produces capability gains that compound — or whether it plateaus on the model’s existing knowledge.
How the Loop Works
Each training cycle runs in three stages. Given a codebase or environment and high-level instructions, the system first proposes a task that goes beyond what the model has already solved — progressively harder problems that expose capability gaps. For each task, the model generates a scaffold: the instructions, tools, and decomposition strategy for approaching the problem. Then it produces a solution rollout.
Reward from the rollout propagates across all three stages. The system learns not only to produce better solutions, but to generate more useful training tasks and construct more effective scaffolds. All three stages are optimised with GRPO — Group Relative Policy Optimization — the same reinforcement learning method that powers DeepSeek’s reasoning models.
The task reward has three components: validity (does the task and scaffold form a working learning environment?), frontier difficulty (is the task near the model’s current capability edge but still solvable enough to produce useful gradients?), and novelty (is it different from previously generated tasks?). The difficulty target is set at a 20 per cent success rate, which favours tasks that are hard but not impossible.
This is the part worth pausing on. The difficulty metric is measured using the model’s own rollouts — if the model gets better at a task, the reward for that task naturally decreases, pushing the generator toward harder problems. The curriculum evolves with the model’s capability. That is a genuine shift from static training distributions.
The Benchmark Picture
The 397B flagship matches Claude Opus 4.8 on Terminal-Bench 2.1 (86.1 vs 85.0) and DeepSWE (56.0 vs 59.0). On SWE-bench Verified it scores 86, compared to Opus 4.8’s 85.8. On HLE with tools — a reasoning benchmark using Claude 4.6 Opus as judge — it reaches 56.1, behind Opus 4.8’s 57.9 but ahead of DeepSeek-V4-Flash (50.8) and GLM-5.2 (54.7).
The 35B mixture-of-experts model, which activates only 3B parameters per token, is where the efficiency story gets interesting. It scores 67.8 on Terminal-Bench 2.1, compared to Gemma 4-31B’s 42.1 and Qwen 3.6-35B’s 52.5. On SWE-bench Verified, 79.0 against 52.0 and 73.4 respectively. A model that activates 3B parameters per token beating a 31B dense model by 25 points on agentic coding is not a marginal gap.
The 9B model, with a quantised mobile version, scores 47.0 on Terminal-Bench 2.1 and 70.6 on SWE-bench Verified. It runs on iPhone and Android. Whether that is practically useful or a benchmark flex is a different question, but the numbers are real.
What Could Go Wrong
The self-improvement loop has an obvious failure mode: if the model generates tasks that are easy to solve but look hard to its own evaluation, it trains on garbage. Ornith’s paper addresses this with the validity gate, which checks that scaffolds run correctly, that high-confidence solutions pass, and that incorrect solutions fail. But the gate is itself evaluated by the model’s own systems.
The novelty term exists to prevent the model from generating small variations of the same task. But novelty is explicitly secondary to validity and difficulty — its role is to reduce redundancy, not to reward unusual tasks. Whether that balance holds at scale is an open question.
There is also the question of what “self-improvement” means here. Ornith-1.5 extends Ornith-1.0’s self-scaffolding framework, and the benchmarks show meaningful gains: Terminal-Bench 2.1 went from 77.5 to 86.1 at the 397B scale, and DeepSWE went from 8 to 56. Those are real improvements. But the model is still trained on a fixed foundation of pre-existing codebases and environments. The self-improvement loop generates new tasks within those environments — it does not create new environments.
The Open-Source Angle
All three model sizes are released as open-source models on Hugging Face. The 397B, 35B, and 9B variants are all available. This puts Ornith in the same open-release category as DeepSeek, Qwen, and GLM — frontier-scale models with open weights.
The 9B mobile model is the one that could matter most for adoption. A model that runs on a phone, scores 70.6 on SWE-bench Verified, and was trained through a self-improvement loop is a proof of concept that the method scales down. If self-improvement works at 9B, it works at 3B, and then the question is whether it works well enough to be useful for on-device coding assistants.
❓ FAQ
Is Ornith-1.5 actually self-improving, or is that marketing? The training method is real — the model generates tasks, scaffolds, and solutions, and all three are optimised with reinforcement learning. But the self-improvement happens within a fixed set of environments and codebases. The model does not create new domains to learn from. “Self-improving within a curriculum” is more accurate than “self-improving without limits.”
How does it compare to DeepSeek or GLM? On coding benchmarks, Ornith-1.5-397B outperforms GLM-5.2 and DeepSeek-V4-Flash-0731 on Terminal-Bench 2.1 and DeepSWE. On reasoning benchmarks (HLE, GPQA Diamond), it is competitive but does not consistently lead. The differentiator is the training method, not the benchmark scores.
Can I run the 9B model on my phone? Yes — Ornith provides a quantised “Mobile” version of the 9B model designed for iPhone and Android deployment. Benchmark results suggest it is functional, though real-world performance will depend on the device and workload.
Does this matter for New Zealand? Open-source models that can be run locally are directly relevant to NZ’s sovereign AI capability. A 9B model that runs on a phone and was trained through self-improvement is the kind of thing that could be fine-tuned for specific NZ use cases without depending on US cloud providers. The open-weight release also means NZ researchers can study the training method directly.