Moonshot AI’s Kimi team has published research showing that a hybrid linear attention architecture can match or outperform standard full attention across short-context, long-context, and reinforcement learning tasks, while reducing memory usage by up to 75% and achieving up to 6x faster decoding throughput for 1 million-token contexts. The paper, published on arXiv, introduces Kimi Delta Attention (KDA), a new linear attention module that the team says works as a drop-in replacement for full attention in production models.
What is attention? In an AI model, attention is the mechanism that decides which parts of the input to focus on when generating each word of output. Standard “full attention” compares every word to every other word, which works well but requires memory and compute that grow quadratically with the input length. Linear attention replaces this with a compressed representation that grows linearly, making it far cheaper to run, but historically at the cost of quality. Kimi Linear is the first architecture to close that quality gap convincingly.
🔍 THE BOTTOM LINE
For years, the assumption has been that linear attention trades quality for efficiency. Kimi Linear challenges that tradeoff directly, matching full attention on benchmarks while using a fraction of the memory. The architecture already powers Kimi K3, the 2.8 trillion parameter open-weight model released last week, which uses 69 KDA layers alongside 24 gated multi-head latent attention layers.
How Kimi Delta Attention Works
KDA extends a previous architecture called Gated DeltaNet with a finer-grained gating mechanism. In plain terms, the model maintains a compressed memory state that updates as it processes each token, rather than storing and attending to every previous token individually. The key innovation is a more expressive gating system that makes better use of the limited memory available in the recurrent state.
The team trained a Kimi Linear model with 3 billion activated parameters and 48 billion total parameters, using a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). With an identical training recipe, this model outperformed full MLA with what the paper describes as “a sizeable margin” across all evaluated tasks.
As Sebastian Raschka notes in his architecture analysis, the evolution from standard attention to KDA follows a clear lineage: attention reformulated as recurrent neural networks, then Mamba2 added decay, then DeltaNet modified the decay, then Gated DeltaNet added a gate matrix on the output, and Kimi Linear Attention is Gated DeltaNet with a more expressive gate. Each step assembles similar building blocks in slightly different configurations.
The 6x Throughput Claim
The practical implications are significant. Kimi Linear reduces KV cache usage by up to 75%, which directly translates to lower memory requirements for serving models. For a 1 million-token context, the architecture achieves up to 6x decoding throughput compared to full attention.
This matters because memory, not raw compute, is often the binding constraint when serving large language models. A model that can process the same context with 75% less cache memory can either serve more users simultaneously, handle longer contexts, or run on smaller hardware. The team has open-sourced the KDA kernel and vLLM implementations, along with pretrained and instruction-tuned checkpoints.
From 48B to 2.8 Trillion
The Kimi Linear paper describes a 48 billion parameter model, but the architecture has already been scaled to production scale. As Raschka confirms, Kimi K3, the 2.8 trillion parameter open-weight model released last week, is essentially a scaled-up production version of Kimi Linear, with 69 KDA layers and 24 gated MLA layers.
K3 also adds several components not present in the original Kimi Linear paper: LatentMoE (a more efficient mixture-of-experts design), attention residuals (connecting residual paths across layers with attention-weighted contributions), and NoPE (No Positional Embeddings) replacing RoPE everywhere. Raschka describes K3 as the first frontier-level model to use NoPE exclusively.
The fact that K3 was already built on this architecture means the paper is not a theoretical proposal but a retrospective description of what is already running. The Moonshot AI IPO plans and the distillation dispute with Anthropic have drawn attention to the company’s strategy, but the architecture itself is the foundation.
What the Community Found
The Hacker News discussion, which reached 209 points and 79 comments, surfaced several useful observations. One commenter noted that the paper’s contribution does not explain the entire success of Kimi K3, pointing out that the training dataset remains immensely important regardless of architecture. Another observed that KDA layers in K3 were inherited from the Kimi Linear research, which was completed months before K3’s pretraining would have started.
The discussion also touched on the broader trend across frontier models. As we have covered in our reporting on the open weights coalition, Nemotron 3, DeepSeek V4, and now Kimi K3 are all moving toward inference-efficiency-tweaked components. MoE becomes LatentMoE, regular attention becomes multi-head latent attention and KDA, and the overall design goal is delivering more intelligence per dollar of compute.
What This Means for the Field
The result that matters most is not the 6x throughput or the 75% memory reduction. It is the claim that linear attention can match full attention on quality at all. If this holds up under independent verification, it changes the economic calculus of running large language models. The dominant cost in AI inference is memory bandwidth, and architectures that reduce memory pressure without sacrificing quality directly lower the cost of serving models.
Moonshot AI has been seeking more Nvidia Blackwell chips for training Kimi K4. If K4 builds further on the KDA architecture, the gap between the compute required to train a frontier model and the compute required to serve it will continue to widen.
❓ FAQ
What is linear attention? A variant of the attention mechanism in AI models that uses a compressed, recurrent memory state instead of comparing every token to every other token. This reduces memory and compute from quadratic to linear growth with input length, but has historically produced lower quality results.
Does this make existing AI models obsolete? No. Kimi Linear is a drop-in replacement for full attention layers in new models, not a retrofit for existing ones. Models already trained with full attention cannot simply swap in KDA layers without retraining.
Is the code available? Yes. Moonshot AI has open-sourced the KDA kernel, vLLM implementations, and both pretrained and instruction-tuned model checkpoints for the 48B Kimi Linear model.
How does this connect to Kimi K3? Kimi K3, the 2.8 trillion parameter open-weight model released last week, uses 69 KDA layers alongside 24 gated MLA layers. K3 is essentially a scaled-up production version of the Kimi Linear architecture described in this paper.
What is NoPE? No Positional Embeddings. Most transformer models use positional encodings (like RoPE) to tell the model the order of tokens in the input. Kimi K3 removes these entirely and uses NoPE everywhere, which Raschka describes as the first frontier-level model to do so.
🔍 THE BOTTOM LINE
The assumption that linear attention trades quality for efficiency has been the received wisdom in AI architecture for years. Kimi Linear challenges it directly, matching full attention on benchmarks while using 75% less memory and delivering 6x faster decoding. The architecture already powers a 2.8 trillion parameter model that is currently the largest open-weight release in the world. If independent verification confirms the claims, the cost of serving frontier AI models just dropped significantly.
📰 Sources
- arXiv — “Kimi Linear: An Expressive, Efficient Attention Architecture” (2510.26692)
- Sebastian Raschka Blog — “Kimi K3 Architecture Notes”
- Doubleword Blog — “You Could Have Come Up With Kimi Delta Attention”
- Hacker News discussion (49082022, 209 points, 79 comments)
