Kimi Linear: Redefining Expressive and Efficient Attention

Summary: The Kimi Team has introduced Kimi Linear, a groundbreaking hybrid linear attention framework. It marks a significant milestone as the first architecture to consistently surpass full attention performance across diverse benchmarks, including short-context, long-context, and RL scaling environments.

---

🛠️ Core Technical Innovations

1. Kimi Delta Attention (KDA)

At the heart of this architecture is the KDA module. This is an advanced linear attention mechanism that builds upon the foundations of Gated DeltaNet.

• The Enhancement: KDA implements a finer-grained gating mechanism.
• The Goal: To optimize the utilization of limited finite-state RNN memory, allowing the model to be more "expressive" than previous linear iterations.

2. Optimized Chunkwise Algorithm

To ensure the model isn't just theoretically sound but also hardware-efficient, the team developed a bespoke chunkwise algorithm.

• DPLR Integration: It utilizes a specialized version of Diagonal-Plus-Low-Rank (DPLR) transition matrices.
• Efficiency Gain: By refining the DPLR formulation, they reduced the computational overhead while maintaining strict adherence to the classical $\text{delta rule}$.

$\text{Delta Rule Concept: } \Delta W_t = \eta (v_t - W_t k_t) k_t^T$

---

🏗️ Model Architecture & Scale

Kimi Linear is not a monolithic structure but a layerwise hybrid. It blends the efficiency of KDA with the power of Multi-Head Latent Attention (MLA).

Model Specifications

Architectural Flow

---

📈 Performance Benchmarks

The team conducted a "fair comparison" using an identical training recipe. The results indicate that full attention is the gold standard Kimi Linear is the new benchmark.

Key Improvements over full MLA:

• KV Cache: Reduced by up to 75%.
• Throughput: Up to $6\times$ increase in decoding speed for contexts reaching 1M tokens.
• Versatility: Superior performance in tasks requiring both extensive input and long-form output.

---

📦 Open Source & Availability

To foster community growth and academic transparency, the Kimi Team has released several assets:

• KDA Kernel (Optimized implementation)
• vLLM Integration (For high-throughput serving)
• Pre-trained Checkpoints
• Instruction-tuned Checkpoints

Example Configuration Snippet

# Conceptual Kimi Linear Configuration
model_config = {
    "architecture": "KimiLinear",
    "activated_params": "3B",
    "total_params": "48B",
    "attention_type": "hybrid_KDA_MLA",
    "dplr_optimized": True,
    "context_window": "1M"
}

---

📄 Metadata & Reference

Paper Details:

• arXiv ID: 2510.26692
• Submission Date: October 30, 2025 (v1) $\rightarrow$ Revised November 1, 2025 (v2).
• Authors: Yu Zhang, Zongyu Lin, Xingcheng Yao, et al. (Kimi Team).
• Categories: cs.CL (Computation and Language), cs.LG (Machine Learning).

Distributed under the license indicated above.