Demystifying Kimi Delta Attention

---

Author: Jamie Dborin, Founder & Member of Technical Staff at Doubleword
Date: July 27, 2026
Topic: Inference API / Linear Attention

📐 A Note on Mathematical Notation

To make the geometric shapes of this derivation intuitive, this article utilizes bra-ket notation (inspired by quantum mechanics).

Notation Reference Table

Key Convention: By default, vectors point right, but keys are written pointing left when they are integrated into the linear-attention state.

Assumptions for this derivation:

• We are analyzing a single causal attention head.
• All vectors are real-valued.
• DeltaNet keys are normalized.
• The state acts as a map from the key space to the value space.

---

The Complexity of Modern Linear Attention

At first glance, contemporary linear attention variants seem impenetrable. Their equations are dense, and the underlying intent is often obscured. To illustrate this, consider the state update logic for Kimi Delta Attention (KDA):

$$\begin{aligned} \widetilde S_t &= S_{t-1}\operatorname{Diag}(\alpha_t) \\ \lvert\widehat v_t\rangle &= \widetilde S_t\lvert k_t\rangle \\ \lvert e_t\rangle &= \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right) \\ S_t &= \widetilde S_t+\lvert e_t\rangle\langle k_t\rvert \\ \lvert o_t\rangle &= S_t\left(d_k^{-1/2}\lvert q_t\rangle\right) \end{aligned}$$

This complexity is a result of "feature creep" within the DeltaNet family. Over several years, these models have evolved, and each iteration has added layers of sophistication, making the latest versions feel inaccessible to outsiders.

However, these equations aren't magic; they are the result of making simple assertions about how a hidden state should behave. We can derive KDA by following this evolutionary path:

Our Roadmap

• Start with Quadratic Attention
• Strip away the Softmax
• Transition to DeltaNet
• Implement Gating
• Arrive at KDA
• Explore Triton implementations (Recurrent & Chunkwise)

---

1. Starting with Quadratic Attention

In standard causal softmax attention, for a query at time $t$, the output is calculated as:

$$\begin{aligned} a_{ti} &= \frac{ \exp\!\left(s\langle k_i\rvert q_t\rangle\right) }{ \sum_{j\leq t} \exp\!\left(s\langle k_j\rvert q_t\rangle\right) }, \qquad s=d_k^{-1/2},\\ \lvert o_t\rangle &= \sum_{i\leq t}a_{ti}\lvert v_i\rangle. \end{aligned}$$

How it works: It computes the similarity between a specific key and query, then uses the softmax function to normalize these scores into a probability distribution.

The Bottleneck:

• For a sequence of length $T$, you must compute $T^2$ key-query pairs.
• While we can cache keys and values during autoregressive inference, the cache grows linearly.
• Every new query must still scan the entire history because the denominator is dependent on both the current query and all previous keys.

1.1 Removing the Softmax

To simplify, let's absorb the scaling factor $s$ into the query and remove the softmax. This gives us a "bare-bones" version of attention:

$\lvert o_t\rangle = \sum_{i\leq t} \langle k_i\rvert q_t\rangle \lvert v_i\rangle$

Because the inner product $\langle k_i\rvert q_t\rangle$ is just a scalar, we can shift it to the right side of the value vector:

$$\begin{aligned} \lvert o_t\rangle &= \sum_{i\leq t} \lvert v_i\rangle \langle k_i\rvert q_t\rangle\\ &= \left( \sum_{i\leq t} \lvert v_i\rangle\langle k_i\rvert \right) \lvert q_t\rangle. \end{aligned}$$

Now, we can encapsulate everything from the past into a single matrix $S_t$ of fixed size $V \times K$:

$\boxed{ S_t = \sum_{i\leq t} \lvert v_i\rangle\langle k_i\rvert }$

Attention is now transformed into a recurrent write followed by a read:

$\boxed{ \begin{aligned} S_t = S_{t-1} + \lvert v_t\rangle\langle k_t\rvert,\\ \lvert o_t\rangle = S_t\lvert q_t\rangle. \end{aligned} }$

The Core Trick: The identity (|v><k>)|q> = <k|q>|v> allows us to store summed outer products in a fixed-size state $S_t$. This reduces the complexity from quadratic to linear relative to sequence length. We only scan the tokens once, updating the $d_v \times d_k$ state at each step.

The trade-off? We lose the selectivity and normalization provided by the softmax. While advanced linear attention uses feature maps to fix this, this basic form highlights the memory issue that DeltaNet aims to solve.

1.2 Addition is not Assignment

Consider what happens if we write a pair $\lvert v_t \rangle \langle k_t \rvert$ and then immediately query the state using that same key:

$S_t \lvert k_t \rangle = ( S_{t-1} + \lvert v_t \rangle \langle k_t \rvert ) \lvert k_t \rangle$