Multi-Head Latent Attention: How DeepSeek Compressed the Cache Further

Day 26 reduced cache size by sharing key/value heads. Could we keep a different view for each query head without storing every expanded key and value?

Multi-Head Latent Attention, or MLA, answers that question with a smaller learned representation. A latent vector is a set of internal numbers that a model learns to use; its entries do not need names such as "noun" or "location." In MLA, one compact vector carries content used to form both keys and values.

The important idea is not simply "compress now, expand the whole cache later." Algebra lets an optimized implementation move some of that expansion into other projections. We will work through that movement with small numbers.

The content cache stays narrow. A separate positional key preserves the position-dependent part of the score.

Start With What Ordinary Attention Saves

At each layer, the hidden state for a token is the vector entering that layer's attention projections. Ordinary multi-head attention makes a key and value for each head and saves them for later queries.

Suppose two heads each have a four-number key and a four-number value. The content cache stores $2\times (4+4)=16$ numbers per token per layer. The query is not included because future tokens do not need old queries.

A later query must read the old keys to make weights, then read the old values to make an output. Wider saved representations therefore cost both space and repeated memory reads.

Our teaching baseline stores sixteen content numbers per token. These are toy dimensions, not a DeepSeek model configuration.

GQA saves fewer KV heads. MLA instead changes the form of the saved content. Those are different restrictions on what the model learns, not interchangeable runtime switches for any existing checkpoint.

Learn a Smaller Shared Representation

A learned down-projection maps the hidden state $h_t$ at token position $t$ into a smaller vector:

$D$ is a matrix learned during training. The letter $c$ names the compressed content, not a probability. For our toy example, let $c_t=[1,2{]}^T$, a column with two numbers.

Each head has transformations that can turn this shared content into its key and value. For one head, write:

$U_K$ and $U_V$ are up-projections: matrices mapping the smaller vector into a wider one. The superscript $C$ marks the content part of a key. We will add the position part later.

The cached content is shared, while different learned matrices can produce different head views.

This is not lossless compression of an arbitrary pre-existing KV cache. Two numbers cannot encode every possible sixteen-number vector without restrictions. The model is trained with this lower-dimensional structure so its attention learns to use the available representation. Whether that preserves useful model quality is an empirical question.

The Direct Form Explains the Meaning

Before optimizing, consider the direct calculation. For one head, choose the teaching matrix:

Multiplying $U_K$ by $c_t=[1,2{]}^T$ gives the four-number content key $[1,2,3,-1{]}^T$. We have not created four independent stored facts. All four entries are combinations of the same two latent numbers.

A query $q=[1,0,1,0{]}^T$ scores this key with a dot product:

That direct form tells us what the attention score means. But expanding the keys for every past token before every decode step would add work and temporary storage.

The expanded key is useful for understanding the math. An optimized decoder does not need to build it for every cached token.

Move the Key Projection Into the Query

Matrix multiplication lets us regroup the score without changing its mathematical value:

Read the two sides in words. The left side expands every stored latent into a key, then compares the query with that key. The right side transforms the new query once, then compares that transformed query directly with the stored latents.

For our numbers, $U_K^{T}q=[2,1{]}^T$. The smaller dot product is $2(1)+1(2)=4$, exactly the same score as before.

This regrouping is called projection absorption. A fixed learned matrix can be combined with another projection, so the repeated calculation uses the compact representation. The DeepSeek-V2 paper describes this absorption and the separate position path used by MLA.

Both paths produce score 4. The absorbed path reads the two-number content cache directly.

Why is moving work to the query side useful? At one decode step there is one newest query per head but potentially thousands of past positions. A transformation done for the newest query is not the same cost pattern as expanding every past key.

We can check the identity with NumPy. It handles the small arrays; no model download is needed.

import numpy as np

We define the exact column-vector values above. One-dimensional arrays here represent those vectors, and @ performs matrix multiplication or a vector dot product as appropriate.

latent = np.array([1.0, 2.0])
up_key = np.array([[1., 0.], [0., 1.], [1., 1.], [1., -1.]])
query = np.array([1.0, 0.0, 1.0, 0.0])
direct_score = query @ (up_key @ latent)
absorbed_score = (up_key.T @ query) @ latent
assert np.isclose(direct_score, absorbed_score)

Both scores are 4. We intentionally check the unscaled content dot product, not complete MLA attention. Complete attention also includes position scores, scaling, visibility rules, and softmax.

Move the Value Expansion After the Weighted Sum

The value side has a related identity. Let $a_t$ be the attention weight for past position $t$. The ordinary weighted sum is:

We can first mix the narrow latents, then expand the one mixed result. We do not have to expand every past value separately.

Take two latent vectors, $c_1=[1,2{]}^T$ and $c_2=[3,0{]}^T$, with illustrative weights 0.25 and 0.75. Their weighted latent is $[2.5,0.5{]}^T$. If $U_V$ uses the same numeric matrix as our teaching $U_K$, the resulting four-number value output is $[2.5,0.5,3,2{]}^T$.

These weights are chosen to illustrate the value identity. They are not claimed to come from the single score in the preceding example.

Linearity lets expansion follow the weighted sum. Head-specific weights still produce head-specific results.

In a complete attention layer, the value up-projection can also combine with that head's contribution to the final output projection. This is why describing MLA only as "rebuild full K and V at every step" misses an important part of its inference design.

Position Rotations Complicate the Absorption

The position information from Day 17 tells attention where tokens occur. Rotary position embeddings, or RoPE, express part of that information by rotating query and key feature pairs. Day 29 will explain those rotations in detail.

The difficulty is algebraic, not simply a risk that compression might forget position. If a position-dependent rotation sits between the key up-projection and the dot product, the transformation is no longer the same fixed matrix for every cached position. It cannot be moved into the newest query in the simple way we just used.

DeepSeek's design separates a content score from a position score. In a simplified one-head notation:

$i$ is the querying position and $t$ is a key position. $C$ names content features. $R$ names features carrying rotary position information. The combined score is then scaled and masked before softmax, as in earlier attention lessons.

The content path can use absorption. The positional path remains available as a separate cached key.

The cache therefore contains more than $c_t$ alone: it also needs the small position-carrying key. Ignoring that extra vector would undercount memory and describe an incomplete attention mechanism.

Count the Cache That Is Actually Stored

Let the shared content latent have width $d_c$ and the separate cached rotary key have width $d_r$. With $L$ layers, $T$ saved positions, $B$ independent equal-length requests, and $b$ bytes per stored number, a simplified uniform-layer count is:

There is no extra factor of two around $d_c$. One shared content latent supports both key and value calculations. The formula describes this MLA storage layout, not ordinary separate K/V arrays.

For our toy comparison, choose $d_c=2$ and $d_r=2$. At two bytes per number, the stored entry needs $(2+2)\times 2=8$ bytes per layer-token. Our earlier ordinary-content baseline needed $16\times 2=32$ bytes. This illustrates counting, not a measured compression ratio for a named model.

Count the arrays that remain in memory, not the wider arrays used in the direct mathematical description.

A fair real-model comparison must use its actual head counts, dimensions, cache dtype, layer types, and implementation. Temporary buffers and model weights are additional costs. A smaller cache alone does not prove a faster complete service.

Sharing Heads and Sharing a Latent Are Different

In MQA, all query heads share one explicit KV head. In GQA, each group of query heads shares an explicit KV head. In MLA, heads can use different transformations of shared compact content, along with the position path.

The difference can be seen with the same two-number latent. One head could project it with $U_K$, while another uses a different learned matrix. They share the stored input but need not produce identical content keys or attention weights.

Shared storage does not require identical head computations.

These choices change how a model represents information. We should not take a trained MHA model, replace its cache with two numbers, and expect unchanged output. The architecture and its learned weights must support the compressed form.

Check Your Understanding

Why is the stored latent not a zipped copy of full K and V? The model learns a restricted representation during training rather than losslessly encoding arbitrary full vectors.

Why does query-side absorption help decoding? One new query can be transformed before it reads many saved latents.

Why must our memory formula include the rotary key? The content latent alone does not provide the complete position-dependent score in this design.

Compact storage works because the later computation is designed to use it, not because earlier context stops mattering.

Recap

MLA saves shared latent content and a separate position-carrying key. The direct equations can expand content into keys and values, but projection absorption lets an optimized path avoid materializing all those expanded cached vectors.

Day 28 keeps the question of storage but changes its scale. Once thousands of requests own growing caches, where should those entries live in GPU memory?