DeepSeek V4 runs a 1M token context window, and it does that without ever letting attention read a million tokens. The part that makes it possible is DeepSeek Sparse Attention, which the team calls DSA, and it sits in every model in the current family: DeepSeek-V4-Pro-0813 released on 13 August 2026, the smaller DeepSeek-V4-Flash-0731, and the experimental multimodal DeepSeek-V4-Flash-Vision-Exp.
In this blog, we will learn what DSA does and why it makes long context cheap. We will build it up one picture at a time, so no background beyond "a model reads text" is needed.
Where These Numbers Come From
Two kinds of number appear in this post, and it is worth separating them up front.
The shapes are read from the published config.json of DeepSeek-V4-Pro-0813 and DeepSeek-V4-Flash-0731. That is where index_n_heads, index_head_dim, index_topk and the 1,048,576 token window come from, and they are exact.
The measurements come from DeepSeek-V3.2-Exp, the release that introduced DSA in September 2025. That is the only generation for which DeepSeek published a full technical report, a reference implementation, an open source kernel write-up and a cost curve. Wherever a figure below is measured rather than declared, it comes from there and we say so.
What Problem Is DSA Solving?
Let's say we hand a model an entire codebase and ask one question about it.
Plain attention has an odd habit here. For every single word it writes, it checks every word that came before. A 1M token window means 1,048,576 things to check, and almost all of them turn out to say nothing useful for that word.
So, here comes DSA to the rescue. Send a cheap scout through the whole document first. The scout is fast and a bit rough. It marks the 1,024 pages worth reading properly. Then the careful reader reads only those.
That is the entire idea. The rest of this post is about why the scout works, what it costs, and how DeepSeek taught it.
What Does Attention Actually Do?
Before the fix, the thing being fixed. In simple words, attention is how a token asks the tokens before it for help.
Every token turns itself into three separate things.
The query is what this token is looking for. The key is what a token can be found by. The value is what a token hands over once it is picked.
To decide how much a past token matters, we take the dot product of our query with that token's key. A dot product is just multiply each pair of numbers and add them all up, and it comes out large when two vectors point the same way. That single number is the score.
Softmax then turns the whole row of scores into weights that add up to 1. The output is the values mixed together in those proportions.
Here, we can see the cost. One token has to do this against every earlier token, and every token has to do it.
Why Is Most of That Work Wasted?
Let's look at the scores we just computed, laid out as a grid. Each row is one token looking back at everything before it.
The grid is mostly dark. Softmax pushes almost every weight close to zero, and a handful of cells per row carry nearly all the weight. Drawn as a profile, one row is a flat floor with a few tall spikes.
This is the observation the whole method is built on. If we only computed the spikes, we would get nearly the same answer for a fraction of the work.
There is one catch, and it is a real one. We cannot see which cells are bright until we compute them, and computing them all is exactly the thing we were trying to avoid. Every sparse attention method is an answer to that catch.
Where Does the Cost Actually Go?
Running a model has two phases, and they get expensive for different reasons.
Prefill is the model reading our prompt. It builds the whole score grid in one go. Fill V4's window and that grid has 1,048,576 x 1,048,576 cells, which is 1.1 trillion of them for one head in one layer. Double the prompt and this gets four times bigger.
Decode is the model writing its answer, one token at a time. Recomputing the past for every new token would be hopeless, so the model stores the keys and values it already built. That store is the KV cache. Every new token reads the whole cache again.
The interesting part is that decode is not slow because of the arithmetic.
Why Is Decoding Held Back by Memory?
A GPU has memory in layers, and they are wildly different sizes.
The KV cache lives in HBM, which is the big memory on the card. The arithmetic happens in SRAM, which sits on the chip itself and holds only around 20 MB. Data has to make the trip from one to the other, and every output token drags the whole cache across.
How big is that trip? The first DSA release gives us an exact figure to hold on to. DeepSeek-V3.2-Exp cached one 576 number entry per token per layer, which is 1,152 bytes in BF16, or 71,424 bytes across its 62 layers. At 128K tokens the whole cache came to 8.72 GB, and DeepSeek's own kernel write-up shows that same arithmetic. V4 caches a compressed entry too, so the shape of the problem is unchanged and only the constant moved.
The tensor cores are fast enough to be idle most of the time, waiting for that data. This is what people mean when they say long context decoding is memory bound.
What Does Compression Fix, and What Does It Not?
DSA does not work alone. DeepSeek models also compress what they cache, and it is worth being precise about what compression fixes, because DSA fixes something else entirely.
Plain attention caches a separate key and value for every head. DeepSeek squeezes all of that into one small entry per token. In V3.2-Exp the squeeze was Multi-head Latent Attention, or MLA, with every token pressed to the same 576 numbers. V4 does it per layer instead: the kv_lora_rank key is gone from both V4 configs, replaced by a compress_ratios list that alternates between 4 and 128 through the middle of the stack.
Let me tabulate the difference for your better understanding.
| What it changes | What stays the same | |
|---|---|---|
| Compression | how many numbers each cached token needs | there are still 1,048,576 cached tokens to read |
| DSA | how many cached tokens get read at all | each one is still a full compressed entry |
Compression made the row thinner. DSA reads fewer rows. Because the two savings are on different axes, they multiply rather than overlap.
One detail matters for later. DSA needs every cached entry to be shared across all query heads, which is why both V4 configs declare num_key_value_heads of 1. One cached entry serves every query head of the token.
How Does the Lightning Indexer Score a Token?
Now the scout itself. DeepSeek calls it the lightning indexer, and it is a small attention shaped scorer that answers one question: how much is this past token likely to matter.
The new token gets projected into 64 small query vectors of 128 numbers each, plus 64 plain numbers called head weights. Every past token gets projected into one key of 128 numbers, shared by all 64 heads. Those two shapes are index_n_heads and index_head_dim in the config, and they are 64 and 128 in every DeepSeek generation that has shipped DSA.
For one past token, the indexer takes the dot product with each of the 64 queries, then puts every result through a ReLU, which is a rule with a short definition: if a number is negative, call it zero. Those 64 numbers are added up in the proportions given by the head weights, and that sum is the index score.
Three choices here are all about speed. There are 64 indexer heads against V4 Pro's 128 real attention heads. Each head is 128 numbers wide, far narrower than the real thing. And the whole scorer runs in FP8, which is half the bytes of BF16.
Let's see what that recipe actually produces. Here is a small run of it over a pretend chat of 12 tokens, with a budget of 3 instead of 1,024.
Token 2 wins easily at 24.2. Tokens 11 and 7 take the other two slots at 6.9 and 6.4. The other nine are dropped, and the real attention will never see them.
Notice that some scores come out negative even though the ReLU already removed negative dot products. That is the head weights doing their job. They are learned numbers with no sign constraint, so a head can vote against a token.
How Does the Selector Pick 1,024?
The selector is the least clever part of DSA, and deliberately so.
It sorts the scores, keeps the highest index_topk, and drops the rest. That is the whole step. What comes out is a list of positions, which the attention kernel uses as an address list into the paged KV cache.
In DeepSeek-V4-Pro-0813 that budget is 1,024. In DeepSeek-V4-Flash-0731 it is 512. Against a 1,048,576 token window, 1,024 is 0.098% of the context, or one token in a thousand.
DeepSeek's reference implementation does the cut in two steps. It calls topk on the row of scores to get the winning positions, then builds a mask that sets every position outside that list to minus infinity, so softmax gives them a weight of zero.
Two things are worth noticing. It asks for the smaller of the budget and the number of tokens that exist so far, so a short prompt is simply read in full and nothing is dropped. And the selection is redone from scratch for every query token, so two neighbouring tokens can end up reading completely different parts of the document.
Why 1,024 and not 4,096? DeepSeek gives no derivation for the number in any generation. It is a chosen setting, not a computed one, and as we will see it has been getting smaller.
What Does the Whole Block Look Like?
We now have every piece, so let's put them together.
The hidden state goes down two paths. On the cheap path, the indexer scores every past token in FP8 and the selector cuts that to 1,024 positions. On the precise path, the token is compressed and written to the KV cache.
The two paths meet at the gate. The position list from the cheap path decides which cached entries get pulled out of HBM. Only those entries are decompressed and attended over, in full BF16 precision.
One consequence is worth stating plainly. The softmax denominator now sums over 1,024 tokens instead of all of them, which is a mathematically different operation from full attention. The model is not asked to tolerate that after the fact. It is trained with this normalisation from the start of sparse training, so it learns inside the constraint.
How Much Arithmetic Does This Actually Save?
The headline is easy. At a full 1M window with a budget of 1,024, attention reads one token in every 1,024, so the attention step alone does 1,024 times less work than it would have done. V4 Flash, at 512, halves that again.
But the scout is not free, and it never stops reading the whole context. To see what the block really costs we need every shape published, and that means going back to the first DSA release at its 128K window. The proportions are what carry over.
We will count multiply-adds, because that is the unit of work a GPU actually does. For one query token against one key token, that generation's attention did 128 heads x 576 numbers for the query and key dot product, plus 128 heads x 512 numbers for mixing the values. That comes to 139,264. The scout did 64 heads x 128 numbers, which is 8,192, and that is still its size in V4 today.
Divide one by the other, and the scout is 17 times cheaper for the same pair of tokens.
Let me tabulate the whole block for your better understanding.
| Work for one pair | Pairs it does | Total multiply-adds | |
|---|---|---|---|
| Dense attention, the old way | 139,264 | 131,072 x 131,072 | 2,393 trillion |
| The scout | 8,192 | 131,072 x 131,072 | 141 trillion |
| Sparse attention | 139,264 | 131,072 x 2,048 | 37 trillion |
| DSA in total, the scout plus sparse attention | 178 trillion |
Here, we can see two things at once.
The attention itself got 64 times cheaper, which is just 131,072 divided by 2,048. But once the scout is added back in, 2,393 trillion against 178 trillion, the honest saving is 13.4 times.
The second thing is in the middle two rows. The scout costs 141 trillion and the sparse attention costs 37 trillion, so at this length the scout does about 3.8 times more arithmetic than the attention it protects. The expensive part of the block is the part that decides what to read, and cutting the budget to 1,024 makes that imbalance sharper, not softer.
That is a real limit, and DeepSeek stated it directly when DSA was introduced. The indexer is still quadratic in context length. DSA shrank the constant by 17 times and moved the heavy work into FP8, but it did not change the shape of the curve.
How Was DSA Trained?
DSA was not trained from scratch. It was fitted onto a finished production model in two stages, and this is the one part of the design where the only published account is the original one.
In stage one, every existing weight is frozen and attention stays dense. Only the brand new indexer learns. Its target is the frozen model's own behaviour: the real attention scores are summed across all heads, normalised into a distribution, and the indexer is trained with a KL divergence loss to match it. In plain words, the scout learns to copy where the big model already looks. This ran for 1,000 steps at a learning rate of 0.001, which is 2.1 billion tokens.
In stage two, the budget is switched on. The selector keeps its top-k, attention sees only those tokens, and every weight in the model is unfrozen. This ran for 15,000 steps at a learning rate of 0.0000073, which is 943.7 billion tokens. The indexer keeps its KL loss, now measured only over the tokens that were selected.
One design choice in stage two deserves attention. The indexer's input is detached from the computational graph. The indexer learns only from its own KL loss, and the main model learns only from the language modelling loss.
Important
That detachment prevents a failure that would be hard to spot. If the gradients were shared, the main model could reduce its loss by making its attention pattern easier for the indexer to guess, rather than by getting better at language. Separating the two losses closes that door.
What Does the GPU Actually Run?
An idea that works on paper still has to map onto hardware. DeepSeek shipped the kernels for this as FlashMLA, and the decode path runs as four steps.
Step one scores everything in one FP8 batched matrix multiply. Step two applies the head weights and runs a fused partial sort for the top-k. Step three gathers those rows out of the paged KV cache. Step four decompresses them and runs the real attention in BF16, on chip.
Step three is where the design pays off, and where it is hardest. GPUs like reading memory in long straight runs, and gathering a thousand scattered addresses is the opposite of that.
The cache is stored in FP8 to make the gather smaller. In the V3.2-Exp layout, which is the one DeepSeek documented byte by byte, each token's entry was 656 bytes: 512 values in float8_e4m3, 4 scale factors in float32, and the 64 position numbers left in bfloat16 because they are too sensitive to quantise.
The scout's own key cache is small by comparison, because it holds only 128 FP8 numbers and one scale per token per layer, against a full compressed entry for the main cache.
Note
The measured throughput comes from DeepSeek's FlashMLA repository, on H800 SXM5 with CUDA 12.8: up to 640 TFLOPS on sparse prefill and 410 TFLOPS on sparse decode. The write-up notes that the earlier version of the sparse decode kernel reached only 250 TFLOPS, and the gain came from letting two thread blocks share the dequantisation work.
How Do the Two Sparsities Fit Together?
DeepSeek V4 Pro is a sparse model twice over, in two unrelated ways.
The first sparsity is the mixture of experts, and it long predates DSA. Each layer of V4 Pro holds 384 routed experts, and the router wakes 6 of them for a given token. That is where the gap between its 1.7T stored parameters and the much smaller number it actually runs comes from.
The second sparsity is DSA, and it is what this post is about. Instead of activating all 1,048,576 past tokens, only 1,024 of them are read.
Both are decided by content rather than by position, and both are learned. The router learns which experts suit this token. The indexer learns which tokens suit this query. Neither is a fixed window or a stride pattern.
How Has the Budget Changed Across Generations?
The scout has not changed shape since it was introduced. Both V4 configs still declare 64 indexer heads of 128 numbers each, exactly as the first DSA model did. What has changed is how many tokens the scout is allowed to keep, and the direction is one way.
| Config key | V3.2-Exp | V4-Flash-0731 | V4-Pro-0813 |
|---|---|---|---|
index_n_heads |
64 | 64 | 64 |
index_head_dim |
128 | 128 | 128 |
index_topk |
2,048 | 512 | 1,024 |
kv_lora_rank |
512 | absent | absent |
compress_ratios |
absent | present, alternating 4 and 128 | present, alternating 4 and 128 |
sliding_window |
absent | 128 | 128 |
num_hidden_layers |
61 | 43 | 61 |
hidden_size |
7,168 | 4,096 | 7,168 |
max_position_embeddings |
163,840 | 1,048,576 | 1,048,576 |
The budget got tighter while the window got bigger. V4 Pro keeps half as many tokens as the first DSA model did, across a window six times longer, and V4 Flash keeps a quarter as many. Whatever DeepSeek learned between these releases, it was that the scout could be trusted with less.
Note
The V4 model cards stop using the DSA name in their prose. They describe the stack as "DFlash attention, MoE, Hyper-Connections, and the DSpark forward path" instead. The indexer keys are still sitting in the config, so the scout is still doing its job under a newer label, but we cannot tell from the cards alone whether DFlash is a rename or a redesign, and DeepSeek has not published a V4 report of the kind it published for V3.2-Exp.
What Does It Cost?
The published cost curve for DSA is the one from its first generation, estimated on H800 clusters at a rental price of $2 per GPU hour and plotted against position in the sequence.
We read these off Figure 3 of that report, so treat them as close rather than exact. At the 128K mark, prefill went from about 0.19, and decode from about 0.25.
The prefill panel also shows something honest. Below roughly 8K tokens the two lines cross, and the sparse model is slightly more expensive. At short context the scout is pure overhead, because there was never much to skip. DeepSeek uses a masked dense mode for short prefills for exactly this reason.
Now the question that decides whether any of this was worth it. Does a model get worse when it stops reading almost all of its own context? The cleanest evidence is still the original comparison, because DSA was added to a finished model and nothing else was changed.
| Benchmark | Before DSA (V3.1-Terminus) | With DSA (V3.2-Exp) |
|---|---|---|
| MMLU-Pro (EM) | 85.0 | 85.0 |
| GPQA-Diamond (Pass@1) | 80.7 | 79.9 |
| Humanity's Last Exam (Pass@1) | 21.7 | 19.8 |
| BrowseComp (Acc.) | 38.5 | 40.1 |
| BrowseComp_zh (Acc.) | 45.0 | 47.9 |
| SimpleQA (Acc.) | 96.8 | 97.1 |
| LiveCodeBench 2408-2505 (Pass@1) | 74.9 | 74.1 |
| Codeforces-Div1 (Rating) | 2046 | 2121 |
| Aider-Polyglot (Acc.) | 76.1 | 74.5 |
| SWE Verified (Agent mode) | 68.4 | 67.8 |
| SWE-bench Multilingual (Agent mode) | 57.8 | 57.9 |
| Terminal-bench (Terminus 1) | 36.7 | 37.7 |
| AIME 2025 (Pass@1) | 88.4 | 89.3 |
| HMMT 2025 (Pass@1) | 86.1 | 83.6 |
Scores move both ways and mostly by less than a point. That is the result the whole V4 line is built on. Sparse attention turned out to be close to free in quality, so the budget kept shrinking.
Where Does It Still Hurt?
Nothing here is free, and DeepSeek was unusually direct about that when it published the method.
- The scout is still quadratic. The constant is 17 times smaller and the work runs in FP8, but the indexer reads the whole context for every query, forever. At a 1M window that is the dominant cost of the block.
- A miss cannot be recovered. Attention is computed only over the selected set. If the indexer leaves out a token that would have mattered, that token is invisible for that query at that layer, and nothing downstream notices. A budget of 512 leaves less room for that mistake than 2,048 did.
- Scattered reads fight the hardware. Gathering a thousand arbitrary rows is the access pattern GPUs are worst at, which is why the kernel needed a dedicated engineering effort of its own.
- The budget is one number for everyone.
index_topkapplies to every query, every layer and every task. Some queries clearly need fewer. Some may genuinely need more. - The kernels are hardware specific. FlashMLA's sparse paths target NVIDIA Hopper and Blackwell, and DeepSeek describes the Blackwell decode kernel as not yet optimised.
- The V4 internals are not documented the way the first DSA release was. We can read the configs, but there is no V4 report, no reference implementation walkthrough and no published cost curve.
Model Card at a Glance
Every value here is a key from the published config.json of DeepSeek-V4-Pro-0813.
| Key | Value |
|---|---|
hidden_size |
7,168 |
num_hidden_layers |
61 |
num_attention_heads |
128 |
num_key_value_heads |
1 |
index_n_heads |
64 |
index_head_dim |
128 |
index_topk |
1,024 |
sliding_window |
128 |
q_lora_rank and o_lora_rank |
1,536 and 1,024 |
n_routed_experts |
384 |
num_experts_per_tok |
6 |
max_position_embeddings |
1,048,576 |
rope_scaling |
YaRN, factor 16, from a 65,536 base |
dspark_target_layer_ids |
58, 59, 60 |
Conclusion
DeepSeek Sparse Attention is one idea carried through carefully. Run something cheap over everything, then run the expensive thing over what survives.
Key takeaways:
- Attention scores are mostly near zero, but we cannot know which ones are useful without computing them. The lightning indexer is a trained guess that costs 17 times less per pair than the attention it stands in front of.
- V4 Pro keeps 1,024 of 1,048,576 tokens, which is 0.098% of a full window, and it re-picks for every single query token. V4 Flash keeps 512.
- The scout has not changed shape in three generations. It is still 64 heads of 128 numbers, still FP8. Only the budget moved, and it moved down.
- Measured on the release that introduced it, attention alone got 64 times cheaper, and the whole block 13.4 times, because the scout still reads everything.
- Training came in two stages: the scout first learns to imitate dense attention against a frozen model, then everything trains with the budget on and two separate gradient paths.
- Benchmarks moved by well under a point in both directions when DSA was first added, which is why the budget has been shrinking ever since.
Next steps:
- Read the GLM 5.3 architecture teardown to see the same indexer idea taken further, with 57 of 78 layers reusing another layer's pick.
- Read the Qwen 3.8 Flash Next teardown for a different answer to the same problem, mixing linear attention layers into the stack.
- See what DeepSeek V4 Flash actually does on one desktop in DeepSeek V4 Flash vs Qwen 3.8 Flash Next at 1-bit.
This is how DeepSeek V4 reads a million tokens without reading a million tokens. A small scout skims the whole window cheaply, marks about a thousand positions, and the real attention never looks at anything else.
