DeepSeek V4.1 Flash came out on 10 September 2026 with a 1M token context window, yet none of its layers ever runs attention over the whole million. Sparse attention is what makes that possible. DeepSeek first shipped it in V3.2 as DeepSeek Sparse Attention, or DSA. V4.1 Flash ships its newest form, Compressed Sparse Attention 2, or CSA2.
In this blog, we will learn how that sparse attention works and why it makes a million tokens cheap. We will build it up one picture at a time, so no background beyond "a model reads text" is needed.
Where These Numbers Come From
Every number in this post comes from one of three places, and it helps to know which is which.
The shapes come from the published config.json of DeepSeek-V4.1-Flash, and they are exact. That is where index_n_heads, index_topk, sliding_window, compress_ratios and the 1,048,576 token window come from. When the config alone does not say how a value is used, we read the reference code in the model's inference/model.py.
The design and its measured results come from the DeepSeek-V4.1-Flash technical report, which ships with the model. It gives the cache sizes, the layer layout and the training recipe.
A few numbers are older. DSA first appeared in DeepSeek-V3.2-Exp in September 2025. That release is where the before and after benchmark test and the dollar cost curve were measured. Wherever a number comes from there, we say so.
What Problem Is Sparse Attention Solving?
Let's say we hand a model an entire codebase and ask one question about it.
Plain attention has an odd habit here. For every single word it writes, it checks every word that came before. A 1M token window means up to 1,048,576 things to check, and almost all of them turn out to say nothing useful for that word.
So, here comes sparse attention to the rescue. Send a cheap scout through the whole document first. The scout is fast and a bit rough. It marks the 512 spots worth reading properly. Then the careful reader reads only those, plus the last 128 tokens right in front of it.
That is the whole idea. The rest of this post is about why the scout works, what it costs, and how V4.1 Flash shares one scout's marks across many layers.
What Does Attention Actually Do?
Before the fix, the thing being fixed. In simple words, attention is how a token asks the tokens before it for help.
Every token turns itself into three separate things.
The query is what this token is looking for. The key is what a token can be found by. The value is what a token hands over once it is picked.
To decide how much a past token matters, we take the dot product of our query with that token's key. A dot product means we multiply each pair of numbers and add them all up. It comes out large when two vectors point the same way. That single number is the score.
Softmax then turns the whole row of scores into weights that add up to 1. The output is the values mixed together in those proportions.
Here, we can see the cost. One token has to do this against every earlier token, and every token has to do it.
Why Is Most of That Work Wasted?
Let's look at the scores we just computed, laid out as a grid. Each row is one token looking back at everything before it.
The grid is mostly dark. Softmax pushes almost every weight close to zero, and a handful of cells per row carry nearly all the weight. Drawn as a profile, one row is a flat floor with a few tall spikes.
This is the idea the whole method is built on. If we only computed the spikes, we would get nearly the same answer for a fraction of the work.
There is one catch, and it is a real one. We cannot see which cells are bright until we compute them, and computing them all is exactly the thing we were trying to avoid. Every sparse attention method is an answer to that catch.
Where Does the Cost Actually Go?
Running a model has two phases, and they get expensive for different reasons.
Prefill is the model reading our prompt. It works out the whole score grid in one go. A full 1M window gives a grid of 1,048,576 x 1,048,576 cells, which is 1.1 trillion. A token only looks back, never forward, so only the lower half of the grid is needed. That is still about 550 billion scores for one head in one layer. Double the prompt and this gets four times bigger.
V4.1 Flash also has a trick of its own for prefill. Its 40 layers are split into a 20 layer encoder and a 20 layer decoder, and most prompt tokens only need the encoder half. The report says this nearly halves prefill work. We will see how when we reach the layer map.
Decode is the model writing its answer, one token at a time. Recomputing the past for every new token would be hopeless, so the model stores the keys and values it already built. That store is the KV cache. With plain attention, every new token reads the whole cache again.
The interesting part is that decode is not slow because of the arithmetic.
Why Is Decoding Held Back by Memory?
A GPU has memory in layers, and they are wildly different sizes.
The KV cache lives in HBM, which is the big memory on the card. The arithmetic happens next to the cores, in on-chip memory called SRAM. Take NVIDIA's H100 as an example. Its HBM holds 80 GB and moves about 3.35 TB a second. Its on-chip memory is 256 KB in each of its 132 SMs, which is about 33 MB in total, plus a 50 MB L2 cache. Data has to make the trip from one to the other.
How big is that trip? V4.1 Flash keeps 890 bytes of long range cache per token. DeepSeek calls this the global KV cache. A full 1M window of it comes to 890 MB. Here, a megabyte means 1,048,576 bytes, so that is about 933 million bytes. That is small for a million tokens, and a good part of this post explains how DeepSeek got it that small.
The tensor cores are fast enough to sit idle much of the time, waiting for that data. This is what people mean when they say long context decoding is memory bound.
What Does Compression Fix, and What Does It Not?
Sparse attention does not work alone. Before it picks anything, V4.1 Flash shrinks what it stores. The V4.1 report names three ways to shrink a cache, and V4.1 Flash uses all three.
First, each entry is small. Plain attention stores a separate key and value for every head. V4.1 Flash stores one entry of 512 numbers per position, shared by all 64 query heads, and the same entry acts as both the key and the value. It keeps that entry in FP4, which means 4 bits per number. The 512 numbers plus their scale factors take 288 bytes.
Second, some layers store fewer positions. In the encoder half, each entry pools 2 neighbouring tokens into one, so a 1M window needs 524,288 entries. In the decoder half the pooling ratio is 1, so every token keeps its own entry.
Third, most layers store nothing of their own. Only 4 of the 40 layers write a global cache. Every other sparse layer borrows the cache of the last layer before it that wrote one.
The table below lines the four savings up side by side.
| What it changes | What stays the same | |
|---|---|---|
| Small entries | how many bytes each cached entry needs | every entry is still there to be read |
| Pooled tokens | how many entries the cache holds | the model still has to find the right ones |
| Shared layers | how many layers keep a cache | each layer still reads a cache |
| Sparse attention | how many entries get read at all | the skipped entries are still stored |
Shrinking makes the cache cheap to keep. Sparse attention makes it cheap to read. The savings sit on different axes, so they multiply rather than overlap.
One detail is easy to misread. The config declares num_key_value_heads of 1, and it is tempting to read that key as the switch that turns sparse attention on. It is not. The first DSA model, V3.2-Exp, declared 128 there, and its report says it still ran sparse attention with one shared entry per token. The key describes how the cache is laid out, not whether the model is sparse.
How Does the Lightning Indexer Score an Entry?
Now the scout itself. DeepSeek calls it the lightning indexer. It is a small attention shaped scorer that answers one question: how much is this past entry likely to matter?
The new token gets projected into 32 small query vectors of 128 numbers each, plus 32 plain numbers called head weights. Every cached entry gets one key of 128 numbers, shared by all 32 heads. In V4.1 Flash that key is projected straight from the main cache entry, so the scout needs no separate path of its own. Those shapes are index_n_heads and index_head_dim in the config.
For one past entry, the indexer takes the dot product with each of the 32 queries. Then it puts every result through a ReLU, which is a rule with a short definition: if a number is negative, call it zero. The 32 numbers are added up in the proportions given by the head weights, and that sum is the index score.
Three choices here are all about speed. There are 32 indexer heads against 64 real attention heads. Each head is 128 numbers wide, a quarter of the real 512. And the indexer queries and keys are stored in FP4, a quarter of the bytes of BF16. DeepSeek trains with that rounding switched on, so the scout learns to work at that precision.
Let's see what that recipe actually produces. Here is a small run of it over a pretend chat of 12 tokens, with a budget of 3 instead of 512.
Token 2 wins easily at 24.2. Tokens 11 and 7 take the other two slots at 6.9 and 6.4. The other nine are dropped, and the real attention will never see them.
Notice that some scores come out negative even though the ReLU already removed negative dot products. That is the head weights doing their job. They are learned numbers with no sign rule, so a head can vote against a token.
How Does the Selector Pick 512?
The selector is the least clever part of the design, and that is on purpose.
It sorts the scores, keeps the highest index_topk, and drops the rest. What comes out is a list of positions, which the attention step uses as an address list into the cache.
In V4.1 Flash that budget is 512. In a decoder layer at a full 1M window, that is 512 of 1,048,576 entries, or 0.049%, one token in every 2,048. In an encoder layer the same 512 is chosen from 524,288 pooled entries. Each of those entries holds 2 tokens, so the pick covers 1,024 tokens.
The pick is not the whole story. Every layer also reads its sliding window, the 128 most recent tokens, in full and without any scoring. So a sparse layer reads at most 512 + 128 = 640 entries for each new token.
Here is where readers often slip. The 512 is not a cap on what the model knows about the document. It is chosen again for every query token, at every layer that runs an indexer. Two neighbouring tokens can read completely different parts of the document, and over a long answer the model touches far more than 512 places.
The reference code asks for the smaller of the budget and the number of entries that exist so far. So a short prompt is simply read in full and nothing is dropped.
Why 512? It is a chosen setting, and the V4.1 report lists it in the model setup next to the other shapes. As we will see later, the budget has moved from one release to the next.
What Does One Layer Look Like?
We now have every piece, so let's put one layer together. The picture shows a Full layer, the kind that runs its own scout.
The hidden state goes down three paths. On the cheap path, the indexer scores the shared cache and the selector cuts it to 512 positions. On the local path, the layer writes its newest token into a small window cache of the last 128 tokens, kept in FP8. On the global path, the layer writes its entry into the shared FP4 cache that later layers borrow.
The paths meet at the attention step. It reads the 128 window entries and the 512 picked entries. The picked FP4 entries are turned back into higher precision numbers first, and then attention runs over all 640 in one go.
One consequence is worth stating plainly. The softmax now adds up over at most 640 entries instead of the whole context. That is a different operation from full attention. It also has one extra term: each head has a learned sink, a slot that can soak up weight when nothing in the list deserves it. The model is trained this way from its first step, so it learns to live inside the budget.
How Does CSA2 Share One Pick Across Layers?
So far we looked at one layer. V4.1 Flash has 40, and this is where it differs most from earlier DeepSeek models.
A full scout and a full cache in every layer would repeat a lot of work and storage. So, CSA2 gives each sparse layer one of three fixed modes.
- Full: the layer builds its own global cache, runs its own indexer, and makes a fresh pick.
- Reindex: the layer borrows the cache of the last Full layer, runs its own indexer queries against it, and makes a fresh pick.
- Reuse: the layer borrows both the cache and the latest pick, and goes straight to attention.
In all three modes the layer still computes its own query and its own window cache.
Let me tabulate the whole stack for your better understanding.
| Layers | Part | Attention | Mode | Tokens per global entry |
|---|---|---|---|---|
| 0 and 1 | encoder | window only | none | no global cache |
| 2, 8, 14 | encoder | window plus CSA2 | Full | 2 |
| the other 15 of layers 3 to 19 | encoder | window plus CSA2 | Reuse | 2 |
| 20 | decoder | window plus CSA2 | Full | 1 |
| 24, 28, 32, 36 | decoder | window plus CSA2 | Reindex | 1 |
| the other 15 of layers 21 to 39 | decoder | window plus CSA2 | Reuse | 1 |
Here, we can see the savings. Only 4 layers build a global cache. Only 8 layers run the indexer. The other 30 of the 38 sparse layers simply reuse a pick. The report says these Reuse layers run with only 15 GPU kernels during prefill and 11 during decode.
Layer 20 has one more job. It is the first decoder layer, and its global cache is projected from the last encoder layer's output. Every decoder layer after it reads that same cache. DeepSeek calls this the Causal Encoder-Decoder design, and it is why most prompt tokens only need to run through the encoder half.
How Does the Hierarchical Indexer Cap the Scout?
Reuse cuts how many layers run a scout. It does not change how much each scout reads. A Full or Reindex layer still scores every entry it can see, and at 1M that is a lot of scoring.
So, V4.1 Flash adds a second trick in the decoder, called the Hierarchical Sparse Indexer.
Layer 20 scores all 1,048,576 entries as usual and takes its own top 512. It also makes a coarser pick. It groups the entries into blocks of 8, gives each block the score of its best entry, and keeps the best 2,048 blocks. The block holding the newest tokens is always kept, so recent context cannot be pushed out. That gives 2,048 x 8 = 16,384 candidate positions, about 1.6% of a full window.
The four Reindex layers after it score only those 16,384 candidates, and each takes its own top 512 from them. So their scoring work stays the same whether the context is 64K or 1M. Only layer 20 still scans the whole range, along with the three Full layers in the encoder.
The candidate pool was added in post-training. The model was trained with it switched on, so the later indexers learn to search inside it.
How Much Arithmetic Does This Actually Save?
Now let's count. We will count multiply-adds, because that is the unit of work a GPU actually does. We take one new token being written at position 1,048,576, the end of a full window. These counts cover attention and the scout only, not the expert layers.
The real attention first. For one query against one entry, each of the 64 heads does 512 multiply-adds for the query and key dot product and 512 more to mix in the value. That is 64 x 1,024 = 65,536 per entry. A sparse layer reads 640 entries, so it does 65,536 x 640 = 41.9 million. Across the 38 sparse layers and the 2 window only layers, that comes to 1.61 billion.
Now the scout. For one query against one entry, it does 32 heads x 128 numbers = 4,096 multiply-adds. That is 16 times cheaper than the real attention for the same entry. But it runs over far more entries.
- Each encoder Full layer (2, 8, 14) scores 524,288 entries, which is 2.15 billion each.
- Decoder layer 20 scores 1,048,576 entries, which is 4.29 billion.
- Each Reindex layer (24, 28, 32, 36) scores 16,384 candidates, which is 67 million each.
Let me tabulate the whole model for your better understanding.
| Entries per layer | Layers | Multiply-adds for one token | |
|---|---|---|---|
| Attention with no selection | every entry, plus the window | 38 sparse, 2 window only | 1.99 trillion |
| Scout, encoder Full layers | 524,288 | 3 | 6.44 billion |
| Scout, decoder Full layer | 1,048,576 | 1 | 4.29 billion |
| Scout, Reindex layers | 16,384 | 4 | 0.27 billion |
| Sparse attention | 640 | 38 sparse, 2 window only | 1.61 billion |
| Scout plus sparse attention | 12.6 billion |
Here, we can see two things at once.
The attention itself got about 1,240 times cheaper: 1.99 trillion against 1.61 billion. Once the scout is added back in, the honest saving for the whole model is about 158 times.
The second thing is in the middle rows. The scout does 11.0 billion multiply-adds and the attention does 1.61 billion. So at 1M the scout does about 6.8 times more arithmetic than the attention it serves. The expensive part is the part that decides what to read.
This is exactly why the candidate pool matters. Without it, the four Reindex layers would each scan 1,048,576 entries, and the scout would cost 27.9 billion instead of 11.0 billion. The scout also runs in FP4, which the report counts at a quarter of the cost of BF16 math. With all of that, the report's own chart shows V4.1 Flash's decode compute growing by only about a quarter from a 4K context to a 1M one.
How Was It Trained?
The way DeepSeek trains sparse attention has changed a lot since the first release.
DSA began as an add-on. In V3.2-Exp, DeepSeek fitted it onto a finished model in two stages. First, every existing weight stayed frozen and attention stayed dense. Only the new indexer learned, by copying where the frozen model already looked. That ran for 1,000 steps and 2.1 billion tokens. Then the budget was switched on, and every weight trained together for 15,000 steps and 943.7 billion tokens. The indexer's input was cut off from the main model's gradients, so each part learned only from its own loss.
V4.1 Flash skips the warm-up entirely. Sparse attention is trained from scratch, with no dense stage at all. Pre-training covers 45T tokens at a sequence length of 64K, and the context is stretched to 1M at the 34T token mark. The Hierarchical Sparse Indexer comes last, in post-training.
What Does the GPU Actually Run?
An idea that works on paper still has to map onto hardware. Per layer and per new token, the sparse path comes down to three steps.
Step one scores every visible entry. It is a batch of FP4 dot products, followed by the ReLU and the head weights. Step two keeps the top 512, and the report credits a TopK kernel from DeepSelect for this job. Step three fetches the 512 picked entries and the 128 window entries, turns them back into higher precision, and runs the real attention on chip. The attention kernel takes the list of positions and does the fetching itself, so step three is one kernel, not two.
Only the 8 indexing layers run steps one and two. The 30 Reuse layers jump straight to step three with the pick they were handed.
Step three is where the design pays off, and where it is hardest. GPUs like reading memory in long straight runs, and gathering 512 scattered entries is the opposite of that. Storing each entry in FP4 makes that gather smaller: 288 bytes per entry, against 584 bytes in V4's FP8 format.
Note
DeepSeek ships these attention kernels in its FlashMLA library. For V4.1, the README lists sparse decoding as available only on SM100, NVIDIA's Blackwell generation. It reports a fused attention kernel reaching up to 1,430 TFLOPS in prefill and 670 TFLOPS in decode on a B200.
How Do the Two Sparsities Fit Together?
DeepSeek V4.1 Flash is a sparse model twice over, in two unrelated ways.
The first sparsity is the mixture of experts, and it long predates sparse attention. Each layer holds 384 routed experts and one shared expert, and the router wakes 6 routed experts for a given token. That is why a model with 552B backbone parameters runs only 16B of them per token while writing, and 8B while reading a prompt.
The second sparsity is the attention pick we followed through this post. Instead of all 1,048,576 past positions, a decoder layer reads 512 picked entries plus a 128 token window.
Both picks are made by content rather than by position, and both are learned. The router learns which experts suit this token. The indexer learns which entries suit this query. The sliding window is the one fixed part, and it only covers the last 128 tokens.
How Has the Scout Changed Across Generations?
The scout has changed shape over four releases, and so has the budget. The table puts the config values side by side.
| Config key | V3.2-Exp | V4-Flash-0731 | V4-Pro-0813 | V4.1-Flash |
|---|---|---|---|---|
index_n_heads |
64 | 64 | 64 | 32 |
index_head_dim |
128 | 128 | 128 | 128 |
index_topk |
2,048 | 512 | 1,024 | 512 |
| indexer number format | FP8 | FP4 | FP4 | FP4 |
| layers that run an indexer | 61 of 61 | 21 of 43 | 30 of 61 | 8 of 40 |
compress_ratios |
absent | 4 and 128 | 4 and 128 | 2 and 1 |
sliding_window |
absent | 128 | 128 | 128 |
| global KV cache per token | 48,068 bytes | 3,514 bytes | about 5,031 bytes | 890 bytes |
max_position_embeddings |
163,840 | 1,048,576 | 1,048,576 | 1,048,576 |
The V4-Pro-0813 cache figure is our own arithmetic from its config. The other three come from Figure 1 of the V4.1 report, and the same arithmetic reproduces all three exactly.
Here, we can see the direction. The indexer got narrower, from 64 heads to 32. It moved from FP8 to FP4. It runs in far fewer layers, from every layer down to 8 of 40. And the cache per token fell from 48,068 bytes to 890, a 54 times drop.
The budget moved too. V3.2-Exp picked 2,048 tokens from its 128K context. V4.1 Flash picks 512 entries from a window eight times longer.
One more thing changed in V4. The V4 models pool tokens as well, with a ratio of 4 in some layers and 128 in others. Only the ratio 4 layers run an indexer. The ratio 128 layers read every one of their heavily pooled entries with no pick at all. V4.1 Flash drops that split and runs CSA2 in every sparse layer.
What Does It Cost?
We now know what V4.1 Flash reads. What does all of this mean in practice?
The cleanest number is the cache. Across DeepSeek generations, the global KV cache per token went from 389,120 bytes in DeepSeek-V1, to 48,068 in V3.2, to 3,514 in V4 Flash, to 890 in V4.1 Flash. That is 437 times smaller than V1, and about 4 times smaller than V4 Flash. The report adds that V4.1 Flash needs only about 1/8 of V4 Flash's persistent cache, the part saved to SSD for reuse. Two things multiply to give that: the smaller global cache, and no window caches saved there at all.
For dollars, the curve we can read comes from the V3.2-Exp report. It compared V3.2-Exp against V3.1-Terminus on H800 GPUs rented at 2 US dollars per GPU hour. We traced the lines from the figure's vector data, so these values are close to exact. At the 128K position, prefill fell from about 0.67 dollars per million tokens to about 0.19 dollars. Decode fell from about 2.15 dollars to about 0.25 dollars, roughly 9 times cheaper.
Below about 10K tokens, the prefill lines cross and the sparse model costs slightly more. At short context the scout is pure overhead, because there was never much to skip. DeepSeek said it uses a masked form of dense attention for short prefills to keep them efficient.
Now the question that decides whether any of this was worth it. Does a model get worse when it stops reading almost all of its own context? The cleanest test is still that first release, because DSA was added to a finished model and nothing else was changed.
| Benchmark | Before DSA (V3.1-Terminus) | With DSA (V3.2-Exp) |
|---|---|---|
| MMLU-Pro (EM) | 85.0 | 85.0 |
| GPQA-Diamond (Pass@1) | 80.7 | 79.9 |
| Humanity's Last Exam (Pass@1) | 21.7 | 19.8 |
| BrowseComp (Acc.) | 38.5 | 40.1 |
| BrowseComp_zh (Acc.) | 45.0 | 47.9 |
| SimpleQA (Acc.) | 96.8 | 97.1 |
| LiveCodeBench 2408-2505 (Pass@1) | 74.9 | 74.1 |
| Codeforces-Div1 (Rating) | 2046 | 2121 |
| Aider-Polyglot (Acc.) | 76.1 | 74.5 |
| SWE Verified (Agent mode) | 68.4 | 67.8 |
| SWE-bench Multilingual (Agent mode) | 57.8 | 57.9 |
| Terminal-bench (Terminus 1) | 36.7 | 37.7 |
| AIME 2025 (Pass@1) | 88.4 | 89.3 |
| HMMT 2025 (Pass@1) | 86.1 | 83.6 |
Scores move both ways. About half of them move by less than a point, and none of the percentage scores moves by more than 3. V4.1 Flash was sparse from its first step, so this kind of before and after test does not apply to it.
Where Does It Still Hurt?
Nothing here is free, and the V4.1 report is direct about where the risks sit.
- The first scouts still read everything. Layers 2, 8, 14 and 20 score every visible entry for every token. Over a whole prompt, that work still grows with the square of the length. The candidate pool fixed this only for the four Reindex layers.
- A miss cannot be recovered. Attention sees only the pick and the window. If the indexer leaves out an entry that mattered, that entry is invisible for that query. With Reuse, one pick serves up to six layers, so one miss is shared by all of them. The report itself names selection errors in CSA2 as a risk in untested cases.
- The window is rebuilt, not stored. To save space, V4.1 Flash does not keep window caches for long. When one is missing, it rebuilds it from only the last 128 tokens, which is close to the original but not exact. The report says the effect on quality was negligible in its tests.
- Scattered reads fight the hardware. Gathering 512 arbitrary entries is the access pattern GPUs are worst at.
- The budget is one number for everyone.
index_topkis 512 for every query, every indexing layer and every task. Some queries clearly need fewer. Some may need more. - The kernels are hardware specific. FlashMLA lists V4.1 sparse decoding only on SM100, NVIDIA's Blackwell generation.
Model Card at a Glance
Every value here comes from the text_config block of the published config.json of DeepSeek-V4.1-Flash.
| Key | Value |
|---|---|
hidden_size |
5,120 |
num_hidden_layers |
40 |
num_attention_heads |
64 |
num_key_value_heads |
1 |
head_dim |
512 |
index_n_heads |
32 |
index_head_dim |
128 |
index_topk |
512 |
sliding_window |
128 |
compress_ratios |
0 for layers 0 and 1, 2 for layers 2 to 19, 1 for layers 20 to 39 |
kv_source_layer_ids |
2, 8, 14, 20 |
index_source_layer_ids |
2, 8, 14, 20, 24, 28, 32, 36 |
candidate_topk_blocks and candidate_block_size |
2,048 and 8 |
q_lora_rank and o_lora_rank |
1,280 and 1,024 |
n_routed_experts and num_experts_per_tok |
384 and 6 |
max_position_embeddings |
1,048,576 |
rope_scaling |
YaRN, factor 16, from a 65,536 base |
Conclusion
Sparse attention is one idea carried through carefully. Run something cheap over everything, then run the expensive thing over what survives.
Key takeaways:
- Attention scores are mostly near zero, but we cannot know which ones matter without computing them. The lightning indexer is a trained guess that costs 16 times less per entry than the attention it stands in front of.
- V4.1 Flash reads 512 picked entries plus the last 128 tokens. In a decoder layer at 1M, that pick is 0.049% of the context, and it is made again for every query token.
- The cache is shrunk three ways before anything is picked: small FP4 entries, 2 tokens per entry in the encoder, and only 4 layers that store a cache. The result is 890 bytes per token.
- CSA2 shares work across layers. Only 8 of 40 layers run the scout, and the later decoder scouts search a fixed pool of 16,384 candidates.
- At 1M, the scout still does about 6.8 times more arithmetic than the attention it serves. The whole block is still about 158 times cheaper than attention with no selection.
- Training moved from a two stage add-on in V3.2-Exp to sparse attention from the very first step in V4.1 Flash.
Next steps:
- Read the GLM 5.3 architecture teardown to see a similar indexer idea, with 57 of 78 layers reusing another layer's pick.
- Read the Qwen 3.8 Flash Next teardown for a different answer to the same problem, mixing linear attention layers into the stack.
- See what DeepSeek V4 Flash actually does on one desktop in DeepSeek V4 Flash vs Qwen 3.8 Flash Next at 1-bit.
This is how DeepSeek V4.1 Flash reads a million tokens without reading a million tokens. We started with a scout that skims every entry cheaply. Then we saw the selector keep 512 entries next to a 128 token window. Finally, we watched CSA2 share one pick across many layers, so the scouting stays small.
