Xiaomi released three MiMo V2.6 models on 21 September 2026. Pro is the big one, with 1.02T parameters. Flash is the smaller one, with 309B. Both are the same design at two sizes. The third model is a 9B distill, and it is the only one most of us can run at home.
In this blog, we will learn how the Pro and Flash backbone is built and how 1.02T parameters fit in 566 GB. We will also see why each repo ships two draft models, and how much of all this reaches the 9B model.
What Did Xiaomi Release?
Let me tabulate the three checkpoints for your better understanding. The parameter counts are ours, counted from the tensor shapes. The vendor row is what Xiaomi says.
| Pro | Flash | Distill 9B | |
|---|---|---|---|
| Total parameters, backbone (our count) | 1,021.25B | 308.78B | 9.41B incl. vision |
| Active parameters per token (our count) | 41.90B | 15.45B | 9.41B (dense) |
| Vendor claim | 1.02T / 42B | 309B / 15B | 9B |
| Layers | 70 (60 SWA + 10 global) | 48 (39 SWA + 9 global) | 32 (24 linear + 8 full) |
| Routed experts (total / active) | 384 / 8 | 256 / 8 | none |
| Vocabulary | 152,576 | 152,576 | 248,320 |
| Main checkpoint, decimal GB | 566.0 | 172.9 | 18.8 |
| Expert storage | MXFP4 | MXFP4 | no experts (dense BF16) |
| Draft models shipped | 3 MTP layers + 5-layer DFlash | 3 MTP layers + 5-layer DFlash | none |
| Image, video, audio input | yes | yes | image and video only |
Here, we can see that our counts match Xiaomi's within rounding. The backbone count includes the embeddings and the output head. The active count takes every weight that is not an expert, plus 8 experts in each layer.
Pro and Flash are mixture of experts models. In simple words, each layer holds hundreds of small feed forward networks, called experts. A small router picks 8 of them for each token, and the rest stay asleep. That is how Flash stores 309B parameters but uses only 15.45B for each token.
How Is the Backbone Built?
Most language models let every layer look at every earlier token. That is called global attention. It sees everything, but its memory grows with every token we add.
MiMo does something cheaper. Most of its layers use sliding window attention, or SWA. In simple words, an SWA layer can only look at the last 128 tokens. Older tokens simply fall out of its view. A few global layers are placed between them, so information from far back can still travel forward.
Let's see both stacks side by side. Black cells are global layers, and grey cells are 128-token window layers.
Here, we can see the pattern. Flash repeats "5 window layers, then 1 global layer" almost perfectly. Its global layers sit at 0, 5, 11, 17, 23, 29, 35, 41 and 47. Pro uses blocks of 7 window layers, with 6 in the first block and in the last two. Its global layers sit at 0, 7, 15, 23, 31, 39, 47, 55, 62 and 69.
Layer 0 is special in both models. It is global, and it is the only layer with a normal dense feed forward network. Every other layer uses experts. There is also no shared expert, which many other MoE models keep.
Let me tabulate the attention details for your better understanding:
| Pro | Flash | |
|---|---|---|
| Hidden size | 6,144 | 4,096 |
| Query heads (SWA and GA) | 128 | 64 |
| KV heads, SWA layers | 8 | 8 |
| KV heads, global layers | 8 | 4 |
| Head dims, query-key / value | 192 / 128 | 192 / 128 |
| Rotary dims | 64 of 192 (partial_rotary_factor 0.334) |
same |
| RoPE base, global / SWA | 10,000,000 / 10,000 | same |
| Attention-sink bias | 60 tensors, one per SWA layer | 39 tensors, one per SWA layer |
Two details are worth a closer look.
First, only the window layers carry an attention sink. In simple words, a sink is a spare slot where a layer can park attention it does not want to spend. A 128-token window has very little to look at, so this spare slot helps. The count matches exactly: 60 sinks in Pro and 39 in Flash, one for each window layer.
Second, only 64 of the 192 query and key dimensions get position information from RoPE. The other 128 dimensions carry no position signal at all. They match tokens by meaning only.
Why Does a 1M Context Stay Cheap?
To answer this, we need the KV cache. In simple words, the KV cache is the model's notes about every token it has already read. Each layer saves a key and a value for each token, so it does not have to read the text again. These notes are what fill GPU memory in long chats.
A window layer only keeps notes for its last 128 tokens. After that, its cache stops growing. So only the global layers grow with the context.
With a 16-bit cache, each token costs:
- Flash: 9 global layers x 4 KV heads x (192 + 128) x 2 bytes = 23.0 KB per token.
- Pro: 10 global layers x 8 KV heads x 320 x 2 bytes = 51.2 KB per token.
The window layers add a fixed 25.6 MB for Flash and 39.3 MB for Pro, no matter how long the context gets.
Let's compare this with a model where every layer is global:
Here, we can see the saving. If the window layers kept every token, Flash would need 222.7 KB per token and Pro 358.4 KB. That is 9.7 and 7.0 times more. At the full 1,048,576-token context, one sequence needs 24.2 GB for Flash and 53.7 GB for Pro.
Flash even beats the small 9B distill, which needs 32.8 KB per token. Both use 4 KV heads in their growing layers. But the 9B stores 256-wide keys and 256-wide values, while Flash stores 192 and 128.
We could check the 9B's number on our RTX 5090. In llama.cpp, going from 32k to 128k context added 33.8 KB per token on the Q4_K_M build and 35.1 KB on the BF16 build. That is close to the 32.8 KB we worked out. The small extra is compute buffers.
How Big Is Each Expert?
Each MoE layer sends a token to 8 experts. The router scores experts with a sigmoid and adds a small correction bias to each score. That bias keeps the load spread evenly, without an extra training loss.
One Flash expert has 25.2M parameters: a gate, an up and a down projection around a 2,048-wide middle layer. One Pro expert has 37.7M. Flash has 256 experts in each layer, and Pro has 384.
Here is the key number. Experts are 98.1% of Flash's backbone and 97.9% of Pro's. So, whatever format the experts are stored in decides the size of the whole checkpoint.
Why Are the V2.6 Files Half the Size of V2.5?
The V2.6 checkpoints are almost exactly half the size of V2.5. Flash went from 315.0 GB to 172.9 GB. Pro went from 1,033.4 GB to 566.0 GB. But the architecture did not shrink at all.
Let's see where the space goes:
Here, we can see that almost the whole bar is experts, and only the experts shrank. They moved from FP8 to MXFP4. In simple words, MXFP4 stores each weight in 4 bits instead of 8. Every group of 32 weights shares one 8-bit scale, which sets how big those 32 numbers are.
Let me tabulate what changed and what stayed:
| Part | V2.5 | V2.6 |
|---|---|---|
| Routed experts | FP8 E4M3, one F32 scale per 128 x 128 block | MXFP4: two 4-bit values per byte (U8), one 8-bit scale per 32 values |
Attention qkv_proj, dense FFN, MTP FFN |
FP8 with block scales | FP8 with block scales (unchanged) |
Attention o_proj |
BF16 | BF16 (listed in ignored_layers) |
| Router | F32 | BF16 (moe_router_dtype) |
| Embeddings, LM head, norms | BF16 | BF16 |
Each byte holds two 4-bit weights, and the shared scales add a little on top. That works out to 4.25 bits per expert weight. So Flash's 302.8B expert weights take 160.9 GB.
The most interesting part comes from Xiaomi's technical report. Xiaomi says it trained the model in MXFP4 during mid-training. During RL, it rounded the experts to MXFP4 after every update. If that is true, these 4-bit experts are not a rough copy of some better original. They are the exact weights the model learned to use.
So, a GGUF or any other 4-bit build of Flash is not squeezing BF16 weights. It is re-encoding weights that are already 4-bit.
Why Are There Two Draft Models?
A big model writes one token at a time, and each step reads billions of weights. That is slow.
Speculative decoding fixes this. In simple words, a small helper model guesses the next few tokens quickly. Then the big model checks all the guesses in one pass. Every correct guess is a token we got almost for free.
Both MiMo repos ship two such helpers.
The first helper lives inside the main checkpoint. It is three MTP layers, which stands for multi-token prediction. Each MTP layer is a window attention layer with a dense feed forward network. It takes the model's last hidden state, joins it with the next token's embedding, and guesses one more token.
There is a small oddity here. Flash's config declares these layers, but Pro's config does not, even though Pro ships them. The Hugging Face code also throws them away on load. So they only matter to a serving engine like SGLang.
The second helper is new in V2.6. It is called DFlash, and it sits in its own dflash/ folder. It is a separate 5-layer model: 1.47B parameters for Flash and 2.77B for Pro. Instead of guessing one token at a time, it fills a block of 8 positions at once: one known token plus 7 blank slots.
DFlash also reads more than the last layer. It joins hidden states from five backbone layers: 0, 11, 23, 35 and 47 for Flash, and 0, 15, 31, 47 and 69 for Pro. These are the triangles in the layer map above.
Xiaomi's report says DFlash's guesses were accepted 31.3% longer on average than the MTP layers' guesses. But the SGLang command on both model cards uses --speculative-algorithm EAGLE with 3 draft steps. That is the MTP path, and nothing in it points at the dflash/ folder. So the serving recipe Xiaomi publishes does not use the drafter it describes on the same page.
What Changed From V2.5 to V2.6?
The skeleton did not change. Flash V2.6 has exactly the same 73,081 tensor names and shapes as MiMo V2.5. Pro V2.6 has the same 159,530 backbone tensors and the same 48 MTP tensors as V2.5 Pro. The one new thing in Pro is the vision and audio encoders, which V2.5 Pro never had.
So, is V2.6 new training, or just the old weights saved in 4 bits? To find out, we compared small tensors between the two versions, byte by byte.
For Flash, every tensor we checked changed, including all 39 sink biases and all 96 norms.
Pro was almost the same story, with one clue. 6 of its 60 sink biases are exactly the same as in V2.5 Pro, all in early layers: 1, 2, 6, 8, 9 and 11. All 140 norms changed. Six tensors of 128 values each do not match by accident. So Pro V2.6 was very likely trained further from V2.5 Pro, not from scratch.
Xiaomi's report also says Pro saw only 3T tokens of image, video and audio data, against 22T for Flash. So Pro learned to see and hear in a much shorter stage.
What Does the 9B Distill Inherit?
Almost nothing from the architecture. MiMo V2.6 Distill Qwen 9B is Qwen 3.5 9B with new weights. It keeps Qwen's 24 linear attention layers, 8 full attention layers and dense feed forward networks. It has no window layers, no sinks, no experts and no MXFP4.
It even keeps Qwen's 248,320-token vocabulary, while Pro and Flash use 152,576. So, the "distill" is plain fine-tuning on text that the MiMo models wrote. Xiaomi says this data was 77.4B tokens.
The one thing the 9B did inherit is a problem. Qwen 3.5 9B ships a one-layer MTP head, made of 15 mtp.* tensors. The distill ships none of them. But its config still says mtp_num_hidden_layers: 1. Xiaomi dropped the head and kept the setting.
We hit this on our own rig. A default llama.cpp conversion trusts the config and builds a 33-block GGUF. Then the server refuses to load it with tensor 'blk.32.attn_norm.weight' not found. Converting with --no-mtp fixes it. It also means the 9B has no draft model at all, while both of its teachers ship two.
Can We Run Pro or Flash on One RTX 5090?
No. Flash's main checkpoint is 172.9 GB, and 160.9 GB of that is experts that are already 4-bit. There is no real way to shrink it onto a 32 GB card. On one machine, it needs roughly 180 GB of combined memory before any KV cache. Pro, at 566.0 GB, is a data center model.
Of the three V2.6 releases, only the 9B fits a consumer GPU, at 18.8 GB in BF16. And as we saw, it shares none of the design above.
What We Did Not Test
- We did not run Pro or Flash, so nothing here says how well they perform. Every benchmark score on the model cards is Xiaomi's.
- The KV cache sizes are arithmetic from the configs, with a 16-bit cache. A real server will allocate memory its own way.
- The V2.5 to V2.6 comparison checked small tensors only: norms, sink biases, router biases and one embedding row. We did not compare expert or attention weights.
- The 31.3% DFlash gain, the MXFP4 training and the token counts are claims from Xiaomi's report that we could not verify.
Conclusion
We opened MiMo V2.6 Pro and Flash and found one design at two sizes. Most layers look at only the last 128 tokens, almost every parameter is an expert, and those experts were trained in 4 bits.
Key takeaways:
- Flash stores 308.78B parameters and uses 15.45B for each token. Pro stores 1,021.25B and uses 41.90B.
- Only 9 of Flash's 48 layers and 10 of Pro's 70 layers are global. That keeps the cache at 23.0 KB and 51.2 KB per token, 9.7 and 7.0 times less than all global layers.
- The V2.6 files are half the size of V2.5 because the experts moved from FP8 to MXFP4. Xiaomi trained in that format, so the 4-bit experts are the real weights.
- Each repo ships two drafters, three MTP layers and a 5-layer DFlash, but Xiaomi's own SGLang command uses the MTP path.
- The 9B distill is a fine-tuned Qwen 3.5 9B. It lost Qwen's MTP head but kept the config line, which breaks a default GGUF conversion.
Next steps:
- Read the GLM 5.3 vs GLM 5.3 Flash teardown for a different way to make a 1M context cheap.
- See how another big sparse model behaves on one desktop in DeepSeek V4 Flash vs Qwen 3.8 Flash Next at 1-bit.
- Use the local LLMs technical reference guide to match a model and a context size to our own card.
This is how MiMo V2.6 Pro and Flash are built. We started with the layer map and its 128-token windows. We saw why the cache stays small and why the files halved. And we finished with the two drafters and the 9B model that shares the name but not the design.
