Qwen 3.8 Flash Next vs Qwen 3.8 27B Architecture Teardown

Qwen published Qwen3.8-Flash-Next on Hugging Face, and calls it an experimental preview of the architecture behind Qwen4. The weights are 360 GB. Our benchmark card is an RTX 5090 with 32 GB, so we cannot load it at all.

So we did the thing we could do. We pulled the config file and the weight map from the repo, pulled the same two files for Qwen 3.8 27B, and compared them. A weight map lists every tensor in a model by name. In simple words, it is the parts list, so every count below comes from the files and not from the model card.

One detail belongs up front, because it sets the scale of everything else. Both models ship at bfloat16. Their config files say dtype: bfloat16, so each parameter takes 2 bytes on disk and in memory. Every size and memory figure below is at that precision, the 360 GB included. The only numbers at a lower precision are our own measurements of Qwen 3.8 27B on the RTX 5090, which run at Q4_K_M with an f16 KV cache, and we say so again where those appear.

In this blog, we will find the four changes that separate the two models, and work out what each one costs. Anyone can download the same two small files from Hugging Face and check the numbers with us.

Here is the short version. Flash Next is not a new family of models. It is the Qwen 3.8 skeleton with four parts replaced, and all four exist to make a long conversation cheap.

How Much of Qwen 3.8 Does Flash Next Keep?

Let's start with what did not move, because most write-ups of this model skip it.

Here, we can see the same vocabulary, the same position encoding, the same recurrent block shape, the same mix of layer types, the same vision tower, and the same context length. A qwen4_exp checkpoint and a qwen3_5 checkpoint agree on all of it.

The differences are all in size. Flash Next has 48 blocks where the 27B has 64, and it is 2,560 wide where the 27B is 5,120. So for each token, the new model runs a shallower and narrower network than the 27B. Everything below is about how it gets away with that.

How Do 512 Experts Replace One Feed Forward Layer?

The feed forward layer is the part of a block that does the plain thinking after attention. Qwen 3.8 27B has one of them per block, 17,408 wide, and every parameter in it runs on every token.

Flash Next has no such tensors at all. In their place, each block carries these:

model.language_model.layers.N.mlp.experts.gate_up_proj
model.language_model.layers.N.mlp.experts.down_proj
model.language_model.layers.N.mlp.gate.weight
model.language_model.layers.N.mlp.shared_expert.{gate,up,down}_proj.weight
model.language_model.layers.N.mlp.shared_expert_gate.weight

This is a mixture of experts layer. In simple words, the model keeps many small feed forward networks and wakes only a few of them for each token. Flash Next holds 512 experts per block, each only 640 wide, and 10 routed experts plus 1 shared expert fire per token.

The arithmetic works out cleanly. Each block stores 512 experts of 640 by 2,560 across three matrices, which is 2.52B parameters. Across 48 blocks that is 121B, which is essentially the whole 125B of core weights the card quotes. So the mixture of experts layer is the model.

Of that 2.52B per block, only 11 experts fire, which is 54M. Across 48 blocks that is 2.6B of expert compute per token instead of 121B.

Let's read that grid as one block's feed forward layer. It holds 2.52B of weights, and 54M of them run. We multiply by 48 blocks and we have the whole stack: 121B stored, 2.6B executed.

This is the oldest trick in the current playbook, and it is still the biggest single lever. A dense 27B reads all 27B of its parameters for every token it writes, whether that token is "the" or the key line of a bug fix. Flash Next reads 6B. Capacity and cost per token stopped being the same number.

The size of each expert matters as much as the count. At 640 wide against a 2,560 hidden size, each expert is a quarter of the model's width, and 10 of them combine per token. That is much finer than the usual 8 large experts, and it gives the router far more combinations to pick from for each token.

How Does Sparse Attention Cap the Cost of a Long Chat?

This is the change Qwen leads with, and it is the one aimed at long agent sessions.

In the 27B, the 16 full attention blocks are ordinary gated attention. Every token that gets decoded attends over the whole cache. The KV cache is the memory a model keeps for every token in the conversation, so reading all of it gets slower as the chat grows.

In Flash Next, three new tensors appear on each of the 12 full attention blocks, and once more inside the draft head:

model.language_model.layers.N.self_attn.indexer.index_qk_proj.weight
model.language_model.layers.N.self_attn.indexer.q_layernorm.weight
model.language_model.layers.N.self_attn.indexer.k_layernorm.weight

The indexer is a cheap scoring network. Its only job is to decide which parts of the cache the real attention should look at. Its settings sit in the config:

Two savings compound here. The first is the cache itself, because 12 blocks with 2 key value heads store less than 16 blocks with 4:

That is 64 KiB per token against 24.75 KiB, which is 16.0 GiB against 6.2 GiB of cache at the full 262k window. We are not asking anyone to take the 64 KiB on trust. It matches what llama-server really allocates for Qwen 3.8 27B on our rig, measured at two context sizes.

The second saving is larger, and it is the real point. Attention traffic per decoded token stops growing with the conversation:

At 262k context, the 27B reads all 16 GiB of its cache to produce one token. Flash Next reads its 2,048 token budget, which is about 48 MiB, plus one compressed key per 4 token block for the scan, which is about 192 MiB. Call it 240 MiB against 16 GiB, so roughly 68 times less.

We should be precise about what that curve is. Sparse attention is not free of context cost. The indexer still scores every block, so the line still rises, just at 768 bytes per token instead of 65,536. Anyone describing this as constant cost attention is overselling it. What it really does is shrink the growing part by 85 times, and pin the expensive part to a fixed 2,048 token window.

Where Did the LayerNorms Go?

This one we did not expect, and it falls straight out of the tensor names.

Qwen 3.8 27B carries an input_layernorm and a post_attention_layernorm on each of its 64 blocks, and one more pair inside its draft head. That is a standard transformer: one norm before attention, one before the feed forward layer, and one residual stream running through the whole stack. The residual stream is the running hidden state that every block adds its result back into. In simple words, it is the model's shared notebook.

Flash Next has neither of those two. The normalization did not disappear from the model, though. It moved somewhere new, and each block carries these tensors instead:

model.language_model.layers.N.attn_hyper_connection.{hc_norm, block_inject_weight,
                              input_mix_weight_down, input_mix_weight_up}.weight
model.language_model.layers.N.mlp_hyper_connection.{...same four...}.weight
model.language_model.hyper_connection_mixer.{hc_norm, input_mix_weight_down,
                              input_mix_weight_up}.weight

That is 96 hyper connection modules across the 48 blocks, two more inside the draft head, and a mixer at the top of each stack. The config gives them their shape with hc_count: 4 and hc_lowrank: 320. Qwen calls this Gated Residual on the card.

In simple words, the model now has four notebooks instead of one. Each block reads a mixture of the four, decides that mixture from the data itself, and writes its result back through a gate. The norm moved inside the connection instead of sitting before the block.

So the residual stream is effectively 4 times 2,560, which is 10,240 wide, while the blocks themselves stay 2,560 wide and cheap. That matters a lot for a model that is 16 blocks shorter and half as wide as the 27B. A narrow, shallow network needs somewhere to put information that should skip several blocks, and the four notebooks are where it goes.

The cost at inference time is the rank 320 projections, which is small next to the attention and expert work. The risk sits at training time instead. Every block depends on this connection, and there is no plain residual path to fall back on. That is likely part of why the release is labelled experimental.

What Does the 51B Lookup Table Do?

The strangest number on the model card is the extra 51B of n-gram embedding. In the weight map it looks like this:

model.language_model.layers.2.ple.ple_embedding.ngram_embedding.shard_0.weight
... 128 shards ...
model.language_model.layers.2.ple.ple_embedding.ngram_embedding.shard_127.weight
model.language_model.layers.2.ple.{conv1d, key_proj, value_proj, norm_*}.weight

It sits at exactly one layer, layer 2, because ple_layer_ids is [2]. The config says ngram_size: 3 and ngram_vocab_size_base: 20000000, and the arithmetic is exact. Twenty million n-grams at 2,560 dimensions each is 51.2B parameters.

An n-gram is just a short run of words. So this is a lookup table of twenty million n-grams, each with its own vector. It is split across 128 shards and injected once near the bottom of the stack. Reading it costs one lookup per token. It does no matrix multiplication at all.

Qwen's framing on the card is simple. An embedding table adds parameters more cheaply than a mixture of experts, because it needs less computation and is easier to offload. That is not marketing. An expert's weights have to be on the GPU when the router picks it, and the router picks differently on every token. A 51B lookup table can live in system RAM, or on an SSD, and be read one row at a time. For anyone thinking about running large models on small cards, this is the most interesting thing in the release, and it is getting the least attention.

There is a fifth, smaller change worth a line. The 27B's multi token prediction head is a plain dense block. Flash Next's is a full expert block with its own indexer and its own hyper connections, which is where the 4B draft head on the card comes from. That head is what makes speculative decoding work, and we measured what it is worth on the 27B in our llama.cpp speed settings post.

That chart is measured on our own card, not derived. The pattern is the reason a bigger draft head matters. The gain tracks acceptance, and acceptance depends on what the model is writing. Counting and JSON are predictable enough that the head guesses right 0.84 to 0.86 of the time, and the model runs at 181 tokens a second. Prose drops acceptance to 0.31, and most of the speedup goes with it. Flash Next spends 4B parameters on that head, which is a lot to spend on guessing.

Where Does Flash Next Actually Win?

Now the part everyone asks about first.

On Qwen's own published numbers, the gap is not spread evenly. The chart above carries all twelve benchmarks. Let me tabulate the two ends for your better understanding.

Every one of those runs the model for many turns with a growing transcript. Now here are the four where the gain almost disappears.

Two things explain that split, and only one of them is architecture.

The architecture half is honest enough. Every benchmark in the first table keeps a transcript that grows for many turns, and Qwen evaluated DeepSWE at a 256k context window. That is exactly the regime the four changes were built for. At that depth the attention read is 68 times cheaper, the cache is 2.6 times smaller, and the cost per token barely moves as the conversation grows.

At 262k, decoding one token costs the 27B about 66.4 GiB of memory traffic, of which 50.3 GiB is weights and 16.0 GiB is cache. Flash Next costs about 11.5 GiB, almost all of it weights, because its attention read is capped and its recurrent state is a constant 108 MiB whatever the context. That is 5.8 times less work per token, at the depth agent harnesses really run at.

Cheaper tokens at depth do not only mean faster. They mean a model can afford to keep the whole transcript, re-read the file it opened forty turns ago, and try again. Long horizon benchmarks reward exactly that.

SWE-bench Pro is the awkward one. Qwen ran it at the same 256k depth as DeepSWE, and it moved 0.8 points. So depth on its own does not buy the gain, and the config files cannot tell us what else these two benchmarks ask for differently.

The other half is not architecture. Flash Next also carries 125B of core weights against 27B, plus a 51B memory table the 27B does not have. More capacity should win on knowledge, and it does, modestly. It is also a separately trained model. Qwen ships no test that separates architecture from training. So nobody outside the lab can say how much of that 22.3 point jump is sparse attention, and how much is a better agent training run. Anyone claiming the architecture caused the benchmark gains is guessing. The architecture explains the cost numbers, which is a claim we can check from the config files, and that is the claim we are making here.

Saturation cuts the other way too. GPQA Diamond at 89.2 has very little room left, so a 2.5 point gain there is not evidence that the new architecture reasons better. It is evidence that the benchmark is finished.

Why Can We Not Run Flash Next Locally?

The 27B is comfortable on a 32 GB card. Here is what we measure on ours, at Q4_K_M with an f16 KV cache and flash attention on:

Flash Next on the same card is not a tuning problem. It is an arithmetic problem:

That last row is a real file on Hugging Face, three shards adding up to 72.5 GB, and it is a dynamic 1 bit quantization. So the smallest thing anyone has made of this model is still more than twice the memory of one 5090.

Two practical notes. llama.cpp has no qwen4_exp support yet, so the GGUF repos that exist are either that single 1 bit build or empty placeholders. And the vendor path is explicit, because Qwen's card recommends SGLang, vLLM, or their own cloud, and points at a hosted Flash model with a 1M window as the production version. This is a datacenter model with the weights published, not a local model.

What would change that is the 51B lookup table. It is a third of the storage, it does no arithmetic, and it is the one part that genuinely suits sitting in system RAM. Let's say a runtime lands that keeps the table on the host and streams only the 6B active path to the GPU. Then the requirement stops being 100 GB of VRAM. It becomes about 128 GB of ordinary system memory with a modest card. That is a different kind of machine, and one people actually own. We would not bet on the timeline.

Which Model Should We Run?

Qwen3.8-Flash-Next is worth paying for through an API when our work is long agent sessions. That means repo scale coding harnesses, multi turn tool use, and anything where the transcript runs past 100k tokens. That is where its 22 point and 16 point gaps live, and where the cost structure it was designed around is real. We should not pay for it to answer single questions, because on that job it is a point or two ahead of a model we can run ourselves for free.

Qwen 3.8 27B stays the model on our disk. It holds 94% of its advertised context on a single 32 GB card, which is a better ratio than anything else we have measured. It reaches 181 tokens a second on structured output with the draft head on. And on the tests that finish in one turn, it gives up at most 5.1 points to a 180B model.

What We Did Not Test

• We never ran Flash Next. It does not fit on our card at any precision that exists today, so every Flash Next score here is Qwen's published number.
• The cost numbers are derived from the config files and weight maps, not measured on hardware. The Qwen 3.8 27B numbers next to them are measured on our own rig.
• Qwen ships no ablation that separates the architecture from the training, so no benchmark gain here can be credited to any one of the four changes.
• The 51B table sitting in system RAM is an idea, not a result. No runtime does it yet.

Conclusion

We compared Qwen3.8-Flash-Next and Qwen 3.8 27B from their config files and weight maps, without downloading 360 GB of weights. The two models share their vocabulary, their position encoding, their layer pattern, their vision tower and their context length. Four parts changed, and all four exist to make long conversations cheap.

Key takeaways:

• The feed forward layer became 512 small experts, of which 11 fire per token. That stores 121B and runs 2.6B.
• Sparse attention pins the expensive read to a 2,048 token window, so a token at 262k context costs about 240 MiB of cache traffic instead of 16 GiB.
• The two LayerNorms per block moved inside a new connection, and the single residual stream became four parallel lanes that give a shallow, narrow stack somewhere to carry information.
• A 51B n-gram lookup table adds capacity with no matrix multiplication, which makes it the one part of this model that could live in system RAM.
• The published gains are concentrated in long agent work, up to 22.3 points, and nearly vanish on the short tests. SWE-bench Pro spoils the pattern at 0.8 points from the same depth, so the architecture explains the cost, not the scores.

Next steps:

• See how the draft head in Qwen 3.8 27B is tuned in Qwen 3.8 27B speed settings on llama.cpp.
• Read the version teardown, Qwen 3.8 27B vs Qwen 3.6 27B vs Qwen 3.5 27B, to see what changed between the releases we can actually run.
• Use the local LLMs technical reference guide to match a model and a context size to our own GPU.

This is how Qwen says Qwen4 will be built. We started with the parts that did not change, we found four that did, and we finished with the arithmetic that says this model needs a datacenter and the one part of it that might not.