GLM 5.3 vs GLM 5.3 Flash Architecture Teardown

What if we told you that two 1 million token models can have almost completely different memory requirements? Z.ai's GLM 5.3 and GLM 5.3 Flash both support a massive 1M token context window and both use 64 attention heads. But there is a shocking difference hiding underneath.

For every token we process, GLM 5.3 needs about 47.6 KB of KV cache. Flash needs just 6.0 KB. That's an 8 times difference. At 1 million tokens, we're talking about approximately 49.9 GB of KV cache for GLM 5.3 versus only 6.3 GB for Flash. On an RTX 5090 with 32 GB of VRAM, one doesn't even fit. The other barely makes a dent.

But here's the real mystery. If both models have 64 heads and the same 1M context window, where did that 8 times difference come from? Why do only 11 of Flash's 45 layers need to store KV cache for every token? What are the other 34 layers actually doing? And why does our calculation from the raw model files produce an 8 times ratio, while Z.ai's own launch material describes the difference as roughly 4.4 times?

To answer that, we're going below the marketing numbers and into the actual model architecture. We inspected the config files and safetensors headers of both checkpoints, without downloading the model weights, to reconstruct exactly how their KV cache is built. And the answer is hiding in the way these two models handle attention.

What Are These Two Models?

GLM 5.3 was announced on 14 August 2026 and Flash on 26 August 2026. The big model is mostly GLM 5 with the context dial turned up. We diffed the two config files, and the skeleton did not move at all: same 78 layers, same hidden size, same 256 experts, same vocabulary. Only three things changed, max_position_embeddings from 202,752 to 1,048,576, rope_theta from 1M to 8M to carry that longer window, and an indexer scheme we come to below. Everything else that is new in this release lives in Flash, which is a fresh model that reads images and video and mixes linear attention with sparse full attention in one stack.

Let me tabulate the two checkpoints for your better understanding. Every row comes from the config files and the safetensors headers, not from a model card.

Here, we can see the number that matters most. Both models keep about 95% of their weights asleep for any single token. GLM 5.3 runs 5.4% of itself, and Flash runs 5.3%. Capacity and cost for each token stopped being the same number a while ago, and these two models are built on that gap.

How Are the Layers Arranged?

Let's look at both stacks side by side.

GLM 5.3 stacks 78 identical DSA blocks. DSA stands for DeepSeek Sparse Attention, and we met it in our DeepSeek V4 Flash post. In simple words, it is normal attention with a filter in front. A small scorer ranks every earlier token, and real attention only reads the top 2,048 of them.

Flash follows a simple 3 to 1 pattern: three KDA linear attention layers, then one DSA sparse attention layer. Across the 45 layer stack that gives us 34 KDA layers and only 11 DSA layers. In both models, layers 0 to 2 are dense, everything above them is mixture of experts, and the last layer is a draft layer for speculative decoding.

And this is the single most important difference in the whole teardown. Only those 11 DSA layers need a growing KV cache. The KV cache is the memory a model keeps for every token in the chat, so it is the thing that gets more expensive as the conversation gets longer. In GLM 5.3, all 78 layers pay it.

What Is Inside a KDA Linear Attention Layer?

Flash's 34 linear layers use Kimi Delta Attention, or KDA. A linear attention layer does not keep a growing cache. It keeps one fixed size memory and edits it as each token arrives, the way we would keep one notepad instead of a folder of every page we ever wrote. A short four token convolution on each channel gives the layer local word order, a forget gate decides how much of the old notepad to keep, and a write strength decides how hard the new token writes into it.

So, here comes the number that matters. The notepad is 64 heads of 128 by 128 for each layer, which is 35.65M elements across all 34 linear layers, or about 71 MB in BF16 for one sequence. That number does not change when the chat gets longer. A conversation of 10 tokens and a conversation of a million tokens cost the same 71 MB in these layers.

Linear attention is not free, and the shapes say where it pays. A KDA layer holds 137.7M parameters and a DSA layer holds 124.9M, so the linear layer is the bigger of the two. It spends 12.8M extra weights for each layer to stop caching. It saves cache and compute, not bytes on disk.

What Is the Lightning Indexer, and Why Is It Shared?

Sparse attention needs something to decide what is worth reading. In DSA that something is called the lightning indexer, and it is a tiny attention shaped scorer with 32 heads of 128 dimensions. It ranks all previous tokens and hands real attention the top 2,048, which the config calls index_topk.

There is a catch. The indexer is the one part of DSA that must still look at the whole context for every token. Attention got cheap, but the thing choosing what attention reads did not.

So, here comes the shared index to the rescue. GLM 5 gave every layer its own indexer. GLM 5.3 keeps one in 21 layers only, and the 57 layers in between reuse the most recent pick, which the config writes as index_topk_freq: 4. We checked that in the checkpoint instead of trusting the config, and exactly 22 layers carry indexer tensors: the 21 main layers plus the draft layer, which reuses the last pick. The other 57 layers have none at all. So the full context scan drops from 78 passes to 21, which is 3.7 times less, and the indexer's own key cache now covers 21 layers instead of 78.

Flash cuts the same cost again. Three config keys and two tensors describe a pooling stage that squeezes every 4 indexer keys into 1 before ranking them, index_kpool: 4 with index_kpool_compress: true, while index_kpool_always_select_tail: true keeps the newest tokens from ever being pooled away. That divides both the scan and the indexer cache by 4.

How Do We Get the KV Cache per Token?

This is the number people actually plan hardware around, and we can derive it from the config alone.

MLA does not cache keys and values for each head. It caches one shared latent for each token in each layer, and rebuilds the heads from it. So the cache is layers times latent width, and nothing else.

GLM 5.3   78 layers x (512 latent + 64 RoPE)  =  44,928
          21 layers x 128 indexer keys        =   2,688
                                                -------
                                                 47,616 elements  ->  47.6 KB per token

Flash     11 layers x 512 latent              =   5,632
          11 layers x 128 / 4 pooling         =     352
                                                -------
                                                  5,984 elements  ->   6.0 KB per token

                                          47,616 / 5,984  =  8.0x less

Our checkpoint level calculation gives 8.0 times less cache for each token. It assumes the pooling compresses the stored indexer keys and not only the scan, which we read from index_kpool_compress: true and the two pooling tensors. Without that assumption the ratio is 6.8 times.

Z.ai reports about 4.4 times smaller KV cache. We could not reproduce that measurement, because the serving method behind it is not in the material we read. So these two numbers should not be treated as competing measurements of the same thing. Ours is a cache calculation from the checkpoint, theirs comes from a different method, and both agree on the reason: only 11 of 45 layers pay for each token at all.

For scale, GLM 5.3's 47.6 KB for each token is still cheap next to the 256 KB we measured for Granite 4.2 30B's dense layers on this rig, in our Granite 4.2 benchmark. MLA plus FP8 is doing a lot of work even in the big model.

Where Do the Parameters Live?

Both models are overwhelmingly mixture of experts. Routed experts hold 734.44B of GLM 5.3, which is 97.5%, and 311.65B of Flash, which is 97.0%. Attention, KDA included, is the largest remainder at 13.04B and 6.09B. Everything else in this teardown, the indexers, the hyper connections, the routers, adds up to less than half a percent between them. The mechanisms are clever and nearly free. The weight is all in the experts.

One GLM 5.3 layer stores 256 experts of 37.75M, which is 9.66B. A token wakes 8 routed experts plus 1 shared one, so 0.34B runs. One Flash layer stores 288 experts of 25.17M, which is 7.25B, and 0.23B runs. Flash uses more experts that are each smaller, which gives its router more combinations to pick from for each token. Routing is identical in both models, scored with a sigmoid and balanced by noaux_tc with no auxiliary loss.

What Else Changed in Flash?

Flash's 11 full attention layers dropped position encoding. The config says qk_rope_head_dim: 0 and mla_use_nope: true, and the shapes confirm it. Flash's kv_a_proj_with_mqa is [512, 4096], the bare latent, while the big model's is [576, 6144], which is 512 latent plus a 64 wide RoPE tail. The KDA layers already carry word order through their convolutions and their running notepad, and the indexer still applies RoPE to its own keys, so the attention path does not need it. There is also no RoPE base left to retune when the window grows.

Flash also replaces the single residual stream with four. hc_mult: 4 widens the stream to four parallel copies of 4,096 dims, and six small tensors in each layer compute a 4 by 4 mixing matrix plus 4 read and 4 write weights for every token. hc_sinkhorn_iters: 20 pushes that matrix toward doubly stochastic, so every row and column sums to 1 and nothing is multiplied up or drained away across 45 layers. The whole mechanism is 35M parameters, which is 0.011% of the model, so it has little to do with the memory story here. We saw four streams before in Qwen 3.8 Flash Next.

Flash is multimodal and GLM 5.3 is text only. The vision tower is compact at 0.56B, 24 blocks at hidden 1,024, taking 448 by 448 images cut into 14 by 14 patches, with temporal_patch_size: 2 packing two video frames into one patch.

Both models ship exactly one draft layer, which the config calls num_nextn_predict_layers. A draft layer guesses the next few tokens cheaply, and the full model then checks the guesses in one pass. These are not small heads: 9.95B in GLM 5.3 and 7.43B in Flash, each a full transformer block with its own experts and its own sparse attention. We measured what a draft head is really worth on hardware in our llama.cpp speed settings post, and the answer was that it depends entirely on how predictable the text is. We have run neither GLM model, so acceptance rate and any speedup are unknown here.

Precision is the last tell. Both checkpoints are natively FP8 in the e4m3 format, with 128 by 128 block scales, but Flash's entire KDA stack stays BF16. Its attention group holds 552 BF16 tensors against only 44 FP8 ones. The delta rule recurrence is too precision sensitive to quantize, which is the same reason A_log, dt_bias and the hyper connection base and scale tensors sit in F32. A running notepad accumulates error at every step, and a cache does not.

What Does Z.ai Report?

Z.ai reports strong agentic scores for Flash against the previous flagship GLM 5.2, Terminal-Bench 2.1 at 84.3 against 81.0 and Toolathlon at 78.4 against 59.9 among them. These are vendor numbers and we measured none of them. One correction we had to make while sourcing them: a deep dive prints those six Flash comparison numbers against a column labelled GLM 5.3, but llm-stats and the Hugging Face card agree they are GLM 5.2's scores, since GLM 5.3 scores 66.9 on DeepSWE and not 46.2.

Can We Run Either Model on One RTX 5090?

Not as shipped. The FP8 checkpoints are 755.6 GB and 328.3 GB, and our card holds 32 GB.

The interesting difference shows up at inference time. GLM 5.3's cache alone reaches about 49.9 GB at 1M tokens, which is more than the whole card holds. Flash's 6.3 GB sits in a corner of it, and Flash wakes only 16.9B parameters for each token. Those two numbers point at the recipe we have used before, a 1 to 2 bit build with the experts pushed out to system memory, which is exactly what we did in Run Qwen 3.8 Flash Next 180B on CPU only.

GGUF conversions of Flash already exist. The thing to check before we spend benchmark time is whether llama.cpp can load a hybrid KDA plus DSA stack at all, because a missing kernel is not a tuning problem.

Method and Limits

We read the config files and the safetensors headers of both checkpoints, and we did not download or run the weights. Every parameter count and cache size here is arithmetic from those files, and every benchmark score is the vendor's own. Our 8.0 times cache ratio assumes the indexer pooling compresses the stored keys and not only the scan, and it is 6.8 times without that assumption. Active parameter counts depend on where the LM head, the embeddings and the draft layer are counted, so we show our own definition next to the vendor's. Cache precision is a serving choice, so the table gives both the FP8 and the BF16 latent variants. Release dates come from third party coverage, since the Hugging Face cards cite the February 2026 GLM 5 paper instead.

Conclusion

We read both GLM 5.3 checkpoints from their config files and their safetensors headers, without downloading a terabyte of weights. The big model turned out to be GLM 5 with the context dial turned to 1M and one new trick. Flash turned out to be a genuinely new stack.

Key takeaways:

• Flash keeps full attention in only 11 of 45 layers. The other 34 are KDA linear layers that hold a fixed 71 MB notepad instead of a growing cache.
• That layout gives 6.0 KB of cache for each token against GLM 5.3's 47.6 KB, which is 8.0 times less by our derivation, against the 4.4 times Z.ai reports from a method we could not reproduce.
• GLM 5.3 keeps a lightning indexer in only 21 of its 78 layers, and the other 57 reuse the most recent pick, which cuts the one pass that still reads the whole context by 3.7 times. Flash pools its indexer keys 4 into 1 on top of that.
• Both models are about 97% experts, and both run about 5% of their weights for each token. GLM 5.3 runs 40.5B of 753B, and Flash runs 16.9B of 321B.
• Flash drops position encoding from its attention layers entirely, and lets the linear layers carry word order instead.

Next steps:

• Read the Qwen 3.8 Flash Next teardown for the same four streams and sparse attention in another vendor's stack.
• See what a 1 bit build of a big sparse model really does on one desktop in DeepSeek V4 Flash vs Qwen 3.8 Flash Next at 1-bit.
• Use the local LLMs technical reference guide to match a model and a context size to our own card.

This is how GLM 5.3 and GLM 5.3 Flash are built. We started with two config files and 194,737 tensor shapes, and we finished with the one number that separates them: 47.6 KB of cache for each token against 6.0 KB.