Z.ai shipped two models under one version number in August 2026, and they are not the same animal. GLM 5.3 is the big text model: 753B parameters, 78 layers, a 1M token window. GLM 5.3 Flash is a new base model at 321B, it reads images and video, and it is the first GLM to mix linear attention with sparse full attention in one stack.
Neither one fits on our RTX 5090. So we did the thing we could do. We pulled the config files, then read the JSON header of every safetensors shard over the network. A shard header lists the name, the shape and the data type of each tensor without sending a single weight. That gave us the complete parts list of both models: 118,629 tensors for GLM 5.3 and 76,108 for Flash.
In this blog, we will open both checkpoints and find out where every parameter sits, what the four new mechanisms actually are in the weights, and why the smaller model keeps eight times less cache for each token.
One line before we start, because it decides how to read everything below. Nothing here ran on our rig. Every parameter count and cache size is derived from the raw files, and it is exact. Every benchmark score is Z.ai's own number, and we label it as such every time it appears.
What Are These Two Models?
GLM 5.3 was announced on 14 August 2026 and Flash on 26 August 2026, per third party launch coverage. The big model reuses the GLM 5.2 base with better post training. Flash is a fresh model, and it carries all the new ideas.
Let me tabulate the two checkpoints for your better understanding. Every row here comes from the raw files, not from a model card.
| GLM 5.3 | GLM 5.3 Flash | |
|---|---|---|
| Total parameters, counted from tensor shapes | 753.33B | 321.32B |
| ...of which the MTP draft layer | 9.95B | 7.43B |
| ...of which the vision tower | none | 0.56B |
| Main stack | 743.4B, the vendor rounds to 744B | 313.3B |
| Active for each token, our derivation | 40.5B | 16.9B, the vendor says 18B |
| Layers in the main stack | 78, all DSA full attention | 45: 34 KDA linear + 11 DSA |
| Hidden size | 6,144 | 4,096 |
| Attention heads | 64 | 64 |
| Mixture of experts | 256 routed + 1 shared, top 8 | 288 routed + 1 shared, top 8 |
| One expert, 3 x hidden x 2048 | 37.75M | 25.17M |
| Dense stem | first 3 layers | first 3 layers |
Context, max_position_embeddings |
1,048,576 | 1,048,576 |
| Position encoding | partial RoPE, 64 of 256 dims, theta 8M | NoPE in the attention layers |
| Vocabulary | 154,880 | 154,880 |
| Reads images | no | yes, image and video |
| Hyper connections | no | yes, 4 streams |
| Checkpoint at FP8 | 755.6 GB, 118,629 tensors | 328.3 GB, 76,108 tensors |
| Architecture id | GlmMoeDsaForCausalLM |
Glm5NextForConditionalGeneration |
Here, we can see the number that matters most. Both models keep about 95% of their weights asleep for any single token. GLM 5.3 runs 5.4% of itself, and Flash runs 5.3%. Capacity and cost for each token stopped being the same number a while ago, and these two models are built on that gap.
How Much of GLM 5 Does GLM 5.3 Keep?
Before the interesting model, the short story of the big one. We diffed the config of zai-org/GLM-5 against zai-org/GLM-5.3. The skeleton did not move at all: 78 layers, hidden 6,144, 64 heads, 256 experts, the same MLA ranks, the same vocabulary.
Exactly three things changed.
| Config key | GLM 5 | GLM 5.3 |
|---|---|---|
max_position_embeddings |
202,752 | 1,048,576 |
rope_theta |
1,000,000 | 8,000,000 |
| Indexer scheme | one indexer per layer | 21 "full" plus 57 "shared", index_topk_freq: 4 |
The window went from 200k to 1M, and the RoPE base went up 8 times to carry it. That is the whole architecture story of GLM 5.3, and it matches the model card, which says the gains come from post training. The third row is genuinely new, and we cover it below. Everything else that is new in this release lives in Flash.
How Are the Layers Arranged?
Let's look at both stacks side by side.
GLM 5.3 stacks 78 identical DSA blocks. DSA stands for DeepSeek Sparse Attention, and we met it in our DeepSeek V4 Flash post. In simple words, it is normal attention with a filter in front. A small scorer ranks every earlier token, and real attention only reads the top 2,048 of them. Layers 0 to 2 have plain dense feed forward layers, layers 3 to 77 are mixture of experts, and layer 78 is a draft layer for speculative decoding.
Flash runs a strict 3 to 1 rhythm. Layers 3, 7, 11 and so on up to 43 are DSA blocks, which is 11 of them. The other 34 layers are KDA linear attention. Layers 0 to 2 are dense again, everything above is mixture of experts, and layer 45 is the draft layer.
That single rhythm is where nearly all of Flash's long context saving comes from. Only 11 of its 45 layers ever grow a KV cache. The KV cache is the memory a model keeps for every token in the chat, so it is the thing that gets more expensive as the conversation gets longer.
What Is the Lightning Indexer, and Why Is It Shared?
Sparse attention needs something to decide what is worth reading. In DSA that something is called the lightning indexer, and it is a tiny attention shaped scorer with 32 heads of 128 dimensions. It ranks all previous tokens and hands the real attention the top 2,048, which the config calls index_topk.
There is a catch, and it is the reason this section exists. The indexer is the one part of DSA that must still look at the whole context for every token. Attention got cheap, but the thing choosing what attention reads did not.
So, here comes the shared index to the rescue. GLM 5 gave every layer its own indexer. GLM 5.3 keeps an indexer in only 21 layers, and the 57 layers in between reuse the most recent pick.
We can read the pattern straight out of the config. The indexer_types list marks layers 0, 1, 2 and then every 4th layer from 6 to 74 as "full", and the rest as "shared". That is index_topk_freq: 4, one fresh pick every 4 layers, with index_skip_topk_offset: 3 keeping the first three layers out of the sharing.
We checked this against the checkpoint instead of trusting the config. Exactly 22 layers carry indexer.wk, indexer.wq_b and indexer.weights_proj tensors: the 21 main layers plus the draft layer, which reuses the last pick through index_share_for_mtp_iteration: true. The other 57 layers have no indexer tensors at all.
Two things get cheaper. The full context scan drops from 78 passes to 21, which is 3.7 times less. And the indexer's own key cache, at 128 numbers for each token, now covers 21 layers instead of 78.
The weight cost is small and easy to see in the shapes. A layer that owns its indexer spends 174.4M parameters on attention. A layer that borrows one spends 165.0M. The indexer itself is 9.4M.
What Is Inside a KDA Linear Attention Layer?
Now the part that makes Flash a different model. Its 34 linear layers use Kimi Delta Attention, or KDA. A linear attention layer does not keep a growing cache. It keeps one fixed size memory and edits it as each token arrives, the way we would keep one notepad instead of a folder of every page we ever wrote.
Here is one layer as it sits in the checkpoint, all in BF16.
| Tensor | Shape | What it does |
|---|---|---|
q_proj, k_proj, v_proj |
[8192, 4096] each | 64 heads of 128 dims |
q_conv1d, k_conv1d, v_conv1d |
[8192, 1, 4] each | short depthwise conv, kernel 4 |
f_a_proj / f_b_proj |
[128, 4096] / [8192, 128] | low rank forget gate |
g_a_proj / g_b_proj |
[128, 4096] / [8192, 128] | low rank output gate |
b_proj |
[64, 4096] | write strength for each head |
A_log (F32) |
[64] | decay for each head |
dt_bias (F32) |
[8192] | time step bias for each channel |
o_norm |
[128] | output RMSNorm for each head |
o_proj |
[4096, 8192] | output |
Let's read that list as one idea. The three conv1d tensors give each channel a four token window, so the layer sees local order without any position encoding. The forget gate decides how much of the old notepad to keep, and the write strength decides how hard the new token writes into it. A_log is the decay rate for each head, which is why it stays in float32.
The notepad itself is 64 heads times 128 by 128 numbers, so 1,048,576 elements for each layer. Across all 34 linear layers that is 35.65M elements, which is about 71 MB in BF16 for one sequence. That number does not change when the chat gets longer. A conversation of 10 tokens and a conversation of a million tokens cost the same 71 MB in these layers.
One surprise falls out of the shapes. A KDA layer holds 137.7M parameters and a DSA layer holds 124.9M, so the linear layer is the bigger of the two. Linear attention here spends weights to save cache and compute. It does not save bytes on disk.
Why Did the Attention Layers Drop Position Encoding?
Flash's 11 full attention layers are MLA, the same latent cache idea the big model uses. Two things about them are new, and both are visible in the config and the shapes.
The first is that position encoding is gone. The config says qk_rope_head_dim: 0 and mla_use_nope: true, which is NoPE, short for no positional encoding. We can confirm it in the tensor shapes. Flash's kv_a_proj_with_mqa is [512, 4096], exactly the 512 wide latent with nothing added. The big model's is [576, 6144], which is 512 latent plus a 64 wide RoPE tail.
So how does Flash know word order? Through the KDA layers. Their four token convolutions and their running notepad carry order for free, so the attention layers do not need to. The indexer still applies RoPE to its own keys through indexer_rope_interleave: true, so the scorer keeps a sense of position. NoPE is also what makes a 1M window cheap to extend, because there is no RoPE base left to retune in the attention path.
The second change is a pooling stage in front of the indexer, and it does not exist in the big model at all. Three config keys and two tensors describe it: index_kpool: 4, index_kpool_compress: true, an index_kpool_compress_gate of [128, 4096] and a learned position embedding index_kpool_compress_ape of [4, 128]. The scorer squeezes every 4 indexer keys into 1 before ranking them, and index_kpool_always_select_tail: true keeps the newest tokens always selected so a fresh token is never pooled away. That cuts both the indexer's scan and its cache by 4 times.
Head geometry differs too. The big model splits each query and key into 192 NoPE dims plus 64 RoPE dims, with 256 dim values. Flash uses a flat 256 NoPE dims for both, which we can read off kv_b_proj at [32768, 512], since 32,768 is 64 heads times 256 keys plus 256 values.
What Are Manifold Constrained Hyper Connections?
Every Flash layer except the draft layer carries six small tensors: hc_attn_base of [24], hc_attn_fn of [24, 16384], hc_attn_scale of [3], and the same three again for the feed forward half.
The residual stream is the running hidden state that every block adds its result back into. In simple words, it is the model's shared notebook. GLM 5.3 has one notebook, like most transformers. Flash has four, because hc_mult: 4 widens the stream to 4 parallel copies of 4,096 dims. That is 16,384, which is exactly the input width of the hc_*_fn tensors.
The 24 numbers those tensors produce are the mixing recipe for the four streams: a 4 by 4 matrix that mixes the streams, plus 4 numbers to read and 4 to write, so 16 plus 4 plus 4. They are computed from the data itself and added to a fixed base, so the mixing changes for every token.
The "manifold constrained" part is the interesting bit, and it is one config key: hc_sinkhorn_iters: 20. The 4 by 4 mixing matrix is pushed toward a doubly stochastic matrix by 20 Sinkhorn steps. A doubly stochastic matrix has rows and columns that each add up to 1, so nothing gets multiplied up or drained away as the token climbs 45 layers. That is the constraint, and it is what keeps four notebooks stable at depth.
The whole mechanism costs 35M parameters, which is 0.011% of the model, across 270 tensors in 45 layers. We saw the same family of ideas in Qwen 3.8 Flash Next, where four streams also replaced one. The Sinkhorn constraint and the per sublayer recipe are the new parts here.
One difference from Qwen is worth a line. In Flash Next the hyper connection replaced the two LayerNorms in each block. In GLM 5.3 Flash the checkpoint still carries input_layernorm and post_attention_layernorm in every layer, so the four streams sit on top of the norms rather than in place of them.
Where Do the Parameters Live?
We grouped every tensor in both checkpoints by what it belongs to. Here is the complete census. FP8 scale tensors are left out, and they only add 0.046B and 0.019B.
| Component | GLM 5.3 | GLM 5.3 Flash |
|---|---|---|
| Routed experts | 734.44B (97.5%) | 311.65B (97.0%) |
| Attention, KDA included | 13.04B | 6.09B |
| Shared experts | 2.87B | 1.08B |
| Embeddings | 0.95B | 0.63B |
| LM head | 0.95B | 0.63B |
| Dense MLP stem, 3 layers | 0.68B | 0.45B |
| Vision tower | none | 0.56B |
| DSA indexers | 0.21B | 0.09B |
| Routers | 0.12B | 0.05B |
| MTP glue | 0.08B | 0.03B |
| Hyper connections | none | 0.035B |
| Norms | 0.001B | 0.0004B |
| Total | 753.33B | 321.32B |
Here, we can see that both models are about 97% experts. Everything else that we spent five sections on, the indexers, the hyper connections, the routers, adds up to less than half a percent between them. The mechanisms are clever and nearly free. The weight is all in the experts.
Let's zoom into one mixture of experts layer to see what that means for a token.
One GLM 5.3 layer stores 256 experts of 37.75M, which is 9.66B. A token wakes 8 routed experts plus 1 shared one, so 0.34B runs. One Flash layer stores 288 experts of 25.17M, which is 7.25B, and 0.23B runs. Flash uses more experts that are each smaller, which gives its router more combinations to pick from for each token.
Routing is identical in both models, and it is the setup that has become standard. Experts are scored with a sigmoid, and noaux_tc keeps them balanced without an auxiliary loss. Each expert carries a learned e_score_correction_bias, routed_scaling_factor is 2.5, and the router itself stays in float32.
Two published numbers need reconciling, and the census does it. Hugging Face shows 753B for the big model because it counts every shard, the 9.95B draft layer included. The vendor says 744B because it leaves the draft layer out, and our count of the main stack is 743.4B. Flash's "320B" works the same way: 313.3B main stack plus 7.43B draft layer plus 0.56B vision tower is 321.32B.
Active parameters need one honest note. Our definition is everything outside the experts that a token touches, plus 8 routed and 1 shared expert in each mixture of experts layer, plus the LM head. That gives 40.5B and 16.9B. The vendor's "40B" matches ours if they drop the LM head. Their "18B" for Flash matches if they instead count the embeddings and the draft layer's active path, since 16.9 plus 0.63 plus about 0.4 is 18.0. We cannot tell which they did, so we show our own definition and leave the gap visible.
How Do We Get the KV Cache per Token?
This is the number people actually plan hardware around, and we can derive it from the config alone.
MLA does not cache keys and values for each head. It caches one shared latent for each token in each layer, and rebuilds the heads from it. So the cache is layers times latent width, and nothing else.
| GLM 5.3 | GLM 5.3 Flash | |
|---|---|---|
| Attention latent | 78 layers x (512 + 64 RoPE) = 44,928 | 11 layers x 512 = 5,632 |
| Indexer keys | 21 layers x 128 = 2,688 | 11 layers x 128 / 4 pooling = 352 |
| Elements for each token | 47,616 | 5,984 |
| FP8 cache | 47.6 KB per token | 6.0 KB per token |
| BF16 latent, FP8 index | 92.5 KB per token | 11.6 KB per token |
| At 1M tokens, FP8 | 49.9 GB | 6.3 GB, plus 0.07 GB of KDA state |
So Flash caches 8.0 times less for each token. If the pooling does not compress the stored indexer keys, only the scan, then it is 6.8 times. We derived the 4 times from index_kpool_compress: true and the two pooling tensors, not from a running server, so we give both numbers.
Z.ai's own launch claim is the more careful "about 4.4 times smaller KV cache". The gap fits a vendor measuring total serving memory at a moderate context, where Flash's fixed 71 MB of KDA state and the allocator's own overhead dilute the ratio. We cannot check their method. Both numbers agree on the direction and on the reason: only 11 of 45 layers pay for each token at all.
For scale, GLM 5.3's 47.6 KB for each token is still cheap next to the 256 KB we measured for Granite 4.2 30B's dense layers on this rig, in our Granite 4.2 benchmark. MLA plus FP8 is doing a lot of work even in the big model. But 1M tokens still needs about 49.9 GB of cache, which is more than an RTX 5090 holds in total, while Flash's 6.3 GB sits in a corner of it.
Which Change Causes the Speedup?
Three separate mechanisms are at work, and they are not equal.
The first is the layer rhythm, and it is the big one. 34 of 45 layers stopped growing state at all. That is where the cache table above comes from, and it is most of the vendor's claim of about 3.0 times less attention compute. Our parameter level view agrees with the direction: only 11 layers still do work that depends on how long the chat is, and even those read at most 2,048 selected tokens.
The second is sparse attention itself. DSA caps the attended set at 2,048 tokens in the layers that are still quadratic in shape. GLM 5.3's shared index cuts the one remaining full context pass from 78 to 21, and Flash cuts its own by a further 4 times with pooling.
The third is the mixture of experts, which is not new in 5.3 but explains the headline. It keeps the work for each token at about 5% of the stored weights in both models. That is why an 18B active model can carry 321B of knowledge.
Hyper connections are the odd one out. They cost 0.011% of the weights, and the vendor credits them with better scaling, but nothing about them makes inference faster. They are a training time aid, and we have no way to isolate what they contributed from outside the lab.
What Do the Draft Layers Do?
Both models ship exactly one draft layer, which the config calls num_nextn_predict_layers. A draft layer guesses the next few tokens cheaply, and the full model then checks all the guesses in one pass. When the guesses are right we get several tokens for the price of one.
These are not small heads. Each is a full transformer block with its own mixture of experts, 256 experts in the big model and 288 in Flash, its own sparse attention with its own indexer, and the usual enorm, hnorm and eh_proj glue. That is 9.95B and 7.43B of parameters spent on guessing.
One detail is easy to miss. Flash's draft layer has no hyper connection tensors, because those exist on layers 0 to 44 only. So the draft path runs on a plain single residual stream while the main stack runs on four.
We measured what a draft head is worth on hardware in our llama.cpp speed settings post, and the answer was that it depends entirely on how predictable the text is. We also measured that it is not always a free switch. Under greedy decoding Qwen's in file draft head gave byte identical output, but with sampling on the text differed in all ten cases we tried. Nothing here changes that, and we have run neither GLM model, so treat acceptance and speedup as unknown for these two.
What Is in the Vision Tower?
Only Flash has one, and it is compact at 0.56B. GLM 5.3 has no vision tensors of any kind, so it is text only.
The tower is 24 blocks at hidden 1,024 with 16 heads, and each head has its own q and k norms. The feed forward layers are SwiGLU at 4,096 wide with swiglu_limit: 10. Images come in at 448 by 448 and are cut into 14 by 14 patches, with temporal_patch_size: 2 packing two video frames into one patch. A 2 by 2 downsample conv and a merger MLP at 10,240 wide then project everything into the 4,096 dim text stream. Images and video get their own token ids, 154854 and 154855, plus begin and end markers.
The tower stays in BF16 even though the rest of the checkpoint is FP8.
What Stays in High Precision?
Both checkpoints are natively FP8 in the e4m3 format, with 128 by 128 block scales and dynamic activation scaling. What the vendor refused to quantize is the interesting part.
In GLM 5.3, every expert and attention matmul is FP8. The norms, router gates, correction biases, embeddings, LM head and MTP glue stay in BF16 or F32.
In Flash, the entire KDA stack is BF16. Its attention group holds 552 BF16 tensors against only 44 FP8 ones, and the FP8 ones are all in the 11 sparse attention layers. Even there, kv_b_proj stays BF16, while the same tensor is FP8 in the big model.
The pattern is clear enough to state. The delta rule recurrence is too precision sensitive to quantize, which is the same reason A_log, dt_bias and the hyper connection base and scale tensors sit in F32. A running notepad accumulates error at every step, and a cache does not.
What Does Z.ai Report?
These are the vendor's own scores. We did not measure any of them.
Flash against the previous flagship GLM 5.2: Terminal-Bench 2.1 at 84.3 against 81.0, Toolathlon at 78.4 against 59.9, DeepSWE v1.1 at 63.4 against 46.2, NL2Repo at 56.3 against 48.9, AutomationBench at 48.8 against 26.2, and Agents' Last Exam at 26.3 against 20.4.
An 18B active hybrid beating last generation's 32B active flagship across an agentic suite is the vendor's core claim. Coverage adds that it approaches Claude Opus 4.8 on several agentic rows at a tenth of the price, while losing NL2Repo at 56.3 against 69.7.
The big model's card highlights Terminal Bench 2.1 at 88.2, CyberGym at 84.5, which is above DeepSeek V4's 83.3, DeepSWE v1.1 at 66.9 and ExploitBench at 54.4. It trails GPT-5.6 Sol on most rows and leads on CyberGym.
One correction we had to make while sourcing these. One deep dive prints the six Flash comparison numbers against a column labelled GLM 5.3, but llm-stats and the Hugging Face card agree those are GLM 5.2's scores, since GLM 5.3 scores 66.9 on DeepSWE and not 46.2. We used the GLM 5.2 labelling.
Can We Run Either Model on One RTX 5090?
Not as shipped. The FP8 checkpoints are 755.6 GB and 328.3 GB, and our card holds 32 GB.
Flash is the one with a realistic local path. At 321B it is heavier than the 180B Qwen 3.8 Flash Next we already ran. But it wakes only 18B for each token, and its cache is 6.3 GB even at 1M. Those two numbers point at the recipe we have used before: a 1 to 2 bit build with the experts pushed out to system memory. That is exactly what we did for Flash Next and DeepSeek V4 in Run Qwen 3.8 Flash Next 180B on CPU only.
GGUF conversions of Flash already exist. The thing to check before we spend benchmark time is whether llama.cpp can load a hybrid KDA plus DSA stack at all, because a missing kernel is not a tuning problem.
The run we want to do is clear. Decode rate against --n-cpu-moe, the effect of a growing chat on decode speed, where there should be no cliff at all in 34 of the 45 layers, and the acceptance rate of that 7.43B draft layer.
Step back from the two checkpoints and the direction is hard to miss. Latent caches from DeepSeek, content selection by a lightning indexer, a 3 to 1 linear attention mix from Kimi Linear and Qwen Flash Next, a widened residual stream, sigmoid routing with no auxiliary loss, and one draft layer as standard equipment. Flash is the first checkpoint we have opened that carries all six at once, and it adds Sinkhorn constrained mixing and indexer key pooling on top.
What We Did Not Test
- Nothing was executed. Every parameter count and cache size is arithmetic from the raw files, and every benchmark score and efficiency factor is the vendor's own, unverified by us.
- Our 8.0 times cache ratio assumes the indexer's pooling compresses the stored keys and not only the scan. Without that assumption it is 6.8 times. The vendor's 4.4 times comes from a method they have not published, and we could not reconcile the two.
- Active parameter counts depend on where the LM head, the embeddings and the draft layer are counted. We show our definition and both readings that agree with the vendor.
- The FP8 ignore list in GLM 5.3's config names 22
self_attn.indexers_projmodules that do not exist anywhere in the checkpoint. It is a harmless artifact, and worth knowing about if we write conversion tooling. - KV cache precision is a serving choice, so we give both the FP8 and the BF16 latent variants.
- The 71 MB KDA state is the notepad only. Conv states and the number of speculative tokens a server uses are runtime details, not checkpoint facts.
- Release dates come from third party coverage. The Hugging Face cards cite the February 2026 GLM 5 paper, which is the paper's date and not the release.
Conclusion
We read both GLM 5.3 checkpoints from their config files and their safetensors headers, without downloading a terabyte of weights. The big model turned out to be GLM 5 with the context dial turned to 1M and one new trick. Flash turned out to be a genuinely new stack.
Key takeaways:
- Both models are about 97% experts, and both run about 5% of their weights for each token. GLM 5.3 runs 40.5B of 753B, and Flash runs 16.9B of 321B.
- GLM 5.3 keeps a lightning indexer in only 21 of its 78 layers, and the other 57 reuse the most recent pick. That cuts the one pass that still has to read the whole context by 3.7 times.
- Flash keeps full attention in only 11 of 45 layers. The other 34 are KDA linear layers that hold a fixed 71 MB notepad instead of a growing cache.
- That layout gives 6.0 KB of cache for each token against GLM 5.3's 47.6 KB, which is 8.0 times less by our derivation and 4.4 times by the vendor's measurement.
- Flash drops position encoding from its attention layers entirely, and lets the linear layers carry word order instead.
- Four residual streams, mixed by a 4 by 4 matrix that 20 Sinkhorn steps push toward doubly stochastic, cost 0.011% of the weights.
Next steps:
- Read the Qwen 3.8 Flash Next teardown for the same four streams and sparse attention in another vendor's stack.
- See what a 1 bit build of a big sparse model really does on one desktop in DeepSeek V4 Flash vs Qwen 3.8 Flash Next at 1-bit.
- Use the local LLMs technical reference guide to match a model and a context size to our own card.
This is how GLM 5.3 and GLM 5.3 Flash are built. We started with two config files and 194,737 tensor shapes, we found where all 753B and 321B of parameters sit, and we finished with the one number that separates the two models: 47.6 KB of cache for each token against 6.0 KB.
