Granite 4.0 was IBM's big architectural bet. Most of its attention layers were replaced with Mamba-2 state space layers. IBM said this cut memory use by more than 70%. Granite 4.1 kept the same design. Granite 4.2 arrived on Ollama a few days ago.
We downloaded it and opened the model file. The Mamba-2 layers are gone.
In this blog, we will find out what that costs. We ran Granite 4.2 against the models people compare it with. All of them face the same memory problem. Gemma 4 solves it with sliding windows. Qwen 3.8 and Ornith 1.5 solve it with SSM layers. Granite 4.2 no longer solves it at all.
Here is the plan. First we open the model files and look at the layers. Then we measure memory cost per token, and turn that into a real context limit. Then speed. Then accuracy on math, code and tool use.
What We Tested
| Group | Models | VRAM for weights |
|---|---|---|
| Small | Granite 4.2 8B, Gemma 4 12B, Ornith 1.5 9B | 5.3 / 7.4 / 5.8 GB |
| Big | Granite 4.2 30B, Gemma 4 31B, Qwen 3.8 27B | 17.7 / 18.3 / 16.8 GB |
We made the two groups by VRAM, not by parameter count. Three models use about 5 to 7 GB. Three use about 17 to 18 GB. What matters is which models fit the same card.
Everything ran on one machine in one session. So no number here is compared against a different day.
- GPU: RTX 5090, 32 GB, all layers on the GPU
- Runtime:
llama-serverbuild b10448 with CUDA on Windows 11 - Quantization: Q4_K_M for all six, so quantization is never the reason one model looks better
- KV cache: f16, flash attention on, prompt caching off
- Sampling: each model uses the settings its own card recommends
- Server flags: same for every model, one slot, no warmup, chat templates on
- Grading: plain Python and real test runs, no judge model anywhere
Important
We used Ollama only to download the weights, never to run them. Ollama starts this same server with its own hidden defaults. So an Ollama speed number tells us about those defaults, not about the model.
What Is Inside the GGUF Files?
A GGUF file is how a model is stored on disk. It begins with a list of every tensor inside it: the name, the shape, and the data type. So we do not have to trust a model card. We can read the file and count.
We did that for all six models. An SSM layer always carries tensors named ssm_, and a draft head always carries tensors named nextn. Granite 4.2 has zero of both, in the 8B and in the 30B.
So every Granite block is a full attention block. A full attention block keeps a cache that grows as our context grows. The 8B has 40 of them. The 30B has 64. All of them cost memory.
The other models do it differently.
Gemma 4 uses sliding window attention on most blocks. A sliding window block only looks at the last 1024 tokens. So its memory stops growing after that. Only 8 of the 12B's 48 blocks are full attention. In the 31B it is 10 out of 60.
Qwen 3.8 and Ornith 1.5 use SSM layers. An SSM layer keeps a small state of fixed size instead of a growing cache. Three out of every four blocks are SSM. Qwen 3.8 has 16 full attention blocks out of 64. Ornith 9B has 8 out of 32.
Here, we can see the irony. The family that made hybrid models famous is now the only one here without a hybrid.
How Much Does the KV Cache Cost per Token?
The KV cache is memory the model keeps for every token in our context. The longer the chat, the bigger it gets. This one number decides our real context limit.
We worked this out from the tensor shapes. Then we checked it against what the server really allocates. Both agree exactly, for all six models.
| Model | Cache per token | Cache at its advertised context |
|---|---|---|
| Granite 4.2 8B | 160 KiB | 21.5 GB at 128k |
| Granite 4.2 30B | 256 KiB | 34.4 GB at 128k |
| Gemma 4 12B | 16 KiB | 4.6 GB at 256k |
| Gemma 4 31B | 80 KiB | 22.3 GB at 256k |
| Qwen 3.8 27B | 64 KiB | 17.2 GB at 256k |
| Ornith 1.5 9B | 32 KiB | 8.6 GB at 256k |
Now let's read the second column, then look at our GPU. Granite 4.2 30B needs 34.4 GB of cache to reach the 128k context printed on its own model page. Its weights need another 17.7 GB. A 32 GB card cannot do this. It is not even close.
The 8B has the same problem in smaller form. It costs 160 KiB per token. Gemma 4 12B costs 16 KiB. That is ten times more, from the smaller model.
How Much Context Can Each Model Really Use?
Caution
On Windows, a CUDA allocation bigger than your VRAM does not fail. The driver quietly uses system RAM instead. The server starts, reports the full context, and never warns you.
So we cannot ask how much context loads. On Windows, everything loads. We ask a better question. How much context can the model hold before it slows down?
We call that the usable context. First we measure the model's speed at a small context. Then we grow the context until speed falls below 90% of that. The last size that passed is the usable context.
| Model | Advertised | Really usable | Share |
|---|---|---|---|
| Granite 4.2 30B | 128k | 58,368 | 45% |
| Gemma 4 31B | 256k | 155,648 | 59% |
| Qwen 3.8 27B | 256k | 245,760 | 94% |
| Granite 4.2 8B | 128k | 131,072 | 100% |
| Gemma 4 12B | 256k | 262,144 | 100% |
| Ornith 1.5 9B | 256k | 262,144 | 100% |
Two models of almost the same size, and a 4.2 times gap. Granite 4.2 30B gets 58k. Qwen 3.8 27B gets 246k.
The drop is sudden. At 58,368 tokens, Granite 30B runs at 78.6 tokens a second. At 61,440 tokens it runs at 15.1. That is 5% more context for 81% less speed.
Both numbers look fine on their own. That is the trap. If we pick a context size and it works, we still do not know how close we are to the edge.
Granite 4.2 8B is the good news here. It reaches its full 128k. Gemma 4 12B and Ornith 1.5 9B reach their full 256k. But the cost is very different. Granite 8B uses 26,565 MiB at 128k. Ornith uses 14,389 MiB at twice that context.
Which Model Writes Tokens Fastest?
Every model wrote exactly 512 tokens, on five kinds of content. Then we ran the whole test again in reverse order, to check the card was not slowing down. Forward and reverse agree within 0.3%.
| Model | Tokens per second |
|---|---|
| Granite 4.2 8B | 214 |
| Ornith 1.5 9B | 192 |
| Gemma 4 12B | 144 |
| Granite 4.2 30B | 78.5 |
| Qwen 3.8 27B | 78.3 |
| Gemma 4 31B | 70.5 |
Granite is the fastest model in both groups. In the small group it is not close. It writes 214 tokens a second, against 192 for Ornith and 144 for Gemma 4 12B.
In the big group all three land within 8 tokens a second of each other. That near tie is the subject of the next section.
Important
These numbers have the draft head turned off. That is the only setting all six models share. Two of them have a draft head and run about twice as fast with it on.
Is the Granite vs Qwen Speed Tie Real?
The table above puts Granite 30B at 78.5 and Qwen 3.8 at 78.3. That looks like a dead heat. It is not.
A draft head is a small extra layer inside the model. It guesses the next few tokens. The full model then checks all the guesses at once and keeps the ones it agrees with. When the guesses are good, the model runs much faster.
Qwen 3.8 and Ornith 1.5 have a draft head. Granite and Gemma do not. So a fair table has to switch it off, and that is what the table above does. But nobody runs a model with a free speedup disabled. So we measured it both ways, turning the head on with the --spec-type draft-mtp flag.
Qwen 3.8 27B, draft head off and on:
| Content | Off | On | Gain | Guesses accepted |
|---|---|---|---|---|
| counting | 76.4 | 179.6 | 2.35x | 84% |
| json | 76.6 | 181.6 | 2.37x | 86% |
| code | 76.2 | 153.7 | 2.02x | 68% |
| fiction | 76.2 | 103.9 | 1.36x | 39% |
| essay | 76.2 | 93.2 | 1.22x | 31% |
Ornith 1.5 9B, the same test:
| Content | Off | On | Gain | Guesses accepted |
|---|---|---|---|---|
| counting | 208.5 | 351.8 | 1.69x | 81% |
| code | 208.6 | 281.5 | 1.35x | 58% |
| json | 208.1 | 231.2 | 1.11x | 43% |
| fiction | 208.6 | 214.6 | 1.03x | 38% |
| essay | 207.8 | 176.8 | 0.85x | 26% |
This changes the big group completely. With its draft head on, Qwen writes code at 154 tokens a second and JSON at 182. That is about twice Granite's rate. Granite has no such switch.
Two things to know before turning it on.
The gain depends on what we write. Let's look at the last column. Counting and JSON accept 84 to 86% of the guesses, and those get the big speedups. Essays accept only 26 to 31%, and barely gain.
It can also make things slower. Ornith 9B writes essays at 176.8 tokens a second with the draft head on, against 207.8 with it off. Most of its guesses were wrong, and checking wrong guesses costs time.
Warning
A draft head also changes the text. With it off, two runs give identical output. With it on, the output differed in all ten cases we tested. Treat it as a different setting, not a free speedup.
How Many Tokens Does Each Model Need per Answer?
Writing tokens quickly is not the same as answering quickly. This is where Granite gets expensive.
We built a new problem set for this round. The old one was too easy: five models scored 39 out of 40, so it measured nothing. The new set has 13 math problems, 7 problems with no valid answer, and 8 formatting tasks. Every math answer was computed by a brute force program first, so the answer key is not our opinion.
Granite ran past our token limit before it finished a single problem. On one hard problem, Granite 4.2 8B needed 21,327 tokens to reach the answer. On another it needed 33,437. So we raised the limit to 32,000 and recorded how many tokens each model actually used.
| Model | 2k | 4k | 8k | 16k | 32k |
|---|---|---|---|---|---|
| Granite 4.2 8B | 3/20 | 5/20 | 10/20 | 13/20 | 15/20 |
| Gemma 4 12B | 2/20 | 3/20 | 4/20 | 9/20 | 13/20 |
| Ornith 1.5 9B | 9/20 | 10/20 | 11/20 | 15/20 | 16/20 |
| Granite 4.2 30B | 8/20 | 10/20 | 12/20 | 14/20 | 16/20 |
| Gemma 4 31B | 6/20 | 10/20 | 12/20 | 20/20 | 20/20 |
| Qwen 3.8 27B | 11/20 | 12/20 | 14/20 | 16/20 | 18/20 |
Both Granite models land in the middle, and they get there the expensive way. Granite 30B solves 16 of 20, close to Qwen's 18. But it needs the full 32,000 tokens to do it. Qwen finishes 11 problems inside 2,000 tokens.
Granite 8B is slower to start. It solves 3 problems inside 2,000 tokens. Ornith 1.5 9B, in the same group, solves 9.
Gemma 4 31B has a different shape. It is average at small budgets, at 6 of 20. Give it 16,000 tokens and it takes everything. That is the only clean sweep in this benchmark.
Let me tabulate the final scores for your better understanding.
| Model | Math | Traps caught | Format | Ran out of room | Median tokens |
|---|---|---|---|---|---|
| Granite 4.2 8B | 9/13 | 6/7 | 8/8 | 3 | 6,533 |
| Gemma 4 12B | 7/13 | 6/7 | 8/8 | 6 | 11,036 |
| Ornith 1.5 9B | 9/13 | 7/7 | 7/8 | 3 | 1,618 |
| Granite 4.2 30B | 9/13 | 7/7 | 8/8 | 3 | 5,240 |
| Gemma 4 31B | 13/13 | 7/7 | 8/8 | 0 | 2,657 |
| Qwen 3.8 27B | 11/13 | 7/7 | 8/8 | 2 | 450 |
"Ran out of room" means the model was still working when it hit the token limit. That is not the same as a wrong answer, so we counted them apart.
We also checked whether our sampling settings caused the long answers. IBM, the Ollama Modelfile, and the Ollama page each suggest different settings. We tried all three. All three run just as long. So it is not sampling. Granite 4.2 simply thinks for a long time.
How Well Does Each Model Code?
Nine coding tasks with 99 hidden tests. The model writes a function, and we run the tests against it in a separate Python process.
These are not interview puzzles, because those are too easy now. They are precise specs with edge cases that are easy to miss. Roman numerals that must reject IIII. INI files with line continuations and duplicate keys. CSV quoting. Negative business day offsets. We wrote our own solutions first and confirmed they pass all 99 tests.
| Model | Tasks passed | Tests passed | Median tokens |
|---|---|---|---|
| Granite 4.2 8B | 9/9 | 100% | 11,947 |
| Gemma 4 31B | 9/9 | 100% | 3,168 |
| Qwen 3.8 27B | 9/9 | 100% | 7,126 |
| Gemma 4 12B | 8/9 | 91% | 11,336 |
| Granite 4.2 30B | 7/9 | 90% | 6,844 |
| Ornith 1.5 9B | 6/9 | 78% | 5,005 |
Granite 4.2 8B passed everything. Granite 4.2 30B did not. The 30B failed 2 of 12 tests on the INI parser, and 1 test on line wrapping. Here the extra 21 billion parameters bought nothing.
But Granite 8B did not sweep alone. Gemma 4 31B and Qwen 3.8 27B also passed all nine tasks, and much more cheaply. Gemma needed 3,168 tokens and Qwen needed 7,126, against Granite's 11,947. Granite 8B writes almost four times as much as Gemma to reach the same score.
Ornith 1.5 9B is the weak one here at 6 of 9. That is where Granite 8B earns its place in the small group.
How Well Does Each Model Use Tools?
Eight tasks with real tool calls. One task pages through data, so answering from the first page is wrong. One tool rejects a bad key once and explains the fix. Two tasks ask for data that does not exist, so the only right answer is UNAVAILABLE.
| Model | Solvable | Traps | Bad calls | Recovered from error |
|---|---|---|---|---|
| Granite 4.2 8B | 5/6 | 2/2 | 0 | 3/4 |
| Gemma 4 12B | 6/6 | 1/2 | 0 | 3/4 |
| Ornith 1.5 9B | 6/6 | 2/2 | 0 | 5/5 |
| Granite 4.2 30B | 6/6 | 2/2 | 0 | 3/4 |
| Gemma 4 31B | 6/6 | 2/2 | 0 | 4/4 |
| Qwen 3.8 27B | 6/6 | 2/2 | 0 | 4/4 |
We will report this the way it came out. Five of six models scored 8 out of 8, and Granite 4.2 30B is one of them. This test did not separate them.
Only two things moved at all. Granite 4.2 8B is the one model that failed the long task. Gemma 4 12B is the one model that made up an answer instead of saying UNAVAILABLE.
One result is clean. Across 312 tool calls, not one was malformed. Granite writes its calls in an XML style instead of JSON, which looked unusual until we read the templates. It is not unusual. Qwen and Ornith use the same style, and Gemma is the odd one out.
Does Granite's Reasoning Effort Setting Work?
Granite 4.2 says we can adjust its reasoning effort. We have been fooled by this kind of claim before, so we did not judge it from the output. Instead we asked the server to build the prompt, and we compared the bytes it produced.
We sent the same short question five times and changed only the effort setting.
| Effort setting | Prompt size | Difference |
|---|---|---|
| not set | 102 bytes | this is the baseline |
medium |
102 bytes | identical to the baseline |
high |
102 bytes | identical to the baseline |
max |
102 bytes | identical to the baseline |
low |
127 bytes | adds {reasoning effort: low} to your message |
Only low changes anything. Setting medium, high or max gives a prompt that is identical to sending nothing at all.
The template explains why. Exactly one line reads the setting, and all it does is check whether the value equals low.
There is no scale. There is low, and there is everything else. Values like medium and max are accepted in silence and do nothing. That is worse than an error, because a config file full of reasoning_effort: high looks like it is working.
To Ollama's credit, its model page says exactly this: two levels. The documentation is right. The phrase "effort level" is what misleads.
Which Model Should We Run?
Granite 4.2 8B is worth running. It passed every coding task, it is the fastest model in its group at 214 tokens a second, and it reaches its full 128k context. Two catches. That 128k needs 21.5 GB of cache, so only a big card can reach it. And it will write tens of thousands of tokens on a hard problem.
Granite 4.2 30B is harder to recommend. We would not use it for long context work on this card. It gets 58k where Qwen gets 246k at the same size. It also lost the coding test to its own 8B. It is fine for short work: 9 of 13 on math, all 7 traps, all 8 formatting tasks, and 8 of 8 on tools.
Qwen 3.8 27B is the model to beat, and Granite does not beat it. It solves 11 of 20 problems inside 2,000 tokens. Its median answer is 450 tokens. It keeps 94% of its advertised context. With its draft head on, it writes code twice as fast as Granite.
Gemma 4 31B is the most accurate. It is the only model that got all 13 math problems right, with nothing cut off. It pays twice for that. It needs a 16,000 token budget, and it is the slowest model here at 70.5 tokens a second.
Ornith 1.5 9B is the quiet one. Its median answer is 1,618 tokens, against 6,533 for Granite 8B. It holds a full 256k context. It recovered from every forced tool error. It only fell down on coding, at 6 of 9.
| Model | Tokens per second | Usable context | Best at |
|---|---|---|---|
| Granite 4.2 8B | 214 | 131,072 | Careful coding on a small card |
| Granite 4.2 30B | 78.5 | 58,368 | Short context work only |
| Ornith 1.5 9B | 192 | 262,144 | Short answers, cheap cache |
| Gemma 4 12B | 144 | 262,144 | The cheapest context here, at 16 KiB per token |
| Qwen 3.8 27B | 78.3 | 245,760 | Long context, and anything you pay tokens for |
| Gemma 4 31B | 70.5 | 155,648 | Hard math, if you can wait |
So, back to the question we started with. Granite 4.0 promised 70% less memory. Granite 4.2 gave that back. The cache and context numbers above are the price of that decision.
What We Got Wrong
Two of our own test items were wrong, and the models were right.
We had marked a train problem as unanswerable, but it was not, and all six models solved it. We had also marked Granite wrong for saying that an impossible event has probability 0, which is a fair answer.
We fixed both graders and rescored from the saved answers. No model was re-run to get a better score.
Conclusion
We opened six GGUF files, counted their layers, and measured what Granite 4.2's move back to full attention really costs. The story is memory. Granite writes tokens fast, but its cache grows faster than any other model here, and that decides how much context it can really hold.
Key takeaways:
- Granite 4.2 dropped the Mamba-2 layers, so every block now pays for a full attention cache. The 30B costs 256 KiB per token, four times what Qwen 3.8 costs.
- That cache bill caps Granite 30B at 58k usable context on a 32 GB card, while Qwen 3.8 reaches 246k at the same size.
- Granite 4.2 8B is the bright spot. It is the fastest model in the small group at 214 tokens a second, and it passed all nine coding tasks.
- Speed per token is not speed per answer. Qwen's median answer is 450 tokens against thousands for Granite, and its draft head doubles its code speed on top.
- Only the
lowreasoning effort setting does anything on Granite 4.2. Themedium,highandmaxvalues are accepted in silence and change nothing.
Next steps:
- See the same measurements run on a new model family in the Ornith 1.5 9B vs 35B benchmark.
- Learn how to turn the draft head on and tune the other llama.cpp flags in Qwen 3.8 27B llama.cpp speed settings.
- Use the local LLMs technical reference guide to match a model and context size to our own GPU.
This is how Granite 4.2 measures up. We started inside the GGUF files where the Mamba layers used to be, we saw the cache bill that decision creates, and we finished with the numbers that say which model our card should run.
