Granite 4.2 vs Gemma 4 vs Qwen 3.8 Benchmark

Granite 4.0 was IBM's architectural bet. It replaced most of the attention layers with Mamba-2 state space layers and was sold on a claim of cutting memory use by more than 70% for long context and concurrent work. Granite 4.1 kept going. Granite 4.2 landed on Ollama a few days ago.

We pulled it, opened the file, and the Mamba is gone.

In this blog, we will measure what that costs. Granite 4.2 is the model under test at both of its sizes. The other four models are the measuring stick, there to tell us whether a Granite number is good, bad, or ordinary. Six models on one RTX 5090 with 32 GB, all of them Q4_K_M, all of them driven by the same llama-server binary, in two brackets.

Setup: One Card, One Runtime, Six Models

The small bracket is matched on memory, not on parameters. 8B against 12B against 9B is not a like for like size fight and we are not going to pretend it is. The big bracket is the clean one: three dense reasoning models within 1.5 GB of each other.

Everything below ran on one machine in one session, so no number here is compared against something measured on a different day.

• GPU: RTX 5090, 32 GB VRAM, full GPU offload
• Runtime: llama-server build b10448 with CUDA on Windows 11
• Quantization: Q4_K_M for all six models, so quant class is never a confound
• KV cache: f16 with flash attention on, prompt caching off
• Sampling: each model uses the settings its own card recommends
• Grading: plain Python functions and executed asserts, no judge model anywhere

llama-server.exe -m granite-4.2-30b-Q4_K_M.gguf --host 127.0.0.1 --port 8099 -c 8192 -ngl 999 -fa on -np 1 --jinja --no-warmup --metrics

GGUF Teardown: The Mamba Is Gone

A GGUF file lists every tensor it contains, with its name, its shape, and its data type. We do not have to believe a model card. We parsed the headers of all six models off our own disk and counted what kind of block each layer actually is.

from gguf import GGUFReader

r = GGUFReader("granite-4.2-30b-Q4_K_M.gguf")
names = [t.name for t in r.tensors]
print(r.fields["general.architecture"].contents())
print("tensors:", len(names))
print("ssm:", sum("ssm_" in n for n in names))
print("experts:", sum("exp" in n for n in names))
print("mtp:", sum("nextn" in n for n in names))

granite
tensors: 579
ssm: 0
experts: 0
mtp: 0

Granite 4.2 declares general.architecture = granite, and nothing in either file is an SSM tensor, an expert tensor, or a speculative head. Forty blocks in the 8B and sixty-four in the 30B, and every single one of them is a full attention block with a KV cache that grows with context.

Its competitors both spend their layers very differently, and neither of them does it the way we expected.

Gemma 4 interleaves sliding window attention 5 to 1. Only 8 of the 12B's 48 blocks and 10 of the 31B's 60 blocks are full attention. The rest are capped at a 1024 token window, so past that window they stop growing.

Qwen 3.8 and Ornith 1.5 are hybrids, and this one corrected a note we had written in an earlier round. We had recorded that the qwen35 family "caches KV on only a quarter of its layers" and filed it as undocumented KV sharing. It is not sharing. Three out of every four blocks carry ssm_* tensors and hold a fixed size recurrent state instead of a KV cache. Qwen 3.8 has 16 full attention blocks out of 64, and Ornith 9B has 8 out of 32. The attention layers sit at indices 3, 7, 11, 15 and so on, a clean 3 to 1 interleave.

Here, we can see the irony. The model whose family made hybrids famous is the only one in this line-up without one. Gemma got there with sliding windows, Qwen got there with SSM layers, and Granite 4.2 pays full price on every layer.

KV Cache Is What That Costs

The KV cache is the memory the model keeps for every token already in context. It is the number that decides how much context your card can hold.

We derived this from tensor shapes, then checked it against what llama-server reports allocating at two different context sizes and took the slope. Derived and measured agree exactly for all six models.

Read the second column, then read the size of your card. Granite 4.2 30B needs 34.4 GB of KV cache to reach the context length printed on its own model page, before we load its 17.7 GB of weights. That is not a tight fit on a 32 GB card. It is not a fit at all.

The 8B is the same story in miniature. 160 KiB per token is ten times what Gemma 4 12B charges and five times what Ornith 9B charges, from the model with the fewest parameters in its bracket.

Granite 4.2 30B Reaches 45% of Its Own Advertised Context

So "the biggest context that loads" is not a measurable quantity on this rig, because the answer is always yes. What is measurable is the decode rate cliff: the largest context where the model still runs within 90% of its own small context speed.

def usable_ceiling(model, floor_fraction=0.90):
    """Largest context where decode stays within 90% of the small-context rate."""
    floor = decode_at(model, ctx=8192) * floor_fraction
    best, ctx = 8192, 65536
    while ctx <= 262144:
        if decode_at(model, ctx) < floor:
            break
        best, ctx = ctx, ctx * 2
    return best

Two models in the matched bracket, near identical weight size, and a 4.2 times difference in usable context. Granite 4.2 30B gets 58k where Qwen 3.8 27B gets 246k.

The drop is a cliff. Granite 30B runs at 78.6 tokens a second at a context of 58,368, and at 15.1 tokens a second at 61,440. A 5% increase in context costs 81% of the throughput. Both measurements look perfectly healthy on their own, and they contradict each other. That is the trap: if you sized your context by trying a number and seeing that it worked, you would never learn which side of the cliff you had landed on.

Granite 4.2 8B is the good news in this section. It reaches its full 128k, and all three small models deliver their advertised number. They just pay wildly different prices for it. At full context the card reads 26,565 MiB in use for Granite 4.2 8B at 128k, against 14,389 MiB for Ornith 9B and 12,977 MiB for Gemma 4 12B at twice that context.

Decode Speed: Granite Is Fast Per Token

Single stream, 512 tokens with ignore_eos so every model produces exactly the same count, five content types, then the whole ladder run again in reverse as a drift control.

Granite is the fastest model in each bracket, and in the small bracket it is not close. The reverse order control matters on this card, which is power capped rather than thermally capped. An hour into a session it is drawing 590 W against a 600 W limit and clocking down. Every number above was measured back to back in one session, and the forward and reverse ladders agree within 0.3%, so none of this is drift.

Granite's Tie With Qwen Only Exists With a Switch Turned Off

That table has Granite 4.2 30B at 78.5 tokens a second and Qwen 3.8 27B at 78.3, which reads as a dead heat. It is not one, and the reason is a tensor Granite does not have.

Qwen 3.8 and Ornith 1.5 both carry a working nextn multi-token prediction head. Granite and Gemma do not, so a like for like comparison has to turn it off, and that is what the table above does. But nobody runs a model with a free speedup disabled, so here is the same measurement with the head enabled, paired off and on in one session.

llama-server.exe -m Qwen3.8-27B-Q4_K_M.gguf -c 8192 -ngl 999 -fa on --jinja --spec-type draft-mtp --spec-draft-n-max 4

This changes the ranking in the matched bracket completely. On MTP off numbers Granite 4.2 30B and Qwen 3.8 27B are tied at about 78 tokens a second. With its head on, Qwen runs at 154 to 182 tokens a second on code and structured output, roughly twice Granite's rate, and Granite has no equivalent switch to flip.

Three things are worth knowing before turning it on.

The gain is entirely content dependent, and it tracks the acceptance rate. Structured and predictable text drafts well, so counting and JSON accept 84 to 86% of drafted tokens. Prose does not, and essays accept 26 to 31%. The speedup follows that curve exactly.

On prose it can be a net loss. Ornith 9B is slower with MTP on for essay generation, 176.8 against 207.8 tokens a second, because it pays to draft tokens that get rejected three times out of four.

It is not a free speedup, it is a different configuration. MTP off output is reproducible bit for bit on this build, while MTP on output differed in all ten content and model combinations we tested. Turning it on changes what the model writes, not just how fast it writes it.

Granite Pays That Speed Back in Tokens

Being fast per token is not the same as being fast to an answer, and this is where Granite gets expensive.

We built a new problem set for this round, because the one from the last round saturated. Five models scored 39 out of 40 on it, and it measured nothing but cost. This set is 13 math items whose answers come from brute force solvers committed next to the questions, 7 items with no valid answer where the only correct reply is IMPOSSIBLE, and 8 format items graded by a Python predicate.

Granite broke the harness before it produced a single score. Our 16k output cap was too small for it. On one tier 3 item, Granite 4.2 8B solved the problem after 21,327 output tokens. On another it produced 33,437. So we re-ran the whole thing with a 32k cap and recorded, for every run, how many tokens it actually took.

Both Granite models sit in the middle of this field, and they get there the expensive way. Granite 30B lands on 16 of 20, within two of Qwen 3.8, but it needs the full 32k budget to do it while Qwen has 11 of those 20 finished inside 2,000 tokens. Granite 8B is worse off still, at 3 of 20 inside 2,000 tokens against 9 for Ornith 1.5 9B in the same bracket.

Gemma 4 31B is the other shape on this chart. It is mid pack at 6 of 20 inside 2,000 tokens, and then it takes everything once it has a 16k budget, the only clean sweep in the benchmark. Granite 4.2 8B climbs steeply too, from 3 to 15, so budget clearly buys it something. The difference is where the climb stops: Gemma reaches 20 and both Granite models flatten out at 15 and 16.

Let me tabulate the final accuracy for your better understanding, with truncations counted separately from wrong answers.

We checked whether the truncation was our sampling rather than the model. Granite ships one sampling profile in its Ollama Modelfile at temperature 1.0 and top_p 0.95, while IBM's standing guidance for Granite 4 is temperature 0, and the Ollama page text quotes a third setting. All three truncate on the same items, and greedy is the worst of the three.

It is not sampling. Granite 4.2 simply thinks for a long time.

Code: The 8B Beats the 30B

Nine coding tasks with 99 hidden assertions, executed in a separate interpreter. These are deliberately not interview classics, because those saturate. They are precise specifications with edge cases that are stated and easy to skip: strict Roman numerals that must reject IIII, INI parsing with line continuations and duplicate keys, RFC style CSV quoting, negative business day offsets. Our reference implementations pass all 99 assertions, checked before any model ran.

Granite 4.2 8B swept the coding set and Granite 4.2 30B did not. The 30B dropped parse_ini at 2 of 12 assertions, plus one assertion of wrap_lines. On this task set the extra 21 billion parameters bought nothing, and the smaller model was better at following a fiddly written specification.

Agentic Tool Use: Granite Ties With Everyone

Eight tasks with real tool calling: pagination that punishes answering from the first page, a tool that rejects a malformed key once and explains the fix, and two tasks whose data does not exist so the only correct answer is UNAVAILABLE.

We will report this the way it came out. Granite 4.2 30B scored 8 out of 8, and so did four other models, so this suite tells us nothing about Granite except that it is not broken. Pagination and forced error recovery, the two mechanisms added specifically to make it harder, separated nobody. The only thing that touched Granite at all was the long horizon probe task, which Granite 4.2 8B is the only model in the group to fail.

One result here is genuinely clean, and Granite is part of it: zero malformed tool calls out of 312 calls across all six models. Granite emits an XML style wrapper rather than a JSON object, which looked like the odd one out until we read the templates. It is not. Granite, Qwen 3.8 and Ornith 1.5 all ship the same wrapper, and Gemma is the one model here using a different syntax.

<tool_call><function=open_manifest><parameter=record_id>0042</parameter></function></tool_call>

Granite's Reasoning Effort Dial Is a Two Position Switch

Granite 4.2 advertises configurable reasoning effort. We have been burned by this claim before, on a different model whose runtime silently never set the template variable at all. So this time we settled it by rendering the prompt and comparing bytes rather than guessing from the output.

import requests

msg = [{"role": "user", "content": "What is 2+2?"}]
for effort in [None, "low", "medium", "high", "max"]:
    body = {"messages": msg}
    if effort:
        body["chat_template_kwargs"] = {"reasoning_effort": effort}
    r = requests.post("http://127.0.0.1:8099/apply-template", json=body)
    print(effort, len(r.json()["prompt"].encode()))

None 102
low 127
medium 102
high 102
max 102

The template itself explains why. It contains exactly one line that reads the setting.

{%- set low_effort = low_effort if low_effort is defined else False %}
{%- if reasoning_effort is defined and reasoning_effort is not none %}
    {%- set low_effort = reasoning_effort == "low" %}
{%- endif %}

There is no scale. There is low, there is everything else, and low is implemented by pasting a sentence of plain text onto the end of your message. Values like medium and max are accepted silently and do nothing at all, which is worse than rejecting them, because a config file full of reasoning_effort: high looks like it is doing something.

To Ollama's credit, its model page describes exactly this: two levels, low and high, with high as the default. The mismatch is with our expectation of what an effort level is, not with the documentation.

Where Granite 4.2 Belongs

Granite 4.2 8B is the one to reach for. It swept the coding set at 9 of 9 tasks and all 99 assertions, it is the fastest model we have measured in its class at 214 tokens a second, and it reaches its full advertised 128k. Two conditions come attached. That 128k costs 21.5 GB of KV cache, so the full window is only reachable on a big card, and the model will spend five figures of output tokens on a hard problem. For fiddly specification work on a small footprint, we would run it.

Granite 4.2 30B is harder to place, and on this card we would not run it for long context work. It gets 58k of usable context where Qwen 3.8 27B gets 246k at the same weight size, because it pays full attention cost on all 64 layers while Qwen pays it on 16. It also lost the coding set to its own 8B sibling, at 7 of 9 against 9 of 9. Where it holds up is short context reasoning: 9 of 13 on math, all 7 traps caught, all 8 format items, and an 8 out of 8 on tool use.

The two models that beat Granite do it on different axes, and it is worth knowing which one you are up against. Gemma 4 31B is the accuracy winner and the only model that swept the math set, but it needs a 16k thinking budget to do it and it is the slowest model here. Qwen 3.8 27B is the efficiency winner: 11 of 20 problems solved inside 2,000 tokens, 94% of an advertised 256k context actually usable, a 450 token median, and with its MTP head on it decodes at roughly twice Granite's rate on code and structured output.

So, back to the question this benchmark started with. Granite 4.0 promised 70% less memory from its hybrid design, and Granite 4.2 gave that back. Every number in the KV cache and context sections is what dropping the hybrid costs. Whether the quality gains were worth it depends on which table you read, which is why we published all of them.

One Thing We Got Wrong

We wrote two bad trap items. All six models answered our train chase problem with 7:00 pm, and they were right: "leaves the station" implies a shared origin and direction, which makes the problem well posed. We had marked it unanswerable. We also marked Granite wrong for answering that the probability of an impossible draw is 0, which is a defensible answer rather than a hallucination.

Both graders were corrected and the results were rescored from the stored answers. Rescoring is a pure function of the recorded output, so no model was re-run to produce a better score.

This is how a local benchmark works. Measure what the hardware actually does, name the number the model card leaves out, and say plainly which of your own graders turned out to be wrong.