Qwen 3.8 27B vs Muse Glimmer 30B vs Gemma 4 26B Consistency Test

Every local model comparison runs each question once, writes the score in a table, and calls it a benchmark. One run. That number hides the thing we care about most: does the model give us the same answer the second time we ask?

consistency@10 is the metric this test is built around. In simple words, we ask the same question ten times and count how often the answers agree.

In this blog, we will run Qwen 3.8 27B, Muse Glimmer 30B, and Gemma 4 26B through twelve generated ledger questions, ten repeats each, on one RTX 5090 through Ollama. That is 360 graded runs on the greedy setting alone, and 720 once we add each model's own shipped sampler.

One Run Per Question Is Not a Benchmark

Greedy decoding is supposed to be repeatable. We set the temperature to zero, the model always picks the most likely next token, and the same prompt should give the same text every time.

That is the theory. On a local machine it is not guaranteed. Batching, cache state, the runtime build, and a handful of silent defaults can all move the output.

So a single score is a weak claim. A model that is 85% correct but changes its answer on a third of the re-runs is a worse tool than a model that is 78% correct and never wavers. No published benchmark tells us which one we are holding.

Setup: Three Models on One RTX 5090

Three models in the same size class, all on one card, all through the same runtime.

• GPU: RTX 5090, 32GB VRAM, driver 610.88
• Runtime: Ollama 0.32.14, and only Ollama
• Quantization: Q4_K_M for all three
• Context: 8,192 tokens for every run
• Output ceiling: 2,560 tokens
• Repeats: 10 per item, per model, per sampler profile

We stayed on Ollama on purpose. Its own defaults are part of what we are measuring, because they are what most people actually run. Where a default is silent, we set it by hand and wrote it into the log instead of switching to another runtime to dodge it.

consistency@10 and the Twelve Ledger Questions

Each test item is a small accounts payable ledger plus one question. The answer is computed from the table, not written by hand:

| invoice | vendor | issued | terms | amount | paid |
| INV-1003 | Granite Courier | 2027-11-01 | Net 15 | 16705.40 | yes |
| INV-1000 | Fairmount Metals | 2027-11-09 | Net 45 | 8773.95  | no  |
   ... 11 more rows ...

What is the total amount of invoices issued by Alder Freight between
2028-01-21 and 2028-03-21, both dates inclusive?

ANSWER: <value>

Every item comes from a seeded generator, so nothing in the set is a puzzle a model may have memorised during training. There are five question types: filtered sums, due dates under Net terms, counting overdue invoices as of a date, finding the largest unpaid invoice, and totalling everything outstanding. A careful person with a calculator scores 100%.

We then report three numbers side by side:

• consistency@10: how often the ten repeats of one item agree with each other.
• Unanimous rate: the share of items where all ten repeats gave the same answer.
• Accuracy with a Wilson 95% interval, so a score of 12 out of 12 does not pretend to be exact.

Speed Knobs and Task Knobs

Here is the trap in every multi model comparison. If we let each model run at its own best settings, the fast one may just be the one we tuned. If we force identical settings on everything, we cripple whichever model has a real optimisation available.

Both are wrong, so every setting went into one of two buckets.

Speed knobs are per model and free to differ. Speculative decoding, layer offload, and cache type belong here, because that is what we can actually get out of each model on our own machine.

Task knobs are identical for everyone. Reasoning tier, output ceiling, context size, prompt, sampler profile, and quantization class belong here. Same task, same budget, same wording.

That split is only honest if a speed knob truly changes speed and nothing else. So we tested it instead of assuming it. We ran the same prompt under greedy at three drafting depths and hashed the output each time.

All three models returned byte identical text at every drafting level. That is what makes the per model settings legitimate. If one hash had differed, that knob would have moved into the locked bucket.

Ollama Ships a 2.3x Speed Flag on Qwen 3.8 27B Only

Speculative decoding in Ollama 0.32.14 is controlled by draft_num_predict, which is a request option and not a server environment variable. There is no OLLAMA_MTP in ollama serve --help. So we read what each model ships with:

ollama show --modelfile qwen3.8:latest

PARAMETER draft_num_predict 4
PARAMETER min_p 0
PARAMETER repeat_penalty 1
PARAMETER temperature 1
PARAMETER top_k 20
PARAMETER top_p 0.95

Here, we can see that Qwen 3.8 ships with drafting already switched on. Muse Glimmer and Gemma 4 ship with nothing.

Read that table again, because the consequence is bigger than the numbers. If we compare these three models at their shipped defaults, we are comparing one model with speculative decoding against two models without it, and nothing on screen tells us so.

That is not a model difference. It is a packaging difference, and it is worth 2.3x.

Muse Glimmer 30B Has No Draft Head Inside Ollama

Drafting did nothing for Muse, and we nearly published that as a fact about the model. It is a fact about Ollama.

We opened the GGUF file and read its tensor list directly. Muse has 52 blocks, indices 0 to 51, and no draft head of any kind. There is no nextn, no eh_proj, no shared_head. Qwen 3.8 carries exactly those tensors as a 65th block sitting behind its 64 transformer layers. So draft_num_predict on Muse was switching on a feature that is not in the file.

Muse does have a speculative path. It just ships as a separate file called DFlash, a 1.6 GB drafter GGUF that llama.cpp loads next to the model:

llama-server -m Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf -md dflash-Muse-Glimmer-30B-Q4_K_M.gguf -ngld 99

The muse-glimmer:latest package in Ollama is two blobs, the model and a vision projector. There is no drafter blob, and Ollama 0.32.14 has no flag for attaching one. The accelerator is sitting in a box that Ollama cannot open.

Now let us check our number against physics. Muse is 15.61 GB and dense. An RTX 5090 has roughly 1.79 TB/s of memory bandwidth, so if every token needs one full pass over the weights, decoding cannot go past about 107 tok/s. We measured 80.4, which is 75% of that ceiling. The vendor's own no drafter baseline on a 5090 is 74.9 tok/s, slightly below what we measured.

Two models on this same card go straight past 107 tok/s, so the card is not the limit. There are only two ways to beat that number. One is to check several drafted tokens in a single pass over the weights, which is what Qwen's draft head does. The other is to avoid reading most of the weights at all, which is what a sparse mixture of experts does. Muse can do the first. Ollama simply cannot ask it to.

Muse Glimmer 30B With DFlash on llama.cpp

So we downloaded the drafter and tested it under llama.cpp build 10448, pointed at the model blob Ollama had already stored so no second copy of 15.61 GB landed on disk.

Our first attempt understated it, and the reason is a good lesson. Passing -md <drafter> alone sends the drafter through llama.cpp's generic speculative path, where --spec-draft-n-max defaults to 3. DFlash drafts blocks of 16. llama.cpp has a dedicated --spec-type draft-dflash mode, and switching it on changes the picture completely.

The community reports of Muse passing 150 tok/s are real. Our median is 145.7, and on the prompt where drafting predicts well it holds 253 tok/s, above the vendor's own 233.4 claim.

Look at the spread as well: 133 to 253 tok/s at the same setting, changing only the prompt. Draft acceptance depends on how predictable the continuation is. So a single headline tok/s figure for a speculative setup means very little unless we say what was being written. Ours vary by nearly 2x across three ordinary prompts.

Depth stops paying off around 16, which matches DFlash's block size. Going to 24 buys another 0.5%.

So the honest version of our speed ranking is this. Qwen ships its accelerator inside the file, Muse ships it beside the file, and only one of those survives contact with Ollama. Muse is not the slow model. It is the model whose accelerator we had to go and fetch.

Reasoning Tiers: Qwen 3.8 27B vs Muse Glimmer 30B vs Gemma 4 26B

All three models accept think values of low, medium, high, true, and false. Setting them all to the same tier looks like parity. It is not.

Those are reasoning characters produced at each tier. Gemma 4 accepts the instruction and ignores it. Any benchmark that sets three models to medium and then compares wall time is measuring three vendors' private definitions of a word.

Then it got worse. On real ledger items at the low tier, Gemma 4 does not simply think for longer. It does not stop.

Raising the ceiling did not help. At 4,096 tokens it used all 4,096 and gave no answer. At 8,192 tokens it used 7,305, produced 15,754 reasoning characters, gave no answer, and returned an empty content field.

Reading the trace is the interesting part. It works out correctly that no invoice matches the filter, and then talks itself out of it:

*   None of these are Alder Freight.
*   Is it possible the question meant "total amount of invoices issued
    by *any* vendor between 2028-01-21 and 20

It re-derives, re-checks, and never commits. So the whole study runs with reasoning off, which is the one setting all three models honour identically. Every consistency number below is a no reasoning number, and that is a real limit on what we can claim.

consistency@10 Results on the Greedy and Default Samplers

Accuracy ranges are Wilson 95% intervals. That is 720 graded runs in total.

Greedy is the controlled case. The shipped default is what we actually get out of the box: temperature 1, with a top_k of 20 or 64 depending on the model, so the sampler really does draw from a distribution instead of taking the top token.

The second block is where we expected everything to fall apart. It mostly did not.

Muse Glimmer scored 100% on every measure with real sampling switched on. Twelve items, ten draws each, temperature 1, top_k 64, and it never once produced a different answer or a wrong one. We did not expect that from any model and we have no tidy explanation for it.

Gemma 4 dropped as predicted, from 99.2% to 93.3% agreement. Qwen slipped to 98.3%, and its slip is the interesting one. It did not give a different answer. It ran away, blowing through the 2,560 token ceiling on 2 of 120 runs and never reaching an answer line. Under greedy it never did that once.

Our founding suspicion was that greedy decoding drifts on local hardware. On this rig it mostly does not. Qwen 3.8 and Muse Glimmer each returned the identical answer on all 120 of their greedy runs.

That is a negative result and we are reporting it as one. It is also the useful half, because it comes with the configuration that bought it:

export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=f16
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_CONTEXT_LENGTH=8192
ollama serve

One model stays resident at a time, and we unload it before the next one loads. No request ever shares a batch with another. Fixed seed, temperature 0, top_k 1. None of those is the default. Reproducibility is something we configure, not something we are given.

Gemma 4 26B Broke Greedy Decoding on Item 9

One red square. That is the whole finding.

Gemma 4 broke on item 9, which asks for the total of all unpaid invoices:

The expected answer is 123910.49. Same prompt, same seed, same settings, two different replies.

Now look at where it happened. Not scattered randomly across the set. It happened on an item the model was already getting wrong, and it never wavered once on any of the ten items it answered correctly.

That is the pattern we would not have found any other way. Instability did not show up as background noise. It clustered exactly at the point where the model had lost the thread. A single run benchmark would have written down one wrong number there and moved on, never learning that the model was also unsure of itself.

Accuracy by Question Type

Gemma 4 26B's 83.3% is not spread thinly across the board. Every one of its 20 wrong runs falls on one question type.

Both instances failed:

• Item 4: expected 150530.55, answered 126529.47 on all ten repeats.
• Item 9: expected 123910.49, answered 123900.49 or 123901.49.

The second one is a ten unit slip on a six figure sum. That is the kind of error that survives a human plausibility check, which makes it the dangerous kind. Date arithmetic across month boundaries, filtering, counting, and finding a maximum are all clean. Adding a long column of numbers is not.

Decode Speed on Ollama and llama.cpp

This chart is in tokens per second and not seconds per item, and that choice is deliberate. On the same twelve items at the same locked settings, llama.cpp generated a median of 708 tokens against Ollama's 250. A wall clock bar would have compared two different amounts of work. Throughput per token normalises it, and the two runtimes agree closely with no drafter, 75 tok/s on Ollama against 79 on llama.cpp. That agreement is what lets us put the hatched bar on the same axis.

With its drafter attached, Muse goes from last place to first, at 278 tok/s against Qwen's 173.

DFlash Accuracy Cost: 3.5x Faster, 75% Correct

Before switching DFlash on, look at what it did to the answers.

We ran the same twelve ledger items twice under llama.cpp, once with the drafter and once without, changing nothing else:

That is 3.5x faster, and a quarter of the answers went wrong.

The failure mode is specific. On 3 of the 12 items, all ten repeats answered:

ANSWER: <value>

That is the literal placeholder from our own prompt template. The drafter proposes the template continuation it can see sitting in the context, and the checker accepts it. The same items are answered correctly on all three control runs. An exact speculative decoder cannot do this, because a drafted token is supposed to survive only if the full model would have written it anyway, and the full model writes a number.

Note the consistency@10 of 100%. It is not flaky. It is confidently and repeatably wrong, which is the exact combination this benchmark was built to expose, and it turned up in the one place we were not looking.

Two Sweeps We Threw Away

This is the part most benchmark posts leave out, so here it is.

The first complete sweep was discarded. It ran with a 1,024 token output ceiling, which looked generous because typical items finish in 155 to 365 tokens. But the "count the overdue invoices" question forces the model to walk the table row by row, and Qwen 3.8 hit the ceiling on 18% of its runs.

A truncated answer is not a wrong answer. Scoring it as one would have quietly punished the most thorough model. Worse, an 18% truncation rate looks exactly like an 18% accuracy deficit once it reaches a summary table. We raised the ceiling to 2,560 and reran everything from scratch for all three models.

Then we corrupted our own data. Mid run, a result row showed completion_tokens: 1024 when the configured ceiling was 2,560. That is impossible under the stated configuration. A kill command had silently failed, so two runner processes were writing to the same output file and one of them carried a stale config in memory.

Those corrupt rows were not detectably wrong from their own contents. The only thing that exposed them was a number that contradicted the configuration it claimed to come from, and the only reason we could see that is that every result row stamps the full options block it was produced under.

There was a third fix, in our own scoring code. Both consistency functions filtered out runs with no readable answer and treated them as missing data. On the item where Qwen ran away twice, those two runs simply vanished, and an 8 of 10 item was scored as unanimous. A run that fails to produce an answer is an outcome, not a gap in the data. It now counts as its own result, which is why Qwen reads 98.3% above instead of a tidy 100%.

Limits of This Test

• One task family. Ledger arithmetic only. Schema conformance, stacked instructions, abstention, and tool use are written but unrun.
• Twelve items. Gemma 4's entire accuracy deficit rests on 2 items, and its instability on 1.
• Reasoning is off, for the reason above.
• Everything ran at 8k context. Nothing here speaks to 32k or 110k.
• One rig, one runtime, one quantization.
• With two models at 100%, this task set is at ceiling and ranks nothing on accuracy. The next version needs bigger tables and more filter conditions.

Conclusion

We asked three local models the same twelve ledger questions ten times each on one RTX 5090, and measured agreement with themselves rather than a single score. Qwen 3.8 and Muse Glimmer answered all 120 of their greedy runs identically and correctly. Gemma 4 26B is roughly twice as fast as Qwen and it is the only one that cannot add a column of numbers.

Key takeaways:

• consistency@10 finds what a single run hides. Gemma 4's one unstable item was an item it was already answering wrongly, so instability marked the spot where the model had lost the thread.
• Ollama ships draft_num_predict 4 inside Qwen 3.8 and nothing inside the other two. A default comparison is a 2.3x packaging comparison, not a model comparison.
• Muse Glimmer is not slow. Its DFlash drafter is a separate file that Ollama cannot load, and with llama.cpp it goes from 75 tok/s to 278 tok/s.
• Speed that changes the answer is not free speed. DFlash on build 10448 cost 25% accuracy, and it was wrong the same way on all ten repeats.
• Reproducible greedy decoding is a configuration, not a property. Five environment variables and an explicit unload bought all 240 identical runs.

Next steps:

• Read our accuracy focused run, Qwen 3.8 27B vs Nemotron 3.5 vs Muse Glimmer, where these models are pushed on sixteen hard problems instead of ledger arithmetic.
• Tune the draft depth on your own card with the Qwen 3.8 27B speed settings on llama.cpp.
• Read the version teardown, Qwen 3.8 27B vs Qwen 3.6 27B vs Qwen 3.5 27B, to see why Qwen 3.8 27B ships that draft head switched on.
• Match a quantization and a context size to your GPU using the local LLMs technical reference guide.

This is how a consistency test works. We picked a metric that needs repeats, locked the task settings and freed the speed settings, ran the same twelve questions ten times each, and found that the interesting failure was not the wrong answer but the run that would not repeat itself.