Why LLMs Can't Count the R's in Strawberry: Tokenization Quirks

Ask a large language model how many times the letter R appears in "strawberry" and it may confidently answer two. The correct answer is three. The same model can write a working program and explain quantum tunnelling, yet it fumbles a question a six-year-old gets right.

This is not a reasoning failure. It is a tokenization failure, and once you see why, a whole family of strange model behaviours suddenly makes sense.

The Model Never Sees Letters

Here is the root of it. By the time text reaches the model, it is no longer text. It is a list of token IDs, and each ID is just a row number.

The word "strawberry" might arrive as three tokens. The model receives something like [496, 675, 15717]. Those numbers carry meaning learned from training, but they do not carry spelling. Nothing in the input says "this piece contains two R's".

# what the model actually receives
# "strawberry" -> ["str", "aw", "berry"] -> [496, 675, 15717]
# the letters are gone by this point

An Analogy: Reading Through Frosted Glass

Imagine someone hands you words printed on cards, but the cards are frosted. You can tell one card from another, and you have learned what each card means from seeing them used millions of times.

Now someone asks you to count the letter R on card number three. You know the card means "berry". You might remember that berry has two R's, because you have read about spelling. But you cannot look and count, because you cannot see through the glass.

That is the model's situation exactly. It can recall facts about spelling; it cannot inspect the letters.

Why the Model Still Sometimes Gets It Right

If the letters are invisible, how does a model ever answer spelling questions correctly? Because spelling appears in its training data as text.

Sentences like "strawberry is spelled s-t-r-a-w-b-e-r-r-y" teach the relationship between a token and its letters. So the model has second-hand knowledge of spelling. That knowledge is patchy, which is why it is right about common words and wrong about odd ones.

Why Splitting Is Inconsistent

There is a second twist. The same letters can be split differently depending on what surrounds them, including whether there is a space in front.

"strawberry" at the start of a sentence and " strawberry" mid-sentence can become different token sequences. So the model's grip on a word's spelling can change with position.

The Same Problem in Arithmetic

Numbers suffer from the same issue. A tokenizer does not know that digits are special. It splits numbers into whatever chunks its vocabulary happens to contain.

So 12345 might become "123" and "45", while 12346 becomes "12" and "346". The model is being asked to add things that were cut into inconsistent pieces, which is why long multiplication often goes wrong even when the reasoning is sound.

Why Rhyming and Wordplay Wobble

Rhyme lives in sound, and sound lives in letters. A model working with chunks has only indirect access to both.

Puns, anagrams, acrostics, and counting syllables all ask the model to manipulate a level of structure that its input does not contain. It can often fake it from memory, and it fails in ways that look careless rather than systematic.

The Simple Workaround

There is an easy fix, and it works because it changes what the model receives.

Ask the model to spell the word out with separators first, then count. Writing "s - t - r - a - w - b - e - r - r - y" forces each letter into its own token, and suddenly the letters are visible. This is why chain-of-thought prompting helps here: it converts a hidden problem into a visible one.

# force the letters into separate tokens, then count
# "spell strawberry letter by letter, then count the r's"
# s - t - r - a - w - b - e - r - r - y   -> now each letter is its own token

Why Models Are Not Simply Built on Letters

If letters cause so much trouble, why not tokenize every character? Because the cost is brutal.

Character tokens make sequences four or five times longer, and attention cost grows with the square of length. You would pay enormously on every task to fix a small family of puzzles. The trade is not worth it, so the quirk stays.

A Quirk, Not a Ceiling

It is worth being precise about what this does and does not prove. The strawberry failure says nothing about whether a model can reason. It says the model was handed a question about a level of detail its input throws away.

Give it the letters and it counts fine. That is a strong hint that this is a plumbing problem, not a thinking problem.

Recap

This is why a capable model miscounts the R's in strawberry. The word arrives as a few chunks, and those chunks are converted to numbers before the model sees anything. Spelling is thrown away in that step, so the model can only recall what it has read about letters rather than look at them.

The same root cause explains shaky arithmetic, wobbly rhymes, and failed anagrams: all of them need a level of detail that subword tokens do not carry. Asking the model to spell a word out first fixes it, because that puts each letter into its own token.

We keep subwords anyway, because character tokens would make every sequence several times longer and attention several times more expensive. That closes our nine days on tokenization. In the next article, we follow the token IDs into the model itself and meet embeddings, the step that turns those row numbers into meaning.