Vocabulary Size vs Context Window: The Core Tokenization Trade-Off

Every tokenizer has one dial you can turn: how many tokens to keep in the vocabulary. Turn it up and each sentence needs fewer tokens. Turn it down and the model gets lighter.

You cannot have both. This is the central trade-off of tokenization, and it quietly shapes how long your prompts can be and how much memory a model needs.

The Same Sentence, Two Vocabularies

Start with the effect you can see. Take one sentence and tokenize it twice, once with a small vocabulary and once with a large one.

With a small vocabulary the model has few whole words, so it must chop text into many small pieces. With a large vocabulary many words already exist as single tokens, so the same sentence becomes a much shorter list.

Why Shorter Sequences Matter So Much

Fewer tokens is not just tidier. It changes what the model can do and what it costs.

Attention compares every token with every other token. Double the tokens and the work grows roughly four times, not two. So a shorter sequence is cheaper in a way that compounds.

Shorter sequences also mean more real text fits inside the context window, and for paid APIs, fewer tokens means a smaller bill.

What a Bigger Vocabulary Costs

Now the other side. Every token in the vocabulary needs its own row in the embedding table, and each row is a vector with one number per dimension.

So the table's size is vocabulary size multiplied by embedding dimension. Grow the vocabulary and that table grows in a straight line, taking memory with it.

# embedding table size = vocab_size x hidden_dim
# 30,000 x 768   -> about 23 million numbers
# 130,000 x 768  -> about 100 million numbers

The Output Layer Pays Twice

The cost does not stop at the input. At the other end of the model, the final layer must produce a score for every token in the vocabulary, so it can pick the next word.

That means a second big matrix of the same shape. A larger vocabulary makes both ends of the model heavier, and it makes the final softmax slower.

The Context Window Is Counted in Tokens

Here is where the trade-off becomes visible to users. A model's context window is a fixed number of tokens, not words and not characters.

So the same document can fit or overflow depending purely on the tokenizer. An efficient tokenizer packs more real text into the same window.

Why This Hits Other Languages Hardest

Most tokenizers are trained mainly on English, so English words tend to be single tokens. Other languages get chopped into many small pieces.

The result is that the same sentence in Hindi or Japanese can cost two or three times more tokens than in English. The window shrinks, and on a paid API the cost rises, for the same meaning.

Where Real Models Land

Given both pressures, most models settle in a familiar band. BERT used about 30,000 tokens. GPT-2 used about 50,000. Recent large models have pushed to 100,000 and beyond, partly to handle many languages and code more fairly.

The trend has been upward, because memory has grown cheaper while long context has grown more valuable.

How the Dial Is Actually Chosen

Nobody picks the number by feel. Teams tokenize a large sample of their real text at several vocabulary sizes and measure the average tokens per word, sometimes called fertility.

The curve flattens: going from 8,000 to 32,000 tokens helps a lot, and going from 100,000 to 200,000 helps far less while costing just as much memory. Teams stop where the curve stops paying.

The Trade-Off in One Picture

Put the two costs together and the shape becomes clear. Sequence length falls as vocabulary grows, while memory rises. The best choice sits where the two lines cross in a useful place, not at either extreme.

Recap

This is the core tokenization trade-off. A bigger vocabulary turns each sentence into fewer tokens, which saves attention work, fits more text in the context window, and lowers cost. But every extra token needs a row in the embedding table and a slot in the output layer, so memory rises in a straight line.

The context window is measured in tokens, so the tokenizer decides how much real text fits, and languages that were under-represented during training pay more for the same meaning. Most models land between 30,000 and 130,000 tokens, chosen by measuring where the savings curve flattens.

In the next article, we look at what happens when tokenization gets strange: why a model that can write an essay cannot reliably count the letter R in the word strawberry.