Every language model starts by cutting text into small pieces called tokens. But there is more than one way to make those cuts, and the choice shapes how well the model reads, how much memory it uses, and whether it ever gets stuck on a new word.
In simple words, there are four ways to split text: by whole words, by single characters, by subwords, and by raw bytes. Each one trades something for something else. By the end of this article, you will know exactly which method a model like BERT, GPT, or Llama uses, and why.
Let's start with the big picture.
Why Split Text at All?
A neural network only does math on numbers. It cannot read the letters in "chess" directly. So the very first step is always to chop the text into tokens and give each token a number, called a token id. The model then works only with those ids.
The question is not whether to split the text, but where to make the cuts. Cut in the wrong place and you either waste memory or lose the meaning. The four methods below are four different answers to that one question.
Word-Level Tokenization
The most natural idea is to treat every word as one token. We just split on spaces:
# word-level: one token per word
"playing chess".split() # -> ["playing", "chess"]This feels clean, and for short sentences it is. But it has two serious problems.
First, the vocabulary explodes. English alone has hundreds of thousands of words, and once you add names, typos, and other languages, the list of known words becomes huge. Every extra word costs memory.
Second, and worse, is the out-of-vocabulary problem. If the model never saw the word "tokenizer" during training, it has no id for it. The word becomes a single "unknown" token with no meaning at all. A tiny typo can turn a real word into a blank.
Character-Level Tokenization
The opposite idea is to make every single character its own token:
# character-level: one token per character
list("chess") # -> ["c", "h", "e", "s", "s"]Now the problems flip. The vocabulary is tiny, just the letters, digits, and a few symbols. And nothing is ever unknown, because any word is just a string of characters the model already knows.
But we pay a heavy price in length. A short sentence becomes a very long list of tokens. The model has to do far more work to see whole words, and it can only fit a fraction of the text in its limited context window. Reading gets slow and shallow.
Subword Tokenization
So, here comes the method almost every modern model uses: subword tokenization. The idea is to keep common words whole and split only rare words into smaller, known pieces.
The word "playing" stays as "play" plus "ing". A rare word like "tokenization" might become "token" plus "ization". Common words are one token, so sentences stay short. Rare words break into familiar parts, so nothing is ever truly unknown.
This is the sweet spot. We get a reasonable vocabulary size and we never get stuck on a new word. The next articles cover the three ways to build these subwords: BPE, WordPiece, and SentencePiece.
Byte-Level Tokenization
There is one more level, below characters: bytes. Every piece of text on a computer is stored as raw bytes, and byte-level tokenization works directly on those.
Why go this low? Because bytes are truly universal. There are only 256 possible byte values, and every language, emoji, and symbol on Earth is made of them. A byte-level model can never hit a character it does not know, not even an emoji or a rare script. It is the ultimate safety net.
Modern systems often combine ideas: they run subword tokenization on top of bytes, so they get short sequences and the byte-level guarantee that nothing ever fails.
The Core Trade-Off: Vocabulary Size vs Sequence Length
Every method above is really balancing two numbers that pull in opposite directions.
A bigger vocabulary keeps more words whole, so each sentence becomes fewer tokens and the model reads faster. But a big vocabulary costs more memory and needs more data to learn well.
A smaller vocabulary uses less memory, but it chops words into more pieces, so sentences become longer and slower, and the model can fit less text at once.
Word-level sits at one extreme (huge vocabulary, short sequences), character-level at the other (tiny vocabulary, long sequences). Subword tokenization is popular precisely because it lands in the comfortable middle.
Handling New Words: The OOV Test
The clearest way to compare the four methods is to hand each one a brand-new word it never saw in training, say "unbelievable", and watch what happens.
Word-level fails outright: the whole word becomes one blank "unknown" token. Character-level and byte-level never fail, but they stretch the word into a long string of tiny pieces. Subword handles it gracefully, splitting into known parts like "un", "believ", and "able", so the meaning mostly survives.
This single test is why subword tokenization won. It keeps sentences short like word-level, yet it never breaks on a new word like character-level.
Where Multilingual Text and Emoji Break the Rules
For plain English, character-level and byte-level look almost the same. The difference shows up the moment you leave English.
Many languages, like Japanese and Chinese, do not put spaces between words, so word-level splitting has nothing to split on. Accented letters and emoji are single characters to a human but several bytes to a computer. This is exactly where byte-level shines: it treats every script and symbol as just bytes, so one tokenizer can handle the whole world.
A Simple Decision Guide
Let me tabulate the four methods for your better understanding.
| Method | Vocabulary | Sequence length | Unknown words | Best for |
|---|---|---|---|---|
| Word-level | Huge | Short | Fails | Simple, small demos |
| Character-level | Tiny | Very long | Never fails | Small, character-heavy tasks |
| Subword | Medium | Medium | Splits gracefully | Almost every modern LLM |
| Byte-level | Tiny (256) | Long | Never fails | Multilingual, emoji, code |
In practice, the answer is almost always "subword, built on top of bytes." That combination gives short sentences, a manageable vocabulary, and a guarantee that nothing ever breaks.
Which Method Do Real Models Use?
Theory is nice, but which method powers the models you actually use? The answer is that they all use a form of subword tokenization, just built in different ways.
BERT uses WordPiece. GPT models use a byte-level BPE. T5 and Llama use SentencePiece. None of them use plain word-level or plain character-level, because the subword sweet spot beats both. The next three articles open up each of these three subword methods in detail.
Recap
This is how text gets split. There are four ways, and each one balances two numbers that fight each other: vocabulary size and sequence length. Word-level keeps sentences short but drowns in a huge vocabulary and breaks on new words. Character-level never breaks but stretches text into long, slow sequences. Byte-level does the same at the level of raw bytes, which makes it perfect for many languages and emoji. And subword tokenization lands in the middle, keeping common words whole while splitting rare ones into known pieces.
That middle ground is why every real model uses a subword method. In the next article, we open up the most popular one of all and watch it build its vocabulary from scratch: Byte-Pair Encoding, explained step by step.
