We have now met all three subword methods. BPE glues the most frequent pair. WordPiece glues the pair that best predicts the text. SentencePiece skips word splitting and reads the raw stream. Three methods, one job.
So how do you tell them apart in the wild? The good news is that each one leaves a fingerprint in its output, and once you know the fingerprint you can name the tokenizer at a glance.
The One Difference That Matters
All three methods start small and build bigger pieces. That part is shared. The difference is the question each one asks before it merges.
BPE asks "which pair appears most often?" WordPiece asks "which pair helps me predict text best?" SentencePiece asks something else entirely, because it changes the input rather than the rule: "what if we never split on spaces at all?"
In simple words, BPE and WordPiece argue about which pair to glue. SentencePiece argues about what counts as a starting piece.
Fingerprint One: The ## Marker
The easiest tell is the ## prefix. If you see tokens like play and ##ing, you are looking at WordPiece. No other common method uses that marker.
# WordPiece output has ## on inside pieces
# "playing" -> ["play", "##ing"]The ## says "join me to the piece before me". It only appears inside a word, never at the start.
Fingerprint Two: The Space Marker
SentencePiece leaves a different mark. Because it keeps spaces as part of the token, its pieces often begin with a small underscore-style symbol that stands for a space.
So a sentence tokenized by SentencePiece looks like _New, _York, _City. That leading mark is the space itself, travelling with the token.
Fingerprint Three: Plain Pieces
If the tokens have no ## and no space marker, and the pieces look like ordinary chunks of words, you are most likely looking at BPE.
Byte-level BPE, the version GPT uses, sometimes shows odd-looking symbols for spaces and accents, because it works on raw bytes. That is its own small tell.
Let me tabulate the fingerprints for your better understanding.
| What you see in the tokens | The method | Example |
|---|---|---|
## in front of inside pieces |
WordPiece | play, ##ing |
| A leading underscore mark | SentencePiece | _New, _York |
| Plain pieces, no markers | BPE | low, est |
How Each One Handles a New Word
Give all three the same unseen word and they all survive, but they cut it differently. BPE replays its merge list. WordPiece takes the longest matching piece first. SentencePiece works from its pruned vocabulary over the raw stream.
The splits often look similar. WordPiece tends to cut at meaningful seams a little more often, because its rule rewards pieces that predict real language.
Which Models Use Which
This is the practical part. BPE, in its byte-level form, powers the GPT family and many open models. WordPiece powers the BERT family. SentencePiece powers T5, Llama, and most multilingual models.
Here, we can see the three methods laid out side by side.
| BPE | WordPiece | SentencePiece | |
|---|---|---|---|
| Merge rule | Most frequent pair | Pair that best predicts | Prune the weakest pieces |
| Starting units | Characters, split on spaces | Characters, split on spaces | The raw character stream |
| Needs spaces? | Yes | Yes | No |
| Marker in output | None | ## on continuations |
Underscore for the space |
| Rebuilds text exactly | By rule | By rule | By construction |
| Typical models | GPT, RoBERTa, BART | BERT, DistilBERT, Electra | T5, mT5, Llama, ALBERT |
If you are loading a model from Hugging Face, you rarely choose the tokenizer yourself. It ships with the model, and it must match, because the token IDs are meaningless to a model trained on a different vocabulary.
Why the Tokenizer Must Match the Model
Here is a mistake worth avoiding. A token ID is just a row number in the model's embedding table. Row 5021 means "able" only because that model learned it that way.
Swap in a different tokenizer and the same word turns into different numbers. The model will still run, but it will read nonsense. This is why every model on Hugging Face ships its tokenizer alongside its weights.
# the tokenizer and the model must come from the same checkpoint
# token id 5021 means "able" only for THIS vocabulary
Choosing a Method for Your Own Model
If you ever train a tokenizer from scratch, the choice is simpler than it looks. For English-only text, BPE is a safe default. For a BERT-style encoder, WordPiece fits the family. For anything multilingual, or text with code, or scripts without spaces, SentencePiece is the natural pick.
In practice most teams start from an existing tokenizer and only train a new one when their text looks nothing like ordinary web text.
The Shared Family Tree
Step back and the three look less like rivals and more like cousins. All three build subwords. All three keep the vocabulary at a fixed size. All three survive words they never saw.
That shared idea, pieces smaller than words but bigger than letters, is what makes modern language models possible at all.
Recap
This is how the three subword methods compare. BPE merges the most frequent pair, WordPiece merges the pair that best predicts real language, and SentencePiece drops word splitting entirely so it can read any script.
You can name them by their fingerprints: ## means WordPiece, a leading space mark means SentencePiece, and plain pieces usually mean BPE. Whichever one your model uses, the rule is the same: the tokenizer must match the model, because token IDs only mean something inside the vocabulary that created them.
In the next article, we look at the special tokens that sit alongside these pieces, the ones in square brackets like [CLS] and [SEP], and what each of them is for.
