BPE and WordPiece both quietly assume one thing: that you can split text into words on spaces first. But many languages do not use spaces at all. SentencePiece, the method behind T5 and Llama, throws out that assumption and treats text as one raw stream of characters.
In simple words, SentencePiece does not care about words or spaces. It reads the text exactly as it is, keeps the spaces as visible markers, and learns its subwords straight from the raw stream. That makes it work for any language on Earth, and for code. Let's see it in action.
The Problem: Languages Without Spaces
BPE and WordPiece start by cutting text into words on spaces, then split each word into subwords. That works fine for English. But Japanese, Chinese, and Thai write sentences with no spaces between words at all. There is nothing to split on.
So, here comes SentencePiece to the rescue. It refuses to depend on spaces. It reads the whole sentence as one stream and finds good subword pieces directly, whether or not spaces exist. One tokenizer, every writing system.
Text as a Raw Stream
The key mental shift is this. BPE and WordPiece work on words. SentencePiece works on the raw character stream, spaces included. It does not pre-split anything. The entire sentence, spaces and all, goes in as one sequence, and SentencePiece learns pieces from it.
Because it never assumes where words begin or end, it treats every language the same way. A space is just another character in the stream, no more special than a letter.
The Space Marker
If SentencePiece keeps spaces in the stream, how does it show them? It replaces every space with a visible marker, a small underscore-like symbol, before it does anything else.
# SentencePiece turns each space into a visible marker (shown here as _)
# "New York" -> ["_New", "_York"]
# because the space is kept, decoding rebuilds the text exactlySo "New York" becomes two pieces that each begin with the space marker. This looks like a small detail, but it is what makes SentencePiece reversible: because the spaces are never thrown away, you can always turn the tokens back into the exact original text, character for character.
The Unigram Model: Start Big, Prune Down
BPE and WordPiece build up: they start small and merge. SentencePiece usually works the other way, using a method called the Unigram model that builds down.
It starts with a big pile of candidate pieces, far more than it needs. Then it repeatedly asks, "which pieces are pulling their weight?" and throws away the least useful ones, shrinking the vocabulary until it hits the target size. Instead of gluing pairs together, it prunes a huge set down to the best pieces.
How Unigram Decides What to Keep
For each candidate piece, Unigram asks a simple question: if we removed this piece, how much worse would the tokenizer get at explaining the training text? Pieces that carry a lot of weight, like "ing" or "tion", hurt a lot to remove, so they stay. Pieces that barely help, or that overlap with better pieces, get dropped.
Because it scores whole pieces this way, Unigram can also offer more than one valid way to cut a word and pick the most likely one. This flexibility is part of why it handles messy, mixed text so well.
Lossless, Reversible Tokenization
This is one of SentencePiece's best features. Because it keeps every space as a marker and never pre-splits the text, you can always decode the tokens back into the original string exactly, with the spaces in the right places.
That sounds obvious, but it is not true for every tokenizer. Some throw away spacing details and have to guess when rebuilding text. SentencePiece never guesses. Tokens in, exact text out. For tasks like translation and code, where every space matters, this is a big deal.
One Vocabulary for Many Languages
Because SentencePiece learns straight from the raw stream, a single model can share one vocabulary across dozens of languages. English, Hindi, Japanese, and Arabic can all live in the same set of subword pieces, with common bits shared where scripts overlap.
This is exactly what multilingual models like mT5 need. Instead of a separate tokenizer per language, one SentencePiece vocabulary covers them all, which keeps the model smaller and lets it transfer knowledge between languages.
SentencePiece and BPE, Side by Side
Before we go further, let me tabulate the difference for your better understanding. Both build subwords, but they treat the space in opposite ways.
| SentencePiece | BPE | |
|---|---|---|
| The space | Kept as a marker inside the token | Removed before learning starts |
| Input it sees | The full character stream | Words, already split |
| Word splitting | Not needed at all | Required first |
| Rebuilding text | Exact, by construction | Needs an external rule |
| Cross-word pieces | Can learn them | Cannot see them |
| Best suited to | Multilingual text, code, any script | English-centric, speed-first work |
Here, we can see that one small design choice, keeping the space, is what gives SentencePiece its coverage and its perfect round trip.
Tokenizing Code
Code is another place where SentencePiece shines. Programming languages are full of spaces, indentation, and symbols that carry real meaning; a stray space can change what code does. Because SentencePiece keeps every space as a marker and rebuilds text exactly, it preserves indentation and layout that other tokenizers might smudge. That reversibility is one reason many code models lean on it.
Where SentencePiece Is Used
SentencePiece powers many of the biggest models. T5 and its multilingual sibling mT5 use it, and so do Llama, ALBERT, and XLNet. Whenever you need one tokenizer that works across languages, keeps spaces exactly, and handles code cleanly, SentencePiece is usually the answer.
That closes our tour of the three subword methods. The next article steps back and compares BPE, WordPiece, and SentencePiece side by side, so you can tell at a glance which one any model is using.
Recap
This is how SentencePiece works. It drops the assumption that we can split text on spaces, and instead reads the whole sentence as one raw stream. It turns each space into a visible marker, which keeps tokenization fully reversible: tokens always decode back to the exact original text. It usually builds its vocabulary with the Unigram model, starting from a huge pile of candidate pieces and pruning away the ones that do not earn their place.
Because it never depends on spaces, one SentencePiece vocabulary can cover many languages at once and preserve the exact layout of code. That is why T5, mT5, and Llama rely on it. Next, we put all three subword methods, BPE, WordPiece, and SentencePiece, side by side, so you can spot which one any model uses.
