WordPiece: How BERT Splits Words

BPE glues the most frequent pair. Its close cousin WordPiece, the method behind BERT, glues the pair that helps the model predict text best. That one small change is the whole story, and it leads to the familiar ## marks you see in BERT's tokens.

In simple words, WordPiece also builds subwords, but it picks its merges with a smarter rule, and it tags the inside pieces of a word so they can be joined back together. Let's see how BERT actually cuts a word.

What Makes WordPiece Different from BPE?

WordPiece and BPE start the same way, from single characters, and both merge pieces into bigger ones. The difference is which pair they choose to merge.

BPE is greedy about counts: it merges the pair that simply appears most often. WordPiece is greedy about usefulness: it merges the pair that most increases the chance of the training text under the model. In plain terms, WordPiece asks "which merge best helps predict real language?" rather than "which pair is most common?"

Most of the time the two agree, but WordPiece's rule tends to produce pieces that line up a little better with real word parts, like prefixes and suffixes.

The ## Continuation Marker

Here is the piece of WordPiece you will actually see in BERT's output. When a subword is inside a word rather than at its start, WordPiece tags it with ##.

# WordPiece tags inside-pieces with ##
# "playing" -> ["play", "##ing"]
# "##ing" means: attach me to the piece before me

So "play" has no marker because it starts the word, and "##ing" carries ## because it continues one. This tiny tag is important: it lets the model tell "ing" the whole word from "ing" the suffix, and it lets us glue the pieces back into the original word without guessing where the spaces were.

Building the Vocabulary by Likelihood

During training, WordPiece grows its vocabulary one merge at a time, just like BPE. But at each step it scores every candidate pair by how much gluing it would raise the likelihood of the training text, and it picks the best-scoring one.

You can picture the score as a simple question: does the pair appear together far more than its two parts would on their own? A pair like "t" and "ion" scores high because "tion" is a real, meaningful chunk. A random pair that just happens to co-occur scores low. This is why WordPiece pieces often feel like natural word parts.

How WordPiece Encodes: Greedy Longest-Match

At tokenize time, WordPiece uses a rule called greedy longest-match-first. For each word, it looks for the longest piece in its vocabulary that matches the start of the word, takes it, then repeats on the rest.

So for "playing", it first grabs the longest matching start, "play", then continues on "ing" and grabs "##ing". Always taking the longest match keeps the token count low and the pieces meaningful.

Encoding a Word BERT Knows Well

For a common word that already sits in the vocabulary, WordPiece is done in one step. The word "hello" is a single token, no splitting, no ##. Most everyday words are like this, which keeps sentences short.

Encoding a Word BERT Has Never Seen

Now the interesting case. Give BERT a rare word like "tokenization", and greedy longest-match kicks in. It grabs "token", then "##ization", or perhaps "##iza" and "##tion" depending on the vocabulary. Either way, the rare word becomes a handful of known pieces, and its meaning largely survives.

This is the same gift as BPE: WordPiece rarely gets truly stuck, because it can always fall back to smaller and smaller subwords.

When Even Subwords Fail: [UNK]

There is one last safety net. If WordPiece cannot match even a single character of a piece to its vocabulary, which is rare, it gives up on that piece and emits a special [UNK] token, meaning "unknown". In practice this almost never happens for normal text, because the vocabulary includes all the common characters. But it is there as a final catch.

Why the Likelihood Rule Matters

You might wonder if choosing merges by likelihood instead of frequency really changes much. It does, in a subtle but useful way. Frequency alone can glue pairs that just happen to sit next to each other a lot. Likelihood favours pairs that carry real meaning together, like true prefixes and suffixes.

The effect is that WordPiece pieces tend to align with the parts of words that matter, which can help a model like BERT understand structure. Same family as BPE, slightly sharper cuts.

Where WordPiece Is Used

WordPiece is the tokenizer behind the BERT family. BERT, DistilBERT, and Electra all use it, which means a large slice of the classification and search systems in production run on WordPiece. Whenever you see tokens with ## in them, you are looking at WordPiece at work.

The next article covers the third big subword method, SentencePiece, which throws out one assumption both BPE and WordPiece quietly make: that we can split on spaces in the first place.

Recap

This is how WordPiece works. It is BPE's close cousin: it also builds subwords by merging small pieces into bigger ones, but it chooses each merge by likelihood, the pair that best predicts real language, rather than by raw frequency. It tags the inside pieces of a word with ## so they can be joined back together, and at tokenize time it uses greedy longest-match to grab the biggest known piece first.

Common words stay whole, rare words split into meaningful subwords, and only in the rarest case does it fall back to [UNK]. This is the tokenizer that powers BERT. Next, we meet SentencePiece, the method that drops the need for spaces entirely and handles any language on Earth.