In the last article we saw that subword tokenization is the sweet spot. Now we open up the most popular subword method of all: Byte-Pair Encoding, or BPE. It powers GPT, RoBERTa, and Llama's cousins, and the idea behind it is beautifully simple.
In simple words, BPE starts with single characters and slowly glues together the pairs that show up most often, building longer and longer pieces. Common chunks like "ing" and "tion" get their own tokens because they appear so much. By the end, you will be able to run BPE by hand.
Let's see the whole process at a glance.
Where Does BPE Start?
BPE begins at the smallest possible pieces: single characters. Before it learns anything, the word "low" is just the characters l, o, w. We also add a special end-of-word marker, written </w>, so the model can tell where one word stops and the next begins.
# every word starts as a list of characters plus an end marker
tokens = list("low") + ["</w>"] # ['l', 'o', 'w', '</w>']At this starting point, the vocabulary is only the individual characters. That is tiny and safe, but it means every word is a long string of letters. BPE's job is to fix that by learning bigger pieces.
Step One: Count the Pairs
Now BPE looks at all the words and counts how often each pair of neighbouring pieces appears together. In a corpus full of words like "newest" and "widest", the pair "e" then "s" shows up again and again, and so does "s" then "t".
This counting is the heart of BPE. The pairs that appear most often are the ones worth gluing, because a shortcut for a common pair saves the most effort overall.
Step Two: Merge the Winner
BPE takes the most frequent pair and glues it into a single new token everywhere it appears. If "e" and "s" is the winner, every "e s" in the corpus becomes one piece, "es". That new piece is added to the vocabulary, and the rule "merge e and s into es" is saved as a merge.
# conceptual: merge the most frequent pair into one token
# ('e','s') is most frequent -> add new token 'es'
# every 'e','s' in the corpus becomes 'es'This is one round. BPE just repeats it: count the pairs again, find the new winner, merge it, and save the rule.
Repeat Until Done
BPE keeps merging, and the pieces keep growing. After "es" it might merge "es" and "t" into "est", then later glue "l" and "o" into "lo", then "lo" and "w" into "low". Each round builds on the last, so short common chunks like "est" and whole common words like "low" both become single tokens.
We stop when the vocabulary reaches the size we chose, often tens of thousands of tokens. The result is a mix: single characters for rare cases, common chunks like "ing", and whole common words.
What BPE Actually Learns
When training finishes, BPE has produced two things: a vocabulary of tokens, and an ordered list of merges. The merge list is the recipe. It says, in order, which pairs to glue.
That order matters. To tokenize any new text later, BPE simply replays these merges from the top, gluing pairs in the exact order it learned them. Nothing is random; it is the same recipe every time.
Encoding a Brand-New Word
Here is where BPE proves its worth. Give it a word it never saw in training, and it does not panic. It starts the word as characters, then applies its learned merges in order, gluing whatever pairs it can.
So a new word like "lowest" starts as l, o, w, e, s, t, and after replaying the merges it becomes "low" plus "est", two pieces it already knows. The word is new, but every piece is familiar. That is how BPE never gets truly stuck.
Byte-Level BPE: How GPT Does It
Plain BPE has one weak spot. If it never saw a character during training, like a rare emoji or a symbol from another script, it still has no token for it. GPT fixes this with a clever twist called byte-level BPE.
Instead of starting from characters, it starts from raw bytes. There are only 256 possible bytes, and every possible character on Earth is made of them. So the base alphabet is complete before training even begins. BPE then merges frequent byte pairs just like normal. The payoff is huge: the tokenizer can never meet something it cannot represent, in any language.
Why the Merge Order Matters
It is worth pausing on one detail, because it trips people up. BPE is not just a dictionary of pieces. It is an ordered recipe. The same word always tokenizes the same way, because the merges always replay in the same order.
This makes BPE fast and predictable. There is no searching and no guessing at tokenize time. It simply walks the merge list from top to bottom and glues what it can, every single time.
Where BPE Is Used
BPE is everywhere. The plain version and the byte-level version together tokenize a huge share of the models people use daily. GPT-2, GPT-3, and GPT-4 use byte-level BPE. So do RoBERTa and many open models. When you hear that a model has a vocabulary of "50,000 tokens", it was almost certainly built with BPE.
The next article looks at a close cousin that BERT uses instead, WordPiece, which merges by a slightly different rule.
Recap
This is how BPE works. It starts every word as characters, counts which neighbouring pairs appear most, and glues the winner into a new token. Then it repeats, so the pieces grow from single letters into common chunks like "est" and whole words like "low". What it learns is an ordered list of merges, a recipe it replays to tokenize any new text, even a word it never saw.
GPT takes this one step lower with byte-level BPE, starting from the 256 raw bytes so it can never meet a character it cannot handle. In the next article, we look at how BERT does it differently with WordPiece, and the small change in the merge rule that sets the two apart.
