Open any tokenizer's vocabulary and you will find strange entries near the top: [CLS], [SEP], [PAD], [UNK]. They are not words. Nobody writes them. Yet without them, a Transformer cannot do its job.
In simple words, these are control tokens. They carry instructions rather than meaning, a little like punctuation marks that only the model can read. Let's meet them one by one.
Why a Model Needs Fake Words
A Transformer only ever sees numbers. Every instruction we want to give it has to arrive as a token ID, because that is the only channel it has.
So when we want to say "this is where the sentence starts", or "ignore this position", we cannot use a comment or a flag. We invent a token, give it an ID, and let the model learn what it means during training.
[UNK]: The Unknown Token
[UNK] stands for unknown. It is the fallback when the tokenizer meets something it cannot represent at all.
With modern subword tokenizers this is rare, because the vocabulary always contains single characters, and byte-level methods contain every possible byte. [UNK] is the last resort, not the normal path.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
tok.tokenize("𓂀") # a glyph outside the vocabulary -> ['[UNK]']![The Fallback Ladder to [UNK]](/images/special-tokens-3.jpg)
[PAD]: Making Sentences the Same Length
Models process text in batches, and every row in a batch must be the same length. Real sentences are not. So we pad the short ones with a filler token until they match the longest.
[PAD] carries no meaning. It exists purely to square off the batch.
The Attention Mask: Telling the Model to Ignore Padding
Padding creates a new problem. If we add filler tokens, the model might pay attention to them and learn from noise.
The fix is the attention mask, a simple list of ones and zeros that travels alongside the token IDs. One means "this is real, look at it". Zero means "this is padding, ignore it".
enc = tok(["the cat sat", "it rained"], padding=True)
enc["attention_mask"] # [[1, 1, 1, 1, 1], [1, 1, 1, 0, 0]]
[CLS]: The Summary Slot
BERT-style models put [CLS] at the very front of every input. It starts as an empty slot with no meaning of its own.
As the model runs, attention lets that slot gather information from every other token. By the final layer it holds a summary of the whole sentence, which is exactly what a classifier needs. When you fine-tune BERT for sentiment, the prediction is read from this one position.
![How [CLS] Fills Up Layer by Layer](/images/special-tokens-6.jpg)
[SEP]: Marking the Boundary
[SEP] marks where one part of the input ends and the next begins. It matters whenever a task involves two pieces of text, like a question and a passage, or two sentences being compared.
Without it, the model would see one long run of words and have no idea where the question stopped and the answer material started.
enc = tok("is it waterproof?", "the case is water resistant")
tok.convert_ids_to_tokens(enc["input_ids"]) # [CLS] ... [SEP] ... [SEP]![What [SEP] Tells the Model](/images/special-tokens-7.jpg)
[MASK]: The Token That Trains BERT
[MASK] is the token behind BERT's training trick. During training, some words are hidden and replaced with [MASK], and the model must guess what was there.
That single game is how BERT learns language. It never sees [MASK] at prediction time in normal use; it is a training-only device.
BOS and EOS: Start and Stop
Generative models like GPT need two more markers. BOS means beginning of sequence, the signal to start. EOS means end of sequence, the signal to stop.
EOS is the important one at generation time. The model generates one token at a time, and it keeps going until it produces EOS. That token is how a model decides an answer is finished.
Let me tabulate them for your better understanding.
| Token | Job | Where it appears |
|---|---|---|
[UNK] |
Text the tokenizer cannot represent | Rare, last resort only |
[PAD] |
Filler so a batch is rectangular | End of short sequences |
[CLS] |
Summary slot for the whole input | Front of BERT inputs |
[SEP] |
Boundary between two parts | Between and after segments |
[MASK] |
Hidden word the model must guess | Training only |
| BOS | Start of generation | Front of generated text |
| EOS | Stop generating here | End of generated text |
Where the Special Tokens Live
Special tokens sit at the very start of the vocabulary, usually holding the lowest IDs. That is a convention, not a rule, but it makes them easy to find and easy to reserve.
Different families use different names for the same ideas. BERT uses [CLS] and [SEP]. GPT-style models often use <|endoftext|> for EOS. The names differ; the jobs do not.
Recap
This is what the square-bracket tokens are for. They are not words, they are instructions delivered through the only channel a model has, its token IDs.
[UNK] catches text the tokenizer cannot represent. [PAD] squares off a batch, and the attention mask tells the model to ignore that padding. [CLS] gathers a summary of the whole sentence for classification, [SEP] marks a boundary between two inputs, and [MASK] powers BERT's fill-in-the-blank training. BOS and EOS start and stop generation.
In the next article, we look at the trade-off hiding behind every tokenizer: a bigger vocabulary means shorter sequences, but it also costs memory, and that tension shapes how models are designed.
