Transformers
Master the Transformer architecture end to end: tokenization, embeddings, the attention family, encoder and decoder families, hands-on fine-tuning, and fast inference with scaling laws.
Transformers Tutorials (9)
Browse, search, and work through all available articles for this category.
Why LLMs Can't Count the R's in Strawberry: Tokenization Quirks
The famous strawberry problem explained: why models that write essays fail at counting letters, plus the arithmetic, spelling, and rhyming quirks tokenization causes.
Read Tutorial
Vocabulary Size vs Context Window: The Core Tokenization Trade-Off
Why a bigger vocabulary makes sequences shorter but the model heavier, how tokens fill a context window, and how the two pull against each other.
Read Tutorial![Special Tokens Explained: [UNK], [PAD], [CLS], [SEP], BOS, and EOS](/_next/image?url=%2Fimages%2Fthumbs%2Fspecial-tokens.jpg&w=3840&q=75)
Special Tokens Explained: [UNK], [PAD], [CLS], [SEP], BOS, and EOS
What the square-bracket tokens mean: [UNK], [PAD], [CLS], [SEP], [MASK], and the BOS and EOS markers, and why a model cannot work without them.
Read Tutorial
BPE vs WordPiece vs SentencePiece: Which One Does Your Model Use
A side-by-side comparison of BPE, WordPiece, and SentencePiece: how each picks its merges, how to spot them in the tokens, and which models use which.
Read Tutorial
SentencePiece: Tokenization for Multilingual Text and Code
How SentencePiece tokenizes raw text with no spaces to split on, the meaning of the underscore marker, the Unigram model, and why T5 and Llama use it.
Read Tutorial
WordPiece: How BERT Splits Words
How WordPiece tokenizes words for BERT, the ## continuation marker, greedy longest-match encoding, and how it differs from BPE by merging on likelihood.
Read Tutorial
Byte-Pair Encoding (BPE) Explained Step by Step
How Byte-Pair Encoding builds a subword vocabulary by merging the most frequent character pairs, how it encodes new words, and why GPT uses byte-level BPE.
Read Tutorial
The Four Ways to Split Text: Word, Character, Subword, and Byte-Level
The four ways to tokenize text, word, character, subword, and byte-level, and the trade-offs in vocabulary size, sequence length, and out-of-vocabulary words.
Read Tutorial
Transformer Architecture and Tokenization Explained
Foundational module on Transformer mechanics and tokenization: embeddings, the attention family, encoder/decoder architectures, and subword tokenization.
Read Tutorial