TransformersJul 23, 202619 min read
Transformer Architecture and Tokenization Explained
Foundational module on Transformer mechanics and tokenization: embeddings, the attention family, encoder/decoder architectures, and subword tokenization.
Category Profile
Transformers
Master the Transformer architecture end to end: tokenization, embeddings, the attention family, encoder and decoder families, hands-on fine-tuning, and fast inference with scaling laws.
Transformers Tutorials (6)
Browse, search, and work through all available articles for this category.
Transformers and Architecture
6 articlesTransformers
TransformersJul 24, 20268 min read
The Four Ways to Split Text: Word, Character, Subword, and Byte-Level
The four ways to tokenize text, word, character, subword, and byte-level, and the trade-offs in vocabulary size, sequence length, and out-of-vocabulary words.
TransformersJul 31, 20267 min read
Byte-Pair Encoding (BPE) Explained Step by Step
How Byte-Pair Encoding builds a subword vocabulary by merging the most frequent character pairs, how it encodes new words, and why GPT uses byte-level BPE.
TransformersAug 4, 20266 min read
WordPiece: How BERT Splits Words
How WordPiece tokenizes words for BERT, the ## continuation marker, greedy longest-match encoding, and how it differs from BPE by merging on likelihood.
TransformersAug 7, 20267 min read
SentencePiece: Tokenization for Multilingual Text and Code
How SentencePiece tokenizes raw text with no spaces to split on, the meaning of the underscore marker, the Unigram model, and why T5 and Llama use it.
TransformersAug 10, 20267 min read
BPE vs WordPiece vs SentencePiece: Which One Does Your Model Use
A side-by-side comparison of BPE, WordPiece, and SentencePiece: how each picks its merges, how to spot them in the tokens, and which models use which.