What Are Embeddings? From Token IDs to Meaning

We spent nine days turning text into numbers. A sentence is now a list of token IDs like [464, 3797, 3332]. But those numbers are just row labels. Token 3797 is not bigger or better than token 464; it is simply the 3,797th entry in a list.

So how does a model get from a row number to meaning? That is the job of embeddings, and it is the first real layer of every Transformer.

A Token ID Is Just a Name Tag

Start with what an ID is not. It is not a measurement. If "cat" is 3797 and "dog" is 3798, those two are neighbours by accident of alphabetical order, not because they mean similar things.

Build a tiny vocabulary and look at what an ID really is. The vocabulary is just a list, and the ID is the position in that list.

vocab = ["<pad>", "a", "cat", "dog", "helicopter", "sat"]

print("cat        ->", vocab.index("cat"))
print("dog        ->", vocab.index("dog"))
print("helicopter ->", vocab.index("helicopter"))

# the id is only a position, so comparing ids says nothing about meaning
print("is dog 'more' than cat?", vocab.index("dog") > vocab.index("cat"))

cat        -> 2
dog        -> 3
helicopter -> 4
is dog 'more' than cat? True

That last line is the point. Python happily answers True, because 3 is greater than 2. The comparison is arithmetically correct and completely meaningless. Token 100 plus token 200 does not equal token 300 in any useful sense. The number identifies a row and nothing more.

The Obvious Idea That Fails: One-Hot

The first idea most people have is one-hot encoding. Give every token a long list of zeros with a single one in its own position.

It works, and it is easy to write. Measure what it costs.

import numpy as np

VOCAB_SIZE = 100_000

def one_hot(token_id):
    vec = np.zeros(VOCAB_SIZE, dtype=np.float32)
    vec[token_id] = 1.0
    return vec

cat = one_hot(3797)
print("numbers per word :", cat.size)
print("non-zero values  :", int(cat.sum()))
print("memory per word  :", cat.nbytes / 1024, "KB")

numbers per word : 100000
non-zero values  : 1
memory per word  : 390.625 KB

One hundred thousand numbers to store a single word, and 99,999 of them are zero. A 500-word paragraph would cost nearly 200 MB.

Waste is the smaller problem. The real failure is that one-hot cannot express relatedness at all. Measure the distance between three words and you get the same answer every time.

dog = one_hot(3798)
helicopter = one_hot(52104)

print("cat vs dog        :", np.linalg.norm(cat - dog))
print("cat vs helicopter :", np.linalg.norm(cat - helicopter))

cat vs dog        : 1.4142135
cat vs helicopter : 1.4142135

Identical, down to the last digit. In one-hot space a cat is exactly as far from a dog as it is from a helicopter, because each word occupies its own private column and no two words ever share anything. The format has no way to say "these two are similar".

The Embedding Table

The real solution is a lookup table. The model keeps a big grid: one row per token in the vocabulary, and a fixed number of columns, often 768 or 4096.

To embed a token, the model does not compute anything clever. It goes to that token's row and reads it. The token ID is the row number, which is the only job the ID ever had.

In PyTorch that table is nn.Embedding. The two numbers it needs are the vocabulary size (how many rows) and the embedding dimension (how many columns).

import torch
import torch.nn as nn

torch.manual_seed(42)

# one row per token, 768 numbers per row
embedding = nn.Embedding(num_embeddings=100_000, embedding_dim=768)

ids = torch.tensor([464, 3797, 3332])

with torch.no_grad():
    vectors = embedding(ids)

print("ids shape     :", ids.shape)
print("vectors shape :", vectors.shape)
print("first 5 numbers of the cat row:", vectors[1][:5])

ids shape     : torch.Size([3])
vectors shape : torch.Size([3, 768])
first 5 numbers of the cat row: tensor([-1.5147, -0.6049, -1.5365, -2.2698,  1.9126])

Read the shapes carefully, because they tell the whole story. Three IDs went in. A 3 × 768 block of numbers came out: one row of 768 values for each token. Compare that with one-hot's 100,000 numbers per word and you can see the trade the model is making.

What the Numbers Mean

A natural question: what does the number in column 42 stand for? Usually nothing you can name.

The dimensions are not designed by hand. They are learned, and meaning is spread across all of them together rather than stored in any single one. Occasionally a direction turns out to track something recognisable, like formality or plurality, but that is a discovery, not a design.

How Embeddings Learn Meaning

At the start of training, the whole table is random noise. Those numbers printed above are exactly that: noise from manual_seed(42). Nothing means anything yet.

Then training begins, and the model is repeatedly asked to predict text. Every time it gets something wrong, the error flows backwards and nudges the rows it used. Words that appear in similar places get nudged in similar directions, over and over, for billions of examples.

Meaning is not inserted. It accumulates, as a side effect of getting better at prediction.

Similar Words Drift Together

The result of all that nudging is the property that makes embeddings useful. Words used in similar contexts end up with similar vectors.

We cannot train a real model here, so use a hand-written stand-in for what a trained table looks like: three words, three columns, with the two animals given similar rows. The usual way to compare two vectors is cosine similarity, which returns roughly 1 for "pointing the same way" and roughly 0 for "unrelated".

import torch
import torch.nn.functional as F

words = ["cat", "dog", "helicopter"]

# a tiny stand-in for a trained table: 3 rows, 3 columns
trained = torch.tensor([
    [0.90, 0.85, 0.10],   # cat
    [0.88, 0.80, 0.15],   # dog
    [0.05, 0.10, 0.95],   # helicopter
])

def similarity(a, b):
    return F.cosine_similarity(trained[a], trained[b], dim=0).item()

print("cat vs dog        :", round(similarity(0, 1), 3))
print("cat vs helicopter :", round(similarity(0, 2), 3))

cat vs dog        : 0.999
cat vs helicopter : 0.189

This is the number one-hot could never produce. "Cat" and "dog" appear near words like pet, feed, and vet, so their rows get pulled in similar directions. "Helicopter" is pulled elsewhere. Nobody told the model that cats and dogs are both animals; it fell out of the training text.

One Vector per Token, Not per Word

A detail that trips people up: embeddings are looked up per token, not per word. If a word splits into three tokens, it gets three separate vectors.

Reuse the same embedding layer from before and feed it the three pieces of "tokenization".

# "tokenization" splits into three tokens, so it needs three lookups
piece_ids = torch.tensor([19205, 528, 341])   # token, iz, ation

with torch.no_grad():
    piece_vectors = embedding(piece_ids)

print("tokens :", piece_ids.shape[0])
print("shape  :", piece_vectors.shape)

tokens : 3
shape  : torch.Size([3, 768])

One word went in, three vectors came out. The later layers combine them. The embedding layer never sees whole words; it only sees the pieces the tokenizer produced.

Static at the Start, Contextual Later

One more important point. The vector from the embedding table is the same every time for a given token. The row for "bank" does not change between a river sentence and a money sentence.

with torch.no_grad():
    first_look  = embedding(torch.tensor([3797]))
    second_look = embedding(torch.tensor([3797]))

print("same vector both times?", torch.equal(first_look, second_look))

same vector both times? True

Context arrives afterwards, in the attention layers. They mix information between positions, so by the upper layers the representation of "bank" differs depending on its neighbours. The embedding is the starting point, not the final meaning.

Where Embeddings Sit in the Model

Put it together and the picture is simple. Text becomes tokens, tokens become IDs, IDs index the embedding table, and the resulting vectors are what actually flow into the Transformer.

Everything that follows, attention included, works on these vectors. The embedding layer is the doorway between language and mathematics.

Recap

This is how a token ID becomes meaning. The ID itself is only a row number, carrying no information about the word. One-hot encoding turns that ID into a vector, but as the code showed, it costs 390 KB per word and puts every pair of words at exactly the same distance.

The embedding table solves it: one row per token, a few hundred or few thousand numbers wide, and looking up a token is simply reading its row. Those numbers start as noise and are shaped by training, so words used in similar contexts drift toward similar vectors, which cosine similarity can measure. Each token gets its own vector, and that vector is the same every time until attention makes it contextual in the layers above.

In the next article, we treat these vectors as points in space and look at the geometry of meaning: how similarity is measured, and what is really going on in the famous king minus man plus woman example.