Multi-Head Attention: Why One Head Is Not Enough

On Day 14, one attention calculation let a word choose which other words mattered. That is a good start, but one set of attention weights answers only one learned question. A sentence often contains several useful relationships at the same time.

Multi-head attention runs several smaller attention calculations in parallel. Every head reads the same input tokens, but each head has its own learned query, key, and value matrices. That gives the model several ways to look at the same sentence.

The diagram below shows the basic idea before we look inside one head.

All heads start from the same token representations, but their separate learned projections can focus on different links.

:::note A real head does not receive a human job title such as "grammar head" or "reference head." We use those labels only to make the idea easy to follow. :::

One View Can Miss Another Useful Clue

Read this sentence: the trophy did not fit in the suitcase because it was too big. To understand it, one attention pattern may need to connect it with trophy. At the same time, the phrase did not fit relates to suitcase, and too big explains why the event happened.

One attention head can focus strongly on the trophy link. If it tries to represent every relationship with the same weight row, its output may become less specific. Several heads give the model room to keep separate clues separate until a later step combines them.

Different relationships can be useful for the same token, so one attention map need not carry every job.

Think of studying a paragraph with three pens. One marks who did something, one marks why it happened, and one marks where it happened. The paragraph did not change. The markings show different questions asked about it.

This sentence-level example assumes all the words are available, as in unmasked encoder attention. A causal decoder state at it cannot read the later words was too big. Heads offer different ways to use allowed context; adding heads never overrides the mask.

One head is not mathematically unable to learn a task with several relationships. Rather, one head uses one score distribution per query for its value mixture. Several heads let the layer form several such mixtures before combining them. That is a specific structural benefit, not a promise that every additional head improves quality.

What Is One Attention Head?

An attention head is one full scaled dot-product attention calculation. It receives the token representations and makes its own queries, keys, and values.

For head $i$, the learned matrices are $W_i^Q$, $W_i^K$, and $W_i^V$. They are different matrices from the ones used by every other head. The same input vector can therefore produce different query, key, and value vectors in different heads.

Separate Q, K, and V matrices let each head score the same words in its own learned way.

During training, no one tells head 1 to look for word reference and head 2 to look for size clues. All heads begin with random values. When a prediction is wrong, training adjusts the matrices. The model can benefit when heads settle into different useful patterns.

For teaching, we might describe three possible views of it like this:

These labels are not promises. A trained model may organize its heads differently from one sentence to the next.

Why Each Head Uses Smaller Vectors

If every head used the full model width, adding more heads would quickly make the attention block much larger. Instead, transformers usually split the model width across heads.

Suppose a small model vector has 12 values and uses 3 heads. Each head works with 4 values, so the joined outputs still have 12 values:

Here $d_{model}=12$, $h=3$, and $d_{head}=4$. Real models use much larger values, but the rule is the same.

Several small head spaces together match the model's usual vector width.

This does not mean that head 1 receives the first four original values and head 2 receives the next four. Each head first uses a learned projection, so all heads can create their own four-value view from the whole input vector.

Project First, Then Reshape

A common implementation applies one wider query matrix and reshapes the result into heads. This is equivalent to placing the per-head projection matrices beside each other. It does not restrict a head to a slice of the original embedding.

We can see the dimensions using NumPy. We choose three token positions, width 12, and three heads. The values below are artificial shape-checking inputs, not a model's learned features.

import numpy as np

The input has one 12-value row per token. Multiplying by a 12-by-12 query matrix gives three projected rows, which we regroup into three four-value heads.

tokens = np.arange(36, dtype=float).reshape(3, 12) / 36
query_matrix = np.full((12, 12), 0.1)
projected_queries = tokens @ query_matrix
query_heads = projected_queries.reshape(3, 3, 4).transpose(1, 0, 2)
assert query_heads.shape == (3, 3, 4)

The final axes are head, token, and head feature. Our simple constant matrix makes heads alike, which is useful for exposing a limitation: reshaping alone does not create different learned views. Real training adjusts the matrix columns, and separate K and V projections complete each head's inputs.

With equal head widths, three projections of shape 12-by-4 contain the same total parameter count as one projection of shape 12-by-12. More heads at fixed total width do not multiply this projection's parameter count by the number of heads. Implementation overhead and score storage can still change.

Every Head Makes Its Own Weight Row

Each head repeats the formula from Day 14 using its own projected Q, K, and V values:

Because $Q_i$ and $K_i$ differ between heads, the attention weights can differ too. The token it may give trophy a large weight in one head and give suitcase a larger weight in another.

The same query position can produce a separate weight distribution in every head.

For example, a teaching table could look like this:

No head is "wrong" because it does not copy another head. Their separate outputs give later parts of the model more than one signal to work with.

Each weight row blends that head's value vectors and returns one small output vector for the token. A three-head layer therefore returns three small results for the same token position.

Concatenation Keeps Every View

The separate outputs must become one vector again before the next layer. First, the model places them beside each other. This operation is called concatenation.

For a separate illustration of concatenation, suppose three four-value heads produce the following outputs. These outputs are not calculated from the earlier weight table, whose value vectors were not specified:

then concatenation makes one 12-value vector:

The vertical bars only help us see the head boundaries. The actual model stores one continuous vector.

Concatenation places each head's small result beside the others instead of averaging them away.

The Output Matrix Lets Heads Share Information

After concatenation, a learned output projection $W_O$ mixes the combined vector back into the normal model space:

Without this projection, the outputs would remain in fixed head-sized sections. $W_O$ lets information discovered by one head influence any part of the next representation.

The learned output projection turns the joined head results into one vector for the next transformer step.

For our width-12 example, $W_O$ has shape $12\times 12$. Its job is to mix head features into the model-width update. This is the attention output projection, not the vocabulary output head that later turns a final hidden state into token scores.

Keeping width 12 also lets the block add this update to its width-12 input through a residual connection. Day 19 will explain that addition and the feed-forward network after it. Attention is one sublayer, not the whole Transformer block.

Heads Learn Rather Than Receive Fixed Jobs

It is tempting to inspect a head and give it a permanent role. Sometimes a head does show a pattern that looks like syntax, word reference, or nearby-context attention. Other heads spread attention widely or appear less important for a particular input.

The useful fact is not that every head can be named. It is that the architecture gives training several independent attention spaces. The next layer can use the combined result even when no single head tells the full story.

Training decides how head projections differ; the labels in this article are teaching shortcuts, not hard-coded rules.

Recap

Multi-head attention performs several smaller attention calculations on the same input. Each head has separate learned Q, K, and V projections, so it can form a different weight pattern. The outputs are concatenated, then mixed through $W_O$ to return one normal-width vector.

On Day 16, we add a rule that decides which tokens a decoder head is allowed to see. That is causal, or masked, attention.