Why Transformers Need Attention: The Trophy Problem

Read this sentence: the trophy did not fit in the suitcase because it was too big.

What does it mean here? We all read it as the trophy, and nobody had to stop and think. Now we change one word: because it was too small. Now it is the suitcase. One word flipped, and the meaning of a different word moved with it.

This is the problem attention was built to solve. In this blog, we will learn why the embeddings from the last two lessons cannot fix it, and what we have to add on top of them.

The Sentence That Breaks a Bag of Vectors

One word, two meanings

Let's put the two sentences side by side. Only the last word changes.

• the trophy did not fit in the suitcase because it was too big → it is the trophy
• the trophy did not fit in the suitcase because it was too small → it is the suitcase

The word it did not move. Nothing next to it changed either. Still its meaning flipped, and the cause is a word four places away.

Change one adjective and the referent of "it" swaps. The word itself never changes.

Why yesterday's embeddings cannot help

Two lessons back we saw that the embedding table gives each token one fixed row. We look up the same token twice, and we get the same numbers twice. Let's see the code as below:

import numpy as np

#                    heavy  roomy  object
trophy   = np.array([0.90, 0.10, 0.80])
suitcase = np.array([0.20, 0.90, 0.80])
it       = np.array([0.50, 0.50, 0.50])

# the same token in both sentences
it_in_sentence_a = it
it_in_sentence_b = it

print("same vector in both sentences?", np.array_equal(it_in_sentence_a, it_in_sentence_b))
print("it =", it)

same vector in both sentences? True
it = [0.5 0.5 0.5]

Here, we can see True, and that is the whole problem. The table hands back one vector for it. It is the same vector in the sentence about a big trophy and in the sentence about a small suitcase. A fixed embedding table has no way to tell the two apart.

What Attention Adds

Every word looks around

So, here comes attention to the rescue. Attention gives every word one new ability. In simple words, a word can look at every other word in the sentence and decide which ones matter to it.

For it, that means looking at trophy and at suitcase, then leaning toward the one that fits better. The rest of the sentence gives the proof, and the word big or small is what tips the balance.

Each word compares itself against all the others, including itself.

Scores, then weights

The comparison is a dot product. We multiply the two vectors position by position, then we add up the results. A bigger number means a stronger match.

The word it does not compare itself raw. It carries a hint from the rest of the sentence. In our small space, too big pushes the query toward the heavy dimension, and too small pushes it toward the roomy one. Let's see the code as below:

q_big   = np.array([0.95, 0.05, 0.10])   # "it ... too big"
q_small = np.array([0.05, 0.95, 0.10])   # "it ... too small"

scores_big = np.array([q_big @ trophy, q_big @ suitcase])
print("scores with 'big'  :", scores_big.round(2))

scores with 'big'  : [0.94 0.32]

Raw scores are hard to read. So we pass them through softmax. Softmax turns the scores into weights, and those weights are always positive and always add up to 1.

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

weights_big = softmax(scores_big * 5)
print("weights with 'big' :", weights_big.round(2))
print("they add up to     :", weights_big.sum().round(2))

weights with 'big' : [0.96 0.04]
they add up to     : 1.0

Here, we can see the answer the model gives us: 96% trophy, 4% suitcase.

Dot product first, softmax second. The scores can be any size; the weights always add to 1.

The new vector for "it"

Those weights now decide how we mix the other words. We take 96% of the trophy vector, 4% of the suitcase vector, and we add them together. Let's see the code as below:

new_it = weights_big[0] * trophy + weights_big[1] * suitcase
print("new vector for 'it':", new_it.round(2))
print("trophy was          :", trophy)

new vector for 'it': [0.87 0.13 0.8 ]
trophy was          : [0.9 0.1 0.8]

Here, we can see that the vector for it has moved a long way. It started at a neutral middle point, and now it sits almost on top of trophy.

The output is a blend of the other words, mixed in proportion to the weights.

Reading the Result

Now we run the same three steps on the other sentence. We change nothing else. Let's see the code as below:

scores_small  = np.array([q_small @ trophy, q_small @ suitcase])
weights_small = softmax(scores_small * 5)
new_it_small  = weights_small[0] * trophy + weights_small[1] * suitcase

print("weights with 'small':", weights_small.round(2))
print("new vector for 'it' :", new_it_small.round(2))
print("suitcase was        :", suitcase)

weights with 'small': [0.03 0.97]
new vector for 'it' : [0.22 0.88 0.8 ]
suitcase was        : [0.2 0.9 0.8]

Here, we can see the same word, the same starting vector, and the same table. Two different answers, because the words around it changed.

This is exactly the job of attention, and we have just watched it happen in nine lines of NumPy.

One token, one embedding row, two outputs. The context did the work.

The Weights Are Learned, Not Written

Nobody wrote a rule saying that big points at trophies. We picked our query vectors by hand so the demo stays easy to read. A real model does not get that help.

A real model starts with random numbers. It reads text and predicts the next word. When it is wrong, it nudges the numbers that caused the mistake. After enough training, the pattern that says "a size word tells us which object it means" is simply the setup that predicted best.

Nobody taught the model that rule. The model found it.

Random at the start, shaped by billions of guesses.

Why Word Order Alone Is Not Enough

An older answer was to read the sentence from left to right and carry a running memory along the way. That works fine until the clue we need sits far away.

Our clue is four words from it. It could just as easily be forty. Reading in order forces the model to keep every word it might need inside one small memory. It also has to drag that memory past every word in between. Long gaps are where those models lost the thread.

Attention removes that journey. Every word can reach every other word in one step, however far apart they sit.

A running memory has to carry the clue the whole way. Attention jumps straight to it.

What This Costs

Connecting every word to every other word is not free. Ten words need one hundred comparisons. One hundred words need ten thousand.

The work grows with the square of the sentence length. We double the sentence, and the cost goes up four times, not two. That one fact drives most of the second half of this series, from the KV cache to Flash Attention.

Every word compares against every word, so the count grows as the square of the length.

Recap

This is how attention works. We started with a sentence that breaks a bag of fixed vectors. The word it means two different things in two sentences, but the embedding table returns the same row both times.

Attention fixes this by letting every word look at every other word. It scores each pair with a dot product, turns those scores into weights with softmax, and builds a new vector by mixing the other words in those proportions. In our small run, it came out 96% trophy in one sentence and 97% suitcase in the other, from the same starting vector.

Nobody wrote those weights. They are what falls out of predicting text well, again and again. And every word reaching every word costs work that grows with the square of the sentence length. That is a bill the rest of this series spends a lot of time paying down.

In the next lesson we name the three vectors this mechanism really uses: query, key and value. We will see exactly which job each one does.