Day 17 introduced positional encoding: attention alone does not know whether dog bites man has the same order as man bites dog. Rotary Position Embeddings, usually called RoPE, add position information in a different way. Instead of adding a position vector to a token vector, RoPE rotates pairs of query and key values by an angle that depends on token position.
The token content stays in Q and K while its position changes their orientation.
Begin With a Two-Dimensional Rotation
Take a pair of numbers (x, y) as a point on a flat plane. Rotating it by an angle theta gives a new pair. The point may point in a new direction, but its distance from the center stays the same.
The horizontal and vertical coordinates are feature values, not locations of words on a page. Rotating a pair changes two learned feature entries through a specific numerical rule. It does not move the token to another place in the sentence.
Also distinguish a vector's angle from an attention probability. After rotation, Q and K still need a dot product, scaling, masking, and softmax. A 90-degree rotation is not an instruction to assign zero attention to every token at that position.
Rotation changes direction without changing vector length.
RoPE applies this simple geometry to pairs of dimensions in a query or key head. Dimension pairs (0,1), (2,3), and so on use different rotation rates.
Use the simple vector and a 90-degree rotation. Since and , the new vector is . A 30-degree rotation gives approximately . In both cases the squared length remains 1.
Computer trigonometry functions normally take radians, an angle unit where a half turn is . A 30-degree turn is radians. Feeding the number 30 to a function expecting radians would not describe a 30-degree turn.
The identity preserves length within each pair. It does not preserve the dot product between two vectors rotated by different angles. Changing that relative orientation is how position enters the attention score.
Position Chooses the Angle
For a token at position p, RoPE rotates each pair by p x theta_i, where theta_i is the rate assigned to that pair. Position 0 rotates by zero. Position 1 rotates once by its rate. Position 5 rotates five times by that rate.
The same content vector points differently at positions 0, 1, and 5.
The rates differ across dimension pairs so the full head has several position patterns rather than repeating one simple clock hand.
Rotate Q and K, Not V
RoPE is applied to queries and keys before their dot product. Values are not rotated for the attention score. The score between a query at one position and a key at another now reflects both their content and their relative orientation.
The value path stays separate because values are mixed after attention weights are made.
This fits neatly into the usual attention pipeline. We still calculate scores, apply a mask and softmax, then mix values.
Relative Distance Appears Naturally
The useful mathematical property is that the dot product of rotated query and key vectors depends on the angle difference. If the query is at position p and the key is at q, the relation involves p - q, their relative distance, rather than only two unrelated absolute labels.
Using rotation matrices, the identity is for a pair with the same rotation rate. The two position rotations combine into one relative rotation. The sign follows the chosen query/key order; the signed offset still carries the relative position.
The underlying content vectors remain in the expression. RoPE introduces relative position into their comparison; it does not replace semantic matching with a distance-only score. Nearby tokens can have weak content matches, and distant tokens can still be useful.
The relationship between positions 7 and 4 has distance 3, wherever that pair occurs.
This helps attention learn patterns such as "look at the previous token" or "look several words back" in a way that can generalize across positions.
Calculate One Relative-Position Score
Hold the unrotated query and key content fixed at so we isolate position. Choose a teaching rate of radians per position. At query position 7 and key position 4, the angle difference is . Their dot product is zero.
At query position 4 and key position 3, the gap is one, and the dot product is . Positions 8 and 5 also have a gap of three and give zero for this same fixed content pair.
We can check this without drawing a model. Python's math module supplies radians-based sine and cosine.
import mathThe helper rotates exactly one pair. We use the same pair and rate for both positions, then take their dot product.
def rotate_pair(pair, angle):
x, y = pair
cosine, sine = math.cos(angle), math.sin(angle)
return (x * cosine - y * sine, x * sine + y * cosine)
rate = math.pi / 6
query_pair = rotate_pair((1., 0.), 7 * rate)
key_pair = rotate_pair((1., 0.), 4 * rate)
score = sum(q * k for q, k in zip(query_pair, key_pair))
assert math.isclose(score, 0.0, abs_tol=1e-12)Real queries and keys contain different learned content, so equal distance does not imply equal attention scores for arbitrary words. The relative-position property tells us how the rotation enters the score while holding the content vectors fixed.
A single rotating pair also repeats. Its score does not fall steadily as distance grows. Multiple rates provide richer distance patterns, but neither this geometry nor a diagram should promise "farther always means less attention."
Different Pairs Track Different Scales
A query head may have 128 dimensions, so RoPE works on 64 pairs. Some pairs rotate quickly, changing a lot over nearby positions. Other pairs rotate slowly, carrying a pattern over a larger distance.
Several rotation rates let one head describe nearby and farther position relationships.
Think of this as many clocks turning at different speeds. One clock can distinguish short gaps well; another changes slowly enough to remain useful farther away.
A common rate schedule is , where counts pairs, is the number of rotated features, and is a chosen base. With four rotated features and base 10,000, the two rates are 1 and 0.01 radians per position.
That is the same idea of fast and slow patterns used in Day 17, but applied through rotations rather than adding a vector. Actual models may choose another base, rotate only part of a head, or use a different feature-pair layout. The checkpoint and implementation must agree on those choices.
The RoFormer paper develops the rotary construction. The easy-to-check two-number example above uses a chosen rate for teaching, not a claim that all model pairs rotate at that rate.
RoPE in the Attention Step
For each token, calculate its ordinary Q and K vectors. Split them into pairs. Rotate every pair using that token's position. Then compute the usual scaled dot products between rotated Q and rotated K.
During cached decoding, older keys are already stored with their position rotation applied. The newest query is rotated for its new position before it compares with those cached keys.
Why Long Context Is Still Hard
RoPE can calculate angles for positions beyond those seen in training, but that does not promise good behavior at every far-away length. The model learned from a particular distribution of distances. Extending context can require a trained long-context model or a carefully chosen adjustment method.
Being able to compute an angle is not the same as being trained to use that distance well.
This is a common lesson in machine learning. A formula may accept a larger input range, while the learned model still needs evidence from training to handle that range well.
There is also a practical cache trap. A cached query slice may have local row 0 while belonging to absolute position 8,192. Its rotation needs the sequence position, not a reset to zero at each decoding call. Stored rotated keys should not be rotated again every time the cache is read.
Changing the rotation scheme while reusing old rotated keys can mix incompatible position representations. Context-extension methods therefore need correct implementation and evaluation, not only a larger maximum-length setting.
A useful long-context test puts necessary information at several distances, asks questions that depend on it, and compares both answer quality and memory use. Accepting a long input and retrieving the right evidence from it are separate checks.
RoPE and Modern Cache Designs
RoPE is also why cache compression designs need care. The stored past keys must retain the position information that future queries use in their dot products. That is why Day 27 separated position-carrying features from compressed content in its MLA overview.
Future queries need both past content and past position relationships.
Recap
This is how RoPE gives attention position information. It treats pairs of Q and K dimensions as small two-dimensional vectors and rotates them by angles tied to token position. When rotated queries and keys take a dot product, relative position differences influence the score.
RoPE changes the Q and K preparation step, while the rest of attention remains the same. Day 30 closes the series by asking a model-building question: given a fixed training budget, how should parameters and data scale together?
