Chapter 2.2 β€” Positional Information

Contents

  1. The problem Chapter 2.1 left open
  2. Sinusoidal absolute positional encoding: the original fix
  3. Learned absolute position embeddings
  4. Relative position representations
  5. Rotary position embeddings (RoPE)
  6. ALiBi: biasing attention scores instead of embeddings
  7. Comparing the four approaches
  8. Interview angle
  9. Self-check questions
  10. Sources

1. The problem Chapter 2.1 left open

Chapter 2.1 ended with an uncomfortable fact sitting underneath everything the Transformer does: self-attention, as defined, is permutation-invariant. The attention weights between any two positions are a function of their query and key vectors alone, and the output at a position is a weighted sum over value vectors, so if you shuffled the order of the input tokens before feeding them into a self-attention layer, and shuffled the corresponding outputs back afterward, you would get the exact same result. A Transformer built purely out of the mechanism from Chapter 2.1 has no way of knowing that "the dog bit the man" and "the man bit the dog" are different sentences, because as sets of token representations with no positional tag, they are literally identical inputs to self-attention.

This is not a minor gap to patch over; it's a direct structural consequence of throwing away recurrence. An RNN gets sequence order for free, because it processes tokens one at a time in order and its hidden state is, by construction, a function of position. The Transformer traded that automatic ordering away in exchange for parallelism, and now has to buy positional information back some other way, deliberately and explicitly. Everything in this chapter is a different answer to the same question: how do you tell a fundamentally order-blind, fully parallel mechanism where each token sits in the sequence, and how do you do it in a way that generalizes well β€” including to sequences longer than anything seen during training?

2. Sinusoidal absolute positional encoding: the original fix

The original Transformer paper's own solution, described directly in Vaswani et al.'s Section 3.5, is to construct a fixed, non-learned vector for each position and add it directly to the token embedding before the first layer. For position $pos$ and embedding dimension index $i$, the encoding is defined as $$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \quad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$ so each dimension of the positional encoding is a sinusoid, and the wavelength of that sinusoid varies geometrically across dimensions, from $2\pi$ up to roughly $10000 \cdot 2\pi$. Adding this vector to the token embedding gives every position a unique, deterministic "signature," and because sine and cosine are used in pairs, that signature has a convenient algebraic property: for any fixed offset $k$, $PE_{pos+k}$ can be written as a linear function of $PE_{pos}$, via the standard angle-addition identities for sine and cosine. Concretely, for the two-dimensional slice at indices $2i, 2i+1$, this linear relationship is a rotation: $$\begin{pmatrix}PE_{pos+k,\,2i}\PE_{pos+k,\,2i+1}\end{pmatrix} = \begin{pmatrix}\cos(k\omega_i) & \sin(k\omega_i)\ -\sin(k\omega_i) & \cos(k\omega_i)\end{pmatrix}\begin{pmatrix}PE_{pos,\,2i}\PE_{pos,\,2i+1}\end{pmatrix}, \quad \omega_i = \frac{1}{10000^{2i/d_{\text{model}}}}$$ This means, at least in principle, that the model could learn to recover relative positional information (how far apart two tokens are) via a linear transformation applied to absolute encodings, even though the encoding itself only ever represents an absolute position directly.

The choice of sinusoids rather than, say, a simple integer count for each position is deliberate on two fronts. First, adding a raw integer position count directly to an embedding would produce values of wildly different scale depending on sequence length, which plays badly with the rest of the network's activations; a bounded, smooth signal like a sinusoid keeps things numerically well-behaved regardless of how long the sequence gets. Second, the authors specifically wanted the encoding scheme to be able to extrapolate, at least somewhat, to sequence lengths not seen during training, since it's defined by a closed-form function of position rather than a lookup table with a fixed number of learned entries β€” you can always evaluate $\sin$ and $\cos$ at a position beyond the training range, whereas a table with a fixed number of rows simply has no entry to look up. In practice this extrapolation turned out to be fairly weak, which is one of the motivations behind the later approaches in this chapter, but the reasoning behind the design is worth understanding on its own terms.

3. Learned absolute position embeddings

The alternative that appeared quickly afterward, and that models like BERT and GPT-2 adopted, is simpler to state: rather than defining a fixed sinusoidal function of position, learn a separate embedding vector for each position index directly, exactly the way you'd learn a separate embedding vector for each vocabulary item, and add it to the token embedding. This is a straightforward application of the idea that if you're not sure the right functional form for something, and you have enough data, let gradient descent find it β€” the model is free to discover whatever positional representation is actually useful for its objective, rather than being constrained to the specific sinusoidal family.

The tradeoff is exactly what you'd expect from replacing a fixed function with a lookup table: learned absolute position embeddings tend to work at least as well as sinusoidal encodings on sequences within the training length, since they have strictly more freedom to fit whatever positional structure is useful. But they have essentially no ability to generalize beyond the maximum sequence length seen during training, because position 5000 simply has no learned embedding if the model only ever saw sequences up to length 2048 β€” there's no functional form to extrapolate, only a table with a fixed number of rows. This limitation, shared to a lesser degree by the sinusoidal approach, is precisely what motivated the shift toward encoding position relatively rather than absolutely, which is where the rest of this chapter goes.

4. Relative position representations

Both of the previous approaches encode where a token is; Shaw, Uszkoreit, and Vaswani's 2018 relative position representations instead encode how far apart two tokens are, and inject that information directly into the attention computation rather than into the token embeddings beforehand. The motivation is a fairly intuitive linguistic observation: what usually matters for understanding a sentence is not that a word sits at absolute position 47, but that it sits three words after the verb it modifies, or two words before the noun it describes. Relative offsets are the more natural unit of positional meaning for language, and an architecture that encodes offsets directly should need less work to learn to exploit them, compared to one that has to reconstruct relative information indirectly from two absolute signatures.

Mechanically, Shaw et al. modify the attention computation itself: for a query at position $i$ attending to a key at position $j$, they introduce a learned vector $a_{ij}$ that depends only on the clipped relative distance $j - i$ (clipped to some maximum range, since very distant relative offsets are grouped together beyond a certain point), and this vector is added into the key (and optionally the value) computation used in the attention score, rather than into the input embeddings. This means the same relative-offset vector is reused for every pair of positions with that offset, anywhere in the sequence β€” the representation of "two positions to the left" is the same whether you're near the start or the end of a long document, which is a strictly more parameter-efficient and generalizable way to represent position than giving every absolute index its own independent vector. This paper is also the conceptual ancestor of both of the modern approaches that follow: once you accept that position should be encoded as a function of relative offset injected into attention itself, rather than as an absolute additive signal to the embeddings, RoPE and ALiBi are best understood as two different, more elegant, more efficient ways of realizing that same idea.

5. Rotary position embeddings (RoPE)

RoPE, introduced by Su et al. in the RoFormer paper, is the approach adopted by most modern open-weight LLMs, and its central trick is genuinely elegant: instead of adding a positional signal anywhere, encode position as a rotation applied to the query and key vectors, chosen so that the dot product between a rotated query and a rotated key depends only on their relative offset, not their absolute positions.

Here's the derivation intuition. Take a 2-dimensional slice of the query vector at position $m$, treat it as a complex number (or equivalently, a 2D vector), and rotate it by an angle proportional to $m$ times some fixed frequency $\theta$: $q_m \to R(m\theta) \, q_m$, where $R(\cdot)$ is a standard 2D rotation matrix. Do the same to the key vector at position $n$: $k_n \to R(n\theta) \, k_n$. Because rotations compose by adding angles, the dot product of the rotated query and rotated key is $$\big(R(m\theta) q_m\big) \cdot \big(R(n\theta) k_n\big) = q_m^\top R(m\theta)^\top R(n\theta) \, k_n = q_m^\top R\big((n-m)\theta\big) \, k_n$$ using the fact that $R(m\theta)^\top R(n\theta) = R\big((n-m)\theta\big)$ for rotation matrices. The absolute positions $m$ and $n$ have vanished from the final expression except through their difference $n - m$ β€” the attention score between two positions depends only on their relative offset, purely as a consequence of the geometry of rotation, with no additive positional signal anywhere and no separate relative-position embedding table to learn. In practice, RoPE applies this rotation independently across many 2D subspaces of the full query and key vectors, each with a different frequency $\theta$ (following a geometric schedule similar in spirit to the sinusoidal encoding's wavelengths), so that different subspaces capture positional relationships at different scales, from very local to very long-range.

RoPE's appeal in practice comes from combining several properties that earlier schemes had to trade off against each other: it's a fixed, deterministic function of position rather than a learned table, so it doesn't run out of positions the way learned absolute embeddings do; it produces relative-position-dependent attention scores natively, without needing an extra learned relative-position term injected into the attention computation the way Shaw et al.'s scheme does; and it's applied directly to queries and keys rather than added to embeddings, which composes cleanly with the rest of the attention mechanism and adds essentially no extra parameters or computation. This combination of properties is a large part of why RoPE became the default choice in models like LLaMA and its many descendants.

6. ALiBi: biasing attention scores instead of embeddings

ALiBi, introduced by Press, Smith, and Lewis, takes a different route to the same underlying goal β€” good behavior on sequences longer than those seen in training β€” and does it with a mechanism that's arguably even simpler than RoPE's rotations. Rather than modifying the query and key vectors at all, ALiBi leaves them untouched and instead adds a static, non-learned penalty directly to the raw attention scores, before the softmax, proportional to the distance between the query and key positions: for a query at position $i$ attending to a key at position $j$, with $j \le i$, the attention score is modified as $$\text{score}(i, j) = q_i \cdot k_j - m \cdot (i - j)$$ where $m$ is a fixed, head-specific slope (different heads get different, geometrically spaced slopes, so some heads penalize distance sharply and others barely at all). The intuition is directly recency-biased: all else being equal, a token should have some baseline preference for attending to nearby tokens over distant ones, and that preference should get built into the architecture as an explicit inductive bias rather than left for the model to discover on its own from scratch through learned embeddings.

The reason this gives strong length extrapolation is worth spelling out. Because the bias term is a simple, unbounded linear function of distance, it behaves consistently regardless of how long the sequence is: a distance of 5000 tokens produces the same proportionally large penalty whether the model was trained on sequences of length 512 or 8192, whereas both sinusoidal and rotary schemes are built out of periodic functions whose behavior at very large positions, unseen during training, is not guaranteed to resemble anything the model learned to interpret correctly during training. In the original paper's experiments, models trained with ALiBi on relatively short sequences and then evaluated on much longer ones at inference time showed markedly better perplexity than models using sinusoidal encodings evaluated in the same extrapolation regime, essentially because the penalty function is monotonic and unbounded rather than periodic, so it doesn't need to "recognize" a distance value it never saw during training β€” larger distances simply produce proportionally larger penalties, extending naturally past the training range.

7. Comparing the four approaches

Stepping back, all four schemes covered in this chapter are answers to the same problem, and it's worth being able to place them on a common set of axes, since interviewers will often probe exactly this comparison. Sinusoidal and learned absolute encodings both inject positional information once, additively, into the embeddings before the first attention layer, and differ only in whether that signal is a fixed function of position or a learned table β€” the fixed function generalizes somewhat better to unseen lengths, the learned table fits the training distribution somewhat better. Relative position representations, RoPE, and ALiBi all instead inject positional information directly into the attention computation itself, at every layer, and differ in mechanism: Shaw et al. use a learned embedding indexed by clipped relative offset, RoPE uses a fixed geometric rotation whose composition algebraically produces relative-offset-dependent dot products, and ALiBi uses a fixed linear penalty subtracted directly from attention scores. Roughly speaking, the field's trajectory across these four ideas has been toward mechanisms that are (a) computed at every attention layer rather than added once at the input, since that gives every layer direct access to relative position rather than relying on it surviving through many layers of transformation, and (b) defined by simple, unbounded, or periodic-but-composable closed-form functions rather than finite learned tables, precisely because that's what makes extrapolation past the training length possible at all.

A modern data point worth adding to this comparison, since it's easy to miss if you only think in terms of "which of the four": some decoder-only models are trained with no explicit positional encoding at all, an approach studied directly by Kazemnejad et al. under the name NoPE. The reason this is even viable is specific to the causal, decoder-only shape (Chapter 2.3): a causal attention mask already breaks the permutation symmetry that would otherwise make position undetectable, since a token's set of visible predecessors changes with its position in the sequence, so the model has an implicit signal it can in principle learn to exploit even with zero explicit positional information injected anywhere. Kazemnejad et al. found that NoPE models not only match explicitly-positioned models on in-distribution lengths, but generalize to longer sequences at least as well as RoPE or ALiBi, without needing any of the periodicity or decay tricks those schemes rely on β€” a striking result given how much engineering effort the rest of this chapter documents going into designing good positional signals. The honest caveat is that NoPE's implicit position signal is weaker and less directly controllable than an explicit one, so it hasn't displaced RoPE as the default choice in frontier models, but it's a useful reminder that "no positional encoding" is itself a legitimate fifth point on this comparison, not just a theoretical curiosity.

This progression β€” from injecting position once at the input, to injecting it directly and repeatedly into attention, in a form built to survive well beyond the training distribution β€” sets up a theme that recurs throughout the rest of this book: many of the most consequential architectural innovations in the LLM era are not changes to what a Transformer can represent in principle, but changes to how gracefully it behaves outside the exact regime it was trained in, whether that's longer sequences, larger models, or different data distributions. With attention and positional information both in place, Chapter 2.3 turns to a different kind of design choice: not how a Transformer represents order, but which of its encoder and decoder halves it keeps at all.

8. Interview angle

"Why does a Transformer need positional encoding at all β€” what specifically breaks without it?" A strong answer states the permutation-invariance argument precisely: self-attention's output is a weighted sum over value vectors where weights come from a softmax over query-key dot products, and this computation is unchanged under any permutation of the input positions, so without an explicit positional signal, "the dog bit the man" and "the man bit the dog" would produce identical token-level representations from self-attention alone. The strongest answers connect this directly to the tradeoff made in Chapter 2.1 β€” recurrence gave positional information for free via sequential processing, and removing recurrence for parallelism means that information has to be bought back explicitly.

"Derive why RoPE's attention scores depend only on relative position." The answer should walk through the rotation-composition argument: rotating the query at position $m$ by angle $m\theta$ and the key at position $n$ by angle $n\theta$, the dot product of the rotated vectors reduces to $q_m^\top R((n-m)\theta) k_n$ because $R(m\theta)^\top R(n\theta) = R((n-m)\theta)$, a basic property of rotation matrices. The absolute positions cancel out algebraically, leaving a function of only $n - m$. A strong answer notes this is exact and structural, not an approximation or something learned.

"Why does ALiBi extrapolate to longer sequences better than sinusoidal or learned absolute encodings?" A strong answer notes that ALiBi's bias is a simple, monotonic, unbounded linear function of distance that behaves consistently regardless of how far apart two positions are, so a distance never seen during training still produces a sensibly scaled penalty. Learned absolute embeddings have literally no representation for positions beyond the training range. Sinusoidal encodings are defined at any position in principle, but their periodicity means very large, unseen positions can look numerically similar to training-range positions in ways the model never learned to handle correctly.

"Compare relative position representations, RoPE, and ALiBi along the axis of where and how they inject positional information." A strong answer places all three as attention-level (not embedding-level) mechanisms, then differentiates by implementation: Shaw et al. add a learned, clipped-relative-offset vector into the key/value computation inside attention; RoPE rotates queries and keys by a position-dependent angle so relative offset emerges algebraically from the dot product; ALiBi adds a fixed linear penalty directly to attention scores based on distance, with no modification to queries or keys at all. The strong answer notes ALiBi is the cheapest and most extrapolation-friendly of the three precisely because it's not tied to any periodic or learned functional form.

"If you had to design a positional scheme for a model expected to handle both very short prompts and 1M-token contexts, what would you consider, and why?" A strong answer reasons from the extrapolation properties discussed in this chapter: prefer a scheme injected at every attention layer rather than once at the input, and prefer one defined by a simple, monotonic, or composably periodic closed-form function over a finite learned table, since the model must handle lengths far beyond anything guaranteed at training time. It's reasonable to mention that RoPE and ALiBi are both real, current choices in production long-context systems, and that further techniques for extending effective context length beyond training length are an active area covered later in the book.

9. Self-check questions

  1. Why exactly is self-attention permutation-invariant, and what specific property of the attention formula from Chapter 2.1 causes this?
  2. Explain the angle-addition property of sinusoidal positional encodings, and what it implies (and doesn't fully guarantee) about relative-position recoverability.
  3. What is the fundamental limitation shared by both sinusoidal and learned absolute position encodings with respect to sequences longer than those seen in training, and why does each fail for a slightly different reason?
  4. What conceptual shift does Shaw et al.'s relative position representations make relative to the two absolute encoding schemes, in terms of where positional information is injected?
  5. Derive, in your own words, why rotating queries and keys by position-dependent angles causes their dot product to depend only on relative offset.
  6. How does ALiBi modify the attention computation, and why does its specific functional form (linear, unbounded, monotonic) matter for extrapolation?
  7. Rank the four schemes in this chapter by how well you'd expect them to extrapolate to sequence lengths well beyond training, and justify the ranking using the mechanisms described in this chapter.

10. Sources

  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762. https://arxiv.org/abs/1706.03762 (sinusoidal positional encoding, Section 3.5; cross-referenced from Chapter 2.1)
  • Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). Self-Attention with Relative Position Representations. NAACL 2018. arXiv:1803.02155. https://arxiv.org/abs/1803.02155
  • Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing 2024. arXiv:2104.09864. https://arxiv.org/abs/2104.09864
  • Press, O., Smith, N. A., & Lewis, M. (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR 2022. arXiv:2108.12409. https://arxiv.org/abs/2108.12409
  • Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., & Reddy, S. (2023). The Impact of Positional Encoding on Length Generalization in Transformers (NoPE). NeurIPS 2023. arXiv:2305.19466. https://arxiv.org/abs/2305.19466

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

Why is self-attention, as defined in Chapter 2.1, permutation-invariant?

Explanation: Shuffling the input tokens (and shuffling the outputs back) leaves the computation unchanged, since nothing in the attention formula references position β€” that's exactly the gap this chapter fills.

What limitation do both sinusoidal and learned absolute position encodings share regarding sequences longer than those seen in training?

Explanation: A learned table simply has no row for position 5000 if it only ever saw sequences up to 2048. Sinusoidal encodings can be evaluated anywhere, but their periodic behavior at unseen lengths isn't guaranteed to be interpreted correctly.

What is RoPE's central mathematical trick for making attention scores depend only on relative position?

Explanation: Because R(mΞΈ)α΅€R(nΞΈ) = R((nβˆ’m)ΞΈ), the absolute positions m and n cancel out algebraically, leaving a function of only the offset nβˆ’m β€” no learned table, no additive signal.

Why does ALiBi extrapolate to longer sequences better than sinusoidal or learned absolute encodings?

Explanation: A distance of 5000 produces a proportionally large penalty whether training length was 512 or 8192 β€” unlike periodic sinusoidal functions, there's no unseen-regime behavior to worry about.

Using the sinusoidal encoding formula $PE_{(pos, 2i)} = \sin\left(\dfrac{pos}{10000^{2i/d_{\text{model}}}}\right)$, what is $PE_{(1, 0)}$ (position 1, dimension index $i=0$)? (Round to 2 decimal places.)

Explanation: At i=0, the denominator 10000^0 = 1, so PE_(1,0) = sin(1/1) = sin(1 radian) β‰ˆ 0.84.

Using ALiBi's score formula $\text{score}(i,j) = q_i \cdot k_j - m \cdot (i - j)$, if $q_i \cdot k_j = 10$, the head-specific slope $m = 0.5$, and $i - j = 6$, what is the modified attention score?

Explanation: 10 βˆ’ 0.5 Γ— 6 = 10 βˆ’ 3 = 7.

In RoPE, a query at position $m = 4$ is rotated by angle $m\theta$ and a key at position $n = 10$ is rotated by angle $n\theta$, with $\theta = 0.5$ radians. What is the effective relative rotation angle $(n - m)\theta$ that ends up governing their dot product?

Explanation: (n βˆ’ m)ΞΈ = (10 βˆ’ 4) Γ— 0.5 = 6 Γ— 0.5 = 3.0 radians β€” only the offset matters, not the absolute positions.

Shaw et al.'s relative position representations clip the relative distance to a maximum range of 16. If two tokens are actually 40 positions apart, what clipped relative-distance value gets used in the computation?

Explanation: Distances beyond the maximum range are grouped together and clipped to that maximum β€” here, 16, regardless of the true distance being 40.