Chapter 3.6 β€” Beyond Attention

Contents

  1. The assumption Part III never questioned
  2. One frame: a linear RNN with a matrix-valued state
  3. State space models: S4 and the convolution-recurrence duality
  4. Mamba: letting the input decide what to remember
  5. RWKV: the same destination from the other direction
  6. DeltaNet: fixing how the state is written
  7. What's still hard, and why the winners are hybrids
  8. Interview angle
  9. Self-check questions
  10. Sources

1. The assumption Part III never questioned

Five chapters of Part III have been about affording attention. Chapter 3.1 made it cheaper in sequence length, Chapter 3.2 made its positional scheme survive longer contexts, Chapter 3.3 bought parameters without buying compute, Chapter 3.4 made a trained model servable, and Chapter 3.5 shrank what serving has to remember. Every one of them left the mechanism itself intact: a token attends to every previous token, softmax-weighted, and the model keeps every previous token's keys and values around so it can. All of the cost in Part III traces back to that one design decision, and none of the fixes touched it.

This chapter is about the line of work that did. The claim it makes is that attention's defining property β€” an unbounded memory that grows with the sequence β€” is not the only way to build a sequence model, and that a fixed-size state updated recurrently can get close, at $O(n)$ total cost and $O(1)$ memory per generated token. Chapter 1.4 already told you what happened the last time the field tried this: RNNs lost to Transformers, decisively, because their sequential dependency made them untrainable in parallel and their fixed state forgot too much. Both objections have real answers now β€” the parallel-training objection was answered by S4 and Mamba's scan formulation, and the forgetting objection is what the entire delta-rule line is about. Whether the answers are good enough is the honest open question this chapter ends on, and the current answer from production models is a compromise nobody proposed in advance.

2. One frame: a linear RNN with a matrix-valued state

The single most useful thing to carry out of this chapter is that Mamba, RWKV, DeltaNet, and vanilla linear attention are not four unrelated architectures. They are one recurrence with four different update rules, and once you see the recurrence the rest is bookkeeping.

Chapter 3.1's section 4 got you most of the way there. Autoregressive linear attention maintains a running sum $S_t = S_{t-1} + \phi(k_t)^\top v_t$ and reads out with $o_t = \phi(q_t) S_t$. The object $S_t \in \mathbb{R}^{d \times d}$ is a matrix-valued hidden state: it is not a summary vector like an LSTM's, it's an associative memory, holding a superposition of key-value bindings written by every token so far. Write the general form as $$S_t = A_t \, S_{t-1} + w_t$$ where $A_t$ says what happens to the existing memory (the forget or decay term) and $w_t$ says what gets written into it. Every model in this chapter is a choice of those two.

Vanilla linear attention chooses $A_t = I$ and $w_t = k_t^\top v_t$: never forget, always add. That combination has an obvious failure mode once you see it as a memory. The state has a fixed capacity of roughly $d$ near-orthogonal key directions; additive writes never remove anything; so as tokens accumulate, new bindings interfere with old ones and the whole memory degrades into a blur. This is not a subtle empirical wrinkle, it's the central problem, and every subsequent model in this chapter is an attempt to fix it β€” either by making $A_t$ shrink the old state (decay and gating), or by making $w_t$ overwrite rather than accumulate (the delta rule), or both.

The most direct fix to try is also one of the most heavily cited: RetNet's retention mechanism (Sun et al., Microsoft), which keeps $A_t$ as a fixed, per-head scalar decay $\gamma < 1$ applied every step ($A_t = \gamma I$) instead of the identity, so old bindings shrink geometrically rather than persisting forever, bounding the interference problem instead of eliminating it outright. What makes RetNet worth naming specifically, beyond being an early instance of the decay idea this chapter keeps returning to, is that its authors show the exact same recurrence can be computed three interchangeable ways depending on what you need: a parallel form (unroll the recurrence into one attention-shaped matrix multiplication, for training throughput), a recurrent form (the $O(1)$-per-token update above, for autoregressive inference), and a chunkwise form (run the parallel form within a chunk of tokens, carry only the chunk-boundary state forward recurrently to the next chunk, for long sequences where neither pure form's tradeoff is acceptable on its own). That three-way equivalence β€” one underlying computation, chosen per use case β€” is a pattern worth watching for in the rest of this chapter's architectures too.

It's worth naming the contrast with attention explicitly, because it's the whole tradeoff in one line. Softmax attention's "state" is the KV-cache, which grows with $n$: it never has to decide what to forget, because it forgets nothing, and it pays for that with the memory and bandwidth costs Chapter 3.5 spent a whole chapter on. A recurrent model's state is fixed size: it costs $O(1)$ per token forever, and in exchange it must decide what to throw away. Everything below is about making that decision well.

3. State space models: S4 and the convolution-recurrence duality

The modern line didn't start from linear attention at all; it started from control theory. A continuous linear state space model maps a signal $x(t)$ to $y(t)$ through a latent state: $$h'(t) = A h(t) + B x(t), \qquad y(t) = C h(t)$$ Discretize it with a step size $\Delta$ and you get exactly a linear RNN, $h_t = \bar{A} h_{t-1} + \bar{B} x_t$, $y_t = C h_t$. Gu, Goel, and RΓ©'s S4 β€” Structured State Space for Sequence modeling, four S-words β€” is built on this, and the reason it's more than a re-labelled RNN comes down to two observations.

The first is the duality that makes it trainable. Because $A$, $B$, and $C$ don't depend on $t$ β€” the system is linear time-invariant β€” unrolling the recurrence gives $y_t = \sum_{j} C\bar{A}^{j}\bar{B}\, x_{t-j}$, which is a convolution with a fixed kernel $\bar{K} = (C\bar{B},\, C\bar{A}\bar{B},\, C\bar{A}^2\bar{B}, \ldots)$. So the same layer has two exact forms: a recurrence you use at inference time, giving constant per-token cost, and a convolution you use at training time, computable for the whole sequence at once via FFT in $O(n \log n)$, fully parallel over positions. That's a direct answer to the objection that killed RNNs β€” there is no sequential dependency during training, because you never run the recurrence during training.

The second is that the choice of $A$ matters enormously, and picking it well is not obvious. A randomly initialized $\bar{A}$ either forgets almost immediately (spectral radius below one) or explodes. HiPPO, the precursor work, derives a specific structured $A$ from the requirement that the state be the coefficients of an optimal polynomial approximation of the input history β€” memory as online function approximation, with a principled answer for what a fixed-size state should store. S4 with HiPPO initialization was the first model to solve Path-X, the 16,000-token pathological task in the Long Range Arena benchmark that every Transformer variant had scored at chance on; without HiPPO initialization the same architecture doesn't work at all.

S4 was a genuine breakthrough on long-range signal tasks and a disappointment on language. The reason is precisely the property that made it fast: linear time-invariance means the same $\bar{A}$ and $\bar{B}$ are applied at every position, so the model's decision about what to remember cannot depend on what the token actually is. It can learn "remember things from roughly this far back," but not "remember this name, ignore these filler words." Language is almost entirely the second kind of problem, and S4-family models failed conspicuously at the diagnostic tasks that require it β€” selective copying, and the induction-head behavior (see the token that followed X last time, predict it again) that Transformers acquire early in training and that correlates strongly with in-context learning.

Sitting between S4 and Mamba historically is H3 (Hungry Hungry Hippos, Fu et al.), a direct attempt to patch exactly this gap without giving up time-invariance: H3 stacks two SSMs per layer with a multiplicative interaction between them, explicitly engineered to emulate the comparison-and-recall behavior an induction head performs, and it closed much of the gap to Transformers on language modeling perplexity while still falling short on the sharpest selective-copying diagnostics. It's a useful waypoint precisely because it shows the field correctly diagnosing which capability SSMs were missing before Mamba's selectivity mechanism went after the underlying cause directly, rather than architecting around the symptom with a fixed, hand-designed interaction.

4. Mamba: letting the input decide what to remember

Gu and Dao's Mamba makes one change, and states plainly that it's the change: make the parameters functions of the input. They call the resulting layer S6 β€” the same S4, plus a Selective mechanism, plus computing it via a Scan, six S-words instead of four. In Mamba's S6 layer, $\Delta$, $B$, and $C$ are all produced by linear projections of $x_t$ rather than being fixed, so the effective $\bar{A}_t$ and $\bar{B}_t$ vary token by token. The step size $\Delta_t$ is the most interpretable of the three β€” a large $\Delta_t$ means "this token matters, reset toward it," a small one means "this token is filler, hold the current state" β€” which is a learned, content-dependent gate in the same family as the LSTM forget gate from Chapter 1.4, now controlling a much larger state. They call this selectivity, and it fixes exactly the failures of section 3: selective copying and induction become solvable, and the resulting model matches Transformers of comparable size on language modeling in their experiments up to a few billion parameters.

The cost is that time-invariance is gone, and with it the convolutional form β€” you cannot FFT a kernel that changes at every position. Mamba's answer is that a linear recurrence is an associative scan, and associative scans parallelize. It's worth spelling out why. Write each step as a pair $(\bar{A}_t, b_t)$, with $b_t = \bar{B}_t x_t$, so applying the step is $h \mapsto \bar{A}_t h + b_t$. Composing two steps is itself an affine map of exactly this same form: $h \mapsto \bar{A}_2(\bar{A}_1 h + b_1) + b_2 = (\bar{A}_2\bar{A}_1) h + (\bar{A}_2 b_1 + b_2)$ β€” which means you can define a combination rule $(\bar{A}_2, b_2) \oplus (\bar{A}_1, b_1) = (\bar{A}_2\bar{A}_1,\ \bar{A}_2 b_1 + b_2)$ that is associative: it doesn't matter how you group a chain of steps, only that you preserve their left-to-right order. Associativity is exactly the property a parallel prefix-sum algorithm needs. Take four consecutive steps 1, 2, 3, 4: instead of combining them strictly in order ($1\oplus2$, then that result $\oplus 3$, then $\oplus 4$ β€” three sequential combinations, depth 3), combine adjacent pairs at the same time ($1\oplus2$ and $3\oplus4$ run concurrently), then combine the two results (depth 2). For $n$ steps arranged this way as a balanced binary tree, the sequential depth β€” the number of combinations that must happen one after another β€” is $O(\log n)$ rather than $O(n)$, even though the total work summed across all the parallel branches is still $O(n)$. This is the same trick a parallel prefix-sum (cumulative sum) uses, just with the addition operator replaced by the $\oplus$ rule above.

On top of that they apply the Chapter 3.1 lesson directly: the internal state is expanded to be much larger than the input dimension, so materializing it in HBM at every position would make the layer memory-bound, and instead the scan is fused into a single kernel that keeps the expanded state in SRAM and only writes the outputs back. It is the same IO-aware argument FlashAttention makes, applied to a recurrence β€” and it's a good illustration that "which algorithm is faster" is now inseparable from "which algorithm maps onto the memory hierarchy."

Mamba-2, published a year later, is worth knowing about for a conceptual reason more than a performance one. Dao and Gu show that a restricted form of the selective SSM and a form of linear attention are the same computation viewed through different algebraic lenses β€” the "state space duality" β€” which means the two lineages in this chapter were never really separate, and which lets Mamba-2's implementation be written in terms of large matrix multiplications instead of a bespoke scan. That matters practically, because matrix multiplication is what modern accelerators are built for, and it made Mamba-2 several times faster than Mamba on the same hardware.

5. RWKV: the same destination from the other direction

RWKV, developed by Peng and a large open-source collaboration, arrived at a very similar place from a completely different starting point β€” not control theory, but a direct attempt to strip attention out of a Transformer block while keeping the block's shape. Each RWKV layer has two sublayers mirroring the Transformer's: time-mixing, which carries the recurrence and replaces attention, and channel-mixing, which is a gated position-wise network in the spirit of Chapter 2.5's FFN.

The time-mixing operator computes, for each channel, a weighted sum over history in which the weight decays exponentially with distance at a learned per-channel rate $w$, plus a separate bonus term $u$ for the current token: $$\text{WKV}_t = \frac{\sum_{i<t} e^{-(t-1-i)w + k_i} v_i + e^{u + k_t} v_t}{\sum_{i<t} e^{-(t-1-i)w + k_i} + e^{u + k_t}}$$ The important structural fact is that the numerator and denominator are both running sums that can be updated incrementally, so this is again a linear recurrence with a fixed-size state β€” parallelizable over the sequence for training, constant-cost per token for generation. The output is then gated by a "receptance" term $\sigma(r_t)$, which is where the R in the name comes from, and which does the same job as Chapter 2.5's SwiGLU gate: a learned, input-dependent decision about how much of the computed value to let through.

RWKV-4, the version that first attracted serious attention by scaling to 14B parameters, used a fixed learned decay $w$ per channel β€” time-invariant, exactly the S4 weakness. The subsequent Eagle and Finch releases (RWKV-5 and RWKV-6) changed that in two steps that should now sound familiar: they promoted the state from a vector to a matrix, and they made the decay data-dependent, computed from the current token rather than fixed. Two lineages that share no mathematical vocabulary, starting from control theory and from attention-free Transformers respectively, converged on "matrix state, input-dependent decay" within about a year of each other. That convergence is decent evidence that the design space really does have the shape section 2 described.

6. DeltaNet: fixing how the state is written

Decay and gating attack the $A_t$ term β€” how the old state fades. The other half of section 2's recurrence is $w_t$, and there's a line of work arguing that's where the real problem is. Schlag, Irie, and Schmidhuber's framing is the sharpest: a linear-attention state $S_t = S_{t-1} + k_t^\top v_t$ is a fast weight matrix being trained by pure Hebbian learning, and Hebbian learning is known to suffer catastrophic interference once the number of stored associations approaches the capacity of the matrix. Writing a new binding for a key similar to one already stored doesn't replace the old association; it superimposes on it. Decay helps only bluntly β€” it fades everything, including the associations you wanted to keep.

The delta rule is the classical fix, and it's sixty years of learning theory arriving exactly where it's needed. Before writing, read: retrieve what the state currently returns for this key, $\hat{v}_t = S_{t-1} k_t$, and write only the correction, scaled by a learned rate $\beta_t$: $$S_t = S_{t-1} + \beta_t (v_t - S_{t-1}k_t)\, k_t^\top = S_{t-1}\big(I - \beta_t k_t k_t^\top\big) + \beta_t v_t k_t^\top$$ The second form is the one to internalize: $I - \beta_t k_t k_t^\top$ is a targeted erasure that removes what was stored at that key specifically, leaving everything stored at other keys untouched, after which the new value is written in. The state now does genuine in-place updating rather than accumulation, and the improvement on associative-recall tasks β€” where a model must retrieve a binding introduced earlier in the context, possibly after it was overwritten β€” is large.

DeltaNet's practical problem was that this looks irreducibly sequential: $S_{t-1}$ appears inside the write, so you can't obviously turn it into a scan or a convolution. It sat mostly unused for three years for that reason. Yang et al. resolved it in 2024 with a chunkwise-parallel reformulation: the product of the per-step erasure matrices $\prod_t (I - \beta_t k_t k_t^\top)$ is a product of rank-one-perturbed identities, which has a compact WY representation that can be computed for a whole chunk with matrix multiplications, letting DeltaNet train at Transformer-competitive speed and, for the first time, at scale.

Gated DeltaNet is the natural closing move, and it's the one that reached production. Yang, Kautz, and Hatamizadeh observe that gating and the delta rule fix different failures β€” a data-dependent global decay $\alpha_t$ is the right tool for "the topic changed, clear the memory," and the delta rule is the right tool for "this one fact was updated, revise that entry" β€” and that there's no reason to choose: $$S_t = S_{t-1}\,\alpha_t\big(I - \beta_t k_t k_t^\top\big) + \beta_t v_t k_t^\top$$ Combining them beats either alone on both language modeling and long-context recall, and the architecture has since been adopted in shipped models, most visibly in Qwen3-Next's hybrid stack. Written next to each other, the whole chapter's progression is one column of a table: $A_t = I$ (linear attention), $A_t = \gamma$ (fixed decay), $A_t = \alpha_t$ (selective gating), $A_t = I - \beta_t k_t k_t^\top$ (delta rule), $A_t = \alpha_t(I - \beta_t k_t k_t^\top)$ (gated delta rule).

7. What's still hard, and why the winners are hybrids

None of this closes the gap, and it's worth being precise about why rather than treating it as a temporary engineering shortfall. A fixed-size state of $d \times d$ numbers can hold a bounded amount of information; a context of $n$ tokens contains information that grows with $n$. For $n$ large enough, exact recall from the state is information-theoretically impossible, no matter how clever the update rule. Attention has no such limit because its state isn't fixed β€” the KV-cache is a lossless record, and that losslessness is precisely what you're paying for when you pay for the cache.

This isn't only a theoretical concern; it shows up exactly where the theory says it should. Jelassi et al. demonstrated a clean separation on copying: Transformers learn to copy long strings from context and generalize to lengths beyond training, while state space models of comparable size fail, and they give both the theoretical argument and the empirical demonstration. The same pattern appears on needle-in-a-haystack retrieval and on multi-query associative recall, where a model must answer several different questions about bindings scattered through a long context β€” one fact might fit in the state, twenty scattered ones don't. Recurrent models are strong at aggregate, compressible properties of a long context and weak at exact retrieval from it, which is not a bug to be fixed but a description of what a compressed state is.

So the field did the obvious thing, and it worked embarrassingly well: use both. A hybrid stack is mostly recurrent layers with full attention layers interleaved at intervals β€” the recurrent layers carry the cheap bulk of the sequence modeling, and the occasional attention layer provides the lossless lookup the recurrent layers can't. AI21's Jamba interleaves one attention layer per eight in a Mamba-plus-MoE stack; Microsoft's Samba pairs Mamba with sliding-window attention; MiniMax-01 places a full-attention layer periodically among linear-attention layers at 456B total parameters; Qwen3-Next combines Gated DeltaNet layers with gated attention at a 3:1 ratio; Google DeepMind's Griffin interleaves a gated linear recurrence (RG-LRU, a learned-decay update in the same family as this chapter's other gating mechanisms) with local sliding-window attention, while its pure-recurrent sibling Hawk drops the attention layers entirely and shows how far the recurrent side alone gets before the recall gap from earlier in this section actually bites. The consistent empirical finding across all of them is that a small minority of attention layers β€” somewhere between one in four and one in eight, depending on the model β€” recovers essentially all of the recall capability while keeping most of the linear-cost benefit β€” a ratio nobody predicted from theory and everybody converged on from experiment.

One more direction worth flagging before closing, since it points at where the field is looking next rather than where it has already landed: test-time training (TTT), proposed by Sun et al., pushes the recurrence idea one step further by making the state itself the weights of a small model that keeps learning online, updated by an actual gradient step on every token rather than one of this chapter's fixed linear update rules β€” the state trains itself on the sequence as generation proceeds, rather than being written to by a rule fixed at pretraining time. It's early-stage relative to the production-proven architectures covered above, but it's a natural extrapolation of the exact question this whole chapter has been asking: what are you allowed to do to a fixed-size state as new tokens arrive, and a learned optimizer step is a strictly more expressive answer than a decay or a delta-rule update, at correspondingly higher cost per token.

That is the honest state of the art, and it's a fitting note for Part III to end on. The question this chapter opened with was "is attention necessary?", and the answer that emerged from five years of work is neither yes nor no: attention is necessary somewhere, in far smaller quantities than a pure Transformer uses, and the interesting engineering question is how little of it you can get away with. Part III as a whole has now taken the architecture Part II specified and asked what it costs β€” in sequence length, in context, in parameters, in serving, in memory, and finally in whether you need the mechanism at all. What none of it has addressed is where the weights come from. Part IV starts at the very beginning of that story, before a single gradient step, with a question so basic it's easy to skip: how does a string of characters become the tokens every chapter so far has silently assumed as its input?

8. Interview angle

This is the most current material in the book, and interviewers use it to separate people who track the field from people who learned it once. Expect questions like:

  • "Explain Mamba to me in terms of something I already know." The strongest answer starts from the recurrence, not from state space theory: it's a linear RNN with a large fixed-size state, made trainable in parallel by the associative-scan formulation, whose forget and input gates are functions of the current token. Naming what selectivity buys β€” content-dependent memory, hence selective copying and induction heads β€” and what it costs β€” the convolutional form, hence the need for a custom scan kernel β€” shows you understand the tradeoff rather than the marketing.
  • "Why did the field move from fixed decay to the delta rule, when both are ways of making room in the state?" The answer must distinguish what each forgets. Decay is untargeted: it fades every stored association equally, including the ones you needed. The delta rule reads the state at the current key first and erases only that key's slot, $I - \beta_t k_t k_t^\top$, leaving other bindings intact. A strong answer adds that Gated DeltaNet uses both because they address different failures, and that the delta rule was stuck for years on the belief that it couldn't be parallelized until the chunkwise WY formulation.
  • "You need a model for 1M-token contexts. Pure Mamba, pure Transformer, or a hybrid β€” and why?" The expected answer names the information-theoretic limit: a fixed state cannot losslessly recall an unbounded context, so pure recurrent models fail on exact retrieval (copying, needle-in-a-haystack, multi-query recall) in a way that isn't fixable by better training. Then it names the hybrid ratio β€” a small minority of layers, roughly one in four to one in eight β€” and cites at least one shipped example. Answering "Mamba, it's linear" without the recall caveat is the trap.
  • "Both RWKV and Mamba ended up with a matrix state and input-dependent decay, from unrelated starting points. What should you take from that?" This one is checking judgment rather than recall. The point is that the design space really is small: given a fixed-size state and a linear recurrence, the only meaningful choices are how the old state is attenuated and how the new one is written, and independent convergence on the same answers is evidence the parameterization details matter less than those two decisions.

The through-line is whether you can hold a tradeoff in mind without collapsing it to a winner. Candidates who have only read announcements tend to describe these models as "Transformer replacements"; candidates who have read the evaluations describe them as a different point on a memory-versus-recall curve, and know which tasks sit on which side of it.

9. Self-check questions

  1. Write the general recurrence $S_t = A_t S_{t-1} + w_t$ and fill in $A_t$ and $w_t$ for vanilla linear attention, a fixed-decay model, and DeltaNet. What failure mode does each successive choice address?
  2. Explain why S4's linear time-invariance is simultaneously what makes it trainable in parallel and what makes it bad at language.
  3. Mamba gives up the convolutional form. What does it use instead, why does that still parallelize over the sequence, and what does the fused-kernel design have in common with FlashAttention?
  4. Write the delta-rule update in the form $S_{t-1}(I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top$ and explain, in terms of stored key-value bindings, what each factor does. Why is this better than decaying the whole state?
  5. What is the information-theoretic argument for why a fixed-size recurrent state cannot match attention on exact long-context recall, and which concrete benchmark behaviors does that argument predict?
  6. Why do production hybrids place attention layers at intervals rather than, say, using attention for the first half of the stack? What would you measure to choose the ratio?
  7. RWKV and Mamba converged on the same two design decisions from unrelated starting points. State both decisions, and explain what that convergence suggests about which parts of these architectures actually matter.

10. Sources

  • Gu, A., Goel, K., & RΓ©, C. (2021). Efficiently Modeling Long Sequences with Structured State Spaces. ICLR 2022. arXiv:2111.00396. https://arxiv.org/abs/2111.00396
  • Gu, A., Dao, T., Ermon, S., Rudra, A., & RΓ©, C. (2020). HiPPO: Recurrent Memory with Optimal Polynomial Projections. NeurIPS 2020. arXiv:2008.07669. https://arxiv.org/abs/2008.07669
  • Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752. https://arxiv.org/abs/2312.00752
  • Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML 2024. arXiv:2405.21060. https://arxiv.org/abs/2405.21060
  • Peng, B., Alcaide, E., Anthony, Q., et al. (2023). RWKV: Reinventing RNNs for the Transformer Era. EMNLP 2023 Findings. arXiv:2305.13048. https://arxiv.org/abs/2305.13048
  • Peng, B., Goldstein, D., Anthony, Q., et al. (2024). Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence. arXiv:2404.05892. https://arxiv.org/abs/2404.05892
  • Schlag, I., Irie, K., & Schmidhuber, J. (2021). Linear Transformers Are Secretly Fast Weight Programmers. ICML 2021. arXiv:2102.11174. https://arxiv.org/abs/2102.11174
  • Yang, S., Wang, B., Zhang, Y., Shen, Y., & Kim, Y. (2024). Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS 2024. arXiv:2406.06484. https://arxiv.org/abs/2406.06484
  • Yang, S., Kautz, J., & Hatamizadeh, A. (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025. arXiv:2412.06464. https://arxiv.org/abs/2412.06464
  • Jelassi, S., Brandfonbrener, D., Kakade, S. M., & Malach, E. (2024). Repeat After Me: Transformers are Better than State Space Models at Copying. ICML 2024. arXiv:2402.01032. https://arxiv.org/abs/2402.01032
  • Lieber, O., Lenz, B., Bata, H., et al. (2024). Jamba: A Hybrid Transformer-Mamba Language Model. arXiv:2403.19887. https://arxiv.org/abs/2403.19887
  • Ren, L., Liu, Y., Lu, Y., Shen, Y., Liang, C., & Chen, W. (2024). Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling. arXiv:2406.07522. https://arxiv.org/abs/2406.07522
  • Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., & Wei, F. (2023, Microsoft). Retentive Network: A Successor to Transformer for Large Language Models (RetNet). arXiv:2307.08621. https://arxiv.org/abs/2307.08621
  • Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., & RΓ©, C. (2022). Hungry Hungry Hippos: Towards Language Modeling with State Space Models (H3). ICLR 2023. arXiv:2212.14052. https://arxiv.org/abs/2212.14052
  • De, S., Smith, S. L., Fernando, A., et al. (2024, Google DeepMind). Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models. arXiv:2402.19427. https://arxiv.org/abs/2402.19427
  • Sun, Y., Li, X., Dalal, K., et al. (2024). Learning to (Learn at Test Time): RNNs with Expressive Hidden States (TTT). arXiv:2407.04620. https://arxiv.org/abs/2407.04620

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

In the unifying recurrence $S_t = A_t S_{t-1} + w_t$, what does vanilla linear attention choose?

Explanation: Linear attention accumulates outer products into a running sum. Because nothing is ever removed, new bindings interfere with old ones once the state's roughly $d$ near-orthogonal key directions fill up.

Why can S4 be trained in parallel over the sequence?

Explanation: With $A$, $B$, $C$ independent of $t$, $y_t = \sum_j C\bar{A}^j\bar{B}x_{t-j}$ β€” a convolution with kernel $(C\bar{B}, C\bar{A}\bar{B}, \ldots)$ evaluated for the whole sequence at once in $O(n\log n)$. The recurrence is only used at inference.

What does linear time-invariance cost S4 on language tasks?

Explanation: S4 can learn "remember things from roughly this far back" but not "remember this name, ignore this filler." That is why S4-family models failed at selective copying and induction-head behavior.

Mamba makes $\Delta$, $B$ and $C$ functions of the input. What does it lose, and what replaces it?

Explanation: A kernel that changes at every position can't be FFT'd. But a linear recurrence is an associative scan, so steps combine pairwise in a tree β€” and Mamba fuses that scan into a kernel keeping the expanded state in SRAM, the same IO-aware argument as FlashAttention.

What is the essential difference between a decay term and the delta rule as ways of making room in the state?

Explanation: The factor $I - \beta_t k_t k_t^\top$ is a targeted erasure of what was stored at $k_t$ specifically, leaving other bindings intact β€” genuine in-place update rather than untargeted fading.

Why did DeltaNet go largely unused for three years after it was proposed?

Explanation: Yang et al. resolved it in 2024: the product of the rank-one-perturbed erasure matrices has a compact WY representation, computable for a whole chunk with matrix multiplications.

What does Gated DeltaNet combine, and why are both needed?

Explanation: $S_t = S_{t-1}\,\alpha_t(I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top$. Gating and the delta rule fix different failures, so combining them beats either alone. Qwen3-Next ships it.

What is the information-theoretic reason a fixed-size recurrent state can't match attention on long-context recall?

Explanation: For large enough $n$, exact recall from a fixed state is impossible regardless of the update rule. Attention has no such limit because the KV-cache is a lossless record β€” which is exactly what you pay for.

Roughly what proportion of attention layers do production hybrids like Jamba and Qwen3-Next use?

Explanation: Jamba uses one in eight, Qwen3-Next a 3:1 Gated-DeltaNet-to-attention ratio. The consistent finding is that a small minority of attention layers recovers essentially all recall ability while keeping most of the linear-cost benefit.