Chapter 3.4 β€” Inference-Time Efficiency

Contents

  1. From bigger-for-free to actually serving the model
  2. Why generation is a different computational regime than training
  3. The KV-cache: eliminating redundant recomputation
  4. Quantization: GPTQ and AWQ
  5. Speculative decoding: trading draft compute for latency
  6. PagedAttention and vLLM: paging ideas applied to the KV-cache
  7. How these techniques stack in a real serving system
  8. Interview angle
  9. Self-check questions
  10. Sources

1. From bigger-for-free to actually serving the model

Chapter 3.3 showed how Mixture of Experts lets a model's total parameter count grow enormously while its per-token compute stays roughly fixed, and closed by flagging that actually serving any large model β€” MoE or dense β€” to real users surfaces its own distinct engineering problems that have nothing to do with training-time FLOPs. This chapter is about exactly that gap. Everything in Part II and the preceding chapters of Part III was implicitly framed around training: given a batch of full sequences, compute a forward and backward pass as efficiently as possible. Inference is a different problem wearing the same architecture's clothes, and the difference matters enough that most of the techniques in this chapter simply don't have an analogue on the training side.

The reason inference deserves its own chapter, rather than being a footnote to training efficiency, is that autoregressive generation is sequential by construction: a decoder-only model produces one token, then must feed that token back in to produce the next one, and so on, one at a time, for as many tokens as the response requires. Training processes an entire sequence's worth of tokens in parallel because every target token is already known in advance. Inference cannot do this β€” token $t+1$ literally does not exist until token $t$ has been sampled β€” and that sequential dependency is the root cause of essentially every technique in this chapter, from the KV-cache to speculative decoding to the memory management problem that PagedAttention solves.

2. Why generation is a different computational regime than training

To see precisely why this sequential structure matters, it helps to compare the arithmetic intensity of the two regimes. Arithmetic intensity is the ratio of compute performed to data moved from memory β€” $I = \text{FLOPs} / \text{bytes moved}$; a workload is compute-bound if it does a lot of arithmetic per byte moved, and memory-bandwidth-bound if it moves a lot of data relative to the arithmetic performed on it. Training processes long sequences and, in practice, large batches, in parallel β€” a single set of weights gets reused across a huge number of tokens within one forward pass, which means the same weight matrices, once loaded from memory, get multiplied against many different token vectors before more data needs to be fetched. That's a favorable ratio of compute to memory movement, and training is generally compute-bound, meaning the GPU's raw FLOP throughput is usually the binding constraint.

It's worth being precise that "generation" is actually two distinct phases with opposite arithmetic profiles, a distinction the rest of this chapter and the next one both lean on by name: prefill, the single forward pass that processes the entire input prompt at once to produce the first output token, and decode, every subsequent step that produces one more token at a time. Prefill looks computationally like training β€” it processes many tokens (the whole prompt) in one parallel pass, reusing loaded weights across all of them, so it's typically compute-bound the same way training is. Decode inverts this. At each decoding step, the model produces exactly one new token, meaning the same enormous set of weight matrices has to be loaded from memory and used to process just one new token vector (or a small batch of them, if serving multiple requests at once) before the process repeats for the next token. The compute done per byte of weight data moved is comparatively tiny, and unless you're serving many concurrent requests, generation tends to be memory-bandwidth-bound rather than compute-bound β€” the GPU spends much of its time waiting for weights to arrive from memory rather than saturating its arithmetic units. This is the same underlying hardware fact that motivated FlashAttention in Chapter 3.1, but it shows up here in a different guise, applying to the entire model's weights on every decoding step rather than specifically to the attention score matrix, and it is the reason essentially every technique in this chapter is, in one way or another, about reducing memory traffic or reducing the number of sequential steps, rather than about reducing raw FLOP count.

3. The KV-cache: eliminating redundant recomputation

The most basic inefficiency in naive autoregressive generation is easy to miss precisely because it's so structural: without any caching, generating token $t+1$ by feeding the model the sequence $x_1, \ldots, x_t$ requires recomputing the key and value projections for every one of those $t$ tokens at every layer, even though $x_1$ through $x_{t-1}$ were already processed, identically, when generating token $t$ one step earlier. Nothing about those earlier tokens' keys and values changes as generation proceeds β€” a token's key and value projections at a given layer depend only on that token's own representation flowing into that layer, not on which later tokens have since been generated β€” so recomputing them at every subsequent step is pure waste, redone from scratch, every single step, for a growing prefix.

The KV-cache eliminates this waste directly: at each layer, cache the key and value vectors for every token as soon as they're computed, and on each subsequent decoding step, compute the query, key, and value only for the single newest token, then attend that one query against the full set of cached keys and values (the new key and value are appended to the cache for future steps). This turns each decoding step's attention computation from "reprocess the entire sequence so far" into "process exactly one new token, against an already-computed history" β€” a large and immediate win, converting what would otherwise be quadratic-in-total-work generation (recomputing an ever-growing prefix at every step) into linear-in-total-work generation. The KV-cache is universally used in production generation and is about as close to a free lunch as this chapter offers, but it isn't entirely free: the cache itself takes memory, and that memory grows linearly with both sequence length and batch size β€” for $L$ layers, $h$ heads of dimension $d_h$, sequence length $n$, batch size $b$, and $p$ bytes per element, $\text{cache bytes} = 2 \cdot L \cdot h \cdot d_h \cdot n \cdot b \cdot p$, with the leading $2$ for keys and values (more concurrent requests means more separate caches to hold simultaneously) β€” which is precisely why Chapter 3.1 and Chapter 3.2's attention-efficiency techniques matter doubly at inference time β€” a sparse attention pattern or a bounded sliding window doesn't just save compute during a single forward pass, it also directly shrinks the size of the KV-cache that has to be held in memory throughout an entire generation, which for long-context serving can become the binding constraint on how many requests a system can serve concurrently.

4. Quantization: GPTQ and AWQ

If generation is typically memory-bandwidth-bound, one of the most direct ways to speed it up is to simply move less data β€” and the most direct way to move less data is to represent the model's weights with fewer bits. Quantization does exactly this: instead of storing weights as 16-bit floating point numbers, store them as 8-bit or 4-bit integers (with a small amount of extra per-group scaling information to map the reduced-precision integers back to a usable range), which shrinks the memory footprint of the weights and, since inference is memory-bandwidth-bound, correspondingly speeds up how quickly they can be streamed from memory during generation. The obvious risk is that throwing away precision degrades the model's outputs, and the interesting engineering content of both methods below is in how they minimize that degradation.

GPTQ approaches this as a layer-by-layer optimization problem with careful error correction rather than naively rounding every weight to the nearest representable low-precision value. It quantizes weights column by column within each layer, and after quantizing each column, it adjusts the remaining, not-yet-quantized columns to compensate for the error just introduced β€” using an approximate, computationally tractable form of second-order (Hessian-based) information about the layer's loss surface to decide how to distribute that compensation, rather than treating every remaining weight as equally important to correct. This second-order-aware, sequential correction process is what allows GPTQ to push weights down to very low precision (commonly 4 bits) while keeping the resulting model's outputs close to the original, full-precision model's outputs β€” a naive independent rounding of every weight, by contrast, tends to accumulate much larger errors, because it never accounts for how an error introduced in one weight interacts with the others.

AWQ takes a different angle on the same underlying problem, motivated by an empirical observation: not all weight channels matter equally to a model's outputs, and a small fraction of channels β€” specifically, those corresponding to activations with disproportionately large magnitudes β€” are disproportionately important to preserving accuracy. Rather than quantizing every weight uniformly, AWQ identifies these salient channels (using activation statistics rather than weight statistics, hence "activation-aware") and protects them, effectively by rescaling weights and activations before quantization so that the small set of important channels retain more effective precision, while the rest of the weights are quantized more aggressively. Because this rescaling can be folded into the model's parameters ahead of time, AWQ avoids the mixed-precision runtime complexity of literally keeping some weights at higher bit-width than others, while still concentrating quantization error away from the channels that matter most. Both methods are examples of the same broader principle: quantization error isn't uniform in its impact, and the state of the art in post-training quantization comes from methods that model where that impact concentrates β€” via second-order loss information in GPTQ's case, via activation-magnitude statistics in AWQ's case β€” rather than distributing error blindly.

5. Speculative decoding: trading draft compute for latency

The KV-cache and quantization both attack the cost of a single decoding step. Speculative decoding attacks a different bottleneck entirely: the fact that generation is strictly sequential, one token at a time, no matter how cheap any individual step becomes. Even with a perfectly optimized single-step cost, producing a hundred tokens still requires a hundred sequential round trips through the model, and each of those round trips carries fixed latency overhead (kernel launch costs, memory-bandwidth-bound weight loading, and so on) that doesn't shrink just because the step itself is efficient. Speculative decoding, introduced concurrently by Leviathan, Kalman, and Matias at Google and by Chen, Borgeaud, Irving, Lespiau, Sifre, and Jumper at DeepMind (under the name speculative sampling), reduces the number of sequential large-model steps needed, rather than the cost of any individual step.

The mechanism works by pairing the large target model with a much smaller, faster draft model that approximates it. At each round, the draft model proposes several tokens ahead β€” say, five tokens β€” generated quickly and cheaply since the draft model is small. The large target model then verifies all five proposed tokens in a single forward pass, computed in parallel across all five positions at once (this is possible because, given a fixed sequence of tokens to score, computing the target model's probabilities for each position doesn't require sequential generation β€” that parallelism is exactly what training-time forward passes already exploit, and speculative decoding borrows it for verification at inference time). The verification step uses a careful acceptance rule based on comparing the draft model's and target model's probabilities for each proposed token, accepting a prefix of the draft's proposals for as long as they remain consistent with what the target model would have generated (accepting outright when the target model agrees strongly, and probabilistically rejecting and resampling when it doesn't), so that the tokens ultimately emitted are guaranteed to be distributed exactly as if the large target model had generated them token by token on its own β€” speculative decoding changes latency, not the output distribution.

The payoff is that whenever the draft model's guesses are good, several tokens get accepted per single target-model forward pass, collapsing what would have been several sequential expensive steps into one. The cost is the extra compute spent running the draft model and verifying proposals that ultimately get rejected when the draft and target disagree, plus the practical burden of maintaining and keeping in sync a second, smaller model purpose-built to approximate the first. This is a clean example of a broader idea worth remembering for its own sake: trading extra, cheaper compute (a small draft model's forward passes) for a reduction in the number of expensive sequential steps (the large model's forward passes) β€” a shape of tradeoff that recurs whenever sequential latency, rather than raw throughput, is the thing actually being optimized.

The draft model itself doesn't have to be a separate, independently trained network β€” that "practical burden" clause above is exactly what more recent approaches attack. Multi-token prediction (MTP) methods attach extra prediction heads directly onto the target model, or a lightweight module trained jointly with it, that predict several tokens ahead from the target model's own forward pass, rather than running a whole second autoregressive model. Medusa bolts several extra heads onto a frozen base model's final hidden state; EAGLE improves on plain logit-level heads by extrapolating in feature space β€” using the target model's own hidden-state dynamics to predict future hidden states, which turns out to be more accurate than predicting future tokens directly; and DeepSeek-V3 trains a dedicated MTP module jointly with the base model from pretraining onward, specifically so it doubles as an accurate speculative drafter at inference time without ever being a separate model to keep synchronized. Diffusion drafters push the same idea further: a small diffusion language model, which denoises a whole block of positions at once rather than generating strictly left to right, can propose an entire multi-token continuation in a single drafting step instead of one token at a time, handing the target model a wider batch of candidates to verify per round without the draft process itself paying any sequential cost.

6. PagedAttention and vLLM: paging ideas applied to the KV-cache

The KV-cache solves the redundant-recomputation problem, but it introduces a resource management problem of its own, which becomes acute the moment a serving system tries to handle many concurrent requests rather than one request at a time. Each request's KV-cache needs to be stored somewhere in GPU memory for the entire duration of that request's generation, and a naive implementation typically allocates a large, contiguous block of memory for each request's cache sized for the maximum sequence length it might ever need β€” which wastes enormous amounts of memory on requests that end up shorter than that maximum, and, because memory is allocated in large contiguous chunks, leads to fragmentation: even when there's technically enough free memory in aggregate to serve a new request, that memory might be scattered across many non-contiguous gaps too small individually to satisfy a new large contiguous allocation.

Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, and Stoica's PagedAttention, implemented in the vLLM serving system, borrows a decades-old idea directly from operating systems to fix this: virtual memory paging. Instead of storing each request's KV-cache in one contiguous block, PagedAttention divides the cache into fixed-size blocks (pages) and allows a request's logical sequence of cache entries to be mapped onto a set of physical blocks that need not be contiguous in memory, exactly the way an operating system's virtual memory maps a process's logical address space onto scattered physical memory pages. This eliminates the fragmentation problem almost by construction, since any free block anywhere in memory can satisfy the next chunk of any request's growing cache, rather than needing to find one large contiguous span. It also enables a further, valuable optimization: when multiple requests share a common prefix (the same system prompt, for instance, or the shared portion of several parallel sampling attempts from the same prompt), their logical caches can map to the same physical blocks for that shared portion, copying only when one of the requests diverges from the others (a copy-on-write scheme directly analogous to how operating systems handle shared memory pages between processes).

The practical payoff is a large increase in achievable throughput via continuous batching β€” the serving system can pack far more concurrent requests into the same GPU memory than a naive contiguous-allocation scheme would allow, because memory is no longer wasted on over-provisioned, under-utilized contiguous blocks and no longer fragmented into unusable gaps. This matters enormously in production, because serving cost is generally dominated by how many requests a fixed amount of GPU memory and compute can serve concurrently, and PagedAttention's contribution is specifically to that dimension, independent of anything covered earlier in this chapter about the cost of any individual token's generation.

7. How these techniques stack in a real serving system

As with Chapter 3.2's long-context techniques, it's worth being explicit that a real production serving system does not pick one of these four ideas β€” it stacks all of them, because each addresses a genuinely different bottleneck. The KV-cache eliminates redundant computation across decoding steps and is essentially assumed by everything else in this chapter. Quantization shrinks the memory footprint and bandwidth demands of the weights themselves (and, in some implementations, of the KV-cache too), directly attacking the memory-bandwidth-bound nature of generation described in section 2. Speculative decoding reduces the number of sequential large-model steps needed to produce a given amount of output, attacking latency specifically rather than throughput or memory. PagedAttention attacks a resource-management problem that only becomes visible once you're serving many concurrent requests rather than one, maximizing how many of those requests a fixed pool of GPU memory can hold at once. None of the four substitute for each other, and a system that only implements one or two of them is leaving a specific, identifiable class of efficiency on the table.

Two further serving techniques target the prefill/decode split named in section 2 directly, and they're worth knowing because they attack a scheduling problem the four techniques above don't touch. A long prompt's prefill pass is compute-heavy enough that running it to completion can block decode steps for every other in-flight request on the same accelerator, inflating their latency even though those requests have nothing to do with the new one arriving. Chunked prefill fixes this by splitting a long prompt's prefill into smaller chunks and interleaving them with other requests' decode steps rather than running the whole prefill as one uninterrupted block, trading a small amount of extra bookkeeping for much more even latency across concurrent requests. Prefill/decode disaggregation goes a step further and separates the two phases onto entirely different accelerators or pools of them β€” one pool tuned and provisioned for prefill's compute-bound profile, another for decode's memory-bandwidth-bound one β€” since the two phases want different hardware tradeoffs and, run on shared hardware, tend to interfere with each other's latency in exactly the way chunked prefill is patching around. Both techniques are about scheduling and resource allocation between requests rather than about any single request's own efficiency, which is precisely the kind of problem that only shows up once a serving system has to juggle many concurrent requests rather than optimize one in isolation.

Zooming out across this entire chapter and Part III so far, the pattern that connects Chapters 3.1 through 3.4 is that the "Transformer" as a pure architectural idea, fully specified by Part II, is nowhere near sufficient by itself to explain how today's largest models are actually trained and served β€” every chapter in this part has been about a gap between the clean mathematical description of the architecture and the messy, hardware- and systems-level reality of running it at scale, whether that gap was attention's quadratic cost, positional encodings' generalization limits, the desire for more capacity without more compute, or the specific sequential and memory-management demands of autoregressive generation. Every fix in this chapter, though, has been applied to a model whose architecture was already fixed β€” you can quantize, page, and window a trained model's cache, but you cannot change how many key and value vectors it produces per token. That number is decided when the architecture is, and it turns out to be the single most consequential number for serving cost. Chapter 3.5 is about the three successive attempts the field has made to shrink it.

8. Interview angle

"Why is LLM inference typically described as memory-bandwidth-bound rather than compute-bound, and how does this differ from training?" A strong answer explains arithmetic intensity directly: training reuses loaded weights across many tokens in parallel within one forward/backward pass, giving a favorable compute-to-memory-movement ratio and making training generally compute-bound. Autoregressive generation processes one new token (or a small batch) per step, so the same large weight matrices must be reloaded from memory to do comparatively little arithmetic each step, making single-request generation typically memory-bandwidth-bound β€” which is precisely why techniques that shrink memory traffic (KV-cache, quantization) or reduce the number of sequential steps (speculative decoding) are central to inference efficiency in a way they aren't to training efficiency.

"Walk me through exactly what would go wrong, computationally, if a serving system didn't use a KV-cache." A strong answer identifies the specific waste: without caching, generating each new token would require recomputing the key and value projections, at every layer, for every token generated so far β€” even though those projections for already-generated tokens never change β€” turning generation into an increasingly wasteful process that redoes an ever-growing amount of prior work at every single step. The KV-cache fixes this by computing keys and values for each token exactly once and reusing them, reducing each step's attention computation to processing only the newest token against a cached history.

"How does GPTQ decide how aggressively to quantize different weights, and why does it need this at all rather than uniform rounding?" A strong answer explains that naive independent rounding of every weight to a lower-precision value accumulates error uncontrolled, because it ignores how an error in one weight interacts with the rest of the layer's output. GPTQ instead quantizes weights column by column and, after quantizing each column, adjusts the remaining unquantized columns to compensate for the error just introduced, using approximate second-order (Hessian) information about the loss surface to decide how to distribute that compensation, which keeps the quantized layer's output much closer to the original than naive rounding would.

"Does speculative decoding change what the model generates, or just how fast it generates it?" A strong answer states clearly that it changes only speed, not the output distribution: the draft model's proposed tokens are verified against the target model in a way (an acceptance/rejection rule comparing draft and target probabilities) specifically designed so the final emitted tokens are distributed exactly as if the target model alone had generated them sequentially. The speedup comes from doing that verification for several proposed tokens in a single parallel forward pass of the target model, collapsing multiple sequential expensive steps into one whenever the draft's guesses are accepted.

"What specific operating-systems idea does PagedAttention borrow, and what problem does it solve that the KV-cache alone doesn't?" A strong answer names virtual memory paging directly: instead of storing each request's KV-cache in one contiguous, worst-case-sized memory block (which wastes memory on overprovisioning and fragments free memory into unusable gaps), PagedAttention divides caches into fixed-size blocks that can map to scattered, non-contiguous physical memory, exactly like an OS's virtual-to-physical page mapping. This solves a resource-management and fragmentation problem specific to serving many concurrent requests, distinct from the redundant-computation problem that the KV-cache itself was invented to solve, and it's the reason vLLM achieves substantially higher throughput via denser continuous batching.

9. Self-check questions

  1. Explain why training a Transformer is typically compute-bound while single-request autoregressive generation is typically memory-bandwidth-bound, in terms of arithmetic intensity.
  2. Precisely describe the redundant computation that occurs during naive (non-cached) autoregressive generation, and explain why a token's cached key and value never need to be recomputed once generated.
  3. Why does the KV-cache's memory cost make Chapter 3.1's and Chapter 3.2's attention-efficiency techniques doubly important at inference time, beyond their original benefit during a single forward pass?
  4. Contrast GPTQ's and AWQ's strategies for minimizing quantization-induced accuracy loss β€” what statistic does each rely on to decide where to concentrate precision?
  5. In speculative decoding, why is verifying several draft-proposed tokens in one target-model forward pass possible, when generating those same tokens autoregressively from the target model would not be?
  6. Explain why speculative decoding's acceptance rule guarantees the same output distribution as standard decoding from the target model alone, despite involving a different, smaller model in the process.
  7. What specific inefficiency in KV-cache memory management does PagedAttention solve, and what OS-level concept does its solution mirror?

10. Sources

  • Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180. https://arxiv.org/abs/2309.06180
  • Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023. arXiv:2210.17323. https://arxiv.org/abs/2210.17323
  • Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., & Han, S. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys 2024. arXiv:2306.00978. https://arxiv.org/abs/2306.00978
  • Leviathan, Y., Kalman, M., & Matias, Y. (2022, Google). Fast Inference from Transformers via Speculative Decoding. ICML 2023. arXiv:2211.17192. https://arxiv.org/abs/2211.17192
  • Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., & Jumper, J. (2023, DeepMind). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318. https://arxiv.org/abs/2302.01318
  • Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., & Dao, T. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv:2401.10774. https://arxiv.org/abs/2401.10774
  • Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. arXiv:2401.15077. https://arxiv.org/abs/2401.15077
  • DeepSeek-AI (2024). DeepSeek-V3 Technical Report (introduces its multi-token prediction module). arXiv:2412.19437. https://arxiv.org/abs/2412.19437
  • Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., & Ramjee, R. (2023). SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv:2308.16369. https://arxiv.org/abs/2308.16369
  • Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2023, Microsoft). Splitwise: Efficient Generative LLM Inference Using Phase Splitting (prefill/decode disaggregation). arXiv:2311.18677. https://arxiv.org/abs/2311.18677

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

Why is training typically compute-bound while single-request autoregressive generation is typically memory-bandwidth-bound?

Explanation: Arithmetic intensity β€” compute per byte moved β€” is high during training (weights reused across many tokens) and low during single-request generation (weights reloaded for one new token at a time).

What redundant computation does the KV-cache eliminate?

Explanation: A token's key/value projections at a layer never change once computed β€” caching them turns each step into "process one new token against cached history" instead of reprocessing the whole prefix.

What is the key difference between GPTQ's and AWQ's approach to minimizing quantization error?

Explanation: GPTQ corrects sequentially using Hessian-based error-compensation; AWQ instead protects the small set of channels that matter most, identified via activation magnitudes, rather than weight statistics.

Does speculative decoding change what tokens a model ultimately generates?

Explanation: Speculative decoding changes latency, not the output distribution β€” draft proposals are verified against the target model with a rule designed to preserve exact equivalence to standard decoding.

A model's KV-cache stores keys and values (2 tensors) for 24 layers, 16 heads per layer, head dimension 64, sequence length 2048, using 2 bytes per number (fp16). What is the total KV-cache size, in megabytes ($1\text{ MB} = 1{,}048{,}576$ bytes)? Formula: $2 \times \text{layers} \times \text{heads} \times \text{head\_dim} \times \text{seq\_len} \times \text{bytes}$.

Explanation: 2 Γ— 24 Γ— 16 Γ— 64 Γ— 2048 Γ— 2 = 201,326,592 bytes Γ· 1,048,576 β‰ˆ 192 MB.

Quantizing a model's weights from 16-bit floating point to 4-bit integers reduces the memory footprint of those weights by roughly what factor?

Explanation: 16 bits / 4 bits = 4x smaller β€” and since inference is memory-bandwidth-bound, this roughly translates into a proportional speedup in weight loading.

A draft model proposes 5 tokens per round in speculative decoding. On average, the first 3 are accepted before a mismatch triggers resampling. How many tokens are emitted from this round in total (accepted tokens plus the one resampled/corrected token)?

Explanation: 3 accepted + 1 resampled (or newly generated) token = 4 tokens emitted from a single target-model verification pass.

Without speculative decoding, generating 100 tokens requires 100 sequential target-model forward passes. If speculative decoding emits an average of 4 tokens per target-model forward pass, roughly how many target-model forward passes are needed to generate the same 100 tokens?

Explanation: 100 / 4 = 25 β€” collapsing what would have been 100 sequential expensive steps down to roughly 25.