Chapter 6.1 β€” System Design for LLM Products

Contents

  1. From components to a product: what this chapter is and isn't
  2. Requirements and constraints: the questions that come before any architecture
  3. Choosing a base model and scale: compute-optimal vs. inference-optimal
  4. Turning a base model into a chat product: the SFT/RLHF layer
  5. Serving at scale: the non-negotiables
  6. Context length as a product decision, not just a capability
  7. RAG vs. fine-tuning vs. agents: matching the lever to the problem
  8. Evaluation as a continuous process, not a launch gate
  9. Walking the whole loop: a worked design narrative
  10. Interview angle
  11. Self-check questions
  12. Sources

1. From components to a product: what this chapter is and isn't

Part V closed by making a point that's easy to nod along with and easy to underestimate: everything this book has covered β€” attention mechanisms, pretraining objectives, scaling laws, alignment techniques, retrieval, agents, evaluation β€” still has to be assembled into something that actually serves requests, at a cost someone is willing to pay, in a way that can be measured honestly once it's live. That assembly step is not a footnote. It is where a surprising fraction of real LLM engineering work actually happens, and it is where the decisions from every earlier chapter start to trade off against each other instead of being considered in isolation.

This chapter is deliberately not going to introduce new techniques. There is no new architecture, no new training objective, no new attention variant. Instead, this chapter's job is to force a kind of recall and synthesis that individual chapters can't: given a concrete product brief, which of the twenty-odd tools you now have does the job, in what order do you reach for them, and β€” just as importantly β€” which do you deliberately not use because the cost isn't justified by the requirement in front of you? This is also, not coincidentally, close to the shape of a real senior-level system-design interview for an LLM role: the interviewer is rarely testing whether you remember that PagedAttention exists, they're testing whether you can reason about when it matters.

2. Requirements and constraints: the questions that come before any architecture

Every real system design starts with questions that have nothing to do with transformers. What is the product? A general-purpose assistant, a customer-support bot for a specific domain, a coding assistant, a search-augmented Q&A tool? Who are the users, and how many of them, and how often do they issue a request? What is the latency budget per request β€” is this a synchronous chat turn that a human is staring at, where time-to-first-token matters enormously, or a batch job that can tolerate minutes? What's the cost budget, expressed as dollars per million tokens served, and how does that budget change as usage grows? What does "acceptable" mean for this product in terms of factual reliability, tone, and safety β€” a legal-research assistant and a creative-writing companion have almost opposite tolerance for hallucination.

These questions matter because they determine which parts of the book are load-bearing for this particular design and which are irrelevant. A low-traffic internal tool for a hundred engineers has completely different serving-layer requirements than a consumer product serving tens of millions of daily active users; a system that needs to reason over an entire multi-hundred-page contract has different context-length requirements than one that answers short factual questions. A senior engineer's first move is almost never "pick the model." It's pinning down these constraints, because they determine the entire shape of everything downstream β€” and an interviewer watching you skip straight to architecture, without asking about scale, latency, and cost, is watching you miss the actual point of the exercise.

3. Choosing a base model and scale: compute-optimal vs. inference-optimal

Once requirements are pinned down, the first real engineering decision is what base model to start from, and at what scale. Chapter 4.3's scaling-law material is the tool that governs this decision, but it's worth restating the tension in product terms rather than pure training-cost terms. The Chinchilla-style compute-optimal frontier tells you, for a fixed training compute budget, the model size and token count that minimizes training loss. But a product that will be served billions of times over its lifetime does not actually want to minimize training loss for a fixed training budget β€” it wants to minimize total cost of ownership, which is dominated by inference cost, not training cost, once the model ships. This is exactly why LLaMA and LLaMA 2 were trained the way they were: Touvron et al.'s LLaMA paper explicitly trained smaller models on far more tokens than the compute-optimal frontier would suggest, because a smaller model that's cheaper to serve at scale, even if it cost more to train per parameter, wins on total lifecycle cost for a widely-deployed product. This is the inference-optimal logic from Chapter 4.3 made concrete: training compute is a one-time cost, serving compute is a cost multiplied by every request for the model's entire deployed lifetime, and at large enough deployment scale the second term dominates the arithmetic completely.

The base-model decision also has an open-weights-versus-proprietary dimension worth naming explicitly, because it recurs in real interviews. Meta's LLaMA line (LLaMA, LLaMA 2, and the LLaMA 3 herd of models) represents a strategy of releasing strong open-weight foundation models and letting an ecosystem fine-tune and serve them; Google's PaLM and Gemini lines represent the alternative strategy of large, often multimodal, tightly-integrated proprietary models served through a single provider's infrastructure. Neither is "correct" in the abstract β€” the right choice for your product depends on whether you need to control your own serving stack and customize deeply (favoring an open-weights base like LLaMA 3, which was trained with post-training and scale choices explicitly aimed at making a herd of models usable across many downstream deployment sizes), versus whether you need best-in-class multimodal capability today and are comfortable depending on a vendor API (favoring something like Gemini). A senior engineer should be able to articulate this tradeoff crisply rather than treating "which model" as a preference question.

4. Turning a base model into a chat product: the SFT/RLHF layer

A pretrained base model, however well-chosen, is not a chat product. Chapter 4.5 covered why: a base model is trained to continue text plausibly, not to follow instructions, refuse unsafe requests, or produce the specific conversational register users expect. Turning it into a product means passing it through supervised fine-tuning on instruction-response pairs, and then some form of preference-based alignment β€” RLHF in its classical form, or one of the more recent direct-preference approaches β€” to shape it toward the responses human raters actually prefer. LLaMA 2's release paper is a genuinely useful concrete reference here because it documents this pipeline end to end for a widely-deployed model: pretraining, supervised fine-tuning, and iterative RLHF with both a helpfulness and a safety reward model, refined over several rounds rather than as a single pass.

This stage is where a lot of a chat product's actual personality and safety profile get decided, and it's also where the product requirements from section 2 feed back in directly. A customer-support bot needs SFT data and a reward model that reflect the specific domain and tone the business wants; a coding assistant needs SFT data saturated with code and tool-use transcripts rather than general conversation. The mistake to avoid, and one an interviewer is listening for, is treating "we'll fine-tune it" as a single undifferentiated step β€” a strong answer distinguishes what SFT is doing (teaching the format and general behavior) from what preference optimization is doing (shaping which of several plausible responses is preferred), and recognizes that these require different data (labeled demonstrations vs. comparison data) and different failure modes to watch for (SFT can overfit to a narrow style; RLHF can reward-hack a flawed reward model).

RLHF's safety reward model shapes the base model's trained-in behavior, but a production system layers explicit, independent guardrails on top rather than trusting trained behavior alone β€” the same training-time-versus-inference-time split Chapter 5.6 draws for structured output applies just as directly to safety. A guardrails layer typically runs a lightweight moderation classifier β€” Meta's Llama Guard is a widely-used concrete example, a model trained specifically to classify a prompt or response against a fixed safety taxonomy rather than to converse β€” over both the incoming request and the model's own output before it reaches the user, checks output against PII or policy-violation patterns, and can reject or rewrite a response independently of whatever the underlying model was inclined to say. The reason this layer exists at all, rather than relying purely on a well-aligned model, is the same reason section 5's serving non-negotiables exist independently of model quality: a mechanical, auditable check that runs on every single request is a hard guarantee in a way that "the model was trained to behave well" is only ever a strong tendency, and a system design answer should be able to name guardrails as their own architectural layer rather than folding all of safety into the training pipeline.

5. Serving at scale: the non-negotiables

Once you have a model, chapter 3.4's serving-efficiency material stops being optional background and becomes the difference between a product that's economically viable and one that isn't. At any meaningful traffic volume, a small number of serving-layer techniques are effectively mandatory rather than optional optimizations. KV-caching is the baseline requirement for autoregressive generation at all β€” without it, every new token would require recomputing attention over the entire prefix from scratch, which is asymptotically wasteful to the point of being unshippable. Quantization (running weights and activations at reduced precision, typically int8 or 4-bit for weights) is close to mandatory at scale because it directly cuts the memory footprint and often the compute cost per request, usually at a small and carefully-measured quality cost. Continuous batching β€” admitting and evicting individual requests from a batch dynamically rather than waiting for a fixed batch to complete together β€” is what makes serving economically efficient under real, bursty, variable-length traffic, since a small number of long requests would otherwise stall a whole static batch of short ones. Prompt caching is the fourth non-negotiable for any product with a shared, repeated prefix across requests β€” a system prompt, a long set of few-shot examples, a retrieved document included on every turn of a conversation: rather than re-running the compute-bound prefill pass (Chapter 3.4) over that identical prefix on every single request, cache the KV-cache entries the prefix produces once and reuse them across every request that shares it, paying the prefill cost once instead of per request. For a product where every request shares a long, expensive-to-process system prompt, this is often the single highest-leverage cost optimization available, precisely because it eliminates repeated work rather than making the same work cheaper.

The canonical open-source treatment of this serving problem is vLLM, built around PagedAttention (Kwon et al., SOSP 2023), which was covered in depth in Chapter 3.4 and is worth citing here only as a cross-reference rather than re-deriving: it treats the KV-cache as a set of fixed-size pages managed the way an operating system manages virtual memory, which eliminates the fragmentation and over-allocation that naive KV-cache management suffers from, and it's the technique most directly responsible for making high-throughput, high-concurrency LLM serving practical outside of proprietary infrastructure. Depending on the product's scale and cost sensitivity, Chapter 3.3's mixture-of-experts material also becomes relevant here for a different reason: MoE lets you increase a model's total parameter count β€” and with it, its capacity to store knowledge and handle diverse request types β€” without proportionally increasing the compute cost of a forward pass, because only a subset of experts activate per token. For a product serving an extremely broad and heterogeneous request distribution at very large scale, that capacity-per-compute argument can outweigh the added serving complexity of routing and expert placement; for a narrower, smaller-scale product it usually isn't worth the complexity, which is itself the kind of judgment call a system-design answer should make explicit rather than reaching for MoE reflexively.

6. Context length as a product decision, not just a capability

How much context a product needs to support is a requirements question with real architectural consequences, not just a number to maximize. Chapters 3.1 and 3.2 covered why naive attention's quadratic cost in sequence length β€” $O(n^2)$ in context length $n$ β€” makes very long contexts expensive, and the various techniques β€” sparse and linear attention variants, positional-encoding schemes designed to extrapolate beyond training length, and the sliding-window and chunking strategies used in practice β€” that make longer effective context tractable. In a system design, this material shows up as a direct question: does this product actually need 100K+ tokens of context, or does it need to feel like it does?

Those are different engineering problems with different price tags. A legal-document-review product genuinely needs to attend over very long documents and the long-context techniques from Chapters 3.1 and 3.2 are load-bearing. A customer-support chatbot that seems to "remember" a long conversation history usually doesn't need a model with a literally enormous context window β€” it needs a retrieval or summarization layer that keeps the actually-relevant recent turns and compresses or discards the rest before it ever reaches the model's context. Conflating these two is a common design mistake: reaching for the most expensive long-context model available when the actual requirement could be satisfied far more cheaply by good context management is exactly the kind of unforced error a strong system-design answer avoids by asking, again, what the requirement really is before picking the tool.

7. RAG vs. fine-tuning vs. agents: matching the lever to the problem

Chapter 5.2 established retrieval-augmented generation and Chapter 5.4 established agents and tool use; a system design has to decide, for a given product, which of these β€” if any β€” apply, and this is one of the most commonly mis-answered parts of real interviews. The distinguishing question is what kind of gap you're trying to close. If the model's knowledge is stale or missing domain-specific facts, but its general reasoning and instruction-following are already adequate, RAG is almost always the right first lever: it's cheaper to update an index than to retrain a model, it comes with a natural provenance trail for citations, and it degrades more gracefully (a bad retrieval produces a wrong-but-explainable answer, whereas a knowledge gap baked into weights produces a confident hallucination with no visible cause). Fine-tuning is the right lever when the gap is behavioral rather than factual β€” the model knows the relevant facts but produces them in the wrong format, tone, or reasoning style for the product β€” since RAG can't teach a model to reason or write differently, only give it different facts to reason and write about.

Agents and tool use, from Chapter 5.4, are the right lever when the task genuinely requires multi-step interaction with the outside world β€” calling APIs, running code, executing a multi-turn plan where later steps depend on the results of earlier ones β€” rather than a single well-informed response. The mistake to flag explicitly in an interview answer is reaching for an agent architecture by default because it's the most sophisticated-sounding option: a single well-retrieved, well-prompted, single-turn call is cheaper, faster, and more reliable than an agent loop whenever the task doesn't actually require iterative interaction with external state, and Chapter 5.4's material on the reliability costs of long-horizon agent execution is precisely the reason not to over-reach for agentic complexity when a simpler pattern would do the job.

8. Evaluation as a continuous process, not a launch gate

Chapter 5.5 made the case that evaluation is not a thing you do once before shipping and then forget; in a real product it has to run continuously, before and after every change, using both offline and online signals. Offline evaluation β€” benchmark suites, held-out test sets, LLM-as-judge scoring β€” is what you use before a change ships, to catch regressions cheaply and quickly, but Chapter 5.5 was explicit that benchmark performance and real-world reliability are not the same thing, and a model that improves on a benchmark can still regress on the actual traffic distribution your product serves. Online evaluation is what catches that gap: A/B testing changes against real user traffic, tracking product-level metrics (task completion, user-reported satisfaction, escalation or complaint rates, retention) rather than only model-level metrics (perplexity, benchmark accuracy), and instrumenting for the specific failure modes that matter for this product β€” hallucination rate on domain-specific facts, refusal rate on benign requests, latency percentiles under real load rather than synthetic load.

The design implication is that a shipped LLM product needs its evaluation infrastructure built at the same time as the product itself, not bolted on afterward. That means logging enough of each interaction to reconstruct what happened, a labeled or judge-scored sample of production traffic reviewed on a cadence, and a rollback or gradual-rollout mechanism so that a regression caught by online metrics doesn't have to wait for the next full release cycle to be fixed. This is also a place where a strong interview answer distinguishes itself: a weaker answer says "we'll evaluate it on standard benchmarks before launch," a stronger one describes a concrete online-and-offline evaluation loop that keeps running after launch, because that's what actually catches the regressions that matter in production.

9. Walking the whole loop: a worked design narrative

Put concretely: suppose the brief is "design a general-purpose chat assistant expected to serve on the order of ten million daily active users, with a latency budget of under a second to first token, a requirement to answer questions about events after any static training cutoff, and a hard requirement to avoid confidently fabricated answers on factual queries." Working through requirements first: this is high enough traffic that serving cost per token dominates the design, latency is tight enough that time-to-first-token and streaming matter, and the freshness and hallucination requirements point directly at retrieval rather than pure parametric knowledge.

From there, the base-model choice follows the inference-optimal logic from section 3: given the enormous request volume, a smaller model trained on more tokens than the raw compute-optimal frontier suggests is very likely the right call, in the spirit of the LLaMA line's design choices, because serving cost dominates lifecycle cost at this scale β€” and if deep customization of the serving stack and fine-tuning pipeline matters, an open-weights base like LLaMA 3 is a defensible starting point over a closed API-only alternative. That base model then goes through SFT and preference optimization (section 4) to become a usable chat model with an appropriate tone and safety profile for a consumer product. Given the freshness and hallucination requirements, RAG (section 7) becomes close to mandatory rather than optional: a retrieval layer over a continuously-updated index handles both the staleness problem and gives the hallucination-avoidance requirement a concrete mechanism β€” answers grounded in retrieved passages with citations, rather than relying on parametric memory alone β€” while fine-tuning alone would not touch the freshness requirement at all. Given that this is a general assistant rather than a narrow API-calling product, a full agent loop is probably not the default path for every request β€” it would add latency and failure surface the ten-million-user latency budget can't easily absorb β€” but a lightweight tool-use capability (for things like calculator-style computation or an explicit search-and-cite tool) is a reasonable middle ground consistent with Chapter 5.4's material on matching agentic complexity to actual task requirements.

On the serving side, at this traffic volume, KV-caching, quantization, and continuous batching (section 5) are simply required, and something in the PagedAttention/vLLM family is a reasonable default rather than a custom-built alternative, precisely because it's the well-tested open-source answer to exactly this problem. It's worth grounding this in rough numbers rather than leaving "serving cost dominates" as an abstract claim: efficiently served open-weight models in this size class run on the order of a fraction of a cent to a few cents per thousand output tokens in raw compute cost, which sounds trivial per request but, multiplied across ten million daily users each exchanging even a modest handful of messages a day, adds up to a serving bill easily in the tens of thousands of dollars a day: spelled out, 10,000,000 users Γ— ~10 messages/day Γ— ~300 output tokens/message Γ— ~$0.002 per thousand tokens works out to roughly $60,000 per day β€” the exact scale at which section 5's techniques stop being nice-to-haves and start being the difference between a viable unit economics story and an unshippable one. The sub-second time-to-first-token requirement is a similarly concrete constraint: at this traffic volume it's really a prefill-latency budget (Chapter 3.4) for however long the retrieved-and-cached context runs, which is exactly why prompt caching and an inference-optimal (rather than maximally compute-optimal) base model both matter here specifically, not just KV-caching in the abstract. Whether MoE is worth the added complexity depends on the breadth of the request distribution: a genuinely general-purpose assistant fielding an extremely heterogeneous mix of request types is a more plausible candidate for MoE's capacity-per-compute benefit than a narrow-domain product would be. Context length should be sized to what the product actually needs β€” a chat assistant's working context is dominated by recent conversation turns and retrieved passages, not an enormous static window, so a moderate context length with good retrieval and summarization is probably more cost-effective than maximizing raw window size. And finally, none of this ships without the evaluation loop from section 8 wired in from day one: offline regression suites gating any model or prompt change, and online metrics β€” hallucination rate on a sampled and judged slice of real traffic, latency percentiles under production load, user-facing satisfaction and escalation signals β€” running continuously after launch, because that is the only way to know whether a change that looked good offline is actually good for the people using the product.

Notice what this walk-through actually did: it didn't introduce a single new technique. It took the compute-vs-inference-optimal tradeoff from Chapter 4.3, the SFT/RLHF pipeline from Chapter 4.5, the serving techniques from Chapter 3.4, the MoE capacity argument from Chapter 3.3, the long-context tradeoffs from Chapters 3.1 and 3.2, the RAG-versus-fine-tuning-versus-agents distinctions from Chapters 5.2 and 5.4, and the continuous-evaluation discipline from Chapter 5.5, and forced them to interact under one concrete set of constraints. That is the actual skill a senior system-design interview is testing, and it's also, not coincidentally, the actual skill the job requires: not knowing that these techniques exist, but knowing which ones a specific set of requirements actually calls for, and being able to say clearly why the others don't apply here. But even the most carefully assembled design of this kind rests on an assumption worth stating plainly: that getting every component right adds up to a system that behaves reliably every time it's called. That assumption is exactly what the rest of the field is still testing.

10. Interview angle

"Design a coding assistant that needs to work over an entire large codebase, not just the current file. Walk me through your approach." A strong answer starts with requirements: what does "work over an entire codebase" require functionally β€” cross-file reference resolution, awareness of project conventions, multi-file edits? It then reasons explicitly about context: a codebase is typically far larger than any practical context window, so this is a retrieval problem (index the codebase, retrieve relevant files/functions/symbols per query) rather than a raw context-length problem, drawing the same RAG-vs-long-context distinction as section 6. It should flag that a coding assistant plausibly needs agentic tool use (running tests, applying multi-file diffs, calling a linter) per Chapter 5.4, and that evaluation here can't be pure offline benchmarks β€” it needs online signals like whether generated diffs actually compile and pass tests, not just whether they look plausible.

"Your product's offline benchmark scores improved after a model update, but users are complaining more. What do you do?" A strong answer immediately identifies this as the Chapter 5.5 gap between benchmark performance and real-world reliability, and proposes a concrete diagnostic path: check whether the benchmark distribution matches the actual production traffic distribution, look at the specific complaint categories against instrumented failure modes (hallucination, tone, refusal rate, latency), and treat the online metrics as ground truth over the offline ones until proven otherwise β€” because the offline suite, by construction, can only test what it was built to test.

"When would you reach for MoE versus just training a smaller dense model?" A strong answer states the actual tradeoff from Chapter 3.3 in product terms: MoE buys parameter capacity β€” useful for a broad, heterogeneous request distribution where different "kinds" of knowledge benefit from different experts β€” without proportionally increasing per-token compute, but it costs meaningfully more serving-infrastructure complexity (expert routing, load balancing across experts, more complex batching). For a narrow-domain product with a homogeneous request distribution, that complexity usually isn't repaid, and a smaller dense model is the more defensible choice.

"Walk me through how you'd decide between fine-tuning and RAG for a customer-support bot that keeps giving outdated policy answers." A strong answer diagnoses the failure mode first: outdated answers are a freshness/factual problem, not a behavioral one, which per section 7 points squarely at RAG over an up-to-date policy index rather than fine-tuning, since fine-tuning bakes facts into weights that will go stale again at the next policy change, while a retrieval index can be updated the moment policy changes without retraining anything.

"At what point does KV-cache memory become the bottleneck, and what do you do about it?" A strong answer connects this to Chapter 3.4 material: KV-cache memory grows with sequence length, batch size, and number of layers/heads β€” concretely, the cache size in bytes is roughly $2 \times L \times H \times d_{head} \times S \times B \times \text{bytes\_per\_element}$, where $L$ is the number of layers, $H$ the number of heads, $d_{head}$ the per-head dimension, $S$ the sequence length, $B$ the batch size, and the leading 2 accounts for storing both keys and values β€” and at high concurrency with long contexts it can dominate GPU memory over the model weights themselves; the practical responses are quantizing the cache, using techniques like multi-query or grouped-query attention to shrink cache size per token, and using paged, non-contiguous cache allocation (PagedAttention) to avoid fragmentation-driven waste, rather than simply provisioning more memory as the default fix.

11. Self-check questions

  1. Why does an inference-optimal model size for a widely-deployed product typically differ from a Chinchilla-style compute-optimal size, and which cost is each one minimizing?
  2. What specific product requirements would push you toward an open-weights base model like LLaMA 3 versus a closed model like Gemini?
  3. Name three serving-layer techniques that go from "nice to have" to "effectively mandatory" as request volume grows, and explain what breaks without each one.
  4. Given a product complaint of "the model gives outdated facts," walk through why RAG is usually the right first lever rather than fine-tuning, and describe a scenario where fine-tuning would be the better answer instead.
  5. Why might a product that seems to require a very long context window actually be better served by good retrieval or summarization than by a genuinely large context length?
  6. What's the difference between what SFT and preference optimization (RLHF or similar) each contribute when turning a base model into a chat product?
  7. Why is offline benchmark improvement insufficient evidence that a model update is safe to fully roll out, and what would you check before trusting it?

12. Sources

  • Touvron, H., Lavril, T., Izacard, G., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971. https://arxiv.org/abs/2302.13971
  • Touvron, H., Martin, L., Stone, K., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. https://arxiv.org/abs/2307.09288
  • Grattafiori, A., Dubey, A., Jauhri, A., et al. (2024, Meta). The Llama 3 Herd of Models. arXiv:2407.21783. https://arxiv.org/abs/2407.21783
  • Chowdhery, A., Narang, S., Devlin, J., et al. (2022, Google). PaLM: Scaling Language Modeling with Pathways. JMLR 2023. arXiv:2204.02311. https://arxiv.org/abs/2204.02311
  • Gemini Team, Google DeepMind (2023). Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805. https://arxiv.org/abs/2312.11805
  • Kwon, W., Li, Z., Zhuang, S., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). SOSP 2023. arXiv:2309.06180. https://arxiv.org/abs/2309.06180
  • Inan, H., Upasani, K., Chi, J., et al. (2023, Meta). Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674. https://arxiv.org/abs/2312.06674

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

Why did Meta train LLaMA models smaller than the Chinchilla compute-optimal size would suggest, on more tokens instead?

Explanation: Training compute is a one-time cost; serving compute is multiplied by every request over the model's entire deployed lifetime β€” at scale, the second term dominates.

According to this chapter, what should be the first step in any real LLM system design, before choosing a model or architecture?

Explanation: These constraints determine which parts of the toolkit are load-bearing for this particular design β€” an interviewer watching you skip straight to architecture is watching you miss the point.

Per the chapter's guidance on RAG vs. fine-tuning, which lever is the right first choice when a model's knowledge is stale or missing domain facts, but its reasoning and instruction-following are already adequate?

Explanation: Fine-tuning is the right lever when the gap is behavioral (wrong format/tone/style), not factual β€” RAG can't teach a model to reason or write differently.

Why is offline benchmark improvement alone insufficient evidence that a model update is safe to fully roll out?

Explanation: This is why online A/B testing against real traffic, tracking product-level metrics, is needed to catch what offline suites can't.

Serving costs 2 USD per million output tokens. An average response is 500 tokens. What is the cost per response, in dollars? (Round to 4 decimal places.)

Explanation: 500 tokens Γ— (2 USD / 1,000,000 tokens) = 0.001 USD.

A product serves 10 million daily active users, each issuing an average of 3 requests per day, each costing 0.001 USD to serve. What is the total daily serving cost, in dollars?

Explanation: 10,000,000 Γ— 3 Γ— 0.001 USD = 30,000 USD per day.

A single GPU can serve 50 requests per second at the target latency budget. How many GPUs are needed to handle a peak load of 2,000 requests per second?

Explanation: 2000 / 50 = 40 GPUs.

A single request's KV-cache uses 200 MB. How much total KV-cache memory is needed to serve 128 concurrent requests simultaneously, in GB (1 GB = 1024 MB)?

Explanation: 200 MB Γ— 128 = 25,600 MB. Since 1 GB = 1024 MB, that's 25,600 / 1024 = 25.0 GB.