Chapter 4.3 β€” Scaling Laws

Contents

  1. From "what data" to "how much of everything"
  2. The empirical discovery: smooth power laws across orders of magnitude
  3. Why this finding changed how the field made decisions
  4. The compute-optimal question: parameters versus tokens
  5. Chinchilla's revision of the recipe
  6. The nuance interviewers probe: compute-optimal is not deployment-optimal
  7. Emergent abilities and running out of data: two cracks in the smooth curve
  8. Interview angle
  9. Self-check questions
  10. Sources

1. From "what data" to "how much of everything"

Chapter 4.2 closed by pointing at a gap: every claim made about data quality and curation was qualitative, a matter of "this helps," with no principled way to predict in advance how much a given change in model size, dataset size, or training compute would actually improve a model's performance. That predictive gap is exactly what scaling laws fill. Rather than treating "should we train a bigger model" or "should we collect more data" as questions answered by intuition or by expensive trial and error, scaling laws turn them into an extrapolation problem: train a family of smaller models cheaply, measure how their loss behaves as you vary size, data, and compute, and fit a functional form that lets you predict β€” with real quantitative confidence β€” what a much larger, much more expensive model will achieve before you spend the money to train it.

2. The empirical discovery: smooth power laws across orders of magnitude

Kaplan et al., in 2020, ran a systematic study of exactly this kind across a striking range of scales, training transformer language models varying in size by several orders of magnitude and measuring test loss as a function of three quantities: the number of model parameters, the size of the training dataset, and the amount of compute used to train the model. The headline empirical result is that, holding the other two roughly fixed or non-bottlenecking, test loss decreases as a smooth power law in each of these quantities β€” that is, loss falls off as roughly $L(N) \approx (N_c / N)^{\alpha_N}$ for parameter count $N$, and analogously for dataset size and compute, each with its own fitted exponent, over several orders of magnitude of scale with no sign of the relationship breaking down or hitting a wall within the range studied. This is a remarkable thing to find empirically: neural network training is generally noisy, full of discrete architectural choices, learning rate schedules, and idiosyncratic hyperparameters, and yet the aggregate loss curve, viewed on a log-log plot against scale, is close to a straight line across a huge span of model and dataset sizes.

The power-law form matters because of what it implies about extrapolation. A relationship that is a smooth power law over the range you've measured gives you real license to extrapolate it somewhat beyond that range, because power laws don't have hidden regime changes lurking arbitrarily β€” you can fit the exponent on cheaper, smaller-scale experiments and use the fitted curve to forecast loss at a scale you haven't yet built. Kaplan et al. also found that these three factors trade off against each other in a describable way, and that larger models are more sample-efficient β€” they need fewer tokens to reach a given loss than smaller models do β€” which fed directly into a prescription about how to allocate a training compute budget between model size and dataset size, a prescription that turned out to need substantial revision two years later.

3. Why this finding changed how the field made decisions

Before this line of work, the decision to train a substantially larger model was, in an important sense, a bet made on intuition and prior anecdote: bigger models had tended to perform better, so making one bigger still seemed promising, but nobody could say with any quantitative confidence how much better, or whether a given increase in scale was worth its dramatically increased cost, without actually training the model and finding out. Scaling laws converted "should we build a bigger model" from a qualitative guess into a budgeting exercise: given the smooth, fitted relationship between loss and scale, you can compute the expected loss of a model at a candidate size and compute budget before committing the resources to train it, and you can ask the inverse question too β€” given a target loss, what is the minimum compute required to reach it, and what parameter/token split achieves that minimum. This is precisely the kind of forecasting tool that lets an organization justify a nine- or ten-figure training run to itself and to funders with something more rigorous than "bigger has worked before." It is hard to overstate how much this reframing shaped the last several years of the field's resource allocation: once loss versus compute became a fittable, extrapolatable curve, the decision to scale stopped being a bet and became closer to an engineering calculation.

4. The compute-optimal question: parameters versus tokens

Granting that scaling laws let you predict loss as a function of compute, there remains a genuinely nontrivial allocation question buried inside "compute": for a fixed training compute budget, measured in floating point operations, you can spend it on a larger model trained on fewer tokens, or a smaller model trained on more tokens, and the two choices are not equivalent β€” total compute for a dense transformer trained for one epoch is, to good approximation, proportional to the product of parameter count and token count β€” the widely used approximation is $C \approx 6ND$, where $C$ is training compute in FLOPs, $N$ is parameter count, and $D$ is the number of training tokens β€” so a fixed compute budget really does trace out a tradeoff curve between the two: $N \cdot D \approx \text{const}$. Kaplan et al.'s original analysis suggested that, given a fixed compute budget, the compute-optimal allocation favored scaling model size considerably faster than dataset size β€” in effect, prioritizing bigger models trained on comparatively modest amounts of data relative to their size. This prescription was influential and was reflected in a number of large models trained in the period following its publication, several of which turned out, in retrospect, to be substantially larger than the amount of training data they were fed would justify.

5. Chinchilla's revision of the recipe

Hoffmann et al.'s 2022 Chinchilla paper revisited this exact question with a more careful experimental design β€” training a large number of models across a range of sizes and token counts and fitting the loss surface directly as a joint function of both, rather than inferring the optimal split indirectly β€” and arrived at a materially different prescription. Their central finding was that many of the large language models trained under the original Kaplan et al. guidance were significantly undertrained relative to their parameter count: for the compute those models consumed, a substantially smaller model trained on proportionally more tokens would have achieved lower loss for the identical compute budget. Concretely, Chinchilla's fitted compute-optimal relationship implies that model size and training tokens should scale roughly in proportion to one another as compute grows, translating in practice to a rule of thumb of roughly twenty training tokens per model parameter β€” $D_{\text{opt}} \approx 20\,N$ β€” a ratio now commonly referred to as "compute-optimal" or "Chinchilla-optimal." Combined with $C \approx 6ND$, this rule of thumb is what underlies the finding that both the optimal parameter count and the optimal token count grow as roughly the square root of compute, $N_{\text{opt}}(C) \propto C^{0.5}$ and $D_{\text{opt}}(C) \propto C^{0.5}$, rather than one dominating the other as compute scales up. Hoffmann et al. demonstrated this concretely by training a model, Chinchilla, with roughly a quarter the parameters of a contemporary large model but on proportionally more data for the same compute budget, and showing it outperformed the larger, more lightly-trained model across a broad suite of downstream evaluations. This result reshaped the field's default recipe almost immediately: the interesting lever for improving a model at fixed compute was not simply "make it bigger" but "get the size-to-data ratio right," and a wave of subsequent models were trained closer to, or explicitly informed by, the Chinchilla-optimal ratio.

6. The nuance interviewers probe: compute-optimal is not deployment-optimal

There is a subtlety in the Chinchilla result that is worth a careful paragraph, because it is precisely the distinction senior interviewers like to probe to separate candidates who've memorized "20 tokens per parameter" from candidates who understand what the number actually optimizes. Chinchilla-optimal is a statement about minimizing training loss for a fixed training compute budget β€” full stop. It says nothing about the cost of running the resulting model at inference time, repeatedly, for months or years, across potentially enormous numbers of user requests, which is precisely the cost structure a deployed production model actually faces. A smaller model is cheaper to run at inference for a fixed level of quality, and inference cost, aggregated over the model's deployed lifetime, can dwarf the one-time training cost many times over for a model serving heavy traffic. This means it is frequently worth deliberately training a smaller-than-Chinchilla-optimal model on far more tokens than the "optimal" ratio would suggest β€” colloquially, "overtraining" it relative to the training-compute-optimal point β€” accepting a training run that is less efficient per training-FLOP in exchange for a smaller, cheaper-to-serve model that reaches comparable quality. Several widely deployed models made exactly this choice explicitly, training deliberately past the Chinchilla-optimal token count for their size because the total cost of ownership calculation, inclusive of inference, favored a smaller model over a larger one even at the price of a less compute-efficient training run. The general lesson, and the one worth stating cleanly in an interview, is that "optimal" in "compute-optimal" always has an implicit objective attached to it, and the objective that minimizes training loss for a given training budget is a different objective from the one that minimizes total cost of ownership for a model that will be served at scale β€” conflating the two is the single most common mistake in loosely reasoning about scaling laws.

7. Emergent abilities and running out of data: two cracks in the smooth curve

Everything so far has treated the power-law relationship as clean and monotonic, but two documented complications are worth knowing before treating "the loss curve is smooth" as the whole story.

The first is the debate over emergent abilities. Wei et al. catalogued a striking pattern across many benchmarks: performance on some tasks doesn't rise gradually with scale the way the aggregate loss curve does β€” it stays near-random until a certain scale threshold, then rises sharply. Arithmetic, certain kinds of multi-step reasoning, and a handful of other capabilities appeared to switch on abruptly rather than improve continuously. Read at face value, this looked like a genuine departure from the smooth power-law story: capabilities simply absent below a threshold, appearing discontinuously above it. Schaeffer et al.'s response, memorably titled "Are Emergent Abilities of Large Language Models a Mirage?", argued that at least some of this apparent discontinuity is a metric artifact rather than a property of the model. Many "emergent" benchmarks are scored with a nonlinear or discontinuous metric β€” exact-match accuracy on a multi-step problem, for instance, where getting any single step wrong scores identically to getting every step wrong β€” so an underlying capability that is genuinely improving smoothly and continuously with scale, measured through a metric with a hard pass/fail threshold, produces a sharp-looking jump in the plotted score even though nothing discontinuous happened in the model's actual output quality. Re-scoring the same checkpoints with a smoother, partial-credit metric erased most of the sharp "emergence" in several of the benchmarks they re-examined. The honest state of the field is that some of both is probably true β€” some benchmark "emergence" is a measurement artifact, and it remains an open, actively debated question whether any genuinely discontinuous capability jump survives once you correct for the metric β€” and the discipline worth carrying into an interview is treating "this capability emerged suddenly" as a claim to interrogate rather than a conclusion to accept at face value, which is exactly the same evaluation-validity skepticism Chapter 5.5 develops in a different context.

The second complication is more mundane and, in some ways, more consequential: the original scaling laws were derived assuming compute is the binding constraint and data is effectively unlimited, an assumption that gets less comfortable every year as the highest-quality text data on the public internet is finite and increasingly fully exploited. Muennighoff et al.'s data-constrained scaling laws extend the Kaplan/Chinchilla framework to the regime where you run out of fresh data before you run out of compute budget, and the central practical finding is that repeating data for multiple epochs is not nearly as costly as the original single-epoch-assumption laws would suggest: training on a fixed, smaller dataset for up to roughly four epochs delivers returns close to what an equivalently-sized fresh dataset would have, before returns to further repetition fall off sharply β€” Muennighoff et al. formalize this by discounting repeated tokens into an "effective dataset size," $D_{\text{eff}}$, a function of the number of epochs that grows more slowly than linearly and captures exactly this diminishing return to repetition. This matters for exactly the reason it sounds like it should: as frontier labs push training-token counts into the tens of trillions, "we've already used most of the good text on the internet" stops being a hypothetical and becomes an actual constraint on the compute-optimal recipe, and data-constrained scaling laws are the tool for reasoning about what to do once fresh tokens, rather than compute, become the scarce resource.

Scaling laws, taken together with the Chinchilla revision and these two caveats, give you a quantitative way to decide how big a model to build and how much data to feed it β€” but they say nothing about how you would actually get such a large model onto hardware and trained in finite wall-clock time, given that neither the model's parameters nor its optimizer state generally fit on a single accelerator at frontier scale. That purely systems-engineering problem, distributing a training run correctly across many devices, is where the next chapter turns.

8. Interview angle

Q: What did Kaplan et al.'s scaling laws actually establish, and why was it a significant finding beyond "bigger models are better"? A strong answer states that loss follows a smooth power law in model size, dataset size, and compute across several orders of magnitude, which is significant not because it shows scale helps (already believed) but because it makes the relationship quantitatively predictable and extrapolatable β€” letting you forecast a large model's loss from experiments at far smaller, cheaper scale, converting scaling decisions from intuition-driven bets into calculations.

Q: What specifically did Chinchilla get wrong about the original Kaplan et al. prescription, and what did it get right instead? A strong answer states that Kaplan et al.'s original compute-optimal allocation favored growing model size faster than dataset size, leading many contemporary models to be undertrained relative to their parameter count; Chinchilla's more careful joint fit showed model size and token count should scale together roughly in proportion, at roughly 20 tokens per parameter, and demonstrated this concretely by showing a smaller, more-trained model can beat a larger, less-trained one at equal compute.

Q: Is training a model well past the Chinchilla-optimal token count ever a good idea? Why would anyone do that? A strong answer says yes, explicitly, and explains the distinction: Chinchilla-optimal minimizes loss for a fixed training compute budget, but says nothing about inference cost; if a model will be served heavily over a long deployment lifetime, total cost of ownership (training plus inference) can be minimized by deliberately choosing a smaller model trained on proportionally more tokens than "optimal," accepting training inefficiency to gain inference cheapness.

Q: How would you use scaling laws in practice to decide on a training run's configuration before spending the compute? A strong answer describes training a suite of much smaller, cheaper models across varied sizes and token counts, fitting the loss as a function of these variables (and compute), and using the fitted curve to choose a parameter count and token count that either hits a target loss at minimum compute or maximizes expected quality at a fixed compute budget β€” explicitly treating this as an extrapolation exercise validated by held-out scale points where feasible.

Q: What's a limitation of relying purely on the scaling-law power-law fit for extrapolation? A strong answer notes the fit is empirical and only validated over the range of scales actually studied; extrapolating far beyond that range assumes the power law continues to hold and that no new regime (e.g. data exhaustion, emergent capabilities, changes in optimal hyperparameters at larger scale) intervenes, which is a real risk the original papers themselves caution about.

Q: A colleague says "reasoning is an emergent ability that only shows up past a certain scale." How would you respond? A strong answer doesn't simply agree or disagree β€” it names the actual debate: Wei et al. documented real, sharp jumps on some benchmarks as scale increases, but Schaeffer et al. showed that at least part of this is a metric artifact, since discontinuous scoring (exact-match on multi-step answers) can turn a smoothly-improving underlying capability into an apparently sudden jump. A strong answer asks what metric the claim is based on before accepting that the underlying capability itself is discontinuous, and notes that resolving this cleanly is still an open research question.

9. Self-check questions

  1. What three quantities does the Kaplan et al. scaling law relate to test loss, and what functional form describes that relationship?
  2. Why does a smooth power-law fit over a wide range of scales give you more confidence extrapolating beyond that range than a fit with visible regime changes would?
  3. In what concrete sense did scaling laws change the nature of the decision to train a larger model, relative to before they existed?
  4. Why is training compute for a dense transformer roughly proportional to the product of parameter count and token count, and what does that imply about the tradeoff between model size and dataset size at fixed compute?
  5. What was Chinchilla's central empirical claim about previously-trained large models, and what evidence supported it?
  6. State the Chinchilla-optimal rule of thumb in terms of tokens per parameter.
  7. Explain precisely why "compute-optimal" and "optimal for a model you intend to deploy at scale" are different objectives, and give a concrete reason an engineering team might deliberately choose to violate the Chinchilla-optimal ratio.
  8. Explain how a discontinuous evaluation metric can make a smoothly-improving capability look like a sudden "emergent" jump, and what evidence would distinguish a real discontinuity from a measurement artifact.
  9. Why do the original Kaplan and Chinchilla scaling laws implicitly assume data is not the binding constraint, and what does data-constrained scaling change about the recipe once that assumption fails?

10. Sources

  • Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
  • Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla). NeurIPS 2022. arXiv:2203.15556. https://arxiv.org/abs/2203.15556
  • Wei, J., Tay, Y., Bommasani, R., et al. (2022). Emergent Abilities of Large Language Models. TMLR 2022. arXiv:2206.07682. https://arxiv.org/abs/2206.07682
  • Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023. arXiv:2304.15004. https://arxiv.org/abs/2304.15004
  • Muennighoff, N., Rush, A., Barak, B., et al. (2023). Scaling Data-Constrained Language Models. NeurIPS 2023. arXiv:2305.16264. https://arxiv.org/abs/2305.16264

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

What did Kaplan et al.'s scaling laws establish about test loss as model size, dataset size, and compute increase?

Explanation: This smooth power-law behavior, visible as a near-straight line on a log-log plot, is what makes loss quantitatively extrapolatable to scales not yet built.

What was Chinchilla's central finding about many large language models trained under the original Kaplan et al. guidance?

Explanation: Hoffmann et al. showed that jointly fitting the loss surface over both size and tokens implied a materially different, more token-heavy allocation than Kaplan et al.'s original prescription.

What is the Chinchilla-optimal rule of thumb for the ratio of training tokens to model parameters?

Explanation: Chinchilla's fitted compute-optimal relationship implies model size and token count should scale together at roughly this ratio as compute grows.

Why might an engineering team deliberately train a model well past the Chinchilla-optimal token count for its size?

Explanation: Total cost of ownership (training + inference over the deployment lifetime) can favor a smaller, more heavily-trained model even at the cost of a less compute-efficient training run.

Using the Chinchilla-optimal rule of thumb of roughly 20 training tokens per parameter, how many billion tokens would be compute-optimal for a 10-billion-parameter model?

Explanation: 10 billion parameters Γ— 20 tokens/parameter = 200 billion tokens.

Using the same 20-tokens-per-parameter rule, how many billion tokens would be compute-optimal for a 70-billion-parameter model?

Explanation: 70 billion parameters Γ— 20 tokens/parameter = 1,400 billion tokens.

Training compute for a dense transformer is roughly proportional to parameters Γ— tokens ($N \times D$). A training run uses $N = 50$ billion parameters and $D = 1000$ billion tokens. If you double the parameter count to 100 billion while holding total compute fixed, what is the new token count $D$ (in billions)?

Explanation: Fixed compute means N Γ— D stays constant: 50 Γ— 1000 = 100 Γ— D, so D = 50,000/100 = 500 billion tokens.

Suppose loss follows the power law $L(N) = (N_c/N)^{\alpha}$ with $N_c = 1$ and $\alpha = 0.5$. What is $L(N)$ when $N = 4$ (in the same units as $N_c$)?

Explanation: L(4) = (1/4)^0.5 = (0.25)^0.5 = 0.5.