IV.11 / Scaling laws, Kaplan, and Chinchilla
Compare two uses of a fixed budget.
Imagine a fixed training budget divided between a larger model with fewer token exposures and a smaller model trained on more data. Plot a hypothetical loss for each allocation and label the measured region separately from any extrapolation. The decision depends on the fitted relationship and the intended evaluation, not parameter count alone.
