Robots Atlas>ROBOTS ATLAS
Training

Compute-Optimal Training

2022ActivePublished: 7 May 2026Updated: 7 May 2026Published
Key innovation
Shows that for a fixed compute budget the number of model parameters and the number of training tokens should be scaled roughly in equal proportion, rather than scaling the model far faster than the data.
Category
Training
Abstraction level
Pattern
Operation level
Training
Use cases
Foundation-model pretraining planningSizing model and dataset against a fixed GPU budgetComparing training efficiency across model familiesDecisions on growing the corpus vs growing the modelEstimating marginal returns from scaling

How it works

The starting point is the standard training-cost approximation C ≈ 6·N·D, where N is the parameter count and D is the number of training tokens. Hoffmann et al. (2022) applied three complementary approaches: (1) training fixed-size models for varying numbers of tokens, (2) IsoFLOP curves — for each budget C the values of N and D are varied while keeping C constant, and the (N*, D*) minimizing validation loss is identified, and (3) a parametric loss model L(N, D) fitted to all 400+ runs. All three methods agreed that the optimal N and D should grow in roughly equal proportion with C, corresponding to scaling exponents a ≈ 0.5 and b ≈ 0.5 in the relations N* ∝ C^a and D* ∝ C^b. In practice this yields the rule of thumb of approximately 20 tokens per parameter to keep training compute-optimal. The rule assumes a sufficiently large, deduplicated corpus, a decoder-only transformer architecture, and a standard learning-rate schedule tuned to the chosen number of steps.

Problem solved

Earlier scaling laws (Kaplan et al., 2020) suggested that under a constrained compute budget one should mainly increase the number of model parameters, which led to the training of very large but significantly undertrained models. Compute-Optimal Training addresses the problem of non-optimal allocation of the FLOPs budget between model size and training data volume.

Implementation

Implementation pitfalls
Confusing compute-optimal with inference-optimalHigh

The 20:1 rule minimizes pretraining loss for a training budget but does not account for inference cost. Models heavily deployed in production are often worth training for longer (more tokens) to reduce inference cost.

Fix:Optimize the combined training + inference cost over the full model lifecycle rather than the pretraining cost alone.
Insufficient corpusMedium

The rule assumes a sufficiently large, deduplicated corpus. Repeating the same data over many epochs breaks the assumption and distorts the relation between D and the effective count of "fresh" tokens.

Fix:Check the size of the unique corpus before planning pretraining and treat D as the number of unique tokens, not tokens with repetition.
Naive extrapolation beyond the experimental rangeMedium

The scaling exponents were fit on models up to around 16B parameters and budgets up to roughly 5e23 FLOPs; extrapolating to budgets that are orders of magnitude larger is not always reliable.

Fix:For very large budgets run your own IsoFLOP curves instead of blindly relying on the Chinchilla exponents.

Evolution

Original paper · 2022 · arXiv 2022; NeurIPS 2022 · Jordan Hoffmann
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre
2020
Kaplan scaling laws

Kaplan et al. publish "Scaling Laws for Neural Language Models", suggesting that under a constrained budget one should mainly grow the model.

Scaling Laws for Neural Language Models (paper)
2022
Chinchilla and compute-optimal scaling
Inflection point

Hoffmann et al. introduce the compute-optimal rule and empirically confirm it with the Chinchilla 70B / 1.4T-token model, outperforming Gopher 280B at the same FLOPs budget.

Training Compute-Optimal Large Language Models (paper)
2023
LLaMA — breaking the 20:1 rule

Touvron et al. train LLaMA on more than one trillion tokens for 7B–13B models, intentionally going beyond the compute-optimal point to achieve cheaper inference at comparable quality.

LLaMA: Open and Efficient Foundation Language Models (paper)

Hyperparameters (configurable axes)

Compute budget (FLOPs)Critical

Total compute budget C in FLOPs allocated for pretraining, jointly determining N and D.

Tokens per parameterCritical

The D/N ratio. According to the Chinchilla results the optimum is around 20 tokens per parameter.

20Value recommended by Hoffmann et al. (2022).
Parameter count (N)High

Number of model parameters, chosen jointly with D for budget C.

Training tokens (D)High

Number of unique tokens processed during pretraining.

Hardware requirements

Primary

Compute-Optimal Training is a rule for allocating a FLOPs budget and does not depend on any specific hardware; it applies equally to GPU and TPU clusters.