Compute-Optimal Training

Scaling Laws are empirical regularities discovered by Kaplan et al. (2020) at OpenAI, describing how the performance of language models changes predictably with model size (parameter count N), dataset size (D), and compute budget (C). Cross-entropy loss scales as power laws with each of these three variables across many orders of magnitude. The study showed that architectural configuration (depth, width) has minimal impact at fixed N and C, that larger models are significantly more sample-efficient, and that optimally efficient training requires very large models on a relatively modest amount of data with early stopping. Hoffmann et al. (Chinchilla, 2022) refined these laws, showing that earlier models (including GPT-3) were massively undertrained and that optimal N and D should scale equally.

Pretraining (self-supervised pretraining) is the first and most expensive stage in building modern foundation models. The model learns to predict missing or next portions of data — next tokens in text, masked words, future video frames, future robot states — without human labels. This unlocks virtually unlimited raw data (web crawls, code, books, YouTube video, robot telemetry). The result is a set of weights encoding "world knowledge" — dense statistical representations that can later be fine-tuned, instruction-tuned, or RLHF-aligned for any downstream task. Pretraining underpins GPT, BERT, CLIP, Llama, Gemini, and robotics foundation models (Pi-Zero, Gemini Robotics, Ti0).

Chinchilla is the result of "Training Compute-Optimal Large Language Models" by DeepMind (Hoffmann et al., 2022). The paper empirically showed that models like GPT-3 (175B parameters, 300B tokens) were **massively undertrained** — for the same compute budget, a smaller model trained on far more data would yield better results. Chinchilla (70B parameters trained on 1.4T tokens) outperformed Gopher (280B parameters) and GPT-3 on most benchmarks at a fraction of the inference cost. The key takeaway: for compute-optimal training, N and D should scale roughly equally (~20 tokens per parameter), not disproportionately faster N (as Kaplan et al. suggested). Later work like Llama, Mistral, and Gemma went even further, training smaller models on 100+ tokens per parameter — because for production-served models, inference cost matters more than pure compute-optimal training.

A Large Language Model (LLM) is a class of machine learning models based on the Transformer architecture, trained on large text datasets via autoregressive language modeling (next-token prediction). These models have billions of parameters and can generate coherent text, answer questions, write code, translate languages, and perform many other language-cognitive tasks without task-specific fine-tuning. The term covers models such as GPT, LLaMA, Gemini, Claude, and Mistral. Most modern LLMs are instruction-tuned (SFT + RLHF) after the pre-training phase.