Intermediate Checkpoint
How it works
At selected moments of training (every N steps or every fixed interval) the process saves the state to durable storage. A full checkpoint includes: the model weights, the optimizer state (for Adam roughly 2× the number of parameters for the moments), the learning-rate schedule state, random seeds/generators and metadata (step, metrics). On resumption these elements are loaded and training continues as if there had been no interruption. At scale, distributed checkpointing is used: the state is sharded across parallel processes (data/tensor/pipeline parallelism) and written in parallel, often asynchronously, so as not to block compute. A retention policy decides how many checkpoints to keep (e.g. the last K plus selected "milestones"), since each takes a multiple of the model size. For release or inference, often only the weight set (without optimizer state) is exported, in a format such as safetensors.
Problem solved
Long training of large models is unreliable and expensive: hardware fails, the loss can destabilise, and the final model does not explain how it came to be. Intermediate checkpoints solve this by persisting state during training — enabling resumption without losing weeks of compute, rollback to a stable state, progress evaluation and scientific analysis of learning dynamics from a series of snapshots.
Components
The values of the model's parameters at a given step — the minimal and always-present component of any checkpoint.
Optimizer statistics (e.g. first and second moments in Adam), typically taking about 2× the number of parameters; required for seamless training resumption.
Official
The step number, learning-rate schedule state, position in the data and random-generator seeds — needed for deterministic resumption.
Official
Step/epoch, metrics (loss, evaluations), code version and configuration — they identify and describe a given snapshot within a series.
Official
The way the state is written to disk/object storage (e.g. safetensors for weights, distributed-checkpoint formats for full state), determining I/O safety and speed.
Official
Implementation
Saving only the weights without the optimizer state, schedule and RNG makes seamless, deterministic training resumption impossible.
A failure during writing (especially distributed) can leave an inconsistent, unreadable checkpoint.
Evolution
Meta releases OPT together with a training logbook and checkpoints, showcasing the practice of saving the state of very large models during training.
Pythia releases 154 intermediate checkpoints per model, making training dynamics and scaling a subject of systematic study.
OLMo releases code, data and a series of intermediate checkpoints, reinforcing the reproducibility of language-model research.
Hyperparameters (configurable axes)
How often (steps or minutes) a checkpoint is saved; a trade-off between I/O cost and the amount of compute lost on failure.
How many checkpoints to keep at once (e.g. the last K plus selected milestones); controls storage usage.
Whether to save only the weights (for inference/release) or the full state (weights + optimizer + training) allowing resumption.
How the state is split across parallel processes in distributed saving (data/tensor/pipeline parallelism).
Whether the save happens in the background, overlapping I/O with further compute to minimise training stall.
Computational complexity
Time complexity: O(S). Space complexity: O(N · S).
Parallelism
Distributed checkpointing shards the state across parallel processes and writes it in parallel, often asynchronously, so as not to block training.
Hardware requirements
Checkpointing is an I/O operation independent of accelerator type; performance is determined by storage and network bandwidth, not the kind of compute unit.