Robots Atlas>ROBOTS ATLAS
Training

Intermediate Checkpoint

2023ActiveUpdated: 17 November 2026Published
Key innovation
Treating a partially trained model state — saved during training rather than only at its end — as a first-class artefact for resumption, evaluation, training-dynamics analysis and release.
Category
Training
Abstraction level
Pojęcie
Operation level
TrainingServingEvaluation (runtime)
Use cases
Resuming training after a failure (fault tolerance)Rollback to a stable state after a loss spikeProgress evaluation and early stoppingStudying training dynamics (e.g. capability emergence, scaling laws)Releasing open checkpoint series for research (Pythia, OLMo)

How it works

At selected moments of training (every N steps or every fixed interval) the process saves the state to durable storage. A full checkpoint includes: the model weights, the optimizer state (for Adam roughly 2× the number of parameters for the moments), the learning-rate schedule state, random seeds/generators and metadata (step, metrics). On resumption these elements are loaded and training continues as if there had been no interruption. At scale, distributed checkpointing is used: the state is sharded across parallel processes (data/tensor/pipeline parallelism) and written in parallel, often asynchronously, so as not to block compute. A retention policy decides how many checkpoints to keep (e.g. the last K plus selected "milestones"), since each takes a multiple of the model size. For release or inference, often only the weight set (without optimizer state) is exported, in a format such as safetensors.

Problem solved

Long training of large models is unreliable and expensive: hardware fails, the loss can destabilise, and the final model does not explain how it came to be. Intermediate checkpoints solve this by persisting state during training — enabling resumption without losing weeks of compute, rollback to a stable state, progress evaluation and scientific analysis of learning dynamics from a series of snapshots.

Components

Model weights snapshotThe core of the artefact; sufficient for inference and release.

The values of the model's parameters at a given step — the minimal and always-present component of any checkpoint.

Optimizer stateEnables resumption without resetting the optimisation dynamics.

Optimizer statistics (e.g. first and second moments in Adam), typically taking about 2× the number of parameters; required for seamless training resumption.

Official

Training and RNG stateEnsures continuity and reproducibility of the training run.

The step number, learning-rate schedule state, position in the data and random-generator seeds — needed for deterministic resumption.

Official

Checkpoint metadataLets one select and compare checkpoints within a series.

Step/epoch, metrics (loss, evaluations), code version and configuration — they identify and describe a given snapshot within a series.

Official

Serialization formatThe persistence layer; it affects write cost and portability.

The way the state is written to disk/object storage (e.g. safetensors for weights, distributed-checkpoint formats for full state), determining I/O safety and speed.

Official

Implementation

Implementation pitfalls
Incomplete checkpoint prevents resumptionHigh

Saving only the weights without the optimizer state, schedule and RNG makes seamless, deterministic training resumption impossible.

Fix:For operational checkpoints save the full state; keep a weights-only version separately, for release and inference.
Corrupted checkpoint on partial writeMedium

A failure during writing (especially distributed) can leave an inconsistent, unreadable checkpoint.

Fix:Use atomic writes (temporary file then rename), checksums and post-write verification.

Evolution

Original paper · 2023 · ICML 2023 · Stella Biderman
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Stella Biderman, Hailey Schoelkopf
2022
OPT-175B logbook and checkpoints

Meta releases OPT together with a training logbook and checkpoints, showcasing the practice of saving the state of very large models during training.

2023
Pythia — intermediate checkpoints as a research tool
Inflection point

Pythia releases 154 intermediate checkpoints per model, making training dynamics and scaling a subject of systematic study.

2024
OLMo — fully open checkpoints

OLMo releases code, data and a series of intermediate checkpoints, reinforcing the reproducibility of language-model research.

Hyperparameters (configurable axes)

Checkpoint frequencyHigh

How often (steps or minutes) a checkpoint is saved; a trade-off between I/O cost and the amount of compute lost on failure.

co 1000 krokówA typical trade-off for large training runs.
co 1–2 h czasu rzeczywistegoA time-based policy for fault tolerance.
Retention countHigh

How many checkpoints to keep at once (e.g. the last K plus selected milestones); controls storage usage.

ostatnie 3Minimal operational retention.
154 (Pythia)A full series for training-dynamics research.
Saved state scopeHigh

Whether to save only the weights (for inference/release) or the full state (weights + optimizer + training) allowing resumption.

weights-onlySmall size, for release and inference.
full stateLarger size, for seamless resumption.
Sharding strategyMedium

How the state is split across parallel processes in distributed saving (data/tensor/pipeline parallelism).

sharded (DCP)Each process writes its own shard of the state.
Asynchronous saveMedium

Whether the save happens in the background, overlapping I/O with further compute to minimise training stall.

enabledReduces stall under frequent checkpointing.

Computational complexity

Time complexity: O(S). Space complexity: O(N · S).

Parallelism

Parallelism level
Partially parallel

Distributed checkpointing shards the state across parallel processes and writes it in parallel, often asynchronously, so as not to block training.

Scope
TrainingAcross devices
Constraints
!In distributed training, processes usually must synchronise (a barrier) at the moment of writing a consistent checkpoint.
!Parallel sharded writes are bounded by network/object-storage bandwidth.

Hardware requirements

Primary

Checkpointing is an I/O operation independent of accelerator type; performance is determined by storage and network bandwidth, not the kind of compute unit.