Robots Atlas>ROBOTS ATLAS
Training

Rollout

2017ActiveUpdated: 17 November 2026Published
Key innovation
Isolating the phase of generating trajectories (samples) from the current policy as a separate, parallelisable and usually cost-dominant step of the reinforcement learning loop.
Category
Training
Abstraction level
Building block
Operation level
TrainingPost-trainingInference
Use cases
RLHF — generating LLM completions scored by a reward modelRLVR / GRPO — group rollouts for reasoning tasks with verifiable rewardTraining robot control policies in simulation (sim-to-real)On-policy RL in games and sequential environmentsGenerating synthetic data for further training

How it works

Step 1: freeze the current policy (a weight snapshot) as the "sampling policy". Step 2: feed a batch of initial states — in classic RL these are environment states, in LLMs they are prompts. Step 3: generation — the policy selects actions (in LLMs: autoregressive token sampling with a given temperature/top-p), recording actions, log-probabilities and possibly values. Step 4: reward collection — the environment or a reward model/verifier assigns rewards to the trajectories (in LLMs often a reward at the end of the sequence). Step 5: store the trajectories in an experience buffer together with the metadata needed for advantage estimation. Only after the rollouts are collected does a separate update phase follow (e.g. PPO or GRPO), in which the gradient is computed on the collected samples and the weights are updated. In group variants (GRPO), multiple rollouts are generated for a single prompt, and the advantage is computed relative to the mean reward within the group.

Problem solved

The RL loop needs fresh behaviour samples generated by the current policy in order to estimate the gradient and improve decisions. The rollout supplies this data in a way explicitly separated from the weight update, which lets one parallelise generation, control the freshness of the samples (on-policy vs off-policy) and optimise the most expensive part of training independently of the optimisation step itself.

Components

Sampling policy (snapshot)The source of actions in the experience-collection phase.

A frozen copy of the current policy weights from which actions are sampled during the rollout; it keeps the samples consistent for the update phase.

Environment / prompt setDefines the task on which trajectories are collected.

The source of initial states and dynamics: an RL environment (states, transitions) or a batch of prompts for a language model.

Official

Sampling / decoding strategyControls the exploration–exploitation balance of the collected samples.

The rule for selecting actions from the policy distribution (e.g. temperature, top-p, top-k), controlling exploration during the rollout.

Official

Reward signal / verifierProvides the scalar learning signal for the update phase.

A reward model, environment function or verifier (e.g. checking answer correctness) that assigns rewards to trajectories after they are generated.

Official

Experience bufferThe bridge between the rollout phase and the policy-update phase.

A structure storing the collected trajectories, log-probabilities, values and rewards for advantage estimation and gradient computation.

Official

Implementation

Implementation pitfalls
Drift between sampling and updated policyHigh

If the policy changes too much between collecting rollouts and updating, the samples become off-policy and the gradient estimator is biased.

Fix:Use importance sampling and clipping (as in PPO) and limit the number of update steps per rollout batch.
Train–inference generation mismatchMedium

Discrepancies between the engine generating rollouts and the engine computing log-probs (e.g. numerical differences, KV-cache length) corrupt the learning signal.

Fix:Unify precision and decoding parameters and verify log-probability agreement across paths.

Evolution

Original paper · 2018 · MIT Press · Richard S. Sutton
Reinforcement Learning: An Introduction (2nd ed.)
Richard S. Sutton, Andrew G. Barto
2018
Rollout as a classic RL concept

Sutton and Barto's textbook systematises trajectories/episodes as the basis of Monte Carlo methods and on-policy learning.

2017
Rollout at the heart of PPO
Inflection point

PPO formalises the cycle: collect rollouts from the old policy, compute advantage, take several clipped update steps.

2022
Completion rollouts in RLHF
Inflection point

InstructGPT uses rollouts of generated LLM answers scored by a reward model in a PPO loop.

2024
Group rollouts (GRPO)
Inflection point

DeepSeekMath introduces GRPO: multiple rollouts per prompt, with advantage computed relative to the group's mean reward — without a separate value network.

Hyperparameters (configurable axes)

Rollout batch sizeHigh

Number of trajectories/prompts collected before a single policy update.

256Typical batch for LLM RLHF.
1024+Larger batches for a more stable gradient.
Samples per prompt (group)Critical

How many rollouts are generated per prompt; critical for group methods such as GRPO, where advantage is computed relative to the group.

8A common choice in GRPO.
16–64Larger groups reduce the variance of the advantage estimator.
Max generation length / horizonHigh

The maximum number of tokens (or environment steps) per trajectory; it directly drives the rollout cost.

1024Standard completions.
8192+Long reasoning chains.
Sampling temperatureMedium

A parameter controlling exploration during rollout generation; too low limits diversity, too high degrades sample quality.

1.0A typical value for exploration in LLM RL.
0.7Lower temperature for more conservative samples.
Rollouts per update (freshness)Medium

How many rollouts are collected between successive weight updates; it regulates the degree of on-policy vs off-policy.

1 epoka na partięStrictly on-policy (e.g. classic PPO).

Computational complexity

Time complexity: O(G · T · C_fwd). Space complexity: O(G · T) + O(T · d · L).

Execution paradigm

Primary mode
Dense

The rollout is the sampling/generation phase, distinct from the gradient-computation and weight-update phase.

Activation pattern
All paths active
Additional modes
Sparse
Routing mechanism

The rollout as a generation phase introduces no routing of its own; the sparse mode appears only when the policy itself is an MoE model.

Parallelism

Parallelism level
Partially parallel

Different rollouts (different prompts/episodes) are independent and massively parallelisable across many GPUs/workers.

Scope
InferenceAcross devices
Constraints
!Autoregressive generation is sequential — each token/step depends on the previous one, so a single trajectory cannot be fully parallelised over time.
!Rollouts from many workers must be gathered and synchronised before the policy-update phase.

Hardware requirements

Primary

Autoregressive generation is matrix-multiply intensive; GPUs with Tensor Cores are the standard for rollout workers.

Good fit

TPUs handle batched policy inference well with suitable serving software.