Rollout
How it works
Step 1: freeze the current policy (a weight snapshot) as the "sampling policy". Step 2: feed a batch of initial states — in classic RL these are environment states, in LLMs they are prompts. Step 3: generation — the policy selects actions (in LLMs: autoregressive token sampling with a given temperature/top-p), recording actions, log-probabilities and possibly values. Step 4: reward collection — the environment or a reward model/verifier assigns rewards to the trajectories (in LLMs often a reward at the end of the sequence). Step 5: store the trajectories in an experience buffer together with the metadata needed for advantage estimation. Only after the rollouts are collected does a separate update phase follow (e.g. PPO or GRPO), in which the gradient is computed on the collected samples and the weights are updated. In group variants (GRPO), multiple rollouts are generated for a single prompt, and the advantage is computed relative to the mean reward within the group.
Problem solved
The RL loop needs fresh behaviour samples generated by the current policy in order to estimate the gradient and improve decisions. The rollout supplies this data in a way explicitly separated from the weight update, which lets one parallelise generation, control the freshness of the samples (on-policy vs off-policy) and optimise the most expensive part of training independently of the optimisation step itself.
Components
A frozen copy of the current policy weights from which actions are sampled during the rollout; it keeps the samples consistent for the update phase.
The source of initial states and dynamics: an RL environment (states, transitions) or a batch of prompts for a language model.
Official
The rule for selecting actions from the policy distribution (e.g. temperature, top-p, top-k), controlling exploration during the rollout.
Official
A reward model, environment function or verifier (e.g. checking answer correctness) that assigns rewards to trajectories after they are generated.
Official
A structure storing the collected trajectories, log-probabilities, values and rewards for advantage estimation and gradient computation.
Official
Implementation
If the policy changes too much between collecting rollouts and updating, the samples become off-policy and the gradient estimator is biased.
Discrepancies between the engine generating rollouts and the engine computing log-probs (e.g. numerical differences, KV-cache length) corrupt the learning signal.
Evolution
Sutton and Barto's textbook systematises trajectories/episodes as the basis of Monte Carlo methods and on-policy learning.
PPO formalises the cycle: collect rollouts from the old policy, compute advantage, take several clipped update steps.
InstructGPT uses rollouts of generated LLM answers scored by a reward model in a PPO loop.
DeepSeekMath introduces GRPO: multiple rollouts per prompt, with advantage computed relative to the group's mean reward — without a separate value network.
Hyperparameters (configurable axes)
Number of trajectories/prompts collected before a single policy update.
How many rollouts are generated per prompt; critical for group methods such as GRPO, where advantage is computed relative to the group.
The maximum number of tokens (or environment steps) per trajectory; it directly drives the rollout cost.
A parameter controlling exploration during rollout generation; too low limits diversity, too high degrades sample quality.
How many rollouts are collected between successive weight updates; it regulates the degree of on-policy vs off-policy.
Computational complexity
Time complexity: O(G · T · C_fwd). Space complexity: O(G · T) + O(T · d · L).
Execution paradigm
The rollout is the sampling/generation phase, distinct from the gradient-computation and weight-update phase.
The rollout as a generation phase introduces no routing of its own; the sparse mode appears only when the policy itself is an MoE model.
Parallelism
Different rollouts (different prompts/episodes) are independent and massively parallelisable across many GPUs/workers.
Hardware requirements
Autoregressive generation is matrix-multiply intensive; GPUs with Tensor Cores are the standard for rollout workers.
TPUs handle batched policy inference well with suitable serving software.