Robots Atlas>ROBOTS ATLAS
Training

Curiosity-driven Exploration

2017ActivePublished: 24 October 2026Updated: 24 October 2026Published
Key innovation
Replacing hand-engineered exploration with an intrinsic reward generated by the model itself (prediction error), letting an agent learn in sparse-reward or reward-free environments.
Category
Training
Abstraction level
Pattern
Operation level
Training
Use cases
Reinforcement learning in sparse-reward environmentsHard-exploration video games (e.g. Montezuma's Revenge)Reward-free explorationRobotics and manipulation skill learningPretraining and representation learning through environment interaction

How it works

The agent maintains a learned predictive model of the environment. At each step the model predicts some quantity (e.g. features of the next state from the current state and action), and the discrepancy between prediction and observation — the prediction error — is used as an intrinsic reward added to the (possibly present) extrinsic reward. In the Intrinsic Curiosity Module (Pathak et al., 2017) an inverse model is trained to predict the action from a pair of states in order to learn a feature space robust to unpredictable, task-irrelevant parts of the environment; a forward model predicts the next state in that feature space and its error is the curiosity reward. In Random Network Distillation (Burda et al., 2018) the reward is the error of a predictor network trained to match the output of a fixed, randomly initialized target network; rarely visited states yield high error and thus high reward. As learning proceeds the model predicts visited states ever better, so their curiosity reward shrinks, naturally shifting exploration toward unseen regions.

Problem solved

In many reinforcement-learning tasks the extrinsic reward is sparse, delayed or entirely absent, so classic exploration methods (e.g. epsilon-greedy) fail to discover useful behaviours. Curiosity-driven exploration provides a dense, self-generated signal that steers the agent toward novel states and enables learning despite little or no environmental feedback.

Components

Intrinsic reward (prediction error)Provides a dense exploration signal

A reward signal generated by the agent itself, proportional to the model's prediction error; added to the extrinsic reward.

Forward dynamics modelSource of the curiosity signal

Predicts the next state (or its features) from the current state and action; its error constitutes the curiosity reward.

Feature encoder / inverse modelFilters out irrelevant sources of unpredictability

In ICM an inverse model predicts the action from a pair of consecutive states, learning a feature space robust to unpredictable, task-irrelevant parts of the environment.

Random Network Distillation (RND)Uses the prediction error of a fixed random target network's output instead of a dynamics model.
Pseudo-counts / count-basedReward derived from an estimated state-visitation count.

Official

Implementation

Reference implementations
noreward-rl (ICM, authors)
Python · Deepak Pathak
Official
random-network-distillation (OpenAI)
Python · OpenAI
Official
large-scale-curiosity (OpenAI)
Python · OpenAI
Official
Implementation pitfalls
The noisy-TV problemHigh

An agent driven by prediction error can get stuck at sources of irreducible stochasticity (e.g. a noisy screen) because they remain permanently unpredictable and keep yielding high reward.

Fix:Compute the error in a learned feature space (ICM) or against a fixed random network (RND) rather than in the raw observation.
Non-stationarity and vanishing intrinsic rewardMedium

The curiosity reward decreases as the model learns, making stable scaling difficult and potentially extinguishing exploration.

Fix:Reward normalization (e.g. by a running standard deviation) and careful tuning of the intrinsic reward coefficient.

Evolution

Original paper · 2017 · Deepak Pathak
Curiosity-driven Exploration by Self-supervised Prediction
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell
1991
Formalization of intrinsic motivation and curiosity
Inflection point

Jürgen Schmidhuber proposes curiosity based on world-model prediction error as a controller's intrinsic motivation.

2015
Curiosity in deep RL (predictive models)

Stadie, Levine and Abbeel use the error of a deep predictive model as an exploration bonus in Atari.

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models (paper)
2016
Count-based and information-theoretic intrinsic motivation

VIME (Houthooft et al.) and pseudo-counts (Bellemare et al.) unify count-based exploration and intrinsic motivation.

Unifying Count-Based Exploration and Intrinsic Motivation (paper)
2017
Intrinsic Curiosity Module (ICM)
Inflection point

Pathak et al. introduce ICM with an inverse model defining a feature space, making curiosity robust to task-irrelevant stochasticity.

Curiosity-driven Exploration by Self-supervised Prediction (paper)
2018
Random Network Distillation and large-scale study
Inflection point

Burda et al. introduce RND (first to exceed human performance on Montezuma's Revenge with pure curiosity) and publish a large-scale study of purely curiosity-driven learning across 54 environments.

Exploration by Random Network Distillation (paper)

Hyperparameters (configurable axes)

Intrinsic reward coefficientHigh

Scaling coefficient balancing the curiosity reward against the extrinsic reward (often denoted β or η).

Feature representationHigh

The space in which prediction error is computed (raw pixels, random features, inverse-model learned features, VAE).

Reward normalizationMedium

Normalization of the non-stationary intrinsic reward (e.g. by a running standard deviation), critical for RND stability.