Robots Atlas>ROBOTS ATLAS
Training

Intrinsic Motivation

2004ActivePublished
Key innovation
Moving the source of reward inside the agent: instead of relying solely on sparse external reward, the agent generates its own reward signal (curiosity, novelty, competence, empowerment), enabling exploration and learning without dense supervision.
Category
Training
Abstraction level
Paradigm
Operation level
TrainingAgent runtime
Use cases
Exploration in sparse-reward environments (sparse-reward RL)Learning without external reward (self-supervised exploration)Developmental robotics and open-ended learningExploratory pre-training of policiesAutomatic curriculum generation

How it works

The agent computes an internal reward signal r_int from its own state of knowledge or competence, then optimises a policy against the sum of the intrinsic and (optionally) external reward. In knowledge-based approaches r_int is proportional to the prediction error of a world model or to state novelty — the agent is rewarded for visiting states it cannot yet predict. In competence-based approaches the reward is the rate of improvement (learning progress), steering the agent toward regions where it learns fastest. Information-theoretic measures such as empowerment reward states that maximise the agent’s influence over the future (the channel capacity from actions to future states). As learning proceeds the intrinsic signal typically fades, because states stop being novel or hard to predict.

Problem solved

It addresses the exploration problem in environments with sparse or absent external reward, where standard RL has nothing to maximise and learning stalls. It provides a signal that drives exploration and skill acquisition even without an explicit goal from the environment.

Components

Intrinsic reward signalLearning signal source independent of the environment

An internally computed reward r_int added to (or replacing) the external reward, steering policy optimisation.

World model / predictorProvides a novelty / unpredictability measure

A model predicting the next state or its features; prediction error feeds the reward in knowledge-based approaches (prediction error, novelty).

Learning-progress estimatorDirects exploration toward fast-learning regions

Measures the rate of improvement in the agent’s competence and rewards regions with high learning progress (competence-based approach).

Empowerment measureRewards control over the environment without an external goal

An information-theoretic objective rewarding states that maximise the agent’s influence over the future (the action→future-state channel capacity).

Implementation

Reference implementations
Random Network Distillation (OpenAI)
Python · OpenAI
Official
noreward-rl (ICM, Pathak)
Python · Deepak Pathak
Official
Implementation pitfalls
Noisy-TV problemHigh

Prediction-error or novelty methods get stuck on inherently stochastic stimuli, generating unbounded intrinsic reward.

Fix:Use RND or learning-progress / competence-based signals instead of raw prediction error.
Non-stationarity of the intrinsic rewardMedium

The intrinsic reward shrinks as the agent learns, complicating value-function estimation.

Fix:Reward normalisation and separate value heads for intrinsic and extrinsic reward.
Miscalibrated intrinsic reward weightMedium

Too high a coefficient dominates the task; too low and exploration is ignored.

Fix:Tune the β coefficient, normalise, and optionally decay it over time.

Evolution

Original paper · 2004 · NeurIPS 2004 · Satinder Singh
Intrinsically Motivated Reinforcement Learning
Satinder Singh, Andrew G. Barto, Nuttapong Chentanez
1990
Early computational curiosity (Schmidhuber)
Inflection point

Schmidhuber introduces curiosity and boredom as intrinsic reward in model-building controllers.

2004
Intrinsically Motivated Reinforcement Learning
Inflection point

Singh, Barto and Chentanez formalise intrinsic motivation within the reinforcement-learning framework.

Intrinsically Motivated Reinforcement Learning (paper)
2007
Typology of computational approaches (Oudeyer & Kaplan)
Inflection point

Oudeyer and Kaplan organise approaches into knowledge-based, competence-based and morphological families.

What is intrinsic motivation? A typology of computational approaches (paper)
2010
Formal theory of creativity (Schmidhuber)

The intrinsic reward becomes the progress in compressing observed data (compression progress).

Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) (paper)
2017
Intrinsic Curiosity Module (Pathak et al.)
Inflection point

Prediction-error reward in a learned feature space drives exploration in deep RL.

Curiosity-driven Exploration (concept)Curiosity-driven Exploration by Self-supervised Prediction (paper)
2018
Random Network Distillation (Burda et al.)

Prediction-error reward against a random fixed network — more robust to the noisy-TV problem.

Exploration by Random Network Distillation (paper)

Hyperparameters (configurable axes)

Intrinsic reward weightCritical

A coefficient scaling the intrinsic reward relative to the extrinsic one (often denoted β). Controls how strongly exploration dominates the task.

β ∈ [0.01, 1.0]typical tuning range
Intrinsic signal typeHigh

The signal family: prediction error, novelty/count-based, learning progress (competence), or empowerment.

Reward normalizationMedium

How the non-stationary scale of the intrinsic reward, which changes during training, is handled.

Decay scheduleMedium

Optional decay of the intrinsic reward over time so that the extrinsic reward prevails later in training.

Hardware requirements

Primary

A reward-design and RL-algorithmic paradigm; its compute cost is inherited from the underlying algorithm and model, with no dependence on a specific hardware type.