Sparse Reward
How it works
In a sparse-reward regime the agent receives a reward of 0 for most of an episode, with a non-zero signal appearing only in the goal state. Because random exploration rarely reaches such a state, standard algorithms (e.g. DQN, PPO, DDPG) may fail to learn anything. Several families of techniques address this: (1) potential-based reward shaping (Ng et al. 1999), which adds an auxiliary signal without changing the optimal policy; (2) intrinsic motivation / curiosity — an extra reward for novelty or model prediction error (Intrinsic Curiosity Module, Random Network Distillation) that drives exploration; (3) goal relabeling (Hindsight Experience Replay) — failed trajectories are treated as successful for the goal actually reached, producing a learning signal from failures; (4) curriculum learning — gradually increasing goal difficulty; (5) expert demonstrations or imitation that steer the agent near rewarding states.
Problem solved
It captures one of the central challenges of RL: how to learn an effective policy when the environment provides feedback only very rarely. It motivates techniques that improve exploration and credit assignment, enabling agents to be trained without hand-crafted dense (and exploit-prone) reward engineering.
Components
A reward function that returns a non-zero (often binary) signal only in a few states — usually upon reaching the goal. It defines the sparse-reward regime.
Official
A strategy (e.g. curiosity, novelty bonuses, count-based exploration) that leads the agent to discover the rare rewarding states when the extrinsic signal alone is insufficient.
Official
The mechanism that propagates a distant, delayed reward back to the actions that led to it (e.g. via the discount factor, TD-learning, goal relabeling).
Official
Implementation
With a purely sparse reward the agent may never reach a rewarding state, so the gradient is effectively zero and learning stalls.
Adding an arbitrary dense signal can shift the optimum and cause reward hacking — the agent optimizes the proxy instead of the true goal.
Even when the agent eventually learns, a sparse reward requires a very large number of environment interactions.
Evolution
Ng, Harada and Russell showed how to add an auxiliary signal that eases reward sparsity without changing the optimal policy.
Pathak et al. introduced the Intrinsic Curiosity Module — an intrinsic reward from prediction error enabling exploration with little or no extrinsic reward.
Andrychowicz et al. demonstrated learning from sparse, binary rewards by relabeling failed trajectories as successful for the goal actually reached.
Burda et al. proposed a simple exploration bonus based on random network distillation, advancing hard sparse-reward games (Montezuma’s Revenge).
Ecoffet et al. introduced the "return, then explore" paradigm, dramatically improving results in environments with extremely sparse rewards.
Hyperparameters (configurable axes)
How often the environment returns a non-zero signal — from fully sparse (terminal only) to partially dense rewards.
The coefficient scaling the exploration bonus (curiosity/novelty) relative to the extrinsic reward.
Determines how far back a delayed reward is propagated; critical for long-horizon credit assignment.
Compute bottleneck
The main bottleneck is not compute but sample efficiency: the number of environment interactions needed to even encounter a non-zero reward and begin learning.
Execution paradigm
Here "sparse" refers to the density of the reward signal over time, not to sparse activation of model parameters.
Parallelism
Experience collection can be parallelized across many actors/environments (distributed RL), but policy updates and propagation of the sparse reward remain sequentially dependent.
Hardware requirements
Sparse reward is an algorithmic challenge (exploration, credit assignment), independent of any specific hardware type.
Policy/value networks and parallel experience collection across many actors benefit from GPU acceleration.