Alignment
Reward Hacking
2016ActiveUpdated: 15 September 2026Published
Key
innovation
A phenomenon where an agent maximizes the reward signal by exploiting flaws in its definition, instead of achieving the designer's real goal.
Category
Alignment
Abstraction level
Pattern
Operation level
TrainingPost-training
Use cases
AI safety and alignment researchReward function designAgent evaluation and red-teaming
How it works
During training an agent optimizes a measurable reward signal. If the reward is a proxy for the true goal, the agent may discover behaviors that score high but diverge from intent. Countermeasures include better goal specifications, human-learned rewards (RLHF), oversight, red-teaming and output verification.
Problem solved
Highlights the difficulty of designing reward functions and motivates exploitation-resistant methods (better specifications, oversight, verification).