Guowei Xu of Tsinghua University, with Mert Yuksekgonul and James Zou of Stanford University, show that reward-related information in a large language model is concentrated in under 1% of a layer’s neurons. Zeroing that one percent drops Qwen-2.5-7B’s accuracy on MATH500 from 75.2% to 20.3%. Zeroing the same number of random neurons changes nothing.
Key takeaways
- Fewer than 1% of a layer’s neurons suffice to predict state value
- Zeroing 1% of value neurons: 75.2% → 20.3% on MATH500
- Zeroing 1% of random neurons: 74.6%, a drop of 0.6 points
- Dopamine neurons encode TD error — the gap between expectation and outcome
- Used as a process reward model they lift accuracy from 72.2% to 77.8%
Two kinds of neurons, one analogy
The authors train a simple two-layer probe on hidden states and prune its inputs step by step. AUC does not fall — the probe works on a handful of neurons. These are called value neurons, because they predict whether the model will solve the task before it writes a single word of the answer.
A second set, dopamine neurons, turns up where prediction and outcome diverge: activation spikes when the model makes unexpected progress and collapses at the moment of an error.
Symbol meaning
- …
- TD error — the gap between what the model expected and what it got
- …
- reward received at step t
- …
- predicted value of the state before the step
- …
- discount factor on future rewards
Controls that separate important from reward-related
An accuracy drop after ablation proves nothing on its own — removing any load-bearing neurons hurts. So the team compared four other selection criteria.
| Criterion for picking 1% of neurons | Average accuracy |
|---|---|
| Value neurons | 20.3% |
| Wanda | 65.8% |
| Magnitude | 71.6% |
| Random | 74.6% |
| NTP neurons (next-token prediction) | 76.4% |
Value neurons and NTP neurons overlap by 0.7% against a random baseline of 0.5%. The pattern repeats on GSM8K, Minerva Math, ARC and MMLU STEM, and across Llama, Gemma and Phi.
From observation to tool
Reading the value neurons predicts model confidence before generation better than a probe on the full hidden state — average AUC 0.67 against 0.60 — and far better than simply asking the model how confident it is (0.52). Dopamine neurons served as a process reward model, picking among four candidates per reasoning paragraph: 77.8% against 75.0% for an implicit PRM and 72.2% for greedy decoding.
Why it matters
Reward models are trained separately today, often at a size comparable to the model being judged. If the reward signal already sits in a hundred neurons in the third layer, part of that work is duplication. The practical consequence is simpler than interpretability itself: a cheap confidence read before generation lets you spend compute on the problems that actually need it.
What's next
- The results come from models up to 14B parameters — whether the picture holds higher up is untested
- The second version, dated 11 May, adds the magnitude, Wanda and NTP controls absent from the February submission
Sources
- arXiv — Sparse Reward Subsystem in Large Language Models
- Hugging Face — Qwen-2.5-7B-SimpleRL-Zoo
