Tag: Reward model

An LLM's Reward Subsystem Fits in 1% of Its Neurons

A probe on hidden states finds "value neurons". Zeroing 1% of them drops Qwen-2.5-7B on MATH500 from 75.2% to 20.3%.