Alignment
Implicit PRM
2024ResearchUpdated: 22 September 2026Published
Key
innovation
Getting per-step rewards without process labels — implicitly from an outcome-trained model.
Category
Alignment
Abstraction level
Building block
Operation level
Post-training
Use cases
Cheap PRMMathematical reasoningRL/reranking
How it works
A reward model is trained on the outcome signal; via a log-likelihood parameterization, per-step rewards emerge implicitly.
Problem solved
Training explicit PRMs requires expensive per-step labels for each reasoning step.