GDPO
How it works
The GDPO pipeline has four stages. Stage 1: Rollout ā policy ĻĪø generates a group of K candidates {y1, ā¦, yK} for input x. Stage 2: Per-reward standardisation ā for each reward R_j and each candidate y_i, a normalised scalar r'_ji = (R_j(y_i) ā Ī¼_j) / (Ļ_j + Īµ) is computed, where Ī¼_j and Ļ_j are computed independently for each reward channel within the group. Stage 3: Sum and batch-normalize ā normalised rewards are summed per candidate (s_i = Ī£_j r'_ji), then a second group-relative normalisation is applied to the summed signal: A_i = (s_i ā Ī¼_s) / (Ļ_s + Īµ). Stage 4: PPO-style policy update ā the model is updated with the standard formula L = -E[A_i Ć log(ĻĪø(y_i|x)/Ļref(y_i|x))], with importance sampling clipping and KL regularisation against Ļref.
Problem solved
Standard GRPO (and before it PPO/RLHF) was designed for a single scalar reward. When modern RL pipelines need to simultaneously optimise multiple heterogeneous rewards (e.g. correctness + format + length), naive summation and normalisation collapses different reward combinations to identical advantage values. This reduces the resolution of the training signal, causes suboptimal convergence and sometimes early training failure. GDPO solves this with decoupled normalisation ā each reward is standardised separately before being summed.
Key mechanisms
Strengths & limitations
Components
For each reward R_j, Ī¼_j and Ļ_j are computed independently within a group of K candidates. Each raw reward is then standardised: r'_ji = (R_j(y_i) ā Ī¼_j) / (Ļ_j + Īµ). This is the key difference from GRPO, where all rewards are summed before any normalisation.
After per-reward normalisation, the normalised values are summed per candidate: s_i = Ī£_j r'_ji. A second normalisation is then applied ā group-relative batch normalisation on the summed signal: A_i = (s_i ā Ī¼_s) / (Ļ_s + Īµ). The resulting A_i is the advantage used in the policy update.
Standard PPO/GRPO policy update formula with importance sampling clipping and KL regularisation against a frozen reference policy Ļref: L = -E[A_i Ć min(ratio, clip(ratio, 1-Īµ, 1+Īµ))] + Ī² Ć KL(ĻĪø || Ļref), where ratio = ĻĪø(y_i|x) / Ļref(y_i|x). This layer is unchanged from GRPO ā the difference lies solely in how A_i is computed.
Official
Optional extension for the constrained formulation in which some rewards are treated as constraints with target thresholds Ļ_c. The composite reward then becomes: R(y_i) = R_primary(y_i) ā Ī£_c Ī»_c(t) Ć ReLU(Ļ_c ā R_c(y_i)), where Ī»_c(t) are PID-controlled time-varying Lagrange multipliers. This pattern is used in Kuaishou's RaG/SCRL.
Official
Implementation
Per-reward standardisation requires reasonable Ī¼_j, Ļ_j statistics within a group. For small K (e.g. K=2, 4) the statistics are unstable ā within-group variance can be zero or very small, leading to exploding advantage values or NaN.
Per-reward standardisation removes information about relative magnitudes between different rewards. If one reward is an order of magnitude more important than another, GDPO has no way to express this ā all are scaled to a similar range.
For constrained problems (constraint thresholds + Lagrangian multipliers), naively applying GDPO without PID-controlled Lagrangian updates can lead to oscillation and overshoot ā Ī»_c grows too aggressively after a constraint violation, then overcorrects.
Evolution
Schulman et al. introduce Proximal Policy Optimization ā the foundation of all later on-policy RL methods for LLMs, including GRPO and GDPO.
OpenAI publishes InstructGPT using RLHF (reward model + PPO) to align LLMs with human annotator preferences ā the standard post-training pipeline.
DeepSeek introduces Group Relative Policy Optimization ā eliminates the value model by using group-relative advantage normalisation. A critical precursor to GDPO.
Liu et al. (NVIDIA Tech Report, arXiv 2601.05242) introduce GDPO ā a direct extension of GRPO with decoupled per-reward normalisation. Solves the collapse of different reward combinations to identical advantage values in multi-reward settings.
GDPO lands in ms-swift (modelscope) as an official rlhf_type option alongside GRPO, giving the community an open-source implementation for the Megatron training framework.
Kuaishou Technology uses GDPO as the core of Synergistic Cross-Domain Reward Learning (SCRL) in the production Recommendation-as-Generation paradigm. The first large-scale production deployment of GDPO (400M+ DAU).
Hyperparameters (configurable axes)
Number of candidates K in a group rollout. Larger K = better per-reward statistics, but higher compute cost.
Weight of KL regularisation against Ļref. Retained from PPO/GRPO. Larger Ī² = slower drift from the reference policy.
PPO-style clip epsilon for the importance sampling ratio. Retained from PPO.
Optional (only for the constrained formulation). Target thresholds for each constraint reward. RaG calibrates them against the SFT baseline distribution as Ļ_c = Ī¼_c + k_c Ć Ļ_c.
Execution paradigm
GDPO is an optimisation algorithm, not a model architecture modification. All reward channels are always active and normalised independently ā there is no conditional routing or sparse activation.
Parallelism
Per-reward normalisation and summation are lightweight statistical operations, fully parallelised within a group of candidates (on GPU) and across multiple groups (across devices). The bottleneck remains the rollout phase (forward passes of the policy for K candidates), not the GDPO optimisation itself.
Hardware requirements
GDPO is a drop-in replacement for GRPO/PPO ā all operations are standard GEMMs and reduce operations on GPU. NVIDIA developed the method with its own training stack (Megatron) in mind.
The algorithm is hardware-agnostic ā it can be implemented on any hardware supporting standard policy optimisation operations (TPU, Habana, AMD MI300 etc.).