- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GPRL for online RL on open-ended tasks. It uses a General Preference Model trained on binary preference labels to obtain several latent preference signals. Instead of immediately combining them into a scalar reward, GPRL computes and normalizes group-relative advantages separately for each signal before aggregation. A drift controller monitors changes in their variance shares and adjusts the dimension weights and KL coefficient. Experiments on Llama-3-8B-Instruct and Gemma-2-9B-it report improvements over offline, iterative, and scalar-reward online baselines on four benchmarks, together with shorter responses and better stability under extended training.
Critical strengths
S1. The paper explores a useful way to use ordinary binary preference data for online RL. Its main idea is to retain the GPM's separate preference signals through advantage estimation, rather than collapsing them into one reward before normalization. This gives a concrete way to prevent a high-variance signal from overwhelming the others, without requiring manually defined reward dimensions. The resulting update also fits into the existing GRPO training framework.
S2. The experiments cover two main base policies and several types of baselines, with ablations on the number of subspaces, normalization, the controller, group size, and reward-model corpus. The LC WR gains on AlpacaEval are substantial: 56.51 versus 41.92 for GRPO+BT on Llama, and 74.31 versus 61.75 on Gemma. The extended-training comparison also provides useful evidence that the controller helps preserve performance later in training.
Critical weaknesses
W1. The main tables lack complete base-policy results.
Tables 1, 2, and 3 omit the scores of base policies, making it difficult to assess what each method gains or loses relative to its base model. In particular, merely being shorter than other methods may not show how GPRL changed or controlled length relative to the starting policy. Reporting base policy scores under identical evaluation settings would make these trajectory comparisons clear.
W2. Response length and quality need more complete evaluation details.
W2.1. Generation Settings & Truncation The generation settings require clarification. Table 11 specifies a maximum rollout length of 512, whereas Table 1 reports average GPRL response lengths of 1600 and 1791 tokens. While this discrepancy may stem from differing training and evaluation settings, key details such as the evaluation cutoff and training truncation rate are omitted. Consequently, it is unclear whether the shorter outputs reflect the intended length-control mechanism or training truncations. The authors should specify these parameters and test whether the length and quality gains hold under different rollout limits.
W2.2. Evaluation Reliability & Judge Bias While AlpacaEval's LC WR is promising, all four benchmarks rely solely on automatic judges without reporting confidence intervals. Given the small margins on benchmarks like Arena-Hard v2 (1.3 vs. 1.0 and 2.7 vs. 2.4), the stability of these gains remains unproven. Providing error bars, consistent decoding setups, and human or independent evaluations would better confirm that GPRL genuinely improves output quality while reducing verbosity.
W3. Score-variance share changes lack validation as a reward-hacking detector.
The controller triggers on shifts in score-variance allocation across dimensions. However, such shifts can signal either superficial exploitation or genuine skill acquisition. This is an ambiguity underscored by Appendix D.5, where a strict threshold (0.05) suppressed legitimate learning and dropped LC WR by 4.07 points. What empirical evidence confirms that the default controller specifically targets harmful exploitation rather than healthy specialization?
Furthermore, the monitor measures relative variance shares rather than absolute quality; uniform variance scaling across dimensions would register zero drift, leaving degradation undetected. While Figure 8 provides an external check on aggregate GPM rankings, it does not prove that downweighted dimensions are the actual sources of reward hacking.
To validate the detector, the paper may need concrete examples and controlled tests showing whether the monitor reacts to artificially lengthened responses and remains stable during real content improvements. Without reporting false positive and false negative rates against ground-truth quality, the results may prove the controller's heuristic utility rather than its validity as a reward-hacking detector.
Questions for authors
Q1. What do the learned dimensions represent? Figure 7 gives a useful interpretation for a two-subspace GPM trained on HelpSteer2, but the main experiments use a three-subspace GPM trained on Skywork-Reward. Is there corresponding evidence about what those main-experiment dimensions capture?
Q2. Why is the GPM's learned gate not used? Table 11 states that the GPM gate is unused and the base weights are uniform. Why discard the prompt-dependent weights learned during preference modeling, and how does retaining them affect the results? Since different decompositions can preserve the aggregate preference score while changing the separately normalized advantages, evidence of robustness would also help assess whether the policy update relies on a stable decomposition.
Q3. How are supervision and computation matched across the methods? The paper describes three iterations of SPPO/GPO and three epochs of GRPO/GPRL as comparable in compute, but these counts do not establish equal budgets. Could you report the training tokens and GPU hours for each method? Please also specify the policy-training data and preference annotations used for DPO/SimPO. Training reward models on the same corpus is a useful control, but it does not fully specify the data or optimization budget seen by each policy.
I would be happy to revise my assessment if the authors can clarify these details and provide the relevant evidence. Looking forward to further discussion.
4 comments
Response to Reviewer UeZz (2/2)
W3 (Validating drift). We address the raised concerns in turn.
-
Uniform variance scaling across dimensions does not alter policy update at all. Per-dimension normalization divides each subspace by its own standard deviation, so a common scale factor cancels in every , and there's essentially nothing for the controller to correct. Degradation of the GPM's absolute accuracy is more of a calibration question, which Appendix E.1 addresses.
-
We record the share of the dominant subspace in a controller-off run:
| Epoch (8B GPM, controller off) | LC WR | Avg. Len | ||
|---|---|---|---|---|
| 1 | 0.42 | 0.02 | 49.42 | 1718 |
| 2 | 0.49 | 0.06 | 54.92 | 1693 |
| 3 | 0.59 | 0.11 | 52.70 | 1742 |
| 5 | 0.76 | 0.27 | 47.71 | 2187 |
and observe that LC WR peaks at epoch 2 and then falls by 7.21 points as one subspace's share grows from 0.49 to 0.76, so the single-axis concentration the controller targets coincides with loss of LC WR, i.e., quality.
- Our existing runs already show both behaviors that the suggested tests focus on. rises with natural length inflation, as in the table above, and during improvement in the first epoch, stays near zero and the controller changes LC WR by at most 0.23 (Appendix D.3). A sweep of the param. (in Appendix D.5) further shows that the default threshold does not interfere with normal training, i.e., any stays within 1.3 LC WR of the default, and only an overly strict suppresses useful learning. We agree that this establishes drift control as an empirically reliable trigger rather than measuring FP and FN rates against ground-truth, and we'll state this in the Limitations.
Q2 (GPM gate). Within a response group, is a positive per-prompt constant, so if applied to the raw subspace scores it gets exactly cancelled by per-dimension normalization since scaling every by scales the group mean and standard deviation to and , so . Used instead as the aggregation weight after normalization, it changes LC WR by at most 0.28, for either better or worse:
| Setting | GPRL, uniform | GPRL, |
|---|---|---|
| Llama-3-8B, 2B GPM | 51.08 | 51.22 |
| Llama-3-8B, 8B GPM | 56.51 | 56.23 |
| Gemma-2-9B, 8B GPM | 74.31 | 74.08 |
This is consistent with our computed statistics that learned gate is close to uniform across subspaces (mean : 0.993, 1.012, 0.998 and coefficient of variation across prompts at most 0.068).
On robustness to the decomposition, writing each embedding as with gives , so the uniform aggregate is . A complex unitary preserves this inner product, unit norm, and block-diagonal operator , but its off-diagonal entries mix blocks and generally change the individual , and hence GPRL's normalized advantages. The aggregate score is thus invariant under uniform weights (though not under non-uniform gate or general rotations in ). We retrained GPRL after applying a random unitary mixing and obtain 55.87 LC WR against 56.51, a difference well within AlpacaEval std. error, so GPRL is empirically robust to this change of GPM basis.
Q3 (Matching supervision and compute). All methods start from the same base policies and use UltraFeedback prompts with the same 512-token generation cap. For Llama-3, we use the released DPO/SimPO and SPPO/GPO checkpoints from prior work rather than retraining them, since they were trained on UltraFeedback under the same setup. Following their published setups, each covers roughly 60k prompts, i.e., DPO and SimPO train for one epoch on pre-generated preference pairs, while SPPO and GPO train for one epoch per iteration using disjoint thirds of the prompts. We roughly estimate their costs at about 25 H100 GPU-hours for DPO/SimPO and 100–110 H100 GPU-hours for SPPO/GPO.
In contrast, GRPO+BT and GPRL train with online rollouts at each step, and require a total of 372 H100 GPU-hours and 746.1M training tokens each. We acknowledge that these online methods are more computationally expensive and will make this distinction clear in the revision. However, GRPO+BT and GPRL share an identical budget, and GPRL still outperforms GRPO+BT by 14.59 LC WR on AlpacaEval. So additional compute alone does not explain GPRL's results, but rather how the reward is structured does.
Thank you for the detailed response and the new experiments. The new experiments and clarifications fully resolve W2.1, W3, and Q2, and largely address W2.2 of mine. I also appreciate the updated scope in the Limitations. One small point remains.
The compute claim in Section 4. You mention that GRPO+BT and GPRL take 372 H100 GPU-hours compared to 100–110 for SPPO and GPO, which contradicts Section 4's current claim that three epochs "matches the three-iteration protocol of SPPO and GPO in compute" by about 3x. I'm not very sure which one is correct. So some clarification on this point would be helpful.
Please also ensure these expanded experiments and insights you made during rebuttal are integrated into the revised manuscript to improve its overall depth.
Overall, this paper presents a compelling perspective and an effective alignment framework. The new results thoroughly demonstrate its efficacy and set a strong baseline for future benchmarks. I have increased my rating and remain open to further discussion.
Thank you for raising your rating and the follow-up.
Compute claim. The 372 H100 GPU-hours figure is the correct one. The wording currently in Section 4 of the manuscript is imprecise, as it refers to matching the number of training rounds (three iterations for SPPO/GPO against three epochs for GRPO+BT and GPRL) rather than equal GPU-hours. In wall-clock time terms, our experiments with online methods cost roughly 3.5 times more than SPPO/GPO, since they generate fresh rollouts at every step. In the revision, we'll replace "matches ... in compute" with "matches the number of training rounds", report the estimated budget of each method explicitly, and note that the compute-matched comparison is between GRPO+BT and GPRL, which share identical GPU-hours yet differ by 14.59 LC WR points.
Revision. Certainly! We'll integrate all experiments and analyses from the discussion into the revised manuscript.
Thank you again for the constructive discussion.
Response to Reviewer UeZz (1/2)
Thank you for the constructive review and the concrete suggestions, which we have followed. Because new results and table take some space, our reply spans two comments.
W2.1 (Generation settings and truncation). Lengths reported in Table 1 follow AlpacaEval's standard length measure, which counts characters rather than tokens, so they aren't directly comparable to the 512-token training cap. We'll correct this unit wherever the manuscript refers to these lengths as tokens. GPRL's 1600 characters correspond to roughly 350 to 400 tokens, which is well under 512 token limit.
At eval., generation uses AlpacaEval's default settings (
max_tokens4096,temperature0.9), so evaluation outputs are not truncated. During training, 512-token cap clips 17.6% of rollouts on average, which decreases from 26.5% in first 10% of training steps to 14.7% in the last 10% steps. To ensure that this token cap is not the main reason for GPRL's improvements, we repeat GPRL and GRPO+BT with a 1024-token cap:With twice the token cap, avg. length increases but LC WR stays essentially unchanged. Only raw WR rises with length, as expected from its length bias [Dubois et al.].
W1 (Base-policy results and length relative to it). We will add the following base policy evals to all main tables:
Relative to initial policy in these results, GPRL shortens Llama-3 responses while raw WR, which is known to favor longer responses, rises from 23.34 to 48.33. On the other hand, Gemma-2's base responses are already short, and each method increases the length, and GPRL does so the least, i.e., 1791, against 1845 to 2452 for all baselines.
W2.2 (Reliability and judge bias). We have repeated a comparison with three training seeds:
Every GPRL seed still outperforms every GRPO+BT seed. We do want to highlight that reporting a single run per config is standard in this line of work (e.g., DPO, SimPO, SPPO, and GPO), since each evaluation repeatedly calls gpt-4-turbo API and repeating all benchmarks across seeds becomes very costly.
As a separate, independent eval., we re-scored the same AlpacaEval outputs with a Llama-3.1-70B-Instruct as a judge:
So, with a distinct judge, absolute scores differ a bit, but GPRL remains the strongest method.
Q1 (What the dimensions represent). Skywork-Reward comparisons carry no specific attribute annotations, so we do not assign semantic labels to these subspaces, and HelpSteer2 analysis of Appendix D.9 does not transfer to them. On held-out responses, the three subspace scores correlate pairwise at 0.16, −0.09, and 0.23, so they carry distinct signals rather than copies of one.