acceptodds
Under review as a conference paper at ICLR 2027

General Preference Reinforcement Learning

Abstract

Online reinforcement learning (RL) with verifiable rewards drives emergent reasoning on math and code, but open-ended tasks have no programmatic verifier. A learned scalar reward model can be applied to these tasks in place of a verifier, yet a scalar is the wrong shape for the job. Quality is multi-dimensional, and any scalar score is an incomplete proxy that lets extended online RL collapse onto whichever axis the score is most sensitive to. Recent multi-reward methods counter this collapse by normalizing each reward separately, but they require every reward to be specified in advance. We instead learn latent quality dimensions from ordinary binary preference labels with the General Preference Model (GPM), and propose General Preference Reinforcement Learning (GPRL), which keeps these dimensions separate through the policy update. GPRL computes group-relative advantages per dimension, normalizes each on its own scale, and aggregates them with per-dimension weights. We characterize when this aggregate rejects responses that improve one dimension at the expense of the others, even when a scalar reward would favor them. The same structure yields a drift monitor that detects such exploitation during training and corrects it online. GPRL reaches length-controlled win rates of from and from on AlpacaEval 2.0 with the shortest responses of any method; it also outperforms offline, iterative, and scalar-reward online baselines on Arena-Hard, MT-Bench, and WildBench, and does not degrade under extended training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
preference optimization, LLM alignment, reward hacking, multi-dimensional preference modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UeZzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GPRL for online RL on open-ended tasks. It uses a General Preference Model trained on binary preference labels to obtain several latent preference signals. Instead of immediately combining them into a scalar reward, GPRL computes and normalizes group-relative advantages separately for each signal before aggregation. A drift controller monitors changes in their variance shares and adjusts the dimension weights and KL coefficient. Experiments on Llama-3-8B-Instruct and Gemma-2-9B-it report improvements over offline, iterative, and scalar-reward online baselines on four benchmarks, together with shorter responses and better stability under extended training.

Critical strengths

S1. The paper explores a useful way to use ordinary binary preference data for online RL. Its main idea is to retain the GPM's separate preference signals through advantage estimation, rather than collapsing them into one reward before normalization. This gives a concrete way to prevent a high-variance signal from overwhelming the others, without requiring manually defined reward dimensions. The resulting update also fits into the existing GRPO training framework.

S2. The experiments cover two main base policies and several types of baselines, with ablations on the number of subspaces, normalization, the controller, group size, and reward-model corpus. The LC WR gains on AlpacaEval are substantial: 56.51 versus 41.92 for GRPO+BT on Llama, and 74.31 versus 61.75 on Gemma. The extended-training comparison also provides useful evidence that the controller helps preserve performance later in training.

Critical weaknesses

W1. The main tables lack complete base-policy results.

Tables 1, 2, and 3 omit the scores of base policies, making it difficult to assess what each method gains or loses relative to its base model. In particular, merely being shorter than other methods may not show how GPRL changed or controlled length relative to the starting policy. Reporting base policy scores under identical evaluation settings would make these trajectory comparisons clear.

W2. Response length and quality need more complete evaluation details.

W2.1. Generation Settings & Truncation The generation settings require clarification. Table 11 specifies a maximum rollout length of 512, whereas Table 1 reports average GPRL response lengths of 1600 and 1791 tokens. While this discrepancy may stem from differing training and evaluation settings, key details such as the evaluation cutoff and training truncation rate are omitted. Consequently, it is unclear whether the shorter outputs reflect the intended length-control mechanism or training truncations. The authors should specify these parameters and test whether the length and quality gains hold under different rollout limits.

W2.2. Evaluation Reliability & Judge Bias While AlpacaEval's LC WR is promising, all four benchmarks rely solely on automatic judges without reporting confidence intervals. Given the small margins on benchmarks like Arena-Hard v2 (1.3 vs. 1.0 and 2.7 vs. 2.4), the stability of these gains remains unproven. Providing error bars, consistent decoding setups, and human or independent evaluations would better confirm that GPRL genuinely improves output quality while reducing verbosity.

W3. Score-variance share changes lack validation as a reward-hacking detector.

The controller triggers on shifts in score-variance allocation across dimensions. However, such shifts can signal either superficial exploitation or genuine skill acquisition. This is an ambiguity underscored by Appendix D.5, where a strict threshold (0.05) suppressed legitimate learning and dropped LC WR by 4.07 points. What empirical evidence confirms that the default controller specifically targets harmful exploitation rather than healthy specialization?

Furthermore, the monitor measures relative variance shares rather than absolute quality; uniform variance scaling across dimensions would register zero drift, leaving degradation undetected. While Figure 8 provides an external check on aggregate GPM rankings, it does not prove that downweighted dimensions are the actual sources of reward hacking.

To validate the detector, the paper may need concrete examples and controlled tests showing whether the monitor reacts to artificially lengthened responses and remains stable during real content improvements. Without reporting false positive and false negative rates against ground-truth quality, the results may prove the controller's heuristic utility rather than its validity as a reward-hacking detector.

Questions for authors

Q1. What do the learned dimensions represent? Figure 7 gives a useful interpretation for a two-subspace GPM trained on HelpSteer2, but the main experiments use a three-subspace GPM trained on Skywork-Reward. Is there corresponding evidence about what those main-experiment dimensions capture?

Q2. Why is the GPM's learned gate not used? Table 11 states that the GPM gate is unused and the base weights are uniform. Why discard the prompt-dependent weights learned during preference modeling, and how does retaining them affect the results? Since different decompositions can preserve the aggregate preference score while changing the separately normalized advantages, evidence of robustness would also help assess whether the policy update relies on a stable decomposition.

Q3. How are supervision and computation matched across the methods? The paper describes three iterations of SPPO/GPO and three epochs of GRPO/GPRL as comparable in compute, but these counts do not establish equal budgets. Could you report the training tokens and GPU hours for each method? Please also specify the policy-training data and preference annotations used for DPO/SimPO. Training reward models on the same corpus is a useful control, but it does not fully specify the data or optimization budget seen by each policy.

I would be happy to revise my assessment if the authors can clarify these details and provide the relevant evidence. Looking forward to further discussion.

4 comments
AuthorsOpenReview ↗

Response to Reviewer UeZz (1/2)

Thank you for the constructive review and the concrete suggestions, which we have followed. Because new results and table take some space, our reply spans two comments.

W2.1 (Generation settings and truncation). Lengths reported in Table 1 follow AlpacaEval's standard length measure, which counts characters rather than tokens, so they aren't directly comparable to the 512-token training cap. We'll correct this unit wherever the manuscript refers to these lengths as tokens. GPRL's 1600 characters correspond to roughly 350 to 400 tokens, which is well under 512 token limit.

At eval., generation uses AlpacaEval's default settings (max_tokens 4096, temperature 0.9), so evaluation outputs are not truncated. During training, 512-token cap clips 17.6% of rollouts on average, which decreases from 26.5% in first 10% of training steps to 14.7% in the last 10% steps. To ensure that this token cap is not the main reason for GPRL's improvements, we repeat GPRL and GRPO+BT with a 1024-token cap:

Method (Llama-3-8B, 8B RM)Max lengthLC WRWRAvg. Len
GRPO+BT51241.9240.511893
GRPO+BT102441.1444.672376
GPRL51256.5148.331600
GPRL102456.8950.741890

With twice the token cap, avg. length increases but LC WR stays essentially unchanged. Only raw WR rises with length, as expected from its length bias [Dubois et al.].

W1 (Base-policy results and length relative to it). We will add the following base policy evals to all main tables:

Base policyLC WRWRAvg. LenAH2MT-BenchWB-ScoreWB-Reward
Llama-3-8B-Instruct23.0723.3419820.57.4030.174.89
Gemma-2-9B-it51.0845.3716161.938.6148.4324.67

Relative to initial policy in these results, GPRL shortens Llama-3 responses while raw WR, which is known to favor longer responses, rises from 23.34 to 48.33. On the other hand, Gemma-2's base responses are already short, and each method increases the length, and GPRL does so the least, i.e., 1791, against 1845 to 2452 for all baselines.

W2.2 (Reliability and judge bias). We have repeated a comparison with three training seeds:

Method (Llama-3-8B, 8B RM)LC WRAvg. LenAlpacaEval SE
GRPO+BT41.90 ±\pm 0.481891 ±\pm 341.73
GPRL56.44 ±\pm 0.391608 ±\pm 311.64

Every GPRL seed still outperforms every GRPO+BT seed. We do want to highlight that reporting a single run per config is standard in this line of work (e.g., DPO, SimPO, SPPO, and GPO), since each evaluation repeatedly calls gpt-4-turbo API and repeating all benchmarks across seeds becomes very costly.

As a separate, independent eval., we re-scored the same AlpacaEval outputs with a Llama-3.1-70B-Instruct as a judge:

Method (Llama-3-8B)RMgpt-4-turbo LC WRLlama-3.1-70B-Instruct LC WR
DPO41.2637.86
SimPO42.8341.16
SPPO8B GPM40.3740.72
GPO8B GPM41.7335.41
GRPO+BT8B BT41.9238.02
GPRL8B GPM56.5153.18

So, with a distinct judge, absolute scores differ a bit, but GPRL remains the strongest method.

Q1 (What the dimensions represent). Skywork-Reward comparisons carry no specific attribute annotations, so we do not assign semantic labels to these subspaces, and HelpSteer2 analysis of Appendix D.9 does not transfer to them. On held-out responses, the three subspace scores correlate pairwise at 0.16, −0.09, and 0.23, so they carry distinct signals rather than copies of one.

AuthorsOpenReview ↗

Response to Reviewer UeZz (2/2)

W3 (Validating drift). We address the raised concerns in turn.

  1. Uniform variance scaling across dimensions does not alter policy update at all. Per-dimension normalization divides each subspace by its own standard deviation, so a common scale factor cancels in every A^l\hat{A}_l, and there's essentially nothing for the controller to correct. Degradation of the GPM's absolute accuracy is more of a calibration question, which Appendix E.1 addresses.

  2. We record the share of the dominant subspace αl∗\alpha_{l^\ast} in a controller-off run:

Epoch (8B GPM, controller off)αl∗\alpha_{l^\ast}D(t)\mathcal{D}(t)LC WRAvg. Len
10.420.0249.421718
20.490.0654.921693
30.590.1152.701742
50.760.2747.712187

and observe that LC WR peaks at epoch 2 and then falls by 7.21 points as one subspace's share grows from 0.49 to 0.76, so the single-axis concentration the controller targets coincides with loss of LC WR, i.e., quality.

  1. Our existing runs already show both behaviors that the suggested tests focus on. D(t)\mathcal{D}(t) rises with natural length inflation, as in the table above, and during improvement in the first epoch, D(t)\mathcal{D}(t) stays near zero and the controller changes LC WR by at most 0.23 (Appendix D.3). A sweep of the τ\tau param. (in Appendix D.5) further shows that the default threshold does not interfere with normal training, i.e., any τ∈[0.1,0.4]\tau \in [0.1, 0.4] stays within 1.3 LC WR of the default, and only an overly strict τ=0.05\tau = 0.05 suppresses useful learning. We agree that this establishes drift control as an empirically reliable trigger rather than measuring FP and FN rates against ground-truth, and we'll state this in the Limitations.

Q2 (GPM gate). Within a response group, λl(x)\lambda_l(x) is a positive per-prompt constant, so if applied to the raw subspace scores it gets exactly cancelled by per-dimension normalization since scaling every s^l(i)\hat{s}_l^{(i)} by λl\lambda_l scales the group mean and standard deviation to λlμl\lambda_l\mu_l and λlσl\lambda_l\sigma_l, so (λls^l(i)−λlμl)/(λlσl)=(s^l(i)−μl)/σl=A^l(i)(\lambda_l \hat{s}_l^{(i)} - \lambda_l\mu_l)/(\lambda_l\sigma_l) = (\hat{s}_l^{(i)} - \mu_l)/\sigma_l = \hat{A}_l^{(i)}. Used instead as the aggregation weight after normalization, it changes LC WR by at most 0.28, for either better or worse:

SettingGPRL, uniform wlw_lGPRL, wl=λl(x)w_l = \lambda_l(x)
Llama-3-8B, 2B GPM51.0851.22
Llama-3-8B, 8B GPM56.5156.23
Gemma-2-9B, 8B GPM74.3174.08

This is consistent with our computed statistics that learned gate is close to uniform across subspaces (mean λl(x)\lambda_l(x): 0.993, 1.012, 0.998 and coefficient of variation across prompts at most 0.068).

On robustness to the decomposition, writing each embedding as z∈Ck\mathbf{z} \in \mathbb{C}^k with zl=v(2l−1)+iv(2l)z_l = v^{(2l-1)} + i v^{(2l)} gives sl(yi,yj)=Im(zl(i)zl(j)‾)s_l(y_i, y_j) = \mathrm{Im}(z_l^{(i)} \overline{z_l^{(j)}}), so the uniform aggregate is Im⟨z(i),z(j)⟩\mathrm{Im}\langle \mathbf{z}^{(i)}, \mathbf{z}^{(j)} \rangle. A complex unitary U∈U(k)U \in U(k) preserves this inner product, unit norm, and block-diagonal operator R≻\mathbf{R}^{\succ}, but its off-diagonal entries mix blocks and generally change the individual sls_l, and hence GPRL's normalized advantages. The aggregate score is thus invariant under uniform weights (though not under non-uniform gate λl(x)\lambda_l(x) or general rotations in O(2k)O(2k)). We retrained GPRL after applying a random unitary mixing z↦Uz\mathbf{z} \mapsto U\mathbf{z} and obtain 55.87 LC WR against 56.51, a difference well within AlpacaEval std. error, so GPRL is empirically robust to this change of GPM basis.

Q3 (Matching supervision and compute). All methods start from the same base policies and use UltraFeedback prompts with the same 512-token generation cap. For Llama-3, we use the released DPO/SimPO and SPPO/GPO checkpoints from prior work rather than retraining them, since they were trained on UltraFeedback under the same setup. Following their published setups, each covers roughly 60k prompts, i.e., DPO and SimPO train for one epoch on pre-generated preference pairs, while SPPO and GPO train for one epoch per iteration using disjoint thirds of the prompts. We roughly estimate their costs at about 25 H100 GPU-hours for DPO/SimPO and 100–110 H100 GPU-hours for SPPO/GPO.

In contrast, GRPO+BT and GPRL train with online rollouts at each step, and require a total of 372 H100 GPU-hours and 746.1M training tokens each. We acknowledge that these online methods are more computationally expensive and will make this distinction clear in the revision. However, GRPO+BT and GPRL share an identical budget, and GPRL still outperforms GRPO+BT by 14.59 LC WR on AlpacaEval. So additional compute alone does not explain GPRL's results, but rather how the reward is structured does.

Thank you for the detailed response and the new experiments. The new experiments and clarifications fully resolve W2.1, W3, and Q2, and largely address W2.2 of mine. I also appreciate the updated scope in the Limitations. One small point remains.

The compute claim in Section 4. You mention that GRPO+BT and GPRL take 372 H100 GPU-hours compared to 100–110 for SPPO and GPO, which contradicts Section 4's current claim that three epochs "matches the three-iteration protocol of SPPO and GPO in compute" by about 3x. I'm not very sure which one is correct. So some clarification on this point would be helpful.

Please also ensure these expanded experiments and insights you made during rebuttal are integrated into the revised manuscript to improve its overall depth.

Overall, this paper presents a compelling perspective and an effective alignment framework. The new results thoroughly demonstrate its efficacy and set a strong baseline for future benchmarks. I have increased my rating and remain open to further discussion.

Thank you for raising your rating and the follow-up.

Compute claim. The 372 H100 GPU-hours figure is the correct one. The wording currently in Section 4 of the manuscript is imprecise, as it refers to matching the number of training rounds (three iterations for SPPO/GPO against three epochs for GRPO+BT and GPRL) rather than equal GPU-hours. In wall-clock time terms, our experiments with online methods cost roughly 3.5 times more than SPPO/GPO, since they generate fresh rollouts at every step. In the revision, we'll replace "matches ... in compute" with "matches the number of training rounds", report the estimated budget of each method explicitly, and note that the compute-matched comparison is between GRPO+BT and GPRL, which share identical GPU-hours yet differ by 14.59 LC WR points.

Revision. Certainly! We'll integrate all experiments and analyses from the discussion into the revised manuscript.

Thank you again for the constructive discussion.

Reviewer_Bax2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work aims to improve preference alignment in language models. It builds on GRPO by replacing scalar rewards with preference signals from GPM. To address reward hacking, the proposed method normalizes within-group scores separately for each latent GPM subspace to obtain per-dimension advantages, which are then aggregated through a weighted sum. It also monitors drift in the per-dimension variance shares to adjust the aggregation weights and the KL regularization coefficient. The authors report performance improvements across multiple alignment benchmarks, as well as improved stability under extended training with the variance-share drift controller.

Critical strengths

1.The paper proposes a connection between variance-share drift across the components of a proxy reward and reward hacking. Its threshold ablation in Appendix D.5 shows that larger terminal drift is generally associated with longer responses. The authors further introduce a controller intended to mitigate such exploitation and report improvements in length-controlled win rates.

2.The paper demonstrates successful policy training by integrating GPM preference signals into a GRPO-style optimization framework, providing empirical evidence for the feasibility of this combination.

Critical weaknesses

1.The connection between variance-share drift across latent preference subspaces and reward hacking remains insufficiently established, despite its central role in motivating the method. In addition, Proposition 2 and Figure 9 appear mathematically inconsistent, as detailed in the Questions for Authors section. These issues weaken the justification for the proposed mechanism.

2.At the level of advantage computation, the method can be expressed as a replacement for the GPM gate. However, the paper lacks a direct comparison with the original gate under the same policy optimization procedure, leaving the contribution of the proposed replacement insufficiently isolated.

Questions for authors

1. Baselines and the interpretation of changes in response length

1.1 I may have overlooked relevant information, but apart from the initial LC WR reported in Appendix D.7, I could not find complete results for the models before further training under the same evaluation settings as the main experiments, particularly their average response lengths. The Llama-3 SPPO/GPO results in Table 1 match the corresponding entries in Table 3 of the original GPM paper exactly, which reports an initial average response length of 1959. Does this initial response-length baseline also apply to this work?

1.2 If so, why does the average response length of GPRL with the 8B GPM already decrease to 1742 when the controller is disabled in Table 6? Together with Table 5, this seems to suggest that per-dimension normalization plays an important role in the change in response length. What evidence distinguishes this substantial decrease as benign from the length increases interpreted as reward hacking?

1.3 If the initial models or evaluation settings differ, please clarify the specific differences and the comparability of the results directly compared in Table 1.

2. Theoretical justification and empirical support for the motivation

2.1 The paper emphasizes that the aggregate advantage can “reject responses that improve one dimension at the expense of the others,” even when a scalar reward would favor those responses. However, the proposition concerns differences in per-dimension advantages between responses, whereas the controller subsequently monitors changes in the within-group variance shares of the scores before normalization. What theoretical or empirical connection links these two quantities? Are there results verifying that the described abnormal growth along a single axis actually occurs and corresponds to a decline in externally evaluated quality?

2.2 The latent dimensions in the main experiments have not been validated as having stable semantic meanings. Consequently, improving one dimension at the expense of the others need not correspond to any deterioration in actual preference satisfaction or performance. Why should this be regarded as reward hacking, and why is the “balanced policy” described in Section 3.2 better? Beyond establishing an ordering of weighted sums, how does Proposition 2 support these claims, and why would per-dimension normalization make its condition more likely to hold?

2.3 There appears to be a mathematical inconsistency between Proposition 2 and Appendix E.2/Figure 9. According to Eq. (5), Eq. (7) is simply a rearrangement of

A^π†−A^π∗<0.\hat A^{\pi^\dagger}-\hat A^{\pi^*}<0.

Therefore, for the same response pairs and weights, satisfying Eq. (7) is equivalent to the balanced response having a higher aggregate advantage. However, Figure 9 reports respective rates of 83.1% and 86.1% at epoch 1, with discrepancies also present at later epochs. The statement that the condition is sufficient but not necessary cannot explain this, could the authors clarify how the two rates were computed and explain this discrepancy?

2.4 Existing work dynamically adjusts multi-objective weights based on reward variance (DVAO) or relative dispersion (SAW), with SAW interpreting changes in these statistics as differences in learning progress across objectives. How does the proposed method distinguish variance-share changes caused by normal learning from those associated with reward hacking? Why should the initial variance profile serve as the reference throughout training, rather than allowing it to evolve with learning progress?

3. Novelty and completeness of the experimental comparisons

3.1 At the level of advantage computation, GPRL’s normalization and dynamic weighting can be equivalently expressed as a replacement for the original GPM gate, excluding adaptive KL regularization. Since the pairwise scores are antisymmetric, the within-group mean of each dimension satisfies μl=0\mu_l=0. For a sampled group YY, define

λleff(x,Y,t)=wlml(t)σl(x,Y)+ϵ,\lambda_l^{\mathrm{eff}}(x,Y,t) =\frac{w_lm_l(t)}{\sigma_l(x,Y)+\epsilon},

setting it to zero when σl=0\sigma_l=0. Then

A^i=1G−1∑j≠i∑lλleff(x,Y,t) sl(yi,yj∣x).\hat A_i =\frac{1}{G-1}\sum_{j\ne i}\sum_l \lambda_l^{\mathrm{eff}}(x,Y,t)\,s_l(y_i,y_j\mid x).

This is exactly the group average of GPM pairwise scores aggregated using the replacement gate. Unlike the original prompt-dependent gate, this effective gate also depends on the sampled group and training history. Given this relationship, what is the additional contribution relative to the existing gating mechanism?

3.2 The original GPM paper already reports the effectiveness of gating, whereas the existing GPO/SPPO comparisons also change the policy optimization algorithm. Could the authors include a GPM baseline that retains the original gate under the same policy optimization procedure, to isolate the contributions of normalization and the controller? Furthermore, how does performance compare with the proposed method when the gate is allowed to be fine-tuned?

The work provides insights into using variance drift across subdimensions to monitor and mitigate reward hacking. Addressing the major issues raised above and providing a matched GRPO+GPM comparison that retains the original gate would help isolate the sources of improvement and could lead me to raise my rating.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Bax2 (1/2)

We highly appreciate your thorough and technical review. Since many questions were raised, we address them in two parts, starting with the inconsistency in Prop. 2 and the GRPO+GPM comparison you requested. We hope the results below address your concerns, and we are happy to provide anything further during the discussion period.

Q2.3 (Proposition 2 and Fig. 9) For a pair satisfying the antecedent, let Δl=A^lπ∗−A^lπ†\Delta _ l = \hat{A} _ l^{\pi^{\ast}} - \hat{A} _ l^{\pi^{\dagger}}. By Eq. (5), this aggregate gap under weights ww is A^wπ†−A^wπ∗=−∑lwlΔl=Hw−Dw\hat{A}^{\pi^{\dagger}} _ w - \hat{A}^{\pi^{\ast}} _ w = -\sum _ {l} w _ l \Delta _ l = H _ w - D _ w, where Hw=−wl∗Δl∗H _ w = -w _ {l^{\ast}} \Delta _ {l^{\ast}} and Dw=∑l≠l∗wlΔlD _ w = \sum _ {l \neq l^{\ast}} w _ l \Delta _ l. Eq. (7) is then Dw>HwD _ w > H _ w, so it holds iff A^wπ†<A^wπ∗\hat{A}^{\pi^{\dagger}} _ w < \hat{A}^{\pi^{\ast}} _ w, and for the exact same pairs and the same ww the two rates in Fig. 9 would coincide. They differed because they were evaluated under different aggregation weights, ww for the condition and w′≠ww' \neq w for the ranking, which was left out of corresponding appendix. Recomputing both columns on the same pairs with the weights used in training makes them identical, as follows:

CheckpointAntecedent covrageCondition satisfied (= balanced ranked higher)
Epoch 118.483.1
Epoch 321.779.4
Epoch 522.581.2

W2, Q3.1, Q3.2 (Relation to GPM gate). Since μl=0\mu_l = 0 by antisymmetry, GPRL's advantage is a group average of GPM scores under the effective gate λleff=wlml(t)/(σl(x,Y)+ϵ)\lambda_l^{\mathrm{eff}} = w_l m_l(t)/(\sigma_l(x,Y)+\epsilon). So this reformulation clarifies our contribution in the below two ways.

  1. Any per-prompt positive gate λl(x)\lambda_l(x) applied to the raw subspace scores is cancelled by per-dimension normalization, because (λls^l−λlμl)/(λlσl)=(s^l−μl)/σl(\lambda_l \hat{s}_l - \lambda_l\mu_l)/(\lambda_l\sigma_l) = (\hat{s}_l-\mu_l)/\sigma_l. The rescaling the gate learns at reward-model training time is performed therefore by 1/σl(x,Y)1/\sigma_l(x,Y) instead. Unlike original GPM gate, this rescaling adapts to the responses actually sampled from current policy, and the controller adjusts it further during training.
  2. GPM's gate adds little to GPRL. In fact, using this gate as the aggregation weight after normalization changes LC. WR of AlpacaEval 2.0 by at most 0.280.28 points, either for better or worse:
SettingGPRL, uniform wlw_lGPRL, wl=λl(x)w_l = \lambda_l(x)
Llama-3-8B, 2B GPM51.0851.22
Llama-3-8B, 8B GPM56.5156.23
Gemma-2-9B, 8B GPM74.3174.08

This is consistent with our computed statistics that learned gate is close to uniform across subspaces (mean λl(x)\lambda_l(x): 0.993, 1.012, 0.998 and coefficient of variation across prompts at most 0.068). Also, Table 4 of [Zhang et al.] shows that GPM's learned gate changes RewardBench accuracy by around one point and again, not consistently for the better.

Comparison with GRPO+GPM. As asked for, to isolate per-dimension normalization and controller's effect, we run GRPO with GPM as a scalar reward, using the same optimization setup as GPRL:

Method (Llama-3-8B, 8B GPM, 3 ep.)WeightingNorm.ControllerLC WRWR
GRPO + GPMlearned gatescalaroff48.1945.83
GRPO + GPMuniformscalaroff47.9645.64
GRPO + GPMlearned gatescalaron52.1747.69
GPRLuniformper-dimensionon56.5148.33

Even under scalar normalization, learned gate is not that far from uniform weighting, i.e., 48.19 vs. 47.96, so GPM's gate simply does not account for GPRL's improvement. Adding controller to the gated scalar baseline gives +3.98 LC WR, and replacing scalar norm. with per-dimension norm. gives us +4.34, so these two components contribute comparably and together account for overall gain between GRPO+GPM and GPRL. We note that this controller-on run also closely matches our global-normalization ablation in Appendix D.2, where LC WR is 52.34. Also, we retain a frozne GPM rather than fine-tuning the gate, because our controller compares against the initial variance profile α(0)\alpha^{(0)}, and updating GPM weights during training would shift that reference.

AuthorsOpenReview ↗

Response to Reviewer Bax2 (2/2)

Q1.1–Q1.3 (Baselines and response length). Llama-3 SPPO/GPO evals. are indeed from the GPM paper, and Llama-3 DPO/SimPO likewise come from prior work (SimPO). These use same UltraFeedback prompts for preference training, same 512-token generation cap, and AlpacaEval 2.0 benchmark setting. All Gemma-2 rows and all online-RL methods (GRPO+BT, GPRL) are our own runs. Re-evaluating the base model ourselves gives an average length of 1982 characters, versus 1959 in the GPM paper, which confirms the evaluation settings match. To remove any doubt, we re-evaluate the released checkpoints of these methods with our setup:

Method (Llama-3-8B)RMLC WRWRAvg. Len
DPO41.2639.731907
SimPO42.8339.141774
SPPO8B BT43.1841.691987
SPPO8B GPM40.3742.162421
GPO8B BT39.4238.631884
GPO8B GPM41.7344.283018

We'll state our sources in a footnote in the revision/report these re-evaluated numbers, and add base policy results as well:

Base policyLC WRWRAvg. LenAH2MT-BenchWB-ScoreWB-Reward
Llama-3-8B-Instruct23.0723.3419820.57.4030.174.89
Gemma-2-9B-it51.0845.3716161.938.6148.4324.67

Also, it's a correct observation that per-dimension normalization, and not the controller, drives most of the length reduction. However, raw WR is not length-controlled and is known to favor longer responses [Dubois et al.], but it still rises from 23.34 to 48.33 on Llama-3 while length falls from 1982 to 1600. Shortening alone would lower raw WR, so the gain must come from response content. Additionally, to rule out 512-token training cap as an issue, we repeat GPRL and GRPO+BT with a 1024-token cap:

Method (Llama-3-8B, 8B RM)Max lengthLC WRWRAvg. Len
GRPO+BT102441.1444.672376
GPRL102456.8950.741890

With twice the max response length, avg. length rises for both methods, but LC WR stays essentially unchanged, i.e., 41.92 to 41.14 and 56.51 to 56.89. Only raw WR increases with length, which is expected from length bias, so 512-token cap does not drive our results, and the gap between the two methods holds at either cap.

Q2.1, Q2.2 A policy exploiting subspace l∗l^\ast produces many antecedent pairs of Prop. 2 favoring l∗l^\ast, which concentrates between-response variance on l∗l^\ast. This relates Prop. 2 to the training step profile taht the controller monitors. We record α(t)\alpha^{(t)} in a controller-off run and get:

Epoch (8B GPM, controller off)αl∗\alpha_{l^\ast}D(t)\mathcal{D}(t)LC WRAvg. Len
10.420.0249.421718
20.490.0654.921693
30.590.1152.701742
50.760.2747.712187

LC WR peaks at epoch 2 and then decreases as training progresses as one subspace's share grows from 0.49 to 0.76, so the single-axis concentration the controller targets coincides with quality loss.

We purposefully do not assign semantic labels to subspaces for the main experiments because Skywork-Reward dataset carry no attribute annotations, and HelpSteer2 analysis of Appendix D.9 does not translate to Skywork subspaces. "Reward hacking" is thus used in an operational sense, i.e., training continues against the GPM proxy while quality (LC WR in this case) falls [Gao et al.], as above. "Balanced" refers only to the aggregate ordering of Prop. 2, and is empirically better, since controller-on run holds 56.47 LC WR at epoch 5 against 47.71 shown in above table.

Prop. 2 further adds that the aggregate can rank a single-axis gain below the losses it causes elsewhere, which a scalar reward cannot express, and its weights are what the controller adjusts. Per-dimension norm. makes Eq. (7) more likely to hold since every A^l\hat{A}_l has unit variance, so the gain is on the same scale as each of the k−1k-1 losses summed in DD, whereas under global norm.m the gain grows with the raw spread of l∗l^\ast, which is what increases during hacking.

Q2.4 (DVAO and SAW). Both these methods up-weight objectives with either higher reward variance (as in DVAO) or coefficient of variation (as in SAW) and treat it as a stronger learning signal. GPRL instead does the opposite, i.e., it down-weights a dimension whose variance share grows past a threshold from α(0)\alpha^{(0)} and treats this growth as a sign of exploitation. We use α(0)\alpha^{(0)} as the anchor instead of a moving reference because a reference that follows the profile would also follow single-axis drift and never trigger, and a frozen GPM keeps this anchor on a fixed scale. Another difference is that DVAO and SAW both reweight rewards that are specified in advance, whereas GPRL's dimensions are learned from binary preferences. In our revision, we'll cite and discuss both.

Reviewer_Bax2OpenReview ↗

Response to Authors

Thank you for the detailed additional results. They address several specific concerns I raised, and the new comparisons make the individual contributions of per-dimension normalization and the controller much clearer.

1. Figure 9 and the baselines. The explanations and additional results adequately address these two issues concerning the completeness of the paper. It remains unclear why the two quantities in the original Figure 9 were evaluated using different aggregation weights, but I do not consider further clarification of this point necessary for my assessment, provided that the figure and its accompanying explanation are corrected in the revision.

Please also check and explain the numerical differences between the paper and the response. For example, Appendix D.7 reports a base-policy LC WR of approximately 22.6 for Llama-3, consistent with the gain reported in Table 8, whereas the response reports 23.07. This difference may reflect generation or judge randomness, or changes in evaluation settings. Please clarify which results were re-evaluated and ensure that the revised paper consistently reports the corresponding configurations and values.

2. Empirical results and the explanation of the gains. I find the comparisons with the GRPO+GPM baseline sufficient to support the effectiveness of per-dimension normalization and the controller, and the reported performance is strong. An aspect that could further strengthen the paper is a clearer explanation of why these components work. Since we agree that per-dimension normalization accounts for most of the reduction in response length, could you connect this observation to the mechanism underlying its performance gains? In particular, an explanation of how normalization changes the optimization incentives, and how the controller complements this effect over training, would be useful.

Overall, this work offers a novel perspective and an effective approach to preference alignment. The expanded experimental evidence supports its effectiveness and establishes a strong baseline for future comparisons. Although the connection between the proposed mechanisms and the observed gains remains incompletely understood. I have raised my rating and would be happy to continue the discussion.

Thank you for raising your rating and for the constructive follow-up.

Numerical differences. To clarify sources, 22.6 in Appendix D.7 is the published base-policy score, while 23.07 is from our own evaluation pipeline, with the small gap due to sampling at temperature 0.9 and judge randomness under otherwise identical settings. In our responses, we re-evaluated base policies and released Llama-3 baseline checkpoints, and re-ran the controller-off run with α(t)\alpha^{(t)} recorded, which matches Table 7 to within one point. All other values come from our original runs. The revision will report our own evaluations throughout the manuscript along with corrected Figure 9 and its correspondng explanation.

Why these components work. Under scalar normalization, the advantage is driven by whichever subspace has the largest raw spread within a group, often the axis a policy can adjust cheaply, such as length. Per-dimension normalization gives every subspace the same unit-variance budget, so growing one axis earns at most that axis's share of the advantage while regressing on the others costs theirs (Proposition 2). Inflating length alone therefore no longer pays off, and optimization moves toward improving several axes at once, which accounts for both the shorter responses and higher LC WR, i.e., 48.19 for GRPO+GPM with scalar normalization against 56.51 for GPRL.

Normalization, however, acts only within each group at each step, and cannot prevent the policy's responses from gradually concentrating their variance on one axis across many steps. GPRL controller targets this slower drift via D(t)\mathcal{D}(t), which is why it changes LC WR by at most 0.23 in the first epoch but adds 3.67 by epoch 3 and keeps LC WR at 56.47 at epoch 5, against 47.71 without it. We'll add this explanation to Section 3 in the revision.

Reviewer_XBVdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GPRL, a preference alignment with reinforcement learning algorithm that leverages the latent subspaces of the general preference model (GPM). Inspired by the GPM that encodes human preferences into latent representation, GPRL simply redefines the advantage aggregation strategy to leverage the information in each subspace of GPM output. The authors demonstrate the strength of GPRL against offline preference learning algorithms and online RL algorithms across diverse generation benchmarks, showing the empirical gain of using GPRL for general preference learning.

Critical strengths
  1. GPRL extends GPM-based preference optimization to the online reinforcement learning regime. It uses GPM's latent representation to compute advantages separately for each subspace.

  2. The paper evaluates GPRL on a wide variety of preference alignment benchmarks, where models trained with GPRL generally outperform baselines, including offline and online preference learning methods.

Critical weaknesses
  1. Insufficient validation of complementary subspace signals: GPRL's proposed mechanism relies on GPM subspaces encoding sufficiently distinct and meaningful preference information. However, the reported training objective does not necessarily prevent redundancy among the subspace signals, and preliminary analysis on those aspects are not clearly shown. Per-subspace normalization and variance-share monitoring do not establish this complementarity, because neither measures cross-subspace correlations. GPRL's proposed resistance to single-axis exploitation relies on its subspaces providing complementary preference information, so preliminary validation of subspace-level signals in GPM would complement the theoretical motivation of the paper.

  2. Measuring Reward Hacking: Although mitigating reward hacking is a core motivation, the reported indicators do not clearly distinguish hacking from other causes of performance deterioration. Variance drift, response length, and the controller ablations provide indirect evidence, while the benchmark results primarily demonstrate improved performance and training stability. The connection between these improvements and reward-hacking mitigation therefore remains unclear. The experimental results mainly establish benchmark performance and training stability, leaving the connection between these improvements and reward-hacking mitigation insufficiently demonstrated. Defining and measuring the clear source of reward hacking that can happen in other baseline RLHF methods will help complement the claim about mitigating reward hacking.

Questions for authors
  1. Figure 4a shows the ablation study on the use of the controller. Did the performance deterioration in the "No controller" runs coincide with the concentration of the variance profile α\alpha in a particular subspace? If so, which subspace was involved, and what preference information did it capture in the GPM used for those runs?

  2. What is the intuition behind using per-subspace variance to define α\alpha? Does larger variance indicate a more informative preference signal, or is the intended interpretation specifically about changes in variance shares relative to initialization?

1 comment
AuthorsOpenReview ↗

Response to Reviewer XBVd

Thank you for the positive assessment and your questions. We address each concern below.

W1 (Complementarity of subspace signals). On held-out responses, the three subspace scores of the Skywork-trained GPM correlate pairwise at 0.16, −0.09, and 0.23, so they carry largely distinct signals rather than copies of one. Two existing ablations further show that this complementarity matters for the policy:

Method (Llama-3-8B, 8B RM)LC WR
GRPO+BT (single scalar)41.92
GPRL, k=1k = 1 (single subspace)44.21
Three orthogonality-regularized BT heads, same normalization and controller (Appendix D.8)50.84
GPRL, k=3k = 356.51

Adding subspaces raises LC WR by 12.30 points, and three heads with comparably low correlations (0.23, 0.28, 0.26) still trail GPRL by 5.67 points, so this improvement comes from distinct signals in GPM relational form than just dimensionality alone.

W2 and Q1 (Measuring reward hacking). We use reward hacking in an operational sense, i.e., training continues against the learned reward while externally judged (gpt-4-turbo for example) quality falls [Gao et al.]. To measure this, we record the share of the dominant subspace αl∗\alpha_{l^\ast} in a controller-off run:

Epoch (8B GPM, controller off)αl∗\alpha_{l^\ast}D(t)\mathcal{D}(t)LC WRAvg. Len
10.420.0249.421718
20.490.0654.921693
30.590.1152.701742
50.760.2747.712187

After peaking at epoch 2, LC WR falls by 7.21 points while one subspace's share grows from 0.49 to 0.76. Two other measurements point to hacking rather than general instability. Over the same epochs, mean response length increases by 494 characters, which is consistent with prior findings that RL optimization against learned reward models often achieves reward gains largely by increasing response length [Singhal et al.]. By epoch 5, frozen GPM also agrees with a held-out Llama-3.1-70B judge on only 63.9% of response pairs, against 69.7% with the controller on (Appendix E.1), so reward stops tracking quality just as the profile concentrates. With the controller on, LC WR instead holds at 56.47 at epoch 5.

As for which subspace was involved, Skywork-Reward comparisons don't carry any attribute annotations, so we can't assign it a semantic label, and HelpSteer2 analysis of Appendix D.9 does not transfer to these subspaces. Its growth alongside response length however does suggest a length- or verbosity-related signal, similar to HelpSteer2 GPM, where one subspace tracks verbosity and complexity.

Q2 (Intuition behind variance shares). The intended interpretation is the second one you make. Larger variance does not indicate a more informative signal; αl\alpha_l measures how much of the between-response spread within a group lies along subspace ll, i.e., where the policy's responses currently differ. Per-dim. normalization fixes every A^l\hat{A}_l at unit variance, so the raw scores before normalization are the only place where such concentration remains visible. In healthy training this profile keeps roughly its initial shape, whereas a policy exploiting one axis makes its responses differ mainly along that axis, so its share grows. Our controller therefore monitors the change D(t)=KL(α(t)∥α(0))\mathcal{D}(t) = \mathrm{KL}(\alpha^{(t)} \| \alpha^{(0)}) relative to initialization rather than absolute variance of any subspace.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion

We thank the reviewers and the AC for their time and constructive feedback throughout the discussion. Below we briefly summarize the main additions, all of which will be integrated into the revision.

  • Isolating effects of components. A matched GRPO+GPM baseline under the same optimization setup separates the effects of per-dimension normalization and the controller, and also shows that the GPM's learned gate does not account for the improvement.

  • Reward hacking and drift. A controller-off run with the variance profile recorded shows that the drop in LC WR coincides with variance concentrating on one subspace and with growing response length, while the three subspaces carry largely distinct signals.

  • Evaluation reliability. We added base-policy results, an independent second judge, a 1024-token rollout cap, and a re-eval. of the released baseline checkpoints, under all of which GPRL's gains hold. GPRL is also shown to be robust to a random unitary change of the GPM basis.

  • Corrections. The revision corrects Figure 9 and its explanation, labels AlpacaEval lengths correctly as characters rather than tokens, states compute budget of each method explicitly, and reports our own evaluations consistently throughout.

We appreciate the reviewers' engagement, which has substantially improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.