acceptodds
Under review as a conference paper at ICLR 2027

GSafePO: Geometric Optimizers for Reward-Safe Preference Learning

Abstract

Direct Preference Optimization (DPO) and related preference-learning methods optimize relative margins between chosen and rejected responses, but provide limited control over their absolute reward dynamics. This can lead to likelihood displacement, where chosen responses become less likely despite improved preference margins, and to reward over-optimization, where excessive absolute reward shifts drive unnecessary policy drift that can degrade generation quality even as the preference objective continues to improve. Recent methods address these effects largely through loss-level interventions. We introduce GSafePO (Geometrically Safe Preference Optimization), an optimizer framework that instead directly constrains the geometry of the realized parameter update. GSafePO projects the effective optimizer step onto a reward-safe set so that rewards for chosen responses do not decrease and rewards for rejected responses do not increase to first order, while deviating minimally from the unconstrained update. We characterize the low-dimensional geometry of such updates, derive conditions under which they remain aligned with descent on the original preference objective, and establish population-level guarantees for stochastic minibatch training. We instantiate the framework as two adaptive optimizers, GSafeRMSProp and GSafeAdam, extending the construction to adaptive preconditioned optimizers. Across multiple preference objectives and model configurations, GSafePO consistently converts undesirable common-mode reward motion into chosen-up/rejected-down dynamics while preserving competitive preference performance. Generation-quality evaluations further show that these corrected dynamics translate into significantly improved behavior in settings exhibiting likelihood displacement. These results establish update-geometry control as a modular mechanism for adding explicit reward safeguards to existing preference-learning objectives.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Direct Preference Optimization, Preference Learning, Likelihood Displacement, Reward Over-Optimization, Adaptive Optimizers, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kBe7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies undesirable chosen/rejected reward dynamics in preference optimization. Specifically, GSafePO projects the optimizer direction onto a set enforcing balanced chosen-up and rejected-down changes to first order. Methodologically, the projection is derived, and the stochastic sufficient conditions are discussed. Moreover, the RMSProp and Adam variants are evaluated. The optimizer-level treatment is useful, and the empirical results are convincing.

Critical strengths
  • The key idea, i.e., constraining the realized optimizer update, provides a reasonable solution to the gap left by pairwise loss modifications. Besides. the derived projection could be directly connected to the desired first-order reward changes.

  • The experiments are comprehensive, which validate the reward trajectories, generation quality, and projection variants across several losses and model settings.

Critical weaknesses
  • There seems to be a gap between the population theorem and the implementation. Specifically, Theorem 4.1 assumes population alignment and safety with second-order control. However, the minibatch diagnostics do not verify the noise conditions of Propositions 4.1 and 4.2. Appendix C changes the safety floor, uses current-batch preconditioning and allows an uncorrected first step. Moreover, lim inf⁡\liminf gradient convergence is weaker than RSTP. Thus, the population claim needs further refinement.

  • The empirical validations could be improved. Specifically, the centered-versus-quadrant ablation measures reward movement, while it does not consider the generation quality; the larger reward separation alone also does not justify the additional balance constraint.

Questions for authors

Q1. Which guarantees remain valid for the exact implementation, including the modified floor and first-step exception in Appendix C? Could the population noise and margin conditions be verified on independent batches?

Q2. Can the projected method converge at a point with a nonzero loss gradient due to the vanished feasible direction? What conditions exclude this case for the implemented losses, beyond assuming uniform descent alignment?

Q3. For the centered and quadrant projections, could generation quality be compared at matched update norm or policy divergence?

9 comments
AuthorsOpenReview ↗

W1 (part 1/3): Connecting the theory to the experiments — minibatch diagnostics and the modified safety floor

We thank the reviewer for the close reading of the theory and for identifying precisely where the analysis and the implementation part ways. Each of the four gaps the reviewer lists is addressed in the revision: Theorem 4.1(a) is strengthened from a lim inf to a limit, the modified floor now has its own norm and alignment bounds (Corollaries C.1 and C.2), the current-batch preconditioner is bounded against the lagged one and compared with it empirically (Appendix C.1.1), and the uncorrected first step is stated explicitly. We have also measured the noise quantities of Proposition 4.2 and tested Assumptions 3 and 4 on the steps actually taken (Appendix E.11), and extended the centered-versus-quadrant ablation to generation quality at matched divergence (Table 10). Detailed responses follow.

There seems to be a gap between the population theorem and the implementation. Specifically, Theorem 4.1 assumes population alignment and safety with second-order control. However, the minibatch diagnostics do not verify the noise conditions of Propositions 4.1 and 4.2. Appendix C changes the safety floor, uses current-batch preconditioning and allows an uncorrected first step. Moreover, lim inf gradient convergence is weaker than RSTP. Thus, the population claim needs further refinement.

Minibatch diagnostics verification. The revision adds measurements of the noise quantities of Prop. 4.2 and realized tests of Assumptions 3 and 4 of Theorem 4.1 (Appendix E.11); please see our responses to Reviewer Kxx7, Q3 and W2(b).

For Prop. 4.1, the condition for population descent is 1/ρmax⁡2>(C/c)2−11/\rho_{\max}^2 > (C/c)^2 - 1, where CC bounds the norm ratio ∥g^∥/∥g∥\Vert\hat{\bf g}\Vert/\Vert{\bf g}\Vert and cc is the alignment constant. Both constants are certified by Corollaries C.1 and C.2 and confirmed by the projection diagnostics (Appendix E.11, Fig. 8):

  • Norm. ∥g^∥/∥g∥≤1\Vert\hat{\bf g}\Vert/\Vert{\bf g}\Vert \le 1 throughout, so C=1C = 1.

  • Alignment, certified. Corollary C.2 gives c≥(1+κ)πmin⁡−κc \ge (1+\kappa)\pi_{\min} - \kappa with πmin⁡\pi_{\min} the smallest value of ∥g⊥∥2/∥g∥2\Vert{\bf g}^\perp\Vert^2/\Vert{\bf g}\Vert^2. The diagnostics show ∥g⊥∥/∥g∥≥0.82\Vert{\bf g}^\perp\Vert/\Vert{\bf g}\Vert \ge 0.82, so πmin⁡=0.67\pi_{\min} = 0.67 and, at κ=0.4\kappa = 0.4, c≥0.54c \ge 0.54.

  • Alignment, observed. The measured ⟨g,g^⟩/∥g∥2\langle {\bf g}, \hat{\bf g}\rangle/\Vert{\bf g}\Vert^2 is at least 0.830.83.

With C=1C = 1 the condition becomes 1/ρmax⁡2>1/c2−11/\rho_{\max}^2 > 1/c^2 - 1. Here 1/ρ21/\rho^2 is a lower bound on the signal-to-noise ratio (SNR) of the minibatch loss gradient, so the condition says how much gradient noise the method can tolerate. Using only the alignment that Corollary C.2 guarantees (c=0.54c = 0.54), the SNR must exceed about 2.42.4: the signal must be at least about 2.42.4 times as strong as the noise. Using the alignment we observed (c=0.83c = 0.83), an SNR of about 0.450.45 is enough: descent holds even when the noise is about twice as strong as the signal.

Safety floor used in experiments. The modified floor used in our experiments (Appendix C.2.1) is covered by the theory. Corollaries C.1 and C.2 (revised) are counterparts of Corollaries 3.1 and 3.2 and supply the norm and alignment constants required by Proposition 4.1; Sections 4 and 6 and Appendix E.11 now refer to them for the floor used in our experiments:

  • Norm (C.1). ∥g^∥≤max⁡{1,κ}∥g∥\Vert\hat{\bf g}\Vert \le \max\lbrace 1,\kappa\rbrace \Vert{\bf g}\Vert, with no eigenvalue bounds on M{\bf M} needed; the update is non-expansive for κ≤1\kappa \le 1.

  • Alignment (C.2). ⟨g,g^⟩≥((1+κ)πmin⁡−κ)∥g∥2\langle {\bf g}, \hat{\bf g}\rangle \ge \big((1+\kappa)\pi_{\min} - \kappa\big)\Vert{\bf g}\Vert^2 whenever ∥g⊥∥2/∥g∥2≥πmin⁡\Vert{\bf g}^\perp\Vert^2/\Vert{\bf g}\Vert^2 \ge \pi_{\min}, for any in-span orientation. This gives a positive alignment constant cc, as Prop. 4.1 requires, when πmin⁡>κ/(1+κ)\pi_{\min} > \kappa/(1+\kappa), i.e. ∥g⊥∥/∥g∥>0.535\Vert{\bf g}^\perp\Vert/\Vert{\bf g}\Vert > 0.535 at our κ=0.4\kappa = 0.4; the observed value is at least 0.820.82 (Fig. 8).

AuthorsOpenReview ↗

W1 (part 2/3): Connecting the theory to the experiments — Current batch preconditioning, first step handling

..uses current-batch preconditioning and allows an uncorrected first step. Thus, the population claim needs further refinement.

Current-batch preconditioning. Our experiments use the current-batch preconditioner because it is the standard form of RMSProp: the second moment is updated with the current gradient before the step is scaled, as in torch.optim.RMSprop, which GSafeRMSProp extends.

Prop. 4.1 assumes zero-mean noise; that assumption transfers to the optimizer’s metric when Pt{\bf P}_t is Ft\mathcal F_t-measurable, which RMSProp satisfies with a one-step-lagged second-moment preconditioner. Such a lagged preconditioner is therefore a departure from the default optimizer, introduced only so that Proposition 4.1 applies in the optimizer's metric: its zero-mean noise assumption, E[ζt∣Ft]=0\mathbb E[\zeta_t \mid \mathcal F_t] = 0, carries over to the preconditioned gradient Pt1/2gt(b)P_t^{1/2} g_t^{(b)} only if PtP_t is Ft\mathcal F_t-measurable.

Our experiments use the current-batch preconditioner, so the revision quantifies the gap in two ways:

  • Theory. Proposition C.1 (Appendix C.1.1 of the revision) shows that, on every coordinate and at every step, the current-batch preconditioner is at most λ−1/2\lambda^{-1/2} times the lagged one, i.e. at most about 5% larger for our λ=0.9\lambda = 0.9.

  • Experiment. A new ablation in Appendix C.1.1 checks that our results do not depend on this difference: we re-ran DPO-S (Llama-3.2-1B / RewardBench-v2) with the one-step-lagged preconditioner Pt−=diag(st−1+ϵ)−1P_t^- = \mathrm{diag}(\sqrt{s_{t-1}} + \epsilon)^{-1}, obtained by swapping lines 1 and 2 of Algorithm 1 for t≥2t \ge 2. All other settings are unchanged.

Preconditionerr+r^+r−r^-Pref. WRGen. WR vs SFT
Current-minibatch (as implemented)0.354−1.1830.7720.742
Lagged (lines 1 and 2 swapped, t≥2t \ge 2)0.291−3.8110.7690.754

The variant with the Ft\mathcal F_t-measurable preconditioner preserves the behaviour of the implemented one: the chosen reward stays positive, the rejected reward stays negative, and preference win rate is essentially unchanged (0.769 vs 0.772). Generation win rate against SFT is slightly better with the lagged variant (0.754 vs 0.742), and it lowers the rejected reward further. Proposition 4.1, and through it Assumption 2 of Theorem 4.1, therefore applies to an optimizer with the same reward-safe dynamics as the one we evaluated, at no cost in preference accuracy or generation quality.

Uncorrected first step. The reviewer is right that the very first update is applied without the projection; we use it because the second moment starts at zero, which otherwise causes early-step blow-ups. This step is best read as part of the initialization: our analysis starts from θ1\boldsymbol\theta_1, and every subsequent update is projected, so the guarantees of Section 4 hold for all t≥1t \ge 1. The first step has also populated the second moment, so the optimizer state at t=1t = 1 depends only on the first minibatch, as the stochastic analysis requires. The cost is one plain RMSProp step of size η\eta that carries no reward-safety guarantee. We now state this exception explicitly in Appendix C.2.2.

AuthorsOpenReview ↗

W1 (part 3/3): Strengthened convergence result (Theorem 4.1(a): to )

Moreover, lim inf gradient convergence is weaker than RSTP. Thus, the population claim needs further refinement.

lim inf vs RSTP: We agree. RSTP is posed for a deterministic sequence, whereas Theorem 4.1 concerns stochastic minibatch iterates, so its conclusions are in expectation over the trajectory. Within that setting the submitted statement was weaker than necessary: the revision strengthens Theorem 4.1(a) to lim⁡t→∞E∥∇L(θt)∥2=0\lim_{t\to\infty}\mathbb E\|\nabla L(\theta_t)\|^2=0 under the same assumptions, using the up-crossing argument of Bertsekas & Tsitsiklis (2000) (new Lemmas A.1–A.2, App. A.5). We also add a sentence after (RSTP) stating that for a stochastic algorithm its conditions are required in expectation over the trajectory, so the problem statement and Theorem 4.1 now match.

Reviewer_ErXdOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on controlling chosen and rejected reward dynamics during preference optimization. It proposes GSafePO, which projects optimizer updates to enforce chosen-up/rejected-down changes to first order, with RMSProp and Adam variants. Experiments across three model configurations and several preference objectives report improved reward dynamics and generation win rates.

Critical strengths
  • The proposed approach can be combined with several existing preference losses.

  • The experiments include multiple random seeds, implementation details, and computational overhead measurements.

Critical weaknesses
  • A directly relevant gradient-projection baseline is missing. DPO-PG (Won et al., 2025, arXiv:2505.23761, Appendix H) already projects gradients to increase chosen likelihood while maintaining or decreasing rejected likelihood. The paper neither discusses nor compares with this method, leaving the added value of its balance constraint and optimizer-aware projection unclear.

  • The theoretical guarantees are not established for the optimizer actually evaluated. The sufficient conditions discussed in Appendix C require a preconditioner determined before the current minibatch, whereas Algorithm 1 uses current-minibatch statistics. The experiments also modify the safety floor, without establishing the same population-level guarantees for this practical variant.

  • The additional balance constraint in Eq. (11) is not sufficiently justified experimentally. Appendix E.9 compares it with a simpler quadrant projection, but Figure 6 only reports reward trajectories. Larger reward shifts do not establish better response quality; preference accuracy and generation results are needed to justify the more restrictive update.

  • The evidence for GSafeAdam is limited to DPO on Llama-3.2-1B and RewardBench in Appendix E.7. The main comparisons across losses and models use RMSProp, and no generation-quality evaluation is provided for Adam. This leaves the claimed benefit across adaptive optimizers insufficiently tested.

I thank the authors for their response. I have carefully read the rebuttal, but my original concerns remain. The other reviews have also raised additional concerns regarding the evaluation protocol and the theoretical claims.

Questions for authors

Please see the weaknesses above.

6 comments
AuthorsOpenReview ↗

W1 Gradient projection baseline (DPO-PG) part 1

We thank the reviewer for the careful reading and for the constructive assessment. The pointer to DPO-PG was especially valuable: we were not aware of it, and the revision now discusses it in the related work and compares against it directly, together with a new ablation that isolates the effect of projecting in the optimizer's metric (Appendix E.12, Tables 14 and 15). The revision also extends the theory to the optimizer as implemented, adds preference accuracy and generation quality to the balance-constraint ablation (Table 10), and reports GSafeAdam results for four losses with generation evaluation (Table 7). We respond to each point below.

A directly relevant gradient-projection baseline is missing. DPO-PG (Won et al., 2025, arXiv:2505.23761, Appendix H) already projects gradients to increase chosen likelihood while maintaining or decreasing rejected likelihood. The paper neither discusses nor compares with this method, leaving the added value of its balance constraint and optimizer-aware projection unclear.

Response. DPO-PG is indeed a relevant prior update-level method that we were unaware of; the revision adds it to Related Work and as a baseline. We state the key differences below.

1. The two methods answer different questions: DPO-PG asks how to raise the chosen likelihood without raising the rejected one, while ignoring the preference loss entirely; GSafePO asks how to optimise a given preference loss subject to chosen-up/rejected-down. While both methods project gradients they are different gradients entirely. DPO-PG projects the gradient of the chosen reward onto a closed half-space that does not increase the rejected reward. So DPO-PG ignores the gradient of the preference loss entirely. GSafePO projects the preference loss gradient onto a hyperplane to protect chosen-up/rejected-down behavior. This distinction has several key structural downstream consequences.

  • a. DPO-PG has no descent guarantees for widely used preference losses: Cor. H.5.1 shows descent only for loss functions ff of the batch-mean margin of the form f(Δˉ)f(\bar\Delta), not for the mean of per-pair losses of the form 1K∑kf(Δk)\tfrac1K\sum_k f(\Delta_k) that are actually used for preference learning. Loss descent is incidental. The per-pair gradient is 1K∑kwk(∇rw,k−∇rl,k)\tfrac1K\sum_k w_k(\nabla r_{w,k} - \nabla r_{l,k}) with wk=−f′(Δk)>0w_k = -f'(\Delta_k) > 0. DPO-PG’s descent argument needs ∑kwk⟨∇rw,k−∇rl,k,v⟩>0\sum_k w_k\langle\nabla r_{w,k} - \nabla r_{l,k}, v\rangle > 0 but only controls the unweighted sums. The two agree when the weights are equal, and they agree approximately when the weighted and unweighted mean gradients are close. GSafePO's alignment results (Cor. 3.2, and Cor. C.2 for the modified floor used in our experiments) are stated for the gradient of whichever preference loss is being trained.

  • b. DPO-PG does not maintain the structure of the underlying loss Without f′(Δk)f'(\Delta_k) the update does not vanish on already-solved pairs, so it keeps raising log⁡π(yw)\log\pi(y_w) on every pair, solved or not, as SFT does. gg carries the Bradley–Terry weights f′(Δk)f'(\Delta_k), so effort concentrates on pairs the model still gets wrong and fades as they are solved; GSafePO structurally preserves this behavior. Also, β\beta, the anchoring to πref\pi_{\rm ref}, SimPO’s length normalisation and margin, Cal-DPO’s targets and DPOP’s penalty all enter through gg and survive the projection; DPO-PG’s update is identical for all of them.

2. DPO-PG is not Optimizer-aware. DPO-PG projects in Euclidean geometry and then passes vPGv_{\rm PG} to Adam/RMSprop; Thms. H.4–H.5 are proved for the SGD step. For a step −ηPv-\eta P v, ⟨μl,v⟩≥0\langle\mu_l,v\rangle\ge 0 does not imply ⟨μl,Pv⟩≥0\langle\mu_l,Pv\rangle\ge 0. Example: μw=(2,1)\mu_w=(2,1), μl=(1,1)\mu_l=(1,1) gives vPG=(−12,12)v_{\rm PG}=(-\tfrac12,\tfrac12); with P=diag(4,1)P=\mathrm{diag}(4,1) the rejected reward increases (Δrl=+1.5η\Delta r_l=+1.5\eta), and with P=diag(1,4)P=\mathrm{diag}(1,4) the chosen reward decreases (Δrw=−η\Delta r_w=-\eta). GSafePO projects in the PP-metric (Eqs. 29–31), so the constraints hold for the realized RMSProp/Adam step, including momentum.

3. DPO-PG does not have balance and margin control. In the active case DPO-PG sets Δrl=0\Delta r_l = 0 to first order, so it does not drive the rejected reward down at all; any movement of rlr_l comes from inactive steps, curvature and the preconditioner, none of which the method controls. Won et al.’s Fig. 8 shows the result ranging from near-flat to a collapse of several hundred nats under the same update rule. GSafePO instead enforces Δrl=−Δrw≤−ηδ\Delta r_l = -\Delta r_w \le -\eta\delta up to first order. The balance equality removes the common-mode term, and the floor δ\delta (in our experiments, the modified floor of App. C.2.1) guarantees a minimum margin, which gives headroom for minibatch safety to carry over to the population. κ\kappa also provides a continuous trade-off rather than a single operating point.

AuthorsOpenReview ↗

W1 Gradient projection baseline (DPO-PG) part 2 (Experiments)

New experiments: We added the experiment DPO-PG (RMSprop, as in Won et al.) vs. DPO-S

Methodr+r^+r−r^-Pref. WRGen. WR vs SFTGen. H2H WR
DPO-PG9.9749.974−14.541-14.5410.7020.7020.7070.7070.3330.333
DPO-S (ours)0.3540.354−1.183-1.1830.772\mathbf{0.772}0.742\mathbf{0.742}0.667\mathbf{0.667}

Remark. DPO-S outperforms DPO-PG on every win-rate metric: +7.0+7.0 points preference win rate, +3.5+3.5 points generation win rate vs. SFT, and a 0.6670.667 head-to-head win rate. DPO-PG's projection is one-sided, so nothing bounds the magnitudes: r+r^+ and r−r^- drift to 9.979.97 and −14.54-14.54, versus 0.350.35 and −1.18-1.18 under our balance constraint.

We now discuss DPO-PG in the related work (Section 1.1), summarize the comparison in Section 6, and report full results in Appendix E.12 (Table 14) of the revision.

DPO-PG differs from DPO-S in several respects at once (what is projected, the balance constraint, and the metric), so the comparison above does not isolate the contribution of optimizer-aware projection. To isolate it, we keep the DPO-S loss, constraint set and hyperparameters fixed and change only the metric in which the projection is computed:

Projection metricr+r^+r−r^-Pref. WRGen. WR vs SFT
RMSProp (optimizer-aware, as implemented)0.3540.354−1.183-1.1830.772\mathbf{0.772}0.742\mathbf{0.742}
Euclidean (raw-space, DPO-PG-style application)1.9531.9530.7010.7010.7370.7370.6700.670

Remark. Projecting in the Euclidean metric and then letting RMSProp rescale the step costs 3.53.5 points of preference win rate and 7.27.2 points of generation win rate vs. SFT. It also loses reward safety in practice: the rejected reward ends above its initial value (r−=+0.70r^- = +0.70). Constraints imposed on ⟨μ,g^⟩\langle \mu, \hat g\rangle do not carry over to the preconditioned step ⟨μ,Pg^⟩\langle \mu, P\hat g\rangle; projecting in the optimizer's metric enforces them for the step actually taken. We have added this ablation to Appendix E.12 of the revision (Table 15).

AuthorsOpenReview ↗

W3 Ablation of balance constraint

The additional balance constraint in Eq. (11) is not sufficiently justified experimentally. Appendix E.9 compares it with a simpler quadrant projection, but Figure 6 only reports reward trajectories. Larger reward shifts do not establish better response quality; preference accuracy and generation results are needed to justify the more restrictive update.

Response. We agree. The revision adds preference accuracy and generation quality to the ablation (Appendix E.9, revised Table 10). At the final checkpoints, for DPO on Llama-3.2-1B / RewardBench-v2:

Projectionr+↑r^+ \uparrowr−↓r^- \downarrowPref. WR ↑\uparrowGen. H2H WR ↑\uparrowGen. WR vs SFT ↑\uparrow
Centered0.354-1.1830.7720.5400.742
Quadrant0.015-0.2820.6930.4600.705

The centered projection has higher preference accuracy and is not worse in generation quality, while the quadrant projection leaves the chosen reward at 0.0150.015, on the boundary of the safe region. The two runs end at somewhat different policy divergence; at matched divergence their generation quality is comparable (0.7160.716 against 0.7050.705 in win rate against SFT). Details are in our responses to Reviewer kBe7 (W2 and Q3).

Reviewer_Kxx7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that DPO-style objectives control only the chosen–rejected reward margin, so the two implicit rewards can drift together (likelihood displacement, rejected-reward inflation). Instead of modifying the loss, it proposes GSafePO, which projects the (preconditioned) minibatch update so that, to first order, the minibatch-average chosen reward rises and the rejected reward falls by equal amounts with a margin δ. The projection has a closed form in a dual basis of span{µw, µl} (Prop. 3.1). The paper adds norm/alignment bounds (Cor. 3.1–3.2) and population-level convergence/safety results for stochastic minibatches (Thm. 4.1, Props. 4.1–4.2). It instantiates GSafeRMSProp and GSafeAdam via a metric change (Sec. 5). Experiments use Qwen-0.5B, Llama-1B and Llama-8B-LoRA with DPO, SimPO, RC-DPO, DPOP and Cal-DPO on UltraFeedback, RewardBench-v2 and MMLU-Pro (ID and OOD). They report reward dynamics, preference accuracy, and LLM-judge head-to-head comparisons between each objective and its "-S" variant.

Critical strengths
  • S1. A modular, loss-agnostic intervention point. The constraint acts on the realized update, so it applies to any pairwise-score objective without changing or retuning the loss (App. D). The projection is a closed-form 1-D problem (Prop. 3.1), and the bar-space metric change (Eq. 28–30) keeps it consistent with adaptive preconditioning. This is the main reason the work could be useful in practice.
  • S2. Broad and consistent evidence on reward dynamics. The study covers 5 objectives × 3 models × ID/OOD with 5 seeds (Table 1, E.1–E.2). The per-example breakdown in Table 2a shows DPO (+,−) going from 5.6% to 66.2% and DPOP (+,+) from 83.4% to 16.9%, and the training trajectories in Fig. 4 show the same shift. This is supported by diagnostics (E.11), measured overhead (Table 6), and ablations (E.8–E.9). The paper's direct claim, that the optimizer changes reward dynamics, is well supported.
Critical weaknesses
  • W1. The central practical claim, "significantly improved" generation (Abstract), is not established.
    • (a) H2H is computed only at the final checkpoint, against baselines tuned for preference accuracy rather than generation quality. For DPO, the selected learning rate (1e-5) is at the upper edge of the grid, and on Llama-1B the chosen reward falls to −12.278 on UltraFeedback (Table 1). In the paper's own Fig. 7, DPO and DPO-S reach nearly the same peak win rate over SFT (~0.76 vs ~0.77, read from the figure); the gap appears mainly after the peak. So the 0.330/0.670 (DPO) and 0.205/0.795 (SimPO) gaps in Table 1 may reflect vanilla over-optimization rather than a better solution.
    • (b) Comparisons are only within each objective. When the loss already keeps the chosen reward up (DPOP), H2H is 0.497/0.503. This suggests the gain comes from preventing chosen-likelihood decrease, which loss-level methods cited in the paper also target: DPOP (Pal et al., 2024) and DPO+NLL (Pang et al., 2024; Liu et al., 2024). There is no DPO-S vs. DPOP or DPO-S vs. DPO+NLL comparison, no absolute benchmark, and no win rate against SFT for Table 1. No variance is reported despite 5 seeds (see also Q5).
    • Fix: add KL-matched or best-checkpoint generation results, cross-objective H2H or an absolute benchmark such as AlpacaEval 2 LC, and report variance.
  • W2. The theory does not deliver the claimed "population-level guarantees", and the geometric novelty is overstated.
    • (a) Thm. 2.1 restates that the dual coordinates equal the constraint values. With the balance constraint, the method is a projection that removes the ∇(r̄w+r̄l) component and clips the margin coordinate. This is in the family of GEM (Lopez-Paz & Ranzato, 2017), A-GEM (Chaudhry et al., 2019), PCGrad (Yu et al., 2020) and MGDA (Sener & Koltun, 2018), none of which is discussed. For δ = 0, Cor. 3.1–3.2 follow from projection onto a convex cone.
    • (b) Thm. 4.1(b) follows directly from Assumptions 3–4, which assume population-level first-order safety that dominates curvature. Prop. 4.2 requires γ_t ≥ max(ρ_w,ρ_l)·S_t with absolute reward-gradient noise ρ, which is never estimated.
    • (c) The paper's own κ = 0 ablation (Table 8: r+ −3.354, r− −4.103, close to vanilla DPO's −3.648/−6.555) shows that the minibatch constraints alone do not produce population-level chosen-up behavior; the effect comes from the tuned floor κ.
    • Fix: position the method relative to gradient-projection methods, qualify the theory claims, and quantify ρ and S_t empirically.
Questions for authors
  • Q1. For each Table 1 pair, what is the win rate against the SFT reference at (i) the final checkpoint and (ii) the best-validation or KL-matched checkpoint? What are the results of DPO-S vs. DPOP and DPO-S vs. DPO+NLL head-to-head? If DPO-S still wins at matched checkpoints, W1 would largely be resolved.
  • Q2. On which split was κ selected? E.3 says κ = 0.4 was not tuned, but E.8 selects it. Why do the Cal-DPO-S κ = 0.4 entries in Table 8 ("validation set": 1.813 / −3.632 / 0.776) coincide exactly with the Table 1 evaluation entries?
  • Q3. With κ = 0 the balance constraint holds on every training minibatch, yet both evaluation rewards decrease. Is this caused by reward-gradient noise (please estimate ρ_w, ρ_l and S_t at batch 512), by higher-order effects, or by train/eval shift?
  • Q4. All main results use RMSProp. What are the results with GSafeAdam/AdamW for SimPO, DPOP and Cal-DPO, including generation evaluation?
  • Q5. For the H2H evaluation: how many prompts were used, with which decoding settings, how were ties handled, and what are the response lengths per pair? Do "-S" models produce systematically shorter outputs?
9 comments
AuthorsOpenReview ↗

W1(a), W1(b), suggested fix, and Q1: evidence for the generation claim (part 1/2: results)

We thank the reviewer for a detailed and specific review. The questions were concrete enough to test directly, and doing so has improved the paper. In response we ran the requested comparisons: win rates against the SFT reference with variance, matched-divergence comparisons (Appendix E.13), cross-objective head-to-head results including a new DPO+NLL baseline, and GSafeAdam results for four losses (Table 7 of the revision). We also measured ρw\rho_w, ρl\rho_l and StS_t and decomposed the κ=0\kappa = 0 reward drop (Appendix E.11), repositioned the method relative to gradient-projection methods, and relabelled Theorem 2.1 as a lemma. Where the new evidence did not support a claim as strongly as the submission stated, for DPOP and for generation quality under Adam, we say so and have narrowed the claim in the abstract, Section 6 and the conclusion.

W1(a): H2H only at the final checkpoint, against baselines tuned for preference accuracy; DPO's learning rate is at the edge of the grid; the gap may reflect vanilla over-optimization.

W1(b): comparisons only within each objective; no DPO-S vs DPOP or DPO+NLL; no absolute benchmark; no win rate against SFT; no variance.

Q1: win rate against SFT at the final and at the best-validation or KL-matched checkpoint; DPO-S vs DPOP and vs DPO+NLL.

Response We ran the requested protocol for three of the Table 1 objectives (DPO, Cal-DPO, DPOP) on Llama-3.2-1B / RewardBench-v2, with Llama-3.1-70B-Instruct as a forced-choice judge and n=8n=8 sampled generations per prompt. The H2H numbers in Table 1 of the paper are on UltraFeedback, so the values below are not expected to match them numerically. Entries are mean ± sd. H2H cells are vanilla win rate / GSafe win rate. Bold marks the better method in each pair; † marks a chosen reward below the reference (r+<0r^+<0) or a rejected reward above it (r−>0r^->0).

(i) Final checkpoint

Methodr+r^+r−r^-Pref. WRGen. H2H (vanilla / safe)Gen. WR vs SFT
DPO−3.648 ± 0.45 †−6.555 ± 0.750.771 ± 0.0150.486 ± 0.018 / 0.514 ± 0.0180.643 ± 0.039
DPO-S0.354 ± 0.09−1.183 ± 0.180.772 ± 0.0120.742 ± 0.009
Cal-DPO0.079 ± 0.08−0.079 ± 0.060.761 ± 0.0160.369 ± 0.013 / 0.631 ± 0.0130.525 ± 0.009
Cal-DPO-S1.813 ± 0.05−3.632 ± 0.080.776 ± 0.0130.651 ± 0.009
DPOP9.115 ± 0.884.508 ± 0.52 †0.710 ± 0.0190.350 ± 0.006 / 0.650 ± 0.0060.780 ± 0.015
DPOP-S1.080 ± 0.17−0.205 ± 0.080.758 ± 0.0140.821 ± 0.009

DPO-S vs. DPOP and DPO-S vs. DPO+NLL. DPO+NLL (L=LDPO+α⋅NLL(yw∣x)\mathcal{L} = \mathcal{L}_{\text{DPO}} + \alpha \cdot \mathrm{NLL}(y_w \mid x), Pang et al. 2024; α=1\alpha = 1, β=0.1\beta = 0.1, lr =10−5= 10^{-5}) is a new baseline implemented for this response. The last column is the baseline's H2H win rate against DPO-S, so values below 0.5 mean DPO-S wins.

Methodr+r^+r−r^-Pref. WRGen. WR vs SFTGen. H2H WR vs DPO-S
DPOP9.115 ± 0.884.508 ± 0.52 †0.710 ± 0.0190.780 ± 0.0150.450 ± 0.021
DPO+NLL23.884 ± 2.87.573 ± 1.9 †0.702 ± 0.0120.732 ± 0.0160.337 ± 0.023
DPO-S0.354 ± 0.09−1.183 ± 0.180.772 ± 0.0120.742 ± 0.009–

Full cross-objective H2H matrix (final checkpoint, n=8n=8 samples per prompt; rows = vanilla, columns = GSafe; each cell is row win rate / column win rate):

DPO-SCal-DPO-SDPOP-S
DPO0.486 / 0.5140.304 / 0.6960.298 / 0.702
Cal-DPO0.197 / 0.8030.369 / 0.6310.198 / 0.802
DPOP0.450 / 0.5500.705 / 0.2950.350 / 0.650

Summary (supporting analysis for point 2 below is in part 2/2).

  1. Each GSafe variant beats its vanilla counterpart against SFT at the final checkpoint: DPO +0.099, Cal-DPO +0.126, DPOP +0.041. Each difference is more than two standard deviations.
  2. The gain is not vanilla over-optimization. For DPO, DPO-S is ahead at matched D^\hat D and both methods are still on the rising part of the curve. For Cal-DPO, the vanilla method has barely left the reference, so the argument does not apply.
  3. The gain is not limited to within-objective comparisons. DPO-S wins head-to-head against both loss-level methods the reviewer names (0.550 vs DPOP, 0.663 vs DPO+NLL), and the GSafe variant is preferred in eight of the nine cells of the cross-objective matrix.
  4. The mechanism differs from loss-level fixes. DPOP and DPO+NLL keep r+r^+ high but also raise r−r^- above the reference (4.508 and 7.573) and lose preference accuracy (0.710 and 0.702, vs 0.772 for DPO-S). GSafe keeps r+≥0r^+ \ge 0 and r−≤0r^- \le 0 without that cost.
AuthorsOpenReview ↗

W1(a), W1(b), suggested fix, and Q1 (part 2/2: checkpoint selection and matched divergence)

Table (i) and the cross-objective matrix are in part 1/2.

Best-validation. Hyperparameters are selected on a validation subset of the training split (Appendix E.3), and the checkpoint we report is the final training step. Fig. 9 (Appendix E.13) plots preference win rate on the evaluation split at every checkpoint for DPO, DPO-S and DPOP. It plateaus by roughly step 20 of 50 and is flat thereafter, and the final step is within 0.01 of the best checkpoint on that split. Even a checkpoint chosen with hindsight on the evaluation split would therefore gain at most 0.01 in preference accuracy, so we report the final checkpoint throughout.

KL-matched, DPO vs DPO-S (Revised Appendix E.13, Fig 10(a)). The reviewer reads Fig. 7 as DPO and DPO-S reaching nearly the same peak win rate over SFT (≈0.76\approx 0.76 vs ≈0.77\approx 0.77), with the gap opening only after the peak. Fig. 7 estimated each point from a single greedy generation per prompt. Fig. 10(a) re-estimates the curve with a Monte Carlo average over n=8n=8 sampled responses per prompt and plots it against D^(θt)\hat D(\theta_t) computed from the same samples (see our response to Q5 for the protocol), so values on the two figures are not directly comparable. The final checkpoints are not at identical divergence (DPO: D^≈21.8\hat D \approx 21.8; DPO-S: D^≈25.6\hat D \approx 25.6), so we compare at matched D^\hat D by reading each curve at the other method's checkpoint. DPO-S remains ahead at both: 0.733 vs 0.643 at D^≈21.8\hat D \approx 21.8, and 0.742 vs 0.672 at D^≈25.6\hat D \approx 25.6. Both final checkpoints lie on the rising part of the curve, so DPO is not being evaluated past its peak.

On the learning rate: 1e-5 was selected for DPO by validation preference accuracy, and it is the largest value in the grid. The matched-divergence comparison is our control for this, since it compares the two methods at equal distance from the reference whatever step size produced it.

Cal-DPO vs Cal-DPO-S. Over-optimization of the vanilla method is not a possible explanation for this pair. Cal-DPO's rewards remain near the reference at the final checkpoint (r+=0.079r^+ = 0.079, r−=−0.079r^- = -0.079; D^≈\hat D \approx 2.71), while Cal-DPO-S moves substantially further (D^≈\hat D \approx 6.39) and is still preferred (0.631 H2H; 0.651 vs 0.525 against SFT). Cal-DPO's preference win rate (0.761) is on par with the other methods, so it is not under-trained by its own selection criterion.

KL-matched, DPOP vs DPOP-S (Revised Appendix E.13, Fig 10(b)). For this pair the final checkpoints are far apart in divergence (DPOP-S: D^≈10.30\hat D \approx 10.30; DPOP: D^≈30.43\hat D \approx 30.43), so the final-checkpoint comparison in (i) is not a matched one. Reading each curve at the other method's checkpoint, as for DPO, DPOP-S is ahead at both: 0.821 vs 0.624 at D^≈10.30\hat D \approx 10.30, and 0.802 vs 0.780 at D^≈30.43\hat D \approx 30.43. DPOP-S reaches its best win rate at about a third of the divergence at which DPOP reaches 0.780, and it does so with r−<0r^- < 0, where DPOP has r−>0r^- > 0.

DPOP's final checkpoint is on the rising part of its curve, so it too is not being evaluated past its peak. For this pair the curve beyond the final checkpoints matters, and we report it: DPOP keeps improving, overtakes DPOP-S shortly after D^≈30\hat D \approx 30, and reaches 0.830 at the largest divergence on the curve, where DPOP-S has declined slowly from its peak to 0.792. The best values of the two methods (0.821 and 0.830) are close. We therefore do not claim that GSafe raises the peak generation quality of DPOP. The claim for this pair is that GSafe reaches that quality at a much smaller divergence from the reference and with the rejected reward kept below it, consistent with the paper's statement that generation quality is approximately preserved for DPOP while its reward dynamics are corrected, and with the reviewer's observation in W1(b).

Two caveats. On this benchmark the direct DPO vs DPO-S head-to-head is close (0.486 / 0.514), so for that pair the evidence is the win rate against SFT and the matched-divergence comparison. DPOP's win rate against SFT (0.780) is above DPO-S's (0.742), although DPO-S wins the direct comparison. In the cross-objective matrix, vanilla DPOP is preferred over Cal-DPO-S (0.705), in line with its higher win rate against SFT (0.780 vs 0.651).

In line with these results, the revision replaces "significantly improved behavior" in the abstract with "improved generation quality in settings exhibiting likelihood displacement", states in Section 6 and the conclusion that generation quality is preserved, not improved, for objectives that already protect the chosen reward, and points to the matched-divergence comparison (Appendix E.13) from the main text.

Table (i), the DPO+NLL comparison and the cross-objective matrix are in Appendix E.14 (Tables 17–19) of the revision.

AuthorsOpenReview ↗

Q4 Adam Experiment

All main results use RMSProp. What are the results with GSafeAdam/AdamW for SimPO, DPOP and Cal-DPO, including generation evaluation?

Response We ran vanilla Adam and GSafe-Adam for SimPO, DPOP and Cal-DPO, in addition to the DPO pair from Appendix E.7, on Llama-3.2-1B / RewardBench-v2. Entries are mean ± sd. Generation is judged by Llama-3.1-70B-Instruct. H2H cells are vanilla win rate / GSafe win rate. Bold marks the better optimizer for each loss; † marks a chosen reward below the reference (r+<0r^+<0) or a rejected reward above it (r−>0r^->0).

LossOptimizerr+r^+r−r^-Pref. WRGen. WR vs SFTGen. H2H (vanilla / GSafe)
DPOAdam−1.648 ± 0.25 †−3.207 ± 0.400.728 ± 0.0180.689 ± 0.0160.488 ± 0.022 / 0.512 ± 0.022
DPOGSafe-Adam0.460 ± 0.07−0.138 ± 0.030.742 ± 0.0140.696 ± 0.015
Cal-DPOAdam0.108 ± 0.03−0.227 ± 0.040.593 ± 0.0210.529 ± 0.0190.482 ± 0.021 / 0.518 ± 0.021
Cal-DPOGSafe-Adam0.811 ± 0.12−0.532 ± 0.090.649 ± 0.0230.530 ± 0.019
DPOPAdam0.248 ± 0.040.036 ± 0.02 †0.609 ± 0.0220.526 ± 0.0190.490 ± 0.022 / 0.510 ± 0.022
DPOPGSafe-Adam0.292 ± 0.05−0.045 ± 0.020.622 ± 0.0210.536 ± 0.019
SimPOAdam−3.171 ± 0.41 †−6.152 ± 0.370.685 ± 0.0240.727 ± 0.0190.460 ± 0.025 / 0.540 ± 0.025
SimPOGSafe-Adam1.974 ± 0.30−2.063 ± 0.310.725 ± 0.0240.745 ± 0.019

Remark. The reward-dynamics result carries over from RMSProp to Adam. All three violations under vanilla Adam (DPO and SimPO with r+<0r^+<0, DPOP with r−>0r^->0) are removed by GSafe-Adam, and every GSafe-Adam run ends with r+>0r^+>0 and r−<0r^-<0. Preference win rate is higher with GSafe-Adam for all four losses, most clearly for Cal-DPO (+0.056) and SimPO (+0.040). On generation, GSafe-Adam is at least as good as vanilla Adam in every comparison, but the margins are small: the differences in win rate against SFT and in H2H are within one standard deviation for DPO, Cal-DPO and DPOP, and largest for SimPO (0.540 H2H). We therefore claim that the reward-safety property holds under Adam with no cost in preference accuracy or generation quality, and do not claim a significant generation gain in this setting. We have included this table in the revision (Appendix E.7, Table 7) and qualify "across adaptive optimizers" in the main text accordingly (Section 6).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of contributions and revisions

We thank all reviewers for their engagement. For the reviewer–AC discussion, we summarize the contributions and the main changes in the revision.

For quick reference, point-by-point summaries are in the threads of Reviewer ErXd ("Request for clarification") and Reviewer Kxx7 ("Summary of responses to the review").

Contributions

  1. Reward safety as a constraint on the update. GSafePO projects the gradient of a given preference loss so that, to first order on each minibatch, the chosen reward rises and the rejected reward falls by at least a safety margin, with a balance condition that removes their common drift.
  2. Optimizer-aware projection. The projection is computed in the metric of the adaptive optimizer (GSafeRMSProp, GSafeAdam), so the constraints are applied for the step actually taken.
  3. Analysis. We give conditions under which minibatch descent and safety carry over to the population, and prove that the expected squared gradient norm of the population preference loss converges to zero. This question arises whenever reward-change properties are enforced on each minibatch, and to our knowledge it has not been analyzed in preference optimization.
  4. Experiments. Across DPO, Cal-DPO, DPOP and SimPO, GSafePO removes chosen-reward decreases and rejected-reward increases while maintaining or improving preference accuracy, and preserves or improves generation quality.
  5. Practical use. GSafeRMSProp and GSafeAdam are drop-in replacements for RMSProp and Adam: the preference loss is unchanged, the safety margin κ\kappa is the one added hyperparameter, and the same optimizer applies to any pairwise preference loss.

Main changes in the revision

  • Baselines: DPO-PG (App. E.12), DPO+NLL and cross-objective comparisons (App. E.14).
  • Generation evaluation: win rates against SFT with variance, and KL-matched and best-checkpoint comparisons (App. E.13).
  • Theory–experiment link: corollaries for the floor used in experiments (App. C.2.1), the lagged-preconditioner bound and ablation (App. C.1.1), measured noise quantities and direct tests of the safety assumptions (App. E.11), and convergence strengthened from liminf to lim.
  • Ablations: generation metrics for the balance-constraint ablation (App. E.9), and GSafeAdam for four losses with generation evaluation (App. E.7).
  • Claims: generation quality is stated as improved where likelihood displacement occurs and preserved otherwise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.