- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies undesirable chosen/rejected reward dynamics in preference optimization. Specifically, GSafePO projects the optimizer direction onto a set enforcing balanced chosen-up and rejected-down changes to first order. Methodologically, the projection is derived, and the stochastic sufficient conditions are discussed. Moreover, the RMSProp and Adam variants are evaluated. The optimizer-level treatment is useful, and the empirical results are convincing.
Critical strengths
-
The key idea, i.e., constraining the realized optimizer update, provides a reasonable solution to the gap left by pairwise loss modifications. Besides. the derived projection could be directly connected to the desired first-order reward changes.
-
The experiments are comprehensive, which validate the reward trajectories, generation quality, and projection variants across several losses and model settings.
Critical weaknesses
-
There seems to be a gap between the population theorem and the implementation. Specifically, Theorem 4.1 assumes population alignment and safety with second-order control. However, the minibatch diagnostics do not verify the noise conditions of Propositions 4.1 and 4.2. Appendix C changes the safety floor, uses current-batch preconditioning and allows an uncorrected first step. Moreover, gradient convergence is weaker than RSTP. Thus, the population claim needs further refinement.
-
The empirical validations could be improved. Specifically, the centered-versus-quadrant ablation measures reward movement, while it does not consider the generation quality; the larger reward separation alone also does not justify the additional balance constraint.
Questions for authors
Q1. Which guarantees remain valid for the exact implementation, including the modified floor and first-step exception in Appendix C? Could the population noise and margin conditions be verified on independent batches?
Q2. Can the projected method converge at a point with a nonzero loss gradient due to the vanished feasible direction? What conditions exclude this case for the implemented losses, beyond assuming uniform descent alignment?
Q3. For the centered and quadrant projections, could generation quality be compared at matched update norm or policy divergence?
9 comments
W1 (part 2/3): Connecting the theory to the experiments — Current batch preconditioning, first step handling
..uses current-batch preconditioning and allows an uncorrected first step. Thus, the population claim needs further refinement.
Current-batch preconditioning. Our experiments use the current-batch preconditioner because it is the standard form of RMSProp: the second moment is updated with the current gradient before the step is scaled, as in torch.optim.RMSprop, which GSafeRMSProp extends.
Prop. 4.1 assumes zero-mean noise; that assumption transfers to the optimizer’s metric when is -measurable, which RMSProp satisfies with a one-step-lagged second-moment preconditioner. Such a lagged preconditioner is therefore a departure from the default optimizer, introduced only so that Proposition 4.1 applies in the optimizer's metric: its zero-mean noise assumption, , carries over to the preconditioned gradient only if is -measurable.
Our experiments use the current-batch preconditioner, so the revision quantifies the gap in two ways:
-
Theory. Proposition C.1 (Appendix C.1.1 of the revision) shows that, on every coordinate and at every step, the current-batch preconditioner is at most times the lagged one, i.e. at most about 5% larger for our .
-
Experiment. A new ablation in Appendix C.1.1 checks that our results do not depend on this difference: we re-ran DPO-S (Llama-3.2-1B / RewardBench-v2) with the one-step-lagged preconditioner , obtained by swapping lines 1 and 2 of Algorithm 1 for . All other settings are unchanged.
| Preconditioner | Pref. WR | Gen. WR vs SFT | ||
|---|---|---|---|---|
| Current-minibatch (as implemented) | 0.354 | −1.183 | 0.772 | 0.742 |
| Lagged (lines 1 and 2 swapped, ) | 0.291 | −3.811 | 0.769 | 0.754 |
The variant with the -measurable preconditioner preserves the behaviour of the implemented one: the chosen reward stays positive, the rejected reward stays negative, and preference win rate is essentially unchanged (0.769 vs 0.772). Generation win rate against SFT is slightly better with the lagged variant (0.754 vs 0.742), and it lowers the rejected reward further. Proposition 4.1, and through it Assumption 2 of Theorem 4.1, therefore applies to an optimizer with the same reward-safe dynamics as the one we evaluated, at no cost in preference accuracy or generation quality.
Uncorrected first step. The reviewer is right that the very first update is applied without the projection; we use it because the second moment starts at zero, which otherwise causes early-step blow-ups. This step is best read as part of the initialization: our analysis starts from , and every subsequent update is projected, so the guarantees of Section 4 hold for all . The first step has also populated the second moment, so the optimizer state at depends only on the first minibatch, as the stochastic analysis requires. The cost is one plain RMSProp step of size that carries no reward-safety guarantee. We now state this exception explicitly in Appendix C.2.2.
W1 (part 3/3): Strengthened convergence result (Theorem 4.1(a): to )
Moreover, lim inf gradient convergence is weaker than RSTP. Thus, the population claim needs further refinement.
lim inf vs RSTP: We agree. RSTP is posed for a deterministic sequence, whereas Theorem 4.1 concerns stochastic minibatch iterates, so its conclusions are in expectation over the trajectory. Within that setting the submitted statement was weaker than necessary: the revision strengthens Theorem 4.1(a) to under the same assumptions, using the up-crossing argument of Bertsekas & Tsitsiklis (2000) (new Lemmas A.1–A.2, App. A.5). We also add a sentence after (RSTP) stating that for a stochastic algorithm its conditions are required in expectation over the trajectory, so the problem statement and Theorem 4.1 now match.
W1 (part 1/3): Connecting the theory to the experiments — minibatch diagnostics and the modified safety floor
We thank the reviewer for the close reading of the theory and for identifying precisely where the analysis and the implementation part ways. Each of the four gaps the reviewer lists is addressed in the revision: Theorem 4.1(a) is strengthened from a lim inf to a limit, the modified floor now has its own norm and alignment bounds (Corollaries C.1 and C.2), the current-batch preconditioner is bounded against the lagged one and compared with it empirically (Appendix C.1.1), and the uncorrected first step is stated explicitly. We have also measured the noise quantities of Proposition 4.2 and tested Assumptions 3 and 4 on the steps actually taken (Appendix E.11), and extended the centered-versus-quadrant ablation to generation quality at matched divergence (Table 10). Detailed responses follow.
Minibatch diagnostics verification. The revision adds measurements of the noise quantities of Prop. 4.2 and realized tests of Assumptions 3 and 4 of Theorem 4.1 (Appendix E.11); please see our responses to Reviewer Kxx7, Q3 and W2(b).
For Prop. 4.1, the condition for population descent is 1/ρmax2>(C/c)2−1, where C bounds the norm ratio ∥g^∥/∥g∥ and c is the alignment constant. Both constants are certified by Corollaries C.1 and C.2 and confirmed by the projection diagnostics (Appendix E.11, Fig. 8):
Norm. ∥g^∥/∥g∥≤1 throughout, so C=1.
Alignment, certified. Corollary C.2 gives c≥(1+κ)πmin−κ with πmin the smallest value of ∥g⊥∥2/∥g∥2. The diagnostics show ∥g⊥∥/∥g∥≥0.82, so πmin=0.67 and, at κ=0.4, c≥0.54.
Alignment, observed. The measured ⟨g,g^⟩/∥g∥2 is at least 0.83.
With C=1 the condition becomes 1/ρmax2>1/c2−1. Here 1/ρ2 is a lower bound on the signal-to-noise ratio (SNR) of the minibatch loss gradient, so the condition says how much gradient noise the method can tolerate. Using only the alignment that Corollary C.2 guarantees (c=0.54), the SNR must exceed about 2.4: the signal must be at least about 2.4 times as strong as the noise. Using the alignment we observed (c=0.83), an SNR of about 0.45 is enough: descent holds even when the noise is about twice as strong as the signal.
Safety floor used in experiments. The modified floor used in our experiments (Appendix C.2.1) is covered by the theory. Corollaries C.1 and C.2 (revised) are counterparts of Corollaries 3.1 and 3.2 and supply the norm and alignment constants required by Proposition 4.1; Sections 4 and 6 and Appendix E.11 now refer to them for the floor used in our experiments:
Norm (C.1). ∥g^∥≤max{1,κ}∥g∥, with no eigenvalue bounds on M needed; the update is non-expansive for κ≤1.
Alignment (C.2). ⟨g,g^⟩≥((1+κ)πmin−κ)∥g∥2 whenever ∥g⊥∥2/∥g∥2≥πmin, for any in-span orientation. This gives a positive alignment constant c, as Prop. 4.1 requires, when πmin>κ/(1+κ), i.e. ∥g⊥∥/∥g∥>0.535 at our κ=0.4; the observed value is at least 0.82 (Fig. 8).