- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies online preference learning with GLMs, including Elo and Bradley-Terry models. The authors analyze the last-iterate convergence of SGD with constant learning rate using existing theoretical results. They also study different types of monotonicity and identify when they hold or fail depending on the geometry and dimension of feature vectors. Synthetic experiments are provided to support the theoretical results on convergence and monotonicity.
Critical strengths
[S1] Clear theoretical analysis of monotonicity.
The paper provides geometric conditions for monotonicity under online SGD. The impossibility result and high-probability guarantee for random features help explain when gap monotonicity can hold. I think these are the main theoretical contributions of this paper.
Critical weaknesses
[W1] Limited novelty compared to previous theoretical results.
I'm not sure that the novelty of theoretical analysis is enough for acceptance. Bareilles et al. (2025, Theorem 5) already provide similar gradient-based conditions for monotonicity, and the convergence guarantees in this paper are based on the existing SGD results. Although the paper includes additional geometric results and analyzes SGD trajectories, I believe the authors should better explain what is new compared to previous work.
[W2] Limited connection to practical LLM preference learning.
The paper motivates its analysis with RLHF and DPO, but the theoretical framework assumes fixed linear feature representations and i.i.d. observations. Also, all experiments are synthetic, so it is difficult to see how the theoretical assumptions apply to actual LLM preference fine-tuning. Along with this, I think the practical relevance of the theoretical findings is somewhat limited.
Questions for authors
[Q1] Novelty compared to previous monotonicity results
Theorem 1 seems closely related to the gradient-based monotonicity conditions in Bareilles et al. (2025, Theorem 5). Could you explain the main theoretical differences between these results? In particular, what is new beyond applying the existing conditions to linear feature representations? I believe it helps to clarify the novelty of this paper.
[Q2] Applicability to RLHF and DPO
In line with W2, could you explain under what assumptions DPO fine-tuning fits into this framework? Also, which monotonicity guarantees would still hold in this setting?
3 comments
[W1, second half.] [...] and the convergence guarantees in this paper are based on the existing SGD results.
We agree, and the paper says so (Introduction, second contribution): Propositions 1 and 2 invoke recent results on constant-step SGD (Attia et al., 2025; Cortild et al., 2026). Our contribution there is to show that GLMs fit their hypotheses, with constants read off the CGF of the base distribution and the second-moment matrix of the features.
The revision goes further. These results require every sample loss to be uniformly smooth, which forces bounded features (). The new Appendix D removes this restriction in two regimes:
- Strongly convex CGF, (Proposition 4). We prove that GLM sample gradients satisfy an expected co-coercivity inequality (Lemmas 8–9), which is all the proof of Proposition 1 needs. Its bound then holds with replaced by an averaged constant .
- Bounded outcomes, (Proposition 5). Stochastic gradients are then uniformly in the parameter (Lemma 10), which yields counterparts of both bounds of Proposition 2, with a uniform noise level in place of .
Convergence and monotonicity guarantees thus now apply in the same feature regime. The main contribution of the paper nonetheless remains the monotonicity analysis discussed above.
[Q2.] Applicability to RLHF and DPO: In line with W2, could you explain under what assumptions DPO fine-tuning fits into this framework? Also, which monotonicity guarantees would still hold in this setting?
Our general response (section "Relevance to RLHF and DPO") answers both parts. In short:
- Assumptions. DPO is exactly our update when the policy is log-linear in fixed features, training starts from the reference policy, and plain SGD processes one pair per step. A DPO step is then our Bradley–Terry SGD step, with features and outcome .
- Monotonicity guarantees. All of them hold: the DPO margin always increases (weak gap monotonicity), and Theorem 1, Corollary 1, Theorem 2 and Proposition 7 apply as stated. In particular, DPO never lowers the probability of the preferred response if and only if the features are gap monotone. These results concern a single update, so they need neither i.i.d. data nor a well-specified model.
- Convergence guarantees may not apply: DPO may lack a minimizer.
On fixed features (W2): for a trained network, Theorem 1 still describes one small step, to first order, with the gradients of the scores as features. Razin et al. (2024) find that the inner products of Theorem 1, computed on hidden embeddings, predict likelihood displacement in real models.
We hope these answers clarify the novelty of our results and their relevance to DPO. If they address your concerns, we would be grateful if you would consider raising your score. We are happy to discuss any remaining point.
Thank you for the detailed rebuttal.
I believe it helps to clarify on the theoretical contributions, particularly the global characterization of monotonicity, trajectory-level analysis, and the new connection to likelihood displacement (W1).
I also appreciate the clarification of the assumptions under which the results apply to DPO (W2). Although the applicability to practical LLM fine-tuning remains limited, I believe the theoretical contributions are still meaningful.
Therefore, I have decided to raise my score from 2 to 3.
We thank the reviewer for their review, and for finding our analysis of monotonicity clear. We answer each point below; the revised PDF shows all changes in blue.
Theorem 1 and Bareilles et al. (2025, Theorem 5) are indeed related: both express monotonicity through gradient inner products. They differ in setting and in what they establish.
Bareilles et al. (2025) handle general, possibly non-linear and non-convex scores, and give sufficient conditions that are local: they hold at a fixed θ, for a step size ε≤ε0 with ε0 not quantified. In that generality, nothing sharper can be expected.
In our setting, the feature representation is linear and the SGD update is explicit. This yields:
The monotonicity identity itself is elementary: it can be read directly off the update rule, and we do not claim technical difficulty. The value lies in the exact and global characterization that this simplicity enables, and in its geometric consequences. Our result is not a corollary of Theorem 5 of Bareilles et al. (2025), which is local and sufficient, and does not give these conditions.
We have added an explicit comparison with Bareilles et al. (2025) in the related work (Appendix A), stating that Theorem 1 follows directly from the update rule. We will also make clear which results are new and which are standard.