- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper targets the problem of direct preference alignment algorithm using an iteratively updated reference model. In an idealized setup, where the policy is constrained in selecting an output response from 9 candidate responses, the authors claim that DPO with iteratively updated reference model fails to learn the optimal. To solve this problem, a safegard mechanism is proposed to ensure conservative reference model update.
Critical strengths
- I agree this is an important problem worth serious exploration and study. As the alignment objective consists a term of reward maximization, as well as base model regularization, a more trustworth regularization target is a key in human-llm alignment.
Critical weaknesses
- I found this paper hard to read and follow. I did my best trying to locate the problem dicussed in the abstract, "turn DPO into a persistent reward loss", but i failed. The authors should do a proper rewrite, ensure the claims are properly supported in your main body.
- DPO is within my research area, but I am overloadf by the way how DPO is presented in the paper. For the sake of all readers, i encourage the authors to use conventionally consistent notations and equations.
- The problem is formulated in a simplified, but unrealistic setup. DPO is used to train LLMs, they are auto-regressive generators, certainly not 9-response policy model.
- The theoretical analysis is a big confusion. Your theories and proof should better serve the mainbody. The connection between then remains unclear to me.
- Lack of meaningful empirical analysis.
Questions for authors
- I appreciate you provide a new perspective framing reference update as an outer loop of LLM alignment (DPO). How would you characterize the objective for such an outer loop?
- Will you please further clarify how DPO and IPO differ towards the problem you identified? Are you suggesting IPO is more robust over DPO?
- There are different approaches to updating the reference model, for example direct replacement and EMA. How will different implementation choice affect this identified problem?
2 comments
Thank you for the clarifications and additional numerical results. However, my major concerns remain unresolved.
My primary concern is that the extensive theoretical analysis lacks a clear, coherent presentation connecting the problem formulation, central claims, and proposed methods. Moreover, the additional experiments remain confined to stylized settings and do not sufficiently establish the practical significance of the identified failure mode for LLM alignment.
I therefore maintain my score of 1.
Thank you for flagging the unclear “persistent reward loss” claim. We will lead with this: from the same initial policy, one-step DPO reaches the in-family optimum while 32-step DPO can approach a cycle with lower reward at both phases. Both reference and sampler are refreshed.
W1 / Q1 — Outer objective and reward gap. We evaluate the adopted policy by J(θ)=Eπθr and ask whether J(θn)→J∗=maxθJ(θ). Adoption specifies the next reference and sampler; it does not differentiate another loss. In Theorem 3.1, r(x,z)=−γx2−δz2. At the optimum, Pr(∣x∣=1)=1/2; at either cycle point a±=(±log3,0) it is 5/8, with the same z marginal. Thus
J∗−J(a±)=γ(5/8−1/2)=γ/8≈0.24329.
From the cycle's open basin the limiting gap is zero for DPO-1 and γ/8 for DPO-32, at the same learning rate and scale. The optimum stays locally stable and every frozen inner solve converges. “Persistent” means a limiting deficit, not a decrease at every round. At a+, inner step 32 even lowers both the frozen loss and candidate reward relative to step 31. We will put this calculation and a common-start reward plot before the general theory.
W2 — Conventional notation. We will use the chosen/rejected log-ratio margin m=log[π(yw∣c)/πref(yw∣c)]−log[π(yl∣c)/πref(yl∣c)] and the usual loss −logσ(βm). In our family the partition terms cancel, so βm=(ϕ(yw)−ϕ(yl))⊤w, where w=β(ϑ−θn). Eq. (3) averages the loss over labels. We will distinguish round n, depth K, and step size η; 32 is normalization-specific.
W3 — Relevance to autoregressive models. With full support on the unrestricted finite simplex, exact population DPO gives π+∝πrefer/β and cannot decrease expected reward. Our class cannot represent this tilt: policy log-ratios are affine in (x,z), but the reward is quadratic. Autoregressive factorization alone does not restore closure. We tested two-token responses with prefix dependence, πλ(z∣x)∝e(t+λx)z. W5 reports the result at λ=0.05.
W4 / Q2 — Proof and IPO comparison. Theorem 3.1 proves the cycle: F(a+)=a− and F(a−)=a+, and both axial and transverse derivatives have magnitude below one. Theorem 3.2 explains the loss difference. For u=βm, matched IPO uses (u−2)2/8 (Eq. (4)). It matches DPO's slope and curvature at zero, hence its first update. IPO's gradient remains affine; DPO's is logistic. Their first nonconstant difference, −u4/192, enters the depth-dependent critical cubic in Eq. (16). IPO converges globally here at every fixed finite depth. This is a separation for this protocol, not a general ranking. We will separate existence (3.1), mechanism (3.2), onset (3.3), and intervention (3.4).
W5 — New numerical evidence. After 1,024 population rounds from an 81-point grid on [−1.5,1.5]2, DPO-32 ended near the cycle from 54 starts and near the optimum from 27; DPO-1 and IPO-32 ended near the optimum from all 81. The prefix-dependent model gave the same counts; its alternating policies averaged 0.24047 below the feasible origin. We ran 64 seeds from (1,0.25), drawing one fresh on-policy batch per round and reusing it within the fit. The table gives the mean deficit over rounds 193–256, then mean ± SD across seeds.
The smaller batch reduces the mean DPO-32 gap and increases its variability. We will add the traces and code.
Q3 — Direct replacement versus EMA. If the actor and reference move together, adopting a fraction α of the proposal equals using scale β/α at fixed inner step in w. Thus α=1/2 maps both original cycle points to zero, whereas α=0.999 leaves an attracting cycle since β32/α∈(0.377,0.378) (Corollary B.3).
If only the reference is averaged, draw from the actor, initialize the fit there, adopt the full actor θ+=U(θ,ρ), then update ρ+=μρ+(1−μ)θ+ once per round. The DPO cycle persists for sufficiently small old-reference weight μ>0: at zero lag the two-round map has the original contracting eigenvalues and two zeros. Larger lag can instead destabilize the optimum for both losses. With R=(3/4)32 and g=3(1−R)/(4β32), the axial characteristic polynomial satisfies Pμ(−1)=2−g−μ(g−2R). For (2−g)/(g−2R)<μ<1, an eigenvalue is below −1; the threshold is about 0.006878. We will add the derivation with this sampler and timing. The threshold is model-specific.