acceptodds
Under review as a conference paper at ICLR 2027

More Optimization, Worse Alignment: Cycles and Safeguards in Iterative DPO

Abstract

Iterative preference learning fits a policy, adopts it as the next reference, and uses it to generate new comparisons. Better fitting appears desirable, but adoption changes the next optimization problem. We show that this feedback can turn deeper direct preference optimization (DPO) into a persistent reward loss. Under a fixed Bradley–Terry reward, we construct a regular nine-response policy family where one-step DPO converges globally, whereas 32-step DPO admits an attracting suboptimal two-cycle. Every frozen inner problem has globally convergent gradient descent, and the unique outer fixed point remains locally stable and reward-optimal within the family. Matched IPO converges globally at the same depth, scale, and learning rate, despite agreeing with DPO through second order at the optimum. We explain the separation by connecting the fourth loss derivative, comparison moments, and inner depth to the leading possible cubic difference. Even an arbitrarily small logistic component in an IPO mixture can produce cycles, with model-dependent depth and coexistence windows. A certified fifth-order analysis quantifies cycle onset and the GD step sizes needed to preserve coexistence. Finally, an IPO-envelope safeguard restores convergence in the product model by controlling the adopted displacement, while allowing arbitrary inner depths and exact minimization. Together, these results show why reliable iterative alignment requires control of policy adoption beyond accurate preference fitting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Direct Preference Optimization, Iterative Preference Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dPNfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper targets the problem of direct preference alignment algorithm using an iteratively updated reference model. In an idealized setup, where the policy is constrained in selecting an output response from 9 candidate responses, the authors claim that DPO with iteratively updated reference model fails to learn the optimal. To solve this problem, a safegard mechanism is proposed to ensure conservative reference model update.

Critical strengths
  1. I agree this is an important problem worth serious exploration and study. As the alignment objective consists a term of reward maximization, as well as base model regularization, a more trustworth regularization target is a key in human-llm alignment.
Critical weaknesses
  1. I found this paper hard to read and follow. I did my best trying to locate the problem dicussed in the abstract, "turn DPO into a persistent reward loss", but i failed. The authors should do a proper rewrite, ensure the claims are properly supported in your main body.
  2. DPO is within my research area, but I am overloadf by the way how DPO is presented in the paper. For the sake of all readers, i encourage the authors to use conventionally consistent notations and equations.
  3. The problem is formulated in a simplified, but unrealistic setup. DPO is used to train LLMs, they are auto-regressive generators, certainly not 9-response policy model.
  4. The theoretical analysis is a big confusion. Your theories and proof should better serve the mainbody. The connection between then remains unclear to me.
  5. Lack of meaningful empirical analysis.
Questions for authors
  1. I appreciate you provide a new perspective framing reference update as an outer loop of LLM alignment (DPO). How would you characterize the objective for such an outer loop?
  2. Will you please further clarify how DPO and IPO differ towards the problem you identified? Are you suggesting IPO is more robust over DPO?
  3. There are different approaches to updating the reference model, for example direct replacement and EMA. How will different implementation choice affect this identified problem?
2 comments

Thank you for flagging the unclear “persistent reward loss” claim. We will lead with this: from the same initial policy, one-step DPO reaches the in-family optimum while 32-step DPO can approach a cycle with lower reward at both phases. Both reference and sampler are refreshed.

W1 / Q1 — Outer objective and reward gap. We evaluate the adopted policy by J(θ)=EπθrJ(\theta)=\mathbb E_{\pi_\theta}r and ask whether J(θn)→J∗=max⁡θJ(θ)J(\theta_n)\to J_*=\max_\theta J(\theta). Adoption specifies the next reference and sampler; it does not differentiate another loss. In Theorem 3.1, r(x,z)=−γx2−δz2r(x,z)=-\gamma x^2-\delta z^2. At the optimum, Pr⁡(∣x∣=1)=1/2\Pr(|x|=1)=1/2; at either cycle point a±=(±log⁡3,0)a_\pm=(\pm\log3,0) it is 5/85/8, with the same zz marginal. Thus

J∗−J(a±)=γ(5/8−1/2)=γ/8≈0.24329.J_*-J(a_\pm)=\gamma(5/8-1/2)=\gamma/8\approx0.24329.

From the cycle's open basin the limiting gap is zero for DPO-1 and γ/8\gamma/8 for DPO-32, at the same learning rate and scale. The optimum stays locally stable and every frozen inner solve converges. “Persistent” means a limiting deficit, not a decrease at every round. At a+a_+, inner step 32 even lowers both the frozen loss and candidate reward relative to step 31. We will put this calculation and a common-start reward plot before the general theory.

W2 — Conventional notation. We will use the chosen/rejected log-ratio margin m=log⁡[π(yw∣c)/πref(yw∣c)]−log⁡[π(yl∣c)/πref(yl∣c)]m=\log[\pi(y_w|c)/\pi_{\mathrm{ref}}(y_w|c)]-\log[\pi(y_l|c)/\pi_{\mathrm{ref}}(y_l|c)] and the usual loss −log⁡σ(βm)-\log\sigma(\beta m). In our family the partition terms cancel, so βm=(ϕ(yw)−ϕ(yl))⊤w\beta m=(\phi(y_w)-\phi(y_l))^\top w, where w=β(ϑ−θn)w=\beta(\vartheta-\theta_n). Eq. (3) averages the loss over labels. We will distinguish round nn, depth KK, and step size η\eta; 32 is normalization-specific.

W3 — Relevance to autoregressive models. With full support on the unrestricted finite simplex, exact population DPO gives π+∝πrefer/β\pi^+\propto\pi_{\mathrm{ref}}e^{r/\beta} and cannot decrease expected reward. Our class cannot represent this tilt: policy log-ratios are affine in (x,z)(x,z), but the reward is quadratic. Autoregressive factorization alone does not restore closure. We tested two-token responses with prefix dependence, πλ(z∣x)∝e(t+λx)z\pi^\lambda(z|x)\propto e^{(t+\lambda x)z}. W5 reports the result at λ=0.05\lambda=0.05.

W4 / Q2 — Proof and IPO comparison. Theorem 3.1 proves the cycle: F(a+)=a−F(a_+)=a_- and F(a−)=a+F(a_-)=a_+, and both axial and transverse derivatives have magnitude below one. Theorem 3.2 explains the loss difference. For u=βmu=\beta m, matched IPO uses (u−2)2/8(u-2)^2/8 (Eq. (4)). It matches DPO's slope and curvature at zero, hence its first update. IPO's gradient remains affine; DPO's is logistic. Their first nonconstant difference, −u4/192-u^4/192, enters the depth-dependent critical cubic in Eq. (16). IPO converges globally here at every fixed finite depth. This is a separation for this protocol, not a general ranking. We will separate existence (3.1), mechanism (3.2), onset (3.3), and intervention (3.4).

W5 — New numerical evidence. After 1,024 population rounds from an 81-point grid on [−1.5,1.5]2[-1.5,1.5]^2, DPO-32 ended near the cycle from 54 starts and near the optimum from 27; DPO-1 and IPO-32 ended near the optimum from all 81. The prefix-dependent model gave the same counts; its alternating policies averaged 0.240470.24047 below the feasible origin. We ran 64 seeds from (1,0.25)(1,0.25), drawing one fresh on-policy batch per round and reusing it within the fit. The table gives the mean deficit over rounds 193–256, then mean ±\pm SD across seeds.

Pairs/roundDPO-1DPO-32Matched IPO-32
4,0960.00026±0.000120.00026\pm0.000120.1178±0.07860.1178\pm0.07860.0223±0.00960.0223\pm0.0096
65,5360.000022±0.0000200.000022\pm0.0000200.2343±0.01840.2343\pm0.01840.0031±0.00240.0031\pm0.0024

The smaller batch reduces the mean DPO-32 gap and increases its variability. We will add the traces and code.

Q3 — Direct replacement versus EMA. If the actor and reference move together, adopting a fraction α\alpha of the proposal equals using scale β/α\beta/\alpha at fixed inner step in ww. Thus α=1/2\alpha=1/2 maps both original cycle points to zero, whereas α=0.999\alpha=0.999 leaves an attracting cycle since β32/α∈(0.377,0.378)\beta_{32}/\alpha\in(0.377,0.378) (Corollary B.3).

If only the reference is averaged, draw from the actor, initialize the fit there, adopt the full actor θ+=U(θ,ρ)\theta^+=U(\theta,\rho), then update ρ+=μρ+(1−μ)θ+\rho^+=\mu\rho+(1-\mu)\theta^+ once per round. The DPO cycle persists for sufficiently small old-reference weight μ>0\mu>0: at zero lag the two-round map has the original contracting eigenvalues and two zeros. Larger lag can instead destabilize the optimum for both losses. With R=(3/4)32R=(3/4)^{32} and g=3(1−R)/(4β32)g=3(1-R)/(4\beta_{32}), the axial characteristic polynomial satisfies Pμ(−1)=2−g−μ(g−2R)P_\mu(-1)=2-g-\mu(g-2R). For (2−g)/(g−2R)<μ<1(2-g)/(g-2R)<\mu<1, an eigenvalue is below −1-1; the threshold is about 0.0068780.006878. We will add the derivation with this sampler and timing. The threshold is model-specific.

Thank you for the clarifications and additional numerical results. However, my major concerns remain unresolved.

My primary concern is that the extensive theoretical analysis lacks a clear, coherent presentation connecting the problem formulation, central claims, and proposed methods. Moreover, the additional experiments remain confined to stylized settings and do not sufficiently establish the practical significance of the identified failure mode for LLM alignment.

I therefore maintain my score of 1.

Reviewer_FBDsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies iterative preference learning under a fixed Bradley–Terry reward, with both reference and sampling policies refreshed between rounds. In a restricted nine-response family, one-step DPO and appropriately normalized IPO at every fixed finite depth converge globally to the within-family reward optimum, whereas 32-step DPO admits an attracting suboptimal two-cycle under the specified common hyperparameters. The optimal fixed point remains locally stable (Theorem 3.1). The paper analyzes the higher-order mechanism and develops an IPO-envelope safeguard for the specified product model.

Critical strengths

A controlled separation between inner and outer convergence. The construction rules out preference intransitivity and unstable frozen optimization as explanations of the cycle. Despite a locally stable reward-optimal fixed point, DPO-32 has an attracting two-cycle with an open basin in R² and an exact reward gap of γ/8 at each phase relative to the within-family optimum (Theorem 3.1).

Mechanistic analysis and a constructive safeguard. DPO-32 and normalized IPO-32 agree through second order at the optimum, yet exhibit different outer dynamics. The fourth-derivative/comparison-moment analysis explains the leading possible cubic difference, while fifth-order analysis quantifies a local fold of two-cycles. The safeguard restores convergence for unbounded, history-dependent inner depths, including exact minimization, in the specified product model with population losses and under the stated parameter conditions (Theorems 3.1–3.4).

Critical weaknesses

Empirical relevance to neural alignment is not evaluated in this paper. The manuscript reports no neural-policy training experiments; its figures and Theorem B.14 concern finite-response models. The safeguard’s convergence guarantee relies on population moments and product structure, and the authors explicitly leave its extension to empirical estimates and general features open. This does not invalidate the theoretical counterexample, but leaves its prevalence and practical significance in LLM training empirically unestablished. A small controlled neural-policy study could strengthen this connection.

Questions for authors

What testable prediction would distinguish this refresh-driven mechanism from finite-sample overfitting or larger policy updates in neural training? Could a small LLM study examine it using refresh/no-refresh and update-KL controls, a fixed label source, and an independent fixed evaluator treated as a reward proxy?

3 comments

Thank you for distinguishing two questions: can reference and sampler refresh cause a persistent reward loss, and does this happen in neural alignment? The manuscript answers the first by construction. It does not establish the second. For this response we ran a controlled neural-policy study. The diagnostic detects a reversal in the paper's example; the neural study did not reproduce a sustained two-cycle. We explain what would count as positive evidence, what we observed, and how to carry the test over to a small pretrained language model.

1. A prediction that a falling reward curve alone does not make. Consider two copies of the same actor, immediately after the same update. In one copy refresh both reference and sampler to that actor; in the other keep their immediately preceding values for one more fit. Start both fits from the current actor. A prediction of the mechanism near the constructed orbit is that the refreshed fit reverses the preceding policy motion, while the unrefreshed fit continues in the old direction. This is a difference in direction at the same state; one can test it at the same adopted update size.

The construction gives a numerical positive control for this test. Write a=log⁡3a=\log 3 and suppress the invariant second coordinate. After a depth-32 step from +a+a to −a-a, joint refresh gives the next proposal +a+a. With reference and sampler instead held at their preceding values, 32 more steps continue the same frozen population loss: the next proposal is a−v64/β32≃−1.11398<−aa-v_{64}/\beta_{32}\simeq-1.11398<-a, using the recurrence in Eq. (9). Interpolating along each proposal from the common actor −a-a to the same forward KL of 10−510^{-5} nats gives signed displacements +0.007294+0.007294 and −0.007312-0.007312, respectively. The comparison uses exact preference expectations, so neither a finite batch nor a larger adopted KL explains this sign difference. It is a one-step intervention, not evidence that a full cycle survives damping to this KL.

There is an equally useful limit to the prediction. On the invariant axis the policy has log-partition A(s)=log⁡(2+2cosh⁡s)A(s)=\log(2+2\cosh s), and

∂∂tDKL(pt∥ps)=(t−s)A′′(t).\frac{\partial}{\partial t}D_{\mathrm{KL}}(p_t\Vert p_s)=(t-s)A''(t).

Consequently, two proposals from ss on the same ray give the same accepted policy if interpolated to the same attainable KL. In particular, matching shallow and deep proposals can remove a depth effect mediated by update size. We are not predicting that all deep DPO runs must remain worse after KL matching. The empirical question is whether a refresh-specific directional effect remains, and whether it recurs often enough to depress reward.

2. What we tested in a neural policy. We trained all 1,036 parameters of a single-head causal-attention decoder. Given a prompt (a,b,c)(a,b,c) over four symbols, it generates three tokens, hence 64 possible completions per prompt. The 64 prompts were split into 32 for training, 16 for KL calibration, and 16 for evaluation. The split, label source, evaluator, depths, and stopping round were fixed across arms. Bradley–Terry labels came from a fixed partial-credit utility

+0.65\,\mathbf 1[y_3=y_1]+0.15\,\mathbf 1[y_1=y_2].$$ The evaluator used a separate rule: $e(x,y)=\mathbf 1[y=(a,b,a)]$. It never supplied a pair label, selected a checkpoint, or set the KL. We report its **expected pass rate** $S_E(\pi)=\mathbb E_{x\sim P_E}\mathbb E_{y\sim\pi(\cdot\mid x)}[e(x,y)]$, with $P_E$ uniform on the fixed evaluation prompts, summing exactly over completions. This is a task-specific reward proxy, not a human alignment score. From each of six paired initializations we compared reference refresh on/off, sampler refresh on/off, and $K\in\{1,8\}$, for 24 rounds. A frozen sampler meant fresh draws from its fixed law, not replay of one dataset. We ran exact population fits and fits with 4,096 newly labeled comparisons per round, reused within the fit. Here “population” integrates response pairs and labels on the training prompts; it does not eliminate generalization error across prompts. The matched arms shared the same **realized** forward KL from their respective previous actors on calibration prompts in each round. We interpolated each proposal to a common attainable value, never enlarging a weak proposal. Maximum between-arm KL mismatch on calibration prompts was below $10^{-9}$ nats per sequence; evaluation prompts were left untouched. Median between-arm KL ranges on those untouched prompts were $8.0\times10^{-7}$ (population) and $2.2\times10^{-6}$ (sampled). Let $S_{rs,K}$ be the mean pass rate over rounds 13–24, with $(r,s)=(1,1)$ denoting both refreshes and $(0,0)$ neither. The depth-by-refresh contrast is $$I_E=(S_{11,8}-S_{11,1})-(S_{00,8}-S_{00,1}).$$ The four score columns below are percentages. The final column is the paired contrast in **percentage points**, with a 95% $t$ interval across the six seed-level contrasts, conditional on the fixed evaluation prompts. The single-refresh arms were also run in the primary experiment. | Training and adoption | Neither, $K=1$ | Neither, $K=8$ | Both, $K=1$ | Both, $K=8$ | $I_E$ [95% interval], pp | |---|---:|---:|---:|---:|---:| | Population, full proposal | 2.31 | 7.18 | 4.07 | 22.33 | $+13.39$ [$-1.03$, $27.81$] | | Population, matched KL | 2.30 | 2.39 | 2.09 | 2.25 | $+0.060$ [$-0.015$, $0.134$] | | Sampled labels, matched KL | 2.25 | 2.50 | 2.33 | 2.63 | $+0.037$ [$-0.013$, $0.086$] | | Already-trained start, population, matched KL | 29.93 | 30.33 | 30.04 | 30.30 | $-0.144$ [$-1.087$, $0.799$] | Under full adoption, deeper training **improved** the observed score in both principal arms. At depth eight, refreshing only the sampler or only the reference gave 10.52% and 27.11%, respectively, with unequal natural step sizes. The primary matched population experiment had a small median attainable KL ($3.24\times10^{-5}$ nats per sequence) and low absolute scores. We therefore repeated the four principal arms from each seed's actor after 24 training rounds, resetting both controls to that common checkpoint. Mean starting pass rate was 28.63%; the last row reports this sensitivity check. Its interval includes effects of either sign. These contrasts are not an equivalence test. We also tracked the policies themselves, since two different policies can have the *same* reward. On the fixed evaluation prompts let $$d_j(t)=\left(\mathbb E_x\, H^2\bigl(\pi_t(\cdot\mid x),\pi_{t-j}(\cdot\mid x)\bigr)\right)^{1/2}, \qquad H^2(p,q)=\tfrac12\sum_y(\sqrt p-\sqrt q)^2.$$ An exact two-cycle has $d_2=0$ and $d_1>0$; an almost stationary policy can have *both* near zero. We counted near alternation only if, over the last 12 rounds, $\overline{d}_1\ge10^{-3}$ and $\overline{d}_2/\overline{d}_1<1/2$. No trajectory met both conditions, with or without KL matching, including the already-trained starts. At rounds 6, 12, and 18 we also performed the common-actor intervention above on the six full-adoption deep trajectories. Only **one of 18** paired interventions reversed the previous motion under refresh while continuing it under no refresh, measured by the sign of the inner product between consecutive changes in $\sqrt{\pi}$ on evaluation prompts. The count remained one when each pair used 95% of its smaller proposed KL, spanning $0.00050$–$0.01267$ nats. These are repeated probes of six runs, not 18 independent replicates. An isolated reversal is not a persistent cycle.

3. A feasible small-LLM test and what would distinguish the alternatives. Yes. We propose starting all arms of each seed from the same pretrained instruction model (for example, Qwen2.5-0.5B-Instruct), with identical trainable adapters, optimizer-state handling, decoding law, prompt allocation, and comparison budget. We would use the same reference-by-sampler factorial design, two inner depths, and multiple batch sizes. The frozen arms would also regenerate comparisons each round. A fixed preference judge would label training pairs. A different, fixed evaluator—for example, sealed executable unit tests on held-out programming prompts—would measure task success and would never enter training, step-size selection, or early stopping. Its score would remain a proxy. We would align comparisons by outer round and also report them against total optimizer steps.

We would run both natural adoption and KL-matched adoption. The quantity to match is Ex∈CDKL(πt+1(⋅∣x)∥πt(⋅∣x))\mathbb E_{x\in C}D_{\mathrm{KL}}(\pi_{t+1}(\cdot\mid x)\Vert\pi_t(\cdot\mid x)) on a separate calibration panel CC, not KL to the DPO reference. For language outputs we would estimate sequence KL with fresh samples from the proposed accepted policy and track output length. We would report realized KL and its uncertainty and identify targets outside common support. On held-out prompts, the policy-distance diagnostic also needs no enumeration: for m=(p+q)/2m=(p+q)/2, H2(p,q)=1−Ey∼m[2p(y)q(y)/(p(y)+q(y))]H^2(p,q)=1-\mathbb E_{y\sim m}[2\sqrt{p(y)q(y)}/(p(y)+q(y))], estimable from sampled sequences and their log-probabilities.

The strongest evidence would be joint: a repeated return toward the policy from two rounds ago; proxy scores below the shallow refreshed comparison at both phases; and an adverse depth-by-refresh contrast under comparable update KL, reproducible across seeds and larger fresh batches. We would check fresh-pair versus training preference loss at the same frozen reference and sampling law. A deficit confined to a reused small batch, with a widening validation gap and attenuation on fresh or larger batches, favors finite-sample overfitting. A deficit under full adoption that vanishes at matched KL is consistent with an update-size explanation or mediation through update size. A common-checkpoint reversal at equal KL tests for a directional effect beyond that scalar explanation; it does not by itself establish a reward cycle. The neural pilot above gives no evidence for the full signature. Its 24 rounds, single synthetic task, and small decoder without language pretraining do not rule out a different outcome at other depths or in language models.

4. Scope of the safeguard and proposed revisions. We agree that Theorem 3.4 uses population moments and product structure. Theorem B.14 tracks a cycle near its basin over finite horizons in a growing-batch limit; it neither estimates prevalence nor validates an empirical safeguard. Within the same nine-response product model, one can replace population moments by estimates with checks: invert only when a confidence lower bound for λmin⁡(M)\lambda_{\min}(M) is positive, certify coordinate signs and minimum accepted progress, and otherwise obtain more comparisons or use a conservative IPO step as appropriate. With adaptive batches this yields high-probability convergence under the model's assumptions (Proposition C.8). This does not justify a raw plug-in rule, arbitrary neural features, or the safeguard's practical superiority. The neural experiments tested the failure mechanism, not that safeguard. We will separate these guarantees in the text, add the controlled study including its negative result, and reserve claims about pretrained LLMs for a direct test.

Reviewer_FVcCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies iterative preference optimization, where the updated policy determines both the reference policy and the comparison distribution in the next round.

Its main result is a controlled counterexample: one-step DPO converges globally, while deeper DPO admits an attracting, suboptimal two-cycle, despite convergent inner optimization and a locally stable, reward-optimal fixed point. Matched IPO converges globally in the same setting.

The paper connects this separation to higher-order loss derivatives and optimization depth, then characterizes cycle onset, coexistence windows, reward gaps, and discretization requirements.

Critical strengths

The main counterexample isolates a meaningful failure mechanism. Fixed Bradley–Terry preferences and convergent inner solvers rule out preference intransitivity and inner optimization instability as explanations for the cycle.

The higher-order analysis goes beyond exhibiting a counterexample. Connecting loss derivatives, comparison moments, and inner depth explains how procedures that agree through second order can have different long-run behavior.

The construction is supported carefully. Exact rational certificates and perturbation arguments provide stronger support than numerical trajectories alone, and the paper explicitly distinguishes its guarantees from broader claims about practical DPO training.

Critical weaknesses

The broader significance of the construction remains uncertain. The paper establishes that the failure is possible, but offers limited guidance on identifying it in more general settings.

Much of the technical development refines the same specialized phenomenon. The fifth-order expansions, mixture limits, and discretization bounds add mathematical precision, but their additional conceptual payoff is less clear.

The robustness results leave the extent of the failure difficult to assess. These results do not establish how readily the cycle arises under less structured models or sustained stochastic training.

The safeguard has limited demonstrated algorithmic value. Its guarantee depends on product structure and population moments, while direct IPO and sufficiently small constant damping already converge in the same model.

Questions for authors

Which conclusions from the higher-order analysis can be used to diagnose cycling outside the constructed families? In particular, what can be inferred when there is no reflection symmetry or invariant scalar direction?

Can the existing perturbation argument yield an explicit bound for a perturbation that breaks product structure?

Does the safeguard retain any convergence guarantee under bounded errors in the estimated comparison moments?

5 comments

Thank you for asking which parts of the construction can actually be used in another model. We will organize the revision around three different questions.

W1 / Q1. What survives without reflection symmetry or an invariant scalar direction? First differentiate the whole adopted round FF, including the change of reference and sampling law. The Hessian of the frozen loss alone is not its Jacobian. Consider a smooth one-parameter family of outer maps, translated along a branch of fixed points. At a critical parameter write

F(x)=Ax+12B(x,x)+16C(x,x,x)+O(∥x∥4).F(x)=Ax+\tfrac12 B(x,x)+\tfrac16 C(x,x,x)+O(\|x\|^4).

Assume Aq=−qAq=-q, p⊤A=−p⊤p^\top A=-p^\top, ∥q∥2=1\|q\|_2=1, p⊤q=1p^\top q=1, the multiplier −1-1 is simple and crosses transversely, and the other multipliers have modulus below one. The center manifold need not be a straight line. Parameterize it by x(u)=qu+h2u2+h3u3+⋯x(u)=qu+h_2u^2+h_3u^3+\cdots, choosing the scalar coordinate to remove its nonresonant quadratic term. Substitution in F(x(u))=x(−u+cu3+⋯ )F(x(u))=x(-u+c u^3+\cdots) gives

cq=(I+A)h_3+B(q,h_2)+\tfrac16 C(q,q,q).$$ Multiplication by $p^\top$ eliminates $h_3$. Thus $$c=\tfrac16 p^\top C(q,q,q) +\tfrac12 p^\top B\!\left(q,(I-A)^{-1}B(q,q)\right).$$ The second term explains why differentiating three times along an arbitrarily chosen axis can give the wrong answer. With multiplier written $-(1+\mu)$, the second iterate minus the identity has leading terms $2\mu u-2c u^3$. If $c\ne0$, the small period-two branch has $u^2\sim\mu/c$. For $c>0$ it is locally attracting on the $\mu>0$ side; for $c<0$ it is unstable on the $\mu<0$ side. At equal finite depth, DPO and matched IPO have the same $A$ and $B$ at a common stationary policy. The entire quadratic correction above therefore cancels in their difference, even when it is nonzero for each loss. Here is a derivation that also specifies how far the manuscript's finite-exponential-family formula (Theorem B.4) extends. Work with smooth, positive finite-response policies in a fixed Euclidean parameterization, fixed Bradley–Terry labels, and ordinary, finite-step inner GD. Let $s_{ij}(\theta)=\log\pi_\theta(i)-\log\pi_\theta(j)$, $d_{ij}=\nabla s_{ij}$, and $b(\theta)=\mathbb E_\theta[(q_{ij}-1/2)d_{ij}]$, where $q_{ij}$ is the preference probability. At $b(\theta_*)=0$, the inner margin for displacement $w$ is $u=\beta[s_{ij}(\theta+w/\beta)-s_{ij}(\theta)]$. The common inner Hessian at $w=0$ is $$H=\tfrac14\mathbb E_*[dd^\top] -\beta^{-1}\mathbb E_*[(q_{ij}-1/2)\nabla^2s_{ij}].$$ The second term vanishes for affine log-ratios; it cannot generally be dropped for nonlinear policies. Set $R=I-\eta H$, $V_0=0$, $V_{j+1}=RV_j+\eta Db(\theta_*)$. For a small outer perturbation $h$, both inner iterates have linear term $V_jh$, so $DF(\theta_*)=I+V_K/\beta$. Since $\sigma(u)=1/2+u/4-u^3/48+O(u^5)$ whereas the matched IPO gradient in the margin is affine, the extra DPO update at step $j$ is $\eta\,\mathcal T_*(V_jh)/48$ to cubic order, where $\mathcal T_*(v)=\mathbb E_*[d(d^\top v)^3]$. Denoting the difference of the two inner iterates by $\Delta_j=w_{\mathrm D,j}-w_{\mathrm I,j}$, we obtain $$\Delta_0=0,\qquad \Delta_{j+1}=R\Delta_j+\frac{\eta}{48}\mathcal T_*(V_jh)+O(\|h\|^4).$$ Iteration transports the difference from step $j$ through $K-1-j$ later steps; division by $\beta$ converts back to the adopted parameter. Hence $$F_{\mathrm D,K}(\theta_*+h)-F_{\mathrm I,K}(\theta_*+h) =\Gamma_K(h)+O(\|h\|^4),\qquad \Gamma_K(h)=\frac{\eta}{48\beta}\sum_{j=0}^{K-1} R^{K-1-j}\mathcal T_*(V_jh),\qquad c_{\mathrm D}-c_{\mathrm I}=p^\top\Gamma_K(q).$$ The projection has no universal sign. It can be computed from derivatives of a specified update without assuming an invariant axis. Take two disjoint closed balls $B_\pm=B(c_\pm,r)$ and a fixed norm. If $$L=\sup_{B_+\cup B_-}\|DF\|<1,\qquad e=\max_\pm\|F(c_\pm)-c_\mp\|<(1-L)r,$$ then $\|F(x)-c_\mp\|\le Lr+e<r$ on $B_\pm$. The two-step map contracts each ball, so there is an attracting orbit of exact period two. To establish *reward loss*, bound reward throughout both balls against a feasible comparator. At an exact pair, the less restrictive local test is $\rho(DF(p_-)DF(p_+))<1$: one phase may expand even while the full return contracts. The box test above also establishes existence for an approximate pair, but its failure does not rule out cycling. Empirically, estimate the exchange residuals and the dominant return multiplier for the implemented update, then evaluate reward at both phases on the same independent evaluator. These are screens, not uniform bounds or ground-truth reward guarantees. There is also a local reward interpretation if the fixed branch is reward-stationary. Taylor expansion at that branch gives $J(\theta_*(\mu))-J(\theta_\pm(\mu))=-\tfrac12 q^\top\nabla^2J(\theta_*(0))q\,(\mu/c)+o(|\mu|)$ for the small cycle. Negative reward curvature along the critical mode then makes both phases worse; stationarity of the preference fit alone does not.

W2. Why keep the fifth-order, mixture, and discretization results?

They answer three different experimental-design questions: how narrow a scale sweep would detect the failure, how deep a fit would reveal it, and could the implementation itself shift the apparent threshold?

  1. A fifth-order term determines resolution and observation time. In Theorem 3.3 set z=a2z=a^2 and ε=τ−τ∗>0\varepsilon=\tau-\tau_*>0. The local amplitude equation has leading part β−βc=βc(τ∗)2(κεz−ψ∗z2)\beta - \beta_c = \frac{\beta_c (\tau_*)}{2}(\kappa \varepsilon z - \psi_*z^2), with κ,ψ∗>0\kappa,\psi_*>0. Maximizing gives af2∼κε/(2ψ∗)a_{\mathrm f}^2\sim\kappa\varepsilon/(2\psi_*) and window width ∼βc(τ∗)κ2ε2/(8ψ∗)\sim\beta_c(\tau_*)\kappa^2\varepsilon^2/(8\psi_*). At a fixed interior fraction of that window the reward gap is of order ε\varepsilon, but the stability margin is only of order ε2\varepsilon^2. A coarse scale sweep can miss a real reward gap, and even a nearby perturbation can take order ε−2\varepsilon^{-2} outer rounds to decay by a fixed factor. These are local statements, not bounds on a trajectory from an arbitrary start.
  2. IPO is not uniformly protected by closeness of losses. On the model sequence of Proposition B.6, the order-λ\lambda logistic correction competes with a residual of order K(2/3)KK(2/3)^K. Balancing them gives Ksgn=[log⁡(1/λ)+log⁡log⁡(1/λ)]/log⁡(3/2)+O(1)K_{\mathrm{sgn}}=[\log(1/\lambda)+\log\log(1/\lambda)]/\log(3/2)+O(1). This is the first negative cubic, not the first cycle. At appropriately greater depths the local coexistence width is Θ(λ2)\Theta(\lambda^2) and its interior reward gap is Θ(λ)\Theta(\lambda). Thus a small mixture can produce the failure, but a fixed depth or scale grid can miss this narrowing window. The model varies with λ\lambda; this is not a prevalence statement for one fixed policy family.
  3. The discretization distinguishes a nonlinear effect from a shifted linear threshold. At τ=Kη\tau=K\eta, the discrete critical scale exceeds the flow value by 38[e−τ/4−(1−η/4)K]=3τe−τ/4η/256+O(η2)\frac38[e^{-\tau/4}-(1-\eta/4)^K]=3\tau e^{-\tau/4}\eta/256+O(\eta^2). An order-η\eta shift can overwhelm an order-ε2\varepsilon^2 window. At the flow scale, η=o(ε2)\eta=o(\varepsilon^2) suffices to preserve the leading predictions. Recentering at the actual discrete flip removes that shift and η=o(ε)\eta=o(\varepsilon) suffices (Proposition B.12). In a new model one should measure the linear multiplier of the implemented map before diagnosing its higher-order terms. The particular rates are not universal learning-rate prescriptions.

W3 / Q2. An explicit perturbation that breaks product structure. Yes. We extended the contracting-neighborhood argument behind Theorem B.13 and Proposition C.9, keeping η=1\eta=1 and β=β32\beta=\beta_{32}. We obtained two distinct guarantees:

Perturbations, fixed across training roundsCertified phase boxes (maximum norm)Certified reward deficit at each phase
All nine log base weights, all 18 feature entries, and all nine rewards can be changed independently by at most 10−710^{-7} per entryRadius 5×10−45\times10^{-4} about (±1.09861228867,0)(\pm1.09861228867,0); sup⁡∥DF∥∞<0.973\sup\lVert DF\rVert_\infty<0.973Jξ(0)−Jξ(p±)>0.240J_\xi(0)-J_\xi(p_\pm)>0.240
Change only the log base weight of response (1,1)(1,1) by λ\lambda, for every λ∈[−0.05,0.05]\lambda\in[-0.05,0.05]Radius 2×10−42\times10^{-4} about separately continued centers; sup⁡∥DF∥∞<0.983\sup\lVert DF\rVert_\infty<0.983Jλ(0)−Jλ(p±)>0.215J_\lambda(0)-J_\lambda(p_\pm)>0.215

In the second row πs,tλ(x,z)∝h(x)exp⁡(sx+tz+λ1(x,z)=(1,1))\pi^\lambda_{s,t}(x,z)\propto h(x)\exp(sx+tz+\lambda\mathbf 1_{(x,z)=(1,1)}). Its cross-product odds ratio π(1,1)π(0,0)/[π(1,0)π(0,1)]=eλ\pi(1,1)\pi(0,0)/[\pi(1,0)\pi(0,1)]=e^\lambda proves nonindependence for every λ≠0\lambda\ne0. The original coordinate reflections are broken. For example, at λ=0.05\lambda=0.05 the two box centers are approximately (1.02043,−0.22012)(1.02043,-0.22012) and (−1.03408,−0.22044)(-1.03408,-0.22044): neither center is on the old axis and they are not negatives of each other. The comparison is to the feasible origin in each perturbed model; we do not assume it remains fixed or globally optimal. Consequently the deficit from the best feasible reward is at least as large.

Here is how the numerical bounds become a proof. For pair mass Pij=πθ(i)πθ(j)P_{ij}=\pi_\theta(i)\pi_\theta(j), difference d=ϕi−ϕjd=\phi_i-\phi_j, and Wj=DθwjW_j=D_\theta w_j, differentiating a frozen fit gives

H_j=\mathbb E[\sigma'(d^\top w_j)dd^\top],\quad B_j=\sum_{i,l}(q_{il}-\sigma(d^\top w_j))\,d\,(\nabla_\theta P_{il})^\top,\quad DF=I+W_{32}/\beta_{32}.$$ For the one-cell tilt we also differentiate the pair masses in $\lambda$ and enclose $D_\lambda w_j$. On each of 10,000 adjacent parameter intervals, outward-rounded dyadic arithmetic bounds the Jacobian *throughout each moving box* and the residual of its center. Explicitly, differentiating $F_\lambda(c_\pm(\lambda))-c_\mp(\lambda)$ along the piecewise linear centers gives $DF_\lambda\,c_\pm'+\partial_\lambda F_\lambda-c_\mp'$. Bounding this derivative and the midpoint residual gives $e<8\times10^{-7}$ uniformly. Since $0.983(2\times10^{-4})+8\times10^{-7}<2\times10^{-4}$, the two-box test applies to the entire interval, including its endpoints. In the unstructured case the center residual is below $4\times10^{-6}$ and $0.973(5\times10^{-4})+4\times10^{-6}<5\times10^{-4}$. These are sufficient radii for the cycle and its phase-wise reward loss, not largest radii, global basin measurements, or explicit radii for every conclusion of Theorem B.13. **Sustained stochastic training is different from deterministic perturbation.** A one-step disturbance bounded uniformly by $\omega$ below the trapping slack preserves alternation in these boxes; the distance from the corresponding deterministic phase is at most $L^nd_0+\omega(1-L^n)/(1-L)$. In particular, $\omega\le2\times10^{-6}$ suffices in the one-cell family. Disturbances tending to zero, while staying below that slack, give convergence to the deterministic phases. Theorem B.14 instead uses a fixed batch within each finite-horizon run, with batch size increasing across horizons. For a *fixed* finite batch of $N$ iid labeled on-policy comparisons per round, permanent trapping in these particular boxes is actually false. Inside them $|t|<1/4$ and the partition function $Z<20$. The two responses $(0,1)$ and $(0,-1)$ have equal reward. The probability of drawing these responses in either order **and** labeling $(0,1)$ the winner is $\pi(0,1)\pi(0,-1)=4/Z^2>1/100$. If all $N$ comparisons do so, the first inner step has $w_z=1$ and all subsequent steps increase $w_z$. The adopted shift in $t$ is at least $1/\beta_{32}$, which exits both boxes. If $\tau_{\mathrm{exit}}$ is the first exit from their union, iterated conditioning gives $$\Pr(\tau_{\mathrm{exit}}>T\mid\theta_0\text{ in the boxes}) \le(1-100^{-N})^T\longrightarrow0.$$ This rare event separates long finite-horizon tracking from a forever guarantee. It does **not** imply permanent escape from the basin, frequent escapes at realistic batch sizes, or recovery of long-run average reward. Neither the certificate nor this bound estimates the frequency of failure in LLMs.

Comments on the paper

1 comment

Thank you to all three reviewers. New and revised material is marked in red throughout the main text and appendices. Figure 1 now shows the reward loss from a common starting policy, Table 1 reports finite-batch results, Table 2 reports the neural-policy experiment, and Table 3 explains how to distinguish the proposed mechanism from overfitting or larger updates.

Reviewer 1 — Readability, the outer objective, and the DPO–IPO comparison. We now start with the standard chosen/rejected DPO log-ratio and distinguish the loss fitted within a round from the expected reward J(θ)=Eπθ[r]J(\theta)=\mathbb E_{\pi_\theta}[r] of the adopted policy. Adoption changes the next reference and sampler; it does not directly optimize JJ. From the same start in the cycle’s basin, one-step DPO and matched 32-step IPO converge to the in-family optimum. By contrast, both phases of the 32-step DPO cycle put probability 5/85/8, rather than 1/21/2, on the penalized event. The resulting gap γ(5/8−1/2)=γ/8\gamma(5/8-1/2)=\gamma/8 is what persists; reward need not decrease every round. Theorems 3.1–3.4 now lead from this example to its mechanism, onset, and prevention. We explain why the nine-response family cannot represent the exact reward tilt and why agreement through second order does not make DPO and IPO equivalent at depth. We do not claim IPO is generally superior. Appendix G.3 also distinguishes damping the adopted policy from averaging only the reference: in this model a small reference lag preserves the cycle, and larger lag can destabilize the optimum for either loss.

Reviewer 2 — A test in a trainable neural policy. In the population example, two copies of the same actor move in opposite directions at the same adopted KL: refreshing the reference and sampler reverses the previous move, while freezing both continues it. We tested this prediction in a small autoregressive decoder over six paired seeds, varying reference refresh, sampler refresh, and depth separately, with a fixed label source and a separate fixed evaluator on held-out prompts. The decoder did not exhibit a sustained two-cycle. With KL matched, the estimated effect of refresh on the benefit of deeper fitting was small and its confidence interval included zero. Section 4 and Appendix G report this negative result alongside the finite-model positive control. The experiment supplies a test for pretrained language models; it does not establish prevalence in them.

Reviewer 3 — Diagnostics, robustness, and the value of the safeguard. Appendix H shows how to diagnose a local period-doubling bifurcation without reflection symmetry or an invariant axis when a single eigenvalue crosses −1-1. The cubic coefficient alone cannot establish an attracting cycle or a reward deficit. The fifth-order analysis explains why a coarse sweep can miss the effect: near onset, the reward deficit is of order ε\varepsilon, but the coexistence window is only of order ε2\varepsilon^2. The mixture and finite-step results specify, respectively, how the required depth grows near IPO and how finely GD must approximate the flow. We also certify a concrete departure from product structure: tilting one response’s log weight by 0<∣λ∣≤0.050<|\lambda|\leq0.05 preserves an attracting cycle with both phases more than 0.2150.215 below a feasible policy. This differs from permanent stochastic trapping: at fixed finite batch size, trajectories leave the corresponding certified boxes almost surely under the protocol studied.

Even small errors in estimated moments can stall an unchecked update. In the product family, checking the estimated cap against its error bound and using a conservative IPO fallback gives geometric convergence with adaptive precision on the confidence event. Fixed precision instead gives an O(eˉ2)O(\bar e^2) bound on the limiting reward deficit, where eˉ\bar e bounds the cap error (Proposition C.8 and Appendix H). This controls how much of an arbitrarily deep fit can be adopted; it establishes no computational advantage over direct IPO or sufficiently small constant damping.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.