acceptodds
Under review as a conference paper at ICLR 2027

Preconditioned Direct Preference Optimization: Geometry and Acceleration

Abstract

Reinforcement learning from human feedback (RLHF) has become a central tool for aligning large language models with human preferences. Direct preference optimization (DPO) recasts RLHF as a policy optimization problem without explicitly estimating the reward function, and enhances the stability and efficiency of the training pipeline. Existing methods for DPO update based on policy gradients, and only the linear convergence rate is guaranteed. Therefore, to achieve acceleration, we consider using second-order information. A natural candidate is the Newton direction, which, however, involves the product of a Hessian inverse and the negative gradient, appearing computationally intractable in large-scale optimization. To circumvent this, we exploit the structure of DPO and, remarkably, characterize the regularized Newton direction via fully first-order information. This direction—which is used to propose a preconditioned method for DPO (PDPO)—encodes the geometry of the comparison graph, where each response is a node and each pairwise preference induces an edge. Consequently, we prove that updating along the Newton direction achieves quadratic convergence, and the updates along preconditioned directions restore the superlinear or the quadratic convergence rate under mild assumptions. Numerical experiments demonstrate that the proposed method achieves superior convergence rate and accuracy than the baselines.

Primary area:
optimization
Keywords:
Direct preference optimization, Preconditioned method, Newton method, Nonlinear Optimization, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_gYVGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PDPO, which uses second-order information to accelerate DPO and introduces random walks to estimate the update direction. It provides convergence analysis and experiments on synthetic problems and language-model fine-tuning.

Critical strengths
  1. The connection between DPO, comparison graphs, and random walks is interesting. It gives a way to understand and compute the proposed update.
  2. The synthetic experiments are carefully designed.
Critical weaknesses
  1. The convergence theory does not cover the algorithm used for language-model training. The theory updates a separate score for each response, while the actual algorithm updates the model parameters. These updates are different, and the paper does not show that the same fast convergence holds for the actual algorithm.

  2. The theory assumes a finite optimum, but the consistent reward-based preferences in the LLM experiments allow the score-space loss to keep decreasing as score gaps grow. The authors should clarify how the convergence guarantee applies to this setting

Questions for authors
  1. Can the authors explain whether the convergence guarantees apply to the actual model-parameter updates?
  2. The paper reports accuracy gains, but it is unclear whether random walks make training faster. The paper does not report the sampling cost or training time compared with DPO.
  3. Appendix F.2 reports preference pairs drawn from responses generated by the initial policy, while Algorithm 4 requires on-policy sampling. It is unclear whether the reported online experiment follows the proposed algorithm.
1 comment

There is no rebuttal, and I keep my score.

Reviewer_MusFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors examine post training and preference alignment through preconditioned DPO. To accelerate traditional DPO, the paper proposes using preconditioning, and seeking to derive information from the Hessian inverse in Newton's. This can't scale, so the authors propose a first order method, and show that the method achieves better convergence and accuracy against the baselines.

Critical strengths
  • The paper provides a clear geometric motivation for preconditioning DPO. Sections 4 and 6 describe the score-space Hessian as a weighted graph Laplacian and correctly use a quotient-space formulation. This yields a well-defined pseudoinverse Newton step and a local convergence results for connected graphs (assuming a finite stationary point exists). Extending the analysis beyond complete comparison graphs is reasonable since preference data can have sparse comparison structure.

  • The random-walk construction gives a nice implementation of the regularized inverse-Hessian action, and Theorem 5.1 constructs an unbiased estimator of the regularized Newton direction. Proposition 5.2 develops this further. This connects the graph structure to an implementable sampling procedure without requiring second derivatives of the neural-network parameters. The construction is a substantive component of the paper, although its computational advantage requires further validation.

  • Synthetic experiments aim to isolate the effect of comparison-graph geometry and evaluation covers six graph families. The first-order baselines show learning-rate tuning, and the experiments demonstrate the iteration-count advantage of exact score-space Newton updates on poorly conditioned graphs. The LM evaluation also uses a different reward model for held-out ranking than for training labels.

Critical weaknesses
  • Theorem 6.1 and Proposition 6.2 analyze updates directly in score space, but Algorithm 1 uses θ+=θ+ηJθ⊤d\theta^{+}=\theta+\eta J_\theta^\top d. Its induced score change is Δu=ηJθJθ⊤d\Delta u=\eta J_\theta J_\theta^\top d, which generally differs from the analyzed Newton step. Appendix C establishes an expectation identity but does not justify transferring the convergence rate. This is a central gap between the theoretical result and the practical algorithm, even before accounting for the use of AdamW.

  • The finite-stationary-point assumption is not satisfied by the hard-labeled score objectives used in the LM experiments. The convergence results require a finite u⋆u^\star satisfying ∇L(u⋆)=0\nabla\mathcal{L}(u^\star)=0. The synthetic experiments ensure this through positive preference weights in both directions. Section 7.2 labels comparisons according to a scalar reward ordering, but scaling scores along that ordering continues decreasing the loss, so a finite score-space minimizer does not exist. Undirected graph connectivity does not resolve the issue, and damping the Newton system does not regularize the objective. The link between the convergence results to the practical experiments is unclear.

  • Computational acceleration of the random-walk method is not convincing. Proposition 6.2 gives an O(1/M)O(1/M) variance statement for fixed damping, but does not track the dependence on γ\gamma, though approaching the Newton direction requires γ→0\gamma\to0. The variance can grow, while Algorithm 4 requires an expected β2/(4γ)\beta^2/(4\gamma) response proposals per walk. These costs can offset improvements in iteration count. Moreover, all synthetic PDPO experiments use exact Newton solves, and the offline LM experiments also use exact solves on only two to five responses per prompt. The paper does not report wall-clock comparisons or directly validate the finite-graph random-walk oracle.

Questions for authors
  • Under what assumptions on JθJ_\theta does the update θ+=θ+ηJθ⊤d\theta^{+}=\theta+\eta J_\theta^\top d preserve descent or inherit the score-space convergence rate? Given the additional factor JθJθ⊤J_\theta J_\theta^\top in the induced score update, should the acceleration claims be restricted to directly optimized response scores, or is there some additional argument?

  • How do Theorem 6.1 and Proposition 6.2 apply to the hard reward-ordered comparisons in Section 7.2? Unrestricted score objectives have no finite stationary point when strict preferences are present. Is an objective regularizer, soft preference target, or another mechanism used in the implementation that changes this?

  • What values or schedules of γ\gamma and the number of averaged samples MM were used, and what were the observed walk lengths and additional generation/scoring costs? Can the authors quantify the sampling requirements needed to satisfy Proposition 6.2’s relative-error conditions and provide a comparison against exact-solve PDPO and DPO using time/total computational work?

1 comment

As it is the final hours of discussion and the authors have not responded, I will keep my original score.

Reviewer_AJAWOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PDPO, a preconditioned method for direct preference optimization. It relates the curvature of the DPO objective to the response comparison graph and expresses a regularized Newton direction through random walks using first-order information. The authors establish local fast convergence in a finite response-score setting and derive how to use the direction for model-parameter updates. Synthetic experiments and language-model evaluations show faster score optimization and improvements in preference ranking and generation win rate.

Critical strengths
  1. The paper develops a useful connection between DPO geometry and faster optimization. The comparison-graph interpretation gives a clear explanation of the Hessian structure. The quotient-space treatment, Newton step, and local convergence analysis fit together well. The random-walk representation also offers an interesting way to obtain a preconditioned direction without forming a large network Hessian.

  2. The experiments examine several graph structures and include useful evaluation controls. The synthetic study covers six graph types and five response-set sizes, with tuned baseline step sizes. The language-model study also tests four offline graph structures. Five-seed evaluation, different response and reward models for ranking, and method-blinded generation judging provide useful support beyond the synthetic optimization results.

Critical weaknesses
  1. The benefits for actual LLM training need a clearer account. The theoretical results concern response-score updates, while the language-model experiments use parameter updates with AdamW. It remains difficult to tell how much faster or better practical LLM training becomes. Training curves showing DPO loss, ranking accuracy, and generation quality over both updates and elapsed time would make this benefit more intuitive.

The LM comparison uses 64 updates and separately tuned learning rates, but reports no runtime or matched-compute results. Fewer optimization iterations do not necessarily mean less computation if candidate scoring or sampling makes each update more expensive. Reporting per-step cost, total training time, and the number of responses processed or generated would help establish the efficiency claim.

  1. Some generation gains are small, and their stability is unclear. Table 1 averages five seeds over 512 generation prompts, with method names hidden from GPT-5.5. These are useful controls. However, the complete-graph and cycle settings have win rates of only 51.07% and 51.27%, without reported variation or confidence intervals. Better ranking accuracy alone does not establish a reliable improvement in generated answers. Per-seed results, paired confidence intervals, repeated decoding, and swapped answer order or repeated judging would help.
Questions for authors
  1. What responses are used in the actual online experiment? Algorithm 4 describes sampling starting responses and walk candidates from the current policy. Appendix F.2 instead describes responses generated before training and one preselected labelled pair per prompt. Does the experiment use this fixed pair or sample a new pair from the current policy? Are walk candidates newly generated or selected from the original two to five responses? Clarifying how these two descriptions fit together would help interpret the online results and their sampling cost.

  2. Do the gains remain useful at a larger training scale? The graph experiments are varied, but all LM training uses one 1.5B model, 256 prompts, 64 updates, and at most five responses per prompt. Longer training and another model or data distribution would help establish whether the benefit persists.

1 comment

My concerns have not been addressed, so I will lower my rating.

Reviewer_9RYUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DPO is usually trained with first-order optimizers such as AdamW. In theory, these can converge slowly when each prompt has several responses but only some pairs of them are compared (e.g., in a chain). This paper keeps the DPO loss unchanged and proposes a new optimizer setup called PDPO which replaces the gradient with a Newton-style direction that accounts for which responses were compared with which. It shows that this direction can be estimated by random walks over the comparison graph, without calculating or inverting the Hessian.

This paper proves quadratic local convergence regardless of how connected the graph is when taking each response's score as a free variable. For experiments, the paper shows that on synthetic problems the exact Newton step converges in 4 iterations while gradient descent needs 1,352 to 32,108 on path graphs. On LLM fine-tuning (Qwen2.5 1.5B, 256 prompts with 2-5 responses each), PDPO reaches 1.2 to 4.4 points higher ranking accuracy than DPO with a 51-55% win rate against it after the same 64 updates.

Critical strengths
  • The paper gives a relatively clear explanation of the scenarios under which DPO with a first-order optimizer would be slow to converge, as well as a proposed fix that works on an arbitrary connected comparison graph. Shi et al.'s quadratic rate requires a complete comparison graph. Extending Newton-type acceleration to arbitrary connected graphs is the main substantive contribution.
Critical weaknesses
  • The original contribution is not clearly stated in the paper. The closest baseline DPO-Mix-P (Shi et al., 2025) is already a Newton method on the complete comparison graph. With its per-pair weights held fixed and unit total weight per pair, the reweighted Hessian on the complete graph is β2(KI−11⊤)\beta^2(KI - \mathbf 1\mathbf 1^\top). Its pseudoinverse on 1⊥\mathbf 1^\perp is I/(β2K)I/(\beta^2K), which is exactly the same as DPO-Mix-P's step size with a quadratic convergence rate proved in Shi et al. In other words, PDPO's acceleration mechanism is DPO-Mix-P's acceleration mechanism extended to general graphs. However, the current paper abstract states that existing DPO methods guarantee "only the linear convergence rate," which contradicts the paper's own account of Shi et al. (2025). The new contribution is extending DPO-Mix-P's Newton step to arbitrary connected graphs and applying random walk estimators to the DPO Newton direction (especially online). That is narrower than what is currently claimed in the paper.

  • The current experiments aren't sufficient to test the paper's new contributions. The new pieces are Newton steps on poorly conditioned, non-complete comparison graphs and the random-walk estimator of the Newton direction. The LLM experiments are relatively weak:

    • The LLM experiments do not test poor conditioning, which is exactly the scenario that PDPO can handle but DPO-Mix-P cannot. With K≤5K \le 5, the graphs have κG\kappa_{\mathcal G} of at most about 9.5 at initialization, well below the 161 and 4.1×1034.1 \times 10^3 values in Table 2 caption.
    • The random-walk estimator is barely evaluated, as the offline walk is never run and the one online setting that uses walks does not report the regularization coefficient or the number of walks.
    • Acceleration is not properly measured. The paper's abstract states that PDPO "achieves superior convergence rate", but convergence rate is measured only on the synthetic problems, and the LLM runs use a fixed number of updates with no loss/accuracy plotted against training steps (only final accuracy and win rate are reported).
Questions for authors
  • Could the authors clarify that, with per-pair weights 1/ω(me)1 / \omega(m_e) held fixed and step size 1/(β2K)1 / (\beta^2 K), DPO-Mix-P is the pseudoinverse Newton step of the reweighted objective on the complete graph? If so, how would the authors state PDPO's contribution relative to Shi et al. (2025)? A clear restatement of the contribution as extending this mechanism to general graphs would be very helpful.

  • Under what realistic scenarios would sparse comparison graphs occur? Standard DPO data is usually one pair per prompt. For example, the UltraFeedback binarized dataset keeps the best response against a random response, and SimPO's construction keeps the best against the worst. There the comparison graph is a single edge. With all pairs labeled, the graph is complete, and thus gradient methods are already well-conditioned and we can just apply DPO-Mix-P directly. In which practical setting do you expect several responses per prompt with only some pairs labeled, for example human or pairwise-judge labeling on a budget? From my understanding, with a reward model you can typically do all pair labeling relatively cheaply, while for human labelers it seems more economical to just focus on quality over quantity and do one pair comparison per prompt? I think a natural setting, with evidence, would raise my assessment of the paper's significance. I think that the practical application of PDPO is somewhat limited/contrived if the advantages are only apparent when available labels are discarded.

1 comment

The authors did not respond so I'm keeping my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.