acceptodds
Under review as a conference paper at ICLR 2027

Graded Equivalence Certificates for Preference Optimisation

Abstract

Preference optimisation has dozens of objectives published as distinct methods, and several frameworks that place them in one parameterised family. Family membership does not tell a practitioner whether switching methods changes anything. Every objective we study gives each response a score, takes the gap between two scores as a margin, and applies a link, a one-dimensional loss, to the margin. We ask at which level two objectives are the same and separate four. Two objectives can share the ranking they induce over responses, the score up to a per-prompt scale and shift, the loss, or the policy that training converges to. Each level has a certificate, an algebraic check that two objectives agree, and a witness, a few numbers that show they do not. The first three levels form a strict chain. The fourth follows from a shared loss and from nothing weaker. The optimum is set by the loss's target map, the score gap the loss drives a labelled pair towards as a function of how often annotators prefer one response. Methods that share DPO's score have different target maps and converge to different policies. Applying the certificates to the methods in current use shows that a shared loss is rare and a shared ranking is common. On a trained model the certified relationships hold exactly, and four links that share DPO's score order their margins as their target maps predict. We also measure the gap the certificates leave open, between scores that use a reference model and scores that use the policy alone. A closed-form experiment shows that when the labelled pairs form a chain, the choice of link alone can reverse which response the policy prefers, and that comparing every response against one anchor removes the effect for every link.

Primary area:
reinforcement learning
Keywords:
preference optimization, direct preference optimization, RLHF, alignment, surrogate losses, calibration, Bradley-Terry model, learning to rank, reward equivalence, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dcEJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for comparing preference optimisation methods at four levels of equivalence. It derives theoretical conditions for these equivalences and conducts experiments to examine the relationships between different objectives.

Critical strengths

The authors attend to explore a fundamental question in RLHF.

Critical weaknesses

The paper reads as though the authors are constantly trying to convince the reader of a complete research narrative: “We pose a question, we solve it, we provide a certificate, we validate it, and we draw practical implications”. So auto-research, so sad.

I find the main takeaway from Table 5 very puzzling. The authors report that: 22 losses fall into 20 equivalence classes, with only two pairs sharing the same loss function, despite being trained on different data. However, why exact loss equivalence should be expected among methods that were specifically proposed with different objectives? This is confuse. And, identifying that two methods use the same loss function but different training data seems to offer limited insight into whether switching between them would meaningfully change the optimization process or the resulting policy.

The paper does not appear to present (in my opinion, the problem is the way to present, I do not say there have no new findings) any new, valuable findings from the perspective of scientific research.

Questions for authors

Many of the Table 5 reported equivalence classes follow immediately from the definitions of the underlying scores and losses, while most loss classes contain only a single method. What nontrivial insights this classification provides beyond a systematic restatement of existing objective formulations?

Reviewer_CmezOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the equivalence of preference optimization objectives. It proposes four levels of equivalence (Rank, Score, Loss, and Opt), along with algebraic certificates and counterexamples. The framework is applied to 33 existing methods. The authors also introduce target maps to characterize population optima under certain assumptions. Finally, they investigate how the choice of loss and the structure of preference comparisons affect the learned rankings.

Critical strengths
  1. The proposed framework offers a useful perspective on the relationships between existing preference optimization methods. The systematic classification of 33 methods is informative and helps clarify which components of these objectives are actually different.

  2. The analysis of preference comparison graphs adds an interesting perspective beyond the taxonomy. In particular, the authors show how a mismatch between the loss function and the preference-label model can lead to incorrect rankings of unobserved pairs. While the analysis relies on idealized assumptions, it provides some useful theoretical insights.

Critical weaknesses
  1. The paper is quite difficult to follow. The paper repeatedly revisits the four equivalence levels, while the formal definitions and several important assumptions are deferred to the appendix. This makes it difficult to distinguish the general claims from results that hold only under specific conditions. Some practical interpretations are also imprecise. For example, Section 7 suggests that Rank-equivalent methods preserve learned rankings, while the formal definition only guarantees ranking agreement at fixed model parameters.

  2. The empirical validation is limited. Experiment 1 largely confirms algebraic identities that hold by construction. Experiment 3 considers only short training runs on a 0.5B model, with observed margins still far from their theoretical targets. Moreover, the comparison-graph results rely on idealized population preferences and unconstrained scores. It remains unclear how well these findings generalize to practical preference optimization.

Questions for authors
  1. Section 7 suggests that Rank-equivalent methods preserve learned rankings. However, the definition compares scores at the same model parameters, while Table 7 shows that different objectives can produce different optimal rankings. Could the authors clarify this distinction?

  2. Definition 1 requires the scaling and shift to be independent of model parameters. However, the proposed certificate seems to check only the scaling factor. How does the certificate handle a shift that depends on model parameters but is constant across responses?

  3. The analysis suggests that star-shaped comparisons can avoid ranking reversals that occur with chain comparisons. Would this finding still hold with a limited number of noisy preference labels? A comparison under a fixed annotation budget would help establish its practical relevance.

Reviewer_bB4VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a graded notion of equivalence for preference-optimization objectives. It decomposes an objective into a per-response score, a pairwise margin, a one-dimensional link, pair weights, extra terms, and a data distribution, and distinguishes four levels of equivalence: Rank, Score, Loss, and Opt. The first three form a strict implication chain, while Opt denotes equality of population minimizers on a particular data distribution and cuts across the other levels. The main theoretical device is the target map Tℓ(p)T_\ell(p), defined as the margin minimizing the expected pairwise loss when the first response is preferred with probability pp. Under calibrated links, suitable realizability assumptions, and jointly realizable pairwise targets, two Score-equivalent objectives share population minimizers exactly when their target maps agree after accounting for the score rescaling. This explains why DPO, IPO, cDPO, and rDPO may share a score while inducing different optima. The paper also analyzes how chain, star, and complete comparison designs interact with the link and may preserve or reverse the latent response ordering. The framework is applied to 33 preference-optimization methods using source-level “loss cards.” Validation consists of numerical checks of certified score relationships on a DPO-trained Qwen2.5-0.5B model, a closed-form synthetic comparison-design experiment, and a small-model experiment comparing the margins learned by four links sharing the DPO score.

Critical strengths

The Rank/Score/Loss/Opt decomposition addresses a real source of confusion in the preference-optimization literature. The accompanying certificates and finite witnesses make the taxonomy more actionable than a purely notational unification.

Critical weaknesses

Definition 1 requires the additive shift in Score equivalence to be b(x)b(x), independent of θ\theta. However, replacing a score by sθ(x,y)+hθ(x)s_\theta(x,y)+h_\theta(x) leaves every pairwise margin, loss, gradient, and optimizer unchanged, while apparently failing the stated definition. The canonical centering procedure would also remove this shift, creating a mismatch between the definition and the proposed certificate. Separately, defining Opt by equality of argmin sets makes two objectives with no minimizer vacuously equivalent because both argmin sets are empty. This is directly relevant to the paper’s own observation that hard-label DPO and rDPO may lack finite minimizers. These issues affect whether the taxonomy is representation-invariant and mathematically non-vacuous.

Experiment 1 evaluates several score formulas on checkpoints produced only by DPO, so the exact certified identities are largely algebraic checks rather than evidence about the training behavior of the corresponding objectives. In Experiment 3, the robust finding is the ordering of the four learned margins; the supplied analysis reports that the stronger target-shape prediction fails for every seed, and the learned margins reach only approximately 2626%--3131% of their theoretical targets. The experiments also use one 0.5B model, one dataset, short training, and constructed soft-label probabilities. Thus, claims about where practical training converges should remain narrower than the population-optimum results. The non-realizable and finite-capacity scope would also benefit from positioning against recent work such as Pukdee et al. (ICML 2026) and Mohri and Zhong (ICML 2026).

Questions for authors
  1. Definition 1 requires s1(x,y)=a(x)s2(x,y)+b(x)s_1(x,y)=a(x)s_2(x,y)+b(x), whereas adding a parameter-dependent prompt shift hθ(x)h_\theta(x) leaves all pairwise margins and losses exactly unchanged. Should Score equivalence instead allow b=bθ(x)b=b_\theta(x) while continuing to require the positive scale a(x)a(x) to be independent of θ\theta? How would this change the canonicalization procedure and the classifications in Table 5?

  2. How is Opt equivalence intended to be defined when one or both population objectives do not attain their infimum? Under the current set-equality definition, two empty argmin sets are equal, which would make some hard-label objectives vacuously Opt-equivalent. Could the authors require nonempty argmin sets or formulate an alternative notion based on limiting policies, infimizing sequences, or implicit-bias directions?

  3. Could the authors more precisely separate what Experiment 3 validates from what remains a population-level prediction? In particular, the released analysis reports that the target-shape prediction fails for all three seeds and that the observed margins reach only about 2626%--3131% of their targets. Do substantially longer training or an additional model/dataset support convergence toward the target maps, or should the empirical conclusion be restricted to the observed ordering of margins?

  4. How do the proposed certificates and graph-design results relate to recent analyses of misspecified pairwise preference learning and finite-capacity surrogate consistency, particularly Pukdee et al. (ICML 2026) and Mohri and Zhong (ICML 2026)? Clarifying whether the present results are complementary to, imply, or require assumptions stronger than these works would materially affect my assessment of novelty and scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.