acceptodds
Under review as a conference paper at ICLR 2027

When Uniform Tie-Breaking Preserves Global Score Structure

Abstract

Pairwise preference optimization learns from binary comparison outcomes, but many annotations also include ties. Prior work has incorporated ties into preference objectives and characterized their effect on population learning targets. Uniform tie-breaking preserves every comparison, but the conditions under which the resulting probabilities admit a single score vector have not been characterized. We establish this compatibility boundary for Davidson comparisons with a shared tie strength through cycle closure, which requires pairwise log-odds to sum to zero around every comparison cycle. We derive the exact residual sign and tight fixed-span extrema, and use inverse-link closure to characterize all feasible tie strengths, including multiple isolated solutions. The analysis explains DPO’s population projection and separates reward-scale changes that temperature calibration can correct from distortions in score direction. On controlled graphs and shared policies, calibration reduces source-policy regret, and cycle mismatch remains distinct from the additional fitting cost of parameter sharing.

Primary area:
learning theory
Keywords:
preference learning, uniform tie-breaking, Davidson model, Bradley–Terry model, cycle consistency, direct preference optimization, temperature calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rkr4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when uniform tie-breaking in Davidson preference models preserves a global-score representation. It characterizes the compatibility boundary through cycle residuals, derives tight residual bounds, and identifies source parameters consistent with the processed comparisons. It further characterizes population DPO as a comparison-weighted KL projection and examines the consequences for policy recovery, ranking, and finite-sample detection of incompatibility. Controlled graph and shared-policy experiments illustrate these theoretical findings.

Critical strengths
  • The paper offers a multifaceted analysis of uniform tie-breaking in Davidson preference models, connecting compatibility, identifiability, population DPO projection, and statistical error within a unified framework.
Critical weaknesses
  • The paper makes several contributions, but it is difficult to follow how the results fit together and assess their significance. In general, more detailed explanations and concrete examples would help readers appreciate the significance of the main claims.

  • Theorem 1 and Proposition 4 seem relatively straightforward given the problem formulations, so their significance could be better explained. Theorem 2 provides a more substantive bound, but the experiments do not directly illustrate its tightness, and its practical usefulness remains unclear.

Questions for authors
  • Could the authors clarify the practical implications of Theorem 2's bound? How close are residuals in realistic data to this bound, and can it inform algorithm design or improve learning?
  • In Proposition 4, how can C⊤j≠0C^\top j \neq 0 be assessed reliably from noisy estimates of lel_e? What does local identification offer in practice when multiple isolated values of ν\nu may remain consistent with the observations?
  • Section 5.4 would benefit from further explanation. In particular, how should the t−6t^{-6} scaling be interpreted, and how does it relate to Makur & Singh (2025)?
1 comment

Thank you for the detailed response and additional experiments. While my questions have been addressed, I still have concerns about the overall significance of the contribution. I will keep my rating for now but remain open to updating it based on discussions with other reviewers.

Reviewer_QRjHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper considers the effect of the tie-breaking practice in preference learning from pairwise comparisons. Tie breaking consists in changing comparisons that are ties (not clear win/lose or lose/wine) of a dataset to a win/lose or lose/win value, at random. The general question investigated here is whether a score model learned from the tie-breaked dataset can recover a model learned on the initial dataset. Section 4 gives necessary conditions (Th. 1) on the statistical model generating the initial dataset that allow its recovery from the tie-breaked dataset. Condition (i) is about the model generating the initial dataset; condition (ii) is on a function hνh_\nu on triple of alternatives that have different rewards. A corollary is that population DPO recovers the initial policy (up to a scaling term) if condition (i) holds on the model generating the initial dataset. Section 5 is a general discussion with two mathematical statements: an optimality condition of the DPO loss (prop. 5), and a statement on the sample complexity required to enforce a certain behavior on "type-I and type-II errors limits of tests". A two page experimental section concludes the paper.

I have substantial concerns about the relevance of the results to the field of Preference Learning, and the readability and accessibility of this paper to a broad audience. While I'm open to hearing the authors' arguments, at this stage I think that this paper is unlikely to be interesting or surprising to a broad audience.

Critical strengths

--

Critical weaknesses
  • My first major concern is that this paper is very hard to read. I have spent a relatively long time on this paper to reach only a partial understanding, particularly of the experimental section. Indeed, there are many different terms used for a much smaller number of ideas and notions. This is particularly damaging in the abstract and introduction. No text guides the reader into the different sections of the paper. The main theorems are not discussed in detail (why are the assumptions / the modeling relevant, what does the conclusion of the theorem mean, what does it imply in practice). This is particularly striking in corollary 6 which characterizes objects that have not been introduced before, "type-I and -II error" of some undefined "tests", and the condition is "nrept6→0n_{rep} t^6 \to 0", which is not discussed. I therefore cannot make sense of what Coro. 6 is about.

  • My second major concern is that, while I recognize that tie-breaking is a clear practice in preference learning, and characterizing the effect of this practice on what can be learned (and what is lost in that process) is an interesting question, I am skeptical about the value of the results proposed. What is the practical value of a necessary condition on whether learning from tie-breaked datasets can recover the underlying score distribution? For instance, some quantitative bound on their distance would be much more insightful. Besides, I fail to see any consequence of this study to Preference Learning practitioners. So I believe that these results are not likely to catch the eye of the Preference Learning community.

Questions for authors

Any elements on the reported weaknesses would be helpful in updating my score.

Reviewer_sPfVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies binary DPO when ties are resolved uniformly at random, or equivalently assigned half labels.

Empirically, exact and controlled graph computations confirm the predicted signs, cubic slopes, and calibration effects. A three-prompt shared-policy example separates cycle mismatch from the cost of parameter sharing. On Llama-3.2-3B and Qwen3-4B, binary DPO is compared with two tie-aware objectives.

Critical strengths
  • It seems the theory is well proved.
Critical weaknesses
  • The practical significance of the central object appears small.

  • The paper's own results suggest this. (e.g. Table2, Table 5)

  • The LLM experiments do not test the theory's actionable prediction. Specifically, the comparison of DPO vs. TODO/DPO-D measures tie-aware versus tie-broken training. It does not measure cycle mismatch or calibration.

Questions for authors
  • How large is the cycle-induced projection error relative to the scale-induced regret under the realistic reward?

  • Does calibrating β\beta after tie-breaking improve real DPO training?

  • For the LLM experiments, how many training seeds were used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.