acceptodds
Withdrawn from ICLR 2027

(In)transitivity: Where Bradley-Terry Fails to Model Human Preferences

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most preference-based alignment methods use Bradley-Terry (BT) preference models, which implicitly assume human preferences are consistent and transitive, i.e., that they can be collapsed onto a single scalar. In reality, however, heterogeneous human preferences can cause intransitivity through cyclic majority relations both at the user, as well as population level—a problem well-studied in social choice theory. While recent game-theoretic alignment methods propose cycle-aware objectives to handle this, whether such cycles matter empirically remains unclear. In this work, we answer this question by first auditing 9 widely used human-preference datasets and find that by construction, most lack the overlapping, repeated judgements required to analyze population-level intransitivity. We then construct INTRANSIT, a controlled dense preference dataset, through preference-completion methods that estimate the missing population-level preferences. We use this as a measurement tool for widely used preference models and show how performance degrades to near-chance agreement on intransitive samples. At the same time, the denser preference structure in the completed datasets contains useful signal in that preference models trained on this data perform as well as models trained on fully complete pairwise annotations. Finally, at inference time, using trained preference model outputs reduces regret and majority-rejected selections, and reinforcement-learning systems trained with preference models over this show improvements for certain preference objectives. Our results identify two challenges for preference alignment: collecting and measuring preferences to allow intransitive preferences to surface, as well as building objectives for the representation of these preferences without imposing a scalar ordering.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference learning, Reward modeling, Intransitive preferences, Social choice theory, RLHF, Alignment, Preference datasets

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pQHtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper tackles three fundamental problem in Reinforcement Learning from Human Feedback. Namely,

  1. Auditing existing, published preference datasets for intransitive properties finding that most don't provide sufficient information to answer the question whether preferences are cyclic or not
  2. Recovering missing preference structures from the partial available information in these datasets.
  3. Measuring model performance on intransitive datasets violating most algorithms' fundamental assumptions. Downstream evaluations show that they offer minor post-training benefits but more significant for inference-time scaling.
Critical strengths
  1. The paper addresses fundamental problems and questions in RLHF which underpin many results but not investigated sufficiently.
Critical weaknesses
  1. Missing background section makes the paper hard to follow for readers not familiar with the domain. Many crucial definitions are missing such as ECE or regret.
  2. Too much information is deferred to the Appendix is Section 3 making the main content of the paper less self-contained and readers have to go back-and-forth to read the section.
  3. I could not find measurements on how well InTransit reproduce human judgement in Section 3.1. As I understand, the authors use PRISM annotator information to estimate preference signals of the annotators using an LLM. However, I could not find any evidence that it actually recover well the annotators' observed signals.
  4. The paper investigates interesting problems but results are trivial. Showing that the Bradley-Terry model does not work well on intransitive datasets is clear as it violates the preference model's main assumption. This means that most of the paper is just underlying issues that are common knowledge without a clear resolution.
Questions for authors
  1. What was the selection criteria for the datasets evaluated in Section 2? How comprehensive is it?
  2. Can you provide more information about PersonalLLM? There is no citation or reference for it in the paper.
  3. In Appendix A.1.4, how was gemini-3.1-flash-lite chosen? Why this model in particular and have the authors carried out any ablation on this?
  4. In Section 4.1, the authors claim "we see that training on completed datasets improves performance when compared to the same model trained on the original dataset". Could this be only a result of the findings in Section 3.3, i.e., that the completed preferences are biased towards transitive preferences?
  5. In Section 4.2, were each model tuned for the right hyperparameters? What was the grid and how do results differ with the hyperparameter selection?
  6. In Section 4.2 and 4.3, how do the better performances transfer to benchmark results?
Reviewer_UmH1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the limitations of scalar Bradley-Terry models in capturing intransitive human preferences. The authors audit nine preference datasets and point out that most lack sufficient annotations to identify population-level preference cycles. They construct INTRANSIT and propose a preference completion method to recover missing pairwise preferences. Experiments show that scalar preference models struggle with intransitive preferences, while completed preference data improves reward modeling and certain downstream alignment tasks, particularly inference-time selection using maximal lotteries.

Critical strengths
  • The authors' critique of the limitations of existing preference benchmarks is well-founded, and the proposed INTRANSIT benchmark provides a controlled setting for studying population-level preference cycles arising from heterogeneous individual preferences.
  • The paper evaluates multiple preference completion methods, reward models, and downstream alignment objectives, providing a comprehensive empirical evaluation of the proposed completion method and benchmark.
Critical weaknesses
  • INTRANSIT relies on constructed preferences assuming linear additive utilities over fixed attributes. This model assumption may oversimplify human preferences and raise concerns about whether the resulting intransitivity accurately reflects real human preferences.

  • It is unclear why the proposed completion method recovers most cycles in INTRANSIT but fails to recover cycles in Habermas. This discrepancy raises concerns about the generalizability of the method.

Questions for authors
  • Could the authors provide further insights into why the proposed preference completion method recovers most cycles in INTRANSIT but fails to recover any in Habermas, and how does this discrepancy affect downstream alignment performance?

  • Could the authors comment on why game-theoretic alignment methods such as NLHF were not included in the downstream evaluation? Is this because NLHF typically requires online learning? It would be interesting to examine whether preference completion also benefits alignment methods designed to handle intransitivity.

  • Typo in the caption of Figure 2: "towards towards"

Reviewer_9onuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors ask whether population-level preference cycles (Condorcet cycles) show up in preference datasets once judgments are aggregated, and whether scalar reward models such as Bradley–Terry (BT) are hurt by them. Their answer starts from the observation that existing datasets are often too sparse or lack repeated, overlapping judgments, so cycles are hard to see in them.

Contributions as I read them:

  • INTRANSIT, a dense, controlled dataset built from 1,379 PRISM annotator profiles and PersonalLLM prompts and responses. Each response gets attribute levels (fluency, safety, helpfulness) from an LLM judge, and profile-specific weights turn these into complete, individually transitive ballots.
  • GPM-Pop, a population-level completion method, compared against BT and other baselines.
  • A certification protocol for cycles (3-cycles, weakest edge at least 0.55, bootstrap survival over annotators and judge repeats), plus an audit of nine existing datasets.

Validation as I understand it: On the 19 certified cyclic sets, scalar models agree with the population majority at close to chance. At inference, a maximal lottery lowers best-of-8 regret on INTRANSIT. In post-training, completion helps DPO but not IPO or KTO. On the real-data check (Habermas), completion fails to recover the cycles. Judge reliability is reported via test–retest and split-half measures, along with position counterbalancing and a length check.

Critical strengths

Here are 2 strengths:

  1. A careful way of spotting contradictions. The authors do not simply claim that human preferences contain cycles. They check that the cases they identify are not due to chance. This matters because the paper's central argument rests entirely on the existence of these collective contradictions.

  2. An automated judge whose reliability has been checked. The scores given by the AI model stay consistent when repeated, and they do not depend on the order in which texts are shown or on their length. Since the rest of the study relies on these scores, this is essential for trusting the results.

Critical weaknesses
  1. The main question is answered by how the data was built. The cycles in INTRANSIT come from a simulator in which every simulated voter is consistent by design, so the cycles only appear because different voters weigh things differently. The paper therefore shows that simple ranking models fail on contradictions that were built into the data. Since the title promises an answer about real human preferences, and no real human judgments test this, the central claim is not supported.

  2. The method does not hold up on real data. The one real dataset where cycles appear, Habermas, is exactly where the method fails: it finds almost none of them, and it removes the structure it was meant to preserve. The gains on alignment are also mixed, helping in one training method, not in another, and hurting in a third. Since the practical message is that these methods improve alignment, and the real-world evidence does not confirm this, I do not think the recommendation is justified.

Questions for authors

Your ballots are simulated, so every voter is consistent by design. Would you be willing to test the cycle structure with a small set of real human judgments on the same prompts and responses, and if not, how would you present this limitation in the abstract?

The cycles you find depend strongly on how much weight is placed on the top attribute. How sensitive are your main conclusions to this weighting choice, and can you show results across a range of plausible weightings?

On the real Habermas data, your method recovers almost none of the cycles and removes the structure it is meant to preserve. Given this, what evidence do you have that the approach would help on real human preference data, and why should the title and abstract not be narrowed accordingly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.