acceptodds
Under review as a conference paper at ICLR 2027

Can Linked Binary Feedback Certify Ranking Disagreement?

Abstract

Several comparisons from the same evaluator reveal response patterns that pooled labels discard. Do these patterns establish different rankings, or can they reflect differences in decisiveness and dependence between answers? Under a logistic response model, every fixed finite linked-binary protocol on at least three stimuli admits a shared-ranking population and a population with a fixed positive fraction of ranking reversals that generate exactly the same joint responses. One distribution of decisiveness serves all comparison bundles. This rules out a uniform certification guarantee from finite linkage alone. We characterize when a fixed shared-ranking direction imposes testable restrictions on the joint response law. In a three-stimulus benchmark, near ties reveal a signal quadratic in ranking heterogeneity, but dependence of the same order can exactly reproduce the complete repeated-comparison law under a shared ranking that permits ties. For a calibrated benchmark with a known dependence budget, matching detection bounds separate the effects of sampling noise and residual dependence. Computations corroborate these transitions. A controlled study of language-model judges illustrates the diagnostic: all three tested models, each using a fixed six-persona roster, violate necessary shared-order conditions on a preselected narrative-rhythm triple at simultaneous 95% confidence. The results connect informative comparison designs to the dependence calibration needed for ranking-disagreement certificates.

Primary area:
learning theory
Keywords:
preference learning, identifiability, pairwise comparisons, mixture models, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RyTTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

From the paper writing, I was not able to understand what the paper is about, except that it addresses multiple pairwise preference selections by LLM judges. Specifically,

Line 038 states what I assume is the central research question of this paper:

Which joint response patterns can rule out every shared-ranking explanation?

However, there is no explanation what the "joint response pattern" is and what "ruling out shared-ranking explanations refers to". Because of the lack of clarity, the motivation and purpose of the following theoretical analysis also remains unclear.


I also tried understanding the paper from the perspective of the empirical validation in Sec. 6. However, the experimental setup is not explained there either. It seems that somehow texts in different styles where judged by closed/API LLMs, but what exactly is judged and with what purpose is not stated.


Finally, the implications for preference learning in Sec. 8 are opaque, too:

Preserve the labelled joint answers, use independent pilot and confirmation units to select promising comparisons, and report the dependence budgets under which a shared-order explanation is rejected.

None of this was directly tested, and how it would be implemented in practice is not stated either.

In summary, if there is actually a meaningful contribution, it needs to be stated in a much clearer way.

Critical strengths

I do not see any strengths in favor of acceptance, except the general topic of preference learning is relevant, and insights that resolve judge nondeterminism or contradictory binary decisions is useful.

Critical weaknesses

As mentioned above, the paper is unclear to an extent that makes judging anything else very difficult. At least, the paper should clearly state its motivation, which problem it addresses, and how it does that. In terms of writing, it should avoid jargon and introduce all concepts and settings properly.

Questions for authors

My main question is to clarify which problem this paper addresses, concretely.

Reviewer_1mUEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Article working on ranking disagreement. Unfortunately, due to several exposition issues detailed below, it is not possible to write a proper summary of the article.

Critical strengths

None

Critical weaknesses

The article is very difficult to understand due to the excessive use of technical jargon. In particular, the introduction fails to properly introduce the considered problem, its importance, the main difference with the literature, the main questions to be answered, and the techniques the article will use for this. Unfortunately, this issue is not solved in Section 2 (setup and observation regimes). Indeed, concepts such as "stimuli", "evaluator", "judgement", "decisiveness", "inferential target", "fresh", "panel", "persistent service", "positive-temperature submodel", "common direction", among others, are used but never properly explained. In particular, the global discussion is impossible to understand.

Unfortunately, the exposition issues only continue during the rest of the article. Due to this, assessing the contributions of the work is not possible. Therefore, the article cannot be recommended for acceptance in a conference such as ICLR.

Questions for authors

I have no questions.

Reviewer_TXDvOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether repeated binary comparisons from the same evaluator can certify genuine ranking disagreement, rather than a shared ranking with heterogeneous decisiveness or dependence between responses. Under a logistic response model the authors show that (1) for any fixed finite protocol on at least three stimuli, a shared-ranking population and a population with a fixed positive fraction of ranking reversals have identical joint response laws, so finite linkage admits no uniform guarantee; (2) a computable rank-deficit criterion characterizes when a protocol imposes testable restrictions, with a quadratic near-tie signal detected at rate K−1/4K^{-1/4} in a three-stimulus benchmark; and (3) dependence of the same quadratic order can reproduce that signal, giving a boundary ε≍max⁡{b, K−1/4}\varepsilon \asymp \max\{\sqrt{b},\, K^{-1/4}\} under a known dependence budget bb. Numerical experiments, an LLM-judge study and a dataset-schema audit support the analysis.

Critical strengths
  1. The question is well posed and directly relevant to multi-preference alignment and personalized RLHF: it turns the implicit assumption that "a few linked comparisons reveal disagreement" into a stated identifiability problem.

  2. The negative result is an exact observational equivalence, not an approximation. A single decisiveness distribution serves all comparison bundles simultaneously, and the construction handles shared items, repeated edges and cycles — a substantive extension of existing local-unrestrictedness results.

  3. The negative and positive results are complementary: beyond the impossibility, the paper provides a computable criterion and matching upper and lower rates, separating the contributions of sampling noise and dependence.

Critical weaknesses
  1. The concrete near-tie results, including the K−1/4K^{-1/4} detection rate and repeated-probe calibration, are established for a specific three-stimulus benchmark. The more general rate result is local and conditional on a fixed utility direction and the order at which the signal emerges.

  2. The robust certificate requires an externally justified dependence budget bb. The experiment treats b=0.2b=0.2 as a sensitivity setting, and the paper leaves calibration of bb from data open. This limits the direct use of the certificate in empirical settings where dependence is unknown.

  3. The LLM study illustrates the diagnostic in a controlled setting with known persona labels and deliberately contrasting preferences. It provides limited evidence about performance when evaluator types are unobserved.

Questions for authors

Please refer to weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.