acceptodds
Under review as a conference paper at ICLR 2027

Balancing Structure and Stability in Leaderboards with Quantal Response Equilibrium

10 views

Abstract

Public leaderboards shape how AI models are perceived and adopted. Yet defining a ranking from preferences is nontrivial, as different aggregation principles can favor different models. In practice, not every pair of models is compared, raising the additional challenge of predicting missing entries while retaining competitive structure. Elo’s scalar model restricts comparisons to additive logits, excluding potentially non-transitive structure. On frontier leaderboards, we show that adding a synthetic model that loses to everyone dethrones Elo’s leader, despite that leader still beating every rival and all original comparisons remaining unchanged. Maximal lotteries (MaxL), an alternative from probabilistic social choice, capture arbitrary tournament structure, but require a complete comparison matrix. Their outputs can change substantially across resamples. To reconcile structure and stability, we decouple leaderboard construction into comparison-matrix estimation and ranking. For estimation, we derive error bounds for low-rank completion that account for structural approximation, regularization, and finite-sample error. These bounds quantify the statistical cost of richer representations. For ranking, we analyze quantal response equilibrium (QRE), an entropy-regularized counterpart to MaxL that yields smooth, payoff-based scores. We derive temperature-dependent stability bounds and approximate guarantees for Condorcet consistency and independence of irrelevant alternatives. Empirically, QRE preserves the incumbent order in the insertion example and retains its leader in nearly all resamples.

Primary area:
datasets and benchmarks
Keywords:
social choice theory, game theory, leaderboards, AI evaluation, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TkyEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares different ranking methods for pairwise comparisons. QRE is one of the good ranking algorithms as it satisfies several nice properties while some other ranking algorithms including BT might not satisfy simultaneously. Then they discuss the trade-off between the social choice guarantees and stability of QRE through the temperature.

Critical strengths

The trade-off is an interesting observation.

Critical weaknesses

The scope is a bit narrow.

Questions for authors

see above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer TkyE (1/2)

We thank the reviewer for reading our paper. We understand the main concern to be that the scope is narrow. Since the review does not specify which aspect of the scope this refers to, we would be grateful for more detail, as it would help us improve the paper. In the meantime, we clarify the scope and contributions below. The other reviews and our responses to them may also provide useful context, as they discuss several of these points in more detail.

Our scope is the reliability of preference-based leaderboards, covering both comparison estimation and ranking. The review’s summary describes only the ranking part and omits the estimation half of the paper (Sec. 3, 5.3). The paper connects social-choice requirements with statistical reproducibility through three contributions.

1. A systematic analysis of why existing leaderboard methods can be unreliable. We compare BT/Elo, SCO, and MaxL through their social-choice properties and statistical stability (Table 1).

  • BT/Elo can produce stable rankings while excluding nonadditive preferences and failing to rank a Condorcet winner first. On Image Bench, one synthetic entrant that loses to every model changes 25/30 ranks and dethrones the leader, although all comparisons are unchanged and the leader still beats every rival. The original submission also shows this with real models: removing two strict losers from Arena changes 10 of 26 survivor ranks while all survivor comparisons stay fixed (Sec. 5.2)..
  • MaxL has the desired social-choice guarantees, but it requires a complete comparison matrix, which practical leaderboards often do not have, and it is sensitive to estimation noise: the set of leaders it reports changes in 27.45% of Arena bootstrap resamples..
  • SCO is Condorcet-consistent only at the global optimum of a nonconvex loss and has no IIA or clone guarantee..

These results explain why sampling stability alone is insufficient to assess leaderboard reliability. We address these limitations with QRE, a solution concept that originated in behavioral game theory (McKelvey & Palfrey, 1995). Unlike BT, it responds to non-transitive structure; unlike SCO, it is a unique, efficiently computable equilibrium; and unlike MaxL, it returns a full ranking and varies continuously with the comparison matrix, so the leaderboard it reports is reproducible under resampling.

2. Quantitative guarantees for addressing these limitations. For comparison estimation, Theorem 1 separates approximation, regularization, and sampling error, quantifying the statistical cost of retaining richer preference structure. For ranking, Theorems 2–3 establish an explicit temperature threshold for Condorcet consistency, a bound on weak-entrant influence that vanishes exponentially as τ→0\tau\to 0, and Lipschitz sensitivity to comparison-matrix errors. Combining Theorems 1 and 3 gives sufficient conditions for preserving population QRE orderings. These contributions explain how representation capacity, available data, and regularization affect the resulting rankings.

AuthorsOpenReview ↗

Response to Reviewer TkyE (2/2)

3. Empirical evidence of the joint benefit. The original Table 1 compares social-choice properties, while the original Table 2 (Table 4 in the revision) evaluates sampling stability. Revised Figure 2 evaluates sampling stability and candidate-pool robustness together: QRE configurations form the observed Pareto frontier among the evaluated BT, SCO, iterative MaxL, and QRE settings, across both datasets and both insertion and deletion experiments. At a single temperature, τ=.25\tau=.25, QRE improves on BT on both axes:

% (BT → QRE)ArenaImage
Resampling Kendall distance2.68 → 2.501.85 → 1.71
Models changing rank, insertion38.83 → 3.3341.63 → 2.85
Models changing rank, deletion12.79 → 3.873.58 → 2.04

This demonstrates the practical value of considering these objectives jointly.

Generality. Theorems 1–3 are stated for general antisymmetric comparison matrices estimated from pairwise outcomes, so nothing is specific to LLMs. The analysis applies wherever pairwise comparisons are aggregated into a ranking, including agent evaluation in games (Bertrand et al., 2023; Lanctot et al., 2023; 2025). Our experiments cover two modalities and scales: text (28 models, 71,866 votes) and image (30 models, 2.16M annotations).

We would greatly appreciate it if the reviewer could let us know which aspect of the scope they find limiting, whether the theoretical setting, the empirical results, or the significance of the contributions, so that we can address it directly. Even a short answer would help us greatly in improving the paper. We hope the clarification above is useful and we kindly request the reviewer to reconsider their rating in light of it.

  • Bertrand, Q., Czarnecki, W. M., & Gidel, G. (2023). On the limitations of the Elo: Real-world games are transitive, not additive. AISTATS.
  • Lanctot, M., et al. (2023). Evaluating agents using social choice theory. arXiv:2312.03121.
  • Lanctot, M., et al. (2025). Soft Condorcet optimization for ranking of general agents. AAMAS.
  • McKelvey, R. D., & Palfrey, T. R. (1995). Quantal response equilibria for normal form games. Games and Economic Behavior, 10(1), 6–38.
Reviewer_aPEcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the problem of building a reliable leaderboard of AI models when having incomplete and low-quality pairwise comparisons. It breaks down the problem into constructing the comparison matrix via DISC-kk representation and converting the matrix to rankings via Quantal Response Equilibrium (QRE), and provides statistical results for the error and stability. Experiments further demonstrate the empirical performance of QRE, but do not provide a strong advantage over existing ranking benchmarks.

Critical strengths
  1. The problem is important, and the approach is practical
  2. The theoretical analysis is nice. The analysis of QER provide quantitiative result showing the influence of the temparature coefficient on the tradeoff between statistical stability and social-choice properties.
  3. The experiment shows how the rankings get influenced by various controlled settings.
Critical weaknesses
  1. Novelty concerns: Both the estimation using Disk-k and the ranking using QER are established methods, and it would be useful to distinguish the contributions
  2. Experiment: It does not provide a clear advantage of QER, especially in Table 2. May want to provide more insights on when each method works the best.
Questions for authors
  1. Novelty: I found a closely related paper that also uses QER for LLM evaluation: Liu et al (ICLR 2026), Re-evaluating Open-ended Evaluation of Large Language Models. Please compare it with the paper.
  2. What is the data requirement (composition in the comparison pair) and the sampling cost? For example, Corollary B.1 assumes every pair is observed, so it should grow with MM, but in Appendix B, it is mentioned that the sampling cost grows with the complexity kk, and not with the number of models MM.
  3. Do you have guidelines on how to choose an appropriate τ\tau?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We thank the Reviewer for their encouraging and helpful feedback. We address their concerns below.

W1: Novelty concerns: Both the estimation using Disk-k and the ranking using QER are established methods, and it would be useful to distinguish the contributions

We use classical tools to provide novel guarantees and analysis, this should be our contribution, not weakness. DISC and QRE provide established mathematical foundations. The contribution lies in identifying distinct failure mechanisms and quantifying how temperature controls Condorcet guarantees, weak-entrant influence, and sensitivity to estimation error. These specific conclusions should be evaluated beyond whether the underlying tools are classical.

W2: The experiments do not show a clear advantage of QRE, especially in Table 2. More insight on when each method works best would help.

We thank the reviewer for this suggestion. We have added a new plot in the revision (Figure 2), which highlights the trade-off more clearly.

QRE’s advantage lies in balancing social-choice properties with sampling stability. The original Table 1 compares social-choice properties, while Table 2 reports sampling stability. Section 5.2 already provides a real-model deletion example: survivor comparisons remain unchanged, yet BT changes 10/26 survivor ranks while QRE preserves their ordering.

The revision makes this joint advantage explicit. In Figure 2, QRE configurations form the observed Pareto frontier between resampling stability and candidate-pool robustness among the evaluated settings, across both datasets and both insertion and deletion experiments. We provide further evidence and discussion in our response to Reviewer BDmi.

Q1: Comparison with Liu et al., "Re-evaluating Open-ended Evaluation of Large Language Models."

We thank the Reviewer for pointing us to this work (published at ICLR 2025). We will cite and discuss it in the revision.

Different settings. Liu et al. address robustness of ratings to redundant (cloned) prompts and models. They model evaluation as a 3-player general-sum game between a prompt player and two model players, with a fully known payoff tensor given by an LLM judge and treated as exact. We address reproducibility of rankings when the comparison matrix is estimated from finite and incomplete votes, while respecting competitive structure. We therefore work with a symmetric 2-player zero-sum game on the estimated pairwise margin matrix.

Liu et al. do not use QRE as a solution concept. Liu et al. rate actions at a selected equilibrium, using two solution concepts: the Nash equilibrium and the coarse correlated equilibrium (CCE). The CCE is computed without QRE. For the Nash equilibrium, they select the limiting logit equilibrium (LLE): they start from the QRE at high temperature, anneal the temperature toward zero while following the QRE path, and take its endpoint which is a Nash equilibrium. QRE therefore serves only as a computational device for selecting one Nash equilibrium among many, and the small end temperature in their implementation is a numerical approximation to the τ→0\tau \to 0 limit.

We propose QRE at fixed τ>0\tau > 0 as the solution concept. A QRE at positive temperature is an equilibrium of the entropy-regularized game and, in general, not a Nash equilibrium of the original game. In our symmetric zero-sum setting, the QRE is unique for every τ>0\tau > 0, and the endpoint of the QRE path is a maximal lottery, so their LLE corresponds to MaxL in our setting. We argue that positive temperature is preferable when rankings are computed from finite, noisy comparisons, and we study the properties it provides and the trade-off between them as τ\tau varies:

  • Stability: the QRE lottery and scores are Lipschitz in the comparison matrix, with constants that improve as τ\tau increases (Theorem 3), whereas MaxL can change discontinuously.
  • Social-choice guarantees: QRE ranks a strict Condorcet winner first for τ\tau below an explicit threshold, and the effect of an irrelevant entrant on score gaps vanishes exponentially as τ→0\tau \to 0 (Theorem 2).

Empirically, we find that temperatures close to the zero limit are less stable under resampling (Figure 2 of the revision – new figure), so choosing τ\tau for stability outputs rankings that differ in practice from those obtained by approximating the Nash equilibrium.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

Q2: What are the data requirements and the sampling cost?

We first clarify scope, then explain the complete-design result and the remark.

Scope. Theorem 1 bounds the estimation error for an arbitrary set of observed pairs EE and vote counts NijN_{ij}, under a restricted strong convexity (RSC) condition (A2) with constant ν>0\nu > 0. This condition measures how strongly the observed pairs constrain the full matrix, and the bound scales as 1/ν1/\nu. The bound is therefore conditional: it does not say which observed graph structures satisfy RSC or how ν\nu scales with the number and arrangement of observed pairs. Providing a sample complexity for incomplete comparison graphs would require exactly this characterization. Such a result, especially for antisymmetric low-rank matrices under non-uniform matchmaking as on Arena, would be a significant contribution to the matrix completion literature and is beyond the scope of this paper. Instead, we study empirically how completing missing pairs affects rankings in Section 5.3.

Why complete graphs. Even when every pair is observed, each win rate is still estimated from finitely many votes. A low-rank model shares information across pairs, so it can reach the same accuracy with fewer votes than estimating each pair separately. The complete graph is where this can be stated explicitly, since RSC holds automatically there (Definition 1). It also matches our experimental setting (Arena28 and Image Bench are complete). Corollary B.1 shows that, when the true matrix has rank at most 2k2k, reaching per-entry accuracy ε\varepsilon requires on the order of kMlog⁡M/ε2kM\log M/\varepsilon^2 total votes for MM models.

The remark in Appendix B. The reviewer is right that the total number of votes grows with MM. However, the remark refers to votes per model. Since each vote involves two models, Corollary B.1 implies that each model needs on the order of klog⁡M/ε2k\log M/\varepsilon^2 votes. The O~(k/ε2)\tilde O(k/\varepsilon^2) notation in the remark absorbs the log⁡M\log M factor to highlight the dependence on kk. Estimating each pair separately would instead need on the order of 1/ε21/\varepsilon^2 votes per pair, i.e. M/ε2M/\varepsilon^2 votes per model. The low-rank structure thus replaces the factor MM with klog⁡Mk\log M. A complete design additionally requires at least one vote per pair, i.e. M−1M-1 votes per model. Even when this dominates, the requirement remains far below the M/ε2M/\varepsilon^2 needed for separate estimation. We will revise the remark to make this more clear and state the one-vote-per-pair condition explicitly.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Q3: Do you have guidelines on how to choose an appropriate τ?

Our results give both theoretical and practical guidance, which we will include in the appendix. The following points matter when setting τ\tau:

  • Any τ>0\tau > 0 restores continuity. MaxL can change discontinuously under arbitrarily small perturbations of the comparison matrix. For every τ>0\tau > 0, QRE's lottery and scores vary continuously with the matrix and τ\tau controls how strongly.

  • Theorem 3 gives a lower bound on τ\tau relative to estimation noise. For every τ>0\tau > 0, ∥pτ(A)−pτ(B)∥1≤ε/τ\lVert p^{\tau}(A)-p^{\tau}(B)\rVert_{1} \le \varepsilon/\tau and ∥sτ(A)−sτ(B)∥∞≤(1+τ−1)ε\lVert s^{\tau}(A)-s^{\tau}(B)\rVert_{\infty} \le (1+\tau^{-1})\varepsilon, with ε=∥A−B∥max\varepsilon=\|A-B\|_{max}. Both bounds decrease in τ\tau, so a smaller τ\tau requires more accurate matrix estimates for the same guarantee. In practice, ε\varepsilon can be estimated from the bootstrap spread of A^\hat A.

  • Theorem 2 gives upper bounds on τ\tau relative to competitive margins. Part (i) guarantees that a strict Condorcet winner is ranked first for τ<τCW=gCW2/(amax⁡log⁡M)\tau < \tau_{\mathrm{CW}} = g_{\mathrm{CW}}^2/(a_{\max}\log M). Part (ii) also yields an explicit condition: for any entrant that loses to every incumbent by margin at least dd, the end of Appendix C.3 shows εIIAij≤4(1+τ−1)e−d/τ\varepsilon_{\mathrm{IIA}}^{ij} \le 4(1+\tau^{-1})e^{-d/\tau}.

Together, these give a range: τ\tau should be large relative to estimation noise and small relative to the competitive margins that we want to respect. These conditions are sufficient rather than necessary.

Empirically, the trade-off is visible across τ\tau. Table 2 reports τ∈.01,.075,.5\tau \in \\{.01, .075, .5\\}, and Fig. 5 sweeps τ∈[.01,5]\tau \in [.01, 5]. Sensitivity to irrelevant alternatives appears only at larger τ\tau: the Image insertion changes 0, 0 and 4 incumbent ranks at τ=.01,.075,\tau = .01, .075, and .5.5.

In practice, we suggest the following procedure:

Step 1: Estimate the noise level ε\varepsilon from the bootstrap spread of A^\hat A, and use Theorem 3 to obtain a minimum τ\tau for the desired stability.

Step 2: Sweep τ\tau above this value and choose the smallest τ\tau at which bootstrap stability has plateaued.

Step 3: If a strict Condorcet winner exists in the estimated matrix, compute τCW\tau_{\mathrm{CW}} directly. If the chosen τ\tau lies below it, Condorcet consistency is guaranteed. Otherwise, since the bound is conservative, check directly whether the Condorcet winner is ranked first.

Since preference leaderboards have no ground-truth ranking, τ\tau cannot be tuned for accuracy. None of the steps above require one: bootstrap stability measures reproducibility, while the Condorcet and insertion checks test properties of the ranking rule on the estimated matrix. For this reason, τ\tau should be selected from these diagnostics rather than from the resulting ranking, and reported alongside the leaderboard, as noted in our limitations.

Hi Authors,

Thanks for the detailed response; they are point-to-point. That clarifies a lot of my questions, and I will raise my assessment to positive point.

Reviewer_BDmiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to rank AI models using pairwise comparisons. It shows that Elo rankings can change when a weak model is added, while maximal lotteries can be sensitive to sampling noise. The authors propose combining an existing method for estimating missing comparisons with quantal response equilibrium (QRE) to produce rankings. They provide theoretical results on estimation error and how QRE balances ranking stability with desirable social-choice properties. Experiments on language and image model leaderboards show that QRE can avoid some of Elo's problems and produce more stable rankings, although the results depend on the choice of parameters.

Critical strengths

I would be inclined to say that the paper provides a reasonably systematic analysis of existing approaches, but I struggle to identify a major technical or empirical strength.

Critical weaknesses

The paper combines existing approaches to matrix completion and QRE-based ranking. The theoretical results appear to largely follow from established techniques in low-rank estimation and entropy-regularized games. It is unclear what substantially new theoretical insights the paper provides beyond applying these techniques to AI leaderboards.

Moreover, the experiments do not establish a clear advantage over existing approaches. QRE does not consistently outperform BT in terms of ranking stability, and the DISC models provide little or no improvement in predicting pairwise preferences. Although the paper demonstrates situations in which BT behaves undesirably, these examples are deliberately constructed. It remains unclear whether the proposed approach provides meaningful improvements in realistic leaderboard settings.

Questions for authors

What is the main theoretical novelty of the paper? How representative are the experiments involving the addition of weak models? In Figure 2, the synthetic model is explicitly selected to disrupt the BT ranking. How frequently do such changes occur when models are added or removed under more realistic conditions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BDmi

The review’s assessment does not adequately account for the specific theoretical guarantees and real-model deletion evidence already provided in the original submission.

1. Established tools do not imply that our theoretical results are already known.

Theorem 1 provides new analyzes for finite-sample DISC estimation, separating approximation, regularization and sampling error. Building on Bertrand et al. (2023), it quantifies the capacity–sample-size tradeoff of retaining nonadditive structure (where ELO fails) under the stated assumptions. Theorems 2–3 establish temperature-dependent Condorcet conditions, social choice guarantees, and matrix-perturbation guarantees. Combining these with Theorem 1 gives sufficient conditions for preserving population QRE orderings.

Which specific result is already known or immediate, and which reference or derivation supports that assessment? If the concern is technical depth or significance, please identify the specific limitation.

2. QRE's advantage is the joint tradeoff between sampling stability and the social choice properties in Table 1; comparing stability alone misses that contribution.

Addition to the synthetic insertion, original Section 5.2 and Appendix D.3 remove two successive strict losers of Arena leaderboard. All 325 survivor comparisons remain fixed, yet BT changes 10/26 ranks, including swapping second and third place. QRE at τ=.075\tau=.075 preserves the survivor ordering. Original Figure 2(b) also exposes a structural mechanism: cyclic perturbations preserving aggregate wins leave BT ratings unchanged even when the Condorcet winner changes. Related concerns and synthetic examples motivate SCO (Lanctot et al., 2025) and MaxL-based evaluation (Khalaf et al., 2026).

Additional frequency evidence: We exhaust all eligible single-model deletions outside the reference top-three union. BT changes the survivor ordering in 20/22 Arena cases (90.9%) and 13/27 Image cases (48.1%), with survivor comparisons unchanged. These are frequencies within the specified eligible cohorts.

3. New experiments show a joint advantage at one common QRE temperature.

Revised Section 5.1 compares 2,000 matched resamples, the deletions above, and 2,000 random weak-model insertions per dataset with independent entrant win probabilities U(0,.4). At τ=.25\tau=.25, QRE improves both resampling stability and candidate-pool robustness:

Metric (%, BT → QRE)ArenaImage
Mean resampling Kendall distance2.68 → 2.501.85 → 1.71
Mean models changing rank: insertion38.83 → 3.3341.63 → 2.85
Mean survivors changing rank: deletion12.79 → 3.873.58 → 2.04

The original Table 1 already compared structural properties alongside Table 2's sampling results; the revision measures this joint benefit directly. Table 2 remains in the appendix.

4. Prediction loss alone does not assess DISC's structural benefit.

DISC achieves held-out losses close to BT while retaining nonadditive structure that BT excludes.

The data support the relevance of this additional structure. In 2,000 simulations from fitted BT with comparison counts fixed, only 7 Arena and 0 Image samples reach the observed nonadditivity: p=.003998 and .000500. These exploratory tests assume independent votes. They provide evidence against these fitted additive nulls.

Higher capacity can become more valuable as data grow. BT is a special case of DISC-1. Where larger k reduces approximation error, more data can offset its additional sampling cost, while BT's structural bias persists. Original Figure 3(a) demonstrates this in recovery from a frozen empirical tournament. Similar predictive loss at current budgets therefore does not establish that richer capacity is unnecessary.

5. The basis for the soundness score remains unspecified.

The review raises novelty and empirical-value concerns without specifying the correctness or evidential issue underlying soundness 1. Could you clarify:

  • Soundness: Which theorem, assumption, proof step, procedure or inference is problematic, and why is it difficult to address?
  • Presentation: Which specific passages or logical connections remain unclear?
  • Overall rating: Which concerns remain decisive for the clear-reject recommendation after considering the analyses and evidence above? If novelty or significance remains central, what specific prior results or technical limitations support that assessment?

We respectfully request reconsideration of the overall ratings.

Dear Reviewer BDmi,

As the discussion deadline approaches, we would appreciate your feedback on our rebuttal and the additional evidence. Could you clarify which concerns remain regarding novelty, empirical benefits, and soundness? We would like the opportunity to address any outstanding issues before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.