- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares different ranking methods for pairwise comparisons. QRE is one of the good ranking algorithms as it satisfies several nice properties while some other ranking algorithms including BT might not satisfy simultaneously. Then they discuss the trade-off between the social choice guarantees and stability of QRE through the temperature.
Critical strengths
The trade-off is an interesting observation.
Critical weaknesses
The scope is a bit narrow.
Questions for authors
see above.
2 comments
Response to Reviewer TkyE (2/2)
3. Empirical evidence of the joint benefit. The original Table 1 compares social-choice properties, while the original Table 2 (Table 4 in the revision) evaluates sampling stability. Revised Figure 2 evaluates sampling stability and candidate-pool robustness together: QRE configurations form the observed Pareto frontier among the evaluated BT, SCO, iterative MaxL, and QRE settings, across both datasets and both insertion and deletion experiments. At a single temperature, , QRE improves on BT on both axes:
| % (BT → QRE) | Arena | Image |
|---|---|---|
| Resampling Kendall distance | 2.68 → 2.50 | 1.85 → 1.71 |
| Models changing rank, insertion | 38.83 → 3.33 | 41.63 → 2.85 |
| Models changing rank, deletion | 12.79 → 3.87 | 3.58 → 2.04 |
This demonstrates the practical value of considering these objectives jointly.
Generality. Theorems 1–3 are stated for general antisymmetric comparison matrices estimated from pairwise outcomes, so nothing is specific to LLMs. The analysis applies wherever pairwise comparisons are aggregated into a ranking, including agent evaluation in games (Bertrand et al., 2023; Lanctot et al., 2023; 2025). Our experiments cover two modalities and scales: text (28 models, 71,866 votes) and image (30 models, 2.16M annotations).
We would greatly appreciate it if the reviewer could let us know which aspect of the scope they find limiting, whether the theoretical setting, the empirical results, or the significance of the contributions, so that we can address it directly. Even a short answer would help us greatly in improving the paper. We hope the clarification above is useful and we kindly request the reviewer to reconsider their rating in light of it.
- Bertrand, Q., Czarnecki, W. M., & Gidel, G. (2023). On the limitations of the Elo: Real-world games are transitive, not additive. AISTATS.
- Lanctot, M., et al. (2023). Evaluating agents using social choice theory. arXiv:2312.03121.
- Lanctot, M., et al. (2025). Soft Condorcet optimization for ranking of general agents. AAMAS.
- McKelvey, R. D., & Palfrey, T. R. (1995). Quantal response equilibria for normal form games. Games and Economic Behavior, 10(1), 6–38.
Response to Reviewer TkyE (1/2)
We thank the reviewer for reading our paper. We understand the main concern to be that the scope is narrow. Since the review does not specify which aspect of the scope this refers to, we would be grateful for more detail, as it would help us improve the paper. In the meantime, we clarify the scope and contributions below. The other reviews and our responses to them may also provide useful context, as they discuss several of these points in more detail.
Our scope is the reliability of preference-based leaderboards, covering both comparison estimation and ranking. The review’s summary describes only the ranking part and omits the estimation half of the paper (Sec. 3, 5.3). The paper connects social-choice requirements with statistical reproducibility through three contributions.
1. A systematic analysis of why existing leaderboard methods can be unreliable. We compare BT/Elo, SCO, and MaxL through their social-choice properties and statistical stability (Table 1).
These results explain why sampling stability alone is insufficient to assess leaderboard reliability. We address these limitations with QRE, a solution concept that originated in behavioral game theory (McKelvey & Palfrey, 1995). Unlike BT, it responds to non-transitive structure; unlike SCO, it is a unique, efficiently computable equilibrium; and unlike MaxL, it returns a full ranking and varies continuously with the comparison matrix, so the leaderboard it reports is reproducible under resampling.
2. Quantitative guarantees for addressing these limitations. For comparison estimation, Theorem 1 separates approximation, regularization, and sampling error, quantifying the statistical cost of retaining richer preference structure. For ranking, Theorems 2–3 establish an explicit temperature threshold for Condorcet consistency, a bound on weak-entrant influence that vanishes exponentially as τ→0, and Lipschitz sensitivity to comparison-matrix errors. Combining Theorems 1 and 3 gives sufficient conditions for preserving population QRE orderings. These contributions explain how representation capacity, available data, and regularization affect the resulting rankings.