- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This submission proposes an agreement-based truncation method for ranking LLMs using repeated pairwise judges. For each input and model, the method collects repeated judegments, keeps comparisons where most frequent outcome exceeds an agreement threshold, and fits a BT model to the comparisons.
The submission provides a theoretical analysis under a statistical model of repeated judgements, showing that moderate truncation can reduce asymptotic ranking error while excessive truncation can increase it. Experiments are conducted on MT-Bench with 20 candidate LLMs, using two different judges. It shows improvements in agreement with reference rankings and gains in held-out pairwise preference prediction.
Critical strengths
-
A simple and interpretable approach with theoretical support. The analysis provides some theoretical justification for the observed tradeoff between filtering noisy comparisons and losing useful information.
-
The evaluation is reasonably thorough, with two different LLM judges and compare against several alternatives. The held-out prediction experiments provide evidence beyond agreement with external reference rankings.
Critical weaknesses
-
Methodological novelty is limited and the positioning relative to established ranking literature is not sufficient. Confidence-based filtering and robust estimation from noisy pairwise comparisons have been extensively studied in ranking and related statistical learning literature. While the paper acknowledges some relevant work, it is unclear what fundamental methodological insight is introduced beyond applying a relatively simple filtering heuristic to LLM evaluation. The theoretical analysis relies on a specific statistical model for prompt-dependent comparison probabilities and a local-signal asymptotic regime, so the generalization to realistic LLM evaluation settings is not clear.
-
Limited empirical evidence of practical benefits, particularly given the modest improvements and substantial evaluation cost. The repeated-comparison requirement also raises an efficiency concern. This is particularly relevant for practical LLM leaderboards, where new models must be incorporated continuously.
Questions for authors
When adding a new LLM to an existing ranking, how many additional comparisons are required to accurately position it?
How does the method compare against alternative evaluation strategies under the same total judge-call budget?
Provide uncertainty estimates for the improvements in Table 2, and clarify how reliably the truncation thresholds can be selected without access to the target ranking?
2 comments
Response to Reviewer qWos (2/2)
Q1. How many comparisons does a new model need?
Selection acts on each unit separately. Adding a model therefore requires judging only its new pairs. No existing unit is judged again or reselected. The cutoff also stays the same, because does not depend on the graph. If the new model is compared with existing models on all prompts, the cost is calls before early stopping.
Theorem 3 quantifies the accuracy. The limiting probability of misordering and is , so the design enters only through the effective resistance . Take 19 existing models compared completely, and a new model compared with of them. The worst case is 0.100, 0.158, 0.263 and 0.579 for , 10, 5 and 2, roughly . Existing pairs have . Placing the new model as accurately as the others thus needs about calls. In our setting, this is 30,400 GPT calls, 10% of the archive. With fewer opponents, the formula gives the accuracy that the budget buys. A simulation with , , and illustrates this. The new model is misordered against 20% of the other models with and 13% with , against limits of 22% and 14%.
Q2. Comparison under the same budget of judge calls.
All eight methods in Table 2 use the same archived calls, 20 per GPT unit. Table 2 therefore already compares aggregation rules under the same budget. Across budgets, the theory gives a clear answer. Truncation with has a smaller limiting risk than dense voting with . See the table in our response to Reviewer 2SFD, W1. Under (8), exact early stopping further saves 17% to 26% of the calls at cutoffs between 0.55 and 0.70, with the same output. For Prometheus, agreement comes from token probabilities in two forward passes per unit, with no repetition.
We also want to state the scope clearly. Repetition cannot remove variation across prompts. Under (8), one prompt judged times carries at most times the information of a prompt judged once. When new prompts and their responses are cheap, more prompts are a better use of calls. Repetition and truncation serve the common case of a fixed and curated prompt set, such as MT-Bench. We will state this scope in the revision.
Q3. Uncertainty and selection of the threshold.
Uncertainty. The released report gives bootstrap standard deviations over whole questions for each method separately (Appendix D.2). Comparing these marginal values ignores the correlation between methods evaluated on the same test questions. The uncertainty of a difference should instead come from paired resamples. We will add to Table 2 paired bootstrap intervals for Truncate minus Dense Vote and Truncate minus Full Vote.
Threshold selection. The thresholds in Table 2 are selected without the reference ranking. Validation holds out one training question at a time and scores the judge's own preferences on it (Section 4.3 and Appendix D.3). The selected GPT cutoffs are 0.55 and 0.80 in the two outer folds, and the Prometheus cutoff is 0.55 in both (Table 8). A default without tuning is also available. For GPT with , Theorem 1 gives , which is the fixed variant Truncate-0.70. It attains 265.78, below Dense Vote (268.34) and Full Vote (267.31). The reference ranking is used only in Figure 4 and Table 1. The stars in Figure 4 mark the best cutoff in hindsight, not a selection rule. We will state this explicitly in the caption.
Response to Reviewer qWos (1/2)
We thank the reviewer for the positive remarks on the simplicity of the method, the theory, and the held out evaluation. We address novelty, cost, and the three questions below.
W1. Novelty and positioning.
LLM evaluation differs structurally from classical noisy ranking. It has two levels of randomness: a latent win probability for each prompt and model pair, and repeated calls given that prompt. Agreement among the repeated calls measures how decisive the prompt is. Classical robust methods treat each comparison as a draw from a probability attached to the pair, or as a possible corruption, and they filter edges or pairs. For example, Agarwal et al. (2020) prune edges by cycle consistency. Our selection acts within a pair at the level of the prompt and keeps the same pair on other prompts. It is also related to selective prediction (Geifman and El-Yaniv, 2017), which trades coverage for accuracy on single predictions. Here the abstention applies to the training units of an aggregate estimator, and the tradeoff appears in the ranking error.
The methodological insight is a precise account of this tradeoff.
We will expand the related work on robust and weighted ranking from noisy comparisons. We would welcome specific references.
The model and the local regime. The scaling δ=m−1/2 is the regime where the problem stays informative. For a fixed signal, every cutoff gives a ranking error that vanishes exponentially in m, and comparing cutoffs becomes a question about error exponents. The local regime keeps the limiting error nondegenerate, as with local alternatives in asymptotic statistics. The linear density (8) is the simplest tilt of a uniform baseline. The new weighted result covers any antisymmetric tilt g, and the tie extension covers three outcomes. Outside the theorem, Appendix C (Figure 8) uses a fixed signal with m=15 and K=96. The same U shape appears, with a paired change of −0.0140 (SE 0.0007) from no truncation to the cutoff 65/96.
W2. Size of the gains.
We agree that the gains are modest in absolute terms. For GPT, Truncate lowers the macro average loss of Dense Vote by 1.3%. The evidence for a real effect lies in its consistency across categories. Truncate beats both random controls in 7 of 8 GPT categories and in all 8 Prometheus categories. Truncate-0.70, whose cutoff is not tuned, beats Full Vote and Agree-Wt in 7 of 8 categories for both judges. Against Dense Vote, the gains are less uniform: 6 of 8 categories for both judges with Truncate-0.70. We will state these counts in the revision, together with the bootstrap intervals described under Q3.