acceptodds
Under review as a conference paper at ICLR 2027

Ranking Large Language Models via Comparison Graph Truncation

Abstract

Large language model (LLM) evaluation is a central problem in modern machine learning. While benchmark scores provide a standard way to compare models on many tasks, evaluating open-ended responses remains challenging because of the lack of a single correct answer. Indeed, pairwise comparison offers a practical approach, but noisy judgments can make the resulting rankings unstable. To address this issue, we repeat each comparison and apply truncation to retain only comparisons with sufficient agreement. Specifically, our method removes comparisons with weak agreement while retaining enough evidence to rank the models. Theoretically, we prove that moderate truncation can reduce ranking error, while removing too many comparisons can increase it. Additionally, we conduct simulations and experiments with real LLMs, showing that our method improves ranking accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, ranking, comparison graph

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qWosOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This submission proposes an agreement-based truncation method for ranking LLMs using repeated pairwise judges. For each input and model, the method collects repeated judegments, keeps comparisons where most frequent outcome exceeds an agreement threshold, and fits a BT model to the comparisons.

The submission provides a theoretical analysis under a statistical model of repeated judgements, showing that moderate truncation can reduce asymptotic ranking error while excessive truncation can increase it. Experiments are conducted on MT-Bench with 20 candidate LLMs, using two different judges. It shows improvements in agreement with reference rankings and gains in held-out pairwise preference prediction.

Critical strengths
  1. A simple and interpretable approach with theoretical support. The analysis provides some theoretical justification for the observed tradeoff between filtering noisy comparisons and losing useful information.

  2. The evaluation is reasonably thorough, with two different LLM judges and compare against several alternatives. The held-out prediction experiments provide evidence beyond agreement with external reference rankings.

Critical weaknesses
  1. Methodological novelty is limited and the positioning relative to established ranking literature is not sufficient. Confidence-based filtering and robust estimation from noisy pairwise comparisons have been extensively studied in ranking and related statistical learning literature. While the paper acknowledges some relevant work, it is unclear what fundamental methodological insight is introduced beyond applying a relatively simple filtering heuristic to LLM evaluation. The theoretical analysis relies on a specific statistical model for prompt-dependent comparison probabilities and a local-signal asymptotic regime, so the generalization to realistic LLM evaluation settings is not clear.

  2. Limited empirical evidence of practical benefits, particularly given the modest improvements and substantial evaluation cost. The repeated-comparison requirement also raises an efficiency concern. This is particularly relevant for practical LLM leaderboards, where new models must be incorporated continuously.

Questions for authors

When adding a new LLM to an existing ranking, how many additional comparisons are required to accurately position it?

How does the method compare against alternative evaluation strategies under the same total judge-call budget?

Provide uncertainty estimates for the improvements in Table 2, and clarify how reliably the truncation thresholds can be selected without access to the target ranking?

2 comments
AuthorsOpenReview ↗

Response to Reviewer qWos (1/2)

We thank the reviewer for the positive remarks on the simplicity of the method, the theory, and the held out evaluation. We address novelty, cost, and the three questions below.

W1. Novelty and positioning.

LLM evaluation differs structurally from classical noisy ranking. It has two levels of randomness: a latent win probability for each prompt and model pair, and repeated calls given that prompt. Agreement among the repeated calls measures how decisive the prompt is. Classical robust methods treat each comparison as a draw from a probability attached to the pair, or as a possible corruption, and they filter edges or pairs. For example, Agarwal et al. (2020) prune edges by cycle consistency. Our selection acts within a pair at the level of the prompt and keeps the same pair on other prompts. It is also related to selective prediction (Geifman and El-Yaniv, 2017), which trades coverage for accuracy on single predictions. Here the abstention applies to the training units of an aggregate estimator, and the tradeoff appears in the ranking error.

The methodological insight is a precise account of this tradeoff.

  1. Theorem 1 gives the exact limiting risk at every cutoff h>K/2h>K/2 and the optimal cutoff h⋆=⌊(2K+3)/3⌋h^\star=\lfloor(2K+3)/3\rfloor. By Theorem 3, h⋆h^\star does not depend on nn, the scores, or the comparison graph.
  2. New: for any weight on agreement, the limiting risk depends on a single index ρ(w)\rho(w). The optimal weight is the positive part of the average signal at each agreement level. In the profiles we examined, truncation stays within 11% of the optimum in effective sample size. Linear weights are optimal under (8) but can lose much more when units with low agreement point the wrong way. See our response to Reviewer 6qV5, point 1.
  3. New: under the half win convention, ties do not change the limiting risk, but they lower the optimal cutoff. See our response to Reviewer 2SFD, W3.

We will expand the related work on robust and weighted ranking from noisy comparisons. We would welcome specific references.

The model and the local regime. The scaling δ=m−1/2\delta=m^{-1/2} is the regime where the problem stays informative. For a fixed signal, every cutoff gives a ranking error that vanishes exponentially in mm, and comparing cutoffs becomes a question about error exponents. The local regime keeps the limiting error nondegenerate, as with local alternatives in asymptotic statistics. The linear density (8) is the simplest tilt of a uniform baseline. The new weighted result covers any antisymmetric tilt gg, and the tie extension covers three outcomes. Outside the theorem, Appendix C (Figure 8) uses a fixed signal with m=15m=15 and K=96K=96. The same U shape appears, with a paired change of −0.0140-0.0140 (SE 0.00070.0007) from no truncation to the cutoff 65/9665/96.

W2. Size of the gains.

We agree that the gains are modest in absolute terms. For GPT, Truncate lowers the macro average loss of Dense Vote by 1.3%. The evidence for a real effect lies in its consistency across categories. Truncate beats both random controls in 7 of 8 GPT categories and in all 8 Prometheus categories. Truncate-0.70, whose cutoff is not tuned, beats Full Vote and Agree-Wt in 7 of 8 categories for both judges. Against Dense Vote, the gains are less uniform: 6 of 8 categories for both judges with Truncate-0.70. We will state these counts in the revision, together with the bootstrap intervals described under Q3.

AuthorsOpenReview ↗

Response to Reviewer qWos (2/2)

Q1. How many comparisons does a new model need?

Selection acts on each unit separately. Adding a model therefore requires judging only its new pairs. No existing unit is judged again or reselected. The cutoff also stays the same, because h⋆h^\star does not depend on the graph. If the new model vv is compared with dd existing models on all mm prompts, the cost is dmKdmK calls before early stopping.

Theorem 3 quantifies the accuracy. The limiting probability of misordering vv and jj is Φ(−2cK∣θv⋆−θj⋆∣t1−t/Rvj)\Phi\big(-\sqrt2c_K|\theta_v^\star-\theta_j^\star|t\sqrt{1-t}/\sqrt{\mathsf R_{vj}}\big), so the design enters only through the effective resistance Rvj\mathsf R_{vj}. Take 19 existing models compared completely, and a new model compared with dd of them. The worst case Rvj\mathsf R_{vj} is 0.100, 0.158, 0.263 and 0.579 for d=19d=19, 10, 5 and 2, roughly 1/d+1/n1/d+1/n. Existing pairs have R=2/n=0.1\mathsf R=2/n=0.1. Placing the new model as accurately as the others thus needs about (n−1)mK(n-1)mK calls. In our setting, this is 30,400 GPT calls, 10% of the archive. With fewer opponents, the formula gives the accuracy that the budget buys. A simulation with n=20n=20, K=20K=20, m=400m=400 and τ=0.70\tau=0.70 illustrates this. The new model is misordered against 20% of the other models with d=5d=5 and 13% with d=19d=19, against limits of 22% and 14%.

Q2. Comparison under the same budget of judge calls.

All eight methods in Table 2 use the same archived calls, 20 per GPT unit. Table 2 therefore already compares aggregation rules under the same budget. Across budgets, the theory gives a clear answer. Truncation with K=10K=10 has a smaller limiting risk than dense voting with K=20K=20. See the table in our response to Reviewer 2SFD, W1. Under (8), exact early stopping further saves 17% to 26% of the calls at cutoffs between 0.55 and 0.70, with the same output. For Prometheus, agreement comes from token probabilities in two forward passes per unit, with no repetition.

We also want to state the scope clearly. Repetition cannot remove variation across prompts. Under (8), one prompt judged KK times carries at most 3K/(K+2)<33K/(K+2)<3 times the information of a prompt judged once. When new prompts and their nn responses are cheap, more prompts are a better use of calls. Repetition and truncation serve the common case of a fixed and curated prompt set, such as MT-Bench. We will state this scope in the revision.

Q3. Uncertainty and selection of the threshold.

Uncertainty. The released report gives bootstrap standard deviations over whole questions for each method separately (Appendix D.2). Comparing these marginal values ignores the correlation between methods evaluated on the same test questions. The uncertainty of a difference should instead come from paired resamples. We will add to Table 2 paired bootstrap intervals for Truncate minus Dense Vote and Truncate minus Full Vote.

Threshold selection. The thresholds in Table 2 are selected without the reference ranking. Validation holds out one training question at a time and scores the judge's own preferences on it (Section 4.3 and Appendix D.3). The selected GPT cutoffs are 0.55 and 0.80 in the two outer folds, and the Prometheus cutoff is 0.55 in both (Table 8). A default without tuning is also available. For GPT with K=20K=20, Theorem 1 gives h⋆/K=14/20=0.70h^\star/K=14/20=0.70, which is the fixed variant Truncate-0.70. It attains 265.78, below Dense Vote (268.34) and Full Vote (267.31). The reference ranking is used only in Figure 4 and Table 1. The stars in Figure 4 mark the best cutoff in hindsight, not a selection rule. We will state this explicitly in the caption.

Reviewer_6qV5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a truncation method to filter unreliable pairwise comparisons, improving LLM ranking accuracy.

Critical strengths

The research problem and main method is easy to understand, comparison to related works makes sense.

Critical weaknesses

Weakness: The paper post a very intuitive and easy method, while it has potential to be a very good one, the current version has several concerns remaining:

  1. Why hard truncation rather than a better soft weighting? Hard truncation can be treated as an extreme case as soft weighing, so why zero? Current theory analysis in section 3 doesn’t cover this part.
  2. Why did your method outperform 2 baselines (even not in all cases)? Though 2 types of indirect evidence are provided, the current version is not convincing enough, especially pointing to the main claim.
  3. You have to dive into the motivation, not just an obvious phenom, but what caused it? Casual relation is hard, but correlation analysis is still needed. These main concerns stop this paper to be presented as a Top-tier venue paper, but instead an incremental work. Typo and minor revisions:
  4. Fig.2 is confusing, regarding the color map. It’s hard to follow which color refers to a ‘stronger’ one.
  5. Stacking Fig.1 and fig.2 is generally not a good idea.
  6. Current sec 3, with some new theoretical analysis of failure mode should appear before the method section.
  7. It’s rare to see current sec 1.1 as a contribution summary, and sec 1.2 as related work.
Questions for authors

No

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6qV5 (1/2)

We thank the reviewer for the detailed comments. The main concerns are the choice of hard truncation, the mechanism behind the gains, and the motivation. We answer them with a new theoretical result and with evidence in the paper that we did not present clearly enough. Our contribution is not only a filtering rule. It is an exact description of how selection by agreement changes the ranking error, which now covers soft weights and ties.

1. Why hard truncation rather than soft weighting?

We have extended Theorem 1 to soft weights. Let w(u)≥0w(u)\ge0 be a weight on the number uu of votes for model ii, not identically zero, with w(u)=w(K−u)w(u)=w(K-u). Then ww depends on a unit only through its agreement C=max⁡{u,K−u}/KC=\max\{u,K-u\}/K. Fit the BT model with weight w(Usij)w(U_{sij}) on each unit. Under the assumptions of Theorem 1, the argument of Theorem 3 gives

lim⁡m→∞Rm,K(w)=1(n2)∑i<jΦ(−n/2∣θi⋆−θj⋆∣ρ(w)),ρ(w)=∑u=0Kw(u)∣2u−K∣(K+2)K+1(∑u=0Kw(u)2)1/2.\lim_{m\to\infty}\mathcal R_{m,K}(w)=\frac{1}{\binom n2}\sum_{i<j}\Phi\Big(-\sqrt{n/2}|\theta_i^\star-\theta_j^\star|\rho(w)\Big),\quad\rho(w)=\frac{\sum_{u=0}^{K}w(u)|2u-K|}{(K+2)\sqrt{K+1}\big(\sum_{u=0}^{K}w(u)^2\big)^{1/2}}.

The numerator measures the signal that the weight keeps, and the denominator the noise. The curvature of the objective cancels. Truncation at hh gives ρ=2cKth1−th\rho=\sqrt2c_Kt_h\sqrt{1-t_h}, which recovers Theorem 1.

By the Cauchy–Schwarz inequality, ρ(w)≤K/(3(K+2))\rho(w)\le\sqrt{K/(3(K+2))}, with equality if and only if w∝2C−1w\propto2C-1. So hard truncation is not optimal under (8). More generally, let the latent probability have density 1+ηg(q)1+\eta g(q), with η=δ(θi⋆−θj⋆)\eta=\delta(\theta_i^\star-\theta_j^\star) and g(1−q)=−g(q)g(1-q)=-g(q). The optimal weight is then w⋆(u)∝max⁡{s(u),0}w^\star(u)\propto\max\{s(u),0\}, where s(u)=sgn(2u−K)E0[g(Q)∣U=u]s(u)=\mathrm{sgn}(2u-K)\mathbb E_0[g(Q)\mid U=u] is the average signal of units with uu votes. The best shape therefore depends on how the signal varies across agreement levels. The table gives (ρ/ρ⋆)2(\rho/\rho^\star)^2, the effective sample size relative to the optimal weight, for K=20K=20 and four profiles. In the last two, prompts with ∣2Q−1∣<s0|2Q-1|<s_0 favor the weaker model.

signal profiledenseAgree-Wt (w=Cw=C)Full Votetruncation
linear, model (8)0.750.901.000.89
signal only where ∣2Q−1∣≥1/3\lvert 2Q-1\rvert\ge1/30.780.910.980.94
reversed where ∣2Q−1∣<1/3\lvert 2Q-1\rvert<1/30.530.710.920.95
reversed where ∣2Q−1∣<1/2\lvert 2Q-1\rvert<1/20.100.250.580.95

Full Vote fits vote proportions. In the binary model, its score at θ=0\theta=0 coincides with that of the weight 2C−12C-1, and only the curvature differs. Hence it is optimal under (8). Truncation, with its best cutoff, stays within 11% of the optimum in all four profiles. Each soft rule is nearly optimal for one profile but loses much more when units with low agreement point the wrong way. A Monte Carlo check confirms the formula under (8), with n=10n=10, K=20K=20, m=400m=400 and 1000 replications (SE about 0.002). The Kendall risks are 0.155 for dense voting (limit 0.154), 0.140 for truncation at 14/2014/20 (limit 0.140), 0.140 for Agree-Wt (limit 0.140), and 0.132 for Full Vote (limit 0.131).

The data match this picture. In Table 1, GPT units in the lowest agreement band agree with the reference only 44.36% of the time, below one half. For Prometheus, the same band gives 54.55%. Correspondingly, for GPT, Truncate beats Full Vote and Agree-Wt in 7 of 8 categories and on average (264.90 against 267.31 and 267.34). For Prometheus, tuned Truncate is close to them on average but not better (326.81 against 326.59 and 326.78). The fixed cutoff 0.70 is best for Prometheus (325.18).

Truncation also has two practical advantages. It depends only on the order of the units by agreement, so any increasing transformation of CC leaves it unchanged. Soft weights also depend on the numerical scale of CC, which has no calibrated meaning here. Truncation also has a single parameter, selected by validation on the judge's own held out preferences. We will add the weighted result and the table above to Section 3 and Appendix A.

AuthorsOpenReview ↗

Response to Reviewer 6qV5 (2/2)

2. Why does truncation outperform the baselines, and why not in all cases?

The paper contains three pieces of direct evidence beyond the two indirect checks. We will make them explicit in the revision.

(a) A randomized control. Random and Random-Tuned remove the same number of units on each question as the selected truncation, but uniformly. They are slightly worse than Dense Vote on average (GPT: 269.24 and 268.94 against 268.34). Truncate is better (264.90). It has a lower loss than both random controls in 7 of 8 categories for GPT and in all 8 for Prometheus. The gain therefore comes from which units are removed, not from sparsity or from the changed balance between data and ridge.

(b) A test on single units. Table 1 measures, before any aggregation, how often a modal judgment agrees with the reference. This rate increases across all six agreement bands for both judges: from 44.36% to 75.21% for GPT and from 54.55% to 90.52% for Prometheus.

(c) A quantitative criterion. In the local regime, removing a group of units lowers the limiting risk if and only if (1−a)2>1−b(1-a)^2>1-b. Here aa is the share of the total signal carried by the group and bb its share of the retained units. To first order, removal helps when the group carries less than half of its proportional share of signal. Units with low agreement carry little signal, so removing them helps at first. Once most remaining units are informative, further removal hurts. This explains the U shape in Theorem 1.

The same criterion explains why the gain is not universal. When units with low agreement still carry signal roughly in proportion to their agreement, vote proportions are competitive (point 1). Table 1 shows this difference between the judges. The lowest band is below one half for GPT but above it for Prometheus. Accordingly, tuned Truncate beats the soft rules for GPT but not for Prometheus.

3. Motivation: what causes low agreement?

In our model, a unit has low agreement when its latent win probability QsijQ_{sij} is near 1/21/2, that is, when the two responses are close in quality on that prompt. Known judge biases also produce low agreement in our design. Each GPT unit pools ten calls in each presentation order. If the judge follows the position, the two orders give opposite verdicts, and the unit has CsijC_{sij} near 1/21/2. Truncation removes such units by construction. Position bias (Zheng et al., 2023; Shi et al., 2025) is thus one concrete cause that truncation filters. In the revision, we will report the correlation of CsijC_{sij} with the reference rank gap of the pair and with the consistency between the two presentation orders. Both are computed from the archived judgments. As the reviewer notes, such an analysis is correlational, but it helps to distinguish these explanations.

4 to 7. Presentation.

  1. We will redraw Figure 2 with two discrete edge styles, solid for retained units and dashed for removed units, and a legend instead of the continuous color map. The figure will also be readable in grayscale.
  2. We will separate Figures 1 and 2 and place each next to the text that cites it.
  3. The theorem analyzes the estimator defined in Section 2, so it needs that definition. Before the method, we will add a short paragraph on the failure mode of dense voting: units with low agreement carry little signal but receive full weight. The criterion in point 2(c) quantifies this failure. The formal analysis stays after the method.
  4. We will fold the contribution list into the introduction and give related work its own section.
Reviewer_2SFDOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a ranking method of large language model (LLM) based on "comparison graph truncation". The main idea of this method is to compare the same model for many times, and then calculate the consistency of the judgment results. The system sets a specific threshold to eliminate the comparison data with weak consistency, and then fit the model on the retained comparison chart to generate the final ranking. This paper constructs a statistical model in theory, and proves that moderate truncation can reduce the U-shaped rule of ranking error. In addition, the experimental results on MT bench show that this method can improve the consistency of ranking and external human preferences, and reduce non contradictory judgments

Critical strengths
  1. Solid theoretical foundation: by building a statistical model, the author mathematically proves the nonlinear U-shaped trade-off relationship between truncated ranking and ranking error, which provides a strong theoretical support for the core idea of this paper, "moderate truncation will be effective", and clearly shows how moderate truncation can reduce noise through the common theorem, while excessive truncation will lead to the loss of useful information.
  2. The experimental effect is obvious: in the experimental verification of MT bench, the fuzzy judgment is eliminated through consistent truncation, which significantly reduces the proportion of "incompatible triangles" in the reserved comparison. From the data level, this method successfully alleviates the common logic cycle problems of a>b, b>c, c>a in LLM referee, and makes the final comparison chart more support the establishment of stable and consistent ranking.
Critical weaknesses
  1. High complexity and lack of scalability of the core module: the core design of the paper requires repeated referee comparisons of cue words and model pairs up to K times, which will lead to a situation: in the context of N candidate models, N2 comparisons are required, and the overall API reasoning call complexity is up to K · N2. In today's large-scale scenarios where LLM evaluation involves a large number of candidate models, this crude global repeated sampling design is very inefficient. The paper does not propose mechanisms such as adaptive sampling or early stop to optimize this short board of complexity.
  2. Lack of comparison with the latest baseline in the same field: Although the paper introduces consistent truncation to denoise, in the experimental part, the comparison baseline of prediction preference is only the basic dense vote, full vote and simple random truncation. According to the latest field research, a large number of advanced calibration and sorting denoising technologies have emerged in the past two years (such as filtering based on graph structure, or anti destruction sorting mechanism, etc.). Although a similar robust ranking method is mentioned in the related work, it is not compared as a baseline in the experiment, which makes it questionable whether the method really reaches or exceeds the current SOTA level.
  3. Disconnection between theoretical derivation and experimental verification: in the part of theoretical proof, in order to simplify the analysis, clear assumptions were made to exclude the existence of a draw, and the judgment results were strictly limited to+1 and -1. however, in the actual MT bench experimental evaluation, LLM referees' draws were a large number of, and in the BT model fitting, the draw was treated as half points for both sides. This simplification led to the fact that the core theoretical proof could not rigorously cover the actual experimental scene, which seriously weakened the effectiveness of the theoretical and experimental closed-loop.
Questions for authors
  1. In view of the high reasoning cost mentioned above, does the author consider introducing a strategy similar to adaptive sampling, such as sampling only K times for the duel with large differences in preliminary judgment, and stopping sampling for the duel extraction with high consistency in preliminary judgment.
  2. In the experimental verification, only non truncation and random truncation are compared, and there is no comparison with the advanced denoising sorting algorithm. Can the author supplement the relevant comparative experiments in the past two years
  3. The theoretical model assumes that there is no tie, but in the real LLM evaluation, the tie is not only universal but also has important information. To what extent does this assumption bias affect the derivation and generalization of the optimal cutoff point in Theorem 1
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2SFD (1/2)

We thank the reviewer for the positive assessment of the theory and of the transitivity results, and for the concrete questions on cost, baselines and ties.

W1/Q1. Cost and adaptive sampling.

The design. The quadratic number of pairs comes from the complete design of our experiments, as in standard pairwise evaluation on MT-Bench. The method does not require it. Truncation acts on each unit separately, so it applies to any comparison design. Theorem 3 in Appendix A holds for every connected comparison graph. Each summand of its limit depends on the cutoff only through th1−tht_h\sqrt{1-t_h}. Hence h⋆=⌊(2K+3)/3⌋h^\star=\lfloor(2K+3)/3\rfloor is optimal for every connected graph. Truncation can therefore be combined with sparse or active pair selection, such as Heckel et al. (2019), without retuning the cutoff.

The number of repetitions. The gain from truncation does not need a large KK. In the local regime, the limiting risk depends on the design through mρ2m\rho^2, where ρ=2cKth1−th\rho=\sqrt2c_Kt_h\sqrt{1-t_h} is the standardized signal of one prompt in Theorem 1. A prompt judged once has ρ=1/3\rho=1/3. The table reports 9ρ29\rho^2, the number of prompts judged once that carry the same information as one prompt judged KK times.

KK461020
dense voting1.441.651.862.04
truncation at h⋆h^\star1.802.012.232.43

Thus truncation with K=10K=10 calls has a smaller limiting risk than dense voting with K=20K=20 calls, and K=6K=6 nearly matches it. Repetition is not even necessary when token probabilities are available. Our Prometheus judge obtains its agreement score from two forward passes per unit, and Truncate-0.70 gives its best result in Table 2 (325.18, against 328.74 for Dense Vote).

Early stopping. Truncation admits exact early stopping. For a cutoff h>K/2h>K/2, stop judging a unit once one outcome reaches hh votes. The unit is then retained, and its modal outcome is fixed. Also stop once no outcome can still reach hh. The unit is then removed. This rule returns the same retained set and the same modal outcomes as all KK calls. Under (8) with K=20K=20, it saves 17% of the calls at τ=0.70\tau=0.70 and 26% at τ=0.55\tau=0.55. A simulation of 200,000 units confirms both the savings and the identical output. The two stage scheme suggested by the reviewer relaxes this rule. It saves more calls but changes the retained set. We will describe both schemes in the revision.

AuthorsOpenReview ↗

Response to Reviewer 2SFD (2/2)

W2/Q2. Baselines.

Methods already compared. Table 2 compares eight methods, not only untruncated and random retention. Full Vote fits the vote proportions of every unit. Agree-Wt weights each modal outcome by its agreement. Random and Random-Tuned remove the same number of units on each question as truncation, but uniformly. All methods use the same archived judgments.

Other aggregators. Truncation selects units before aggregation, so it complements robust aggregators. Our theory shows that, in the local regime, the choice among common aggregators does not change the effect of truncation. The BT fit is linear in the net retained wins BhB_h to first order (Appendix A). HodgeRank (Jiang et al., 2011), weighted by retained counts, solves a least squares problem with the same first order solution. On the complete graph, both reduce to Borda count of BhB_h (Shah and Wainwright, 2018). These three aggregators therefore have the same limiting risk at every cutoff. A simulation confirms this. It uses model (8) with n=10n=10 equally spaced scores, K=20K=20, m=400m=400 and 1000 replications, and applies the three aggregators to the same data. The SE is about 0.002.

cutoff τ\tau0.50 (no truncation)0.700.85
BT0.15350.13900.1489
HodgeRank0.15340.13900.1488
Borda count0.15340.13960.1484
limit0.15370.14010.1534

The three aggregators agree at every cutoff. At τ=0.85\tau=0.85, all three sit slightly below the limit, which reflects finite mm. Appendix C gives a further check for Borda count: its exact finite sample risk differs from the BT limit by at most 1.4×10−31.4\times10^{-3} at m=25m=25.

Methods with a different goal. Agarwal et al. (2020) prune whole model pairs by cycle consistency to resist adversarial corruption. On a complete design, this removes all prompts of a pair. Truncation removes single units and keeps the pair on other prompts. Huang et al. (2026) select entire comparison graphs built from prompt perturbations. Truncation needs no perturbations and selects within each graph.

In the revision, we will report Borda and Davidson fits with and without truncation, computed from the archived judgments. If the reviewer has specific recent methods in mind, we would be glad to discuss them.

W3/Q3. Ties.

We agree that this gap should be closed, and we now extend the theory to ties. Suppose each call is a tie with probability π\pi and otherwise follows the binary model (8). The proof of Theorem 3 carries over with two changes.

Retained ties do not change the limiting risk. A retained unit whose modal outcome is a tie has target 1/21/2. Its gradient at θ=0\theta=0 vanishes. It therefore adds curvature to the objective but no signal or noise to the score equation. It shrinks the fitted scores without changing the limit of their signs. As a consequence, the half win convention and removing tie units give the same limiting risk.

Ties lower the optimal cutoff. Ties dilute the largest vote share CsijC_{sij}, which moves the optimal cutoff down. The limit keeps the form of Theorem 1, with 2cKt1−t\sqrt2c_Kt\sqrt{1-t} replaced by μτ/vτ\mu_\tau/\sqrt{v_\tau}. Here μτ\mu_\tau is the derivative of E[Y^1{C≥τ}]\mathbb E[\widehat Y\mathbf 1\{C\ge\tau\}] with respect to η=δ(θi⋆−θj⋆)\eta=\delta(\theta_i^\star-\theta_j^\star) at η=0\eta=0. The quantity vτv_\tau is the probability that a unit is retained with a strict modal outcome. For K=20K=20:

tie rate π\pi00.10.20.3
optimal cutoff τ⋆\tau^\star0.700.600.550.50
gain over no truncation in effective sample size1.191.161.131.10

The optimal cutoff stays close to 2/32/3 of the expected share of strict votes, (2/3)(1−π)(2/3)(1-\pi). The gain decreases with π\pi but persists.

A Monte Carlo check uses n=10n=10 equally spaced scores, K=20K=20, m=400m=400, π=0.2\pi=0.2 and 600 replications. At τ=0.35\tau=0.35 (no truncation), 0.550.55 and 0.700.70, the Kendall risks are 0.152, 0.145 and 0.163, against limits 0.155, 0.146 and 0.168. Removing tie units instead gives 0.153, 0.145 and 0.163.

We will add this extension to Section 3 and Appendix A, and report the empirical tie rates of both judges.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.