acceptodds
Under review as a conference paper at ICLR 2027

Rank Confidence Sequences: Anytime-Valid Leaderboards

231 views

Abstract

Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model’s rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level , in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models’ scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM evaluation, leaderboards, ranking, e-values, anytime-valid inference, multiple testing, family-wise error rate

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ccdQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper considers MM models and constructs anytime-valid confidence intervals for each model’s rank. Strictly speaking, it first constructs confidence sets and then converts them into intervals. The approach is to construct an e-process for each possible ordering. An interesting aspect of the construction is that, for a candidate ordering WW, the authors identify the pairwise order constraints implied by WW, construct an e-process for each constraint, and then average these pairwise e-processes to obtain an e-process for WW.

Critical strengths
  1. The paper studies an interesting problem and connects statistical tools (i.e. e-process) with LLM evaluation. This connection is valuable because it demonstrates how statistical methods can make model comparisons more reliable.
  2. The method is intuitive: it progressively eliminates candidate orderings using pairwise evidence and constructs rank confidence sets from the remaining orderings.
Critical weaknesses

The rank confidence intervals appear relatively wide, and exact computation can be expensive. Although the paper provides computational shortcuts that preserve anytime validity, these shortcuts may introduce additional conservatism and further widen the intervals.

Questions for authors
  1. Could λt\lambda_t be chosen adaptively from the observed data? Although the appendix discusses adaptive strategies, the reported results seem to favor averaging over a prespecified grid. I am not fully convinced that reliance on a fixed grid is the most suitable approach. I would expect a data-driven choice of λt\lambda_t to adapt better to the underlying model differences.
  2. With the same pairwise e-processes and model-specific retirement rule, how much evaluation cost does the proposed method save compared with e-Bonferroni?
3 comments
AuthorsOpenReview ↗

New experiments: interval width, computation, adaptive betting, and evaluation savings

Thank you for your constructive feedback. We ran additional experiments to address your questions and concerns.

W1. Width and computation. We compared our mean rank-interval widths with Romano–Wolf at halfway in the finite-benchmark setting (F) across twelve 20-model settings. Our widths differed by at most 0.71 ranks and were narrower in nine settings. Our method also provides anytime-valid guarantees. Table 7 shows that exact certification adds only 1.4 true dominances on average over the shortcut at halfway, out of 179–190. Across all checkpoints in the 20-model experiments, the shortcut's mean interval width was at most about 0.33 ranks larger than the exact test's.

Algorithm 3 solved all tested integer programs. We also measured runtime for different numbers of models. Mean seconds per run (checks approximately every 1% of the benchmark, 10 item orders per benchmark):

ModelsExact test (s)Shortcut (s)Shortcut's extra width at halfway (ranks)
80.20.010.15
120.60.010.18
208.40.010.20
30810.020.42

The last column reports the largest benchmark-mean width difference. The shortcut greatly reduces computation with limited additional conservatism.

Q1. Why not a data-driven λt\lambda_t? Data-driven bets are already permitted. Any predictable bet in [0,1][0,1] preserves validity (Lemma B.1). Our default grid is itself a wealth-weighted adaptive mixture, equivalent to a predictable effective bet (Remark B.3).

We compared alternatives on six real benchmarks and two 20-model panels, with 50 item orders each. Changes in certified true dominances at halfway, in percentage points relative to ours:

Betting methodDifference from ours
Ours: default 5-point mixture0 (baseline)
Plug-in (approximate GRAPA)−1.6 to +1.9
Online Newton−1.8 to +1.3
Dense 50-point mixture−0.6 to +2.6
Ours + online Newton as a sixth component−0.2 to +0.6

The default five-point mixture remains competitive with data-driven alternatives without benchmark-specific tuning. Adding online Newton as a sixth mixture component also preserves validity (Lemma B.1). This extension improves final certification in all 12 settings by 0.04–0.98 percentage points. It also reduces top-3 evaluation cost in all 12 settings by 0.02–0.62 percentage points of full evaluation.

Q2. Savings versus e-Bonferroni. We ran new experiments comparing evaluation costs with e-Bonferroni. All methods used the same pairwise betting construction and identical item orders. Each method applied the top-3 retirement rule based on its own certification decisions.

On all 395 models (50 orders per benchmark), the shortcut's evaluation cost is within about 0.01 percentage points of e-Bonferroni's in every order, and e-Bonferroni is never cheaper.

On the 8- and 20-model panels (six benchmarks, 50 orders each), exact weak-order certification reduces average evaluation cost in all 18 settings. Mean savings relative to e-Bonferroni, in percentage points of full evaluation is shown below:

Exact vs. e-BonferroniMMLUHellaSwagGSM8KWinoGrandeARCTruthfulQA
Top 80.311.511.580.260.981.33
Top 200.080.540.750.290.250.39
Spread 200.120.740.310.780.890.80

The exact test never requires more evaluations in any of the 900 runs. The shortcut remains within 0.03 percentage points of e-Bonferroni on average. These experiments demonstrate the benefit of weak-order certification. The exact test consistently saves more evaluations than e-Bonferroni across all 18 settings. The shortcut provides a faster alternative and certifies at least what e-Bonferroni does (Section 4.1).

Plan. We will add the width and runtime results to Appendix E, the betting comparisons and online Newton mixture extension to E.6, and the evaluation-cost comparison to E.3.

Thank you for the additional experiments and detailed responses. My questions have been largely addressed, and I maintain my positive assessment of the paper.

Thank you for your follow-up and positive assessment. We appreciate your time and feedback.

Reviewer_2zLDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses anytime-valid confidence sets for the ranks of M models scored on the same benchmark items, where scores are dependent and the leaderboard may be inspected after every item. The paper provides a few theoretical and simulation results. On the Open LLM Leaderboard per-item data, the method certifies a growing fraction of true dominances. It is roughly on par with fixed-sample methods at a pre-planned look at half the benchmark.

Critical strengths
  1. Section 3 is a clear description of the method, and the “what needs a guarantee / result / relation to existing results” structure is helpful. However, the paper is extremely dense. Key claims are scattered across a very long appendix, and the main text compresses many numbers into single paragraphs (E1–E4). The tier definitions, the shortcut statistic and the finite-benchmark offset are hard to digest without the appendices. Several results (Prop A.2, Prop B.4, Cor B.15) are of secondary importance and take up a lot of space.

  2. It is informative. Closed testing restricted to weak orders (Shaffer-style logical restriction) combined with e-process averaging gives rank sets, and the finite-benchmark offset gives a clean anytime-valid story for fixed benchmarks. The quantification of monitoring-induced error inflation is also informative.

Critical weaknesses

The readability of the paper is a challenge. Please see critical strength 1.

Questions for authors
  1. How often do exact ties in benchmark means occur in the real data (binary scores, N around 1000)? A tied pair is never certified and a tied model is never retired under the all-pairs rule. What does this imply for the stated compute savings in practice?
  2. The (F) guarantee requires a recorded random permutation and lockstep evaluation of all active models. How does this match how leaderboards are actually run (parallel batches, models arriving over time)? Is a batch-wise version covered by the theory?
1 comment
AuthorsOpenReview ↗

Presentation, exact ties, and batch-wise evaluation

Thank you for your constructive feedback.

Presentation. Based on your feedback, we will provide clearer explanations of the key concepts in Section 3. The finite-benchmark offset bounds the expected score difference on the next item under a null comparison, using observed scores (zero under i.i.d. sampling). The shortcut combines direct pairwise evidence with indirect evidence through each third model. A tier gives a certified lower bound on rank, without implying that models in the same tier are tied.

We will also shorten the presentation of secondary theoretical results and improve the readability of E1–E4 by breaking up the paragraphs.

Q1. Exact ties and compute savings. We counted exact-tie pairs among all 395 models. The cost and settlement results come from Table 3, with all ties present.

All 395 modelsMMLUHellaSwagGSM8KWinoGrandeARCTruthfulQA
Exact-tie pairs (new)13681714362904
Top-3 cost (% of full)6.215.216.747.336.927.3
Top-3 statuses settled (%)100.0100.099.299.599.597.7

On the correct-certification event, tied models cannot retire under the all-pairs rule (Cor. B.15). Our reported savings instead use top-3 retirement, which need not resolve ties away from the decision boundary. We also verified that no benchmark has an exact tie between ranks 3 and 4. Thus, despite the ties, the method settles 97.7–100% of top-3 statuses on average using only 6.2–47.3% of full evaluation.

Q2. Parallel batches and new models. Yes, batch-wise evaluation is covered when the precommitted random item order is respected. Models and items may be scored in parallel, but statistical updates must follow that order and use only the completed shared prefix. Checking results only at batch ends preserves anytime validity.

Our validity and retirement guarantees (Theorem 4.2 and Proposition 4.5) assume a fixed model set. New entrants would require online closed testing or an error budget allocated in advance (Appendix D).

Plan. We will incorporate your presentation suggestions, add the exact-tie counts beside Table 3, and clarify batch-wise evaluation and new model arrivals in Appendix D.

Reviewer_NAtGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the setting of testing a set of LLMs on a benchmark with the goal of forming a ranking across the models. It extends existing work on FWER control on e-values to rank objects. The authors develop anytime-valid confidence sequences on model rankings. This procedure certifies ranks, top-k sets, and tiers, while maintaining validity even when the benchmark is checked multiple times during testing. The authors provide efficient ways to compute this method and evaluate it on empirical data.

Critical strengths

The theory fills a gap to extend existing work to the setting with rank objects. The paper provides a satisfying characterization of how to construct e-processes per pair of models, then average over these to build a wealth process for weak orders. This ensures coverage of the true ranking within a confidence set. The authors also provide guarantees on other related rank objects such as top-k sets and tiers. A nice consequence of this procedure is that it allows isolation of uncertainty to parts of a weak ordering.

The paper also includes considerations that bridge the gap to practice. First, the paper considers two sampling models: finite benchmark and i.i.d. distribution. The finite sample analysis is helpful as it instantiates the motivating benchmark evaluation problem more directly. Second, the authors provide algorithms for efficiently computing the steps of the main procedure. Third, the authors extend the results to show that the method allows for early stopping of evaluation on a model by model basis, which is useful to save cost and compute.

The paper is clearly structured and written. Assumptions are stated clearly, main ideas easy to follow, and theoretical insights explicated.

Critical weaknesses

Experiment E1 examines rank errors on six tied models, which seems to be a least favorable setting. What is the error rate when evaluated on all 395 models on the OpenLeaderboard data? Without model ties, are repeated looks a real problem empirically? I'd like to have more convincing empirical support on realistic leaderboard datasets for the problem motivation.

E1 also shows that a single pre-planned look remains valid. Curious if there is a way to either empirically or theoretically investigate the relationship between repeated look frequency/spacing and error rate on this dataset. Similarly, experiment E2 seems to suggest that when model abilities are spread out, the method certifies rankings more quickly. It would be nice to have a more direct experimental analysis of this.

On evaluating power, it would be nice to see Table 1 and Figure 2 demonstrate the advantages of the proposed method compared to other anytime-valid constructions (e.g. e-Bonferroni). Also, for the reasons stated in Appendix E.2 (Why the end of a benchmark flatters our sets.), Figure 2 doesn't seem like a fair comparison of power between the proposed method and the fixed-sample baselines. What does the analogous figure look like in the i.i.d. case?

Questions for authors

The betting parameter lambda is set at a default grid in Section 3. Why is this chosen? What about other alternatives, such as a fixed parameter or continuous mixture?

Under Scope and cost, the paper states "Pairwise-battle data are out of scope." However, the setting of an online leaderboard, such as Arena, seems to directly motivate a similar problem: items arrive online, models are scored, and rankings are examined along the way. Just out of curiosity, does the theory easily extend to this setting? One could imagine collecting a set of items per model pair, constructing the same e-process, then averaging over these to build an analogous wealth process to certify weak orders.

3 comments
AuthorsOpenReview ↗

New real-data experiments: monitoring errors, ability spread, and i.i.d. power

Thank you for your constructive feedback. We ran new experiments.

W1. Repeated looks without ties. We ran Holm on all 395 Open LLM Leaderboard models (1,000 item orders per benchmark). Orders with a false dominance claim:

Holm, wrong orders of 1,000MMLUHellaSwagGSM8KWinoGrandeARCTruthfulQA
Every 1% (103–107 looks)40184557
One look at halfway000000

No falsely ordered pair was tied. Ours made no errors in the first 50 orders per benchmark (0/300). Of Holm's 88 wrong orders, 67 failed at the first look (8–140 items), where the zz approximation can be unreliable, and monitored leaderboards are read that early. On the paper's top-8 panels (1,000 orders each), looks every 1% from 10% versus one look at 10% gave 51 vs. 16 wrong orders for the zz-test (of 6,000) and 26 vs. 5 for exact McNemar (of 5,000). 50 of 51 zz errors involved non-tied pairs.

For close real pairs (200 orders, McNemar on binary data), false-winner rates under looks every 1% were:

Pair typezz-testMcNemarOurs
Non-identical tied23–26%11–21%1.3–2.1%
Non-tied, total-score gap ≤511–13%5–10%≤0.8%

One halfway look erred below 1%. Among tied pairs, no single look exceeded 5.4% (zz) or 4.0% (McNemar). Repeated looks matter even with exact tests and without ties.

W2a. Look frequency and spacing. For tied pairs under (F), our new analysis approximates the zz-statistic after a fraction uu of the items by B(u)/uB(u)/\sqrt{u}, with BB a Brownian bridge. Increasing equally spaced looks from 10 to 100 raises real-data errors from 7.3–8.1% to 23.0–25.8% (Brownian-bridge approximation: 7.7% to 25.3%). With 10 looks, early-heavy versus late-heavy spacing gives 15.7–18.9% vs. 2.3–5.4%.

W2b. Ability spread. We spaced 20 models evenly across increasingly broad rank ranges. True dominances certified at halfway (50 orders, %):

Panel spreadMMLUHellaSwagGSM8KWinoGrandeARCTruthfulQA
Top 2050.148.722.21.60.217.3
Top 4066.760.436.10.64.047.7
Top 8079.358.951.62.811.058.6
Top 16070.167.874.416.230.063.4
All 39591.077.085.348.360.264.5

The widest panel leads on all six benchmarks. Prop. B.4 under (S) suggests ρ/(NΔ2)\rho/(N\Delta^2), where Δ\Delta is the score gap, ρ\rho the mean squared difference, and NN the benchmark size. Empirically, every pair below 0.01 was certified by halfway, versus only 0.05% above 0.3.

W3a. e-Bonferroni comparison. Section 4.1 shows our shortcut certifies at least what e-Bonferroni does. Table 7 reports that the exact test certifies 1.4 more true dominances than the shortcut on average at halfway. New e-Bonferroni runs on the Figure 2 settings and all 395 models, with identical wealths and orders, stay within 0.1 points of our shortcut. The finite-benchmark offset raises top-20 halfway certification from 11.2% to 22.2% (GSM8K) and 4.3% to 17.3% (TruthfulQA) versus clipping it at zero (also valid under (F)).

W3b. Power under i.i.d. sampling. We reran Figure 2 under (S), sampling with replacement while preserving model dependence. Top-20 true dominances certified at halfway (50 replications, %):

Setting / methodMMLUHellaSwagGSM8KWinoGrandeARCTruthfulQA
Finite (F), ours50.148.722.21.60.217.3
Finite (F), Romano–Wolf49.448.522.01.30.121.1
i.i.d. (S), ours44.945.112.50.80.06.0
i.i.d. (S), Romano–Wolf49.550.324.33.31.222.6

Figure 2 combines the finite-benchmark information gain with the cost of anytime validity. At halfway under (F), ours matches or beats Romano–Wolf on five of six. Under (S), Romano–Wolf has greater single-look power, while ours remains anytime-valid. We will clarify the scope and add full i.i.d. curves.

Q1. Why the λ\lambda grid? The geometric grid adapts through wealth weighting (Remark B.3). Under (S), Prop. B.4(d) gives a grid bet with expected log-growth of at least 0.18Δ2/ρ0.18\Delta^2/\rho per item for 0.06≤Δ/ρ≤10.06\le\Delta/\rho\le1, at a mixing cost of at most log⁡5\log 5 in log-wealth. On real data, each fixed value trails the best by 11–68 points somewhere, while ours stays within 4.8 points of the hindsight-best fixed value. A 50-point mixture changes halfway certification by −0.6 to +2.6 points. Any predictable bet in [0,1][0,1] preserves validity (Lemma B.1).

Q2. Arena extension. Yes, conditionally, with a fixed model set. Preferences must induce a transitive scalar ranking (e.g., Bradley–Terry). Battles need fresh i.i.d. prompts independent of adaptively chosen pairs, with outcomes depending only on pair and prompt. Scoring unshown pairs as zero gives nonpositive conditional means under each pairwise null. Lemma B.1 then gives valid pairwise e-processes for the weak-order test.

Plan. We will add the new results and clarify Figure 2 and its comparison with other methods.

Thank you for the detailed response! I appreciate the additional experiments, which largely address my concerns regarding practical relevance of the problem and the effects of look frequency/spacing. While the method does not seem to provide substantial empirical advantage over e-Bonferroni on the experiments, I find the theoretical contributions interesting and well-formulated. I maintain my positive assessment.

Thank you for your follow-up and positive assessment. We appreciate your time and feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript Uploaded: Summary of Changes Addressing Reviewer Comments

Dear Area Chair and Reviewers,

We have uploaded the revised manuscript. The tables below summarize the changes made in response to each reviewer.

Reviewer NAtG

CommentTypeChangeWhere
W1New experimentRepeated looks on real leaderboard dataSec. 5 (E1), Table 2. App. E.1, Table 3
W2aNew experimentError by number and timing of looksApp. E.1, Table 4
W2bNew experimentAbility spread experimentSec. 5 (E2). App. E.2, Table 6
W3aNew experimente-Bonferroni added on real dataFigure 2. App. E.6
W3bClarification, new experimentScope clarified, i.i.d. curves, offset ablationSec. 5 (E4). App. E.4, Figure 3, Table 10
Q1New experimentFixed, mixture and adaptive bets comparedApp. E.6, Table 16
Q2DiscussionAnswered in our replyReply only

Reviewer 2zLD

CommentTypeChangeWhere
PresentationClarificationPlain explanations of offset, shortcut and tiersSec. 3
PresentationPresentationParagraphs split, close-pair numbers in a tableSec. 5, Table 2
PresentationPresentationProp. A.2, Prop. B.4 and Cor. B.15 shortenedApp. A and B. Supporting discussion moved to App. C
PresentationPresentationAdded pointers to proofs and experimentsAppendix contents (p. 14, before App. A)
Q1New analysisExact-tie counts addedApp. E.3, Table 7
Q2ClarificationBatch-wise evaluation and fixed model setApp. D

Reviewer ccdQ

CommentTypeChangeWhere
W1New experimentRank-interval widths versus Romano–WolfApp. E.4, Table 11
W1New experimentRun time and the shortcut's extra widthApp. E.6, Table 15
Q1New experimentAdaptive betting alternatives and online Newton mixture extensionApp. E.6, Table 16
Q2New experimentRetirement cost against e-BonferroniApp. E.3, Table 8

The core method and theoretical guarantees are unchanged. All previously reported results are preserved. The revision adds experiments and clarifies the presentation.

Our earlier replies to the reviewers refer to the table numbering in the original submission. Some table numbers have changed in the revised manuscript. In particular, the original Table 3 is now Table 7, and the original Table 7 is now Table 14. Figures 1 and 2 retain their original numbers, and Figure 3 is new.

Thank you for your time and feedback,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.