acceptodds
Under review as a conference paper at ICLR 2027

Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized -Processes

35 views

Abstract

Large language model (LLM) leaderboards compare model capabilities by ranking models according to their mean performance on fixed benchmarks. However, variability in evaluation outcomes across runs may produce unsupported claims of model superiority on the benchmark, a risk compounded by leaderboard updates. In this paper, we propose BB-EDGE (Benchmark-Weighted and Block-Factorized -processes for Directed Graph Evaluation), a principled framework that represents an LLM leaderboard as a directed graph whose edges certify pairwise mean-performance advantages, with anytime-valid family-wise error rate (FWER) control. Concretely, for each direction, BB-EDGE constructs an empirical-Bernstein -process by factorizing evidence over protocol-defined blocks and assigning stakes proportional to the corresponding block weights, then applies direct -Holm across these -processes to certify directional advantages as edges. Theoretically, we characterize weight-proportional linear stakes under heterogeneous benchmark-average nulls and prove anytime FWER control under arbitrary within-block and cross-pair dependence. BB-EDGE further supports anytime-valid Top- certification and simultaneous rank intervals. Extensive experiments on synthetic data and four real-world benchmarks demonstrate that BB-EDGE maintains anytime FWER control while achieving high efficiency.

Primary area:
generative models
Keywords:
Large Language Models, Anytime-Valid Inference, Leaderboards

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JTypOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes BB-EDGE for certifying pairwise mean-performance advantages on fixed LLM benchmarks. It combines block-factorized empirical-Bernstein e-processes, stakes proportional to benchmark weights, and direct e-Holm to provide anytime-valid family-wise error control under explicit assumptions. Experiments on synthetic data and four real-world benchmarks evaluate error control and certification efficiency.

Critical strengths
  1. The paper provides a rigorous mathematical formulation and false-positive control guarantees under explicit assumptions, supported by a fairly complete theoretical analysis.
  2. The ablation studies are reasonably thorough and demonstrate improved evaluation efficiency while controlling false positives in the tested settings.
Critical weaknesses
  1. The practical scope of the method needs further clarification. Real-data validation of its treatment of within-block dependence focuses mainly on multi-turn dialogue and tool use. It would be helpful to demonstrate benefits in other evaluation settings with dependent scores and explain how the partitioning assumptions can be justified in those settings.
  2. The incremental contribution over existing methods needs further clarification. SERPANT-R already substantially mitigates false positives in the heterogeneity-only experiments. The paper should more clearly explain BB-EDGE's efficiency advantage over this baseline and the conditions under which that advantage holds.
Questions for authors
  1. Theoretical efficiency. Please clarify when the positive-growth condition in Theorem 4 holds for the chosen stake grid, and provide a more explicit theoretical characterization of the evaluation budget needed for certification.
  2. Practical applicability. Please add experimental evidence from some other practical settings involving dependent scores beyond multi-turn dialogue and tool use, and explain how the blocks are defined and their independence justified.
  3. Incremental benefit. Building on Figure 11, please quantify BB-EDGE's evaluation-cost savings over SERPANT-R at matched certification targets, especially in settings where both methods keep empirical false-positive rates within the target level.
1 comment
AuthorsOpenReview ↗

Response to Reviewer JTyp

We appreciate your insightful comments on our work. Please find our response below:

1. Evaluation on more benchmarks [W1, Q2]

Thank you for highlighting the concern. We include two additional real-world benchmarks with dependent scores: IFEval [1] for instruction following and ClassEval [2] for class-level code generation.For IFEval, the instructions of each prompt are evaluated on the same response, so we group them into one block, i.e., ∣Cm∣∈1,2,3|C_m| \in \\{1,2,3\\}. In ClassEval, we group the methods of each class into a block, with ∣Cm∣∈2,…,10|C_m| \in \\{2,\ldots,10\\}, since they share class attributes and are generated together. Each prompt or class is generated in a separate stateless execution with independent randomization and no shared context across blocks. Since independence cannot be tested directly, we assess correlation as a diagnostic for dependence. As shown in the table below, all 95% confidence intervals for the cross-replicate and cross-block correlations contain zero, indicating no detectable correlation across replicates or blocks. We have added these tests across replicates and across blocks in Lines 474–480 and Table 2.

IFEvalClassEval
Cross-replicate correlation[-0.09, 0.10][-0.21, 0.19]
Cross-block correlation[0.00, 0.00][0.00, 0.01]

Additionally, we also add the certified graphs, Top-kk certification, and rank intervals for IFEval and ClassEval in Appendix F.

2. Efficiency advantage over SERPANT-R [W2, Q3]

Thank you for the great suggestion. Former Figure 11 now appears as Figure 7 in the revised manuscript. The table below summarizes the results shown in Figure 7(a) for a synthetic i.i.d. item stream (N=100N=100, ∣Cm∣=1|C_m|=1, hi,ℓ=0h_{i,\ell}=0) that satisfies SERPANT-R’s assumptions. The results show that both BB-EDGE and SERPANT-R control the anytime FWER, while BB-EDGE achieves higher power. For instance, BB-EDGE reaches 80% power at 50 replicates, whereas SERPANT-R requires 100 replicates to reach 81%, thereby halving the evaluation cost.

Replicates1050100
FWER/Power
SERPANT-R0.00/0.210.00/0.640.00/0.81
BB-EDGE0.00/0.340.00/0.800.00/0.99

3. Clarification of theoretical efficiency [Q1]

Thank you for this insightful comment. We have moved the former Theorem 4 (power consistency) to the main text as Theorem 2, and added Proposition 2 to make the positive-growth condition and the certification budget explicit. These results characterize BB-EDGE’s efficiency in two respects:

  • Positive growth. With our running-mean predictions, the average log growth converges to the deterministic rate ae,g∞=λgδe/w⋆−∑mψE(λg,m)σe,m2,a^\infty_{e,g}=\lambda_g\delta_e/w_\star-\sum_m\psi_E(\lambda_{g,m})\sigma^2_{e,m}, where the first term is the evidence gained from the true gap δe\delta_e and the second is a variance penalty from block-level noise. Hence, the condition holds if and only if ae⋆=max⁡g:ρg>0ae,g∞>0a_e^\star=\max_{g:\rho_g>0}a^\infty_{e,g}>0, i.e., at least one stake on the grid gains more from the signal than it loses to noise. A simple sufficient condition is Δa,bB−τ>λmin⁡/[4(1−λmin⁡)]\Delta_{a,b}^{\mathcal B}-\tau>\lambda_{\min}/[4(1-\lambda_{\min})], where λmin⁡\lambda_{\min} is the smallest positive stake with positive mixture weight. Since this threshold vanishes as λmin⁡→0\lambda_{\min}\to0, including small stakes in the grid allows even small gaps above the margin to be eventually certified, although smaller stakes yield slower evidence growth.

  • Evaluation budget. With the design fixed, the first certification time satisfies lim sup⁡α↓0Te(α)log⁡(1/α)≤1ae⋆\limsup_{\alpha\downarrow0}\frac{T_e(\alpha)}{\log(1/\alpha)}\le\frac{1}{a_e^\star} almost surely. That is, for small α\alpha, certifying an edge requires at most about log⁡(1/α)/ae⋆\log(1/\alpha)/a_e^\star replicates. The budget therefore grows only logarithmically in 1/α1/\alpha: asymptotically, halving α\alpha adds at most about log⁡2/ae⋆\log 2/a_e^\star replicates. Moreover, fewer replicates are needed as ae⋆a_e^\star increases, which occurs when the gap is wider or the block variances are smaller.

[1] Instruction-following evaluation for large language models, arXiv 2023.

[2] Evaluating Large Language Models in Class-Level Code Generation, ICSE 2024.

Reviewer_Q6YwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BB-EDGE, a method for building reliable LLM leaderboards. It uses block-factorized e-processes and direct e-Holm to compare models while controlling the family-wise error rate (FWER). The authors provide theoretical proofs and experiments on synthetic and real-world benchmarks to show the effectiveness of the method.

Critical strengths
  1. The paper provides a clear theoretical justification for the method. Proposition 1 explains why the stakes should be proportional to the benchmark weights, and Theorem 1 proves the anytime FWER guarantee under the stated assumptions.

  2. The experiments are well designed. The authors compare BB-EDGE with both existing methods and internal baselines. In particular, BFH-e-Holm and CR-EDGE help show the benefits of variance adaptation and block factorization, respectively.

Critical weaknesses

The novelty of the method is somewhat limited. It mainly combines existing statistical techniques, and the authors could better explain what is new compared with previous methods.

Questions for authors

The theoretical guarantees rely on independence assumptions across evaluation runs and blocks. How can these assumptions be ensured in real-world LLM evaluations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Q6Yw

Thank you for the valuable comments. Please find our response below.

1. Clarification of our contributions [W1]

Thank you for raising this concern. In this work, we formulate LLM leaderboard certification as anytime-valid multiple testing of pairwise benchmark-average differences. This differs from fixed-budget inference [1–2], whose guarantees do not generally extend to repeated leaderboard updates, and from online pairwise preference ranking [3], which targets a different estimand. Moreover, existing online methods [3–5] assume i.i.d. items, ignoring the heterogeneity and dependence of structured benchmarks. The key difficulty is heterogeneity: since the null constrains only the weighted average, some blocks may favor one model even when it holds, so upweighting them creates spurious evidence. We prove that, among nonnegative linear stake allocations, weight-proportional stakes are necessary and sufficient for nonpositive linear drift over the full benchmark-average null. Building on this, we derive BB-EDGE’s block-factorized ee-processes, which allow arbitrary within-block dependence under cross-block independence, with weight-proportional stakes ensuring nonpositive null drift despite heterogeneous block means. We also establish BB-EDGE's power consistency and an asymptotic bound on the certification budget. We have highlighted this distinction in the Section 2.

2. Concerns of Independence Assumption [Q1]

Thank you for the question. To support independence across replicates, we run each with its own random seed and stochastic decoding, with no conversational state or feedback shared across replicates. For the condition of independence across blocks, we group all items from the same dialogue or task trajectory into a single block according to the predefined evaluation protocol. If such a grouping cannot be justified, we use CR-EDGE, which treats the entire benchmark within each evaluation run as a single block and thus requires only independence across replicates. Since independence cannot be tested directly, we assess correlation as a diagnostic for dependence. As shown in the table below, all 95% confidence intervals for the cross-replicate and cross-block correlations of score differences across all 45 model pairs contain zero, indicating no detectable correlation across replicates or blocks. Additionally, Table 1 shows that BB-EDGE maintains empirical anytime FWER control under low to moderate levels of cross-run and cross-block correlations. We have added these tests across replicates and across blocks in Lines 474–480 and Table 2.

GSM8KMMLU-ProMT-BenchACEBench
Cross-replicate correlation[-0.06, 0.05][-0.01, 0.02][-0.20, 0.22][-0.22, 0.29]
Cross-block correlation[0.00, 0.00][0.00, 0.00][0.00, 0.01][0.00, 0.00]

[1] Confidence Diagram of Nonparametric Ranking for Uncertainty Assessment in Large Language Models Evaluation, arXiv 2024.

[2] Pluralistic leaderboards, ICML 2026.

[3] Anytime-Valid Inference for Online Ranking of Large Language Models, ICML 2026.

[4] CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes, ICML 2026.

[5] Efficient Sequential Evaluation of Large Language Models, NeurIPS 2026.

Reviewer_41EEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes BB-EDGE, a method for constructing statistically certified LLM leaderboards on a fixed benchmark. Rather than reporting a total ranking, it constructs a directed graph in which each edge certifies that one model’s expected benchmark score exceeds another's by a prespecified margin. The guarantee controls the probability of ever reporting any false edge across model pairs and repeated evaluation runs.

The method groups dependent observations into protocol-defined blocks, constructs empirical-Bernstein e-processes using stakes proportional to benchmark block weights, and applies direct e-Holm testing. The authors characterize the proportional-stake requirement for preserving the benchmark-average null and prove anytime family-wise error control under their assumptions. Experiments include synthetic settings with heterogeneous item effects and within-block dependence, along with four real benchmarks. Extensions cover top-k certification, rank intervals, benchmark updates, and model additions.

Critical strengths
  • The main proof appears correct. Proportional stakes preserve the benchmark-average null, conditional block independence permits factorization and the resulting directional supermartingales support anytime FWER control through the direct e-Holm argument. The proof correctly permits dependence across model-pair tests.

  • The paper targets expected mean performance on a fixed benchmark, rather than silently replacing it with average pairwise preference or an assumption that every item favors the same model. Proposition 1 explains why stakes proportional to block weights are necessary and sufficient for the stated nonpositive linear-drift condition. This gives a principled basis for handling heterogeneous item effects while retaining the original benchmark weighting. The resulting guarantee simultaneously covers both comparison directions, all model pairs, and repeated monitoring.

  • The experiments include useful controlled comparisons of the method's main components. BFH-e-Holm retains the partition and testing procedure while replacing the empirical-Bernstein penalty with a Hoeffding penalty; CR-EDGE instead aggregates each complete replicate into one block. These comparisons help separate the benefits of variance adaptation and finer evidence aggregation. The reported GSM8K result that resolvs 41 of 45 model pairs after two replicates rather than six for BFH-e-Holmdemonstrates a meaningful gain against a baseline with the same target and an anytime guarantee. The top-k and rank-interval extensions also turn the graph into useful decision outputs.

  • The paper clearly defines the target quantity, the testing procedure and the distinction between within-block dependence and independence across blocks. It also explicitly limits its claims to performance on the fixed benchmark under the specified evaluation protocol.

Critical weaknesses
  • The external comparisons mainly demonstrate failures under mismatched assumptions, leaving the efficiency comparison relatively narrow. EMR does not control statistical error, several baselines are designed for fixed-time inference, and the SERPANT adaptation is deliberately applied outside its item-stream assumptions in the heterogeneous and dependent settings. Appendix G.2 appropriately explains these mismatches. Their failures are useful diagnostics, but provide limited evidence that BB-EDGE is more efficient than other methods designed for the same target and sampling protocol.

  • The synthetic heterogeneity construction also places opposite effects in the first and second halves of the benchmark, while the item-stream baseline monitors intermediate prefixes. This is a valid counterexample, but makes its failure especially sensitive to item order. The practically relevant efficiency claim therefore rests mainly on the internal baselines. A stronger comparison would include a competitive finite-sample sequential method adapted to the same benchmark-average target, or a valid time-adjusted fixed-time baseline, with comparable tuning and query accounting.

  • The practical reliability of the efficient block factorization is not sufficiently stress-tested. The largest efficiency gains rely on treating items or dialogues as independent blocks. The execution protocol described in Appendix G.4 provides a reasonable justification for the locally hosted experiments. However, the permutation diagnostic in Appendix G.1 compares aggregate variance and uses only 20 replicates, failure to reject does not establish mutual independence. It also tests each model separately, whereas the theorem assumes independence of the joint model-score vectors across blocks.

  • The synthetic experiments vary dependence within correctly specified blocks, but do not show what happens when weak dependence remains across the chosen blocks. This matters particularly for shared judges or API execution. The one-block fallback protects validity under arbitrary within-run dependence, but can substantially reduce efficiency. A sensitivity analysis with misspecified partitions would help practitioners understand when the reported savings are reliable and when coarser blocks are necessary.

Questions for authors
  • How quickly does FWER deteriorate under a small shared run-level or judge-level effect spanning nominally independent blocks? Can a prespecified coarser partition retain useful power in these settings?

  • How much of SERPANT's failure under heterogeneity depends on the ordered positive/negative item effects? Please distinguish the existing completed-replicate SERPANT-R comparison from a comparison using randomized item order or a method explicitly adapted to the same fixed-benchmark target.

  • The implementation uses 41 equally spaced stakes on [0,0.95], so the smallest positive stake is 0.02375. Theorem 4 requires a grid component with positive long-run expected log growth, which need not exist for arbitrarily small positive gaps with nonzero variance. How sensitive are certification times and unresolved pairs to the grid near zero? Does a finer or geometrically spaced grid materially change the results?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 41EE

5. Analysis of SERPANT [Q2]

Thank you for this insightful question. We consider two types of heterogeneity: (i) binary scores with opposing item-level effects, and (ii) graded scores where one model wins narrowly on most items but loses substantially on a minority. In case (i), the failure of SERPANT indeed stems mainly from the ordered item effects: SERPANT-S and SERPANT-R reduce the anytime FWER from 0.70 to at most 0.01, while BB-EDGE still attains higher power. For case (ii), SERPANT, SERPANT-S, and SERPANT-R all exceed the target level, indicating that neither completed-replicate monitoring nor randomized item order can mitigate this failure. In contrast, BB-EDGE maintains zero empirical anytime FWER. We have added this analysis at Line 412–416 of the revised manuscript.

Replicates1050100
FWERr/Powerr\mathrm{FWER}_r/\mathrm{Power}_r
BinarySERPANT0.70/0.300.70/0.670.70/0.83
SERPANT-S0.01/0.230.01/0.640.01/0.82
SERPANT-R0.00/0.210.00/0.640.00/0.81
BB-EDGE0.00/0.380.00/0.860.00/1.00
GradedSERPANT0.01/1.000.11/1.000.61/1.00
SERPANT-S0.01/1.000.12/1.000.63/1.00
SERPANT-R0.00/1.000.09/1.000.59/1.00
BB-EDGE0.00/1.000.00/1.000.00/1.00

6. Sensitivity to the stake grid Λ\Lambda [Q3]

Thank you for the helpful suggestion. We evaluate all four combinations of two grid sizes (41 and 401 points) and two spacing schemes (uniform on [0,0.95][0, 0.95] and geometric on [10−4,0.95][10^{-4}, 0.95]) to assess sensitivity to the stake grid. The table below summarizes the results of this analysis on synthetic data with 100 items, ∣Cm∣=1|C_m|=1, and hi,ℓ=0h_{i,\ell}=0. We find that our method’s empirical anytime FWER and power are largely insensitive to grid size and spacing scheme. We have added this sensitivity analysis at Line 420–422 of the revised manuscript.

Replicates1050100
FWERr/Powerr\mathrm{FWER}_r/\mathrm{Power}_r
Uniform, GG=410.00/0.340.00/0.800.00/0.99
Uniform, GG=4010.00/0.340.00/0.800.00/0.99
Geometric, GG=410.00/0.310.00/0.790.00/0.99
Geometric, GG=4010.00/0.320.00/0.790.00/0.99

The table below reports the percentage of the 45 model pairs resolved by BB-EDGE for GSM8K at τ=0\tau=0 under the four stake grids. The results show that our method is also insensitive to the grid size and spacing scheme on real data.

Replicates1251020
Uniform, GG=4180.0091.1193.3393.3395.56
Uniform, GG=40180.0091.1193.3393.3395.56
Geometric, GG=4175.5691.1193.3393.3395.56
Geometric, GG=40177.7891.1193.3393.3395.56

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the chairs and reviewers for their reviewing efforts and the constructive suggestions on our work. We are pleased that the reviewers recognize our proposed BB-EDGE and its extensions as reliable and useful [Q6Yw, 41EE], our theory as clear, rigorous and fairly complete [Q6Yw, JTyp], and our experiments as well designed and reasonably thorough [Q6Yw, JTyp], with meaningful gains [41EE].

In the following responses, we have addressed the reviewers' comments and concerns point by point. The reviews allow us to strengthen our manuscript, and the changes are summarized below:

  • Added the positive-growth condition and an asymptotic bound on the certification budget in Proposition 2. [JTyp]
  • Added analysis of SERPANT under two types of heterogeneity in Lines 412–416 and Figure 2. [41EE]
  • Added comparison of two valid time-adjusted fixed-time baselines in Lines 417-420 and Figure 7. [41EE]
  • Added sensitivity analysis of the stake grid in Lines 420–422 and Figure 8. [41EE]
  • Added discussion of potential failure cases in Lines 428–431, Lines 459–460 and Table 1. [41EE]
  • Added correlation diagnostics across replicates and blocks in Lines 474–480 and Table 2. [41EE, JTyp, Q6Yw]
  • Added two additional real-world datasets, IFEval and ClassEval, in Section 6.2 and Appendix F. [JTyp]
  • Highlighted the efficiency of BB-EDGE over SERPANT in the i.i.d. setting in Lines 401–403 and Lines 417–420. [41EE, JTyp]
  • Highlighted the distinction from existing methods in Lines 106-111. [Q6Yw]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.