- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper considers models and constructs anytime-valid confidence intervals for each model’s rank. Strictly speaking, it first constructs confidence sets and then converts them into intervals. The approach is to construct an e-process for each possible ordering. An interesting aspect of the construction is that, for a candidate ordering , the authors identify the pairwise order constraints implied by , construct an e-process for each constraint, and then average these pairwise e-processes to obtain an e-process for .
Critical strengths
- The paper studies an interesting problem and connects statistical tools (i.e. e-process) with LLM evaluation. This connection is valuable because it demonstrates how statistical methods can make model comparisons more reliable.
- The method is intuitive: it progressively eliminates candidate orderings using pairwise evidence and constructs rank confidence sets from the remaining orderings.
Critical weaknesses
The rank confidence intervals appear relatively wide, and exact computation can be expensive. Although the paper provides computational shortcuts that preserve anytime validity, these shortcuts may introduce additional conservatism and further widen the intervals.
Questions for authors
- Could be chosen adaptively from the observed data? Although the appendix discusses adaptive strategies, the reported results seem to favor averaging over a prespecified grid. I am not fully convinced that reliance on a fixed grid is the most suitable approach. I would expect a data-driven choice of to adapt better to the underlying model differences.
- With the same pairwise e-processes and model-specific retirement rule, how much evaluation cost does the proposed method save compared with e-Bonferroni?
3 comments
Thank you for the additional experiments and detailed responses. My questions have been largely addressed, and I maintain my positive assessment of the paper.
Thank you for your follow-up and positive assessment. We appreciate your time and feedback.
New experiments: interval width, computation, adaptive betting, and evaluation savings
Thank you for your constructive feedback. We ran additional experiments to address your questions and concerns.
W1. Width and computation. We compared our mean rank-interval widths with Romano–Wolf at halfway in the finite-benchmark setting (F) across twelve 20-model settings. Our widths differed by at most 0.71 ranks and were narrower in nine settings. Our method also provides anytime-valid guarantees. Table 7 shows that exact certification adds only 1.4 true dominances on average over the shortcut at halfway, out of 179–190. Across all checkpoints in the 20-model experiments, the shortcut's mean interval width was at most about 0.33 ranks larger than the exact test's.
Algorithm 3 solved all tested integer programs. We also measured runtime for different numbers of models. Mean seconds per run (checks approximately every 1% of the benchmark, 10 item orders per benchmark):
The last column reports the largest benchmark-mean width difference. The shortcut greatly reduces computation with limited additional conservatism.
Q1. Why not a data-driven λt? Data-driven bets are already permitted. Any predictable bet in [0,1] preserves validity (Lemma B.1). Our default grid is itself a wealth-weighted adaptive mixture, equivalent to a predictable effective bet (Remark B.3).
We compared alternatives on six real benchmarks and two 20-model panels, with 50 item orders each. Changes in certified true dominances at halfway, in percentage points relative to ours:
The default five-point mixture remains competitive with data-driven alternatives without benchmark-specific tuning. Adding online Newton as a sixth mixture component also preserves validity (Lemma B.1). This extension improves final certification in all 12 settings by 0.04–0.98 percentage points. It also reduces top-3 evaluation cost in all 12 settings by 0.02–0.62 percentage points of full evaluation.
Q2. Savings versus e-Bonferroni. We ran new experiments comparing evaluation costs with e-Bonferroni. All methods used the same pairwise betting construction and identical item orders. Each method applied the top-3 retirement rule based on its own certification decisions.
On all 395 models (50 orders per benchmark), the shortcut's evaluation cost is within about 0.01 percentage points of e-Bonferroni's in every order, and e-Bonferroni is never cheaper.
On the 8- and 20-model panels (six benchmarks, 50 orders each), exact weak-order certification reduces average evaluation cost in all 18 settings. Mean savings relative to e-Bonferroni, in percentage points of full evaluation is shown below:
The exact test never requires more evaluations in any of the 900 runs. The shortcut remains within 0.03 percentage points of e-Bonferroni on average. These experiments demonstrate the benefit of weak-order certification. The exact test consistently saves more evaluations than e-Bonferroni across all 18 settings. The shortcut provides a faster alternative and certifies at least what e-Bonferroni does (Section 4.1).
Plan. We will add the width and runtime results to Appendix E, the betting comparisons and online Newton mixture extension to E.6, and the evaluation-cost comparison to E.3.