acceptodds
Under review as a conference paper at ICLR 2027

Anytime-Valid False Discovery Rate Control for Adaptive Foundation Model Evaluation

Abstract

Foundation model leaderboards are adaptive experiments: evaluators repeatedly choose model comparisons, allocate samples to promising claims, and stop when results look publishable, so standard fixed-time p-value procedures do not generally retain valid false-discovery guarantees after data-dependent stopping. We formulate leaderboard evaluation as sequential FDR control over directed pairwise model claims, and show that fixed-time BH fails under adaptive stopping while pairwise e-processes combined with e-BH control FDR at arbitrary stopping times. The main practical obstacle is power: greedy allocation concentrates nearly all budget on one comparison, leaving other true alternatives undiscovered. We make this quantitative: greedy allocation exhibits a 1/K power ceiling up to an explicit finite-sample correction (K true alternatives), allocation-aware sampling recovers power, and we prove a Theta(m/s) separation in discovery budget between top-k and uniform sampling that holds for any FDR-controlling rule (not only e-BH). We propose allocation-aware anytime-valid protocols (top-k round-robin, threshold reallocation) and validate them on synthetic adaptive leaderboards and adaptive replays of Chatbot Arena and MT-Bench, showing inflated fixed-time error rates, stable e-BH control, and substantial power recovery.

Primary area:
datasets and benchmarks
Keywords:
False discovery rate, anytime-valid inference, e-processes, e-BH, foundation model evaluation, leaderboards, adaptive sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ed5VOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies FDR control and power in adaptive pairwise evaluation of foundation models. It uses e-processes and e-BH to control FDR. Its main focus is how sample allocation affects power: greedy sampling can concentrate on one comparison and leave other true alternatives undiscovered. The paper studies top-k round-robin sampling and threshold reallocation, and compares their discovery budgets with uniform sampling.

Critical strengths

The paper studies an interesting problem and applies statistical multiple testing tools to LLM evaluation I think this connection is important because it shows how statistics can help make model comparisons more reliable and what statistics can contribute in the LLM era.

Critical weaknesses
  1. I think the writing in this paper is a bit unclear. Sections 2, 3, and 4 are short and cover fairly basic material. It would be more appropriate to combine them into one section. In addition, Proposition 3 uses “drift γ\gamma”, but this quantity is not introduced or explained in the main text. There are other examples: Kmax⁡K_{\max} first appears in Proposition 6 without a definition, and Lemma 1 uses Nh(τ)N_h(\tau), which has not been defined in the main text.

  2. The main theoretical contributions are in Propositions 3–8. However, there are problems with some of the statements and proofs, particularly in Propositions 3, 5, and 7. My specific questions are listed below.

Questions for authors
  1. Proposition 3. Could Equation (6) be simplified to 1/K1/K under the stated burn-in condition? The last two terms seem unnecessary. In addition, the following inequality is a key step in the proof in Appendix D is correct:

    \exists t\le T:D_h(t)\le0 \mid D_h(0)=D_0>0 \right) \le T\exp\left[ -\frac{2(D_0+\gamma_{\log}T)^2}{Tw^2} \right].$$

As T→∞T\to\infty, RHS tends to zero, while LHS is nondecreasing in TT. This would imply Pr⁡(∃t≤T:Dh(t)≤0∣Dh(0)=D0>0)=0for every T.\Pr\left(\exists t\le T:D_h(t)\le 0\mid D_h(0)=D_0>0\right)=0\quad\text{for every }T. Could you explain how this inequality is obtained?

Besides, The last two terms in Equation (6) seem unnecessary. Greedy only updates the largest e-value, while all other e-values remain unchanged, so the second-largest e-value cannot increase. Under the stated condition, it stays below 1/α1/\alpha, which seems to limit e-BH to at most one rejection. Could you check whether this observation gives a bound of 1/K1/K and allows a simpler proof?

  1. Proposition 5, Appendix F.1. Let L(n)L(n) denote the lower bound on the evidence you proved, so that log⁡Eh(n)≥L(n).\log E_h(n)\ge L(n).

    Could you clarify why L(n)≥log⁡mαrL(n)\ge\log\frac{m}{\alpha r} is treated as a necessary condition for discovery and used to derive a lower bound on the minimum budget? Would this only be a sufficient condition? Even if the lower bound is below the threshold, the actual evidence may still exceed it.

  2. Proposition 5, Appendix F.2. Does the proof assume that the active set contains all ss true alternatives? The proof only establishes evidence growth for true alternatives within the active set, but concludes that all true alternatives can be discovered. Could you clarify this step, since k≥sk\ge s alone does not guarantee that the active set contains all true alternatives?

  3. Proposition 7. Could you state the assumptions required for this result? They are not stated in the proposition in the main text. Without these assumptions, the proposition as stated in the main text admits counterexamples.

Reviewer_EHc7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This article studies the adaptive evaluation of foundation models. The pairwise comparisons are sampled sequentially and evaluation budgets are allocated adaptively. The e-processes and e-BH are combined to achieve anytime-validity for FDR control. Moreover, the authors investigate how different allocation strategies would affect statistical power and sampling efficiency. Sample-complexity results are developed. The proposed approaches are evaluated through simulations and real-data replays.

The problem itself is interesting and relevant. However, the paper is not well organized and the presentation is unclear. The central procedure has not been described as a complete end-to-end algorithm; standard validity ingredients, which are well known, occupy substantial space; and the potentially novel allocation results are mixed with various technical discussions. It is difficult to determine which theoretical guarantees apply to which implementations and what the main independent contribution of the paper is.

Critical strengths

-The paper considers an interesting and practically relevant problem, especially when evaluation budgets are limited. The separation between statistical validity and allocation-dependent power is a reasonable way to formulate the problem.

-The power limitations of greedy allocation and various sampling strategies are investigated; the top-kk round-robin and threshold reallocation look promising.

Critical weaknesses
  1. Yang et al. (2017) [1] already developed a framework combining adaptive sampling, always-valid sequential inference, and online FDR control, with theoretical guarantees on statistical power and sample complexity. The underlying motivations and methodological ideas appear closely related. This connection has not been carefully explained in the present work.

[1] Yang, F., Ramdas, A., Jamieson, K. G., and Wainwright, M. J. (2017). A Framework for Multi-A(rmed)/B(andit) Testing with Online FDR Control. Advances in Neural Information Processing Systems, 30, 5957–5966.

  1. The theory and practical algorithms in Section 5 do not appear to match. The authors should have clearly presented the main method as an end-to-end algorithm. It makes one feel that the manuscript presents several partially analyzed allocation rules without specifying which one is the principal proposal. The presentation should be revised to make clear which theoretical guarantee applies to which implementation.

  2. Section 3 is trivial, and Section 4 largely restates well-known results. Section 5 contains a large amount of material, but it is poorly organized and lacks a clear overview. It should be reorganized around a coherent theme of efficient statistical decision-making. More fundamentally, even the allocation objective is not clearly specified, making it difficult to understand what the proposed procedures are intended to optimize or what readers should take away from the analysis.

  3. There are some other relatively small issues that are worthy of discussion. The FDR guarantee relies on the validity of each e-process with respect to the global filtration. This assumption can be restrictive when different data streams are dependent. Moreover, the sample-complexity analysis for top-kk allocation appears to require that the active set contain the true alternatives. This can be problematic in practice where the active set is selected adaptively?

Questions for authors

Q1. Equation (2) is conceptually misstated. FDR is a deterministic risk functional of a multiple-testing procedure under a data-generating distribution. Writing FDR(τ)\mathrm{FDR}(\tau) makes one feel that the FDR is a functional of a random stopping time. A deterministic quantity cannot be equal to a random quantity. This notation should be revised to avoid confusion. Moreover, I do not think that this is not an “adaptation” of classical FDR to pairwise claims. Pairwise claims merely index the hypothesis family; the definition of FDR itself is unchanged.

Q2. Under what practical conditions does the conditional null assumption in Equation (1) hold with respect to the global filtration, particularly when observations across model pairs are dependent? Does the data-generating mechanism used in Section 6 satisfy this assumption, and, if so, how is this ensured?

Q3. Does Proposition 5 account for the cost of identifying an active set that contains the true alternatives? If this condition is not guaranteed by the proposed adaptive top-kk procedure, to what extent does the stated sample-complexity separation apply to the actual algorithm?

Reviewer_wpdROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies anytime-valid false discovery rate (FDR) control for adaptive foundation model evaluation. The authors formulate model evaluation as a sequential multiple hypothesis testing problem over directed pairwise model comparisons, where the evaluator can adaptively allocate samples and stop based on accumulated evidence.

The proposed framework combines pairwise e-processes with e-BH to guarantee FDR control under adaptive sampling and arbitrary stopping times. To improve statistical power, the paper investigates several sampling strategies, including greedy allocation, top-k round-robin, and threshold-based reallocation. The authors analyze the power limitations of greedy sampling and derive sample complexity bounds comparing top-k and uniform allocation. Experiments on synthetic data and adaptive replays of Chatbot Arena and MT-Bench demonstrate improved power from allocation-aware sampling while maintaining FDR control.

Critical strengths
  1. The paper addresses a relevant problem in adaptive evaluation and clearly distinguishes statistical validity from discovery power.
  2. The theoretical analysis provides insights into greedy allocation's power limitations and the sample complexity benefits of allocation-aware sampling.
  3. The experiments cover different signal regimes and demonstrate substantial power improvements over greedy and uniform sampling.
Critical weaknesses
  1. The core framework substantially overlaps with Xu et al. (https://proceedings.neurips.cc/paper_files/paper/2021/file/8c460674cd61bf189e62b4da4bd9d7c1-Paper.pdf) and Wang et al, (https://arxiv.org/pdf/2502.08539), which already establish anytime-valid FDR control using e-processes under arbitrary adaptive sampling and stopping, while explicitly separating exploration from statistical inference. The paper does not establish a sufficiently clear methodological or theoretical advance over these closely related works. The claimed allocation-dependent sample complexity separation appears intuitive, and its incremental novelty remains unclear.

  2. The greedy lock-in probability bound in Appendix D appears problematic: the proposed upper bound on the probability of crossing within the first (T) steps converges to zero as (T) increases, whereas the probability of this event must be nondecreasing. This raises concerns about the proof of the (1/K) power ceiling.

Additionally, the top-k sample complexity guarantee assumes sufficient coverage of the true alternatives, without establishing that the proposed adaptive selection strategy achieves this coverage. The lower bound also requires restrictive sparsity assumptions and applies only under specified sampling constraints. These limitations weaken the generality of the claimed theoretical separation.

3 The framework is a generic sequential multiple testing formulation and does not exploit LLM-specific properties. Chatbot Arena and MT-Bench primarily provide empirical sampling distributions. The authors should clarify the application-specific contribution and discuss which challenges of real-world LLM evaluation are addressed.

  1. The experiments primarily compare against greedy, uniform, and oracle sampling. Comparisons with existing adaptive bandit multiple testing algorithms are necessary to establish the practical benefits of the proposed strategies.
Questions for authors

see weaknesses, particularly 1 & 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.