- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies FDR control and power in adaptive pairwise evaluation of foundation models. It uses e-processes and e-BH to control FDR. Its main focus is how sample allocation affects power: greedy sampling can concentrate on one comparison and leave other true alternatives undiscovered. The paper studies top-k round-robin sampling and threshold reallocation, and compares their discovery budgets with uniform sampling.
Critical strengths
The paper studies an interesting problem and applies statistical multiple testing tools to LLM evaluation I think this connection is important because it shows how statistics can help make model comparisons more reliable and what statistics can contribute in the LLM era.
Critical weaknesses
-
I think the writing in this paper is a bit unclear. Sections 2, 3, and 4 are short and cover fairly basic material. It would be more appropriate to combine them into one section. In addition, Proposition 3 uses “drift ”, but this quantity is not introduced or explained in the main text. There are other examples: first appears in Proposition 6 without a definition, and Lemma 1 uses , which has not been defined in the main text.
-
The main theoretical contributions are in Propositions 3–8. However, there are problems with some of the statements and proofs, particularly in Propositions 3, 5, and 7. My specific questions are listed below.
Questions for authors
-
Proposition 3. Could Equation (6) be simplified to under the stated burn-in condition? The last two terms seem unnecessary. In addition, the following inequality is a key step in the proof in Appendix D is correct:
\exists t\le T:D_h(t)\le0 \mid D_h(0)=D_0>0 \right) \le T\exp\left[ -\frac{2(D_0+\gamma_{\log}T)^2}{Tw^2} \right].$$
As , RHS tends to zero, while LHS is nondecreasing in . This would imply Could you explain how this inequality is obtained?
Besides, The last two terms in Equation (6) seem unnecessary. Greedy only updates the largest e-value, while all other e-values remain unchanged, so the second-largest e-value cannot increase. Under the stated condition, it stays below , which seems to limit e-BH to at most one rejection. Could you check whether this observation gives a bound of and allows a simpler proof?
-
Proposition 5, Appendix F.1. Let denote the lower bound on the evidence you proved, so that
Could you clarify why is treated as a necessary condition for discovery and used to derive a lower bound on the minimum budget? Would this only be a sufficient condition? Even if the lower bound is below the threshold, the actual evidence may still exceed it.
-
Proposition 5, Appendix F.2. Does the proof assume that the active set contains all true alternatives? The proof only establishes evidence growth for true alternatives within the active set, but concludes that all true alternatives can be discovered. Could you clarify this step, since alone does not guarantee that the active set contains all true alternatives?
-
Proposition 7. Could you state the assumptions required for this result? They are not stated in the proposition in the main text. Without these assumptions, the proposition as stated in the main text admits counterexamples.