- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies , the probability of succeeding on all independent attempts, averaged over tasks. It derives the upward bias of the plug-in estimate, analyzes the variance of the existing unbiased estimator, and shows how resampling trials within tasks can put the bias back into a bootstrap interval. It then applies bounded-mean confidence intervals and examines the implications for comparing agents. The simulations and rollout experiments support the estimation problem, although the broader ranking conclusions are less uniform.
The target is an average over a task population of the probability that all k attempts succeed, not the probability that at least one attempt succeeds. The per-task unbiased estimate is a falling-factorial ratio, defined when the number of trials is at least k. Across independent sampled tasks this gives a bounded observation to which standard mean-inference tools can be applied. The empirical sections combine simulations, released multi-trial data, new preregistered rollouts and a separate accuracy-level leaderboard audit. These components answer different questions, so the paper’s overall interpretation depends on keeping their scope distinct.
Critical strengths
-
The paper clearly explains a nonlinear-estimation pitfall. Its exact bias calculation quantifies an effect that can be large at small trial budgets, and the bootstrap argument makes the centering issue understandable. The authors appropriately credit the existing unbiased estimator and distinguish the problematic resampling variant from the functioning task-level bootstrap.
The factorial-moment calculation provides more information than merely observing Jensen’s inequality. It shows how the discrepancy changes with trial depth and k, and helps quantify regimes where a plug-in number is misleading. I also appreciate the distinction between absolute and relative bias: a small reliability probability can have a large relative error without an equally large absolute error. Those are useful explanatory choices. My criticism of novelty should not obscure that the presentation can help practitioners use existing statistical tools correctly.
-
The comparison of interval width and coverage is useful, and the authors report their preregistered hypotheses that did not hold. Those admissions help establish the scope of the evidence. The paper does not present every experiment as a confirmation of the desired conclusion.
The discussion of the hierarchical Bayesian alternative is also balanced in several respects. The authors show settings in which it gives narrower intervals, rather than treating a distribution-free guarantee as automatically preferable. They explain the model-shape concern and acknowledge that a functioning task-level bootstrap can be narrower. This is a useful way to present a tradeoff between finite-sample guarantees and practical precision. Reporting the null or contrary outcomes in the higher-k experiments is a strength that should remain prominent in any revision.
Critical weaknesses
-
As a statistician, I appreciate efforts to bring statistical reasoning into AI evaluation. However, a contribution to this area should be motivated by a consequential evaluation problem and demonstrate what changes when that problem is addressed. Applying familiar techniques to an AI metric is not sufficient on its own. Here, the main ingredients are largely established: binomial factorial moments for the plug-in bias, a classical U-statistic estimator and variance calculation, and existing confidence procedures for bounded means. The authors acknowledge that τ-bench already uses the unbiased estimator, and Section 3.1 says they cannot identify a published benchmark using the criticized hierarchical-bootstrap variant. This leaves the practical motivation insufficiently established relative to the paper’s broad framing. I would like evidence identifying the affected evaluation pipelines and the substantive model comparisons or deployment conclusions that change after correction. A related caution is Li, Xin, Long, and Weijie J. Su (2025), Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?. Its theorem experiment extrapolates counts of verified generated names, rather than establishing proof or application ability, a limitation the authors acknowledge. This illustrates why estimating a well-defined output statistic does not by itself establish a broader capability claim. For the present submission, the analogous requirement is to show that correcting the statistical target materially improves real evaluation decisions.
The key issue for me is therefore not whether the bias exists; it does under the stated sampling model. It is whether the paper establishes that this bias is a binding problem for the AI comparisons it foregrounds. The existence of a natural but incorrect implementation is useful as a warning, yet it is different from demonstrating that existing published evaluations have been materially affected. Where the deployed suite already uses the classical unbiased statistic, the remaining contribution is inference and interpretation, and the comparison should be made against the working task-level procedure rather than only against a defective resampling variant.
A finite-sample guarantee can still be valuable, particularly when the cost of an incorrect deployment decision is high. However, the paper needs to connect that guarantee to an actual evaluation decision and show when the extra conservatism is worth its cost. Its experiments also report settings where the betting rule cannot distinguish any models. That outcome is not a theoretical failure, but it limits the practical claim. A stronger empirical contribution would show what a user would select or conclude differently with the recommended procedure, at the budgets and task structure of a real suite.
-
The empirical ranking conclusions do not bridge that gap convincingly. The six-leaderboard audit is at k=1, where the nonlinear plug-in bias vanishes. That is evidence about ordinary accuracy-ranking uncertainty, not direct validation of the paper’s pass^k-specific problem. The four-model higher-k experiment does not establish the proposed leaderboard-level deterioration, and the paper acknowledges that higher k can separate agents with similar accuracy. Reporting these outcomes is good practice, but the abstract and lower-bound language still suggest a broader conclusion than they support. The paired planning rule also needs clarification about cross-arm covariance and its approximate power assumptions. Taken together, I see a potentially useful statistical note whose current evidence does not establish a sufficiently substantial methodological or empirical contribution for acceptance.
There are further interpretation issues around statistical indistinguishability. A confidence interval containing zero does not establish that two models have equal capability, and overlap of two marginal intervals is not equivalent to a paired difference test. Pairwise confidence statements across many model comparisons also do not automatically provide simultaneous ranking guarantees.
The task-population assumption deserves equal attention. If a benchmark is treated as a fixed set of tasks, the uncertainty target differs from inference to a broader distribution. If task families share templates or state, counting each task as an independent observation can overstate the effective sample size. The paper acknowledges clustering and distinguishes fixed-suite inference in its limitations. I would like the empirical audit to make those choices operational: identify the task sampling unit, explain complete-case selection and variable trial depth, and show which conclusions survive a dependence-aware analysis. Correcting centering is only one part of making the final comparison trustworthy.
Overall Assessment
Presentation: The mathematical argument is understandable, but the interpretation of the ranking results needs to be more consistent.
The manuscript often states important qualifications, but the surrounding framing does not always retain them. A reader should be able to identify which claim concerns estimator bias, which concerns interval coverage, and which concerns ranking stability without reconstructing that distinction from the limitations section.
Informativeness: The statistical exposition is useful, but the methodological novelty and demonstrated consequences for existing AI evaluations are limited.
Soundness: I have fixable concerns about the paired planning formula and the scope of the empirical claims.
I recommend rejection in its current form. The paper offers useful exposition and a plausible warning about reliability estimation, but its statistical novelty is modest and its demonstrated impact on actual agent-evaluation practice remains too limited. My concern is the scale of the contribution, not the simplicity of the proofs. A straightforward analysis can be important when it establishes a real problem and changes meaningful conclusions; I do not think the current evidence meets that standard convincingly. This assessment does not imply misconduct, and I appreciate the authors’ explicit reporting of limitations and negative results.
I encourage the authors to develop the strongest practical version of the contribution. A clear audit of an affected pipeline, a reproducible corrected analysis, and an explanation of the decision consequences would be more persuasive than expanding the number of general claims about ranking noise. The transparent reporting of unfavorable results should be retained. My rejection is compatible with recognizing that the estimator discussion and implementation warning could be useful to the community in a more narrowly framed contribution.
Questions for authors
-
Which published agent-evaluation pipelines actually use the biased plug-in estimator or the criticized hierarchical-bootstrap procedure? After correcting those pipelines, which model rankings, reliability decisions or reported scientific conclusions change materially? How does this differ from using the unbiased estimator already employed by τ-bench and a task-level paired bootstrap?
-
Which methodological results are new beyond the established binomial, U-statistic and bounded-mean tools, and what is the strongest pass^k-specific empirical conclusion supported by the higher-k experiments? Can the authors reconcile that conclusion with the k=1 audit and lower-bound framing, and specify the paired variance term in Equation 4, including cross-arm covariance?
-
In which realistic task/trial-budget settings does the guaranteed interval support a consequential comparison that a correctly implemented task-level paired bootstrap does not? When the guaranteed rule is uninformative, what recommendation does the paper make to a benchmark user?
-
Are the reported statistical tiers based on paired difference tests or overlap of marginal intervals, and what uncertainty claim is intended across the full collection of model pairs? How do the reported rates change when the independent unit is a task family rather than an individual task?