acceptodds
Under review as a conference paper at ICLR 2027

Are Agent Reliability Rankings Real? A Statistical Treatment of the Metric

Abstract

The reliability of LLM agents — their tendency to succeed repeatably, not merely once — is increasingly measured by , the probability that an agent succeeds on all independent attempts of a task. It is reported by -bench, -bench and agent-memory benchmarks, and used to rank models. We show that its statistics have never been checked, and that its failures share one mechanism: mis-centering. The ubiquitous plug-in estimator is provably upward biased. We give its exact bias in closed form via Stirling numbers; the relative bias grows without bound in (at , : 3% at , 101% at ). That bias then propagates silently into interval estimates. The hierarchical (task trial) bootstrap resamples within-task counts as , which makes even the unbiased estimator unbiased for rather than : its resampling distribution is centred exactly on the plug-in. Coverage therefore falls as tasks are added — 91% to 36% to 7% at (, ) — which no variance-based account predicts, and re-centring does not repair. We report equally that the plain task-level bootstrap does not fail. We then give the first interval with a finite-sample, distribution-free coverage guarantee, via betting concentration on the bounded unbiased estimator, and compare it against a hierarchical Beta-Binomial baseline. That baseline is narrower where its Beta shape holds; but on a shape agent suites actually have — an atom at , tasks every agent solves — it loses coverage as tasks are added (0.59 at ), the same mis-centering signature, this time from shrinkage rather than resampling. And at the budgets suites run, neither it nor the bootstrap reaches nominal (0.90-0.93 at ), where the guaranteed interval covers 1.00. Inverting the closed-form variance yields a design rule: separation is task-limited, not trial-limited. Finally we test the theory on data. On 4,600 agent rollouts we run ourselves (pre-registered), Theorem 1 predicts the observed bias with no free parameters (correlation 0.969), and the plug-in overstates by 20-147% — worst for the least reliable agent, where deployment is actually in question. A audit of six leaderboards — the regime most favourable to separation, and therefore a lower bound on the problem — finds 33% of all model pairs statistically indistinguishable and 16 of 30 models sharing -bench's top tier. We release a drop-in tool.

Primary area:
datasets and benchmarks
Keywords:
pass^k, agent reliability, LLM agents, evaluation, benchmarking, leaderboards, confidence intervals, distribution-free inference, betting concentration, e-values, unbiased estimation, bootstrap, pre-registration, statistical power, tau-bench

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qL2xOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies passk\mathrm{pass}^k, the probability of succeeding on all kk independent attempts, averaged over tasks. It derives the upward bias of the plug-in estimate, analyzes the variance of the existing unbiased estimator, and shows how resampling trials within tasks can put the bias back into a bootstrap interval. It then applies bounded-mean confidence intervals and examines the implications for comparing agents. The simulations and rollout experiments support the estimation problem, although the broader ranking conclusions are less uniform.

The target is an average over a task population of the probability that all k attempts succeed, not the probability that at least one attempt succeeds. The per-task unbiased estimate is a falling-factorial ratio, defined when the number of trials is at least k. Across independent sampled tasks this gives a bounded observation to which standard mean-inference tools can be applied. The empirical sections combine simulations, released multi-trial data, new preregistered rollouts and a separate accuracy-level leaderboard audit. These components answer different questions, so the paper’s overall interpretation depends on keeping their scope distinct.

Critical strengths
  1. The paper clearly explains a nonlinear-estimation pitfall. Its exact bias calculation quantifies an effect that can be large at small trial budgets, and the bootstrap argument makes the centering issue understandable. The authors appropriately credit the existing unbiased estimator and distinguish the problematic resampling variant from the functioning task-level bootstrap.

    The factorial-moment calculation provides more information than merely observing Jensen’s inequality. It shows how the discrepancy changes with trial depth and k, and helps quantify regimes where a plug-in number is misleading. I also appreciate the distinction between absolute and relative bias: a small reliability probability can have a large relative error without an equally large absolute error. Those are useful explanatory choices. My criticism of novelty should not obscure that the presentation can help practitioners use existing statistical tools correctly.

  2. The comparison of interval width and coverage is useful, and the authors report their preregistered hypotheses that did not hold. Those admissions help establish the scope of the evidence. The paper does not present every experiment as a confirmation of the desired conclusion.

    The discussion of the hierarchical Bayesian alternative is also balanced in several respects. The authors show settings in which it gives narrower intervals, rather than treating a distribution-free guarantee as automatically preferable. They explain the model-shape concern and acknowledge that a functioning task-level bootstrap can be narrower. This is a useful way to present a tradeoff between finite-sample guarantees and practical precision. Reporting the null or contrary outcomes in the higher-k experiments is a strength that should remain prominent in any revision.

Critical weaknesses
  1. As a statistician, I appreciate efforts to bring statistical reasoning into AI evaluation. However, a contribution to this area should be motivated by a consequential evaluation problem and demonstrate what changes when that problem is addressed. Applying familiar techniques to an AI metric is not sufficient on its own. Here, the main ingredients are largely established: binomial factorial moments for the plug-in bias, a classical U-statistic estimator and variance calculation, and existing confidence procedures for bounded means. The authors acknowledge that τ-bench already uses the unbiased estimator, and Section 3.1 says they cannot identify a published benchmark using the criticized hierarchical-bootstrap variant. This leaves the practical motivation insufficiently established relative to the paper’s broad framing. I would like evidence identifying the affected evaluation pipelines and the substantive model comparisons or deployment conclusions that change after correction. A related caution is Li, Xin, Long, and Weijie J. Su (2025), Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?. Its theorem experiment extrapolates counts of verified generated names, rather than establishing proof or application ability, a limitation the authors acknowledge. This illustrates why estimating a well-defined output statistic does not by itself establish a broader capability claim. For the present submission, the analogous requirement is to show that correcting the statistical target materially improves real evaluation decisions.

    The key issue for me is therefore not whether the bias exists; it does under the stated sampling model. It is whether the paper establishes that this bias is a binding problem for the AI comparisons it foregrounds. The existence of a natural but incorrect implementation is useful as a warning, yet it is different from demonstrating that existing published evaluations have been materially affected. Where the deployed suite already uses the classical unbiased statistic, the remaining contribution is inference and interpretation, and the comparison should be made against the working task-level procedure rather than only against a defective resampling variant.

    A finite-sample guarantee can still be valuable, particularly when the cost of an incorrect deployment decision is high. However, the paper needs to connect that guarantee to an actual evaluation decision and show when the extra conservatism is worth its cost. Its experiments also report settings where the betting rule cannot distinguish any models. That outcome is not a theoretical failure, but it limits the practical claim. A stronger empirical contribution would show what a user would select or conclude differently with the recommended procedure, at the budgets and task structure of a real suite.

  2. The empirical ranking conclusions do not bridge that gap convincingly. The six-leaderboard audit is at k=1, where the nonlinear plug-in bias vanishes. That is evidence about ordinary accuracy-ranking uncertainty, not direct validation of the paper’s pass^k-specific problem. The four-model higher-k experiment does not establish the proposed leaderboard-level deterioration, and the paper acknowledges that higher k can separate agents with similar accuracy. Reporting these outcomes is good practice, but the abstract and lower-bound language still suggest a broader conclusion than they support. The paired planning rule also needs clarification about cross-arm covariance and its approximate power assumptions. Taken together, I see a potentially useful statistical note whose current evidence does not establish a sufficiently substantial methodological or empirical contribution for acceptance.

    There are further interpretation issues around statistical indistinguishability. A confidence interval containing zero does not establish that two models have equal capability, and overlap of two marginal intervals is not equivalent to a paired difference test. Pairwise confidence statements across many model comparisons also do not automatically provide simultaneous ranking guarantees.

    The task-population assumption deserves equal attention. If a benchmark is treated as a fixed set of tasks, the uncertainty target differs from inference to a broader distribution. If task families share templates or state, counting each task as an independent observation can overstate the effective sample size. The paper acknowledges clustering and distinguishes fixed-suite inference in its limitations. I would like the empirical audit to make those choices operational: identify the task sampling unit, explain complete-case selection and variable trial depth, and show which conclusions survive a dependence-aware analysis. Correcting centering is only one part of making the final comparison trustworthy.

Overall Assessment

Presentation: The mathematical argument is understandable, but the interpretation of the ranking results needs to be more consistent.

The manuscript often states important qualifications, but the surrounding framing does not always retain them. A reader should be able to identify which claim concerns estimator bias, which concerns interval coverage, and which concerns ranking stability without reconstructing that distinction from the limitations section.

Informativeness: The statistical exposition is useful, but the methodological novelty and demonstrated consequences for existing AI evaluations are limited.

Soundness: I have fixable concerns about the paired planning formula and the scope of the empirical claims.

I recommend rejection in its current form. The paper offers useful exposition and a plausible warning about reliability estimation, but its statistical novelty is modest and its demonstrated impact on actual agent-evaluation practice remains too limited. My concern is the scale of the contribution, not the simplicity of the proofs. A straightforward analysis can be important when it establishes a real problem and changes meaningful conclusions; I do not think the current evidence meets that standard convincingly. This assessment does not imply misconduct, and I appreciate the authors’ explicit reporting of limitations and negative results.

I encourage the authors to develop the strongest practical version of the contribution. A clear audit of an affected pipeline, a reproducible corrected analysis, and an explanation of the decision consequences would be more persuasive than expanding the number of general claims about ranking noise. The transparent reporting of unfavorable results should be retained. My rejection is compatible with recognizing that the estimator discussion and implementation warning could be useful to the community in a more narrowly framed contribution.

Questions for authors
  1. Which published agent-evaluation pipelines actually use the biased plug-in estimator or the criticized hierarchical-bootstrap procedure? After correcting those pipelines, which model rankings, reliability decisions or reported scientific conclusions change materially? How does this differ from using the unbiased estimator already employed by τ-bench and a task-level paired bootstrap?

  2. Which methodological results are new beyond the established binomial, U-statistic and bounded-mean tools, and what is the strongest pass^k-specific empirical conclusion supported by the higher-k experiments? Can the authors reconcile that conclusion with the k=1 audit and lower-bound framing, and specify the paired variance term in Equation 4, including cross-arm covariance?

  3. In which realistic task/trial-budget settings does the guaranteed interval support a consequential comparison that a correctly implemented task-level paired bootstrap does not? When the guaranteed rule is uninformative, what recommendation does the paper make to a benchmark user?

  4. Are the reported statistical tiers based on paired difference tests or overlap of marginal intervals, and what uncertainty claim is intended across the full collection of model pairs? How do the reported rates change when the independent unit is a task family rather than an individual task?

Reviewer_9SS8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the statistical reliability of the pass^k metric used to evaluate LLM agents, which measures the probability that an agent successfully completes a task across all k independent trials. The paper investigates whether commonly used estimators and confidence intervals can provide reliable performance estimates and support meaningful comparisons between agents.

The authors derive the bias of the plug-in estimator, analyze failures of certain confidence interval methods, and propose a distribution-free confidence interval with finite-sample guarantees. They also study sample-size requirements for reliable model comparisons and evaluate their findings through simulations and real agent benchmarks. Compared with prior work, the main contributions are the statistical characterization of pass^k estimation, the analysis of confidence interval failures, and the development of a finite-sample-valid evaluation procedure. The unbiased estimator itself is already known and used in the original τ-bench.

Critical strengths
  1. Interesting and relevant problem. The paper studies an important problem in LLM-agent evaluation: how to reliably measure agent success rates and determine whether performance differences between agents are statistically meaningful. This question is particularly relevant as LLM agents are increasingly evaluated for reliability across repeated executions rather than single-attempt success.

  2. The bias analysis of the plug-in pass^k estimator is interesting and provides useful insights into potential estimation errors. The analysis of how bias can propagate into confidence interval estimation is also valuable, particularly under limited evaluation budgets.

Critical weaknesses
  1. Unclear practical motivation. The paper does not clearly establish how widely the identified statistical problems occur in existing LLM evaluations. For example, τ-bench already uses the unbiased estimator, and the authors acknowledge that the problematic hierarchical bootstrap is not known to be used in published benchmarks. This raises concerns about the practical significance of the analysis.
  2. Unclear contributions and practical insights. Despite extensive statistical analysis, the paper does not clearly highlight its main novel contributions or explain their practical implications. In particular, what actionable insights can practitioners take from these results, and what are the benefits over existing unbiased estimators and standard task-level bootstrapping?
  3. Limited empirical evidence for pass^k rankings. The leaderboard analysis mainly focuses on k=1, where pass^k reduces to standard accuracy, and some evaluated benchmarks do not report pass^k. The practical impact of the proposed analysis on actual pass^k-based model rankings therefore remains unclear.
Questions for authors
  1. Could the authors provide more concrete evidence of how frequently the biased plug-in estimator and the problematic hierarchical bootstrap are used in existing LLM-agent benchmarks or industrial evaluation pipelines? Given that τ-bench already uses the unbiased estimator, which widely adopted evaluation practices are directly affected by the identified problems?

  2. Figure 2: Could the authors clarify whether the results in Figure 2, particularly the confidence intervals and conclusions about model rankings, are newly computed by the authors using the released τ²-bench data or are directly taken from the original τ²-bench paper? If they are newly computed, it would be helpful to clarify the data processing and confidence interval construction and explain how these results extend the original benchmark analysis.

  3. Main novelty: Could the authors clarify which theoretical or methodological contributions are genuinely new compared with established results on unbiased estimation, U-statistics, and concentration-based confidence intervals? What is the most important new insight beyond applying these existing techniques to pass^k?

Reviewer_859YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes estimation and uncertainty for pass^k, the probability of k successful attempts averaged over tasks. It derives the exact upward bias of the plug-in estimator and the variance of the classical unbiased estimator, explains a centering failure in a task-by-trial bootstrap, and constructs intervals by applying existing bounded-mean concentration methods. Simulations, accuracy-leaderboard audits, and multi-trial agent data illustrate the practical effects. A paired-comparison design rule relates the task and trial budgets needed to resolve a reliability gap.

Critical strengths
  1. The exact plug-in bias, U-statistic variance, and bootstrap-centering identity explain practical pitfalls and support concrete reporting guidance (pp.3-4; Appendices A,G). The unbiased estimator's classical status and the working task-only bootstrap are clearly acknowledged.
  2. The paper reports the hierarchical Bayesian comparator's width advantage where its assumptions hold, the cost of guaranteed intervals, and failed or uninformative real-data hypotheses (pp.4-5, Table1; p.8; Appendices E-F). This provides useful evidence beyond favorable examples.
Critical weaknesses
  1. Eq.4's infinite-trial floor needs latent paired variance Var(p_A^k-p_B^k), not the variance of noisy observed differences. The correction and uncertainty behind 0.0580/0.05438 are not specified, and the finite-n term is defined using a single-agent variance (pp.6,8). At n=10,k=8, the needed latent variance involves moments beyond those identified by arbitrary binomial mixtures at that trial depth. This leaves the 242/574-task impossibility interpretations unestablished, although the general task-limited phenomenon is valid when the latent variance is known.
  2. The 16-of-30 top tier is defined through marginal interval overlap, which does not imply that paired-difference intervals include zero (pp.6-7; Fig.2, p.9). The all-pair fraction and top-tier count also depend on the model slate, and the k=1 lower-bound wording conflicts with the qualification and counterexample in Sec.5. These issues weaken the headline interpretation of real rankings.
Questions for authors
  1. Please specify the latent paired-variance estimator, finite-trial noise correction, assumptions, and uncertainty behind the 242/574-task floors; derive the paired finite-n term, including both arms and any conditional covariance. How is the k=8,n=10 moment-identification issue handled? If the reported variances are current-depth observed variances, do the claims hold after narrowing their interpretation?
  2. Does the 16-model tier persist under the paper's paired-difference criterion, with a consistent task-ID protocol? Please state the comparison-wise or simultaneous scope, clarify the 278/228 task definitions, and align the k=1 interpretation throughout with the acknowledged lack of a general lower-bound result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.