Are Agent Reliability Rankings Real? A Statistical Treatment of the ${\mathrm{pass}}^k$ Metric

The reliability of LLM agents — their tendency to succeed repeatably, not merely once — is increasingly measured by ${\mathrm{pass}}^k$, the probability that an agent succeeds on all $k$ independent attempts of a task. It is reported by $\tau$-bench, ${\tau }^2$-bench and agent-memory benchmarks, and used to rank models. We show that its statistics have never been checked, and that its failures share one mechanism: mis-centering. The ubiquitous plug-in estimator $(c/n{)}^k$ is provably upward biased. We give its exact bias in closed form via Stirling numbers; the relative bias grows without bound in $k$ (at $p=0.75$, $n=10$: 3% at $k=2$, 101% at $k=8$). That bias then propagates silently into interval estimates. The hierarchical (task $\times$ trial) bootstrap resamples within-task counts as $\mathrm{Binomial}(n,\hat{p})$, which makes even the unbiased estimator unbiased for ${\hat{p}}^k$ rather than $p^k$: its resampling distribution is centred exactly on the plug-in. Coverage therefore falls as tasks are added — 91% to 36% to 7% at $M=25,115,230$ ($k=8$, $n=10$) — which no variance-based account predicts, and re-centring does not repair. We report equally that the plain task-level bootstrap does not fail. We then give the first ${\mathrm{pass}}^k$ interval with a finite-sample, distribution-free coverage guarantee, via betting concentration on the bounded unbiased estimator, and compare it against a hierarchical Beta-Binomial baseline. That baseline is narrower where its Beta shape holds; but on a shape agent suites actually have — an atom at $p=1$, tasks every agent solves — it loses coverage as tasks are added (0.59 at $M=230$), the same mis-centering signature, this time from shrinkage rather than resampling. And at the budgets suites run, neither it nor the bootstrap reaches nominal (0.90-0.93 at $M\le 50$), where the guaranteed interval covers 1.00. Inverting the closed-form variance yields a design rule: separation is task-limited, not trial-limited. Finally we test the theory on data. On 4,600 agent rollouts we run ourselves (pre-registered), Theorem 1 predicts the observed bias with no free parameters (correlation 0.969), and the plug-in overstates ${\mathrm{pass}}^8$ by 20-147% — worst for the least reliable agent, where deployment is actually in question. A $k=1$ audit of six leaderboards — the regime most favourable to separation, and therefore a lower bound on the problem — finds 33% of all model pairs statistically indistinguishable and 16 of 30 models sharing ${\tau }^2$-bench's top tier. We release a drop-in tool.