acceptodds
Under review as a conference paper at ICLR 2027

What pass@k Measures When Attempts Share State: Count Runs, Not Attempts

Abstract

Search trees, planners and self-correcting agents make the attempts of one run share a random state, yet their results are often scored with a pass@k estimator built for independent draws, which public harnesses apply to whatever attempts they receive. Prior work separates the procedure's success probability from the independent-attempt one; we ask which of them such a number measures and how to report it validly, taking the complete run as the sampling unit. Whether a run succeeds within k attempts is an unbiased estimate of the procedure's success probability succ@k under any within-run dependence, and succ@k is the only quantity estimable from the first k attempts, with estimates in [0,1], that equals pass@k when attempts are independent. Pooling attempts across runs estimates a third quantity, which we give in closed form and which, for attempts conditionally i.i.d. within a run, lies between succ@k and the independent-attempt target pass@k^iid. Under a calibration condition and over a rich class of dependence structures, pass@k^iid has a uniformly unbiased estimator exactly when at least k complete runs are observed, however much is recorded inside each run, while per-question run-level confidence sequences are valid at any number of runs, for succ@k and, under the same condition, for pass@k^iid. In 8 designs spanning four dependence mechanisms, the pooled report exceeds the run-level estimate of succ@4 in every design, by +0.0205 to +0.1766, and on one design a run-level interval contains a fresh reference 0.952 of the time, against 0.104 for an attempt-level one. The fix is to name the target and count fresh runs.

Primary area:
datasets and benchmarks
Keywords:
pass@k, evaluation of language models, dependent samples, estimands, unbiased estimation, anytime-valid inference, confidence sequences, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jNjgOpenReview ↗
Rating
2(−1)
Confidence
1(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies what pass@k measures when attempts within an agent run share state. It distinguishes procedure success (succ@k), independent-attempt success (pass@k_iid), and pass@k computed from pooled attempts. It characterizes the pooled estimator under a conditionally i.i.d. latent-variable model, establishes a uniqueness result for succ@k under specified constraints, and studies the run-count threshold for unbiased estimation of pass@k_iid, minimax bias, confidence sequences, and sampling costs. Experiments across eight designs and four dependence mechanisms show that pooled pass@4 exceeds the run-level estimate of succ@4.

Critical strengths
  1. The paper provides a rigorous characterization of what pass@k measures when attempts share state. By distinguishing procedure-level success, independent-attempt success, and pooled reporting, it clarifies how evaluation design can affect reported scores and lead to misleading interpretations.

  2. The paper systematically examines unbiased estimability, statistical inference, and sampling costs under dependent attempts. The theoretical results, together with experiments across multiple dependence mechanisms, provide a coherent framework and practical guidance for evaluating multi-attempt agents.

Critical weaknesses
  1. Prior work has already distinguished dependent- and independent-attempt success and emphasized the importance of independent complete runs. The proofs and constructions for several other results, including the run-count threshold and confidence sequences, rely on established statistical principles. Although the exact pooled-score characterization and uniqueness result are valuable, the extent of the conceptual novelty beyond these prior insights remains limited.

  2. The ordering between pooled pass@k, succ@k, and pass@k_iid in Proposition 3.2 relies on the conditionally i.i.d. latent-variable model in Definition 3.1. This assumption does not generally hold for adaptive agents, where later attempts depend on earlier outputs or feedback, including the self-debugging and tree-search settings evaluated in the paper. Although the authors acknowledge this limitation, the exact pooled-score characterization does not directly extend to these settings, limiting the generality of this particular theoretical contribution.

Questions for authors
  1. Although Tables 38–39 and 51 examine decision-level consequences through simulations, measured runs, and constructed examples, the impact on comparisons between distinct real systems remains unclear. Could the authors examine whether run-aware evaluation changes model rankings, the direction of performance differences, or statistical significance under matched evaluation settings? Reporting all comparable pairs examined, including those whose conclusions remain unchanged, would help establish the practical impact.

  2. Theorem A.6 establishes a sharp minimax-bias bound over real-valued estimators, which need not lie in [0,1]. Does this bound remain attainable when estimators are constrained to [0,1]? If not, could the authors provide a simple example illustrating the difference and discuss its implications for practical probability reporting?

1 comment
AuthorsOpenReview ↗

What Pooled pass@k Measures When Attempts Share State

Thank you for asking what our work adds. We make four main contributions.

1. What pooled pass@k measures. There are two target probabilities. Following Liu et al. (2025), succ@k is the chance of at least one success within kk attempts in one run. Independent-attempt pass@k is the chance of at least one success in kk fresh, independent attempts. Pooled pass@k is a score computed by combining attempts across runs of the same problem. We derive its exact expectation for a finite evaluation (Proposition A.21). Under the conditionally i.i.d. model, this expectation lies between the two targets (Proposition 3.2). Changing the number of runs can therefore change the expected score of an unchanged procedure. For adaptive agents, this ordering can fail. Scoring each run 1 if it succeeds within kk attempts and 0 otherwise still estimates succ@k without bias (Proposition 3.4). Under Theorem 3.5's conditions, this is the unique scoring rule in [0,1][0,1] on one run of kk attempts that agrees in expectation with classical pass@k under i.i.d. sampling.

2. How many runs are needed. Applying a classical statistical criterion, Theorem 4.1 shows that, under its assumptions, unbiased estimation of independent-attempt pass@k requires at least kk independent runs, and kk suffice. More attempts within a run cannot make up the difference. Below this threshold, Theorem A.6 characterizes the smallest achievable worst-case absolute bias.

3. How to report uncertainty. We adapt confidence sequences to both targets so that, under the stated assumptions, intervals remain valid when evaluators use the data to decide when to stop. We also show which part of the variance only fresh runs can reduce. This guides the choice between starting a run and adding an attempt.

4. What happens in practice. Across eight designs covering four forms of dependence, pooled pass@4 exceeded the observed run-success rate by 2.05 to 17.66 percentage points (the pooled excess). Run-level intervals also had better interval scores in all eight designs. On five additional open-weight models, four fresh, independent attempts had a higher estimated chance of success than four attempts sharing one prefix (a positive estimand gap).

Does scoring change the model ranking? For the Qwen2.5 pair in Table 24, 7B led 1.5B under both methods. Both used the same self-debugging procedure on the same 164 HumanEval problems, with 16 runs per model per problem. At k=4k=4, the 7B lead was 22.29 percentage points under pooled scoring (95% CI 17.31 to 27.36) and 28.43 points under run-level scoring (95% CI 23.36 to 33.50). We computed both intervals from the submitted outcomes using 10,000 paired bootstrap resamples of problems. Both intervals exclude zero. Pooling preserved the ranking but narrowed the measured lead by 6.14 points.

Can a bounded estimator reach the bound in Theorem A.6? The lower bound still holds, but it need not be attainable. Take k=2k=2 and one run in which all attempts succeed with probability qq and all fail otherwise. The run reveals one binary outcome XX, and the independent-attempt target is 2q−q22q-q^2. The estimator X+1/8X+1/8 attains the optimal worst-case absolute bias of 0.125, but outputs 9/89/8 on success. If every output must lie in [0,1][0,1], the optimal rule reports 3−223-2\sqrt{2} on failure and 1 on success. Its worst-case absolute bias is 3−22≈0.17163-2\sqrt{2}\approx0.1716. This quantifies the cost of requiring every reported estimate to be a valid probability.

Reviewer_T8mqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses an issue with the pass@k score for assessing multi-attempt LLM systems. Pass@k assumes each solution attempt to be independent; however, in practice, many attempts are not independent (for example, an LLM repeatedly revising a coding solution). The paper proposes a solution to this issue, proposing that independent executions of the entire procedure, rather than individual attempts, should be used as the unit of evaluation. The authors provide theoretical results and experimental validation demonstrating the effects of dependence on pass@k estimates and confidence intervals.

Critical strengths

The paper addresses an important issue in evaluating multi-attempt LLM systems, where multiple attempts at solutions are not independent and thus pass@k estimates can be misleading.

Critical weaknesses

Unfortunately, I found the presentation of this paper to be inscrutable; I find much of the exposition almost impossible to follow. Much of the manuscript is written in highly compressed, telegraphic language that introduces new terminology, mathematical objects, and theoretical/experimental claims without sufficient definition or context. Many sentences combine methodology, results, conclusions, and experiments into one sentence. I found the narrative structure of the paper confusing, jumping forward and backward in time under the assumption that the reader is familiar with the entire content of the paper from the outset. I frequently found myself unable to understand what was being claimed without substantial inference on my part. Some representative examples:

  • Introduction, paragraph 2: Terms such as "pooling all attempts of a question," "the pooled excess," "Arm A," a "fresh 128-run reference," and "the run-level estimate of the procedure’s own success" are all jargon internal to the paper, introduced without any context and before the concepts and experimental procedures necessary to interpret them have been established.

  • 3.1, line 137: The authors introduce a deluge of mathematical concepts and notation with no explanation of what these objects represent, why they are important, or how they relate to each other.

  • 5.3, line 288: The paper shifts abruptly from a theoretical discussion into a densely compressed experimental evaluation; Here, they introduce experimental configurations, statistical procedures, and conclusions without clearly explaining what is being compared, why, or what the results establish.

I don't find these to be isolated examples, as the entire paper is rife with these issues. I found myself repeatedly reconstructing what the authors appear to intend rather than evaluating clearly stated claims.

As such, I feel I am unable to meaningfully evaluate any scientific contributions this paper may make. I understand the general problem being addressed, but cannot confidently determine what the specific theoretical contributions are, how they are different from prior work, or what the experiments establish. These are elements that the paper must clearly communicate, and, in my view, the manuscript in its current form does not adequately accomplish this. Thus, I recommend this paper be rejected.

Questions for authors

I do not have any specific technical questions that could be resolved through clarification in the rebuttal. It is my position that the presentation of the manuscript requires substantial work outside the scope of the rebuttal period, including a heavy rewrite that clearly communicates the problem, the proposed solution, and the theoretical/experimental evidence supporting the paper's claims.

2 comments
AuthorsOpenReview ↗

Response to Comments on Presentation and Contributions

We thank the reviewer for the detailed feedback. We recognize that the presentation could be improved to make the contributions more accessible.

Our specific contributions include the finite-design characterization of the pooled estimator (Proposition 3.2), the uniqueness of succ@k as a range-preserving estimable extension (Theorem 3.5), and the complete-run threshold under calibrated kernels (Theorem 4.1).

We will refine the organization, introduce key concepts before experimental details, and more clearly distinguish our contributions from established statistical results.

Thank you for your clarifications. However, my concerns about the paper's clarity and technical substance remain unresolved. I therefore maintain my recommendation.

Reviewer_JewaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is an LLM generated paper with zero clarity and zero merit. I will not review LLM output.

Critical strengths

None.

Critical weaknesses

Everything. The paper is unrecoverable.

Questions for authors

None.

1 comment
AuthorsOpenReview ↗

Response to Unsupported Ethics Allegation

We strongly disagree with the reviewer's assertion that our paper is "entirely LLM generated" and formally contest the associated ethics allegation.

Our submission explicitly discloses AI assistance in both the submission form and the AI Use Statement (p. 9). ICLR 2027 permits AI-assisted research and writing subject to its disclosure requirements. Disclosed AI assistance does not, by itself, constitute an ethics violation.

The manuscript presents specific theoretical results, complete mathematical proofs (Appendix A), empirical evaluations, and reproducibility materials. The authors take full responsibility for the scientific content.

The review provides no specific evidence of a disclosure violation, research-integrity issue, or other breach of ICLR's ethics policy.

We therefore request an evidence-based reconsideration of the ethics flag. An allegation of this seriousness should be supported by identifiable facts and an applicable policy provision, rather than an unsupported characterization of the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.