acceptodds
Under review as a conference paper at ICLR 2027

Success Has a Shape: TailPass@k for Repeated Sampling Evaluation

Abstract

Repeated-sampling evaluation measures language models over multiple non-deterministic attempts on each task, but scalar summaries such as accuracy, , , or a single thresholded repeated-sampling score can conflate systems with different operational behavior. A model that occasionally finds a correct answer can have high discovery but low repeatability, while another model may have lower single-sample accuracy but much stronger consistency across attempts. We characterize failure modes of commonly used evaluation metrics and address information loss and uncertainty through a discovery–stability tail profile: for each task, reporting budget , and threshold , the profile is the probability that the model produces at least correct responses among attempts. We place a Dirichlet–categorical model on each task's latent outcome distribution, with binary exact match recovered as the Beta–Bernoulli special case, and propagate this uncertainty to dataset-level profiles (across tasks), yielding closed-form posterior means and covariances, together with posterior Monte Carlo credible intervals. The resulting uncertainty-aware report is TailPass@. The framework recovers discovery, , stability, and single-sample accuracy as special cases and shows that accuracy is the area under the profile. Utilities over this profile combine its threshold probabilities into scalar scores that express preferences over discovery and repeatability, with uncertainty propagated to each score (e.g., power-moment utilities shift emphasis from discovery toward repeatability as their parameter increases). Exact enumeration of response patterns shows that endpoint metrics and uniform-threshold area can merge response patterns with the same total correctness but different discovery–stability structure, while the full profile exposes these distinctions.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Large Language Models, Evaluation Metrics, Bayesian Methods, Uncertainty Quantification, Model Ranking, Reasoning, Statistical Significance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3ereOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The key idea seems to be that LLMs are often evaluated based on whether they get the correct answer under some number of attempts. This can obfuscate performance differences, since, if a model is evaluated on k attempts, each model will have a distribution over the number of correct answers in its k attempts. Depending on how you choose to evaluate, the rankings of different models may change: some models may get one correct answer out of k more frequently, while others get higher percentages of their k attempts correct. As such, reporting the full distribution of successes over k attempts would be more informative about model behavior.

Critical strengths

None

Critical weaknesses

I have serious concerns with the presentation of this paper and found it very difficult to understand: the style is excessively verbose yet often lacking in crucial explanatory detail, and relatively simple concepts are obfuscated with breezy statements that are somehow both matter-of-fact and contentless. This isn't helped by the sudden notation and results dumps. I recommend that the authors do a full, human rewrite. As a reviewer, I struggled with how much effort I should put into deciphering this and ultimately decided that, regardless of the content, I would strongly recommend rejection based on the presentation alone.

Questions for authors

Did you use an LLM to generate the entire paper? If so, did you proofread the generation? Did you run any of the experiments yourselves?

2 comments
AuthorsOpenReview ↗

Request for specific feedback

First and foremost, your criticisms of the presentation of this paper and difficulty to understand are well taken. We revised the paper so that it can now be read without a background in Bayesian methods or advanced statistics. Also, the appendices only provide supporting details on the proposed method, additional evaluations, and mathematical proofs. The main text is sufficient to communicate the idea behind TailPass and its utility without needing to go back and forth between the main text and the appendices. Please let us know if you find the revised version clearer and easier to follow.

We understand that your decision, as you stated, is based on the presentation alone and that your recommendation holds regardless of the content, but we would like to ask for more specific feedback: Did you find a claim or result that is incorrect, or were you unable to determine its correctness from the exposition?

Questions

Did you use an LLM to generate the entire paper? If so, did you proofread the generation?

We used AI no more than what is disclosed in our AI use statement. The proposed methods and all mathematical derivations are the authors' own work.

Did you run any of the experiments yourselves?

There are 15 distinct experimental setups (7 empirical and 8 numerical; Section 4 and Appendices C, F-H, J-K) on 22 LLM configurations, as follows:

ExperimentDescriptionReference
1. Model comparisons and uncertaintyModel rankings and uncertainty across discovery, repeatability, and utility scores.Sections 4.1-4.2; Appendices F.2 and H
2. Greedy-prior validationWhether one greedy response improves utility estimates with limited stochastic data.Section 4.3; Appendices F.3 and I.3
3. Categorical rubric evaluationRankings under correctness, format, efficiency, and confidence rubrics.Section 4.4; Appendices F.4 and I.6
4. Ranking convergence and correlationConvergence and Kendall correlation against full-response accuracy rankings.Appendix F.1
5. Math and SuperGPQA profile comparisonsProfiles and utilities on 186 mathematics and 3,600 SuperGPQA questions.Appendices G.1-G.2
6. SuperGPQA held-out model comparisonsSelection regret and ranking agreement using disjoint fitting and held-out responses.Appendices G.3-G.4
7. Temperature sweepTemperature effects on Qwen3.5-4B's discovery and repeatability on AIME'26.Appendices H.1 and I.2
8. Profile-resolution enumerationInformation retained by full profiles versus endpoint scores at 50% accuracy.Appendix C.1
9. Matched-endpoint model selectionModel selection with equal accuracy and endpoints but differing intermediate tails.Appendices C.2-C.3
10. Random-histogram ranking reversalsBudget-dependent ranking reversals in sampled Dirichlet success-count histograms.Appendix J.1.3, "Randomly sampled histograms"
11. Exact ranking-reversal enumerationRanking reversals across 1,287 sorted count vectors and 827,541 pairs for five questions and eight attempts.Appendix J.1.3, "Exact finite-sample enumeration"
12. Finite-sample convergenceFinite-sample pass@N versus coverage and accuracy under a fixed zero-inflated Beta population.Appendix J.2.4, "Finite-sample convergence"
13. Random-population ranking disagreementAccuracy/pass@k disagreement across 200,000 zero-inflated Beta population pairs.Appendix J.2.4, "Comparisons between synthetic models"
14. Non-identifiability enumerationScalar-score collisions and how additional budgets distinguish tied distributions.Appendix J.3.4
15. Imperfect-verifier ranking disagreementRankings by verifier acceptance versus returned correctness for 200,000 synthetic model pairs.Appendix K.1, "Comparisons between synthetic models"

The experiments involved generating over 9.5 billion tokens and took us several months to complete. How could AI conduct such a variety of experiments to examine different aspects of evaluation? We think that if we could scaffold AI models to do so, that would be a contribution in its own right!

Please do let us know if you would like any details or explanations about any of the above experiments. Is there a particular result whose provenance you question? If so, please identify it so that we can address the concern directly.

Request

We understand that every reviewer is busy and that reviewing a long paper like ours takes more time, but you can skip the appendices and focus on the main text. In the main text, we discuss the problems with existing evaluation methods and how our proposed framework can address them. We welcome any criticism and will respond to questions about any technical aspect. But please provide specific criticisms and identify the weaknesses. If you wish to maintain your score, please at least clarify your specific concerns.

The issue is not that the appendix is long or that your paper has math, it's that the entire manuscript was written in Claudish. I'm not sure how the conference publication system should be adjusted to handle this kind of thing, but, in the current paradigm, relying on peer review to parse your slop for you is antisocial behavior. If you don't have time to write the paper (or clean up the AI writing), don't submit. I maintain my score.

Reviewer_2LGqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I have serious concerns about the paper and I was not able to understand it. It reads like its fully AI generated and regardless, does not contribute to scholarship in this field.

Critical strengths

N/A

Critical weaknesses

Unreadable

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Request for clarification on technical concerns and the ethics flag

Concerns

We would like to ask you for clarification on your serious concerns. We need at least some basis for discussion. Please point to the specific issues and explain your criticisms to give us the opportunity to discuss them with you and clear up any misunderstandings.

If you find that the contribution is not good enough, that's fair, but could you at least write a paper summary? That would help us distinguish a disagreement about its scientific value from a misunderstanding of the work, and give us something specific to address.

Ethics

We would like to ask for clarification: What in our method, mathematical derivations, or experiments led you to think that the submission is "AI Slop"? What specific ethical concern are you raising?

Honestly, we do not know how to use AI to propose a method that outperforms all existing evaluation methods (methods that have been widely used for years). How would we ask it to make the proposed metric tunable, measure uncertainty, use prior knowledge, and reduce collisions (when a metric outputs the same value for different answers)?

We have 15 distinct experimental setup (7 empirical and 8 numerical; Section 4 and Appendices C, F-H, J-K), evaluating 22 LLM configurations with over 9.5 billion tokens. How could AI conduct such a variety of experiments to examine different aspects of evaluation? In fact, if you ask frontier models (commercial or open-source) what the best metric is for evaluating LLMs in test-time scaling, their answer is a single metric: pass@k. Sorry, but we believe that simply calling our work "AI slop" is unfair to the effort we put into it.

We used AI no more than what is disclosed in our AI use statement. In the revised version, we explicitly state: We used commercial large language models and a local in-house model .. polishing the manuscript, summarizing research content ... However, the proposed methods and all mathematical derivations are the authors’ own work. Could you let us know which aspect of this disclosed use you consider ethically improper?

Request

We would like to ask you to assess the work on its scientific merits and explain your objections. We welcome any criticism and will respond to questions about the method, experiments, and all technical aspects. If you wish to maintain the ethics flag, please at least clarify your concerns and identify the violation with supporting evidence.

I have read the response and retain my score. Please do not submit a paper like this to any future conference.

Reviewer_nRiGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes replacing the ubiquitous pass@k metric for evals with a discovery-stability curve, which offers a more complete picture of both success probability and reliability. They show that in several cases, using pass@k model rankings can reverse as a function of k. The authors also provide a (relatively standard) mathematical framework for estimating the posterior probability that a model will pass a test under a given budget of attempts, under a given prior, and thus for providing probabilistic models that one model is more likely to do so than another. The approach offers researchers to flexibly weight discovery and reliability.

This is a pretty simple and straightforward contribution. It's obvious in hindsight that existing metrics can bias discovery over stability or vice versa. There isn't much in the way of novel theory or experiments here (nice to know about the reversal, but probably not a big surprise to most readers). It's more like an advert for something overlooked. I imagine this paper will be cited. I wouldn't be surprised if people adopted the proposed approach. So I think, although simple, it is useful.

Critical strengths

Overlooked problem, elegant solution.

Critical weaknesses

Seems a bit obvious.

Questions for authors

I honestly struggle to know what to ask.

3 comments
AuthorsOpenReview ↗

What TailPass adds beyond existing methods

We appreciate your description of the work as an "overlooked problem, elegant solution," and your recognition of the approach's usefulness in practice. We would, however, like to address "Seems a bit obvious." We do not think that this description adequately reflects the method or the theoretical and experimental results supporting it.

  1. Information that accuracy, discovery, and all-correct scores can miss. The contribution goes beyond observing that rankings change with the attempt budget. At a fixed budget of 16 attempts, we construct two models with exactly equal accuracy, discovery probability (at least one correct response), and all-correct probability. Yet one is preferable when at least four correct responses are required, and the other when at least thirteen are required, with opposing gaps of 18.70 percentage points. The TailPass profile records the probability of meeting each required success count (Appendix C.2).

  2. Joint uncertainty in the evaluation. Every success threshold uses the same responses. TailPass preserves the resulting dependence when calculating uncertainty in utilities and model comparisons. We derive the covariance between profile probabilities and use shared posterior draws to obtain credible intervals and probabilities that one model has a higher utility. This allows an evaluator to distinguish a higher estimated score from sufficient evidence of a higher score (Section 3.2; Appendix B.3).

  3. A tunable criterion with an explicit meaning. Utility weights specify how much each additional correct response is worth for the application. Discovery, average accuracy, and all-correct success are special cases. The weights are chosen independently of the observed rankings. TailPass therefore retains pass@k\mathrm{pass}@k when finding one correct response is the appropriate objective, while supporting other requirements within the same framework (Section 3.1; Appendix D).

  4. Prior evidence when responses are scarce. Separate responses, such as a greedy response, can inform the prior, with adjustable strength. We test the effect against held-out responses using U2(16)U_{2}(16), the expected squared fraction of correct responses in 16 future attempts. With four stochastic attempts per question, adding one greedy response per question lowers mean absolute error from 0.0691 to 0.0650. The benefit depends on the available data and utility (Section 4.3, Table 3).

  5. Evaluation beyond binary correctness. The categorical extension estimates the probability that the average rubric score over several attempts reaches a target. It supports partial credit and criteria such as answer format or response length, with uncertainty propagated through a Dirichlet model. The empirical rubric comparisons show how these criteria change model rankings (Sections 3.3 and 4.4).

Selection regret measures how much less likely the chosen model is to meet the required success count than the better model in each pair. On SuperGPQA, with 16 fitting and 64 held-out attempts per question, mean regret is 0.112 percentage points for TailPass, compared with 2.633 for choosing by pass@16\mathrm{pass}@16 across all requirements. Controls estimating the same profile give 0.197 for the finite-sample estimator and 0.118 for the binomial maximum-likelihood plug-in estimator (Appendix G.3, Table 26). These comparisons separate the benefit of choosing for the required threshold from the smaller additional benefit of Bayesian estimation. The evaluation holds out attempts on the same questions.

Unlike the existing methods such as pass@k\mathrm{pass}@k, passk\mathrm{pass}^{k}, and mg-pass@k\mathrm{mg\text{-}pass}@k, TailPass measures uncertainty and supports rubric-based scoring, tunable utilities, and prior evidence within one evaluation framework (Sections 3 and 5).

Please let us know if you have any further questions or would like any clarification.

I don't share the other authors' concerns around this paper being AI generated (although the replies to my review look pretty Claudish) I haven't seen anything to make me change my scores. I think this paper has at least one nice idea, but it does not feel groundbreaking to me.

AuthorsOpenReview ↗

Appreciation and Invitation to Use TailPass

We thank Reviewer nRiG especially for their evaluation of the scientific merit of the submission.

Let us emphasize its significance:

pass@k has been widely used in the community since 2021, when the metric was introduced in OpenAI's HumanEval paper. The origin of the metric goes back even further; it was introduced in 2019 in SPoC (Kulal et al.) to measure the fraction of problems successfully solved within a fixed budget of trials. We reviewed many papers (even from big names) that use the metric in a wrong way. For example, reporting pass@k when the total number of trials, N, is only slightly larger than k can lead to estimates with very high variance. Even reporting pass@1 (which is very common in model cards) with a small number of trials can lead to estimates with high variance; thus, rankings and comparisons between models may become unreliable. Besides that, reporting pass@1 without bootstrapping is technically just reporting average accuracy, without accounting for statistical uncertainty.

Our proposed method not only solves that problem (in a way that works even with few trials, since it's a Bayesian estimation with uncertainty), but also includes more properties (as mentioned in the "TailPass: Main Contributions" comment, with a concrete example).

Regardless of whether you raise your score, we are glad you find the idea nice, and we would like to invite you to use TailPass in your research; the implementation is available in the supplementary material. You can test it with your data and tweak it (e.g., lambda) for your specific applications, whether the evaluation results are binary (e.g., on math questions with final answers) or rubric-based (e.g., correct-short, correct-long, incorrect-short, incorrect-long), or for repeated sampling like top-p and incorporating prior knowledge (e.g., using one greedy sample), or for ranking models with uncertainty, or for using TailPass's uncertainty to find how many trials are needed to reach a certain level of uncertainty, and so on.

We would love to discuss more about the mathematical proof of why we claim TailPass is better than existing methods. Please let us know if you have any questions about that.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Properties of TailPass

Dear Reviewers and AC,

Our submission is more than 100 pages long, and we appreciate the time it takes to review it. The main text presents the paper's contributions and can be read on its own. Much of the appendix examines where existing evaluation metrics fail. During the rebuttal, we simplified some of the notation and expressions. Readers can follow the main argument without working through the proofs.

We began with a practical question about evaluating LLMs. Given several models that can answer the same question differently on each attempt (e.g., using non-deterministic decoding), how should we rank them or choose one for a task?

We want a metric that can distinguish between models consistently and tell us how certain we are about the ranking or evaluation. These two goals guide the design of TailPass. Bayes@N [1] estimates single-attempt performance and its uncertainty. Its estimates can't show how the chance of getting correct answers changes over more attempts. The scores pass@k\mathrm{pass}@k [2], passk\mathrm{pass}^{k} [3], and mG-pass@k\mathrm{mG\text{-}pass}@k [4] depend on that budget, so they can measure LLM performance under different attempt budgets but they hide differences between models and give no indication of how uncertain a ranking or evaluation is.

Our experiments and theorems examine where current metrics fail and how TailPass distinguishes models in those cases. Our reply to Reviewer 3ere includes a table of the experiments. As an example, consider the response matrices RAR_A and RBR_B for models A and B. Each row is a question and each column is an attempt (11 for a correct answer, 00 otherwise).

RA=[00000100 10001100 11000010 01100001 00110010 00011001 11111111]R_A= \begin{bmatrix} 0&0&0&0&0&1&0&0\\\ 1&0&0&0&1&1&0&0\\\ 1&1&0&0&0&0&1&0\\\ 0&1&1&0&0&0&0&1\\\ 0&0&1&1&0&0&1&0\\\ 0&0&0&1&1&0&0&1\\\ 1&1&1&1&1&1&1&1 \end{bmatrix} RB=[01000010 00010100 00100010 00001010 10100101 01011101 11111001]R_B= \begin{bmatrix} 0&1&0&0&0&0&1&0\\\ 0&0&0&1&0&1&0&0\\\ 0&0&1&0&0&0&1&0\\\ 0&0&0&0&1&0&1&0\\\ 1&0&1&0&0&1&0&1\\\ 0&1&0&1&1&1&0&1\\\ 1&1&1&1&1&0&0&1 \end{bmatrix}

Let's first evaluate the models using pass@k\mathrm{pass}@k:

kk12345678
pass@k\mathrm{pass}@k (A)0.42860.63780.78320.87760.93370.96430.98211.0000
pass@k\mathrm{pass}@k (B)0.41070.64290.78320.87550.93880.97961.00001.0000
WinnerABTieABBBTie

As you can see, A ranks higher at k=1k=1, B at k=2k=2, and they tie at k=3k=3. A leads again at k=4k=4, B at k=5,6,7k=5,6,7, and they tie at k=8k=8. This is one of the motivations behind our proposal.

Now, let's use U1/2(k)U_{1/2}(k), the TailPass utility with λ=1/2\lambda=1/2 (Section 3.1, Line 233):

kk12345678
U1/2(k)U_{1/2}(k) (A)0.42860.51520.56370.59230.60920.61920.62550.6308
U1/2(k)U_{1/2}(k) (B)0.41070.50690.55470.58300.60130.61310.62020.6234
WinnerAAAAAAAA

U1/2(k)U_{1/2}(k) ranks A higher at every budget. When model scores are close, TailPass's uncertainty estimates also tell us how confident we can be in distinguishing the models (Section 3.2).

In summary, TailPass has the following properties:

  • Quantifies uncertainty in success distributions and utility scores (Section 3.2).
  • Offers tunable utilities that reflect the value of additional correct responses (Section 3.1).
  • Incorporates prior evidence (Section 3.2).
  • Supports partial credit and graded evaluation through categorical rubrics (Section 3.3).
  • Distinguishes success-count distributions that accuracy and existing methods cannot separate (Table 5).

We invite Reviewers to assess the paper on its scientific merits. We welcome any criticisms of the technical aspects and scholarly contribution of the work. We have learned that two papers have cited our anonymous submission. Although a paper under review may have limited visibility, we would also like to invite anyone in the community who comes across it to write a review and judge whether the proposed method makes a meaningful contribution to the field.

References:

  • [1] Hariri et al. Don't Pass@k. ICLR 2026.
  • [2] Chen et al. Evaluating Large Language Models Trained on Code.
  • [3] Yao et al. τ\tau-bench. ICLR 2025.
  • [4] Liu et al. Are Your LLMs Capable of Stable Reasoning? Findings of ACL 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.