acceptodds
Withdrawn from ICLR 2027

Where Best-of-\(N\) Gains Are Lost: The Limits of Self-Sourced Code Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sampling multiple candidate programs is a standard way to scale inference-time performance for code generation. Although increasing the sample count makes it more likely that a correct program is present, realizing this gain requires reliably identifying a correct one. We study this problem when selection relies on signals available from the generator itself, including candidate agreement, self-generated tests, and self-judgment. We find a striking mismatch between where resampling creates opportunity and where these signals are effective: across four models on LiveCodeBench, 78–86% of the gap between single-sample and Best-of- performance is concentrated on minority-correct problems, where only 1–5 of 10 candidates are correct. This is precisely the regime in which self-sourced selectors are weakest, and none significantly outperforms a simple filter that executes the worked examples provided in the problem statement. We isolate two mechanisms behind this shortfall. First, even an oracle behavioral partition leaves plurality selection weak: identifying behavioral classes does not identify the correct class when correct behavior is a minority. Second, generated tests are often discriminating but mislabeled; validating their expected outputs raises a simple test filter 5.7–11.9 points above the example filter on minority-correct problems. Together, these results show that producing additional distinctions among candidates is not enough: the central challenge is grounding those distinctions in correctness. Effective inference-time scaling through resampling therefore requires selection evidence whose connection to correctness can be independently grounded.

Primary area:
foundation or frontier models, including LLMs
Keywords:
code generation, best-of-N sampling, inference-time scaling, candidate selection, verification, self-generated tests, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2SgmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically examined the true bottlenecks of the Best-of-N reasoning extension in code generation. It was found that when all the signals were derived from the generation model itself rather than external annotations, the four mainstream self-source selection methods were almost unable to surpass an extremely simple baseline on LiveCodeBench and HumanEval+. This baseline simply used the sample examples provided by the question for test filtering, and even the oracle voting for perfect behavior clustering was no exception. The paper further proved through oracle decomposition that the re-sampling gain accounted for about 70-80% of it was concentrated on a few correct problems where only one or two pairs were selected. However, on these types of problems, the selection accuracy based on majority consistency was only about 30%, because the correct answers were in the minority. At the same time, the author verified that the input discrimination of the generation test was actually sufficient. The assertion pool after cleaning could bring a more than five-point improvement. What truly failed was the model-generated expected output. From this, the core conclusion was drawn that the self-source methods could distinguish candidate behaviors but could not independently determine which behavior was correct. The advantage of the sample filter lies not in the amount of information but in the external grounding. The subsequent improvement direction should be to build a mechanism that can independently verify correctness, rather than stacking more complex self-source selectors. All conclusions remained consistent under multiple robustness tests involving four approximately 4B models, cross-model assertion pools intersection, N=50 candidate pools amplification

Critical strengths

The author did not merely rely on the empirical conclusion that the effect of the source selector was poor. Instead, he constructed a behavioral grouping based on a complete grading suite, decoupling the two questions of "whether the candidate behaviors can be distinguished" and "which behavior is correct". This experimental design of splitting the black-box failure into verifiable mechanisms is the core contribution of this paper compared to ordinary benchmark evaluation articles.

Critical weaknesses
  1. The model size is seriously insufficient, and the extrapolation of the conclusion is questionable. All the experiments are based on four models with approximately 4B parameters, but the title and conclusion of the paper claim a universal judgment of "limitations of self-source code selection". The assertion ability of the small model, the reliability of self-evaluation, and the quality of behavioral consistency are inherently weak. A natural counterexample is: when the model is large enough to generate correct assertion expected outputs and accurately judge whether the code conforms to the intention, will self-source selection become effective? The paper itself admits this in the "Limitations" section, but the narrative intensity of the main text far exceeds the scope that the evidence can support.
  2. The strength of the example filter baseline is highly coupled with the selected benchmark, and the comparison is unfair. The topic of LiveCodeBench naturally carries rich and high-quality sample input and output (this is a characteristic of competition questions), and the paper regards this "built-in question bank" baseline as an external grounded gold standard, but this condition is not possessed by all code tasks. For scenarios such as function completion, bug fixing, refactoring, and API design, there are often no ready-made worked examples, and example filter is simply not applicable. More importantly, on HumanEval+, only a few correct questions are 11–33, and the statistical power is extremely weak. The paper itself also says that it can only be considered as auxiliary evidence. The core comparison is based on a benchmark that is inherently disadvantageous to the self-source method, and the applicable boundary of the conclusion is seriously exaggerated.
  3. The "self-sourced" category is too broad. The paper packages three methods with completely different failure mechanisms - behavior clustering (plain agreement), assertion scoring generation (CodeT/SRank), and model self-evaluation (self-judge) - into one large category and uniformly declares "self-source selection fails". However, the problems of CodeT are incorrect expected outputs, the problem of plain agreement is the failure of majority voting when it is in the minority, and the problem of self-judge is unreliable subjective judgment. These three have completely different improvement directions, and packaging them together leads to an overly generalized conclusion. At the same time, the paper only tested four relatively basic selectors and did not cover more complex self-source schemes (such as iterative selection combined with execution feedback, multi-round test generation and screening, and mixed evaluation combining code understanding and execution results), and the core discovery of "example filter defeats all self-source methods" may be partially due to weak sampling, rather than the limitations of the paradigm itself.
Questions for authors
  1. All the conclusions of the paper are based on a model with approximately 4B parameters, but the title and conclusion present it as questioning the universal nature of the "self-source code selection" paradigm. Does the author have any plans or preliminary evidence indicating that the expected output reliability, self-evaluation accuracy, and behavioral consistency of the generated assertions will significantly change on models of 30B, 70B, or even larger scales? If the performance of self-source selection on large models significantly improves, will the paper limit the conclusion to "self-source selection fails in small model scenarios" instead of a broader paradigm criticism?
  2. The effectiveness of the Example Filter is highly dependent on the worked examples provided in the LiveCodeBench questions, which is a special design for competition-type questions. Does the author have verified this baseline in code tasks without existing sample input and output code (such as function completion, bug fixing, code refactoring)? If the example filter cannot be constructed in such tasks, how should the core comparison (self-source method vs. external grounding baseline) be rephrased in the paper? Additionally, on HumanEval+, only 11-33 of the questions are correct, and the statistical power is insufficient. Why not supplement a benchmark with a larger sample size and different question examples to support the universality of the conclusion?
  3. The paper unifies three distinct failure mechanisms of methods such as behavior clustering, assertion scoring, and model self-evaluation under the category of "self-sourced" and draws a paradigmatic conclusion. However, the problems of CodeT are incorrect expected output, plain agreement's problem of failure when the majority vote is in the minority, and self-judge's problem of unreliable subjective judgment. The improvement paths for these are completely different. Why did the author discuss them together? Has the author tested more complex self-source schemes (such as iterative selection combined with execution feedback, multi-round test generation and screening, and mixed evaluation combining code semantic understanding and execution results)? If these stronger self-source methods can approach or even exceed the example filter, should the core narrative of the paper be revised from "the self-source paradigm fails" to "current self-source methods have specific repairable flaws"?
Reviewer_RTv7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Best-of-N code selection when the selection signals come only from the generator itself (candidate agreement, generated tests, and self-judgment), using four ~4B models on LiveCodeBench and HumanEval+. It reports that no self-sourced selector significantly outperforms a filter that executes the examples in the problem statement, and that 78–86% of the resampling gap lies in minority-correct problems (1–5 of 10 candidates correct), where agreement-based selection is weakest. Diagnostic analyses with an oracle behavioral partition and with generated assertions validated against known-correct programs attribute the shortfall to a lack of grounded correctness information rather than to insufficient discrimination among candidates.

Critical strengths

(S1) Thorough statistical reporting. Nearly all comparisons come with paired bootstrap intervals, and the main conclusions are checked with leave-one-model-out difficulty, a regrade on hidden tests only, and an extension to N = 50.

Critical weaknesses

(W1) Limited novelty. The three main findings largely restate known results.

(W1-1)

Self-sourced selectors rarely outperform a simple example filter. (Sec. 1)

This is already known: AlphaCode filters candidates on example tests before clustering [1], and the paper itself cites DOCE for the same observation [2].

(W1-2)

We find a striking mismatch between where resampling creates opportunity and where these signals are effective: across four models on LiveCodeBench, 78–86% of the gap between single-sample and Best-of-N performance is concentrated on minority-correct problems, where only 1–5 of 10 candidates are correct. (Abstract)

Agreement is weakest where selection is needed most. (Sec. 1)

The concentration follows largely from the gap definition: a problem with c correct samples contributes 1 − c/10 to the gap, so the reported bin shares alone, assuming c is uniform within each bin, already place about 75% of the gap in the 1–5 bin (e.g., 74.8% vs. 78.6% reported for Qwen3-4B). The weakness of agreement in this regime is likewise expected by construction, since plurality cannot select the correct class at c = 1, and prior work has shown that practical selectors fall well short of pass@N [3, 4].

(W1-3)

Generated tests contain useful inputs but unreliable expected outputs. (Sec. 1)

Prior work has already reported this as a trade-off between error-revealing inputs and output accuracy [5], which the paper does not cite, and input-only clustering such as AlphaCode's [1] exists precisely because generated outputs are unreliable. The validated-assertion experiment quantifies this issue, but it relies on known-correct programs and so does not suggest an actionable remedy.

Overall, the paper mainly re-measures known phenomena at the 4B scale rather than offering new insight or methods.

[1] Y. Li et al. Competition-Level Code Generation with AlphaCode. Science, 2022.

[2] H.-S. Li et al. DOCE: Finding the Sweet Spot for Execution-Based Code Generation. arXiv:2408.13745, 2024.

[3] B. Brown et al. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787, 2024.

[4] B. Stroebl, S. Kapoor, and A. Narayanan. The Limits of Inference Scaling Through Resampling. ICLR 2026.

[5] A. Prasad et al. Learning to Generate Unit Tests for Automated Debugging. COLM 2025.

Questions for authors

Beyond the findings already established in prior work [1–5], what are the paper's unique takeaways, and how should they change how practitioners design Best-of-N selection?

Reviewer_46JiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies some limitations of using a self-sourced selector in Best-of-NN sampling for code generation tasks. To perform quantitative analysis, this paper introduces resampling gap, that is the difference between pass@NN and pass@1. Then, this paper identifies the cases where Best-of-NN sampling does not perform well. More specifically, this paper finds three main findings: (1) limited performance gain of self-sourced selectors, (2) minority-correct problems in aggreement, and (3) incorrect assertion generation.

Critical strengths
  • S1. This paper identifies some limitations of Best-of-NN for code generation tasks by performing comprehensive experiments.
Critical weaknesses
  • W1. This paper is a basically analysis paper. Therefore, its technical contribution may be limited compared to other algorithm papers.

  • W2. The main findings from the analysis seem somewhat limited, since this paper mainly focuses on the cases where self-sourced selector are used in Best-of-NN sampling. In other words, trained selectors are not considered in the analysis. Therefore, the analysis results may not reflect the performance results of Best-of-NN sampling.

Questions for authors
  • Q1. Is there any suggestions that may address the limitations of the self-sourced selector in Best-of-NN?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.