- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper systematically examined the true bottlenecks of the Best-of-N reasoning extension in code generation. It was found that when all the signals were derived from the generation model itself rather than external annotations, the four mainstream self-source selection methods were almost unable to surpass an extremely simple baseline on LiveCodeBench and HumanEval+. This baseline simply used the sample examples provided by the question for test filtering, and even the oracle voting for perfect behavior clustering was no exception. The paper further proved through oracle decomposition that the re-sampling gain accounted for about 70-80% of it was concentrated on a few correct problems where only one or two pairs were selected. However, on these types of problems, the selection accuracy based on majority consistency was only about 30%, because the correct answers were in the minority. At the same time, the author verified that the input discrimination of the generation test was actually sufficient. The assertion pool after cleaning could bring a more than five-point improvement. What truly failed was the model-generated expected output. From this, the core conclusion was drawn that the self-source methods could distinguish candidate behaviors but could not independently determine which behavior was correct. The advantage of the sample filter lies not in the amount of information but in the external grounding. The subsequent improvement direction should be to build a mechanism that can independently verify correctness, rather than stacking more complex self-source selectors. All conclusions remained consistent under multiple robustness tests involving four approximately 4B models, cross-model assertion pools intersection, N=50 candidate pools amplification
Critical strengths
The author did not merely rely on the empirical conclusion that the effect of the source selector was poor. Instead, he constructed a behavioral grouping based on a complete grading suite, decoupling the two questions of "whether the candidate behaviors can be distinguished" and "which behavior is correct". This experimental design of splitting the black-box failure into verifiable mechanisms is the core contribution of this paper compared to ordinary benchmark evaluation articles.
Critical weaknesses
- The model size is seriously insufficient, and the extrapolation of the conclusion is questionable. All the experiments are based on four models with approximately 4B parameters, but the title and conclusion of the paper claim a universal judgment of "limitations of self-source code selection". The assertion ability of the small model, the reliability of self-evaluation, and the quality of behavioral consistency are inherently weak. A natural counterexample is: when the model is large enough to generate correct assertion expected outputs and accurately judge whether the code conforms to the intention, will self-source selection become effective? The paper itself admits this in the "Limitations" section, but the narrative intensity of the main text far exceeds the scope that the evidence can support.
- The strength of the example filter baseline is highly coupled with the selected benchmark, and the comparison is unfair. The topic of LiveCodeBench naturally carries rich and high-quality sample input and output (this is a characteristic of competition questions), and the paper regards this "built-in question bank" baseline as an external grounded gold standard, but this condition is not possessed by all code tasks. For scenarios such as function completion, bug fixing, refactoring, and API design, there are often no ready-made worked examples, and example filter is simply not applicable. More importantly, on HumanEval+, only a few correct questions are 11–33, and the statistical power is extremely weak. The paper itself also says that it can only be considered as auxiliary evidence. The core comparison is based on a benchmark that is inherently disadvantageous to the self-source method, and the applicable boundary of the conclusion is seriously exaggerated.
- The "self-sourced" category is too broad. The paper packages three methods with completely different failure mechanisms - behavior clustering (plain agreement), assertion scoring generation (CodeT/SRank), and model self-evaluation (self-judge) - into one large category and uniformly declares "self-source selection fails". However, the problems of CodeT are incorrect expected outputs, the problem of plain agreement is the failure of majority voting when it is in the minority, and the problem of self-judge is unreliable subjective judgment. These three have completely different improvement directions, and packaging them together leads to an overly generalized conclusion. At the same time, the paper only tested four relatively basic selectors and did not cover more complex self-source schemes (such as iterative selection combined with execution feedback, multi-round test generation and screening, and mixed evaluation combining code understanding and execution results), and the core discovery of "example filter defeats all self-source methods" may be partially due to weak sampling, rather than the limitations of the paradigm itself.
Questions for authors
- All the conclusions of the paper are based on a model with approximately 4B parameters, but the title and conclusion present it as questioning the universal nature of the "self-source code selection" paradigm. Does the author have any plans or preliminary evidence indicating that the expected output reliability, self-evaluation accuracy, and behavioral consistency of the generated assertions will significantly change on models of 30B, 70B, or even larger scales? If the performance of self-source selection on large models significantly improves, will the paper limit the conclusion to "self-source selection fails in small model scenarios" instead of a broader paradigm criticism?
- The effectiveness of the Example Filter is highly dependent on the worked examples provided in the LiveCodeBench questions, which is a special design for competition-type questions. Does the author have verified this baseline in code tasks without existing sample input and output code (such as function completion, bug fixing, code refactoring)? If the example filter cannot be constructed in such tasks, how should the core comparison (self-source method vs. external grounding baseline) be rephrased in the paper? Additionally, on HumanEval+, only 11-33 of the questions are correct, and the statistical power is insufficient. Why not supplement a benchmark with a larger sample size and different question examples to support the universality of the conclusion?
- The paper unifies three distinct failure mechanisms of methods such as behavior clustering, assertion scoring, and model self-evaluation under the category of "self-sourced" and draws a paradigmatic conclusion. However, the problems of CodeT are incorrect expected output, plain agreement's problem of failure when the majority vote is in the minority, and self-judge's problem of unreliable subjective judgment. The improvement paths for these are completely different. Why did the author discuss them together? Has the author tested more complex self-source schemes (such as iterative selection combined with execution feedback, multi-round test generation and screening, and mixed evaluation combining code semantic understanding and execution results)? If these stronger self-source methods can approach or even exceed the example filter, should the core narrative of the paper be revised from "the self-source paradigm fails" to "current self-source methods have specific repairable flaws"?