- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of LLM-generated code reranking when both candidate codes and LLM-generated tests may not be reliable. The authors propose viewing the expected outputs of generated tests as an implicit (n+1)-th candidate program, thereby unifying test-based verification and cross-code consensus. Building on this perspective, they introduce N+1 Propagation Consistency (N+PC), which combines multiple agreement views through graph Laplacian propagation with task-adaptive weighting. Experiments on HumanEval, HumanEval+, and MBPP demonstrate consistent improvements over existing execution-based reranking methods.
Critical strengths
The idea of considering the LLM-generated test cases as potential unstated codes is interesting. This perspective unifies many existing methods into a unified framework.
Critical weaknesses
The experimental part is not comprehensive or convincing enough. The authors only evaluate and compare methods on MBPP and HumanEval(+), which are often considered easy benchmarks. The authors are expected to conduct experiments on one or two more challenging benchmarks, such as LiveCodeBench and APPS. The authors only apply all of the discussed methods on GPT3.5-Turbo. Experiments on more recent proprietary models are expected to show whether the proposed methods are still effective. Besides, there is an important baseline B4, that is not compared in the experiments. B4 also discusses the unreliability of both code and test cases and tries to address this problem, which is closely related work.
Questions for authors
Please see my comments in the weakness section.
3 comments
[Response 2/2] Comparison with B4
W1.2: How does N+PC compare with B4, given that both address unreliable programs and tests?
We compared the authors’ B4 implementation with our submitted N+PC model (with ACES-O as its inner scorer) on the same candidate programs and generated tests. B4 uses assertion pass/fail outcomes to estimate the reliability of programs and tests. N+PC also uses agreement between programs that return the same value even when both fail an assertion.
The table reports Pass@1, Pass@2, and Pass@5 averaged equally across HumanEval, HumanEval+, and MBPP. Following B4 (Chen et al., 2024a, Table 1), we classify a problem as hard if at least one but fewer than half of its candidate programs pass the benchmark tests. Adding N+PC to B4 improves the mean at all three cutoffs, both overall and on hard problems.
| Problem subset | Method | Pass@1 (%) | Pass@2 (%) | Pass@5 (%) |
|---|---|---|---|---|
| All problems | B4 | 75.59 | 76.20 | 76.81 |
| N+PC with B4 | 76.42 | 76.62 | 78.05 | |
| N+PC with ACES-O | 78.25 | 79.75 | 80.72 | |
| Hard problems | B4 | 39.24 | 41.24 | 43.40 |
| N+PC with B4 | 40.88 | 41.72 | 46.72 | |
| N+PC with ACES-O | 53.44 | 57.27 | 59.27 |
I thank the authors for their responses. The rebuttal is good, and I will raise my score.
[Response 1/2] Harder Benchmarks and Newer Generators
W1.1: Does N+PC improve reranking on harder benchmarks than MBPP and HumanEval(+)? Do the gains extend to other candidate generators, including proprietary models newer than GPT-3.5-Turbo?
We extended the evaluation to LeetCodeDataset, LiveCodeBench (511 problems), and the full CodeContests test split (165 problems). On LiveCodeBench with Qwen2.5-Coder-7B, adding N+PC raised ACES-O Pass@1 from 27.79% to 34.83%. On CodeContests, adding N+PC raised ACES-O Pass@1 from 69.70% to 70.30% with GPT-5.6 Luna and from 21.21% to 26.06% with GPT-5.4 mini.
Table 2 (revised manuscript). Contest benchmarks. Each cell reports Pass@1 / Pass@2 (%).
On LeetCodeDataset, adding N+PC also improved Pass@1 and Pass@2 for all three inner scorers on each generator’s post-release subset.
Table 4 (revised manuscript). LeetCodeDataset problems published after each generator’s release (estimated dates). Each cell reports Pass@1 / Pass@2 / Pass@5 (%). The 7B column uses the same rankings as Table 2, restricted to the 185 eligible problems.
Table 5 extends the generator comparison to HumanEval and HumanEval+ using the same candidate pools under both test suites. Across all 16 benchmark–generator pairs, adding N+PC to ACES-O raised mean Pass@1 from 58.43% to 61.78%, a gain of 3.35 percentage points (conservative 95% CI [1.43, 5.30], two-sided paired permutation test, p<0.001985).
Table 5 (revised manuscript). HumanEval and HumanEval+ use the same candidate programs and generated tests for each generator. Each cell reports Pass@1 / Pass@2 / Pass@5 (%).