acceptodds
Under review as a conference paper at ICLR 2027

Tests Reveal an Unstated Code: (N+1) Propagation Consistency for LLM Code Generation

Abstract

Selecting LLM-generated code without ground-truth tests relies on LLM-generated test inputs and expected outputs. Yet whether the LLM-generated tests' expected outputs or the candidate codes are the more reliable signal varies sharply across tasks. Existing methods either trust the LLM-generated tests' expected outputs as ground truth or discard them in favor of cross-code consensus. To our knowledge, no prior framework unifies these two stances as endpoints of a principled, task-adaptive continuum; existing combinations remain heuristic. Our unifying insight: the LLM-generated tests' expected outputs are the input-output behavior of an unstated -th candidate code implicit in the LLM's solution distribution, on equal footing with the explicit ones. Each of the codes defines an agreement view, and we identify codes that score consistently across all views. We propose -Propagation Consistency (), which couples Laplacian propagations via a task-adaptive scalar weighting the unstated-code view against the rest. At its two endpoints, reduces to each prior stance, making a principled unification rather than a heuristic mixture. consistently improves every inner baseline and advances the state-of-the-art Pass@ among execution-only rerankers as a plug-and-play framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
code generation, large language models, test-time scaling, self-consistency, graph-based ranking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pFkvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge of LLM-generated code reranking when both candidate codes and LLM-generated tests may not be reliable. The authors propose viewing the expected outputs of generated tests as an implicit (n+1)-th candidate program, thereby unifying test-based verification and cross-code consensus. Building on this perspective, they introduce N+1 Propagation Consistency (N+PC), which combines multiple agreement views through graph Laplacian propagation with task-adaptive weighting. Experiments on HumanEval, HumanEval+, and MBPP demonstrate consistent improvements over existing execution-based reranking methods.

Critical strengths

The idea of considering the LLM-generated test cases as potential unstated codes is interesting. This perspective unifies many existing methods into a unified framework.

Critical weaknesses

The experimental part is not comprehensive or convincing enough. The authors only evaluate and compare methods on MBPP and HumanEval(+), which are often considered easy benchmarks. The authors are expected to conduct experiments on one or two more challenging benchmarks, such as LiveCodeBench and APPS. The authors only apply all of the discussed methods on GPT3.5-Turbo. Experiments on more recent proprietary models are expected to show whether the proposed methods are still effective. Besides, there is an important baseline B4, that is not compared in the experiments. B4 also discusses the unreliability of both code and test cases and tries to address this problem, which is closely related work.

Questions for authors

Please see my comments in the weakness section.

3 comments
AuthorsOpenReview ↗

[Response 1/2] Harder Benchmarks and Newer Generators

W1.1: Does N+PC improve reranking on harder benchmarks than MBPP and HumanEval(+)? Do the gains extend to other candidate generators, including proprietary models newer than GPT-3.5-Turbo?

We extended the evaluation to LeetCodeDataset, LiveCodeBench (511 problems), and the full CodeContests test split (165 problems). On LiveCodeBench with Qwen2.5-Coder-7B, adding N+PC raised ACES-O Pass@1 from 27.79% to 34.83%. On CodeContests, adding N+PC raised ACES-O Pass@1 from 69.70% to 70.30% with GPT-5.6 Luna and from 21.21% to 26.06% with GPT-5.4 mini.

Table 2 (revised manuscript). Contest benchmarks. Each cell reports Pass@1 / Pass@2 (%).

ScorerLeetCodeDataset · Qwen2.5-Coder-7BLiveCodeBench · Qwen2.5-Coder-7BCodeContests · GPT-5.4 miniCodeContests · DeepSeek V4 FlashCodeContests · GPT-5.6 Luna
MV14.04 / 15.7927.98 / 35.0320.61 / 25.4560.00 / 63.0370.91 / 75.76
MV + N+PC22.37 / 22.3734.64 / 37.9626.67 / 29.0961.21 / 64.2471.52 / 75.76
ACES-C17.54 / 18.8628.38 / 35.6221.21 / 25.4560.61 / 63.6470.91 / 75.15
ACES-C + N+PC21.49 / 21.4934.25 / 37.5726.67 / 28.4861.21 / 63.6470.91 / 75.15
ACES-O17.54 / 18.4227.79 / 34.6421.21 / 24.2460.00 / 62.4269.70 / 75.15
ACES-O + N+PC19.30 / 20.6134.83 / 37.7726.06 / 28.4860.61 / 63.0370.30 / 74.55

On LeetCodeDataset, adding N+PC also improved Pass@1 and Pass@2 for all three inner scorers on each generator’s post-release subset.

Table 4 (revised manuscript). LeetCodeDataset problems published after each generator’s release (estimated dates). Each cell reports Pass@1 / Pass@2 / Pass@5 (%). The 7B column uses the same rankings as Table 2, restricted to the 185 eligible problems.

ScorerQwen2.5-Coder-7BQwen2.5-Coder-14BDeepSeek-Coder-V2-16B
MV12.97 / 15.14 / 18.9221.54 / 24.62 / 29.2317.11 / 20.18 / 24.12
MV + N+PC21.62 / 21.62 / 21.6226.92 / 28.46 / 30.0025.44 / 26.32 / 27.19
ACES-C15.14 / 16.76 / 17.8426.92 / 29.23 / 31.5425.00 / 25.88 / 26.75
ACES-C + N+PC20.54 / 20.54 / 21.0830.00 / 31.54 / 33.0826.32 / 26.32 / 27.19
ACES-O16.76 / 17.84 / 18.9221.54 / 25.38 / 30.0021.49 / 22.37 / 24.12
ACES-O + N+PC18.92 / 20.54 / 21.0823.85 / 26.15 / 27.6925.44 / 26.75 / 27.63

Table 5 extends the generator comparison to HumanEval and HumanEval+ using the same candidate pools under both test suites. Across all 16 benchmark–generator pairs, adding N+PC to ACES-O raised mean Pass@1 from 58.43% to 61.78%, a gain of 3.35 percentage points (conservative 95% CI [1.43, 5.30], two-sided paired permutation test, p<0.001985p < 0.001985).

Table 5 (revised manuscript). HumanEval and HumanEval+ use the same candidate programs and generated tests for each generator. Each cell reports Pass@1 / Pass@2 / Pass@5 (%).

GeneratorScorerHumanEvalHumanEval+
Qwen2.5-Coder-7BMV89.02 / 90.24 / 90.2480.49 / 84.15 / 84.15
MV + N+PC91.46 / 91.46 / 91.4682.93 / 84.15 / 85.37
ACES-C89.63 / 90.24 / 90.2481.71 / 84.15 / 84.76
ACES-C + N+PC90.85 / 90.85 / 90.8583.54 / 84.15 / 85.37
ACES-O90.85 / 91.46 / 91.4683.54 / 85.37 / 85.37
ACES-O + N+PC92.07 / 92.07 / 92.6885.37 / 85.98 / 86.59
DeepSeek-Coder-V2-16BMV84.76 / 85.98 / 86.5976.83 / 77.44 / 78.05
MV + N+PC86.59 / 86.59 / 87.2078.66 / 79.27 / 79.27
ACES-C87.20 / 87.20 / 87.2078.66 / 78.66 / 78.66
ACES-C + N+PC87.20 / 87.20 / 87.8078.66 / 78.66 / 79.27
ACES-O85.37 / 86.59 / 86.5978.05 / 78.05 / 78.05
ACES-O + N+PC85.37 / 85.98 / 85.9878.05 / 79.27 / 79.27
DeepSeek-Coder-6.7BMV64.63 / 70.12 / 77.4456.71 / 65.24 / 73.17
MV + N+PC79.27 / 80.49 / 81.1073.17 / 74.39 / 77.44
ACES-C70.73 / 73.78 / 78.0563.41 / 69.51 / 73.78
ACES-C + N+PC79.27 / 81.10 / 81.1071.34 / 75.00 / 78.05
ACES-O67.07 / 70.73 / 77.4459.76 / 65.85 / 73.17
ACES-O + N+PC78.66 / 79.27 / 81.7173.78 / 75.00 / 76.83
AuthorsOpenReview ↗

[Response 2/2] Comparison with B4

W1.2: How does N+PC compare with B4, given that both address unreliable programs and tests?

We compared the authors’ B4 implementation with our submitted N+PC model (with ACES-O as its inner scorer) on the same candidate programs and generated tests. B4 uses assertion pass/fail outcomes to estimate the reliability of programs and tests. N+PC also uses agreement between programs that return the same value even when both fail an assertion.

The table reports Pass@1, Pass@2, and Pass@5 averaged equally across HumanEval, HumanEval+, and MBPP. Following B4 (Chen et al., 2024a, Table 1), we classify a problem as hard if at least one but fewer than half of its candidate programs pass the benchmark tests. Adding N+PC to B4 improves the mean at all three cutoffs, both overall and on hard problems.

Problem subsetMethodPass@1 (%)Pass@2 (%)Pass@5 (%)
All problemsB475.5976.2076.81
N+PC with B476.4276.6278.05
N+PC with ACES-O78.2579.7580.72
Hard problemsB439.2441.2443.40
N+PC with B440.8841.7246.72
N+PC with ACES-O53.4457.2759.27

I thank the authors for their responses. The rebuttal is good, and I will raise my score.

Reviewer_cuosOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for generated program selection, from a pool of generated programs and tests. The method accounts for program outputs, as well as generated test answers, acknowledging the reliability of model generated tests and answers, which can be used to select a candidate program. This method defines an approach combining both test outputs and model generated test answers. By treating generated program outputs, plus the model generated answer as an additional candidate (n+1 sources), the method scores each source as a provisional answer compared to other candidates, creates an agreement graph based on the scores, and provides higher weighting for agreement. They select the candidate with the highest score. The paper provides ablations for different weighting and versions of agreement to provide study for their approach.

Critical strengths
  • Ablations are useful to determine which part of the method improves performance, exploring alternative graph constructions, fixed vs adaptive weighting, and different information sources.
Critical weaknesses
  • The paper uses a single set of generated programs and tests from GPT-3.5-Turbo, which are re-used from Enhancing large language models in coding through multi-perspective self-consistency (Huang et al., 2024). The method therefore isn’t given enough evidence to verify if it transfers to other models that have come out more recently, or in different pools of generated programs and tests. It would have been nice to see the method compared to the additional models they provide in Section 4.1. Improvements are marginal, and need stronger support to back claims. *The paper provides narrow benchmark coverage, with benchmarks that are old. HumanEval+ does include stricter testing, but it is hard to measure if the method defined in this paper would be relevant to recent models and benchmarks.
Questions for authors
  • Appendix C.5 reports a 19 × 20 hyper-parameter sweep, and Appendix B.5 states that the sensitivity results use all benchmark tasks. How were the reported dataset-specific parameters selected? Were they fixed before examining benchmark performance, or chosen using a separate validation set?
5 comments
AuthorsOpenReview ↗

[Response 1/2] Hyperparameter Selection and Validation

Q1: How were the reported dataset-specific parameters selected? Were they fixed before examining benchmark performance, or chosen using a separate validation set?

We fixed the hyperparameters before examining benchmark performance. Drawing on MPSC’s use of generated data for parameter selection (Huang et al., 2024, Appendix E), we used a separate model, DeepSeek-V3, to generate validation tests for each problem. GPT-3.5-Turbo generated both the candidate programs and the tests used for reranking. We selected λ\lambda and κmax⁡\kappa_{\max} by five-fold cross-validation over problems. In each round, we compared settings on four folds by checking the top-ranked programs’ outputs against the expected outputs of the validation tests. Only after fixing the selected setting did we use the hidden tests of the held-out fold to measure Pass@kk (§4.1 and Appendix B.2).

We did not use performance on the evaluation sets to choose the hyperparameters reported in Table 1. For example, with ACES-O as the inner scorer, the reported setting yields 84.76% Pass@1 on HumanEval, 0.61 percentage points below the best score in the Figure 8 sweep (85.37%). On HumanEval+, the same setting yields 76.22%, 1.22 points below the best score (77.44%). On MBPP, N+PC outperforms the strongest execution-only baseline in Table 1 (ACES-O, 72.37% Pass@1) at 307 of the 380 settings (80.8%). The gain on MBPP thus persists across a broad range of hyperparameter settings.

AuthorsOpenReview ↗

[Response 2/2] Generalization Across Generators and Benchmarks

W1: How does N+PC perform with other generators, on different pools of generated programs and tests, and on newer benchmarks? What further evidence supports the reported gains?

We evaluated N+PC on programs generated by Qwen2.5-Coder-7B, Qwen2.5-Coder-14B, DeepSeek-Coder-V2-16B, DeepSeek-Coder-6.7B, GPT-5.4 mini, DeepSeek V4 Flash, and GPT-5.6 Luna. The tables cover HumanEval, HumanEval+, and three additional benchmarks (LeetCodeDataset, LiveCodeBench, and CodeContests). Within each pool, each inner scorer and its N+PC variant ranked the same programs using the same generated tests.

Table 2 (revised manuscript). Contest benchmarks. Each cell reports Pass@1 / Pass@2 (%).

ScorerLeetCodeDataset · Qwen2.5-Coder-7BLiveCodeBench · Qwen2.5-Coder-7BCodeContests · GPT-5.4 miniCodeContests · DeepSeek V4 FlashCodeContests · GPT-5.6 Luna
MV14.04 / 15.7927.98 / 35.0320.61 / 25.4560.00 / 63.0370.91 / 75.76
MV + N+PC22.37 / 22.3734.64 / 37.9626.67 / 29.0961.21 / 64.2471.52 / 75.76
ACES-C17.54 / 18.8628.38 / 35.6221.21 / 25.4560.61 / 63.6470.91 / 75.15
ACES-C + N+PC21.49 / 21.4934.25 / 37.5726.67 / 28.4861.21 / 63.6470.91 / 75.15
ACES-O17.54 / 18.4227.79 / 34.6421.21 / 24.2460.00 / 62.4269.70 / 75.15
ACES-O + N+PC19.30 / 20.6134.83 / 37.7726.06 / 28.4860.61 / 63.0370.30 / 74.55

To test transfer to newer problems, we restricted LeetCodeDataset to problems whose estimated publication dates follow each generator’s release. On these subsets, adding N+PC improved both Pass@1 and Pass@2 for all three inner scorers with all three generators.

Table 4 (revised manuscript). LeetCodeDataset problems published after each generator’s release (estimated dates). Each cell reports Pass@1 / Pass@2 / Pass@5 (%). The 7B column uses the same rankings as Table 2, restricted to the 185 eligible problems.

ScorerQwen2.5-Coder-7BQwen2.5-Coder-14BDeepSeek-Coder-V2-16B
MV12.97 / 15.14 / 18.9221.54 / 24.62 / 29.2317.11 / 20.18 / 24.12
MV + N+PC21.62 / 21.62 / 21.6226.92 / 28.46 / 30.0025.44 / 26.32 / 27.19
ACES-C15.14 / 16.76 / 17.8426.92 / 29.23 / 31.5425.00 / 25.88 / 26.75
ACES-C + N+PC20.54 / 20.54 / 21.0830.00 / 31.54 / 33.0826.32 / 26.32 / 27.19
ACES-O16.76 / 17.84 / 18.9221.54 / 25.38 / 30.0021.49 / 22.37 / 24.12
ACES-O + N+PC18.92 / 20.54 / 21.0823.85 / 26.15 / 27.6925.44 / 26.75 / 27.63

The original comparison included direct inference with DeepSeek-Coder. Table 5 evaluates reranking with DeepSeek-Coder-6.7B. Adding N+PC to ACES-O raised Pass@1 from 67.07% to 78.66% on HumanEval and from 59.76% to 73.78% on HumanEval+. Across all 16 benchmark–generator pairs, adding N+PC to ACES-O raised mean Pass@1 from 58.43% to 61.78%, a gain of 3.35 percentage points (conservative 95% CI [1.43, 5.30], two-sided paired permutation test, p<0.001985p < 0.001985).

Table 5 (revised manuscript). HumanEval and HumanEval+ use the same candidate programs and generated tests for each generator. Each cell reports Pass@1 / Pass@2 / Pass@5 (%).

GeneratorScorerHumanEvalHumanEval+
Qwen2.5-Coder-7BMV89.02 / 90.24 / 90.2480.49 / 84.15 / 84.15
MV + N+PC91.46 / 91.46 / 91.4682.93 / 84.15 / 85.37
ACES-C89.63 / 90.24 / 90.2481.71 / 84.15 / 84.76
ACES-C + N+PC90.85 / 90.85 / 90.8583.54 / 84.15 / 85.37
ACES-O90.85 / 91.46 / 91.4683.54 / 85.37 / 85.37
ACES-O + N+PC92.07 / 92.07 / 92.6885.37 / 85.98 / 86.59
DeepSeek-Coder-V2-16BMV84.76 / 85.98 / 86.5976.83 / 77.44 / 78.05
MV + N+PC86.59 / 86.59 / 87.2078.66 / 79.27 / 79.27
ACES-C87.20 / 87.20 / 87.2078.66 / 78.66 / 78.66
ACES-C + N+PC87.20 / 87.20 / 87.8078.66 / 78.66 / 79.27
ACES-O85.37 / 86.59 / 86.5978.05 / 78.05 / 78.05
ACES-O + N+PC85.37 / 85.98 / 85.9878.05 / 79.27 / 79.27
DeepSeek-Coder-6.7BMV64.63 / 70.12 / 77.4456.71 / 65.24 / 73.17
MV + N+PC79.27 / 80.49 / 81.1073.17 / 74.39 / 77.44
ACES-C70.73 / 73.78 / 78.0563.41 / 69.51 / 73.78
ACES-C + N+PC79.27 / 81.10 / 81.1071.34 / 75.00 / 78.05
ACES-O67.07 / 70.73 / 77.4459.76 / 65.85 / 73.17
ACES-O + N+PC78.66 / 79.27 / 81.7173.78 / 75.00 / 76.83

I thank the authors for a very thorough set of additional results and ablations based on the feedback provided. I am glad to hear care was taken care in selecting the hyper-parameters, and no leakage happened.

The new results demonstrate varied levels of improvement with frontier models. Particularly looking at the CodeContest results, there are marginal improvements for certain models leading me to believe that N+PC has little benefit. I do note however, for GPT-5.4 Mini there seems to be meaningful improvement.

What may help is to see the oracle accuracy for the candidate pools; what is the upper limit to improvements for these new results?

For now, I have decided to keep my score unchanged.

Reviewer_fjh7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes N+PC, a reranker for LLM-generated code that treats the LLM-generated tests’ expected outputs as an unstated (n+1)-th candidate, rather than either trusting them as ground truth or discarding them in favor of cross-candidate agreement. Each of the n+1 codes defines an agreement view over the candidates. N+PC combines these views through a joint graph-Laplacian objective, with a task-adaptive weight on the expected-output view computed from pass density and cross-candidate disagreement. The paper shows that boundary settings of this weight recover existing test-trusting and consensus-only rerankers. On HumanEval, HumanEval+, and MBPP with a GPT-3.5-Turbo candidate pool, wrapping N+PC around MV, ACES-C, and ACES-O improves or matches each baseline on every reported Pass@k, with the largest gains on tasks where the generated tests are mostly wrong.

Critical strengths

Strengths:

  1. The diagnostic analysis clarifies when each signal helps. Figure 5's MV vs MBR-exec crossover, the stratification by test accuracy (Figures 3c and 7c), and the endpoint ablation show where generated tests help and where reranking headroom lies. These findings don't depend on the headline margins, which makes them the most durable contribution.

  2. The ablations are thorough and transparent. Each component of the adaptive weight and graph construction is ablated, and full hyperparameter heatmaps are reported. This makes it possible to verify which parts carry the gains, and it makes the remaining concerns checkable.

  3. The method is practical. It is training-free, closed-form, about 8 ms per task, and wraps any pass-matrix reranker. If the gains hold up, it is easy to adopt.

Critical weaknesses

Weakness: 1.The evidence for the gains isn’t established. The hyperparameters appear to be selected on the evaluation sets, and on MBPP the full sweep spans about 1 point, compared with a 1.4 gain over ACES-O. Everything comes from one GPT-3.5 pool that has likely seen these benchmarks, with no second generator. The ACES-O gains are 1, 3, and 6 problems, with no significance tests.

2.The novelty and conceptual claims are overstated. CodeT already combines both signals, which the paper’s own description of CodeT acknowledges. MPSC already does graph-based consistency ranking on the same pool, with no comparison. The “unstated code” framing amounts to appending one row, and the endpoint proposition holds by construction.

3.The motivation is flawed. Figure 1 compares accuracies rather than whether wrong expected outputs coincide with wrong candidates. The ρ justification also assumes independent errors across candidates from one model.

Questions for authors

1.How were λ and κmax selected? Was any held-out split used?

2.Which candidate serves as the “correct” reference in the test-accuracy diagnostic, and what happens when correct candidates disagree?

3.Do the gains survive if δ excludes failed cells and outputs are canonicalized (for example, 9 vs 9.0)?

4.How does the method compare with MPSC’s graph optimization on the same pool?

5.How does it compare with a simple adaptive-weighted mixture of MV and MBR-exec scores?

6.Where do the Group 3 and direct-inference numbers come from?

7.Does Pass@k at k>1 change with deduplicated candidates?

6 comments
AuthorsOpenReview ↗

[Response 1/4] Hyperparameter Selection and Sensitivity

Q1/W1.1: How were hyperparameters chosen, and was a held-out split used?

For the results in Table 1, we held out the benchmarks’ hidden tests during hyperparameter selection and used them only for final evaluation. Drawing on MPSC’s use of generated data for parameter selection (Huang et al., 2024, Appendix E), we used a separate model, DeepSeek-V3, to generate validation tests for each problem. GPT-3.5-Turbo generated both the candidate programs and the tests used for reranking. We selected λ\lambda and κmax⁡\kappa_{\max} by five-fold cross-validation over problems. In each round, we compared settings on four folds by checking the top-ranked programs’ outputs against the expected outputs of the validation tests. We then fixed the selected setting and measured Pass@kk on the held-out fold using its hidden tests.

We did not use performance on the evaluation sets to choose the hyperparameters reported in Table 1. In the sensitivity analysis in Figure 8, the stars mark those settings, and nearby settings give similar Pass@1 scores on HumanEval and HumanEval+. Tuning for the highest Pass@1 in each of those sweeps would have yielded higher scores than we report. For example, with ACES-O as the inner scorer, we report 84.76% Pass@1 on HumanEval, while the best of the 380 settings reaches 85.37%. On HumanEval+, we use the same setting because the reranker inputs are identical. We report 76.22% Pass@1 there, below the best score of 77.44%. The revision describes the selection procedure in §4.1 and Appendix B.2 and the sensitivity analysis in Appendix C.6.

W1.2: How sensitive is the MBPP gain to those settings?

On MBPP, the reported setting raises Pass@1 from 72.37% with ACES-O to 73.77% with N+PC (+1.40 percentage points). N+PC outperforms ACES-O at 307 of the 380 settings in Figure 8. In a contiguous region around the reported setting, spanning eight tested values of λ\lambda (70 to 1,000) and twelve of κmax⁡\kappa_{\max} (40 to 3,000), all 96 combinations outperform ACES-O, with Pass@1 ranging from 72.60% to 73.77%. Thus, N+PC consistently outperforms its inner baseline, across a broad range of hyperparameter settings on MBPP.

AuthorsOpenReview ↗

[Response 2/4] Generator Transfer, Statistical Significance, and Test Accuracy

W1.3: Given that GPT-3.5 may have seen the benchmarks, do the gains hold with other generators? Are the gains over ACES-O statistically significant?

We evaluated N+PC on programs generated by Qwen2.5-Coder-7B, Qwen2.5-Coder-14B, DeepSeek-Coder-V2-16B, DeepSeek-Coder-6.7B, GPT-5.4 mini, DeepSeek V4 Flash, and GPT-5.6 Luna. The tables cover HumanEval, HumanEval+, and three additional benchmarks (LeetCodeDataset, LiveCodeBench, and CodeContests).

Table 2 (revised manuscript). Contest benchmarks. Each cell reports Pass@1 / Pass@2 (%).

ScorerLeetCodeDataset · Qwen2.5-Coder-7BLiveCodeBench · Qwen2.5-Coder-7BCodeContests · GPT-5.4 miniCodeContests · DeepSeek V4 FlashCodeContests · GPT-5.6 Luna
MV14.04 / 15.7927.98 / 35.0320.61 / 25.4560.00 / 63.0370.91 / 75.76
MV + N+PC22.37 / 22.3734.64 / 37.9626.67 / 29.0961.21 / 64.2471.52 / 75.76
ACES-C17.54 / 18.8628.38 / 35.6221.21 / 25.4560.61 / 63.6470.91 / 75.15
ACES-C + N+PC21.49 / 21.4934.25 / 37.5726.67 / 28.4861.21 / 63.6470.91 / 75.15
ACES-O17.54 / 18.4227.79 / 34.6421.21 / 24.2460.00 / 62.4269.70 / 75.15
ACES-O + N+PC19.30 / 20.6134.83 / 37.7726.06 / 28.4860.61 / 63.0370.30 / 74.55

To check for possible benchmark exposure, we restricted LeetCodeDataset to problems whose estimated publication dates follow each generator’s release. Adding N+PC raised ACES-O Pass@1 from 16.76% to 18.92% on the 185 problems for Qwen2.5-Coder-7B, from 21.54% to 23.85% on the 130 for Qwen2.5-Coder-14B, and from 21.49% to 25.44% on all 228 for DeepSeek-Coder-V2-16B.

Table 4 (revised manuscript). LeetCodeDataset problems published after each generator’s release (estimated dates). Each cell reports Pass@1 / Pass@2 / Pass@5 (%). The 7B column uses the same rankings as Table 2, restricted to the 185 eligible problems.

ScorerQwen2.5-Coder-7BQwen2.5-Coder-14BDeepSeek-Coder-V2-16B
MV12.97 / 15.14 / 18.9221.54 / 24.62 / 29.2317.11 / 20.18 / 24.12
MV + N+PC21.62 / 21.62 / 21.6226.92 / 28.46 / 30.0025.44 / 26.32 / 27.19
ACES-C15.14 / 16.76 / 17.8426.92 / 29.23 / 31.5425.00 / 25.88 / 26.75
ACES-C + N+PC20.54 / 20.54 / 21.0830.00 / 31.54 / 33.0826.32 / 26.32 / 27.19
ACES-O16.76 / 17.84 / 18.9221.54 / 25.38 / 30.0021.49 / 22.37 / 24.12
ACES-O + N+PC18.92 / 20.54 / 21.0823.85 / 26.15 / 27.6925.44 / 26.75 / 27.63

Across all 16 benchmark–generator pairs, adding N+PC to ACES-O raised mean Pass@1 from 58.43% to 61.78%, a gain of 3.35 percentage points (conservative 95% CI [1.43, 5.30], two-sided paired permutation test, p<0.001985p < 0.001985).

Table 5 (revised manuscript). HumanEval and HumanEval+ use the same candidate programs and generated tests for each generator. Each cell reports Pass@1 / Pass@2 / Pass@5 (%).

GeneratorScorerHumanEvalHumanEval+
Qwen2.5-Coder-7BMV89.02 / 90.24 / 90.2480.49 / 84.15 / 84.15
MV + N+PC91.46 / 91.46 / 91.4682.93 / 84.15 / 85.37
ACES-C89.63 / 90.24 / 90.2481.71 / 84.15 / 84.76
ACES-C + N+PC90.85 / 90.85 / 90.8583.54 / 84.15 / 85.37
ACES-O90.85 / 91.46 / 91.4683.54 / 85.37 / 85.37
ACES-O + N+PC92.07 / 92.07 / 92.6885.37 / 85.98 / 86.59
DeepSeek-Coder-V2-16BMV84.76 / 85.98 / 86.5976.83 / 77.44 / 78.05
MV + N+PC86.59 / 86.59 / 87.2078.66 / 79.27 / 79.27
ACES-C87.20 / 87.20 / 87.2078.66 / 78.66 / 78.66
ACES-C + N+PC87.20 / 87.20 / 87.8078.66 / 78.66 / 79.27
ACES-O85.37 / 86.59 / 86.5978.05 / 78.05 / 78.05
ACES-O + N+PC85.37 / 85.98 / 85.9878.05 / 79.27 / 79.27
DeepSeek-Coder-6.7BMV64.63 / 70.12 / 77.4456.71 / 65.24 / 73.17
MV + N+PC79.27 / 80.49 / 81.1073.17 / 74.39 / 77.44
ACES-C70.73 / 73.78 / 78.0563.41 / 69.51 / 73.78
ACES-C + N+PC79.27 / 81.10 / 81.1071.34 / 75.00 / 78.05
ACES-O67.07 / 70.73 / 77.4459.76 / 65.85 / 73.17
ACES-O + N+PC78.66 / 79.27 / 81.7173.78 / 75.00 / 76.83

Q2: Which candidate program serves as the reference, and how are disagreements among correct candidates handled?

In the test-accuracy diagnostic shown in Figures 3c and 7c, we did not use any generated candidate program as a reference. We ran each generated test against the benchmark’s canonical_solution and marked it correct if its assertion passed. Disagreement among candidate programs therefore does not affect the label. We used these labels only to group problems by test accuracy for the figures. N+PC ranks candidates without access to the reference solution or these labels. Both revised captions now identify the reference solution.

AuthorsOpenReview ↗

[Response 3/4] Output Agreement and Comparisons with Prior Methods

Q3: Do the gains persist when failed cells are excluded from δ\delta and outputs are canonicalized?

On problems with complete execution traces, we excluded failed cells from δ\delta. N+PC still improved mean Pass@1 over the base scorers by 2.87 percentage points (p=0.0027p=0.0027, exact two-sided paired permutation test grouped by problem), compared with 3.18 points under the original rules on the same problems. We retain failed cells because counting only successful returns can overstate how much of the candidate pool supports a given output.

Base scorer → N+PC with failed cells excluded from δ\delta, Pass@1 (%)HumanEvalHumanEval+MBPP
MV80.25 → 83.3369.75 → 75.3168.40 → 74.06
ACES-C82.72 → 85.1971.60 → 75.3170.99 → 74.06
ACES-O83.95 → 83.3374.69 → 75.9372.17 → 73.82

The original rule compares repr strings (Appendix A.1), so it treats 9 and 9.0 as distinct outputs. Merging equal integer and floating-point outputs left Pass@1 unchanged in all nine benchmark–scorer combinations, both with the original δ\delta and with failed cells excluded.

Q4: How does N+PC compare with MPSC on the same candidate pool?

Table 1 of our original submission compares the MPSC results reported by Huang et al. (2024) with our own N+PC results. As described in Section 4.1, we ran all our main reranking experiments on the same GPT-3.5-Turbo candidate programs released with MPSC for each benchmark. With each of the three inner scorers, N+PC achieved higher Pass@1 than MPSC on all three benchmarks.

Pass@1 (%)HumanEvalHumanEval+MBPP
MPSC74.1765.0569.34
N+PC with MV82.9376.2274.00
N+PC with ACES-C84.7675.6174.00
N+PC with ACES-O84.7676.2273.77

Q5/W2: Does CodeT already use both signals? How does N+PC differ from MPSC, what do the additional reference and the endpoints contribute, and how does N+PC compare with an adaptive mixture of MV and MBR-exec scores?

Compared with CodeT: N+PC uses the generalized pass matrix B\mathcal B to capture both agreement with the generated expected outputs and agreement between candidate outputs. CodeT uses only the first signal. As Section 5 describes, CodeT groups programs by their pass patterns and weights each group by its size and the number of tests passed. Both quantities come from the same pass matrix BB, the expected-output slice of B\mathcal B. When two programs fail the same assertion, BB cannot tell whether they returned the same alternative answer. N+PC captures that agreement directly.

Compared with MPSC: MPSC propagates scores on a single graph linking programs, tests, and specifications. Its program–test edges record whether each program passes each test. N+PC builds a graph for each reference. An edge between two programs is weighted by the fraction of tests on which both produce the same output as that reference. By combining these graphs in one objective, N+PC can use agreement on an alternative output even when both programs fail the generated assertion.

The expected-output row and the two endpoints: N+PC can reduce to either of two ranking regimes. At λ=0\lambda=0 and κ→∞\kappa\to\infty, it recovers the base scorer’s ranking on the expected-output pass matrix BB. At κ=0\kappa=0, it ranks programs solely by agreement among candidates. These reductions work in one direction. The expected-output row of YY in Eq. (3) defines one full candidate-by-test view, BB, and using each program as a reference gives the other nn views of B\mathcal B. Once the execution data have been summarized as in CodeT and MPSC, combining those summaries cannot in general recover these views. In Table 3, replacing their reference-specific graphs with one shared graph lowered mean Pass@1 from 78.25% to 77.12%, with the views, scores, and weights held fixed.

Adaptive mixture of MV and MBR-exec: We combined normalized MV and MBR-exec scores using N+PC’s task-specific weighting rule. Both methods were evaluated on the same problems with complete execution traces.

Method (Pass@1, %)HumanEvalHumanEval+MBPPMean
Adaptive MV + MBR-exec*83.3372.8469.1075.09
N+PC with MV83.3375.3174.2977.64

* The starred result is the upper bound for direct score mixing over the tested grid because its hyperparameter was chosen to maximize mean Pass@1 on the evaluation sets, a choice unavailable in practice. N+PC with MV exceeds this upper bound (77.64% vs. 75.09%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.