acceptodds
Under review as a conference paper at ICLR 2027

Beyond Execution Agreement with Few Tests: Relational Transmission for Code Generation

Abstract

Large language models (LLMs) have become increasingly capable code generators, yet reliable assessment of generated code remains essential to improving inference and learning. Existing approaches draw on model likelihood for independent assessment and execution agreement for relational assessment on available tests. However, inadequate tests can cause correct and incorrect candidates to exhibit identical observed behavior, leaving execution agreement alone unable to rank the correct candidates higher. We therefore seek finer relational evidence in code differences to move beyond execution agreement with few tests. To this end, we propose Trace-Conditioned Relational Transmission (TCRT). For each candidate pair, TCRT uses each response's reasoning trace or other non-code text in turn to score changed tokens on both sides of the code diff. For each response's context, the mean token log probability defines a directed local log potential. These potentials, with unary model likelihood and execution agreement, define a relational kernel. TCRT applies finite-step relational transmission through this kernel to redistribute evidence and rank the candidates. Theoretically, our unified analysis explains why execution agreement can leave likelihood-based misrankings unresolved. We prove that TCRT can reverse these misrankings when informative contexts receive sufficient support. Empirically, TCRT consistently outperforms strong baselines across LLMs and code-generation benchmarks, with particularly strong gains when few tests are available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
code generation, large language models, test-time scaling, code reranking, reasoning traces

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JLo2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select a correct program from multiple code candidates when only a few tests are available. Correct and incorrect programs can produce identical outputs on the available inputs, so aggregation based solely on execution agreement cannot distinguish candidates within the same execution-equivalence class. The proposed method, TCRT, uses code candidates and their accompanying reasoning traces to construct finer-grained relations. For each ordered candidate pair, it uses teacher forcing under the source candidate’s context to score the changed tokens in both programs, averaging their log probabilities to obtain a directed local potential. It then combines full-code likelihoods on the diagonal, local potentials off the diagonal, and execution agreement into a relational kernel PP. Candidates are ranked after a finite number of updates, μ(t+1)=μ(t)P\mu^{(t+1)}=\mu^{(t)}P. Experiments cover four 6.7B–9B backbones and four benchmarks: HumanEval+, MBPP+, LiveCodeBench, and CodeContests+. Evaluation is restricted to problems whose candidate pools contain both correct and incorrect programs. The paper reports improvements over the strongest baseline in all 48 settings, averaging 5.24 percentage points.

Critical strengths
  1. The paper clearly identifies a limitation of execution-agreement aggregation. Equation (4) separates source weights from candidate relations, and Corollary 2.1 shows that candidates with identical columns in the relation matrix cannot be distinguished by changing source weights. For aggregation methods using the execution-agreement matrix AA, reweighting sources alone cannot resolve within-class ties. This analysis provides a clear motivation for introducing finer-grained candidate relations and distinguishes reweighting across classes from discrimination within a class.

  2. The experiments cover a broad range of settings, with consistently positive reported gains and an ablation of propagation depth. TCRT outperforms the evaluated baselines across four backbones, four benchmarks, and three test budgets, with larger average gains when fewer tests are available. Figure 6 also shows that, in the fixed Qwen3-8B/HumanEval+ setting, selection accuracy increases from 72.62% at L=0L=0 to 77.38% at L=1L=1, and remains at 82.14% over the evaluated depths from L=14L=14 to 3232. This provides evidence that multiple propagation steps affect selection performance in this setting.

Critical weaknesses
  1. Configuration selection and final evaluation use the same correctness labels, leaving the main results without independent validation. Appendix C.3 states that scoring configurations are selected through a finite-grid search using correctness labels from the same mixed subset on which results are reported. This amounts to model selection on the evaluation problems, so the reported gains may partly reflect adaptation to those problems. Until configuration selection is separated from final evaluation, I cannot determine whether the average improvement of 5.24 percentage points would hold.

  2. The connection between the relational kernel and correctness ranking is insufficiently validated, and the current theory and ablations do not close this gap. Treating PijP_{ij} as an allocation weight is reasonable; the key question is whether local likelihood preferences and propagation improve ranking by correctness. Theorem 3.1 gives sufficient conditions for ranking recovery, but the paper does not show whether the proposed scores produce kernels satisfying these conditions in practice. Its relevance to the observed gains remains unclear. Meanwhile, the changed-span/full-code comparison in Table 2 changes both the scoring scope and which side of the pair is scored: the former pools changed tokens from the source and target, whereas the latter scores only the target code. The two configurations are also tuned separately. Without a control removing the reasoning traces and an ablation that varies only the scoring scope, the main gains cannot yet be attributed specifically to trace-conditioned code-difference relations.

Questions for authors
  1. Could you specify the configuration search space and provide evaluation results independent of configuration selection? Please report the complete grid, including propagation depth LL, agreement coefficient β\beta, normalization method, and any other tunable parameters, together with the selected configuration for each backbone–benchmark combination. Please select configurations on a disjoint set of problems and freeze them before evaluation, or use another protocol that separates configuration selection from performance estimation.

  2. How much do the reasoning traces and local difference scoring each contribute? Could you include an ablation that removes the accompanying reasoning text while retaining the problem statement, as well as a changed-span/full-code comparison that keeps the scoring sides fixed and varies only the token scope? Since Table 2 changes both locality and one-sided versus two-sided scoring, how can their contributions be distinguished from the current results?

  3. How does Equation (7) handle the relative scales of full-code and local-difference scores, as well as duplicate candidates? For identical code, the local potential is set to zero, while the self-score is usually negative. Could this favor duplicate candidates and affect the final ranking? Please clarify the normalization and any score scaling used in the main experiments, and whether the gains persist after deduplication.

  4. Do reductions in candidate and test counts translate into lower total computational cost? Figure 5 shows that TCRT needs fewer candidates and tests, but it also requires additional model scoring. How does its total runtime, including generation, scoring, and execution, compare with SC at similar accuracy?

5 comments
AuthorsOpenReview ↗

[Response 1/4] Benchmark-Level Configuration Selection and Search Space

Q1/W1: Which correctness labels were used for configuration selection? Do the gains persist on held-out problems, and which settings were searched and used for each backbone and benchmark?

During development, we ran an exploratory search to see how high Pass@1 could reach when tuning directly on the evaluation candidates. We recognized then that using this search for the main comparison would reuse the same correctness labels for both selection and evaluation. We therefore chose to fix one configuration per benchmark before the main evaluation. In Appendix C.3 of the submitted manuscript, we incorrectly described this historical exploratory search as the procedure used to select the configurations for Table 1. We are very sorry that this oversight misled you.

Configuration selection. For the results reported in Table 1, however, we selected one configuration per benchmark using a held-out pool of candidate programs generated by a separate LLM. We ran these programs against the benchmark’s hidden tests and used only their correctness labels for cross-validation. We froze each benchmark’s configuration before evaluation and used it across all four backbones and all test budgets. The separate auxiliary LLM served only for hyperparameter selection. Revised §4.1 and Appendix C.3 describe this procedure. The selected configurations are listed below.

BenchmarkRow normalizationInitializationβ\betaLL
HumanEval+entmax-1.5SC0.69
MBPP+entmax-1.5RPC0.964
LiveCodeBenchz-scored softmax (T=0.7T=0.7)SC0.99
CodeContests+Rank power 16SC0.92

Search space. The table summarizes the candidate settings for benchmark-level configuration selection. We use β\beta for the agreement coefficient in Eq. (7).

ParameterCandidate settings
InitializationUniform (SC), likelihood-based (PPL), or based on the likelihood mass of each execution class (RPC)
Agreement coefficient β\beta0 to 0.9 in increments of 0.1
Propagation depth LLEvery integer from 1 through 64, plus 128 and 256
Row normalizationsoftmax, z-scored softmax, rank-power normalization, sparsemax, and entmax-1.5
AuthorsOpenReview ↗

[Response 2/4] Held-Out Validation and Evaluation-Set Comparisons

Evaluation on held-out problems using generated tests. In the benchmark-level protocol above, we used the benchmark’s hidden tests only to label the auxiliary model’s held-out programs for configuration selection. To address your concern about the use of those tests, we also performed five-fold cross-validation by problem. Following MPSC’s use of generated tests to assess candidate programs, we selected one configuration per split based solely on the programs’ performance on our generated tests across the four training folds. This search used neither the benchmark’s hidden tests nor the candidate programs evaluated in Table 1. We then froze the configuration and applied it to the held-out fold at all three test budgets. The five-fold results below come from the Qwen3-8B run on LiveCodeBench:

TestsPPLSCRPCTCRT (reported)TCRT (five-fold)Δ vs. best Table 1 baseline (pp)TCRT (evaluation-set search)
247.951.051.655.755.9+4.357.1
547.955.255.258.357.6+2.459.1
10047.956.355.257.857.8+1.659.9

Pass@1 (%). The first four score columns are from Table 1. Δ compares five-fold TCRT with the strongest Table 1 baseline. For the last column, we searched directly on the evaluation set to gauge how high Pass@1 could reach with one configuration shared across all three test budgets.

On Qwen3-8B/LiveCodeBench, TCRT’s five-fold Pass@1 was within 0.8 percentage points of the corresponding Table 1 score and exceeded every evaluated baseline at all three test budgets. If requested, we can use five-fold cross-validation to select one configuration per benchmark for all experiments.

Evidence that we did not tune the hyperparameters for Table 1 directly on the evaluation set. For Qwen3-8B on LiveCodeBench, direct evaluation-set tuning yielded higher Pass@1 than Table 1 at all three test budgets (last column above). Both HumanEval+ scores were already reported in our original submission. With Qwen3-8B and 100 tests, the depth sweep (Figure 7 in the revised manuscript) showed 82.1% Pass@1 at L=14L=14, while Table 1 reported 81.0% at L=9L=9, the depth selected using the auxiliary model’s held-out programs, but not the best configuration on hidden evaluation set.

AuthorsOpenReview ↗

[Response 3/4] Scoring Ablations and Candidate Deduplication

Q2/W2: What do reasoning traces, token scope and scoring side each contribute? Do local scores favor correct programs when the public tests cannot distinguish them?

Revised Table 2 reports three ablations on Qwen3-8B with 100 tests per problem. Within each benchmark, we kept the scoring hyperparameters fixed across the ablation runs. The variant that retained the problem statement and instructions but omitted the reasoning trace scored 2.7 percentage points below TCRT in mean Pass@1 across four benchmarks. With both programs scored, full-code scoring averaged 2.0 points below changed-token scoring. With scoring restricted to changed tokens, scoring only the target program averaged 2.6 points below scoring both programs.

Scoring variantHumanEval+MBPP+LiveCodeBenchCodeContests+MeanΔ (pp)
TCRT81.070.957.861.767.8—
Remove the reasoning trace81.067.152.160.565.2−2.7
Score all tokens on both sides81.067.756.858.065.9−2.0
Score only target-side changes79.868.456.356.865.3−2.6

Mean is Pass@1 (%) averaged equally across the four benchmarks. Δ is the difference from TCRT in percentage points, computed before rounding.

Ranking programs that agree on the public tests. In a supplementary HumanEval analysis with Qwen2.5-Coder-1.5B-Instruct, we compared pairs in which a correct and an incorrect program agreed on the public tests. TCRT ranked the correct program higher in 59.3% of pairs, compared with 49.1% for PPL. When a third candidate’s reasoning trace was used to score the two programs, the local score favored the correct program in 57.8% of comparisons.

Q3: How does Eq. (7) weight identical programs, how are full-code and local scores scaled and normalized, and do the gains persist after deduplication?

We retain repeated samples because the frequency of a program cc among responses to a prompt xx estimates its generation probability, Pgen(c∣x)P_{\mathrm{gen}}(c\mid x). Identical programs can also come with different reasoning traces. An exact copy has a local score of zero before normalization. When the source program has a negative self-score, its row can assign more weight to the copy than to the source program itself. The final ranking aggregates support from all reasoning traces, allowing a less frequent program to rank first when other traces favor its implementation.

Score scaling and normalization. Both terms in Eq. (7) are mean token log probabilities under the source context. The self-score averages over the full program, and the local score pools the changed tokens from both programs. Agreement on the available tests scales the corresponding cross-score by 1−β1-\beta. For each benchmark, we fixed the relative scale of the self- and cross-scores before evaluation and applied the normalization listed in the row normalization column of the configuration table above. Duplicate counts received no additional weight.

Gain after deduplication. On Qwen3-8B/CodeContests+, exact-code deduplication removed 18.94% of candidates. Within each problem, we kept the response with the lowest original sample ID for each distinct code string. At each test budget, all methods reranked the same reduced pools using their original settings and test inputs. After deduplication, TCRT led the strongest baseline by 1.85, 1.85 and 0.62 Pass@1 points at 2, 5 and 100 tests, respectively.

TestsTCRT before (%)TCRT after (%)Best baseline after (%)Gain before (pp)Gain after (pp)
260.4957.4155.56+5.56+1.85
561.7359.8858.02+4.32+1.85
10061.7361.1160.49+1.23+0.62

Gains use unrounded Pass@1 against the best of PPL, SC and RPC on the corresponding pool.

Reviewer_9QEHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In test-time computation and evaluation for large language model (LLM) code generation, existing mainstream methods face two major limitations: independent scoring based on model likelihood (PPL) tends to favor code that is functionally incorrect but has lower perplexity; on the other hand, relational assessment based on execution agreement over available tests (e.g., Self-Consistency) heavily relies on the adequacy of the test suite. When available tests are sparse, a significant portion of both functionally correct and incorrect code candidates fall into the same execution-agreement class, rendering them indistinguishable. To address these limitations, this paper proposes a novel code evaluation method named Trace-Conditioned Relational Transmission (TCRT). Empirical results demonstrate that TCRT consistently outperforms strong baselines across matched candidate pools and test budgets, achieving significant average gains.

Critical strengths
  • Clever combination of code differences (diffs) and reasoning traces for fine-grained relational assessment: Most existing code selection approaches treat code as a whole or rely coarsely on execution outputs. This paper keenly observes that even when two code candidates exhibit identical behavior on a small set of available tests, they still differ in their specific implementation details. TCRT leverages a candidate's own reasoning trace as a conditional context to cross-evaluate these diff regions, effectively bypassing the bottleneck of inadequate test suites.
  • Solid and rigorous theoretical foundation: Many papers on reasoning reranking or test-time compute tend to pile up empirical experiments while lacking deep theoretical explanations. This work not only establishes a unified scoring framework in Section 2 (viewing PPL and SC as special cases), but also derives a finite-step ranking recovery theorem in Section 3, strictly proving the conditions under which relational transmission can reverse likelihood-based misrankings.
Critical weaknesses
  • Heavy reliance on LLM reliability and unverified generalization: The reliability of the current evaluation method heavily depends on the underlying LLM itself, and its robustness as well as generalization capability across different model families require further verification.
  • Insufficient analysis of computational overhead and complexity: As the candidate pool size K further expands, the cross-validation overhead scaling at O(K^2) quickly explodes. The paper lacks an explicit comparison of wall-clock inference time against baselines (such as SC and RPC).
Questions for authors
  • Regarding computational efficiency and scalability: Could the authors add a comparison of wall-clock time overhead in a revision to clarify how many times the computational cost increases compared to SC or RPC while achieving the Pass@1 performance gains?
  • Regarding the robustness of hyperparameter \beta and transmission depth L: Could the authors elaborate on the sensitivity and robustness of the agreement coefficient \beta and transmission depth L across different model architectures and benchmark difficulties?
3 comments
AuthorsOpenReview ↗

[Response 1/2] Runtime at Comparable Accuracy

Q1/W2: How does TCRT’s wall-clock time compare with SC and RPC at comparable accuracy?

TCRT took less estimated time than both SC and RPC at all five accuracy targets they reached. Only TCRT reached the two higher targets. On HumanEval+ with Qwen3-8B, TCRT also achieved the highest mean Pass@1 at all seven corresponding time limits (revised Figure 6, p. 9).

Time to reach the same accuracy. We take targets from the joint Pareto frontier underlying Figure 6(a). For each candidate count KK with a TCRT point on the frontier below 1,000 s, we show the fastest and the most accurate such point. Each entry is the shortest estimated runtime at which that method met or exceeded the target. Both tables use mean Pass@1 over 20 seeds and round values only for display.

Target mean Pass@1TCRT candidates/tests (K/mK/m)TCRT (s)PPL (s)SC (s)RPC (s)
73.15%2/210.7531.8916.4217.54
75.65%4/226.31Not reached32.8234.21
75.77%4/3226.43Not reached32.8234.21
78.04%16/4221.95Not reached263.01534.21
78.15%16/16222.23Not reached263.01534.21
78.87%32/4755.24Not reachedNot reachedNot reached
79.29%32/16755.82Not reachedNot reachedNot reached

“Not reached” means that the method did not attain the target in the evaluated configurations. At the 78.04% target, TCRT needed 16 candidates, compared with 64 for SC and 128 for RPC. Using the smaller pool, TCRT took 15.6% less estimated time than SC and 58.5% less than RPC.

Estimated runtimes are totals for the evaluation subset on eight GPUs. They include candidate generation, scoring, test execution, final selection and, for RPC, pruning. We exclude test generation for all methods because tests can be generated offline and reused. For the 78.04% row, TCRT used four tests per problem and SC used eight, so generating tests on demand at the same cost per test would further increase TCRT’s time saving.

Best accuracy within the same time budget. Each entry gives the highest mean Pass@1 that method achieved within the stated time limit. At 755.82 s, the best mean Pass@1 was 79.29% for TCRT, 78.75% for SC and 78.57% for RPC.

Time limit (s)TCRT candidates/tests (K/mK/m)TCRTPPLSCRPC
10.752/273.15%71.19%72.74%71.19%
26.314/275.65%72.14%75.24%74.82%
26.434/3275.77%72.14%75.24%74.82%
221.9516/478.04%73.27%77.98%77.02%
222.2316/1678.15%73.27%77.98%77.02%
755.2432/478.87%73.27%78.75%78.57%
755.8232/1679.29%73.27%78.75%78.57%
AuthorsOpenReview ↗

[Response 2/2] Hyperparameter Sensitivity and Generalization Across Backbones

Q2: How sensitive is TCRT to the agreement coefficient β\beta and propagation depth LL across model architectures and benchmark difficulties?

The original submission reported sensitivity analyses for propagation depth LL and agreement coefficient β\beta on HumanEval+ with Qwen3-8B (Figures 7 and 8, respectively, in Appendix D of the revised manuscript). To assess sensitivity across model families and difficulty levels, we also evaluated DeepSeek-Coder-6.7B-Instruct on the Easy, Medium, and Hard subsets of LiveCodeBench. For DeepSeek-Coder, we report results at β=0.9\beta=0.9 and L=9L=9. Both tables show Pass@1 (%).

Agreement coefficient β\beta (fixed L=9L=9). The reported setting (β=0.9\beta=0.9) yielded the same Pass@1 as β=0.7\beta=0.7 in all three groups. On Medium and Hard, Pass@1 remained constant across all tested β\beta values from 0.3 to 0.9. On Easy, it ranged from 67.6% to 68.5% across the four values shown from 0.6 to 0.9.

Difficulty / β0.00.10.20.30.40.50.60.70.80.9
Easy64.864.864.864.866.766.767.668.567.668.5
Medium35.135.137.837.837.837.837.837.837.837.8
Hard18.227.327.336.436.436.436.436.436.436.4

Propagation depth LL (fixed β=0.9\beta=0.9). In all three groups, Pass@1 remained constant across all integer depths from 5 through 45, including L=9L=9. For an irreducible, aperiodic propagation matrix, the candidate weights converge as LL increases.

Difficulty / L012–345–45
Easy52.863.066.767.668.5
Medium29.737.837.837.837.8
Hard9.136.436.436.436.4

W1: How much does TCRT’s reliability depend on the underlying LLM, and do its gains generalize across model families?

TCRT outperformed the strongest of PPL, SC and RPC in all 48 settings in Table 1. These settings span four backbones from the DeepSeek and Qwen families, four benchmarks and three test budgets. The table reports mean Pass@1 for each backbone. We calculated the gain against the strongest baseline in each setting, then averaged the 12 gains for that backbone.

BackbonePPLSCRPCTCRTGain (pp)
DeepSeek-Coder-6.7B-Instruct50.65%61.47%61.55%67.95%+6.37
Qwen2.5-Coder-7B-Instruct60.88%67.44%67.20%73.28%+5.84
Qwen3-8B57.06%64.17%64.07%67.68%+3.47
Qwen3.5-9B64.36%71.63%71.57%76.92%+5.29

TCRT can correct the backbone’s own likelihood ranking. Even when two programs agree on every available test, other candidates’ reasoning traces can favor one implementation over the other. TCRT uses the backbone’s token probabilities to score this support where the code differs. Theorem 3.1 shows when that support can reverse an initial preference for the incorrect program.

On Qwen3-8B, mean Pass@1 rose from 57.06% with likelihood ranking (PPL) to 67.68% with TCRT on the same candidate pools, a 24.7% relative reduction in selection error.

Dear Reviewer,

Thank you very much for your time and for your constructive feedback on our submission. As the discussion period approaches its deadline, we would like to follow up and ask whether you have had a chance to review our detailed rebuttal and the additional experimental results. We would be grateful for any indication of whether our response has addressed your concerns.

We fully understand that your schedule may be busy. If you have any remaining questions or would like further clarification on any point, we would be very happy to continue the discussion and address them promptly.

Thank you again for your time and consideration.

Best regards,

Reviewer_ZxAWOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

TCRT ranks generated code by scoring code-diff tokens under each candidate's reasoning context, combining with likelihood and execution agreement, then propagating support iteratively. It outperforms baselines, especially with few tests.

Critical strengths

The motivation seems sound but I am unable to follow the rest of the paper to provide any constructive feedback. Will mark as weak accept so that the paper is not unnecessarily penalized. Will defer to other reviewers.

Critical weaknesses

The motivation seems sound but I am unable to follow the rest of the paper to provide any constructive feedback. Will mark as weak accept so that the paper is not unnecessarily penalized. Will defer to other reviewers.

Questions for authors

N/A.

1 comment
AuthorsOpenReview ↗

[Response 1/1] Method Overview

W1: Clarify the approach that follows from the paper’s motivation.

Thank you for your feedback on the presentation. Two programs can produce the same outputs on every available test even if only one is correct. In that case, the solution described in one answer’s reasoning trace can still help us assess the code in another. TCRT scores how well the parts where the two programs differ fit that reasoning trace. We combine these scores with test agreement to build a graph of support among the candidates. TCRT then uses graph propagation to select the program with the strongest overall support. The opening paragraph of Section 3 (p. 3) now explains this intuition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.