- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to select a correct program from multiple code candidates when only a few tests are available. Correct and incorrect programs can produce identical outputs on the available inputs, so aggregation based solely on execution agreement cannot distinguish candidates within the same execution-equivalence class. The proposed method, TCRT, uses code candidates and their accompanying reasoning traces to construct finer-grained relations. For each ordered candidate pair, it uses teacher forcing under the source candidate’s context to score the changed tokens in both programs, averaging their log probabilities to obtain a directed local potential. It then combines full-code likelihoods on the diagonal, local potentials off the diagonal, and execution agreement into a relational kernel . Candidates are ranked after a finite number of updates, . Experiments cover four 6.7B–9B backbones and four benchmarks: HumanEval+, MBPP+, LiveCodeBench, and CodeContests+. Evaluation is restricted to problems whose candidate pools contain both correct and incorrect programs. The paper reports improvements over the strongest baseline in all 48 settings, averaging 5.24 percentage points.
Critical strengths
-
The paper clearly identifies a limitation of execution-agreement aggregation. Equation (4) separates source weights from candidate relations, and Corollary 2.1 shows that candidates with identical columns in the relation matrix cannot be distinguished by changing source weights. For aggregation methods using the execution-agreement matrix , reweighting sources alone cannot resolve within-class ties. This analysis provides a clear motivation for introducing finer-grained candidate relations and distinguishes reweighting across classes from discrimination within a class.
-
The experiments cover a broad range of settings, with consistently positive reported gains and an ablation of propagation depth. TCRT outperforms the evaluated baselines across four backbones, four benchmarks, and three test budgets, with larger average gains when fewer tests are available. Figure 6 also shows that, in the fixed Qwen3-8B/HumanEval+ setting, selection accuracy increases from 72.62% at to 77.38% at , and remains at 82.14% over the evaluated depths from to . This provides evidence that multiple propagation steps affect selection performance in this setting.
Critical weaknesses
-
Configuration selection and final evaluation use the same correctness labels, leaving the main results without independent validation. Appendix C.3 states that scoring configurations are selected through a finite-grid search using correctness labels from the same mixed subset on which results are reported. This amounts to model selection on the evaluation problems, so the reported gains may partly reflect adaptation to those problems. Until configuration selection is separated from final evaluation, I cannot determine whether the average improvement of 5.24 percentage points would hold.
-
The connection between the relational kernel and correctness ranking is insufficiently validated, and the current theory and ablations do not close this gap. Treating as an allocation weight is reasonable; the key question is whether local likelihood preferences and propagation improve ranking by correctness. Theorem 3.1 gives sufficient conditions for ranking recovery, but the paper does not show whether the proposed scores produce kernels satisfying these conditions in practice. Its relevance to the observed gains remains unclear. Meanwhile, the changed-span/full-code comparison in Table 2 changes both the scoring scope and which side of the pair is scored: the former pools changed tokens from the source and target, whereas the latter scores only the target code. The two configurations are also tuned separately. Without a control removing the reasoning traces and an ablation that varies only the scoring scope, the main gains cannot yet be attributed specifically to trace-conditioned code-difference relations.
Questions for authors
-
Could you specify the configuration search space and provide evaluation results independent of configuration selection? Please report the complete grid, including propagation depth , agreement coefficient , normalization method, and any other tunable parameters, together with the selected configuration for each backbone–benchmark combination. Please select configurations on a disjoint set of problems and freeze them before evaluation, or use another protocol that separates configuration selection from performance estimation.
-
How much do the reasoning traces and local difference scoring each contribute? Could you include an ablation that removes the accompanying reasoning text while retaining the problem statement, as well as a changed-span/full-code comparison that keeps the scoring sides fixed and varies only the token scope? Since Table 2 changes both locality and one-sided versus two-sided scoring, how can their contributions be distinguished from the current results?
-
How does Equation (7) handle the relative scales of full-code and local-difference scores, as well as duplicate candidates? For identical code, the local potential is set to zero, while the self-score is usually negative. Could this favor duplicate candidates and affect the final ranking? Please clarify the normalization and any score scaling used in the main experiments, and whether the gains persist after deduplication.
-
Do reductions in candidate and test counts translate into lower total computational cost? Figure 5 shows that TCRT needs fewer candidates and tests, but it also requires additional model scoring. How does its total runtime, including generation, scoring, and execution, compare with SC at similar accuracy?
5 comments
[Response 2/4] Held-Out Validation and Evaluation-Set Comparisons
Evaluation on held-out problems using generated tests. In the benchmark-level protocol above, we used the benchmark’s hidden tests only to label the auxiliary model’s held-out programs for configuration selection. To address your concern about the use of those tests, we also performed five-fold cross-validation by problem. Following MPSC’s use of generated tests to assess candidate programs, we selected one configuration per split based solely on the programs’ performance on our generated tests across the four training folds. This search used neither the benchmark’s hidden tests nor the candidate programs evaluated in Table 1. We then froze the configuration and applied it to the held-out fold at all three test budgets. The five-fold results below come from the Qwen3-8B run on LiveCodeBench:
| Tests | PPL | SC | RPC | TCRT (reported) | TCRT (five-fold) | Δ vs. best Table 1 baseline (pp) | TCRT (evaluation-set search) |
|---|---|---|---|---|---|---|---|
| 2 | 47.9 | 51.0 | 51.6 | 55.7 | 55.9 | +4.3 | 57.1 |
| 5 | 47.9 | 55.2 | 55.2 | 58.3 | 57.6 | +2.4 | 59.1 |
| 100 | 47.9 | 56.3 | 55.2 | 57.8 | 57.8 | +1.6 | 59.9 |
Pass@1 (%). The first four score columns are from Table 1. Δ compares five-fold TCRT with the strongest Table 1 baseline. For the last column, we searched directly on the evaluation set to gauge how high Pass@1 could reach with one configuration shared across all three test budgets.
On Qwen3-8B/LiveCodeBench, TCRT’s five-fold Pass@1 was within 0.8 percentage points of the corresponding Table 1 score and exceeded every evaluated baseline at all three test budgets. If requested, we can use five-fold cross-validation to select one configuration per benchmark for all experiments.
Evidence that we did not tune the hyperparameters for Table 1 directly on the evaluation set. For Qwen3-8B on LiveCodeBench, direct evaluation-set tuning yielded higher Pass@1 than Table 1 at all three test budgets (last column above). Both HumanEval+ scores were already reported in our original submission. With Qwen3-8B and 100 tests, the depth sweep (Figure 7 in the revised manuscript) showed 82.1% Pass@1 at , while Table 1 reported 81.0% at , the depth selected using the auxiliary model’s held-out programs, but not the best configuration on hidden evaluation set.
[Response 3/4] Scoring Ablations and Candidate Deduplication
Q2/W2: What do reasoning traces, token scope and scoring side each contribute? Do local scores favor correct programs when the public tests cannot distinguish them?
Revised Table 2 reports three ablations on Qwen3-8B with 100 tests per problem. Within each benchmark, we kept the scoring hyperparameters fixed across the ablation runs. The variant that retained the problem statement and instructions but omitted the reasoning trace scored 2.7 percentage points below TCRT in mean Pass@1 across four benchmarks. With both programs scored, full-code scoring averaged 2.0 points below changed-token scoring. With scoring restricted to changed tokens, scoring only the target program averaged 2.6 points below scoring both programs.
| Scoring variant | HumanEval+ | MBPP+ | LiveCodeBench | CodeContests+ | Mean | Δ (pp) |
|---|---|---|---|---|---|---|
| TCRT | 81.0 | 70.9 | 57.8 | 61.7 | 67.8 | — |
| Remove the reasoning trace | 81.0 | 67.1 | 52.1 | 60.5 | 65.2 | −2.7 |
| Score all tokens on both sides | 81.0 | 67.7 | 56.8 | 58.0 | 65.9 | −2.0 |
| Score only target-side changes | 79.8 | 68.4 | 56.3 | 56.8 | 65.3 | −2.6 |
Mean is Pass@1 (%) averaged equally across the four benchmarks. Δ is the difference from TCRT in percentage points, computed before rounding.
Ranking programs that agree on the public tests. In a supplementary HumanEval analysis with Qwen2.5-Coder-1.5B-Instruct, we compared pairs in which a correct and an incorrect program agreed on the public tests. TCRT ranked the correct program higher in 59.3% of pairs, compared with 49.1% for PPL. When a third candidate’s reasoning trace was used to score the two programs, the local score favored the correct program in 57.8% of comparisons.
Q3: How does Eq. (7) weight identical programs, how are full-code and local scores scaled and normalized, and do the gains persist after deduplication?
We retain repeated samples because the frequency of a program among responses to a prompt estimates its generation probability, . Identical programs can also come with different reasoning traces. An exact copy has a local score of zero before normalization. When the source program has a negative self-score, its row can assign more weight to the copy than to the source program itself. The final ranking aggregates support from all reasoning traces, allowing a less frequent program to rank first when other traces favor its implementation.
Score scaling and normalization. Both terms in Eq. (7) are mean token log probabilities under the source context. The self-score averages over the full program, and the local score pools the changed tokens from both programs. Agreement on the available tests scales the corresponding cross-score by . For each benchmark, we fixed the relative scale of the self- and cross-scores before evaluation and applied the normalization listed in the row normalization column of the configuration table above. Duplicate counts received no additional weight.
Gain after deduplication. On Qwen3-8B/CodeContests+, exact-code deduplication removed 18.94% of candidates. Within each problem, we kept the response with the lowest original sample ID for each distinct code string. At each test budget, all methods reranked the same reduced pools using their original settings and test inputs. After deduplication, TCRT led the strongest baseline by 1.85, 1.85 and 0.62 Pass@1 points at 2, 5 and 100 tests, respectively.
| Tests | TCRT before (%) | TCRT after (%) | Best baseline after (%) | Gain before (pp) | Gain after (pp) |
|---|---|---|---|---|---|
| 2 | 60.49 | 57.41 | 55.56 | +5.56 | +1.85 |
| 5 | 61.73 | 59.88 | 58.02 | +4.32 | +1.85 |
| 100 | 61.73 | 61.11 | 60.49 | +1.23 | +0.62 |
Gains use unrounded Pass@1 against the best of PPL, SC and RPC on the corresponding pool.
[Response 1/4] Benchmark-Level Configuration Selection and Search Space
Q1/W1: Which correctness labels were used for configuration selection? Do the gains persist on held-out problems, and which settings were searched and used for each backbone and benchmark?
During development, we ran an exploratory search to see how high Pass@1 could reach when tuning directly on the evaluation candidates. We recognized then that using this search for the main comparison would reuse the same correctness labels for both selection and evaluation. We therefore chose to fix one configuration per benchmark before the main evaluation. In Appendix C.3 of the submitted manuscript, we incorrectly described this historical exploratory search as the procedure used to select the configurations for Table 1. We are very sorry that this oversight misled you.
Configuration selection. For the results reported in Table 1, however, we selected one configuration per benchmark using a held-out pool of candidate programs generated by a separate LLM. We ran these programs against the benchmark’s hidden tests and used only their correctness labels for cross-validation. We froze each benchmark’s configuration before evaluation and used it across all four backbones and all test budgets. The separate auxiliary LLM served only for hyperparameter selection. Revised §4.1 and Appendix C.3 describe this procedure. The selected configurations are listed below.
Search space. The table summarizes the candidate settings for benchmark-level configuration selection. We use β for the agreement coefficient in Eq. (7).