acceptodds
Under review as a conference paper at ICLR 2027

TEST-TO-TEACH: EXECUTION-CONDITIONED ONPOLICY DISTILLATION FOR AUTONOMOUS CODINGAGENTS

Abstract

Unit tests are widely used to evaluate and refine code generated by autonomousagents, but their outcomes are often reduced to binary rewards or repair prompts.We introduce Test-to-Teach, an execution-conditioned, on-policy distillationframework that uses program behavior as supervision. The student generates code,tests, and repair trajectories. The framework then constructs counterfactual testsby perturbing variables, inverting boundary conditions, substituting dependencies,and contaminating state. These tests target implementations that pass the existingsuite for the wrong reason. Without seeing reference solutions, the teacher derives local behavioral constraints, causal failure graphs, and minimal repair principles from execution traces. Execution information gain determines the weightof each distillation example. Under a matched training budget on SWE-benchVerified, Test-to-Teach improves the resolved-task rate from 34.8% to 43.6% overstandard trajectory distillation trained on the same teacher traces. Across threecross-repository benchmarks, it raises the hidden-test pass rate by 9.7 percentagepoints and reduces patch overfitting by 31.2%. In online repair on HumanEval+,it reaches a solved-task rate of 86.59%. The results show how execution evidencecan support code generation, failure attribution, and repair planning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-to-Teach

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k6JLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The research background of this paper is that while current code-generation models are capable of producing and modifying programs—with the generated code often passing existing public tests—these models do not necessarily possess a true understanding of how the programs function. In certain scenarios, such as those involving edge cases, state transitions, or shifts in dependencies, the generated code may fail to execute correctly; in other words, models that pass public tests often lack sufficient generalization capabilities. To address this, the authors propose "Test-to-Teach," a framework designed to transform execution-based checks into effective signals for both code repair and model learning. The goal is to resolve errors that persist in code even after it has passed public tests, thereby improving the success rate of repairs on held-out (hidden) test cases. The Test-to-Teach workflow functions as follows: a "student" model generates code, which is then executed to identify potential issues; a "teacher" model uses the evidence gathered from these executions to propose repairs; and finally, the student model is trained using these verified repair records. Specifically, the student model receives a task and public test cases to generate candidate code. A "Counterfactual Evidence Collection" module then alters specific conditions to uncover potential, previously overlooked flaws. The findings from these checks are formatted into a structured output, which serves as input for the teacher model. The teacher model’s verified outputs are then used as training samples for "Offline Student Distillation." The primary contribution of this paper is the introduction of an execution-condition distillation framework, which converts observed program behaviors into weighted supervision for code repair, and integrates counterfactual operators and failure descriptions into a structured repair process.

Critical strengths

The proposed framework integrates fixed counterfactual execution operators and failure characterizations derived from execution into a structured repair process. It quantifies the contributions of these components through paired ablation studies and execution audits. The design is well-conceived, with a clear focus on how execution evidence can be transformed into supervisory signals.

Critical weaknesses

The paper contains an excessive number of tables, which severely hampers readability. There is only one primary architectural diagram in the entire paper, with all other data presented in tables; this makes it difficult to grasp the content and prevents a clear, intuitive comparison between the proposed approach and existing methods. The abstract is too vague, offering almost no insight into the actual work presented in the paper. Furthermore, the paper lacks sufficient innovation: the workflow—in which a student model generates erroneous code, the code is tested, and a teacher model provides feedback to facilitate repairs and subsequent retraining of the student model—has already been explored in numerous existing studies.

Questions for authors

All references cited in the paper date from 2024 or earlier; have the authors surveyed literature from the last two years—is there truly no relevant work from this period? Furthermore, have the authors read the paper "Personalized Distillation"? If so, why is there no comparison with the approach proposed in that work? Removing the counterfactual tests and the "failure cases" analysis resulted in success rate decreases of 1.46% and 1.22%, respectively; however, given that the p-values for the clustering tests are 0.085 and 0.120, can the independent contribution of the core modules really be established? Additionally, why are all experimental data presented exclusively in tables rather than through more intuitive formats, such as line graphs?

2 comments

Dear Reviewer k6JL,

Thank you for requesting a clearer technical contribution, fuller related work, and an easier-to-read account of what the experiments establish. The revision adds 7,305 SWE, 16,400 CF×graph, and 16,400 filtering rows.

1. Contribution and related work. Test-to-Teach closes an execution-to-training feedback loop: the current student generates a candidate; counterfactual probes and the failure graph expose its behavior; a teacher proposes a repair; a fixed visible-suite non-regression gate admits the trace; and acceptance-associated evidence weights the next student update. Individual ingredients have precedents. The distinct learning signal is that the same student's execution failures determine both the teacher's repair evidence and its subsequent training examples. Section 9 now discusses Personalized Distillation, PLUM, CodeRL, LeDex, CodeT, MT-LAPR, TestGen-LLM, SWE-agent, and Agentless by their actual overlap with this mechanism.

2. Main results and training interventions. We keep held-out scoring and the statistical unit visible in the reply:

ComparisonT2TControlDifference
HumanEval+ hidden success, five-run mean733/820Extra-call 708/820+3.05 pp
SWE-bench Verified, primary run206/487Extra-call 184/487+4.52 pp
HumanEval+ no on-policy collection733/820707/820+3.17 pp
HumanEval+ uniform instead of IG weights733/820712/820+2.56 pp
HumanEval+ neither training mechanism733/820698/820+4.27 pp

HumanEval+ training variants share 12,000 samples/2.4M tokens; choices precede checkpoint freezing. Primary T2T-minus-extra-call run differences are +25, +21, +29, +23, +27/820 (two-sided sign p=0.0625p=0.0625), distinct from task-slot pairing. The HE-train-v1 isolation audit reports 0 confirmed overlaps across 420 source and 164 evaluation tasks.

The main HumanEval+ comparison records public passes 778/820 vs 752/820 and hidden passes 733/820 vs 708/820. Conditional hidden success is 733/778 vs 708/752; the clear result is +25/820 hidden passes, not a large change in conditional overfitting.

3. Component attribution and adapted methods. We added a four-arm CF×graph matrix at each of five fixed checkpoints on the same 820 slots:

SettingFive-run mean solved /820
Neither712.6
Graph only721.0
CF only723.8
Both733.0

With the graph present, CF adds 12.0/820 (+1.46 pp) on average; with CF present, the graph adds 9.2/820 (+1.12 pp). Each conditional gain is positive in all five runs. Interaction counts are 0, +3, +2, +1, −2; we therefore make no positive-synergy claim. This fixed-checkpoint design answers a different question from the earlier separately trained Table 36.

The adapted SWE-bench controls now have five training seeds, 487 instances each:

MethodResolved /2,435Five-run mean /487Calls/taskTokens/task
T2T1,0302062.9911,033.5
PLUM adapted9501902.8151,035.9
LeDex adapted9051812.7861,034.7

All three record the same base model, source-task manifest hash, 350 source tasks, 12,000 samples, and 2.4M training tokens. Because T2T uses more actual calls, this is a training-volume-matched descriptive comparison, not strict inference-compute matching or a claim of paired statistical superiority.

SWE per-seed resolved triples (T2T / PLUM / LeDex) are 202/186/178, 206/191/183, 210/189/180, 203/194/186, 209/190/178. T2T leads in every run; the 2,435 outcomes are not independent training replicates.

4. Test semantics and reproducibility scope. Of 2,870 expanded HumanEval+ probe attempts, 1,558 execute; among those, 1,286 are semantically supported, 219 ambiguous, and 53 invalid. Five-seed retraining after excluding invalid probes from gate, IG, or both gives mean solved counts 734.2, 735.0, 736.0/820, versus 733.0/820 original. These are new checkpoints, distinct from cached replay. The corrected IG diagnostic has 669/820 hidden passes and is kept separate from the primary 733/820 result. Appendix G.4 maps the concrete example to official HumanEval/7 (filter_by_substring); a four-case ledger records all four operator types, written expected-output bases, repairs, visible-gate transitions, and hidden outcomes. The four cases cover HumanEval/7 (variable), /0 (boundary), /2 (dependency), and /3 (state). Each ledger reports 1/2 → 2/2 and a locked hidden pass; public replay checks I/O, while the audit measures population-level semantic support. Run-artifact paths are listed; the workbook does not embed those files. Single-pass transfer is kept separate from primary reference-free distillation.

Thank you for the additional experiments and clarifications, which substantially address my concerns about the component contributions and the relationship to prior work. I will raise my rating to 3 (Weak acceptance), while noting the remaining limitations in training-stage attribution and compute matching.

Reviewer_Z6ogOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Test-to-Teach, an on-policy distillation loop for code repair: a 7B student samples candidates, four fixed operators generate counterfactual tests (variable/boundary/dependency/state), a teacher turns execution traces into a minimal patch, and accepted repairs train a weighted loss where weights are the (bias-corrected) mutual information between execution evidence and the visible-suite acceptance label. Hidden tests run only after the candidate is locked. Headline results are HumanEval+ 89.39% vs 86.34% for a budget-matched baseline (+3.05 pp, Holm p=0.014) and SWE-bench Verified 42.3% vs 37.8% for a 7B student (+4.5 pp, Holm p=0.003); BugsInPy/Defects4J gains are non-significant.

Critical strengths
  1. The information weight uses a Miller–Madow finite-sample correction for sparse evidence tables, and the paper explicitly states the value model predicts visible acceptance, not hidden correctness (Sec. 4.6).

  2. On-policy collection and information-gain weighting give the largest paired drops, corroborated by a fixed-budget ratio sweep and a seven-scheme weighting comparison (Tables 2, 8, 9).

Critical weaknesses
  1. PLUM (reference-free generated tests + on-policy execution-labelled candidates), CodeT (consensus selection), CodeRL/LeDex (execution-feedback repairer distillation), MT-LAPR/TestGen-LLM (mutation operators, non-regression filters), and Agentless are neither cited nor compared, so novelty is overstated.

  2. Weights predict visible acceptance Y, but the paper never shows Î or Y correlates with post-lock hidden success; up-weighting may simply sharpen overfitting to the visible suite, and the POR difference is negligible (5.78% vs 5.85%).

  3. Dependency substitution and state contamination execute in only 5.5%/1.2% of attempts (Table 17), yet are kept as core contributions with no ablation restricted to the two active operators.

  4. The 487-task result has no multi-seed training variance (the McNemar CI is conditional on one trained model), while the 24-task stress test swings 37.5% vs 25.0% across seeds.

Questions for authors

See the weaknesses

1 comment

Dear Reviewer Z6og,

Thank you for challenging the novelty claim, baseline fairness, visible-test overfitting, low-coverage operators, and training-seed stability. The revision adds 7,305 SWE and 16,400 CF×graph item records, with observed compute.

1. The mechanism and its intervention. We do not claim that on-policy sampling, generated tests, or teacher repair were individually invented here. Our unit of learning is an accepted teacher repair trace conditioned on the current student's execution failure: candidate-conditioned probes expose behavioral differences; structured failures guide teacher repair; a fixed visible-suite non-regression gate admits traces; acceptance-associated information weights the next student update. CodeT selects with tests; CodeRL uses RL; LeDex trains debugging; PLUM uses on-policy preferences. Our claim is the execution-to-training feedback path, tested by changing the policy's training rather than merely adding inference calls:

HumanEval+ training conditionSolved /820Difference from full
Full T2T733—
No on-policy collection707−3.17 pp
Uniform weights instead of IG712−2.56 pp
Neither mechanism698−4.27 pp
Extra-call distillation control708−3.05 pp

The variants use 12,000 samples/2.4M tokens; checkpoints are frozen before online repair. The ablations thus test pre-evaluation training, not a claim that each ingredient is new. Five T2T-minus-extra-call run differences are +25, +21, +29, +23, +27/820 (two-sided sign p=0.0625p=0.0625).

2. Adapted PLUM/LeDex controls across five training seeds. Each seed evaluates 487 listed SWE-bench Verified instances per method. All three controls record one base model, source-task manifest hash, 350 source tasks, 12,000 samples and 2.4M tokens. Separately, the HE-train-v1 audit finds 0 confirmed overlaps among 420 source and 164 HumanEval+ evaluation tasks (two near-duplicates reviewed).

Training seedT2T /487PLUM /487LeDex /487
300202186178
301206191183
302210189180
303203194186
304209190178
Total /24351,030950905

The five-run means are 206/487, 190/487, and 181/487; T2T exceeds adapted PLUM by 3.29 pp and adapted LeDex by 5.13 pp. T2T's seedwise margins over PLUM are 16, 15, 21, 9, 19 and over LeDex 24, 23, 30, 17, 31. All favor T2T; five-run sign-test p=0.0625p=0.0625. Actual resource use is:

MethodCalls/taskTokens/task
T2T2.9911,033.5
PLUM adapted2.8151,035.9
LeDex adapted2.7861,034.7

T2T uses more calls, so these are training-volume-matched descriptive controls, not strictly equal realized inference compute or a paired-significance claim. The config records inference_cap=3 while item rows record candidate_cap=4; we therefore use measured calls for fairness. The primary extra-call comparison is a separate protocol: 206/487 vs 184/487 on SWE-bench and 733/820 vs 708/820 on HumanEval+.

Primary HumanEval+ public passes are 778/820 vs 752/820 and hidden passes 733/820 vs 708/820. Conditional hidden success is 733/778 (94.22%) vs 708/752 (94.15%): more hidden successes overall, but only about 0.07 pp difference conditional on public pass.

3. IG and hidden success. IG is fitted to visible-suite repair acceptance; its AUROC is not a hidden-correctness metric. In an independent 820-slot diagnostic with 669 hidden passes (distinct from the primary 733/820 run), we stratified by public pass and sorted unrounded pooled IG:

Public statusLow IG (Q1)High IG (Q4)
Pass157/182166/183
Fail3/226/23

Intermediate Q2/Q3 counts are 160/182, 168/182 for public passes and 5/23, 4/23 for public failures. Q4 is higher than Q1 in both strata, but intermediate quartiles are not monotonic and public-fail cells are small. This is a descriptive association, not proof that IG causes hidden generalization. The causal evidence is the uniform-weight retraining ablation above; a separate invalid-probe IG-filtering retrain changes mean solved from 733 to 735/820 across five seeds.

4. Operators and stability. A five-checkpoint fixed-model CF×graph matrix shows CF adds 11–13/820 with the graph present and the graph adds 8–11/820 with CF present in every seed; the factorial interaction ranges −2 to +3/820, so we do not claim synergy. The four-versus-two operator contrast is 733 vs 729/820 (+0.49 pp; task-clustered 95% interval [−2.56, 3.54] pp). Dependency and state remain conditional auxiliary probes, not proven independent average contributors. The five SWE runs characterize adapted-control training variation; the separate primary task-paired test does not replace run-level inference.

Reviewer_egPJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Test-to-Teach, a code-repair and distillation workflow that turns public-test and generated counterfactual execution evidence into structured teacher feedback, filters repairs by visible-test improvement without regression, and weights accepted training examples by estimated information about acceptance. The reference-free primary comparisons report 89.39% versus 86.34% on HumanEval+ and 42.3% versus 37.8% on SWE-bench-Verified against extra-call controls with matched reported resources. The paper also studies component removal, reference-assisted validation, and transfer to other repair benchmarks.

Critical strengths
  1. Learning from execution failures is a meaningful direction for coding agents. The paper provides a concrete pipeline for turning these failures into repair supervision, rather than relying only on whether the final program passes the tests.

  2. The main empirical results are convincing. Under matched additional-call budgets, the method improves HumanEval+ from 86.34% to 89.39% and SWE-bench Verified from 37.8% to 42.3%. The paired analysis and resource accounting make these comparisons more informative than reporting accuracy alone.

Critical weaknesses
  1. The reference-free test oracle is central to the method, but I am not clear how it works in practice. Section 4.4 and Appendix D.1 list failed assertions and exceptions among invalid tests, while such failures can also be exactly what a valid test should reveal about incorrect code. It is also unclear how the expected behavior is established after a counterfactual transformation without using a reference solution. I would appreciate one complete example showing how a test is constructed, validated, and then used to decide whether a repair is correct.

  2. The training ablations do not clearly connect to the HumanEval+ evaluation setting. Tables 18 and 19 describe this setting as one-pass online repair, with training resources marked N/A and no training split, whereas the largest drops in Table 2 come from removing on-policy collection and information-gain weighting. This does not imply that no prior training was used, but the paper should specify which checkpoints were trained on which data, when they were frozen, and how each ablation changes the evaluated system. Appendix C.1 also acknowledges that a single observation cannot estimate mutual information, but the actual pooling across tasks, tests, or operators is not sufficiently specified to reproduce the weights.

  3. The evidence for the counterfactual tests and failure graph is weaker than the evidence for the overall pipeline. In Table 2, removing these components reduces performance by 1.46 and 1.22 points, with clustered p-values of .085 and .120, while the larger and statistically stronger effects come from on-policy collection and information-gain weighting. These results do not show that the two components are ineffective, but they leave it unclear how much of the main improvement should be attributed to them. The test audit in Table 23 is useful, though it does not by itself establish their advantage over a simpler execution-conditioned repair pipeline.

Questions for authors

How does the reference-free oracle distinguish an invalid test from a valid test that exposes a bug in the candidate? Could the authors provide a concrete example showing how the expected behavior is determined after a counterfactual edit, without a gold implementation?

HumanEval+ is described as one-pass online repair, with training listed as N/A in Table 18. How does this setting relate to the on-policy and information-gain ablations in Table 2? Please clarify which checkpoint is evaluated, what data it was trained on, and how the information-gain weights are computed and used in this setting.

The full method shows gains under matched budgets, but the individual contributions of counterfactual tests and the failure graph are less clear from Table 2. Can the authors clarify when these two components help, and whether their benefits depend on each other? This would help distinguish their contribution from that of on-policy collection and information-gain weighting.

1 comment

Dear Reviewer egPJ,

Thank you for asking how reference-free probes obtain defensible outputs, how training choices affect a frozen online evaluator, and what CF tests and the failure graph contribute separately. The new evidence includes 16,400 CF×graph and 16,400 filtering-retrain rows.

1. CF × failure graph: five fixed-checkpoint, four-arm comparisons. Each row below evaluates the same checkpoint on the same 820 task-seed slots under four inference settings. The columns show solved counts; the last column is the factorial interaction count, both − CF only − graph only + neither.

Training seedNeitherGraph onlyCF onlyBothInteraction
11017127217237320
1102712718721730+3
1103716722727735+2
1104710720722733+1
1105713724726735−2
Mean / 820712.6721.0723.8733.0+0.8

With the graph present, CF adds 12.0/820 (+1.46 pp) on average; with CF present, the graph adds 9.2/820 (+1.12 pp). Both conditional gains are positive in all five checkpoints. The interaction averages only +0.10 pp and changes sign, so we do not claim established synergy. This design resolves the main attribution ambiguity in the earlier Table 36, where the four arms had separate training configurations. It tests inference-stage component effects; the pretraining contribution is addressed by the training ablations below.

Each seed fixes the task IDs, evaluator seeds, generator, visible suite, and candidate cap across arms. Source manifests differ across checkpoints, so inference effects are paired within each checkpoint. Five positive run-level differences give a two-sided sign-test p=0.0625p=0.0625; we report effect sizes without a component significance claim.

2. Semantic validity and invalid-probe sensitivity. A probe's ability to execute is not proof that its expected output matches the task specification. Reference-free admission checks interface, harness execution, public constraints, and consistency without a canonical solution. The expanded audit reports:

HumanEval+ probe auditCountDenominator
Executable1,5582,870 attempts
Semantically supported1,2861,558 executable
Ambiguous2191,558 executable
Invalid531,558 executable

The selected 23-invalid-probe trace found 2 admitted; we do not extrapolate that selected fraction to all generated probes. To measure the training consequence, we retrained four filtering arms, with a distinct checkpoint per arm and seed and 12,000 samples/2.4M tokens per arm:

Exclude invalid probes fromFive solved counts (/820)Mean /820
Neither (original)732, 730, 735, 733, 735733.0
Gate only733, 732, 736, 734, 736734.2
IG only734, 732, 737, 735, 737735.0
Both735, 733, 738, 736, 738736.0

The corresponding mean public-pass counts are 778.0, 779.2, 780.0, and 781.0/820. The gains are small and descriptive. This is a training intervention, unlike our earlier cached gate-only replay. It quantifies sensitivity to invalid tests while leaving the reference-free oracle limitation explicit.

The admission gate does not automatically identify all 219 ambiguous and 53 invalid executable probes. The filtering retrain measures how their use affects the learned policy, while the case's expected output follows from the written task specification, not from hidden tests; hidden scoring occurs after candidate selection.

3. Concrete cases and why training matters online. The official HumanEval/7 task is filter_by_substring. The written containment rule requires ['ba','ab'] for (['ba','ab'],'a'); a prefix-only candidate yields ['ab'], and the repair changes startswith to substring containment. The case ledger reports a 1/2 → 2/2 gate change; public replay verifies candidate and repair I/O, not the gate count. It also records HumanEval/0, /2, and /3 for boundary, dependency, and state probes; all four rows identify the probe, expected-output basis, candidate, teacher call, patch, gate transition, and locked hidden outcome. The workbook lists artifact paths, but the run files are not embedded in it.

On-policy collection and IG change the student before it is frozen for online evaluation. Under the same stated 12,000-sample/2.4M-token budget, full T2T solves 733/820, versus 707/820 without on-policy collection, 712/820 with uniform weights, and 698/820 without both. Against extra-call standard distillation, the full method solves 733/820 vs 708/820 on held-out HumanEval+ tests. Training-run differences are +25, +21, +29, +23, +27/820; their two-sided sign test is p=0.0625p=0.0625. We separate run-level inference from task-slot pairing and do not claim significance for every module.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.