- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The research background of this paper is that while current code-generation models are capable of producing and modifying programs—with the generated code often passing existing public tests—these models do not necessarily possess a true understanding of how the programs function. In certain scenarios, such as those involving edge cases, state transitions, or shifts in dependencies, the generated code may fail to execute correctly; in other words, models that pass public tests often lack sufficient generalization capabilities. To address this, the authors propose "Test-to-Teach," a framework designed to transform execution-based checks into effective signals for both code repair and model learning. The goal is to resolve errors that persist in code even after it has passed public tests, thereby improving the success rate of repairs on held-out (hidden) test cases. The Test-to-Teach workflow functions as follows: a "student" model generates code, which is then executed to identify potential issues; a "teacher" model uses the evidence gathered from these executions to propose repairs; and finally, the student model is trained using these verified repair records. Specifically, the student model receives a task and public test cases to generate candidate code. A "Counterfactual Evidence Collection" module then alters specific conditions to uncover potential, previously overlooked flaws. The findings from these checks are formatted into a structured output, which serves as input for the teacher model. The teacher model’s verified outputs are then used as training samples for "Offline Student Distillation." The primary contribution of this paper is the introduction of an execution-condition distillation framework, which converts observed program behaviors into weighted supervision for code repair, and integrates counterfactual operators and failure descriptions into a structured repair process.
Critical strengths
The proposed framework integrates fixed counterfactual execution operators and failure characterizations derived from execution into a structured repair process. It quantifies the contributions of these components through paired ablation studies and execution audits. The design is well-conceived, with a clear focus on how execution evidence can be transformed into supervisory signals.
Critical weaknesses
The paper contains an excessive number of tables, which severely hampers readability. There is only one primary architectural diagram in the entire paper, with all other data presented in tables; this makes it difficult to grasp the content and prevents a clear, intuitive comparison between the proposed approach and existing methods. The abstract is too vague, offering almost no insight into the actual work presented in the paper. Furthermore, the paper lacks sufficient innovation: the workflow—in which a student model generates erroneous code, the code is tested, and a teacher model provides feedback to facilitate repairs and subsequent retraining of the student model—has already been explored in numerous existing studies.
Questions for authors
All references cited in the paper date from 2024 or earlier; have the authors surveyed literature from the last two years—is there truly no relevant work from this period? Furthermore, have the authors read the paper "Personalized Distillation"? If so, why is there no comparison with the approach proposed in that work? Removing the counterfactual tests and the "failure cases" analysis resulted in success rate decreases of 1.46% and 1.22%, respectively; however, given that the p-values for the clustering tests are 0.085 and 0.120, can the independent contribution of the core modules really be established? Additionally, why are all experimental data presented exclusively in tables rather than through more intuitive formats, such as line graphs?
2 comments
Thank you for the additional experiments and clarifications, which substantially address my concerns about the component contributions and the relationship to prior work. I will raise my rating to 3 (Weak acceptance), while noting the remaining limitations in training-stage attribution and compute matching.
Dear Reviewer k6JL,
Thank you for requesting a clearer technical contribution, fuller related work, and an easier-to-read account of what the experiments establish. The revision adds 7,305 SWE, 16,400 CF×graph, and 16,400 filtering rows.
1. Contribution and related work. Test-to-Teach closes an execution-to-training feedback loop: the current student generates a candidate; counterfactual probes and the failure graph expose its behavior; a teacher proposes a repair; a fixed visible-suite non-regression gate admits the trace; and acceptance-associated evidence weights the next student update. Individual ingredients have precedents. The distinct learning signal is that the same student's execution failures determine both the teacher's repair evidence and its subsequent training examples. Section 9 now discusses Personalized Distillation, PLUM, CodeRL, LeDex, CodeT, MT-LAPR, TestGen-LLM, SWE-agent, and Agentless by their actual overlap with this mechanism.
2. Main results and training interventions. We keep held-out scoring and the statistical unit visible in the reply:
HumanEval+ training variants share 12,000 samples/2.4M tokens; choices precede checkpoint freezing. Primary T2T-minus-extra-call run differences are +25, +21, +29, +23, +27/820 (two-sided sign p=0.0625), distinct from task-slot pairing. The HE-train-v1 isolation audit reports 0 confirmed overlaps across 420 source and 164 evaluation tasks.
The main HumanEval+ comparison records public passes 778/820 vs 752/820 and hidden passes 733/820 vs 708/820. Conditional hidden success is 733/778 vs 708/752; the clear result is +25/820 hidden passes, not a large change in conditional overfitting.
3. Component attribution and adapted methods. We added a four-arm CF×graph matrix at each of five fixed checkpoints on the same 820 slots:
With the graph present, CF adds 12.0/820 (+1.46 pp) on average; with CF present, the graph adds 9.2/820 (+1.12 pp). Each conditional gain is positive in all five runs. Interaction counts are 0, +3, +2, +1, −2; we therefore make no positive-synergy claim. This fixed-checkpoint design answers a different question from the earlier separately trained Table 36.
The adapted SWE-bench controls now have five training seeds, 487 instances each:
All three record the same base model, source-task manifest hash, 350 source tasks, 12,000 samples, and 2.4M training tokens. Because T2T uses more actual calls, this is a training-volume-matched descriptive comparison, not strict inference-compute matching or a claim of paired statistical superiority.
SWE per-seed resolved triples (T2T / PLUM / LeDex) are 202/186/178, 206/191/183, 210/189/180, 203/194/186, 209/190/178. T2T leads in every run; the 2,435 outcomes are not independent training replicates.
4. Test semantics and reproducibility scope. Of 2,870 expanded HumanEval+ probe attempts, 1,558 execute; among those, 1,286 are semantically supported, 219 ambiguous, and 53 invalid. Five-seed retraining after excluding invalid probes from gate, IG, or both gives mean solved counts 734.2, 735.0, 736.0/820, versus 733.0/820 original. These are new checkpoints, distinct from cached replay. The corrected IG diagnostic has 669/820 hidden passes and is kept separate from the primary 733/820 result. Appendix G.4 maps the concrete example to official
HumanEval/7(filter_by_substring); a four-case ledger records all four operator types, written expected-output bases, repairs, visible-gate transitions, and hidden outcomes. The four cases cover HumanEval/7 (variable), /0 (boundary), /2 (dependency), and /3 (state). Each ledger reports 1/2 → 2/2 and a locked hidden pass; public replay checks I/O, while the audit measures population-level semantic support. Run-artifact paths are listed; the workbook does not embed those files. Single-pass transfer is kept separate from primary reference-free distillation.