acceptodds
Withdrawn from ICLR 2027

TMT: Co-Evolving Models and Harnesses with Local Test Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The performance of a coding agent depends on both the model and the harness. A complete task rollout may succeed despite ineffective actions, or fail despite correctly implemented subtasks. We propose Task–Method–Test (TMT), a framework that uses test feedback to improve both components. TMT splits each original task into subtasks and, following test-driven development (TDD), generates corresponding tests for them. The tests are admitted and fixed before implementation. Each subtask is then addressed by one or more methods. Each method comprises one or more model responses and their associated tool calls. During rollouts on training tasks, TMT records these methods and their test outcomes. Records whose tests pass are used for model fine-tuning, including records from rollouts that fail whole-task evaluation. Records whose tests fail guide harness adaptation through aggregation, diagnosis, and targeted repair, even when their rollouts pass whole-task evaluation. We evaluate the first TMT update cycle, in which the model and harness are each updated once, using a Model × Harness matrix on SWE-bench to measure the individual and combined effects of these updates. We then assess the updated system on eight zero-shot coding benchmarks. The updated model–harness pair outperforms the vanilla system and the systems with only a model update or only a harness update in whole-task pass rate, while using fewer rollout tokens. The updated harness generalizes well to frozen external models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, large language models, model-harness co-evolution, local credit assignment, test-driven development

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J6pEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Task–Method–Test (TMT), a coding agent training framework that decomposes SWE tasks into subtasks and constructs, admits, and freezes local tests before implementation. Locally passing records supply selected actions for supervised fine-tuning, while failure records guide the updated model in diagnosing and repairing its harness. The authors evaluate one teacher-assisted update cycle through a model–harness factorial comparison, cross-task coding benchmarks, and harness transfer to frozen external models.

Critical strengths
  1. The fine-grained supervision design is reasonable and novel to me. Decomposing SWE tasks into independently verifiable subtasks allows the framework to retain successful work from globally unsuccessful trajectories and identify failed attempts within globally successful ones. This provides a potentially useful improvement over supervision based solely on whole-task outcomes.

  2. The evaluation separates model and harness contributions. The Model-Harness comparison and transfer experiments with frozen external models provide useful evidence about the individual effects and reusability of the updates.

Critical weaknesses
  1. Missing comparison or discussion with reinforcement learning. Training coding models using executable test feedback is closely related to RL with verifiable rewards, yet the paper provides neither an RL baseline nor a substantive discussion of this relationship. In particular, the proposed local tests could also provide process rewards to address sparse feedback in long trajectories. The authors should explain why TMT-SFT is preferable in their setting and discuss its expected advantages and limitations relative to RL. A controlled comparison using the same model, tasks, and explicitly reported compute budgets would strengthen the paper.

  2. Insufficiently specified benchmark protocols. Benchmarks such as MBPP and BigCodeBench are reasonable supplementary tests of cross-task transfer, but offer limited coverage of SWE agent capabilities. The paper does not adequately explain how these tasks are adapted to the agent setting: whether multi-turn tool use is allowed, what execution feedback is exposed, how final code is extracted, and how termination is determined. The initial 27B model solving only 1/50 MBPP tasks particularly warrants investigation of the adaptation and scoring pipeline. The version and selection procedure for the 170-task SWE-bench subset should also be specified.

  3. The inference-token savings lack sufficient analysis. Most of the aggregate zero-shot token reduction is driven by xCodeEval, where the joint system’s pass rate simultaneously decreases from 52% to 48%. The joint update also does not consistently reduce tokens across individual benchmarks. These aggregate results therefore do not distinguish more efficient problem solving from earlier termination or reduced exploration. Per-task token distributions, separate statistics for successful and failed tasks, tool-call and retry counts, and a precise token-accounting definition would help. Moreover, fewer supervised tokens do not directly imply proportionally lower training compute.

  4. Insufficient dataset reporting undermines the interpretability of the Raw-SFT comparison. The appendix reports 1,021 training windows for TMT-SFT but does not provide the corresponding sample count for Raw-SFT, the number of distinct source tasks represented in either corpus, or the number of retained Raw-SFT trajectories. Sharing the same pool of 1,000 source tasks does not ensure comparable training coverage: TMT can retain locally successful actions from globally failed rollouts, whereas Raw-SFT retains only globally successful trajectories. Moreover, the two pipelines generate different trajectories and use unmatched training-token budgets. Consequently, the comparison cannot distinguish gains from finer-grained supervision from those due to different task coverage, trajectory construction, or training volume. The authors should report these corpus statistics and provide controlled comparisons with matched task coverage and clearly specified training settings.

  5. Dependence on a stronger external teacher and unclear applicability to smaller models (minor concern). The first update cycle relies on a stronger external model for task decomposition, test construction, and rollout generation. The authors report that Qwen3.5-9B failed to complete the required steps and subsequently adopt Qwen3.6-27B, leaving the framework’s applicability to smaller models or the scalability unestablished.

Questions for authors

Please refer to weakness.

Reviewer_9oksOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes TMT, a framework for using local executable test feedback to update both a coding model and its surrounding harness. Instead of treating an entire software engineering trajectory as a single success or failure, TMT decomposes a task into subtasks, associates each subtask with an admitted and frozen test, and records the methods used between test executions. Passing records are used to select actions for LoRA fine-tuning, while failing records are used to group failures, diagnose harness defects, and propose harness repairs. The first update cycle is teacher-assisted: DeepSeek-V4-Flash-0731 performs task decomposition, test construction, and rollout collection, while Qwen3.6-27B is fine-tuned on selected local records and subsequently used for harness diagnosis and repair. The authors evaluate the model and harness updates using a Model × Harness matrix on SWE-bench, compare TMT-SFT with trajectory-level SFT, and test the updated system on several zero-shot coding benchmarks and two frozen external models.

Critical strengths

The paper addresses a real limitation of current coding-agent training pipelines. Whole-task evaluation is indeed too coarse for reliable credit assignment: a failed trajectory may contain useful partial work, while a successful trajectory may contain failed or redundant actions. The idea of using local executable evidence as a shared interface between model training and harness repair is conceptually clean and potentially useful.

The TMT record abstraction is clearly specified, and the separation between PASS records for model training and FAIL records for harness adaptation is easy to understand. The test admission and freezing procedure is also a sensible attempt to prevent test drift and to keep local supervision tied to a fixed code state.

Critical weaknesses

The main concern is that the paper’s strongest claims go beyond what is actually demonstrated. The experiments cover only one teacher-assisted update cycle. In this cycle, the external teacher performs the central task decomposition, test construction, and method rollout steps. The paper does not show that the updated student can independently generate valid subtask tests, pass test admission, collect its own TMT records, and drive a subsequent cycle. Thus, the current evidence supports a teacher-assisted data-generation and post-hoc harness-repair pipeline more directly than an autonomous or recursive co-evolution framework.

The interpretation of local PASS records as process supervision is also stronger than the evidence supports. A passing test validates the resulting code state, but it does not establish that every selected action in the preceding method sequence was correct, necessary, or causally responsible for the result. The proposed training data therefore appears to be outcome-conditioned action filtering rather than genuine action-level process supervision. The paper should provide evidence that the selected actions are better training targets than simpler alternatives.

The test construction procedure relies heavily on a strong teacher and private reference code. Although the reference implementation is hidden during solver rollouts, it plays a central role in constructing and admitting local tests. The paper does not report test admission rates, subtask statistics, test coverage, mutation scores, or the relationship between local test outcomes and final task correctness. Without such analysis, it is difficult to determine whether the observed improvements arise from TMT itself or from high-quality teacher-generated, reference-assisted supervision.

Questions for authors

In the first-cycle data collection, which model actually generates the method rollouts and code edits: the DeepSeek teacher, Qwen3.6-27B, or a combination of both? Please clarify the exact role of each model in task decomposition, test generation, implementation, and repair.

What are the test admission rate, fallback rate, average number of subtasks and tests per task, and PASS/FAIL record distributions? How do local test outcomes correlate with the final benchmark verifier?

Can the authors provide a matched-budget comparison between TMT-SFT and Raw-SFT, together with ablations separating action selection, loss masking, recovery examples, and local-test supervision?

Reviewer_CRSKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Task–Method–Test (TMT), a framework for jointly improving coding models and their harnesses through local test feedback. TMT decomposes coding tasks into verifiable subtasks and associates execution methods with test outcomes. Locally successful records are selected for model fine-tuning, while failed records guide harness adaptation through failure aggregation, diagnosis, and repair.

Critical strengths
  1. Fine-grained supervision. TMT addresses the limitations of trajectory-level supervision by preserving successful local actions from failed rollouts while avoiding direct supervision on failed attempts within successful rollouts.
  2. Unified use of execution feedback. The framework provides a coherent approach to leveraging local test outcomes for both model training and harness improvement.
Critical weaknesses
  1. Limited evidence for co-evolution. Although the paper emphasizes model–harness co-evolution, the experiments evaluate only a single sequential update cycle. The proposed framework's ability to support sustained, mutually reinforcing evolution remains unverified.
  2. Insufficient differentiation from prior work. The framework combines established techniques, including test-driven development, fine-grained supervision, SFT, and failure-driven harness optimization. Its methodological novelty relative to existing approaches such as SWE-Flow, HarnessForge, HELIX, and Self-Harness requires clearer justification.
  3. Incomplete ablation of TMT-SFT. While TMT-SFT demonstrates promising data efficiency, the comparison with Raw-SFT evaluates two complete data-construction pipelines with different training volumes and supervision targets. As acknowledged in Appendix B.2, the individual contributions of local verification, action selection, and loss masking remain unclear.
  4. Limited scope of harness adaptation. Although the framework describes seven harness components, the demonstrated update only modifies the verification component. This limits the evidence supporting the generality of the proposed harness evolution mechanism.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.