- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CoTrace, a framework for jointly improving terminal agents and their execution harnesses. Its main idea is to carefully manage how execution trajectories are used during co-evolution. Failed trajectories guide harness updates, while successful trajectories train the model under the adopted harness. The framework also refreshes training tasks and separately validates model and harness updates. Experiments on Tmax show improvements from 78 to 88 solved tasks with SFT and 90 with RL. Additional evaluations on Terminal-Bench 2.1 and SWE-bench Lite highlight the importance of harness compatibility.
Critical strengths
I think the focus on trajectory provenance is interesting and useful. The paper identifies an important issue in co-evolution: successful trajectories from different harnesses may not provide suitable training signals for the deployed model.
The experimental analysis is thoughtful. I particularly appreciate the component-wise evaluation, which helps distinguish improvements from harness updates and model training. The cross-harness experiments also provide useful insights into how runtime compatibility affects performance.
Critical weaknesses
I am not fully convinced that harness matching is the main reason for the observed improvements. The compared data recipes differ in several other aspects, including corpus size, task coverage, and fresh rollout generation. Without more controlled ablations, it is difficult to isolate the actual contribution of trajectory provenance.
The main results rely on the same 102-task validation split throughout the co-evolution process. Repeatedly selecting updates against this split could lead to overfitting. The external evaluations provide limited evidence of generalization, as the SFT model does not consistently outperform the baseline, while the RL improvements are relatively modest.
Questions for authors
Could the authors isolate the effect of harness matching while keeping corpus size, task coverage, and freshness fixed? This would help establish the central contribution.
Could the authors evaluate the final models on an independent held-out Tmax split? Results across multiple runs would also help establish whether the improvements are robust.
4 comments
Response to Reviewer hsZk (2/2)
W2: The main results rely on the same 102-task validation split throughout the co-evolution process. Repeatedly selecting updates against this split could lead to overfitting. The external evaluations provide limited evidence of generalization, as the SFT model does not consistently outperform the baseline, while the RL improvements are relatively modest.
A: The 102-task validation split has a specific role: comparing candidate updates against the incumbent on a fixed set of tasks. This avoids confounding component-wise promotion with changes in evaluation-task composition. Training and harness search draw on separate task pools. To independently assess the resulting policies, we added a frozen 300-task test outside both promotion and learning, with no further updates based on its outcomes.
| System | Pass rate | Gain over base; task 95% CI |
|---|---|---|
| Base pair | 74.7 | Reference |
| Early SFT θ1 + H2 | 75.6 | +0.9 [-2.3, +4.2] |
| RL iteration-3 weights + H3 | 80.3 | +5.7 [+2.7, +8.8] |
| Same RL weights + H0 | 81.6 | +6.9 [+3.9, +9.9] |
Each row averages three executions. A separate three-seed SFT experiment also improves by +4.2 pp over base weights under fixed Hmix (seed-and-task CI [+0.7, +7.8]).
The positive RL and controlled SFT comparisons demonstrate policy improvements on unseen tasks with evaluation runtime held fixed. They directly address whether the training recipe yields benefits beyond promotion-validation scores. The table separately identifies the earlier SFT pair and its interval.
The external results probe a further shift in domain and runtime: SFT is similar to the base on TB 2.1 and scores 35.0% versus 37.3% on SWE-bench Lite; RL scores 41.0% versus 37.3% (p = 0.16). The 10.3 pp recovery from changing only SFT's runtime (24.7% to 35.0%) illustrates the compatibility effect; the independent Tmax test establishes the learned policies' in-domain gains.
Q2: Could the authors evaluate the final models on an independent held-out Tmax split? Results across multiple runs would also help establish whether the improvements are robust.
A: We added 300 independent Tmax tasks, three execution repeats per fixed system, and three paired training seeds per controlled SFT arm.
The test continues the original sampling order, excluding the 102 promotion tasks and all 461 task IDs in project training/search/probe/pilot records. Of the next 400 eligible tasks, we retain the first 300 whose containers build and start, without using model outcomes. The manifest is frozen before evaluation, disjoint from the new SFT generation pools, and never used for promotion. All comparisons share greedy decoding, an 80-step cap, and 4,096 output tokens per turn.
We identify the tested configurations explicitly:
- SFT θ1 + H2 (Table 3): the policy promoted after iteration 1's SFT update, paired with the harness adopted at iteration 2's stage B.
- RL iteration-3 policy: the last promoted RL policy, since the fourth proposal was rejected (Table 8), evaluated with H3 and H0. The paper's 90/102 point pairs this policy with the later H4.
The RL pair scores 79.3/81.3/80.3% versus 74.3/72.3/77.3% for the base, giving +5.7 pp (task CI [+2.7, +8.8]). With H0 fixed, its weights gain +6.9 pp. The new controlled SFT models gain +4.2 pp under fixed Hmix (seed-and-task CI [+0.7, +7.8]).
These results establish independent-test policy gains and quantify execution repeatability and SFT training variation separately. We will label stage-specific configurations explicitly: the later H4 pairing and iteration-5 SFT policy are distinct from the systems in this test.
Thanks for providing the experiment results for RL gains on unseen tasks. However, it seems that the ablation gain for harness matching is merely +0.4 pp, with a CI of [−2.2, +3.1]. The +6.9 pp RL gain mostly resolves W2, but the final SFT system still lacks independent evaluation.
Response to Reviewer hsZk follow-up
Thank you for the follow-up. We would like to clarify the interpretation of the +0.4 pp matching contrast and provide an additional independent evaluation of the final SFT checkpoint.
1. On the +0.4 pp matching contrast.
The +0.4 pp figure is computed by averaging two runtime-specific matching contrasts, which differ in direction:
- Under H0: H0-trained models achieve 78.1%, compared with 76.7% for R2c1-trained models, yielding a +1.4 pp matching advantage.
- Under R2c1: R2c1-trained models achieve 76.0%, compared with 76.6% for H0-trained models, yielding a −0.6 pp contrast.
The resulting average is (+1.4 − 0.6)/2 = +0.4 pp. This average aggregates two heterogeneous effects and should not be interpreted as a universal measure of the benefit of harness-aware data selection.
Importantly, H0 and R2c1 share the same system prompt and tool schema, differing only in loop detection and lifecycle self-verification. The experiment therefore isolates provenance effects under a relatively narrow runtime variation, rather than evaluating the full range of harness differences encountered during co-evolution.
The broader CoTrace contribution concerns how experience is routed, verified, and used throughout harness–model co-evolution. The original mixed-sibling comparison examines this broader setting, while the controlled ablation measures a narrower provenance contrast. We will make this distinction clearer in the revised manuscript.
2. Independent evaluation of the final SFT checkpoint (θ5).
We have now additionally evaluated θ5, the final SFT checkpoint that achieved 88/102 solved tasks on the original validation split, on the frozen independent 300-task Tmax test.
It achieves 78.3% (235/300), compared with 74.7% for the base system (three-execution average), corresponding to an observed improvement of +3.6 percentage points.
Due to the limited discussion-period compute budget, this additional θ5 evaluation currently contains one execution, whereas the base reference averages three executions. We therefore present this as a preliminary held-out result rather than a statistically established gain. Nevertheless, it provides direct evidence on the previously unevaluated final SFT checkpoint, with a positive observed difference on tasks excluded from training and promotion. We will extend the repeated evaluation and report the corresponding uncertainty in the revised results.
Together with the independently verified RL improvement (+6.9 pp under fixed H0) and the three-seed controlled SFT improvement (+4.2 pp under fixed Hmix), this new result further supports the effectiveness of the CoTrace training recipe beyond the original promotion split.
We are incorporating these additional results, checkpoint correspondences, and more precise interpretations of the provenance ablation into the revised manuscript to make the experimental evidence and scope of our claims easier to follow.
Response to Reviewer hsZk (1/2)
Thank you for recognizing the value of provenance and component-wise analysis. We added the requested controlled ablation and independent evaluation.
A: The original experiment measures the benefit of the complete data recipe. To separately identify the provenance contrast, we added a task-paired ablation using the same generating policy, 50 common-success tasks, fresh collection campaign, one verified trajectory per task, identical per-task pair counts (556 per arm), serializer, LoRA settings, and paired seeds 42/43/44.
The intervention changes only the generating runtime: H0 versus R2c1. They share the same system prompt and tools; R2c1 changes loop detection and lifecycle self-verification. All six checkpoints are evaluated under both runtimes on the independent test.
Means average three training seeds. Matched data lead by +1.4 pp under H0, with seed-and-task CI [-2.4, +5.6]; one seed favors sibling data. The mean cross-runtime matching advantage is +0.4 pp (seed-and-task CI [-2.2, +3.1]).
These results provide the requested controlled provenance comparison, with a +1.4 pp matched-data estimate and its confidence interval. The compact H0-data recipe additionally improves over base weights by +4.2 pp under fixed Hmix (seed-and-task CI [+0.7, +7.8]). Together, they establish the practical training benefit and quantify the matching contrast within the shared prompt/tool interface.
A: Yes. We collected both corpora on the same 50 task IDs using the same policy in one fresh campaign, with no replay. Each contains one verified trajectory per task and 556 prompt/completion pairs with identical per-task counts. The serializer, LoRA settings, and seeds 42/43/44 are identical; completion-character counts are within 3.3%. The tasks are solved under both generating harnesses, preventing a between-arm coverage difference.
All six checkpoints are evaluated under both H0 and R2c1. Under H0, H0-data models score 80.7/76.0/77.7%, and sibling-data models score 77.0/76.7/76.3%: +1.4 pp for matched data (task CI [-1.3, +4.2]; seed-and-task CI [-2.4, +5.6]). The mean diagonal-minus-off-diagonal contrast is +0.4 pp (seed-and-task CI [-2.2, +3.1]).
This experiment controls the requested factors and measures a +1.4 pp matched-data advantage within a compatible runtime family, with uncertainty quantified above. The full cross-runtime table and per-seed results make the provenance comparison directly inspectable alongside the recipe's policy-training benefit.