- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoIn, a framework that combines harness evolution with model fine-tuning to improve agents' decision-making capabilities. Specifically, a stronger model (Claude Opus 4.7) analyzes failed trajectories and proposes improved decision-making procedures through harness evolution. The target model (Qwen3.5-35B-A3B) then generates successful trajectories under the evolved harness. These trajectories are rewritten by TAILOR to remove dependencies on the evolved harness and are subsequently used for supervised fine-tuning. Experiments demonstrate substantial improvements of 10.9 and 9.2 percentage points on in-domain and out-of-domain benchmarks, respectively.
Critical strengths
-
The paper addresses a meaningful question: how to internalize effective decision-making procedures discovered through harness evolution into the model itself, rather than relying on external instructions at inference time.
-
Significant empirical improvements. EvoIn achieves substantial performance gains across multiple benchmarks. The ablation studies also demonstrate the effectiveness of harness evolution and trajectory rewriting, particularly for out-of-domain generalization.
-
Clear presentation. The paper is well-written and easy to follow. The motivation and overall framework are clearly explained, and the case studies provide intuitive examples of the proposed approach.
Critical weaknesses
-
The idea of internalizing external strategies or reasoning procedures into model parameters has been extensively explored. Recent works such as HarnessForge[1] and Co-Harness[2] have also investigated harness evolution followed by model fine-tuning using trajectories generated under improved harnesses. Therefore, the main novelty of EvoIn appears to lie in the TAILOR trajectory rewriting stage. However, TAILOR not only removes harness-specific instructions but also repairs invalid tool calls, reasoning inconsistencies, and other trajectory quality issues. It remains unclear whether the observed improvements come from genuine procedure internalization or simply from higher-quality SFT data.
-
EvoIn relies on Claude Opus 4.7 to discover effective decision-making procedures, which are then expressed as harness instructions to guide trajectory generation by a weaker model. However, if Opus 4.7 already possesses strong agentic decision-making capabilities, why not directly use it to generate high-quality trajectories for distillation? The proposed pipeline introduces additional stages of harness evolution, trajectory generation, and rewriting, but the paper does not demonstrate whether these stages provide benefits over direct distillation from a strong teacher. This makes it difficult to assess the necessity of the proposed framework.
[1] Chen, Ming-Ju et al. “HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems.” [2] Chen, Zheng-Yu et al. “Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents.”
Questions for authors
What is the key methodological novelty of EvoIn compared with existing harness evolution and distillation methods? In particular, is TAILOR the main contribution? Could the authors provide more fine-grained ablations separating the effects of removing harness-specific dependencies from general trajectory cleaning and quality improvement?
5 comments
Response to reviewer EVdk (part 2)
Weakness 1 & Question (continued)
Our response to Reviewer nbkV reports case studies of how internalization changes the behavior of the model. For your convenience we reproduce two of them here, both on OOD benchmarks, where Base and EvoIn answer the same question under the same seed harness. If the model itself did not have this decision-making ability, it would have to rely on the harness to spell out the procedure for each task. Such decisions vary widely across tasks, and it is hard to summarize all of these behaviors into a single general harness. More case studies can be found in our response to Reviewer nbkV.
(1) Giving each stage a scope, and repairing a failed method. On Table-Longer, an out-of-domain benchmark, a 2,482-line table document is queried in two stages. The first stage collects the companies that have a project which is both delayed and of environmental grade 3, and the second keeps those of them that also have a safety project with expected ROI strictly above 20. The trap is that one qualifying safety project is recorded as completed rather than delayed, so the second condition has to be checked against all records of each candidate, not only the records that made it a candidate. Both models stated the two stages in their reasoning. Base then tested all conditions on single records and lost that company. EvoIn wrote the two stages into its notes with a scope for each. When its first parser returned no records, it re-read the raw lines, found the company name on the line after the record tag, and rewrote it. In the second stage it searched all records of the 15 candidates, which is how it recovered the company Base missed.
(2) Producing the missing labels instead of searching for them. Oolong, another out-of-domain benchmark, gives the model a 3,193-line log in which each line holds one question with its date and user id, but no class label. Asked whether "numeric value" is more or less common than "abbreviation" among one user's entries, the model must therefore classify those entries itself before it can count them. Base retrieved all lines of that user in its second turn, then spent 27 further turns re-running the same search and looking for a label field that does not exist, and its 30 turns ran out with no answer. EvoIn read the hits in segments and wrote one ledger entry per record for the 13 records, each with its line number and the class assigned to it, counted the two classes from the ledger, and answered at turn 10.
Response to reviewer EVdk (part 3)
Weakness 1 & Question (continued)
We also train on trajectories sampled from GLM-5.3 directly under the seed harness, which can be regarded as high-quality SFT data. With the same rollout model and the same training questions, EvoIn still outperforms this direct distillation on both ID and OOD.
| Method | ID Pass | ID Score | OOD Pass | OOD Score |
|---|---|---|---|---|
| Base + seed harness | 35.96 | 44.58 | 18.99 | 31.02 |
| Direct distillation from GLM-5.3 under the seed harness | 49.26 | 58.38 | 22.87 | 47.10 |
| EvoIn with GLM-5.3 rollouts | 52.96 | 61.83 | 24.56 | 48.26 |
Weakness 2: If Opus 4.7 already possesses strong agentic decision-making capabilities, why not directly use it to generate high-quality trajectories for distillation?
Answer: We thank the reviewer for raising this question. The Claude Opus 4.7 API returns only summarized thinking, so the complete reasoning process is not accessible and cannot be used directly for SFT. To address the reviewer's concern, we instead use GLM-5.3, whose full reasoning traces are available, to generate high-quality trajectories for distillation. Collecting GLM-5.3 trajectories under the evolved harness outperforms collecting them directly under the seed harness, which indicates that EvoIn is more effective than directly learning from high-quality trajectories. Besides the ID test set used in the evolution stage, the table also reports an independent ID test set of 2,300 held-out questions that we constructed in response to Reviewer nbkV and that was used neither in harness evolution nor in training, and the three settings keep the same ranking on both.
| Method | (2,300) | Held-out ID (2,300) | Gain on | Gain on held-out |
|---|---|---|---|---|
| Base + seed harness | 44.58 | 43.07 | — | — |
| Direct distillation from GLM-5.3 under the seed harness | 58.38 | 57.87 | +13.80 | +14.80 |
| EvoIn with Qwen rollouts | 54.41 | 52.75 | +9.83 | +9.68 |
| EvoIn with GLM-5.3 rollouts | 61.83 | 58.38 | +17.25 | +15.31 |
Moreover, EvoIn is not limited to knowledge transfer from a stronger teacher to a weaker student. Rather, our insight is that a model can discover better decision-making procedures, then internalize them to improve its own capabilities. Actually, we also evaluate EvoIn using the same model for both evolution and internalization. Specifically, as reported in Section 5 of our manuscript, we use Qwen3.5-35B-A3B to drive the EvoIn pipeline and fine-tune the same model, achieving performance gains over the baseline. From this perspective, EvoIn offers a potential path toward autonomous model improvement and motivates further exploration of recursive self-improvement.
[1] HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems, 2026.
[2] Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents, 2026.
[3] Rethinking the Evaluation of Harness Evolution for Agents, 2026.
Response to reviewer EVdk (part 1)
Thank you for the careful review and constructive suggestions. Both of your concerns ask where EvoIn's gains actually come from, which gives us a good opportunity to make this clear. Weakness 1 and the Question concern the same issue, so we answer them together.
Answer: We thank the reviewer for raising this question. We would like to clarify that EvoIn differs from existing harness–model co-evolution methods in its core motivation and corresponding methodological design. HarnessForge [1] and Co-Harness [2] jointly optimize models and harnesses to improve performance on target tasks. This optimization direction faces two generalization risks. First, models may over-adapt to the evolved harness, limiting the transfer of their gains to other harnesses [1]. Second, the system may increasingly exploit patterns specific to the target tasks without acquiring strategies that generalize to unseen tasks [3]. Unlike these works, our core insight is that models can discover and validate effective decision-making and reasoning strategies through harness evolution, then internalize the reusable logic to improve their own capabilities. Here, the harness actually provides a space for exploration, while the ultimate goal is to improve the model's own reasoning and decision-making abilities beyond a specific harness or task domain. Moreover, the OOD improvements reported in Table 2 of our manuscript further support our motivation.
To address your concerns about Tailor, we conduct a 2×2 ablation separating harness-dependency removal from general trajectory quality improvement. The four settings are no rewriting, dependency removal only, quality improvement only, and full Tailor. We use the same task set, training-set size, and training recipe across these settings, with results reported in the table below. Neither dependency removal alone nor quality improvement alone yields a statistically significant Pass improvement over no rewriting. However, full Tailor significantly outperforms all three ablated settings on OOD Pass by 4.94–6.40 percentage points. These results show that general trajectory cleaning alone is insufficient to explain the full method's performance. Removing either component from full Tailor significantly reduces OOD performance, supporting the contribution of both components when used together.