- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies long-horizon recursive co-evlution between language model and their execution harnesses. the authors argue that harness-only optimization can exacerbate model-interface mismatch, while shared-trajectory updates and fixed routing rules may erode model harness compatibility and eventually lead to performance saturation. To address these issues, the authors propose CoEvo-RSI, which combines Behavioral-Change and Self-Judgement to allocate trajectories seperately to model and harness updates, with their relative influence adjusted through intervention feedback. Across 40 co-evolution rounds on four benchmarks, CoEvo-RSI continues to improve under equal rollout caps and ultimately outperforms seven adapted baselines. Overall, the paper tackles an important problem with a coherent method and extensive experiments. However, the reliability of its attribution mechanism, the additional computational overhead, and its generalization across models and real-world deployment settings warrant further investigation.
Critical strengths
-
The paper addresses an important and forward-looking research problem. The co-evolution of models and agent harnesses is a critical step toward Recursive Self-Improvement (RSI). Compared with optimizing either component in isolation, joint evolution can mitigate the model-harness mismatch that emerges over long optimization horizons, offering a promising path toward sustainable and autonomous self-improving agent systems.
-
The empirical evaluation is comprehensive. To demonstrate the effectiveness of CoEvo-RSI, the authors conduct 40 rounds of co-evolution across four benchmarks and compare their approach against seven adapted published systems. The evaluation is further supported by ablation studies providing evidence for the paper’s central claims from multiple perspectives.
Critical weaknesses
-
The convergence properties of the joint optimization process require further justification. CoEvo-RSI alternates between model post-training and autonomous harness evolution, but the paper does not sufficiently demonstrate that this coupled process consistently moves toward the Pareto frontier. Because the two components are interdepedent, updating either one changes the optimization landscape faced by the other. Consequently, local improvement may not translate into gains at the system level and could instead leads to oscillation, path dependence, or convergence to a suboptimal local optimum. The paper would benefit from a joint-objective analysis or more systematic experiments.
-
A commonly held view in industry is that capability gains introduced by a harness will eventually be internalized into the foundation model’s weights through long-term developing. Although the paper shows that some harness mechanisms become redundant as the model improves, it does not systematically examine whether the harness’s independent contribution continues to diminish with stronger backbones or longer training.
-
Appendix S extends the Terminal-Bench 2 experiment to 80 rounds and suggests that performance begins to plateau after approximately 60 rounds. However, this experiment covers only one benchmark and uses two random seeds, while the SearchQA curve in Figure 3 is still clearly improving at round 40. Longer-horizon results on the remaining benchmarks, together with a principled stopping criterion, would help establish whether performance genuinely converges.
-
Although the paper uses disjoint evolution and held-out task sets and reports additional hidden-set and transfer results, repeatedly updating both components on a fixed evolution pool may still cause them to co-adapt to a particular task distribution. Given that only a limited fraction of the in-domain gains is preserved under cross-domain transfer, broader evaluation on unseen tasks and out-of-distribution settings would strengthen the paper’s claims regarding generalization.
Questions for authors
Please check the section of critical weaknesses
All changes are within run-to-run noise except small coding gains after the Terminal-Bench 2 and SWE-bench Pro runs, so the rank-64 LoRA updates do not trade general capability for benchmark fit.
4. Patch generality audit. Two authors labelled every committed patch, pooled over five seeds (Cohen’s κ = 0.71). Task-family-agnostic, 171 of 195 on Terminal-Bench 2 (87.7%), 152 of 170 on SWE-bench Pro (89.4%), 128 of 155 on GAIA (82.6%) and 121 of 135 on SearchQA (89.6%). The rest target a task family, such as Makefile-specific log parsing; no patch targets an individual task. GAIA, with the smallest evolution split, has the most family-specific patches, consistent with its gap results above.
5. Mixed-pool co-evolution. One run on the union of the Terminal-Bench 2 and SWE-Gym pools (two seeds) reaches 43.1 on Terminal-Bench 2 and 54.7 on SWE-bench Pro, within 1.5 and 1.3 points of the single-benchmark runs. HELIX-A under the same mixed protocol reaches 38.2 and 51.1. A single co-evolved system can therefore serve two distributions at little cost.
Further out-of-distribution results, on tasks created after our runs finished, on MuSiQue and Bamboogle, with a newer Wikipedia index, under environment changes, and with a non-repeating task stream, are in our response to Reviewer AMRX, point 4.