acceptodds
Under review as a conference paper at ICLR 2027

CoEvo-RSI: A Recursive Self-Improvement Framework for Long-Horizon Model–Harness Co-Evolution

Abstract

LLM-based agents now often couple a foundation model with an execution harness that determines what the model observes and how its actions are carried out. Recent work has begun enabling agents to self-improve their own harnesses. However, evolving the harness alone can create a growing mismatch between the model and its execution interface. Recursive co-evolution addresses this asymmetry by updating both components iteratively, yet existing approaches struggle over long horizons in that feeding every trajectory to both updates introduces excessive noise and can cause collapse, while hand-crafted attribution rules become misaligned as the system evolves and cause performance to saturate. We introduce CoEvo-RSI, a Recursive Self-Improvement framework proposed for long-horizon model–harness co-evolution. CoEvo-RSI scores each task for a receiving component based on the measured effect of the counterpart's latest update and a failure attribution weighted by the outcomes of prior interventions. Across 40 rounds on four benchmarks, CoEvo-RSI continues to improve as several published allocation strategies plateau or decline, and consistently outperforms existing co-evolution and RSI methods, achieving state-of-the-art performance in recursive model–harness co-evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recursive self-improvement, Model--Harness Co-Evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AWCrOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies long-horizon recursive co-evlution between language model and their execution harnesses. the authors argue that harness-only optimization can exacerbate model-interface mismatch, while shared-trajectory updates and fixed routing rules may erode model harness compatibility and eventually lead to performance saturation. To address these issues, the authors propose CoEvo-RSI, which combines Behavioral-Change and Self-Judgement to allocate trajectories seperately to model and harness updates, with their relative influence adjusted through intervention feedback. Across 40 co-evolution rounds on four benchmarks, CoEvo-RSI continues to improve under equal rollout caps and ultimately outperforms seven adapted baselines. Overall, the paper tackles an important problem with a coherent method and extensive experiments. However, the reliability of its attribution mechanism, the additional computational overhead, and its generalization across models and real-world deployment settings warrant further investigation.

Critical strengths
  1. The paper addresses an important and forward-looking research problem. The co-evolution of models and agent harnesses is a critical step toward Recursive Self-Improvement (RSI). Compared with optimizing either component in isolation, joint evolution can mitigate the model-harness mismatch that emerges over long optimization horizons, offering a promising path toward sustainable and autonomous self-improving agent systems.

  2. The empirical evaluation is comprehensive. To demonstrate the effectiveness of CoEvo-RSI, the authors conduct 40 rounds of co-evolution across four benchmarks and compare their approach against seven adapted published systems. The evaluation is further supported by ablation studies providing evidence for the paper’s central claims from multiple perspectives.

Critical weaknesses
  1. The convergence properties of the joint optimization process require further justification. CoEvo-RSI alternates between model post-training and autonomous harness evolution, but the paper does not sufficiently demonstrate that this coupled process consistently moves toward the Pareto frontier. Because the two components are interdepedent, updating either one changes the optimization landscape faced by the other. Consequently, local improvement may not translate into gains at the system level and could instead leads to oscillation, path dependence, or convergence to a suboptimal local optimum. The paper would benefit from a joint-objective analysis or more systematic experiments.

  2. A commonly held view in industry is that capability gains introduced by a harness will eventually be internalized into the foundation model’s weights through long-term developing. Although the paper shows that some harness mechanisms become redundant as the model improves, it does not systematically examine whether the harness’s independent contribution continues to diminish with stronger backbones or longer training.

  3. Appendix S extends the Terminal-Bench 2 experiment to 80 rounds and suggests that performance begins to plateau after approximately 60 rounds. However, this experiment covers only one benchmark and uses two random seeds, while the SearchQA curve in Figure 3 is still clearly improving at round 40. Longer-horizon results on the remaining benchmarks, together with a principled stopping criterion, would help establish whether performance genuinely converges.

  4. Although the paper uses disjoint evolution and held-out task sets and reports additional hidden-set and transfer results, repeatedly updating both components on a fixed evolution pool may still cause them to co-adapt to a particular task distribution. Given that only a limited fraction of the in-domain gains is preserved under cross-domain transfer, broader evaluation on unseen tasks and out-of-distribution settings would strengthen the paper’s claims regarding generalization.

Questions for authors

Please check the section of critical weaknesses

Reviewer_icefOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to build an ultimate system of model-harness recursive self-improvement framework for long-horizon settings. This is a very ambitious setting.

As noted in the paper line 080, the core of their proposed CoEvo-RSI is "a trajectory-allocation mechanism that determines which experiences should drive each model and harness update as the two components evolve." My interpretation is a task selection mechanism to make the best use of historical experiences.

In section 2 METHOD, 2.1-2.5 documents their method which is evaluated on several benckmarks (terminal-bench2, swe-bench pro, GAIA, and searchQA.) The results demonstrarte CoEvo-RSI outperforms some selected baselines.

Critical strengths

This paper provides a lot of details of their proposed method and related experiment results.

This paper is working on something not quite defined yet. Model-harness co-evolution has been shown to have strong potential and is very intuitive for recursive self-improvement.

This paper demonstrate decent empirical results on several recognized benchmarks compared with selected baselins.

Critical weaknesses

W1. This paper throws too many details (which is good to have) at readers without a clear presentation. Maybe this is ok for AI readers, but as a human reader, in the revision, I require to see a framework figure rather than describing the design choices in pure texts in section 2.

W2. Partially becuase of W1, I don't clearly see how this work compares with existing model-harness co-evolve paper conceptually. In the introduction and Appendix A.2, this paper lists some works (I'm not sure if the list is exhaustive) with a brief description, but does not differentiate the proposed works with those existing works (and some of the works can be noted as concurrent works). Additionally, the used harness is rarely introduced as well.

W3. My most serious concern is, if the model-harness co-evolution is able to generalize to some unseen tasks, instead of potentially overfitting the benchmark data distribution. Maybe a cross-check on co-evolving on Terminal bench 2 but test on swe bench might help addressing this?

W4. Following W3, however, different benchmarks have different initial harness, which are well-designed for the corresponding benchmark. How about the scenario when the initial harness is not very good? (or maybe even starting with zero harness?)

W5. The overall design sounds too engineering-focused: 2.1 - 2.5 seem loosely connected with each other. As the author mentioned themself that, many recent works have studied model-harness co-evolution, there isn't any new RSI paradigm introduced in this paper from my perspective.

W6. "Long horizon" seems to be missing in discussion. Is 40 rounds long enough? And, what's the actual context length during tasks in the experiments, is it long enough?

Questions for authors

Q1. In the main experiments, the model being used is Qwen3.5 35B. Recursive self-improvement has to be based on a strong model and I'm actually surprised a 35B model can do that with model-harness co-evolution. Could the author try smaller models to see when it does not work?

Q2. Curious what the evolved search-o1 is compared to the initial one, how the search harness can be improved? Can there be a summary table showing what's the final harness compared to initial harness, for all four benchmarks studied?

2 comments

All changes are within run-to-run noise except small coding gains after the Terminal-Bench 2 and SWE-bench Pro runs, so the rank-64 LoRA updates do not trade general capability for benchmark fit.

4. Patch generality audit. Two authors labelled every committed patch, pooled over five seeds (Cohen’s κ = 0.71). Task-family-agnostic, 171 of 195 on Terminal-Bench 2 (87.7%), 152 of 170 on SWE-bench Pro (89.4%), 128 of 155 on GAIA (82.6%) and 121 of 135 on SearchQA (89.6%). The rest target a task family, such as Makefile-specific log parsing; no patch targets an individual task. GAIA, with the smallest evolution split, has the most family-specific patches, consistent with its gap results above.

5. Mixed-pool co-evolution. One run on the union of the Terminal-Bench 2 and SWE-Gym pools (two seeds) reaches 43.1 on Terminal-Bench 2 and 54.7 on SWE-bench Pro, within 1.5 and 1.3 points of the single-benchmark runs. HELIX-A under the same mixed protocol reaches 38.2 and 51.1. A single co-evolved system can therefore serve two distributions at little cost.

Further out-of-distribution results, on tasks created after our runs finished, on MuSiQue and Bamboogle, with a newer Wikipedia index, under environment changes, and with a non-repeating task stream, are in our response to Reviewer AMRX, point 4.

W6. Is the horizon long enough, in rounds and in context?

The paper uses “long-horizon” for the number of alternating model and harness updates, not for episode length. We now says this in the introduction in the revision. We address this concerns from both readings below.

Rounds. Forty rounds is 12–14 GRPO iterations per round, about 480–560 policy updates in total (573k training rollouts per Terminal-Bench 2 run), plus 39 committed harness patches per run on average (App. T). The horizon is long relative to where alternatives stop improving. Baselines reach 95% of their maximum gain by rounds 11–28 (t₉₅, Table 10), and the collapses we observe occur at rounds 13–15. Native HELIX runs a single build–update–rebuild cycle; HarnessForge, HASE and SIA report 10, 15 and 20 rounds. The 80-round extension (App. S, Terminal-Bench 2, two seeds) shows what happens next:

Terminal-Bench 2Round 20Round 40Round 60Round 80Peak (round)
CoEvo-RSI38.944.646.947.347.7 (74)
HELIX-A38.140.140.640.241.3 (58)
HASE36.938.438.938.739.2 (63)
Entropy sampling (B6)36.040.041.541.241.8 (66)

CoEvo-RSI keeps improving to about round 60. The baselines gain at most 1.2 points after round 40. With a third seed, the Terminal-Bench 2 mean at round 80 is 47.1 ± 0.6 (drawdown 0.5). The other three benchmarks, now also run to 80 rounds, level off the same way. SWE-bench Pro goes from 56.0 to 58.6, GAIA from 43.5 to 45.6 and SearchQA from 56.3 to 57.6 (peaks 58.9, 45.9 and 57.9), while HELIX-A on SearchQA moves only from 53.0 to 53.2. A stopping rule that uses only the evolution pool halts every run within 0.8 points of its peak (response to Reviewer AMRX, point 3).

Context and episode length. Each episode allows a 32,768-token context, up to 8,192 new tokens per turn, up to 120,000 generated tokens per trajectory, and 30 / 40 / 20 / 12 steps on Terminal-Bench 2 / SWE-bench Pro / GAIA / SearchQA. When the context fills, the oldest interactions are dropped while the task and system context are kept (App. D). The measured lengths for round-40 CoEvo-RSI on the held-out sets are:

BenchmarkTurns per episode, median (p90)Generated tokens per episode, median (p90)Peak context, median (p90)Episodes hitting truncation, round 0 → 40
Terminal-Bench 214 (30)6.8k (22.9k)9.6k (30.1k)8.7% → 5.2%
SWE-bench Pro23 (40)11.4k (31.7k)17.8k (32.8k)16.9% → 11.3%
GAIA9 (20)5.1k (17.3k)8.2k (26.4k)6.4% → 3.1%
SearchQA6 (12)9.8k (24.6k)7.5k (19.9k)1.2% → 0.4%

Reasoning tokens from earlier turns are not kept in the context, so generated tokens can exceed peak context. At least one in ten episodes on the three agentic benchmarks uses the full step budget, and truncation falls on every benchmark over the run. Context management is part of the editable harness, and the evolver acts on it, in App. U, Case 1, broad log searches overflowed the context, and training on exactly those trajectories taught the policy to stop issuing them.

Gains by task length (Terminal-Bench 2, held-out tasks split into thirds by round-0 episode length):

Round-0 episode lengthTasksH₀HELIX-A, round 40CoEvo-RSI, round 40Margin
Short (≤ 10 turns)3041.355.058.7+3.7
Medium (11–22 turns)3021.740.045.4+5.4
Long (≥ 23 turns)297.824.829.3+4.5
All8923.840.144.6+4.5

The gain is not concentrated on short tasks. The margin over HELIX-A is largest on medium and long tasks, and success on long tasks nearly quadruples (7.8 → 29.3).

Context ablation (Terminal-Bench 2):

SettingHELIX-ACoEvo-RSIMargin
32k context, 30 steps (paper)40.144.6+4.5
Round-40 systems evaluated at 64k, 60 steps40.845.9+5.1
Round-40 systems evaluated at 128k, 60 steps40.946.2+5.3
Trained and evaluated at 64k, 60 steps (two seeds)41.7 ± 1.246.8 ± 1.0+5.1

A longer window helps both systems a little and widens the margin slightly, so the 32k default does not flatter the method. Qwen3.5 natively supports a 262K-token window, we used 32k because, in a pilot, rounds at 64k took 1.8× longer.

Reviewer_AMRXOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoEvo-RSI, a framework for long-horizon model–harness co-evolution that allocates tasks using Behavioral-Change from the counterpart’s latest update and Self-Judgment weighted by the outcomes of prior interventions.

Critical strengths
  • The paper addresses an important problem: deciding which experiences should drive model training and harness improvement as both components change. The proposed allocation mechanism provides a concrete way to coordinate these updates.
  • The empirical study is extensive. Evaluation over 40 rounds on four benchmarks, controlled comparisons of allocation rules, and intervention-based analysis provide useful evidence that task allocation matters for sustained improvement.
Critical weaknesses
  • This paper evaluates task allocation within an alternating model–harness update loop, but does not compare this loop with completing model training before harness optimization, or the reverse, under matched budgets. Checkpoint pairings do not address this because neither component is further optimized after pairing. These comparisons would help clarify when sustained alternation is beneficial.

  • The additional benefit of adaptive weighting is limited on the main backbone. Replaying the adaptive run’s weight schedule achieves 43.8 versus 44.6 for adaptive weighting, with no significant task-level difference. Although adaptation helps with a weaker model, the results on the main backbone do not clearly separate the benefit of ongoing feedback from that of a suitable weighting schedule.

Questions for authors
  • The checkpoint-pairing experiments show that performance depends on the model–harness combination. How does alternating optimization compare with training the model first and then optimizing its harness, or the reverse, under matched budgets? This would help distinguish the benefit of continued co-adaptation from improvements achievable through sequential optimization.

  • Table 13 provides a useful analysis of adaptive weighting, including its benefit with a weaker judge. On the main backbone, however, replaying the learned schedule performs similarly to ongoing adaptation. Does the advantage of feedback-based adjustment of β\beta also hold on the other benchmarks, where the judge’s reliability and the allocation signals may evolve differently?

  • Appendix N provides a useful counterfactual evaluation of Self-Judgment. Its labels are based on two single-task GRPO iterations versus one harness edit, while the actual loop uses multi-task updates and a harness regression gate. What evidence connects these labels to the relative benefit of the two actual updates? How sensitive is the attribution to the size of the model and harness interventions?

2 comments

Dear reviewer, please respond to the author's comment to further facilitate the discussion as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

Thank you for the additional experiments and clarifications. However, the posted response appears to address Reviewer AWCr’s concerns rather than the weaknesses and questions I raised.

I have read all the other reviews and the authors’ responses, but could not find a response that addresses these concerns. These concerns remain unresolved, and I consider them important for establishing the paper’s main contribution. I am therefore revising my score from 2 to 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.