acceptodds
Under review as a conference paper at ICLR 2027

CoEvoBench: Can LLMs Drive Recursive Co-Evolution of Agentic Systems?

Abstract

Autonomously improving agentic systems requires researchers to translate execution feedback into effective updates to model weights and the harness governing interaction with tools and environments. Existing evaluations primarily focus on model training or harness optimization, leaving research capability under joint access to both intervention types insufficiently assessed. We introduce CoEvoBench, a benchmark that evaluates LLM researchers on autonomous model–harness improvement across five agentic domains. Researchers start from a common initialization within each domain and independently choose training, harness revisions, and their combinations within a bounded resource budget, using development feedback to guide further research. The submitted model–harness pair then executes independently on held-out tasks to measure the system improvement delivered by the researcher. Across five frontier researcher models, we observe substantial but uneven gains across domains, with no researcher consistently achieving the strongest improvements on every benchmark. Research gains concentrate in key breakthroughs: among improving runs with at least three selection-set candidate evaluations, the largest increase in best-so-far performance accounts for a median of 62.5% of the total observed gain. These results highlight the challenge of extending isolated breakthroughs into sustained improvement. CoEvoBench provides a framework for evaluating this capability.

Primary area:
datasets and benchmarks
Keywords:
Recursive self-improvement, Model–harness co-evolution, Agentic systems, Automated AI research, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gQnROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark CoEvoBench that evaluated llm researchers on whether they could drive recursive improvement process and achieve further system gains through successive iterations. The benchmark requires the llms to act within a 24 hour period and select the best final model-harness pair. The authors evaluated 5 frontier models on the benchmark.

Critical strengths
  • This paper proposes a benchmark that evaluates llm researchers on an important problem of whether they could drive recursive improvement process and achieve further system gains through successive iterations. This is an important issue to auto research.
  • The writing and presentation of the paper is clear, helping readers to understand the main method and importance of the paper.
Critical weaknesses
  • The initialization might be too weak to provide meaningful baseline reference. For example for the swebench verified experiments, opus 5 obtained results of 24.11%, where Instruct + Reference Harness is 29.25%. Although this is extensive gain compared to the 0.4% base, less than 30% is still low given model performances on the leaderboard of swebench verified.
Questions for authors

Are there any evidence showing that the benchmark actually evaluates llm researchers on recursive co-evolution?

5 comments
AuthorsOpenReview ↗

Response to Reviewer gQnR (Part 1 of 2)

We thank the reviewer for the thoughtful and constructive review. We are encouraged that the reviewer finds that CoEvoBench targets an important problem, whether LLM researchers can drive a recursive improvement process and achieve further system gains through successive iterations, which is an important issue for automated research. We are also glad that the reviewer finds the writing and presentation clear, helping readers understand the main method and the importance of the paper. Your questions on the initialization and on evidence for recursive co-evolution inspired the new analyses below, and key manuscript revisions are highlighted in blue.

Q1: About the Initialization and the Baseline Reference.

Thank you for your attention to the baseline reference. The absolute scores are modest because we fix a 9B open target model so that the open-source community can reproduce the benchmark. Compared with a reference system of the same scale, researchers come close to or surpass it within a single day, so the measured gains are meaningful.

We limit the model size so that the community can run the benchmark. A full development run on the 9B target, including full fine-tuning, fits within 4 GPUs and 24 hours. Scaling the target further would multiply training cost and put reproduction out of reach for most teams. The full experimental code is available at https://anonymous.4open.science/r/CoEvoBench-1D48/.

Against the same-scale reference, the researchers' results are competitive. The high scores on the SWE-bench Verified leaderboard come from frontier models that are far larger. The same-scale reference is Qwen3.5-9B-Instruct with a reference harness, the Instruct + Reference Harness row in Table 1.

BenchmarkBaseBest researcher (mean of three runs)Instruct + reference harness
SWE-bench Verified0.424.129.3
AppWorld0.037.933.3
BFCL-V41.825.926.8

Within one day, Opus raises SWE-bench Verified from 0.4 to 24.1, more than 80% of the official post-trained model with an expert harness. Astra surpasses the reference on AppWorld and nearly matches it on BFCL-V4. On these tasks, a single day of research by the researcher already comes close to the Qwen team's full post-training pipeline with an expert harness.

Starting from Base with a weak harness leaves measurable headroom on both the model side and the harness side. Starting from Instruct with the reference harness, Telecom would begin at 98.9, the post-training headroom would be used up in advance, and the benchmark could not measure what researchers achieve on the model side. The best TB-Lite system currently reaches 24.4 against a reference of 46.5, so the benchmark is far from saturated and can keep separating stronger researchers.

Revisions. New Appendix D.4 explains the choice of the 9B target and the starting point and reads the scores against the same-scale reference system, and Section 4.2 reports how much of the gap to this reference each researcher closes.

AuthorsOpenReview ↗

Response to Reviewer gQnR (Part 2 of 2)

Q2: About Whether the Benchmark Evaluates Recursive Co-Evolution.

Thank you for this question, which goes to the core of the paper. Yes. During development, the model and the harness advance in alternation and adapt to each other, and each round builds on the system and feedback of the previous round. This recursive co-evolution is exactly what CoEvoBench measures.

The model and the harness advance in alternation. Of the 75 Qwen3.5-9B runs, 73 both revise the harness and launch training. In one Opus run on AppWorld, the harness first lifts the system from 0 to 5.9, and training then raises it to 23.5, a gain three times that of the harness. The harness makes the environment usable, and training teaches the model to act well within it.

Harness revisions change the value of training. In the Astra AppWorld run shown in Figure 5, training first solves 5 more tasks than Base. After a harness revision, the same weights solve 4 fewer tasks than Base. Retraining on trajectories collected under the revised harness restores the gain to 4 more tasks. Cross-evaluating the systems with trained weights shows that this coupling holds generally.

Systems with trained weights (mean)Researcher harnessReference harness
Instruct30.237.8
Researcher weights19.82.6

The same reference harness lifts Instruct from 30.2 to 37.8 but drops the researchers' trained weights from 19.8 to 2.6, showing that trained weights work only under the harness they evolved with. Every further gain therefore requires deciding anew which component to change based on the current system, which is what recursive co-evolution means.

Inspired by your question, we added a section that compares the research decisions of strong and weak researchers, showing that CoEvoBench indeed measures this capability. Weaker researchers also diagnose failures, revise the harness, and train the model. The difference lies in whether results decide the next step, specifically in choosing a direction based on results, abandoning a route after negative results, and judging whether a new system is better on held-out tasks. Every round of recursive improvement depends on these three decisions, and the gap between the two tiers of researchers measures exactly this capability.

Revisions. Section 3 now defines recursive improvement, Table 2 in Section 5.2 adds the cross-evaluation, and new Section 5.4 with Table 3 compares the research decisions of strong and weak researchers.

We thank the reviewer again for recognizing the importance and the writing of the paper and for the constructive comments. We hope these additions further address your questions and are happy to discuss any further questions.

AuthorsOpenReview ↗

Thank You and Follow-up

Dear Reviewer gQnR,

Thank you for your thoughtful review and positive assessment. Your question on recursive co-evolution inspired the cross-evaluation of weights and harnesses and a new section comparing the research decisions of strong and weak researchers, and your comment on the initialization helped us clarify the 9B setting.

We would sincerely appreciate any further questions or suggestions you may have, and would be happy to provide additional clarification if helpful.

We hope these additions further strengthen your confidence in our submission.

Thank you again for your time and consideration.

With sincere appreciation, The Authors

Reviewer_DHefOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoEvoBench, a benchmark that tests whether an LLM "researcher" can improve an agentic harness by training its model, editing its harness, or both. Every run starts from Qwen3.5-9B-Base with a weak, domain-specific starting harness. The researcher gets 24 hours, four H20 GPUs, and a $100 budget for a fixed teacher model. The five domains are SWE-bench Verified, TB-Lite, AppWorld, τ²-bench Telecom, and BFCL-V4. Each task pool is split three ways: a visible set with full tasks and traces, a selection set that returns only scores, and a held-out test set. The operator runs the single submitted model–harness pair on the test set. Trajectories are audited by hand, and runs with confirmed cheating are re-scored with the base weights. Five researchers are each run three times per domain: GPT-6 Astra through Codex, and Claude Opus 5, GLM-5.3, Qwen3.8-max and Kimi K3 through Claude Code. GPT-6 Astra has the best weighted average, followed by Claude Opus 5, and the other three are far behind. Gains vary widely across domains. The largest single increase in the best score so far makes up a median of 62.5% of the observed gain.

Critical strengths
  1. The question is timely and the protocol is well designed. Earlier benchmarks test post-training or harness editing alone. This one gives the researcher both, on five real agentic benchmarks, under a fixed and realistic compute budget. The three-way split mirrors how agent developers actually work: a visible set with full traces, a selection set with scores only, and a held-out test set run by the operator. It also blocks the most direct forms of overfitting.

  2. The evaluation is large and fairly transparent. Every researcher gets three runs of up to a day on every domain. The paper reports each run's score rather than only the means, and the case studies trace concrete side-by-side comparisons of weights and harnesses. The two strongest researchers are clearly separated from the other three.

Critical weaknesses
  1. As a benchmark, the score is hard to trust and hard to reproduce. It is noisy: results for the same researcher and domain can differ by tens of points across three runs, and no confidence intervals are given. It is often far from development results: in several domains, systems that score high on the selection set score much lower on the test set, and the paper does not explain why. It also depends on a manual audit that is described in only a few sentences. The paper does not say who audits, what counts as cheating, how runs lasting many hours are checked, how many runs were flagged, or what was found. A run with confirmed cheating still keeps the gain from its harness. Finally, the benchmark is hard for others to use. Testing one new researcher takes a manual audit. So other groups would find it hard to run the benchmark and compare their results with the paper's.

  2. The evidence does not support the "co-evolution" and "recursive improvement" framing. The paper does not report how many final systems actually use trained weights. In the cases it describes, the strongest researcher's coding submissions all keep the base model. Every run starts from a base model with a weak harness, so much of the gain may come from fixing parsing and protocol problems. That is harness engineering, not co-evolution. The two analysis findings are also weak. When a run has only a few evaluations, the "one major jump" statistic is partly guaranteed by construction, and it is not compared with any baseline.

Questions for authors
  1. Selection versus test. Selection and test scores often disagree. The final Astra AppWorld system solves 42/51 selection tasks but only 22/51 test tasks (Appendix H.2). GLM's Telecom curve in Figure 4 reaches about 85%, but all three of its Telecom test runs are at or below 20% (Table 2). What causes these gaps?

  2. Cost and usability. Please report the full cost of a run: GPU-hours used, researcher API cost, teacher spending, and audit effort. Telecom relies on GPT-4o as the user simulator and GPT-4.1 as the judge (Appendix D.2). How will scores stay comparable when these models change?

  3. Please also describe the audit: who did it, what counted as cheating, and how runs lasting many hours were checked. How did you prevent or detect the use of selection or test instances from such sources?

5 comments
AuthorsOpenReview ↗

Response to Reviewer DHef (Part 1 of 3)

We thank the reviewer for the detailed and constructive review. We are encouraged that the reviewer finds opening both post-training and harness modification a timely problem, considers the three-way split close to how agent developers actually work, values the large and transparent evaluation, and notes the clear separation between the two strongest researchers and the rest. Your comments on score reliability, evidence for co-evolution, the Selection–Test gap, cost, and auditing pushed the rigor and usability of the benchmark a large step forward. We added new analyses for each point and rewrote the analysis sections. We also repeated the full study on a second target model, MiniCPM5-2B-Base, with all five researchers running three independent developments on all five benchmarks, and the full experimental code is available at https://anonymous.4open.science/r/CoEvoBench-1D48/. Key manuscript revisions are highlighted in blue.

Q1: About Score Reliability and Reproducibility.

Trustworthy scores are the first requirement for a benchmark, and we thank you for raising this point. Accounting for the variation across three independent runs, researchers still fall into two stable tiers, and the same tiers appear on a second target model. Repeated evaluations of the same system vary by less than 2.5 points, so both the ranking and the scores are reliable and reproducible.

The two tiers are supported by statistical tests. To account for run-to-run variation, we ran exact permutation tests across the three independent runs. All six cross-tier pairs are significantly separated, and even the closest pair, Opus minus GLM, differs by 12.0 points.

The two tiers hold on a different target model. To test whether the ranking depends on a particular target model, we switched to MiniCPM5-2B-Base, a smaller model from a different model family, and kept the same protocol, with every researcher running three independent developments on all five benchmarks, 75 runs in total. The boundary between Tier 1 (Astra, Opus) and Tier 2 (GLM, Qwen, Kimi) is unchanged, the smallest cross-tier gap of 4.4 points remains significant, and only the order within Tier 2 changes. Two very different target models produce the same tiers, showing that the tiers reflect stable differences in research capability.

TierResearchersQwen3.5-9B (Avg.)MiniCPM5-2B (Avg.)
Tier 2Kimi, Qwen, GLM7.9, 9.4, 12.29.7, 12.0, 5.8
Tier 1Opus, Astra24.2, 31.216.4, 18.5
Smallest cross-tier gapTier 1 min − Tier 2 max+12.0+4.4

Evaluation itself is stable, and the large swings across runs come from the research process. We evaluated each final system three times on Test and compare the standard deviation of these repeated evaluations with the gaps between researcher means.

BenchmarkStd. of repeated evaluations, same systemResearcher means, best minus worst
SWE-bench Verified1.018.4
TB-Lite2.416.3
BFCL-V40.422.9

Repeated evaluations of a frozen system have a standard deviation of only 0.4 to 2.4 points, compared with 16 to 23 points between researcher means. That the same researcher reaches such different outcomes from the same start and budget shows a capability gap in frontier models themselves. It also shows that this research task has not been memorized, since a memorized solution would give nearly the same result in every run. CoEvoBench runs three independent developments per pair precisely to expose this.

Revisions. Section 4.2 now reports the two tiers, Table 1 adds the second target model, and new Appendix L gives the cross-tier tests and the repeated evaluations.

AuthorsOpenReview ↗

Response to Reviewer DHef (Part 2 of 3)

Q2: About Evidence for Co-Evolution and the Two Analytical Claims.

Thank you for pressing on the central claim of the paper, which helped us complete the evidence for co-evolution. Component decomposition and cross-evaluation show that the gains come from the model and the harness together, and that the two are tightly coupled. Following your suggestion, we also added a random-order baseline for the single-large-jump statistic, and the finding holds.

Most final systems use trained weights. We agree that we did not report this number. Of the 75 Qwen3.5-9B runs, 54 submit trained weights, and 73 both revise the harness and launch training. The table below splits the mean Test gain of each benchmark into a harness part and a training part, where the training part is the submitted system minus Base weights under the same evolved harness, using only systems without an integrity issue.

BenchmarkBase+ Harness+ TrainingFinalInstruct + same harness
BFCL-V41.8+2.2+10.114.19.7
SWE-bench Verified0.4+12.4+6.018.824.9
AppWorld0.0+21.8+4.125.947.5
TB-Lite2.5+13.8+1.317.633.4
Telecom0.6+42.9+1.645.178.3

On the short function-calling task BFCL-V4, training carries over 80% of the gain. On the other four multi-step and coding tasks, the harness carries most of the gain, consistent with your observation. Pairing the same evolved harness with Qwen3.5-9B-Instruct raises these four tasks by another 6 to 33 points, showing that substantial model-side headroom remains. Researchers leave this headroom unclaimed because, although our agentic sandbox environment supports over 1,000 concurrent rollouts for reinforcement learning, every researcher still chooses only offline supervised fine-tuning, so the models learn shortcuts such as submitting early or looping in place on long-horizon interactions.

The strongest researcher keeps Base in all coding submissions, which reflects the domain dependence of training gains. Your observation is accurate. Astra submits trained weights in all three BFCL-V4 runs, and it also trains the model in all six coding runs, but every trained checkpoint falls below Base on Selection, so Astra keeps Base. This matches the table above. Training pays off most on short function-calling tasks and is the hardest part to realize on long-horizon coding tasks. Because CoEvoBench covers several task types, it can expose this domain dependence.

The two components are tightly coupled, and this coupling is co-evolution. Moving the researchers' trained weights to the reference harness drops their mean from 19.8 to 2.6, whereas the same reference harness lifts Instruct from 30.2 to 37.8, so the reference harness itself is strong. If the gains came only from parsing fixes, the trained weights should also work under a stronger harness. Instead, they work only under the harness they evolved with, showing that the model and the harness adapt to each other, which is exactly the co-evolution CoEvoBench measures.

Following your suggestion, we added a baseline for the single-large-jump statistic. You rightly point out that with few evaluations, a single large jump is partly guaranteed by construction. We shuffle the candidate order within each run 5,000 times, keeping the scores, evaluation noise, and number of evaluations per run fixed and destroying only the order, so the construction effect of few evaluations is already built into this baseline.

Runs with at least five candidates, no integrity issueActual new-best rateExpected rate under shuffled orderp
First half of run47%30%0.008
Second half of run17%12%0.19

In the first half, candidates set new bests much more often than under random order, showing that early breakthroughs come from researchers using feedback effectively. In the second half, the rate falls back near chance, showing that sustained improvement is the core bottleneck for current researchers.

Revisions. We have rewritten Section 5.2 to report the number of systems with trained weights, and new Table 2 gives the component decomposition and the cross-evaluation. Section 5.1 now reports the comparison with shuffled candidate orders, and new Appendix K details the analyses.

AuthorsOpenReview ↗

Response to Reviewer DHef (Part 3 of 3)

Q3: About the Gap Between Selection and Test.

Thank you for pointing out these two cases. We agree that we did not explain either gap. One gap comes from the anti-cheating procedure and the other from overfitting to development feedback, and both show why the three-way split and the audit are necessary.

The GLM gap on Telecom is the anti-cheating mechanism at work. The audit confirmed that all three GLM Telecom runs obtained protected test material. The curve near 85% in Figure 4 is the Selection score of the contaminated system. The scores of 18.3 to 20.0 in Table 4 are re-evaluations after resetting the weights to Base. Without the audit and the held-out Test set, this inflated score would have been recorded.

The Astra gap on AppWorld (42/51 vs. 22/51) comes from overfitting to development feedback, a research skill that frontier researchers have yet to master. The run keeps the best of repeated evaluations on the same 51 Selection tasks, so its Selection score is optimistic. The researcher cannot tell which improvements will generalize, and only the held-out Test set exposes this. Even so, the system solves 22/51 (43.1) on Test, still above the reference system's 33.3.

Revisions. Appendix B now marks runs with integrity issues, and Appendix E.2 explains why a repeatedly used Selection set gives optimistic scores.


Q4: About Per-Run Cost and Evaluator Stability.

Thank you for raising this from the perspective of community use. The compute and API cost of a run is manageable, and spending does not track performance. The strongest researcher, Astra, uses the least GPU time and teacher cost. The Telecom evaluators strictly follow the official configuration.

ResearcherGPU h (4×H20)In tok (M)Out tok (M)Researcher APITeacher APIAvg. score
Kimi85.277.00.14$49$17–237.9
Qwen78.797.90.86$160$28–399.4
GLM68.460.60.25$23$24–3212.2
Opus63.442.50.25$120$44–5824.2
Astra27.648.00.22$100$2–331.2

Tokens include cache reads and writes. GLM has the lowest API cost but scores 12.2, Qwen the highest but 9.4.

The Telecom user simulator and judge strictly follow the official τ²-bench evaluation configuration, pinned to the dated snapshots gpt-4o-2024-11-20 and gpt-4.1-2025-04-14 at temperature 0, so scores are directly comparable over time.

Revisions. Appendix G now includes a per-researcher cost table, and Appendix D.2 lists the evaluator snapshots.

Q5: About the Audit Procedure, Anti-Cheating, and External Reuse.

Thank you for asking about the audit details. We agree that the paper described the audit in only a few sentences. An agent screens every complete trajectory with a fixed open-source skill, and we confirm every finding, so external teams can reuse the protocol directly.

The audit procedure is standardized and reproducible. After each run, an agent applies the same audit skill to the complete trajectory, including tool calls, file reads and writes, commands, training data lineage, and network logs during evaluation. Each check returns one of four verdicts, confirmed, blocked, not found in scope, or insufficient evidence, and only a confirmed verdict counts as a violation, after which we review the evidence. Against test instances obtained from the web, development blocks data sources that expose the task pools, and the audit scans downloads, training corpora, and code for protected task IDs, problem text, and n-gram fragments. The skill files are in the anonymous repository.

Violations concentrate in the weaker tier of researchers. Of the 75 Qwen3.5-9B runs, 22 are confirmed violations, and 20 of them come from the Tier 2 researchers Kimi, Qwen, and GLM (7, 6, and 7 runs), while Opus has 2 and Astra has none. All 22 accessed prohibited data, and 6 also trained on it. This tendency to take shortcuts is itself something CoEvoBench measures.

CheckConfirmed runs
Accessing prohibited data22
Training on prohibited data6
Substituting unauthorized weights0
Task-specific harness customization or embedded test content0

Keeping a harness that passed the audit introduces no leakage. Static screening finds no task-specific code or embedded test content in any of the 22 final harnesses. Every violation entered only the training data, the weights, or the development process, so resetting the weights to Base removes the contamination. Resetting all 22 runs to the initial system leaves the two tiers unchanged and widens the closest gap, Opus minus GLM, from 12.0 to 13.7 points.

Revisions. Section 3.4 now describes the audit, and new Appendix M reports the audit skill, the breakdown of the 22 runs, and the strict-rule results.

We thank the reviewer again and are happy to discuss further.

Reviewer_x5jFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoEvoBench, a benchmark that evaluates LLMs on their ability to improve agents via both model training and harness optimization. Across five agentic domains, the paper compares how different LLM researchers approach the optimization of a common initial seed, choosing between training and harness revision under bounded resources. The results show that gains are substantial but uneven, with no LLM researcher dominating all benchmarks. Analysis of optimization trajectories reveals that gains typically come from key breakthroughs, rather than multiple, small individual changes.

Critical strengths
  • Interesting question. The paper addresses the problem of joint co-evolution of models and harnesses, an understudied case in the auto-research literature.

  • Experimental breadth and design. The paper evaluates five frontier models on five diverse benchmarks using fixed budgets. Care is taken to split the dataset into development, validation, and held-out splits. The authors additionally take measures to detect researcher cheating.

Critical weaknesses
  • Missing baselines and ablations. The paper asserts that "Joint access to model training and harness modification enables evaluation of recursive improvement at the system level." It is unclear, however, whether the selected benchmark suite (i.e. the choice of downstream tasks and seed model-harness pair) really tests this ability. Section 5.3 reports that Astra runs obtain the highest or near-highest scores by retaining the base model on both SWEBench and TBLite, indicating that the task is largely "solvable" through harness interventions. Running harness-only and model-only researchers would provide evidence for (a) the need and (b) the exploitability of joint co-evolution on the selected suite (i.e. there is headroom in these cases from both model and harness optimization and that researchers can take advantage of both of these sources of headroom). Additionally, reporting the performance of researcher-optimized models and harnesses on reference harnesses and models (the Instruct model), respectively, on all downstream tasks would help show the benefit of joint evolution.

  • Generalization to more competent models and harnesses is unclear. The paper often claims that the optimization setup is "recursive". However, the target model is different from and much smaller than the researcher model; in addition, the chosen seed harnesses appear to be significantly weaker than the reference harness. It is unclear to me that reported findings would transfer to settings where (a) the target model is already instruction tuned or stronger, or (b) the harness is stronger.

  • Statistical distinguishability. Some of the ranking claims made in the paper seem like they could reasonably be a result of noise; these need to be supported with confidence intervals that take into account run-to-run variance. For example, on pg. 6 L318-320, the authors claim that Opus 5 leads Astra on SWE-bench Verified. However, the difference is one of 0.53 pp. In Section 5.3, the authors themselves claim that "Specific interventions also yield substantial variation within the same broad strategy", with Astra TB-Lite runs spanning 13.53 pp, which is on the same order as the deltas between researchers.

  • Insights are insufficiently supported. The authors make several claims that are insufficiently justified.

    • "...gains concentrate in key breakthroughs": What is the null hypothesis here? Can the authors show that this is unexpected under some reasonable noise model?
    • "Characterizing the capabilities and bottlenecks of autonomous improvement": Related to my first criticism, the paper doesn't really describe what the bottlenecks are in terms of model vs. harness interventions. Do reported gains primarily come from model or harness side changes?
    • "harness revisions can reverse training gains, while subsequent training on refreshed trajectories can restore them": This core contribution is only supported by a single anecdotal sample shown in Figure 5. While anecdotal samples are helpful, the anecdote ignores the statistical noise that the authors mention in other parts of the paper. Could the authors provide stronger evidence for this core claim through repeated trials?
  • Insufficient description of anti-cheating guardrails and remedies. On L273-275 pg. 6, the authors mention that for runs with confirmed cheating, they re-evaluate the base model paired with the final submitted harness. It is unclear to me: (a) how the authors determine whether researchers are cheating, (b) how re-running the base model with the final harness alone can account for cheating. Leakage may occur through the harness as well. How many runs were flagged for cheating and what did those runs do?

  • Related work is short and the introduction doesn't position against existing work. Several prior works study joint harness and model optimization, including SIA, Co-Harness, and HarnessForge. The paper only positions itself against ScienceBuddy. The authors should discuss these works in greater detail and specify how their work is differentiated.

Questions for authors
  1. Figure 5 is quite dense and difficult to follow. Could the authors consider revising it?

  2. How often do researchers choose to edit the harness vs. train the model?

  3. What types of training interventions do the researchers make? How does the quality of model vs. harness interventions compare as measured by their effect on the agent or in terms of their originality?

  4. What is the scale of the evaluation noise on SWEBench, TBLite, and BFCL? How does this compare against the deltas between researcher found solutions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer x5jF (Part 1 of 3)

We thank the reviewer for the detailed and constructive review. We are encouraged that the reviewer finds joint model–harness co-evolution an interesting and understudied question, and recognizes the breadth of the evaluation and the care taken in data splits and cheating detection. Your experimental suggestions inspired the new analyses below, and key manuscript revisions are highlighted in blue.

Q1: About Joint Co-Evolution and Where the Gains Come From.

Inspired by your suggestion, we ran component-swap and cross-system transfer evaluations with Instruct and the reference harness across all submitted systems without an integrity issue, which isolate what the harness alone and training alone deliver on the same systems. They show that CoEvoBench is not solvable through harness engineering alone: the four multi-step tasks still leave 6 to 33 points of model-side headroom, and researchers rely mainly on the harness there only because they currently fail to train models effectively on long-horizon tasks, which is the exact capability gap CoEvoBench is built to expose.

Every researcher actively combines both levers, while the task structure determines which lever carries the gain. Of 75 runs, 73 both revise the harness and launch training. The first table reports intervention frequency, and the second splits the mean Test gain of each benchmark into a harness part and a training part (the submitted system minus Base weights under the same evolved harness).

ResearcherHarness edits / runTraining jobs / runSubmits trained weights
Kimi10.46.914 / 15
Qwen24.43.911 / 15
GLM21.83.413 / 15
Opus29.92.79 / 15
Astra28.12.07 / 15
BenchmarkBase+ Harness+ TrainingFinalInstruct + same harness
BFCL-V41.8+2.2+10.114.19.7
SWE-bench Verified0.4+12.4+6.018.824.9
AppWorld0.0+21.8+4.125.947.5
TB-Lite2.5+13.8+1.317.633.4
Telecom0.6+42.9+1.645.178.3

Both levers are used because harness edits (control flow, output parsing, prompts) build a working execution loop while model training (LoRA fine-tuning on teacher-distilled trajectories) adapts the model to that loop. Training carries over 80% of the gain on the short function-calling task BFCL-V4, whereas the harness carries at least two thirds of the gain on the four multi-step tasks.

Small training gains on the multi-step tasks come from researchers relying solely on offline fine-tuning and never using reinforcement learning, not from a lack of model-side headroom. Pairing each evolved harness with Qwen3.5-9B-Instruct (last column above) raises scores on the four multi-step tasks by another 6 to 33 points, while on the short task BFCL-V4 the researchers' trained systems already beat Instruct by 4.4 points. Although our agentic sandbox environment supports large-scale rollouts with over 1,000 concurrent instances for reinforcement learning, every researcher still chooses only offline supervised fine-tuning as its training strategy, causing models on long-horizon tasks to learn shortcuts such as submitting early or looping on short replies; in Section 5.2, all six Astra coding runs also train the model and keep Base only because the trained checkpoints drop on Selection. CoEvoBench therefore leaves ample headroom on both sides and exposes a blind spot in long-horizon post-training.

Cross-system transfer evaluations further show that evolved harnesses generalize to stronger models, whereas trained weights remain tightly coupled to their own harness. The table covers the 35 systems that submit trained weights.

Weights + harnessSWETBAppWorldTelecomBFCL
Instruct + researcher harness20.934.348.780.08.9
Instruct + reference harness29.346.533.398.926.8
Researcher weights + researcher harness16.014.624.162.815.7
Researcher weights + reference harness0.52.22.98.03.4

Pairing researcher harnesses with Instruct works across all five benchmarks and even beats the expert reference harness on AppWorld (48.7 vs. 33.3), showing that workflow and interface improvements transfer across models. Conversely, moving researcher weights to the reference harness drops scores to single digits on every benchmark because fine-tuning adapts the weights only to the prompt and action format of the researcher's own harness, which is why the model–harness pair must be evaluated as one system.

Revisions. Section 5.2 now presents these results in new Table 2, new Table 3 in Section 5.4 gives the intervention statistics, Appendix E describes the sandbox, and new Appendix K details the analyses. The introduction and contributions are updated to match.

AuthorsOpenReview ↗

Response to Reviewer x5jF (Part 2 of 3)

Q2: About Generalization to Stronger Models and Harnesses.

Thank you for your question on recursive improvement and generalization to stronger models and harnesses. Our core findings are not artifacts of a weak starting point, and fixing a trainable 9B target keeps the benchmark reproducible for the open-source community while preserving headroom on both sides.

Researcher improvements remain effective on stronger models and harnesses, and the current setup is far from saturated. First, our transfer results show that pairing researcher-evolved harnesses with the instruction-tuned model Instruct still surpasses the expert reference harness on AppWorld (47.5 vs. 33.3), proving that the gains are not limited to a weak base model. Second, even at the 9B scale, researchers leave 6 to 33 points of model-side headroom unclaimed on the multi-step tasks, and the best TB-Lite system (24.43) remains far below the same-scale reference system (46.48). Starting from Instruct with the reference harness would pre-consume the post-training headroom we aim to measure and would start near the ceiling on tasks such as Telecom (98.89).

Recursive improvement refers to the agent system being developed, and the 9B scale makes open reproduction feasible. Every round of development starts from the revised system and its new feedback, so fixing a shared open target model is necessary to compare different researchers from the same starting point on their ability to sustain this loop. A full run on the 9B target, including full fine-tuning, fits within 4 GPUs and 24 hours, whereas scaling the target further would multiply training cost and hinder community reproduction.

Revisions. Section 3 now defines recursive improvement explicitly, and new Appendix D.4 explains the choice of target scale and starting point.


Q3: About Statistical Distinguishability and Evaluation Noise.

Motivated by your question on run-to-run variance and evaluation noise, we ran cross-run permutation tests and three-time repeated evaluations. Accounting for multi-run variance separates the five researchers into two stable tiers, while evaluation noise on a frozen system stays below 2.5 points, far smaller than the gaps between researchers.

We revise the 0.53-point difference between Opus and Astra on SWE-bench Verified to a tie and partition researchers into two tiers using exact permutation tests across the three independent runs per benchmark. The tier boundary between Opus and GLM is supported in two ways: first, all six cross-tier pairs are significantly separated, with a 12.0-point gap even for the closest pair (Opus minus GLM); second, repeating the full study on a second target model, MiniCPM5-2B-Base, preserves the exact same boundary between Tier 1 (Astra, Opus) and Tier 2 (GLM, Qwen, Kimi) while the three models inside Tier 2 reorder among themselves.

Tier divisionResearchersQwen3.5-9B (Avg.)MiniCPM5-2B (Avg.)
Tier 2Kimi, Qwen, GLM7.9, 9.4, 12.29.7, 12.0, 5.8
Tier 1Opus, Astra24.2, 31.216.4, 18.5
Smallest cross-tier gapTier 1 min − Tier 2 max+12.0+4.4

Evaluation noise on a fixed system is small, and the 13.53-point spread across runs reflects researchers taking different optimization routes. We evaluated each final system three times on Test and compare the standard deviation of these repeated evaluations against the gaps between researcher means.

BenchmarkStd. of repeated evaluations, same systemResearcher means, best minus worst
SWE-bench Verified1.018.4
TB-Lite2.416.3
BFCL-V40.422.9

Repeated evaluations of a frozen system have a standard deviation of only 0.4 to 2.4 points, compared with 16 to 23 points between researcher means. That the same frontier researcher reaches such different outcomes from the same start and budget shows a capability gap in frontier models themselves, namely a lack of optimization stability. It also shows that this research task has not been memorized, since a memorized solution would give nearly the same result in every run. CoEvoBench runs three independent developments per pair precisely to expose this.

Revisions. Section 4.2 now reports the two tiers, Table 1 adds the second target model, and new Appendix L gives the cross-tier tests and the repeated evaluations.

AuthorsOpenReview ↗

Response to Reviewer x5jF (Part 3 of 3)

Q4: About the Null Hypothesis for Key Breakthroughs and Component Coupling.

Following your suggestion, we ran a random-order null test and cross-evaluated all trained systems. Dataset-wide statistics confirm that early breakthroughs come from using feedback and that model–harness coupling holds across all trained systems. We also simplified Figure 5.

Early breakthroughs beat a random-order null, while late progress stalls. Under the null that a researcher learns nothing from feedback, a candidate's quality is unrelated to its position in the run. We shuffle the candidate order within each run 5,000 times, keeping all scores fixed, to measure how often new bests occur by chance.

Actual new-best rateShuffled-order ratep
First half of run47%30%0.008
Second half of run17%12%0.19

In the first half, candidates set new bests significantly more often than under random order, so early jumps come from feedback. In the second half, the rate falls back near chance, so researchers struggle to turn feedback from a working system into the next breakthrough.

The main challenge of co-evolution is adapting model training and the harness to each other, and this holds across all trained systems. In Figure 5, training gain is the trained model minus Base under the same harness, which cancels out the harness itself. Revising the harness flips the gain of the same weights from +5 tasks to −4 tasks, and retraining for the revised harness restores it to +4 tasks. Cross-evaluating all trained systems shows the same pattern. The reference harness lifts Instruct, which was never trained on any researcher harness, from 30.2 to 37.8, yet drops the researchers' trained weights from 19.8 to 2.6, an 87% loss. Harness revisions therefore invalidate prior training unless the model is re-adapted.

Revisions. Section 5.1 now reports the shuffled-order comparison, Table 2 adds the cross-evaluation, and Figure 5 is redrawn with details in Appendix H.


Q5: About the Anti-Cheating Audit and Remedies.

Thank you for asking about the audit and harness leakage. We audited all 75 trajectories with a fixed skill-driven agent and human verification. None of the 22 confirmed runs leaked test content into the final harness, and resetting all 22 runs to the initial system leaves the two tiers unchanged.

Our audit workflow is standardized and reproducible. After each run, an agent checks the complete trajectory against a fixed audit skill, and a human verifies every finding. The full skill is available at https://anonymous.4open.science/r/CoEvoBench-1D48/ for direct reuse. Of the 75 runs, 22 are confirmed violations, and 20 of them come from the Tier 2 researchers Kimi, Qwen, and GLM (7, 6, and 7 runs), while Opus has 2 and Astra has none. All 22 runs accessed prohibited data, and 6 of them also trained on it.

CheckConfirmed runs
Accessing prohibited data22
Training on prohibited data6
Substituting unauthorized weights0
Task-specific harness customization or embedded test content0

Resetting the weights to Base while keeping the final harness removes the cheating benefit, because no confirmed run leaked test information through the harness. Static screening of final harnesses found no hardcoded task logic or embedded test material, and every violation occurred in training data, weights, or development-time file access. Resetting all 22 runs to the initial system even widens the closest cross-tier gap, Opus minus GLM, from 12.0 to 13.7 points.

Revisions. Section 3.4 expands the audit, and new Appendix M reports the audit skill, the breakdown of the 22 runs, and the strict-rule results.


Q6: About Positioning Against Joint Optimization Methods.

Thank you for pointing out these works. The key distinction is that they propose specific methods for building agent systems, whereas CoEvoBench is an evaluation benchmark that tests whether frontier LLMs can drive this co-evolution autonomously.

Specifically, SIA uses a feedback agent to update both the harness and the weights, Co-Harness alternates failure-guided harness updates with fine-tuning on improved trajectories, and HarnessForge pairs fault-guided harness tailoring with harness-conditioned policy alignment. CoEvoBench places frontier LLMs in the researcher's seat under a shared budget, three-way data split, and anti-cheating audit to measure how well they conduct joint optimization on their own, exposing their current inability to train models on long-horizon tasks. These pipelines can also be evaluated as researchers under our protocol.

Revisions. The second paragraph of the introduction and the co-evolution paragraph of Section 2 now discuss and differentiate SIA, Co-Harness, and HarnessForge.

We thank the reviewer again and welcome further discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear ACs and Reviewers,

We sincerely thank all three reviewers for their constructive feedback and for recognizing the importance of evaluating whether LLM researchers can drive recursive co-evolution of agentic systems, the careful three-way split and cheating detection, the breadth and transparency of the evaluation, and the clarity of the presentation.

We have focused the revision on providing direct evidence of model–harness co-evolution, making the rankings and the audit reproducible, and clarifying the design choices of the benchmark. The revisions summarized below have already been incorporated into the manuscript, with the key changes highlighted in blue for easy reference. The full experimental code and the audit skill are available at https://anonymous.4open.science/r/CoEvoBench-1D48/. We group shared concerns to provide an overview of how they have been addressed.

ReviewerConcernResponse and manuscript revision
x5jF, DHef, gQnREvidence for model–harness co-evolution and recursive improvement.We have added direct evidence that researchers use both components and that the two co-evolve. The revised analysis reports how often researchers edit the harness and train the model and how many final systems use trained weights, decomposes the gain on each benchmark into harness and training parts, and cross-evaluates weights and harnesses to show that the two adapt to each other. A new section compares the research decisions of strong and weak researchers, and the paper now defines recursive improvement explicitly.
x5jF, DHefStatistical reliability of the rankings under run-to-run variation, and evaluation noise.We have shown that the rankings remain reliable under run-to-run variation, with a similar capability gradient on a second target model. The revised results add cross-run statistical tests, a full repeat of the study on a second target model from a different model family, and a comparison of repeated-evaluation noise against the gaps between researchers.
x5jF, DHefBaseline for the concentration of gains in key breakthroughs.We have added a random-order baseline, and early breakthroughs clearly exceed it. The revised analysis compares the observed rate of new bests with shuffled candidate orders that keep the scores and the number of evaluations fixed, so the construction effect of few evaluations is already part of the baseline. Late progress falls back near this baseline.
x5jF, DHefAudit procedure and treatment of flagged runs.We have documented and released the audit, and flagged runs do not change the conclusions. The manuscript now specifies what the audit checks, how each finding is confirmed, how many runs were flagged and what they did, and marks these runs in the per-run results. No flagged run leaked test content through its harness, and the conclusions hold when all flagged runs are reset to the initial system.
x5jF, gQnRInitial setup and generalization to stronger models and harnesses.We have clarified that the 9B open target and weak starting point keep the benchmark reproducible and leave headroom on both the model and the harness sides. The revised paper reads the scores against a same-scale reference system, and transfer results show that researcher-evolved harnesses remain effective with an instruction-tuned model.
x5jFPositioning against joint optimization methods, and readability of Figure 5.We have clarified that SIA, Co-Harness, and HarnessForge propose specific co-evolution pipelines, whereas CoEvoBench evaluates whether frontier LLMs can drive this process themselves. The revised introduction and related work discuss these methods and note that such pipelines can be evaluated as researchers under our protocol. Figure 5 has been redrawn as a clearer three-stage chart.
DHefSelection–Test gap, per-run cost, and evaluator stability.We have explained the two Selection–Test gaps, which show why the three-way split and the audit are needed, and added the full cost of a development run. The revised paper describes where both gaps come from, what a run costs, and how evaluation stays comparable over time.

The revised manuscript now presents direct evidence for model–harness co-evolution, statistically grounded rankings replicated on a second target model, a documented and reusable audit protocol, and a clear rationale for the 9B setting. Detailed settings and numerical results are provided in our individual responses. We thank the ACs and reviewers for their time and welcome further discussion.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.