- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark CoEvoBench that evaluated llm researchers on whether they could drive recursive improvement process and achieve further system gains through successive iterations. The benchmark requires the llms to act within a 24 hour period and select the best final model-harness pair. The authors evaluated 5 frontier models on the benchmark.
Critical strengths
- This paper proposes a benchmark that evaluates llm researchers on an important problem of whether they could drive recursive improvement process and achieve further system gains through successive iterations. This is an important issue to auto research.
- The writing and presentation of the paper is clear, helping readers to understand the main method and importance of the paper.
Critical weaknesses
- The initialization might be too weak to provide meaningful baseline reference. For example for the swebench verified experiments, opus 5 obtained results of 24.11%, where Instruct + Reference Harness is 29.25%. Although this is extensive gain compared to the 0.4% base, less than 30% is still low given model performances on the leaderboard of swebench verified.
Questions for authors
Are there any evidence showing that the benchmark actually evaluates llm researchers on recursive co-evolution?
5 comments
Response to Reviewer gQnR (Part 2 of 2)
Q2: About Whether the Benchmark Evaluates Recursive Co-Evolution.
Thank you for this question, which goes to the core of the paper. Yes. During development, the model and the harness advance in alternation and adapt to each other, and each round builds on the system and feedback of the previous round. This recursive co-evolution is exactly what CoEvoBench measures.
The model and the harness advance in alternation. Of the 75 Qwen3.5-9B runs, 73 both revise the harness and launch training. In one Opus run on AppWorld, the harness first lifts the system from 0 to 5.9, and training then raises it to 23.5, a gain three times that of the harness. The harness makes the environment usable, and training teaches the model to act well within it.
Harness revisions change the value of training. In the Astra AppWorld run shown in Figure 5, training first solves 5 more tasks than Base. After a harness revision, the same weights solve 4 fewer tasks than Base. Retraining on trajectories collected under the revised harness restores the gain to 4 more tasks. Cross-evaluating the systems with trained weights shows that this coupling holds generally.
| Systems with trained weights (mean) | Researcher harness | Reference harness |
|---|---|---|
| Instruct | 30.2 | 37.8 |
| Researcher weights | 19.8 | 2.6 |
The same reference harness lifts Instruct from 30.2 to 37.8 but drops the researchers' trained weights from 19.8 to 2.6, showing that trained weights work only under the harness they evolved with. Every further gain therefore requires deciding anew which component to change based on the current system, which is what recursive co-evolution means.
Inspired by your question, we added a section that compares the research decisions of strong and weak researchers, showing that CoEvoBench indeed measures this capability. Weaker researchers also diagnose failures, revise the harness, and train the model. The difference lies in whether results decide the next step, specifically in choosing a direction based on results, abandoning a route after negative results, and judging whether a new system is better on held-out tasks. Every round of recursive improvement depends on these three decisions, and the gap between the two tiers of researchers measures exactly this capability.
Revisions. Section 3 now defines recursive improvement, Table 2 in Section 5.2 adds the cross-evaluation, and new Section 5.4 with Table 3 compares the research decisions of strong and weak researchers.
We thank the reviewer again for recognizing the importance and the writing of the paper and for the constructive comments. We hope these additions further address your questions and are happy to discuss any further questions.
Thank You and Follow-up
Dear Reviewer gQnR,
Thank you for your thoughtful review and positive assessment. Your question on recursive co-evolution inspired the cross-evaluation of weights and harnesses and a new section comparing the research decisions of strong and weak researchers, and your comment on the initialization helped us clarify the 9B setting.
We would sincerely appreciate any further questions or suggestions you may have, and would be happy to provide additional clarification if helpful.
We hope these additions further strengthen your confidence in our submission.
Thank you again for your time and consideration.
With sincere appreciation, The Authors
Response to Reviewer gQnR (Part 1 of 2)
We thank the reviewer for the thoughtful and constructive review. We are encouraged that the reviewer finds that CoEvoBench targets an important problem, whether LLM researchers can drive a recursive improvement process and achieve further system gains through successive iterations, which is an important issue for automated research. We are also glad that the reviewer finds the writing and presentation clear, helping readers understand the main method and the importance of the paper. Your questions on the initialization and on evidence for recursive co-evolution inspired the new analyses below, and key manuscript revisions are highlighted in blue.
Q1: About the Initialization and the Baseline Reference.
Thank you for your attention to the baseline reference. The absolute scores are modest because we fix a 9B open target model so that the open-source community can reproduce the benchmark. Compared with a reference system of the same scale, researchers come close to or surpass it within a single day, so the measured gains are meaningful.
We limit the model size so that the community can run the benchmark. A full development run on the 9B target, including full fine-tuning, fits within 4 GPUs and 24 hours. Scaling the target further would multiply training cost and put reproduction out of reach for most teams. The full experimental code is available at https://anonymous.4open.science/r/CoEvoBench-1D48/.
Against the same-scale reference, the researchers' results are competitive. The high scores on the SWE-bench Verified leaderboard come from frontier models that are far larger. The same-scale reference is Qwen3.5-9B-Instruct with a reference harness, the Instruct + Reference Harness row in Table 1.
Within one day, Opus raises SWE-bench Verified from 0.4 to 24.1, more than 80% of the official post-trained model with an expert harness. Astra surpasses the reference on AppWorld and nearly matches it on BFCL-V4. On these tasks, a single day of research by the researcher already comes close to the Qwen team's full post-training pipeline with an expert harness.
Starting from Base with a weak harness leaves measurable headroom on both the model side and the harness side. Starting from Instruct with the reference harness, Telecom would begin at 98.9, the post-training headroom would be used up in advance, and the benchmark could not measure what researchers achieve on the model side. The best TB-Lite system currently reaches 24.4 against a reference of 46.5, so the benchmark is far from saturated and can keep separating stronger researchers.
Revisions. New Appendix D.4 explains the choice of the 9B target and the starting point and reads the scores against the same-scale reference system, and Section 4.2 reports how much of the gap to this reference each researcher closes.