- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ScholarEvolve, a training-free framework that uses research literature to guide the evolution of agent harnesses around a fixed language model. The framework organizes the harness into five functional modules: tool interfaces, context management, skills, memories, and agentic workflows. It analyzes execution failures to identify capability gaps, retrieves relevant papers, and groups them by mechanism to diversify candidate modifications. Research agents translate selected methods into implementation blueprints, while coding agents produce executable module variants. Candidates undergo individual evaluation and module crossover, with complete configurations evaluated to account for interactions and a paired-bootstrap criterion governing champion replacement. New publications can initiate further evolution cycles. Experiments on AppWorld and τ²-Bench Telecom show improved task performance, including gains on held-out AppWorld applications and tasks requiring broader API usage.
Critical strengths
The paper systematically turns research literature into harness improvements, broadening the knowledge sources available for automatic evolution.
Its modular search and mechanism-diversity design fit together well, and the paper explicitly validates the effect of combining them.
The method is tested on two challenging general tool-use benchmarks, which provides some evidence for the effectiveness of SCHOLAREVOLVE.
Critical weaknesses
The authors divide the harness into five modules and, when generating each candidate modification, replace only one module while keeping the other four fixed. They do not ensure that the modules are independent, so there is no guarantee that the other modules can remain unchanged when one module is replaced.
The genetic-algorithm-style evolutionary approach is reasonable, but the same overall idea has already been used in many prior works, including EvoFlow: Evolving Diverse Agentic Workflows On The Fly, AgentEvo: Cost-Aware Agentic Workflow Generation via Adaptive Multi-Stage Evolution, and EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms. In terms of the main pipeline, I do not see a clear advantage of this work over those methods.
The method is evaluated only on two general tool-use tasks. It is not tested on other benchmarks aimed at deep-research tasks, so the authors do not sufficiently demonstrate the generality of their method.
The authors do not include out-of-domain generalization experiments, so it is unclear whether the method is at risk of overfitting.
Questions for authors
If a failure comes from the environment—for example, a search API that keeps returning failures—how should the framework repair or adapt to that kind of failure?
Why does summing the individual gains of the modules to predict the combined effect suffice to “select a small number of candidates worth actually testing”? The authors do not explain this clearly.
The paper mainly evolves from various kinds of literature as its data source, yet the main experiments study AppWorld and τ²-Bench, two general tool-use tasks. Is this mismatched with the proposed method?
4 comments
Response to Reviewer p292 (Part 2)
[W2] The evolutionary approach resembles prior work (EvoFlow, AgentEvo, EvoAgent), with no clear advantage in the main pipeline.
Evolutionary search provides a foundation for automated agent design. ScholarEvolve advances this approach in three ways.
- Research-guided variation. Retrieved papers provide concrete mechanisms, while topic modeling spreads exploration across distinct approaches rather than repeated local edits.
- Broader execution design. The search modifies five executable harness modules and their persistent skill and memory artifacts, rather than focusing only on workflow structures or role configurations.
- Lifelong updates. New publications initiate further evolution rounds after earlier search cycles, introducing mechanisms that were not available in the original pool.
We test the contribution of literature with a matched no-paper modular ablation. The five modules, coding model, candidate count, DEV budget, crossover, and selection remain fixed. Only the source of mutation proposals changes from research papers to training failure trajectories. We also evaluate additional spending through equal-cost Meta Harness and the value of fresh literature through a frozen-pool comparison (Table 4). TEST scores average three runs.
[Table 4] Research guidance, additional search spending, and literature refresh on TEST TGC
| Control | Setting | ScholarEvolve: initial → final | Δ | Control: initial → final | Δ |
|---|---|---|---|---|---|
| Meta Harness at equal cost ($310) | Qwen3.5-27B, Normal | 69.0 → 81.4 | +12.4 | 69.0 → 76.4 | +7.4 |
| Meta Harness at equal cost ($310) | Qwen3.5-27B, Challenge | 49.6 → 63.6 | +14.0 | 49.6 → 59.2 | +9.6 |
| No-paper modular search | Qwen3.5-27B, Normal | 69.0 → 81.4 | +12.4 | 69.0 → 69.6 | +0.6 |
| No-paper modular search | Qwen3.5-27B, Challenge | 49.6 → 63.6 | +14.0 | 49.6 → 55.2 | +5.6 |
| No-paper modular search | GPT-5.4-mini, Normal | 67.1 → 72.4 | +5.3 | 67.1 → 69.6 | +2.5 |
| No-paper modular search | GPT-5.4-mini, Challenge | 46.1 → 55.6 | +9.5 | 46.1 → 50.4 | +4.3 |
| Frozen paper pool, lifelong round 3 | Qwen3.5-27B, Normal | 69.00 → 81.55 | +12.55 | 69.00 → 79.56 | +10.56 |
In lifelong evolution, both branches keep searching with the same budget. The branch that adds new papers reaches 81.55 TGC in round 3, compared with 79.56 when the initial paper pool remains fixed. Fresh literature adds 1.99 points beyond continued search alone (Table 4).
The matched no-paper ablation gives ScholarEvolve TGC advantages of 11.8 and 8.4 points for Qwen3.5-27B on Normal and Challenge, and 2.8 and 5.2 for GPT-5.4-mini (Table 4). The equal-cost Meta Harness baseline also remains behind ScholarEvolve. These controls isolate the value of research-guided mutations and show how new publications sustain improvement across rounds.
[W3&Q3] Evaluating research-guided evolution on deep-research tasks.
We evaluate ScholarEvolve on BrowseComp-Plus to test its applicability beyond general tool use. We evolve a Qwen3.5-27B harness using the 154 examples in EvoAgentBench's training split, with 100 for evolution and 54 for validation. The remaining 676 BrowseComp-Plus examples form the held-out TEST set. GPT-5.5 judges answer accuracy.
[Table 5] BrowseComp-Plus held-out TEST accuracy (%) with Qwen3.5-27B
| Agent | Original Harness | Meta Harness | ScholarEvolve |
|---|---|---|---|
| Accuracy | 27.1 | 25.8 | 31.4 |
ScholarEvolve reaches 31.4% accuracy, exceeding the initial harness at 27.1% by 4.3 points and Meta Harness at 25.8% by 5.6 points. The evolved modules apply research ideas to deep search. Context management uses belief-state tracking to record established and missing evidence. Episodic memory retrieves search procedures and outcomes from similar questions. The tool interface supports parallel retrieval to gather independent evidence. These mechanisms show how research-guided evolution adapts the harness to a different task domain.
Response to Reviewer p292 (Part 3)
[W4] Generalization beyond evolution tasks and source benchmarks.
We assess generalization at three levels. The BrowseComp-Plus experiment in [W3&Q3] tests the evolution procedure on deep-research tasks. Within AppWorld, we evaluate the same evolved harnesses on unseen applications and on tasks with broader API requirements. We also audit the papers behind selected modules to check for benchmark-specific sources.
- Unseen applications. Challenge includes Amazon and Gmail APIs that are absent from evolution and validation data. This tests the evolved harness on new application interfaces (Table 1 of the paper).
- Broader API requirements. We examine Challenge tasks whose reference solutions use more distinct APIs than any training reference solution. The training maximum is 12 (Table 4 of the paper).
- Paper provenance. We audit whether the source papers of selected harness modules were developed or evaluated on their target benchmarks.
We report the held-out AppWorld results and provenance analysis below.
[Table 6] Transfer to unseen applications and broader API requirements
| Evidence | Backbone | Initial harness | ScholarEvolve | Δ |
|---|---|---|---|---|
| Applications unseen during search, TGC | Qwen3.5-27B | 49.6 | 63.6 | +14.0 |
| Applications unseen during search, TGC | GPT-5.4-mini | 46.1 | 55.6 | +9.5 |
| Applications unseen during search, SGC | Qwen3.5-27B | 28.3 | 44.8 | +16.5 |
| Applications unseen during search, SGC | GPT-5.4-mini | 21.1 | 32.4 | +11.3 |
| Tasks needing more APIs than any training task, TGC | Qwen3.5-27B | 34.97 | 44.44 | +9.5 |
| Tasks needing more APIs than any training task, TGC | GPT-5.4-mini | 38.24 | 45.42 | +7.2 |
On AppWorld Challenge, Qwen3.5-27B gains 14.0 TGC and 16.5 SGC points, and GPT-5.4-mini gains 9.5 and 11.3 points (Table 6). For tasks with more than 12 reference APIs, the TGC gains are 9.5 and 7.2 points. The provenance audit finds no final module drawn from a paper developed or evaluated on the target benchmark. Together with BrowseComp-Plus in [W3&Q3], these results demonstrate transfer within AppWorld and the applicability of ScholarEvolve to a distinct deep-research workload.
[Q1] Adapting to environment failures.
We test resilience to tool failures by injecting HTTP 503 errors into Qwen3.5-27B AppWorld DEV. At each step, with probability p, the environment skips execution and returns an HTTP 503 response. Both harnesses receive the same fault sequence. Comparing their outcomes under these matched faults measures recovery robustness. We then explain how the evolution procedure can select mechanisms for persistent failures.
[Table 7] Robustness to injected HTTP 503 failures on AppWorld DEV
| p (per-step failure probability) | TGC: initial → final | Δ TGC | SGC: initial → final | Δ SGC |
|---|---|---|---|---|
| 0 | 70.2 → 82.5 | +12.3 | 45.6 → 64.9 | +19.3 |
| 0.1 | 70.2 → 78.9 | +8.7 | 47.4 → 55.3 | +7.9 |
| 0.3 | 69.3 → 71.1 | +1.8 | 42.1 → 42.1 | 0.0 |
At a 10% per-step fault rate, the evolved harness gains 8.7 TGC and 7.9 SGC points over the initial harness (Table 7). At a 30% fault rate, it remains at or above the initial harness on both metrics. The evolved execution mechanisms retain an advantage under injected failures.
The framework can adapt to persistent failures by including fault-exposed trajectories in evolution and validation. These failures guide literature retrieval toward retries, alternative tools, fallbacks, and response verification. Candidate recovery mechanisms are then evaluated under the same failure conditions. We will add the fault-injection experiment and this adaptation procedure to the paper.
We thank the reviewer again for the thoughtful feedback and would be happy to address any further questions.
Follow-up on Our Rebuttal
Dear Reviewer p292,
Thank you again for your careful evaluation of our work. We have provided detailed responses and additional experiments addressing the concerns raised in your reviews. As the discussion period is approaching its end, we would greatly appreciate it if you could review our rebuttal and let us know whether it addresses your concerns or if any further clarification would be helpful.
Thank you for your time and consideration!
Response to Reviewer p292 (Part 1)
We thank the reviewer for the detailed comments. We address module interactions, the contribution beyond prior evolutionary methods, generalization to deep-research tasks, and adaptation to tool failures.
[W1&Q2] How module interactions are handled and how additive screening selects combinations.
The five harness modules have defined interfaces, so each module can be changed and checked separately. Executable probes verify interface compatibility before evaluation. Because modules interact during execution, we measure complete configurations before selecting the next champion. Additive gains are used only to shortlist combinations for joint evaluation. We examine this design with three analyses.
Tables 1 through 3 report the results.
[Table 1] Leave one module out on the final Qwen3.5-27B AppWorld harness, TEST Normal (Δ vs. the final harness)
[Table 2] Predicted and measured gains of all combinations measured during search
[Table 3] The additive screen in Qwen3.5-27B AppWorld
The three analyses explain the design:
The roles of the three steps are clear. Isolated mutations produce comparable candidates, additive screening allocates the joint evaluation budget, and full-harness validation selects the champion. We will add these results to clarify the method.