acceptodds
Under review as a conference paper at ICLR 2027

Learning from Research: Toward Lifelong Agent Harness Evolution

Abstract

Language agents are expected to solve increasingly complex tasks, creating a growing need for continual improvement. One promising approach is to evolve the agent harness, the software that governs tool use, memory management, and task execution, while keeping the underlying language model fixed. Recent methods automate this process by using a meta coding agent to modify the harness based on execution feedback. However, relying on that agent's existing knowledge and observed failures can restrict exploration and make adaptation reactive. Inspired by how human experts learn from the research literature for new solutions, we introduce ScholarEvolve, a framework that automatically draws on state-of-the-art research to guide harness evolution. ScholarEvolve organizes the harness evolution directions into functional modules and uses topic modeling to identify distinct improvement strategies for each module. It implements these strategies and evaluates their combinations to improve task performance. Moreover, the framework is designed to incorporate new publications over time, allowing research advances to drive proactive lifelong evolution. Experiments demonstrate improvements on AppWorld and τ²-Bench. ScholarEvolve raises Qwen3.5-27B task goal completion from 49.6% to 63.6% on AppWorld Challenge, and raises GPT-5.4-mini pass¹ from 72.7% to 81.9% on τ²-Bench Telecom.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Harness Evolution, Automated Research

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_p292OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ScholarEvolve, a training-free framework that uses research literature to guide the evolution of agent harnesses around a fixed language model. The framework organizes the harness into five functional modules: tool interfaces, context management, skills, memories, and agentic workflows. It analyzes execution failures to identify capability gaps, retrieves relevant papers, and groups them by mechanism to diversify candidate modifications. Research agents translate selected methods into implementation blueprints, while coding agents produce executable module variants. Candidates undergo individual evaluation and module crossover, with complete configurations evaluated to account for interactions and a paired-bootstrap criterion governing champion replacement. New publications can initiate further evolution cycles. Experiments on AppWorld and τ²-Bench Telecom show improved task performance, including gains on held-out AppWorld applications and tasks requiring broader API usage.

Critical strengths

The paper systematically turns research literature into harness improvements, broadening the knowledge sources available for automatic evolution.

Its modular search and mechanism-diversity design fit together well, and the paper explicitly validates the effect of combining them.

The method is tested on two challenging general tool-use benchmarks, which provides some evidence for the effectiveness of SCHOLAREVOLVE.

Critical weaknesses

The authors divide the harness into five modules and, when generating each candidate modification, replace only one module while keeping the other four fixed. They do not ensure that the modules are independent, so there is no guarantee that the other modules can remain unchanged when one module is replaced.

The genetic-algorithm-style evolutionary approach is reasonable, but the same overall idea has already been used in many prior works, including EvoFlow: Evolving Diverse Agentic Workflows On The Fly, AgentEvo: Cost-Aware Agentic Workflow Generation via Adaptive Multi-Stage Evolution, and EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms. In terms of the main pipeline, I do not see a clear advantage of this work over those methods.

The method is evaluated only on two general tool-use tasks. It is not tested on other benchmarks aimed at deep-research tasks, so the authors do not sufficiently demonstrate the generality of their method.

The authors do not include out-of-domain generalization experiments, so it is unclear whether the method is at risk of overfitting.

Questions for authors

If a failure comes from the environment—for example, a search API that keeps returning failures—how should the framework repair or adapt to that kind of failure?

Why does summing the individual gains of the modules to predict the combined effect suffice to “select a small number of candidates worth actually testing”? The authors do not explain this clearly.

The paper mainly evolves from various kinds of literature as its data source, yet the main experiments study AppWorld and τ²-Bench, two general tool-use tasks. Is this mismatched with the proposed method?

4 comments
AuthorsOpenReview ↗

Response to Reviewer p292 (Part 1)

We thank the reviewer for the detailed comments. We address module interactions, the contribution beyond prior evolutionary methods, generalization to deep-research tasks, and adaptation to tool failures.

[W1&Q2] How module interactions are handled and how additive screening selects combinations.

The five harness modules have defined interfaces, so each module can be changed and checked separately. Executable probes verify interface compatibility before evaluation. Because modules interact during execution, we measure complete configurations before selecting the next champion. Additive gains are used only to shortlist combinations for joint evaluation. We examine this design with three analyses.

  1. Leave-one-out evaluation. We revert one module at a time in the final Qwen3.5-27B AppWorld harness while retaining the other four. TEST Normal performance over three runs measures each module's contribution within the complete system.
  2. Predicted versus measured gains. For every combination tested during search, we compare its actual DEV gain with the sum of its modules' standalone gains. The difference measures interactions that additive scoring cannot capture.
  3. Shortlist analysis. Qwen3.5-27B AppWorld has 3,125 possible configurations with five choices per module. We examine whether the additive score keeps promising combinations and excludes candidates with large individual regressions.

Tables 1 through 3 report the results.

[Table 1] Leave one module out on the final Qwen3.5-27B AppWorld harness, TEST Normal (Δ vs. the final harness)

Reverted moduleTGCΔ TGCSGCΔ SGC
None (final harness, paper)81.4—69.0—
Agentic workflows73.2−8.253.6−15.4
Context management76.8−4.660.7−8.3
Memories77.4−4.060.7−8.3
Skills77.8−3.662.5−6.5
Tool interface77.8−3.660.1−8.9
All five (initial harness, paper)69.0−12.448.8−20.2

[Table 2] Predicted and measured gains of all combinations measured during search

SettingMeasured onCombinationsPredicted gainMeasured gainMeasured − predicted
Qwen3.5-27B, AppWorldDEV4+40.4 to +42.1+11.1 to +17.0−30.4 to −23.4
GPT-5.4-mini, AppWorldDEV3+15.8 to +17.0−0.6 to +11.7−16.4 to −4.7
GPT-5.4-mini, AppWorldDEV2+17.3 to +17.5+2.7 to +11.1−14.6 to −6.4
Qwen3.5-27B, τ² TelecomVAL3+6.8+2.0 to +3.0−4.8 to −3.8
GPT-5.4-mini, τ² TelecomVAL4+35.1 to +37.2−48.6 to +17.6−83.7 to −19.2

[Table 3] The additive screen in Qwen3.5-27B AppWorld

QuantityValue
Configurations (5 options per module)3,125
Candidates individually worse than −30 DEV points3
Configurations containing at least one of them1,625 (52%)
Shortlisted combinations containing one of them0 of 4
DEV gain of the shortlisted combinations+11.1 to +17.0
Spearman correlation of additive score and measured gain, among the shortlisted−0.8

The three analyses explain the design:

  • All five modules contribute to the final harness. Reverting any one lowers TGC by 3.6 to 8.2 points and SGC by 6.5 to 15.4 points on TEST Normal (Table 1). These TGC losses sum to 24.0 points, nearly twice the final harness's 12.4-point gain over the initial harness. This confirms that module effects interact rather than simply add together.
  • The additive prediction overestimates every measured combination (Table 2). For GPT-5.4-mini on Telecom, one combination had a predicted gain of 35.1 points but an actual loss of 48.6 points. Two individually beneficial modules performed poorly together, and joint validation rejected the configuration. Measuring full harnesses is therefore essential for selection.
  • The additive score reduces the combinations that require costly joint evaluation. In Qwen3.5-27B AppWorld, 1,625 of the 3,125 possible configurations contain one of three individually harmful candidates, and none appears in the four shortlisted combinations (Table 3). All four shortlisted combinations improve on DEV. We use their measured gains to select the final harness.

The roles of the three steps are clear. Isolated mutations produce comparable candidates, additive screening allocates the joint evaluation budget, and full-harness validation selects the champion. We will add these results to clarify the method.

AuthorsOpenReview ↗

Response to Reviewer p292 (Part 2)

[W2] The evolutionary approach resembles prior work (EvoFlow, AgentEvo, EvoAgent), with no clear advantage in the main pipeline.

Evolutionary search provides a foundation for automated agent design. ScholarEvolve advances this approach in three ways.

  1. Research-guided variation. Retrieved papers provide concrete mechanisms, while topic modeling spreads exploration across distinct approaches rather than repeated local edits.
  2. Broader execution design. The search modifies five executable harness modules and their persistent skill and memory artifacts, rather than focusing only on workflow structures or role configurations.
  3. Lifelong updates. New publications initiate further evolution rounds after earlier search cycles, introducing mechanisms that were not available in the original pool.

We test the contribution of literature with a matched no-paper modular ablation. The five modules, coding model, candidate count, DEV budget, crossover, and selection remain fixed. Only the source of mutation proposals changes from research papers to training failure trajectories. We also evaluate additional spending through equal-cost Meta Harness and the value of fresh literature through a frozen-pool comparison (Table 4). TEST scores average three runs.

[Table 4] Research guidance, additional search spending, and literature refresh on TEST TGC

ControlSettingScholarEvolve: initial → finalΔControl: initial → finalΔ
Meta Harness at equal cost ($310)Qwen3.5-27B, Normal69.0 → 81.4+12.469.0 → 76.4+7.4
Meta Harness at equal cost ($310)Qwen3.5-27B, Challenge49.6 → 63.6+14.049.6 → 59.2+9.6
No-paper modular searchQwen3.5-27B, Normal69.0 → 81.4+12.469.0 → 69.6+0.6
No-paper modular searchQwen3.5-27B, Challenge49.6 → 63.6+14.049.6 → 55.2+5.6
No-paper modular searchGPT-5.4-mini, Normal67.1 → 72.4+5.367.1 → 69.6+2.5
No-paper modular searchGPT-5.4-mini, Challenge46.1 → 55.6+9.546.1 → 50.4+4.3
Frozen paper pool, lifelong round 3Qwen3.5-27B, Normal69.00 → 81.55+12.5569.00 → 79.56+10.56

In lifelong evolution, both branches keep searching with the same budget. The branch that adds new papers reaches 81.55 TGC in round 3, compared with 79.56 when the initial paper pool remains fixed. Fresh literature adds 1.99 points beyond continued search alone (Table 4).

The matched no-paper ablation gives ScholarEvolve TGC advantages of 11.8 and 8.4 points for Qwen3.5-27B on Normal and Challenge, and 2.8 and 5.2 for GPT-5.4-mini (Table 4). The equal-cost Meta Harness baseline also remains behind ScholarEvolve. These controls isolate the value of research-guided mutations and show how new publications sustain improvement across rounds.


[W3&Q3] Evaluating research-guided evolution on deep-research tasks.

We evaluate ScholarEvolve on BrowseComp-Plus to test its applicability beyond general tool use. We evolve a Qwen3.5-27B harness using the 154 examples in EvoAgentBench's training split, with 100 for evolution and 54 for validation. The remaining 676 BrowseComp-Plus examples form the held-out TEST set. GPT-5.5 judges answer accuracy.

[Table 5] BrowseComp-Plus held-out TEST accuracy (%) with Qwen3.5-27B

AgentOriginal HarnessMeta HarnessScholarEvolve
Accuracy27.125.831.4

ScholarEvolve reaches 31.4% accuracy, exceeding the initial harness at 27.1% by 4.3 points and Meta Harness at 25.8% by 5.6 points. The evolved modules apply research ideas to deep search. Context management uses belief-state tracking to record established and missing evidence. Episodic memory retrieves search procedures and outcomes from similar questions. The tool interface supports parallel retrieval to gather independent evidence. These mechanisms show how research-guided evolution adapts the harness to a different task domain.

AuthorsOpenReview ↗

Response to Reviewer p292 (Part 3)

[W4] Generalization beyond evolution tasks and source benchmarks.

We assess generalization at three levels. The BrowseComp-Plus experiment in [W3&Q3] tests the evolution procedure on deep-research tasks. Within AppWorld, we evaluate the same evolved harnesses on unseen applications and on tasks with broader API requirements. We also audit the papers behind selected modules to check for benchmark-specific sources.

  1. Unseen applications. Challenge includes Amazon and Gmail APIs that are absent from evolution and validation data. This tests the evolved harness on new application interfaces (Table 1 of the paper).
  2. Broader API requirements. We examine Challenge tasks whose reference solutions use more distinct APIs than any training reference solution. The training maximum is 12 (Table 4 of the paper).
  3. Paper provenance. We audit whether the source papers of selected harness modules were developed or evaluated on their target benchmarks.

We report the held-out AppWorld results and provenance analysis below.

[Table 6] Transfer to unseen applications and broader API requirements

EvidenceBackboneInitial harnessScholarEvolveΔ
Applications unseen during search, TGCQwen3.5-27B49.663.6+14.0
Applications unseen during search, TGCGPT-5.4-mini46.155.6+9.5
Applications unseen during search, SGCQwen3.5-27B28.344.8+16.5
Applications unseen during search, SGCGPT-5.4-mini21.132.4+11.3
Tasks needing more APIs than any training task, TGCQwen3.5-27B34.9744.44+9.5
Tasks needing more APIs than any training task, TGCGPT-5.4-mini38.2445.42+7.2

On AppWorld Challenge, Qwen3.5-27B gains 14.0 TGC and 16.5 SGC points, and GPT-5.4-mini gains 9.5 and 11.3 points (Table 6). For tasks with more than 12 reference APIs, the TGC gains are 9.5 and 7.2 points. The provenance audit finds no final module drawn from a paper developed or evaluated on the target benchmark. Together with BrowseComp-Plus in [W3&Q3], these results demonstrate transfer within AppWorld and the applicability of ScholarEvolve to a distinct deep-research workload.


[Q1] Adapting to environment failures.

We test resilience to tool failures by injecting HTTP 503 errors into Qwen3.5-27B AppWorld DEV. At each step, with probability p, the environment skips execution and returns an HTTP 503 response. Both harnesses receive the same fault sequence. Comparing their outcomes under these matched faults measures recovery robustness. We then explain how the evolution procedure can select mechanisms for persistent failures.

[Table 7] Robustness to injected HTTP 503 failures on AppWorld DEV

p (per-step failure probability)TGC: initial → finalΔ TGCSGC: initial → finalΔ SGC
070.2 → 82.5+12.345.6 → 64.9+19.3
0.170.2 → 78.9+8.747.4 → 55.3+7.9
0.369.3 → 71.1+1.842.1 → 42.10.0

At a 10% per-step fault rate, the evolved harness gains 8.7 TGC and 7.9 SGC points over the initial harness (Table 7). At a 30% fault rate, it remains at or above the initial harness on both metrics. The evolved execution mechanisms retain an advantage under injected failures.

The framework can adapt to persistent failures by including fault-exposed trajectories in evolution and validation. These failures guide literature retrieval toward retries, alternative tools, fallbacks, and response verification. Candidate recovery mechanisms are then evaluated under the same failure conditions. We will add the fault-injection experiment and this adaptation procedure to the paper.


We thank the reviewer again for the thoughtful feedback and would be happy to address any further questions.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer p292,

Thank you again for your careful evaluation of our work. We have provided detailed responses and additional experiments addressing the concerns raised in your reviews. As the discussion period is approaching its end, we would greatly appreciate it if you could review our rebuttal and let us know whether it addresses your concerns or if any further clarification would be helpful.

Thank you for your time and consideration!

Reviewer_3F47OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ScholarEvolve, a framework for continually improving LLM agents by evolving their harnesses without finetuning. Instead of relying solely on execution failures, ScholarEvolve retrieves research papers and organizes their methods into diverse topics using topic modeling across tool use, context management, skills, memory, and workflows. A coding agent implements research-inspired modifications, and validation guides systematic module selection and recombination. Experimental results on AppWorld and τ2\tau^2-Bench benchmark with Qwen3.5-27B and GPT-5.4-mini demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The main idea of improving agent harnesses by learning from research papers is intuitive and closely aligns with how human researchers develop new methods. Leveraging external scientific knowledge provides a promising alternative to evolution methods that rely primarily on execution feedback.
  2. The topic modeling mechanism helps organize retrieved papers into distinct methodological directions, reducing redundant exploration and encouraging diversity in candidate modifications. This structured exploration strategy is a meaningful contribution to automated harness evolution.
  3. Periodically incorporating newly published research papers to update harnesses is an interesting and practically valuable idea. It provides a potential pathway toward proactive and continuous improvement without repeatedly retraining the underlying language model.
Critical weaknesses
  1. It's a good work but my main concern is that since the experiments are conducted on AppWorld and τ2\tau^2-Bench, the retrieval process may identify papers that specifically propose methods or optimizations targeting these two benchmarks. Incorporating such methods could improve benchmark performance without necessarily providing generalizable improvements in realistic scenarios, where task-specific research papers may be unavailable. Although the paper adopts benchmark-agnostic query generation, it does not explicitly establish benchmark-related paper filtering. These two mechanisms are not equivalent, and the current approach may still introduce benchmark-specific knowledge into the evolution process. Consequently, the reported improvements could overestimate the effectiveness of ScholarEvolve in more general settings. An ablation study excluding papers that use the evaluation benchmarks during method development or evaluation would help clarify whether the observed gains arise from generalizable research knowledge or benchmark-specific optimization experience.
  2. Although ScholarEvolve achieves improvements on AppWorld and τ2\tau^2-Bench, it remains unclear whether the evolved harnesses preserve the agents' original general-purpose capabilities. Optimizing a harness for specific benchmarks may lead to over-specialization, potentially improving performance on the target benchmarks at the expense of other tasks. While the paper evaluates generalization to unseen applications within AppWorld, this does not fully establish general capability preservation across diverse task domains. Evaluating the evolved harnesses on additional, unrelated benchmarks would help determine whether the improvements reflect generalizable capability gains or benchmark-specific optimization.
Questions for authors
  1. Does the retrieval pipeline explicitly filter out papers related to the evaluation benchmarks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3F47 (Part 1)

We thank the reviewer for the careful assessment. We provide a paper provenance audit, a benchmark filtering analysis, and new evidence of generalization across applications and task domains.

[W1&Q1] Whether research retrieval introduces benchmark-specific methods.

The original retrieval process uses benchmark-agnostic queries rather than an explicit benchmark-name exclusion filter. We now examine the selected papers and replay selection with an explicit filter. These checks test whether the final harnesses depend on methods developed for AppWorld or τ²-Bench.

  1. Full-text provenance audit. We examined 69 setting–paper pairs across four main settings and three lifelong rounds. We checked each paper's full text for references to AppWorld or the τ-bench family. We classified each paper as (a) developed or evaluated on the target benchmark, (b) mentioning or citing it only, or (c) not mentioning it. For GPT-5.4-mini on AppWorld, the audit covers four evaluated paper sources, including all three sources in the selected final harness. This identifies whether a benchmark-specific method contributes to the final design.
  2. Filtered selection replay. We remove candidate papers whose titles or abstracts mention either benchmark, then repeat selection with the same topic and paper allocation rules. We compare the selected papers and resulting final modules. This tests whether the chosen research mechanisms change when benchmark-related papers are excluded.

Tables 1 through 3 report the results.

[Table 1] Full-text provenance audit of research papers across settings

SettingAudited papers(a)(b)(c)Paper modules in selected harnessesOf which (a) / (b) / (c)
Qwen3.5-27B / AppWorld20101950 / 0 / 5
GPT-5.4-mini / AppWorld400430 / 0 / 3
Qwen3.5-27B / τ² Telecom15011450 / 0 / 5
GPT-5.4-mini / τ² Telecom15011450 / 0 / 5
Lifelong, Qwen3.5-27B / AppWorld (rounds 1–3)15011460 / 1 / 5
Total691365240 / 1 / 23

[Table 2] Candidate-pool records removed by the explicit filter

PoolRecordsRemoved by the filterSelected papers among the removed
Qwen3.5-27B / AppWorld1,95622 (1.1%)1 (SAGE, not in the final harness)
τ² Telecom1,70722 (1.3%)0

[Table 3] Paper selection replayed on the filtered Qwen3.5-27B AppWorld pool

ItemAfter filtering
Topic clusters and their order (all five modules)Unchanged
Selected papers19 of 20 unchanged; SAGE is replaced by the next paper in its cluster
Final harnessDoes not use SAGE's candidate

The audit and filtering replay show that the selected harnesses draw their mechanisms from general research.

  • None of the 24 selected module-source assignments comes from a paper developed or evaluated on the corresponding target benchmark (Table 1). Of the 69 audited setting–paper pairs, 65 never mention the target benchmark.
  • The sole paper developed or evaluated on the target benchmark did not enter any evaluated module combination.
  • The filter removes just 22 of 1,956 AppWorld records and 22 of 1,707 Telecom records (Table 2). In the Qwen3.5-27B AppWorld replay, 19 of 20 selected papers stay the same. The final harness uses none of the filtered papers (Table 3).

These results show that ScholarEvolve benefits from research mechanisms beyond methods designed for the evaluation benchmarks. We will include the provenance audit and explicit filter in the paper.

AuthorsOpenReview ↗

Response to Reviewer 3F47 (Part 2)

[W2] Generalization to unseen applications and unrelated task domains.

ScholarEvolve keeps the backbone fixed and evolves a harness for the target environment. We test whether improvements extend beyond the tasks used during search. First, we evaluate the same AppWorld harnesses on unseen applications and tasks with broader API requirements. We also evaluate the evolution procedure on BrowseComp-Plus, a separate deep-research benchmark.

  1. Unseen applications. AppWorld Challenge requires Amazon or Gmail APIs that never appear in evolution or validation. This tests whether the evolved harness transfers to new application interfaces (Table 4).
  2. Broader API requirements. We group Challenge tasks by the number of distinct APIs used in their reference solutions. The maximum in training is 12. This tests whether gains persist on tasks with wider reference API usage than the training tasks (Table 5).

Both AppWorld analyses use the same harnesses frozen after DEV selection. Scores are averages over three TEST runs.

[Table 4] AppWorld Challenge results on applications unseen during search

BackboneTGC: initial → ScholarEvolveΔ TGCSGC: initial → ScholarEvolveΔ SGC
Qwen3.5-27B49.6 → 63.6+14.028.3 → 44.8+16.5
GPT-5.4-mini46.1 → 55.6+9.521.1 → 32.4+11.3

[Table 5] Challenge TGC by reference API breadth

BackboneReference APIsInitial harnessScholarEvolveΔ
Qwen3.5-27B≤ 963.6976.79+13.1
Qwen3.5-27B10–1243.5461.90+18.4
Qwen3.5-27B> 12 (more than any training task)34.9744.44+9.5
GPT-5.4-mini≤ 956.3561.71+5.4
GPT-5.4-mini10–1239.9155.78+15.9
GPT-5.4-mini> 12 (more than any training task)38.2445.42+7.2

The held-out results show transfer beyond the evolution tasks.

  • On unseen Challenge applications, Qwen3.5-27B gains 14.0 TGC and 16.5 SGC points, while GPT-5.4-mini gains 9.5 and 11.3 points (Table 4).
  • On Challenge tasks whose reference solutions use more than 12 distinct APIs, Qwen3.5-27B gains 9.5 TGC points and GPT-5.4-mini gains 7.2 (Table 5).
  • On BrowseComp-Plus, ScholarEvolve evolves a Qwen3.5-27B deep-research harness and raises accuracy from 27.1 to 31.4 on 676 held-out tasks, compared with 25.8 for Meta Harness. The paper audit also finds no selected final module developed or evaluated on its target benchmark (Table 1).

The same evolution procedure thus improves two distinct task domains, while the AppWorld harnesses transfer to unseen applications and broader API requirements. We will clarify these generalization results in the paper.


We thank the reviewer again for the thoughtful feedback and would be happy to address any further questions.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer 3F47,

Thank you again for your careful evaluation of our work. We have provided detailed responses and additional experiments addressing the concerns raised in your reviews. As the discussion period is approaching its end, we would greatly appreciate it if you could review our rebuttal and let us know whether it addresses your concerns or if any further clarification would be helpful.

Thank you for your time and consideration!

Reviewer_HYuWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ScholarEvolve uses research papers to guide executable changes to an agent harness while keeping the backbone model fixed. It organizes the search across five modules, uses topic grouping to select different research mechanisms, and evaluates combinations of the resulting module implementations. Experiments on AppWorld and τ²-Bench show gains over the initial harness and Meta Harness. A separate three-round experiment introduces papers from successive publication windows to study continued harness improvement.

Critical strengths

The gains on held-out tasks are substantial. On AppWorld Challenge, Qwen’s TGC rises from 49.6% to 63.6%, compared with 54.6% for Meta Harness. The gains extend to tasks involving Amazon and Gmail, which are absent from the evolution and validation data, providing evidence of transfer beyond the applications used during search.

The paper clearly separates the data used for evolution, validation, and testing. Persistent skills and memories are built from evolution data and frozen during validation and testing. In the three-round experiment, all search decisions are also frozen before test evaluation. These choices make the held-out results more convincing.

Critical weaknesses
  1. The cumulative ablations in Table 2 do not isolate the benefit of external papers within the full modular framework. It remains unclear how much research guidance contributes when modular mutation and recombination are retained.

  2. The three-round experiment does not separate the benefit of newly published papers from that of continued search. It therefore provides limited evidence for the specific role of fresh literature in lifelong adaptation.

  3. The computational cost of the improvements is unclear. The paper reports matched rollout budgets, but does not specify their accounting in enough detail or report actual optimization costs and per-task inference costs and latency.

Questions for authors
  1. How does the full modular search perform without external papers, while retaining recombination and matching the coding model, candidate count, and evaluation budget?

  2. Across the same three rounds, how does refreshing the literature compare with continuing to search using a fixed initial paper pool?

  3. Could you report the actual token usage or cost of the optimization process, together with the inference cost and latency of the initial and evolved harnesses?

6 comments
AuthorsOpenReview ↗

Response to Reviewer HYuW (Part 1)

We thank the reviewer for the detailed assessment. We provide a matched no-paper ablation, a literature-refresh control, and measurements of optimization and inference costs to address the concerns.

[W1&Q1] The contribution of research guidance under matched modular search.

The cumulative ablation in the paper changes several search components together. To measure the contribution of research guidance, we now compare ScholarEvolve with a version that keeps the full modular search procedure but proposes mutations without papers. We also extend Meta Harness to a comparable optimization cost to test whether additional search alone explains the gains.

  1. Matched no-paper modular search. Both methods use the same five modules, coding model, candidate count, DEV evaluation budget, crossover, and validation selection. The control replaces research-based mutation blueprints with proposals based on training trajectories and observed failures. This isolates the effect of research guidance while keeping the search procedure fixed.
  2. Equal-cost Meta Harness. We extend Meta Harness through additional search sessions until its estimated optimization cost reaches approximately $310, matching ScholarEvolve at API-equivalent rates (Table 1). This tests whether spending more on trajectory-driven search can close the performance gap.

We report three-run averages on held-out TEST. Table 2 compares the equal-cost searches. Table 3 compares the two modular searches with and without research papers.

[Table 1] Estimated search spending, Qwen3.5-27B AppWorld (OpenRouter API-equivalent rates)

ItemMeta Harness (paper)Meta Harness at equal costScholarEvolve
Coding agent≈ $16$92≈ $12
Research model and literature stage——a few dollars
Task-model execution≈ $70$218$295
Total≈ $86$310≈ $310
Rollouts (DEV + TRAIN)1155 + 3604,104 + 5404,617 + 90
Candidate harnesses evaluated on DEV42324

[Table 2] Equal-cost Meta Harness comparison on AppWorld TEST, Qwen3.5-27B

MethodSplitTGC: initial → finalΔ TGCSGC: initial → finalΔ SGC
Meta Harness (paper)Normal69.0 → 76.8+7.848.8 → 62.5+13.7
Meta Harness (paper)Challenge49.6 → 54.6+5.028.3 → 32.9+4.6
Meta Harness at equal costNormal69.0 → 76.4+7.448.8 → 56.5+7.7
Meta Harness at equal costChallenge49.6 → 59.2+9.628.3 → 39.8+11.5
ScholarEvolveNormal69.0 → 81.4+12.448.8 → 69.0+20.2
ScholarEvolve (paper)Challenge49.6 → 63.6+14.028.3 → 44.8+16.5
Final-score gap: ScholarEvolve − equal-cost Meta HarnessNormal+5.0+12.5
Final-score gap: ScholarEvolve − equal-cost Meta HarnessChallenge+4.4+5.0
AuthorsOpenReview ↗

Response to Reviewer HYuW (Part 2)

[Table 3] Modular search with and without research papers on AppWorld TEST

Backbone, splitMethodTGC: initial → finalΔ TGCSGC: initial → finalΔ SGC
Qwen3.5-27B, TEST NormalNo-paper modular search69.0 → 69.6+0.648.8 → 48.8+0.0
Qwen3.5-27B, TEST NormalScholarEvolve (paper)69.0 → 81.4+12.448.8 → 69.0+20.2
Qwen3.5-27B, TEST ChallengeNo-paper modular search49.6 → 55.2+5.628.3 → 34.5+6.2
Qwen3.5-27B, TEST ChallengeScholarEvolve (paper)49.6 → 63.6+14.028.3 → 44.8+16.5
GPT-5.4-mini, TEST NormalNo-paper modular search67.1 → 69.6+2.546.4 → 48.2+1.8
GPT-5.4-mini, TEST NormalScholarEvolve (paper)67.1 → 72.4+5.346.4 → 55.4+9.0
GPT-5.4-mini, TEST ChallengeNo-paper modular search46.1 → 50.4+4.321.1 → 25.2+4.1
GPT-5.4-mini, TEST ChallengeScholarEvolve (paper)46.1 → 55.6+9.521.1 → 32.4+11.3

The matched ablation isolates research guidance. The equal-cost baseline tests the effect of additional search spending.

  • With the modular search fixed, ScholarEvolve outperforms the no-paper control on both backbones and both TEST splits (Table 3). For Qwen3.5-27B, the TGC and SGC advantages are 11.8 and 20.2 points on Normal and 8.4 and 10.3 points on Challenge. For GPT-5.4-mini, they are 2.8 and 7.2 points on Normal and 5.2 and 7.2 points on Challenge. The no-paper control also raises TGC in all four settings. Modular evolution helps, and research-grounded proposals deliver much larger gains.
  • At approximately the same $310 optimization cost, Meta Harness remains below ScholarEvolve by 5.0 TGC and 12.5 SGC points on Normal, and 4.4 TGC and 5.0 SGC points on Challenge (Table 2). Increasing the budget of trajectory-driven search does not close the gap.

Together, these controls show the value of learning mutation mechanisms from papers within a fixed modular search, beyond gains from additional search spending. We will add both experiments and their protocols to the paper.


[W2&Q2] The benefit of refreshing literature across evolution rounds.

We compare two branches that share round 1 and use the same budget, prescreening, and promotion rule in later rounds. Streaming adds newly published papers before rounds 2 and 3. Frozen continues searching untried mechanisms from the round 1 paper pool. Since both branches keep searching, their difference measures the value of fresh literature.

Table 4 reports the TEST scores of each round's best candidate.

[Table 4] TEST Normal score of each round's best candidate (Streaming and Meta Harness from Table 5 of the paper)

RunMetricRound 0Round 1Round 2Round 3Δ (round 3 − round 0)
StreamingTGC69.0073.4179.7681.55+12.55
StreamingSGC48.8054.1761.9166.67+17.87
FrozenTGC69.0073.4168.8579.56+10.56
FrozenSGC48.8054.1745.2459.47+10.67
Meta Harness, same protocolTGC69.0071.0370.4469.64+0.64
Meta Harness, same protocolSGC48.8047.6251.1951.19+2.39

Refreshing the paper pool produces stronger candidates in both later rounds.

  • In round 2, Streaming reaches 79.76 TGC, compared with 68.85 for Frozen. In round 3, Streaming reaches 81.55 TGC and 66.67 SGC, compared with 79.56 and 59.47 for Frozen (Table 4). The final advantage is 1.99 TGC and 7.20 SGC points. Meta Harness shows little improvement over the same three rounds.

Fresh literature therefore drives additional improvement beyond continued search over the initial paper pool. We will add the frozen-pool comparison to the paper.

AuthorsOpenReview ↗

Response to Reviewer HYuW (Part 3)

[W3&Q3] Search budget, optimization cost, and inference cost.

We measure how the search budget is used and report the cost of optimization and inference. We also compare the evolved harness with repeated initial-harness executions to test whether extra inference alone explains its gains.

  1. Budget accounting. We count each evaluation episode toward the search budget. The cap is 4,617 DEV episodes for Qwen3.5-27B on AppWorld and 6,512 validation simulations for Telecom. Table 5 shows the AppWorld episode breakdown. Table 6 gives the estimated search cost at API rates.
  2. Inference cost. We measure model calls, tokens, latency, and estimated cost per task for the initial and evolved harnesses at the same concurrency. Table 7 reports both backbones on both benchmarks. For GPT-5.4-mini, latency measures model time without API queueing.
  3. Comparable inference budget. We execute the initial harness three times per task and combine its outputs with either majority vote or a same-model verifier, without using reference answers. We compare each approach with one evolved-harness run. This tests whether repeating the initial harness can match the benefits of changing its design.

The results are summarized in Tables 5 through 8.

[Table 5] Use of the rollout budget, Qwen3.5-27B AppWorld

ItemScholarEvolveMeta Harness (paper)
Initial harness, 3 runs171—
Repeated initial-harness runs (noise estimate)342—
20 single-module candidates × 3 runs3,420—
4 combinations × 3 runs684—
DEV episodes (cap 4,617)4,617 (100%)1155 (25.0%)
TRAIN episodes90360

[Table 6] Estimated cost of one ScholarEvolve search on Qwen3.5-27B AppWorld at API-equivalent rates

ComponentCost
Task-model execution (DEV and TRAIN)$295
Coding agent (20 sessions)≈ $12
Research model≈ $0.3
Literature stagea few dollars
Total≈ $310

[Table 7] Per-task inference cost and latency, initial → final harness

SettingModel callsTokensLatencyCost
Qwen3.5-27B, AppWorld20.8 → 67.9175K → 596K84 s → 250 s$0.047 → $0.173
Qwen3.5-27B, τ² Telecom56.0 → 88.5511K → 894K218 s → 451 s$0.111 → $0.211
GPT-5.4-mini, AppWorld16.0 → 34.884K → 276K20 s → 47 s$0.042 → $0.120
GPT-5.4-mini, τ² Telecom58.0 → 73.7439K → 631K67 s → 99 s$0.081 → $0.144

[Table 8] One evolved-harness run versus three initial-harness runs at comparable inference cost

SettingFinal-harness cost (× one initial run)Initial × 3, verifier picks oneInitial × 3, majority voteFinal harness, one runFinal − better of the two
Qwen3.5-27B, AppWorld Normal3.0–3.7×73.872.081.4+7.6
Qwen3.5-27B, AppWorld Challenge3.0–3.7×51.848.063.6+11.8
GPT-5.4-mini, AppWorld Normal2.2–3.3×65.569.172.4+3.3
GPT-5.4-mini, AppWorld Challenge2.2–3.3×48.449.655.6+6.0

The measurements show the cost of harness evolution and its benefit over repeated inference.

  • At API-equivalent rates, one ScholarEvolve search costs approximately $310, with about 95% spent on task-model execution (Tables 5 and 6). ScholarEvolve uses 4,617 DEV episodes. The original Meta Harness run uses 1,155, while the equal-cost Meta Harness run uses 4,104.
  • One evolved-harness run outperforms three initial-harness runs with either aggregation method. Its TGC advantage over the better aggregation method ranges from 3.3 to 11.8 points across the four AppWorld settings (Table 8). The gains come from a better harness, not simply from repeating the initial execution.

We will add these budget, cost, latency, and repeated-run results to the paper.


We thank the reviewer again for the thoughtful feedback and would be happy to address any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal and Discussion Summary

We thank the Area Chair and reviewers for their careful assessment and constructive feedback. We summarize the key additional evidence provided during rebuttal.

Reviewers recognized the novelty of research-guided harness evolution, the structured modular exploration, and the substantial improvements on held-out benchmarks.

During rebuttal, we addressed the main concerns through new experiments:

(1) Matched no-paper ablation isolating the contribution of research guidance, showing gains of up to 11.8 TGC and 20.2 SGC points over the same modular search without papers;

(2) Equal-cost Meta Harness and frozen-pool controls demonstrating that gains cannot be explained by additional search alone, and that refreshing literature provides further improvements;

(3) Generalization experiments on deep research agents also demonstrate the improved accuracy, together with a paper provenance audit confirming that no selected final module originates from research developed or evaluated on its target benchmark;

(4) Module interaction, fault-injection, and computational analyses validating the search design, execution robustness, and performance advantages over repeated inference.

Reviewer HYuW explicitly stated that the new experiments fully addressed all concerns and significantly strengthened the paper, and raised their rating to 4.

We believe these additional experiments provide strong evidence for ScholarEvolve's central contribution and directly address the key concerns raised during review. We hope the Area Chair will consider the new results and positive reviewer feedback in the final decision.

We thank the Area Chair and reviewers again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.