acceptodds
Under review as a conference paper at ICLR 2027

EvoG: A Modular Framework for Evolutionary Optimization of Agentic Workflows

Abstract

Designing effective agentic workflows for large language model (LLM) applications requires repeated, costly revision and evaluation. Evolutionary optimization can automate this process and existing methods have instantiated specific combinations of candidate selection, revision, example sampling, and evaluation strategies. Making these choices explicit and composable would help users construct and compare configurations suited to their task objectives and budget. Therefore, we introduce EvoG (Evolve on Graph), a modular framework for composing evolution strategies within a shared optimization loop. EvoG separates evolution into four components: a Selector that chooses parent candidates; a Generator that produces revisions; a Sampler that selects evaluation examples; and an Evaluator that assesses candidates. A shared history links candidate lineage, per-example outcomes, and execution traces to inform subsequent iterations. We evaluate EvoG on HotpotQA, MuSiQue-Ans, BigCodeBench-Hard, ScienceWorld, and WebShop. It achieves the highest mean test scores among the evaluated methods on four of five benchmarks, improving over the strongest baseline by 1.10 points on HotpotQA, 3.76 points on MuSiQue-Ans, 7.00 points on ScienceWorld, and 4.89 points on WebShop. On BigCodeBench-Hard, the best EvoG configuration achieves 35.33 points versus AgentSquare's 37.00 points, at approximately one-eighth its search cost and half its inference cost. For reproducibility, we release our code at https://anonymous.4open.science/r/EvoG-D6C2/

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agentic Workflow Optimization, Evolutionary Search, Composable Frameworks, Self-Evolving Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DyerOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoG is a framework for improving agent workflows by repeatedly selecting an earlier version, editing it, choosing examples, and evaluating the result. These four decisions are separate components, linked by a shared record of previous edits, scores, and execution traces. Its editing process uses an Analyzer, Optimizer, and Reviewer. Experiments compare selection choices on five benchmarks, and sampling choices on the two QA benchmarks, using one shared language model. Each configuration is searched once, then its selected workflow is tested three times. The best EvoG settings lead on four benchmarks. On BigCodeBench-Hard, EvoG scores 35.33 versus AgentSquare's 37.00, while costing less to search and run.

Critical strengths

The framework makes optimizer choices explicit. Selection, editing, sampling, and evaluation share a consistent interface and history. Within each search, a fixed set of examples is used to compare candidates. Testing combinations of selection and sampling policies gives useful information beyond evaluating one fixed recipe.

The paper reports practical trade-offs. It evaluates question answering, coding, and interactive tasks, and reports both search and execution costs. It acknowledges where EvoG loses or where another method is cheaper. Removing traces or resetting the Analyzer's history lowers the reported scores, and the WebShop example shows how that evidence can lead to a concrete repair (Table 3; Appendix C).

Critical weaknesses

Repeated tests of one workflow do not establish that the search itself is reliable. Different search runs can select different parents, sample different examples, and generate different edits. Testing the same final workflow three times does not measure those effects. The headline QA gains also select the best test score from 16 EvoG configurations. Reporting the full grid is useful, but a user still needs a way to choose a setting before seeing test results. This matters particularly for the 1.10-point HotpotQA gain and the claimed changes in policy rankings.

The evidence for the broader modular framework is limited. The experiments exchange selectors and samplers, but keep the same generator and evaluator. They support a useful implementation, while leaving less evidence for the value of swapping all four components. A controlled component exchange would strengthen that contribution. The lower search cost relative to AgentSquare also needs its stated context: AgentSquare evaluates up to nine candidates per iteration, while EvoG evaluates one. The observed saving does not establish which method is better at the same total search budget.

Questions for authors

Do the main gains, policy-ranking changes, and trace/history results hold across independent search runs? Please report search variability separately from repeated testing of one final workflow.

How should a user choose a selection and sampling configuration without seeing test scores? What result and total cost would a fixed default or independently selected configuration achieve?

Can you compare alternative generators or evaluators within the same framework and total budget? Also, was cross-example test memory enabled for AgentSquare, as allowed by the adaptation in Appendix B.2.2?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Dyer (1/3)

Q1: The results do not establish reliability across independent search runs.

A1: We thank the reviewer for highlighting search variability. As detailed in General Response Q2, we have repeated the complete search procedure three times for each configuration of EvoG and the search baselines in the main experiments. Each run's selected workflow is tested three times, and the average is used as that search run's test score. Tables 1 and 2 report the mean and sample standard deviation across these three search-run scores. The reported standard deviation therefore reflects variation across complete searches. We have also updated Figures 2 to 6 and the additional figures in the appendix.

The best EvoG configuration for each task achieves the highest mean test score among the evaluated methods on four of the five benchmarks. The gains over the strongest baseline are 2.36 points on HotpotQA, 1.79 points on MuSiQue-Ans, 5.29 points on ScienceWorld, and 1.27 points on WebShop.

Selector rankings still vary across Samplers in the averages across three searches. For example, on MuSiQue-Ans, soft-mixed scores higher than UCT-tree with full-pool sampling (83.23 vs. 82.61), but lower with random-feedback sampling (81.83 vs. 83.26). This observation supports comparing selection and sampling policies in combination.

We have also completed three independent search runs for each condition in the ablation study on all five benchmarks, evaluating each run's selected workflow three times. We have updated the paper's ablation section and marked the revisions in blue. Table 3 from the revised paper is reproduced below.

Table 3. Evidence ablations under soft-mixed selection and full-pool sampling. Scores are on a scale from 0 to 100 and report the mean ± sample standard deviation across three search runs, with each run's score averaging three test executions. Bold marks the highest mean score for each task. When we reran the ablations, the model served by DeepSeek's official API had changed from DeepSeek-V4.1-Flash to DeepSeek-V4.1-Flash-1017. All three conditions, including Full evidence, used the updated model, with three independent search runs per condition. So the Full evidence results differ from the corresponding results in Tables 1 and 2 of General Response Q2.

TaskFull evidenceNo traces to AnalyzerReset Analyzer
HotpotQA82.59 ± 0.6582.32 ± 0.5682.12 ± 0.68
MuSiQue-Ans83.60 ± 0.6681.17 ± 1.9283.48 ± 0.27
BigCodeBench-Hard36.33 ± 1.1535.44 ± 0.9634.78 ± 1.39
ScienceWorld98.42 ± 1.1395.68 ± 2.6397.25 ± 1.04
WebShop80.45 ± 1.4877.50 ± 1.0279.33 ± 1.50

Both ablations yield lower mean test scores than Full evidence on all five benchmarks. Removing detailed traces has its largest observed effect on WebShop, while resetting the Analyzer's persistent context has its largest effect on BigCodeBench-Hard. These results support the use of detailed traces to diagnose intermediate failures and persistent analysis to connect earlier edits with subsequent outcomes when planning workflow revisions.

AuthorsOpenReview ↗

Response to Reviewer Dyer (2/3)

Q2: How can users choose a configuration without using test scores, and what is the associated cost?

A2: We thank the reviewer for raising this practical question. We would like to clarify that EvoG is designed to support exploring and comparing different combinations of rules for evolving workflows on a given task, rather than prescribe a single default configuration across tasks. Users can choose how to explore these combinations according to evaluation cost and the available budget, as discussed in General Response Q3. Existing methods often implement a particular combination of these rules. When the rules are intertwined in the code, changing their combination can require modifying several parts of the implementation. EvoG gives these decisions separate interfaces, making it easier to add, replace, and compare alternatives for a given task.

When evaluation is relatively inexpensive and sufficient budget is available, users can apply grid search over predefined component combinations. They can specify the search budget for each trial, run these configurations, and compare the resulting workflows on a common validation set, reserving the test set for final evaluation. The total cost includes the search costs for all configurations tried and the cost of evaluating their resulting workflows on the validation set.

When evaluation is expensive or the budget is limited, exhaustive configuration sweeps may be impractical. As a future direction, an agent could choose which combination to try next based on earlier scores, search costs, and the remaining budget. This follows the direction of optimizing the evolution process itself discussed in General Response Q3. As a harness for workflow evolution, EvoG provides configurable component interfaces, shared execution and evaluation infrastructure, and records of candidate performance and resource usage, which provide a foundation for implementing such an adaptive extension.

AuthorsOpenReview ↗

Response to Reviewer Dyer (3/3)

Q3: The experiments vary Selectors and Samplers but keep the Generator and Evaluator fixed.

A3: We thank the reviewer for drawing attention to these two components. The benefits of EvoG’s modular design include both the flexibility to combine different component implementations, as illustrated by our Selector and Sampler experiments, and the engineering convenience of integrating different generation backends and evaluation pipelines. The engineering benefits are particularly relevant to the Generator and Evaluator. A Generator implementation encapsulates the model or agent harness used to produce revisions, while an Evaluator implementation supports task execution and scoring pipelines tailored to the underlying infrastructure. Shared interfaces allow these implementations to be integrated with the same evolution loop and history infrastructure. In the main experiments, we use a common Generator model and keep the evaluation protocol fixed within each benchmark, providing consistent conditions for comparing Selector and Sampler choices.

We conducted an additional experiment replacing the Generator's DeepSeek Harness (DSH) backend with Codex, while retaining the evidence-guided generation procedure and the underlying DeepSeek model. We kept the Selector, Sampler, and Evaluator fixed, using soft-mixed selection, full-pool sampling, and ten search iterations. Each harness was evaluated with three independent search runs per benchmark, with three test executions of each run's selected workflow.

Table: Generator harness comparison. Scores are mean ± sample standard deviation across three search-run averages, on a scale from 0 to 100. The DSH results are the Full evidence reruns reported in Q1.

BenchmarkDSHCodex
HotpotQA82.59 ± 0.6583.02 ± 0.15
MuSiQue-Ans83.60 ± 0.6683.34 ± 0.65
BigCodeBench-Hard36.33 ± 1.1534.67 ± 0.33
ScienceWorld98.42 ± 1.1398.84 ± 0.43
WebShop80.45 ± 1.4879.43 ± 1.24

Codex achieves higher mean scores on HotpotQA and ScienceWorld, while DSH achieves higher means on the other three benchmarks. This comparison provides a concrete example of integrating an alternative Generator harness while reusing the same evolution loop, history, and evaluation infrastructure.

Q4: How should the AgentSquare comparison be interpreted regarding search budget and test memory?

A4: We thank the reviewer for requesting clarification of the AgentSquare settings. In our evaluated setting, AgentSquare can evaluate up to nine candidates per iteration through its module evolution and recombination procedure, whereas EvoG generates one candidate per iteration. We retain this difference in their search structures, and the reported search costs include both optimization and candidate evaluation. In the updated repeated-search results, the best EvoG configurations achieve higher mean test scores than AgentSquare on all three benchmarks where both methods are evaluated.

We retained AgentSquare’s memory mechanism as part of its search space and allowed candidates with memory modules to use and update memory during both search and testing. Whether the final workflow uses memory follows the candidate selected by the search. Across the three search repetitions, the final candidate used memory in the second run on BigCodeBench-Hard and the third run on ScienceWorld.

Reviewer_rXAHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

A method for optimizing agentic workflow that's in 4 steps, select / generate / sample / eval

Critical strengths

it is a complete paper with all the components

Critical weaknesses

There are not much insights gained. The paper says it is "modular" but what modular means is simply that in their approach there are 4 steps. No justification was given on why those are the 4 steps. The ablation studies "detailed execution traces and persistent analysis to guide program revision" rather than the "modular" aspect of this work.

Shouldn't the title be "Detailed execution traces and persistent anlysis are useful in workflow evolution" ?

At the end of the day I did not learn much from reading this work, other than they built an end-to-end system, and it somewhat has better scores compared to baselines (but that could be due to a whole host of reasons such as better prompting or such).

Questions for authors

no questions

3 comments
AuthorsOpenReview ↗

Response to Reviewer rXAH

Q1: What does EvoG contribute beyond a complete workflow optimization system?

A1: We thank the reviewer for raising this question about the contribution. As clarified in General Response Q1, EvoG makes key design choices in evolution explicit through interchangeable Selector, Generator, Sampler, and Evaluator components with shared interfaces and history.

Modularity allows users to combine and compare policies to find suitable configurations for their tasks. Our experimental results support this motivation. With full-pool sampling, soft-mixed achieves the highest EvoG mean score on HotpotQA, while Pareto-coverage achieves the highest on MuSiQue-Ans. These configurations also achieve the highest mean scores among the evaluated methods on their respective QA benchmarks. Selector rankings also change across Samplers, illustrating why these choices should be compared in combination.

Modularity also supports engineering reuse. The Generator interface allows different agent harnesses to inspect and revise workflow code. The Evaluator supports task execution and scoring pipelines tailored to the underlying infrastructure. These implementations can reuse the same evolution loop and history management.

Q2: Why is workflow evolution decomposed into these four components?

A2: We would like to clarify the basis for this decomposition. As explained in General Response Q1, it reflects recurring choices in prior work on evolving prompts, agents, and workflows, as well as the engineering convenience of separating generation and evaluation implementations from the evolution loop. We use these choices to define the following components:

  • The Selector determines which candidate to revise. For example, AFlow defines a search policy for selecting workflows for further revision.
  • The Generator determines how to revise a candidate using available evidence. For example, Meta-Harness uses the source code, scores, and execution traces of prior candidates to guide code revisions.
  • The Sampler determines which examples provide evaluation and revision feedback. For example, ReCreate uses small batches of tasks to collect execution feedback for scaffold updates.
  • The Evaluator determines how candidates are executed and scored, encapsulating task execution, scoring, and feedback collection.

EvoG gives each decision a separate interface, allowing users to combine and compare component policies while reusing the same evolution loop and historical evidence. These interfaces also support engineering reuse. Different agent harnesses can be integrated into the Generator, and different execution and scoring pipelines can be integrated into the Evaluator.

Q3: How do the ablation studies relate to the modular framework?

A3: We thank the reviewer for requesting clarification of the purpose of the ablations. Our main experiments support the practical value of modularity. The best EvoG configurations achieve higher mean test scores than all evaluated baselines on both QA benchmarks, and the best-performing combination differs between HotpotQA and MuSiQue-Ans. These results demonstrate the practical value of comparing different policy combinations.

The ablation studies assess specific design choices within our Generator implementation. We withhold detailed execution traces from the Analyzer or reset its persistent context between rounds, while keeping the Selector, Sampler, and Evaluator fixed. Both variants yield lower scores on all five benchmarks, supporting the contributions of detailed execution evidence and persistent analysis to workflow revision.

Dear Reviewer rXAH,

Thank you again for your feedback. Our response clarifies the rationale for the four-component decomposition, the practical and engineering benefits of the modular framework, and how the Generator ablations relate to the broader contribution.

We would appreciate your feedback on whether these clarifications address your concerns. If any aspect of the contribution or supporting evidence remains unclear, we would welcome the opportunity to discuss it further.

AuthorsOpenReview ↗

Additional Response to Reviewer rXAH

As the discussion period draws to a close, we would like to follow up on your concern that “what modular means is simply that in their approach there are 4 steps.” Our previous response emphasized the benefits of combining strategies. Here, we would like to explain more clearly the design needed to make them interchangeable within a shared evolution loop. The following examples illustrate why strategies from existing implementations often cannot be combined through a simple substitution.

  • AFlow: Parent selection uses a shortlist formed by average scores. A per-example coverage strategy may need workflows outside this shortlist, including those with lower average scores but complementary strengths on particular examples. It also requires per-example results, while the existing selection input supplies aggregate scores. This strategy therefore cannot be directly substituted using the same inputs. See the AFlow implementation.
  • ADAS: The evaluation procedure internally chooses which examples to run, while the archive used for later proposals summarizes performance as aggregate fitness. A sampler based on previous mistakes needs per-example history and a way to specify the next evaluation examples. The existing evaluation and archive interfaces do not directly provide these capabilities, so adding a sampling rule alone cannot enable this combination. See the ADAS implementation.
  • ReCreate: The current workflow is evaluated to collect experience, which then guides revisions and their combination into an updated workflow. The batch scores describe the version executed before revision. A selector comparing generated workflows needs evaluations of those revised versions. The existing batch results therefore cannot directly support this selection, and adding a score-based selector alone would leave it without the required scores. See the ReCreate implementation.

EvoG’s design contribution is to provide the interfaces and shared records that allow these strategies to be replaced and combined within one evolution loop. Its history records which workflow was selected, which new version was generated, and how each evaluated version performed on each example. These records provide the information needed by alternative strategies, allowing users to combine their implementations while reusing the surrounding search and evaluation infrastructure.

Reviewer_6HacOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to automate the design of agentic workflows for large language model applications. Observing that existing evolutionary optimization methods hardcode specific combinations of candidate selection, revision, example sampling, and evaluation, the authors introduce EvoG, a modular framework that separates the evolution loop into four interchangeable components, namely a Selector that chooses parent candidates, a Generator that produces revisions, a Sampler that selects evaluation examples, and an Evaluator that assesses candidates, all linked by a shared history of candidate lineage, per-example outcomes, and execution traces. The authors instantiate four Selector policies, four Sampler settings, and an evidence-guided Generator composed of an Analyzer, an Optimizer, and a Reviewer with mechanical validation and bounded repair. Experiments on HotpotQA, MuSiQue-Ans, BigCodeBench-Hard, ScienceWorld, and WebShop report that the best EvoG configuration attains the highest mean test score on four of the five benchmarks, and that on BigCodeBench-Hard it trails AgentSquare by 1.67 points at roughly one-eighth of the search cost. Ablations indicate that detailed execution traces and persistent Analyzer context both contribute to revision quality.

Critical strengths
  1. Motivation: The problem is well motivated and clearly positioned. Evolutionary workflow optimizers such as ADAS, AFlow, and GEPA each hardcode interdependent design choices, and Section 1 argues convincingly that making these choices explicit and composable lets users match configurations to task objectives and budgets while reusing the same execution and feedback infrastructure. The observed Selector ranking reversal across Sampler settings in Table 1, for instance soft-mixed beating UCT-tree with full-pool sampling, 84.48 against 83.62, but losing with random-feedback sampling, 82.49 against 84.65 on MuSiQue-Ans, is direct evidence that these choices interact and therefore deserve joint, configurable study rather than fixed instantiation.

  2. Method design: The decomposition is clean and technically sound. Section 3.1 formalizes the loop with a fixed select set and optional per-round propose sets, Eq. 1 defines the shared comparison metric, and Appendix A gives exact, implementable rules for all four Selector policies in Eq. 2 through Eq. 7, with tie-breaking and parameter values documented in Appendix B.3. The evidence-guided Generator sensibly combines trace-guided reflection in the style of GEPA and ReCreate with history-conditioned code revision in the style of Meta-Harness, and the mechanical validation, read-only Reviewer check, bounded repair, and eligibility gating form a practical safeguard against broken candidates. The paper is also unusually honest about negative results, reporting that low-score-feedback loses six of eight matched comparisons and that EvoG trails AgentSquare on BigCodeBench-Hard.

  3. Experiments: The evaluation spans five benchmarks across three task families under one common backbone model, DeepSeek-V4.1-Flash, and the cost analysis is a standout feature: Figures 3 and 5 give score-cost Pareto views, Figures 6 and 7 separate optimization cost from candidate-evaluation cost, and Figures 9 through 14 report token usage and wall-clock time. The evidence ablations in Table 3 isolate the contribution of traces and of persistent analysis across all five tasks, and the WebShop case study in Appendix C concretely shows how the shared history turns a diagnosis into an effective program revision. The anonymized code release plus the detailed documentation of splits, baseline adaptations, and execution limits in Appendix B support reproducibility.

Critical weaknesses
  1. Statistical support for the headline claims is thin. Each configuration is searched only once, and just the test stage is repeated three times, as stated in Section 4. Search is stochastic, so single-run margins such as the 1.10-point gain over AdaptFlow on HotpotQA in Table 1 are hard to interpret, and the paper's central interaction claim, that Selector rankings vary across Samplers, rests on one run per cell of Table 1. Configuration sensitivity is substantial: on HotpotQA, five of the sixteen EvoG configurations fall below AdaptFlow's 82.60, and on MuSiQue-Ans the best-observed with low-score-feedback configuration scores 77.85, below even the IO baseline at 80.14.

  2. The practical payoff of modularity remains post hoc. The conclusion that the best configuration varies by task is empirically supported, but the paper offers no procedure, heuristic, or meta-selection scheme for choosing a Selector and Sampler given a new task and budget. A practitioner must still sweep configurations to benefit, which undercuts the framework's promise of matching strategies to objectives and budgets.

Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6Hac

Q1: The results lack repeated search runs to account for search stochasticity.

A1: We thank the reviewer for highlighting search variability. As detailed in General Response Q2, we have repeated the complete search procedure three times for each configuration of EvoG and the search baselines in the main experiments. Each run's selected workflow is tested three times. We have updated Tables 1 and 2, Figures 2 to 6, and the appendix figures and tables to reflect these repeated searches. Tables 1 and 2 now report the mean and sample standard deviation across the three run averages. The best EvoG configuration for each task achieves the highest mean test score among the evaluated methods on four of the five benchmarks. Selector rankings still vary across Samplers when results are averaged across the three searches. For example, on MuSiQue-Ans, soft-mixed scores higher than UCT-tree with full-pool sampling (83.23 vs. 82.61), but lower with random-feedback sampling (81.83 vs. 83.26).

Q2: Performance is sensitive to configuration choices, and some configurations underperform the baselines.

A2: We would like to clarify this point. As discussed in General Response Q1, different tasks may favor different combinations of selection and sampling policies, depending on whether their failures share common causes or require different fixes. It is therefore expected that some combinations are less suitable for a given task and may underperform the baselines. Providing a broader space of combinations is important for comparing alternatives and finding suitable configurations for each task. Our experimental results support this motivation. Soft-mixed with full-pool sampling on HotpotQA and Pareto-coverage with full-pool sampling on MuSiQue-Ans achieve the highest mean test scores among the evaluated methods on their respective benchmarks.

Q3: How can users choose a configuration for a new task and budget?

A3: We thank the reviewer for raising this practical question. As discussed in General Response Q3, when evaluation is relatively inexpensive and sufficient budget is available for multiple trial searches and evaluations, one option is grid search over component combinations, such as different Selector and Sampler policies. Existing workflow optimization methods typically implement a particular combination of rules, such as how to choose a workflow for revision, generate changes, and select examples for feedback. When these choices are intertwined in the implementation, trying a different combination for a new task may require changes to multiple parts of the code. This makes it harder to introduce alternative rules and compare different combinations. EvoG gives these decisions separate interfaces, allowing users to add, replace, and combine their implementations while reusing the same evolution loop, evaluation infrastructure, and history records. Users can select a configuration based on their performances on a common validation set, with the total budget including all trial searches and validation evaluations.

When evaluation is expensive or the budget is limited, exhaustive configuration sweeps may be impractical. Following the recent work discussed in General Response Q3, a future extension could use an agent to propose component combinations. The agent could use earlier scores and search costs to decide which combination to try next within the remaining budget. As a harness for workflow evolution, EvoG provides configurable component interfaces, shared execution and evaluation infrastructure, and records of candidate performance and resource usage, which provide a foundation for implementing such an adaptive extension.

Dear Reviewer 6Hac,

Thank you again for your thoughtful feedback. In our rebuttal, we report three complete search runs for each search-based method and configuration in the main experiments, with updated means and standard deviations. We also clarify the practical value of modularity and discuss configuration selection for new tasks and budgets.

We would appreciate hearing whether these additional results and clarifications address your concerns, and whether there are any remaining points we could clarify during the discussion.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank all reviewers for their careful reading and feedback. We appreciate Reviewer 6Hac’s positive assessment of the motivation, method design, and experimental coverage, and Reviewer Dyer’s recognition of the explicit optimizer choices and practical performance and cost analysis. The reviews raise three common questions concerning the contribution of the modular framework (rXAH), the reliability of stochastic search (6Hac and Dyer), and configuration selection for new tasks (6Hac and Dyer). We address these questions below. Changes in the revised paper are marked in blue.

Q1: What is the contribution of our modular framework?

A1: We would like to clarify the intended contribution of EvoG. EvoG makes key design choices in workflow evolution explicit and allows their implementations to be replaced and combined. Our decomposition is motivated by recurring choices in prior work on evolving prompts, agents, and workflows, as well as the engineering convenience of separating generation and evaluation implementations from the evolution loop. These choices concern which candidates to revise, how to generate revisions, which examples provide feedback, and how candidates are executed and scored. For example, AFlow [1] defines a policy for selecting workflows for further revision. Meta-Harness [2] uses the source code, scores, and execution traces of prior candidates to guide code revisions. ReCreate [3] uses small batches of tasks to collect execution feedback for scaffold updates. EvoG organizes these choices into the Selector, Generator, Sampler, and Evaluator, connected through shared interfaces and a common record of candidates, scores, and execution traces. These interfaces also support engineering reuse. Different agent harnesses can be integrated into the Generator to inspect and revise workflow code, while different task-execution and scoring pipelines can be integrated into the Evaluator. These implementations can reuse the same evolution loop and history management.

Modularity allows users to combine and compare evolution policies to find suitable configurations for their tasks. Intuitively, different tasks can favor different policy combinations because their failures may arise from a common weakness or from several distinct weaknesses. When many failures have the same cause, feedback from a few representative examples can guide revisions that improve performance across many examples. This can make focused sampling and continued refinement of a strong candidate effective. When different types of examples require different fixes, a narrow feedback set can miss important weaknesses. Sampling more varied examples and exploring candidates with different strengths may then offer better opportunities for improvement. EvoG allows users to combine these policies, compare alternatives, and select a configuration based on their performances. Our experimental results support the practical value of combining and comparing policies for different tasks. The best EvoG configurations achieve the highest mean test scores among the evaluated methods on both QA benchmarks, using soft-mixed with full-pool sampling on HotpotQA and Pareto-coverage with full-pool sampling on MuSiQue-Ans.

AuthorsOpenReview ↗

General Response (2/3)

Q2: How reliable are the results given the stochastic search process?

A2: We have repeated the complete search procedure three times for every method and configuration that performs search in the main experiments. We evaluate the selected workflow from each search on the test set three times and use the average as that search run's test score. For IO and CoT, which do not perform search, we use three groups of three test evaluations and apply the same aggregation. We have updated Tables 1 and 2, Figures 2 to 6, and the additional figures in the appendix of the paper to reflect these repeated searches. The corresponding revisions are marked in blue. Tables 1 and 2 from the revised paper are reproduced below and report the mean and sample standard deviation across the three run averages.

The updated results show that the best EvoG configuration for each task achieves the highest mean test score among the evaluated methods on four of the five benchmarks. The gains over the strongest baseline on each task are 2.36 points on HotpotQA, 1.79 points on MuSiQue-Ans, 5.29 points on ScienceWorld, and 1.27 points on WebShop. On BigCodeBench-Hard, EvoG with soft-mixed selection scores 34.22 ± 1.39, compared with 34.56 ± 1.84 for Meta-Harness. Its mean score is 0.33 points lower, with a smaller observed standard deviation across search runs. It also has 14.4% lower search cost and 23.0% lower inference cost than Meta-Harness.

Table 1. QA test answer-token F1 on 800 examples per benchmark, on a scale from 0 to 100. Sampler entries show the sizes of the fixed select set and the additional feedback set used in each iteration. Bold marks the highest mean in each column.

MethodSelectorSamplerHotpotQAMuSiQue-Ans
ION/AN/A63.58 ± 0.4379.95 ± 0.18
CoTN/AN/A80.41 ± 0.2279.49 ± 0.50
ADASN/AN/A81.02 ± 0.4081.59 ± 0.65
AFlowN/AN/A80.33 ± 1.4378.83 ± 1.36
AdaptFlowN/AN/A81.03 ± 1.3781.74 ± 1.47
AdaptFlow + TTAN/AN/A80.91 ± 0.9381.14 ± 1.80
EvoGBest-observedFull (200/0)82.90 ± 0.2083.37 ± 0.55
EvoGBest-observedFixed (100/0)83.31 ± 0.4482.71 ± 0.75
EvoGBest-observedRandom (50/50)82.67 ± 0.2183.06 ± 0.25
EvoGBest-observedLow-score (50/50)82.82 ± 0.4081.40 ± 3.18
EvoGSoft-mixedFull (200/0)83.39 ± 0.2883.23 ± 1.09
EvoGSoft-mixedFixed (100/0)82.86 ± 0.6881.76 ± 0.83
EvoGSoft-mixedRandom (50/50)82.78 ± 0.6781.83 ± 0.78
EvoGSoft-mixedLow-score (50/50)82.49 ± 0.5582.83 ± 0.28
EvoGUCT-treeFull (200/0)82.53 ± 0.7582.61 ± 0.92
EvoGUCT-treeFixed (100/0)82.51 ± 0.2182.95 ± 0.33
EvoGUCT-treeRandom (50/50)82.74 ± 0.3183.26 ± 1.22
EvoGUCT-treeLow-score (50/50)82.00 ± 0.0382.88 ± 0.96
EvoGPareto-coverageFull (200/0)81.07 ± 1.3983.53 ± 0.44
EvoGPareto-coverageFixed (100/0)82.53 ± 0.3782.63 ± 0.47
EvoGPareto-coverageRandom (50/50)82.72 ± 0.0782.80 ± 0.79
EvoGPareto-coverageLow-score (50/50)82.33 ± 0.1882.65 ± 0.84

Table 2. Test scores on 100 tasks per benchmark, on a scale from 0 to 100. BigCodeBench-Hard reports the percentage of tasks passing all unit tests. ScienceWorld reports its task score, and WebShop reports its purchase reward. All EvoG configurations use full-pool sampling with 20 search tasks. Bold marks the highest mean in each column.

MethodBigCodeBench-HardScienceWorldWebShop
IO33.44 ± 1.5086.85 ± 2.3078.12 ± 0.26
CoT33.56 ± 1.2689.12 ± 0.7478.12 ± 0.58
ReCreate32.56 ± 1.3991.93 ± 4.4777.93 ± 1.41
AgentSquare34.00 ± 3.0076.99 ± 3.5475.31 ± 0.65
Meta-Harness34.56 ± 1.8488.99 ± 1.9879.44 ± 1.64
EvoG best-observed30.67 ± 3.0695.28 ± 3.1180.71 ± 2.65
EvoG soft-mixed34.22 ± 1.3997.22 ± 2.3078.71 ± 4.39
EvoG UCT-tree33.67 ± 1.5396.00 ± 2.8779.05 ± 3.32
AuthorsOpenReview ↗

General Response (3/3)

Q3: How can users choose a configuration for a new task?

A3: EvoG provides a common framework for exploring combinations of its four components when optimizing workflows for tasks. Examples include a workflow for handling a company’s after-sales inquiries or a workflow for comparing supplier quotes.

When evaluation is relatively inexpensive and sufficient budget is available for multiple trial searches and evaluations, grid search over component combinations is one way to choose a configuration. Users can run different configurations and compare their proformances on a common validation set. Existing workflow optimization methods typically implement a particular combination of rules for choosing workflows, generating revisions, and selecting examples for feedback. When these choices are intertwined in the implementation, trying a different combination for a new task may require changes to multiple parts of the code. This makes it harder to introduce alternative rules and compare different combinations. EvoG gives these decisions separate interfaces, allowing users to add, replace, and combine their implementations while reusing the same evolution loop, evaluation infrastructure, and history records.

When evaluation is expensive or the budget is limited, exhaustive configuration sweeps may be impractical. Recent work has begun to explore new approaches by optimizing the evolution process itself. Dream-RSI [4] revises the code that controls exploration and evaluates alternative policies by replaying past search records. FreeEvolve [5] improves an editable set of instructions that guides agent evolution, evaluating each version by the performance of the agents it produces. Following this direction, a future extension of EvoG could use an agent to choose component combinations. The agent could propose a configuration, run a trial search, inspect the resulting workflow’s performance and search cost, and use these results to choose the next configuration within the remaining budget. As a harness for workflow evolution, EvoG provides configurable component interfaces and records of candidate performance and resource usage, which provide a foundation for implementing such an extension.

References

[1] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, XiongHui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating Agentic Workflow Generation. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/5492ecbce4439401798dcd2c90be94cd-Abstract-Conference.html.

[2] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv preprint arXiv:2603.28052, 2026. URL https://arxiv.org/abs/2603.28052.

[3] Zhezheng Hao, Hong Wang, Jian Luo, Jianqing Zhang, Yuyan Zhou, Qiang Lin, Can Wang, Hande Dong, and Jiawei Chen. ReCreate: Reasoning and Creating Domain Agents Driven by Experience. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 31018–31046, 2026. doi:10.18653/v1/2026.acl-long.1432. URL https://aclanthology.org/2026.acl-long.1432/.

[4] Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, and Yunsong Guo. Dream-RSI: Recursive Self-Improvement through Evolving Worlds. arXiv preprint arXiv:2609.14858, 2026. URL https://arxiv.org/abs/2609.14858.

[5] Lecheng Kong, Like Hui, Nikos Kanakaris, Prithwish Jana, Sahika Genc, and Narayanan Sadagopan. FreeEvolve: Learning to Evolve Beyond Fixed Loops. arXiv preprint arXiv:2610.09197, 2026. URL https://arxiv.org/abs/2610.09197.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.