- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EvoG is a framework for improving agent workflows by repeatedly selecting an earlier version, editing it, choosing examples, and evaluating the result. These four decisions are separate components, linked by a shared record of previous edits, scores, and execution traces. Its editing process uses an Analyzer, Optimizer, and Reviewer. Experiments compare selection choices on five benchmarks, and sampling choices on the two QA benchmarks, using one shared language model. Each configuration is searched once, then its selected workflow is tested three times. The best EvoG settings lead on four benchmarks. On BigCodeBench-Hard, EvoG scores 35.33 versus AgentSquare's 37.00, while costing less to search and run.
Critical strengths
The framework makes optimizer choices explicit. Selection, editing, sampling, and evaluation share a consistent interface and history. Within each search, a fixed set of examples is used to compare candidates. Testing combinations of selection and sampling policies gives useful information beyond evaluating one fixed recipe.
The paper reports practical trade-offs. It evaluates question answering, coding, and interactive tasks, and reports both search and execution costs. It acknowledges where EvoG loses or where another method is cheaper. Removing traces or resetting the Analyzer's history lowers the reported scores, and the WebShop example shows how that evidence can lead to a concrete repair (Table 3; Appendix C).
Critical weaknesses
Repeated tests of one workflow do not establish that the search itself is reliable. Different search runs can select different parents, sample different examples, and generate different edits. Testing the same final workflow three times does not measure those effects. The headline QA gains also select the best test score from 16 EvoG configurations. Reporting the full grid is useful, but a user still needs a way to choose a setting before seeing test results. This matters particularly for the 1.10-point HotpotQA gain and the claimed changes in policy rankings.
The evidence for the broader modular framework is limited. The experiments exchange selectors and samplers, but keep the same generator and evaluator. They support a useful implementation, while leaving less evidence for the value of swapping all four components. A controlled component exchange would strengthen that contribution. The lower search cost relative to AgentSquare also needs its stated context: AgentSquare evaluates up to nine candidates per iteration, while EvoG evaluates one. The observed saving does not establish which method is better at the same total search budget.
Questions for authors
Do the main gains, policy-ranking changes, and trace/history results hold across independent search runs? Please report search variability separately from repeated testing of one final workflow.
How should a user choose a selection and sampling configuration without seeing test scores? What result and total cost would a fixed default or independently selected configuration achieve?
Can you compare alternative generators or evaluators within the same framework and total budget? Also, was cross-example test memory enabled for AgentSquare, as allowed by the adaptation in Appendix B.2.2?
6 comments
Response to Reviewer Dyer (2/3)
Q2: How can users choose a configuration without using test scores, and what is the associated cost?
A2: We thank the reviewer for raising this practical question. We would like to clarify that EvoG is designed to support exploring and comparing different combinations of rules for evolving workflows on a given task, rather than prescribe a single default configuration across tasks. Users can choose how to explore these combinations according to evaluation cost and the available budget, as discussed in General Response Q3. Existing methods often implement a particular combination of these rules. When the rules are intertwined in the code, changing their combination can require modifying several parts of the implementation. EvoG gives these decisions separate interfaces, making it easier to add, replace, and compare alternatives for a given task.
When evaluation is relatively inexpensive and sufficient budget is available, users can apply grid search over predefined component combinations. They can specify the search budget for each trial, run these configurations, and compare the resulting workflows on a common validation set, reserving the test set for final evaluation. The total cost includes the search costs for all configurations tried and the cost of evaluating their resulting workflows on the validation set.
When evaluation is expensive or the budget is limited, exhaustive configuration sweeps may be impractical. As a future direction, an agent could choose which combination to try next based on earlier scores, search costs, and the remaining budget. This follows the direction of optimizing the evolution process itself discussed in General Response Q3. As a harness for workflow evolution, EvoG provides configurable component interfaces, shared execution and evaluation infrastructure, and records of candidate performance and resource usage, which provide a foundation for implementing such an adaptive extension.
Response to Reviewer Dyer (3/3)
Q3: The experiments vary Selectors and Samplers but keep the Generator and Evaluator fixed.
A3: We thank the reviewer for drawing attention to these two components. The benefits of EvoG’s modular design include both the flexibility to combine different component implementations, as illustrated by our Selector and Sampler experiments, and the engineering convenience of integrating different generation backends and evaluation pipelines. The engineering benefits are particularly relevant to the Generator and Evaluator. A Generator implementation encapsulates the model or agent harness used to produce revisions, while an Evaluator implementation supports task execution and scoring pipelines tailored to the underlying infrastructure. Shared interfaces allow these implementations to be integrated with the same evolution loop and history infrastructure. In the main experiments, we use a common Generator model and keep the evaluation protocol fixed within each benchmark, providing consistent conditions for comparing Selector and Sampler choices.
We conducted an additional experiment replacing the Generator's DeepSeek Harness (DSH) backend with Codex, while retaining the evidence-guided generation procedure and the underlying DeepSeek model. We kept the Selector, Sampler, and Evaluator fixed, using soft-mixed selection, full-pool sampling, and ten search iterations. Each harness was evaluated with three independent search runs per benchmark, with three test executions of each run's selected workflow.
Table: Generator harness comparison. Scores are mean ± sample standard deviation across three search-run averages, on a scale from 0 to 100. The DSH results are the Full evidence reruns reported in Q1.
| Benchmark | DSH | Codex |
|---|---|---|
| HotpotQA | 82.59 ± 0.65 | 83.02 ± 0.15 |
| MuSiQue-Ans | 83.60 ± 0.66 | 83.34 ± 0.65 |
| BigCodeBench-Hard | 36.33 ± 1.15 | 34.67 ± 0.33 |
| ScienceWorld | 98.42 ± 1.13 | 98.84 ± 0.43 |
| WebShop | 80.45 ± 1.48 | 79.43 ± 1.24 |
Codex achieves higher mean scores on HotpotQA and ScienceWorld, while DSH achieves higher means on the other three benchmarks. This comparison provides a concrete example of integrating an alternative Generator harness while reusing the same evolution loop, history, and evaluation infrastructure.
Q4: How should the AgentSquare comparison be interpreted regarding search budget and test memory?
A4: We thank the reviewer for requesting clarification of the AgentSquare settings. In our evaluated setting, AgentSquare can evaluate up to nine candidates per iteration through its module evolution and recombination procedure, whereas EvoG generates one candidate per iteration. We retain this difference in their search structures, and the reported search costs include both optimization and candidate evaluation. In the updated repeated-search results, the best EvoG configurations achieve higher mean test scores than AgentSquare on all three benchmarks where both methods are evaluated.
We retained AgentSquare’s memory mechanism as part of its search space and allowed candidates with memory modules to use and update memory during both search and testing. Whether the final workflow uses memory follows the candidate selected by the search. Across the three search repetitions, the final candidate used memory in the second run on BigCodeBench-Hard and the third run on ScienceWorld.
Response to Reviewer Dyer (1/3)
Q1: The results do not establish reliability across independent search runs.
A1: We thank the reviewer for highlighting search variability. As detailed in General Response Q2, we have repeated the complete search procedure three times for each configuration of EvoG and the search baselines in the main experiments. Each run's selected workflow is tested three times, and the average is used as that search run's test score. Tables 1 and 2 report the mean and sample standard deviation across these three search-run scores. The reported standard deviation therefore reflects variation across complete searches. We have also updated Figures 2 to 6 and the additional figures in the appendix.
The best EvoG configuration for each task achieves the highest mean test score among the evaluated methods on four of the five benchmarks. The gains over the strongest baseline are 2.36 points on HotpotQA, 1.79 points on MuSiQue-Ans, 5.29 points on ScienceWorld, and 1.27 points on WebShop.
Selector rankings still vary across Samplers in the averages across three searches. For example, on MuSiQue-Ans, soft-mixed scores higher than UCT-tree with full-pool sampling (83.23 vs. 82.61), but lower with random-feedback sampling (81.83 vs. 83.26). This observation supports comparing selection and sampling policies in combination.
We have also completed three independent search runs for each condition in the ablation study on all five benchmarks, evaluating each run's selected workflow three times. We have updated the paper's ablation section and marked the revisions in blue. Table 3 from the revised paper is reproduced below.
Table 3. Evidence ablations under soft-mixed selection and full-pool sampling. Scores are on a scale from 0 to 100 and report the mean ± sample standard deviation across three search runs, with each run's score averaging three test executions. Bold marks the highest mean score for each task. When we reran the ablations, the model served by DeepSeek's official API had changed from DeepSeek-V4.1-Flash to DeepSeek-V4.1-Flash-1017. All three conditions, including Full evidence, used the updated model, with three independent search runs per condition. So the Full evidence results differ from the corresponding results in Tables 1 and 2 of General Response Q2.
Both ablations yield lower mean test scores than Full evidence on all five benchmarks. Removing detailed traces has its largest observed effect on WebShop, while resetting the Analyzer's persistent context has its largest effect on BigCodeBench-Hard. These results support the use of detailed traces to diagnose intermediate failures and persistent analysis to connect earlier edits with subsequent outcomes when planning workflow revisions.