acceptodds
Under review as a conference paper at ICLR 2027

WHAT IS ONE SEARCH OPERATION WORTH? EVALUATING ITS DOWNSTREAM VALUE IN AUTOMATIC AGENT DESIGN

Abstract

Automatic agent design searches for better prompts, agent connections, and workflows. Yet a strong final system does not reveal which individual search decisions improved it. We introduce a protocol for evaluating a search operation through the optimization that follows it. From the same recorded state, we execute the original operation or a specified alternative, then resume optimization under the same objective and budget. We apply this protocol to GPTSwarm, AFlow, MaAS, and EvoMAS across six benchmarks. A dedicated GPTSwarm–MMLU study includes 480 executions at 20 states, with same-operation controls and separate runs for selection and validation. Operations selected using four repetitions per alternative show an apparent mean advantage of 2.17 percentage points over the unselected alternatives. Four additional repetitions on the same evaluation items estimate this advantage at 0.65 points (95% CI ), yielding a paired selection–validation gap of 1.52 points . GPTSwarm and EvoMAS/MMLU also show measurable separation in subsequent search trajectories, while estimated associations with downstream value remain imprecise. These measurements distinguish how an operation changes search from how well a selected operation performs on new executions, enabling evaluation of individual decisions within automatic agent design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, automated design of agentic systems, agentic system optimization, adaptive search, search evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SmpcOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an operator-level evaluation protocol for automatic agent design, addressing the limitation that a strong final agent does not establish that the search decisions leading to it were valuable. The protocol restores a recorded optimiser state, replaces one operation with a prespecified alternative, and resumes the native optimiser under the same objective and computational budget. It distinguishes three questions: whether an operation changes the subsequent search trajectories, whether it improves downstream performance, and whether its apparent advantage generalises to independent executions. The authors apply the protocol to four agent-design frameworks: GPTSwarm, AFlow, MaAS, EvoMAS, across six benchmarks. The primary GPTSwarm-MMLU study evaluates 480 executions across 20 frozen states, using same-operation controls and separate selection and validation waves. The apparent advantage of operations selected using four repetitions is 2.17 % points, but falls to 0.65 points on independent validation. The paired selection-validation gap is 1.52 points. GPTSwarm and EvoMAS experiments also show measurable changes in subsequent search trajectories, but the link between trajectory changes and downstream value remains uncertain. The main contribution is a generalisable experimental methodology for evaluating individual search decisions through counterfactual replay, native optimisation continuation, repeated-execution controls, and independent validation. The results highlight selection bias and demonstrate that changing a search trajectory does not necessarily improve performance. However, the strongest independent validation is limited to 20 GPTSwarm states, one model route, and a fixed set of 153 MMLU evaluation terms, leaving broader generalisability unresolved.

Critical strengths
  1. A principled protocol for evaluating individual search decisions. The paper introduces a clear and generalisable framework that restores a recorded optimiser state, intervenes on a single operation and continues optimisation under the original procedure. By combining counterfactual comparisons, same-operation controls, and independent validation, it distinguishes search trajectory changes from downstream performance improvements and selection bias. This is a meaningful methodological contribution to evaluating automatic agent-design systems.

  2. Empirical demonstrations of the gap between apparent and independently validated value. Across four agent-design frameworks, the authors investigate whether individual operations influence subsequent search and whether these changes translate to downstream benefits. The GPTSwarm-MMLU study is particularly informative: an apparent selection advantage of 2.17 points falls to 0.65 points on independent validation, with a 95% confidence interval spanning 0. The result illustrates the importance of independent evaluation and challenges the assumption that locally promising search decisions reliably improve downstream outcomes.

Critical weaknesses
  1. Limited Evidence for the paper’s central empirical conclusion. The strongest independent validation study uses only 20 GPTSwarm states, one model route, and a fixed set of 153 MMLU evaluation items. The estimated advantage of the selected operation drops from 2.17 points during selection to 0.65 during independent validation, with the validation confidence interval spanning 0. Although this demonstrates the possibility of selection bias, the small validation cohort and narrow evaluation setting make it difficult to establish how consistently this effect generalises across states, tasks and search algorithms. This limits the strength of the paper’s broader empirical conclusions.

  2. The relationship between trajectory changes and downstream value remains weakly established. A central motivation is to distinguish whether a search operation changes the optimisation trajectory from whether it improves the final agent. However, the reported local signals - including graph displacement, immediate score differences, and gradient-based surrogates - show weak and imprecise associations with independently validated performance. In some settings, trajectory separation is measurable while task-level performance contrasts are zero or too coarse to resolve. The methodology therefore provides a useful framework for evaluating search operations, but the paper offers limited evidence that its measurements can reliably identify valuable operations or support better search decisions in practice.

Questions for authors
  1. Clarification: Robustness of independent validation results. The estimated advantage of the selected operation decreases from 2.17 to 0.65 % points in Wave 2, with the latter’s confidence spanning 0. How much statistical power does the 20-state validation study provide to distinguish a genuinely positive effect from no effect? Do the conclusions remain consistent under alternative aggregation or uncertainty-estimation procedures?

  2. Clarification: Interpretation of the Native-A/Native-B offset. The same operation control and exhibit a signed mean difference of -1 % point, suggesting a non-negligible variability event without changing the selected operation. How does this execution offset affect the estimated counterfactual treatment effect, and what evidence supports attributing observed differences to the operation intervention rather than stochastic variation in continuation?

  3. Clarification: Generalisability across search frameworks and tasks. The strongest independent validation comes from GPTSwarm on a fixed MMLU evaluation set, while some other settings have coarse or uninformative task-level performance contrasts. What evidence supports extending the conclusions about operational value and selection reliability beyond this particular model route and evaluation setting?

Reviewer_wY32OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to evaluate the contribution of individual search operations in automatic agent design. The authors propose a protocol that restores a recorded search state, replaces one operation with an alternative, and continues optimization under the same settings and budget. By comparing the resulting outcomes, the framework aims to measure how individual decisions affect subsequent search and final performance.

The authors evaluate the protocol on four agent design frameworks (GPTSwarm, AFlow, MaAS, and EvoMAS) across six benchmarks. The main GPTSwarm experiment shows that an apparent 2.17 percentage-point advantage from selecting an operation decreases to 0.65 percentage points in independent evaluation. Additional experiments examine changes in search trajectories, execution variability, and the relationship between local signals and downstream outcomes. The results highlight the difficulty of reliably estimating the value of individual search operations.

Critical strengths
  • Meaningful evaluation perspective: The paper considers an important question of how individual search decisions affect later optimization, rather than evaluating only the final agent system. The proposed protocol offers a systematic way to compare alternative operations while accounting for subsequent search behavior. This provides a useful perspective for understanding automatic agent design methods.

  • Careful experimental design and reporting: The authors include repeated evaluations, same-operation controls, and independent validation to account for execution variability. They also report uncertainty estimates and acknowledge limitations in their findings. The supplementary materials provide extensive experimental records and analysis code, which support reproducibility.

Critical weaknesses
  • Limited practical relevance of the evaluated search operations: Although the proposed protocol provides a systematic way to compare search operations, some of the evaluated alternatives do not clearly represent realistic decisions made during optimization. For example, GPTSwarm uses a researcher-defined reassignment of observed utility values, while EvoMAS selects alternative parent pairs with minimal overlap. It is unclear whether conclusions drawn from these interventions would hold for alternative operations that naturally arise during optimization. The paper also does not demonstrate how the resulting measurements could inform practical search decisions, leaving the broader usefulness of the protocol insufficiently established.

  • Limited evidence for reliable and generalizable operation-level evaluation: The main independent validation is restricted to 20 recorded GPTSwarm states, using the same 153 MMLU questions across evaluation waves. Although the selection–validation gap is statistically supported, the independently validated advantage remains uncertain (0.65 percentage points, 95% CI [-0.56, 1.82]). Moreover, the same-operation controls show an unexplained difference of approximately 1.00 percentage point between identical operations, raising concerns about execution variability. Evidence from the other frameworks is also limited: EvoMAS shows no task-accuracy differences across its 20 complete states, while AFlow produces substantially different results across independent evaluation waves. Overall, the experiments do not provide sufficiently convincing evidence that the protocol can reliably assess operation value across different settings.

Questions for authors
  • How sensitive are the findings to the choice of alternative operations? Would the conclusions remain consistent if GPTSwarm and EvoMAS used alternative operations sampled from their native search procedures rather than the specific alternatives used in this study?

  • Table 4 reports a -1.00 percentage-point difference between Native-A and Native-B despite executing the same operation. Can the authors explain this difference and clarify how it affects the reliability of the operation-level comparisons?

  • Does the observed selection–validation gap remain consistent on newly sampled search states and unseen evaluation questions? Can the authors provide similar independent validation evidence for the other agent design frameworks?

Reviewer_T7EVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work introduces a protocol for evaluating a search operation within automatic agent design. This protocol quantifies the gap between the apparent and independently evaluated advantage of finite-budget operation selection in the methods. The search operation is applied to GPTSwarm, AFlow, MaAS, and EvoMAS over six tasks. The experimental results reveal that a high-scoring candidate may not lead to a final valuable system decision.

Critical strengths
  1. The protocol provides a good way to quantify the improvement from each search decision during the optimization.
  2. The protocol is evaluated on a few representative automatic agent design methods.
Critical weaknesses
  1. Generally speaking, the writing in this work is very hard to follow, and the presentation requires improvement.

  2. The work brings marginal technical contributions and rarely discusses how this protocol can become beneficial to automatic agent design methods.

Questions for authors

Could you please justify how this protocol will be crucial for the development of automatic agent design approach?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.