- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces an operator-level evaluation protocol for automatic agent design, addressing the limitation that a strong final agent does not establish that the search decisions leading to it were valuable. The protocol restores a recorded optimiser state, replaces one operation with a prespecified alternative, and resumes the native optimiser under the same objective and computational budget. It distinguishes three questions: whether an operation changes the subsequent search trajectories, whether it improves downstream performance, and whether its apparent advantage generalises to independent executions. The authors apply the protocol to four agent-design frameworks: GPTSwarm, AFlow, MaAS, EvoMAS, across six benchmarks. The primary GPTSwarm-MMLU study evaluates 480 executions across 20 frozen states, using same-operation controls and separate selection and validation waves. The apparent advantage of operations selected using four repetitions is 2.17 % points, but falls to 0.65 points on independent validation. The paired selection-validation gap is 1.52 points. GPTSwarm and EvoMAS experiments also show measurable changes in subsequent search trajectories, but the link between trajectory changes and downstream value remains uncertain. The main contribution is a generalisable experimental methodology for evaluating individual search decisions through counterfactual replay, native optimisation continuation, repeated-execution controls, and independent validation. The results highlight selection bias and demonstrate that changing a search trajectory does not necessarily improve performance. However, the strongest independent validation is limited to 20 GPTSwarm states, one model route, and a fixed set of 153 MMLU evaluation terms, leaving broader generalisability unresolved.
Critical strengths
-
A principled protocol for evaluating individual search decisions. The paper introduces a clear and generalisable framework that restores a recorded optimiser state, intervenes on a single operation and continues optimisation under the original procedure. By combining counterfactual comparisons, same-operation controls, and independent validation, it distinguishes search trajectory changes from downstream performance improvements and selection bias. This is a meaningful methodological contribution to evaluating automatic agent-design systems.
-
Empirical demonstrations of the gap between apparent and independently validated value. Across four agent-design frameworks, the authors investigate whether individual operations influence subsequent search and whether these changes translate to downstream benefits. The GPTSwarm-MMLU study is particularly informative: an apparent selection advantage of 2.17 points falls to 0.65 points on independent validation, with a 95% confidence interval spanning 0. The result illustrates the importance of independent evaluation and challenges the assumption that locally promising search decisions reliably improve downstream outcomes.
Critical weaknesses
-
Limited Evidence for the paper’s central empirical conclusion. The strongest independent validation study uses only 20 GPTSwarm states, one model route, and a fixed set of 153 MMLU evaluation items. The estimated advantage of the selected operation drops from 2.17 points during selection to 0.65 during independent validation, with the validation confidence interval spanning 0. Although this demonstrates the possibility of selection bias, the small validation cohort and narrow evaluation setting make it difficult to establish how consistently this effect generalises across states, tasks and search algorithms. This limits the strength of the paper’s broader empirical conclusions.
-
The relationship between trajectory changes and downstream value remains weakly established. A central motivation is to distinguish whether a search operation changes the optimisation trajectory from whether it improves the final agent. However, the reported local signals - including graph displacement, immediate score differences, and gradient-based surrogates - show weak and imprecise associations with independently validated performance. In some settings, trajectory separation is measurable while task-level performance contrasts are zero or too coarse to resolve. The methodology therefore provides a useful framework for evaluating search operations, but the paper offers limited evidence that its measurements can reliably identify valuable operations or support better search decisions in practice.
Questions for authors
-
Clarification: Robustness of independent validation results. The estimated advantage of the selected operation decreases from 2.17 to 0.65 % points in Wave 2, with the latter’s confidence spanning 0. How much statistical power does the 20-state validation study provide to distinguish a genuinely positive effect from no effect? Do the conclusions remain consistent under alternative aggregation or uncertainty-estimation procedures?
-
Clarification: Interpretation of the Native-A/Native-B offset. The same operation control and exhibit a signed mean difference of -1 % point, suggesting a non-negligible variability event without changing the selected operation. How does this execution offset affect the estimated counterfactual treatment effect, and what evidence supports attributing observed differences to the operation intervention rather than stochastic variation in continuation?
-
Clarification: Generalisability across search frameworks and tasks. The strongest independent validation comes from GPTSwarm on a fixed MMLU evaluation set, while some other settings have coarse or uninformative task-level performance contrasts. What evidence supports extending the conclusions about operational value and selection reliability beyond this particular model route and evaluation setting?