I have several concerns as follows:
(1) The main contribution is a three-arm evaluation that decomposes performance differences between single-agent, frozen-team, and evolved-team policies. While useful, I am concerning about its methodological novelty. Prior studies have already examined optimized and unoptimized agent structures (e.g., GPTSwarm), with/without-evolution comparisons (e.g., Mobile-Agent-E), and controlled evaluations of fixed and evolving multi-agent workflows (e.g., BenchAgent). Although these studies do not implement the same matched three-arm decomposition, they establish relevant precedents for evaluating the incremental value of optimization or evolution. In addition, the proposed decomposition measures differences between complete system policies rather than identifying the underlying sources of performance improvement. It remains unclear what substantive methodological advance this framework provides beyond existing controlled ablations.
(2) The empirical evaluation relies primarily on SwarmEvolve, a single failure-guided structure-search framework. The additional baselines and ablations examine fixed policies or variations of the same search procedure, rather than independent evolution methods. Furthermore, although the paper is framed around adaptive LLM teams, adaptation can involve mechanisms beyond structure evolution, for which different frozen controls may be required. The applicability and additional value of the proposed evaluation framework across other evolution or adaptation approaches remain insufficiently demonstrated.
Besides, I think the paper was not well written and difficult to follow. For example, the narrative moves between the three-arm decomposition, fixed-policy comparisons, search-rerun uncertainty, and proposal-coverage analysis without clearly distinguishing the primary research questions from exploratory findings. Figure 1, Figures 2 and 5 are particularly difficult to interpret. For example, text jumping out from the boxes, the arrow crossing boxes. The font size, font types are not consistent with those in the manuscript.
Dear Reviewer Gxsq,
Thank you for your careful review and for raising questions about the controls and models. We appreciate your support for measuring team gains separately from the incremental gain from evolution. We have made targeted changes to the description of the evaluation design and plan to include them in the next revision.
Thank you again for your review and suggestions.