acceptodds
Under review as a conference paper at ICLR 2027

Decomposing Adaptive LLM Team Gains with a Frozen-Team Counterfactual

Abstract

Adaptive LLM teams combine two interventions: a fixed collaboration policy and a search procedure that adapts it. Standard evolved-versus-single comparisons conflate their contributions. We introduce a three-arm evaluation that measures Team Lift (frozen team minus single-agent policy) and Evolution Lift (evolved team minus the same frozen starting team) under a common allocated token budget. On a static SWE-derived patch-output proxy—scored from generated patch text without running repository tests—the evolved team exceeds the single agent by 6.875 percentage points across the observed runs. Arithmetically, this total consists of 5.350 points of Team Lift and 1.525 points of Evolution Lift; on GPQA, the corresponding decomposition is 3.0=3.8−0.8 points. Exploratory controls show that fixed policies with calibrated or deterministic selection can reach or exceed the evolved policy’s proxy score, while cold-start traces motivate broader proposal coverage. Rerun-aware and execution-based analyses distinguish uncertainty over search trajectories from validity of the evaluation endpoint. These results motivate a practical standard for collaboration search: report whether adaptation adds repeatable, task-valid value beyond a strong frozen policy instead of attributing the full team gain to evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, LLM agents, agentic systems, structure evolution, self-improvement, test-time scaling, allocated-budget evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GxsqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an in-depth study of the effect of team adaptation by separating team lift from evolution lift.

Critical strengths
  • The decomposition makes sense.
Critical weaknesses
  • There are many other dimensions that could affect the performance. It is unclear why this particular setup provides valuable insights.
Questions for authors

What are the effects of models?

1 comment

Dear Reviewer Gxsq,

Thank you for your careful review and for raising questions about the controls and models. We appreciate your support for measuring team gains separately from the incremental gain from evolution. We have made targeted changes to the description of the evaluation design and plan to include them in the next revision.


  1. Why three arms? All three arms share the same backbone, task splits, and evaluator. We match token allocations, not actual usage. The score difference between the frozen team and the single agent measures the gain from a fixed team policy. The evolved and frozen arms start from the same team, so their score difference measures the incremental gain from allocating part of the budget to structural search. This comparison evaluates complete policies rather than isolating the effect of any individual role or edit.
  2. Sensitivity to the model backend. On GPQA, at the low and high budgets respectively, the score differences (evolved minus frozen) were −3.2 and +1.6 percentage points on the primary backend (Table 3), and −15.3 and −14.0 percentage points on Qwen3-8B. With Qwen3-8B, the three arms scored similarly on the SWE patch-output proxy (Appendix Table 14). The revision will highlight these results: score differences between policies are sensitive to the backend setting, and this comparison does not measure model capability in isolation.

Thank you again for your review and suggestions.

Reviewer_u9JyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper notes that multi-agent improvement in literature is often confounded by not properly ablating against frozen team initialisation (pre evolution) and single agent performance. They instantiate this decomposition on SWE bench verified and GPQA and report the results of the decomposition.

Critical strengths
  • They argue for a decomposition that should be standard in literature
Critical weaknesses
  • This is not novel - there are many blog posts and papers that now point to properly controlling in multi-agent teams. For instance, see https://iclr.cc/virtual/2025/poster/31346
  • Figure 2 is poorly rendered with overlapping elements
  • The evolution protocol is not novel
Questions for authors

No questions, but rejection on grounds of no novelty and restatement of what previous works have done.

1 comment

Dear Reviewer u9Jy,

Thank you for reviewing our manuscript and pointing out relevant prior work. We appreciate your support for making it standard practice to report team gains separately from gains due to search. We have made targeted changes to how we describe our contribution and to the figures, and plan to include them in the next revision.


  1. Prior work and contribution. The related work section now cites Hangfan Zhang et al. (ICLR 2025 Blog Track) for their comparison of multi-agent debate and strong single-agent baselines, which also examined inference efficiency. We count search toward the same total budget and use a frozen team with the same starting configuration as a control. This comparison shows that the frozen team accounts for most of the gain on the static SWE patch-output proxy. Our contribution is to separate the gain from using a fixed team from the incremental gain from search, not to propose a new algorithm for team evolution.
  2. Figures. We redrew the score–cost plot (original Figure 2, revised Figure 3), offsetting the point labels and using different marker shapes to distinguish the three arms from the supplementary fixed policies.
  3. AI use. The manuscript’s AI Use Statement discloses how generative AI helped refine the research framework and hypotheses, provide feedback on experimental design, and assist with coding, analysis, figure preparation, and writing. It also describes verification steps, including code testing, citation checks, recomputing results, and cross-checking against archived records. The authors take responsibility for the final content.

Thank you again for your review and for raising these points.

Reviewer_njA5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce a three-arm evaluation framework for adaptive LLM teams, comparing a single agent, a frozen team, and an evolved team. The authors decompose the overall performance gain into Team Lift (frozen minus single) and Evolution Lift (evolved minus frozen). The purpose is to distinguish the benefit of a fixed collaboration structure from that of structure search. The framework is evaluated using SwarmEvolve on a SWE-derived patch-output proxy and GPQA under matched allocated token budgets. The experimental results show that most of the observed improvement over the single agent is already achieved by the frozen team, while the additional gain from evolution is small and statistically uncertain.

Critical strengths

I think the paper targets an important issue in adaptive multi-agent research, that is to say, improvement over a single-agent baseline does not necessarily demonstrate the added value of structure search. Introducing a frozen-team control provides a useful comparison for evaluating this incremental benefit. I also found that the experiment is quite carefully designed and analyzed.

Critical weaknesses

I have several concerns as follows: (1) The main contribution is a three-arm evaluation that decomposes performance differences between single-agent, frozen-team, and evolved-team policies. While useful, I am concerning about its methodological novelty. Prior studies have already examined optimized and unoptimized agent structures (e.g., GPTSwarm), with/without-evolution comparisons (e.g., Mobile-Agent-E), and controlled evaluations of fixed and evolving multi-agent workflows (e.g., BenchAgent). Although these studies do not implement the same matched three-arm decomposition, they establish relevant precedents for evaluating the incremental value of optimization or evolution. In addition, the proposed decomposition measures differences between complete system policies rather than identifying the underlying sources of performance improvement. It remains unclear what substantive methodological advance this framework provides beyond existing controlled ablations. (2) The empirical evaluation relies primarily on SwarmEvolve, a single failure-guided structure-search framework. The additional baselines and ablations examine fixed policies or variations of the same search procedure, rather than independent evolution methods. Furthermore, although the paper is framed around adaptive LLM teams, adaptation can involve mechanisms beyond structure evolution, for which different frozen controls may be required. The applicability and additional value of the proposed evaluation framework across other evolution or adaptation approaches remain insufficiently demonstrated.

Besides, I think the paper was not well written and difficult to follow. For example, the narrative moves between the three-arm decomposition, fixed-policy comparisons, search-rerun uncertainty, and proposal-coverage analysis without clearly distinguishing the primary research questions from exploratory findings. Figure 1, Figures 2 and 5 are particularly difficult to interpret. For example, text jumping out from the boxes, the arrow crossing boxes. The font size, font types are not consistent with those in the manuscript.

Questions for authors

1.While the proposed three-arm protocol offers a more systematic decomposition with budget accounting and uncertainty analysis, what substantive methodological advance does it provide beyond existing controlled ablations? In particular, what new insights into the sources of performance improvement can be obtained from these policy-level comparisons? 2. The evaluation is demonstrated primarily with SwarmEvolve, which implements one form of collaboration-structure evolution. How broadly are the conclusions intended to apply? What evidence supports the applicability and usefulness of the proposed framework to other evolution or adaptation mechanisms that may require different frozen counterfactuals? 3. The reported Evolution Lift is statistically uncertain, while the SWE proxy does not directly measure executable repair success. Given these limitations and the competitive fixed-policy baselines, what robust conclusions about the practical value of structure evolution are supported by the experiments?

1 comment

Dear Reviewer njA5,

Thank you for your careful review and constructive suggestions. We appreciate your recognition of the value of the frozen-team control and the care taken in the experimental design. In response to your comments on the contribution, scope, and presentation, we have made targeted changes that we plan to incorporate into the revised manuscript.


  1. W1 and Q1: How this differs from prior ablations. GPTSwarm, Mobile-Agent-E, and BenchAgent set important precedents. We charge search and execution to the same token budget: the single agent is the reference for the evolved team's total gain, and the frozen team with the same starting structure is the reference for the gain from search. We also account for variation across search reruns to assess whether that additional gain is repeatable. The comparison separates the gain associated with the fixed team from the additional gain associated with search; it does not isolate the effect of an individual role or edit.
  2. W2 and Q2: Scope. Our experiments test structure search in SwarmEvolve. In the revision, we will use prompt search and tool-policy search to illustrate how to construct a frozen control by retaining the initial configuration and matching the token allocation.
  3. Q3: Practical takeaway. On the static SWE patch-output proxy, with equal weight across the two budget tiers, the frozen team scored 5.350 percentage points above the single agent, and the evolved team scored 1.525 points above the frozen team. The 95% interval for the latter difference spans zero when search reruns are accounted for. In our exploratory comparison, a fixed policy based on parallel sampling and selection outscored the evolved team at both budget tiers. We therefore recommend benchmarking against strong fixed policies under the same token allocation before devoting part of that budget to search. The revision will make clear that the proxy scores patch output; it does not test whether applying the patch fixes the issue.
  4. Presentation. We redrew Figures 1, 2, and 5 from the original submission, standardized typography within the figures, and adjusted the layout of labels, text boxes, and arrows. We now present the primary comparisons separately from the exploratory analyses.

Thank you again for your thoughtful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.