acceptodds
Under review as a conference paper at ICLR 2027

Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systems

Abstract

LLM-based multi-agent systems (MAS) have emerged as an effective paradigm for complex and long-horizon tasks. However, in real-world tasks, MAS often exhibit various failures during execution and such failures are difficult to eliminate during design. This motivates experience-driven MAS evolution, where a system improves based on its own execution experience. Yet such evolution is challenging because MAS experience is prolonged and intricate, interleaving multiple agents’ execution chains and communication messages, which makes it difficult to identify what should be improved. To address this challenge, we propose Meta-Team, an experience-driven MAS evolution framework based on collaborative self-evolution. Meta-Team preserves the execution context of each agent and coordinates post-task communication, enabling agents to exchange distributed evidence for evolution. Building on this design, Meta-Team conducts multi-scale self-evolution, transforming execution experience into reusable improvements to agent behaviors, inter-agent coordination, and team-level organization. Across six long-horizon agent benchmarks, Meta-Team consistently outperforms single-agent systems, hand-crafted MAS, and prior MAS evolution methods; further analyses demonstrate that Meta-Team enables more reliable and scalable MAS self-evolution.\footnotetext[3]Code is available at https://anonymous.4open.science/r/Meta-Team-CD4D.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, LLM, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U6frOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Meta-Team, an experience-driven framework for optimizing LLM-based multi-agent systems. It preserves agent-local execution contexts and coordinates post-task evidence exchange to update individual behaviors and skills, teammate profiles and interaction notes, and team organization and shared rules. In the main experiments, configurations evolve on approximately 20 instances per benchmark or subset and are then frozen for evaluation on disjoint instances. Validation covers six long-horizon agent benchmarks, failure attribution, component ablations, context-length and cross-language evaluation, and additional backbone and evolution-stability experiments.

Critical strengths
  1. meta-Team has consistent empirical gains on multiple challenging task Settings: Table 1 includes several existing evolution methods, and Meta-Team achieves the highest reported mean in all nine columns. Its cross-column average is 62.7, compared with 56.4 for MASFly and 57.8 for AOrchestra. Paired significance tests and repeated full evolution runs on three representative settings provide additional support. This establishes credible practical value for task-family adaptation.
  2. There are key ablation experiments that support the core claim. In Table 2, collaborative experience organization exceeds the centralized control by 4.1 points on Ansible and 2.9 points on ResearchRubrics, and also outperforms isolated local analysis. Removing any update scope in Table 3 reduces performance. These experiments provide direct evidence for the practical usefulness of the proposed claimed design.
Critical weaknesses
  1. Centralized control can also be more comprehensive: Table 2 supports the effectiveness of the method. However, the centralized baseline uses only one analysis call, while the collaborative method allows repeated exchanges and checks. It remains unclear how much of the advantage would remain if the centralized baseline could make several passes under a comparable reflection budget.
  2. Proposition 1 lacks necessary conditions: The assumptions in Appendix C.2 do not guarantee the crossover claimed in Proposition 1. For example, if the recovery rate stays at 1, global analysis never becomes worse than local analysis. The proof also uses continuity and a recovery rate approaching zero, neither of which is stated in the assumptions.
Questions for authors
  1. How many model calls and how much cost did each Table 2 variant use for reflection? Does the centralized variant have to diagnose the problem and generate configuration updates in the same call?
  2. On Ansible or ResearchRubrics, could you allow the centralized baseline several analysis passes under a comparable reflection budget and check whether the advantage remains?
  3. Could you add the conditions needed for Proposition 1, or revise the claim to address the counterexample above?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer U6fr [1/2]

We sincerely thank the reviewer for recognizing our empirical results and ablations, and for the concrete suggestions on the centralized control and the theoretical analysis. Key revisions in the revised manuscript are marked in blue.

[W1 & Q1 & Q2] Centralized control can also be more comprehensive: Table 2 supports the effectiveness of the method. However, the centralized baseline uses only one analysis call, while the collaborative method allows repeated exchanges and checks. It remains unclear how much of the advantage would remain if the centralized baseline could make several passes under a comparable reflection budget.

How many model calls and how much cost did each Table 2 variant use for reflection? Does the centralized variant have to diagnose the problem and generate configuration updates in the same call?

On Ansible or ResearchRubrics, could you allow the centralized baseline several analysis passes under a comparable reflection budget and check whether the advantage remains?

[Response] Even with separate diagnosis and update calls and a matched reflection budget, Centralized gains almost nothing and remains clearly below Meta-Team. We address these questions in three parts: (1) how the original Centralized is called; (2) the calls and reflection cost of each scheme; and (3) multi-pass Centralized under a comparable budget.

(1) The original Centralized performs diagnosis and update generation in a single call. As described in Section 4.3, Centralized feeds the flattened whole-team trajectory to a single LLM analyzer call, which both diagnoses the failure and generates configuration updates. The new multi-pass variant separates the two steps into different calls and allows multiple rounds of self-critique and revision.

(2) Calls and reflection cost. The original Centralized calls the analyzer once per task; Partitioned assigns one analyzer to each agent; Collaborative additionally exchanges evidence on demand. The evolution overhead of Collaborative is $13 on Ansible and $25 on ResearchRubrics (Table 13 in the manuscript). Agents exchange evidence only when needed, and pairs without problematic interactions do not communicate, so the cost does not grow quadratically with team size.

(3) Budget-matched multi-pass Centralized. Following the reviewer's suggestion, we let Centralized repeatedly run "diagnose → generate updates → self-critique → revise" over the same flattened trajectory on both Ansible and ResearchRubrics, until its reflection cost matches that of Collaborative. All other conditions are identical to the original experiment, including the feedback, the L1/L2/L3 update space, and the tools.

Table R1: Comparison of experience schemes, including the budget-matched multi-pass Centralized variant. The evolution overhead of the multi-pass variant is matched to that of Collaborative ($13 on Ansible and $25 on ResearchRubrics, Table 13 in the manuscript).

Experience schemeAnsibleResRub.
Centralized (single call)49.852.9
Centralized (multi-pass, budget-matched)50.252.4
Partitioned44.551.0
Collaborative (Meta-Team)53.955.8

The advantage remains under a comparable reflection budget. Multi-pass Centralized only rises from 49.8 to 50.2 on Ansible and drops from 52.9 to 52.4 on ResearchRubrics, remaining 3.7 and 3.4 points below Collaborative. Moreover, Partitioned and Collaborative use the same number of analyzers over the same local trajectories, yet Collaborative is still higher by 9.4 and 4.8 points, showing that the gain comes from communication rather than more calls.

The bottleneck is that a single context must read the entire interleaved long trajectory, and more analysis passes do not remove this burden. This is consistent with the sharp drop of the global scheme on trajectories longer than 128K in Figure 1. Table R1 is included as Table 15 in Appendix F.2 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal to Reviewer U6fr [2/2]

[W2 & Q3] Proposition 1 lacks necessary conditions: The assumptions in Appendix C.2 do not guarantee the crossover claimed in Proposition 1. For example, if the recovery rate stays at 1, global analysis never becomes worse than local analysis. The proof also uses continuity and a recovery rate approaching zero, neither of which is stated in the assumptions.

Could you add the conditions needed for Proposition 1, or revise the claim to address the counterexample above?

[Response] We thank the reviewer for the careful examination. In Appendix C.2 of the revised manuscript, we state the conditions of Proposition 1 explicitly and give a sharper proposition and proof. The conclusion is unchanged: on sufficiently long trajectories, global analysis is strictly worse than local analysis, while collaborative analysis is at least as good as both at any length and strictly better than global analysis on long trajectories.

Added assumptions.

  • (A3) Context dilution vanishes: In addition to Assumption 2 (ρ is non-increasing), we require lim⁡x→∞ρ(x)=0\lim_{x\to\infty}\rho(x)=0. This rules out the counterexample ρ ≡ 1.
  • (A4) Trajectories grow longer through more division of labor: The total length L grows with the number of agents N; each agent's local trajectory is bounded, ℓi≤ℓmax⁡\ell_i \le \ell_{\max}; local evidence is bounded below, si≥smin⁡>0s_i \ge s_{\min} > 0; and interaction evidence grows at most linearly with the number of agents, B≤cNB \le cN.

Revised Proposition 1.

  • Long trajectories: Under (A3)–(A4), there exists a finite LcL_c such that Vglobal<VlocalV_{\text{global}} < V_{\text{local}} for all L>LcL > L_c.
  • Short trajectories: At any length L0L_0 with ρ(L0) B≥(1−ρ(L0)) S\rho(L_0)\,B \ge \big(1-\rho(L_0)\big)\,S, we have Vglobal≥VlocalV_{\text{global}} \ge V_{\text{local}}. That is, the global scheme is preferable when context dilution is weak and the interaction evidence B>0B>0 is substantial.

Proof sketch. By (A4),

f(L)=Vglobal−Vlocal≤N[c ρ(L)−smin⁡(ρ(ℓmax⁡)−ρ(L))].f(L) = V_{\text{global}} - V_{\text{local}} \le N\big[c\,\rho(L) - s_{\min}\big(\rho(\ell_{\max}) - \rho(L)\big)\big].

By (A3), the right-hand side is negative once ρ(L)<smin⁡ ρ(ℓmax⁡)/(c+smin⁡)\rho(L) < s_{\min}\,\rho(\ell_{\max})/(c + s_{\min}), so f(L)<0f(L) < 0 for all sufficiently large L. Taking Lc=sup⁡{L:f(L)≥0}L_c = \sup\{L : f(L) \ge 0\} completes the proof, which no longer requires continuity.

Proposition 3 is strengthened accordingly. Whenever ℓi,ℓ⋆≤L\ell_i, \ell^\star \le L, and under (A4), the advantage of Collaborative over Global satisfies

Vcollab−Vglobal≥N smin⁡(ρ(ℓmax⁡)−ρ(L)),V_{\text{collab}} - V_{\text{global}} \ge N\,s_{\min}\big(\rho(\ell_{\max}) - \rho(L)\big),

which is positive for sufficiently large L and grows with the number of agents N.

This result matches Figure 1: global falls behind local on trajectories longer than 128K, while collaborative is the best in both length regimes.

We are grateful to the reviewer for the careful reading and valuable suggestions. We hope our responses have resolved the reviewer's concerns, and we would welcome any further feedback.

AuthorsOpenReview ↗

Author Follow-up for Reviewer U6fr

Dear Reviewer U6fr,

Thank you again for your time and effort in reviewing our submission.

As the discussion period will end in less than a day, we would like to kindly remind you that we have posted our responses to all of your comments, including the budget-matched multi-pass centralized baseline you suggested and the revised conditions and proof for Proposition 1. We hope they address your concerns, and we would be happy to clarify anything further.

We sincerely appreciate your valuable suggestions and thank you again for your time and consideration.

Thanks for adding the budget-matched experiment. It addresses my concern about the single-call baseline. I’m still unsure whether Meta-Team would retain its advantage over a centralized method that can selectively inspect the same local evidence. On balance, I’m keeping my rating at 2.

Reviewer_4cKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether agents can learn better from execution experience by keeping their local contexts and exchanging evidence after a task. Meta-Team updates 1) individual behavior and skills, 2) teammate profiles, and 3) team roles and shared rules without changing LLM weights. It evolves teams on small task sets, then freezes the updates for evaluation on unseen tasks. Using Claude Sonnet 4.6, it compares against nine baselines on selected subsets of SWE-bench Pro, BeyondSWE, LOCA-Bench, GAIA, LoCoBench, and ResearchRubrics. It reports the best scores in all nine benchmarks, averaging 62.7 versus 56.1 for the initial MAS and 56.4 for MASFly. Validation includes failure attribution, ablations, transfer across context lengths and programming languages, and a smaller GPT-5.5 evaluation. The paper also reports paired significance tests, three independent evolution runs, and favorable performance-cost trade-offs under reduced evolution budgets.

Critical strengths
  1. The paired significance tests and three independent evolution runs are useful, distinguishing significant gains on six columns from nonsignificant gains on three.
  2. The reduced-budget evaluation and separate evolution and deployment costs help assess practical value. Meta-Team reports the best performance-cost trade-off on four settings.
  3. The paper addresses a practical problem: improving multi-agent systems from their execution failures without retraining the underlying models.
  4. The evaluation covers software engineering, tool use, and open-ended research, with additional checks across context lengths, programming languages, and a second backbone.
Critical weaknesses
  1. Centralized reflection in Table 2 already beats every Table 1 baseline on both tested benchmarks: 49.8 on Ansible and 52.9 on ResearchRubrics. Collaboration adds 4.1 and 2.9 points, but the centralized control uses only one analyzer call. It brings question to whether Table 1 effective proves the superiority of Meta-team as a framework or it's mainly the confounding of more turns / reflections / ICL effects in the centralized context of Meta-team. It may be more helpful to 1) compare different ways of organizing the same experience with matched feedback, update options, length, and reflection budgets, and 2) prior methods much closer to Meta-team listed below. This would help distinguish better context organization from extra rethinking.
  2. The comprehensiveness of related work / novelty of idea. Please clarify L3's novelty relative to LIET, which updates shared cooperation guidance; Criticize–Reflect, which revises organizational prompts; and EvoMAC, which also changes agent rosters and workflows. These methods overlap with Meta-team's ingredients more than some currently listed related work.
Questions for authors
  1. How were Figure 1 and Tables 2–3 averaged? For example, 106 traces with three runs each give 318 binary outcomes, which cannot produce 77.5% after rounding. Were different sample counts or averaging rules used?
  2. Does each agent initially receive only its own trace or everyone's traces? The main text describes local contexts, but Appendix D.2 says preserved traces are broadcast to all agents. Please clarify what is shared before agents communicate.
  3. I'd like to see some more qualitative sample of how agents evolve L3 team structure. How do you ensure that L1 and L2 config do not break upon L3 changes?
6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 4cKa [1/3]

We sincerely thank the reviewer for appreciating our significance tests, cost analysis, and evaluation coverage, and for the constructive questions. The main revisions in the revised manuscript are highlighted in blue.

[W1] Centralized reflection in Table 2 already beats every Table 1 baseline on both tested benchmarks: 49.8 on Ansible and 52.9 on ResearchRubrics. Collaboration adds 4.1 and 2.9 points, but the centralized control uses only one analyzer call. It brings question to whether Table 1 effective proves the superiority of Meta-team as a framework or it's mainly the confounding of more turns / reflections / ICL effects in the centralized context of Meta-team. It may be more helpful to 1) compare different ways of organizing the same experience with matched feedback, update options, length, and reflection budgets, and 2) prior methods much closer to Meta-team listed below. This would help distinguish better context organization from extra rethinking.

[Response] Meta-Team's advantage comes from how experience is organized, not from extra reflection: even with a matched reflection budget, a multi-pass centralized analyzer gains almost nothing and remains clearly below Meta-Team. We address this concern in three parts: (1) why Centralized already outperforms the Table 1 baselines; (2) whether Collaborative remains better when feedback, update options, length, and reflection budget are matched; and (3) whether Meta-Team's advantage over the Table 1 baselines comes from a larger budget. The comparison with the listed prior methods is discussed in [W2].

(1) Centralized is a variant of Meta-Team itself, and its strong results reflect the effectiveness of multi-scale evolution. Centralized in Table 2 is not an external baseline: it shares the same L1/L2/L3 update space, tools, and pre-commit checks with Meta-Team, and differs only in how experience is organized (Section 4.3). Meta-Team's advantage therefore comes from two complementary designs, each verified by an ablation:

  • Multi-scale evolution (Table 3): Removing any level lowers performance. Centralized also has this design, which is why it already outperforms the Table 1 baselines.
  • Collaborative experience organization (Table 2): With the same multi-scale evolution, Collaborative further improves over Centralized by 4.1 and 2.9 points on Ansible and ResearchRubrics.

(2) Under matched conditions, the advantage comes from how experience is organized, not from more calls. All schemes in Table 2 already read the same execution experience and use the same feedback, update space, and tools; they differ only in how experience is organized. To address the reflection budget, we add a budget-matched multi-pass Centralized variant on Ansible and ResearchRubrics: it repeatedly runs "diagnose → generate updates → self-critique → revise" over the same flattened trajectory until its reflection cost matches that of Collaborative.

Table R1: Comparison of experience schemes, including the budget-matched multi-pass Centralized variant. The evolution overhead of the multi-pass variant is matched to that of Collaborative ($13 on Ansible and $25 on ResearchRubrics, Table 13 in the manuscript).

Experience schemeAnsibleResRub.
Centralized (single call)49.852.9
Centralized (multi-pass, budget-matched)50.252.4
Partitioned44.551.0
Collaborative (Meta-Team)53.955.8
  • More analysis passes bring almost no gain. With a matched budget, multi-pass Centralized only rises from 49.8 to 50.2 on Ansible and drops from 52.9 to 52.4 on ResearchRubrics, remaining 3.7 and 3.4 points below Collaborative. The bottleneck is that a single context must read the entire interleaved long trajectory, and reading it multiple times does not remove this burden. This is consistent with the sharp drop of the global scheme on trajectories longer than 128K in Figure 1.
  • The number of calls is not the deciding factor. Partitioned and Collaborative both assign one analyzer per agent over the same local trajectories and differ only in communication, yet Collaborative is higher by 9.4 and 4.8 points.

Table R1 is included as Table 15 in Appendix F.2 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal to Reviewer 4cKa [2/3]

(3) Meta-Team's advantage over the Table 1 baselines does not come from a larger budget.

  • Comparable cost and higher performance than single-analyzer evolution. MASFly [1] also revises the MAS with a single analyzer. The total evolution cost is $348 for Meta-Team and $268 for MASFly (Table 14 in the manuscript), while Meta-Team is 6.3 points higher on average.
  • The advantage holds under a reduced budget. When the evolution budget of all methods is reduced to 1/3 of the main setting, Meta-Team still achieves the best performance–cost trade-off on all four benchmarks (Figure 4 in the manuscript).

[1] Liu, Guangyi, et al. "MAS-on-the-Fly: Dynamic Adaptation of LLM-based Multi-Agent Systems at Test Time." arXiv:2602.13671, 2026.

[W2] The comprehensiveness of related work / novelty of idea. Please clarify L3's novelty relative to LIET, which updates shared cooperation guidance; Criticize–Reflect, which revises organizational prompts; and EvoMAC, which also changes agent rosters and workflows. These methods overlap with Meta-team's ingredients more than some currently listed related work.

[Response] We thank the reviewer for pointing out these related works, which are now discussed in the Related Work (Section 5.2) of the revised manuscript. We address this concern in two parts: (1) the novelty of L3; and (2) the comparison with these methods.

(1) L3 differs not in what it revises, but in who revises it and based on what evidence. In Criticize–Reflect and EvoMAC, a single analyzer reads the full experience and revises the organization. In LIET, agents do reflect, but they only update a shared list that guides communication, without changing the team composition or structure. In Meta-Team, team-level revisions arise bottom-up: each agent proposes improvements based on its own preserved local execution context, and the chairman makes the final decision by combining these proposals with the L1/L2 conclusions.

Table R2: Comparison of L3 with related methods.

MethodWho analyzes the experienceWhat evolvesScope of evolution
LIET [2]Message receivers reflect on received messages and update a shared listA shared cooperation knowledge list that guides communication; roster and structure fixedAccumulated across episodes in embodied environments
Criticize–Reflect [3]A single critic reads the full dialogue and action history; a coordinator writes a new promptOrganizational prompt; roster fixedIteratively refines the organizational prompt
EvoMAC [4]A single gradient agent analyzes test feedback; an updating agent revises the networkAgent prompts, roster, workflowIterated at test time for each task; not reused across tasks
Meta-Team (L3)Each agent proposes improvements from its local trajectory; the chairman decides with L1/L2 conclusionsRoster, organizational structure, shared constitutionAccumulated across tasks; frozen and reused on held-out tasks

Moreover, L3 co-evolves with L1 (agent behavior) and L2 (collaboration), and the complementary roles of the three levels are verified in Table 3 in the manuscript.

(2) The core mechanisms of these methods are already compared under controlled conditions. LIET and Criticize–Reflect target embodied environments, and EvoMAC iterates within a single task without producing a configuration reusable across tasks. None of them directly applies to our setting of evolving on an evolution set and evaluating the frozen system on a held-out set. Their core mechanisms are already compared under controlled conditions:

  • Criticize–Reflect and EvoMAC: A single analyzer reads the full experience and revises the organization. This is exactly what Centralized implements under identical conditions in Table 2, and MASFly, a method of the same type, is included in Table 1. Collaborative performs better in both comparisons.
  • LIET: It only updates shared collaboration knowledge without changing the team composition or structure, which corresponds to removing team-level evolution. In Table 3, removing L3 lowers performance by 2.0 and 3.7 points on Ansible and ResearchRubrics, respectively.

[2] Li, Xinran, et al. Learn as Individuals, Evolve as a Team: Multi-agent LLMs Adaptation in Embodied Environments. arXiv:2506.07232, 2025.

[3] Guo, Xudong, et al. Embodied LLM Agents Learn to Cooperate in Organized Teams. arXiv:2403.12482, 2024.

[4] Hu, Yue, et al. Self-Evolving Multi-Agent Collaboration Networks for Software Development. ICLR, 2025.

AuthorsOpenReview ↗

Rebuttal to Reviewer 4cKa [3/3]

[Q1] How were Figure 1 and Tables 2–3 averaged? For example, 106 traces with three runs each give 318 binary outcomes, which cannot produce 77.5% after rounding. Were different sample counts or averaging rules used?

[Response] All results are computed as avg@3: each instance is run three times independently, the results are first averaged over the three runs, and then arithmetically averaged over instances.

  • Figure 1: The 220 TraceElephant traces are split by length into 106 traces (≤128K) and 114 traces (>128K). Within each split, Agent Accuracy and Step Accuracy are exact-match rates against the annotated (mistake_agent, mistake_step) tuple, averaged as above (Appendix C.1).
  • Tables 2–3: As in Table 1, results are computed as avg@3 on the held-out evaluation set of each benchmark (Section 4.1).

The revised manuscript states this computation explicitly in Appendix C.1.

[Q2] Does each agent initially receive only its own trace or everyone's traces? The main text describes local contexts, but Appendix D.2 says preserved traces are broadcast to all agents. Please clarify what is shared before agents communicate.

[Response] Before communication, each agent receives only its own execution trace, together with the outcome feedback shared by all agents (the final deliverable, the evaluation verdict, an error diagnosis, and pairwise communication summaries). Other agents' traces are not placed in its context; when needed, an agent requests evidence from the relevant teammates on demand during communication (Appendix D.2, L1). This design is precisely how Meta-Team avoids the single-context bottleneck. We have made this explicit in Appendix D.2 of the revised manuscript.

[Q3] I'd like to see some more qualitative sample of how agents evolve L3 team structure. How do you ensure that L1 and L2 config do not break upon L3 changes?

[Response] We address this question in two parts: (1) qualitative examples of L3 evolution; and (2) how L3 changes avoid breaking the L1/L2 configurations.

(1) Examples of L3 evolution. We give two representative examples below; the revised manuscript adds the LOCA-Bench example to Appendix D.3.

  • LOCA-Bench: adjusting team size. During evolution on 96K-token contexts, team-level evolution adds two worker agents to the team (Section 4.4). This change keeps Meta-Team effective at the longest 256K context, where both the single agent and the fixed MAS degrade clearly (Figure 3a in the manuscript).
  • GAIA: revising team rules. In the case study of Appendix D.3, L2 finds that the WebSearcher reports estimated word counts as exact values. In L3, members submit their observations via suggest_team_improvement, and the chairman, combining them with the L2 finding, writes a rule into the team constitution: "Any externally looked-up value must be tagged with its source and confidence, and low-confidence values must be re-verified before use in critical computation."

(2) L3 changes do not break the L1/L2 configurations.

  • L3 runs after L1/L2 and takes their conclusions as input. The chairman reads the summarized conclusions of L1/L2 when making decisions (Appendix D.2), so team-level revisions are made with full knowledge of the changes at the agent and interaction levels.
  • Pre-commit checks. All updates must pass checks on role consistency, tool availability, format validity, and budget constraints before being committed; updates that fail are not written into the system (Appendix D.1).
  • Empirical evidence. The full Meta-Team outperforms the variant without L3 (53.9 vs. 51.9 on Ansible; 55.8 vs. 52.1 on ResearchRubrics, Table 3 in the manuscript). If L3 changes broke the L1/L2 configurations, adding L3 would lower performance.

We sincerely thank the reviewer again for the constructive feedback. We hope our responses address the reviewer's concerns, and we would be happy to discuss any further questions.

Reviewer_5nxhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Waggle, a parameter-shared anonymous local policy through which LLM agents jointly select task actions, bounded communication updates, and local commitments from local views. It introduces Swarm-Consistent Distillation, combining equivariant-view supervision with next-local-field prediction, and evaluates scaling, frozen transfer, and counterevidence-driven reorganization across swarm coordination benchmarks.

Critical strengths
  1. The paper gives a coherent and unusually concrete formulation of anonymous local coordination. Sections 3–4 precisely separate the learned semantic policy from the fixed runtime, define the bounded local interface in Equation 1, jointly decode actions, communication, and commitment updates in Equation 2, and give a conditional local-to-global relabeling result in Proposition 1. This makes the intended object of learning substantially more precise than the usual high-level claims about emergent LLM-agent organization.

  2. The empirical evidence is broad and includes useful mechanism tests rather than only aggregate benchmark scores. Table 1 shows consistent advantages across two substantially different substrates and population sizes, while Table 3 and Table 15 isolate the effect of the two SCD objectives: complete SCD improves held-out anonymous-view agreement by 22.8 points and late-counterevidence recovery by 8.5 points relative to decision-only distillation. The four-arm design is particularly valuable because it connects the proposed objectives to deployed behavior after the training-only heads are removed.

Critical weaknesses
  1. The central empirical positioning against learned communication and collaboration training methods is incomplete. Section 5.1 compares Waggle to organization/topology systems, fixed swarm rules, and an agent-distillation baseline, but omits a direct comparison to recent post-training approaches that explicitly learn reusable information-sharing behavior, particularly CoSIR. This matters because the paper’s main empirical takeaway is not merely that a hand-designed local runtime can work, but that SCD is an effective way to train a reusable coordination law under changing interaction conditions. A strong communication-training baseline could plausibly change the conclusion about whether the observed gains come specifically from anonymous local-law learning and SCD, rather than from teacher distillation plus the carefully engineered runtime and validator. An interface-matched CoSIR-style comparison, or a clear explanation of an unavoidable incompatibility and a narrowed comparative claim, would resolve this concern.
Questions for authors
  1. [Clarification] Was CoSIR, or another learned information-sharing post-training method, considered for the primary evaluation in Section 5.1? If it could not be made interface-compatible, please specify the concrete incompatibility and whether an ablation approximating its information reward could be implemented within the Waggle runtime. A competitive result would materially strengthen my confidence in the comparative claim.

  2. [Clarification] Proposition 1 requires exact equivariance of the complete deployed decode-and-admit pipeline, but Table 3 reports 91.1% exact admitted-record equivariance for Waggle rather than exact equivariance. What are the dominant sources of the remaining failures, decoder sensitivity versus validator/admission behavior, and do these cases measurably affect closed-loop outcomes under relabeling? This would clarify the practical scope of the theoretical framing.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer 5nxh

We thank the reviewer for the time spent on this review. However, the methods, experiments, and theory it discusses (Waggle, Swarm-Consistent Distillation, the comparison with CoSIR, the 91.1% equivariance in Table 3, and Proposition 1 on the decode-and-admit pipeline) do not appear in our submission. The review appears to be intended for a different submission.

To facilitate verification, we briefly summarize the core contributions of our paper:

  • Finding: an architectural mismatch in MAS evolution. MAS distribute a task across agents to relieve the burden on any single context, yet existing evolution methods flatten the whole-team trajectory into a single analyzer, reintroducing the very single-context bottleneck that MAS are designed to avoid. On TraceElephant, a failure-attribution benchmark for MAS, this global scheme degrades sharply on trajectories longer than 128K and falls behind analyzing local trajectories alone (Figure 1).
  • Method: a MAS that not only executes as a team, but also evolves as a team. Meta-Team lets each agent preserve its own local execution context. After each task, agents exchange evidence to jointly identify the cause of failure, and turn it into reusable improvements at three levels: agent behavior, inter-agent collaboration, and team organization.
  • Results: Across six long-horizon agent benchmarks (SWE-bench Pro, BeyondSWE, LOCA-Bench, GAIA, LoCoBench, and ResearchRubrics), Meta-Team achieves the best result in all nine columns, with an average of 62.7, compared with 56.4 for MASFly, the strongest MAS evolution method, and 56.1 for the initial MAS. The evolved team generalizes to longer contexts and other programming languages, and remains effective with GPT-5.5 as the backbone.

We would be glad to answer any questions if the reviewer re-evaluates our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.