- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Meta-Team, an experience-driven framework for optimizing LLM-based multi-agent systems. It preserves agent-local execution contexts and coordinates post-task evidence exchange to update individual behaviors and skills, teammate profiles and interaction notes, and team organization and shared rules. In the main experiments, configurations evolve on approximately 20 instances per benchmark or subset and are then frozen for evaluation on disjoint instances. Validation covers six long-horizon agent benchmarks, failure attribution, component ablations, context-length and cross-language evaluation, and additional backbone and evolution-stability experiments.
Critical strengths
- meta-Team has consistent empirical gains on multiple challenging task Settings: Table 1 includes several existing evolution methods, and Meta-Team achieves the highest reported mean in all nine columns. Its cross-column average is 62.7, compared with 56.4 for MASFly and 57.8 for AOrchestra. Paired significance tests and repeated full evolution runs on three representative settings provide additional support. This establishes credible practical value for task-family adaptation.
- There are key ablation experiments that support the core claim. In Table 2, collaborative experience organization exceeds the centralized control by 4.1 points on Ansible and 2.9 points on ResearchRubrics, and also outperforms isolated local analysis. Removing any update scope in Table 3 reduces performance. These experiments provide direct evidence for the practical usefulness of the proposed claimed design.
Critical weaknesses
- Centralized control can also be more comprehensive: Table 2 supports the effectiveness of the method. However, the centralized baseline uses only one analysis call, while the collaborative method allows repeated exchanges and checks. It remains unclear how much of the advantage would remain if the centralized baseline could make several passes under a comparable reflection budget.
- Proposition 1 lacks necessary conditions: The assumptions in Appendix C.2 do not guarantee the crossover claimed in Proposition 1. For example, if the recovery rate stays at 1, global analysis never becomes worse than local analysis. The proof also uses continuity and a recovery rate approaching zero, neither of which is stated in the assumptions.
Questions for authors
- How many model calls and how much cost did each Table 2 variant use for reflection? Does the centralized variant have to diagnose the problem and generate configuration updates in the same call?
- On Ansible or ResearchRubrics, could you allow the centralized baseline several analysis passes under a comparable reflection budget and check whether the advantage remains?
- Could you add the conditions needed for Proposition 1, or revise the claim to address the counterexample above?
4 comments
Rebuttal to Reviewer U6fr [2/2]
[W2 & Q3] Proposition 1 lacks necessary conditions: The assumptions in Appendix C.2 do not guarantee the crossover claimed in Proposition 1. For example, if the recovery rate stays at 1, global analysis never becomes worse than local analysis. The proof also uses continuity and a recovery rate approaching zero, neither of which is stated in the assumptions.
Could you add the conditions needed for Proposition 1, or revise the claim to address the counterexample above?
[Response] We thank the reviewer for the careful examination. In Appendix C.2 of the revised manuscript, we state the conditions of Proposition 1 explicitly and give a sharper proposition and proof. The conclusion is unchanged: on sufficiently long trajectories, global analysis is strictly worse than local analysis, while collaborative analysis is at least as good as both at any length and strictly better than global analysis on long trajectories.
Added assumptions.
- (A3) Context dilution vanishes: In addition to Assumption 2 (ρ is non-increasing), we require . This rules out the counterexample ρ ≡ 1.
- (A4) Trajectories grow longer through more division of labor: The total length L grows with the number of agents N; each agent's local trajectory is bounded, ; local evidence is bounded below, ; and interaction evidence grows at most linearly with the number of agents, .
Revised Proposition 1.
- Long trajectories: Under (A3)–(A4), there exists a finite such that for all .
- Short trajectories: At any length with , we have . That is, the global scheme is preferable when context dilution is weak and the interaction evidence is substantial.
Proof sketch. By (A4),
By (A3), the right-hand side is negative once , so for all sufficiently large L. Taking completes the proof, which no longer requires continuity.
Proposition 3 is strengthened accordingly. Whenever , and under (A4), the advantage of Collaborative over Global satisfies
which is positive for sufficiently large L and grows with the number of agents N.
This result matches Figure 1: global falls behind local on trajectories longer than 128K, while collaborative is the best in both length regimes.
We are grateful to the reviewer for the careful reading and valuable suggestions. We hope our responses have resolved the reviewer's concerns, and we would welcome any further feedback.
Author Follow-up for Reviewer U6fr
Dear Reviewer U6fr,
Thank you again for your time and effort in reviewing our submission.
As the discussion period will end in less than a day, we would like to kindly remind you that we have posted our responses to all of your comments, including the budget-matched multi-pass centralized baseline you suggested and the revised conditions and proof for Proposition 1. We hope they address your concerns, and we would be happy to clarify anything further.
We sincerely appreciate your valuable suggestions and thank you again for your time and consideration.
Thanks for adding the budget-matched experiment. It addresses my concern about the single-call baseline. I’m still unsure whether Meta-Team would retain its advantage over a centralized method that can selectively inspect the same local evidence. On balance, I’m keeping my rating at 2.
Rebuttal to Reviewer U6fr [1/2]
We sincerely thank the reviewer for recognizing our empirical results and ablations, and for the concrete suggestions on the centralized control and the theoretical analysis. Key revisions in the revised manuscript are marked in blue.
[Response] Even with separate diagnosis and update calls and a matched reflection budget, Centralized gains almost nothing and remains clearly below Meta-Team. We address these questions in three parts: (1) how the original Centralized is called; (2) the calls and reflection cost of each scheme; and (3) multi-pass Centralized under a comparable budget.
(1) The original Centralized performs diagnosis and update generation in a single call. As described in Section 4.3, Centralized feeds the flattened whole-team trajectory to a single LLM analyzer call, which both diagnoses the failure and generates configuration updates. The new multi-pass variant separates the two steps into different calls and allows multiple rounds of self-critique and revision.
(2) Calls and reflection cost. The original Centralized calls the analyzer once per task; Partitioned assigns one analyzer to each agent; Collaborative additionally exchanges evidence on demand. The evolution overhead of Collaborative is $13 on Ansible and $25 on ResearchRubrics (Table 13 in the manuscript). Agents exchange evidence only when needed, and pairs without problematic interactions do not communicate, so the cost does not grow quadratically with team size.
(3) Budget-matched multi-pass Centralized. Following the reviewer's suggestion, we let Centralized repeatedly run "diagnose → generate updates → self-critique → revise" over the same flattened trajectory on both Ansible and ResearchRubrics, until its reflection cost matches that of Collaborative. All other conditions are identical to the original experiment, including the feedback, the L1/L2/L3 update space, and the tools.
Table R1: Comparison of experience schemes, including the budget-matched multi-pass Centralized variant. The evolution overhead of the multi-pass variant is matched to that of Collaborative ($13 on Ansible and $25 on ResearchRubrics, Table 13 in the manuscript).
The advantage remains under a comparable reflection budget. Multi-pass Centralized only rises from 49.8 to 50.2 on Ansible and drops from 52.9 to 52.4 on ResearchRubrics, remaining 3.7 and 3.4 points below Collaborative. Moreover, Partitioned and Collaborative use the same number of analyzers over the same local trajectories, yet Collaborative is still higher by 9.4 and 4.8 points, showing that the gain comes from communication rather than more calls.
The bottleneck is that a single context must read the entire interleaved long trajectory, and more analysis passes do not remove this burden. This is consistent with the sharp drop of the global scheme on trajectories longer than 128K in Figure 1. Table R1 is included as Table 15 in Appendix F.2 of the revised manuscript.