- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EvolveMAS, a unified self-evolution framework for decentralized multi-agent systems. Motivated by the fact that existing approaches build adaptation mechanisms for one particular interaction regime, and that accumulating long interaction histories inside the reasoning context causes context bloat and memory pollution, the authors ask how a single mechanism can support both test-time adaptation and reliable experience reuse across cooperative, mixed-motive, and adversarial settings. EvolveMAS pairs each main agent with a dedicated MemoryAgent that maintains Episodic, Semantic, and Working memory outside the agent's reasoning context and updates it turn by turn (Within-Episode Online Adaptation), while at episode end the main agent distills reusable Interaction Strategies and Partner Profiles conditioned on a task-native outcome signal (Cross-Episode Knowledge Evolution) — all without updating model parameters. Its contributions are the resulting Execute–Evaluate–Evolve framework, the external MemoryAgent governance design, and an evaluation across five benchmarks and six configurations spanning the three regimes, where EvolveMAS attains the highest normalized average score of 77.2.
Critical strengths
- The evaluation breadth is stronger than most work in this line. Five benchmarks spanning cooperative, mixed-motive, and adversarial regimes — BBH, HumanEval, SOTOPIA, LLM-Deliberation, and Werewolf — are genuinely heterogeneous rather than variants of a single paradigm.
- The supplementary system-level analyses are practically useful. The team-size sweep (stable at 83–86 across 1–6 agents), the comparison against static MoA aggregation (84.0 vs. 81.0), and the cost/context-occupancy trajectories provide information that reviewers typically want but most papers omit.
- The code is released via an anonymous repository, which partially supports reproducibility. This is preferable to the common "available upon acceptance" practice, though reproducibility remains partial because the MemoryAgent prompts and the distillation procedure are not specified in the paper.
Critical weaknesses
-
Weak statistical support and no repeated-trial statistics. Each cell of the main ablation comes from a single execution, and column means are reported without SDs or confidence intervals. The paper's own precision analysis gives a standard error of about 4 points on BBH, yet the BBH gain over the native baseline is only +4, and the reported McNemar tests find almost nothing significant. The one confidence interval that is reported treats 50 sessions of a single evolving stream as independent samples, so the effective sample size is one, not fifty.
-
Baselines miss the paper's own contribution class. All five baselines are MAS architecture families transferred across settings; no memory or self-evolution method is compared, even though ReasoningBank, A-Mem, Mem0, ExpeL, and Reflexion are all cited. The result therefore shows EvolveMAS beats transferred scaffolds, not that it beats alternative memory mechanisms. Two stated contributions are also left unsupported: "unified" is never tested by transferring knowledge across regimes, and the "plug-in extension" claim is never tested by attaching the memory layer to any baseline family.
Questions for authors
- Could the author report mean ± SD over at least three seeds for every cell of Table 1, with paired significance tests for the Werewolf and SOTOPIA columns? Given the standard errors quoted in Appendix A.1 (≈4 points on BBH, ≈5 on each Werewolf audit), how much of the headline +9.7 survives? Also, could the LLM-Deliberation ablation be repeated over independent streams, so that the stream rather than the session is the replication unit?
- All five baselines are MAS architecture families; no memory or self-evolution method is compared despite ReasoningBank, A-Mem, and Mem0 being cited. Could the author add one such baseline, and attach the EvolveMAS memory layer to a baseline family (e.g., AgentNet) to separate the contribution of memory governance from that of native team structure?
5 comments
Comment 2|Response to Weaknesses (1/2): W1
W1 (statistical support):
We clarify that our paper already reports uncertainty analyses, and that our protocol follows prior test-time self-evolution studies. Appendix A.1 (L871–879) reports the sampling precision of each column, and Appendix E.1 further reports paired McNemar tests.
Evaluating a single evolution run is the common practice in self-evolving agent studies. For example, ReasoningBank (ICLR 2026) evaluates test-time self-evolution on sequential task streams and reports point estimates without run-to-run variance, and Unified Context Evolution (EMNLP 2026) reports a single evolution process with task-level McNemar and paired bootstrap tests rather than repeated evolution runs. Because each task in our setting is encountered only once along a continuous experience stream, one complete run over the benchmark is the natural unit of evaluation.
Nevertheless, to further address your concern, we conducted two additional independent full evolution runs for all six configurations (three runs in total), including complete LLM-Deliberation streams, so that the evolution stream rather than the session is the replication unit. Across the three runs, the normalized average is 77.9 ± 1.1 (77.2, 79.1, and 77.2), and every run exceeds the strongest transferred MAS baseline (67.5) by 9.7–11.6 points. The per-benchmark results (mean ± SD) are listed in our response to Q1. We have reported these results in Appendix H of the revised manuscript.
Response to Weaknesses (2/2): W2
Response to Weaknesses (2/2)
W2 (memory baselines, Unified, and Plug-in):
We would like to clarify that the central contribution of EvolveMAS is a unified self-evolution framework for decentralized MAS, not a standalone memory method, and both claims should be read in this sense.
- Unified refers to the mechanism, not to knowledge transfer. Across settings, the self-evolution process, memory representation, and update schedule remain unchanged (Sec. 4.3, L404). Our paper never claims that Knowledge evolved in one regime transfers to another; prior self-evolution methods such as Reflexion likewise reuse experience within the evaluated tasks.
- Plug-in follows from the architecture. EvolveMAS preserves the native decentralized organization and pairs each main agent one-to-one with a dedicated MemoryAgent (Sec. 3.1, L187–191). In every column of Table 1, it runs on the same team as the native system and only adds per-agent memory governance and evolution. This is why we describe it as a plug-in extension to existing agents (Conclusion, L481), consistent with modular memory augmentation in prior work such as Proactive Memory Agent.
- Memory baselines target a different task. Mem0 and A-MEM are designed and evaluated for question answering over accumulated conversation sessions, whereas our benchmarks require memory to be updated during ongoing multi-agent interactions. Their published implementations therefore cannot be transferred as equivalent baselines without redesigning their workflows. Our memory-governance claim is instead supported by controlled comparisons with the same interaction structure: Self-Memory (Sec. 4.1, L316), Figure 4, Table 6, and Appendix F.
Comment 1|Overview
We thank the reviewer for the detailed assessment, particularly the recognition of our evaluation breadth and supplementary system-level analyses. We appreciate the concerns regarding statistical reliability, baseline selection, and the interpretation of our Unified and Plug-in claims. We would like to clarify several important points supported by both the original manuscript and our additional experiments.
First, we have strengthened the evidence for experimental reproducibility. We conducted two additional independent complete evolution runs across all six evaluation configurations, giving three runs in total. The normalized average scores are 77.2, 79.1, and 77.2, consistently exceeding the strongest transferred MAS baseline (67.5). We distinguish these independent-run statistics from the task-level uncertainty and within-stream variability originally reported in Appendix A.1, without treating dependent sessions as independent evolution runs.
Second, EvolveMAS does not introduce a different native MAS team architecture. As specified in Sec. 4.1, Self-Memory preserves the same interaction structure, while Figure 4 and Table 6 evaluate memory governance and cross-episode Knowledge Evolution under controlled configurations. These comparisons directly examine the proposed mechanisms without changing the underlying team organization.
Third, Unified and Plug-in have specific architectural meanings. Unified refers to applying the same agent-local evolution mechanism across heterogeneous interaction regimes, not transferring task-specific knowledge between unrelated environments. Plug-in refers to a modular, training-free extension operating through local interaction trajectories, memory-conditioned decisions, and task-native feedback, without imposing centralized policy control or a fixed team topology.
Our principal contribution remains a unified framework for continuous, interaction-driven adaptation among autonomous decentralized agents. The following responses provide the detailed statistical results, controlled comparisons, and methodological reasoning supporting these distinctions.