- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoEvolve, a framework that jointly evolves collaboration skills, skill memory, and executor memory in multi-agent systems. Skills provide a shared interface between collaboration patterns and their supporting experience. Counterfactual continuations, generated by removing an executor from a failed trajectory, supply signals for coordinated updates. Experiments evaluate the framework on ALFWorld and WebShop.
Critical strengths
-
Jointly adapting collaboration and its supporting memory is a useful system-design contribution. The distinction between skill memory for stage allocation and executor memory for stage execution provides a concrete way to maintain compatibility as collaboration patterns evolve.
-
The component ablations provide meaningful evidence for the complete framework. Table 2(a) evaluates all eight combinations of the three evolving components, and the full system outperforms every partial configuration. Table 1 reports improvements over the strongest listed baseline on both ALFWorld, from 0.7537 to 0.7985, and WebShop, from 0.6681 to 0.7072. Appendix E.1 also explicitly separates evolution tasks from validation and test tasks.
Critical weaknesses
-
The experiments do not sufficiently distinguish the proposed counterfactual and alignment mechanisms from additional search and experience. Removing an executor changes the subsequent allocation and execution trajectory, while the counterfactual ablation does not replace these additional continuations with ordinary resampling under the same budget. This matters because Appendix F.2 reports an average of 3.912M tokens per round for counterfactual sampling on ALFWorld. Similarly, the factorial ablation establishes that evolving multiple components is beneficial, but does not isolate alignment itself: a configuration where all components evolve without conditioning memory updates on the updated skill space is missing. Consequently, the specific mechanism claims are less well supported than the overall system results.
-
The advantage over strong baselines requires clearer evolution-stage resource accounting and uncertainty estimates. Evaluation backbones are aligned, but the paper does not adequately document each baseline’s evolution models, training-task exposure, additional rollouts, and optimization budget. The ALFWorld improvement over AutoGen-GMemory corresponds to approximately six additional successes among 134 tasks, yet no repeated evolution runs or paired uncertainty estimates are reported. The results therefore establish an advantage for the reported configurations, but do not yet demonstrate its robustness to sampling and resource allocation.
-
CoEvolve should be positioned against Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems, which evolves reusable orchestration knowledge through multi-trajectory feedback and contrastive reflection. Please clarify the additional contribution of jointly updating local collaboration skills, skill memory, and executor memory, and distinguish the proposed executor-removal interventions from trajectory-based contrastive reflection.
Questions for authors
See Weakness
Thank you for your response. I decide to raise my rating. Good luck!