- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TacoMAS, a test-time co-evolution framework for multi-agent LLM systems. The key idea is to adapt two things jointly but at different time scales: a fast capability loop that refines agent memory/prompts each round based on trajectory feedback, and a slow topology loop driven by a meta-LLM that every K rounds performs bounded birth-death and edge edits. The design is motivated by a replicator–mutator surrogate model, with an explicit convergence bound. Experiments on four benchmarks against baselines show consistent gains.
Critical strengths
- The idea is well motivated and timely. Jointly evolving capability and topology at different time scales is a clean, principled contribution that fills a real gap between topology-only and capability-only test-time methods.
- The evaluation is honest and thorough. Four benchmarks, about 20 baselines, multiple backbones, and the authors are unusually upfront about their assumptions, confounds like unmatched compute, and what the theory does not actually show.
Critical weaknesses
1.The novelty feels a bit incremental. It basically combines two existing ingredients, CORAL-style memory refinement and SelfOrg-style rewiring, with a scheduling heuristic, and each piece on its own is fairly standard.
2.Compute is not controlled. The gains might largely come from extra LLM calls and judge rounds, and the equal-compute comparisons are weak or missing.
3.The theory is somewhat decorative. The surrogate theorem relies on unverifiable assumptions, applies to a made-up weight space instead of real text agents, and the authors repeatedly admit it does not establish anything useful.
4.There are reproducibility and rigor concerns. The paper leans on archived summaries, mixes protocols, uses unpaired comparisons and non-factorial sweeps, and lacks error bars, so several numbers cannot be fairly compared.
5.Scalability and cost are questionable. Wall time is about 86 minutes per instance on finance, the controller can eat up to 93% of the cost, and the single meta-controller is a bottleneck, which makes it impractical for real deployment.
Questions for authors
Please see the Weaknesses.
Dear Reviewer,
Please reply to the authors' rebuttal.
Thanks!
AC.