I have two primary concerns, which I outline below in order of importance.
I. The paper's core methodological contribution has not been experimentally verified independently.
My biggest concern is that the existing experiments are insufficient to demonstrate whether EMMA's reported performance improvement truly stems from its proposed scoped and guarded update mechanism. The rationale behind EMMA is clear: allowing an agent's evolutionary process to modify its structure without constraints risks making the system overly dependent on the underlying base model. However, the ablation experiments in Table 5 mainly focus on the contributions of the agent pool and skill pool, as well as the difference between episodic learning during training and adaptation during testing, without providing a truly matched key control.
Specifically, the paper does not compare EMMA against an otherwise identical system in which the reflector can freely choose update actions without the proposed scope restrictions. Nor does it examine the role of each programmatic guard separately. This is a critical omission because the differences between EMMA and the baselines extend beyond the update mechanism to include controller architecture, memory organization, step-by-step execution, and planning strategy. Therefore, EMMA's better performance on ACE, GEPA, EvoMAS, and ADAS indicates the effectiveness of the overall system, but it does not establish that its core scoped and guarded update mechanism is necessary.
There is also a conceptual issue regarding failure attribution. Although the guards ensure that the predicted failure scope is compatible with the corresponding update action, the scope itself is still determined by the LLM-based reflector. In other words, the guards can constrain updates, but they cannot guarantee that the diagnosed scope accurately reflects the true cause of the failure. Constraining updates and correctly attributing failures are distinct processes. Given that the scoped and guarded update mechanism is the paper's primary contribution, the absence of a directly matched ablation study significantly weakens the persuasiveness of the core argument.
II. Incomplete evaluation of computational efficiency and limited evidence of training robustness
The paper states that EMMA requires fewer training task executions than the baselines, but the number of task executions is not necessarily a comparable cost metric across different agent architectures. A single EMMA execution may involve multiple controller decisions, sub-agent calls, tool interactions, and replanning. Therefore, fewer task executions do not necessarily mean lower token consumption, fewer LLM calls, or shorter actual execution time. Without these measurements, it is difficult to determine whether EMMA provides a genuine computational efficiency advantage over simpler approaches.
In terms of training robustness, the framework performs single-pass training in a fixed task order, and memory updates are retained even when subsequent attempts fail. In this way, the learned configuration depends on cumulative updates, and both the training order and stochastic execution behavior can affect the final memory and evaluation performance. The paper does not report variability across different training orders or repeated runs, which limits the evidence for robustness beyond comparisons across different backbones. These limitations do not invalidate the paper's reported performance improvements, but they do weaken its broader claims regarding learning efficiency and consistency.
Rebuttal Response to Reviewer 7kug
We thank the reviewer for finding the observation interesting, and for recognizing that clarifying which component owns each correction makes the update process easier to inspect. We address the concern below.
We understand the concern and would like to clarify the cause of the observed phenomenon and how EMMA addresses it. Existing multi-agent evolution methods leave one question to the backbone: given a failure, which part of a multi-agent system should change, and by how much. EMMA answers this question explicitly rather than tuning a complete system.
Why the phenomenon occurs: In existing methods, the edit types are unconstrained and the feedback is a task score that does not indicate which component failed, so the backbone's own preference over edit types decides the update. Updates therefore follow the backbone rather than the failure (Figure 1: Opus 5 creates 18 skills on one benchmark, while DeepSeek v3.2 creates 114 agents over 83 training tasks), and performance follows the backbone: on AppWorld challenge, EvoMAS and ADAS reach 89.0 and 84.4 TGC with Opus 5 but fall to 7.2 (EvoMAS, GLM-5) and 0.5 (ADAS, DeepSeek v3.2). The update target should thus be determined by the diagnosed failure, not delegated to the backbone.
How EMMA addresses it: In our design, each memory level answers a distinct question, so each correction has exactly one level that owns it:
The reflector reads the full trajectory, localizes the failing step and classifies its scope, and the selection rules restrict the edit to the level that owns the correction. The selection rules are a credit-assignment procedure over the memory hierarchy and do not depend on the benchmark. As the constraints depend only on the memory and the revision history, they apply identically to every backbone, and EMMA's weakest backbone outperforms every baseline's weakest backbone on all three benchmarks.
Component-wise Analysis of EMMA Framework: To establish whether the advantage comes from the update mechanism, we have added an architecture-matched ablation on KramaBench that keeps the controller, the three memory pools, the execution procedure and the training budget fixed, and varies only the update mechanism with three model families:
Removing either component lowers accuracy on every backbone, and the drop is largest on the open-weight backbones (up to 15.7% for DeepSeek v3.2 without selection rules), consistent with Figure 1, where DeepSeek v3.2 creates agents regardless of the failure. We have also added AFlow, GPTSwarm and Memento-Skills as baselines (see our response to Reviewer FtEy). None of them restricts the update target, and they show the same backbone dependence: AFlow nearly matches EMMA with Opus 5 on AppWorld challenge (88.0 vs. 89.2 TGC) but falls to 15.6 and 13.7 with DeepSeek v3.2 and Kimi-K2.5. The gains therefore stem from the update mechanism rather than from the architecture alone.
We have added this analysis to Section 1 and Sections 3.2.1–3.2.2, the component-wise ablation to Section 5.3 (Table 6a), and the new baselines to Section 5.2 and Table 3 (AppWorld) and to Appendix D (BBEH-mini, Table 13).
Thank you again for the insightful comments and we have incorporated your suggestions in the revised version of the manuscript, highlighted in blue. If there are any additional questions or concerns, we would be happy to address them.
Should you find that our responses sufficiently address your feedback, we would sincerely appreciate your consideration of raising the score, if deemed appropriate.