acceptodds
Under review as a conference paper at ICLR 2027

EMMA: Structured Episodic Memory for Multi-Agent Workflow Synthesis

Abstract

Deploying LLM agents in specialized domains often require fine-grained task-specific procedures and knowledge a general model lacks, yet parameter training is costly and non-interpretable, and hand-engineered agent systems shift the cost to expert labor. Episodic memory updates at context level, but single agent adaptation struggles to alleviate cross-task interference, especially with complex tasks such as multi-application navigation or data pool analysis. Although multi-agent evolution alleviates this problem, existing methods lack clear definitions on update rules and rely on the backbone model to decide where and how much the system changes. This ill-scoped evolution makes learning follow the backbone's structural preferences rather than the observed failures: one backbone accumulates skills while another spawns agents. We present EMMA (structured Episodic Memory for Multi-Agent workflow synthesis), which treats the configuration of a multi-agent system (controller routing memory, specialists inheriting a generalist prompt, and per-step procedural skills) as episodic memory. EMMA makes the ownership and admissibility of updates explicit: a reflector assigns each diagnosed failure to the scope owning its correction and proposes one edit within it, and backbone-independent guards keep every accepted edit scoped, incremental and bounded. Across KramaBench, AppWorld and BBEH with five backbones and four baselines (ACE, GEPA, EvoMAS and ADAS), EMMA attains the best overall performance on every benchmark, up to 32.5% over the strongest baseline, maintains consistent improvements across backbones, and uses 17–38% fewer training executions than the most efficient baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, episodic learning, multi agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7kugOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents EMMA, a framework for multi-agent workflow synthesis using structured episodic memory. By mapping execution failures to explicit scopes with programmatic update guards, EMMA aims to make memory updates targeted and backbone-independent. Across three benchmarks, EMMA outperforms several baselines with fewer training executions.

Critical strengths
  1. The observation is interesting. Following a failure in a multi-agent system, changing task routing, agent prompts, or procedural skills can have different consequences. The paper attempts to clarify which component should own each correction and to limit the scope of individual edits, making the update process easier to inspect.
Critical weaknesses
  1. The authors provide an interesting finding, but the experiments are limited. The results only show that the complete system performs well, but do not establish whether its advantage comes from the update mechanism emphasized as the main contribution. More importantly, the paper does not sufficiently investigate why the observed phenomenon occurs or how the proposed method directly addresses its underlying causes. Without such analysis, the method appears more like an empirically effective engineering solution than a principled approach to the identified problem.
Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Rebuttal Response to Reviewer 7kug

We thank the reviewer for finding the observation interesting, and for recognizing that clarifying which component owns each correction makes the update process easier to inspect. We address the concern below.

The authors provide an interesting finding, but the experiments are limited. The results only show that the complete system performs well, but do not establish whether its advantage comes from the update mechanism emphasized as the main contribution. More importantly, the paper does not sufficiently investigate why the observed phenomenon occurs or how the proposed method directly addresses its underlying causes. Without such analysis, the method appears more like an empirically effective engineering solution than a principled approach to the identified problem.

We understand the concern and would like to clarify the cause of the observed phenomenon and how EMMA addresses it. Existing multi-agent evolution methods leave one question to the backbone: given a failure, which part of a multi-agent system should change, and by how much. EMMA answers this question explicitly rather than tuning a complete system.

Why the phenomenon occurs: In existing methods, the edit types are unconstrained and the feedback is a task score that does not indicate which component failed, so the backbone's own preference over edit types decides the update. Updates therefore follow the backbone rather than the failure (Figure 1: Opus 5 creates 18 skills on one benchmark, while DeepSeek v3.2 creates 114 agents over 83 training tasks), and performance follows the backbone: on AppWorld challenge, EvoMAS and ADAS reach 89.0 and 84.4 TGC with Opus 5 but fall to 7.2 (EvoMAS, GLM-5) and 0.5 (ADAS, DeepSeek v3.2). The update target should thus be determined by the diagnosed failure, not delegated to the backbone.

How EMMA addresses it: In our design, each memory level answers a distinct question, so each correction has exactly one level that owns it:

Memory levelQuestion it answersFailure scopeAllowed edits
Controller memoryHow is the task divided and routed?routingadd a memory note, revise the plan
Agent promptHow does an agent work on any subtask?agent behaviorrefine the prompt
SkillHow is a class of sub-problems solved?procedurerevise or create a skill

The reflector reads the full trajectory, localizes the failing step and classifies its scope, and the selection rules restrict the edit to the level that owns the correction. The selection rules are a credit-assignment procedure over the memory hierarchy and do not depend on the benchmark. As the constraints depend only on the memory and the revision history, they apply identically to every backbone, and EMMA's weakest backbone outperforms every baseline's weakest backbone on all three benchmarks.

Component-wise Analysis of EMMA Framework: To establish whether the advantage comes from the update mechanism, we have added an architecture-matched ablation on KramaBench that keeps the controller, the three memory pools, the execution procedure and the training budget fixed, and varies only the update mechanism with three model families:

ConfigurationSonnet 5GLM-5DSv3.2
EMMA, full0.6410.6930.585
No selection rules0.6310.6210.493
No update constraints0.6340.6260.556
Neither0.5410.6260.556

Removing either component lowers accuracy on every backbone, and the drop is largest on the open-weight backbones (up to 15.7% for DeepSeek v3.2 without selection rules), consistent with Figure 1, where DeepSeek v3.2 creates agents regardless of the failure. We have also added AFlow, GPTSwarm and Memento-Skills as baselines (see our response to Reviewer FtEy). None of them restricts the update target, and they show the same backbone dependence: AFlow nearly matches EMMA with Opus 5 on AppWorld challenge (88.0 vs. 89.2 TGC) but falls to 15.6 and 13.7 with DeepSeek v3.2 and Kimi-K2.5. The gains therefore stem from the update mechanism rather than from the architecture alone.


We have added this analysis to Section 1 and Sections 3.2.1–3.2.2, the component-wise ablation to Section 5.3 (Table 6a), and the new baselines to Section 5.2 and Table 3 (AppWorld) and to Appendix D (BBEH-mini, Table 13).


Thank you again for the insightful comments and we have incorporated your suggestions in the revised version of the manuscript, highlighted in blue. If there are any additional questions or concerns, we would be happy to address them.

Should you find that our responses sufficiently address your feedback, we would sincerely appreciate your consideration of raising the score, if deemed appropriate.

Reviewer_WZU7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EMMA, a framework developed to improve the performance of large language model (LLM)-based multi-agent systems. EMMA utilizes structured "episodic memory" to enhance performance without altering the underlying LLM parameters. The authors contend that current evolutionary methods for multi-agent systems face limitations because they depend on the LLM to autonomously determine modifications. This reliance, coupled with insufficient constraints, can result in inefficient structural changes and performance highly contingent on the specific LLM.

EMMA organizes reusable knowledge across three layers: a controller memory layer for task breakdown and distribution, an agent pool for specialized functions, and a skill pool for procedural knowledge. A reflection module analyzes unsuccessful execution paths during training. It categorizes these failures and initiates updates specifically within the pertinent memory layer. Additionally, programmatic constraints dictate permissible actions, the scope of modifications, and the extent of memory growth.

Evaluations were conducted on KramaBench, AppWorld, and BBEH-mini, employing five distinct LLMs. EMMA's performance was compared against four baseline methods: ACE, GEPA, EvoMAS, and ADAS. The results indicate that EMMA achieved superior average performance across these benchmarks, securing the top position in 17 out of 20 benchmark-model pairings. Furthermore, based on the evaluation metrics, EMMA demonstrated a reduced need for task executions during training.

Critical strengths
  1. Comprehensive empirical assessment utilizing diverse benchmarks and varying large language model (LLM) architectures.

A significant contribution of this study is its extensive experimental validation. The research encompasses three benchmarks designed to capture distinct task attributes and five model backbones, incorporating both closed-source and open-weight models. EMMA demonstrates consistent, high performance when compared against established methods for prompt optimization, episodic memory, and multi-agent evolutionary processes.

The observed performance gains, particularly with less capable backbones, are notable. This suggests that the proposed framework may mitigate some of the performance deficits commonly associated with agent evolution that are dependent on specific backbone models. The breadth of this evaluation furnishes substantial empirical evidence for the practical efficacy of the presented system, thereby strengthening the case for its adoption.

  1. A well-defined and replicable methodology for systematic multi-agent adaptation.

This work delineates a clear distinction between controller memory, agent behaviors, and procedural capabilities, coupled with a failure-based update mechanism that confines modifications to predefined components. While certain individual elements draw upon existing agent frameworks, their unified integration establishes a structured and comprehensible method for managing enduring adaptations within multi-agent systems.

The inclusion of detailed algorithms, update restrictions, reflective prompts, and examples of emergent capabilities enhances the transparency and reproducibility of the proposed framework. This degree of methodological clarity is crucial for advancing future research into self-evolving agent systems.

Critical weaknesses

I have two primary concerns, which I outline below in order of importance.

I. The paper's core methodological contribution has not been experimentally verified independently.

My biggest concern is that the existing experiments are insufficient to demonstrate whether EMMA's reported performance improvement truly stems from its proposed scoped and guarded update mechanism. The rationale behind EMMA is clear: allowing an agent's evolutionary process to modify its structure without constraints risks making the system overly dependent on the underlying base model. However, the ablation experiments in Table 5 mainly focus on the contributions of the agent pool and skill pool, as well as the difference between episodic learning during training and adaptation during testing, without providing a truly matched key control.

Specifically, the paper does not compare EMMA against an otherwise identical system in which the reflector can freely choose update actions without the proposed scope restrictions. Nor does it examine the role of each programmatic guard separately. This is a critical omission because the differences between EMMA and the baselines extend beyond the update mechanism to include controller architecture, memory organization, step-by-step execution, and planning strategy. Therefore, EMMA's better performance on ACE, GEPA, EvoMAS, and ADAS indicates the effectiveness of the overall system, but it does not establish that its core scoped and guarded update mechanism is necessary.

There is also a conceptual issue regarding failure attribution. Although the guards ensure that the predicted failure scope is compatible with the corresponding update action, the scope itself is still determined by the LLM-based reflector. In other words, the guards can constrain updates, but they cannot guarantee that the diagnosed scope accurately reflects the true cause of the failure. Constraining updates and correctly attributing failures are distinct processes. Given that the scoped and guarded update mechanism is the paper's primary contribution, the absence of a directly matched ablation study significantly weakens the persuasiveness of the core argument.

II. Incomplete evaluation of computational efficiency and limited evidence of training robustness

The paper states that EMMA requires fewer training task executions than the baselines, but the number of task executions is not necessarily a comparable cost metric across different agent architectures. A single EMMA execution may involve multiple controller decisions, sub-agent calls, tool interactions, and replanning. Therefore, fewer task executions do not necessarily mean lower token consumption, fewer LLM calls, or shorter actual execution time. Without these measurements, it is difficult to determine whether EMMA provides a genuine computational efficiency advantage over simpler approaches.

In terms of training robustness, the framework performs single-pass training in a fixed task order, and memory updates are retained even when subsequent attempts fail. In this way, the learned configuration depends on cumulative updates, and both the training order and stochastic execution behavior can affect the final memory and evaluation performance. The paper does not report variability across different training orders or repeated runs, which limits the evidence for robustness beyond comparisons across different backbones. These limitations do not invalidate the paper's reported performance improvements, but they do weaken its broader claims regarding learning efficiency and consistency.

Questions for authors

Q1. How much does the scoped and guarded update mechanism actually contribute?

  • I’d like to ask the authors to provide an architecture-matched ablation: keep the controller, agent pool, skill pool, execution procedure, and training budget exactly the same, but remove the proposed scope restrictions and let the reflector freely select update actions. This would directly test whether the reported gains and backbone consistency come from constrained update selection, rather than from other architectural components.
  • It would also be helpful to isolate the effect of the individual programmatic guards. Otherwise, it is still hard to tell how much the constraints themselves contribute.
  • Relatedly, how reliable is the reflector’s failure-scope classification? Have the authors manually examined a sample of failure trajectories to check whether the predicted scopes actually correspond to the real causes of failure? This also seems important for judging whether the overall mechanism holds up.

Q2. Computational cost and efficiency

  • In addition to the number of task executions, could the authors report total LLM calls, input/output tokens, and wall-clock time for both training and inference? Since EMMA and the baselines have substantially different execution structures, task execution counts alone do not tell us much about actual computational cost.
  • These measurements would help establish whether the reported “fewer training task executions” advantage really translates into a computational efficiency improvement, rather than just a less comparable metric across different architectures.

Q3. Sensitivity to training order and stochasticity

  • How sensitive are the learned agent configurations and final performance to the order of training tasks and to stochastic variation in LLM execution?
  • Results from multiple training orders or repeated runs, even on just one representative benchmark, would help substantiate the authors’ claims about consistency and robustness.
2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer WZU7 (1/2)

We thank the reviewer for the positive assessment and for recognizing the breadth of the evaluation and the clearly specified, reproducible framework. We address each point below.

Insufficient evidence isolating the paper's central methodological contribution...

Q1. Contribution of the scoped and guarded update mechanism...

We address the weakness and Q1 together, as they concern the same claim. We agree that the central claim requires an architecture-matched ablation, and that constraining an update is distinct from attributing a failure correctly. We address this from three aspects.

The update mechanism is responsible for the gains. We have added an architecture-matched ablation on KramaBench that keeps the controller, the three memory pools, the execution procedure and the training budget fixed, and varies only the update mechanism: the reflector either selects any action for any failure (no selection rules), or its proposals are applied without the programmatic checks (no update constraints), or both.

ConfigurationSonnet 5GLM-5DeepSeek v3.2
EMMA, full0.6410.6930.585
No selection rules0.6310.6210.493
No update constraints0.6340.6260.556
Neither0.5410.6260.556

Removing either component lowers accuracy on every backbone, by up to 15.7% (DeepSeek v3.2 without selection rules), and the drop is largest on the open-weight backbones, consistent with Figure 1, where DeepSeek v3.2 creates agents regardless of the failure. The gains and the backbone consistency therefore stem from the update mechanism rather than from the controller, memory organization or execution procedure. We agree that removing each constraint individually would further separate their contributions, and we will include this in the final version.

The reflector's scope labels agree with manual inspection. We manually inspected five failure updates randomly sampled from the training traces, comparing the reflector's scope and action with our own reading of each failure. The scope and action agree with human judgment in all five cases; the annotated cards are provided in the supplementary material, and the following example is included in Appendix E.1:

SA003 (KramaBench, Kimi-K2.5). The task asks for the average potassium concentration (ppm) at the first and last recorded human presence in Malta, using linear interpolation. The agent took 6005 and 6412 years BP as the boundary dates, directly from the minimum and maximum of the radiocarbon dates, without verifying them against the age axis of the potassium record, and obtained 8347.1887 instead of 8577.5298. The evaluator attributes the error to the dates rather than to the interpolation.

LabelReflectorManual checkReason
Scopeagent_behaviorAgreeThe interpolation was correct; the agent did not verify the date boundaries before computing, a general working habit rather than a method for a class of sub-problems.
Actionrefine_promptAgreeAn instruction to verify selected boundaries and their chronological convention before computing prevents the same error on other tasks.

We have added the component-wise ablation to the revised paper (Section 5.3, Component-wise Analysis of EMMA Framework, Table 6a), discussed the distinction between constraining an update and attributing a failure in Section 3.2.2, added the inspected example to Appendix E.1 (Table 16), and included all five inspected trajectories in the supplementary material.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer WZU7 (2/2)

Q2. Computational cost and efficiency...

We agree that the number of training executions alone does not measure computational cost across different execution structures. We report input/output token counts for training and inference on KramaBench below, as ranges over the five backbones. We use token counts rather than wall-clock time because all runs call hosted API endpoints with high parallelism, and retries under rate limits make wall-clock time highly variable: the same training run can take between 5 and 27 hours depending on API status. ACE and GEPA adapt a single agent; the other methods learn multi-agent systems.

KramaBench, token counts (ranges over backbones)

MethodTrain exec.Train tokens (in / out)Test tokens / task (in / out)
ACE (single)92–10012–39M / 0.21–1.1M—
GEPA (single)141–16824–47M / 0.20–1.2M—
EvoMAS168≥5.9–69M / ≥0.11–3.6M≥47k–412k / ≥1–32k
ADAS1405.4–12M / 0.39–1.1M29k–69k / 2–9k
EMMA49–7310–51M / 0.47–2.1M265k–640k / 8–28k

≥: lower bound, as EvoMAS does not log every call. —: not recorded in log.

EMMA's training cost is comparable to the multi-agent evolution baseline. EvoMAS is the baseline closest to EMMA, as both evolve a complete multi-agent configuration. EMMA's training input tokens (10–51M) fall within the range of EvoMAS (at least 5.9–69M), and below the upper end of GEPA (47M), while EMMA uses the fewest training executions of all methods.

EMMA costs more per task than lighter workflow search methods. We agree that EMMA uses more tokens than ADAS, in training and especially at inference (265k–640k vs. 29k–69k input tokens per task). The additional cost comes mainly from step-wise execution: the controller selects memory, plans, replans and curates the context of each sub-agent, which is also what allows a failure to be attributed to a single step.

The additional cost yields consistent gains. On KramaBench, EMMA improves mean accuracy over ADAS by 16.4% and over EvoMAS by 18.1%, and its weakest backbone (0.585) exceeds the five-backbone mean of every baseline. Training is a one-time cost, as the memory is frozen and reused for every held-out task. EMMA therefore trades additional computation for substantial and backbone-consistent gains.


We have added the token counts and this analysis to Appendix B.1 (Table 9), revised Section 5.2.2 to state that EMMA's advantage holds in training executions, not in tokens, and noted the higher per-task token cost as a limitation in Section 6.

Q3. Sensitivity to training order and stochasticity...

We agree that a single training run does not establish robustness to training order and stochastic execution. On KramaBench, we repeat training three additional times in the paper's task order on three backbones, and twice with shuffled task orders on Opus 5. Each run trains a new memory from scratch and evaluates it on the same held-out tasks.

KramaBench, repeated runs in the paper's task order (macro accuracy, mean ± std)

SettingOpus 5Sonnet 5Kimi-K2.5
Paper (single run)0.7660.6410.586
Paper order, 3 repeated runs0.793 ± 0.0120.624 ± 0.0250.579 ± 0.030

Performance is stable across stochastic runs. The run-to-run standard deviation is 0.012–0.030 across backbones. The paper's single-run results lie within one standard deviation of the repeated runs for Sonnet 5 and Kimi-K2.5, and below their mean for Opus 5, so the reported numbers are not favorably selected. On Opus 5 and Sonnet 5, EMMA's margin over the strongest baseline (0.084 and 0.040) exceeds the standard deviation.

Performance is stable across training orders. We compare the paper-order runs on Opus 5 with two runs on shuffled task orders:

Training orderRunsAccuracy
Paper order30.793 ± 0.012
Shuffled20.783 ± 0.015

Shuffling the training order changes mean accuracy by 0.010, within one standard deviation of either setting. This stability is expected from the update mechanism: each accepted edit is small and confined to one scope, so no single task, wherever it appears in the order, can restructure the memory. EMMA's results are therefore robust to both stochastic execution and training order.


We have added the repeated runs and shuffled training orders to the revised paper (Appendix B.2, Table 10) and summarized them in Section 5.2.


Thank you again for the insightful comments and we have incorporated your suggestions in the revised version of the manuscript, highlighted in blue. If there are any additional questions or concerns, we would be happy to address them.

Should you find that our responses sufficiently address your feedback, we would sincerely appreciate your consideration of raising the score, if deemed appropriate.

Reviewer_FtEyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

EMMA improves multi-agent workflows by organizing memory into different levels and using failure feedback to guide updates. The authors evaluate it on five backbone models across three benchmarks covering data analysis, interactive tool use, and reasoning, compare it with four baselines, and report improved task performance.

Critical strengths
  • The experimental results are strong overall, with improvements across a range of settings.
  • Separating memory adaptation from held-out evaluation is a useful design choice for examining whether knowledge transfers to new tasks.
Critical weaknesses
  • The writing style of this paper is generally hard to follow. Terms such as “attempt chains,” “ownership rules,” and “envelope of learning” describe operations that could be expressed more directly as iterative refinement, rules for selecting update targets, and update constraints. The author should not reinvent terms to make them seem novel.
  • The methodological contribution and the main takeaway are unclear. EMMA combines several components through heuristic rules. For example, “failure-guided updates” follows the obvious idea of using failure feedback to decide what to change. The proposed rules appear to be practical heuristics, with limited explanation of why they should work more generally. While the results show that the full system works, they offer limited insight beyond the effectiveness of this specific combination of components.
  • Several closely related methods are missing from the comparison. As discussed in the paper, AFlow [1] optimizes workflows using execution feedback, GPTSwarm [2] optimizes agent prompts and connections, and Memento-Skills [3] updates reusable skills based on failure feedback. These methods share important features with EMMA, but they are not included as baselines. Without comparisons using the same backbone models and similar budgets, it is difficult to judge EMMA’s improvement over these methods.

[1] Zhang et al. AFlow: Automating Agentic Workflow Generation. ICLR, 2025. Paper

[2] Zhuge et al. GPTSwarm: Language Agents as Optimizable Graphs. ICML, 2024. Paper

[3] Zhou et al. Memento-Skills: Let Agents Design Agents. Technical report, 2026. Paper

[4] He et al. Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. ICML, 2026. Paper

Questions for authors
  • Could the authors clarify what “training” means here, given that no model parameters are updated?
  • Do all methods have comparable adaptation budgets and access to feedback? How does freezing memory at test time affect baselines originally designed for online adaptation, such as EvoMAS?
10 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer FtEy (1/3)

We thank the reviewer for the careful reading and for noting the strength of the results and the separation of memory learning from held-out evaluation. We address each point below.

The writing style of this paper is generally hard to follow...

We appreciate the suggestion and have revised the paper with plain terms throughout the text, tables and algorithms. The original purpose is to avoid repetitive long phrases for better readability, but too condensed words introduced unnecessary confounds.

CurrentRevised
attempt chainiterative refinement loop (execute → verify → revise)
ownership rules (Table 1)update-target selection rules (failure scope → allowed edit actions)
guards (§3.2.2)update constraints
envelope of learningremoved; we name the selection rules and the update constraints directly

We have applied these terms in Sections 3.2–3.3, Table 1, Figure 3, Algorithms 1 and 3, and the appendices.


The methodological contribution and the main takeaway are unclear...

We understand the concern and would like to clarify that using failure feedback is not our contribution; Reflexion, GEPA, ACE and EvoMAS all do so. Our contribution addresses a question these methods leave to the backbone: given a failure, which part of a multi-agent system should change, and by how much.

Main takeaway: In episodic learning of multi-agent systems, the update target should be determined by the diagnosed failure, not delegated to the backbone. Otherwise, updates follow backbone preferences rather than the failure (Figure 1: Opus 5 creates 18 skills on one benchmark, while DeepSeek v3.2 creates 114 agents over 83 training tasks), and performance follows the backbone: on AppWorld challenge, EvoMAS and ADAS reach 89.0 and 84.4 TGC with Opus 5 but fall to 7.2 (EvoMAS, GLM-5) and 0.5 (ADAS, DeepSeek v3.2). With the update target fixed by the failure, EMMA's weakest backbone outperforms every baseline's weakest backbone on all three benchmarks.

Why the rules hold more generally: In our design, each memory level answers a distinct question, so each correction has exactly one level that owns it:

Memory levelQuestion it answersFailure scopeAllowed edits
Controller memoryHow is the task divided and routed?routingadd a memory note, revise the plan
Agent promptHow does an agent work on any subtask?agent behaviorrefine the prompt
SkillHow is a class of sub-problems solved?procedurerevise or create a skill

The selection rules are a credit-assignment procedure over the memory hierarchy and do not depend on the benchmark. As the constraints depend only on the memory and the revision history, they apply identically to every backbone, which is consistent with EMMA attaining the best overall performance on every benchmark across five backbone models.

Component-wise Analysis of EMMA Framework: To provide more insights beyond the effectiveness of the EMMA combination, we have added an architecture-matched ablation on KramaBench that keeps the controller, the three memory pools, the execution procedure and the training budget fixed, and varies only the update mechanism with three model families:

ConfigurationSonnet 5GLM-5DSv3.2
EMMA, full0.6410.6930.585
No selection rules0.6310.6210.493
No update constraints0.6340.6260.556
Neither0.5410.6260.556

Removing either component lowers accuracy on every backbone, and the drop is largest on the open-weight backbones (up to 15.7% for DeepSeek v3.2 without selection rules), consistent with Figure 1, where DeepSeek v3.2 creates agents regardless of the failure. The gains therefore stem from the update mechanism rather than from the architecture alone.


We have added the explanation of why the update target should be determined by the failure to Sections 1 and 3.2, the credit-assignment view of the selection rules to Section 3.2.1, and the component-wise ablation to Section 5.3 (Table 6a).


AuthorsOpenReview ↗

Rebuttal Response to Reviewer FtEy (2/3)

Several closely related methods are missing from the comparison...

We thank the reviewer for the suggestion and have added all three methods. They belong to the families our baselines cover: AFlow and GPTSwarm search over workflows or agent graphs, with the optimizer proposing each change (as in EvoMAS and ADAS), and Memento-Skills learns a skill library for a single agent, which corresponds to one level of EMMA's memory.

None of them restricts which component a failure may change or by how much. All three use the same backbones, splits and ground-truth feedback as EMMA, and the same code-execution tool as EMMA's sub-agents.

AppWorld, TGC / SGC (%)

Backbonenormal: AFlownormal: EMMAchallenge: AFlowchallenge: EMMA
Opus 592.9 / 85.796.4 / 94.688.0 / 82.089.2 / 83.5
Sonnet 570.2 / 55.489.9 / 82.153.0 / 38.884.7 / 73.4
GLM-555.4 / 30.478.6 / 57.144.1 / 21.671.5 / 51.1
DeepSeek v3.239.3 / 26.877.4 / 57.115.6 / 6.562.6 / 40.3
Kimi-K2.532.1 / 17.978.6 / 64.313.7 / 5.861.2 / 31.6
Mean58.0 / 43.284.2 / 71.042.9 / 30.973.8 / 56.0

BBEH-mini, accuracy (368 held-out examples)

BackboneAFlowGPTSwarmMemento-SkillsEMMA
Opus 50.7280.7040.6770.804
Sonnet 50.5710.6140.5080.647
GLM-50.4020.3100.3020.617
DeepSeek v3.20.2120.478-0.563
Kimi-K2.50.4380.451-0.685

EMMA attains the best result in every setting, and its mean is 72.2% above AFlow's in TGC on AppWorld challenge and 29.7% above the strongest new baseline (GPTSwarm) on BBEH-mini.

The new baselines show the same backbone dependence as EvoMAS and ADAS: AFlow nearly matches EMMA with Opus 5 on AppWorld challenge (88.0 vs. 89.2 TGC) but falls to 15.6 and 13.7 with DeepSeek v3.2 and Kimi-K2.5, and on BBEH-mini GPTSwarm and Memento-Skills fall to 0.310 and 0.302 with GLM-5, against EMMA's 0.617. The improvement is thus largest where the backbone's own update choices are least reliable, which supports the main takeaway of EMMA. The Memento-Skills runs with DeepSeek v3.2 and Kimi-K2.5 are taking longer than expected and are left blank; we will add them once they complete.


We have added the three methods to Section 5.1 and Appendix D, AFlow's AppWorld results to Table 3, the BBEH-mini results of all three methods to Table 13 (Appendix D), and the three methods to Table 15 (Table 10 in the original submission), which compares how each method constrains a single update.


Could the authors clarify what "training" means here, given that no model parameters are updated?

As defined in section 2.1, definition 1, "Training" refers to the phase in which EMMA learns its memory from the training split, with the backbone parameters θ fixed throughout. It differs from parameter training only in what is updated:

Parameter trainingEMMA training
What is updatedmodel parameters θmemory: controller memory, agent pool, skill pool (θ fixed)
Datatraining splittraining split
Update signalgradient of a lossa failed execution trace with ground truth
Update stepgradient stepone memory edit within the diagnosed scope
Outputupdated weightsa memory frozen for evaluation, with no further updates

Concretely, training is a single pass over the training tasks. Each task is processed by the iterative refinement loop: it is executed, verified against its reference (an LLM judge on KramaBench, unit tests on AppWorld, exact match on BBEH-mini) and, if incorrect, revised by one memory edit and re-run, up to five revisions. A training execution (Table 4) is one full execution of the system on a training task, re-runs included. We use the term in the same sense as GEPA and ACE, which also learn non-parametric artifacts from a training split and evaluate them on held-out tasks.


We have added this clarification after Definition 1 (Section 2.1) and defined a training execution in Section 3.3.


AuthorsOpenReview ↗

Rebuttal Response to Reviewer FtEy (3/3)

Do all methods have comparable adaptation budgets and access to feedback? How does freezing memory at test time affect baselines originally designed for online adaptation, such as EvoMAS?

Yes. We clarify the budgets and feedback, and have added experiments on EvoMAS to isolate the effect of freezing.

Adaptation budgets are comparable. Every method is capped at the same interaction budget per step: each agent code-execution step allows at most 30 rounds of environment interaction. For adaptation, we count the training task executions of every method (Table 4). EMMA uses the fewest on every benchmark, 17–38% fewer than the most efficient baseline, so no baseline received a smaller adaptation budget than EMMA.

All methods learn from ground-truth feedback through their own mechanisms. All methods adapt on the same training split with ground-truth feedback. Baselines such as GEPA and EvoMAS support ground-truth feedback natively, and we use these mechanisms without modification (Appendix D).

Freezing memory does not explain the gap. Table 6c (Table 5 in the original submission) already compares EMMA with methods that adapt on the test tasks themselves, all of which remain below EMMA. To isolate freezing for EvoMAS, we additionally run EvoMAS-online, which continues to evolve across the test tasks without ground truth, as in Table 6c:

EvoMAS-online vs. EMMA: AppWorld (TGC / SGC, %) and BBEH-mini (accuracy)

Backbonenormal: EvoMAS-onlinenormal: EMMAchallenge: EvoMAS-onlinechallenge: EMMABBEH: EvoMAS-onlineBBEH: EMMA
Opus 596.4 / 96.496.4 / 94.691.8 / 83.589.2 / 83.50.6820.804
Sonnet 576.8 / 60.789.9 / 82.154.4 / 33.184.7 / 73.40.5650.647
GLM-541.1 / 17.978.6 / 57.139.8 / 19.471.5 / 51.10.4380.617
DeepSeek v3.256.5 / 44.677.4 / 57.136.2 / 16.562.6 / 40.30.5080.563
Kimi-K2.558.3 / 33.978.6 / 64.347.2 / 28.161.2 / 31.60.4480.685
Mean65.8 / 50.784.2 / 71.053.9 / 36.173.8 / 56.00.5280.663

Online adaptation improves EvoMAS over its frozen configuration (Table 3) by 28.4% in mean TGC on AppWorld normal, most on the open-weight backbones (16.7 to 41.1 on GLM-5). Even so, EMMA with frozen memory improves over EvoMAS-online by 27.9% in mean TGC on AppWorld normal, 37.0% on AppWorld challenge and 25.6% in mean accuracy on BBEH-mini. EvoMAS-online matches or exceeds EMMA only with Opus 5 on AppWorld, the same backbone dependence shown in Figure 4. The gains therefore stem from the memory EMMA learns during training rather than from the evaluation protocol.


We have added the EvoMAS-online results to the revised paper (Section 5.3, Table 5, with per-backbone results in Table 14, Appendix D), and stated the interaction budget and feedback of every method in Section 5.1 and Appendix D.


Thank you again for the insightful comments and we have incorporated your suggestions in the revised version of the manuscript, highlighted in blue. If there are any additional questions or concerns, we would be happy to address them.

Should you find that our responses sufficiently address your feedback, we would sincerely appreciate your consideration of raising the score, if deemed appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.