acceptodds
Under review as a conference paper at ICLR 2027

ECHO: Evolving Coordination from Human Operations for Training-Free LLM-Based Multi-Agent Decision Making

Abstract

Large language models (LLMs) enable training-free multi-agent decision-making, but human interventions in these systems are typically executed as transient, single-agent commands. This local execution fails to re-coordinate the remaining agents, forcing human operators to repeatedly perform similar manual interventions across episodes. We formulate human-guided multi-agent adaptation as inducing reusable, anchor-conditioned coordination patterns from localized human operations. We introduce ECHO, a training-free framework that grounds each human intervention as an immutable anchor within a shared collaboration graph and dynamically reorganizes the roles, intents, subgoals, and coordination dependencies of responder agents around it without modifying the underlying LLM parameters. The resulting responder-side adjustments are abstracted into identity-agnostic, relational coordination operators that encode transferable team reorganization knowledge while accumulating outcome evidence and human feedback. A structured matching mechanism and an evidence-gated Shadow deployment pipeline progressively promote candidate operators from confirmation-dependent execution to fully autonomous application. Evaluations on SMAC and Overcooked-AI show that ECHO improves immediate complementary coordination and supports cross-episode knowledge reuse. Across continual evolution, ECHO maintains a 100.0% success rate while reducing human intervention rate by 51.8%. Frozen operators also reduce human interaction on held-out tasks, with no harmful automatic applications observed in the evaluated transfer trials. These results show that human operations can provide persistent coordination supervision rather than only momentary control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human-AI Collaboration, Multi-Agent Coordination, Training-Free Continual Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Hsh1OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2(−1)
Presentation
2
Soundness
2(−1)
Read review
Paper summary

The authors propose ECHO. ECHO studies how a local human intervention can improve coordination across a team and be reused later. It preserves the human command, adjusts the other agents through a collaboration graph, and stores those adjustments as relational operators. Operators gain permission for automatic reuse through execution evidence and human feedback. Experiments on SMAC and Overcooked include a participant study, continual adaptation, and transfer with frozen memory. The distinction between immediate coordination and accumulated memory is important here. The former asks whether teammates respond better to a human command; the latter asks whether similar situations need fewer interventions over time. The paper evaluates both, rather than attributing every improvement to memory. “Training-free” refers to frozen LLM parameters, while expert guidance and operator collection still require effort.

Critical strengths
  • The authors discuss clearly the choice of what to remember: how teammates should respond to a protected human action. This is more useful for coordination than storing the command alone.
  • The controls are reasonably thorough. The 12-participant comparison improves success without memory (Table 2). In the three-seed evolution study, interactions fall from 29.7 to 13.5 per episode while success stays at 100% on the evaluated map. The confirmation-only and no-safeguard variants help explain why the full design works.
  • The ablation results show why this distinction matters in practice: NoSafeguard becomes less successful as memory accumulates, whereas full ECHO maintains performance. Simply collecting more reusable rules is therefore not enough. The frozen-memory transfer test adds useful evidence that operators capture something beyond episode-specific unit identities.
Critical weaknesses
  • The practical workload benefit is less clear than the interaction counts suggest. Inference pauses the environment, and automatic reuse has a 15-second veto window. Fewer clicks may still require substantial monitoring. The NASA-TLX results help, but do not establish how much time the system saves.
  • Transfer is tested on only three related map pairs. The harmful automatic applications is encouraging, but repeated uses share operators and episodes. It is therefore limited evidence for reliability beyond these settings. There is also a difference between reducing intervention frequency and solving the harder tasks. On the difficult transfer targets, success remains around one third (Table 4). The benefit is therefore mainly lower human involvement at comparable or modestly improved performance, rather than consistently successful autonomous coordination.
  • The novelty lies in a fairly specific representation, and its advantage over prior procedural memory is not fully isolated. Existing works already induces reusable routines from experience and retrieves them for later tasks. From my view ECHO's more specific contribution is the anchor-conditioned responder-side graph rewrite, together with its reuse gate. TextRAG removes structure and gating together, while IDSpecific deliberately retains instance-bound selectors. Neither establishes that graph-difference memory is better than a transferable procedure or rule representation given the same relational selectors, feedback, and authorization mechanism. The current evidence supports the integrated system more clearly than it establishes a new memory principle.
  • I have carefully read Reviewer jWhw’s comments and the authors’ rebuttal. The explanations are reasonable, and I appreciate the corrections, but the errors across multiple tables still reduce my confidence in the experimental results. Even if the overall trends remain unchanged, the number and range of corrections raise concerns about the reliability of the data aggregation and reporting process.
Questions for authors
  • What can an anchor-conditioned graph-difference operator express or transfer that a relational procedure/rule memory cannot? Could the authors compare these representations while keeping anchor protection, relational selectors, feedback, and reuse gating fixed? Table 11 tests identifier abstraction, but does not answer this representation question.
  • How much wall-clock and active supervision time is saved over Static and ConfirmOnly, including veto windows and memory collection? How was the Static versus evolved-ECHO workload comparison controlled for order and prior exposure?
  • How many distinct operators and episodes account for the 786 automatic applications, and does uncertainty assessed at that level support the same conclusion?
6 comments

We thank the reviewer for the constructive feedback. (1) We clarify ECHO’s contribution and compare its graph-difference operators with equivalent relational rules. (2) We also report wall-clock measurements, explain how we assessed supervision and workload, and (3) add an episode-level uncertainty analysis of harmful automatic applications. (4) Finally, we clarify the scope of our transfer results and distinguish reductions in human involvement from gains in task success.


1. Comparison with relational procedural memory (Q1/W3).

[Core Contribution] ECHO's core contribution is the kind of coordination knowledge it extracts: how the rest of the team should adjust its current plan in response to a protected human intervention. For the same environment state,

    1. ECHO compares an autonomous plan with a plan that preserves the intervention as an anchor. The differences isolate which responder roles, intents, and dependencies need to change relative to what the team would otherwise do.
    1. ECHO abstracts these adjustments so that they can be applied to different teammates in a new scenario when the relevant relational conditions hold.

Graph differences provide one way to encode that knowledge, and sufficiently expressive relational rules can encode it as well.

[Supplementary Experiments] Table 11 compares relational and instance-specific selectors. To address the encoding question directly, we added a controlled comparison between ECHO's graph-difference operators and relational rules converted from the same operators. Both conditions use the same learned coordination adjustments, anchor protection, relational selectors, feedback, and reuse gating. We froze the source memory and evaluated both encodings on the same held-out transfer tasks.

Source task → Target taskMemory encodingSR (%)HI / episodeAutomatic apps. / episode
3s_vs_4z → 3s_vs_3zECHO100.0 (30/30)13.5 ± 2.67.5 ± 2.4
3s_vs_4z → 3s_vs_3zRelational Rules100.0 (30/30)15.3 ± 2.89.1 ± 2.4
3s_vs_4z → 3s_vs_5zECHO36.7 (11/30)34.8 ± 5.012.6 ± 3.4
3s_vs_4z → 3s_vs_5zRelational Rules33.3 (10/30)36.7 ± 5.414.2 ± 3.5
2s3z → 3s5zECHO33.3 (10/30)19.7 ± 3.66.1 ± 2.3
2s3z → 3s5zRelational Rules40.0 (12/30)22.3 ± 4.35.4 ± 2.6

(SR denotes success rate and HI denotes human interactions. Per-episode counts are reported as mean ± SD.)

Both encodings transferred the same learned coordination adjustments to held-out tasks, with similar success rates and no consistent advantage for either encoding.

[Why We Use Graph Encoding] We use graph encoding because typed edges make inter-agent dependencies explicit and graph differences record only the assignments and relationships that change. We therefore expect operators to remain compact as team size grows when interventions affect only a few units. In our 3s_vs_4z source memory, converting the 96 retained operators and their variants yielded 3,162 rule instructions. Prior work (Fatemi et al., ICLR 2024) also shows that graph encoding choices can affect LLM performance on relational reasoning tasks.

2. Time savings and workload controls (Q2/W1).

[Supplementary Experiments] We compared wall-clock time for evolved ECHO, ECHO-Static, and ECHO-ConfirmOnly on 3s_vs_3z, 3s_vs_4z, and 2c_vs_64zg, measuring from episode start to termination and including inference pauses, confirmation waits, and applicable veto windows. ECHO collects operator evidence during ordinary interactions, including when users accept or reject Shadow proposals, so evidence collection takes no additional user time.

Method3s_vs_3z3s_vs_4z2c_vs_64zgOverall
ECHO-Static793.87 ± 22.541255.60 ± 82.41652.80 ± 41.49900.76 ± 265.80
ECHO-ConfirmOnly752.60 ± 73.811307.67 ± 63.70619.87 ± 28.11893.38 ± 306.67
ECHO736.33 ± 58.651312.27 ± 65.18614.20 ± 26.83887.60 ± 312.15

(Episode wall-clock time (s, mean ± SD), each method was evaluated over 15 episodes per map.)

Under the same stable network and model API conditions, ECHO averaged 13.16 s less per episode than ECHO-Static and 5.78 s less than ECHO-ConfirmOnly across the three maps. ECHO was faster than both baselines on two maps and slower on 3s_vs_4z. All three methods draw on the same underlying coordination knowledge, which can lead to similar task progression and comparable episode durations. However, in some 3s_vs_3z runs, we observed that users in ECHO-Static had to work out how to adjust team coordination themselves and sometimes made redundant interventions. ECHO reused a learned coordination structure, avoided those extra interventions, and completed the task sooner.

[Active Supervision Time and Mental Effort] In all three conditions, users are expected to follow the task from start to termination, so we use each episode's wall-clock time as the duration of required monitoring. Comparable monitoring durations can still involve different levels of mental effort. For example,

  • With ECHO-Static, users must continually assess whether the current coordination is adequate. When they spot a problem, they must decide which unit to intervene on and how to change its intent to improve team coordination.

  • With ECHO, autonomous planning and operator reuse handle routine coordination, allowing users to focus on following task progress for much of the episode. Users step in when coordination needs correction or a proposed adjustment needs review.

In the latter case, users primarily judge whether the proposed adjustment is appropriate. Human intervention counts capture how often users step in, and NASA-TLX, including its temporal demand subscale, captures how demanding they find the task. Together, these measures characterize the workload within the time users spend monitoring.

[Condition Order and Prior Exposure] After standardized training, all 12 participants completed ECHO first, followed by the ECHO-Direct and ECHO-Static block under the protocol described in the manuscript. Prior exposure could have helped participants perform the later conditions through greater familiarity with the task. Participants nevertheless reported lower workload for ECHO than for Static. This result supports a perceived workload benefit for ECHO under the tested protocol. We agree that the fixed order of evolved ECHO and Static could have influenced the ratings, and we will acknowledge this limitation in the manuscript.

3. Distinct operators, episodes, and uncertainty (Q3/W2).

[Supplementary Data] Because repeated applications can share operators and episode histories, we counted the distinct operators and episodes underlying the 786 automatic applications in Table 15. These applications involved 20 distinct operators across 90 episodes (30 per transfer pair). The first two pairs used 15 distinct operators from their shared frozen source memory after deduplication, and the third pair contributed five additional operators.

Source → Target taskAuto apps.Distinct operatorsEpisodesApps. per operator
3s_vs_4z→3s_vs_3z2251330/3017 [8–31]
3s_vs_4z→3s_vs_5z3781530/3025 [14–42]
2s3z→3s5z183530/3035 [29–48]
Total78620 (15+5)90/9036.5 [22, 73]

("Apps. per operator" is reported as median [min–max].)

  • Applications of 13 operators on 3s_vs_3z:

    validated operators: 31,16,23,17,19,22,15,21,20;

    auto-ready operators: 8,11,12,10;

  • Applications of 15 operators on 3s_vs_5z:

    validated operators: 42,19,31,25,37,28,22,20,18,30,34;

    auto-ready operators: 14,17,15,26;

  • Applications of 5 operators on 3s5z:

    validated operators: 48,40,31;

    auto-ready operators: 35,29;

Source → Target taskAuto apps.Distinct operatorsEpisodesApps. per operator
3s_vs_4z→3s_vs_4z1341010/106 [4, 53]
  • Applications of 10 operators on 3s_vs_4z:

    validated operators: 34,7,9,5,53,5;

    auto-ready operators: 4,5,4,8;

[Episode-Level Uncertainty] We observed no harmful outcomes in 920 (786+134) automatic applications across 100 episodes (90 transfer + 10 nomal) and 20 distinct operators. We classified an episode as harmful if it contained at least one harmful automatic application. The nominal Wilson 95% interval for the harmful-episode rate was 0–4.1% (0/90) on the transfer tasks. Ten additional episodes on the source map, 3s_vs_4z, also had no harmful automatic applications. Pooling the source and transfer episodes gave a nominal interval of 0–3.7% (0/100), corresponding to 96.3%–100% for the proportion of harm-free episodes under the evaluated conditions.

[Implications for Our Reliability Claims] The expanded evaluation supports our original conclusion: we observed no harmful outcomes in 920 automatic applications of 20 distinct operators across 100 episodes. Because applications within an episode can be correlated, we use the episode as the unit of analysis. The Wilson 95% confidence interval for the harmful-episode rate was 0–3.7%, corresponding to a 96.3% lower confidence bound on the proportion of episodes free of harmful automatic applications under the evaluated conditions.


4. Transfer scope and task performance (W2).

Complementary coordination can improve task success. On 3s_vs_5z, adding complementary coordination to human input increased the observed success rate from 16.7% with ECHO-Direct to 30.0% with ECHO-Static, with operator memory disabled in both conditions (Section 4.2).

Operator evolution is designed to reduce repeated human involvement while retaining task performance, not to achieve further substantial gains in success rates. On the two difficult transfer targets in Table 4, reusing evolved operators reduced human interactions relative to ECHO-Static, while success remained around one third. The primary benefit of evolution on these targets was therefore reduced human involvement at comparable or modestly improved performance.

We will clarify this distinction in Section 4.4 and emphasize that reliable autonomous completion on difficult transfer targets remains an open challenge. ECHO provides a promising foundation for investigating how operator evolution could further improve success rates on these targets.

Reviewer_jWhwOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

ECHO is a training-free framework that converts local human interventions into reusable coordination knowledge for LLM-based multi-agent systems. It treats human operations as anchors in a shared collaboration graph and reorganizes the remaining agents around them. These adjustments are abstracted into relational coordination operators, whose reuse is governed by structured matching and evidence-gated lifecycle management. Experiments examine immediate coordination, continual operator evolution, and transfer to held-out tasks without updating the underlying LLM parameters.

Critical strengths
  • The anchor mechanism provides a clear way to translate local human interventions into complementary team-wide responses.
  • Relational operators capture reusable coordination patterns across episodes and tasks without updating model parameters.
Critical weaknesses
  • Several entries in Tables 1, 6, and 9 appear inconsistent with the stated metric definitions, raising concerns about the reliability of the reported results.
  • The NASA-TLX evaluation uses operator memory obtained after 60 evolution episodes. This assessment does not capture the workload throughout the preceding process of learning and validating operators.
Questions for authors
  • Some reported metrics appear inconsistent under the stated definitions. In Table 1, ECHO-Auto’s 80% success rate on 2c_vs_64zg implies at least 51.20 mean enemy kills, yet 47.13 is reported; similar issues appear in Table 6. In Table 9, ECHO-ConfirmOnly’s interaction components for episodes 31–40 sum to 23.0, but the reported total is 21.7. Could the authors clarify these discrepancies and whether they affect the main conclusions?
8 comments

We sincerely thank the reviewer for carefully examining our results and identifying these data inconsistencies. We fully acknowledge these unintentional oversights and sincerely apologize for the errors. In response, we have systematically re-examined the original experimental records, corrected the discrepancies, and carefully assessed their impact on our findings.

  • Sources of discrepancies. Our examination confirmed that these discrepancies arose from inadvertent oversights during data aggregation and manuscript preparation, rather than errors in the underlying experiments. Specifically, they involved the accidental inclusion of outdated success criterion in Tables 1 and 5, the misapplication of map-specific counts in Table 6, and an incorrect episode-block entry in the mean calculation in Table 9. These were isolated reporting and processing errors affecting a limited number of entries.
  • Impact on the conclusions. We have corrected the identified errors and updated the corresponding tables accordingly. Importantly, the revised results remain consistent with our original findings and do not alter the main conclusions regarding complementary human–agent coordination and the reduction in human intervention achieved through operator evolution. Although these corrections lead to minor numerical changes, the overall trends and comparative findings remain unchanged.
  • Supporting evidence. For transparency and reproducibility, we provide the original experimental records, detailed calculations, and explanations of each correction. All supporting data have also been uploaded to our anonymous repository (https://anonymous.4open.science/r/-ECHO/) for independent verification.
AuthorsOpenReview ↗

[1/4] Analysis of Table 9

[Analysis of Causes] In Table 9, for ECHO-ConfirmOnly over episodes 31–40, the direct-operation mean mistakenly used Seed 3’s value from episodes 21–30 (22.4) instead of the correct value (18.6). This produced (18.1 + 17.7 + 22.4) / 3 = 19.4.

Below, each array follows the episode-block order from 1–10 to 51–60.

Seed 1
"total_human_interactions_per_episode": {"ECHO-ConfirmOnly": [29.5, 27.4, 24.1, 21.7, 21.2, 20.4]},
"direct_operations_per_episode": {"ECHO-ConfirmOnly": [28.0, 25.5, 21.8, 18.1, 16.7, 15.5]},
"shadow_decisions_per_episode": {"ECHO-ConfirmOnly": [1.5, 1.9, 2.3, 3.6, 4.5, 4.9]}

Seed 2
"total_human_interactions_per_episode": {"ECHO-ConfirmOnly": [28.9, 26.8, 23.7, 21.5, 21.5, 20.0]},
"direct_operations_per_episode": {"ECHO-ConfirmOnly": [27.5, 24.8, 21.1, 17.7, 16.6, 14.9]},
"shadow_decisions_per_episode": {"ECHO-ConfirmOnly": [1.4, 2.0, 2.6, 3.8, 4.9, 5.1]}

Seed 3
"total_human_interactions_per_episode": {"ECHO-ConfirmOnly": [29.5, 27.6, 24.5, 22.0, 21.3, 20.5]},
"direct_operations_per_episode": {"ECHO-ConfirmOnly": [28.3, 25.8, 22.4, 18.6, 17.1, 15.8]},
"shadow_decisions_per_episode": {"ECHO-ConfirmOnly": [1.2, 1.8, 2.1, 3.4, 4.2, 4.7]}

[Data Correction] The correct values for episodes 31–40 are:

  • Direct operations: (18.1 + 17.7 + 18.6) / 3 = 18.1.
  • Shadow decisions: (3.6 + 3.8 + 3.4) / 3 = 3.6.
  • Total human interactions: (21.7 + 21.5 + 22.0) / 3 = 21.7.

The direct-operation mean should therefore be corrected from 19.4 to 18.1. The Shadow-decision and total-interaction means remain unchanged, and the displayed values satisfy 18.1 + 3.6 = 21.7.

We will provide the complete Table 9 data in the anonymous repository.

AuthorsOpenReview ↗

[2/4] Analysis of Tables 1 and 5

[Analysis of Causes] Tables 1 and 5 retained success labels for 2c_vs_64zg calculated under an earlier criterion, although the manuscript stated the updated criterion. This was a failure to synchronize the reported success rates during manuscript revision.

  • Earlier criterion: To account for the imbalance between 2 allied units and 64 enemies, success meant either eliminating all enemies within 64 seconds or having at least one allied unit alive at the time limit. Thus, 80% success did not require 24 episodes with all enemies eliminated and could coexist with 47.13 average enemy kills.
  • Updated criterion: Success requires eliminating all enemies within the prescribed time limit on every map. The reviewer's lower bound of 51.20 is correct under this definition. The discrepancy arose because the tables retained earlier results while the manuscript stated the updated criterion.

[Data Correction] We recomputed success rates from the same episode records (success indicator, enemy kills, allied deaths) using the criterion stated in the manuscript. Episode trajectories and time limits are unchanged. Successful episodes with fewer than 64 kills indicate survival at the time limit under the earlier criterion; these episodes are classified as failures under the updated criterion.

  • LLM-PySC2: (0, 25, 2), (0, 27, 2), (0, 22, 2), (0, 23, 2), (0, 26, 2), (0, 15, 2), (0, 20, 2), (0, 17, 2), (0, 18, 2), (0, 15, 2), (0, 18, 2), (0, 16, 2), (0, 16, 2), (0, 17, 2), (0, 14, 2), (0, 23, 2), (0, 20, 2), (0, 24, 2), (0, 21, 2), (0, 28, 2), (0, 24, 2), (0, 14, 2), (0, 28, 2), (0, 29, 2), (0, 19, 2), (0, 26, 2), (0, 21, 2), (0, 19, 2), (0, 22, 2), (0, 27, 2);

    In earlier criterion: 0.0%(0/30), 21.13/2.00

    In updated criterion: 0.0%(0/30), 21.13/2.00


  • RuleExpert: (1, 64, 1), (1, 64, 0), (0, 37, 2), (1, 64, 1), (1, 64, 1), (0, 28, 2), (0, 39, 2), (1, 36(<64), 0), (1, 64, 1), (1, 64, 0), (1, 64, 0), (0, 29, 2), (0, 40, 2), (0, 38, 2), (1, 35(<64), 1), (0, 34, 2), (1, 64, 1), (0, 41, 2), (1, 64, 0), (1, 40(<64), 0), (0, 32, 2), (1, 64, 1), (0, 30, 2), (1, 40(<64), 1), (1, 64, 1), (0, 43, 2), (0, 35, 2), (1, 42(<64), 0), (1, 64, 0), (0, 33, 2);

    In earlier criterion: 56.7%(17/30), 47.33/1.17

    In updated criterion: 40.0%((17-5)/30), 47.33/1.17


  • HEP (adapted): (0, 29, 2), (1, 64, 0), (1, 64, 0), (1, 33(<64), 0), (0, 35, 2), (0, 28, 2), (1, 64, 1), (1, 64, 1), (0, 32, 2), (0, 33, 2), (0, 36, 2), (0, 39, 2), (0, 38, 2), (1, 33(<64), 1), (1, 29(<64), 1), (1, 64, 1), (1, 64, 0), (1, 64, 0), (1, 64, 0), (0, 26, 2), (1, 64, 0), (1, 25(<64), 1), (1, 24(<64), 0), (0, 30, 2), (0, 41, 2), (1, 64, 0), (1, 64, 1), (1, 64, 1), (1, 64, 0), (1, 32(<64), 1);

    In earlier criterion: 63.3%(19/30), 45.83/1.03

    In updated criterion: 43.3%((19-6)/30), 45.83/1.03


  • ECHO-Auto: (1, 11(<64), 1), (1, 64, 1), (0, 16, 2), (1, 64, 1), (1, 64, 1), (1, 64, 1), (1, 64, 0), (0, 25, 2), (0, 21, 2), (1, 64, 0), (1, 16(<64), 1), (1, 64, 0), (1, 64, 0), (1, 64, 1), (1, 64, 0), (0, 28, 2), (0, 19, 2), (1, 64, 1), (1, 64, 0), (1, 64, 1), (1, 64, 1), (1, 64, 0), (0, 13, 2), (1, 16(<64), 1), (1, 64, 1), (1, 64, 1), (1, 64, 1), (1, 15(<64), 0), (1, 18(<64), 1), (1, 64, 1);

    In earlier criterion: 80.0%(24/30), 47.13/0.93

    In updated criterion: 63.3%((24-5)/30), 47.13/0.93


  • ECHO-Direct: (1, 64, 1), (1, 64, 1), (1, 64, 1), (1, 64, 1), (0, 18, 2), (1, 64, 1), (1, 45(<64), 1), (1, 37(<64), 0), (1, 64, 1), (0, 23, 2), (1, 64, 1), (1, 64, 1), (1, 64, 1), (1, 64, 1), (1, 64, 1), (1, 46(<64), 0), (0, 28, 2), (1, 64, 1), (1, 64, 1), (1, 64, 0), (1, 64, 1), (1, 64, 1), (1, 64, 1), (1, 64, 0), (1, 42(<64), 1), (0, 14, 2), (1, 64, 1), (1, 64, 1), (1, 64, 0), (1, 64, 1);

    In earlier criterion: 86.7%(26/30), 55.37/0.97

    In updated criterion: 73.3%((26-4)/30), 55.37/0.97


  • ECHO-Static: (1, 64, 0) × 29 episodes, (1, 64, 1) × 1 episode;

    In earlier criterion: 100.0%(30/30), 64.00/0.03

    In updated criterion: 100.0%(30/30), 64.00/0.03

Reviewer_mumrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ECHO, a training-free framework that converts localized human interventions into complementary multi-agent coordination and reusable coordination knowledge. The method preserves a grounded human operation as an immutable anchor in a collaboration graph, reorganizes the remaining agents around that anchor, and abstracts the resulting responder-side graph changes into identity-agnostic relational coordination operators. These operators accumulate execution outcomes and human feedback, and their reuse authority evolves through an evidence-gated lifecycle, from candidate operators to autonomous application.

The framework is evaluated on SMAC and Overcooked-AI, with experiments covering immediate coordination, continual operator evolution, component ablations, frozen-memory transfer, and human interaction. The results suggest that ECHO can reduce repeated human intervention while maintaining task performance in the evaluated settings, and can reuse learned coordination patterns on related held-out tasks without updating the underlying LLM parameters.

Overall, I find the problem meaningful and the proposed framework reasonably well motivated. The main contribution is the integration of anchor-constrained team reconfiguration, relational coordination-operator abstraction, and evidence-gated reuse into a unified mechanism. Although several underlying ideas have precedents in the literature, I do not find sufficient evidence that an existing method already subsumes this complete design.

Critical strengths

1.A meaningful and clearly formulated problem. The paper identifies a practical limitation of LLM-based multi-agent systems: a local human command may not induce the complementary responses required from the rest of the team, and similar interventions must consequently be repeated across episodes. Treating human interventions as persistent coordination supervision is a useful perspective.

2.A coherent representation of reusable coordination knowledge. ECHO does not simply store human commands or complete interaction trajectories. Instead, it extracts responder-side graph changes while preserving the human-specified anchor, and represents these changes through relational selectors rather than episode-specific agent identities. This provides a concrete abstraction for reusing coordination patterns across compatible situations.

3.A useful separation between memory and execution authority. The evidence-gated operator lifecycle distinguishes storing an observed coordination pattern from authorizing its reuse. Structured matching, ambiguity-based abstention, rejection handling, human-confirmed Shadow proposals, and eventual autonomous application collectively provide a more controlled reuse mechanism than unconditional retrieval and execution.

4.Mechanism-oriented experimental evidence. The paper includes targeted comparisons and ablations, including ECHO-Direct versus ECHO-Static, structured operator memory versus textual retrieval, mandatory confirmation versus autonomous reuse, removal of safeguards, and relational selectors versus identity-specific selectors. These experiments provide evidence for the proposed mechanisms rather than relying exclusively on aggregate task performance.

5.Evidence of reduced human involvement. During continual evolution on 3s vs 4z, ECHO maintains a 100% success rate while reducing the reported human intervention rate from approximately 57.0% to 27.5%. Total human interactions decrease from approximately 29.7 to 13.5 per episode. On the more difficult 3s vs 5z task, the intervention rate also decreases over time, although task success remains substantially lower. The held-out transfer experiments further show reductions in human interactions while maintaining or improving success rates across the evaluated transfer pairs.

6.Attention to reuse safety and reproducibility. The paper reports rejection and harmful-application statistics, provides explicit matching and promotion thresholds, and includes implementation details and code. The safety evaluation is limited to the tested settings, but it is a useful component of the empirical study.

Critical weaknesses

1.The novelty relative to prior work could be positioned more precisely. Graph-structured agent memory, human-feedback-driven adaptation, reusable skills, and evidence-based experience management have all been explored in previous work. In particular, the discussion would benefit from a more direct comparison with G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems (NeurIPS 2025) and Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026). Nevertheless, these conceptual overlaps do not by themselves establish that ECHO lacks novelty. The potentially distinctive contribution is the particular combination of immutable human anchors, relational responder-side graph rewrites, and an evidence-gated operator authorization lifecycle. I did not identify a prior work that clearly subsumes this complete mechanism. The related-work discussion should sharpen this distinction, but the current evidence does not justify treating the contribution as merely a renamed existing method.

2.The evaluation remains limited in scale and scope. Most method-task comparisons use 30 trials, the continual-evolution experiments use three random seeds, and the human study involves 12 participants. These are useful initial evaluations, but some success-rate estimates remain uncertain. Performance on the more difficult 3s vs 5z task is also modest, with success rates fluctuating rather than approaching saturation. The paper should therefore frame its conclusions primarily around reduced human involvement and coordination reuse under the evaluated conditions, rather than implying uniformly robust task-solving performance.

3.The sensitivity of the operator-management mechanism is not fully established. The approach relies on several manually specified thresholds for operator matching, merging, promotion, rejection, and autonomous reuse. The ablations demonstrate the importance of the overall safeguards, but they do not fully establish how sensitive the results are to these individual threshold choices or how readily the settings transfer to new environments.

4.The generalization claims should remain appropriately bounded. The frozen-memory transfer experiments provide useful evidence of transfer between related SMAC tasks. The separate Overcooked-AI evaluation also demonstrates that the framework can be instantiated in another coordination environment. However, the current experiments do not establish that operators themselves transfer across substantially different domains or that the same parameterization works without environment-specific adaptation.

5.The safety results should be interpreted narrowly. The paper reports no harmful outcomes among 786 automatic operator applications, but also reports three harmful outcomes among 502 human-confirmed applications. These results support the usefulness of the proposed safeguards in the tested settings, but they do not establish a general safety guarantee. The paper appropriately acknowledges some of these limitations; the distinction between empirical safety evidence and broader safety claims should remain explicit.

Questions for authors

1.Could the authors clarify the distinction between ECHO's relational coordination operators and existing graph-structured agent memories or reusable agent skills, particularly G-Memory and HILA? Which aspects of the representation and authorization lifecycle are essential to ECHO's contribution?

2.How sensitive are operator reuse and continual-evolution results to the matching, merging, and promotion thresholds? Even a limited sensitivity analysis or a clearer discussion of how these values were selected would strengthen the case for robustness.

3.Can the authors clarify the circumstances behind the harmful human-confirmed applications and discuss how the current safeguards might address similar failures during autonomous reuse? The reported absence of harmful automatic applications is encouraging, but its interpretation should account for the finite evaluation sample.

2 comments

We thank the reviewer for the constructive feedback. We conducted a threshold sensitivity experiment on 3s_vs_4z and found that task success remained stable across the tested configurations. We also clarify

  • how ECHO differs from prior work,
  • how harmful applications occurred,
  • what safeguards govern autonomous reuse,
  • and how the evaluation bounds our claims about operator evolution and generalization.

We will include the new results in the manuscript and revise the abstract and the related work, discussion, and limitations sections to reflect these clarifications.

Below, we address the three questions first, followed by the remaining concerns.


1. Distinction from prior work (Q1/W1).

  • G-Memory distills generalizable insights from past collaboration trajectories and uses a graph hierarchy to retrieve them alongside relevant interactions. These insights guide future collaboration while leaving the specific adjustments that improve team coordination implicit. ECHO explicitly extracts and represents those adjustments. For the same environment state, ECHO compares an autonomous plan with a plan that preserves the human intervention as an anchor. It stores the resulting adjustments to responders’ roles, intents, and dependencies as a relational rewrite that can be reused across teammates in compatible scenarios.
  • HILA uses cost-aware reinforcement learning to train agents to decide when to seek expert help and turns expert feedback into supervision for continual policy training. Its focus on policy improvement leaves open how teammates should coordinate around a local human intervention during execution. ECHO addresses that team coordination problem and uses feedback to update operator reuse authority with fixed model parameters. Relational matching determines whether a rewrite applies in the current context. Execution outcomes and human feedback determine whether a matched rewrite requires confirmation or can run autonomously.

Together, these mechanisms let ECHO learn reusable coordination adjustments from protected human interventions and use accumulated evidence to decide when those adjustments can be applied autonomously. We will clarify these distinctions in the related work.


2. Threshold sensitivity (Q2/W3).

Threshold sensitivity on 3s_vs_4z. Each configuration uses 60 evolution and 30 evaluation episodes, with other settings fixed.

ConfigurationMatching thresholdMerging thresholdMin. positive outcomesSuccess rate (%)Human interactions / episodeAutomatic app. / episode
Lenient0.650.702100.015.311.5
Default0.750.803100.013.610.6
Strict0.850.904100.014.88.9

We compared three joint threshold configurations: Lenient, Default, and Strict. Performance varied only modestly within the tested range.

However, overly permissive matching can generate irrelevant proposals requiring human inspection and rejection, increasing supervision burden, while permissive promotion can authorize insufficiently supported operators. Conversely, overly strict thresholds limit retrieval or delay autonomous reuse. Our defaults balance these effects, favoring missed reuse opportunities over inappropriate applications.


3. Harmful applications and safeguards (Q3/W5).

  • Circumstances. The harmful applications occurred after human confirmation of candidate or provisional operators that passed matching checks but were insufficiently suited to the current context. For illustration, a retreat-cover operator may preserve the retreating unit’s anchor while assigning an overly vulnerable ally to provide cover. Human approval permits execution but does not establish reliability; an adverse outcome does not justify promotion.
  • Safeguards for autonomous reuse. Protection operates at two levels. Before authorization, promotion to auto-ready requires positive outcomes across episodes, bounded negative-outcome and rejection rates, and sufficient Shadow confirmations. A single approval does not grant autonomous authority. During execution, operators remain subject to compatibility checks, ambiguity abstention, anchor preservation, action-validity checks, and a 15-second pre-execution veto window. Freezing memory disables evidence updates but retains previously earned authorization and runtime safeguards.

Three of 502 human-confirmed applications were harmful. We observed no harmful applications in 786 automatic executions (Wilson 95% CI: 0–0.5%), providing empirical evidence of safety under the evaluated conditions. This finding does not constitute an absolute safety guarantee.

AuthorsOpenReview ↗

Remaining Concerns

4. Evaluation scope (W2).

We agree that the trial counts, three seeds, and 12-participant study limit the precision and scope of our conclusions. Although absolute success on 3s vs 5z remains modest, Section 4.2 shows an improvement with operator memory disabled: success increases from 3.3% for ECHO-Auto to 16.7% for ECHO-Direct and 30.0% for ECHO-Static, supporting the benefits of human input and complementary coordination.

Operator evolution serves a distinct purpose central to this work: reducing repeated human involvement while retaining task performance. On 3s_vs_5z, mean human interactions fell from 49.7 per episode in the first 10-episode window to 38.0 in the last, averaged across three seeds. Success ranged from 26.7% to 40.0% across the six 10-episode windows. These results demonstrate reduced human involvement on this task. We will also clarify that the abstract’s 100% success result refers to continual evolution on 3s_vs_4z.

Building on this work, a promising direction for future research is to explore how ECHO can realize the potential of operator evolution to further improve success rates on more challenging tasks.


5. Generalization boundaries (W4).

Our experiments support operator transfer between related SMAC tasks and framework applicability across SMAC and Overcooked-AI. Our claim is that ECHO can be instantiated across domains to incorporate coordination knowledge from human interventions and reduce human involvement.

However, coordination requirements differ: SMAC involves adversarial team coordination, whereas Overcooked-AI involves non-adversarial cooperative coordination. Differences in coordination structure require environment-specific grounding, evidence definitions, and parameter settings; broader operator reuse may require higher-level abstractions. We will explicitly distinguish this limitation from the framework's applicability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.