- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2(−1)
- Presentation
- 2
- Soundness
- 2(−1)
Read review
Paper summary
The authors propose ECHO. ECHO studies how a local human intervention can improve coordination across a team and be reused later. It preserves the human command, adjusts the other agents through a collaboration graph, and stores those adjustments as relational operators. Operators gain permission for automatic reuse through execution evidence and human feedback. Experiments on SMAC and Overcooked include a participant study, continual adaptation, and transfer with frozen memory. The distinction between immediate coordination and accumulated memory is important here. The former asks whether teammates respond better to a human command; the latter asks whether similar situations need fewer interventions over time. The paper evaluates both, rather than attributing every improvement to memory. “Training-free” refers to frozen LLM parameters, while expert guidance and operator collection still require effort.
Critical strengths
- The authors discuss clearly the choice of what to remember: how teammates should respond to a protected human action. This is more useful for coordination than storing the command alone.
- The controls are reasonably thorough. The 12-participant comparison improves success without memory (Table 2). In the three-seed evolution study, interactions fall from 29.7 to 13.5 per episode while success stays at 100% on the evaluated map. The confirmation-only and no-safeguard variants help explain why the full design works.
- The ablation results show why this distinction matters in practice: NoSafeguard becomes less successful as memory accumulates, whereas full ECHO maintains performance. Simply collecting more reusable rules is therefore not enough. The frozen-memory transfer test adds useful evidence that operators capture something beyond episode-specific unit identities.
Critical weaknesses
- The practical workload benefit is less clear than the interaction counts suggest. Inference pauses the environment, and automatic reuse has a 15-second veto window. Fewer clicks may still require substantial monitoring. The NASA-TLX results help, but do not establish how much time the system saves.
- Transfer is tested on only three related map pairs. The harmful automatic applications is encouraging, but repeated uses share operators and episodes. It is therefore limited evidence for reliability beyond these settings. There is also a difference between reducing intervention frequency and solving the harder tasks. On the difficult transfer targets, success remains around one third (Table 4). The benefit is therefore mainly lower human involvement at comparable or modestly improved performance, rather than consistently successful autonomous coordination.
- The novelty lies in a fairly specific representation, and its advantage over prior procedural memory is not fully isolated. Existing works already induces reusable routines from experience and retrieves them for later tasks. From my view ECHO's more specific contribution is the anchor-conditioned responder-side graph rewrite, together with its reuse gate. TextRAG removes structure and gating together, while IDSpecific deliberately retains instance-bound selectors. Neither establishes that graph-difference memory is better than a transferable procedure or rule representation given the same relational selectors, feedback, and authorization mechanism. The current evidence supports the integrated system more clearly than it establishes a new memory principle.
- I have carefully read Reviewer jWhw’s comments and the authors’ rebuttal. The explanations are reasonable, and I appreciate the corrections, but the errors across multiple tables still reduce my confidence in the experimental results. Even if the overall trends remain unchanged, the number and range of corrections raise concerns about the reliability of the data aggregation and reporting process.
Questions for authors
- What can an anchor-conditioned graph-difference operator express or transfer that a relational procedure/rule memory cannot? Could the authors compare these representations while keeping anchor protection, relational selectors, feedback, and reuse gating fixed? Table 11 tests identifier abstraction, but does not answer this representation question.
- How much wall-clock and active supervision time is saved over Static and ConfirmOnly, including veto windows and memory collection? How was the Static versus evolved-ECHO workload comparison controlled for order and prior exposure?
- How many distinct operators and episodes account for the 786 automatic applications, and does uncertainty assessed at that level support the same conclusion?
6 comments
2. Time savings and workload controls (Q2/W1).
[Supplementary Experiments] We compared wall-clock time for evolved ECHO, ECHO-Static, and ECHO-ConfirmOnly on 3s_vs_3z, 3s_vs_4z, and 2c_vs_64zg, measuring from episode start to termination and including inference pauses, confirmation waits, and applicable veto windows. ECHO collects operator evidence during ordinary interactions, including when users accept or reject Shadow proposals, so evidence collection takes no additional user time.
| Method | 3s_vs_3z | 3s_vs_4z | 2c_vs_64zg | Overall |
|---|---|---|---|---|
| ECHO-Static | 793.87 ± 22.54 | 1255.60 ± 82.41 | 652.80 ± 41.49 | 900.76 ± 265.80 |
| ECHO-ConfirmOnly | 752.60 ± 73.81 | 1307.67 ± 63.70 | 619.87 ± 28.11 | 893.38 ± 306.67 |
| ECHO | 736.33 ± 58.65 | 1312.27 ± 65.18 | 614.20 ± 26.83 | 887.60 ± 312.15 |
(Episode wall-clock time (s, mean ± SD), each method was evaluated over 15 episodes per map.)
Under the same stable network and model API conditions, ECHO averaged 13.16 s less per episode than ECHO-Static and 5.78 s less than ECHO-ConfirmOnly across the three maps. ECHO was faster than both baselines on two maps and slower on 3s_vs_4z. All three methods draw on the same underlying coordination knowledge, which can lead to similar task progression and comparable episode durations. However, in some 3s_vs_3z runs, we observed that users in ECHO-Static had to work out how to adjust team coordination themselves and sometimes made redundant interventions. ECHO reused a learned coordination structure, avoided those extra interventions, and completed the task sooner.
[Active Supervision Time and Mental Effort] In all three conditions, users are expected to follow the task from start to termination, so we use each episode's wall-clock time as the duration of required monitoring. Comparable monitoring durations can still involve different levels of mental effort. For example,
-
With ECHO-Static, users must continually assess whether the current coordination is adequate. When they spot a problem, they must decide which unit to intervene on and how to change its intent to improve team coordination.
-
With ECHO, autonomous planning and operator reuse handle routine coordination, allowing users to focus on following task progress for much of the episode. Users step in when coordination needs correction or a proposed adjustment needs review.
In the latter case, users primarily judge whether the proposed adjustment is appropriate. Human intervention counts capture how often users step in, and NASA-TLX, including its temporal demand subscale, captures how demanding they find the task. Together, these measures characterize the workload within the time users spend monitoring.
[Condition Order and Prior Exposure] After standardized training, all 12 participants completed ECHO first, followed by the ECHO-Direct and ECHO-Static block under the protocol described in the manuscript. Prior exposure could have helped participants perform the later conditions through greater familiarity with the task. Participants nevertheless reported lower workload for ECHO than for Static. This result supports a perceived workload benefit for ECHO under the tested protocol. We agree that the fixed order of evolved ECHO and Static could have influenced the ratings, and we will acknowledge this limitation in the manuscript.
3. Distinct operators, episodes, and uncertainty (Q3/W2).
[Supplementary Data] Because repeated applications can share operators and episode histories, we counted the distinct operators and episodes underlying the 786 automatic applications in Table 15. These applications involved 20 distinct operators across 90 episodes (30 per transfer pair). The first two pairs used 15 distinct operators from their shared frozen source memory after deduplication, and the third pair contributed five additional operators.
| Source → Target task | Auto apps. | Distinct operators | Episodes | Apps. per operator |
|---|---|---|---|---|
3s_vs_4z→3s_vs_3z | 225 | 13 | 30/30 | 17 [8–31] |
3s_vs_4z→3s_vs_5z | 378 | 15 | 30/30 | 25 [14–42] |
2s3z→3s5z | 183 | 5 | 30/30 | 35 [29–48] |
| Total | 786 | 20 (15+5) | 90/90 | 36.5 [22, 73] |
("Apps. per operator" is reported as median [min–max].)
-
Applications of 13 operators on
3s_vs_3z:validated operators: 31,16,23,17,19,22,15,21,20;
auto-ready operators: 8,11,12,10;
-
Applications of 15 operators on
3s_vs_5z:validated operators: 42,19,31,25,37,28,22,20,18,30,34;
auto-ready operators: 14,17,15,26;
-
Applications of 5 operators on
3s5z:validated operators: 48,40,31;
auto-ready operators: 35,29;
| Source → Target task | Auto apps. | Distinct operators | Episodes | Apps. per operator |
|---|---|---|---|---|
3s_vs_4z→3s_vs_4z | 134 | 10 | 10/10 | 6 [4, 53] |
-
Applications of 10 operators on
3s_vs_4z:validated operators: 34,7,9,5,53,5;
auto-ready operators: 4,5,4,8;
[Episode-Level Uncertainty] We observed no harmful outcomes in 920 (786+134) automatic applications across 100 episodes (90 transfer + 10 nomal) and 20 distinct operators. We classified an episode as harmful if it contained at least one harmful automatic application. The nominal Wilson 95% interval for the harmful-episode rate was 0–4.1% (0/90) on the transfer tasks. Ten additional episodes on the source map, 3s_vs_4z, also had no harmful automatic applications. Pooling the source and transfer episodes gave a nominal interval of 0–3.7% (0/100), corresponding to 96.3%–100% for the proportion of harm-free episodes under the evaluated conditions.
[Implications for Our Reliability Claims] The expanded evaluation supports our original conclusion: we observed no harmful outcomes in 920 automatic applications of 20 distinct operators across 100 episodes. Because applications within an episode can be correlated, we use the episode as the unit of analysis. The Wilson 95% confidence interval for the harmful-episode rate was 0–3.7%, corresponding to a 96.3% lower confidence bound on the proportion of episodes free of harmful automatic applications under the evaluated conditions.
4. Transfer scope and task performance (W2).
Complementary coordination can improve task success. On 3s_vs_5z, adding complementary coordination to human input increased the observed success rate from 16.7% with ECHO-Direct to 30.0% with ECHO-Static, with operator memory disabled in both conditions (Section 4.2).
Operator evolution is designed to reduce repeated human involvement while retaining task performance, not to achieve further substantial gains in success rates. On the two difficult transfer targets in Table 4, reusing evolved operators reduced human interactions relative to ECHO-Static, while success remained around one third. The primary benefit of evolution on these targets was therefore reduced human involvement at comparable or modestly improved performance.
We will clarify this distinction in Section 4.4 and emphasize that reliable autonomous completion on difficult transfer targets remains an open challenge. ECHO provides a promising foundation for investigating how operator evolution could further improve success rates on these targets.
We thank the reviewer for the constructive feedback. (1) We clarify ECHO’s contribution and compare its graph-difference operators with equivalent relational rules. (2) We also report wall-clock measurements, explain how we assessed supervision and workload, and (3) add an episode-level uncertainty analysis of harmful automatic applications. (4) Finally, we clarify the scope of our transfer results and distinguish reductions in human involvement from gains in task success.
1. Comparison with relational procedural memory (Q1/W3).
[Core Contribution] ECHO's core contribution is the kind of coordination knowledge it extracts: how the rest of the team should adjust its current plan in response to a protected human intervention. For the same environment state,
Graph differences provide one way to encode that knowledge, and sufficiently expressive relational rules can encode it as well.
[Supplementary Experiments] Table 11 compares relational and instance-specific selectors. To address the encoding question directly, we added a controlled comparison between ECHO's graph-difference operators and relational rules converted from the same operators. Both conditions use the same learned coordination adjustments, anchor protection, relational selectors, feedback, and reuse gating. We froze the source memory and evaluated both encodings on the same held-out transfer tasks.
3s_vs_4z→3s_vs_3z3s_vs_4z→3s_vs_3z3s_vs_4z→3s_vs_5z3s_vs_4z→3s_vs_5z2s3z→3s5z2s3z→3s5z(SR denotes success rate and HI denotes human interactions. Per-episode counts are reported as mean ± SD.)
Both encodings transferred the same learned coordination adjustments to held-out tasks, with similar success rates and no consistent advantage for either encoding.
[Why We Use Graph Encoding] We use graph encoding because typed edges make inter-agent dependencies explicit and graph differences record only the assignments and relationships that change. We therefore expect operators to remain compact as team size grows when interventions affect only a few units. In our 3s_vs_4z source memory, converting the 96 retained operators and their variants yielded 3,162 rule instructions. Prior work (Fatemi et al., ICLR 2024) also shows that graph encoding choices can affect LLM performance on relational reasoning tasks.