- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AECG, a memory governance framework for LLM-based multi-agent systems. AECG preserves the coordination scope of stored experience, uses asymmetric evidence updates and dual-timescale reliability estimates to detect degraded skills, and prioritizes reviews according to downstream exposure. Revised skills are activated only after replay-based validation. Experiments across three multi-agent frameworks and four benchmarks show that AECG achieves the best performance in 11 of 12 settings, demonstrating the benefits of scope preservation and continuous memory governance.
Critical strengths
-
AECG provides a complete memory governance process for multi-agent systems. It preserves the scope of stored experience, detects degraded skills, prioritizes risky skills, and validates revisions before reuse. This helps reduce memory pollution and unsafe knowledge transfer.
-
The experiments are broad and support the main design choices. AECG is tested on three multi-agent frameworks and four benchmarks, achieving the best result in 11 of 12 settings. The ablation studies also show that its main components improve performance.
Critical weaknesses
-
The treatment of positive and negative evidence is asymmetric. The paper emphasizes that spuriously successful but unreliable procedures may enter long-term memory. However, an applied skill can receive weighted positive evidence whenever the task succeeds, whereas negative evidence requires agreement between two LLM judges. Although credit sharing and promotion thresholds mitigate this issue, the method still cannot determine whether a skill actually contributed to success and therefore does not fully address the problem of spurious success highlighted in the motivation.
-
The failure-attribution mechanism lacks independent validation. Agreement between two LLM judges does not necessarily imply correct attribution, particularly when their errors may be correlated. The paper does not evaluate attribution accuracy, recall, or confidence calibration using human-annotated cases, even though these decisions directly affect reliability updates, skill repair, and isolation. This weakens confidence in the framework’s core governance mechanism.
Questions for authors
-
How does AECG prevent an irrelevant or harmful skill from receiving positive evidence simply because the overall task succeeds? Did the authors consider positive attribution or counterfactual replay?
-
Why does AECG underperform ConMem on MacNet–PDDL? Could the authors provide failure cases or analyze whether strict scope matching limits useful knowledge reuse?
-
What is the additional computational cost of dual-LLM attribution, skill review, revision, and paired replay in terms of model calls, tokens, and runtime?
-
Has the dual-LLM failure-attribution mechanism been evaluated against human-annotated cases? If so, what are its accuracy, recall, and confidence-calibration results?
7 comments
(Response to Q2) ConMem’s Advantage and the Effect of Scope Matching on MacNet–PDDL
Thank you for asking why ConMem performs better on MacNet–PDDL and whether scope matching limits useful reuse. On MacNet–PDDL, ConMem achieved a mean success rate of 54.12% across three seeds, 2.56 percentage points above AECG’s 51.56%. ConMem is particularly well suited to this setting because it can recover action dependencies during MacNet’s plan handoffs. MacNet passes revised plans from node to node along a DAG. Preconditions and ordering constraints raised in upstream discussions need to be written into the plan to reach downstream nodes. In PDDL, omitting a step that establishes a precondition can make a later action inapplicable. Within a PDDL domain, fixed operators define action preconditions and state transitions, so dependencies learned from past tasks can be reused across instances. ConMem’s graph of memory cards can supply a downstream node with both the current action and the steps that establish its preconditions, helping the node complete the plan while accounting for ordering constraints and conflicts.
On the same PDDL benchmark, AutoGen exchanges revisions through multi-turn dialogue, and AgentNet aggregates subtask results through dynamic routing. Relative to each framework’s no-memory baseline, ConMem improved success by 8.81 percentage points on MacNet, 3.71 on AutoGen, and 4.75 on AgentNet. AECG outperformed ConMem on AutoGen and AgentNet by 6.44 and 4.63 percentage points, respectively. Within MacNet, FEVER, ScienceWorld, and MBPP-Plus require skills to adapt to instance-specific conditions such as the evidence for the current claim, the current environment state, and the current function specification. AECG outperformed ConMem on all three benchmarks and in 11 of the 12 framework–benchmark combinations overall.
To assess the effect of scope matching on MacNet–PDDL, we compared full AECG with w/o Scope on the same 60 tasks.
Table 4. Cross-Scope Candidates Blocked
| Framework–benchmark | Total retrievals | Retrievals with blocked candidates | Share of retrievals affected | Blocked candidate instances |
|---|---|---|---|---|
| MacNet–PDDL | 243 | 57 | 23.5% | 86 |
We counted memories that would have entered the candidate set with scope filtering disabled. A single retrieval could block multiple candidates.
Table 5. Task Outcomes With and Without Scope Matching
| With scope matching | Without scope matching | Tasks |
|---|---|---|
| Correct | Correct | 22 |
| Correct | Incorrect | 10 |
| Incorrect | Correct | 2 |
| Incorrect | Incorrect | 26 |
In MacNet’s node-by-node plan revision, a stored skill can assume the responsibilities and preceding steps associated with its original position in the workflow. Scope matching limits reuse to compatible collaboration contexts, reducing the chance that a downstream node adopts assumptions from another position in the plan. Disabling scope matching recovered 2 of 60 tasks and caused 10 previously correct tasks to fail. Scope matching therefore yielded 8 more correct solutions, raising success from 40.0% (24/60) to 53.3% (32/60), a gain of 13.3 percentage points on this paired evaluation.
(Response to Q3) Governance Cost and Task Performance
Thank you for asking how much computation each governance component adds. Skill revision preserves experience that can still be reused. When degradation affects only one instruction or a subset of the conditions under which a skill applies, disabling the entire skill also removes procedures that remain effective in other settings. AECG can correct the faulty instruction or narrow the skill’s applicability so that later tasks can still benefit from that experience.
Paired replay checks whether a revised skill is ready to be reactivated. A revision can resolve the failure that prompted it while breaking a previously successful use. Because later tasks can repeatedly retrieve the same version, a regression introduced by one revision can affect multiple runs. AECG therefore replays a historical failure to check whether the original problem is resolved and a historical success to check whether the skill preserves its previously effective behavior. AECG incurs this validation cost before reactivation, and later tasks can reuse the validated version.
On the same AgentNet–SciWorld evaluation tasks, we compared full AECG, AECG without governance, and two variants that separately removed skill revision and paired replay.
Table 6. Governance Cost and Accuracy on AgentNet–SciWorld
| Setting or component | Accuracy (%) | Triggers | Model API calls | Input tokens | Output tokens | Runtime (s) |
|---|---|---|---|---|---|---|
| AECG w/o Governance | 54.71 | — | 3,954 | 7,105,728 | 527,133 | 18,285 |
| Full AECG | 68.18 | — | 4,691 | 8,671,893 | 631,389 | 21,704 |
| Change with governance enabled | +13.47 pp | — | +18.64% | +22.04% | +19.78% | +18.70% |
| Dual-judge failure attribution | — | — | 124 | 366,803 | 29,850 | 672 |
| Skill review | — | 7 | — | — | — | 8 |
| Skill revision | 64.37 (without revision) | 8 | 51 | 56,189 | 7,484 | 499 |
| Paired replay | 61.22 (without replay) | 14 | 562 | 1,143,175 | 66,922 | 2,240 |
Removing skill revision reduced accuracy by 3.81 percentage points, showing the benefit of recovering experience that later tasks can reuse. Paired replay accounted for 76.3% of governance model calls. Removing replay reduced accuracy by 6.96 percentage points, eliminating about half of the accuracy gain from full governance. After accounting for the additional computation, full AECG completed about 5.0% more tasks correctly both per hour and per 1,000 model calls than AECG without governance. On this task set, the full governance pipeline improved both accuracy and the number of correct solutions per unit of runtime and per model call.
(Response to W1 & Q1) Testing Misassigned Positive Credit with Counterfactual Replay
Thank you for raising the concern that a successful task can give positive credit to a skill that hindered it. AECG distributes task-level credit across the skills used in the run. Before promotion, a candidate memory must pass seed screening by exceeding the mean Ready score of active memories in the same scope by a preset margin. We used counterfactual replay to assess whether memories that received credit helped the task, then recomputed seed screening to test whether a single successful run could change the screening decision.
From each of 12 successful trajectories, we selected one target memory that had received positive evidence. We ran paired replays from the same state, retaining the target in one replay and removing it in the other.
Table 1. Counterfactual Replay Outcomes
We reviewed the four trajectories with different replay outcomes. Three target memories contributed to task success. The fourth hindered the task but received positive evidence in the original successful run. We recomputed seed screening from the same snapshot taken before the positive observation was recorded. Let Δ be the candidate memory’s Ready score minus the mean Ready score of the same-scope reference pool. For AGENT-scope memories, the reference pool also matches the agent’s identity. Standard AECG requires Δ>0.05, while the zero-margin control requires Δ>0. The observation-removal control omits only the target memory’s positive observation from that run and updates all other memories as usual.
Table 2. Seed Screening Pass Counts
Table 3. Ready Margins and Screening Outcomes by Case
Case D illustrates the reviewer’s concern. The replay succeeded only when D was removed, yet D received positive credit in the original successful run. That credit raised its Δ from −0.01 to +0.03, enough to pass screening under the zero-margin control. AECG’s 0.05 margin kept D from qualifying as a seed for promotion while retaining both contributing memories that passed under the zero-margin control. In Case B, the positive observation raised a contributing memory’s Δ from +0.02 to +0.06, allowing it to pass standard screening. Together, these cases show how task-level credit helped a contributing memory pass screening while the Ready margin prevented the observed instance of misassigned credit from carrying a detrimental memory past the promotion screen.