acceptodds
Under review as a conference paper at ICLR 2027

AECG: Asymmetric Experience Consolidation and Governance In Multi-Agent Systems

Abstract

Large language model (LLM)-based multi-agent systems increasingly rely on memory to transform execution trajectories into reusable procedural knowledge. Yet repeated retrieval also makes memory errors persistent: memory pollution arises when outdated, weakly supported, or spuriously successful procedures become recurring components of future reasoning. Multi-agent execution introduces an additional structural risk. Scope collapse occurs when procedural knowledge escapes the coordination scope in which it was shown effective and is repeatedly reused at incompatible decision levels, allowing local errors to influence cascades of downstream decisions. Meanwhile, task-level failures provide ambiguous supervision because they rarely reveal which recalled knowledge was responsible. We introduce AECG, a framework for asymmetric experience consolidation and governance formulti-agent systems. AECG turns memory from static experience storage into a dynamic reliability-governance loop, preserving coordination scope and using multi-scale, confidence-aware reliability to detect degradation. It then combines degradation with downstream impact to prioritize high-risk knowledge under a bounded review budget, applies targeted interventions, and reactivates revised skills only after paired replay. Across three multi-agent frameworks and four benchmarks, AECG achieves the best score in 11 of 12 framework–benchmark settings and improves over the strongest competing memory method by as much as 10.23 percentage points; removing scope preservation reduces accuracy by up to 16.89 points. AECG thereby reframes multi-agent memory from passive accumulation into auditable reliability governance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory, Multi-Agent, Long-Horzon Agents, Memory Governance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DZ7vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AECG, a memory governance framework for LLM-based multi-agent systems. AECG preserves the coordination scope of stored experience, uses asymmetric evidence updates and dual-timescale reliability estimates to detect degraded skills, and prioritizes reviews according to downstream exposure. Revised skills are activated only after replay-based validation. Experiments across three multi-agent frameworks and four benchmarks show that AECG achieves the best performance in 11 of 12 settings, demonstrating the benefits of scope preservation and continuous memory governance.

Critical strengths
  1. AECG provides a complete memory governance process for multi-agent systems. It preserves the scope of stored experience, detects degraded skills, prioritizes risky skills, and validates revisions before reuse. This helps reduce memory pollution and unsafe knowledge transfer.

  2. The experiments are broad and support the main design choices. AECG is tested on three multi-agent frameworks and four benchmarks, achieving the best result in 11 of 12 settings. The ablation studies also show that its main components improve performance.

Critical weaknesses
  1. The treatment of positive and negative evidence is asymmetric. The paper emphasizes that spuriously successful but unreliable procedures may enter long-term memory. However, an applied skill can receive weighted positive evidence whenever the task succeeds, whereas negative evidence requires agreement between two LLM judges. Although credit sharing and promotion thresholds mitigate this issue, the method still cannot determine whether a skill actually contributed to success and therefore does not fully address the problem of spurious success highlighted in the motivation.

  2. The failure-attribution mechanism lacks independent validation. Agreement between two LLM judges does not necessarily imply correct attribution, particularly when their errors may be correlated. The paper does not evaluate attribution accuracy, recall, or confidence calibration using human-annotated cases, even though these decisions directly affect reliability updates, skill repair, and isolation. This weakens confidence in the framework’s core governance mechanism.

Questions for authors
  1. How does AECG prevent an irrelevant or harmful skill from receiving positive evidence simply because the overall task succeeds? Did the authors consider positive attribution or counterfactual replay?

  2. Why does AECG underperform ConMem on MacNet–PDDL? Could the authors provide failure cases or analyze whether strict scope matching limits useful knowledge reuse?

  3. What is the additional computational cost of dual-LLM attribution, skill review, revision, and paired replay in terms of model calls, tokens, and runtime?

  4. Has the dual-LLM failure-attribution mechanism been evaluated against human-annotated cases? If so, what are its accuracy, recall, and confidence-calibration results?

7 comments

(Response to W1 & Q1) Testing Misassigned Positive Credit with Counterfactual Replay

Thank you for raising the concern that a successful task can give positive credit to a skill that hindered it. AECG distributes task-level credit across the skills used in the run. Before promotion, a candidate memory must pass seed screening by exceeding the mean Ready score of active memories in the same scope by a preset margin. We used counterfactual replay to assess whether memories that received credit helped the task, then recomputed seed screening to test whether a single successful run could change the screening decision.

From each of 12 successful trajectories, we selected one target memory that had received positive evidence. We ran paired replays from the same state, retaining the target in one replay and removing it in the other.

Table 1. Counterfactual Replay Outcomes

Paired replay outcomeTarget retainedTarget removedTrajectories
Both succeededSuccessSuccess6/12 (50.0%)
Only retention succeededSuccessFailure3/12 (25.0%)
Only removal succeededFailureSuccess1/12 (8.3%)
Both failedFailureFailure2/12 (16.7%)

We reviewed the four trajectories with different replay outcomes. Three target memories contributed to task success. The fourth hindered the task but received positive evidence in the original successful run. We recomputed seed screening from the same snapshot taken before the positive observation was recorded. Let Δ\Delta be the candidate memory’s Ready score minus the mean Ready score of the same-scope reference pool. For AGENT-scope memories, the reference pool also matches the agent’s identity. Standard AECG requires Δ>0.05\Delta > 0.05, while the zero-margin control requires Δ>0\Delta > 0. The observation-removal control omits only the target memory’s positive observation from that run and updates all other memories as usual.

Table 2. Seed Screening Pass Counts

Replay assessmentStandard AECGPositive observation removedPositive observation retained, zero margin
Contributed to success2/31/32/3
Hindered the task but received positive evidence0/10/11/1

Table 3. Ready Margins and Screening Outcomes by Case

CaseReplay assessmentΔ\Delta with observationΔ\Delta without observationStandard AECGObservation removedZero margin
AContributed to success+0.08+0.06PassPassPass
BContributed to success+0.06+0.02PassFailPass
CContributed to success−0.01−0.03FailFailFail
DHindered the task+0.03−0.01FailFailPass

Case D illustrates the reviewer’s concern. The replay succeeded only when D was removed, yet D received positive credit in the original successful run. That credit raised its Δ\Delta from −0.01 to +0.03, enough to pass screening under the zero-margin control. AECG’s 0.05 margin kept D from qualifying as a seed for promotion while retaining both contributing memories that passed under the zero-margin control. In Case B, the positive observation raised a contributing memory’s Δ\Delta from +0.02 to +0.06, allowing it to pass standard screening. Together, these cases show how task-level credit helped a contributing memory pass screening while the Ready margin prevented the observed instance of misassigned credit from carrying a detrimental memory past the promotion screen.

(Response to Q2) ConMem’s Advantage and the Effect of Scope Matching on MacNet–PDDL

Thank you for asking why ConMem performs better on MacNet–PDDL and whether scope matching limits useful reuse. On MacNet–PDDL, ConMem achieved a mean success rate of 54.12% across three seeds, 2.56 percentage points above AECG’s 51.56%. ConMem is particularly well suited to this setting because it can recover action dependencies during MacNet’s plan handoffs. MacNet passes revised plans from node to node along a DAG. Preconditions and ordering constraints raised in upstream discussions need to be written into the plan to reach downstream nodes. In PDDL, omitting a step that establishes a precondition can make a later action inapplicable. Within a PDDL domain, fixed operators define action preconditions and state transitions, so dependencies learned from past tasks can be reused across instances. ConMem’s graph of memory cards can supply a downstream node with both the current action and the steps that establish its preconditions, helping the node complete the plan while accounting for ordering constraints and conflicts.

On the same PDDL benchmark, AutoGen exchanges revisions through multi-turn dialogue, and AgentNet aggregates subtask results through dynamic routing. Relative to each framework’s no-memory baseline, ConMem improved success by 8.81 percentage points on MacNet, 3.71 on AutoGen, and 4.75 on AgentNet. AECG outperformed ConMem on AutoGen and AgentNet by 6.44 and 4.63 percentage points, respectively. Within MacNet, FEVER, ScienceWorld, and MBPP-Plus require skills to adapt to instance-specific conditions such as the evidence for the current claim, the current environment state, and the current function specification. AECG outperformed ConMem on all three benchmarks and in 11 of the 12 framework–benchmark combinations overall.

To assess the effect of scope matching on MacNet–PDDL, we compared full AECG with w/o Scope on the same 60 tasks.

Table 4. Cross-Scope Candidates Blocked

Framework–benchmarkTotal retrievalsRetrievals with blocked candidatesShare of retrievals affectedBlocked candidate instances
MacNet–PDDL2435723.5%86

We counted memories that would have entered the candidate set with scope filtering disabled. A single retrieval could block multiple candidates.

Table 5. Task Outcomes With and Without Scope Matching

With scope matchingWithout scope matchingTasks
CorrectCorrect22
CorrectIncorrect10
IncorrectCorrect2
IncorrectIncorrect26

In MacNet’s node-by-node plan revision, a stored skill can assume the responsibilities and preceding steps associated with its original position in the workflow. Scope matching limits reuse to compatible collaboration contexts, reducing the chance that a downstream node adopts assumptions from another position in the plan. Disabling scope matching recovered 2 of 60 tasks and caused 10 previously correct tasks to fail. Scope matching therefore yielded 8 more correct solutions, raising success from 40.0% (24/60) to 53.3% (32/60), a gain of 13.3 percentage points on this paired evaluation.

(Response to Q3) Governance Cost and Task Performance

Thank you for asking how much computation each governance component adds. Skill revision preserves experience that can still be reused. When degradation affects only one instruction or a subset of the conditions under which a skill applies, disabling the entire skill also removes procedures that remain effective in other settings. AECG can correct the faulty instruction or narrow the skill’s applicability so that later tasks can still benefit from that experience.

Paired replay checks whether a revised skill is ready to be reactivated. A revision can resolve the failure that prompted it while breaking a previously successful use. Because later tasks can repeatedly retrieve the same version, a regression introduced by one revision can affect multiple runs. AECG therefore replays a historical failure to check whether the original problem is resolved and a historical success to check whether the skill preserves its previously effective behavior. AECG incurs this validation cost before reactivation, and later tasks can reuse the validated version.

On the same AgentNet–SciWorld evaluation tasks, we compared full AECG, AECG without governance, and two variants that separately removed skill revision and paired replay.

Table 6. Governance Cost and Accuracy on AgentNet–SciWorld

Setting or componentAccuracy (%)TriggersModel API callsInput tokensOutput tokensRuntime (s)
AECG w/o Governance54.71—3,9547,105,728527,13318,285
Full AECG68.18—4,6918,671,893631,38921,704
Change with governance enabled+13.47 pp—+18.64%+22.04%+19.78%+18.70%
Dual-judge failure attribution——124366,80329,850672
Skill review—7———8
Skill revision64.37 (without revision)85156,1897,484499
Paired replay61.22 (without replay)145621,143,17566,9222,240

Removing skill revision reduced accuracy by 3.81 percentage points, showing the benefit of recovering experience that later tasks can reuse. Paired replay accounted for 76.3% of governance model calls. Removing replay reduced accuracy by 6.96 percentage points, eliminating about half of the accuracy gain from full governance. After accounting for the additional computation, full AECG completed about 5.0% more tasks correctly both per hour and per 1,000 model calls than AECG without governance. On this task set, the full governance pipeline improved both accuracy and the number of correct solutions per unit of runtime and per model call.

Reviewer_r86mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the continuous management of procedural memory in multi-agent systems. AECG retains three types of coordination scopes - TEAM, EVENT, and AGENT - and restricts the reuse across scopes. Successful tasks provide weighted positive evidence, while failure only results in negative evidence after unanimous attribution and confidence threshold by two LLM judges. The system uses dual-timescale reliability beliefs to detect skill degradation and combines usage frequency and the downstream reachable range discounted by distance in the execution graph to determine the review priority through downstream exposure. Based on the failure evidence, the system proposes scope contraction (NARROW) or process revision (REPAIR), and then decides whether to enable the new version through paired replay of failure cases and historical successful cases. The experiments cover three multi-agent frameworks and four benchmarks. The authors report that AECG achieved the best results in eleven out of twelve combinations of frameworks and benchmarks, and analyzed the effects of coordination scope, dual-timescale reliability estimation, and review prioritization through ablation and controlled corruption experiments.

Critical strengths

S1. The research question has a good correspondence with the collaborative structure of multi-agent systems. This paper distinguishes the scope of experience usage in team planning, collaboration stages, and role execution, and retains these scopes during skill abstraction and retrieval. The method does not directly consider a single task failure as a negative evidence for all the used skills, but instead, through the consistent attribution of the double judge and the control of the confidence threshold, it regulates the acceptance of evidence. These designs directly address the issues of experience misuse and failure attribution in multi-agent systems.

S2. The experiment covered multiple frameworks and implemented reasonable controls for some mechanisms. The main experiment covered three frameworks and four benchmarks, and the results were of a certain breadth. The controlled contamination and review sorting experiments used shared warm-up memory snapshots and maintained the same task sequence and execution conditions. The sorting experiments also uniformly adopted isolated operations to fix the intervention method. These controls helped to compare the effects of different detection and sorting strategies.

Critical weaknesses

W1. The actual incremental improvement of the proposed method compared to the existing skill management work is not yet clear. The existing work discussed in the paper already includes experience stratification, reliability maintenance, and revision based on feedback on success and failure, but the main experiment did not directly compare with methods such as LEGOMem, MACLA, and MSCE. AECG combines the scope constraint with mechanisms such as dual time-scale estimation and downstream impact ranking. The existing results show the overall performance and the role of some components, but have not fully explained what substantial improvements this combination has brought compared to the aforementioned related methods.

W2. The actual value of the complete skill update process lacks direct verification. The paper not only detects unreliable skills but also revises and validates them for re-use. However, the existing ablation studies mainly analyze the scope of effect, reliability estimation, and review ranking. The ranking experiments uniformly deactivate the selected skills. These results demonstrate the overall performance and the role of individual components, but do not directly compare the effects of continuing to use the revised skills versus directly deactivating the problematic skills.

Questions for authors

Q1. Compared to the closest skill management method, what are the main improvements of AECG? Could you provide a representative direct comparison to illustrate the actual benefits of the new design?

Q2. Compared to simply discontinuing the problematic skills, does the comprehensive governance process that includes skill revision and replay verification lead to better task performance? Can corresponding comparative results be provided?

5 comments

(Response to W1 & Q1) AECG’s Contributions and Comparisons with Existing Methods

Thank you for asking what AECG adds to existing skill management methods and how those additions affect task performance. We first summarize the different emphases of AECG, MSCE, MACLA, and LEGOMem.

  • MSCE emphasizes abstraction across levels, progressively transforming execution experience into callable skills.
  • MACLA emphasizes procedure utility, using execution feedback to select and improve procedures for solving tasks.
  • LEGOMem emphasizes role specialization, assigning memories at different levels of granularity to different roles within a team.
  • AECG emphasizes how skill effectiveness changes over time, focusing on how collaborative skills remain usable across repeated reuse and successive revisions.

We describe the agent memory lifecycle in five stages: M1 Formation → M2 Organization → M3 Retrieval and Use → M4 Maintenance → M5 Updating. Established methods for M1 and M3 extract experience from execution trajectories and retrieve relevant memories for new tasks. AECG’s main contributions are in M2, M4, and M5. These stages address how to organize multi-agent experience by collaboration scope, when to review a skill as its reliability changes, and what evidence is required to reactivate a revised skill.

M2 Organization — Preserving the Collaboration Scope of Experience

Existing methods can group experience by semantic similarity, task, or agent role. In a multi-agent system, the same role may handle similar subtasks at different stages, under different initial states, with different outputs from preceding steps and different handoff conditions. If these experiences are merged during abstraction, the resulting skill may be semantically relevant to the current task while relying on conditions that hold only at another stage.

AECG represents team-level, collaboration-stage, and individual scopes as TEAM, EVENT, and AGENT, respectively. A skill inherits its source scope when promoted. EVENT lets agents working on the same stage accumulate shared experience while preserving the applicability boundaries of experience that an individual agent gains at different stages. The same collaboration scope is thus preserved from experience formation through skill reuse.

Table 1. Cross-Scope Candidates Blocked by Scope Matching

Framework–benchmarkTotal retrievalsRetrievals with blocked candidatesRetrieval blocking rateBlocked candidate instances
MacNet–PDDL2435723.5%86

Table 2. Paired Task Outcomes With and Without Scope Matching

With scope matchingWithout scope matchingTasks
CorrectCorrect22
CorrectIncorrect10
IncorrectCorrect2
IncorrectIncorrect26

Disabling scope matching yielded 2 additional correct solutions and caused 10 previously correct tasks to fail. Across the 60 MacNet–PDDL tasks, retaining scope matching increased the number of correct solutions from 24 to 32, a gain of 13.33 percentage points. Collaboration scope provides applicability information beyond semantic similarity, helping the system preserve successful execution paths when reusing skills.

M4 Maintenance — Detecting Changes in Skill Reliability

Existing methods can estimate skill reliability from past outcomes, such as MACLA’s Bayesian posterior and MSCE’s smoothed skill invocation success rate. In multi-agent tasks, several decisions jointly determine the final outcome. Attributing a task failure to every skill used can cause the system to stop reusing skills that remain effective. Even when failure attribution is accurate, a long record of success can mask a sustained decline in recent performance.

AECG makes two distinct decisions: whether to accept evidence and whether that evidence warrants review. It records weighted positive evidence for skills used in successful runs. For failures, it records negative evidence only when both judge models agree that the same skill materially contributed to the failure and the attribution meets the confidence threshold. The fast timescale then tracks recent performance, while the slow timescale provides a historical reference for that skill. A substantial decline relative to its own history can trigger review even when cumulative reliability remains high.

The 18 failed trajectories contained 45 candidate trajectory–skill pairs. Human reviewers judged that the skill materially contributed to the failure in 18 pairs, did not materially contribute in 24, and had insufficient evidence for a decision in 3. We computed precision and recall over pairs with conclusive human judgments and the abstention rate over the 18 failed trajectories.

Table 3. Human Evaluation of Failure Attribution

Attribution policyThresholdTP / FP / FNPrecisionRecallAbstention rate
Single judge A0.7012 / 4 / 675.0%66.7%11.1%
Single judge B0.7011 / 4 / 773.3%61.1%16.7%
Dual-judge agreement0.4013 / 3 / 581.3%72.2%11.1%
Dual-judge agreement (default)0.7012 / 1 / 692.3%66.7%27.8%
Dual-judge agreement0.908 / 0 / 10100%44.4%55.6%

At the same 0.70 threshold, the default dual-judge policy and single judge A each identified 12 correct attributions, while the dual-judge policy reduced false attributions from 4 to 1. This preserves the same number of correct failure signals for reliability tracking and reduces the chance that failures caused by other decisions are counted against the skill.

Table 4. Paired Replay of Skills That Received Positive Evidence

Skill retainedSkill removedTrajectories
SuccessSuccess6/12 (50.0%)
SuccessFailure3/12 (25.0%)
FailureSuccess1/12 (8.3%)
FailureFailure2/12 (16.7%)

Across the 12 replay pairs, 9 replays succeeded with the skill retained and 7 succeeded with it removed. Six trajectories succeeded under both conditions. These outcomes support treating task success as weak positive evidence for reuse, since a skill’s contribution varies across trajectories.

Table 5. Review Delay Under Controlled Degradation

Reliability trackingMean skill uses from first failed use to first review
Cumulative reliability only9.7
AECG (Dual-Time)3.8

Table 6. Task Accuracy Across AgentNet Task Streams

ConfigurationSciWorldMBPP-Plus
Confidence Filtering62.50%72.13%
Fast-Time Only65.91%74.59%
Slow-Time Only63.89%72.73%
AECG (Dual-Time)68.18%77.87%

Table 5 shows that using a skill’s own historical performance as a reference reduced the mean delay to review from 9.7 to 3.8 skill uses. Table 6 evaluates task performance over the full streams. Full AECG outperformed Confidence Filtering by 5.68 percentage points on SciWorld and 5.74 on MBPP-Plus. With conservative attribution and the rest of the governance pipeline held fixed, dual-timescale tracking still outperformed the best single-timescale variant by 2.27 and 3.28 percentage points, respectively. Conservative attribution improves the quality of failure evidence entering the tracker. Dual-timescale tracking uses that evidence to trigger review sooner.

M5 Updating — Validating Revisions Before Reactivation

Existing methods can revise skills using new trajectories or disable skills whose reliability has declined. In multi-agent tasks, changing an instruction or narrowing its applicability can change the state that the next agent receives. A revision can resolve the failure that prompted it while breaking behavior the skill previously supported on other tasks. Disabling the entire skill also removes procedures that remain effective.

AECG uses failure evidence to revise a skill or narrow its applicability, then requires two kinds of replay before reactivation. Replaying the failure that prompted the revision checks whether the known problem is resolved. Replaying a historical success within the same collaboration scope checks whether the revision preserves previously successful behavior. AECG activates the candidate version only after it passes both checks, so later tasks can reuse the validated procedure.

Table 7. Task Accuracy With Skill Disabling or Validated Reactivation

Skill handlingSciWorldMBPP-Plus
Direct skill disabling64.37%75.22%
AECG (revision, paired replay, and reactivation)68.18%77.87%
Accuracy gain+3.81 pp+2.65 pp

Revision followed by replay validation and reactivation outperformed direct disabling on both task streams. This comparison quantifies the benefit of continuing to use an existing skill after repair. When a targeted change resolves the problem, later tasks can reuse the validated procedure without having to relearn it.

Direct Comparison with Related Methods

We compared AECG directly with MSCE and MACLA on the same benchmark–framework combinations. Across three multi-agent frameworks and four benchmarks, AECG achieved the highest accuracy in all 12 combinations.

Table 8. Task Accuracy by Framework and Benchmark (%)

MASMemory methodPDDLFEVERSciWorldMBPP-PlusAvg.
AutoGenMSCE42.6770.4161.1179.5166.48
MACLA44.4472.4557.0280.3366.61
AECG46.6774.4963.3582.7869.79
MacNetMSCE50.9568.7565.9172.5966.31
MACLA51.2270.4163.6476.3067.48
AECG51.5671.4371.5979.5170.70
AgentNetMSCE53.1368.3764.7775.4167.24
MACLA49.3671.4365.9163.1963.63
AECG54.6973.4768.1877.8770.43
Reviewer_tj6HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AECG, a framework for improving the reliability of procedural memory in multi-agent systems. The authors identify two challenges in existing memory systems, namely memory pollution from unreliable or outdated procedures and scope collapse when experiences are reused outside their original coordination scope. AECG addresses these issues through scope-aware memory retrieval, conservative failure attribution, dual-timescale reliability estimation, and exposure-aware review prioritization. It also uses targeted repair and paired replay to validate updated skills before reactivation. Experiments across three multi-agent frameworks and four benchmarks show consistent improvements over existing memory baselines, with additional studies evaluating the contributions of individual components.

Critical strengths
  1. The paper addresses an important problem that has received relatively little attention in agent memory research. Most existing work focuses on how to construct and retrieve memories, while this paper studies how to maintain their reliability as they are repeatedly reused. The discussion of scope collapse is particularly relevant to multi-agent systems.

  2. The proposed framework is well motivated, and the individual components fit together naturally. I especially like the idea of maintaining separate estimates of historical and recent reliability, as well as considering the downstream impact of a skill when deciding which memories to review.

  3. The experiments are fairly comprehensive, covering three multi-agent frameworks and four benchmarks. The method achieves the best results in 11 out of 12 settings, suggesting that the improvements are not limited to a particular agent architecture.

  4. The paper includes useful experiments that directly examine the proposed mechanisms. In particular, the scope ablation shows a substantial performance drop, while the controlled corruption experiments demonstrate that dual-timescale reliability can detect degraded skills earlier. These results help explain why the method works beyond the overall performance improvements.

Critical weaknesses
  1. The evaluation of memory degradation relies heavily on controlled corruption. While these experiments help isolate the proposed mechanisms, it is less clear how frequently similar degradation occurs naturally in long-running multi-agent systems. I would like to see more evidence that the proposed governance mechanisms are useful under naturally occurring failures.

  2. The reliability estimation depends on LLM-based failure attribution, but the accuracy of this attribution process is not sufficiently evaluated. Requiring agreement between two judges may reduce incorrect negative feedback, but it could also miss many genuine failures. There is also a potential imbalance between positive evidence from successful tasks and negative evidence that must pass a stricter attribution process.

  3. Although the paper compares against several memory methods, the comparisons do not fully establish how much of the improvement comes from the proposed reliability-governance framework rather than simpler memory filtering or confidence-tracking strategies. More direct reliability-aware baselines would strengthen the contribution.

  4. The framework introduces additional overhead from failure attribution, memory review, skill repair, and replay. The paper does not fully characterize the trade-off between this additional computation and task performance. It is also unclear whether strict scope matching sometimes prevents useful knowledge transfer across coordination levels.

Questions for authors
  1. How often does memory degradation occur naturally in the evaluated task streams? Can the authors quantify how many governance interventions are triggered without injected corruption and how much these interventions improve performance?

  2. Have the authors evaluated the accuracy of the failure attribution mechanism, particularly its precision, recall, and abstention rate? How sensitive are the results to the judge agreement and confidence thresholds?

  3. Could the authors compare against simpler reliability-aware memory methods, such as confidence-based filtering or single-timescale reliability tracking with conservative failure attribution?

  4. Could the authors provide a more detailed analysis of the additional inference cost introduced by the governance pipeline? Also, how often does strict scope preservation prevent a potentially useful cross-scope memory reuse?

6 comments

(Response to W1 & Q1) Naturally Occurring Skill Degradation and the Benefits of Governance

Thank you for suggesting that we evaluate the governance mechanism on a task stream without injected contamination. We conducted an additional ablation on AgentNet × MBPP+. Both variants ran the same tasks in the same order, and the ablated variant disabled only governance actions. Every 10 tasks, we report the success rate over the preceding 10 tasks and the overall skill degradation rate at that checkpoint.

**Table 1. Task Success and Skill Degradation With and Without Governance

Metric (%) / Tasks completed102030405060708090100
Task success rate (full AECG)30205080708070908080
Task success rate (without governance)30205070807070606060
Skill degradation rate (full AECG)011712101398108
Skill degradation rate (without governance)091115182022262526

Without injected contamination, the skill degradation rate in the variant without governance rose as the run progressed, reaching 25–26% at the final three checkpoints. On the same task stream, full AECG carried out four governance interventions and repaired seven skills, with degradation rates of 8–10% at those checkpoints. This shows that routine operation can produce enough skill degradation to trigger intervention and that AECG can identify and repair degraded skills during the run.

Over tasks 31–70, the two variants had similar success rates (75.0% for full AECG and 72.5% without governance). At the 70-task checkpoint, both variants had a 70% success rate over the preceding 10 tasks, yet skill degradation had already reached 22% without governance, compared with 9% for full AECG. In each of the next three 10-task windows, the success rate without governance dropped to 60%. Full AECG achieved 83.3% over the final 30 tasks, a 23.3-percentage-point advantage. Across all 100 tasks, enabling governance raised the success rate from 57.0% to 65.0%, an improvement of 8.0 percentage points.

Both variants used the same task stream and degradation diagnostics, so the ablation isolates the effect of acting on those diagnostics. The gap in skill degradation emerged before the sustained gap in task success. Over the final 30 tasks, the variant that acted on those diagnostics and repaired skills maintained both lower degradation and higher task success. These results show that, on the evaluated task stream, acting on detected skill degradation reduces the prevalence of degraded skills and sustains task performance later in the run.

(Response to W2 & Q2) Failure Attribution Accuracy

Thank you for pointing out the need for independent validation of failure attribution. Using human judgments as the reference, we evaluated 45 candidate trajectory–skill pairs from 18 failed trajectories. Human reviewers found that the skill materially contributed to the failure in 18 pairs, did not materially contribute in 24 pairs, and could not be assessed conclusively in 3 pairs. We computed precision and recall over the 42 pairs with conclusive human judgments and the abstention rate over the 18 failed trajectories. The dual-judge policy requires both judges to identify the same skill within the same scope and applies the confidence threshold to their mean confidence score.

Table 2. Failure Attribution by Judge Policy and Confidence Threshold

Attribution policyThresholdTP / FP / FNPrecisionRecallAbstention rate
Single judge A0.7012 / 4 / 612/16 (75.0%)12/18 (66.7%)2/18 (11.1%)
Single judge B0.7011 / 4 / 711/15 (73.3%)11/18 (61.1%)3/18 (16.7%)
Dual-judge agreement0.4013 / 3 / 513/16 (81.3%)13/18 (72.2%)2/18 (11.1%)
Dual-judge agreement (default)0.7012 / 1 / 612/13 (92.3%)12/18 (66.7%)5/18 (27.8%)
Dual-judge agreement0.908 / 0 / 108/8 (100%)8/18 (44.4%)10/18 (55.6%)

At the same 0.70 threshold, the dual-judge policy accepted 12 human-confirmed attributions, matching judge A and retaining one more than judge B. It reduced false attributions from four under each single-judge policy to one, raising precision from 75.0% and 73.3% to 92.3%. Compared with dual-judge agreement at 0.40, the default retained nearly as many correct attributions (12 versus 13) and reduced false attributions from three to one. Compared with dual-judge agreement at 0.90, the default retained four additional human-confirmed attributions (12 versus 8). Each false attribution prevented keeps a failure from being recorded as negative reliability evidence against a skill that did not materially contribute to it.

(Response to W2 & Q2) Treatment of Positive and Negative Evidence

AECG handles positive and negative evidence differently because they have different consequences for skills. A negative attribution directly lowers a skill’s reliability and can affect whether it is subsequently repaired or quarantined, so AECG requires stronger evidence before accepting it. Success on a long-horizon task typically depends on several steps, so AECG treats task success as weak evidence and distributes credit across the participating skills. It then compares each candidate memory’s Ready score with those of memories in the same scope to determine whether the candidate passes seed screening.

From each of 12 successful trajectories, we selected one target memory that had received positive evidence. We ran paired replays from the same state, retaining the target memory in one replay and removing it in the other.

Table 3. Paired Replay With and Without the Target Memory

Paired replay outcomeTarget retainedTarget removedTrajectories
Both succeededSuccessSuccess6/12 (50.0%)
Only the retention replay succeededSuccessFailure3/12 (25.0%)
Only the removal replay succeededFailureSuccess1/12 (8.3%)
Both failedFailureFailure2/12 (16.7%)

We reviewed the four trajectories with different replay outcomes. In three cases, the target memory contributed to success. In the fourth, the target memory had a detrimental effect on the task but received positive evidence in the original successful run. We recomputed seed screening from the same snapshot, taken before the positive observation was recorded. Let Δ\Delta be the candidate memory’s Ready score minus the mean Ready score of the reference pool for the same scope. For AGENT-scope memories, the reference pool also matches the agent identity. Standard AECG requires Δ>0.05\Delta > 0.05, and the zero-margin control requires Δ>0\Delta > 0. In the observation-removal control, we removed only the target memory’s positive observation from that run and updated all other memories as usual.

Table 4. Seed Screening by Positive Evidence and Ready Margin

(a) Pass counts by replay assessment

Replay assessmentStandard AECGPositive observation removedPositive observation retained, zero margin
Contributed to success2/31/32/3
Detrimental but received positive evidence0/10/11/1

(b) Ready differences and screening outcomes by case

CaseReplay assessmentΔ\Delta with observationΔ\Delta without observationStandard AECGObservation removedZero margin
AContributed to success+0.08+0.06PassPassPass
BContributed to success+0.06+0.02PassFailPass
CContributed to success−0.01−0.03FailFailFail
DDetrimental+0.03−0.01FailFailPass

In Case B, the positive observation raised Δ\Delta from +0.02 to +0.06, allowing the memory to pass standard screening. Case D also received credit from the original successful run, which raised Δ\Delta from −0.01 to +0.03 and was enough to pass the zero-margin control. AECG’s 0.05 margin screened out Case D while retaining both contributing memories that passed under the zero-margin control. Together, the two experiments show how AECG handles each type of evidence. Dual-judge agreement reduced erroneous negative updates while retaining a similar number of correct attributions. On the positive side, credit helped a contributing memory pass the Ready screening, and the margin filtered out the case of misassigned credit observed in this experiment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal

We thank the reviewers for their constructive feedback. Our rebuttal adds experiments and clarifications addressing AECG’s practical relevance, attribution reliability, methodological contribution, computational cost, and sensitivity to collaboration scope.

  • Effectiveness under natural memory degradation. Without injected contamination, AECG carried out four governance interventions and repaired seven degraded skills. The corresponding ablation demonstrates the performance benefit of these interventions, showing that AECG addresses degradation arising during ordinary memory reuse.

  • Attribution reliability and safeguards against spurious credit. We report attribution precision, recall, abstention rates, and confidence-threshold comparisons. The default dual-judge configuration with a 0.7 threshold reduces false positives while preserving comparable true-positive detection, supporting conservative intervention decisions. Counterfactual replay and follow-up analysis further assess the effects of spurious positive credit on reliability estimates and memory promotion during continued updates.

  • Contribution beyond confidence filtering and existing memory methods. AECG connects outcome attribution, longitudinal reliability assessment, and memory governance. Confidence captures certainty in an individual judgment, whereas reliability accumulates evidence across repeated reuse. Comparisons with confidence-only filtering, single-timescale variants, and additional memory baselines substantiate this distinction and demonstrate the complementary roles of fast and slow reliability tracking.

  • Necessity and computational cost of governance components. Each component addresses a distinct source of recurring failure: attribution identifies the responsible skills, reliability tracking accumulates evidence about their behavior, and governance acts on that evidence to limit repeated errors. Component ablations test these roles, while cost analyses quantify the associated model calls, tokens, and runtime alongside the performance benefits.

  • Role of collaboration scope. Scope ablations and comparisons across benchmark–MAS combinations support conditioning skill reuse on collaboration context. We also explain why ConMem performs better in the specific MacNet–PDDL setting, relating this result to task and collaboration structure and clarifying the broader value of scope-aware reuse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.