- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies memory-induced overconfidence in long-running multi-agent systems, where historical memories may strongly and consistently guide agent behavior despite having low reliability in previous tasks. To address this problem, the authors propose C2Mem, which separates memory influence into memory commitment and memory competence. Commitment measures the strength of memory guidance based on retrieval support and structural consistency in a behavioral tree, while competence estimates node-level reliability from historical successes and failures. When C2Mem detects memories with high commitment but low competence, it reduces their weights before injecting them into the agents’ contexts. Historical experiences are summarized by an LLM as reusable behavioral patterns and organized in a hierarchical behavioral tree. After each task, node reliability is updated according to the final outcome and the degree of memory exposure. Experiments across three multi-agent frameworks and five question-answering and reasoning benchmarks show that C2Mem achieves the best average performance in all three frameworks. Ablation studies, trajectory analysis, and case studies further support the effectiveness of memory reliability calibration.
Critical strengths
-
The paper identifies an important but underexplored problem: semantically relevant memories may provide strong and consistent behavioral guidance even when they are unreliable. By separating memory influence into commitment and competence, the paper offers a clear and novel perspective for understanding and mitigating memory-induced overconfidence.
-
C2Mem provides a complete memory calibration mechanism with broad empirical support. It connects a behavioral tree, historical reliability posteriors, and pre-injection weight calibration. The method achieves strong average performance across three multi-agent frameworks and five different tasks, while the ablation studies, trajectory analysis, and case studies provide additional evidence for the contributions of its main components.
Critical weaknesses
-
Memory competence does not accurately measure the true utility of an individual memory. The paper updates competence using the final task outcome and node exposure, but task success or failure may also be caused by other memories, agents, or tool outputs. Therefore, competence mainly captures a correlation between memory usage and task outcomes rather than whether a particular memory is beneficial. The unknown node is also not a true no-memory counterfactual, which weakens its role as a reference baseline.
-
Commitment and competence are not clearly separated in the implementation. The LLM evaluator that generates memory-support weights can see each node’s historical_competence, while these weights are later used to calculate commitment. Competence may therefore already influence commitment and then be used again during calibration, weakening the paper’s central two-dimensional formulation.
Questions for authors
-
What are the node-splitting criterion, clustering method, maximum tree depth, Top-K value, and tree-distance normalization method? Have the authors verified that tree distance accurately represents consistency between behavioral strategies?
-
C2Mem relies on benchmark ground-truth answers to determine whether historical tasks succeed. How would memory competence be estimated and updated for open-ended tasks with no ground truth or only noisy feedback?
-
Calibrated weights affect subsequent exposure, which is then used to update competence. How does the method prevent a node that is incorrectly downweighted early from receiving too little new evidence to recover?
-
Could the authors add more direct ablations, such as a flat memory structure, hiding historical_competence from the support evaluator, and using a true no-memory reference, to separately verify the necessity of the behavioral tree, the separation of the two dimensions, and the unknown node?
5 comments
Question 1Behavioral Tree Construction and Validity
The behavioral tree groups similar execution strategies across tasks to share reliability evidence. An LLM summarizes 30 memory-free training trajectories into three initial behavioral patterns. Experiences are inserted by behavioral similarity. Every new trajectories per node (default n=10), an LLM decides whether and how to split it; children inherit the parent’s Beta parameters proportionally to their experience counts (Appendix A.2, Eqs. A.1–A.2). Depth is unrestricted to avoid forcing distinct strategies to share a reliability estimate. Retrieval uses top-K = 3 nodes plus the unknown node. Shortest-path distances are divided by the current tree’s maximum shortest-path length, yielding values in [0,1].
To test whether tree distance reflects behavioral compatibility, we compare node summaries and conduct same-prefix experiments on 200 HotpotQA tasks. We inject memory pairs at different distances individually or jointly, measuring next-action agreement between individual branches and average agreement between the joint action and the two individual actions. Closer memories have more similar summaries and higher agreement under both measures, supporting tree distance as a measure of behavioral coherence.
| Normalized Tree Distance | Mean Summary Cosine Distance | Pairwise Agreement (%) | Joint–Individual Agreement (%) |
|---|---|---|---|
| (0,1/3] | 0.168 | 54.7 | 68.9 |
| (1/3,2/3] | 0.176 | 45.2 | 61.7 |
| (2/3,1] | 0.233 | 24.1 | 45.7 |
Question 2Competence Updates for Open-Ended Tasks and Noisy Feedback
Thank you for your valuable suggestion. We considered the effect of feedback quality on competence estimation when designing our experiments. We therefore chose benchmarks with ground-truth answers to evaluate memory calibration under clear, consistent feedback while limiting interference from evaluation noise.
For open-ended tasks, LLM-as-a-Judge [1] or reward models [2] could provide proxy feedback, but evaluator errors and biases [3] may be mistaken for memory failures. The reviewer’s suggestion motivates future work on incorporating feedback reliability into competence estimation for more robust updates.
[1] Wei et al. Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards. EMNLP 2025.
[2] Chen et al. RM-R1: Reward Modeling as Reasoning. ICLR 2026.
[3] Xu et al. A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth. ICML 2026.
Question 3Re-Retrieval and Continued Learning after Early Downweighting
Thank you for raising this important question. C²Mem preserves opportunities for low-exposure nodes to participate again in later matching states. Retrieval prioritizes nodes whose behavioral patterns match the current execution state based on behavioral similarity (Eq. 2). Thus, an early-downweighted node can re-enter the top-K candidate set when a subsequent state closely matches its strategy.
During training, a retrieved node’s support weight and calibration are reassessed in the current state. Nodes whose calibrated weights meet the injection threshold can participate again and update competence through new feedback. Early downweighting therefore does not permanently exclude a node from use. Posterior confidence also modulates the penalty, distinguishing uncertain early estimates from judgments supported by sufficient evidence.
Question 4(1)The Role of the Behavioral Tree in Calibration and Interpretability
An advantage of the behavioral tree is that it identifies coherent guidance across memories and makes their strategic relationships interpretable. Nearby nodes supporting compatible strategies can jointly yield high commitment despite distributed support weights. Flattening removes the structural basis for this assessment and for tracing behaviors to broader strategies. On HotpotQA, full C²Mem outperforms its flat-memory variant, supporting the contribution of the tree structure.
| Method | EM (%) | F1 (%) |
|---|---|---|
| C²Mem with flat memory structure | 49.00 | 61.89 |
| C²Mem | 50.50 | 63.51 |
Our response to Question 1 further examines how tree distance reflects strategy compatibility. Nearby memories induce more consistent actions and better preserve their individual guidance when used jointly.
We are very grateful for your constructive comments and suggestions, which have been invaluable in improving our paper. We sincerely hope to continue our discussions with you and would welcome the opportunity to address any questions or concerns that remain.
We sincerely thank the reviewer for the thoughtful feedback on reliability estimation, the two-dimensional formulation, and memory updates. Below, we provide point-by-point responses with additional experimental evidence and implementation details.
Thank you for raising this important question. An advantage of C²Mem is that memory reliability estimates can be continually revised as new execution evidence becomes available. Even when a task fails because of other memories, agents, or tool outputs, that outcome contributes only part of a node’s accumulated evidence. Subsequent exposure in successful executions can still increase its competence and correspondingly reduce the calibration penalty.
We randomly select 200 HotpotQA tasks where calibration changes memory weights under AutoGen + C²Mem (GPT-4o, fixed behavioral tree). From the same execution prefix at the calibration step, we compare uncalibrated memories, removal of downweighted memories, and calibrated memories, holding other conditions fixed. Removal improves performance, while calibration yields further gains. These controlled comparisons support competence as a basis for memory intervention, beyond its association with task outcomes.
Section 3.8 and Appendix B, Figure 9 provide a detailed case study showing how calibration downweights low-competence guidance, allowing identity verification to redirect execution while preserving later reliable guidance.
The unknown node provides a reliability reference learned from training experience. Both ordinary nodes and the unknown node update their competence using accumulated exposure and task outcomes. To examine whether this reference corresponds to actual memory utility, we conduct same-prefix continuation experiments.
On 200 HotpotQA tasks with AutoGen, we freeze the behavioral tree and reliability posteriors and continue each task from the same execution prefix under three conditions: injecting a memory with competence below μ0, injecting one above μ0, or injecting no memory. The two memory conditions use the selected memories directly without additional calibration, with all other execution conditions held constant.
Under the same tasks and execution prefixes, memories below the reference reduce performance, while those above it improve performance. These results support μ0 as a useful comparison boundary in the evaluated states, helping distinguish memory guidance worth relying on from guidance that should be suppressed.
Weakness 2 & Question 4(2)Distinct Roles of Commitment and CompetenceThank you for your insightful observation. C²Mem examines memory from two perspectives—current guidance strength and historical execution reliability—which do not require entirely independent inputs. historical_competence provides the support evaluator with execution history to inform its assessment of behavioral compatibility with the current state. In computing competence, the same information quantifies the reliability of these memories. The two quantities draw on shared information but answer different questions.
Specifically, commitment depends on current support and the structural relationships between nodes (Eq. 6), whereas competence aggregates reliability estimates derived from historical successes and failures (Eq. 7). For example, memories with the same historical reliability may jointly support a coherent strategy or point toward different behavioral directions. They can therefore have identical aggregated competence but different commitment. This distinction allows calibration to apply different intervention strengths to memories with equal reliability but different levels of current guidance.