- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes MCMEM, a metacognitive memory mechanism for long-horizon LLM agents that calibrates reliance on retrieved memories using provenance-grounded evidence. The framework constructs a hierarchical memory graph that preserves provenance links to supporting evidence. It further introduces a metacognitive layer that disentangles correctness, adoption, and execution signals from episode-level evidence to revise memories and control their delivery. Experiments across multiple benchmarks and model scales demonstrate the effectiveness of MCMEM.
Critical strengths
- The paper is well motivated, addressing the important problem of memory reliability calibration. The hierarchical memory graph with explicit provenance edges and audited evidence provides clear traceability.
- The evaluation across four long-horizon benchmarks and multiple model sizes demonstrates the generality and effectiveness of the proposed approach.
Critical weaknesses
- The method requires additional model calls to verify and maintain memories, but the computational overhead is not evaluated or compared with baselines. The performance gains may partly stem from increased computation rather than the proposed mechanism itself.
- The paper is difficult to follow due to excessive formalization and overly complex definitions, which obscure the main ideas and reduce readability.
Questions for authors
- Could you provide a comparison of computational costs and evaluate whether MCMEM still outperforms the baselines under matched computation budgets?
- Does MCMEM use the same LLM for the assessor, applicability/evidence judges, adoption judge, and editor/merge judge, or are different models used?
- The method involves numerous hyperparameters that require task-specific configuration. How sensitive is performance to these settings?
- The paper does not separately ablate correctness, adoption, and execution beliefs. Could you provide ablations to clarify the individual contributions of these components?
7 comments
[W2] The paper is difficult to follow due to excessive formalization and overly complex definitions, which obscure the main ideas and reduce readability.
Thanks for the valuable comment. We will revise the methodology section accordingly to enhance the delivery of key ideas and insights of our method.
[Q2] Does MCMEM use the same LLM for the assessor, applicability/evidence judges, adoption judge, and editor/merge judge, or are different models used?
Thanks for the question. MCMem uses the same frozen LLM as the actor for all memory management and utilization procedures, including the assessor, applicability/evidence judges, adoption judge, and editor/merge judge. This design aligns with our goal of enabling an LLM agent to progressively recognize its own memory-augmented capability boundaries and calibrate its reliance on accumulated experience, without relying on a separate, more powerful model for memory evaluation or management.
[Q4] The paper does not separately ablate correctness, adoption, and execution beliefs. Could you provide ablations to clarify the individual contributions of these components?
Thanks for the insightful comment. We conduct additional three-repeat ablation studies to more clearly isolate the effects of three beliefs. As shown below, all these modules contribute to the performance superiority and stability across datasets. We will add these results into the revised manuscript.
| Variants | SW-9B | AW-9B |
|---|---|---|
| MCMem | 56.3 ± 3.39 | 81.6 ± 2.40 |
| w/o correctness belief only | 53.3 ± 4.01 | 76.9 ± 1.97 |
| w/o adoption belief only | 53.0 ± 3.21 | 77.9 ± 2.28 |
| w/o execution belief only | 51.5 ± 4.45 | 79.6 ± 0.860 |
Thank you for the additional experiments. However, some concerns remain. The sensitivity analysis shows that performance is considerably affected by hyperparameter choices, making the tuning protocol particularly important. How were the hyperparameters selected and were comparable tuning efforts applied to all baselines? Finally, the readability concern remains unaddressed, as no concrete revisions are provided. I maintain my original Weak Reject rating.
Thanks for the constructive comment. We agree the necessity of comparing total computation cost and conduct matched-total-budget experiments. We have additionally conducted the following experiments to address the reviewer's concern.
On SW 9B backbone setting, we have added an end-to-end cost table covering input and output tokens, all LLM calls and wall-clock time, with separate accounting for actor decision inference, within-step memory processing, and post-episode extraction and maintenance.
We also added two comparable computation budget setting for no-memory baseline and MCMem: 1) Enhanced no memory: we added additional self-review computation for the no-memory baseline to correct its potential decision errors; 2) MCMem, budget-comparable: we reduced the number of memory entries processed during agent retrieval, memory judging, and periodic review, and adopted batch processing for step-level applicability assessment and source-evidence verification instead of prompting the LLM separately for each entry.
At comparable measured token consumption, MCMem still achieves superior performance, and beat enhanced no-memory baseline across all four datasets. These results support the value of allocating computation to memory management and utilization and demonstrate the superiority of our method design is not purely from higher computational cost.
Thanks for the comment. We apologize for listing hyperparameters without discussing their sensitivity.
First, we clarify that MCMem uses a single fixed hyperparameter configuration across all benchmarks and actor scales without task-specific tuning.
Second, despite many hyperparameters, they serve different purposes: some govern core memory decisions, such as evidence requirements, retrieval, and maintenance, while others serve as implementation safeguards or computational limits.
Third, we conduct further sensitivity experiments on ScienceWorld SW and ALFWorld AW with the 9B backbone, varying representative parameters governing reliability estimation, memory retrieval, maintenance, and context budgets, one at a time, while keeping other settings fixed within each experiment.
The results show that 1 core decision settings matter: lowering the reliability threshold reduces success on both benchmarks; narrower retrieval budget also yield lower success in the tested runs; 2 maintenance sensitivity is environment-dependent: decreasing review capacity does not consistently degrade performance; and 3 auxiliary settings are relatively stable within the tested ranges: halving the context budget preserves performance, and changes to the execution threshold have comparatively smaller effects.
We will incorporate these results and reorganize Appendix B.1 by functional role, distinguishing core decision parameters from auxiliary implementation controls.