- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how spurious correlations in retrieved memories can influence LLM-agent decisions. It constructs a benchmark covering explicit confounding, unmeasured confounding, and collider bias across ALFWorld, ScienceWorld, LoCoMo, and WebShop. It evaluates four memory systems and proposes CAMEL, a calibration mechanism operating at both write and retrieval time. At write time, CAMEL subtracts a step-level running mean from memory representations and applies a content-novelty criterion. At retrieval time, it identifies directions with greater between-step than within-step variation and reranks candidates using their median absolute projections onto those directions. The paper reports reduced spurious reasoning and improved clean accuracy across three LLM backbones, with component ablations, adaptive attacks, and latency measurements.
Critical strengths
-
The paper addresses an important aspect of agentic memory reliability. Memory can preserve misleading associations and repeatedly expose agents to them during later decisions. Studying this problem across both embedding-based and graph-based architectures is useful, and the distinction between write-time and retrieval-time intervention provides a meaningful way to organize mitigation strategies. The benchmark construction also attempts to distinguish several mechanisms of spurious association rather than treating every misleading memory as the same phenomenon.
-
The proposed calibration is computationally attractive and accompanied by a broad evaluation design. Its main operations use representation arithmetic rather than additional generative-model calls or fine-tuning. The evaluation includes four datasets, four memory systems, three backbones, component-level ablations, and architecture-specific adaptive attacks. This breadth could support a valuable empirical contribution if the mathematical claims and experimental reporting issues below are resolved.
Critical weaknesses
-
The causal interpretation is stronger than the evidence currently establishes. The method uses statistical properties of memory representations to distinguish shared context from entry-specific information, but these properties do not necessarily separate causal content from non-causal associations. The paper would benefit from a clearer distinction between the assumptions motivating calibration, the guarantees that actually hold, and the empirical evidence supporting its effectiveness. This matters because the contribution is presented as causality-informed calibration rather than simply a useful representation-processing heuristic.
-
The experimental reporting requires clarification before the breadth of the claimed gains can be assessed confidently. In Tables 6 and 7, every memory–dataset configuration exhibits exactly the same CAMEL-minus-Vanilla changes: +7.6 percentage points in clean accuracy and −19.0, −19.3, and −18.4 points in the three spurious-reasoning metrics. This pattern holds across different datasets, memory architectures, and two LLM backbones. It is an unusual reporting pattern that warrants explanation and supporting run-level results; it does not, by itself, establish an experimental error. Given that these tables support the claim of generality across backbones, resolving this issue is consequential to my assessment. I currently favor weak rejection while recognizing the value of the research question and the proposed approach.
Questions for authors
Tables 6 and 7 show identical additive changes between CAMEL and Vanilla across all sixteen memory–dataset configurations in each table: +7.6 points for clean accuracy, −19.0 for T1, −19.3 for T2, and −18.4 for T3. Could the authors explain why these improvements are exactly identical across both backbones and all configurations? Please provide the underlying per-run results and describe how the reported means were computed, including whether any shared post-processing or table-generation procedure contributed to this pattern. like:
| position | Org. | |||
|---|---|---|---|---|
| p.35,A-Mem / ALFWorld | ||||
| p.35,AriGraph / WebShop | ||||
| p.36,A-Mem / ALFWorld | ||||
| p.36,AriGraph / WebShop |
8 comments
1.4. Limitations and Revised Positioning
When task-relevant signals share the same step-level covariance structure as nuisance factors, or when latent confounders leave no directional footprint in the encoder's representation space, representation-level calibration is less effective (as analyzed in Appendix H.7). We have therefore narrowed our positioning throughout Sections 4–5 and Appendix C: rather than claiming universal removal of non-causal features, we present CAMEL as a causality-informed calibration framework grounded in explicit structural assumptions that improves agentic memory reliability without requiring causal graph reconstruction or extra LLM calls.
We have clarified the motivating assumptions, exact prefix-centering guarantees, and the distinction from batch orthogonal projection in Appendix F.1; removed the overstated (m)=0 equivalence and qualified the ranking statistic in Appendices F.5–F.6; and updated our framing and limitations in Sections 4–5 and Appendix C.
Weakness 2 and Question 1
In Tables 6 and 7, every memory–dataset configuration exhibits exactly the same CAMEL-minus-Vanilla changes: +7.6 percentage points in clean accuracy and −19.0, −19.3, and −18.4 points in the three spurious-reasoning metrics. This pattern holds across different datasets, memory architectures, and two LLM backbones.
Could the authors explain why these improvements are exactly identical across both backbones and all configurations? Please provide the underlying per-run results and describe how the reported means were computed, including whether any shared post-processing or table-generation procedure contributed to this pattern.
Thank you for catching this reporting issue. We agree that identical additive deltas across distinct memory architectures, datasets, and LLM backbones are anomalous and should not appear in valid experimental comparisons.
2.1. Origin of the Identical Deltas in the Original Tables 6 and 7
CAMEL does not induce constant additive offsets across configurations; its actual effect depends on the stored representation distribution, dataset-specific spurious correlations, and backbone reasoning behavior.
As the reviewer conjectured, the uniform deltas (+7.6, -19.0, -19.3, -18.4) in the original Tables 6 and 7 stemmed from a shared post-processing step in our supplementary table-generation script. Specifically, when formatting the appendix tables to align with the Vanilla baselines reproduced from Table 1, the export script inadvertently applied a single summary-level delta vector across all (memory, dataset) entries rather than aggregating each configuration's raw evaluation outputs independently.
We also verified that this post-processing artifact was isolated to the supplementary appendix script for Tables 6 and 7; all main-text tables (Tables 1–4) were compiled directly from their respective per-configuration run logs and are unaffected.
2.2. Multi-Run Re-Evaluation and Revised Reporting
To provide a clean comparison, we conducted three independent evaluation runs for every memory–dataset configuration on both Mistral-Small-3.2-24B and Qwen3.6-27B, replacing the previous entries in Tables 6 and 7.
- Paired evaluation and statistics. Within each run, Vanilla and CAMEL are evaluated under matched task instances, random seeds, retrieval budgets, and decoding parameters, with each configuration aggregated independently from its own raw outputs. For each metric, we report the arithmetic mean and sample standard deviation (
ddof = 1) over the three runs: $ x=13r=13xr,s=12r=13(xr-x)2.$ - Manuscript updates and per-run traceability. While the Vanilla rows in the original Tables 6 and 7 were copied from the six-run temperature/seed sweep in Table 1, Revised Tables 6 and 7 report paired three-run
mean ± stdvalues under a unified setting so that Vanilla and CAMEL are directly comparable on the exact same runs. In the revised manuscript, we have replaced the original Tables 6 and 7 with these updated tables and highlighted both entire tables in blue. Section 4 below lists all underlying per-run results from which everymean ± stdentry is computed.
We thank the reviewer for the careful reading and constructive feedback. Below, we clarify the motivating assumptions, conditional guarantees, and empirical scope of CAMEL (Critical Weakness 1), explain the aggregation bug in the original Tables 6 and 7, and provide corrected multi-run results alongside all underlying per-run scores and reproducibility details (Critical Weakness 2 and Question 1).
Weakness 1: Causal Interpretation and Theoretical Guarantees
We agree that statistical variation in memory representations does not by itself establish causal relevance, and that our original presentation blurred the boundary between the structural assumptions motivating CAMEL, the formal guarantees that hold under those assumptions, and the empirical evidence supporting the method. We have revised the manuscript to separate these three aspects explicitly.
1.1. Motivating Assumptions
CAMEL is motivated by the observation that memory representations often entangle task-relevant content with contextual signals that induce spurious associations. We do not assume that all step-shared information is non-causal; task goals and persistent environmental states can also be shared across entries within a step while remaining necessary for correct decisions.
Instead, CAMEL relies on a task-relative nuisance assumption: varying a nuisance factor should leave the target task outcome invariant when the true causal variables are held fixed. For retrieve-stage calibration, we additionally assume that nuisance and task-relevant components exhibit separable covariance structures—specifically, that nuisance directions yield larger between-step-to-within-step variance ratios than task-relevant directions. These are sufficient conditions under which our operational definitions align with non-causal nuisance variation, rather than properties guaranteed for arbitrary agent trajectories.
1.2. Scope of Theoretical Guarantees
We have revised Appendices F.1, F.5, and F.6 to state precisely what each calibration stage guarantees under these assumptions and to remove overstated claims:
1.3. Empirical Evidence
Because exact causal identification in latent embedding spaces is generally unattainable without strong structural knowledge, our empirical claims rest on downstream behavioral improvements rather than proving that every suppressed direction is strictly non-causal. Across the evaluated benchmarks (Table 2), CAMEL reduces spurious reasoning ratios (T1–T3) without degrading clean-task accuracy. On A-Mem with ALFWorld, for instance, CAMEL increases clean accuracy from 58.4% to 66.0% while reducing T1 and T2 spurious reasoning ratios from 31.7% to 12.7% and 48.3% to 29.0%, respectively. In Appendix H.4 (Table 9), our component-level ablations further show that write-stage residualization and retrieve-stage reranking address complementary failure modes.