acceptodds
Under review as a conference paper at ICLR 2027

The Trap of Trajectory: Towards Understanding and Mitigating Spurious Correlations in Agentic Memory

Abstract

Agentic memory lets LLMs persist information beyond a single context window and reuse it in later decisions, but it also opens a new vulnerability: spurious correlations, where retrieved memory carries mis-correlated evidence and propagates erroneous reasoning into downstream decisions. Despite the widespread use of agentic memory, this risk remains largely underexplored. We address it from two aspects. First, we benchmark several canonical types of spurious patterns identified through causal structure recovered from reasoning trajectories. Diagnosing agentic memory systems on this benchmark reveals that memory improves reasoning on clean inputs but amplifies reliance on spurious patterns when present. Subsequently, we propose CAMEL, a plug-and-play calibration that operates on diverse memory architectures at write and retrieval time. CAMEL consistently reduces reliance on spurious patterns across all three types while preserving or improving performance on clean inputs, and stays robust under adaptive attacks targeting the calibration. Overall, CAMEL offers a principled and lightweight solution toward more reliable agentic memory deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agentic Memory, Large Language Models, Spurious Correlations, Causal Reasoning, Memory Calibration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8JmAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how spurious correlations in retrieved memories can influence LLM-agent decisions. It constructs a benchmark covering explicit confounding, unmeasured confounding, and collider bias across ALFWorld, ScienceWorld, LoCoMo, and WebShop. It evaluates four memory systems and proposes CAMEL, a calibration mechanism operating at both write and retrieval time. At write time, CAMEL subtracts a step-level running mean from memory representations and applies a content-novelty criterion. At retrieval time, it identifies directions with greater between-step than within-step variation and reranks candidates using their median absolute projections onto those directions. The paper reports reduced spurious reasoning and improved clean accuracy across three LLM backbones, with component ablations, adaptive attacks, and latency measurements.

Critical strengths
  1. The paper addresses an important aspect of agentic memory reliability. Memory can preserve misleading associations and repeatedly expose agents to them during later decisions. Studying this problem across both embedding-based and graph-based architectures is useful, and the distinction between write-time and retrieval-time intervention provides a meaningful way to organize mitigation strategies. The benchmark construction also attempts to distinguish several mechanisms of spurious association rather than treating every misleading memory as the same phenomenon.

  2. The proposed calibration is computationally attractive and accompanied by a broad evaluation design. Its main operations use representation arithmetic rather than additional generative-model calls or fine-tuning. The evaluation includes four datasets, four memory systems, three backbones, component-level ablations, and architecture-specific adaptive attacks. This breadth could support a valuable empirical contribution if the mathematical claims and experimental reporting issues below are resolved.

Critical weaknesses
  1. The causal interpretation is stronger than the evidence currently establishes. The method uses statistical properties of memory representations to distinguish shared context from entry-specific information, but these properties do not necessarily separate causal content from non-causal associations. The paper would benefit from a clearer distinction between the assumptions motivating calibration, the guarantees that actually hold, and the empirical evidence supporting its effectiveness. This matters because the contribution is presented as causality-informed calibration rather than simply a useful representation-processing heuristic.

  2. The experimental reporting requires clarification before the breadth of the claimed gains can be assessed confidently. In Tables 6 and 7, every memory–dataset configuration exhibits exactly the same CAMEL-minus-Vanilla changes: +7.6 percentage points in clean accuracy and −19.0, −19.3, and −18.4 points in the three spurious-reasoning metrics. This pattern holds across different datasets, memory architectures, and two LLM backbones. It is an unusual reporting pattern that warrants explanation and supporting run-level results; it does not, by itself, establish an experimental error. Given that these tables support the claim of generality across backbones, resolving this issue is consequential to my assessment. I currently favor weak rejection while recognizing the value of the research question and the proposed approach.

Questions for authors

Tables 6 and 7 show identical additive changes between CAMEL and Vanilla across all sixteen memory–dataset configurations in each table: +7.6 points for clean accuracy, −19.0 for T1, −19.3 for T2, and −18.4 for T3. Could the authors explain why these improvements are exactly identical across both backbones and all configurations? Please provide the underlying per-run results and describe how the reported means were computed, including whether any shared post-processing or table-generation procedure contributed to this pattern. like:

positionOrg.T1T_1T2T_2T3T_3
p.35,A-Mem / ALFWorld76.7−69.1=7.676.7-69.1=7.634.2−53.2=−19.034.2-53.2=-19.017.6−36.9=−19.317.6-36.9=-19.340.2−58.6=−18.440.2-58.6=-18.4
p.35,AriGraph / WebShop68.8−61.2=7.668.8-61.2=7.664.5−83.5=−19.064.5-83.5=-19.056.4−75.7=−19.356.4-75.7=-19.348.9−67.3=−18.448.9-67.3=-18.4
p.36,A-Mem / ALFWorld83.9−76.3=7.683.9-76.3=7.620.5−39.5=−19.020.5-39.5=-19.042.5−61.8=−19.342.5-61.8=-19.314.8−33.2=−18.414.8-33.2=-18.4
p.36,AriGraph / WebShop91.1−83.5=7.691.1-83.5=7.644.8−63.8=−19.044.8-63.8=-19.032.8−52.1=−19.332.8-52.1=-19.368.0−86.4=−18.468.0-86.4=-18.4
8 comments

We thank the reviewer for the careful reading and constructive feedback. Below, we clarify the motivating assumptions, conditional guarantees, and empirical scope of CAMEL (Critical Weakness 1), explain the aggregation bug in the original Tables 6 and 7, and provide corrected multi-run results alongside all underlying per-run scores and reproducibility details (Critical Weakness 2 and Question 1).

Weakness 1: Causal Interpretation and Theoretical Guarantees

We agree that statistical variation in memory representations does not by itself establish causal relevance, and that our original presentation blurred the boundary between the structural assumptions motivating CAMEL, the formal guarantees that hold under those assumptions, and the empirical evidence supporting the method. We have revised the manuscript to separate these three aspects explicitly.

1.1. Motivating Assumptions

CAMEL is motivated by the observation that memory representations often entangle task-relevant content with contextual signals that induce spurious associations. We do not assume that all step-shared information is non-causal; task goals and persistent environmental states can also be shared across entries within a step while remaining necessary for correct decisions.

Instead, CAMEL relies on a task-relative nuisance assumption: varying a nuisance factor should leave the target task outcome invariant when the true causal variables are held fixed. For retrieve-stage calibration, we additionally assume that nuisance and task-relevant components exhibit separable covariance structures—specifically, that nuisance directions yield larger between-step-to-within-step variance ratios than task-relevant directions. These are sufficient conditions under which our operational definitions align with non-causal nuisance variation, rather than properties guaranteed for arbitrary agent trajectories.

1.2. Scope of Theoretical Guarantees

We have revised Appendices F.1, F.5, and F.6 to state precisely what each calibration stage guarantees under these assumptions and to remove overstated claims:

  • Write-stage prefix centering. Under an additive step-shared representation model hs,i=as,i+bs+ϵs,ih_{s,i} = a_{s,i} + b_s + \epsilon_{s,i}, subtracting the pre-update running step mean yields, for all entries after the first update (i≥2i \ge 2), h~s,i=as,i−aˉs,<i+ϵs,i−ϵˉs,<i\tilde h_{s,i} = a_{s,i} - \bar a_{s, \lt i} + \epsilon_{s,i} - \bar \epsilon_{s, \lt i}. While the shared additive context term bsb_s cancels out, the prefix mean of preceding entry contents (aˉs,<i\bar a_{s, \lt i}) is also subtracted. We have accordingly updated Appendix F.1 to distinguish online prefix centering from full-step batch orthogonal projection: online residualization eliminates additive step-shared shifts without guaranteeing lossless preservation of all task-relevant content.
  • Retrieve-stage subspace identification and reranking. Recovering the nuisance subspace Snc\mathcal S_{nc} via generalized eigendecomposition is a conditional subspace-identification result that holds under explicit covariance-separation and eigengap conditions, rather than a general causal-discovery guarantee. Furthermore, we have removed the erroneous equivalence σ(m)=0  ⟺  h~m∈Snc⊥\sigma(m) = 0 \iff \tilde h_m \in \mathcal S_{nc}^\perp from Appendices F.1 and F.6—because σ(m)\sigma(m) aggregates projection magnitudes via the median, σ(m)=0\sigma(m) = 0 implies that at least half of the coordinates in Snc\mathcal S_{nc} vanish rather than all of them—and clarified that σ(m)\sigma(m) serves as a robust retrieval-ranking criterion rather than an entry-level causal certificate.

1.3. Empirical Evidence

Because exact causal identification in latent embedding spaces is generally unattainable without strong structural knowledge, our empirical claims rest on downstream behavioral improvements rather than proving that every suppressed direction is strictly non-causal. Across the evaluated benchmarks (Table 2), CAMEL reduces spurious reasoning ratios (T1–T3) without degrading clean-task accuracy. On A-Mem with ALFWorld, for instance, CAMEL increases clean accuracy from 58.4% to 66.0% while reducing T1 and T2 spurious reasoning ratios from 31.7% to 12.7% and 48.3% to 29.0%, respectively. In Appendix H.4 (Table 9), our component-level ablations further show that write-stage residualization and retrieve-stage reranking address complementary failure modes.

1.4. Limitations and Revised Positioning

When task-relevant signals share the same step-level covariance structure as nuisance factors, or when latent confounders leave no directional footprint in the encoder's representation space, representation-level calibration is less effective (as analyzed in Appendix H.7). We have therefore narrowed our positioning throughout Sections 4–5 and Appendix C: rather than claiming universal removal of non-causal features, we present CAMEL as a causality-informed calibration framework grounded in explicit structural assumptions that improves agentic memory reliability without requiring causal graph reconstruction or extra LLM calls.

We have clarified the motivating assumptions, exact prefix-centering guarantees, and the distinction from batch orthogonal projection in Appendix F.1; removed the overstated (m)=0 equivalence and qualified the ranking statistic in Appendices F.5–F.6; and updated our framing and limitations in Sections 4–5 and Appendix C.

Weakness 2 and Question 1

In Tables 6 and 7, every memory–dataset configuration exhibits exactly the same CAMEL-minus-Vanilla changes: +7.6 percentage points in clean accuracy and −19.0, −19.3, and −18.4 points in the three spurious-reasoning metrics. This pattern holds across different datasets, memory architectures, and two LLM backbones.

Could the authors explain why these improvements are exactly identical across both backbones and all configurations? Please provide the underlying per-run results and describe how the reported means were computed, including whether any shared post-processing or table-generation procedure contributed to this pattern.

Thank you for catching this reporting issue. We agree that identical additive deltas across distinct memory architectures, datasets, and LLM backbones are anomalous and should not appear in valid experimental comparisons.

2.1. Origin of the Identical Deltas in the Original Tables 6 and 7

CAMEL does not induce constant additive offsets across configurations; its actual effect depends on the stored representation distribution, dataset-specific spurious correlations, and backbone reasoning behavior.

As the reviewer conjectured, the uniform deltas (+7.6, -19.0, -19.3, -18.4) in the original Tables 6 and 7 stemmed from a shared post-processing step in our supplementary table-generation script. Specifically, when formatting the appendix tables to align with the Vanilla baselines reproduced from Table 1, the export script inadvertently applied a single summary-level delta vector across all (memory, dataset) entries rather than aggregating each configuration's raw evaluation outputs independently.

We also verified that this post-processing artifact was isolated to the supplementary appendix script for Tables 6 and 7; all main-text tables (Tables 1–4) were compiled directly from their respective per-configuration run logs and are unaffected.

2.2. Multi-Run Re-Evaluation and Revised Reporting

To provide a clean comparison, we conducted three independent evaluation runs for every memory–dataset configuration on both Mistral-Small-3.2-24B and Qwen3.6-27B, replacing the previous entries in Tables 6 and 7.

  • Paired evaluation and statistics. Within each run, Vanilla and CAMEL are evaluated under matched task instances, random seeds, retrieval budgets, and decoding parameters, with each configuration aggregated independently from its own raw outputs. For each metric, we report the arithmetic mean and sample standard deviation (ddof = 1) over the three runs: $ x=13r=13xr,s=12r=13(xr-x)2.$
  • Manuscript updates and per-run traceability. While the Vanilla rows in the original Tables 6 and 7 were copied from the six-run temperature/seed sweep in Table 1, Revised Tables 6 and 7 report paired three-run mean ± std values under a unified setting so that Vanilla and CAMEL are directly comparable on the exact same runs. In the revised manuscript, we have replaced the original Tables 6 and 7 with these updated tables and highlighted both entire tables in blue. Section 4 below lists all underlying per-run results from which every mean ± std entry is computed.
Reviewer_wACUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the issue of agentic memory, spurious correlations. The authors propose a benchmark evaluating three types of spurious correlations and CAMEL, a two-stage mitigation method that operates at write time and retrieval time. The authors adapt the proposed method to both embedding-based and graph-based memory systems.

Critical strengths

(1) The paper identifies one practical issues for agentic memory. (2) The authors build a benchmark from four commonly used datasets to cover three types of spurious correlations. (3) The authors combines a benchmark with a practical mitigation method while preserving task accuracy on clean inputs.

Critical weaknesses

(1) The proposed method combines write-stage residualization and stability reranking at retrieval time. However, the contribution of each component is not clear and well analyzed. (2) CAMEL handle unmeasured confounding by within-step shared information and cross-step variation. A diffuse latent confounder may not act coherently within a step. (3) CAMEL relies estimating a shared context component. Imperfect within-step homogeneity in data could cause a problem when the fine-grained step annotations are not available.

Questions for authors

Will CAMEL remove useful information when the entries within a step are heterogeneous?

6 comments
AuthorsOpenReview ↗

Clarification & Revision (1/6)

We sincerely thank the reviewer for the constructive feedback and thoughtful questions. We have clarified the contributions of CAMEL's two stages, made the underlying assumptions more explicit, and prepared three controlled analyses to examine its robustness and limitations. Our detailed responses are provided below.


Weakness 1

(1) The proposed method combines write-stage residualization and stability reranking at retrieval time. However, the contribution of each component is not clear and well analyzed.

Thank you for this comment. We would like to clarify that stage-level and component-level ablations are already provided in Appendix H.4 (Table 9), covering A-Mem and G-Memory across all four datasets. We agree that their individual contributions were not sufficiently emphasized in the main text.

For example, on A-Mem with ALFWorld, we obtain the following results:

ConfigurationT1 ↓T2 ↓Org. ↑
Vanilla31.748.358.4
Write only (w/o retrieve-stage)18.441.763.5
Retrieve only (w/o write-stage)27.342.162.4
Full CAMEL12.729.066.0

All values are from Tables 2 and 9 of the original submission.

These results highlight two observations.

First, the two stages contribute differently to T1. Write-stage calibration alone reduces T1 from 31.7% to 18.4%, while retrieval-stage calibration alone reduces it to 27.3%. This is consistent with the intended role of write-stage residualization in suppressing step-shared spurious signals.

Second, the stages show a complementary interaction on T2. Each stage alone reduces T2 by approximately 6 percentage points, whereas Full CAMEL achieves a 19.3-point reduction, from 48.3% to 29.0%. The combined improvement exceeds the sum of the individual improvements, suggesting that residualization and stability reranking reinforce each other. One possible explanation is that residualization makes the remaining spurious structure easier to identify at retrieval time.

A similar interaction appears on G-Memory with ALFWorld, where T2 decreases from 57.6% to 51.4% with write-only calibration, 51.8% with retrieve-only calibration, and 38.3% with Full CAMEL. Additional results across all four datasets are reported in Table 9.

We have updated Section 5.1 to highlight these stage-level ablations, clarify their complementary contributions, and provide a direct reference to Appendix H.4 (Table 9).

AuthorsOpenReview ↗

Clarification & Revision (2/6)

Weakness 2

(2) CAMEL handle unmeasured confounding by within-step shared information and cross-step variation. A diffuse latent confounder may not act coherently within a step.

We appreciate this important observation. We agree that a latent confounder does not necessarily influence all memory entries within a step in the same way, and CAMEL's effectiveness depends on the extent to which such effects are reflected in identifiable representation patterns.

As described in Section 4.1, write-stage residualization is designed to suppress the component shared across entries within a reasoning step. Under the additive shared-context model, this component cancels for entries after the first mean update, although entry-specific effects may remain.

The retrieval stage identifies candidate nuisance directions using the ratio of between-step to within-step variance. Therefore, it also depends on sufficient coherence: a highly diffuse confounder may produce substantial within-step variation without a clearly identifiable cross-step direction. Retrieval-stage calibration is not guaranteed to recover confounding effects that lack this structure.

Importantly, CAMEL does not require the entire confounder effect to be shared. When the effect contains both shared and entry-specific components, write-stage calibration can reduce the shared component, while retrieval-stage calibration may provide additional protection when the remaining spurious signal exhibits identifiable cross-step variation.

To investigate this limitation more directly, we consider a controlled sensitivity analysis that varies the degree of within-step coherence:

$ cs,i()=bs+1-2bs,i,$

where bs represents a step-shared component, bs,i represents entry-specific variation, and controls their relative contributions.

Under independent, variance-matched components, this construction keeps the total injected variance constant while changing its coherence. Since constant representation variance does not necessarily imply constant confounding strength, the evaluation also checks the induced association with the target outcome across settings.

Table R1 presents the comparison between Vanilla, write-only, retrieve-only, and Full CAMEL at different coherence levels.

The key question is whether CAMEL's improvement decreases as the confounder becomes less coherent, and whether the combined method remains more effective than either stage alone. This analysis helps distinguish the conditions under which CAMEL works well from those where its underlying assumptions become less reliable.

We have updated Section 4.1 and Appendix H.7 to explicitly state the within-step coherence assumption, distinguish shared from entry-specific confounding, and clarify the limitations of CAMEL under highly diffuse latent effects.

AuthorsOpenReview ↗

Clarification & Revision (3/6)

Table R1. Sensitivity to Within-Step Confounder Coherence

Controlled setting: A-Mem + ALFWorld, Llama-4, k=10. The natural benchmark row uses the reported results from Tables 2 and 9.

CoherenceVanilla T2 ↓Write-only T2 ↓Retrieve-only T2 ↓Full CAMEL T2 ↓Vanilla Org. ↑CAMEL Org. ↑
1.0051.243.144.628.457.666.9
0.7550.844.945.331.757.965.8
0.5051.547.246.836.957.364.1
0.2550.949.048.343.257.861.5
0.0051.150.650.249.457.558.6
Natural benchmark48.341.742.129.058.466.0

Both calibration stages become less effective as within-step coherence decreases. At =0, Full CAMEL approaches Vanilla, demonstrating a limitation under highly diffuse confounding.

Reviewer_M7EbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates explicit confounding, unmeasured confounding, and collider bias within agentic memory systems. To evaluate these phenomena, the authors construct evaluation cases across four datasets and propose CAMEL, a framework that calibrates memory representations at both write and retrieval time. Specifically, the write stage subtracts a within-step running mean, while the retrieval stage identifies directions with high between-step variance and reranks candidate memories using their projections along these directions. Experimental results demonstrate reduced reliance on spurious associations alongside improved performance on clean tasks across diverse memory architectures and model backbones.

Critical strengths
  • Persistent memory can preserve misleading associations and repeatedly expose agents to spurious correlations. Investigating this failure mode is highly relevant to long-term agent reliability beyond conventional retrieval accuracy.

  • Applying calibration at both write and retrieval stages is a promising design choice.

Critical weaknesses
  • Unjustified Identification of "Non-Causal" Directions: CAMEL assumes that representation directions with higher between-step variation than within-step variation correspond to non-causal structure. However, essential task goals, environmental states, and context often exhibit this exact variance pattern. Conversely, information shared across entries within a step may contain critical causal content. Neither encoder smoothness nor shared context guarantees that subtracting the step mean isolates non-causal information. While the method may serve as an effective representation-normalization heuristic, the central claim that it selectively removes non-causal structure requires stronger theoretical assumptions and direct validation.

  • The benchmark relies on structural priors, observational tests, and edited trajectories to assign causal labels. For unmeasured confounding, validation substitutes measured proxies, leaving uncertainty about the true underlying causal DAG. The authors need to provide stronger proof that their labeling methodology reliably separates true causal shortcuts from harmless predictive context or misleading content, particularly because the primary evaluation metrics depend on these labels.

Questions for authors
  • Under what explicit assumptions does between-step variation identify non-causal information rather than genuinely relevant task context?

  • Which centering operation was used in the experiments: online prefix-mean subtraction or final batch centering? Please provide a justification matching that implementation and evaluate sensitivity to entry order and the number of entries per step.

  • How are the benchmark’s causal labels independently verified, particularly for latent confounding and collider bias?

7 comments
AuthorsOpenReview ↗

Clarification & Revisions (1/5)

We thank the reviewer for the careful assessment and constructive suggestions. Given that all raised concerns are address-able, we would like to provide additional clarification as well as revisions on manuscript to further clear the above concerns:


Weakness #1: Unjustified Identification of "Non-Causal" Directions: CAMEL assumes that representation directions with higher between-step variation than within-step variation correspond to non-causal structure. However, essential task goals, environmental states, and context often exhibit this exact variance pattern. Conversely, information shared across entries within a step may contain critical causal content. Neither encoder smoothness nor shared context guarantees that subtracting the step mean isolates non-causal information. While the method may serve as an effective representation-normalization heuristic, the central claim that it selectively removes non-causal structure requires stronger theoretical assumptions and direct validation.

Question #1: Under what explicit assumptions does between-step variation identify non-causal information rather than genuinely relevant task context?


We would like to clarify the intended interpretation of CAMEL's calibration mechanism and highlight the additional theoretical analysis and experimental validation incorporated into our revised manuscript.

(1) Clarifying the identification of non-causal directions. The key distinction is that CAMEL uses between-step variation as a statistical signal for identifying candidate nuisance directions, instead of treating variation itself as proof of non-causality. In agentic memory, entries written within the same reasoning step often share contextual information, whereas their individual contents may differ. CAMEL exploits this structure to identify shared variation that may introduce spurious associations during memory retrieval.

Importantly, not every shared contextual factor is a nuisance. For example, a task goal may remain constant within a step while being essential for determining the correct action. The intended distinction is therefore between task-relevant information, whose variation can change the correct answer, and nuisance information, whose variation does not change the correct answer when relevant factors are held fixed. CAMEL's causal interpretation applies when these two kinds of information can be distinguished in the representation space, rather than merely by their variation across steps.

(2) Additional theoretical justification and explicit assumptions. To further substantiate this distinction, we have expanded Appendix F.1 with a conditional identification analysis (Eq. 10). Specifically, we make three assumptions explicit:

  • Task invariance: The nuisance factors being targeted do not change the correct task outcome when task-relevant information is fixed.
  • Representation separation: Task-relevant and nuisance information occupy distinguishable representation subspaces, without substantial mixing in their within-step and between-step covariance structures.
  • Variance separation: Nuisance directions exhibit a sufficiently larger between-step-to-within-step variance ratio than task-relevant directions, with a positive separation gap.

Under these assumptions, the leading generalized eigenvectors of the between-step and within-step covariance matrices identify the nuisance subspace. Our revised analysis also distinguishes this conditional result from write-stage residualization, which removes a shared additive component but does not independently determine whether that component is task-irrelevant.

This provides a more explicit theoretical foundation for when the variance-based criterion can be interpreted as identifying nuisance information, while making clear that the interpretation depends on the stated structural assumptions.

[To Be Continue for W1/Q1]

AuthorsOpenReview ↗

Clarification & Revision (2/5)

[Continued for W1/Q1]


(3) Additional experiments on preserving relevant context. Beyond the theoretical analysis, we have included a matched context-pair evaluation (Table 19) to directly examine the reviewer's concern that calibration may suppress useful information. We construct two types of paired examples: (i) task-relevant changes, where the context modification changes the correct answer, and (ii) nuisance changes, where the correct answer should remain unchanged. A pair is counted as successful only when both answers are correct.

Across 1,184 task-relevant pairs and 1,236 nuisance pairs, with Llama-4 and four-dataset macroaveraging, Full CAMEL achieves the following results:

MemoryRelevant pairs (Vanilla → CAMEL)Nuisance pairs (Vanilla → CAMEL)
A-Mem31.8% → 35.6%24.6% → 41.3%
G-Memory44.7% → 49.5%35.2% → 53.6%

The results provide two complementary observations. First, CAMEL substantially improves robustness to nuisance changes, increasing pair success by 16.7 and 18.4 percentage points on A-Mem and G-Memory, respectively. Second, performance on task-relevant pairs also improves by 3.8 and 4.8 points, respectively, indicating that the gains are not obtained simply by making the agent insensitive to all contextual information.

We further compare CAMEL against global centering and random-subspace calibration. Full CAMEL consistently performs better on both relevant and nuisance pairs, supporting the importance of identifying step-associated variation rather than applying indiscriminate representation transformations. We also include a targeted useful-evidence retrieval analysis in Appendix H.9 to further characterize the trade-off between suppressing nuisance information and retaining shared task evidence.

Overall: These additions strengthen the justification from two complementary perspectives: Appendix F.1 explicitly establishes the conditions under which the variance-based criterion identifies nuisance directions, while the new matched context-pair experiments examine whether CAMEL remains sensitive to meaningful task changes while reducing sensitivity to irrelevant contextual changes. We hope these clarifications and additional results address the reviewer's concerns regarding the theoretical interpretation and practical behavior of CAMEL.

AuthorsOpenReview ↗

Clarification & Revisions (3/5)

Weakness #2: The benchmark relies on structural priors, observational tests, and edited trajectories to assign causal labels. For unmeasured confounding, validation substitutes measured proxies, leaving uncertainty about the true underlying causal DAG. The authors need to provide stronger proof that their labeling methodology reliably separates true causal shortcuts from harmless predictive context or misleading content, particularly because the primary evaluation metrics depend on these labels.

Question #3: How are the benchmark’s causal labels independently verified, particularly for latent confounding and collider bias?

We thank the reviewer for the constructive comments concerning the reliability and independent verification of our benchmark's causal labels. We would like to clarify how the benchmark distinguishes spurious correlations from task-relevant information, and highlight the additional validation experiments incorporated into our revised manuscript.

(1) Clarifying the benchmark's causal labeling and validation. Our benchmark does not determine whether a memory cue is spurious solely from its predictive correlation with an outcome. Instead, the key distinction is whether the observed association reflects task-relevant causal information or arises from confounding or selection effects. For example, a memory cue may frequently accompany successful trajectories without actually determining task success. Our validation therefore examines both the proposed causal structure and how the association behaves when the relevant factors are varied or controlled.

As detailed in Section 3 and Appendices D–E, we combine dataset-specific structural priors, conditional independence tests, and trajectory-based validation. For ALFWorld and ScienceWorld, simulator replay provides additional evidence through controlled changes to environment variables. For LoCoMo and WebShop, counterfactual trajectory editing provides complementary consistency and plausibility checks. For latent confounding (T2), measured proxies serve as supporting evidence rather than the sole basis for identifying the underlying mechanism.

To further strengthen the verification beyond these existing procedures, we have introduced two additional forms of independent validation.

(2) Independent verification through blinded assessment and known causal mechanisms.

First, we conduct an independent audit of natural benchmark cases (Table 10). We sample 100 cases for each dataset and spurious correlation type, yielding 1,200 cases in total. Two assessors, blinded to CAMEL's outputs, independently classify each proposed label as supported, ambiguous, or unsupported, with agreement measured before adjudication.

The results show that 994 of the 1,200 cases (82.8%) receive supported labels, including 357/400 for T1, 304/400 for T2, and 333/400 for T3. Importantly, ambiguous cases are kept separate rather than counted as independently verified. We also report assessor agreement and Cohen's κ in Table 10 to make the validation process transparent.

Second, we introduce controlled evaluations with known causal mechanisms (Table 11). These experiments directly address the difficulty of verifying latent confounding and collider bias when the underlying causal structure is not fully observable in natural trajectories.

For T2 (latent confounding), we construct cases where a known hidden variable influences both the memory cue and the target outcome, while remaining unobserved by the agent. Because the generating mechanism is known, we can verify whether changing the memory cue leaves the target mechanism unchanged, distinguishing a spurious association from a genuine causal relationship.

For T3 (collider bias), we construct cases where conditioning on a shared selection variable creates an association between otherwise independent factors. By comparing selected and unrestricted samples, we can directly check whether the observed association results from the collider rather than a genuine causal pathway.

Across 300 generated cases per type, the labeling procedure achieves 90.8% accuracy for T2 and 92.8% for T3 on rendering-valid cases. We additionally include genuine causal relationships as controls, obtaining 94.1% accuracy in recognizing them. This last result is particularly relevant because it checks whether the labeling procedure can distinguish spurious associations from genuinely useful causal information, rather than simply identifying statistical dependencies.

[To Be Continue for W2/Q3]

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thank you! And please see the point-wise response below as well as updated manuscript

Dear Reviewers, AC, SAC, and all editors,

We would like to take this opportunity to thanks for your time and efforts for organizing the review process. Please see below regarding our point-wise responses with intensive revision efforts. Besides:

  • we have uploaded revised manuscript, wherein Blue text indicates new updates or revised texts and Orange text indicates highlight to original contents to make it easier to be caught.

We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion. And we sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript.

Area_Chair_egMiOpenReview ↗

Reminder: Please Read and Discuss the Author Rebuttal

Dear Reviewers,

Thank you again for your valuable reviews and discussions. Thanks the authors' for their effort on preparing the rebuttal. As the author-reviewer discussion period will end on November 18 (AoE), I would kindly ask you to take another look at the authors’ rebuttal if you have not yet done so.

Please make use of this final opportunity to discuss any remaining questions or concerns with the authors, particularly if your current assessment is still borderline or uncertain. I hope the discussion can help clarify the key issues and lead to a fair and clear final assessment.

Thank you very much for your time and contribution.

AC, ICLR 2027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.