acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoning

Abstract

Autonomous agents increasingly rely on memory to generalize beyond their training environments. However, agents are bounded by what they have seen and believed, and leveraging such memories in unseen environments can introduce biases into their internal beliefs. We formalize this phenomenon as false memory, which can arise from spurious correlations, environment shifts, and knowledge conflicts. Despite its importance, false memory is difficult to evaluate because it stems from agent internal beliefs and is easily confounded with ordinary generalization failures. Therefore, we propose FAME, a training-free framework that evaluates false memory through the evolution of agent beliefs under counterfactual reasoning. Specifically, counterfactual scenarios reveal how beliefs change as the latent concept of memory shifts under hypothetical interventions; thus, measuring the resulting concept drift provides a signal of faithful versus false memory. Such concepts can be estimated from agent hidden states before answer generation, eliminating the need for reward design or answer sampling. Empirical experiments reveal that simply monitoring answers often fails to detect false memory, while FAME achieves AUROCs of 76.2% – 96.7% across false-memory settings, and outperforms the best baseline by 3.4% – 23.3% across realistic benchmarks, spanning math reasoning (GSM-Symbolic), code generation (GitChameleon), and complex reasoning (BigBench-Hard). We further release corresponding counterfactual templates and facilitate future research on false memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Autonomous Agents, False Memory, Memory-Augmented Agents, Counterfactual, In-Context Learning, Trustworthy AI, Agent Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hKVuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FAME, a framework that detects “false memory” before answer generation by measuring changes in hidden representations under counterfactual queries. Its central idea is to distinguish when memory should remain stable from when it should adapt, with evaluations spanning mathematical reasoning, code generation, and complex reasoning tasks.

Critical strengths
  • A valuable research perspective. Connecting memory reliability, counterfactual interventions, and internal representations offers a useful perspective on when memory becomes unreliable in new contexts.

  • A reasonable methodological design. Distinguishing robustness from adaptation avoids simply equating representation drift with errors, and the ablation studies support this design choice.

Critical weaknesses
  • Memory-induced errors are not sufficiently distinguished from general reasoning failures. Labels primarily reflect answer or behavioral correctness, without systematic memory-removal or replacement controls. The theory mainly connects representation changes to answer preferences, rather than establishing that memory causes the errors.

  • Evidence of generalization remains limited. The method relies on reference queries constructed separately for each template, and evaluation primarily uses curated template-based tasks. Transfer to unseen templates and realistic long-term agent memory remains unclear.

Questions for authors
  • How can the authors establish that memory causes the errors? How do error rates and FAME scores change when memory is removed, replaced, or corrected?
  • Does the advantage persist under fair calibration? What happens when baselines can calibrate their score directions and access the same reference information as FAME?
  • Can the method generalize across templates and settings? Does it remain effective on unseen templates, natural counterfactuals without explicit misleading cues, and realistic multistep memory interactions?
Reviewer_oyyiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FAME, a training-free framework for detecting potential false-memory behavior before answer generation. The proposed method constructs counterfactual queries and measures the resulting hidden-state changes relative to leave-one-out memory demonstrations. It then assesses whether these changes reflect the expected behavior: preserving relevant knowledge when robustness is required or updating it when adaptation is needed. Deviations from these expectations serve as signals of potential false memory. Experiments on curated sentiment, mathematical, coding, and reasoning tasks demonstrate promising failure detection, although distinguishing memory-induced errors from ordinary task failures remains a central issue.

Critical strengths
  1. The distinction between adaptation and drift could be useful. The manuscript recognizes that some counterfactual changes require preserving existing knowledge, whereas others require updating it. This distinction provides a meaningful basis for evaluating whether an agent responds appropriately.
  2. Promising empirical detection results. The paper reports useful failure-prediction performance across mathematical, coding, and reasoning tasks. Additional-model experiments and component ablations provide further support for the proposed detection approach.
Critical weaknesses
  1. Unclear attribution of errors to memory. The paper aims to detect errors caused by memory, but its ground-truth labels are based on arithmetic operations, syntax, or final-answer correctness. These labels identify task failures without establishing whether memory caused them. Moreover, changing the query while holding memory fixed measures the effect of the query in the presence of memory; it does not isolate memory’s contribution. To support the central claim, the authors should therefore compare identical queries under original, absent, and corrected-memory conditions.
  2. Unfair baseline comparison. The paper reports a cloud-distance AUROC of 0.078 and a FAME AUROC of 0.922 for spurious keeping in Table 1. However, reversing the baseline’s score gives 1 − 0.078 = 0.922, matching FAME exactly. Hence, this result mainly shows that the baseline’s score points in the opposite direction but not that it lacks useful predictive information. A fair comparison should give both methods the same robustness/adaptation information when setting score direction. Separately, Table 3 describes 50% subsamples of a “top-70% template pool,” whereas the text states 60% random template selection. The authors should reconcile these descriptions because template selection affects how the reported results are interpreted.
  3. Mismatch between theory and implementation. The paper uses Theorem 3 to relate representation changes to logit changes under an assumed linear readout. However, this relationship does not establish that those changes identify memory-specific errors. Furthermore, the implementation uses intermediate-layer representations and template-based readout directions that differ from the theorem’s answer-head assumption, so the connection between the theorem and the implemented detector needs clarification. A related implementation issue arises in Algorithm 1: it requires an inverse covariance, but clouds of four or six demonstrations have rank at most three or five. The authors should specify the projection, regularization, or pseudoinverse used to make this computation well defined.
  4. Dependence on task-specific setup. Although FAME is described as training-free, the realistic-task evaluation constructs a separate reference direction for each template using 20–50 held-out query pairs. The method therefore still requires task-specific preparation, which should be accounted for in baseline comparisons and efficiency claims. To clarify how well the approach generalizes beyond this setup, the authors are encouraged to report performance on new templates without constructing fresh reference directions.
Questions for authors

Please see weakness

Reviewer_EyE8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies false memory, where an LLM agent uses past examples inappropriately, leading to failure on a new query. It proposes FAME, a training-free method that compares hidden states of the original and counterfactual queries. Experiments on sentiment analysis, mathematical reasoning, and coding benchmarks demonstrate that FAME consistently outperforms the evaluated baselines.

Critical strengths
  1. The paper studies a relevant problem, and the methodological insight looks sensible. The counterfactual approach is novel, and it's indeed true that naively treating a large representation gap as false memory is not a good idea.
  2. There are consistent empirical gains, and the paper's method is technically solid.
Critical weaknesses

My biggest concern is with respect to the evaluation, and it is twofold.

  1. Although the paper motivates the problem through autonomous agents, the experiments primarily involve fixed datasets and constructed in-context demonstrations, rather than agents interacting with an environment and accumulating memories through experience. This setup allows the authors to preconstruct counterfactual queries and reference directions tailored to each scenario. It remains unclear how these would be obtained for an autonomous agent facing open-ended tasks and evolving memories.
  2. The evaluation relies on relatively small models, primarily Llama-3.2-3B-Instruct, with additional experiments on models up to 8B parameters. Many of the constructed failures appear simple enough that stronger models might directly avoid them. Without evaluating stronger models or more challenging agentic settings, it is difficult to assess whether FAME addresses a practically significant memory failure or mainly detects weaknesses of smaller models under artificial prompting conditions.
Questions for authors
  1. How do the authors establish that the observed errors are caused by “false memories,” rather than errors the model would also make without any demonstrations?
  2. Can the authors clarify how the reported experiments support the practical relevance of FAME, particularly for autonomous agents operating in real-world settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.