- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FAME, a framework that detects “false memory” before answer generation by measuring changes in hidden representations under counterfactual queries. Its central idea is to distinguish when memory should remain stable from when it should adapt, with evaluations spanning mathematical reasoning, code generation, and complex reasoning tasks.
Critical strengths
-
A valuable research perspective. Connecting memory reliability, counterfactual interventions, and internal representations offers a useful perspective on when memory becomes unreliable in new contexts.
-
A reasonable methodological design. Distinguishing robustness from adaptation avoids simply equating representation drift with errors, and the ablation studies support this design choice.
Critical weaknesses
-
Memory-induced errors are not sufficiently distinguished from general reasoning failures. Labels primarily reflect answer or behavioral correctness, without systematic memory-removal or replacement controls. The theory mainly connects representation changes to answer preferences, rather than establishing that memory causes the errors.
-
Evidence of generalization remains limited. The method relies on reference queries constructed separately for each template, and evaluation primarily uses curated template-based tasks. Transfer to unseen templates and realistic long-term agent memory remains unclear.
Questions for authors
- How can the authors establish that memory causes the errors? How do error rates and FAME scores change when memory is removed, replaced, or corrected?
- Does the advantage persist under fair calibration? What happens when baselines can calibrate their score directions and access the same reference information as FAME?
- Can the method generalize across templates and settings? Does it remain effective on unseen templates, natural counterfactuals without explicit misleading cues, and realistic multistep memory interactions?