acceptodds
Withdrawn from ICLR 2027

When Relevant Memories Are Wrong: A Controlled Reliability Audit of Episodic Memory Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Episodic memory systems retrieve prior episodes by semantic similarity, but relevance does not guarantee a correct outcome. We construct paired GSM8K memories that preserve the task and most of the worked derivation while changing the final conclusion, and evaluate retrieval and downstream use under matched memory contexts. Across all 1,319 test questions, Qwen3-Embedding-4B ranks the verified episode first in 40.3% of final-only pairs and 66.4% of structured pairs; BGE-M3 reaches 32.1% and 39.8%, respectively. For BGE-M3, paraphrasing changes selection by at most 0.2 percentage points, whereas removing the duplicated question changes Good@1 by 9.3–12.6 percentage points. In a frozen response bank, verified memory raises accuracy from 93.0% to 99.0%, while corrupted memory reduces it to 56.5%. Semantic Top-1 reaches 81.0% versus 77.5% for matched random selection; the paired 95% CI includes zero. Natural incorrect-solution and temporal-evidence evaluations show that retrieval relevance, episode correctness, and generator utilization should be reported as distinct quantities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
episodic memory, language agents, memory retrieval, retrieval reliability, trustworthy AI, robustness evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CXMiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether retrievers in episodic memory systems can distinguish a correct past episode from a near-duplicate one with a wrong conclusion. Using GSM8K, the authors pair each official solution with a copy whose final answer is altered, and evaluate BM25, bge-m3, and Qwen3-Embedding-4B on which episode is ranked first. They then feed fixed memory contexts (none, correct only, corrupted only, both) to Qwen3-8B and Llama-3.1-8B to measure the downstream effect, and extend the analysis to naturally written incorrect solutions and LongMemEval-S. The results show that dense retrievers frequently rank the corrupted episode first, and that semantic top-1 retrieval is not significantly better than random selection. Based on these findings, the paper argues that retrieval selection, memory correctness, and generator utilization should be reported as separate quantities in episodic-memory evaluation.

Critical strengths
  • The paper highlights that relevance and correctness can diverge in episodic memory retrieval, which is an increasingly important issue as LLM agents rely on stored past experiences.
  • The counterfactual memory pairs isolate the effect of the final conclusion while keeping the task and most of the derivation fixed, and the frozen response bank allows retrieval policies to be compared without confounding from additional generation sampling.
  • The use of question-level clustering, paired tests, transition matrices, and matched random and oracle baselines makes the reported differences more interpretable than end-task accuracy alone.
Critical weaknesses

W1. The main claim is unclear and the paper is difficult to follow

  • The central conclusion is essentially that retrieval selection, memory correctness, and generator utilization should be reported separately. While reasonable, this recommendation is fairly intuitive and not by itself a substantial contribution.
  • The more concrete findings (e.g., dense retrievers barely reflect a changed conclusion when the question is duplicated; corrupted memories can even help weaker generators) are not elevated into clear takeaways.
  • No method for reliability-aware retrieval is proposed or validated. The suggestions in Appendix F (reliability signals, conflict-aware retrieval) remain untested.

W2. Inconsistent results across tables for the same setting

  • Table 3 and Table 5 both report Qwen3-8B on the same 100 frozen questions, yet the numbers differ substantially: corrupted-only accuracy is 56.5% in Table 3 but 73.0% in Table 5. A 16.5 pp gap in the corrupted condition is difficult to attribute to the number of seeds alone.
  • Similarly, the "exact query + full memory" control in Table 2 should match the bge-m3 setting in Table 1, but the values differ (31.5 vs. 32.1 for final-only, 36.0 vs. 39.8 for structured). Since embedding computation is deterministic, the paper should explain what differs between these runs (e.g., backend, preprocessing, corruption variant).
  • These discrepancies raise concerns about the reliability of the reported numbers and the consistency of the experimental pipeline.

W3. No downstream evaluation with bge-m3 as the retriever

  • bge-m3 is the most striking case in the retrieval audit, as it ranks the corrupted episode first more often than chance (32.3% and 39.9% Pairwise AUC). However, all downstream experiments appear to use Qwen3-Embedding-4B (or leave the retriever unspecified, as in Table 5 and Table 6b).
  • The query ablation in Table 2 is conducted only with BGE-M3, while the downstream replay uses a different embedding model, so the two analyses are not directly connected.
  • Since the response bank already stores both singleton responses, bge-m3-based semantic Top-1 can be computed without any additional generation. This analysis should be added.

W4. Conflicting results are not reconciled with the paper's narrative

  • In Table 5, corrupted memory raises Llama-3.1-8B accuracy from 69.0% to 90.0%. This suggests that the final-answer edit is a weak corruption This undermines the general claim that corrupted memories are harmful and questions whether the corruption mechanism captures realistic unreliable memories.
  • Section 7.5 argues that presenting both episodes enables recovery (98.5% / 96.5%). However, with naturally written incorrect solutions (Table 6b), presenting both yields only 43.0% / 42.0% for Qwen3-8B, far below verified-only (77.0%). The recovery effect thus appears limited to the synthetic setting, yet Appendix F still recommends conflict-aware multi-retrieval without acknowledging this.
  • The no-memory accuracy of Qwen3-8B drops from 93.0% (synthetic bank) to 41.0% (natural-error bank), indicating a strong selection bias toward harder questions that is not discussed.

W5. Concerns of the setting

  • It is not stated which corruption variant (final-only or structured) is used in the response bank and cross-generator replay.
  • BGE-M3 is run with a maximum sequence length of 256. Since the only difference between e+ and e- lies at the end of the text, truncation could make the two embeddings identical.
  • The rationale for disabling Qwen3-8B's reasoning mode is not given, although it may affect whether the generator detects arithmetic inconsistencies in corrupted memories.
Questions for authors
  • What explains the retrieval patterns in Table 1? In particular, why does Qwen3-Embedding-4B select the verified episode more often under structured edits (66.4%) than under final-only edits (40.3%), even though structured edits change more numbers? And why does bge-m3 systematically prefer the corrupted episode ?
  • Could you clarify why Table 3 and Table 5 report different numbers for Qwen3-8B on the same 100 questions?
  • Would the downstream results change if Qwen3-8B's reasoning mode were enabled? A reasoning model may be more likely to detect the arithmetic inconsistency introduced by final-only corruption.
  • In LongMemEval-S, why does BM25 achieve much higher Current@1 (89.7%) than both dense retrievers (50.0%)? Is there a specific reason that both dense models obtain exactly the same value?
Reviewer_KDynOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether retrieving a memory that is similar to the current question also helps a language model answer correctly. For each of the 1,319 GSM8K test questions, the authors pair the official worked solution with a version containing numerical errors. They test whether BM25, Qwen3-Embedding-4B, and BGE-M3 rank the correct version first. For the downstream experiment, they save model responses to 100 questions under five conditions: no memory, correct memory only, incorrect memory only, and both memories in either order. Retrieval policies are compared by reusing these saved responses. Further experiments examine query and memory formatting, a second answering model, naturally written incorrect solutions, and time-dependent information in LongMemEval-S.

The results show that selecting the correct memory depends on the retriever and the type of error, while the effect on answering also depends on the language model. In the main response bank, correct memory improves accuracy and incorrect memory reduces it, but semantic selection does not show a statistically reliable advantage over random selection.

Critical strengths
  • Measuring which memory is retrieved separately from how the model answers helps distinguish retrieval errors from failures to use the supplied information. Reusing saved model responses also allows retrieval policies to be compared without generating a new answer for each policy.

  • The results show that an incorrect memory can affect different models differently. In Table 5, synthetic incorrect memories hurt Qwen3-8B but help Llama-3.1-8B; in Table 6(b), naturally incorrect solutions hurt both models. These observations motivate evaluating a memory's correctness separately from its effect on answering.

Critical weaknesses
  • Key experimental comparisons cannot yet be verified from the provided PDF. Several reported results need a clearer account of how the experiments differ. Qwen3-8B's incorrect-memory accuracy changes from 56.5% in Table 3 to 73.0% in Table 5. No-memory and correct-memory-only accuracies also differ substantially between Tables 5 and 6(b), without a clear mapping of questions, prompts, and generation settings to each table. In Table 6(b), Random Top-1 reaches 65.0% and 52.0%, whereas choosing each memory with probability one half gives expected accuracies of 49.5% and 40.0% from the saved single-memory responses. A particular random draw can differ from its expectation, but the selection records are needed to assess this comparison.

    The construction of the evaluation data is also not sufficiently specified. The PDF does not clearly identify which numerical-error variant is used in the main response bank. Section 7.7 reports candidate counts and a human-review rubric, but does not give enough detail to reproduce candidate generation, model validation, or final subset selection. Complete prompts and their association with each experiment are also needed. If the supplementary materials contain these details, the authors should identify the relevant files. These gaps matter because the paper's contribution depends on controlled and reproducible comparisons.

  • The added scientific insight and its generality remain insufficiently established. Prior work already studies misleading retrieved content and the difference between retrieval relevance and usefulness for a language model (e.g., Hong et al., 2024; Pan et al., 2024; Trappolini et al., 2026). The paired memories and saved responses provide useful experimental control, but the paper needs to explain more clearly what new finding this design establishes beyond that literature.

    Most experiments use the current GSM8K question's own worked solution and a nearly identical incorrect version. The two synthetic error variants modify the final numerical answer and, in one variant, the preceding reasoning line. The naturally incorrect solutions broaden the error construction but remain within GSM8K. LongMemEval-S adds a temporal-update setting, but does not repeat the same paired correctness audit on another task. The main downstream comparisons also use only 100 questions each. The differences across error types and the opposite effects across models are interesting, but their causes and stability across tasks remain unclear. The current evidence therefore offers limited support for general conclusions about memories from agent interactions.

Questions for authors
  1. Issues of data authenticity
  • Different results in Tables 3 and 5. Why does Qwen3-8B's incorrect-memory accuracy change from 56.5% in Table 3 to 73.0% in Table 5? If Table 5 retains the two runs from Table 3 and adds one generation seed, the third run would require 106 correct answers out of 100. Please identify the differences in runs, prompts, and model settings that account for these results.

  • Different reference results in Tables 5 and 6(b). Between these tables, no-memory accuracy changes from 91.0% to 41.0% for Qwen3-8B and from 69.0% to 38.0% for Llama-3.1-8B. Correct-memory-only accuracy changes from 98.0% to 77.0% and from 97.0% to 62.0%, respectively. Please report the question-set overlap, selection procedures, prompts, and generation settings. Neither condition uses incorrect memories, so these changes need to be explained separately from the incorrect-memory construction.

  • The random baseline in Table 6(b). Choosing the correct or incorrect memory with equal probability gives expected accuracies of 49.5% for Qwen3-8B and 40.0% for Llama-3.1-8B, based on their saved single-memory responses. Table 6(b) instead reports 65.0% and 52.0%. A particular random draw can differ from its expectation. Please provide the memory chosen for each question and generation seed, the selection probabilities and random seeds, and whether the same choice is reused across generation seeds. Please also report the random baseline obtained by averaging the two saved outcomes for each question, with paired confidence intervals that group repeated generations of the same question together. Does this change the comparison with Semantic Top-1?

  • Errors despite access to the official solution. Does the Verified condition in Table 6(b) supply the current question's official worked solution, as defined in Section 4.2? Is the full solution visible in the model input? If so, please analyze the 23% and 38% of responses still scored incorrect for Qwen3-8B and Llama-3.1-8B. Representative outputs should distinguish wrong answers from answer-extraction failures, missing answer markers, and output truncation.

  1. Error construction in the main response bank. Do Tables 3 and 5 use final-answer-only edits, edits that also change the preceding reasoning line, or a mixture? Please identify the construction used for each table. If variants are mixed, provide their counts and assignment rule.

  2. How naturally incorrect solutions were constructed. How were the initial 3,957 solutions in Section 7.7 generated: ordinary attempts later screened for errors, or prompts that deliberately requested incorrect solutions? Please specify the generation model, sampling settings, model-validation procedure, and whether the generator or validator saw gold answers. After model validation and human review, how were the final 200 retrieval items and 100 downstream items selected? Please clarify their relationship to the 371 model-validator-retained candidates and the overlap between the two evaluation subsets.

  3. Similarity scores and input truncation. How large are the similarity-score differences between correct and incorrect memories, and how stable are rankings when the scores are nearly tied? With BGE-M3's 256-token input limit, how often are the edited answer regions actually retained? Does the improvement from removing the repeated question persist when the full edited region is visible in both conditions? This would help distinguish the effect of repeated wording from a change in which text fits within the input limit.

  4. Reproducing the retrieval runs. Table 1 and Table 2 use independent runs, but their nominally matching BGE-M3 conditions give different results. Which preprocessing, tokenizer, pooling, truncation, precision, or backend settings differ? Please also clarify whether remote and local embedding implementations used equivalent configurations, including in the LongMemEval-S experiments.

  5. Dependence on nearly identical memory texts. The correct and incorrect memories share the current question and most of its solution. Section 7.2 already tests query paraphrasing and removal of the question from memory. As a further robustness check, would BGE-M3 still select the correct memory less than 50% of the time if the question text inside the stored memories were paraphrased while the query remained unchanged? The correct and incorrect versions should remain matched in wording and mathematical content apart from the intended error.

  6. Why incorrect memory helps one model but hurts another. In Table 5, synthetic incorrect memory reduces Qwen3-8B accuracy by 18 percentage points but increases Llama-3.1-8B accuracy by 21 points. What evidence explains this difference? For example, an analysis of the generated answers could show whether models follow the incorrect conclusion, use the remaining correct reasoning, or answer independently. These are possible explanations to test, not conclusions established by the current aggregate results.

  7. Temporal retrieval and actual answering. On the 78 LongMemEval-S update questions, BM25 ranks the later answer-bearing session first in 89.7% of cases, compared with 50.0% for each dense retriever. What explains this difference, given that BGE-M3 has higher Recall@1 than BM25 on the full 500-question set? Separately, Table 7(b) directly supplies old or current sessions instead of using each retriever's actual selections. Can the authors evaluate answers using those retrieved sessions, so that retrieval errors can be linked to downstream answering errors?

  8. What the temporal evaluation metrics mean. Appendix A.1 states that answers are often closer to the stale answer when both sessions are supplied. How is this closeness calculated? Table 7(b) measures whether the current answer string appears in the response. Please distinguish correctly adopting the updated information from merely mentioning its answer string, including responses that also endorse the outdated answer. This distinction is needed to assess whether the model resolves the old/new conflict.

  9. Scope of the conclusions. What new finding does this audit establish beyond prior work on misleading retrieved content and retrieval usefulness? The main downstream experiments each use 100 GSM8K questions, and the paired memories are constructed question–solution texts. Please explain which findings should extend to memories from agent interactions. Additional tasks, distinct error types, or larger independent question sets would strengthen that claim; otherwise, the conclusions should be limited to the tested settings. Repeated generation seeds should be distinguished from additional independent questions.

  10. Prompts and reproducibility materials. Please provide complete prompt templates and filled examples for candidate generation and validation, query paraphrasing and validation, the five memory-context conditions, and old/current evidence injection. Please map each table to its prompts, model versions, and sampling settings. If these details are in the supplementary materials described in Appendix I, please identify the exact files and summarize the essential construction steps in the appendix.

  11. Figure presentation. Figure 1 on page 2 has misaligned elements and text extending beyond graphical boundaries. Figure 2 on page 6 has misaligned y-axis labels, and Figure 3 on page 14 has numerical annotations overlapping the axes. Please correct these layout problems.

Reviewer_LtLKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper offers a useful controlled decomposition of retrieval relevance, memory correctness, and downstream utilization, but the contribution is primarily diagnostic, the semantic result is inconclusive, and the evidence depends heavily on a synthetic construction with substantial generator dependence. It constructs two deterministic GSM8K corruption variants and evaluates all test questions with three retrievers. A frozen 100-question response bank replays five ordered contexts and compares semantic and random Top-1 using stored singleton responses. The main bank shows a substantial harm from corrupted memory, whereas cross-generator and naturally written error evaluations reveal that this harm depends on the generator and error surface.

Critical strengths

1.A clean decomposition of memory failure. The paper separates retrieval relevance, episode correctness, and downstream utilization through matched counterfactual pairs and a frozen response bank. This is consequential because it localizes failures that end-to-end accuracy alone would conflate. 2.Careful validation beyond one synthetic result. The study combines full-split retrieval experiments, multiple retrievers and generators, naturally written incorrect solutions, and temporal updates in LongMemEval-S. This breadth makes the central finding more credible than a single benchmark demonstration.

Critical weaknesses

1.Limited novelty. The central observation that semantic relevance does not imply factual or outcome reliability is already intuitive and closely related to prior work on counterfactual RAG and credibility-aware retrieval. The paper offers a useful audit, but no new retrieval or verification method, which limits its technical contribution. 2.Heavily on a narrow construction. The primary corrupted memories alter the final number while retaining most of the original derivation, sometimes creating detectable inconsistencies, and both candidates repeat the exact query. The paper itself shows that removing this duplication changes Good@1 by 9–13 points. The smaller natural-error and temporal studies do not fully establish that the main conclusions transfer to realistic agent memories.

Questions for authors

1.The cross-generator results reverse the effect of corrupted memory: it strongly harms Qwen3-8B but improves Llama-3.1-8B over no memory. How do the authors explain this reversal, and which conclusions are intended to hold independently of the generator? 2. Can the authors provide the corresponding downstream response-bank results with question-omitted memories? This seems necessary to determine how much of the main finding depends on the duplicated-query construction. 3.For the naturally written incorrect solutions, can the authors report pre-adjudication annotator agreement and clarify how candidate filtering affected the retained error distribution? This would help assess whether the extension meaningfully addresses the synthetic construction concern. 4. Are the Table 5 seed sets disjoint from those used in Table 3, and what other configuration differences account for the Qwen corrupted-only and semantic Top-1 changes? 5. What sequence limit and backend are used for each BGE result, and after its actual preprocessing, how many paired inputs retain the edited answer tokens or become token-identical?

Suggestion for future work: A genuinely naturalistic, end-to-end agent-memory benchmark would be valuable, but I view this as future work rather than a required addition to the present paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.