- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether retrievers in episodic memory systems can distinguish a correct past episode from a near-duplicate one with a wrong conclusion. Using GSM8K, the authors pair each official solution with a copy whose final answer is altered, and evaluate BM25, bge-m3, and Qwen3-Embedding-4B on which episode is ranked first. They then feed fixed memory contexts (none, correct only, corrupted only, both) to Qwen3-8B and Llama-3.1-8B to measure the downstream effect, and extend the analysis to naturally written incorrect solutions and LongMemEval-S. The results show that dense retrievers frequently rank the corrupted episode first, and that semantic top-1 retrieval is not significantly better than random selection. Based on these findings, the paper argues that retrieval selection, memory correctness, and generator utilization should be reported as separate quantities in episodic-memory evaluation.
Critical strengths
- The paper highlights that relevance and correctness can diverge in episodic memory retrieval, which is an increasingly important issue as LLM agents rely on stored past experiences.
- The counterfactual memory pairs isolate the effect of the final conclusion while keeping the task and most of the derivation fixed, and the frozen response bank allows retrieval policies to be compared without confounding from additional generation sampling.
- The use of question-level clustering, paired tests, transition matrices, and matched random and oracle baselines makes the reported differences more interpretable than end-task accuracy alone.
Critical weaknesses
W1. The main claim is unclear and the paper is difficult to follow
- The central conclusion is essentially that retrieval selection, memory correctness, and generator utilization should be reported separately. While reasonable, this recommendation is fairly intuitive and not by itself a substantial contribution.
- The more concrete findings (e.g., dense retrievers barely reflect a changed conclusion when the question is duplicated; corrupted memories can even help weaker generators) are not elevated into clear takeaways.
- No method for reliability-aware retrieval is proposed or validated. The suggestions in Appendix F (reliability signals, conflict-aware retrieval) remain untested.
W2. Inconsistent results across tables for the same setting
- Table 3 and Table 5 both report Qwen3-8B on the same 100 frozen questions, yet the numbers differ substantially: corrupted-only accuracy is 56.5% in Table 3 but 73.0% in Table 5. A 16.5 pp gap in the corrupted condition is difficult to attribute to the number of seeds alone.
- Similarly, the "exact query + full memory" control in Table 2 should match the bge-m3 setting in Table 1, but the values differ (31.5 vs. 32.1 for final-only, 36.0 vs. 39.8 for structured). Since embedding computation is deterministic, the paper should explain what differs between these runs (e.g., backend, preprocessing, corruption variant).
- These discrepancies raise concerns about the reliability of the reported numbers and the consistency of the experimental pipeline.
W3. No downstream evaluation with bge-m3 as the retriever
- bge-m3 is the most striking case in the retrieval audit, as it ranks the corrupted episode first more often than chance (32.3% and 39.9% Pairwise AUC). However, all downstream experiments appear to use Qwen3-Embedding-4B (or leave the retriever unspecified, as in Table 5 and Table 6b).
- The query ablation in Table 2 is conducted only with BGE-M3, while the downstream replay uses a different embedding model, so the two analyses are not directly connected.
- Since the response bank already stores both singleton responses, bge-m3-based semantic Top-1 can be computed without any additional generation. This analysis should be added.
W4. Conflicting results are not reconciled with the paper's narrative
- In Table 5, corrupted memory raises Llama-3.1-8B accuracy from 69.0% to 90.0%. This suggests that the final-answer edit is a weak corruption This undermines the general claim that corrupted memories are harmful and questions whether the corruption mechanism captures realistic unreliable memories.
- Section 7.5 argues that presenting both episodes enables recovery (98.5% / 96.5%). However, with naturally written incorrect solutions (Table 6b), presenting both yields only 43.0% / 42.0% for Qwen3-8B, far below verified-only (77.0%). The recovery effect thus appears limited to the synthetic setting, yet Appendix F still recommends conflict-aware multi-retrieval without acknowledging this.
- The no-memory accuracy of Qwen3-8B drops from 93.0% (synthetic bank) to 41.0% (natural-error bank), indicating a strong selection bias toward harder questions that is not discussed.
W5. Concerns of the setting
- It is not stated which corruption variant (final-only or structured) is used in the response bank and cross-generator replay.
- BGE-M3 is run with a maximum sequence length of 256. Since the only difference between e+ and e- lies at the end of the text, truncation could make the two embeddings identical.
- The rationale for disabling Qwen3-8B's reasoning mode is not given, although it may affect whether the generator detects arithmetic inconsistencies in corrupted memories.
Questions for authors
- What explains the retrieval patterns in Table 1? In particular, why does Qwen3-Embedding-4B select the verified episode more often under structured edits (66.4%) than under final-only edits (40.3%), even though structured edits change more numbers? And why does bge-m3 systematically prefer the corrupted episode ?
- Could you clarify why Table 3 and Table 5 report different numbers for Qwen3-8B on the same 100 questions?
- Would the downstream results change if Qwen3-8B's reasoning mode were enabled? A reasoning model may be more likely to detect the arithmetic inconsistency introduced by final-only corruption.
- In LongMemEval-S, why does BM25 achieve much higher Current@1 (89.7%) than both dense retrievers (50.0%)? Is there a specific reason that both dense models obtain exactly the same value?