acceptodds
Under review as a conference paper at ICLR 2027

Answer Error Forgetting: Disentangling Memory from Temporal Reasoning over Continual Knowledge Streams

Abstract

Long-term memory is essential for language models operating over continual knowledge streams, where information is presented sequentially as knowledge accumulates, is updated, or is superseded over time. Recent long-term memory benchmarks extend beyond factual recall to evaluate capabilities such as knowledge updating and temporal reasoning over long histories. However, these capabilities are typically assessed through end-to-end question answering, making it difficult to determine whether an incorrect answer reflects a memory-related failure or insufficient temporal reasoning. We introduce DiMeR, a diagnostic benchmark that presents knowledge units sequentially and evaluates the same temporal questions under controlled evidence conditions. DiMeR contains 1,956 temporal question–answer pairs constructed from 674 Wikipedia articles, each associated with source-linked gold evidence in the corresponding knowledge stream. We further divide the questions into Easy and Hard groups according to whether the temporal cues expressed in the question are explicitly present in the knowledge stream. For each question, we construct five core evaluation conditions that vary when and how the supporting evidence is available. To more directly separate memory-dependent degradation from reasoning limitations, we additionally evaluate memory conditions on questions that the same model answers correctly when the relevant evidence is provided directly. Beyond answer accuracy, we measure evidence completeness and introduce Refresh-4K, which reintroduces the original gold evidence after continued knowledge ingestion. Experiments show that Hard questions are more sensitive to continued ingestion and that similar answer accuracy can correspond to different levels of evidence completeness. Evidence reintroduction also does not always recover the correct answer, indicating that answer correctness alone is insufficient to characterize long-term memory performance.

Primary area:
datasets and benchmarks
Keywords:
Long-Term Memory, Temporal Reasoning, Continual Knowledge Streams, Evidence Recovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BdgLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces DiMeR, a benchmark for temporal question answering over sequential knowledge streams. It constructs 1,956 questions from 674 Wikipedia articles with source-linked reference evidence. The same question, answer, and evidence are compared under no-evidence, direct-evidence, immediate, and delayed conditions. An additional diagnostics condition on direct-evidence success reintroduces the reference evidence after delayed ingestion. Answer accuracy and final citation coverage are jointly analyzed across base models and adapted memory backends.

Critical strengths
  1. The matched conditions are useful: they hold the historical question and target fixed while changing the input prefix. The appendix specifies state isolation and independent reconstruction, preventing earlier questions or answers from contaminating later conditions. Conditioning on the same model's direct-evidence successes also reduces one important source of confounding.

  2. Jointly inspecting answers and citations provides useful behavioral information. In particular, Appendix F.2 shows faithful quotation of the reference evidence alongside a temporally incompatible answer. The paper also includes cardinality/relation analyses and paired uncertainty estimates, and explicitly acknowledges the limits of interpreting citations and reinjection as internal mechanisms.

Critical weaknesses
  1. The reported results cannot currently be reconciled across tables and declared sample sizes. All three memory systems have different Immediate-Memory accuracies in Tables 2 and 9; for example, Mem0 Hard is 35.96% versus 44.06%. Appendix E.2 states that every model uses 1,219 Easy and 737 Hard questions, but several percentages, including Mem0's Table 9 Immediate values, cannot arise from any integer correct count under these denominators and the defined accuracy metric. Table 8's Easy direct-evidence-success counts also differ from those implied by Table 2, and Table 12 reports Qwen/GLM Easy gains of 2.47/1.81 rather than Table 9's 2.38/1.72.

  2. The diagnostic measurements need stronger validation for the claimed separation. Final citation coverage does not independently measure retention or evidence sufficiency, and reinjection changes recency, repetition, and memory processing. I appreciate that E.2 explicitly acknowledges this: the supported contribution is a behavioral diagnostic, rather than an identification of internal failure mechanisms. Its validity also depends on the reference targets. B.1 removes month/day precision while retaining one answer; F.3 uses an interval-target convention for a 2009–2010 question despite a July 2010 team change. That convention and acceptable alternatives are not operationally specified. Fact-level verification and a few inspected cases do not quantify ambiguity or citation fidelity across the benchmark.

Questions for authors
  1. Can you reconcile Tables 2, 8, 9, and 12 using explicit question IDs, effective denominators, exclusions, and run/config identifiers? Please explain whether different releases, technical recoveries, or averaging are involved, and regenerate the direct-evidence-success subsets and paired statistics from a consistent set of predictions.

  2. Which observable failure categories does DiMeR distinguish, and which remain unidentifiable? Please specify the year-level interval/answer convention and treatment of multiple valid answers and alternative sufficient evidence, and provide a quantified human audit of these targets and citation fidelity. Item-level answer–coverage transitions would also help establish what the reinjection diagnostic adds beyond aggregate differences.

Reviewer_BmroOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces DiMeR, a benchmark for investigating why LLMs fail at temporal QA after continuously reading new information. It constructs ordered knowledge streams from Wikipedia articles and evaluates models under controlled evidence conditions, measuring answer accuracy and evidence citation coverage. Experiments on several LLMs and memory systems show that continued reading can impair temporal QA performance, while reintroducing gold evidence does not consistently recover correct answers.

Critical strengths

S1: Meaningful research question. Errors after continued reading do not necessarily mean that a model has forgotten the relevant facts. Distinguishing this from failures in temporal reasoning is an important research question.

S2: Complementary evaluation. Evaluating evidence coverage alongside answer accuracy provides a more nuanced assessment of model failures than answer accuracy alone. The evidence refresh condition further tests whether reintroducing the original evidence helps recover correct answers.

Critical weaknesses

W1: Knowledge-unit segmentation quality. The paper uses an LLM to merge sentences into knowledge units but does not report an independent audit or quantitative assessment of the resulting segmentation. The manual checks of evidence annotations do not establish the reliability of the unit boundaries themselves.

W2: Evidence coverage does not establish evidence use. EH, ER, and AR measure the overlap between cited units and gold evidence, but cannot determine whether the cited evidence actually informs the final answer or is merely retrieved or mentioned during reasoning. Full evidence coverage therefore does not establish that the answer is derived from the cited evidence.

W3: Uncontrolled relevance of continued-reading content. The authors acknowledge that appended content is not independently verified to be irrelevant to the question. Performance degradation may therefore reflect interference from related or competing facts, rather than difficulty accessing earlier evidence alone.

W4: Limited isolation of experimental factors. Direct-Evidence removes the surrounding reading context, while Refresh-4K reintroduces the gold evidence at the end of the input. These conditions also differ in context and evidence recency or repetition, making it difficult to attribute performance changes solely to evidence availability.

W5: The Easy/Hard split captures cue explicitness, not difficulty alone. The split is based on whether the queried time point or both interval endpoints appear in the knowledge stream. This is a reasonable way to distinguish explicit from implicit temporal cues, but it does not directly measure reasoning difficulty. Differences between the groups may also reflect other factors, such as evidence requirements or relation types.

Questions for authors

Q1: Knowledge-unit quality. Did the authors conduct any independent manual audit or quantitative assessment of the LLM-generated knowledge-unit boundaries? The reported manual checks concern evidence annotations, but it is unclear whether the segmentation and merging decisions themselves were validated.

Q2: Evidence use. Citing gold evidence does not necessarily mean that it supports the final answer. A model may cite the correct evidence while relying primarily on other contextual information or its parametric knowledge to produce the answer. Did the authors assess whether the cited evidence actually influences the final answer, for example, by testing whether removing or altering it changes the answer?

Q3: Continued-reading content. Have the authors analyzed whether the appended articles contain facts relevant to the target questions or competing temporal information? If so, how do the results change when such cases are accounted for? This would help clarify whether the observed degradation can be attributed to continued reading alone.

Reviewer_vx8HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DiMeR, a benchmark with 1,956 temporal QA pairs. It compares direct evidence, continued reading, and evidence reintroduction to examine memory and reasoning failures. The main finding is that answer accuracy and evidence coverage do not always change together.

Critical strengths

The focus on why an answer goes wrong is interesting. Looking at evidence coverage alongside accuracy gives a more useful picture than accuracy alone.

Testing the same questions across conditions is a sensible design. Conditioning on Direct-Evidence success also helps account for questions the model cannot solve even with the evidence.

Critical weaknesses

I’m not fully convinced that the setup separates memory from reasoning. The evidence metrics measure final citations, while full-context models still receive the earlier evidence. Missing a citation does not establish forgetting.

Some reported results need reconciling. For example, Mem0’s Immediate-Memory accuracy on Hard questions is 35.96% in Table 2 but 44.06% in Table 9. That is a substantial difference for the same condition, making the memory-system comparisons difficult to assess.

Questions for authors

Which runs and question sets produced the different Immediate-Memory results in Tables 2 and 9?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.