acceptodds
Under review as a conference paper at ICLR 2027

When Stale Facts Win: Why Language Models Lose Context in Corrections, and How to Measure, Predict, and Fix It

Abstract

If a context mentions a fact and later explicitly corrects it, a language model should respond with the new value. Often the old value remains in effect. We investigate this failure as a distinct ability to track updates and distinguish it from the mere retrieval of information. A causal audit first examines, for each model, whether attention-based sentence rankings are in fact necessary and sufficient for the response. A pattern emerges: the models focus on the correction but still select the outdated value. Using StaleBench, we measure this behavior across different context lengths, update types, and model sizes. The results show that update tracking neither reliably scales with model size nor is predicted by standard longcontext tests. A position swap also provides evidence that smaller models often follow a positional heuristic; for large models, this finding regarding the mechanism remains tentative. Common prompt and decoding methods do not robustly resolve the problem. We therefore evaluate a training-free context pruning method that removes outdated value-binding sentences before generation. It significantly improves the responses, provided that the conflicting statements are detected. It is this detection that proves to be a central bottleneck on natural Wikipedia text. This paper does not offer the familiar recommendation to delete old values, but rather a causally verified measurement and diagnostic procedure, a benchmark for update tracking, and an explicit analysis of the assumptions and limitations of context-based repairs. In practice, this implies that agent stores and retrieval systems should resolve conflicting versions prior to generation and test update tracking separately from information retrieval.

Primary area:
datasets and benchmarks
Keywords:
Update Tracking, StaleBench, Knowledge Conflicts, Causal Attention Auditing, Context Pruning, Agent Memory

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_nocwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents StaleBench to evaluate how LLMs handle stale facts in context. It first conducts a casual audit analysis showing the model attends to the corrected fact while can still get misled by outdated information in the context. Then it evaluates several LLMs of different families and scales, showing how different models may rely on different mechanisms in handling stale facts. The paper also studies several context-side repairs and discusses the limitations.

Critical strengths
  1. The paper explores a timely topic in how LLMs handle stale facts and presents a benchmark evaluating several models of different scales;
  2. The paper presents several analyses that try to investigate the mechanism from different perspectives;
  3. The Casual Audit analysis is appreciated;
Critical weaknesses
  1. As one of the major contributions, current writing of section 3 (The Causal Audit) is extremely dense and hard to follow; the author should at least provide concrete formulation or implementation details in appendix for a grounded understanding, especially for readers who may not have a substantial background in interpretability; similar when presenting the countermeasures, the paper has zero description of the CAD method and is not self-contained.
  2. Some major claims need to be re-scoped or have limited generality:
  • Section 4.1 claims one of the major structural findings of StaleBench that "performance does not increase monotonically with model size". While this is true for Qwen2.5-1.5B and Qwen2.5-0.5B (the other comparison between Llama3.3-70B and Qwen2.5-32B is weak due to a substantial difference in model family), this does not strongly support the intended claim, especially with the limited model scale, this can be easily undermined by the other comparisons between Qwen2.5-0.5/1.5B and 3B/32B; without testing with more model families/scales, this single inversion may just be specific to tiny Qwen models under 3B;
  • Section 4.3 states that "Table 3 presents three tasks in identical contexts", conflicting with what the caption of Table 3 says "Three tasks in the same context window"; the paper also did not provide any details on the "identical context" or the implementation of the three tasks, making the results of Table 3 hard to interpret;
  • Section 5 directly compares Llama-3.3-70B with smaller models from Qwen2.5 to conclude about "position swap"; this comparison, again, is inconclusive based on the huge gap between Llama and Qwen model families; as "a hypothesis about different strategies, not as a proven scaling effect", it should not be listed as one of the contributions in the introduction;
  1. Limited evaluation and the choice of evaluated models makes the comparison results (section 4 & 5) hard to interpret. The experiments only establishes claims around model scale, if any, with Qwen2.5 models of 4 scales (ranging from 0.5B to 32B). The rest of choices of Llama-3.2-3B, Phi-4-mini, and Llama-3.3-70B provides no direct comparison due to a significant difference in model families. This weakness is closely related to the limited generality point above. The reviewer strongly recommends adding at least one more model family such as Qwen3 or Gemma4 for a more diverse and comprehensive evaluation to strengthen the major claims.
  2. Novelty of the findings are limited: the major insight about how model generation can be constrained by context information, despite of explicit instructions or rejections, is already explored in multiple existing works [1, 2, 3]. While the paper revisited it under a knowledge-editing setting and provided several additional analyses, it does not provide more effective mitigations than context-control methods, which are already explored in [2].
  3. Limited connection to agent memory: the paper does not establish any results related to agent memory or retrieval systems or have a minimal relevant setup in the evaluations while having two claims about "First, agent memory should not append corrections, but rather delete or over-write old values" and "Second, retrieval systems should not assemble isolated snippets with conflicting values; a correction plus an old value without context is the measurably worst form of context" in the introduction. These should be marked as informal implications instead.

  1. Huang, Y., Bie, B., Na, Z., Ruan, W., Lei, S., Yue, Y., & He, X. (2025). Understanding the Anchoring Effect of LLM with Synthetic Data: Existence, Mechanism, and Potential Mitigations. arXiv preprint arXiv:2505.15392.

  2. Cheng, Y., Zhu, X., Zhao, H., & Arora, S. (2026). Contextual Drag: How Errors in the Context Affect LLM Reasoning. arXiv preprint arXiv:2602.04288.

  3. Simhi, A., Barez, F., Tutek, M., Belinkov, Y., & Cohen, S. B. (2026). Old habits die hard: How conversational history geometrically traps llms. arXiv preprint arXiv:2603.03308.

Questions for authors
  1. Do similar observations of section 4.1 and 5.1 hold for models of the same family and different scales such as Qwen 3 and Gemma 4?
  2. How is the mechanism of LLM in handling stale facts significantly different from existing work such as anchoring bias and contextual drag?
  3. What are some quantitative evidences that show the downstream impact of StaleBench on agent memory and retrieval systems?
Reviewer_WZgCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how language models respond to factual updates when outdated information remains in context. Through attention-based analysis and sentence masking, the authors observe that models can assign the highest attention rank to a correction while answering with an outdated value. They introduce StaleBench to evaluate update tracking across context lengths, update formulations, update counts, and distractors. Results show non-monotonic performance with model size, improved accuracy with longer contexts for some models, and update failures despite high retrieval accuracy. Position-swapping experiments examine reliance on sentence order versus explicit update markers. The authors also evaluate a context-pruning repair that removes outdated statements using positional or update-marker rules. Experiments on Wikipedia articles with inserted updates show similar failures in small models and improvements from repair, with lower repaired accuracy than in synthetic settings.

Critical strengths
  • The paper studies a relevant and interesting problem: language models may return outdated values despite explicit corrections in context, which can affect their reliability in evolving conversations and documents.
  • The study examines this behavior through several complementary experiments, including attention-based analysis, controlled evaluations on StaleBench, position-swapping interventions, and evaluations on Wikipedia articles with inserted updates.
  • The paper shows that a simple repair that removes statements identified as outdated improves accuracy on both StaleBench and the Wikipedia-based evaluation, illustrating the benefit of context cleanup when outdated information can be identified.
Critical weaknesses

The novelty appears limited.

Prior work has examined interference from outdated values [a], benchmarks for following updated instructions and preferences [b], and removal- and retention-based tests of attribution faithfulness [c]. The paper’s distinction lies mainly in its targeted analysis of factual corrections and the conditions under which context pruning succeeds or fails, which appears to offer an incremental contribution.

The presentation is difficult to follow.

The paper often reads as a catalogue of experiments, with implementation details introduced before the motivating questions and analysis procedures are clearly explained. For example, Section 3 moves directly into attention aggregation and masking configurations without first explaining the audit’s logic and how it supports the subsequent claims. Clearer links between the hypotheses, experiments, and conclusions are needed throughout the paper.

Several empirical claims are not sufficiently supported by the reported results.

Section 5.4 states that attention masking and text deletion produce nearly identical results, but provides neither a quantitative comparison nor a clear evaluation scope. Section 6 attributes the reduced repair performance on Wikipedia text to incomplete detection of outdated statements, whereas Table 7 reports only answer-level performance. A comparison with oracle masking of all outdated mentions would be helpful.

The relevance of the findings to stronger models remains unclear.

In Table 2, Qwen2.5-32B and Llama-3.3-70B already achieve 99% and 91% accuracy, respectively. The 70B model also achieves 92% update-tracking accuracy in Table 3, but is evaluated on only 24 examples there and six examples in Table 4. Neither model is evaluated in Tables 6–7. These results leave open whether the reported failures largely diminish with model scale, limiting the broader significance of the proposed diagnosis and repair.


[a] Wang, Chupei, and Jiaqiu Vince Sun. “Unable to Forget: Proactive Interference Reveals Working Memory Limits in LLMs Beyond Context Length.” ICML 2025 Workshop on Long Context Foundation Models, 2025.

[b] Qian, Yuli, et al. “Do LLMs Forget What They Should? Evaluating In-Context Forgetting in Large Language Models.” International Conference on Learning Representations, 2026.

[c] DeYoung, Jay, et al. “ERASER: A Benchmark to Evaluate Rationalized NLP Models.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.

Questions for authors

See weaknesses

Reviewer_vXCaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how language models handle conflicting facts when outdated information is explicitly corrected within the same context. The problem is relevant to long-context reasoning and agent memory systems. However, I have several concerns regarding the novelty, causal analysis, and experimental evaluation.

  1. The related work overlooks several recent studies on outdated and conflicting knowledge, including HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation (ACL 2025), When Facts Change: Temporal Knowledge Conflict Resolution in LLMs (Findings of ACL 2026), and Tracking the Limits of Knowledge Propagation: How LLMs Fail at Multi-Step Reasoning with Conflicting Knowledge (EACL 2026). Although these studies consider different conflict settings, they are relevant to the problem investigated here. The authors should discuss these works and clarify how StaleBench differs from existing benchmarks, particularly in its treatment of conflicting information within the same context.

  2. The causal analysis does not fully support the proposed explanation. The authors argue that models attend to corrected facts but still select outdated values. However, high attention to a sentence does not necessarily imply that the model has correctly interpreted its temporal or semantic relationship with other statements. The masking interventions establish the importance of certain contextual evidence, but they do not fully explain why outdated values dominate the final prediction. Stronger analysis is needed to distinguish failures of semantic update understanding from positional bias and competition between conflicting values.

  3. The benchmark design limits the generalizability of the findings. StaleBench relies heavily on synthetic numerical facts and predefined update patterns. The position-swapping experiment is also potentially ambiguous, since moving the correction before the original statement may change how the temporal order of the facts is interpreted. The authors should consider more diverse factual relationships and explicitly timestamped updates. The Wikipedia evaluation is useful, but it still relies on artificially inserted corrections and includes only 103 examples.

  4. Several conclusions are stronger than the experimental evidence supports. The claim that longer contexts improve update tracking may be affected by changes in the relative positions and separation of conflicting statements rather than context length itself. These factors should be controlled independently. Furthermore, the two largest models are evaluated on only 100 examples, and the 70B position-swapping result is based on just six examples. These results are insufficient for drawing meaningful conclusions about model scaling or differences in update-tracking strategies.

  5. The proposed repair method offers limited methodological novelty. Removing outdated statements before generation is a straightforward strategy whose effectiveness largely depends on correctly identifying conflicting facts. The near-perfect results on synthetic data rely on controlled detection, while performance drops substantially on natural text. More importantly, the authors do not compare against stronger conflict-resolution approaches, such as explicit fact extraction, temporal reasoning, or two-stage verification. An end-to-end evaluation with realistic conflict detection is needed to establish the practical value of the proposed method.

Critical strengths

None

Critical weaknesses

None

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.