- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents StaleBench to evaluate how LLMs handle stale facts in context. It first conducts a casual audit analysis showing the model attends to the corrected fact while can still get misled by outdated information in the context. Then it evaluates several LLMs of different families and scales, showing how different models may rely on different mechanisms in handling stale facts. The paper also studies several context-side repairs and discusses the limitations.
Critical strengths
- The paper explores a timely topic in how LLMs handle stale facts and presents a benchmark evaluating several models of different scales;
- The paper presents several analyses that try to investigate the mechanism from different perspectives;
- The Casual Audit analysis is appreciated;
Critical weaknesses
- As one of the major contributions, current writing of section 3 (The Causal Audit) is extremely dense and hard to follow; the author should at least provide concrete formulation or implementation details in appendix for a grounded understanding, especially for readers who may not have a substantial background in interpretability; similar when presenting the countermeasures, the paper has zero description of the CAD method and is not self-contained.
- Some major claims need to be re-scoped or have limited generality:
- Section 4.1 claims one of the major structural findings of StaleBench that "performance does not increase monotonically with model size". While this is true for Qwen2.5-1.5B and Qwen2.5-0.5B (the other comparison between Llama3.3-70B and Qwen2.5-32B is weak due to a substantial difference in model family), this does not strongly support the intended claim, especially with the limited model scale, this can be easily undermined by the other comparisons between Qwen2.5-0.5/1.5B and 3B/32B; without testing with more model families/scales, this single inversion may just be specific to tiny Qwen models under 3B;
- Section 4.3 states that "Table 3 presents three tasks in identical contexts", conflicting with what the caption of Table 3 says "Three tasks in the same context window"; the paper also did not provide any details on the "identical context" or the implementation of the three tasks, making the results of Table 3 hard to interpret;
- Section 5 directly compares Llama-3.3-70B with smaller models from Qwen2.5 to conclude about "position swap"; this comparison, again, is inconclusive based on the huge gap between Llama and Qwen model families; as "a hypothesis about different strategies, not as a proven scaling effect", it should not be listed as one of the contributions in the introduction;
- Limited evaluation and the choice of evaluated models makes the comparison results (section 4 & 5) hard to interpret. The experiments only establishes claims around model scale, if any, with Qwen2.5 models of 4 scales (ranging from 0.5B to 32B). The rest of choices of Llama-3.2-3B, Phi-4-mini, and Llama-3.3-70B provides no direct comparison due to a significant difference in model families. This weakness is closely related to the limited generality point above. The reviewer strongly recommends adding at least one more model family such as Qwen3 or Gemma4 for a more diverse and comprehensive evaluation to strengthen the major claims.
- Novelty of the findings are limited: the major insight about how model generation can be constrained by context information, despite of explicit instructions or rejections, is already explored in multiple existing works [1, 2, 3]. While the paper revisited it under a knowledge-editing setting and provided several additional analyses, it does not provide more effective mitigations than context-control methods, which are already explored in [2].
- Limited connection to agent memory: the paper does not establish any results related to agent memory or retrieval systems or have a minimal relevant setup in the evaluations while having two claims about "First, agent memory should not append corrections, but rather delete or over-write old values" and "Second, retrieval systems should not assemble isolated snippets with conflicting values; a correction plus an old value without context is the measurably worst form of context" in the introduction. These should be marked as informal implications instead.
-
Huang, Y., Bie, B., Na, Z., Ruan, W., Lei, S., Yue, Y., & He, X. (2025). Understanding the Anchoring Effect of LLM with Synthetic Data: Existence, Mechanism, and Potential Mitigations. arXiv preprint arXiv:2505.15392.
-
Cheng, Y., Zhu, X., Zhao, H., & Arora, S. (2026). Contextual Drag: How Errors in the Context Affect LLM Reasoning. arXiv preprint arXiv:2602.04288.
-
Simhi, A., Barez, F., Tutek, M., Belinkov, Y., & Cohen, S. B. (2026). Old habits die hard: How conversational history geometrically traps llms. arXiv preprint arXiv:2603.03308.
Questions for authors
- Do similar observations of section 4.1 and 5.1 hold for models of the same family and different scales such as Qwen 3 and Gemma 4?
- How is the mechanism of LLM in handling stale facts significantly different from existing work such as anchoring bias and contextual drag?
- What are some quantitative evidences that show the downstream impact of StaleBench on agent memory and retrieval systems?