- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies a confound in LLM knowledge-update benchmarks. Since evidence is usually presented chronologically, temporal recency and positional recency are essentially indistinguishable. Therefore, the authors perform controlled interventions by exchanging the timestamps or positions of competing statements to test whether models rely on temporal information or positional shortcuts when resolving knowledge conflicts. Experiments across several benchmarks, LLMs, and memory systems show different cue preferences. On this basis, the authors suggest randomizing evidence order and reporting controlled exchanges in future benchmarks.
Critical strengths
S1. The evaluation problem is interesting and relevant. The concern for the reliability of existing memory benchmarks is meaningful.
S2. The paper evaluates a diverse set of LLMs and memory systems under time- and position-exchange interventions.
S3. Based on the evaluation results, the authors make practical suggestions for future benchmark design.
Critical weaknesses
W1. The writing is very difficult to follow. Many specialized terms, such as "reader", "native accuracy", "aligned layout", "inversion control", and "pair members", are used before the underlying concepts are explained. In many cases, these technical terms are not defined at all. The introduction involves six benchmarks, eleven reader settings, and four memory systems without an accessible overview of their roles. Many long and complicated sentences contain a lot of numerical numbers and obscure relatively simple ideas.
W2. Despite extensive notation, the paper never gives a self-contained definition of the knowledge-update task. What are the inputs and expected outputs? How are conflicting statements identified? For example, Sec 4 introduces "air members" and "eight sessions retrieved under the official BM25 contract" without explaining either. The relationship among the six audited benchmarks, the three used for controlled exchanges, and the four memory systems also needs to be stated explicitly. The authors should also use a concrete running example to explain the setup throughout.
W3. The core idea of this paper is a sensitivity analysis comparing behavior when timestamps and positions are exchanged. The central claim that similar native accuracy hides different cue preferences is supported mainly by selected comparisons. How strongly do native accuracy and position preference correlate across models and benchmarks? Does the relationship vary with model capability, benchmark difficulty, or context length? A quantitative correlation analysis with confidence intervals would be very valuable here.
W4. The presentation of the experiment results is very confusing. The main figures compress many model labels, conditions, intervals, and abbreviations into small plots. Figures 3 and 4 combine multiple comparisons without clear statements of conditions or takeaways. The text also alternates between absolute accuracy, accuracy differences, and cue-preference statistics. Please consider substantially reorganizing the experiment section around explicit research questions, adding readable tables of the main results, and simplifying the figures.
W5. Several confounding factors are not adequately controlled. The paper reports that reasoning changes cue preference, but configurations differ across these settings. For example, DeepSeek-Flash thinking-enabled and -disabled settings have very different token budgets. In addition, the exchanged timestamps can conflict with other relevant temporal expressions in the text, and the original reference answer is still retained even when the modified timestamps change which statement is chronologically correct. The identifiability experiments cover only three of six audited benchmarks, and the LongMemEval paired analysis uses only 49 questions. This largely limits model-specific conclusions. It would also be helpful to check whether the observed preferences are specific to the temporal-positional conflicts, or reflect a broader prompt sensitivity, such as formatting, few-shot examples, option ordering, etc.
Questions for authors
See weaknesses.
Thanks for the additional explanations. However, my concerns regarding the presentation and clarity remain. I encourage the authors to fully revise the paper. I'll keep my current rating.