acceptodds
Under review as a conference paper at ICLR 2027

Sorted by Date: Knowledge-Update Scores Cannot Tell Dates from Position

Abstract

Knowledge-update benchmarks are used to compare assistants and measure the benefit of memory updates. We show that a routine choice in evaluation code can undermine both conclusions: presenting evidence chronologically places the current statement after the superseded one. Time reading and position following then earn the same score, even when averaged over time-ordered layouts. We identify this confound in three update settings across six audited memory benchmarks. Paired exchanges of time labels or positions preserve input length and format, yet expose opposite cue preferences among eleven reader settings with similar native scores. Four LongMemEval readers differ by just 1.4 percentage points natively but by 16.9 points when averaged over five random orders; the lowest native scorer ranks first. Placing the most relevant evidence last puts the superseded session last on 27-44% of questions; GPT-4o-mini loses 23 points overall with lexical retrieval. Even an oracle update note shows no significant native gain across five readers. Its gain is 11.7 points larger when the superseded session is last (95% CI [6.3, 17.1]; exploratory paired analysis). Four memory systems extend the diagnosis to arrival order and stored dates. We recommend retaining native accuracy, adding random-order accuracy split by the competing pair's orientation, and using paired exchanges as diagnostics.

Primary area:
datasets and benchmarks
Keywords:
evaluation methodology, benchmark validity, long-term memory, knowledge update, LLM evaluation, position bias, identifiability, cue conflict

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iuztOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies a confound in LLM knowledge-update benchmarks. Since evidence is usually presented chronologically, temporal recency and positional recency are essentially indistinguishable. Therefore, the authors perform controlled interventions by exchanging the timestamps or positions of competing statements to test whether models rely on temporal information or positional shortcuts when resolving knowledge conflicts. Experiments across several benchmarks, LLMs, and memory systems show different cue preferences. On this basis, the authors suggest randomizing evidence order and reporting controlled exchanges in future benchmarks.

Critical strengths

S1. The evaluation problem is interesting and relevant. The concern for the reliability of existing memory benchmarks is meaningful.

S2. The paper evaluates a diverse set of LLMs and memory systems under time- and position-exchange interventions.

S3. Based on the evaluation results, the authors make practical suggestions for future benchmark design.

Critical weaknesses

W1. The writing is very difficult to follow. Many specialized terms, such as "reader", "native accuracy", "aligned layout", "inversion control", and "pair members", are used before the underlying concepts are explained. In many cases, these technical terms are not defined at all. The introduction involves six benchmarks, eleven reader settings, and four memory systems without an accessible overview of their roles. Many long and complicated sentences contain a lot of numerical numbers and obscure relatively simple ideas.

W2. Despite extensive notation, the paper never gives a self-contained definition of the knowledge-update task. What are the inputs and expected outputs? How are conflicting statements identified? For example, Sec 4 introduces "air members" and "eight sessions retrieved under the official BM25 contract" without explaining either. The relationship among the six audited benchmarks, the three used for controlled exchanges, and the four memory systems also needs to be stated explicitly. The authors should also use a concrete running example to explain the setup throughout.

W3. The core idea of this paper is a sensitivity analysis comparing behavior when timestamps and positions are exchanged. The central claim that similar native accuracy hides different cue preferences is supported mainly by selected comparisons. How strongly do native accuracy and position preference correlate across models and benchmarks? Does the relationship vary with model capability, benchmark difficulty, or context length? A quantitative correlation analysis with confidence intervals would be very valuable here.

W4. The presentation of the experiment results is very confusing. The main figures compress many model labels, conditions, intervals, and abbreviations into small plots. Figures 3 and 4 combine multiple comparisons without clear statements of conditions or takeaways. The text also alternates between absolute accuracy, accuracy differences, and cue-preference statistics. Please consider substantially reorganizing the experiment section around explicit research questions, adding readable tables of the main results, and simplifying the figures.

W5. Several confounding factors are not adequately controlled. The paper reports that reasoning changes cue preference, but configurations differ across these settings. For example, DeepSeek-Flash thinking-enabled and -disabled settings have very different token budgets. In addition, the exchanged timestamps can conflict with other relevant temporal expressions in the text, and the original reference answer is still retained even when the modified timestamps change which statement is chronologically correct. The identifiability experiments cover only three of six audited benchmarks, and the LongMemEval paired analysis uses only 49 questions. This largely limits model-specific conclusions. It would also be helpful to check whether the observed preferences are specific to the temporal-positional conflicts, or reflect a broader prompt sensitivity, such as formatting, few-shot examples, option ordering, etc.

Questions for authors

See weaknesses.

1 comment

Thanks for the additional explanations. However, my concerns regarding the presentation and clarity remain. I encourage the authors to fully revise the paper. I'll keep my current rating.

Reviewer_vZbeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is complete AI slop, end-to-end. I doubt the authors even read the paper. I tried to read the introduction and could not follow a single argument. The writing is complete slop. Reviewing this paper is a waste of researchers' valuable time, and I strongly suggest a desk rejection for this paper.

If the AC manages to see this review before the other reviewers start reviewing the paper, I recommend asking them not to review it so they don't waste their time.

If the authors want a review, they can use AI to review their paper (just as they did to write it). For a conference, this is completely unacceptable.

Critical strengths

none

Critical weaknesses

This paper is complete AI slop

Questions for authors

why?

Reviewer_kPU5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Knowledge-update benchmarks often present evidence in time order, so the current statement also appears later in the context. This paper asks whether the resulting scores can tell time reading from position following, and how this affects model comparisons and the measured benefit of memory updates. The study focuses on update questions where explicit time cues are provided. Of the six audited benchmarks, three print such a cue and score updates: LongMemEval and STALE print session dates, and MemoryAgentBench numbers its facts. The authors audit six benchmarks and test eleven reader settings on LongMemEval, STALE, and MemoryAgentBench. They exchange either the time labels or the positions of two conflicting statements, keeping the input length, format, and final slot unchanged. Every answer is scored against the original reference answer. The score with time labels exchanged minus the score with positions exchanged is called position preference. They also use random orders, date deletion, bootstrap, and Holm correction. For four memory systems, they separately change arrival order and stored dates to examine what happens during ingestion. The paper reports that readers with similar native accuracy show opposite cue preferences, and that evidence order changes model rankings and the measured benefit of an update signal.

Critical strengths

The paper identifies a specific evaluation confound: under time-ordered layouts, the same score can come from reading dates or following positions. The paper states this formally: under aligned layouts, and under any average over them, a time rule and a position rule earn the same accuracy. The exchange experiments reveal opposite cue preferences among readers with similar native accuracy. On STALE, GPT-4o-mini and GPT-5.6 Sol score 0.942 and 0.989 under the native layout, yet their position preferences are +0.642 and -0.758. This supports a diagnosis of the evaluation design, but does not identify the models’ internal mechanisms.

The paper uses controlled experiments to show the confound’s practical effects: evidence order changes model rankings and the measured benefit of an update signal. Four readers differ by only 1.4 percentage points under the native layout, but by 16.9 points when averaged over five random orders. These are LongMemEval readers on 70 knowledge-update questions, and the lowest native scorer, Qwen3-235B, ranks first among the four under random orders. An oracle note gives an 11.7-point larger benefit when the superseded statement is placed last than under the native layout, with a 95% CI of [6.3, 17.1]. The 11.7-point figure is pooled over five readers; under the native layout alone, the note’s gain ranges from -2.9 to +2.9 points, with all intervals including zero. This result is exploratory.

Critical weaknesses

The memory-system results combine effects from storage, retrieval, and answering, so the sources of failure within the memory systems are not fully separated. When Mem0 2.0 retrieves the gold value under both conditions, GPT-4o-mini still falls from 37/40 to 30/40 correct under the arrival exchange, and the paper notes that the retrieved context and its ordering can still contribute.

Questions for authors
  1. Exchanging dates can conflict with expressions such as “last month.” Only 14 LongMemEval items have no relative time expressions. These 14 are out of 49 eligible LongMemEval items, and the paper reports that the date-following preferences shrink on them. Position exchanges and serial-number experiments support the finding, but would the effect sizes hold on a larger sample without these conflicts?
  2. The four systems ran with their default configurations and GPT-4o-mini as the only writer model. How far can findings from default configurations and one writer model be generalized?

Comments on the paper

1 comment
Area_Chair_8qyvOpenReview ↗

Discussion Period

Dear Reviewers,

The authors have posted their response. Please take a moment to review it and engage in discussion with both the authors and fellow reviewers, especially on any remaining concerns or points of disagreement. Brief updates to your assessment would be very helpful.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.