- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks whether a memory task distinguishes fixed-width state from an index allowed to read k records per query. It defines resolving power as the gap between the two oracle optima, argues that observed model gaps cannot bound it, and derives record-cardinality conditions. On a two-record arithmetic task, an updating 240-character scratchpad scores 0.695; the authors claim a 1/900 upper bound for index@1. Increment-chain and recurrent-model experiments explore whether the result extends beyond this witness.
Critical strengths
- The paper identifies a real evaluation concern: single-record retrieval can be mistaken for cross-record memory, and the distinction depends on budget and segmentation.
- Separating observed score gaps from gaps between optimal systems is useful. The paper also reports its unsuccessful recurrent-model experiments transparently.
Critical weaknesses
- The motivation assumes that memory benchmarks should distinguish fixed-width state from a one-record index, without showing that this is their intended claim. The result also depends heavily on the chosen k=1 budget and record segmentation.
- The novelty is limited. RULER and BABILong already test reasoning over distributed facts; the paper's main additions are an oracle-score definition and relatively elementary conditions on record access.
- The claimed 1/900 bound is not established for the stated index-only class. The paper restricts what the reader sees but does not adequately constrain what the retriever may inspect or encode through its record choice. Answer absence alone does not bound every permitted strategy.
- The experiments do not support an architecture-level conclusion. The positive result uses repeated scratchpad updates, while genuine recurrent models score only 0.035 and 0.065; the long-chain comparison also gives the scratchpad more sequential computation. Evaluation stays within synthetic arithmetic tasks, without testing the diagnostic on RULER, BABILong[3], or LongMemEval[1][2], or positioning it against recent multi-step retrieval.
- It seems that the authors changed the official ICLR template, including the font, line height, and so on. This behavior is forbidden for a valid submission.
[1] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, ICLR 2025.
[2] LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues, arXiv 2026.
[3] BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack, NeurIPS 2024.
Questions for authors
Please refer to the Critical Weaknesses section.