- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how the retrieval budget required to locate answers in long-term memory scales with record length. Fixing the task, the retriever (BM25 over events), the scorer (lexical containment of the gold answer), and a threshold, the authors define a boundary function k*(L) and its exponent α ∈ [0,1]. On deletion-built nested records from LongMemEval, the pooled exponent is ≈1.01 over a 62× length ladder (required share ≈0.40–0.42), refuting their pre-registered sublinear hypothesis, and exponents differ across item quantiles. A second corpus and an undeleted 10× lever give different exponents, and the authors argue that an exponent is identified only as a (task, lever, retriever) triple, with distinguishing the levers requiring 12–54× more items.
Critical strengths
The paper examines retrieval behavior across multiple record lengths, pre-registers its hypotheses, and reports uncertainty (bootstrap intervals) and potential confounding factors transparently, including a refuted hypothesis. This careful reporting is commendable, but it does not compensate for the weaknesses below.
Critical weaknesses
-
Questionable significance and limited novelty. The motivation for treating single-pass top-k retrieval as a fundamental constraint of long-term memory systems is weak. Modern agents can perform adaptive retrieval, repeated searches, and structured queries, so the practical implications of the proposed scaling analysis are unclear. Moreover, the observation that retrieval requirements depend on the task, retriever, and data construction is largely expected. The main theoretical results follow directly from the definitions (e.g., α ∈ [0,1] is essentially a consequence of k ≤ L) and provide limited new insight.
-
Weak construct validity and a narrow protocol. The experiments primarily measure whether BM25 retrieves an event containing the literal answer string, rather than whether sufficient evidence is available to answer the question. This is a poor proxy for actual memory retrieval and reasoning difficulty. The authors acknowledge that the exponent is measured only on the lexically reachable subset, but this directly undermines the headline claims: with a single lexical retriever and a literal-match scorer, the paper studies a narrowly defined retrieval protocol yet draws conclusions framed around broader memory scaling behavior.
Additional note on presentation: the manuscript has noticeable formatting and typography inconsistencies, and the figures are overly simplistic and insufficiently polished. The writing is sometimes repetitive and unnecessarily elaborate. Although the paper discloses the use of generative AI for substantial manuscript rewriting, the final presentation would benefit from more careful editorial review.
Overall, I do not find the problem formulation sufficiently motivated or the technical and empirical contributions compelling enough for acceptance.
Questions for authors
- (Affects assessment) Do the exponent estimates and their quantile ordering hold with a dense retriever, or with iterative/adaptive retrieval (multiple queries per item)? If α changes substantially, the main conclusions would appear to be properties of BM25 rather than of the task.
- (Affects assessment) How does the required share change if an item counts as reached when the annotated evidence needed to answer the question is retrieved, rather than when an event contains the literal answer string?
- (Clarification) Concretely, which evaluation or system-design decision would change given α ≈ 1 under this protocol? Please state the practical implication for deployed memory systems that use adaptive retrieval.