acceptodds
Withdrawn from ICLR 2027

The Infinite-Record Limit: A Retrieval Budget That Stays Constant for the Hardest Items and Vanishes for the Median

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-context and memory benchmarks routinely claim that a task requires reaching deep into a record, but that claim is about a retrieval budget, and whether a budget counts as small depends on how long the record is. No evaluation measures how the required budget scales as the record grows, so a claim established at one length is silently carried to another; and because no fixed-corpus lever moves record length alone, even a measured scaling may belong to the lever rather than to the task. That confound is not hypothetical: it produced the field's most expensive scaling disagreement, between two well-known compute-optimal scaling laws for the same nominal resource. We make the scaling the estimand. Pinning the retriever, the scorer and the task and sending record length to infinity exposes a boundary function — the budget at which the unreachable share of items falls below threshold — and its exponent alpha in the unit interval: an exponent below 1 makes "a small budget" meaningful, an exponent of 1 makes it empty. On deletion-built nested records the pooled exponent is 1.011 over a 62x ladder, with the required share stuck at 0.400 to 0.417, refuting the sublinear scaling we had frozen in advance; the exponent differs at every quantile, and that distribution is the finding — the hardest few per cent of items demand a constant share of the record while the median item's demand vanishes. The shape transports to a second corpus as unlike the first as our data allows, but the constant does not, and an undeleted 10x lever reproduces neither the exponents nor their ordering, with neither version wrong: an exponent is identified only as a (task, lever, retriever) triple, and telling the two levers apart is not decidable at this benchmark's item counts, needing 12 to 54 times the items. The reporting rule with a proof behind it is to publish an exponent with the lever and the retriever that produced it; the one unconditional result is that the required share does not vanish as the record grows.

Primary area:
datasets and benchmarks
Keywords:
long-context evaluation, retrieval-augmented generation, scaling laws, identification, benchmark evaluation, memory benchmarks, pre-registration, evaluation methodology, retrieval budget, LongMemEval

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_8KrSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how the retrieval budget required to locate answers in long-term memory scales with record length. Fixing the task, the retriever (BM25 over events), the scorer (lexical containment of the gold answer), and a threshold, the authors define a boundary function k*(L) and its exponent α ∈ [0,1]. On deletion-built nested records from LongMemEval, the pooled exponent is ≈1.01 over a 62× length ladder (required share ≈0.40–0.42), refuting their pre-registered sublinear hypothesis, and exponents differ across item quantiles. A second corpus and an undeleted 10× lever give different exponents, and the authors argue that an exponent is identified only as a (task, lever, retriever) triple, with distinguishing the levers requiring 12–54× more items.

Critical strengths

The paper examines retrieval behavior across multiple record lengths, pre-registers its hypotheses, and reports uncertainty (bootstrap intervals) and potential confounding factors transparently, including a refuted hypothesis. This careful reporting is commendable, but it does not compensate for the weaknesses below.

Critical weaknesses
  1. Questionable significance and limited novelty. The motivation for treating single-pass top-k retrieval as a fundamental constraint of long-term memory systems is weak. Modern agents can perform adaptive retrieval, repeated searches, and structured queries, so the practical implications of the proposed scaling analysis are unclear. Moreover, the observation that retrieval requirements depend on the task, retriever, and data construction is largely expected. The main theoretical results follow directly from the definitions (e.g., α ∈ [0,1] is essentially a consequence of k ≤ L) and provide limited new insight.

  2. Weak construct validity and a narrow protocol. The experiments primarily measure whether BM25 retrieves an event containing the literal answer string, rather than whether sufficient evidence is available to answer the question. This is a poor proxy for actual memory retrieval and reasoning difficulty. The authors acknowledge that the exponent is measured only on the lexically reachable subset, but this directly undermines the headline claims: with a single lexical retriever and a literal-match scorer, the paper studies a narrowly defined retrieval protocol yet draws conclusions framed around broader memory scaling behavior.

Additional note on presentation: the manuscript has noticeable formatting and typography inconsistencies, and the figures are overly simplistic and insufficiently polished. The writing is sometimes repetitive and unnecessarily elaborate. Although the paper discloses the use of generative AI for substantial manuscript rewriting, the final presentation would benefit from more careful editorial review.

Overall, I do not find the problem formulation sufficiently motivated or the technical and empirical contributions compelling enough for acceptance.

Questions for authors
  1. (Affects assessment) Do the exponent estimates and their quantile ordering hold with a dense retriever, or with iterative/adaptive retrieval (multiple queries per item)? If α changes substantially, the main conclusions would appear to be properties of BM25 rather than of the task.
  2. (Affects assessment) How does the required share change if an item counts as reached when the annotated evidence needed to answer the question is retrieved, rather than when an event contains the literal answer string?
  3. (Clarification) Concretely, which evaluation or system-design decision would change given α ≈ 1 under this protocol? Please state the practical implication for deployed memory systems that use adaptive retrieval.
Reviewer_dM8JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the retrieval budget scales with record length. Specifically, the authors define k*(L), the smallest budget at which BM25 finds the answer for 95% of items, and its exponent alpha. The authors also prove that on a fixed corpus, no lever can change the length alone. They find that (1) on nested LongMemEval records built by deletion, the hardest items need a constant share of about 40% of the record, while the median item’s share vanishes; (2) with the short and long files as the two lengths, the exponents differ, and the ranking of question types is reversed. On InfiniteBench, the share again does not vanish, but is lower.

Critical strengths
  1. Studying how the retrieval budget scales with record length is commendable, which facilitates the understanding of what memory benchmarks actually test.

  2. The authors show that the short and long LongMemEval files share almost no sessions, which is useful for other studies that use both files.

Critical weaknesses
  1. In abstract, the authors claim that “the hardest few per cent of items demand a constant share of the record”. However, in Appendix E, this result comes from only 35 items. Since k* allows only 1 of them to miss, each value is decided by the second-hardest item. The pooled values are even the same as the temporal-reasoning row of Table 4, which has 4 items, and the caption calls Table 4 “the thing to register next, not a result”. Therefore, I think this claim is not that convincing. The authors need to add confidence intervals to show that the share is not decided by one or two items. Also, the main text should say that the headline numbers come from these 35 items.

  2. From my understanding, k* only takes power-of-two values, and each record length is about twice the previous one. Is this correct? If so, a slowly vanishing share could give the same k* values as a constant one. In Appendix L, the authors say that with power-of-two lengths, a constant share is the default, not a finding, and that the main corpus “escapes it only by accident”, since its lengths are not powers of two. However, I don’t see why lengths that nearly double each time would behave differently. One possible way would be to report the interpolated pooled share and exponent with an interval, as Appendix I does for each dimension.

Questions for authors
  1. BM25 is the only retriever tested apart from a random baseline, and Section 3 notes that random ranking already gives alpha = 1. So is “the one unconditional result” in the abstract about the task, or about BM25? Results with at least one dense retriever should be added.

  2. Proposition 2 says the share vanishes for any alpha below 1. In Section 4, the second corpus still counts as not vanishing because its interval [0.893, 1.070] is above the floor of 0.80, although it includes values below 1. Which values of alpha count as not vanishing?

  3. [Missing references] Zhang, Xinrong, et al. "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens." arXiv preprint arXiv:2402.13718 (2024).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.