acceptodds
Withdrawn from ICLR 2027

Two Counts over Text: What a Memory Benchmark Can See, Measured Before Anything Is Run

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A long-term memory benchmark is cited for an architectural distinction: a system that maintains a composable state against one that indexes what it was told. That citation is never checked against the benchmark's own text, although it can be. Where an item's gold answer is announced verbatim inside a single logged record, returning that one record suffices and no composition is being tested — but auditing this today means running systems, a judge and an API bill, so it happens after a benchmark ships rather than before. We build an instrument that needs no model, no judge and no API call: two counts over text and a retriever. V is the share of items whose gold answer appears verbatim in one record; R-str(k) is the share of that containment gap a budget-k retriever fails to close, and k* is the budget where it closes, the audit's one free constant swept and priced within 0.008 of optimal. A proved proposition makes the instrument one-sided: it can condemn a task as blind and can never acquit one. Run blind on four corpora, k* recovers class labels fixed by role before any dataset was named — single-hop dimensions close at 8, 8 and 16, composition dimensions do not close at 32, and a needle task returns the k* of 1 a working retriever must return. On a flagship benchmark built to be immune to context stuffing, V runs 0.55 to 0.64 on knowledge_update against 0.00 to 0.07 elsewhere: more than half its checkable update answers are announced verbatim in one turn, and across a 74x range of that benchmark's own headline axis the counts barely move. Priced on a corpus built to defeat it, V travels while k* does not, since k* also measures a reader's lexical reach. The reporting rule is one-sided too: a high V condemns, a low reader score proves nothing, and authors can publish V and k* per dimension before shipping, at the budget the baseline was allowed.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, construct validity, long-term memory, retrieval-augmented generation, evaluation methodology, memory benchmarks, pre-registration, state space models, information retrieval, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xH78OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on whether the long-term memory baseline can distinguish the system that can maintain and compose persistent states from simply retrieve information from historical records. It contains low-cost auditing method, LLM judge and API calls with several metrics, such as the proportion of gold answers, the remaining answer-coverage gap, and the minimum budget needed to reduce the gap below a predefined threshold. Experiments show that more than half of the checkable answers in the knowledge-update dimension appear in individual records.

Critical strengths

This paper does not propose a new memory framework, but investigate whether the baseline have enough construct validity, which is very important for rapidly developed LLM memory research. This method can be easily adapted into existing benchmarks, because the metrics V and Rstr are very simple without any inference of LLM. This paper remains the unsuccessful hypothesis and unrecognizable results, which is highly recommended.

Critical weaknesses

The condition of proposition 1 is too general. The paper proposes that If the answer is the function of write record, the index-only system can reach to the upper bound, which contradicts the retrieval budget constraint established in the paper. The proposed metrics are adaptable to belief questions with clear and definite answers, which is too limited. Because many tasks do not have clear answers, this paper can only be deployed in few benchmarks. The structure of this paper is so complex, which contains many preregistered criteria and analysis that obscure the core contributions and conclusions. The metric Rstr uses absolute value instead of normalization, which may lead to unfair comparison between different benchmarks. The experiments are highly relied on BM25 and last-occurence reader, without sufficiently evaluating iterative retrieval methods.

Questions for authors

Please see the weakness above to explain some details in your paper clearly. Moreover, some questions are listed as follows: If the same answer appears in many events, how to determine which events provide correct evidence? Does the answer necessarily means that the event contains sufficient information to answer the question? Please mention the length threshold used to define “short gold” answer. The conclusion of this paper is stable if the threshold is changed? What text procedures are used to calculate V?

Reviewer_hX8fOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper seems to be testing some sort of benchmark without calling a model or API. This paper is almost impossible to follow and seems to be written with AI.

Critical strengths

None

Critical weaknesses

The paper is illegible and is impossible to follow. I have no idea what this paper is trying to say, what they are trying to test, and what the figures even mean. The paper is extremely confusing and I am not sure what the authors are trying to say, or what the task is. The wording is extremely verbose while always circumventing.

I could be wrong, but it seems that this paper might be written by AI.

Questions for authors

This paper is almost impossible to follow and seems to be written with AI.

Reviewer_CugGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I have serious concerns about the quality and scholarly rigor of this submission.

  1. The paper is extremely verbose and difficult to follow. The abstract and introduction fail to clearly articulate the research problem, while the contributions are presented in lengthy, formulaic, AI-style prose without establishing a coherent technical argument. The figures, e.g, Figure 1, are also poorly designed and add little technical insight. Overall, the presentation falls well below the clarity and rigor I would expect from an ICLR submission.

  2. More concerningly, the paper cites seven anonymous companion manuscripts (Anonymous, 2026a–g), likely AI slop from the same research program. Their identifiers and repositories are withheld, yet the submission repeatedly refers to them for theoretical foundations, experimental interpretations, and supporting claims. This unusual volume of mutually referencing, inaccessible manuscripts raises serious questions about the independence and verifiability of the work. It also raises concerns about possible large-scale AI-assisted manuscript production, although I cannot independently establish how the underlying research was produced.

Given above problems, I have difficulty treating this as a sufficiently rigorous and self-contained research contribution. I would encourage the AC to scrutinize these issues, like the provenance and role of the seven companion manuscripts. In its current form, I do not believe this paper meets ICLR's standards.

Critical strengths

N/A

Critical weaknesses

See Summary

Questions for authors

Please respect reviewer's time

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.