acceptodds
Withdrawn from ICLR 2027

Resolving Power: Whether a Memory Task Can See the Distinction It Claims to Test

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory benchmarks are cited to settle an architectural claim: that a system maintaining a compressive state does something a scalable verbatim index cannot. Whether a given task can see that distinction is never checked, and a task that cannot scores the two systems identically, however large, leakage-free or reliably judged it is. The natural estimator is unavailable too: a difference of two realised scores bounds the quantity from neither side, since both realisations bound their own optima in the same direction. We define a task's resolving power at access budget k as the score gap between an access-complete and an index-only oracle, give a checkable sufficient condition for blindness and a condition on cardinality at a budget and a segmentation for sightedness, and prove that a sound positive result requires a task whose subtracted optimum is bounded by construction. We then build one: a 240-character scratchpad, rewritten once per record and never shown the question, whose probe answers appear in no record at all. That bounds the index-only optimum at a budget of 1 by 1/900 by design rather than by any claim about retrievers; the carrier scores 0.695, putting resolving power at 0.694 or above against a band of 0.151 fixed in advance, while a reader with one record and unlimited chain-of-thought writes the answer on 0 of 200 probes. Sending the cardinality to 32, the bounded-width carrier holds where the access-complete arm collapses, so from cardinality 8 upward a system that sees every record scores below one that sees only a fixed-width summary. Eight recurrent checkpoints screened against a floor fixed in advance all fail it, so the architecture-level question stays not decidable at current scale. The residue is a two-clause design rule — make the answer depend on more records than the budget holds, and let it appear verbatim in none of them — shipped as a generator with 500 probes that re-derives both premises on demand.

Primary area:
datasets and benchmarks
Keywords:
memory benchmarks, evaluation theory, identification, construct validity, long-context evaluation, state-space models, benchmark design, pre-registration, differential item functioning, partial identification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_VfjJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether a memory task distinguishes fixed-width state from an index allowed to read k records per query. It defines resolving power as the gap between the two oracle optima, argues that observed model gaps cannot bound it, and derives record-cardinality conditions. On a two-record arithmetic task, an updating 240-character scratchpad scores 0.695; the authors claim a 1/900 upper bound for index@1. Increment-chain and recurrent-model experiments explore whether the result extends beyond this witness.

Critical strengths
  1. The paper identifies a real evaluation concern: single-record retrieval can be mistaken for cross-record memory, and the distinction depends on budget and segmentation.
  2. Separating observed score gaps from gaps between optimal systems is useful. The paper also reports its unsuccessful recurrent-model experiments transparently.
Critical weaknesses
  1. The motivation assumes that memory benchmarks should distinguish fixed-width state from a one-record index, without showing that this is their intended claim. The result also depends heavily on the chosen k=1 budget and record segmentation.
  2. The novelty is limited. RULER and BABILong already test reasoning over distributed facts; the paper's main additions are an oracle-score definition and relatively elementary conditions on record access.
  3. The claimed 1/900 bound is not established for the stated index-only class. The paper restricts what the reader sees but does not adequately constrain what the retriever may inspect or encode through its record choice. Answer absence alone does not bound every permitted strategy.
  4. The experiments do not support an architecture-level conclusion. The positive result uses repeated scratchpad updates, while genuine recurrent models score only 0.035 and 0.065; the long-chain comparison also gives the scratchpad more sequential computation. Evaluation stays within synthetic arithmetic tasks, without testing the diagnostic on RULER, BABILong[3], or LongMemEval[1][2], or positioning it against recent multi-step retrieval.
  5. It seems that the authors changed the official ICLR template, including the font, line height, and so on. This behavior is forbidden for a valid submission.

[1] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, ICLR 2025.

[2] LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues, arXiv 2026.

[3] BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack, NeurIPS 2024.

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_mtpfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In the current form, the paper is incomprehensible, at least to me. It doesn't clearly state what kinds of models it's considering, what they mean by memory. There are no examples, and no context for framing the results. The main body of the paper (Sec 3) starts with a sequence of definitions and proofs, again even with some effort, I just could not understand what they refer to. Almost half the references are to papers by "anonymous" which are neither published nor on arXiv, all with cryptic titles in the style of the title of this paper. So I am afraid I can't summarize the paper in any meaningful manner.

Critical strengths

See above

Critical weaknesses

While it's entirely possible that there is a line of research that I am not familiar with, and this paper makes sense to folks in that area, authors should make an effort to make the paper accessible to broader ICLR community (at least to the extent that they can understand the abstract and the statement of the result). Perhaps this problem was caused by use of AI to compress the paper, in which case could be fixed by getting comments on the draft by human readers.

Questions for authors

See above

Reviewer_xVf7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper defines a state channel as a persistent carrier whose width does not depend on the span it serves. An index channel is a store that grows with the span and is read under a per-query budget of k records. Resolving power is R_state(T;k) = S(access-complete oracle) − S(index-only oracle at budget k). Five propositions follow:

  • Prop. 1 (blindness): a task is blind if its answers are deterministic functions of the write record.
  • Prop. 2: acquittal needs construction, because condemning a task is existential while acquitting it quantifies over every k-subset and every function.
  • Prop. 3 (sightedness): a task needs some query that no k records determine under a given segmentation σ.
  • Prop. 4: a cardinality that grows with the stream satisfies Prop. 3 at every budget and chunk width.
  • Prop. 5 (non-identifiability): realised arms only lower-bound their own optima, so a difference of two realised scores bounds R from neither side. A valid lower bound needs a constructive upper bound on the index-only optimum.

The authors conclude that typical ceiling-versus-reader audits only measure the index class's internal budget spread. Differential item functioning (psychometrics) and Manski's partial identification serve as framing.

§5 builds the construction. One record gives a base value for an entity, a link record gives an offset, and the query asks for their sum. The generator rejects any probe whose answer appears verbatim in a record, which yields bound (2) ≤ 1/900. The leading side is filled by an LLM that reads each record once, in order, and rewrites a 240-character notebook whose update prompt has no question field. Controls are an oracle at k=2, a full-context arm, and a CoT reader. The paper distills a two-clause design rule (R1: more bearing records than the budget holds; R2: the answer verbatim in none) and includes a generator with 500 probes. §6 sweeps cardinality on an "increment" family, and §7 tries the protocol on state-space and RWKV models after screening eight checkpoints. Roughly 26 pages of appendix contain:

  • three order-sensitivity runs, none gradeable;
  • the SSM defects;
  • a budget-axis factorisation S = a·ι, where a is the share of bearing records retrieved and ι is the scorer's credit rate;
  • a first, confounded cardinality run;
  • a preregistration record generated from git;
  • a 115-row scorecard, much of which grades results of eight anonymous companion manuscripts.
Critical strengths

The benchmarking and evaluation questions genuinely matter. Memory systems such as Mem0 and A-MEM routinely cite benchmark wins as evidence for their architectural claims, so asking whether a given item can actually separate the two classes is the right move. Proposition 5 correctly identifies a real and common error: subtracting a reader’s score from a ceiling does not yield a valid bound, because both realizations go in the same direction. The proposed fix, bounding the subtracted optimum by construction rather than by measurement, is logical. Table 3 (p. 14) demonstrates why rule R2 is necessary. On a family in which the answer appears verbatim in a single record, a tempting 1/7200 bound is easily beaten by k=1 strategies that score 0.359, 0.485, and 1.000. The authors further make the dependence on budget and segmentation explicit and quantify it (oracle index@2 reaches 0.990 while index@1 is 0.000. The sign flips at k=8), which are details that most benchmark papers never report.

Transparency is well above the norm:

  • Criteria were frozen before data, and preregistered and post-hoc analyses are reported side by side (App. B).
  • Exact-containment scoring removes judge noise.
  • The CoT control shows the gold answer in 0 of 200 one-record traces.
  • The generator ships with a --verify mode.
  • Self-corrections are reported against the authors' own interest.

Every number I could re-derive recomputes correctly: R̂ = 0.694, the Wilson interval [0.758, 0.931], κ = 0.702, the standard-error ratios in Table 8, and the §6 margins. The §6 observation, that an incremental bounded carrier stays roughly flat while a single-pass reader collapses under repeated updates to one entity, is interesting in its own right.

Critical weaknesses

The theory is mostly definitional and contains errors, and the headline is nearly tautological.

  • Prop. 1 is false as written (p.3). §5's answer (base + offset) is a deterministic function of the write record, yet the paper claims R > 0 at k=1. The proof ("an index-only system can compute any function of the write record") ignores the budget that Def. 2 makes essential. The statement also claims identical score distributions for any two systems and zero mutual information, while the proof shows only that two optima are equal.
  • Table 13's caption contradicts Prop. 5 (p.26). It calls S_ceil − S_oracle "the measured upper bound on R_state". By Prop. 5 that difference is a lower bound if S_oracle is the exact optimum, and no bound at all otherwise.
  • The headline is almost guaranteed. At k=1 with one record per chunk, a sum whose operands sit in two records cannot be produced from one record. So (3) reduces to "an LLM with a notebook adds two numbers it saw in sequence 70% of the time". The 0.151 band passes whenever S(state) > 0.152.
  • The setting is narrow, with the advantage vanishing at k=2 or under coarser chunking, which is how deployed retrieval memory actually operates (k ≥\geq 5, multi-sentence chunks). The witness streams are also short: k=8 covers 57% of the transcript (p.6), so about 14 records.
  • Bound (2), the "by construction" step, is under-specified. The joint distribution of base, offset and gold is never given. Table 20, row 40f (p.32) says "900 is the gold vocabulary", which would let a reader holding the base record narrow the sum.
  • The strategy class shifts between sections. Def. 1 and Table 16 count records read. §5 (p.7) counts records "exposed to the reader". Table 3 scores a two-hop "read the link record, then fetch the value record" strategy as k=1. Under that permissive class, a retriever can read both operands and encode the sum in which record it exposes. Bound (2) is therefore a claim about retrievers after all. The practical effect is small, but (2) is not the proof it is advertised as.

The empirical interpretation is weak and the models are unnamed.

  • §6 has a serial-compute confound. The abstract says "a system that sees every record scores below one that sees only a fixed-width summary". But the carrier gets one LLM call, and one addition, per record. The full-context arm must sum up to 31 increments "in a single forward pass with no scratchpad of its own" (p.8). No full-context + CoT arm is run (Table 4 covers only index@1 and the carrier, on the §5 family), and n = 60 per rung. "A realized access-complete arm is not an upper bound" is Prop. 5's own premise, not a new finding.
  • No model is named. No LLM is identified anywhere for the carrier, full-context, reader, extract or CoT arms. There are no prompts and no decoding settings, only "local weights or one pinned model" (Table 11, p.23). Table 15 (p.27) lists 0 result files for the cot, extract, oracle and ceiling arms whose numbers are reported.
  • §7 overstates its screen. "Eight recurrent checkpoints … all fail it" includes two that never loaded and one disclaimed as a harness failure (Table 7, p.18). Ceilings of 0.340 and 0.535 on adding two given operands, together with the admitted left-padding defect (p.19), point to the harness rather than the architecture. "A fortieth" (p.9) should be about 1/68.

The novelty claim is overstated. "Never checked" ignores:

  • the multi-hop shortcut literature: Min et al. 2019; Chen & Durrett 2019; Trivedi et al. 2020 (DiRe); Trivedi et al. 2022 (MuSiQue);
  • long-context work that probes beyond retrieval: Goldman et al. 2024; Vodrahalli et al. 2024 (Michelangelo); Modarressi et al. 2025 (NoLiMa). The paper runs an E_NOLIMA experiment (Table 12) but never cites NoLiMa;
  • theory of fixed-state versus copying/recall: Jelassi et al. 2024; Arora et al. 2023 (Zoology); Merrill et al. 2024
Questions for authors
  1. Which LLM(s), prompts and decoding settings implement the carrier, full-context, reader, extract and CoT arms? Is the scratchpad update prompt task-aware, for example instructed to track entity values and offsets?
  2. State the joint distribution of base, offset and gold, and the exact index-only strategy class under which (2) is a proof. How does this reconcile with Table 3 counting a two-hop strategy as k=1?
  3. Does the §6 crossover survive if the full-context arm may reason step by step or keep its own scratchpad at m = 8, 16 and 32?
  4. What is R_state for the §5 family under realistic segmentation (multi-record chunks) and k = 5-10? If it is ≤\leq 0 there, what does the design rule give deployed benchmarks beyond "make cardinality exceed k"?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.