Resolving Power: Whether a Memory Task Can See the Distinction It Claims to Test
Abstract
Memory benchmarks are cited to settle an architectural claim: that a system maintaining a compressive state does something a scalable verbatim index cannot. Whether a given task can see that distinction is never checked, and a task that cannot scores the two systems identically, however large, leakage-free or reliably judged it is. The natural estimator is unavailable too: a difference of two realised scores bounds the quantity from neither side, since both realisations bound their own optima in the same direction. We define a task's resolving power at access budget k as the score gap between an access-complete and an index-only oracle, give a checkable sufficient condition for blindness and a condition on cardinality at a budget and a segmentation for sightedness, and prove that a sound positive result requires a task whose subtracted optimum is bounded by construction. We then build one: a 240-character scratchpad, rewritten once per record and never shown the question, whose probe answers appear in no record at all. That bounds the index-only optimum at a budget of 1 by 1/900 by design rather than by any claim about retrievers; the carrier scores 0.695, putting resolving power at 0.694 or above against a band of 0.151 fixed in advance, while a reader with one record and unlimited chain-of-thought writes the answer on 0 of 200 probes. Sending the cardinality to 32, the bounded-width carrier holds where the access-complete arm collapses, so from cardinality 8 upward a system that sees every record scores below one that sees only a fixed-width summary. Eight recurrent checkpoints screened against a floor fixed in advance all fail it, so the architecture-level question stays not decidable at current scale. The residue is a two-clause design rule — make the answer depend on more records than the budget holds, and let it appear verbatim in none of them — shipped as a generator with 500 probes that re-derives both premises on demand.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.