acceptodds
Under review as a conference paper at ICLR 2027

MEMORY AS A CACHE: EXACT CONTEXT REUSE AND DELETION BY CONSTRUCTION

Abstract

The KV cache of a transformer entangles every token's representation with its entire prefix: a passage encoded once cannot be reused under a different prefix, and cannot be removed without recomputing everything after it. Exact cache reuse in serving is therefore limited to shared prefixes, and systems that reuse more must approximate. We present SMem (static memory), an architecture whose context representation is a cache by construction. A block-local encoder maps each block of tokens to memory rows independently of all other blocks; a reader conditions generation on the union of those rows through cross-attention. Three properties then hold for every parameter setting: memory composes exactly at fixed block indices, deleting a block is exact and is an memory update for -token blocks, and the memory state is independent of the edit path. What the cache buys is measured directly. At the training context, under the shared training recipe, SMem still retrieves planted needles beyond any window of the trained length (exact-match accuracy – at distances 31 and 63 blocks) where a learned-position transformer, a RoPE transformer, a Block-Attention-style two-stream arm, and windowed or saturated readings of the transformers all score at most ; with two identical keys it returns the nearer one's value. Because reader computation is block-local, a fully cached context is served by computing one block alone (exact up to floating-point rounding) at a near-constant – ms against the same model's cold prefill, which grows with context; batched decode holds – fewer KV rows and runs – faster than the transformer where decode is bandwidth-bound, and end-to-end deletion beats suffix recomputation by at 512 blocks, rising to at 4096 blocks; both are – the trained length and probe the cost model rather than a served regime. What it costs is a perplexity gap of to (negative favors SMem) against a parameter-matched transformer with the same positional scheme, at 160M–1.5B on FineWeb-Edu across two training recipes and a learning-rate search; the band's lower endpoint is set by the recipe and its upper by the learning-rate search, and the recipe's change moves the transformer twice as far as it moves SMem at 160M. SMem also composes with RoPE: the composite leads both the matched transformer and SMem on every seed pair, closing – of SMem's distance to a RoPE transformer, which leads SMem by –. Context representation entangled with the full prefix is a design choice; an architecture that drops that entanglement reaches comparable perplexity and gains a cache that composes, deletes, and edits exactly and, under the shared recipe, retrieves at range where every transformer variant we test does not.

Primary area:
foundation or frontier models, including LLMs
Keywords:
static memory, KV-cache reuse, prefix caching, block-local encoding, cross-attention memory, exact deletion, machine unlearning, context editing, long-context retrieval, length extrapolation, efficient LLM serving, language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sot3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SMem. Each block of bb tokens is encoded independently into memory rows, and a reader combines block-local self attention with cross attention to the rows of earlier blocks. Exact composition, O(b)O(b) deletion, and path independence follow from the masking pattern. Models from 160M to 1.5B are trained on FineWeb-Edu with T=1024T=1024 and B=16B=16, under two recipes. I think the experimental design of this paper needs improvement, and the use of AI throughout the paper is too obvious and heavy, making it difficult for me to establish a complete logical chain for the core idea.

Critical strengths

The paper directly tests whether block independent context encoding can be trained from scratch at moderate scale. It includes several relevant controls: a Block-Attention-style arm, a CEPE-style reader, and windowed transformer readings.

Critical weaknesses
  1. Many propositions and formulas bear little relation to the main idea and confuse the logic. Proposition 1, Theorem 1, and Lemma 1 restate what block-local masking already implies. The containment argument, the round elimination sketch, and the permutation chain testbed in Section 5 are not connected to SMem's perplexity, retrieval, or serving behavior.

  2. I think this manuscript shows heavy AI traces, and its logic is hard to follow. Sentences stack parenthetical qualifications and numerical ranges, terms such as roll gap and retrievable-rare stratum lack clear definitions, and paragraphs switch between recipes and scales without signaling the comparison.

  3. The evidence does not establish SMem as a practical long-context architecture. Training uses 1024 tokens, and there is no RULER or LongBench evaluation. Under the tuned recipe, SMem trails the matched transformer by 1.7% to 2.8% and the RoPE transformer by 7.4% to 8.5%, and its retrieval beyond the window falls to near zero.

Questions for authors
  1. Under the tuned recipe, what is the perplexity gap at 1.5B, and does far-range retrieval survive at that scale?
  2. With a transformer given the same DD and LL depth split as SMem, how much of the decode gain remains?
  3. How does SMem perform on RULER or LongBench at contexts beyond 4096 tokens?
  4. How do the transformer baselines perform at 4×4\times length after brief fine-tuning on longer contexts?
Reviewer_vVtLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Exact reuse of a transformer's KV states is confined to contexts that share a prefix verbatim, and removing one passage invalidates the states of everything after it. SMem partitions the context into blocks and encodes each block to memory rows with attention masked to within the block, then conditions generation on the union of those rows through cross-attention from a reader whose self-attention never crosses a block boundary. Because a block's rows do not depend on its neighbors, composition at fixed indices, deletion, and path independence hold for every parameter setting, and a fully cached context is served by reading the final block alone. Matched-parameter pretraining at 160M to 1.5B measures what dropping prefix entanglement costs against a transformer on the same positional scheme, and needle probes measure retrieval beyond the trained length.

Critical strengths
  1. The structural claims are checked rather than asserted, and one control attributes the quality. Composition, deletion, and path independence follow from the attention mask and hold for every parameter setting, and the numerical gates verify them to machine precision on random parameters and on trained checkpoints. The CEPE-style arm on the same memory at the same parameters and depth split trails SMem by 9.7% with its roll gap falling to 0.12 from 0.85-0.97, so what carries quality is the block-local reader together with the memory and not the memory alone. This is the property the retrofit neighbors lack and it is what the paper is for.

  2. The retrieval evaluation is built to kill alternative explanations. The recipe axis replicates the separation at short range, the positional-scheme variant keeps it once its offsets are capped, and windowed and sink-plus-window readings of the transformer arms read 0.00 beyond 15 blocks. A Pythia-410M trained on two orders of magnitude more tokens also reads 0.00 at twice its context, and the two-key probe returns the nearer value at 0.65 against 0.01, so the separation at distances of 31 blocks and beyond is reach rather than re-indexing.

Critical weaknesses
  1. The three serving numbers in the abstract do not sit in one system. The near-constant 3.1-6.2 ms block-skip is measured at 1k to 8k tokens and the paper attributes its flatness to the launch-bound regime, while the same read inside the delete-then-generate cycle at 4096 blocks lands in a 125-126 ms total at 1.5B. The 1.4-1.7x decode gain is the contiguous layout and the flat deletion curve is the index-mapped one, under which decode runs at 0.55-0.74x the transformer's throughput and no layout in the implementation gives both. The 8.5-452x deletion ratio runs at 512 and 4096 blocks against a trained length of 16 blocks, which the paper says probes the cost model rather than a served regime. Report the serving claims as two alternative operating points at a length the model was trained for.

  2. Both favorable endpoints come from the same recipe. Regime 1 has no warmup, a choice the paper says was fixed during SMem pilots, and it is where SMem reaches -4.7% perplexity and 0.14-0.28 exact match at distances of 31 and 63 blocks. Under the tuned recipe the same needle grid reads 0.01-0.11 across three seeds against the transformer's 0.00-0.02, and the symmetric learning-rate search the paper ran at 160M leaves the perplexity contrast at +2.4 to +2.8% with the best arm still 4.8% to 5.5% behind a RoPE transformer. State both claims at the tuned-recipe endpoints, and put the reach claim on the composite, whose 0.14 at 410M is a single seed.

Questions for authors
  1. The delete-then-generate grid runs at 512 and 4096 blocks. What is the ratio at the trained length of 16 blocks, which the deletion latency sweep should already contain.

  2. Under the index-mapped layout decode runs at 0.55-0.74x the transformer's throughput. Is there a layout or kernel in the implementation that holds both the flat deletion curve and the 1.4-1.7x decode gain, or should the serving claims be stated as two alternative operating points.

  3. Under the tuned recipe the far-range exact match is 0.01-0.11 and the composite reads 0.14 at 410M on one seed. Can the retrieval claim be stated on the composite, and how many seeds does that arm have at 410M.

  4. The symmetric learning-rate search leaves the contrast at +2.4 to +2.8%. Should the perplexity claim be stated at that contrast rather than as a band whose lower endpoint comes from the recipe fixed on SMem pilots.

  5. Contribution 5 says the reader-round requirement is met exactly by trained models. The grid at width 4 and below searched only k rounds and above, so the floor is not tested from below there, and k=6 formed in zero of 88 cells. Which cells support the exact reading.

Reviewer_UWoiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SMem, an architecture that enables exact context reuse and deletion through independent block encoding and cross-attention-based reading. The idea is interesting, but I have significant concerns about the presentation and the practical implications of the claimed contributions.

Critical strengths

The paper presents an interesting architecture for exact context reuse and deletion. The separation of block-independent encoding and context-dependent reading is conceptually appealing, and the structural guarantees are supported by theoretical analysis and numerical verification.

Critical weaknesses
  1. Poor writing and organization. The manuscript is verbose and difficult to follow. The abstract alone attempts to summarize nearly every theoretical claim, experimental result, and implementation detail. Throughout the paper, numerical results, qualifications, and repeated claims of exactness are densely interleaved, making the paper hard to understand. I strongly recommend a substantial rewrite rather than incremental polishing.

  2. Limited scope of the exactness guarantees. The original architecture guarantees exact deletion only when the remaining blocks retain their positional indices. This differs from ordinary sequence editing, where subsequent content shifts position. Although the RoPE variant partially addresses this limitation, the distinction is insufficiently emphasized relative to the paper's broad claims.

  3. Insufficient practical evidence. The strongest long-range retrieval results depend on training configurations, while the tuned RoPE Transformer still achieves lower perplexity. Furthermore, the fastest deletion and decoding results rely on different memory layouts. The paper has not convincingly demonstrated that these advantages can be realized simultaneously in practical workloads.

Questions for authors

The manuscript suffers from substantial clarity and organization issues. Several terms and experimental configurations are introduced without sufficient explanation at their first occurrence, including "primary/replication harness," "saturated readings," "width substitution," etc. Important methodological definitions are frequently deferred to the appendices, making the main text difficult to follow independently.

Furthermore, the paper repeatedly uses vague expressions such as "reads" to report different evaluation metrics. Combined with the excessive density of numerical results and qualifications, this significantly undermines readability. I recommend a substantial restructuring of the manuscript, with clear definitions and consistent terminology.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.