acceptodds
Under review as a conference paper at ICLR 2027

SpMem: Sparse Attention via Memory Tokens

Abstract

Most sparse attention methods store the entire key–value (KV) cache and are sparse only in how they read it. A selector must keep every entry in case a later query wants it, so the cache still grows with the context. These methods also pay for sparsity with machinery outside attention itself: a KV block compressor, a top-k selector, and a training recipe that departs from the base model's. We present SpMem, a sparse attention that needs neither a compressor nor a selector and keeps the KV cache compressed throughout inference. On the global-attention layers, SpMem appends m memory tokens after every C real input tokens. Memory tokens are processed through a transformer forward pass, generating KV features that carry the core information of the prior input chunks. To this end, we train a model using next-token prediction with modified attention masking. During inference, real and memory tokens share one fixed retention rule, Tapered Window Attention (TWA), which keeps the most recent W tokens and halves the number retained with each further window of age. At 128K context a SpMem layer holds 16K entries against dense attention's 128K, and that number does not grow with context. On Gemma 4 12B and 31B, against a dense model trained on the same data, SpMem stays close on knowledge and long-context reasoning and the cost falls mostly on retrieval. Keeping one of the eight global layers at full attention recovers most of that retrieval with a 4.3× smaller KV cache at 12B, and puts that model at or above dense on five of seven benchmarks, MRCR-v2 among them. At 1M tokens, 8× its training length, it finds every needle in single-needle NIAH once its memory cache is widened at inference, where the dense model finds 85%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, KV cache compression, long-context language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_h3dcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets the resident KV cache in long-context decoding. It modifies only the global-attention layers of a local/global interleaved transformer. After every 8 real tokens, 1 learned memory token is appended. It is processed by the ordinary forward pass, and its keys/values act as a summary of the preceding chunk. Both real and memory tokens are retained under Tapered Window Attention. The most recent W keys are kept verbatim, and each older band of W positions keeps half as many keys, selected by absolute position. TWA is a closed-form, monotone mask, so the same rule is used in training and serving, and the per-layer cache is fixed at about 2W - 1 entries per stream. Models are obtained by continued pretraining and compared against a dense model uptrained on the same data.

With every global layer bounded, SpMem roughly preserves short-context ability and LongBench-v1 but loses substantial retrieval. Keeping one of the eight global layers at full attention recovers most of the gap at a 4.3x reduction. With the memory cache widened at inference only, that variant finds all 100 single needles at ~1M tokens. The paper also reports decode latency, memory and throughput on TPU, and extensive ablations.

Critical strengths
  1. Simple, practical design. There is no router, top-k selector or separate compressor. The only new parameters are the m x d memory-token embeddings, and the retention rule is closed-form shared by training and serving. Adoption in existing local/global models looks straightforward. This is the paper's main practical selling point. A method this cheap to adopt is valuable even with modest gains, which is why I see the paper as borderline rather than a clear reject.

  2. Clear problem framing. The distinction between methods that are sparse in what they read and methods that are sparse in what they store (Sec. 3) is useful and correctly identifies resident KV memory as the binding constraint for long-context serving. It fixes the right metric for judging the paper (resident memory, not FLOPs).

Critical weaknesses
  1. Training-schedule confound in the dense reference. SpMem variants use a 5K-step cosine schedule and are evaluated at step 5K, i.e., after full decay. The dense reference uses a 50K-step schedule and is also evaluated at step 5K, early in its decay. Comparing a fully annealed model with a mid-schedule one likely favors SpMem. A dense model trained with the same 5K-step schedule is needed. This is my main reason for not recommending acceptance. It affects every headline comparison with dense, and its direction plausibly favors SpMem.

  2. The bounded-memory claim and the competitive-accuracy claim come from different configurations. The abstract and conclusion lead with a cache that does not grow with context. The configuration that is competitive with dense (SpMem + 1 global) keeps one layer at full attention, so its cache is O(L) again. Its reduction is 4.3x at 128K and can never exceed 8x. The fully bounded configuration loses 18.2 MRCR-v2 and 35.8 NIAH points at 12B. The 1M-token NIAH result also relies on the +1 global variant. The framing should make this trade-off explicit up front. The paper's main selling point and its main accuracy result do not hold at the same time. The realized contribution is therefore a 4.3x (at most 8x) reduction, not a constant-size cache, which lowers the significance relative to the framing.

  3. The simplest hybrid baseline is missing. For the +1 global configuration, the natural baseline keeps one global layer at full attention and converts the other seven to plain sliding windows (no TWA, no memory tokens), trained identically. Table 2 includes TWA + 1 global without memory tokens, and Table 15 includes sliding window + memory tokens with K retained layers, but plain SWA + 1 full global layer is absent. Without it, the value of TWA and memory tokens over simply converting most global layers to local layers is not isolated. Without this baseline I cannot attribute the +1 global results to TWA and memory tokens rather than to the simple choice of keeping one full-attention layer. This is my second main reason for the recommendation.

Questions for authors

Q1: Can you report a dense reference trained with the same 5K-step schedule as the SpMem variants? Do the Table 1 conclusions still hold? Q2: How does SpMem + 1 global compare to plain sliding windows (no TWA, no memory tokens) on seven global layers plus one full-attention global layer, trained identically? Q3: How were the Beacon and SPLA points in Fig. 3 obtained? Were they retrained on Gemma 4 with your data, number of steps and budget?

6 comments

We thank the reviewer for the detailed feedback. We address each point below. We have uploaded a revised version of the paper incorporating these changes; results of runs still in progress will follow in a later comment and revision.

W1 / Q1. Dense trained on the same 5K-step schedule. We retrained dense with SpMem's schedule (lr 1e-5, 500 warmup, cosine over 5K, same data and optimizer).

12B, step 5KMMLUIFEvalAIMELB-v1LB-v2MRCR-v2NIAH
Dense, submitted (50K schedule @5K)69.059.933.334.642.553.295.8
Dense, matched (5K schedule)66.565.840.040.350.756.596.4
SpMem + 1 global, submitted (C8M1)70.761.943.338.639.555.390.1
SpMem + 1 global, C=128, m=4 + memory forcing (main configuration)73.463.243.338.645.953.085.6
SpMem + 1 global, budget-matched C=64, m=4(training; posted in a follow-up)

Matching the schedule strengthens dense, mostly on long-context reasoning and retrieval. Against this, the submitted SpMem + 1 global is within 1.2 on MRCR-v2 and 1.7 on LongBench-v1, ahead on MMLU (+4.2), and behind on LongBench-v2 (−11.2), IFEval (−3.9) and NIAH (−6.3).

The last rows are improvements made since submission to the same method: a coarser configuration (C=128, m=4) and memory forcing, which restricts a fraction of training sequences to their most recent real tokens so that older context must come through memory. The final recipe forces half the sequences, with every layer limited to its last 128 real tokens. Against the matched dense, it is within 1.7 on LongBench-v1 and 3.5 on MRCR-v2 at a 4.8× smaller cache; against the submitted configuration, it ties LongBench-v1, gains 6.4 on LongBench-v2, and is 2–5 points lower on retrieval. The revision uses it as the main configuration.

The revision reports all comparisons against this model. At 31B, the revision labels its rows with their configuration (C=8, m=1); the matched dense and main-configuration 31B runs are scheduled after the 12B runs, and we will post them if they finish within the discussion period.

W2. Bounded-memory claim vs. the competitive configuration. We agree that the two claims come from different configurations, and the revision states the trade-off up front, as two operating points rather than a constant cache. (i) Every global layer bounded: a constant cache per layer at any length (15,872 entries at the submitted configuration, 12,287 at the improved one; 8× and 10.7× smaller than dense at 128K), at a real cost in retrieval: against the schedule-matched dense, the submitted model loses 21.5 on MRCR-v2 and 36.4 on NIAH. (ii) One global layer kept at full attention: most of the retrieval comes back (within 1.2 of dense on MRCR-v2 at the submitted configuration), but that layer's cache grows with the context, so the reduction is 4.3× at 128K (4.8× at C=128, m=4) and, as the reviewer notes, can never exceed 8×; the 1M result also uses (ii). We recommend (ii) and present (i) as the constant-memory limit. The bounded run with the new recipe has finished, and we post it in a follow-up.

W3 / Q2. Plain sliding window on seven globals + one full-attention global. Trained identically (same data, 5K schedule, sphere projection, same kept layer), with plain sliding windows of 8K and 12K entries and, as a stronger window baseline, TWA 16K without memory (16,383 entries, W = 8K; caches are named by their size in entries, as in the revision).

12B, + 1 global, step 5KKV per bounded layerLB-v1LB-v2MRCR-v2NIAHRULER-128K
Dense, matched schedule131,07240.350.756.596.485.1
SWA 8K8,19232.828.948.473.342.8
SWA 12K12,28836.737.448.979.244.9
TWA 16K, no memory16,38337.836.751.983.246.2
SpMem, submitted (C=8, m=1)15,87238.639.555.390.165.4
SpMem, C=128, m=4 + memory forcing (main configuration)12,28738.645.953.085.662.1

Keeping one global layer and windowing the rest is not enough, so the + 1 global results are not explained by the kept layer alone. With the same full-attention layer in every row, the window baselines stay well below SpMem, which isolates what memory tokens add over spending the budget on more real tokens. At the submitted configuration's budget (about 16K entries), SpMem + 1 global beats TWA 16K + 1 global without memory on all five axes, by 19.2 on RULER-128K and 6.9 on NIAH; plain sliding windows with the same layer reach only 42.8 and 44.9 on RULER-128K at 8K and 12K entries. Since submission, with the improved training recipe (W1 / Q1 above), SpMem + 1 global at C=128, m=4 has the 12K sliding window's budget and beats it on every axis: LongBench-v1 +1.9, LongBench-v2 +8.5, MRCR-v2 +4.1, NIAH +6.4, RULER-128K +17.2. NIAH is the 8-subtask RULER mean over six lengths, as in the paper.

Q3. How the Beacon and SPLA points were obtained. Both are our reimplementations in the same training stack, on Gemma 4 12B with the same data, optimizer and batch. Beacon uses its recommended 8× configuration (1,024-token intervals condensed into 128 beacons). It differs from the original in four ways: the base model is trained rather than frozen, there are no dedicated beacon projections, the raw-token window slides across intervals, and the ratio is fixed rather than sampled. Its schedule did not match (read at 5K of a 50K schedule), and a 5K-schedule Beacon is not in this rebuttal. SPLA used SpMem's 5K schedule. The revision's appendix describes how both were adapted to our stack.

Follow-up: the every-global-bounded run with the final recipe. The run we said was in progress has finished, and the revision now fills its rows (Tables 1–3, Figure 3). Gemma 4 12B, C=128, m=4, every global layer bounded at 12,287 entries (10.7× smaller than dense at 128K, at any length):

12B, every global boundedKV per layerLB-v1LB-v2MRCR-v2NIAHRULER-128K
SWA 8K8,19231.630.319.447.116.9
TWA 8K, no memory8,19130.231.626.347.611.7
TWA 16K, no memory16,38337.729.928.562.420.3
SpMem (TWA 8K + memory)12,28731.729.919.659.729.6
SpMem + 1 global (main)12,287 (+1 full)38.645.953.085.662.1
Dense131,07240.350.756.596.485.1

With every global layer bounded, memory tokens on top of TWA 8K raise NIAH by 12.1 and RULER-128K by 17.9 for 4K extra entries, and RULER-128K exceeds TWA 16K, which holds more entries. They do not help MRCR-v2. MMLU and IFEval stay close to dense (69.6 / 63.4 vs 66.5 / 65.8). As the reviewers noted, fully bounded retrieval remains well below dense, and we present this setting as the constant-memory limit.

The memory cache can also be extended after training without a full-attention layer. At ~1M tokens, the fully bounded model finds 12 of 100 single needles with its trained memory cache and 63 with a 25K-entry memory cache swapped in at inference (32× smaller than dense across the eight global layers), against 90 for the matched dense model. So long-range recall can come through memory alone, though less reliably than through one retained layer.

Our recommended operating point therefore remains SpMem + 1 global: within 1.7 of dense on LongBench-v1 and 3.5 on MRCR-v2 at 4.8× smaller cache (7.4× at 1M), and 100 of 100 needles at 1M with the extended memory cache. Only that one layer grows with context.

Reviewer_4DBDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpMem, a sparse-attention approach that reduces KV-cache memory usage by inserting learned memory tokens after fixed-size chunks of input tokens. It introduces Tapered Window Attention (TWA), a deterministic retention policy that keeps recent tokens and progressively retains fewer older tokens, applying the same policy to both regular and memory tokens.

The method is trained through standard next-token prediction and evaluated on Gemma 4 models at 12B and 31B parameters.

Critical strengths
  • The proposed approach is relatively simple and does not require an additional learned compressor or token-selection network.

  • The deterministic TWA policy provides a predictable KV-cache budget, with consistent attention masking during training and inference.

  • The experiments cover two relatively large models and several short- and long-context benchmarks, and the authors provide detailed discussions of some limitations.

Critical weaknesses

Main Concerns

W1. Limited methodological novelty

The main ideas used in SpMem have already been explored in prior work. Inserting special memory tokens to summarize previous context has been studied in Recurrent Memory Transformer (RMT), Gist Tokens, AutoCompressor, Compressed Context Memory (CCM), and Activation Beacon. Similarly, retaining progressively fewer tokens from older parts of the context has been explored in LongNet and Cascading KV Cache, although their exact retention mechanisms differ from TWA. Even combining memory tokens with restricted attention patterns is not a new idea. SpMem mainly applies a deterministic exponential retention policy to both regular and memory tokens, while training the model to work with this pattern. Although this combination is different from previous approaches, I find its novelty rather limited, and the paper does not clearly establish what the main new technical contribution is beyond these established ideas.

W2. Lack of controlled comparisons with closely related methods

Although the paper discusses closely related approaches such as RMT, Gist Tokens, AutoCompressor, and CCM, it does not experimentally compare SpMem against any of them. Instead, most of the reported baselines are sparse-attention or training-free KV-cache eviction methods, which use rather different approaches. The authors include Activation Beacon, one of the closest methods, in Figure 3 and report better results, but the paper does not clearly establish whether the comparison uses matched base models, training data, and training budgets. This is especially important because SpMem uses full-parameter continued pretraining, whereas several previous compression methods use different or substantially cheaper adaptation procedures. Since these methods already explore using learned representations to compress previous context, more direct comparisons are necessary to demonstrate the actual benefits of SpMem. Without such experiments, it is difficult to determine whether SpMem offers a meaningful improvement over existing approaches with similar methodology.

W3. Incomplete related work and unclear positioning against retrieval-based KV-cache management

The paper categorizes existing methods into approaches that are sparse in what they read and approaches that are sparse in what they store, and argues that the former cannot address KV-cache memory limitations because they need to retain the entire cache. However, this discussion largely overlooks an important line of work on retrieval-based KV-cache management and CPU–GPU offloading, including InfLLM, RetrievalAttention, RetroInfer, MagicPIG, PQCache, Louver, HiSparse, and SPIN. Several of these methods store older KV entries in CPU memory and retrieve or process only relevant entries during inference, while others develop efficient indexing methods for sparse KV access, meaning that retaining the full history does not necessarily require keeping it in GPU memory.

Unlike SpMem, which permanently discards older KV entries, these approaches can preserve access to the original history, although they introduce additional indexing, retrieval, and data-transfer costs. The paper briefly discusses related ideas such as Landmark Attention and Memorizing Transformers, but does not adequately explain this alternative approach to addressing the GPU memory bottleneck. Given that reducing serving memory is the main motivation of SpMem, I find this omission important: the related-work discussion does not clearly establish what gap remains unresolved by these existing approaches or what advantages SpMem provides over them in terms of accuracy, GPU memory, and inference latency.

W4. Limited evidence that memory tokens improve over a larger retention window

The ablation results do not consistently show that the proposed memory tokens are more useful than simply retaining more original tokens under the same KV-cache budget. For example, in Table 2, with all global layers bounded and a budget of approximately 16K entries, TWA without memory tokens achieves 35.2 on MRCR-v2 and 64.4 on NIAH, compared with 35.0 and 60.0 for SpMem with memory tokens. The memory-token configuration performs slightly better on RULER-128K, but neither configuration consistently dominates the other. Memory tokens provide clearer improvements when one global layer is retained at full attention, particularly on RULER, but this configuration no longer has a fully bounded KV cache. I think the paper needs stronger evidence explaining why learned memory tokens are preferable to allocating the same memory budget to a larger TWA window, particularly in the fully bounded setting that motivates the method.

W5. Significant retrieval degradation and dependence on a full-attention layer

Although the paper emphasizes bounded KV-cache memory, the fully bounded configuration suffers substantial retrieval degradation, and the strongest results depend on retaining one full-attention layer. For example, at 128K context, SpMem achieves only 24.7 on RULER, compared with 78.8 for dense attention, while retaining one dense global layer improves this to 65.4 (Table 2). More importantly, at approximately 1M tokens, the fully bounded SpMem configuration retrieves only 2 out of 100 needles, even after widening its memory cache, whereas the configuration with one dense global layer and a widened memory cache retrieves all 100 (Table 3). However, retaining this layer reintroduces an O(L) KV-cache component, so the strongest retrieval configuration no longer satisfies the paper's central constant-memory property. The 1M-token result is also limited to single-needle retrieval: LongBench-v2 and BABILong remain near chance, and Appendix A.12 shows that the retrieval improvement from widening the memory cache does not transfer to the W = 16K configuration. Overall, these results raise concerns about how effectively the proposed compressed-memory mechanism preserves long-range information and how much of the reported retrieval performance comes from retaining full attention.

W6. Inference-efficiency experiments do not evaluate the final proposed configuration

One important concern is that the inference-efficiency measurements in Section 5.3 and Figures 4 and 6 were conducted before TWA was introduced. These experiments use sliding-window configurations retaining approximately 6K–12K KV entries per global layer, rather than the final SpMem configuration with approximately 16K entries, and the authors assume that the final configuration would exhibit similar performance. Furthermore, the measurements use a single TPU v5p chip, zero-initialized model weights, and a cache fast-forward procedure for long contexts instead of performing the complete prefill. The authors explain these decisions and validate some of their effects, but optimized fused attention kernels are also disabled for both models, and the results do not measure the final method in a fully optimized serving environment. In particular, there is no direct end-to-end latency and throughput evaluation of the final TWA-based SpMem configuration on a realistic GPU serving stack. Since inference efficiency is one of the paper's main contributions, I think it is important to evaluate the actual proposed configuration rather than relying on measurements from an earlier variant and assumptions about its behavior.

W7. Limited evaluation across model architectures

The experiments are restricted to Gemma 4 12B and 31B, which belong to the same model family and share a similar architecture with interleaved local and global attention layers.

W8. Concerns about training data and the fairness of accuracy comparisons

The continued-pretraining data include synthetic question-answering and multi-round retrieval examples similar to MRCR-v2, which is also one of the main evaluation benchmarks. As the authors acknowledge in Appendix A.8, continued pretraining substantially changes the base model's performance: the dense reference gains 13.5 points on MRCR-v2 but loses 9.3 points on LongBench-v1 and 7.8 points on RULER compared with the released checkpoint. Although the dense and SpMem variants use the same training mixture, this raises questions about how much the reported performance depends on the particular training distribution and whether the improvements generalize to independent long-context tasks. Another concern is that the dense reference and SpMem use different learning-rate schedules: the dense model follows a 50K-step schedule with 1K warmup steps, while SpMem follows a 5K-step schedule with 500 warmup steps, even though both are evaluated at step 5K. Therefore, their optimization states are not directly matched, which could affect the relatively small accuracy differences reported on some benchmarks. I would like to see stronger evaluation on independent task distributions and a more closely controlled training comparison.

W9. Additional training cost is not sufficiently reflected in the overall efficiency discussion

Although SpMem is presented as a relatively simple and efficient approach, Appendix A.15 reports that its attention pattern makes training more expensive rather than cheaper. In particular, the authors measure approximately 1.63× the forward/backward cost of dense attention for a global layer and estimate around 1.36× the end-to-end training cost, largely because memory tokens increase the sequence length and produce inefficient attention-mask blocks. The paper also reports no meaningful prefill speedup from the proposed sparsity. These limitations are acknowledged, but I think they deserve more emphasis when presenting the overall efficiency benefits, particularly given the substantial continued-pretraining cost required to adapt the model. A clearer discussion of the tradeoff between additional training cost, inference speed, and memory savings would strengthen the practical contribution.

Refernces

Memory tokens and context compression

Sparse attention and hierarchical KV-cache retention

Retrieval-based attention and CPU–GPU KV-cache management

Questions for authors

Please see the weakness I mentioend above.

7 comments

We thank the reviewer for the detailed feedback. We address each point below. We have uploaded a revised version of the paper incorporating these changes; results of runs still in progress will follow in a later comment and revision.

W1. Novelty. We agree that each ingredient has precedent: memory tokens (RMT, Gist, AutoCompressor, CCM, Activation Beacon) and retention that coarsens with age (LongNet, Cascading KV). We take that resemblance as support for the design rather than a claim to the pattern. Our contribution is what the combination makes possible in an attention cache. SpMem applies one closed-form retention rule to both the real tokens and the memory tokens, so the resident cache of each global layer is fixed at any context length (2W − 1 real entries plus a bounded summary pool) while a tapered sample of every raw token stays addressable, and the training mask and the serving cache are the same rule, with no compressor, selector or added attention parameters. The revision adds a table that sets these methods side by side by how the resident state grows with length, which raw tokens remain addressable, and what is trained. Only SpMem combines a fixed cache, addressable raw tokens from the whole context, and learned summaries: Cascading KV also bounds the cache but keeps no summaries and is training-free, while the memory-token methods either grow with the input or drop the raw tokens they compress. We apply it to pretrained 12B and 31B local/global models and evaluate it to 1M tokens. Since submission we also found that memory tokens need an explicit training signal in a local/global model, because the local layers otherwise answer most targets. Memory forcing, which routes a fraction of training sequences through memory, is the resulting change to the training recipe (W4).

W2. Controlled comparisons with memory-token methods. For Activation Beacon, the closest method, the comparison is matched on base model, data and batch: it is our reimplementation in the same training stack, on Gemma 4 12B with the same data, optimizer and global batch as SpMem, and it is trained with the same full-parameter procedure, so the comparison does not favor SpMem on adaptation. Beacon uses its recommended 8× configuration (1,024-token intervals condensed into 128 beacons) and differs from the original in four ways: the base model is trained rather than frozen, there are no dedicated beacon projections, the raw-token window slides across intervals, and the ratio is fixed rather than sampled. One part is not matched: Beacon was read at step 5K of a 50K-step schedule, while SpMem used a 5K-step schedule. A 5K-schedule Beacon is not in this rebuttal. Full-parameter training is a deliberate choice, and the submission tests the cheaper alternative (Appendix A.18, Table 15, on an earlier configuration; Appendix A.16, Table 14 in the submission): trained with LoRA (rank 128), the same model learns less from the same data (MRCR-v2 45.0 against 55.4 with full attention) and loses more when SpMem is switched on (MRCR-v2 −7.2 against −2.4, NIAH −12.6 against −9.2). A frozen base keeps its original abilities but does not learn to use the new attention pattern. Retraining AutoCompressor and CCM under our recipe is not feasible within the rebuttal window, so for those two the revision gives a qualitative comparison (Appendix A.1.8).

W3. Retrieval-based KV management and offloading. The revision adds an appendix section and a sentence in related work. Offloading keeps the cache O(L) and moves the cost to the host link and the index: each decode step fetches selected entries over a link far slower than accelerator memory, so per-token latency grows with what is fetched. Most of these works report throughput rather than TPOT. SpMem bounds what is resident and adds no index or transfer; the two compose (dropped entries could be offloaded instead of discarded). A roofline for Gemma 4 12B makes the cost concrete. Each global layer stores 2 KB per token (K and V, 512 dims, bf16), and there are 8 global layers. Fetching a top-k of 2K entries per layer is then 32 MiB per decode step per sequence, or 64 MiB at 4K. Over PCIe Gen4 x16 at an effective 25 GB/s that is 1.3–2.7 ms per sequence (0.7–1.3 ms at Gen5's 50 GB/s), i.e. 8–16% of SpMem's whole 17 ms step at batch 1, and it scales with the batch: 43–86 ms per step at batch 32. The fetch also sits on the critical path, because a layer's top-k is known only once its query exists. Selecting the top-k needs scores against every key, so the keys stay resident and grow with L: 1 GiB at 128K and 8 GiB at 1M, about 0.4 ms and 3.1 ms of HBM time per step just to scan them. SpMem's whole global read is a fixed 192 MiB (12,287 entries per layer), 0.07 ms at v5p's 2.8 TB/s, at any L. This is an analytic estimate, not a measurement of a specific offloading system.

W4. Memory tokens vs. a larger window at the same budget. We retrained the window baselines identically to SpMem for this response, at the submitted configuration (C=8, m=1) and its budget. Caches are named by their size in entries, as in the revision: TWA 16K holds 16,383 entries (W = 8K).

12B, step 5KKV per bounded layerLB-v1LB-v2MRCR-v2NIAHRULER-128K
TWA 16K, every global bounded16,38337.729.928.562.420.3
SpMem, every global bounded (submitted)15,87234.637.435.060.024.7
TWA 16K + 1 global16,38337.836.751.983.246.2
SpMem + 1 global (submitted)15,87238.639.555.390.165.4

With one global layer kept, memory beats the larger window on all five axes at the same budget. With every global layer bounded the result is mixed: memory wins LongBench-v2 (+7.5), MRCR-v2 (+6.5) and RULER-128K (+4.4) and loses LongBench-v1 (−3.1) and NIAH (−2.4). (Retrained identically, the bounded TWA row scores 28.5 on MRCR-v2 and 62.4 on NIAH; the submission's Table 2 listed 35.2 and 64.4 from an earlier run.) TWA 8K with no memory (8,191 entries) is the lower bracket: LB-v1 30.2, LB-v2 31.6, MRCR-v2 26.3, NIAH 47.6, RULER-128K 11.7.

Since submission: a better training recipe and configuration for the same method. The bounded gap is what memory forcing targets. Next-token prediction trains the memory tokens weakly, because most targets are predictable from the local window, so memory forcing restricts a fraction of training sequences to their most recent real tokens and older context must come through memory. The final recipe forces a random half of the training sequences, with every layer limited to its last 128 real tokens. Paired with a coarser configuration, C=128, m=4, and one global layer kept, it ties the submitted configuration on LongBench-v1 (38.6), gains 6.4 on LongBench-v2 (45.9) and is 2–5 points lower on retrieval, in a 23% smaller cache; at the same budget (12,287 entries per bounded layer) it beats a 12K sliding window with the same retained layer on every axis (LongBench-v2 45.9 vs 37.4, RULER-128K 62.1 vs 44.9, NIAH 85.6 vs 79.2, MRCR-v2 53.0 vs 48.9). The fully bounded run with this forcing, the setting the reviewer asks about, is in progress, and we will post it in a follow-up.

W5. Retrieval and dependence on the full-attention layer. We agree, and the revision states the trade-off as two operating points rather than one claim: (i) every global layer bounded, a constant cache per layer (15,872 entries at the submitted configuration), where retrieval is the weak point (RULER-128K 24.7 and MRCR-v2 35.0, against the schedule-matched dense's 85.1 and 56.5); and (ii) one global layer kept at full attention, a 4.3× smaller cache at 128K that still grows in that one layer and recovers most of the retrieval (RULER-128K 65.4, MRCR-v2 55.3). We recommend (ii) and present (i) as the constant-memory limit, not as a match for dense. With the improved training recipe and configuration (W4), (ii) is within 1.7 of dense on LongBench-v1 and 3.5 on MRCR-v2 at 4.8×, and at ~1M tokens it finds 94 of 100 needles with its trained cache (48 of the 52 beyond 512K) and all 100 once its memory cache is extended at inference, against 90 for the matched dense model. Because its bounded layers reach only 128K, this long-range recall runs through the retained layer, as the reviewer suggests; the every-global-bounded run with the new recipe is in progress and will be posted in a follow-up.

We also agree on the scope of the 1M result. It is single-needle retrieval only: at 1M, LongBench-v2 stays near chance and BABILong near its floor for the matched dense model as well (LongBench-v2 31 vs 32.2, BABILong 2–3% vs 7.6%), so these tasks do not separate the methods at that length, and we do not claim long-range reasoning at 1M.

W6. Efficiency of the final configuration. Re-measured with C=128, m=4: decode latency and HBM from 8K to 1M, prefill TTFT and training cost (Gemma 4 12B, bf16, one TPU v5p chip, batch 1, same protocol as the appendix). Decode at 1M tokens: 17.0 ms/token and 22.9 GiB peak HBM against dense's 36.6 ms and 38.7 GiB (2.15× faster, 15.8 GiB less); 1.53× at 512K and 1.04× at 128K. SpMem's latency and memory are flat from 8K to 1M (16.8–17.1 ms); with one global layer kept at full attention it is 21.1 ms and 24.9 GiB at 1M (1.74×). The crossover with dense lies between 32K and 128K; below it SpMem is up to 12% slower per token, because at short contexts dense's cache is small and SpMem still reads two caches. The new configuration is also cheaper than the submitted one at the same length (17.1 vs 17.4 ms at 128K) because its memory pool is 4,096 slots rather than 7,681. Across the batch axis at 128K the bounded cache pays off further. Dense runs out of memory at batch 32 and peaks at 248 tokens/s, while SpMem fits batch 128 and reaches 395 tokens/s (1.59×), and at batch 16 it decodes 1.31× faster. Fast-forward decode agrees with real prefill to within 1.1%. In prefill, with every global layer bounded and chunked prefill (8,192 tokens per chunk), a 128K prompt takes 15.8 s against dense's 26.1 s (1.65× faster), while a single-shot 32K prefill is 23% slower (3.15 s vs 2.57 s), the cost of partially masked attention blocks. Training cost is in W9. Zero-initialized weights do not change these costs, which depend on tensor shapes and cache size rather than weight values. Both variants use the same standard attention path; at batch 1 the fused kernel gives dense no gain (18.7 vs 18.66 ms at 128K).

W7. One model family. Correct; all results are Gemma 4. We acknowledge this limits the claim and plan a second model family for the camera-ready version.

Reviewer_fiQVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SpMem, a memory-token-based approach to reducing KV-cache storage in pretrained local/global transformers. It inserts one memory token after every eight real tokens and applies Tapered Window Attention (TWA) separately to real-token and memory-token histories. This bounds the KV storage of each compressed global layer without a separate compressor or selector. The model is adapted through continued next-token pretraining.

Critical strengths

The paper presents a clear bounded-storage attention design with consistent training and inference masks, demonstrating substantial KV-cache reductions on large pretrained models. Matched-budget ablations also reveal a useful interaction between memory tokens and a retained full-attention layer.

Critical weaknesses
  1. The practical benefits of the proposed cache design are not yet clear.

TWA permanently drops parts of the history, potentially losing information that later queries need. When all global layers use this scheme, retrieval accuracy drops substantially. Keeping one full-attention layer helps recover accuracy, but the cache then grows with context length again. Table 2 also shows that, with the same cache budget, adding memory tokens is not consistently better than simply keeping more original-token KV entries.

  1. The end-to-end efficiency advantage over lighter compression mechanisms remains unclear.

Inserting one memory token per eight real tokens increases the internal sequence length by 12.5%, adding projection and MLP computation throughout the model, including the already-windowed local layers. Interleaved memory positions also create partially masked attention blocks that limit efficient exploitation of sparsity. The paper reports no prefill acceleration, and its efficiency measurements use an earlier configuration rather than the final accuracy configurations.

Questions for authors
  1. Does TWA work well in practice? In practical long-context applications, retaining access to the full KV history still seems important for accuracy. Even with one full-attention layer retained, SpMem falls short of the dense baseline on several benchmarks. Given these accuracy losses, in which practical workloads do TWA’s memory savings outweigh the cost of discarding historical information?

  2. When does the extra computation pay off? Inserting one memory token per eight real tokens increases the internal sequence length by 12.5%, and these extra positions go through the entire Transformer. I would expect this to add prefill work and potentially increase time to first token (TTFT). In what application scenarios does this trade-off pay off? The memory-token idea also appears in earlier prompt-compression work, such as Gist. How does this approach compare with chunk-based compression methods that aim to reduce the computation needed to process long inputs?

  3. Can the sparse mask actually improve prefill efficiency? Different queries attend to different KV subsets. This can still be computed in parallel, but interleaved memory positions and partially masked blocks may make it difficult to turn logical sparsity into actual speedups. The paper does not demonstrate a prefill speedup. Can the authors provide direct prefill measurements for the final configuration and explain how efficiently the kernel handles this mask?

  4. What is the advantage over CSA-style compression? DeepSeek-V4’s CSA is discussed as related work, but there is no direct experimental comparison. Even setting top-k selection aside, CSA-style learned KV aggregation avoids running additional memory positions through the full Transformer, which seems more economical.

5 comments

We thank the reviewer for the detailed feedback. We address each point below. We have uploaded a revised version of the paper incorporating these changes; results of runs still in progress will follow in a later comment and revision.

W1. Dropped history, bounded retrieval, and memory tokens vs. more real tokens (the submission's Table 2). We agree that retrieval is the weak point when every global layer is bounded, and we now present two operating points: (i) every global layer bounded, a constant cache per layer (15,872 entries at the submitted configuration, 12,287 at the improved one below), and (ii) one global layer kept at full attention, a 4.3× (4.8×) smaller cache than dense at 128K that still grows in that one layer. We recommend (ii) and report (i) as the bounded limit.

On the submission's Table 2, we retrained the baselines identically at the submitted configuration (C=8, m=1). With one global layer kept, memory tokens beat spending the same budget on more real tokens on every long-context axis: SpMem + 1 global (15,872 entries per bounded layer) against TWA 16K + 1 global without memory (16,383 entries): LongBench-v1 38.6 vs 37.8, LongBench-v2 39.5 vs 36.7, MRCR-v2 55.3 vs 51.9, NIAH 90.1 vs 83.2, RULER-128K 65.4 vs 46.2. A plain sliding window with the same retained layer trails further (RULER-128K 42.8 at 8K and 44.9 at 12K entries). With every global layer bounded, the result is mixed: against TWA 16K, memory wins LongBench-v2 (+7.5), MRCR-v2 (+6.5) and RULER-128K (+4.4) and loses LongBench-v1 (−3.1) and NIAH (−2.4).

Since submission: a better training recipe and configuration for the same method. Next-token prediction trains the memory tokens weakly, because most targets are predictable from the local window. Memory forcing restricts a fraction of training sequences to their most recent real tokens, so older context must come through memory; it is a change to the training recipe, not to the method. The final recipe forces a random half of the training sequences, with every layer limited to its last 128 real tokens. We pair it with a coarser configuration, C=128, m=4. With one global layer kept, against the submitted C=8, m=1, it ties LongBench-v1 (38.6), gains 6.4 on LongBench-v2 (45.9 vs 39.5) and is 2–5 points lower on retrieval (MRCR-v2 53.0 vs 55.3, NIAH 85.6 vs 90.1, RULER-128K 62.1 vs 65.4), in a 23% smaller cache (12,287 vs 15,872 entries per bounded layer), at 25% lower training cost per global layer. At the same budget it beats a 12K sliding window with the same retained layer on every axis (LongBench-v2 45.9 vs 37.4, RULER-128K 62.1 vs 44.9, NIAH 85.6 vs 79.2, MRCR-v2 53.0 vs 48.9). The revised paper uses C=128, m=4 with this forcing as its main configuration. The every-global-bounded run with the new forcing is in progress, and we will post it in a follow-up.

W2. End-to-end efficiency on the final configuration. At C=128, m=4 memory tokens add 4 positions per 128 reals, 3.1% of the sequence, against 12.5% at the submitted C=8, m=1, so one global layer costs 1.36× causal attention in training against 1.81×, and a full training step costs 1.30× dense with one global layer kept (main configuration, including memory forcing). We re-measured efficiency on the final configuration (Gemma 4 12B, one TPU v5p chip, batch 1). With every global layer bounded, decode at 1M tokens takes 17.0 ms per token and 22.9 GiB against dense's 36.6 ms and 38.7 GiB (2.15× faster), flat from 8K to 1M; with one global layer kept it is 21.1 ms and 24.9 GiB (1.74×). The submitted configuration decodes at nearly the same speed (17.4 vs 17.1 ms at 128K). With every global layer bounded and chunked prefill (8,192 tokens per chunk), a 128K prompt takes 15.8 s against dense's 26.1 s (1.65× faster), because each chunk attends to a bounded cache rather than to everything before it. At 32K the two are even (3.9 s vs 4.0 s). A single-shot 32K prefill is 23% slower (3.15 s vs 2.57 s), which is the block-occupancy cost described in Q3. The revision replaces the submission's prefill statement with these measurements.

Q1. Does TWA work in practice; which workloads. TWA does drop history, but by design. The point is a resident cache that stays fixed as a session grows, which is what limits long sessions and concurrent users in serving. Against a dense model retrained on the same schedule, the submitted SpMem + 1 global is within 1.2 on MRCR-v2 and 1.7 on LongBench-v1 at a 4.3× smaller cache at 128K; with the improved recipe (W1), C=128, m=4 is within 1.7 on LongBench-v1 and 3.5 on MRCR-v2 at 4.8×. At 1M tokens the improved model finds 94 of 100 needles with its trained cache, and all 100 once its memory cache is extended at inference, where the matched dense model finds 90. Where exact recall of arbitrary old tokens is essential, keeping one global layer at full attention is the supported operating point.

Q2. When the extra computation pays off, and relation to Gist / chunk-based compression. The extra positions pay off at long context. With every global layer bounded, decode overtakes dense between 32K and 128K and is 2.15× faster at 1M (1.74× with one global layer kept), and chunked prefill of a 128K prompt is 1.65× faster (W2); below the crossover SpMem is up to 12% slower per token, so the trade-off favors long sessions and many concurrent long-context requests. Prompt-compression methods (Gist, AutoCompressor, CCM) and Activation Beacon compress context into memory states to cut the cost of processing long inputs, but their resident state is fixed per prompt (Gist) or grows with the input (AutoCompressor, Beacon, and CCM in its concatenation form), and the raw tokens before the current segment are no longer addressable. SpMem's resident cache is fixed (2W − 1 real entries plus a bounded summary pool) at any length, a tapered sample of all raw tokens stays addressable, and it applies to pretrained 12B and 31B local/global models. The revision adds a table that sets these methods side by side by resident state, addressable raw tokens and what is trained. We also compare against Activation Beacon directly, reimplemented in our training stack on Gemma 4 12B with the same data, optimizer and batch.

Q3. Can the sparse mask improve prefill. Not with the current layout. A block-sparse attention kernel (block size 512×512) skips empty blocks and runs full ones without mask work. Memory positions are interleaved with the reals, so almost no standing block is full: at 128K, causal attention has 30,720 full blocks and 256 partial, while C=128, m=4 has 1,736 full and 31,296 partial. The bounded cache already pays off in chunked prefill (above). A one-shot speedup would need memory positions grouped into their own blocks, which is a layout change, not a change to the method. We list it as future work.

Q4. Advantage over CSA-style compression. CSA learns a compressor and a top-k selector while SpMem uses neither, and its memory tokens are ordinary positions read by ordinary attention, so it drops into an existing local/global model with one embedding table. We did not retrain a CSA model under our recipe; the revision discusses the cost trade-off (CSA avoids running memory positions through the stack, SpMem avoids the compressor and selector) in the related work.

Follow-up: the every-global-bounded run with the final recipe. The run we said was in progress has finished, and the revision now fills its rows (Tables 1–3, Figure 3). Gemma 4 12B, C=128, m=4, every global layer bounded at 12,287 entries (10.7× smaller than dense at 128K, at any length):

12B, every global boundedKV per layerLB-v1LB-v2MRCR-v2NIAHRULER-128K
SWA 8K8,19231.630.319.447.116.9
TWA 8K, no memory8,19130.231.626.347.611.7
TWA 16K, no memory16,38337.729.928.562.420.3
SpMem (TWA 8K + memory)12,28731.729.919.659.729.6
SpMem + 1 global (main)12,287 (+1 full)38.645.953.085.662.1
Dense131,07240.350.756.596.485.1

With every global layer bounded, memory tokens on top of TWA 8K raise NIAH by 12.1 and RULER-128K by 17.9 for 4K extra entries, and RULER-128K exceeds TWA 16K, which holds more entries. They do not help MRCR-v2. MMLU and IFEval stay close to dense (69.6 / 63.4 vs 66.5 / 65.8). As the reviewers noted, fully bounded retrieval remains well below dense, and we present this setting as the constant-memory limit.

The memory cache can also be extended after training without a full-attention layer. At ~1M tokens, the fully bounded model finds 12 of 100 single needles with its trained memory cache and 63 with a 25K-entry memory cache swapped in at inference (32× smaller than dense across the eight global layers), against 90 for the matched dense model. So long-range recall can come through memory alone, though less reliably than through one retained layer.

Our recommended operating point therefore remains SpMem + 1 global: within 1.7 of dense on LongBench-v1 and 3.5 on MRCR-v2 at 4.8× smaller cache (7.4× at 1M), and 100 of 100 needles at 1M with the extended memory cache. Only that one layer grows with context.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers for their valuable feedback. Below we summarize the changes in the revised PDF.

  • Schedule-matched dense reference (h3dc W1/Q1, 4DBD W8). Dense is retrained on SpMem's 5K-step schedule, data and optimizer at 12B and 31B, and every main comparison (Tables 1–3) is against it (Section 4, Appendix A.8).
  • Training recipe: memory forcing (4DBD W4/W5). On a random half of the training sequences, attention is restricted so that older context must come through the memory tokens. Forcing is not applied at inference. It changes the training recipe, not the method (Section 2.2; ablation in Table 7, Appendix A.3.2).
  • Final configuration and both operating points (h3dc W2, fiQV W1, 4DBD W5). C=128, m=4 with memory forcing. Results are reported for every global layer bounded (constant cache) and for one global layer kept at full attention (the recommended operating point), at 12B and 31B (Section 2.3, Table 1).
  • Same-budget baselines (h3dc W3/Q2, 4DBD W4). With every global layer bounded: a plain 8K sliding window and TWA 8K, 16K and 32K without memory tokens. With one global layer kept: a plain 12K sliding window and TWA 16K at the same or a larger budget, and a C=64, m=4 configuration at TWA 16K's budget. All are trained identically (Table 2, Section 5.2).
  • Baseline provenance (h3dc Q3, 4DBD W2). Activation Beacon and SPLA are retrained on the same base model and data as SpMem (Section 4, Appendix A.1.5).
  • Efficiency on the final configuration (fiQV W2/Q2/Q3, 4DBD W6/W9). Decode latency and memory from 8K to 1M, batch throughput at 128K, chunked prefill, and the end-to-end training step (Section 5.3, Figure 4, Appendices A.5, A.11, A.15 and A.17).
  • Related work (4DBD W1/W3, fiQV Q2). An appendix section on retrieval-based KV management and offloading (Appendix A.1.7), and a table placing SpMem among memory-token and summary methods, including Gist tokens, AutoCompressor and Activation Beacon, and Cascading KV (Appendix A.1.8, Table 4).
  • Abstract and framing (h3dc W2). These now state the two operating points and their cache reductions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.