- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper targets the resident KV cache in long-context decoding. It modifies only the global-attention layers of a local/global interleaved transformer. After every 8 real tokens, 1 learned memory token is appended. It is processed by the ordinary forward pass, and its keys/values act as a summary of the preceding chunk. Both real and memory tokens are retained under Tapered Window Attention. The most recent W keys are kept verbatim, and each older band of W positions keeps half as many keys, selected by absolute position. TWA is a closed-form, monotone mask, so the same rule is used in training and serving, and the per-layer cache is fixed at about 2W - 1 entries per stream. Models are obtained by continued pretraining and compared against a dense model uptrained on the same data.
With every global layer bounded, SpMem roughly preserves short-context ability and LongBench-v1 but loses substantial retrieval. Keeping one of the eight global layers at full attention recovers most of the gap at a 4.3x reduction. With the memory cache widened at inference only, that variant finds all 100 single needles at ~1M tokens. The paper also reports decode latency, memory and throughput on TPU, and extensive ablations.
Critical strengths
-
Simple, practical design. There is no router, top-k selector or separate compressor. The only new parameters are the m x d memory-token embeddings, and the retention rule is closed-form shared by training and serving. Adoption in existing local/global models looks straightforward. This is the paper's main practical selling point. A method this cheap to adopt is valuable even with modest gains, which is why I see the paper as borderline rather than a clear reject.
-
Clear problem framing. The distinction between methods that are sparse in what they read and methods that are sparse in what they store (Sec. 3) is useful and correctly identifies resident KV memory as the binding constraint for long-context serving. It fixes the right metric for judging the paper (resident memory, not FLOPs).
Critical weaknesses
-
Training-schedule confound in the dense reference. SpMem variants use a 5K-step cosine schedule and are evaluated at step 5K, i.e., after full decay. The dense reference uses a 50K-step schedule and is also evaluated at step 5K, early in its decay. Comparing a fully annealed model with a mid-schedule one likely favors SpMem. A dense model trained with the same 5K-step schedule is needed. This is my main reason for not recommending acceptance. It affects every headline comparison with dense, and its direction plausibly favors SpMem.
-
The bounded-memory claim and the competitive-accuracy claim come from different configurations. The abstract and conclusion lead with a cache that does not grow with context. The configuration that is competitive with dense (SpMem + 1 global) keeps one layer at full attention, so its cache is O(L) again. Its reduction is 4.3x at 128K and can never exceed 8x. The fully bounded configuration loses 18.2 MRCR-v2 and 35.8 NIAH points at 12B. The 1M-token NIAH result also relies on the +1 global variant. The framing should make this trade-off explicit up front. The paper's main selling point and its main accuracy result do not hold at the same time. The realized contribution is therefore a 4.3x (at most 8x) reduction, not a constant-size cache, which lowers the significance relative to the framing.
-
The simplest hybrid baseline is missing. For the +1 global configuration, the natural baseline keeps one global layer at full attention and converts the other seven to plain sliding windows (no TWA, no memory tokens), trained identically. Table 2 includes TWA + 1 global without memory tokens, and Table 15 includes sliding window + memory tokens with K retained layers, but plain SWA + 1 full global layer is absent. Without it, the value of TWA and memory tokens over simply converting most global layers to local layers is not isolated. Without this baseline I cannot attribute the +1 global results to TWA and memory tokens rather than to the simple choice of keeping one full-attention layer. This is my second main reason for the recommendation.
Questions for authors
Q1: Can you report a dense reference trained with the same 5K-step schedule as the SpMem variants? Do the Table 1 conclusions still hold? Q2: How does SpMem + 1 global compare to plain sliding windows (no TWA, no memory tokens) on seven global layers plus one full-attention global layer, trained identically? Q3: How were the Beacon and SPLA points in Fig. 3 obtained? Were they retrained on Gemma 4 with your data, number of steps and budget?
6 comments
W3 / Q2. Plain sliding window on seven globals + one full-attention global. Trained identically (same data, 5K schedule, sphere projection, same kept layer), with plain sliding windows of 8K and 12K entries and, as a stronger window baseline, TWA 16K without memory (16,383 entries, W = 8K; caches are named by their size in entries, as in the revision).
| 12B, + 1 global, step 5K | KV per bounded layer | LB-v1 | LB-v2 | MRCR-v2 | NIAH | RULER-128K |
|---|---|---|---|---|---|---|
| Dense, matched schedule | 131,072 | 40.3 | 50.7 | 56.5 | 96.4 | 85.1 |
| SWA 8K | 8,192 | 32.8 | 28.9 | 48.4 | 73.3 | 42.8 |
| SWA 12K | 12,288 | 36.7 | 37.4 | 48.9 | 79.2 | 44.9 |
| TWA 16K, no memory | 16,383 | 37.8 | 36.7 | 51.9 | 83.2 | 46.2 |
| SpMem, submitted (C=8, m=1) | 15,872 | 38.6 | 39.5 | 55.3 | 90.1 | 65.4 |
| SpMem, C=128, m=4 + memory forcing (main configuration) | 12,287 | 38.6 | 45.9 | 53.0 | 85.6 | 62.1 |
Keeping one global layer and windowing the rest is not enough, so the + 1 global results are not explained by the kept layer alone. With the same full-attention layer in every row, the window baselines stay well below SpMem, which isolates what memory tokens add over spending the budget on more real tokens. At the submitted configuration's budget (about 16K entries), SpMem + 1 global beats TWA 16K + 1 global without memory on all five axes, by 19.2 on RULER-128K and 6.9 on NIAH; plain sliding windows with the same layer reach only 42.8 and 44.9 on RULER-128K at 8K and 12K entries. Since submission, with the improved training recipe (W1 / Q1 above), SpMem + 1 global at C=128, m=4 has the 12K sliding window's budget and beats it on every axis: LongBench-v1 +1.9, LongBench-v2 +8.5, MRCR-v2 +4.1, NIAH +6.4, RULER-128K +17.2. NIAH is the 8-subtask RULER mean over six lengths, as in the paper.
Q3. How the Beacon and SPLA points were obtained. Both are our reimplementations in the same training stack, on Gemma 4 12B with the same data, optimizer and batch. Beacon uses its recommended 8× configuration (1,024-token intervals condensed into 128 beacons). It differs from the original in four ways: the base model is trained rather than frozen, there are no dedicated beacon projections, the raw-token window slides across intervals, and the ratio is fixed rather than sampled. Its schedule did not match (read at 5K of a 50K schedule), and a 5K-schedule Beacon is not in this rebuttal. SPLA used SpMem's 5K schedule. The revision's appendix describes how both were adapted to our stack.
Follow-up: the every-global-bounded run with the final recipe. The run we said was in progress has finished, and the revision now fills its rows (Tables 1–3, Figure 3). Gemma 4 12B, C=128, m=4, every global layer bounded at 12,287 entries (10.7× smaller than dense at 128K, at any length):
| 12B, every global bounded | KV per layer | LB-v1 | LB-v2 | MRCR-v2 | NIAH | RULER-128K |
|---|---|---|---|---|---|---|
| SWA 8K | 8,192 | 31.6 | 30.3 | 19.4 | 47.1 | 16.9 |
| TWA 8K, no memory | 8,191 | 30.2 | 31.6 | 26.3 | 47.6 | 11.7 |
| TWA 16K, no memory | 16,383 | 37.7 | 29.9 | 28.5 | 62.4 | 20.3 |
| SpMem (TWA 8K + memory) | 12,287 | 31.7 | 29.9 | 19.6 | 59.7 | 29.6 |
| SpMem + 1 global (main) | 12,287 (+1 full) | 38.6 | 45.9 | 53.0 | 85.6 | 62.1 |
| Dense | 131,072 | 40.3 | 50.7 | 56.5 | 96.4 | 85.1 |
With every global layer bounded, memory tokens on top of TWA 8K raise NIAH by 12.1 and RULER-128K by 17.9 for 4K extra entries, and RULER-128K exceeds TWA 16K, which holds more entries. They do not help MRCR-v2. MMLU and IFEval stay close to dense (69.6 / 63.4 vs 66.5 / 65.8). As the reviewers noted, fully bounded retrieval remains well below dense, and we present this setting as the constant-memory limit.
The memory cache can also be extended after training without a full-attention layer. At ~1M tokens, the fully bounded model finds 12 of 100 single needles with its trained memory cache and 63 with a 25K-entry memory cache swapped in at inference (32× smaller than dense across the eight global layers), against 90 for the matched dense model. So long-range recall can come through memory alone, though less reliably than through one retained layer.
Our recommended operating point therefore remains SpMem + 1 global: within 1.7 of dense on LongBench-v1 and 3.5 on MRCR-v2 at 4.8× smaller cache (7.4× at 1M), and 100 of 100 needles at 1M with the extended memory cache. Only that one layer grows with context.
We thank the reviewer for the detailed feedback. We address each point below. We have uploaded a revised version of the paper incorporating these changes; results of runs still in progress will follow in a later comment and revision.
W1 / Q1. Dense trained on the same 5K-step schedule. We retrained dense with SpMem's schedule (lr 1e-5, 500 warmup, cosine over 5K, same data and optimizer).
Matching the schedule strengthens dense, mostly on long-context reasoning and retrieval. Against this, the submitted SpMem + 1 global is within 1.2 on MRCR-v2 and 1.7 on LongBench-v1, ahead on MMLU (+4.2), and behind on LongBench-v2 (−11.2), IFEval (−3.9) and NIAH (−6.3).
The last rows are improvements made since submission to the same method: a coarser configuration (C=128, m=4) and memory forcing, which restricts a fraction of training sequences to their most recent real tokens so that older context must come through memory. The final recipe forces half the sequences, with every layer limited to its last 128 real tokens. Against the matched dense, it is within 1.7 on LongBench-v1 and 3.5 on MRCR-v2 at a 4.8× smaller cache; against the submitted configuration, it ties LongBench-v1, gains 6.4 on LongBench-v2, and is 2–5 points lower on retrieval. The revision uses it as the main configuration.
The revision reports all comparisons against this model. At 31B, the revision labels its rows with their configuration (C=8, m=1); the matched dense and main-configuration 31B runs are scheduled after the 12B runs, and we will post them if they finish within the discussion period.
W2. Bounded-memory claim vs. the competitive configuration. We agree that the two claims come from different configurations, and the revision states the trade-off up front, as two operating points rather than a constant cache. (i) Every global layer bounded: a constant cache per layer at any length (15,872 entries at the submitted configuration, 12,287 at the improved one; 8× and 10.7× smaller than dense at 128K), at a real cost in retrieval: against the schedule-matched dense, the submitted model loses 21.5 on MRCR-v2 and 36.4 on NIAH. (ii) One global layer kept at full attention: most of the retrieval comes back (within 1.2 of dense on MRCR-v2 at the submitted configuration), but that layer's cache grows with the context, so the reduction is 4.3× at 128K (4.8× at C=128, m=4) and, as the reviewer notes, can never exceed 8×; the 1M result also uses (ii). We recommend (ii) and present (i) as the constant-memory limit. The bounded run with the new recipe has finished, and we post it in a follow-up.