- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper aims to enable long-context modeling with a bounded cache by introducing State Space Attention (SSA), a sparse attention mechanism with learnable eviction. Specifically, a learnable scorer dynamically selects which KV chunks to retain, while a multi-stage training strategy gradually transitions from soft to hard retention. Counterfactual recovery is further introduced to provide training signals for incorrectly evicted chunks. Experiments show that SSA achieves performance comparable to full-attention Transformer on several synthetic retrieval tasks. However, it incurs substantially higher training overhead and exhibits worse language modeling performance than the compared baselines.
Critical strengths
The paper is motivated by an important problem: improving long-context modeling capabilities under a bounded cache budget. This is a critical challenge in efficient LLM inference with significant practical value.
Critical weaknesses
-
Insufficient methodological justification and validation. The paper lacks important implementation details, comparisons with key baselines to establish its contributions, and controlled ablations to validate the effectiveness of its training strategy. See Questions (1)–(3) for details.
-
Questionable practical value and limited experimental evidence. The reported results raise concerns about SSA's practical applicability, including substantial training overhead, insufficient experimental validation, and relatively poor language modeling performance. See Questions (4)–(5) for details.
Questions for authors
1. Insufficient comparisons with existing sparse attention methods. SSA can be viewed as sparse attention with learned KV-cache eviction, a direction already explored in prior work. For example, eviction can be learned during training (Dynamic Context Pruning, NeurIPS 2023) or applied after pretraining (e.g., H2O and SnapKV). However, the paper lacks comparisons with these closely related sparse attention baselines, especially in the 400M pretraining experiments. The absence of strong sparse-attention-only baselines makes it difficult to assess the actual benefits of SSA's learned retention policy and complex training strategy.
2. Insufficient implementation details of the Scorer. The Scorer is the core component of SSA, but its architecture and implementation are not sufficiently described. For example, how are chunk-level scores generated from token representations? What attention pattern does the Scorer use, and how are positional and segment information encoded? The authors should provide a more detailed explanation of the Scorer's implementation.
3. Insufficient justification of the multi-stage training strategy. The paper lacks controlled ablations to demonstrate the necessity and effectiveness of its key training components, including soft retention warmup, STE-based hard retention, and counterfactual recovery. In particular, Table 14 only evaluates adding recovery during language pretraining, rather than its contribution during retrieval training. Could the authors provide component-wise ablations (e.g., Hard-only vs. Soft-to-Hard; with vs. w/o STE; with vs. w/o recovery) to justify their individual benefits?
4. Significant training overhead and unclear practical efficiency. Section 4.4 shows that SSA incurs substantially higher training costs, even without counterfactual recovery, whose potentially expensive overhead is not evaluated. This raises serious concerns about SSA's practical value, as the paper fails to demonstrate a competitive performance–efficiency trade-off against existing sparse-attention or constant-cache baselines in realistic language model settings.
5. Limited experimental evidence to support the effectiveness of SSA. The current experiments provide insufficient evidence for the benefits of the proposed method:
- Section 4.1: Although SSA successfully retains relevant information, all linear/hybrid baselines (GDN, HOLA, laLTE) achieve 100% accuracy, making this task insufficiently discriminative.
- Section 4.2: The entire sequence fits within SSA's memory budget, so no eviction is required. The results mainly demonstrate the advantage of softmax attention models over linear-attention/hybrid models on this specific task, rather than the effectiveness of SSA and learned retention.
- Section 4.3: Insufficient comparisons against relevant baselines make it difficult to isolate the benefits of SSA.
- Section 4.5: The 400M pretraining experiments report only NLL, without evaluating long-context retrieval, reasoning, or downstream language modeling benchmarks. Moreover, SSA achieves worse NLL than all reported baselines at the training context length.
Overall, these experiments do not sufficiently establish the effectiveness or practical advantages of SSA, particularly in realistic LLM settings.
3 comments
Comment 2 of 2.
Q3. Ablations of the training stages. We ran the ablations you list (hard-only vs soft-to-hard, with and without STE, with and without recovery) during retrieval post-training at 400M. We also ran both forms of the scorer without rescoring. Each arm has two runs of 1,560 updates from the same pretrained SSA (Table 21).
The outcome at this budget is bimodal and does not favor the full recipe with the soft updates. It learned the retrieval task in 1 of 3 runs (MKVT at 8K: 90.9%, 7.8% and 7.3%). Hard-only training learned it in 2 of 2 (64.4% and 95.7%), as did both scorers without rescoring (contextual: 80.0% and 98.4%; non-contextual: 98.3% and 98.0%). Without recovery it was learned in 0 of 2 (8.0% and 7.1%), and without STE in part in 1 of 2 (31.5% and 8.0%). We report this as is, and we have removed the claim that these components speed up retrieval learning. We believe due to the observed diminishing benefits of the soft phase at realistic scales, this might not be necessary anyway.
On this, we thank you once again as this point has made it clear the importance of counterfactual recovery, which seems to be essential since no run without recovery learned, compared to the soft phase, and how, given the decreased comparative benefit of running soft updates, and given that component level ablations show it might destabilize training, it might not be necessary after all.
Q4. Training overhead and efficiency. We agree the training cost is high. Table 27 repeats the submission's measurements: SSA takes 3.6 times a Transformer's time per update in the soft phase and about 6 times in the hard phase, without recovery. Recovery adds to this: in pretraining, a hard update with recovery takes 13 times the Transformer's time (32 s against 2.5 s, a new measurement on the recovery-pretrained model). At the pilot settings, it selects 8 of each hard update's 64 rows and replays them in one pass every 8 hard updates. In post-training with sequences up to 8K, on the same GPUs and task mixture, SSA takes 3.0 times a Transformer's time per update in the soft phase, 2.3 times in the hard phase without recovery, and 3.3 times with recovery; recovery adds 45% to a hard update (Table 27), but the context size difference ends up bringing the update times closer than our estimates above. These ratios also depend on the head layout. Our 400M models use 4 attention heads of dimension 256, as do GDN and HOLA, and on this layout the Transformer trains 1.7 to 4.0 times more slowly than with 16 heads of dimension 64. In a synthetic grid (Tables 28&29), the hard-phase ratio for our configuration falls from 2.8 at 2K to 0.7 at 64K; with 16 heads of dimension 64 it is 5.0 at 2K, 6.8 at 8K and 3.2 at 64K, so with standard head sizes SSA is slower to train at every context length we measured.
On the inference side, SSA reads 576 tokens at any context length. At 400M (24 layers, width 1024, bf16), SSA's key-value storage is a fixed 54 MiB per sequence (measured allocation), against 768 MiB for a full-attention Transformer at 8K (computed from the formula in Section 4.3), and it does not grow with length. With optimized decoders for both models (Table 32), SSA's decoding time per token is flat in context (1.48 ms at 8K, 1.44 ms at 64K) while the Transformer's grows (1.92 ms at 8K, 5.66 ms at 64K). SSA reads the prompt more slowly, because its eviction decisions are sequential, so its total time is lower only after about 380 generated tokens at 8K and 200 at 64K.
We do not yet show a better accuracy-per-cost trade-off than the baselines at 400M, and the paper now says so.
Q5. Strength of the experimental evidence.
- Section 4.1. We agree that cued copying does not separate SSA from the recurrent baselines, and the paper says so. We keep it as a feasibility check and rely on the new 400M tasks for discrimination.
- Section 4.2. We agree that the sequence fits SSA's memory, so it tests the reader, not retention; the paper states this. The new 8K haystack version (Table 24) requires about 480 eviction decisions. There, with the source at the start, SSA reaches 77.4% token accuracy from its mixed-task checkpoint, while the 576-window Transformer, the full-attention Transformer, GDN and HOLA stay at chance after the same training. From their k-hop checkpoints, SSA reaches 99.2% and the full-attention Transformer 99.9%.
- Section 4.3. We now evaluate MKVT at 2K, 4K and 8K for all nine 400M models (Table 2) and with the eviction baselines (Table 22). SSA reaches 96.13%, 95.97% and 96.57%, against 28.90%, 14.67% and 8.13% for the 576-window Transformer.
- Section 4.5. The 400M models are now evaluated on retrieval and reasoning (Tables 2 and 24), not only NLL. We agree that SSA's language modeling is worse at 2K: its perplexity after post-training is 23.07, against 18.77 to 21.49 for the other models. The paper now states this directly.
I appreciate the authors' detailed responses and additional experiments, which have significantly improved the paper's experimental validation.
However, the results still indicate that SSA incurs considerable training overhead and does not demonstrate a clear overall performance–efficiency advantage over existing sparse attention or constant-cache methods. In particular, its substantially worse language modeling performance remains a major concern, limiting its practical value. Therefore, I will maintain my original score.
Comment 1 of 2.
Thank you for the thorough review and the concrete list of questions. We agree with the main concerns and have used the rebuttal period to address them with new 400M experiments.
Q1. Comparisons with sparse-attention and eviction baselines. We now compare against H2O and SnapKV, as you suggest, and against StreamingLLM and TOVA (Table 22). We apply each method to our 400M full-attention Transformers after the same long-context post-training. Each runs at SSA's budget of 576 readable tokens, and H2O and TOVA also run with their papers' own cache sizes.
At 8K, the eviction rules that do not see the question lose most of MKVT: 5 to 8% on the 2K-pretrained Transformer, against 96.6% for SSA. SnapKV, which reads the whole prompt including the question before choosing, reaches 99.1% on MKVT and 51.0 on RULER, where SSA reaches 56.4.
These methods are training-free: they were not trained under eviction, while SSA was. The comparison therefore shows the value of learned retention over rule-based retention. As you suggest, we also trained a learned-eviction baseline at 400M, Dynamic Memory Sparsification (DMS; Łańcucki et al., 2025). We chose DMS as a baseline as it is a modern method we saw papers compare to, it had easily accessible released code and ran relatively fast for results within the rebuttal period. It starts from the pretrained Transformer and goes through the same 9,560 post-training updates, with the hyperparameters of its paper and official code (we did not search them ourselves) and a compression ratio chosen to match SSA's 576 tokens at 8K (Table 23). DMS chooses separately for each layer and head: each head keeps 344 tokens on average against SSA's 576, but across its 96 layer and head pairs DMS holds keys and values from about 4,480 distinct tokens, where SSA holds the same 576 at every layer and head. We acknowledge this as an advantage of DMS; SSA can be extended to choose per layer and head as well, which we have not done at this scale. The two are nearly equivalent on MKVT at 2K and 4K, with DMS leading by a few percentage points, and on MQAR, and DMS's clear win is perplexity (20.92 against 23.07 at 2K, and 22.21 against 26.91 at 8K). On MKVT at 8K they are nearly equivalent, with SSA leading by one percentage point (96.57 against 95.43). SSA is ahead on RULER at 8K (56.40 against 50.84) and on k-hop (99.98 / 99.97 / 99.54 against 99.95 / 97.55 / 38.61 token accuracy at 4 / 8 / 16 hops, excluding the null symbol). At the 3M scale, the appendix already compares SSA with laLTE, which learns token eviction in a hybrid model. As the general response explains, these methods also compose with SSA: they can be run on top of its retained set.
Q2. The scorer's implementation. We have added a description and a diagram to the main text (Section 3.1) and the full specification to the appendix. In short: