acceptodds
Under review as a conference paper at ICLR 2027

State Space Attention: Learning Context-Dependent Memory Retention

Abstract

The key–value cache used by standard attention grows linearly with context length. Bounding this cache raises a central question: what should a sequence model retain, and how can it learn to make those decisions to support long-context reasoning? We introduce State Space Attention (SSA), an attention architecture that learns to maintain a bounded set of past representations and reads them using standard attention. A learned scorer repeatedly rescores all chunks in light of newly arriving context and the other contents of memory. We train the scorer and reader jointly, gradually moving from fully differentiable, softly weighting candidate memories to retaining a hard, fixed number. Moreover, the scorer receives training feedback based on whether retaining otherwise discarded information would improve the model’s predictions. Across our evaluated retrieval and reasoning benchmarks, SSA maintains full-attention Transformer-level performance while its active inference memory remains independent of sequence length. We compare SSA with recurrent models, bounded-memory attention methods, and full-attention Transformers across retrieval, variable tracking, multi-hop reasoning, and language modeling. We further pretrain 400M-parameter models on 15B tokens, and measure the associated computational and memory costs along with language modeling performance. Together, these experiments examine the capabilities and limitations of learned memory retention for long-context sequence modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context modeling, bounded memory, attention mechanisms, learned memory retention, sequence modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DyvMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to enable long-context modeling with a bounded cache by introducing State Space Attention (SSA), a sparse attention mechanism with learnable eviction. Specifically, a learnable scorer dynamically selects which KV chunks to retain, while a multi-stage training strategy gradually transitions from soft to hard retention. Counterfactual recovery is further introduced to provide training signals for incorrectly evicted chunks. Experiments show that SSA achieves performance comparable to full-attention Transformer on several synthetic retrieval tasks. However, it incurs substantially higher training overhead and exhibits worse language modeling performance than the compared baselines.

Critical strengths

The paper is motivated by an important problem: improving long-context modeling capabilities under a bounded cache budget. This is a critical challenge in efficient LLM inference with significant practical value.

Critical weaknesses
  1. Insufficient methodological justification and validation. The paper lacks important implementation details, comparisons with key baselines to establish its contributions, and controlled ablations to validate the effectiveness of its training strategy. See Questions (1)–(3) for details.

  2. Questionable practical value and limited experimental evidence. The reported results raise concerns about SSA's practical applicability, including substantial training overhead, insufficient experimental validation, and relatively poor language modeling performance. See Questions (4)–(5) for details.

Questions for authors

1. Insufficient comparisons with existing sparse attention methods. SSA can be viewed as sparse attention with learned KV-cache eviction, a direction already explored in prior work. For example, eviction can be learned during training (Dynamic Context Pruning, NeurIPS 2023) or applied after pretraining (e.g., H2O and SnapKV). However, the paper lacks comparisons with these closely related sparse attention baselines, especially in the 400M pretraining experiments. The absence of strong sparse-attention-only baselines makes it difficult to assess the actual benefits of SSA's learned retention policy and complex training strategy.

2. Insufficient implementation details of the Scorer. The Scorer is the core component of SSA, but its architecture and implementation are not sufficiently described. For example, how are chunk-level scores generated from token representations? What attention pattern does the Scorer use, and how are positional and segment information encoded? The authors should provide a more detailed explanation of the Scorer's implementation.

3. Insufficient justification of the multi-stage training strategy. The paper lacks controlled ablations to demonstrate the necessity and effectiveness of its key training components, including soft retention warmup, STE-based hard retention, and counterfactual recovery. In particular, Table 14 only evaluates adding recovery during language pretraining, rather than its contribution during retrieval training. Could the authors provide component-wise ablations (e.g., Hard-only vs. Soft-to-Hard; with vs. w/o STE; with vs. w/o recovery) to justify their individual benefits?

4. Significant training overhead and unclear practical efficiency. Section 4.4 shows that SSA incurs substantially higher training costs, even without counterfactual recovery, whose potentially expensive overhead is not evaluated. This raises serious concerns about SSA's practical value, as the paper fails to demonstrate a competitive performance–efficiency trade-off against existing sparse-attention or constant-cache baselines in realistic language model settings.

5. Limited experimental evidence to support the effectiveness of SSA. The current experiments provide insufficient evidence for the benefits of the proposed method:

  • Section 4.1: Although SSA successfully retains relevant information, all linear/hybrid baselines (GDN, HOLA, laLTE) achieve 100% accuracy, making this task insufficiently discriminative.
  • Section 4.2: The entire sequence fits within SSA's memory budget, so no eviction is required. The results mainly demonstrate the advantage of softmax attention models over linear-attention/hybrid models on this specific task, rather than the effectiveness of SSA and learned retention.
  • Section 4.3: Insufficient comparisons against relevant baselines make it difficult to isolate the benefits of SSA.
  • Section 4.5: The 400M pretraining experiments report only NLL, without evaluating long-context retrieval, reasoning, or downstream language modeling benchmarks. Moreover, SSA achieves worse NLL than all reported baselines at the training context length.

Overall, these experiments do not sufficiently establish the effectiveness or practical advantages of SSA, particularly in realistic LLM settings.

3 comments

Comment 1 of 2.

Thank you for the thorough review and the concrete list of questions. We agree with the main concerns and have used the rebuttal period to address them with new 400M experiments.

Q1. Comparisons with sparse-attention and eviction baselines. We now compare against H2O and SnapKV, as you suggest, and against StreamingLLM and TOVA (Table 22). We apply each method to our 400M full-attention Transformers after the same long-context post-training. Each runs at SSA's budget of 576 readable tokens, and H2O and TOVA also run with their papers' own cache sizes.

At 8K, the eviction rules that do not see the question lose most of MKVT: 5 to 8% on the 2K-pretrained Transformer, against 96.6% for SSA. SnapKV, which reads the whole prompt including the question before choosing, reaches 99.1% on MKVT and 51.0 on RULER, where SSA reaches 56.4.

These methods are training-free: they were not trained under eviction, while SSA was. The comparison therefore shows the value of learned retention over rule-based retention. As you suggest, we also trained a learned-eviction baseline at 400M, Dynamic Memory Sparsification (DMS; Łańcucki et al., 2025). We chose DMS as a baseline as it is a modern method we saw papers compare to, it had easily accessible released code and ran relatively fast for results within the rebuttal period. It starts from the pretrained Transformer and goes through the same 9,560 post-training updates, with the hyperparameters of its paper and official code (we did not search them ourselves) and a compression ratio chosen to match SSA's 576 tokens at 8K (Table 23). DMS chooses separately for each layer and head: each head keeps 344 tokens on average against SSA's 576, but across its 96 layer and head pairs DMS holds keys and values from about 4,480 distinct tokens, where SSA holds the same 576 at every layer and head. We acknowledge this as an advantage of DMS; SSA can be extended to choose per layer and head as well, which we have not done at this scale. The two are nearly equivalent on MKVT at 2K and 4K, with DMS leading by a few percentage points, and on MQAR, and DMS's clear win is perplexity (20.92 against 23.07 at 2K, and 22.21 against 26.91 at 8K). On MKVT at 8K they are nearly equivalent, with SSA leading by one percentage point (96.57 against 95.43). SSA is ahead on RULER at 8K (56.40 against 50.84) and on k-hop (99.98 / 99.97 / 99.54 against 99.95 / 97.55 / 38.61 token accuracy at 4 / 8 / 16 hops, excluding the null symbol). At the 3M scale, the appendix already compares SSA with laLTE, which learns token eviction in a hybrid model. As the general response explains, these methods also compose with SSA: they can be run on top of its retained set.

Q2. The scorer's implementation. We have added a description and a diagram to the main text (Section 3.1) and the full specification to the appendix. In short:

  1. Inputs. The scorer reads the reader's token embeddings, not its hidden states, for the M resident chunks, the candidate chunk and the w recent-window chunks. A linear map projects them to the scorer width. Each token gets a learned within-chunk position embedding (16 positions) and a learned segment embedding for its slot: one per resident slot, one for the candidate, and one for the window.
  2. Chunk summaries. Each resident and the candidate gets a learned summary token, carrying its slot's segment embedding.
  3. Layers. Pre-norm Transformer blocks (RMSNorm, 4-head attention, SwiGLU feed-forward) attend without a mask over all summary tokens and all tokens of the residents, the candidate and the window. The last block updates only the summary tokens.
  4. Scores. A linear head maps each summary token to one keep score. The lowest-scoring of the M+1 chunks is evicted: deterministically at evaluation, and sampled with probability proportional to exp(-score/T_e) during hard training, with sampling temperature T_e.
  5. Sharing. One score set, and therefore one memory, is shared by all layers and heads of the reader.
  6. Size. At 400M the scorer has 2 layers of width 640, at about 3.5% the size of the model (13.8M). A larger 4-layer, width-896 scorer (13%, 52.4M), which takes its parameters from the reader's feed-forward layers to keep the total near 400M, raises MKVT 8K from 98.4% to 99.9% after 7,800 updates. RULER is unchanged (68.7 vs 68.8 at 4K), and 2K perplexity is slightly worse (21.40 vs 22.05). As such, we hypothesize the scorer can stay relatively small.

Comment 2 of 2.

Q3. Ablations of the training stages. We ran the ablations you list (hard-only vs soft-to-hard, with and without STE, with and without recovery) during retrieval post-training at 400M. We also ran both forms of the scorer without rescoring. Each arm has two runs of 1,560 updates from the same pretrained SSA (Table 21).

The outcome at this budget is bimodal and does not favor the full recipe with the soft updates. It learned the retrieval task in 1 of 3 runs (MKVT at 8K: 90.9%, 7.8% and 7.3%). Hard-only training learned it in 2 of 2 (64.4% and 95.7%), as did both scorers without rescoring (contextual: 80.0% and 98.4%; non-contextual: 98.3% and 98.0%). Without recovery it was learned in 0 of 2 (8.0% and 7.1%), and without STE in part in 1 of 2 (31.5% and 8.0%). We report this as is, and we have removed the claim that these components speed up retrieval learning. We believe due to the observed diminishing benefits of the soft phase at realistic scales, this might not be necessary anyway.

On this, we thank you once again as this point has made it clear the importance of counterfactual recovery, which seems to be essential since no run without recovery learned, compared to the soft phase, and how, given the decreased comparative benefit of running soft updates, and given that component level ablations show it might destabilize training, it might not be necessary after all.

Q4. Training overhead and efficiency. We agree the training cost is high. Table 27 repeats the submission's measurements: SSA takes 3.6 times a Transformer's time per update in the soft phase and about 6 times in the hard phase, without recovery. Recovery adds to this: in pretraining, a hard update with recovery takes 13 times the Transformer's time (32 s against 2.5 s, a new measurement on the recovery-pretrained model). At the pilot settings, it selects 8 of each hard update's 64 rows and replays them in one pass every 8 hard updates. In post-training with sequences up to 8K, on the same GPUs and task mixture, SSA takes 3.0 times a Transformer's time per update in the soft phase, 2.3 times in the hard phase without recovery, and 3.3 times with recovery; recovery adds 45% to a hard update (Table 27), but the context size difference ends up bringing the update times closer than our estimates above. These ratios also depend on the head layout. Our 400M models use 4 attention heads of dimension 256, as do GDN and HOLA, and on this layout the Transformer trains 1.7 to 4.0 times more slowly than with 16 heads of dimension 64. In a synthetic grid (Tables 28&29), the hard-phase ratio for our configuration falls from 2.8 at 2K to 0.7 at 64K; with 16 heads of dimension 64 it is 5.0 at 2K, 6.8 at 8K and 3.2 at 64K, so with standard head sizes SSA is slower to train at every context length we measured.

On the inference side, SSA reads 576 tokens at any context length. At 400M (24 layers, width 1024, bf16), SSA's key-value storage is a fixed 54 MiB per sequence (measured allocation), against 768 MiB for a full-attention Transformer at 8K (computed from the formula in Section 4.3), and it does not grow with length. With optimized decoders for both models (Table 32), SSA's decoding time per token is flat in context (1.48 ms at 8K, 1.44 ms at 64K) while the Transformer's grows (1.92 ms at 8K, 5.66 ms at 64K). SSA reads the prompt more slowly, because its eviction decisions are sequential, so its total time is lower only after about 380 generated tokens at 8K and 200 at 64K.

We do not yet show a better accuracy-per-cost trade-off than the baselines at 400M, and the paper now says so.

Q5. Strength of the experimental evidence.

  • Section 4.1. We agree that cued copying does not separate SSA from the recurrent baselines, and the paper says so. We keep it as a feasibility check and rely on the new 400M tasks for discrimination.
  • Section 4.2. We agree that the sequence fits SSA's memory, so it tests the reader, not retention; the paper states this. The new 8K haystack version (Table 24) requires about 480 eviction decisions. There, with the source at the start, SSA reaches 77.4% token accuracy from its mixed-task checkpoint, while the 576-window Transformer, the full-attention Transformer, GDN and HOLA stay at chance after the same training. From their k-hop checkpoints, SSA reaches 99.2% and the full-attention Transformer 99.9%.
  • Section 4.3. We now evaluate MKVT at 2K, 4K and 8K for all nine 400M models (Table 2) and with the eviction baselines (Table 22). SSA reaches 96.13%, 95.97% and 96.57%, against 28.90%, 14.67% and 8.13% for the 576-window Transformer.
  • Section 4.5. The 400M models are now evaluated on retrieval and reasoning (Tables 2 and 24), not only NLL. We agree that SSA's language modeling is worse at 2K: its perplexity after post-training is 23.07, against 18.77 to 21.49 for the other models. The paper now states this directly.

I appreciate the authors' detailed responses and additional experiments, which have significantly improved the paper's experimental validation.

However, the results still indicate that SSA incurs considerable training overhead and does not demonstrate a clear overall performance–efficiency advantage over existing sparse attention or constant-cache methods. In particular, its substantially worse language modeling performance remains a major concern, limiting its practical value. Therefore, I will maintain my original score.

Reviewer_Qw17OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of preserving attention-based retrieval in long-context modeling under a fixed inference-memory budget. It proposes State Space Attention (SSA), which combines context-dependent chunk retention with joint training of the scorer and reader to preserve useful information in a bounded KV cache. The method is evaluated on small-scale synthetic retrieval and reasoning tasks and on 400M-parameter language-model pretraining.

Critical strengths

SSA bounds inference memory while allowing direct attention to retained KV chunks. On cued copying, learned retention substantially outperforms drop-oldest and random eviction with the same reader and memory budget. On multi-hop induction, the bounded attention reader approaches Transformer accuracy and clearly outperforms recurrent baselines with a comparable state size.

Critical weaknesses
  1. The paper lacks direct ablations of context-dependent rescoring and counterfactual recovery during retrieval training. Their benefits over simpler alternatives remain unclear. Since these components distinguish SSA from prior learned-eviction methods, this weighs heavily in my assessment.
  2. Long-context reasoning under sustained eviction is insufficiently validated. The strongest compression result comes from cued copying, which recurrent baselines also solve. The main multi-hop experiment requires no eviction. At the 400M scale, SSA degrades substantially beyond its training context, while recurrent baselines remain stable. Since constant-memory long-context reasoning is the central motivation of the architecture, this gap is critical.
Questions for authors
  1. Could the authors provide separate ablations for context-dependent rescoring and counterfactual recovery during retrieval training?
  2. Could the authors provide results on long-context multi-hop tasks that require sustained memory eviction?
3 comments

Comment 1 of 2.

Thank you for a focused and constructive review. You identified the two experiments the paper really needed, and we ran both at the 400M scale.

Q1. Separate ablations of rescoring and counterfactual recovery during retrieval training. We ran these as post-training pilots (Table 21). Each pilot starts from the same pretrained 400M SSA and trains for 1,560 updates on the mixed long-context tasks: 624 soft updates, then 936 hard. Each arm changes one thing:

  • No recovery: counterfactual recovery is off.
  • Contextual, no rescoring: the scorer still sees the residents, the candidate and the recent window. Each chunk is scored once, when it becomes a candidate, and keeps that score from then on.
  • Non-contextual: the same scorer sees only the new chunk, so a chunk's score never changes.
  • Hard only: no soft phase; all 1,560 updates use hard retention.
  • No STE: the straight-through read gradient is off.

To be able to separate run-to-run variation from the effect of each change at least a little, we ran every arm twice. As some recipes were shown to pick up learning later than others, these are meant as early signal.

The outcome was not what we expected, and we report it as such. On MKVT the pilots are bimodal, a run either learns the task or stays near 30% at 2K and 8% at 8K. MQAR and mixed-task k-hop do not separate the arms.

  • The full recipe broke out in 1 of 3 runs at this budget (the original pilot and two same-configuration reruns).
  • Without recovery, it broke out in 0 of 2.
  • The two scorers without rescoring and the hard-only schedule broke out in both of their runs (MKVT 8K 64.4 to 98.4).
  • Without STE, one run partly broke out but did not learn well (31.5%).

We checked that the reruns are equivalent to the original pilot. The evaluator, the GPU model and the training schedule are also the same. These were checked because the interesting part is that the runs in the full recipe can differ in whether they break out with the exact same recipe, including the seed, and preliminary analysis of the loss and gradient norms suggests this may be floating point nondeterminism.

With two or three runs per arm we cannot rank the arms precisely. These pilots do not show that rescoring is necessary for retrieval learning at this budget, and the simpler scorers learned more reliably here. We do, however, believe rescoring remains important from an expressivity perspective, and aim to train fully expressive scorers. The revised paper states that the full recipe was not more reliable than the simpler scorers at this budget.

The runs suggest that full hard training with recovery might be necessary for more consistent performance, and that the soft-phase training provided is a compromise, or even something that the model worked despite, not thanks to. New training speed ablations at 8K, 400M scale suggest that this might be OK nonetheless, as the difference between the two seems to diminish drastically even at that scale.

What the evidence does support is narrower. At 7,800 updates, the full recipe reaches 98.4% on MKVT 8K and 61.9 on RULER 8K (99.9% and 61.9 with a larger scorer), with MQAR above 99.9 either way.

The hard-only arm also differs from the control in hard-phase exposure: 1,560 hard updates and 195 recovery passes against 936 and 117.

Comment 2 of 2.

Q2. Long-context multi-hop reasoning under sustained eviction. We built a k-hop task inside 8,192-token rows of SlimPajama text (Table 24). The k-hop source block (258 tokens) is placed in the text, and the question and the 256-token answer come at the end. With the block at the start, SSA must keep its 17 chunks through about 480 eviction decisions while reading only 576 tokens at a time.

Every model was trained for 1,760 updates from its own mixed-task checkpoint (after 7,800 post-training updates), with half of the rows using a random block position. We report token accuracy excluding the null symbol, where chance is about 25%. We do not discuss whole-answer exact match, which is unstable for 256-token answers and depends on run-to-run randomness. With the block at the start, SSA reaches 77.4%: 100.0% at 4 hops, 99.9% at 8 hops and chance at 16 hops. The 576-window Transformer, the full-attention 8K Transformer, GDN and HOLA all stay at chance (22.5 to 26.7%).

With the block at a random position, no model succeeds, SSA included: all stay at chance per token. A retention probe shows that SSA does keep the block:

  • 97.1% of its chunks are in memory at the first answer step;
  • the whole block stays in memory at every answer step in 51% of rows.

Accuracy is the same whether or not the whole block was kept, so the failure is in reading the block at a variable offset.

Models started from their k-hop-trained checkpoints show the same split but meaningfully different performance. With the block at the start, the full-attention Transformer reaches 99.9% and SSA 99.2% (98.9% at 16 hops), while the 576-window Transformer, GDN and HOLA stay at chance. At a random position every model is at chance.

Degradation beyond the training context. We agree this is a real weakness at 400M. Capping query-key distances at the pretraining length removes about half of SSA's perplexity rise from 2K to 8K (Table 26). Replacing the learned retention rule with a sliding window over the most recent chunks at evaluation removes all of it (20.81 at 2K, 20.16 at 8K), so the rise comes from reading the old chunks the policy keeps. A sliding window gives up long-range retrieval, so we state this in the limitations.

Thanks for the additional experiments. The new results help address my concern about reasoning under sustained eviction. However, the benefit of contextual rescoring remains unclear from these short ablation runs, and the long-context perplexity degradation remains a concern. I appreciate the authors' efforts, I will keep my rating at 2.

Reviewer_nVfcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

SSA replaces the KV cache, which grows linearly with the context, with a fixed set of retained chunks along with a local window that, at some point, remains constant. This is done with a learned scorer that decides which chunks to evict or keep.

Critical strengths
  • The idea is interesting and can be useful if it's done properly.
Critical weaknesses
  • The paper's own results contradict some of the claims:

    • The abstract says SSA "maintains full-attention Transformer-level performance." However, Table 1 (fixed data) shows 70.05 vs. 94.35.
    • L453 says SSA is better than the Transformer at 8K. Both models break down at that length (4.33 vs. 5.04 nats), while the 8K Transformer, GDN, and HOLA reach about 2.55. At 2K, SSA is the worst of the five models (Table 2).
  • Scale is insufficient. Most of the results come from 3M–20M models, even on synthetic tasks. The one realistic-scale experiment (400M) has a single seed and reports perplexity. There is no realistic long-context evaluation and no scaling analysis.

  • Other baselines are discussed but not compared.

  • The AI-use statement says AI helped "review the proof concerning equivalence between the soft and hard phases," but the paper contains no proof.

  • The title is misleading. "State Space Attention" suggests an SSM, but the paper says SSA is "closer to attention than SSMs" (L64). It is a Transformer with learned KV-cache eviction.

  • The writing is draft quality.

    • A leftover note remains at L377 ("Seed details go in the appendix").
    • One sentence is duplicated (L263–265), right in the key credit-assignment passage.
    • The language is informal in some places.
    • The main text has only two tables, and almost every claim depends on the appendix.

It needs major revision and does not meet the ICLR bar.

Questions for authors

Check weaknesses

3 comments

Comment 1 of 2.

Thank you for the detailed review. You are right that several claims went beyond the evidence, and that the submitted draft was not in the state it should have been. We address each point below and have corrected the paper accordingly.

The abstract's claim of Transformer-level performance. We agree that this claim was too broad. Currently, SSA comes within 3.4 percentage points of the full-attention Transformer on four of our tasks while reading at most 576 tokens:

  • cued copying, 98.70 vs 100;
  • multi-hop induction with fresh data, 99.65 vs 99.75;
  • the new 400M results on MKVT at 8K, 96.57 vs 99.60 and 99.90;
  • MQAR, 99.95 vs 99.98 and 99.99.

It trails clearly in two places: by 24.3 points on multi-hop induction with a fixed training set (70.05 vs 94.35, as you note) at small scale, and by 14 to 20 points on RULER at 8K (56.40 vs 70.72 and 76.86) at large scale. We have replaced the sentence in the abstract with a scoped one (SSA matches or comes within a few points of full attention on most tasks, with the largest remaining accuracy gap at 400M scale on RULER), and the main text reports both gaps. However, SSA’s score is still the highest RULER result in our experiments among attention models reading 576 tokens.

The recurrent baselines beat SSA on RULER 8K. GDN reaches 99.50 on MKVT 8K and 68.91 on RULER 8K, and HOLA 99.53 and 58.04 (SSA: 96.57 and 56.40). SSA's advantage over them is on 256-token k-hop induction, where SSA reaches 99.84% token accuracy and both recurrent models stay at chance (26.1% and 26.5%; Table 2). Those k-hop rows fit within SSA's 576 tokens, so this is a reader comparison, not a retention one, and it is meant to highlight that SSA keeps the benefits of quadratic attention. As such, we have narrowed the claim: the paper now says that, at similar state budgets, SSA stays at Transformer accuracy on multi-hop induction that fits its readable state when trained on fresh data, where GDN and HOLA do not. We also find training instability, which the paper reports.

The 8K language-modeling claim (L453). We agree, and we withdraw the sentence. Both 2K-pretrained attention models degrade at 8K, while GDN, HOLA and the 8K Transformer do not. We also agree that SSA has the highest 2K NLL of the five submitted models (2.656 nats vs 2.590 to 2.623). Among the eight models now in the pretraining table, the three SSA variants (2.650 to 2.665) are above every model except the new 576-window Transformer (2.695). After post-training, SSA has the highest perplexity of all models at both 2K and 8K (Table 2). We provide a diagnosis of the language modeling degradation below.

To understand the 8K degradation, we capped every query-key distance at 2,047 tokens at evaluation time, which removes distances never seen in pretraining (Table 26). The cap removes all of the 2K-pretrained Transformer's 8K degradation (22.28 to 18.77 perplexity). It removes about half of SSA's (26.91 to 24.92, against 23.07 at 2K). Replacing SSA's learned retention rule at evaluation with a sliding window over the 34 most recent chunks removes the whole rise: the same weights give 20.81 at 2K and 20.16 at 8K (Table 25). The degradation therefore comes from reading the old chunks the policy keeps, about half of it through distances beyond the pretraining length. A sliding window gives up the long-range retrieval the policy is trained for, so we report the degradation as a limitation. An SSA configuration with a larger ww may be better for overcoming this problem, but it has not been tested.

Scale. We agree the submission's 400M evidence was thin. All new experiments are at 400M:

  • post-training of nine models on long-context retrieval and reasoning (Table 2);
  • eviction baselines (Tables 22 and 23);
  • component ablations with two runs per arm (Table 21);
  • multi-hop reasoning inside 8K haystacks (Table 24).

The nine-model comparison is still one seed, which we state in the paper. We do not have a rigorous scaling-law analysis, and the paper now says so rather than implying one.

Comment 2 of 2.

Baselines that were discussed but not compared. We now compare against StreamingLLM, H2O, TOVA and SnapKV on our 400M full-attention Transformers, at SSA's 576-token budget and with each paper's own settings for H2O and TOVA (Table 22). We also trained a learned-eviction method, Dynamic Memory Sparsification (DMS), through the same post-training (Table 23).

At 8K on the 2K-pretrained Transformer, at SSA's budget, the training-free eviction rules reach 5 to 8% on MKVT, against 96.6% for SSA. With their papers' own settings, H2O keeps about 1,600 tokens (20% of the prompt) and reaches 40.3% on MKVT and 24.1 on RULER, and TOVA keeps 1,024 tokens (1/8 of the context) and reaches 6.4% and 6.8. SnapKV, which sees the question before choosing what to keep, reaches 99.1% on MKVT and 51.0 on RULER (SSA: 56.4). These methods were not trained under eviction, so the comparison aims to isolate learned versus rule-based retention. Learned-eviction methods are the fairer comparison for SSA. We trained DMS at 400M: it starts from the pretrained Transformer and goes through the same 9,560 post-training updates, with the hyperparameters of its paper and official code (we did not search them ourselves) and a compression ratio chosen to match SSA's 576 tokens at 8K (Table 23). We apply DMS from the pretrained Transformer and do not pretrain with it, because of time constraints and because DMS is designed to be applied to a trained model. DMS chooses separately for each layer and head: each head keeps 344 tokens on average against SSA's 576, but across its 96 layer and head pairs DMS holds keys and values from about 4,480 distinct tokens, where SSA holds the same 576 at every layer and head. We acknowledge this as an advantage of DMS; SSA can be extended to choose per layer and head as well, which we have not done at this scale. The two are nearly equivalent on MKVT at 2K and 4K, with DMS leading by a few percentage points, and on MQAR, and DMS's clear win is perplexity (20.92 against 23.07 at 2K, and 22.21 against 26.91 at 8K). On MKVT at 8K they are nearly equivalent, with SSA leading by one percentage point (96.57 against 95.43). SSA is ahead on RULER at 8K (56.40 against 50.84) and on k-hop (99.98 / 99.97 / 99.54 against 99.95 / 97.55 / 38.61 token accuracy at 4 / 8 / 16 hops, excluding the null symbol). The 3M comparison with laLTE, which learns token eviction in a hybrid model, remains in the appendix. As the general response explains, these methods also compose with SSA: they can be run on top of its retained set.

The AI-use statement. Thank you for catching this. The paper contains a derivation, not a proof, and the wording "proof" was used colloquially. We have corrected the statement: it now says that AI tools were used to check the identity in Section 3.2 that relates the soft-phase gradient to counterfactual credit.

The title. We take the point that "State Space" suggests an SSM. We used it to mean a bounded state, in the sense that SSA's memory is a fixed-size set of key-value chunks. We will consider renaming the paper, if the reviewers and the area chair agree.

Writing. We apologize for the state of the draft. We have made these changes:

  • removed the leftover note at L377;
  • removed the duplicated clause in the credit-assignment passage (L263 to 265);
  • moved the main 400M results into the main text, so that the central claims no longer depend on the appendix;
  • revised informal phrasing throughout.

Thank you for the response and revision. The overclaims and draft errors are fixed. However, the 400M results are single-seed/scale and fragile, and they are well below the scales used in competing work. I believe the paper needs a major revision, and I would suggest testing at least three scales (e.g., ~400M, 760M, and 1.3B), as is standard in this literature. I maintain my rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers 1/2

Comment 1 of 2.

We thank the three reviewers for their careful reading and their time. The reviews agree on several points, and we agree with them:

  • The abstract and two sentences in the results overstated what the experiments show.
  • The 400M evidence was limited to pretraining perplexity.
  • The training components lacked ablations.
  • Learned-eviction and KV-compression baselines were discussed but not compared.
  • The writing had draft-quality errors.

We have used the rebuttal period, and the period since submission, mainly to run new experiments. All of them are at the 400M scale. Besides the five models of the submission's Table 2, we pretrained three new models on the same 15B SlimPajama tokens: a Transformer with a 576-token window, SSA with recovery during pretraining, and SSA with a larger scorer (4 layers of width 896, with the reader's feed-forward width reduced to keep about 400M parameters). Their held-out NLL at 2K / 8K is 2.695 / 2.686, 2.650 / 3.626 and 2.665 / 4.191 nats; the revised pretraining table reports all eight models. The post-training experiments below start from these pretrained models. Numbers are single-seed unless stated. Table numbers refer to the revised paper. As there are many updates, we report all headline numbers here, and the rebuttal can be read in isolation. The supplementary code is the version submitted with the original paper; we will release the code for the new experiments with the final version.

  1. Long-context post-training at 400M (new). We post-trained nine models on a mixture of long-context tasks: SSA with its scorer frozen or trained in the final stage; Transformers with full 8K attention (pretrained at 2K or 8K) or with 2K or 576-token sliding windows, including a new Transformer pretrained natively with a 576-token window to control for training context; GDN; and HOLA. The main task mixture is:

    • multi-key variable tracking (MKVT) at 2K, 4K and 8K;
    • multi-query associative recall (MQAR);
    • the 13 RULER tasks at 4K and 8K;
    • k-hop induction with 4, 8 and 16 hops.

    This came to 9,560 updates (611,840 examples) per model, with SSA reading 576 tokens at every context length and the full-attention Transformer reading up to 8,192. With 576 readable tokens SSA reaches 96.57% on MKVT at 8K, 99.95% on MQAR and 56.40 on RULER at 8K. The full-attention Transformers reach 99.60 to 99.90%, 99.98 to 99.99% and 70.72 to 76.86, and a Transformer with a 576-token window 8.13%, 85.74% and 14.61 (Table 2). On k-hop, which fits in SSA’s 576 readable tokens, SSA reaches 99.984%, 99.971% and 99.536% token accuracy at 4, 8 and 16 hops, against 99.997%, 99.976% and 99.457% for the 8K-pretrained Transformer; GDN and HOLA stay at chance, about 26% (Table 2).

  2. KV-cache eviction and compaction baselines (new). We applied StreamingLLM, H2O, TOVA and SnapKV to the full-attention 400M Transformers at SSA's 576-token budget. We also ran H2O and TOVA with the cache sizes and rules of their own papers. At SSA's budget, the rules that do not see the question reach 5 to 8% on MKVT at 8K, against 96.6% for SSA; SnapKV, which sees the question, reaches 99.1% and 81.4% on the two Transformers (Table 22). We also trained a learned-eviction method, Dynamic Memory Sparsification (DMS; Łańcucki et al., 2025), at 400M through the same post-training as the full-attention Transformer, with a compression ratio that matches SSA's 576 tokens at 8K (Table 23). DMS chooses separately for each layer and head: each head keeps 344 tokens on average against SSA's 576, but across its 96 layer and head pairs DMS holds keys and values from about 4,480 distinct tokens, where SSA holds the same 576 at every layer and head. With that advantage it is nearly equivalent to SSA on MKVT at 2K and 4K, with DMS leading by a few percentage points, and on MQAR, and its clear win is perplexity (20.92 against 23.07 at 2K). On MKVT at 8K the two are nearly equivalent, with SSA leading by one percentage point (96.57 against 95.43). SSA is ahead on RULER at 8K (56.40 against 50.84) and on k-hop (99.98 / 99.97 / 99.54 against 99.95 / 97.55 / 38.61 token accuracy at 4 / 8 / 16 hops).

AuthorsOpenReview ↗

General Response 2/2

Comment 2 of 2.

  1. Component ablations (new). Starting from the same pretrained SSA, each arm removes one component during post-training: counterfactual recovery, rescoring of residents, the context given to the scorer, the soft phase, or the straight-through estimator. Each arm has two runs. No run without recovery learned the retrieval task (0 of 2); the full recipe learned it in 1 of 3 runs, and hard-only training and both scorers without rescoring in 2 of 2 (Table 21).
  2. Multi-hop reasoning under sustained eviction (new). A k-hop task is hidden in 8,192-token rows of SlimPajama text, so SSA must keep the source across about 480 eviction decisions. With the source at the start and training from each model's k-hop checkpoint, SSA reaches 99.2% token accuracy against 99.9% for full attention, while the 576-window Transformer, GDN and HOLA stay at chance; with the source at a random position every model is at chance (Table 24).
  3. Diagnostics (new). Capping query-key distances at the pretraining length explains about half of SSA's perplexity rise from 2K to 8K. Replacing the learned retention rule with a sliding window over the most recent chunks at evaluation removes all of it (20.81 at 2K, 20.16 at 8K), so the rise comes from reading the old chunks the policy keeps (Tables 25 and 26). We also implement a small retention probe to separate what the memory keeps from what the reader can use in the multi-hop task under sustained eviction.
  4. Writing. We have corrected the overstated claims and removed the leftover note and the duplicated sentence. We have fixed the AI-use statement, described the scorer in full, and moved the main 400M results into the main text.

How SSA relates to cache eviction and compression. The comparisons in item 2 put SSA and these methods at the same budget, but we believe the two answer different questions. Eviction and compression methods start from a model that reads the whole context and reduce what it stores. SSA is a model with a fixed readable budget of R tokens, and learns to make those R tokens useful for retrieval and reasoning over inputs many times longer: 24 times its budget in our small-scale experiments and 14 times at 400M. Its natural baseline is therefore the R-token Transformer, which it far exceeds on long inputs (96.6% against 8.1% on MKVT at 8K with the same 576 tokens). The two approaches also compose. SSA's reader is standard attention over its retained set, so an eviction or sparsification method such as DMS can be run on top of SSA's memory to evict further. For a sketch, an SSA with a readable budget of 1M tokens that extends to 10M-token inputs could have DMS applied within its 1M tokens. We have not run this combination, and the paper states it as a property of the design, not as a result. What the paper does test (at small scale) is stacking: three GDN layers followed by one SSA layer at the 3M scale (appendix).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.