- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SketchSSM to reduce recurrent-state memory access during decoding in Mamba-2, GDN, and KDA models. Existing methods such as ReplaySSM reduce state-write overhead by buffering keys and values, but still require full-state reads at every decoding step. SketchSSM preserves exact full-state updates while approximating state-read outputs using compact, low-rank representations. It calibrates query basis vectors offline and precomputes their outputs whenever the full state is updated. During subsequent decoding steps, the model reconstructs outputs from these precomputed vectors and query-dependent coefficients, avoiding repeated full-state reads. The method also allocates different sketch ranks across attention heads based on reconstruction error and loss sensitivity. Experiments on four models show approximately 10× lower state-access traffic while largely preserving average accuracy. SketchSSM achieves up to 7.78× linear-attention kernel speedup and 2.64× higher end-to-end decode throughput on Nemotron 3 Super.
Critical strengths
1: Effective reduction of state-read overhead. Unlike ReplaySSM and KVBuffer, which mainly reduce state-write overhead through buffered updates, SketchSSM also reduces repeated full-state reads. It preserves exact state updates while approximating only the read outputs, avoiding errors caused by directly compressing the recurrent state. The ablation results support this design choice.
2: Strong experimental and implementation evidence. The paper evaluates four models across Mamba-2, GDN, and KDA, with comparisons against state quantization, pruning, and ReplaySSM. It reports both kernel-level speedups and end-to-end throughput improvements, supported by detailed ablation studies. These results provide useful evidence of its practical benefits for large-batch decoding.
Critical weaknesses
1、Limited evaluation of long-context reasoning and long-generation stability. The paper mainly evaluates model accuracy on MATH-500, AIME25, GPQA Diamond, and LiveCodeBench. Long-context evaluation is limited to four RULER retrieval tasks at 16K context length. Some models already achieve 100% accuracy on these tasks, making it difficult to assess whether SketchSSM preserves more challenging long-context reasoning and information integration abilities. Moreover, although full-state updates remain exact, approximate read errors may still affect subsequent token generation. The paper lacks a detailed analysis of error accumulation during long autoregressive generation.
2、Lack of training-based adaptation experiments.
The paper focuses on a training-free approach, using offline calibration to determine query bases and head-specific sketch ranks without updating model weights. However, accuracy still drops noticeably at lower sketch ranks. The paper does not investigate whether SFT or RL could help models adapt to approximate state reads and recover lost accuracyLack of training-based adaptation experiments.
Questions for authors
What is the main source of the low-rank property observed in state-read outputs?
Figure 2 shows that a small number of query basis vectors can preserve most of the state-read output energy. However, it remains unclear whether this property mainly comes from the query distribution, the recurrent state matrix, or their interaction.
Could the authors compare the singular value or eigenvalue distributions of the query covariance, the state Gram matrix, and the state-weighted query covariance?
It would also be useful to examine whether this property remains consistent across layers, heads, context lengths, and tasks.
5 comments
Response to Reviewer uNHi (2/4)
W1. Long-context reasoning and long-generation stability
Limited evaluation of long-context reasoning and long-generation stability. The paper mainly evaluates model accuracy on MATH-500, AIME25, GPQA Diamond, and LiveCodeBench. Long-context evaluation is limited to four RULER retrieval tasks at 16K context length. Some models already achieve 100% accuracy on these tasks, making it difficult to assess whether SketchSSM preserves more challenging long-context reasoning and information integration abilities. Moreover, although full-state updates remain exact, approximate read errors may still affect subsequent token generation. The paper lacks a detailed analysis of error accumulation during long autoregressive generation.
(1) The evaluated benchmarks are long-generation tasks: MATH-500, AIME25, GPQA Diamond and LiveCodeBench are reasoning benchmarks whose responses are long: under Standard, the mean number of generated tokens per problem is 1.2K–5.3K on MATH-500, 18.9K–22.6K on AIME25, 10.8K–18.9K on GPQA Diamond and 14.1K–24.5K on LiveCodeBench across the four models, with generations up to the cap of 32K (Nemotron Nano) or 65K tokens (the other models). At W=16, a 20K-token response spans more than 1,200 windows of sketched reads. This is the setting in which read errors would have to accumulate, and SketchSSM at Ḡ=8 matches Standard on all four models. The revised Section 5 states this explicitly.
(2) Error accumulation over long generations (new analysis in the revised paper, Section 5.1 and Fig. 6): We group the problems of the four benchmarks by how many tokens Standard generates for them and measure every method on the same problems. If errors accumulate during generation, a method falls further behind Standard as the generation gets longer. The table gives the relative change in accuracy from Standard on Nemotron 3 Super, with the traffic reduction in parentheses:
| Generated tokens (Standard) | <4K | 4–8K | 8–16K | 16–32K | ≥32K |
|---|---|---|---|---|---|
| Problems | 569 | 119 | 105 | 117 | 133 |
| Standard accuracy (%) (1.0×) | 98.8 | 96.4 | 94.0 | 85.7 | 34.7 |
| SketchSSM Ḡ=8 (9.4×) | 0% | 0% | 0% | −4% | +16% |
| State quantization 8-bit (3.8×) | 0% | −1% | −9% | −25% | −67% |
| State quantization 6-bit (5.1×) | 0% | −7% | −23% | −65% | −85% |
| State pruning 50% (2.0×) | −3% | −16% | −27% | −55% | −60% |
| State pruning 62.5% (2.7×) | −21% | −54% | −76% | −86% | −87% |
Quantization and pruning match Standard on short generations but lose a growing fraction of their accuracy as the generation gets longer: 8-bit quantization goes from no loss below 4K tokens to 25% at 16–32K tokens and 67% beyond 32K. This is the signature of errors compounding through state updates. SketchSSM at Ḡ=8 shows no such trend and matches Standard in every group, with a 2.5× larger traffic reduction than 8-bit quantization. The ≥32K group is hard for every method (Standard reaches 34.7%), so small absolute differences become large relative ones there; the +16% of SketchSSM corresponds to 5.5 points.
(3) Long-context benchmarks with short answers: Long-context benchmarks with short answers, such as RULER and LongBench, mainly measure the quality of the state built by prefill, which SketchSSM leaves unchanged, so we examined long generations instead (point (2) above). SketchSSM changes only decode: prefill runs the same full-state computation as Standard, so the state after a long context is identical, and the approximation enters only through reads during generation. On RULER at 16K, recall stays within 0.4 points of Standard at Ḡ ∈ {8, 4}, and for the same reason we did not add further benchmarks of this kind, such as LongBench.
Response to Reviewer uNHi (3/4)
Q. Source of the low-rank property
What is the main source of the low-rank property observed in state-read outputs?
Figure 2 shows that a small number of query basis vectors can preserve most of the state-read output energy. However, it remains unclear whether this property mainly comes from the query distribution, the recurrent state matrix, or their interaction.
Could the authors compare the singular value or eigenvalue distributions of the query covariance, the state Gram matrix, and the state-weighted query covariance?
It would also be useful to examine whether this property remains consistent across layers, heads, context lengths, and tasks.
We thank the reviewer for this question. We have added the requested comparison to Appendix H.2 (Fig. 13). In short:
(1) The low rank of state-read outputs comes mainly from the state: The read output retained by a calibrated sketch at rank Ḡ equals the share of the Ḡ leading eigenvalues of the state-weighted query covariance M, so Figure 2 shows that M is concentrated in a few directions. Comparing the three spectra, the state Gram is by far the most concentrated, the query covariance is much flatter, and M lies close to the state.
(2) The property is consistent across tasks, context lengths, layers, and heads: The states are approximately low-rank, as prior work has observed, and our measurements show that their leading directions are shared across corpora and context lengths; that is, they are a property of the trained model rather than the input. The degree of concentration varies across layers and heads, which is why SketchSSM allocates the rank per head.
(3) As a result, the calibration does not depend on the dataset: Recalibrating the sketching matrices and the rank allocation on other corpora leaves accuracy unchanged within the standard deviation across seeds.
(4) What differs from prior work is where the low rank is used: Prior work uses it to compress the state itself, so the approximation error carries forward through every later update. SketchSSM keeps exact full-state updates and uses the low rank only in the read.
We give the details below.
Comparison of the three distributions: Under the calibration objective, the read output energy that a calibrated test matrix Ω retains at rank Ḡ equals the share of the Ḡ leading eigenvalues of the state-weighted query covariance M (Appendix A.1). The observation in Figure 2 therefore means that the spectrum of M is concentrated in a few directions. The table below compares the eigenvalue distributions of the three matrices on Nemotron Nano 9B v2: the energy in the leading one and four directions (mean over heads) and the participation rank (Σλᵢ)²/Σλᵢ², the effective number of directions (median over heads).
| Corpus | Matrix | Top-1 energy | Top-4 energy | Participation rank |
|---|---|---|---|---|
| General | Query covariance | 59% | 73% | 2.8 |
| State Gram | 84% | 91% | 1.3 | |
| State-weighted query covariance | 75% | 87% | 1.5 | |
| Code | Query covariance | 59% | 74% | 2.8 |
| State Gram | 84% | 91% | 1.3 | |
| State-weighted query covariance | 76% | 87% | 1.4 | |
| Math | Query covariance | 60% | 75% | 2.7 |
| State Gram | 84% | 91% | 1.3 | |
| State-weighted query covariance | 77% | 88% | 1.4 |
The property comes mainly from the low rank of the state. The state is the most concentrated, the query alone is much flatter, and M lies close to the state. The state strongly amplifies its leading direction even when the query barely points to it: the mean query is nearly orthogonal to that direction (squared cosine 0.001), yet 82–89% of its read output comes from it.
Response to Reviewer uNHi (1/4)
Thank you for your thoughtful review and constructive feedback. We address your concerns below.
Common: Revised evaluation
Revised evaluation. We first note that we re-ran the entire accuracy evaluation after submission, under a single protocol:
Average accuracy over the four tasks (state-access traffic reduction in parentheses):
At Ḡ=8, SketchSSM matches Standard on all four models within one standard deviation, at about a 10× reduction of state-access traffic. Lower ranks reduce traffic further at the cost of accuracy. However, Ḡ=8 already accelerates decoding substantially, and lowering the rank to Ḡ=4 raises throughput by only 1–7% further.
W2. No training-based adaptation
We agree that training-based adaptation is a promising direction, but SketchSSM already removes most of the linear-attention cost without training, so we found little need for the lower ranks where adaptation would help. SketchSSM is designed to be training-free, so that it applies to any released checkpoint, and at Ḡ=8 it already matches Standard on all four models at about 10× less state-access traffic. Going further to Ḡ=4 raises the maximum end-to-end throughput by only 6–7% on Nemotron 3 Super and 1–2% on Qwen3.8 Flash-Next and GLM 5.3 Flash (Section 5.2). We therefore see the training-free Ḡ=8 setting as sufficient in practice, and leave training-based extensions to future work.