acceptodds
Under review as a conference paper at ICLR 2027

SketchSSM: Write to the Full State, Read from a Compact Sketch

Abstract

Hybrid-attention models replace most softmax attention layers with linear attention, reducing KV-cache growth and enabling larger decode batches where recurrent-state access becomes a major bottleneck. ReplaySSM amortizes state updates by buffering keys and values, but each new query still requires a full-state read even though the state remains unchanged between state updates. We observe that low-rank state-weighted query approximation accurately preserves state-read outputs. Although future queries are unknown, the basis vectors used to approximate them can be fixed offline. Based on this observation, we introduce SketchSSM, which preserves full-state updates while approximating reads. At each state update, SketchSSM reads the full state once to precompute outputs for these basis vectors, storing them in a compact sketch. Each subsequent decode step combines the sketch vectors with query-dependent coefficients to reconstruct the output without a full-state read. Across four Mamba-2-, GDN-, and KDA-based models, SketchSSM reduces state-access traffic by approximately 10 while largely preserving average accuracy across four decode benchmarks and recall on four RULER retrieval tasks. On one NVIDIA B300, linear-attention kernel speedups over the standard vLLM baseline reach 7.78, 5.22, and 5.20 for Mamba-2, GDN, and KDA, respectively, with up to 2.64 higher decode throughput on Nemotron 3 Super.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
linear attention, state space model, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uNHiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SketchSSM to reduce recurrent-state memory access during decoding in Mamba-2, GDN, and KDA models. Existing methods such as ReplaySSM reduce state-write overhead by buffering keys and values, but still require full-state reads at every decoding step. SketchSSM preserves exact full-state updates while approximating state-read outputs using compact, low-rank representations. It calibrates query basis vectors offline and precomputes their outputs whenever the full state is updated. During subsequent decoding steps, the model reconstructs outputs from these precomputed vectors and query-dependent coefficients, avoiding repeated full-state reads. The method also allocates different sketch ranks across attention heads based on reconstruction error and loss sensitivity. Experiments on four models show approximately 10× lower state-access traffic while largely preserving average accuracy. SketchSSM achieves up to 7.78× linear-attention kernel speedup and 2.64× higher end-to-end decode throughput on Nemotron 3 Super.

Critical strengths

1: Effective reduction of state-read overhead. Unlike ReplaySSM and KVBuffer, which mainly reduce state-write overhead through buffered updates, SketchSSM also reduces repeated full-state reads. It preserves exact state updates while approximating only the read outputs, avoiding errors caused by directly compressing the recurrent state. The ablation results support this design choice.

2: Strong experimental and implementation evidence. The paper evaluates four models across Mamba-2, GDN, and KDA, with comparisons against state quantization, pruning, and ReplaySSM. It reports both kernel-level speedups and end-to-end throughput improvements, supported by detailed ablation studies. These results provide useful evidence of its practical benefits for large-batch decoding.

Critical weaknesses

1、Limited evaluation of long-context reasoning and long-generation stability. The paper mainly evaluates model accuracy on MATH-500, AIME25, GPQA Diamond, and LiveCodeBench. Long-context evaluation is limited to four RULER retrieval tasks at 16K context length. Some models already achieve 100% accuracy on these tasks, making it difficult to assess whether SketchSSM preserves more challenging long-context reasoning and information integration abilities. Moreover, although full-state updates remain exact, approximate read errors may still affect subsequent token generation. The paper lacks a detailed analysis of error accumulation during long autoregressive generation.

2、Lack of training-based adaptation experiments.

The paper focuses on a training-free approach, using offline calibration to determine query bases and head-specific sketch ranks without updating model weights. However, accuracy still drops noticeably at lower sketch ranks. The paper does not investigate whether SFT or RL could help models adapt to approximate state reads and recover lost accuracyLack of training-based adaptation experiments.

Questions for authors

What is the main source of the low-rank property observed in state-read outputs?

Figure 2 shows that a small number of query basis vectors can preserve most of the state-read output energy. However, it remains unclear whether this property mainly comes from the query distribution, the recurrent state matrix, or their interaction.

Could the authors compare the singular value or eigenvalue distributions of the query covariance, the state Gram matrix, and the state-weighted query covariance?

It would also be useful to examine whether this property remains consistent across layers, heads, context lengths, and tasks.

5 comments
AuthorsOpenReview ↗

Response to Reviewer uNHi (1/4)

Thank you for your thoughtful review and constructive feedback. We address your concerns below.

Common: Revised evaluation

Revised evaluation. We first note that we re-ran the entire accuracy evaluation after submission, under a single protocol:

  • Repeated runs. To reduce run-to-run noise, we now run every SketchSSM configuration with 3 seeds (8 seeds for AIME25, which has 30 problems), and we report mean ± std.
  • One rank set for all models. The submission used a different set of mean ranks per model, which made it hard to compare up to which rank each model preserves Standard accuracy. All four models are now evaluated at Ḡ ∈ {8, 4, 2}.
  • Ridge λ. The ridge in the coefficient solve only keeps the BF16 solve numerically stable. A value near the BF16 precision (2⁻⁸ ≈ 0.004) is therefore enough, and anything larger biases the coefficients. The submitted version mistakenly used λ = 0.1, which added 10% of the mean eigenvalue of the normalized state metric to its diagonal. We lowered it to λ = 0.003 for every model and rank and re-ran the evaluation. Correcting this bias turned out to recover part of the sharp accuracy drop at low ranks (Nemotron Nano at Ḡ=2: 56.37 → 68.24 under the same protocol).

Average accuracy over the four tasks (state-access traffic reduction in parentheses):

ModelStandardReplaySSMSketchSSM Ḡ=8Ḡ=4Ḡ=2
Nemotron Nano 9B v2 (Mamba-2)74.36 ± 0.9975.1574.41 ± 0.91 (9.94×)71.93 (12.26×)68.24 (13.88×)
Nemotron 3 Super (Mamba-2)84.56 ± 0.8084.5084.67 ± 1.19 (9.39×)84.43 (11.84×)81.76
Qwen3.8 Flash-Next (GDN)90.65 ± 0.6491.0190.63 ± 0.74 (10.68×)88.59 (12.81×)83.15 (14.23×)
GLM 5.3 Flash (KDA)86.03 ± 1.3086.8588.48 ± 1.19 (10.68×)87.3780.91

At Ḡ=8, SketchSSM matches Standard on all four models within one standard deviation, at about a 10× reduction of state-access traffic. Lower ranks reduce traffic further at the cost of accuracy. However, Ḡ=8 already accelerates decoding substantially, and lowering the rank to Ḡ=4 raises throughput by only 1–7% further.

W2. No training-based adaptation

Lack of training-based adaptation experiments. The paper focuses on a training-free approach, using offline calibration to determine query bases and head-specific sketch ranks without updating model weights. However, accuracy still drops noticeably at lower sketch ranks. The paper does not investigate whether SFT or RL could help models adapt to approximate state reads and recover lost accuracy.

We agree that training-based adaptation is a promising direction, but SketchSSM already removes most of the linear-attention cost without training, so we found little need for the lower ranks where adaptation would help. SketchSSM is designed to be training-free, so that it applies to any released checkpoint, and at Ḡ=8 it already matches Standard on all four models at about 10× less state-access traffic. Going further to Ḡ=4 raises the maximum end-to-end throughput by only 6–7% on Nemotron 3 Super and 1–2% on Qwen3.8 Flash-Next and GLM 5.3 Flash (Section 5.2). We therefore see the training-free Ḡ=8 setting as sufficient in practice, and leave training-based extensions to future work.


AuthorsOpenReview ↗

Response to Reviewer uNHi (2/4)

W1. Long-context reasoning and long-generation stability

Limited evaluation of long-context reasoning and long-generation stability. The paper mainly evaluates model accuracy on MATH-500, AIME25, GPQA Diamond, and LiveCodeBench. Long-context evaluation is limited to four RULER retrieval tasks at 16K context length. Some models already achieve 100% accuracy on these tasks, making it difficult to assess whether SketchSSM preserves more challenging long-context reasoning and information integration abilities. Moreover, although full-state updates remain exact, approximate read errors may still affect subsequent token generation. The paper lacks a detailed analysis of error accumulation during long autoregressive generation.

(1) The evaluated benchmarks are long-generation tasks: MATH-500, AIME25, GPQA Diamond and LiveCodeBench are reasoning benchmarks whose responses are long: under Standard, the mean number of generated tokens per problem is 1.2K–5.3K on MATH-500, 18.9K–22.6K on AIME25, 10.8K–18.9K on GPQA Diamond and 14.1K–24.5K on LiveCodeBench across the four models, with generations up to the cap of 32K (Nemotron Nano) or 65K tokens (the other models). At W=16, a 20K-token response spans more than 1,200 windows of sketched reads. This is the setting in which read errors would have to accumulate, and SketchSSM at Ḡ=8 matches Standard on all four models. The revised Section 5 states this explicitly.

(2) Error accumulation over long generations (new analysis in the revised paper, Section 5.1 and Fig. 6): We group the problems of the four benchmarks by how many tokens Standard generates for them and measure every method on the same problems. If errors accumulate during generation, a method falls further behind Standard as the generation gets longer. The table gives the relative change in accuracy from Standard on Nemotron 3 Super, with the traffic reduction in parentheses:

Generated tokens (Standard)<4K4–8K8–16K16–32K≥32K
Problems569119105117133
Standard accuracy (%) (1.0×)98.896.494.085.734.7
SketchSSM Ḡ=8 (9.4×)0%0%0%−4%+16%
State quantization 8-bit (3.8×)0%−1%−9%−25%−67%
State quantization 6-bit (5.1×)0%−7%−23%−65%−85%
State pruning 50% (2.0×)−3%−16%−27%−55%−60%
State pruning 62.5% (2.7×)−21%−54%−76%−86%−87%

Quantization and pruning match Standard on short generations but lose a growing fraction of their accuracy as the generation gets longer: 8-bit quantization goes from no loss below 4K tokens to 25% at 16–32K tokens and 67% beyond 32K. This is the signature of errors compounding through state updates. SketchSSM at Ḡ=8 shows no such trend and matches Standard in every group, with a 2.5× larger traffic reduction than 8-bit quantization. The ≥32K group is hard for every method (Standard reaches 34.7%), so small absolute differences become large relative ones there; the +16% of SketchSSM corresponds to 5.5 points.

(3) Long-context benchmarks with short answers: Long-context benchmarks with short answers, such as RULER and LongBench, mainly measure the quality of the state built by prefill, which SketchSSM leaves unchanged, so we examined long generations instead (point (2) above). SketchSSM changes only decode: prefill runs the same full-state computation as Standard, so the state after a long context is identical, and the approximation enters only through reads during generation. On RULER at 16K, recall stays within 0.4 points of Standard at Ḡ ∈ {8, 4}, and for the same reason we did not add further benchmarks of this kind, such as LongBench.


AuthorsOpenReview ↗

Response to Reviewer uNHi (3/4)

Q. Source of the low-rank property

What is the main source of the low-rank property observed in state-read outputs?

Figure 2 shows that a small number of query basis vectors can preserve most of the state-read output energy. However, it remains unclear whether this property mainly comes from the query distribution, the recurrent state matrix, or their interaction.

Could the authors compare the singular value or eigenvalue distributions of the query covariance, the state Gram matrix, and the state-weighted query covariance?

It would also be useful to examine whether this property remains consistent across layers, heads, context lengths, and tasks.

We thank the reviewer for this question. We have added the requested comparison to Appendix H.2 (Fig. 13). In short:

(1) The low rank of state-read outputs comes mainly from the state: The read output retained by a calibrated sketch at rank Ḡ equals the share of the Ḡ leading eigenvalues of the state-weighted query covariance M, so Figure 2 shows that M is concentrated in a few directions. Comparing the three spectra, the state Gram is by far the most concentrated, the query covariance is much flatter, and M lies close to the state.

(2) The property is consistent across tasks, context lengths, layers, and heads: The states are approximately low-rank, as prior work has observed, and our measurements show that their leading directions are shared across corpora and context lengths; that is, they are a property of the trained model rather than the input. The degree of concentration varies across layers and heads, which is why SketchSSM allocates the rank per head.

(3) As a result, the calibration does not depend on the dataset: Recalibrating the sketching matrices and the rank allocation on other corpora leaves accuracy unchanged within the standard deviation across seeds.

(4) What differs from prior work is where the low rank is used: Prior work uses it to compress the state itself, so the approximation error carries forward through every later update. SketchSSM keeps exact full-state updates and uses the low rank only in the read.

We give the details below.

Comparison of the three distributions: Under the calibration objective, the read output energy that a calibrated test matrix Ω retains at rank Ḡ equals the share of the Ḡ leading eigenvalues of the state-weighted query covariance M (Appendix A.1). The observation in Figure 2 therefore means that the spectrum of M is concentrated in a few directions. The table below compares the eigenvalue distributions of the three matrices on Nemotron Nano 9B v2: the energy in the leading one and four directions (mean over heads) and the participation rank (Σλᵢ)²/Σλᵢ², the effective number of directions (median over heads).

CorpusMatrixTop-1 energyTop-4 energyParticipation rank
GeneralQuery covariance59%73%2.8
State Gram84%91%1.3
State-weighted query covariance75%87%1.5
CodeQuery covariance59%74%2.8
State Gram84%91%1.3
State-weighted query covariance76%87%1.4
MathQuery covariance60%75%2.7
State Gram84%91%1.3
State-weighted query covariance77%88%1.4

The property comes mainly from the low rank of the state. The state is the most concentrated, the query alone is much flatter, and M lies close to the state. The state strongly amplifies its leading direction even when the query barely points to it: the mean query is nearly orthogonal to that direction (squared cosine 0.001), yet 82–89% of its read output comes from it.


Reviewer_NHEZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SketchSSM to address the recurrent-state access bottleneck of linear-attention layers in hybrid-attention LLM decoding.

Critical strengths
  1. Sharp problem identification and strong motivation: profiling clearly establishes linear-attention state-read as the dominant decode bottleneck and shows that ReplaySSM addresses only writes, not reads, filling a well-defined optimization gap.

  2. Clever “exact update + approximate read” decoupling with ablation support: retaining the full FP32 state for flush updates while approximating only the read path prevents state-compression errors from propagating through recurrent updates; Table 12 shows clear gains over direct quantization or low-rank state representations.

  3. Complete derivations, solid engineering, and broad evaluation: the method derives PCA calibration, least-squares coefficients, and a range-finding form, with low-rank-plus-diagonal and Woodbury acceleration; it is integrated into vLLM with kernel and end-to-end speed measurements, covering three architectures, multiple benchmarks, RULER, and comparisons against quantization/pruning.

Critical weaknesses
  1. Offline fixed Ω risks distribution shift: the basis is fixed from WikiText-2-calibrated average state geometry and never updated at inference; there is no OOD calibration, cross-dataset calibration, or calibration-size sensitivity study, so the low-rank subspace may be suboptimal under distribution shift in real serving.

  2. Clear accuracy and verbosity degradation at very low rank: at G_hat=2, Nemotron Nano average accuracy drops from 73.38 to 55.21 and verbosity rises to 1.42; the method is not robust in the aggressive-compression regime, and generated length increases, while the paper under-emphasizes these operating boundaries.

  3. Insufficient analysis of flush overhead and window-size sensitivity: each flush still reads the full S_0 and constructs U,C; W=16 is a fixed hyperparameter with no sensitivity study; exact coefficient-map construction is expensive and relies on P, λ approximations, adding engineering complexity.

  4. Complex rank-allocation deployment and limited optimality: head-level allocation requires offline loss gradients and reconstruction residuals plus a multiple-choice knapsack/Lagrangian solve; the score is a first-order separable surrogate that omits cross-head terms, and stability across batch sizes and serving scenarios is not validated.

  5. Narrow end-to-end evaluation scope: throughput is reported only on Nemotron 3 Super, while other models are evaluated mainly at kernel latency; accuracy benchmarks and 16K RULER coverage are limited, and there is no end-to-end validation on longer contexts, multi-turn dialogue, prefill/scheduling interactions, or different hardware bandwidths.

Questions for authors

In real service distribution shifts and long-context scenarios, can offline fixed query basis Ω still reliably support low-rank reconstruction of state-read?

4 comments
AuthorsOpenReview ↗

Response to Reviewer NHEZ (1/3)

Thank you for your thoughtful review and constructive feedback. We address your concerns below.

Common: Revised evaluation

Revised evaluation. We first note that we re-ran the entire accuracy evaluation after submission, under a single protocol:

  • Repeated runs. To reduce run-to-run noise, we now run every SketchSSM configuration with 3 seeds (8 seeds for AIME25, which has 30 problems), and we report mean ± std.
  • One rank set for all models. The submission used a different set of mean ranks per model, which made it hard to compare up to which rank each model preserves Standard accuracy. All four models are now evaluated at Ḡ ∈ {8, 4, 2}.
  • Ridge λ. The ridge in the coefficient solve only keeps the BF16 solve numerically stable. A value near the BF16 precision (2⁻⁸ ≈ 0.004) is therefore enough, and anything larger biases the coefficients. The submitted version mistakenly used λ = 0.1, which added 10% of the mean eigenvalue of the normalized state metric to its diagonal. We lowered it to λ = 0.003 for every model and rank and re-ran the evaluation. Correcting this bias turned out to recover part of the sharp accuracy drop at low ranks (Nemotron Nano at Ḡ=2: 56.37 → 68.24 under the same protocol).

Average accuracy over the four tasks (state-access traffic reduction in parentheses):

ModelStandardReplaySSMSketchSSM Ḡ=8Ḡ=4Ḡ=2
Nemotron Nano 9B v2 (Mamba-2)74.36 ± 0.9975.1574.41 ± 0.91 (9.94×)71.93 (12.26×)68.24 (13.88×)
Nemotron 3 Super (Mamba-2)84.56 ± 0.8084.5084.67 ± 1.19 (9.39×)84.43 (11.84×)81.76
Qwen3.8 Flash-Next (GDN)90.65 ± 0.6491.0190.63 ± 0.74 (10.68×)88.59 (12.81×)83.15 (14.23×)
GLM 5.3 Flash (KDA)86.03 ± 1.3086.8588.48 ± 1.19 (10.68×)87.3780.91

At Ḡ=8, SketchSSM matches Standard on all four models within one standard deviation, at about a 10× reduction of state-access traffic. Lower ranks reduce traffic further at the cost of accuracy. However, Ḡ=8 already accelerates decoding substantially, and lowering the rank to Ḡ=4 raises throughput by only 1–7% further.

W2. Accuracy and verbosity degradation at very low rank

Clear accuracy and verbosity degradation at very low rank: at G_hat=2, Nemotron Nano average accuracy drops from 73.38 to 55.21 and verbosity rises to 1.42; the method is not robust in the aggressive-compression regime, and generated length increases, while the paper under-emphasizes these operating boundaries.

We agree, and the revision states the operating boundaries explicitly. Ḡ=8 is the recommended setting: with about 10× less state-access traffic, its average accuracy is within one standard deviation of Standard on all four models. Lower ranks trade accuracy for traffic: Ḡ=2 costs 2.8–7.5 points of average accuracy, depending on the model, for about 14× less traffic. We do not claim that Ḡ=2 preserves the accuracy of Standard.

The Ḡ=2 numbers that the reviewer cites come from the submission. With the unified protocol and lower-coefficient regularization (see the common response), Nemotron Nano 9B v2 at Ḡ=2 reaches 68.24 instead of 55.21, and its normalized verbosity is 1.21 instead of 1.42. Verbosity follows the same boundary as accuracy: at Ḡ=8 it is 1.04, 0.95, 1.00 and 0.86 on the four models, and it rises only at Ḡ=2 (1.21, 1.03, 1.58 and 1.06).

In practice, there is also little reason to go below Ḡ=8, because Ḡ=8 already removes most of the linear-attention cost. Linear attention falls to 22.3%, 10.2% and 8.7% of the decode step on Nemotron 3 Super, Qwen3.8 Flash-Next and GLM 5.3 Flash, and Ḡ=4 raises the maximum end-to-end throughput over Ḡ=8 by only 6–7% on Nemotron 3 Super and 1–2% on the other two models, at 0.2–2.0 points of average accuracy (Section 5.2). Therefore, even though there are some score drops in certain settings with Ḡ=2 and 4, we achieve most of the acceleration possible for linear attention with Ḡ=8 that matches the score of Standard.


AuthorsOpenReview ↗

Response to Reviewer NHEZ (2/3)

W1/Q1. Distribution shift and long contexts with a fixed basis

In real service distribution shifts and long-context scenarios, can offline fixed query basis Ω still reliably support low-rank reconstruction of state-read?

Yes. A single fixed basis covers different data distributions and long contexts, for the reasons below, which we verify in the revision (Appendix H).

(1) A low-rank basis suffices because the states are approximately low-rank. It is well known that linear-attention states are approximately low-rank, and we observe that the low rank of state-read outputs comes mainly from this property of the state (Appendix H.2).

(2) The leading directions of the states are set by the model, not by the input. For the same group of heads, the leading state direction is nearly the same across general text, code, and mathematics (squared cosine 0.94–0.96, versus 0.01 for random directions) and across context lengths from 512 to 8,192 tokens (0.96–0.99). It depends on the model itself rather than the calibration dataset. This cross-domain stability of state is also well known; offline state-pruning methods rely on the same stability when they fix the pruned dimensions offline.

(3) A basis calibrated on WikiText-2 therefore transfers to other domains and longer contexts. On code and mathematics, it retains 96.5% and 96.8% of the read output at G=4, close to a basis calibrated on the test domain itself (97.2% and 97.5%). Its retention stays unchanged as the context grows from 512 to 8,192 tokens (97.0% to 96.9%). Accordingly, SketchSSM at Ḡ=8 matches Standard on all four models on benchmarks unrelated to WikiText-2 (mathematics, science, and code).

(4) Recalibrating on other datasets gives the same accuracy. To probe this further, we recalibrated Nemotron Nano on the Pile, code, and mathematics corpora. Accuracy changes by at most 1.4 points on LiveCodeBench and 0.4 points on MATH-500, within the standard deviation across seeds, and matching the calibration domain to the benchmark does not help (table below).

Calibration datasetLCBMATH-500
Standard (no calibration)68.99 ± 1.2097.33 ± 0.42
WikiText-2 (default)68.15 ± 1.6397.07 ± 0.58
Pile67.94 ± 0.8496.73 ± 0.50
Code68.04 ± 1.0297.13 ± 0.42
Math69.31 ± 0.4897.13 ± 0.31

(5) What, then, differs from prior work that uses the low rank of the states? Where the low rank is used. Prior work uses it to compress the state itself, keeping a reduced state and updating it. The approximation then enters the state that stores the information, and its error is carried forward by every later update. SketchSSM keeps exact full-state updates and uses the low rank only in the read, with a sketch rebuilt from the exact state at every flush step. It thus does not directly compress the state that stores the information, which cuts the propagation of approximation error through subsequent updates. The difference shows over long generations (Section 5, Fig. 6). On Nemotron 3 Super, 8-bit state quantization matches Standard below 4K generated tokens but trails it by 21.7 points at 16–32K tokens, whereas SketchSSM at Ḡ=8 stays within 3.6 points of Standard in every group, at a 2.5× larger traffic reduction.

W4. Rank allocation is complex to deploy

Complex rank-allocation deployment and limited optimality: head-level allocation requires offline loss gradients and reconstruction residuals plus a multiple-choice knapsack/Lagrangian solve; the score is a first-order separable surrogate that omits cross-head terms, and stability across batch sizes and serving scenarios is not validated.

The allocation adds no cost at serving time. It is computed in the same offline calibration pass that produces the query basis Ω, which runs once per model, takes about 11 minutes for Nemotron Nano 9B v2 on one RTX PRO 6000, and produces a single 11 MB file. The mean rank is then chosen when the model is loaded, without recalibration, and the solve is a CPU-side step at startup that takes under a second. The allocation is per head and static, so it does not depend on the batch size. The score is indeed a first-order separable surrogate, and we do not claim it is optimal; the ablation shows that it is effective, improving average accuracy by 10.0 points over a uniform rank at Ḡ=4.


AuthorsOpenReview ↗

Response to Reviewer NHEZ (3/3)

W3. Flush overhead and window-size sensitivity

Insufficient analysis of flush overhead and window-size sensitivity: each flush still reads the full S_0 and constructs U,C; W=16 is a fixed hyperparameter with no sensitivity study; exact coefficient-map construction is expensive and relies on P, λ approximations, adding engineering complexity.

We added both analyses to the revised paper (Appendices I and J).

Window size: Accuracy is insensitive to the window size W ∈ {8, 16, 32}, and our default W=16 gives the lowest average latency (Nemotron Nano 9B v2, Ḡ=8, three seeds; kernel latency on one RTX PRO 6000 at batch size 256, µs per layer):

WLiveCodeBenchMATH-500Non-flush step (µs)Flush step (µs)Average step (µs)
Standard68.99 ± 1.2097.33 ± 0.42––1,836
868.15 ± 0.7397.13 ± 0.311011,962334
16 (default)68.15 ± 1.6397.07 ± 0.581181,995236
3268.25 ± 1.3896.27 ± 0.811642,617240

A larger window amortizes the flush step over more decode steps, but each non-flush step then reads more of the key-value buffer: on average 68, 158 and 338 KiB per request and layer at W=8, 16 and 32, against 293–304 KiB for the sketch and coefficient map. W=16 balances the two, and it is also the default window of ReplaySSM in vLLM.

Flush step: We added Appendix J to the revised paper, which describes and analyzes our flush kernel. The kernel incorporates three optimizations, detailed below. SketchSSM adds to the buffered-update flush the construction of the sketch and the coefficient map. Both need the updated full state, so building them in a separate pass reads the state twice. Our first optimization fuses their construction into the state update, so that the state is read once. Second, the flush kernel is memory-bound (roofline analysis in Appendix J), and its shared-memory copy pipeline looks only one tile ahead, so the kernel often waits for HBM; we therefore prefetch state blocks two blocks ahead into the L2 cache. Third, we tune the launch configuration for each GPU. On Nemotron Nano 9B v2 (one RTX PRO 6000, batch size 256, µs per layer):

Flush kernelLatencySpeedupMemory traffic (MiB)Bandwidth utilization
Separate construction pass3,8751.00×15.674%
+ Fusion3,3141.17×10.659%
+ L2 prefetch2,9281.32×10.667%
+ Tuned launch (ours)1,9951.94×10.698%
No sketch construction (lower bound)1,9182.02×10.499%

Together, the three optimizations make the flush step 1.94× faster, within 4% of a flush that constructs no sketch at all, at 98% of the memory bandwidth that a plain copy achieves on this GPU.

W5. Narrow end-to-end evaluation scope

Narrow end-to-end evaluation scope: throughput is reported only on Nemotron 3 Super, while other models are evaluated mainly at kernel latency; accuracy benchmarks and 16K RULER coverage are limited, and there is no end-to-end validation on longer contexts, multi-turn dialogue, prefill/scheduling interactions, or different hardware bandwidths.

Evaluation scope: The revision reports measured end-to-end decode throughput and the decode-step breakdown for all three model families (Section 5.2). With Ḡ=8, SketchSSM improves maximum decode throughput over Standard by 1.60× and 1.55× on Qwen3.8 Flash-Next and GLM 5.3 Flash, respectively, while ReplaySSM improves it by 1.17× and 1.15×.

Long-generation tasks: The reasoning benchmarks are already long-generation tasks: under Standard, the mean number of generated tokens per problem is 1.2K–5.3K on MATH-500, 18.9K–22.6K on AIME25, 10.8K–18.9K on GPQA Diamond, and 14.1K–24.5K on LiveCodeBench across the four models, with generations up to the cap of 32K (Nemotron Nano) or 65K tokens (the other models).


AuthorsOpenReview ↗

Reminder: Discussion Period Ending in a Few Days

Dear Reviewer NHEZ,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we showed that a single fixed basis holds under distribution shift and long contexts: recalibrating on general, code-only, and math-only corpora changes accuracy by at most 1.4 points, within the seed-to-seed deviation, and a basis calibrated on WikiText-2 retains 96.9–97.0% of the read output from 512 to 8,192 tokens of context. With the unified protocol, Nemotron Nano at Ḡ=2 reaches 68.24 instead of 55.21, and we now state Ḡ=8 as the recommended setting, which matches Standard at about 10× less traffic. We added the window-size and flush-step analyses: accuracy is insensitive to W ∈ {8, 16, 32}, and our flush kernel is 1.94× faster than a separate construction pass, within 4% of a flush that builds no sketch. The rank allocation is computed once offline in about 11 minutes, adds no cost at serving time, and is independent of the batch size. End-to-end throughput is now measured on all three architectures (1.60× and 1.55× on GDN and KDA).

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

Reviewer_a6PrOpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SketchSSM speeds up decoding by reducing reads of the recurrent state. It keeps the full state for updates and uses a small sketch to approximate outputs between updates. The method adapts the sketch to the query and state, and assigns different ranks to different heads. Experiments show a good trade-off between accuracy and memory traffic.

Critical strengths
  1. Keeping full-state updates while approximating reads is a well-motivated design. It avoids feeding state compression errors directly into later updates.
  2. The calibrated basis, state-dependent coefficients, and head-level rank allocation are supported by useful ablation studies.
  3. Results cover Mamba-2, GDN, and KDA models, with up to 2.64× higher decode throughput on Nemotron Super.
Critical weaknesses
  1. Some task scores drop noticeably at faster settings. For example, LiveCodeBench on Nemotron Super drops from 82.86 to 76.83, which is less clear from the average score.
  2. End-to-end speed is evaluated only on Nemotron Super. The practical gains on GDN and KDA models remain unclear.
Questions for authors
  1. How sensitive are the results to the calibration data, especially on code and math tasks?
  2. How does runtime compare with the quantized-read baseline in Appendix G at similar accuracy?
  3. Can you report end-to-end speed on GDN and KDA models?
5 comments
AuthorsOpenReview ↗

Response to Reviewer a6Pr (1/2)

Thank you for your thoughtful review and constructive feedback. We address your concerns below.


Common: Revised evaluation

Revised evaluation. We first note that we re-ran the entire accuracy evaluation after submission, under a single protocol:

  • Repeated runs. To reduce run-to-run noise, we now run every SketchSSM configuration with 3 seeds (8 seeds for AIME25, which has 30 problems), and we report mean ± std.
  • One rank set for all models. The submission used a different set of mean ranks per model, which made it hard to compare up to which rank each model preserves Standard accuracy. All four models are now evaluated at Ḡ ∈ {8, 4, 2}.
  • Ridge λ. The ridge in the coefficient solve only keeps the BF16 solve numerically stable. A value near the BF16 precision (2⁻⁸ ≈ 0.004) is therefore enough, and anything larger biases the coefficients. The submitted version mistakenly used λ = 0.1, which added 10% of the mean eigenvalue of the normalized state metric to its diagonal. We lowered it to λ = 0.003 for every model and rank and re-ran the evaluation. Correcting this bias turned out to recover part of the sharp accuracy drop at low ranks (Nemotron Nano at Ḡ=2: 56.37 → 68.24 under the same protocol).

Average accuracy over the four tasks (state-access traffic reduction in parentheses):

ModelStandardReplaySSMSketchSSM Ḡ=8Ḡ=4Ḡ=2
Nemotron Nano 9B v2 (Mamba-2)74.36 ± 0.9975.1574.41 ± 0.91 (9.94×)71.93 (12.26×)68.24 (13.88×)
Nemotron 3 Super (Mamba-2)84.56 ± 0.8084.5084.67 ± 1.19 (9.39×)84.43 (11.84×)81.76
Qwen3.8 Flash-Next (GDN)90.65 ± 0.6491.0190.63 ± 0.74 (10.68×)88.59 (12.81×)83.15 (14.23×)
GLM 5.3 Flash (KDA)86.03 ± 1.3086.8588.48 ± 1.19 (10.68×)87.3780.91

At Ḡ=8, SketchSSM matches Standard on all four models within one standard deviation, at about a 10× reduction of state-access traffic. Lower ranks reduce traffic further at the cost of accuracy. However, Ḡ=8 already accelerates decoding substantially, and lowering the rank to Ḡ=4 raises throughput by only 1–7% further.


W1. Task scores drop at faster settings.

Some task scores drop noticeably at faster settings. For example, LiveCodeBench on Nemotron Super drops from 82.86 to 76.83, which is less clear from the average score.

We agree that some task scores drop noticeably at faster settings. We do not claim that every setting, including Ḡ=2 and Ḡ=4, preserves the accuracy of Standard. We now state the trade-off more explicitly: Ḡ=8 is the recommended setting, with about 10× less state-access traffic and average accuracy within one standard deviation of Standard on all four models. Lower ranks such as Ḡ=2 trade 2.8–7.5 points of average accuracy, depending on the model, for about 14× less traffic.

In practice, Ḡ=8 already removes most of the linear-attention cost, so lower ranks have little speed left to gain. At Ḡ=8, SketchSSM already accelerates linear attention by 7.3×, 5.0× and 5.2× on Nemotron 3 Super, Qwen3.8 Flash-Next and GLM 5.3 Flash. Its share of the decode step falls to 22.3%, 10.2% and 8.7%, and the models reach 78%, 90% and 91% of the throughput they would have if linear attention took no time at all (Section 5.2). From this point, further acceleration to Ḡ=4 raises the maximum end-to-end throughput over Ḡ=8 by only 6–7% on Nemotron 3 Super and 1–2% on Qwen3.8 Flash-Next and GLM 5.3 Flash, at the cost of 0.2–2.0 points of average accuracy. Therefore, even though there are some score drops in certain settings with Ḡ=2 and 4, we achieve most of the acceleration possible for linear attention with Ḡ=8 that matches the score of Standard.

AuthorsOpenReview ↗

Response to Reviewer a6Pr (2/2)

W2/Q3. End-to-end speedup on GDN and KDA models.

Can you report end-to-end speed on GDN and KDA models?

We measured end-to-end throughput on the GDN and KDA models. The revision reports it alongside the decode-step breakdown for all three models (Section 5.2, Fig. 8 and Fig. 9). With Ḡ=8, SketchSSM improves maximum decode throughput over Standard by 1.60× on Qwen3.8 Flash-Next (GDN) and 1.55× on GLM 5.3 Flash (KDA). In comparison, ReplaySSM, which targets only state-write traffic, improves decode throughput over Standard by 1.17× and 1.15×, showing that SketchSSM further accelerates linear attention and increases throughput. With SketchSSM, the linear attention share falls to 10.2% and 8.7%, down from 36.2% and 33.3% under Standard, and it is no longer the main decoding bottleneck.


Q1. Sensitivity to the calibration dataset.

How sensitive are the results to the calibration data, especially on code and math tasks?

Thanks for the valuable feedback. We show that the results are not sensitive to the calibration data, and the revision explains the details (Appendix H).

The default calibration corpus we used across all models, WikiText-2, is a general-purpose corpus unrelated to the evaluation benchmarks in math, science, and code, so every result in the paper is already out of domain.

To further explore calibration sensitivity, we recalibrated Nemotron Nano 9B v2 on three other corpora: a general one (The Pile), a code-only one (CodeSearchNet), and a math-only one (NuminaMath-CoT). We evaluate the code benchmark (LiveCodeBench) and math benchmark (MATH-500) with three seeds at Ḡ=8.

Calibration datasetLCBMATH-500
Standard (no calibration)68.99 ± 1.2097.33 ± 0.42
WikiText-2 (default)68.15 ± 1.6397.07 ± 0.58
Pile67.94 ± 0.8496.73 ± 0.50
Code68.04 ± 1.0297.13 ± 0.42
Math69.31 ± 0.4897.13 ± 0.31

It shows the insensitivity clearly. Accuracy varies by at most 1.4 points on LiveCodeBench and 0.4 points on MATH-500, within the seed-to-seed deviation of the default calibration. Also, matching the calibration domain to the task does not help, and mismatching it does not degrade.


Q2. Runtime against the quantized-read baseline of Appendix G

How does runtime compare with the quantized-read baseline in Appendix G at similar accuracy?

We thank the reviewer for this question. We first note that the baseline in Appendix G is itself an instance of our proposed decoupled design, i.e., keeping the exact full-state update and approximating only the read. The read representation can be a quantized copy, truncated SVD, or a sketch (ours). In the submitted paper, we provided only the accuracy comparison for these ablations. We additionally provide the runtime speedup of each design in Appendix G of the revised paper. Our new Figure 12 illustrates each method's accuracy and speedup.

Here is the comparison table (Nemotron Nano 9B v2, RTX PRO 6000, batch size 256, W=16, µs per layer). While a quantized copy could reduce memory access traffic, it should be dequantized and computed in a higher-precision format. So its relative speedup is far less than the sketch even if the memory-access traffic reduction is comparable. At similar accuracy (4-bit copy 71.07%, sketch at Ḡ=4 71.93%), the sketch is 1.49× faster per step.

ReadNon-FlushFlushAverage StepSpeedupAverage Accuracy
full state (Standard)1,8351,8351,8351.00×74.36
4-bit state copy1892,0483056.02×71.07
2-bit state copy1362,0082537.25×46.14
sketch, Ḡ=81191,9952367.78×74.41
sketch, Ḡ=4871,9672058.96×71.93


AuthorsOpenReview ↗

Reminder: Discussion Period Ending in a Few Days

Dear Reviewer a6Pr,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we addressed each of your points. On task scores at faster settings, we now state Ḡ=8 as the recommended setting: it matches the accuracy of Standard at about 10× less state-access traffic. For the example you raised, LiveCodeBench on Nemotron 3 Super is 80.74 ± 0.80 at Ḡ=8 against 80.21 ± 0.80 for Standard, and the drop appears only at lower ranks. On end-to-end speed, we measured the GDN and KDA models, where SketchSSM improves maximum decode throughput by 1.60× and 1.55× over Standard, against 1.17× and 1.15× for ReplaySSM. On calibration sensitivity, recalibrating on code-only and math-only corpora changes accuracy by at most 1.4 points on LiveCodeBench and 0.4 points on MATH-500, within the seed-to-seed deviation. On the quantized-read baseline of Appendix G, the sketch is 1.49× faster per step than a 4-bit quantized state copy at similar accuracy.

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their constructive feedback. We are encouraged that the reviewers find our design of keeping exact full-state updates while approximating only the reads well motivated (@a6Pr, @NHEZ, @uNHi), and our evaluation across Mamba-2, GDN, and KDA models strong (@a6Pr, @NHEZ, @uNHi).

SketchSSM is a training-free optimization of linear attention that attains most of the possible throughput gain. At a mean sketch rank of Ḡ=8, it reduces state-access traffic by about 10× while matching the accuracy of Standard. Linear attention is then no longer the main decode bottleneck: the models reach 78%, 90%, and 91% of the throughput they would have if linear attention took no time at all.

ModelAvg. accuracy (Standard / Ḡ=8)Linear-attention share of decode (Standard → Ḡ=8)Max. decode throughput
Nemotron 3 Super (Mamba-2)84.56 / 84.6768.5% → 22.3%2.77×
Qwen3.8 Flash-Next (GDN)90.65 / 90.6336.2% → 10.2%1.60×
GLM 5.3 Flash (KDA)86.03 / 88.4833.3% → 8.7%1.55×

SketchSSM is stable across models, datasets, tasks, and context lengths. It matches the accuracy of Standard on all four evaluated models: Nemotron Nano 9B v2 and Nemotron 3 Super (Mamba-2), Qwen3.8 Flash-Next (GDN), and GLM 5.3 Flash (KDA). As several reviewers asked (@a6Pr, @NHEZ), we recalibrated it on another general corpus, a code-only corpus, and a math-only corpus and obtained the same accuracy, showing that SketchSSM is insensitive to the calibration dataset. SketchSSM is also stable over long generations, which the reviewers raised as a concern (@NHEZ, @uNHi). Our four reasoning benchmarks are already decode-heavy, with up to 24.5K generated tokens per problem on average and up to 65K at most. A new analysis in the revision (Section 5.1, Fig. 6) groups their problems by generation length: SketchSSM matches Standard at every length, whereas state quantization and pruning fall further behind as the generation gets longer (8-bit quantization trails Standard by 21.7 points at 16–32K generated tokens).

Why SketchSSM works: the low rank of the recurrent state, used differently. Thanks to the question of @uNHi, we analyzed during the rebuttal where the accuracy of SketchSSM comes from, and found that it rests on a well-known property: the recurrent states of linear attention are approximately low-rank. SketchSSM applies matrix sketching to the state read, which is accurate whenever the state is approximately low-rank, and this property is what lets SketchSSM perform well and stably across models and tasks. Our novelty comes from how we use this low-rank property. Prior work uses it to compress the state itself, so the approximation enters the stored state and accumulates through every later update. We found that avoiding this accumulation requires decoupling the state write from the state read: SketchSSM keeps the write exact and compresses only the representation used for reading, which turns the low rank into system efficiency. We hope this makes SketchSSM useful for the linear-attention layers of future frontier models.

New in the revision. The main revisions are highlighted in blue in the updated PDF: Section 5.1 (error accumulation over long generations), Section 5.2 (end-to-end throughput on all three architectures), and Appendices E–J, including the new Appendices H (analysis of the state-weighted query basis and calibration sensitivity), I (window size), and J (flush step). We first re-ran the entire accuracy evaluation under one protocol, with repeated seeds and the same ranks Ḡ ∈ {8, 4, 2} for all four models. Following the reviewers' suggestions, we also added:

  • End-to-end speed on GDN and KDA (@a6Pr, @NHEZ): 1.60× and 1.55× over Standard, against 1.17× and 1.15× for ReplaySSM.
  • Calibration sensitivity (@a6Pr, @NHEZ) and the source of the low rank (@uNHi), summarized above.
  • Error accumulation over long generations (@NHEZ, @uNHi), compared with state quantization and pruning.
  • Window size and flush step (@NHEZ) and runtime against quantized reads (@a6Pr).

We address each reviewer's questions in the individual responses below.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.