acceptodds
Under review as a conference paper at ICLR 2027

ReplaySSM: Cache SSM Inputs, Not State

11 views

Abstract

State-space models (SSMs) maintain a fixed-size recurrent state, but reading and writing it at every decoding step can bottleneck hybrid models. This eager state management causes large state I/O with low arithmetic intensity, leaves speculative decoding no cheap rollback, and feeds state quantization error into subsequent recurrent updates. We introduce ReplaySSM, which unrolls the recurrence to represent the state as a checkpoint and a buffer of recent SSM inputs. This separates output computation from checkpoint updates. By updating the checkpoint periodically, ReplaySSM avoids state writes on most steps and roughly halves the dominant state memory traffic. Keeping recent inputs also makes speculative rollback a buffer-pointer operation and enables output-only verification without materializing intermediate states. The same separation allows a low-precision checkpoint to serve output computation on most steps while a higher-precision checkpoint serves recurrent updates, so rounding error in the low-precision copy never enters later updates. We implement ReplaySSM in vLLM for Mamba-2 and Gated DeltaNet. At serving batch size 256, it accelerates standard decoding by – over a tuned vLLM baseline across four hybrid models, and delivers about the throughput of vLLM's speculative decoding on the two 120B-class models. Separate state precisions also reduce output KL divergence and downstream accuracy losses compared with a shared low-precision checkpoint.

Primary area:
foundation or frontier models, including LLMs
Keywords:
State Space Models, Hybrid Models, LLM Inference, Linear Attention, Speculative Decoding, Quantization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_MhgHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReplaySSM, which decomposes the recurrent state of SSMs into a checkpoint and a buffer of recent inputs. By decoupling output computation from checkpoint updates, ReplaySSM avoids state writebacks at most decoding steps. This design also simplifies state rollback in speculative decoding and mitigates the accumulation of quantization errors. Experiments on several large models demonstrate 1.2–1.5× decoding speedups, substantial throughput improvements in speculative decoding, and better accuracy preservation when checkpoints are quantized to lower precision.

Critical strengths
  1. The paper identifies a practical and important bottleneck: the memory bandwidth overhead of reading and writing the full SSM state at every decoding step.
  2. The proposed design is clean and intuitive. Decoupling output computation from state updates is a straightforward yet effective approach, supported by a solid implementation.
Critical weaknesses
  1. Limited novelty. The idea of buffering recent inputs to avoid frequent state updates and simplify rollback has been explored in prior work on SSM speculative decoding, including STree, The Mamba in the Llama, and Snakes and Ladders. The paper does not sufficiently clarify its fundamental differences from these methods, nor does it provide direct comparisons.

  2. Incomplete experimental evaluation. The low-precision state evaluation mainly consists of ablations of ReplaySSM's own configurations, without direct comparisons against specialized SSM state quantization methods such as Q-Mamba (Q-Mamba: Towards more efficient Mamba models via post-training quantization). This makes it difficult to assess its advantages over existing approaches. Additionally, the window size L is fixed at 16 throughout the experiments, without investigating whether the optimal window size changes with batch size.

Questions for authors
  1. Prior work such as STree has also explored input buffering to avoid state rollback. What are the fundamental differences between ReplaySSM and these approaches?
  2. Could the authors provide direct experimental comparisons with state-of-the-art SSM quantization methods, such as Q-Mamba?
1 comment
AuthorsOpenReview ↗

Response to Reviewer MhgH

We are glad the reviewer finds decoupling output computation from state updates a clean and effective design.

W1 and Q1. Differences from STree, The Mamba in the Llama, and Snakes and Ladders.

We thank the reviewer for raising this comparison. The submission includes both an algorithmic comparison (Appendix A) and a direct experimental comparison (Appendix B). With only a pointer in Section 3.2, they are easy to miss, and we will move the key differences into the main text. The main differences are:

(1) Output computation is separated from state updates. This is the main contribution of ReplaySSM. Prior work also caches SSM inputs, but only to recover the accepted state after verification, so it still updates and writes the state at every step. ReplaySSM computes the outputs at every step and updates the checkpoint once per window of LL committed tokens. The separation does not depend on speculation, so it also speeds up standard decoding and allows separate state precisions, which the prior methods do not address.

(2) State traffic per step. Every prior method moves the full state at least three times per verification step, for example two reads and one write for STree. A non-flush ReplaySSM step reads the checkpoint once and writes nothing. Nsight measurements confirm these counts (Table 3).

(3) Serving batch sizes. The prior papers report the speedup of speculative over standard decoding at small batch sizes, where the gain comes mostly from amortizing the weight loads across the TT verified tokens. Serving batch sizes are harder. The weights are already amortized across the batch, the GEMMs approach compute bound as B⋅TB \cdot T grows, and state traffic is the bottleneck, so the cost of handling the state decides whether speculation pays off. At B=256B = 256, ReplaySSM verifies 2.3 to 5.3x faster than STree and The Mamba in the Llama (Figure 7). Every baseline is tuned to its best launch configuration, and STree is measured as its algorithm, without the extra state copies of its released path.

(4) Delta-rule models. The prior work evaluates Mamba layers, while recent hybrid models are moving to the delta-rule family. ReplaySSM covers both. For the delta rule, output-only verification is not straightforward, because the draft corrections depend on one another, and ReplaySSM obtains them from a single T×TT \times T triangular solve.

W2 and Q2. Comparison with Q-Mamba.

Q-Mamba and ReplaySSM reduce different parts of the quantization error, so they combine.

Let DtD_t be the deviation of the stored quantized state from the exact state, and ete_t the rounding error added at step tt. When the quantized state is fed back at every step, Dt=αtDt−1+etD_t = \alpha_t D_{t-1} + e_t (Appendix D.5): each rounding adds ete_t, and the recurrence carries the earlier errors forward.

(1) Q-Mamba reduces ete_t. Decoupled Scale Quantization (DSQ) gives the state separate scales along its two dimensions, and Efficient Selectivity Reconstruction fine-tunes about 2% of the parameters. The quantized state still feeds the next update, so the error still propagates. Q-Mamba's training simulation neglects the propagated term αtDt−1\alpha_t D_{t-1}, and its evaluation covers at most 8K tokens. We measure out to 128K tokens, where the accumulated error shows up (Figure 5).

(2) ReplaySSM removes the propagation. The low-precision copy is read only for outputs and never enters the recurrence, which removes most of the error growth out to 128K tokens (Figure 5). It breaks the propagation path instead of reducing ete_t, so it works with any method that reduces ete_t.

To confirm this, we added DSQ as a state format and measured the output KL on Qwen3.5-4B under the protocol of Figure 5. We use INT4 states, the setting of Q-Mamba's main results, and DSQ without the fine-tuning step, which modifies the model weights and is built for pure Mamba-2 models.

State configurationNN = 1K8K32K128K
DSQ-INT4, fed back every step0.517.9divergeddiverged
DSQ-INT4 read, FP16+SR recurrence (ReplaySSM)0.0180.0210.0210.025

Fed back at every step, DSQ-INT4 accumulates error until the state diverges past 8K tokens: its values grow without bound and leave the FP16 range. ReplaySSM, which keeps a higher-precision checkpoint for the recurrence, breaks the error propagation path and keeps the KL roughly flat out to 128K tokens.

W2. Window size.

The window size is a tradeoff: a shorter window flushes more often and pays for more checkpoint writes, while a longer window loads more buffered inputs at every step. Appendix F.2.2 sweeps L∈{8,16,32}L \in \{8, 16, 32\}, and L=16L = 16 is best on all four models. The optimal window is set by this tradeoff and by the acceptance length, not by BB, since state traffic and buffer traffic both scale with BB.

Reviewer_pUp3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors propose ReplaySSM to improve the inference efficiency of SSMs. By storing the recurrent state as a checkpoint alongside an LL-step input buffer, state write-backs are performed only when the buffer is full. This significantly reduces memory bandwidth by half, streamlines speculative decoding rollbacks into simple pointer updates, and facilitates dual-precision state management. Implementations on Mamba-2 and GDN using vLLM show up to a 1.47× end-to-end decoding speedup, ~1.8× higher speculative decoding throughput on 120B models, and better performance retention than shared NVFP4 quantization.

Critical strengths
  1. The paper introduces a mechanism that yields three distinct advantages. The proposed dual-precision checkpoint is novel, and the handling of SSMs is clearly detailed and well-executed.

  2. The empirical validation is rigorous. The experimental setup features well-tuned baselines, Nsight-verified memory traffic analysis for previous kernels, paired accuracy assessments, and a comprehensive length-stratified analysis.

Critical weaknesses
  1. The core mechanism proposed in Section 4 (checkpointing combined with input replay and periodic flushing) closely mirrors the "periodic cache checkpointing" of Nemotron 3 Ultra. Although cited as concurrent work, this overlap overstates the theoretical novelty of the first contribution, reducing the paper's actual novelties primarily to empirical speedup measurements and the precision split. Furthermore, the demonstrated gains are highly regime-specific: they depend on single-GPU configurations without Tensor Parallelism (TP) and require large batch sizes (B≥256), as Table 6 indicates only a marginal ~1.1× speedup at B=32. The claimed 1.8× advantage over vLLM speculative decoding also appears to stem largely from vLLM's snapshot mechanism hitting memory capacity limits.
Questions for authors
  1. Could the authors provide end-to-end speedups under realistic parallelism (e.g., TP=2–8, per-GPU batch 32–128) to account for communication overhead and reduced per-GPU state volume? Does TP=kk at batch BB behave exactly like single-GPU B/kB/k?

  2. How does Section 4 differ algorithmically from Nemotron 3 Ultra’s periodic cache checkpointing, excluding the precision split?

3 comments
AuthorsOpenReview ↗

Response to Reviewer pUp3 (Part 1/2)

We thank the reviewer for the careful reading and for noting the rigor of the empirical validation. Our response is in two parts: Part 1 answers Q1 on tensor parallelism, and Part 2 answers Q2 together with W1.

Q1. Speedups under tensor parallelism.

Yes for the SSM layers: TP =k= k at batch BB behaves like a single GPU at batch B/kB/k. TP splits the SSM heads across GPUs while every GPU still sees all BB sequences, so each GPU runs the kernel on BB sequences and H/kH/k heads. The kernel is independent per (sequence, head) pair, so this is the same work and the same state traffic as B/kB/k sequences with all HH heads. TP communicates once per block, through an all-reduce after the output projection, which sits outside the SSM kernel and is the same with or without ReplaySSM.

To confirm this, we measured the kernel speedup over the tuned conventional kernel for TP = 1 to 8 and per-GPU batch B/TPB/\mathrm{TP} from 32 to 512:

Nemotron-3-Super-120B-A12B (Mamba-2)

B/TPTP=1TP=2TP=4TP=8
321.18x1.19x1.21x1.20x
1281.46x1.45x1.45x1.44x
2561.52x1.51x1.51x1.46x
5121.55x1.55x1.47x1.54x

Qwen3.5-122B-A10B (GDN)

B/TPTP=1TP=2TP=4TP=8
321.37x1.36x1.38x1.40x
1281.44x1.45x1.47x1.47x
2561.46x1.47x1.48x1.46x
5121.48x1.48x1.47x1.48x

At a fixed B/TPB/\mathrm{TP}, the columns agree within a few percent, so the speedup depends on B/TPB/\mathrm{TP} and not on TP.

End to end, TP changes the rest of the decode step and not the SSM kernel: it adds an all-reduce after each block and changes the shapes of the projections. The e2e gain at a given B/TPB/\mathrm{TP} then follows the SSM share of the step, as Amdahl's law predicts. The models we tested also fit on a single GPU, where data parallelism avoids the all-reduce and each GPU serves its own batch.

AuthorsOpenReview ↗

Response to Reviewer pUp3 (Part 2/2)

Q2. Algorithmic difference from Nemotron 3 Ultra.

Nemotron 3 Ultra presents periodic cache checkpointing as a way to reduce Mamba cache quantization error, and notes a possible speedup for Mamba-2 standard decoding with kernels still under development (Section 4.5). ReplaySSM is concurrent work, first made public at a similar time. It covers standard and speculative decoding and the delta-rule family, besides the precision split. The differences:

(1) What fundamentally changes. Nemotron 3 Ultra describes storing the state less often. The main contribution of ReplaySSM is the separation of output computation from state update, a concept Nemotron 3 Ultra does not mention. Caching SSM inputs is the mechanism, and the separation is the idea behind it. All three benefits come from it: lower state traffic, cheap rollback for speculative decoding, and state quantization. The precision split is a direct consequence of this separation, not an independent finding. The outputs and the recurrence read different copies of the state, so each copy can use its own precision. Periodic checkpointing alone does not lead there, since its single quantized checkpoint still feeds its rounding error into the recurrence.

(2) Speculative decoding and serving. Nemotron 3 Ultra's checkpointing covers standard decoding, where memory traffic is the issue. Speculative decoding raises three more, and ReplaySSM addresses each:

  • Computation. Verifying TT draft tokens multiplies the computation under the same memory load. ReplaySSM verifies all TT tokens in one output-only evaluation from the checkpoint and buffer (Section 5.2).
  • Memory capacity. Per-token state snapshots cost capacity as well as traffic. ReplaySSM keeps one checkpoint per sequence, and rollback is a buffer-pointer move.
  • Serving. Each request accepts a different number of tokens per step, which makes CUDA Graph support difficult. ReplaySSM manages the per-request buffer and flush decisions inside the graph, so the kernel gain carries through to serving throughput.

All of this is implemented in vLLM and measured end to end on four models.

(3) Delta-rule family. Nemotron 3 Ultra covers Mamba-2 only, while recent hybrid models are moving to the delta-rule family (Qwen3.8, Kimi-K3, GLM-5.3-Flash). Extending input replay to this family is not straightforward. In Mamba-2, each step adds a term that depends only on the current input, so the state is a decayed sum of the cached inputs and replaying them rebuilds it. The delta rule overwrites instead: it first removes what the state already stores for the current key, then writes the new value (Eq. 2). Its update depends on the current state as well as the current input, so the cached inputs cannot simply be summed onto a checkpoint. ReplaySSM caches the correction ut=βt(vt−αtSt−1kt)u_t = \beta_t(v_t - \alpha_t S_{t-1} k_t), the new value minus what the state already stores, in place of vtv_t. This restores the decayed-sum form (Eq. 3). Under speculative decoding, the TT draft corrections depend on one another, and ReplaySSM obtains them from a single T×TT \times T triangular solve.

W1. The 1.8x speculative-decoding gain.

The 1.8x comes from two sources: a faster verification kernel, and a higher maximum concurrency because ReplaySSM stores no per-token state snapshots. The kernel gain holds even where memory capacity is not a limit.

(1) The kernel gap at a fixed batch. At B=256B = 256, conventional verification of eight tokens costs 4.66x a standard-decoding step, while ReplaySSM costs 0.89 to 1.21x (Figure 4). This is a kernel latency, so memory capacity does not enter. In serving, ReplaySSM already delivers 1.29x and 1.43x the throughput of conventional speculation at B=128B = 128, T=4T = 4 (Tables 8 and 9).

(2) Why the gap appears at serving batch sizes. At small BB, speculative decoding gains mainly from amortizing the weight loads across the verified tokens, so the SSM kernel is a small share of the step and the two methods are close in speed. ReplaySSM still saves the memory of the TT snapshots there, which matters on edge devices where memory is scarce. At serving batch sizes, the weights are already amortized across the batch, and the per-token state writes are the cost that remains. Conventional speculation stops gaining beyond B=128B = 128 and falls below standard decoding on Qwen3.5-122B at B=512B = 512, while ReplaySSM keeps scaling on both models (Figure 3). A larger maximum concurrency alone does not explain the gap: at these batch sizes the GEMMs approach compute bound, so a larger batch does not translate one-to-one into throughput.

Thank you for the detailed response. The authors have addressed my concerns and I have no further questions. I will maintain my positive assessment and look forward to seeing these clarifications incorporated into the final manuscript.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer eKkR (Part 1/2)

We thank the reviewer for the helpful feedback and for noting that ReplaySSM is highly practical and well-motivated.

W1. Memory-bound regime and small batch sizes.

The memory-bound regime is not a hardware context chosen to favor us. It is the nature of LLM decoding. A conventional SSM decoding step has an arithmetic intensity of 0.63 FLOP per byte (Appendix D.1), while an H100 needs about 300 to keep its BF16 tensor cores busy, and batching does not change this because every sequence brings its own state. The baselines are tuned too: each is vLLM at its best swept launch configuration, not the shipped default (Appendix F.2.1).

We see ReplaySSM as more than a deployment-specific gain. ReplaySSM fundamentally changes how the state is represented and updated. An SSM layer has two jobs per step: compute the output and update the state. Conventional decoding follows the recurrence directly, which fuses the two at every step. ReplaySSM separates them, and this one change addresses all three challenges of SSM decoding (Figure 1): state traffic, rollback under speculative decoding, and state quantization. Nothing in the change depends on batch size. What depends on batch size is whether the SSM kernel is the bottleneck of the decode step.

A hybrid model has three major components: GEMM (MLP and projections), attention, and SSM. Their per-step memory traffic scales as follows:

ComponentPer-step memory traffic
GEMM (MLP and projections)constant (weight parameters)
AttentionO(B⋅N)O(B \cdot N)
SSM (conventional)O(B⋅T)O(B \cdot T)

where BB is the batch size, NN is the sequence length, and TT is the number of tokens verified in parallel.

At small batch sizes (e.g., B=1B = 1), the step latency is dominated by loading the weights, not by the SSM or attention kernels, so the SSM kernel barely affects e2e performance (ReplaySSM matches the tuned baseline within 1 to 4%). ReplaySSM still helps here through memory capacity. Small batches are typical of edge devices, where the weights and the KV cache take most of the memory. Conventional speculative decoding stores TT extra state snapshots per sequence for rollback, which can limit the draft length. ReplaySSM keeps only one checkpoint and a small input buffer.

As the batch grows, the weight load is amortized across sequences, while SSM and attention traffic grow with BB. Of the two, the SSM state is the larger up to about 80K tokens of context (see W2), so the SSM layers become the bottleneck. This is the regime production serving runs in, and where ReplaySSM's trade of more computation for less memory traffic pays off.

At any batch size, ReplaySSM also breaks error propagation, which allows a lower state precision.

W2 and Q2. Long context and the crossover between attention and SSM.

We agree that attention traffic grows with sequence length while the SSM state does not, so attention dominates at very long context lengths.

(1) With full attention, the crossover is far out. For Nemotron-3-Super-120B-A12B-NVFP4 (Figure 1, bottom left), the per-step traffic is:

  • KV cache: NN (context length) ×\times 8 (attention layers) ×\times 2 (K and V) ×\times 2 (KV heads) ×\times 128 (head dim) ×\times 1 byte (FP8) =4N= 4N KiB per sequence.
  • SSM state: 40 (Mamba-2 layers) ×\times 128 (heads) ×\times 64 (dvd_v) ×\times 128 (dstated_{state}) ×\times 4 bytes (FP32) ×\times 2 (load and store) =320= 320 MiB per sequence.

The two are equal at N=81,920N = 81{,}920 tokens, at any batch size, since both scale with the number of sequences. In latency, the SSM layers remain the bottleneck up to about 100K tokens (Figure 1). Speculative decoding pushes the crossover further out: the KV cache is still loaded once per step, but conventional SSM decoding moves the state T+1T+1 times instead of twice, which gives about 370K tokens at T=8T = 8. Past the crossover, the speedup remains, with a margin that shrinks with the SSM share of the step, as Amdahl's law predicts. At N=160N = 160K, the SSM layers still move a third of the bytes under standard decoding, and more under speculative decoding.

(2) Recent hybrid models are moving to sparse attention for this reason. Kimi K3.1 Pro [1], GLM-5.3-Flash [2], and Qwen3.8-Flash-Next [3] replace full attention with sparse attention. This pushes the crossover much further out and makes ReplaySSM more important.

We find the comparison between SSM and attention valuable and will add the analysis above to the appendix.

AuthorsOpenReview ↗

Response to Reviewer eKkR (Part 2/2)

Q1. Hardware generality.

ReplaySSM pays off as long as the conventional SSM kernel is memory-bound, and it is by a wide margin. The conventional kernel performs 0.63 FLOP per byte of state traffic, while an H100 needs about 300 to keep its BF16 tensor cores busy [4]. Memory bandwidth would have to grow by two orders of magnitude relative to compute before the kernel stopped being memory-bound. Recent hardware has not moved that way: from H100 to B300, HBM bandwidth grew about 2.3x, while tensor-core throughput grew 2.3x (BF16, FP8) to 7x (NVFP4) [4, 5]. Over the longer history of accelerators, peak FLOPS has grown faster than DRAM bandwidth [6].

The same holds across accelerator classes, not only on data center GPUs. A DGX Spark (GB10) desktop has about 1.6x the compute per byte of memory bandwidth of B300 [7]. We expect this for any LLM accelerator: one balanced near 0.63 FLOP per byte would be far too slow for the GEMMs of the same model, whose arithmetic intensity grows with batch size while the SSM kernel's does not.

Our experiments cover H100 and B300. B300 has 2.3x the HBM bandwidth of H100, and ReplaySSM speeds up standard decoding on both: 1.27 to 1.47x e2e on H100 and 1.20 to 1.42x on B300 at B=256B = 256 (Table 6).

The window is less sensitive to the ratio than it may seem. It mainly trades flush frequency against the buffered inputs loaded per step, and both are memory traffic, so to first order the optimal LL does not move with the ratio. The replay computation starts to matter at larger windows under speculative decoding, and there, more bandwidth per FLOP would favor a smaller LL. LL is a runtime parameter, so retuning it per chip is a sweep, not a redesign. We appreciate the reviewer raising this point and will add this discussion to the appendix.

Two contributions do not depend on this ratio at all: breaking error propagation allows a lower state precision, and dropping the TT state snapshots raises the maximum concurrency, which is set by memory capacity, not by the ratio.

References

[1] Kimi K3.1 Pro model card. https://huggingface.co/moonshotai/Kimi-K3.1-Pro

[2] GLM-5.3-Flash: Frontier Intelligence, Flash Cost. https://z.ai/blog/glm-5.3-flash

[3] Qwen3.8-Flash-Next model card. https://huggingface.co/Qwen/Qwen3.8-Flash-Next

[4] NVIDIA H100 Tensor Core GPU, product specifications. https://www.nvidia.com/en-us/data-center/h100/

[5] NVIDIA Blackwell Ultra datasheet, HGX B300 technical specifications. https://resources.nvidia.com/en-us-blackwell-architecture/blackwell-ultra-datasheet

[6] A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer. AI and memory wall. IEEE Micro, 44(3):33-39, 2024.

[7] NVIDIA DGX Spark, product specifications. https://www.nvidia.com/en-us/products/workstations/dgx-spark/

Thank you for the detailed response and the additional experiments. The new results and clarifications adequately address my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.