- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReplaySSM, which decomposes the recurrent state of SSMs into a checkpoint and a buffer of recent inputs. By decoupling output computation from checkpoint updates, ReplaySSM avoids state writebacks at most decoding steps. This design also simplifies state rollback in speculative decoding and mitigates the accumulation of quantization errors. Experiments on several large models demonstrate 1.2–1.5× decoding speedups, substantial throughput improvements in speculative decoding, and better accuracy preservation when checkpoints are quantized to lower precision.
Critical strengths
- The paper identifies a practical and important bottleneck: the memory bandwidth overhead of reading and writing the full SSM state at every decoding step.
- The proposed design is clean and intuitive. Decoupling output computation from state updates is a straightforward yet effective approach, supported by a solid implementation.
Critical weaknesses
-
Limited novelty. The idea of buffering recent inputs to avoid frequent state updates and simplify rollback has been explored in prior work on SSM speculative decoding, including STree, The Mamba in the Llama, and Snakes and Ladders. The paper does not sufficiently clarify its fundamental differences from these methods, nor does it provide direct comparisons.
-
Incomplete experimental evaluation. The low-precision state evaluation mainly consists of ablations of ReplaySSM's own configurations, without direct comparisons against specialized SSM state quantization methods such as Q-Mamba (Q-Mamba: Towards more efficient Mamba models via post-training quantization). This makes it difficult to assess its advantages over existing approaches. Additionally, the window size L is fixed at 16 throughout the experiments, without investigating whether the optimal window size changes with batch size.
Questions for authors
- Prior work such as STree has also explored input buffering to avoid state rollback. What are the fundamental differences between ReplaySSM and these approaches?
- Could the authors provide direct experimental comparisons with state-of-the-art SSM quantization methods, such as Q-Mamba?
Response to Reviewer MhgH
We are glad the reviewer finds decoupling output computation from state updates a clean and effective design.
W1 and Q1. Differences from STree, The Mamba in the Llama, and Snakes and Ladders.
We thank the reviewer for raising this comparison. The submission includes both an algorithmic comparison (Appendix A) and a direct experimental comparison (Appendix B). With only a pointer in Section 3.2, they are easy to miss, and we will move the key differences into the main text. The main differences are:
(1) Output computation is separated from state updates. This is the main contribution of ReplaySSM. Prior work also caches SSM inputs, but only to recover the accepted state after verification, so it still updates and writes the state at every step. ReplaySSM computes the outputs at every step and updates the checkpoint once per window of L committed tokens. The separation does not depend on speculation, so it also speeds up standard decoding and allows separate state precisions, which the prior methods do not address.
(2) State traffic per step. Every prior method moves the full state at least three times per verification step, for example two reads and one write for STree. A non-flush ReplaySSM step reads the checkpoint once and writes nothing. Nsight measurements confirm these counts (Table 3).
(3) Serving batch sizes. The prior papers report the speedup of speculative over standard decoding at small batch sizes, where the gain comes mostly from amortizing the weight loads across the T verified tokens. Serving batch sizes are harder. The weights are already amortized across the batch, the GEMMs approach compute bound as B⋅T grows, and state traffic is the bottleneck, so the cost of handling the state decides whether speculation pays off. At B=256, ReplaySSM verifies 2.3 to 5.3x faster than STree and The Mamba in the Llama (Figure 7). Every baseline is tuned to its best launch configuration, and STree is measured as its algorithm, without the extra state copies of its released path.
(4) Delta-rule models. The prior work evaluates Mamba layers, while recent hybrid models are moving to the delta-rule family. ReplaySSM covers both. For the delta rule, output-only verification is not straightforward, because the draft corrections depend on one another, and ReplaySSM obtains them from a single T×T triangular solve.
W2 and Q2. Comparison with Q-Mamba.
Q-Mamba and ReplaySSM reduce different parts of the quantization error, so they combine.
Let Dt be the deviation of the stored quantized state from the exact state, and et the rounding error added at step t. When the quantized state is fed back at every step, Dt=αtDt−1+et (Appendix D.5): each rounding adds et, and the recurrence carries the earlier errors forward.
(1) Q-Mamba reduces et. Decoupled Scale Quantization (DSQ) gives the state separate scales along its two dimensions, and Efficient Selectivity Reconstruction fine-tunes about 2% of the parameters. The quantized state still feeds the next update, so the error still propagates. Q-Mamba's training simulation neglects the propagated term αtDt−1, and its evaluation covers at most 8K tokens. We measure out to 128K tokens, where the accumulated error shows up (Figure 5).
(2) ReplaySSM removes the propagation. The low-precision copy is read only for outputs and never enters the recurrence, which removes most of the error growth out to 128K tokens (Figure 5). It breaks the propagation path instead of reducing et, so it works with any method that reduces et.
To confirm this, we added DSQ as a state format and measured the output KL on Qwen3.5-4B under the protocol of Figure 5. We use INT4 states, the setting of Q-Mamba's main results, and DSQ without the fine-tuning step, which modifies the model weights and is built for pure Mamba-2 models.
Fed back at every step, DSQ-INT4 accumulates error until the state diverges past 8K tokens: its values grow without bound and leave the FP16 range. ReplaySSM, which keeps a higher-precision checkpoint for the recurrence, breaks the error propagation path and keeps the KL roughly flat out to 128K tokens.
W2. Window size.
The window size is a tradeoff: a shorter window flushes more often and pays for more checkpoint writes, while a longer window loads more buffered inputs at every step. Appendix F.2.2 sweeps L∈{8,16,32}, and L=16 is best on all four models. The optimal window is set by this tradeoff and by the acceptance length, not by B, since state traffic and buffer traffic both scale with B.