- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SSMs carry a fixed-size internal state vector along the sequence, which could be thought of as the short-term memory the model holds. The paper proposes POISE, a post-training method that shrinks this state without labels or retraining, using a few hundred unlabeled calibration sequences. For each SSM layer, POISE changes the layer's internal coordinates so that each direction is ranked by how much it matters on the data the layer actually sees, then deletes the lowest-ranked directions. It then decides how many directions each layer keeps under one global budget. For the linear SSMs in S4D, SaShiMi and S5, the authors prove that the shrunk layer stays stable and give an exact formula and a simple upper bound for its error. Mamba-2, whose dynamics change with the input, is handled empirically without these theoretical guarantees. Experiments on multiple architectures and tasks compare against five prior state-pruning rules and report improvements.
Critical strengths
-
Well-motivated approach with strong theoretical guarantees for LTI systems (Thm 1, Table 10). POISE uses calibration data to estimate the input-induced state covariance (P) and combines it with the observability Gramian (Q) to identify important state directions. Theorem 1 proves that the reduced LTI system remains stable and provides an upper bound on its output error. Importantly, the paper proves that using the input-induced covariance minimizes this bound. Table 10 supports the practical value of this approach, showing improvements over classical balanced truncation across the evaluated LTI systems.
-
Thorough ablations (Appendix B). The paper separately evaluates key components of POISE, including which state directions to remove and how to allocate the state budget across systems. It also analyzes computational costs, calibration-set size, and sensitivity to calibration-distribution mismatch, helping identify which components contribute to the improvements.
-
Strong memory–accuracy tradeoff on Mamba-2 (Fig. 1). On Mamba-2 1.3B, POISE retains only 30% of the recurrent states while increasing perplexity slightly (13.17 to 13.51). At batch size 64, total inference memory decreases from approximately 5.9 GB to 3.67 GB. The compressed model also achieves better perplexity than the smaller dense Mamba-2 780M (14.97) while using slightly less memory (3.67 vs. 3.98 GB).
Critical weaknesses
-
POISE's stability and error guarantees apply to individual stable LTI systems, not selective Mamba-2 or the end-to-end compressed network. Although Proposition 2 characterizes network-level error propagation, it does not guarantee the accuracy of the practical allocation surrogate. On Mamba-2 1.3B (budget 0.10), spectral allocation without sensitivity probing performs worse than uniform allocation (20.99 vs. 18.60 perplexity), while adding the heuristic probe improves perplexity to 15.12. Thus, the strongest allocation gains rely on an empirically effective heuristic without a corresponding theoretical guarantee. (Fixable: clarify the scope of the guarantees and analyze the probe's robustness.)
-
Mamba-2 is calibrated on WikiText-2 training sequences and evaluated using WikiText perplexity with 2,048-token windows. The paper does not report evaluation on other text corpora, downstream language tasks, or longer contexts. This is particularly important because POISE explicitly uses calibration data to decide which states matter for compression. States that seem unimportant for WikiText could be important for other types of text or OOD tasks. (Fixable with additional experiments.)
-
Despite motivating POISE through inference efficiency, the paper reports memory savings without measuring end-to-end Mamba-2 decoding throughput or latency. Table 5 shows that a more memory-efficient recurrence layout can be substantially slower on S4D (3.60 vs. 1.49 ms/step), demonstrating that reduced state memory does not necessarily translate into faster inference. Without compressed-versus-original Mamba-2 speed measurements, the practical runtime benefits remain unclear. (Fixable with additional measurements.)
Questions for authors
-
Given that spectral allocation without the sensitivity probe performs worse than uniform allocation on Mamba-2, how robust is the proposed allocation to the choice of probe level (15%)? Could the authors provide a sensitivity analysis across different probe levels on Mamba-2, and clarify what Proposition 2 implies about the accuracy of the practical allocation surrogate?
-
How well does POISE perform when Mamba-2 is evaluated on other text distributions, downstream tasks, or longer contexts? In particular, how sensitive is the compressed model to a mismatch between the calibration and evaluation distributions?
-
Can the authors report end-to-end decoding throughput or latency for compressed versus original Mamba-2 at different batch sizes? Given the memory–runtime tradeoff observed, it would be valuable to establish whether the reported memory savings also translate into practical speedups.
3 comments
Part 2/2
Our general response reports the same experiment on Mamba-2 across three corpora, with consistent findings: calibration on the evaluation distribution is preferable, but mismatched calibration incurs only a small penalty, and POISE still outperforms GHOST calibrated on the evaluation distribution.
W3/Q3 - "Reduced state memory does not necessarily translate into faster inference". Our general response reports decoding throughput, latency, and peak memory for the reduced Mamba-2 1.3B models at batch sizes from 1 to 1,024. The reduced models are faster at all tested batch sizes of 64 and above. At budget 0.1, POISE achieves speed-ups of 2.0x, 3.0x, and 3.3x over the uncompressed model at batch sizes 64, 256, and 1,024, respectively. At batch size 1, where kernel-launch overhead dominates, the two additional projection maps in each reduced layer make decoding 14-17% slower.
Table 5 addresses a different issue: it compares execution layouts of the same reduced S4D/SaShiMi model at budget 0.15. These models contain many single-channel systems, which are batched into groups to limit kernel launches. The table illustrates how group size affects the trade-off between memory and speed. Mamba-2 updates the states within each layer using a single state-update kernel, so this channel-grouping trade-off does not arise. We will clarify this distinction alongside Table 5.
Reviewer response to rebuttal
I thank the authors for the clarifications and the additional experiments. The new results resolve the main concerns that I raised in my review:
- The probe-level sweep on Mamba-2 shows that the allocation is robust to the choice of probe level.
- The 0-shot, longer-context and calibration-mismatch evaluations show that the method generalizes beyond the calibration distribution.
- The throughput measurements show real speed-ups at practical batch sizes.
These results would be a useful additions to the revision, including the small slowdown at bs 1. I'm maintaining my positive score.
Part 1/2
We thank the reviewer for the positive assessment and for identifying areas where we can improve the paper.
W1 - Stability and performance guarantees. Theorem 1 indeed provides per-system guarantees for stable LTI systems. Our general response, available to all reviewers, explains what carries over to Mamba-2: the layerwise error identity and stability of the reduced recurrence hold exactly, whereas optimality of the balanced projection relies on the moment-factorization assumption.
At the network level, Proposition 2 characterizes the loss caused by reducing one system alone as ∣δj(rj)∣Γj2. Our allocation procedure approximates this quantity by cjEj(rj), where the sensitivity cj is estimated by probing at a single order. The guarantee thus applies to the exact one-at-a-time loss terms. It does not establish the accuracy of their probe-based approximations or their sum as a model of joint reduction. We will state this scope explicitly.
This distinction also explains why spectral allocation without probing can perform worse than uniform allocation. Setting c≡1 treats equal readout errors in different layers as having equal effects on network loss. In Mamba-2 1.3B, however, the measured sensitivities vary across layers by more than five orders of magnitude.
Q1 - "How robust is the proposed allocation to the choice of probe level?" On S4D, varying the probe level from π=0.01 to 0.65 changes accuracy by at most 0.4 percentage points (Table 16). We have also evaluated this sensitivity on Mamba-2 1.3B. The table below reports perplexity for five probe levels, alongside uniform allocation and allocation without probing (c≡1).
Across probe levels, perplexity varies by only 0.01 at budget 0.5 and 0.04 at budget 0.3. Even at budget 0.1, the spread is just 0.31, and every probe level substantially outperforms both allocation without probing and uniform allocation, whose perplexities are 20.99 and 18.60, respectively.
The probe-based scaling also benefits other reduction scores. Applied to LAST and AIRE-Prune using their respective scores, it improves AIRE-Prune at every tested budget and LAST on SaShiMi (Appendix B.4).
Q1 - "What Proposition 2 implies about the accuracy of the practical allocation surrogate". The surrogate reproduces the one-at-a-time loss terms of Proposition 2 exactly when the sensitivity cj(r)=∣δj(r)∣Γj2/Ej(r) is independent of the order r, since a single probe then determines it. Proposition 2 does not establish this independence, but three empirical checks on Mamba-2 show that the approximation is adequate for the allocation. First, the table above shows that the final perplexity is insensitive to the probe order. Second, re-probing each layer at its allocated order changes cj by more than a factor of two in only 2 of the 24 layers of Mamba-2 130M (Appendix B.8). Third, on Mamba-2, the sum of the measured one-at-a-time losses at the allocated orders differs from the surrogate estimate by at most 25%, and by at most 2% at budget 0.1. Establishing conditions under which cj(r) remains approximately constant for balanced truncations is a promising direction for future theoretical work.
W2/Q2 - Downstream tasks, longer contexts. We kindly refer the reviewer to the general response which now includes zero-shot evaluation and longer contexts. In all tasks and budgets, POISE outperforms GHOST and also shows robustness to calibration mismatch (as further discussed below).
W2/Q2 - POISE calibration. We thank the reviewer for correctly noting that POISE uses calibration data to achieve strong performance. Compared with data-agnostic methods such as LAST, AIRE-Prune, and classical balanced truncation, a potential risk is a mismatch between the calibration and evaluation distributions. This risk can often be mitigated because POISE requires only unlabelled inputs, which can be collected at deployment, as in test-time adaptation.
We explicitly evaluate this mismatch on SaShiMi in Appendix B.6. Calibrating on the evaluation distribution, whether speech or piano, reduces excess loss by a factor of 2-8 compared with calibrating on the other distribution. Even under this mismatch, however, POISE incurs over 20 times less excess loss than classical balanced truncation and over 3,000 times less than LAST, neither of which uses calibration data.