acceptodds
Under review as a conference paper at ICLR 2027

POISE: Post-training Input-aware State Elimination for Deep State Space Models

Abstract

State space models (SSMs) trade long-context computation for a recurrent state, making state dimension a direct cost of autoregressive inference. Existing post-training reduction methods either prune modes in the trained coordinates or apply classical balanced truncation, whose controllability Gramian reflects generic white-noise excitation rather than the inputs encountered at deployment. We introduce `POISE`, a label-free, post-training method that instead balances each SSM against the state covariance induced by a small set of calibration inputs and allocates a global state budget across the network. For stable linear time-invariant (LTI) SSMs, `POISE` yields stable reduced systems and provides error guarantees for arbitrary inputs and finite horizons. We apply it to the LTI backbones S4D, SaShiMi and S5, and extend the method empirically to selective Mamba-2. On Mamba-2 B, `POISE` removes of the recurrent states for only additional WikiText perplexity. At a -token context, the reduced model comes within perplexity of Llama--B with a four-bit KV cache while using less memory.

Primary area:
learning on time series and dynamical systems
Keywords:
State Space Models, Model Order Reduction, Balanced Truncation, Post-training Model Compression, Data-driven Model Reduction

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_VoKROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SSMs carry a fixed-size internal state vector along the sequence, which could be thought of as the short-term memory the model holds. The paper proposes POISE, a post-training method that shrinks this state without labels or retraining, using a few hundred unlabeled calibration sequences. For each SSM layer, POISE changes the layer's internal coordinates so that each direction is ranked by how much it matters on the data the layer actually sees, then deletes the lowest-ranked directions. It then decides how many directions each layer keeps under one global budget. For the linear SSMs in S4D, SaShiMi and S5, the authors prove that the shrunk layer stays stable and give an exact formula and a simple upper bound for its error. Mamba-2, whose dynamics change with the input, is handled empirically without these theoretical guarantees. Experiments on multiple architectures and tasks compare against five prior state-pruning rules and report improvements.

Critical strengths
  1. Well-motivated approach with strong theoretical guarantees for LTI systems (Thm 1, Table 10). POISE uses calibration data to estimate the input-induced state covariance (P) and combines it with the observability Gramian (Q) to identify important state directions. Theorem 1 proves that the reduced LTI system remains stable and provides an upper bound on its output error. Importantly, the paper proves that using the input-induced covariance minimizes this bound. Table 10 supports the practical value of this approach, showing improvements over classical balanced truncation across the evaluated LTI systems.

  2. Thorough ablations (Appendix B). The paper separately evaluates key components of POISE, including which state directions to remove and how to allocate the state budget across systems. It also analyzes computational costs, calibration-set size, and sensitivity to calibration-distribution mismatch, helping identify which components contribute to the improvements.

  3. Strong memory–accuracy tradeoff on Mamba-2 (Fig. 1). On Mamba-2 1.3B, POISE retains only 30% of the recurrent states while increasing perplexity slightly (13.17 to 13.51). At batch size 64, total inference memory decreases from approximately 5.9 GB to 3.67 GB. The compressed model also achieves better perplexity than the smaller dense Mamba-2 780M (14.97) while using slightly less memory (3.67 vs. 3.98 GB).

Critical weaknesses
  1. POISE's stability and error guarantees apply to individual stable LTI systems, not selective Mamba-2 or the end-to-end compressed network. Although Proposition 2 characterizes network-level error propagation, it does not guarantee the accuracy of the practical allocation surrogate. On Mamba-2 1.3B (budget 0.10), spectral allocation without sensitivity probing performs worse than uniform allocation (20.99 vs. 18.60 perplexity), while adding the heuristic probe improves perplexity to 15.12. Thus, the strongest allocation gains rely on an empirically effective heuristic without a corresponding theoretical guarantee. (Fixable: clarify the scope of the guarantees and analyze the probe's robustness.)

  2. Mamba-2 is calibrated on WikiText-2 training sequences and evaluated using WikiText perplexity with 2,048-token windows. The paper does not report evaluation on other text corpora, downstream language tasks, or longer contexts. This is particularly important because POISE explicitly uses calibration data to decide which states matter for compression. States that seem unimportant for WikiText could be important for other types of text or OOD tasks. (Fixable with additional experiments.)

  3. Despite motivating POISE through inference efficiency, the paper reports memory savings without measuring end-to-end Mamba-2 decoding throughput or latency. Table 5 shows that a more memory-efficient recurrence layout can be substantially slower on S4D (3.60 vs. 1.49 ms/step), demonstrating that reduced state memory does not necessarily translate into faster inference. Without compressed-versus-original Mamba-2 speed measurements, the practical runtime benefits remain unclear. (Fixable with additional measurements.)

Questions for authors
  1. Given that spectral allocation without the sensitivity probe performs worse than uniform allocation on Mamba-2, how robust is the proposed allocation to the choice of probe level (15%)? Could the authors provide a sensitivity analysis across different probe levels on Mamba-2, and clarify what Proposition 2 implies about the accuracy of the practical allocation surrogate?

  2. How well does POISE perform when Mamba-2 is evaluated on other text distributions, downstream tasks, or longer contexts? In particular, how sensitive is the compressed model to a mismatch between the calibration and evaluation distributions?

  3. Can the authors report end-to-end decoding throughput or latency for compressed versus original Mamba-2 at different batch sizes? Given the memory–runtime tradeoff observed, it would be valuable to establish whether the reported memory savings also translate into practical speedups.

3 comments

We thank the reviewer for the positive assessment and for identifying areas where we can improve the paper.

W1 - Stability and performance guarantees. Theorem 1 indeed provides per-system guarantees for stable LTI systems. Our general response, available to all reviewers, explains what carries over to Mamba-2: the layerwise error identity and stability of the reduced recurrence hold exactly, whereas optimality of the balanced projection relies on the moment-factorization assumption.

At the network level, Proposition 2 characterizes the loss caused by reducing one system alone as ∣δj(rj)∣Γj2\\|\boldsymbol\delta_j(r_j)\\|^2_{\Gamma_j}. Our allocation procedure approximates this quantity by cjEj(rj)c_jE_j(r_j), where the sensitivity cjc_j is estimated by probing at a single order. The guarantee thus applies to the exact one-at-a-time loss terms. It does not establish the accuracy of their probe-based approximations or their sum as a model of joint reduction. We will state this scope explicitly.

This distinction also explains why spectral allocation without probing can perform worse than uniform allocation. Setting c≡1c\equiv1 treats equal readout errors in different layers as having equal effects on network loss. In Mamba-2 1.3B, however, the measured sensitivities vary across layers by more than five orders of magnitude.

Q1 - "How robust is the proposed allocation to the choice of probe level?" On S4D, varying the probe level from π=0.01\pi=0.01 to 0.65 changes accuracy by at most 0.4 percentage points (Table 16). We have also evaluated this sensitivity on Mamba-2 1.3B. The table below reports perplexity for five probe levels, alongside uniform allocation and allocation without probing (c≡1c\equiv1).

1.3B (uncompressed 13.17)
budget0.50.30.1
uniform allocation13.7614.9118.60
c≡1c\equiv114.6916.7320.99
cπc^\pi, π=0.05\pi=0.0513.3113.5515.43
cπc^\pi, π=0.10\pi=0.1013.3013.5315.40
cπc^\pi, π=0.15\pi=0.15 (default)13.3013.5115.12
cπc^\pi, π=0.25\pi=0.2513.3113.5215.31
cπc^\pi, π=0.5\pi=0.513.3113.5215.41

Across probe levels, perplexity varies by only 0.01 at budget 0.5 and 0.04 at budget 0.3. Even at budget 0.1, the spread is just 0.31, and every probe level substantially outperforms both allocation without probing and uniform allocation, whose perplexities are 20.99 and 18.60, respectively.

The probe-based scaling also benefits other reduction scores. Applied to LAST and AIRE-Prune using their respective scores, it improves AIRE-Prune at every tested budget and LAST on SaShiMi (Appendix B.4).

Q1 - "What Proposition 2 implies about the accuracy of the practical allocation surrogate". The surrogate reproduces the one-at-a-time loss terms of Proposition 2 exactly when the sensitivity cj(r)=∣δj(r)∣Γj2/Ej(r)c_j(r)=\\|\boldsymbol\delta_j(r)\\|^2_{\Gamma_j}/E_j(r) is independent of the order rr, since a single probe then determines it. Proposition 2 does not establish this independence, but three empirical checks on Mamba-2 show that the approximation is adequate for the allocation. First, the table above shows that the final perplexity is insensitive to the probe order. Second, re-probing each layer at its allocated order changes cjc_j by more than a factor of two in only 2 of the 24 layers of Mamba-2 130M (Appendix B.8). Third, on Mamba-2, the sum of the measured one-at-a-time losses at the allocated orders differs from the surrogate estimate by at most 25%, and by at most 2% at budget 0.1. Establishing conditions under which cj(r)c_j(r) remains approximately constant for balanced truncations is a promising direction for future theoretical work.

W2/Q2 - Downstream tasks, longer contexts. We kindly refer the reviewer to the general response which now includes zero-shot evaluation and longer contexts. In all tasks and budgets, POISE outperforms GHOST and also shows robustness to calibration mismatch (as further discussed below).

W2/Q2 - POISE calibration. We thank the reviewer for correctly noting that POISE uses calibration data to achieve strong performance. Compared with data-agnostic methods such as LAST, AIRE-Prune, and classical balanced truncation, a potential risk is a mismatch between the calibration and evaluation distributions. This risk can often be mitigated because POISE requires only unlabelled inputs, which can be collected at deployment, as in test-time adaptation.

We explicitly evaluate this mismatch on SaShiMi in Appendix B.6. Calibrating on the evaluation distribution, whether speech or piano, reduces excess loss by a factor of 2-8 compared with calibrating on the other distribution. Even under this mismatch, however, POISE incurs over 20 times less excess loss than classical balanced truncation and over 3,000 times less than LAST, neither of which uses calibration data.

Our general response reports the same experiment on Mamba-2 across three corpora, with consistent findings: calibration on the evaluation distribution is preferable, but mismatched calibration incurs only a small penalty, and POISE still outperforms GHOST calibrated on the evaluation distribution.

W3/Q3 - "Reduced state memory does not necessarily translate into faster inference". Our general response reports decoding throughput, latency, and peak memory for the reduced Mamba-2 1.3B models at batch sizes from 1 to 1,024. The reduced models are faster at all tested batch sizes of 64 and above. At budget 0.1, POISE achieves speed-ups of 2.0x, 3.0x, and 3.3x over the uncompressed model at batch sizes 64, 256, and 1,024, respectively. At batch size 1, where kernel-launch overhead dominates, the two additional projection maps in each reduced layer make decoding 14-17% slower.

Table 5 addresses a different issue: it compares execution layouts of the same reduced S4D/SaShiMi model at budget 0.15. These models contain many single-channel systems, which are batched into groups to limit kernel launches. The table illustrates how group size affects the trade-off between memory and speed. Mamba-2 updates the states within each layer using a single state-update kernel, so this channel-grouping trade-off does not arise. We will clarify this distinction alongside Table 5.

Reviewer_VoKROpenReview ↗

Reviewer response to rebuttal

I thank the authors for the clarifications and the additional experiments. The new results resolve the main concerns that I raised in my review:

  1. The probe-level sweep on Mamba-2 shows that the allocation is robust to the choice of probe level.
  2. The 0-shot, longer-context and calibration-mismatch evaluations show that the method generalizes beyond the calibration distribution.
  3. The throughput measurements show real speed-ups at practical batch sizes.

These results would be a useful additions to the revision, including the small slowdown at bs 1. I'm maintaining my positive score.

Reviewer_tG5bOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes POISE, a post-training compression framework for deep state space models (SSMs). The method introduces input-aware balanced truncation, using empirical state covariances from calibration data to identify state directions that should be preserved. It further proposes a global state-budget allocation procedure based on estimated local reduction errors and network-level sensitivity.

The authors provide theoretical guarantees for stable linear time-invariant SSMs and evaluate the method on S4D, SaShiMi, S5, and Mamba-2. The reported results show substantial improvements over existing state-pruning approaches, especially at aggressive compression ratios.

While the method is technically interesting and achieves promising compression results, I have significant concerns about the experimental validation, particularly the lack of direct evidence for practical inference efficiency on large-scale models and the limited evaluation of the proposed method's generalization.

Critical strengths

The paper presents a principled connection between balanced truncation and post-training state compression, supported by theoretical analysis for stable LTI systems.

The input-aware formulation is well motivated, and the global budget allocation provides a practical way to distribute compression across multiple SSM layers.

The results on Mamba-2 are promising. In particular, the method achieves substantially lower perplexity than GHOST at matched recurrent-state budgets.

Critical weaknesses
  1. Insufficient end-to-end inference efficiency evaluation on Mamba-2.

A central motivation of the paper is to reduce the memory and computational cost of autoregressive inference. However, the most important large-scale experiments do not provide sufficiently direct measurements of the actual deployment benefits.

As described in Appendix A, Mamba-2 perplexity is primarily evaluated by running the original full-size model with a modified output projection, rather than executing the reduced-state recurrence throughout the reported benchmark.

Although the authors provide numerical equivalence checks for the reduced realization, these checks do not replace an end-to-end evaluation of inference throughput, latency, and peak GPU memory under realistic decoding workloads.

The memory–perplexity comparison in Section 5.3 is based primarily on the recurrent-state footprint and a memory accounting model. Moreover, the reduced Mamba-2 implementation retains the full-width input projection and introduces additional projection matrices, which may affect practical efficiency.

This is particularly important because the proposed method changes the internal state representation rather than simply eliminating existing state channels. Thus, matching methods by retained state count does not necessarily establish comparable inference costs.

Without a systematic evaluation of actual decoding speed, runtime memory consumption, and kernel efficiency, it is difficult to assess the practical significance of the reported compression gains.

  1. Limited evaluation of large-scale language modeling and long-context capabilities.

Although the paper evaluates several SSM architectures, the large-scale language-model experiments are restricted to WikiText-2 perplexity.

No additional language-model evaluation datasets or downstream tasks are included to assess whether the compressed representations preserve broader model capabilities.

Moreover, the paper emphasizes the advantages of recurrent states over Transformer KV caches for long-context inference, but the reported comparison uses a fixed context length of 2048 tokens.

The potential advantages at substantially longer context lengths are therefore not directly demonstrated.

A more convincing experimental evaluation would include multiple language-model benchmarks, substantially longer contexts, and a wider range of generation workloads.

  1. The extension to selective SSMs lacks sufficient validation of its approximation assumptions.

The main theoretical guarantees apply to stable linear time-invariant systems.

For Mamba-2, however, the state transitions and output maps depend on the inputs, so these guarantees do not directly apply.

As explained in Appendix A, the authors replace the joint moments of the state and output maps with products of separate moments. This approximation is exact under an independence condition that generally does not hold for Mamba-2.

The resulting balancing procedure therefore minimizes a surrogate rather than the actual output reconstruction error.

While the reported empirical results are encouraging, the paper provides limited direct evidence about the approximation error introduced by this step or its sensitivity to changes in the input distribution.

Since Mamba-2 constitutes the most important practical application of the proposed method, this limitation deserves further investigation.

  1. The global allocation surrogate is not sufficiently characterized.

The proposed global budget allocation replaces the true network-level compression objective with a sum of local errors weighted using a single probe configuration.

This approximation neglects the interactions between simultaneously compressed systems and assumes that sensitivity coefficients estimated at one compression level remain informative at other compression levels.

Appendix B.7 reports that the resulting one-at-a-time surrogate objective is approximately four to five times its directly optimized value on S4D, despite the relatively small difference in final classification accuracy.

This suggests that the surrogate may be poorly calibrated as an approximation to the objective it is intended to minimize.

It would be helpful to understand whether the same discrepancy persists for larger models and more aggressive compression settings, particularly where classification accuracy is not saturated.

  1. Additional concerns about evaluation protocols and baseline comparisons.

Appendix A states that, for LRA-Text, the training checkpoint is selected using the test split because no validation split is available. Although the same checkpoint-selection procedure is applied across compression methods, using test data for model selection is methodologically undesirable.

Furthermore, the comparison against GHOST primarily matches recurrent-state counts, while the resulting compressed realizations have different parameter structures and computational requirements.

An evaluation under matched measured inference latency, memory consumption, or FLOP budgets would provide a more direct assessment of the practical advantages of the proposed compression method.

Questions for authors

Can the authors report actual autoregressive decoding throughput, latency, and peak GPU memory for the deployed reduced-state Mamba-2 implementation across multiple batch sizes?

Does POISE preserve performance on additional language-model benchmarks and long-context evaluation tasks beyond WikiText-2?

How accurate is the moment-factorization approximation used for selective Mamba-2, and how does its error depend on the input distribution?

Can the authors compare POISE and GHOST under matched runtime or FLOP budgets rather than retained state counts alone?

How well does the proposed allocation surrogate predict actual network-level degradation at different compression budgets, particularly on larger models?

4 comments

We thank the reviewer for the detailed review. Below, we address all concerns, including those on the implementation and evaluation of POISE on Mamba-2. We also provide new measurements of throughput, latency, and peak memory, comparisons at comparable runtime, and evaluations on additional datasets and longer contexts. Finally, we address the theoretical assumptions, allocation surrogate, and evaluation protocol, drawing on both the submitted results and the additional experiments.

W1/Q1 - Implementation and evaluation of Mamba-2. We agree that the performance of Mamba-2 must be evaluated using the actual reduced rr-dimensional state layer. This is specifically important for memory (see the results in Section 5.3). For perplexity, two evaluation methods are theoretically equivalent: (i) running the original full-size model with a modified output projection, as in GHOST, which we used, and (ii) running the reduced rr-dimensional model directly. Appendix A verifies that this equivalence also holds numerically: the two methods agree within 2×10−52\times10^{-5} in perplexity on four 2,048-token test windows. Our intention was to explain and validate the perplexity-evaluation method used by GHOST. We will revise the paper to clarify. All new rebuttal numbers use the reduced rr-dimensional state layer.

W1/Q1. End-to-end evaluation: throughput, latency and peak memory. Our initial submission provides an analytical expression for peak memory and reports measurements of memory per state, perplexity, and the total runtime of POISE. We agree that direct measurements of throughput, latency, and peak memory provide a very valuable complement, and we have now added them. All results are reported in our general response, available to all reviewers. Specifically, we measured decoding speed and peak memory for the deployed rr-state models at batch sizes ranging from 1 to 1,024. These additional measurements further demonstrate the practical benefits of POISE, and we thank the reviewer for suggesting them. They are consistent with the paper's analytical memory accounting (an 81% reduction measured, against 86% in the accounting).

W1/Q1 - Impact of projection matrices. Indeed POISE introduces additional maps VV and U⊤U^\top. They add 2rN2rN weights per layer. Together with the full-width input projection, which GHOST prunes, this represents only 1.7% more parameters and FLOPs per token than GHOST at budget 0.1 on 1.3B, as already reported in Appendix A. Their effect on practical efficiency is small. At batch 1 they add 0.24 to 0.28 ms per step, and from batch 64 on the decoding time is set mainly by the stored state, which is where the speed-ups come from.

W1/W5/Q4 - Matched runtime rather than retained state count. All methods in the paper indeed use the retained state count as their budget. This is consistent with the definition of state sparsity used in the GHOST paper. To address the reviewer's request, we timed GHOST's deployed implementation, with pruned rows and convolution channels removed, and remeasured POISE in the same session under identical conditions. The timings differ from those in our general response by at most 6%. The table reports milliseconds per decoding step on Mamba-2 1.3B:

batch1642561,024
uncompressed1.685.7818.5971.37
POISE, budget 0.51.964.9214.4953.90
GHOST, budget 0.51.653.7410.2537.87
POISE, budget 0.31.973.9910.6039.10
GHOST, budget 0.31.835.1114.5754.02
POISE, budget 0.11.922.986.3721.94
GHOST, budget 0.11.803.026.5822.30

At equal state counts, neither method is uniformly faster. Runtime also depends on how efficiently the decoding kernel handles the retained state widths. The kernel is tuned for powers of two: at budget 0.5, GHOST retains exactly 64 of the 128 states in every layer, whereas at budget 0.3, it retains 38 or 39. These widths are handled less efficiently, making GHOST slower at 0.3 than at 0.5 despite retaining roughly 40% fewer states.

At comparable latency, POISE achieves lower perplexity. At batch 1,024, POISE with budget 0.3 decodes at nearly the same speed as GHOST with budget 0.5 (39.10 versus 37.87 ms per step), while achieving perplexity 13.51 versus 14.23 (Table 9). At budget 0.1, the two decode at the same speed (21.94 versus 22.30 ms per step), with perplexity 15.12 versus 25.44. Moreover, POISE at budget 0.1 outperforms GHOST at budget 0.3 in perplexity on both models (15.12 versus 16.18 on 1.3B, and 34.34 versus 37.82 on 130M), using one-third as much recurrent state. On 1.3B at batch 256, it also decodes 2.3x faster (6.37 versus 14.57 ms per step).

W2/Q2 - Additional datasets and longer contexts for Mamba-2. We followed GHOST's evaluation protocol, WikiText-2 with 2,048-token contexts, to ensure a direct comparison. As requested, we have expanded the evaluation to zero-shot downstream tasks, two additional corpora with calibration on each, and 4,096-token contexts, twice the pretraining length. All results are reported in our general response. POISE substantially outperforms GHOST across these evaluations. We thank the reviewer for suggesting these additions.

W3/Q3 - Mamba-2 guarantees and its moment factorization assumption. The LTI guarantees indeed do not apply to Mamba-2, and our general response explains what carries over. The moment factorization assumption is not required for (U,V)(U,V) to be well-defined. It is used to choose (U,V)(U,V), and to derive the spectral variant of POISE on Mamba-2 since in that case the reduction error factorizes into tr⁡((I−UV)⊤∑tCtCt⊤(I−UV)∑ththt⊤)\operatorname{tr}\bigl((I-UV)^\top\sum_t C_tC_t^\top(I-UV)\sum_t h_th_t^\top\bigr) which simplifies into a tail of singular values, similar to Theorem 1. The exact variant also uses these (U,V)(U,V), but allocates on the true error, without the assumption.

To answer your concern on "the approximation error introduced by this step or its sensitivity to changes in the input distribution", Table 9 shows that allocating on the true or the factorized curve gives the same perplexity within 0.18 with the probed scale. We give a further explanation. For layer jj at order rr, let Ej(r)=∑i>rσj,i2E_j(r)=\sum_{i>r}\sigma_{j,i}^2 be the factorized read-out error, Ejtrue(r)E_j^{\mathrm{true}}(r) the true read-out error on the calibration blocks, and Fj(r)=Ejtrue(r)/Ej(r)F_j(r)=E_j^{\mathrm{true}}(r)/E_j(r). The factorization underestimates the true error by a factor of about 2 (1.6 to 2.7 across models and draws, and 1.7 to 2.2 across WikiText-2, C4 and PTB calibration). However, this has little impact on the allocation, since the minimization ignores a factor that is the same at every order. Allocating on the true curve instead would replace each term cjπEj(rj)c^\pi_jE_j(r_j) by cjπEj(rj) Fj(rj)/Fj(rjπ)c^\pi_jE_j(r_j)\ F_j(r_j)/F_j(r^\pi_j). Only the ratio Fj(rj)/Fj(rjπ)F_j(r_j)/F_j(r^\pi_j) can therefore change the allocation, and its median across layers at the deployed orders is 0.75 to 1.06, on both models and all three corpora, which answers the concern on changes in the input distribution.

W4 - Global allocation and its surrogate. The sum of one-at-a-time losses is the standard surrogate of layer-wise post-training compression (LeCun et al., 1990; Yuan et al., 2023), and Section 4.1 explains why we adopt it. This allows Proposition 2 to make each term exact, and the sum is separable, so a single sort or dynamic programming minimizes it. These works show the efficiency of such approaches and ours follows the same methodology.

W4 - The probe-based solution for the one-at-a-time loss. The main interest of our probe-based allocation is to avoid evaluating the network at every order of every system, as explained in Section 4.2. An allocation surrogate does not need to reproduce the value of the exact objective, but rather yield an allocation close to the exact minimizer in accuracy, which it does at budgets 0.10 to 0.30. Without the probe (c≡1c\equiv1) this ratio is 8 to 20, so the probe brings the allocation much closer to the optimum. The probe-based allocation comes within 0.08 accuracy points of dynamic programming on the exact one-at-a-time curve at every budget from 0.10 to 0.30, with 64 times fewer network passes.

W4 - Sensitivity coefficients estimated at one compression level. The reviewer's concern that sensitivities measured at one compression level are used at another is exactly what the one-step correction of Appendix B.8 addresses: it re-probes every system at its allocated order and repeats the sort. Without it, POISE already outperforms every baseline in the tables of the paper. With it, it improves the result where the two orders differ most, on S4D at budget 0.03 (89.26 to 93.91) and on SaShiMi at every budget tested. On Mamba-2 it does not help, because the rates change little at the deployed orders: re-probing changes the rate by more than a factor of two in only 2 of the 24 layers of 130M (see also our reply to Reviewer VoKR).

W4/Q5 - Larger models, aggressive budgets and unsaturated accuracies for the probe. On larger models, the exact one-at-a-time curve needs one network pass per candidate order of every system, about 2×1052\times10^5 passes on SaShiMi and an estimated 70 GPU-hours on Mamba-2 1.3B, against 30 minutes for the probes. Avoiding this cost is the purpose of the probe. Nonetheless, on Mamba-2 (130M and 1.3B), where the full curve is out of reach, we can still measure the one-at-a-time losses and the full loss at the allocation found by the surrogate. We observe that the sum of the one-at-a-time losses differs from the surrogate by at most 25% at every budget (at most 2% at budget 0.1). The loss of the whole reduced network is 1.2 to 1.35 times the sum of the one-at-a-time losses at budgets 0.5 and 0.3, and about twice that sum at 0.1. This excess is the interaction between layers that the surrogate leaves out. At 0.1, the surrogate's allocation still gives 15.12 against 25.44 for GHOST (Table 9).

On S4D at aggressive and unsaturated budgets 0.03 and 0.05, dynamic programming on the exact one-at-a-time curve reaches 96.64 and 98.40, against 89.26 and 97.73 for the probed sort (Table 15 for the latter). Optimized exactly, the additive objective therefore remains accurate at 3% of the state, and the gap comes from measuring each sensitivity at a single order, which the one-step correction of Appendix B.8 reduces (93.91 and 98.21).

W5 - Checkpoint selection on LRA-Text. This follows the standard training pipeline of the S5 authors. LRA-Text has no validation split, and the official S5 code (Smith et al., 2023), which the code of LAST also uses, selects the training epoch on the test split for this task. We kept this pipeline so that our checkpoints reproduce the published accuracies, and we disclosed it in Appendix A. It has no bearing on POISE or on any comparison in the paper. The epoch is fixed before any compression, from the accuracy of the uncompressed model alone, and every method, POISE and the baselines alike, then compresses the same checkpoint. No step of POISE uses test data: its calibration inputs are training sequences, disjoint from the test set. We will state this more explicitly in the revised version.

References

  • Y. LeCun, J. S. Denker and S. A. Solla. Optimal brain damage. In NeurIPS, 1990.
  • J. T. H. Smith, A. Warrington and S. W. Linderman. Simplified state space layers for sequence modeling. In ICLR, 2023.
  • Z. Yuan, Y. Shang, Y. Song, D. Yang, Q. Wu, Y. Yan and G. Sun. ASVD: Activation-aware singular value decomposition for compressing large language models. arXiv:2312.05821, 2023.

Thanks for the response. I am satisfied with the response and I will raise my score to 3.

Reviewer_7JzZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes POISE for post-training state reduction in deep SSMs. It has two main components: (1) input-weighted balanced truncation, which uses calibration data to adapt the state coordinates to directions activated by the input distribution, and (2) layer-wise state budget allocation using probe-based layer-wise impact estimation on network-level output error. For stable LTI systems, the paper provides a finite-horizon output-error bound, and shows that the reduced system remains stable. Experiments cover S4D, SaShiMi, S5, and Mamba-2 SSMs using sequential MNIST, SC09, LRA-Text, and WikiText. On WikiText, the compressed Mamba-2 1.3B achieves lower perplexity than a dense Mamba-2 780M at a similar memory footprint. The theoretical guarantees apply to LTI systems, and Mamba-2 is evaluated empirically.

Critical strengths
  1. The paper addresses the practically setting of correlated, non-white inputs with an input-aware reduction method. The combination of calibration-based state coordinates and layer-wise budget allocation is supported by theoretical analysis for LTI systems.
  2. The Mamba-2 results demonstrate a potentially useful compression trade-off: the reduced 1.3B model outperforms a smaller dense Mamba-2 model at a similar memory footprint.
Critical weaknesses
  1. The evaluation does not provide a sufficiently broad view of the performance–compression trade-off. Figure 2 in the appendix gives more informative curves than Table 1’s selected budgets, but covers only a limited range. The LRA-Text result at the particularly aggressive 3% budget shows a large gap over the baselines, but the reduced model’s absolute performance is low. For Mamba-2, the paper does not evaluate a classical BT baseline, making it difficult to isolate the benefit of POISE’s input-aware weighting on this model. Finally, Table 2 reports per-stream state memory, whereas end-to-end memory comparisons are shown only for Mamba-2 in Figure 1. Reporting total or peak inference memory for the other benchmarks would help clarify the practical efficiency gains.
  2. Some methodological details are difficult to follow. In particular, the motivation for the probe-based surrogate in Section 4.2 and how Steps 5–7 of Algorithm 1 produce the final allocation could be explained more clearly. Since Mamba-2 is a major empirical result and differs from the LTI setting, its implementation details would also benefit from a fuller explanation in the main text.
Questions for authors
  1. Could the authors present a detailed comparison of the methods’ performance–compression trade-offs? For example, comparing the state budgets each method requires to reach matched performance degradation would clarify the each benefit of POISE that require calibration and baselines that require less calibration or setup. Showing the full 0–100% budget range for all tasks in Figure 2 could help.
  2. Since stability preservation is presented as an advantage of POISE, could the authors summarize how it compares with the reduction methods discussed in the paper? A concise table indicating whether each method provides the stability would clarify how distinctive POISE’s guarantee is.
  3. Could the authors add a classical BT baseline for Mamba-2, using the same extension from LTI SSMs to Mamba as in POISE, but removing POISE’s input-aware weighting and allocations?
4 comments

We thank the reviewer for their insightful questions, which we answer below, along with all the concerns raised.

W1/Q1 - Broader view on the performance-compression trade-off. On S4D and SaShiMi, POISE already matches uncompressed performance within the budget range shown in Figure 2. On Mamba-2 1.3B, it comes within 0.13 perplexity at budget 0.5. For S5, we agree with the reviewer that the original budgets were aggressive, chosen to distinguish the methods, and that larger budgets are relevant in practice.

We have therefore extended the S5 curves to larger budgets for every method on all three tasks, including the newly added LRA-Image. In these runs, we use a revised POISE rule that automatically selects between c≡1c\equiv1 and cπc^\pi at each budget by choosing the smaller KL divergence from the uncompressed model on 32 held-out training inputs, without labels. This adjustment follows our observation that, on S5 alone, c≡1c\equiv1 can be more robust than the probe-based choice. The same rule applies to every model, and it keeps cπc^\pi on SaShiMi and Mamba-2 at every budget and in most cases on S4D. There, its accuracy is never more than 0.02 points below that of cπc^\pi, and up to 3.6 points above it at the most aggressive budgets. Results are presented in the table below, which will be added in the revised version.

TextImageListOps
budget0.050.070.100.300.500.700.670.80
uncompressed88.6788.6788.6787.1387.1387.1361.4261.42
Global H∞\mathcal H_\infty57.3065.6573.1518.7080.8086.9230.8044.95
LAST52.4353.8959.4131.6073.7485.7040.1048.67
AIRE-Prune53.3953.9353.6124.3676.3985.7349.6756.38
GHOST55.9552.1252.1671.4885.2386.8746.9053.10
classical BT80.5785.9187.8167.6686.8287.14--
POISE81.4486.8288.3878.2086.9287.1449.5259.87

At these larger budgets, POISE comes within 0.3 points of the uncompressed accuracy on LRA-Text at budget 0.10, matches it on LRA-Image at 0.70, and comes within 1.6 points on LRA-ListOps at 0.80. The matched-degradation comparison requested by the reviewer follows from these curves and the paper's tables: for each method, we can identify the smallest tested budget that keeps degradation within a specified tolerance of the uncompressed model, often far below 100%. At those budgets, several baselines still remain far below, demonstrating the benefit of balancing, and of calibration data on top of it. On LRA-Text, POISE stays within 2 points from budget 0.07, compared with 0.10 for classical balanced truncation. On LRA-Image, both methods stay within 1 point from budget 0.50, whereas GHOST and Global H∞\mathcal H_\infty reach this tolerance only at 0.70. On Mamba-2 1.3B, POISE at budget 0.1 achieves lower perplexity than GHOST at 0.3 (15.12 versus 16.18), despite using one-third as much recurrent state.

W1/Q3 - A classical balanced truncation baseline on Mamba-2, isolating the input-aware weighting. We thank the reviewer for this interesting suggestion. Classical BT does not apply directly to Mamba-2's input-dependent, time-varying dynamics. Our POISE construction nevertheless provides a natural analogue (see our general response): replace the deployment-input weighting with a white-noise reference. Let WB,WCW_B,W_C denote the in_proj rows producing Bt,CtB_t,C_t, and let KB,KCK_B,K_C denote the corresponding depthwise convolution kernels. For a temporally white layer input with identity covariance and a linearised activation, the input-side moment becomes E[BtBt⊤]=(WBWB⊤)∘(KBKB⊤),\mathbb{E}[B_tB_t^\top] =(W_BW_B^\top)\circ(K_BK_B^\top), where ∘\circ denotes the Hadamard product. This matrix is positive semidefinite by the Schur product theorem. We retain the empirical readout moment ∑tCtCt⊤\sum_t C_tC_t^\top, changing only the input-side weighting that the reviewer asks us to remove. Classical BT and the input-aware weighting both use the unscaled allocation c≡1c\equiv1, and the last row adds the probed scale, so each row changes one component.

The table below reports perplexity on Mamba-2 130M and 1.3B, comparing GHOST with the BT analogue described above and then adding input-aware weighting and probed allocation in turn. Lower values are better.

130M1.3B
budget0.50.30.10.50.30.1
uncompressed25.8525.8525.8513.1713.1713.17
GHOST29.0337.8265.1214.2316.1825.44
classical BT31.6542.1662.0715.4618.0123.81
+ input-aware weighting (c≡1c\equiv1)29.1632.6441.3414.6916.7320.99
+ probed allocation (POISE)26.3627.2334.3413.3013.5115.12

On both models, classical BT performs worse than GHOST at budgets 0.5 and 0.3, but better at 0.1. Each component of POISE lowers perplexity at every tested budget on both models. Making the readout side data-free as well worsens performance at every tested budget, yielding perplexities of 17.03, 22.53, and 120.4 on 1.3B and above 10710^7 on 130M. These results support the premise of POISE: effective state reduction should account for the inputs a selective SSM actually receives, for which white noise is a poor substitute. We will add this baseline to the revised paper.

W1 - Total/peak inference memory on other benchmarks. We thank the reviewer for this suggestion. The paper reports end-to-end memory savings for Mamba-2 in Figure 1, complemented by decoding-throughput measurements in our general response, and for S4D and SaShiMi at budget 0.15 in Tables 2 and 5. We now report additional budgets for S4D and SaShiMi, using the default four channel groups per layer, and measurements for S5, where the main benefit is faster inference. For S4D and SaShiMi, we measure peak memory during a full recurrent pass, excluding the memory occupied by model weights and inputs. S4D results are averaged over ten seeds.

S4D, budgetuncompressed0.030.050.100.150.30
state per stream (kB)65.52.54.38.913.925.4
peak memory, batch 64 (MB)6.930.350.520.951.322.39
peak memory, batch 1,024 (MB)110.95.68.315.121.843.6
SaShiMi, budgetuncompressed0.020.050.100.150.25
state per stream (MB)1.840.0460.1140.2240.3100.511
peak memory, batch 64 (MB)2406.415.129.140.165.9
peak memory, batch 1,024 (MB)3,8361082685037141,092

Peak memory decreases approximately in proportion to state size, as the recurrent state accounts for most of the memory retained during the pass. At budget 0.10 and batch size 64, peak memory falls by 86% on S4D and 88% on SaShiMi. At batch size 1,024, SaShiMi's peak memory falls from 3.8 GB to 0.5 GB.

The S5 models on LRA are bidirectional: each layer scans the sequence in both directions, so outputs depend on future inputs and cannot be produced autoregressively. Inference uses parallel scans over the full 4,096-step sequence, with peak memory dominated by scan activations rather than the recurrent state. State reduction therefore primarily lowers inference time. The following measurements cover a JIT-compiled inference pass on a batch of 64 LRA-Text sequences, evaluated across three checkpoints:

S5, budgetuncompressed0.100.050.03
peak memory (MB)1,7081,6131,6241,635
ms per pass39.318.0 (2.2x)17.1 (2.3x)12.0 (3.3x)

S5 achieves speed-ups of 2.2-3.3x, with modest reductions in peak memory.

W2 - Methodological details. We thank the reviewer for the opportunity to clarify the design of our method. The probe-based surrogate in Section 4.2 is motivated by computational cost. The exact alternative evaluates the network loss at every candidate order of every system, requiring one network pass per order: 16,384 passes on S4D and 6,144 on Mamba-2 1.3B, with an estimated cost of 70 GPU-hours for the latter. Probing requires only one pass per system: 256 and 48 passes, respectively, taking about 30 minutes on Mamba-2 1.3B. On S4D, where the exact alternative is affordable, probe-based allocation comes within 0.08 accuracy points of dynamic programming on the exact one-at-a-time loss curves at budgets 0.10-0.30 (Appendix B.7).

The steps of Algorithm 1 serve distinct purposes. Step 5 computes each system's reduction-error curve using input-weighted balanced truncation, with the finite-horizon guarantee of Theorem 1 in the LTI setting. Step 6 estimates how these errors affect the loss of the full nonlinear network. Proposition 2 expresses the loss caused by reducing one system alone as a weighted norm of that system's reduction error. We approximate this weighting by a scalar sensitivity cjc_j, estimated at a single order with one probe pass. Step 7 uses the resulting loss estimates to allocate the state budget across systems.

We agree that Mamba-2 is "a major empirical result". We will move its implementation details from Appendix A to the main text and clarify which guarantees carry over, as explained in our general response.

Q2 - Stability across methods, and how distinctive the guarantee of POISE is. We thank the reviewer for this question, which highlights the theoretical core of our contribution. All baselines except classical BT truncate in the trained coordinates. They keep a subset of the original modes (Global H∞\mathcal H_\infty, LAST, AIRE-Prune) or of the original states (GHOST). The retained eigenvalues are then eigenvalues of the trained model, so stability is preserved trivially whenever that model is stable. The price is that the states are not mixed before truncation, and no error guarantee follows. Balanced truncation first changes coordinates so that the retained states carry the most energy, which gives the classical H∞\mathcal H_\infty bound. Stability is then no longer automatic.

methodreduced model stableerror guarantee
Global H∞\mathcal H_\infty, LAST, AIRE-Prune, GHOSTyes, triviallyno input-dependent bound
classical BTyesH∞\mathcal H_\infty, 2∑i>rσi2\sum_{i>r}\sigma_i
two-sided weighted or time-limited BTnot in generalrealization-dependent
POISEyes, for a stable system, any P,M≻0P,M\succ0exact, any input and horizon

For classical BT, the stability of the reduced model is classical (Pernebo and Silverman, 1982). Extension to general Gramians has been an active research direction. Once weights are placed on both the input and the output side, as in frequency-weighted BT (Enns, 1984), the reduced system can lose stability, and no error bound in terms of the weighted singular values exists (van Gestel et al., 2001). The available repairs (Wang et al., 1999) carry constants that depend on the realization. Theorem 1(i) answers this for our weighting. It holds for every P≻0P\succ0 and every M≻0M\succ0, with no condition on the weight, on the input process or on the Hankel spectrum. We will merge this table into our existing Table 19 and move it to the main body in the revised version.

References

  • D. F. Enns. Model reduction with balanced realizations: An error bound and a frequency weighted generalization. In IEEE Conference on Decision and Control, 1984.
  • L. Pernebo and L. M. Silverman. Model reduction via balanced state space representations. IEEE Transactions on Automatic Control, 1982.
  • T. Van Gestel, B. De Moor, B. D. O. Anderson and P. Van Overschee. On frequency weighted balanced truncation: A constructive counterexample to Enns' conjecture. In European Control Conference, 2001.
  • G. Wang, V. Sreeram and W. Q. Liu. A new frequency-weighted balanced truncation method and an error bound. IEEE Transactions on Automatic Control, 1999.

We thank the authors for the detailed responses. The comparison with classical BT clearly demonstrates POISE’s performance. As the paper explains, classical BT includes established Hankel singular value–based reduction methods, making this comparison informative.

Another point that could be addressed is the missing classical BT evaluation on ListOps. Since stability can be ensured during training, evaluating classical BT on a stable S5 checkpoint could provide a useful information, completing the comparison.

I also encourage the authors to follow through on the planned revisions, particularly by broadening the baseline comparisons across budgets and clarifying the extension to Mamba. I have increased my score to 3.

Reviewer_hsf5OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose POISE a label-free post-training state-reduction framework. The reported results improve over previous compression methods. The authors provide a well written theoretical analysis for Linear Time Invariant Systems (LTIs). The theoretical results depend on linearity; e.g., Theorem 1 relies on the discrete Lyapunov/Stein equation. The Stein equation ATQMA−QM+CTMC=0A^T Q_M A - Q_M + C^T M C = 0 is defined only for fixed matrices. The assumption does not hold for selective SSMs like Mamba or Mamba-2. However, the experimental results suggest that extending the results to non LTI systems could be possible. POISE method is summarized in Algorithm 1.

Critical strengths
  • Finite-horizon distribution-depend bound and optimality results (non-trivial)
  • End to end budget allocation via network probes (avoids expensive combinatorial evaluations)
  • Going from LTI theoretical results to practical Mamba-2 testing using sample moments and showing improvement over GHOST shows practical relevance.
Critical weaknesses

None

Questions for authors

None

1 comment

We thank the reviewer for the positive assessment and for highlighting the two contributions we consider central and of broad interest: the finite-horizon identity in Theorem 1 and the probe-based allocation. We briefly elaborate on their significance below.

Theorem 1 for control theory. The identity holds for any weights P,M≻0P,M\succ0, any horizon, and any input process. It also gives an exact H2\mathcal H_2 identity for classical balanced truncation (P=WcP=W_c, M=IM=I, white noise, T→∞T\to\infty), with a remainder of definite sign and hence the a priori bound ∣G−Gr∣H22≤∑i>rσi2\\|G-G_r\\|_{\mathcal H_2}^2\le\sum_{i>r}\sigma_i^2. We will add this corollary and its proof to the appendix.

Probe-based allocation. Proposition 2 characterizes the loss from compressing one system through a KL divergence between network outputs. The probe evaluates this loss at a single order to estimate the system's sensitivity, requiring JJ passes rather than the ∑jnj\sum_j n_j passes needed to evaluate every candidate order. This reduces the cost from 16,384 to 256 passes on S4D and from 6,144 to 48 on Mamba-2 1.3B: about 30 minutes rather than an estimated 70 GPU-hours on the latter. On S4D, where evaluating the full curve is affordable, probe-based allocation comes within 0.08 accuracy points of dynamic programming on that curve at every tested budget from 0.10 to 0.30 (Appendix B.7).

Scope of the LTI theory. The reviewer correctly identifies the scope of Theorem 1: it relies on the Stein equation and applies to LTI systems. The Mamba-2 extension provides additional empirical evidence beyond this setting. As our general response explains, its layerwise error identity and inherited recurrence stability hold exactly, while optimality of the balanced projection requires the moment-factorization assumption. We will move the construction and this clarification from Appendix A to the main text.

We would be happy to address any further questions.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General comment (Part 1/2)

We thank all reviewers for their thoughtful questions and suggestions. Below, we address the key recurring points, which clarify the intended interpretation of POISE without affecting its soundness, and present additional experiments that further support its effectiveness.

What our theory covers, and what carries over to Mamba-2. As noted by all reviewers, our main theoretical contribution concerns stable linear time-invariant (LTI) state-space systems. For these systems, Theorem 1 establishes an exact finite-horizon output-error identity for POISE, stability of every truncation, and the input weight that minimizes the resulting error bound. In particular, the optimal weight is the state covariance induced by the inputs of interest. These guarantees are central to the paper and distinguish POISE from the diagonal-LTI baselines LAST, AIRE-Prune, and Global H∞\mathcal H_\infty, which do not provide the same input-dependent output-error guarantee.

For Mamba-2, Theorem 1 does not apply, but the layerwise error identity and stability of the reduced recurrence hold exactly. Optimality of the balanced projection further requires a moment-factorization assumption. We agree with Reviewer VoKR that this distinction should be explicit. We will clarify it in the main text and move the Mamba-2 construction there from Appendix A. Specifically, Mamba-2's scalar transition atIa_tI ensures that the projected state zt=Vhtz_t=Vh_t follows the exact reduced recurrence. With readout y^t=Ct⊤Uzt\hat y_t=C_t^\top Uz_t, this gives yt−y^t=Ct⊤(I−UV)hty_t-\hat{y}_t=C_t^\top(I-UV)h_t at every step along each input trajectory. The reduced transition is atIra_tI_r, so stability is inherited. Neither property requires a statistical assumption. To choose the projection, we balance the state moment P^=∑ththt⊤\widehat P=\sum_t h_th_t^\top and readout moment Q^=∑tCtCt⊤\widehat Q=\sum_t C_tC_t^\top. This minimizes an output-error surrogate obtained by factorizing their joint moments, with minimum ∑i>rσi2\sum_{i>r}\sigma_i^2. As Appendix A explains, this need not minimize the true error when the factorization fails. The exact calibration error at every order is nevertheless available from one accumulated N×NN\times N matrix, without this assumption. Probing and allocation then follow Algorithm 1.

Our numerical experiments support the extension, which Reviewers 7JzZ and hsf5 also highlight. On Mamba-2 1.3B, removing 70% of the recurrent state increases perplexity by 0.34, versus 3.01 for GHOST (Table 9). Removing 90% increases it by 1.95, versus 12.3 (Table 1).

Inference speed on Mamba-2. The paper focuses on the memory savings from state reduction, as memory is a key constraint in autoregressive serving. We thank Reviewers tG5b and VoKR for suggesting direct throughput measurements, which we now report and will include in the revision. We decoded the stored reduced Mamba-2 1.3B models one token at a time on a single GH200, using mamba_ssm's native decoding path: its fused kernels and the single CUDA graph per batch size used by generate. We report the median of seven runs of 128 decoding steps, after 32 warm-up steps, with the state in float16:

ms per step (speed-up), batchpeak memory
perplexity1642561,024batch 1,024
uncompressed13.171.705.9819.675.756.7 GB
POISE, budget 0.513.301.975.02 (1.19x)15.1 (1.29x)57.2 (1.32x)31.0 GB
POISE, budget 0.313.511.984.03 (1.48x)11.1 (1.76x)41.0 (1.85x)20.8 GB
POISE, budget 0.115.121.942.97 (2.01x)6.58 (2.98x)22.9 (3.30x)10.7 GB

The reduced models are faster at all tested batch sizes of 64 and above, with speed-ups increasing with both batch size and the degree of state reduction. At batch 1,024, budget 0.1 yields a 3.30x speed-up and lowers peak memory from 56.7 to 10.7 GB. This measured 81% reduction supports the substantial memory savings reported in Figure 1, whose accounting gives 86%. At batch 64 and 256, at budget 0.1, peak memory falls from 6.36 and 16.43 GB to 3.87 and 5.25 GB. At batch 1, the reduced models are slightly slower and their peak memory rises by 0.1 to 0.4 GB: the two projection maps VBtVB_t and U⊤CtU^\top C_t in each reduced layer add 0.24-0.28 ms (14-17%) to a decoding step of roughly 2 ms. At this batch size, kernel-launch overhead limits performance rather than state size. Fusing these maps into the state-update kernel could reduce this overhead.

AuthorsOpenReview ↗

General comment (Part 2/2)

Mamba-2 beyond WikiText-2. The Mamba-2 experiments follow GHOST's protocol, WikiText-2 perplexity at 2,048 tokens, so that the comparison with GHOST is direct. Since POISE selects states from unlabelled calibration inputs, the natural further question, raised by Reviewers tG5b and VoKR, is how the reduced model behaves on other text and other tasks.

(i) Other corpora. As in the SaShiMi experiment of Appendix B.6, we calibrate POISE and GHOST on each of three corpora, WikiText-2, C4 and the Penn Treebank (PTB), and evaluate every reduction on all three, on Mamba-2 1.3B. WikiText-2 is scored by word perplexity, as in the paper, C4 and PTB by token perplexity.

evaluated on
methodcalibrated onWikiText-2C4PTB
uncompressednone13.1714.7817.72
budget 0.3
GHOSTWikiText-216.1817.8921.77
GHOSTC416.7917.9822.20
GHOSTPTB16.6317.9120.95
POISEWikiText-213.5115.3318.14
POISEC413.5515.0818.13
POISEPTB13.7015.2417.97
budget 0.1
GHOSTWikiText-225.4426.8938.34
GHOSTC425.3326.2042.02
GHOSTPTB25.1126.5731.70
POISEWikiText-215.1217.2820.86
POISEC415.6016.6921.48
POISEPTB15.7416.9919.15

At budget 0.1, calibrating on the evaluation corpus helps most on PTB. There, POISE reaches 19.15 when calibrated on PTB, against 20.86 and 21.48 when calibrated on WikiText-2 or C4. On WikiText-2 and C4 a mismatched calibration costs POISE at most 0.62. At budget 0.3 every calibration keeps POISE within 0.55 of the uncompressed model on every corpus. At both budgets and on every corpus, POISE calibrated on another corpus is ahead of GHOST calibrated on the evaluation corpus itself.

(ii) Longer contexts. Mamba-2 is pretrained on sequences of 2,048 tokens. We evaluate at 4,096 tokens, twice that length. POISE and GHOST are calibrated on blocks of 4,096 tokens of WikiText-2.

budgetmethodWikiText-2 at 4,096 tokens
noneuncompressed13.80
0.3GHOST16.09
0.3POISE14.03
0.1GHOST26.54
0.1POISE15.89

POISE stays within 0.23 of the uncompressed model at budget 0.3 and within 2.1 at budget 0.1, where GHOST loses 2.3 and 12.7. The memory advantage over a key-value cache grows with the length, since the recurrent state does not grow with the context while a key-value cache grows linearly with it.

(iii) Downstream tasks. Items (i) and (ii) evaluate perplexity on held-out text, measuring the next-token prediction objective used in pretraining. Downstream tasks assess performance on specific language-understanding and reasoning problems. Following GHOST, we evaluate these tasks zero-shot with lm-eval, without worked examples in the prompt or task-specific fine-tuning. For multiple-choice tasks, the model selects the option with the highest log-likelihood. Length-normalized accuracy instead selects the option with the highest log-likelihood per unit length, reducing the preference for shorter answers.

taskwhat it testsmetric
LAMBADAthe last word of a passage, which needs its wider contextaccuracy
PIQAphysical commonsense, the right one of two solutionsaccuracy, normalized
ARC-e, ARC-cgrade-school science, four options (ARC-c: the hard set)accuracy, normalized
WinoGrandewhich of two entities fills a blank in a sentenceaccuracy
HellaSwagthe plausible continuation of an everyday scene, four optionsaccuracy, normalized

Both POISE and GHOST are calibrated exclusively on WikiText-2, without access to inputs from these downstream tasks. The table reports accuracy (%) for Mamba-2 1.3B, with length-normalized accuracy in parentheses where applicable. The mean is computed over the six unnormalized accuracies.

LAMBADAPIQAARC-eARC-cWinoGrandeHellaSwagmean
uncompressed65.5573.34 (73.78)64.23 (60.35)30.03 (33.36)61.2545.87 (59.95)56.71
POISE, budget 0.363.0573.07 (73.67)63.80 (59.55)29.44 (32.59)59.3545.26 (59.15)55.66
GHOST, budget 0.337.0171.98 (73.07)62.63 (56.82)27.39 (30.46)56.0442.57 (55.64)49.60
POISE, budget 0.147.5672.25 (72.69)63.01 (58.08)28.50 (30.63)56.6743.23 (56.92)51.87
GHOST, budget 0.117.1069.80 (71.38)59.01 (51.98)24.83 (28.33)51.0738.36 (49.32)43.36

POISE outperforms GHOST on every task and metric. At budget 0.3, its mean accuracy is only 1.05 percentage points below the uncompressed model. On LAMBADA, the drop is 2.50 points, compared with 28.54 for GHOST. At budget 0.1, POISE retains a mean accuracy of 51.87%, versus 43.36% for GHOST, and a LAMBADA accuracy of 47.56%, versus 17.10%.

Dear reviewers,

Thank you again for reviewing our paper. We believe we have addressed all your questions and concerns by providing additional experiments demonstrating the strong advantages of our algorithms and clarifying our theoretical contributions. We would appreciate if you could let us know what you think!

Many thanks!

The authors

AuthorsOpenReview ↗

Closing comment

We thank the reviewers and the AC for this productive discussion. As it nears its end, we give a short summary. Three of the four reviewers recommend acceptance. Of the three who raised concerns, two responded positively to our rebuttal, and the third has not yet responded.

  • Reviewer hsf5 recommended clear acceptance and raised no weaknesses or questions.

  • Reviewer VoKR confirmed that the new results resolve the main concerns of their review, and maintained their positive view of the paper: robustness of the allocation to the probe level, generalization beyond the calibration distribution, and real speed-ups at practical batch sizes. As they suggest, the revision will include these results, including the small slowdown at batch size 1.

  • Reviewer 7JzZ found that the new comparison with classical balanced truncation clearly demonstrates POISE’s performance, and increased their score. They suggested one addition, classical BT on a stable S5 checkpoint for ListOps, which we will include in the revision along with the changes planned in the rebuttal, which they also encourage.

  • Reviewer tG5b has not yet responded. Their concerns on measured inference efficiency on Mamba-2 and on evaluation beyond WikiText-2 (W1, W2) were shared with Reviewer VoKR, who confirmed that the new experiments resolve them. Their remaining points are answered in our response to them: a comparison with GHOST at matched latency, a direct measurement of the moment-factorization error, the accuracy of the allocation surrogate on Mamba-2, and the LRA-Text checkpoint protocol. We believe this addresses all their concerns.

All new results and clarifications will be included in the revision, and we thank the reviewers for helping strengthen the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.