acceptodds
Withdrawn from ICLR 2027

Training-Aware, Calibration-Free Model Reduction for Selective State Space Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modeling long sequences, such as text and audio, requires computationally efficient architectures; Mamba addresses this need using selective state-space models (SSMs). However, two structural dimensions intrinsic to these selective SSMs—the state order and the step-size scheduling dimension—directly affect per-step cost and parameter count. We introduce a training-aware framework that uses structural regularization during full-order training to enable joint reduction of both dimensions. A per-layer finite-horizon output-error bound motivates a scale-invariant, decay-aware state score and a factorized low-rank surrogate for the step-size map. After training, we perform one-shot reduction using state selection and truncated SVD to construct a model with the requested dimensions from the learned parameters alone, without calibration data or post-compression fine-tuning. On Selective Copying, SC09, and The Pile, the jointly reduced models outperform vanilla post-hoc reduction in reported point estimates at matched reduction levels. In isolated state reduction, our calibration-free method outperforms the adapted calibration-based baselines on Selective Copying and SC09 and remains competitive on The Pile.

Primary area:
learning on time series and dynamical systems
Keywords:
selective state-space models, Mamba, model compression, model order reduction, low-rank approximation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uELNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a model reduction method for selective state space models (SSMs) like Mamba, aiming to jointly reduce the state size and low-rank size within the SSM core. The approach first trains a model with loss regularization for both state and low-rank dimensions, creating a distinct variance in output contribution across dimensions to make them easier to reduce. Once trained, dimensions are retained based on a state importance score and singular value decomposition (SVD). Compared to baselines like GHOST, the proposed method operates in a calibration-free manner on pre-trained checkpoints, requiring no additional calculation process once the retention rate is specified. The proposed method is evaluated across synthetic (Selective Copying), audio (SC09), and text (Pile) datasets, demonstrating that the combination of regularization, state importance score-based reduction, and SVD-based reduction achieves the smallest compression loss at the same retention rates.

Critical strengths
  1. Empirical validation was conducted across a wide range of environments, spanning synthetic tasks, audio modality, and text modality.
  2. The comparison against training a smaller model from scratch (rather than reducing a pre-trained model) directly addresses a natural question regarding the approach. The authors transparently note that training a small model from scratch can be competitive or superior, which provides a clear positioning that these reduction methods are actually useful when reusing pre-trained checkpoints.
Critical weaknesses
  1. The novelty is limited. The individual components of the proposed method, such as regularization, state importance score, and SVD-based reduction, are largely drawn from prior work. Additionally, the evaluation of joint regularization is primarily benchmarked against random reduction, making it difficult to substantiate claims that the proposed method is uniquely tailored or superior for joint reduction.
  2. It is questionable whether model reduction for pre-trained checkpoints is a major practical bottleneck. Since training a smaller model from scratch often yields significantly better performance, the practical benefits of reusing pre-trained checkpoints via model reduction may be negligible due to performance degradation, especially given that training a new compact model from scratch is a one-time cost.
Questions for authors
  1. Even at aggressive retention rates, the actual speedup and memory allocation reduction appear small (e.g., 1.015x speedup and 5.39% memory reduction for Pile decode). Given that Mamba is already a highly efficient backbone architecture, what are the absolute latency and memory figures for the full versus reduced models, and what are the absolute practical efficiency gains?
  2. The current experiments lack comparisons combining alternative state reduction and rank reduction methods with joint regularization. Without these, the work risks appearing as a straightforward combination and simple extension of existing techniques. Please clarify which specific component of the proposed method is explicitly tailored for joint reduction.
Reviewer_9sG8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Training-aware compression of Mamba-1 by jointly reducing state order (N) and step-size rank (R), using a finite-horizon error bound, structural regularization, state selection, and truncated SVD.

Critical strengths
  • Flexible compression: Multiple target dimensions from one trained checkpoint, without calibration or post-reduction optimization.

  • Reasonable evaluation: Isolated reductions, random controls, direct-small baselines, and measured inference efficiency.

Critical weaknesses
  • Novelty Concern: How is it different from related work like AIRE-Prune and LAST?

  • Missing state-only versus rank-only versus joint reduction ablation on the same jointly regularized checkpoint.

  • Limited Gains: At 50% joint retention on The Pile: only 2.70% fewer parameters and 1.015× decoding speedup

  • Theorem 1 provides only a single-layer guarantee; quadratic horizon dependence may yield loose bounds without demonstrated empirical tightness.

Questions for authors

Q1: How does the proposed state score compare with AIRE-Prune on identical Mamba-1 checkpoints and retention budgets?

Q2: Can state-only, rank-only, and joint compression be evaluated from the same jointly regularized checkpoint to establish the additional benefit of reducing (R)?

Q3: Why do 18/40 SC09 blocks have zero step-size maps after rank regularization? Is input-dependent scheduling being suppressed rather than compressed?

Q4: How well do the theoretical bound terms correlate with measured output errors, particularly for long sequences where (\Phi_R) scales quadratically with (L)?

Reviewer_qYtcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper extends model order reduction to the selective sub sequence pipeline of language models. This is a setting not covered by standard pruning/quantization. It describes a training aware, structure preserving reduction framework for the selective sub sequence processing components that appear in modern language models. It introduces a formula for the join reduction of the selective order (the number of dynamical channels) and the step size scheduling dimension (the dimension that controls input dependent step sizes). The learned parameters are use to develop a reduced sub sequence model. The reduction is calibration free at inference time and requires no positive reduction tuning. An out of TPU error analysis separates an input dependent factor from model dependent structural terms. This guides the design of two structure inducing surrogates, a scale invariant coupling and dynamics regularizer for reduced order reduction, and factorized low rank surrogates for the step size map. The method is evaluated empirically and shows improved performance.

Critical strengths
  • Mathematical analysis with an explicit error bound and a proof sketch.

  • Training aware surrogate losses that shape the learned parameters

  • Calibration free reduction at inference that avoids the need for positive reduction tuning.

  • Experiments with efficiency gains (speed up and memory savings) on two distinct tasks.

  • Joint treatment of order and step size dimensions

Critical weaknesses
  • The method is designed for the selective sub sequence processing of a particular language model architecture; applicability to other models is not considered.

  • The assumptions of zero initial states and negative diagonal elements may limit the generality of the error bound.

  • Performance on non selective tasks (classification, reasoning, etc.) is not assessed.

  • The manuscript is hard to read

Questions for authors
  • Can the learned reduction structure be transferred to other language model families or to non selective sequence processing components?

  • Are the error bound assumptions (zero initial states, negative diagonal entries) necessary, or can the bound be extended to more general settings?

  • What are the quantitative trade offs of the proposed calibration free reduction versus pruning, quantization, or low rank factorization on accuracy, speed, and memory?

  • What is the computational cost of the training aware framework during full order training, and how does it compare to the inference time savings?

Does the method preserve performance on tasks that do not rely on selective copying (e.g., classification, reasoning, multi modal tasks)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.