- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a model reduction method for selective state space models (SSMs) like Mamba, aiming to jointly reduce the state size and low-rank size within the SSM core. The approach first trains a model with loss regularization for both state and low-rank dimensions, creating a distinct variance in output contribution across dimensions to make them easier to reduce. Once trained, dimensions are retained based on a state importance score and singular value decomposition (SVD). Compared to baselines like GHOST, the proposed method operates in a calibration-free manner on pre-trained checkpoints, requiring no additional calculation process once the retention rate is specified. The proposed method is evaluated across synthetic (Selective Copying), audio (SC09), and text (Pile) datasets, demonstrating that the combination of regularization, state importance score-based reduction, and SVD-based reduction achieves the smallest compression loss at the same retention rates.
Critical strengths
- Empirical validation was conducted across a wide range of environments, spanning synthetic tasks, audio modality, and text modality.
- The comparison against training a smaller model from scratch (rather than reducing a pre-trained model) directly addresses a natural question regarding the approach. The authors transparently note that training a small model from scratch can be competitive or superior, which provides a clear positioning that these reduction methods are actually useful when reusing pre-trained checkpoints.
Critical weaknesses
- The novelty is limited. The individual components of the proposed method, such as regularization, state importance score, and SVD-based reduction, are largely drawn from prior work. Additionally, the evaluation of joint regularization is primarily benchmarked against random reduction, making it difficult to substantiate claims that the proposed method is uniquely tailored or superior for joint reduction.
- It is questionable whether model reduction for pre-trained checkpoints is a major practical bottleneck. Since training a smaller model from scratch often yields significantly better performance, the practical benefits of reusing pre-trained checkpoints via model reduction may be negligible due to performance degradation, especially given that training a new compact model from scratch is a one-time cost.
Questions for authors
- Even at aggressive retention rates, the actual speedup and memory allocation reduction appear small (e.g., 1.015x speedup and 5.39% memory reduction for Pile decode). Given that Mamba is already a highly efficient backbone architecture, what are the absolute latency and memory figures for the full versus reduced models, and what are the absolute practical efficiency gains?
- The current experiments lack comparisons combining alternative state reduction and rank reduction methods with joint regularization. Without these, the work risks appearing as a straightforward combination and simple extension of existing techniques. Please clarify which specific component of the proposed method is explicitly tailored for joint reduction.