- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReBound, a training-free method for reducing prefill computation in Mamba language models. The method selects a pruning layer using held-out KL divergence, processes the full sequence through that layer, and removes low-impact token representations before subsequent layers. Token importance is estimated by combining the selective timescale parameter (Δ), selective-update energy, and proximity to the final autoregressive readout. The score uses separately normalized terms in log space. The evaluation covers Mamba-1 1.4B and 2.8B, along with Mamba-2 2.7B, using language-model benchmarks, WikiText-103 continuation perplexity, LongBench, component ablations, and H100 measurements. The results suggest a useful trade-off between computational efficiency and prediction quality, although the comparative evidence requires clarification.
Critical strengths
-
ReBound presents a reasonable readout-aware pruning strategy for causal Mamba models. The seven-way ablation in Table 2 supports its design, with the complete Δ+E+P score increasing perplexity by only 0.69%, compared with 2.52% for Δ alone. The results suggest that combining selective-update energy and readout proximity is more effective than relying solely on local timescale information.
-
The calibration procedure uses held-out KL divergence without downstream task labels. Figure 4 shows relatively stable safe pruning regions across different pruning ratios and calibration sets, with the selected boundary changing from L44 on Mamba-1 to L52 on Mamba-2. The reported 1.15× H100 speedup further supports its practical value, although it is measured at a different operating point from the main accuracy results.
Critical weaknesses
-
Table 1 raises concerns about baseline comparability. The uncompressed Mamba-1 1.4B results match Gu and Dao (2024), and the UTRC results match Zhan et al. (2024), while the uncompressed 2.8B baseline differs. Using UTRC's published baseline gives a 2.50-point accuracy reduction rather than the reported 2.38 points. Furthermore, DTP performs worse than random pruning on both models, raising questions about its adaptation from multimodal Mamba. These discrepancies require clarification, particularly given ReBound's reported 0.62-point accuracy reduction.
-
The theoretical analysis motivates the proposed score but does not fully justify its implementation. Equation (32) analyzes errors in the layer following pruning, while Algorithm 2 uses statistics from the selected layer. The independently normalized score terms also require justification, and Equation (21) appears inconsistent with the skip-adjusted energy definition in Section 3.3. Furthermore, PassageRetrieval-en baseline accuracies are only 0.115 and 0.065, making it difficult to assess whether pruning preserves meaningful long-context retrieval capability.
Questions for authors
-
Were the uncompressed and UTRC baselines evaluated using the same harness, few-shot settings, and FLOP accounting, and what explains the different Mamba-1 2.8B baseline values?
-
What exactly does "DTP (Δ)" implement for text-only prefill, and why does it perform worse than random pruning on both models?
-
How do the selected-layer statistics and independently normalized score components follow from the theoretical bound, and how should Equation (21) be reconciled with Section 3.3?
-
Since proximity-only pruning effectively retains recent tokens, how does it compare with a keep-last-K baseline, and what pruning ratio and FLOP reduction were used for the Table 2 ablation?
-
Can the authors provide more informative long-context retrieval results and repeated H100 timing measurements at the same FLOP reduction used for the main accuracy comparison?
5 comments
Q5 — Informative Retrieval and Repeated H100 Timing
Additional RULER S-NIAH-1/2/3 retrieval. We evaluated Mamba-2 2.7B using 300 paired prompts per row. Unpruned accuracy is 82.7% across the selected retrieval settings.
| Layer | ReBound | Keep-last-K | Difference (95% CI) |
|---|---|---|---|
| L20 | 83.0% | 72.0% | +11.0 [6.3, 15.7] pp |
| L32 | 85.0% | 78.3% | +6.7 [3.0, 10.3] pp |
| L44 | 81.3% | 76.0% | +5.3 [2.0, 9.0] pp |
| L52 (calibrated) | 82.0% | 77.1% | −4.9 [1.9, 8.3] pp |
At the calibrated boundary, ReBound preserves retrieval accuracy close to the unpruned baseline. Earlier layers also reveal the benefit of score-aware selection over retaining only recent tokens.
Additional H100 prefill profiling (~15% FLOP reduction). We repeated measurements on one NVIDIA H100 80GB GPU using FP16, with 5 warm-ups and 30 interleaved timed runs. Timing includes decoding-state allocation and initialization, score computation, CPU token selection, device synchronization, and gathering. Token counts are scan-aligned, with 14.8% prefill FLOP reduction.
| Batch | Tokens | Unpruned (ms) | ReBound (ms) | Speedup |
|---|---|---|---|---|
| 1 | 1,024 | 42.5 | 44.1 | 0.969× |
| 1 | 4,096 | 73.6 | 67.4 | 1.110× |
| 8 | 1,024 | 145.1 | 127.4 | 1.147× |
| 8 | 2,048 | 284.8 | 246.8 | 1.155× |
| 8 | 4,096 | 568.3 | 490.5 | 1.160× |
| 32 | 1,024 | 563.2 | 486.0 | 1.160× |
| 32 | 2,048 | 1,140.8 | 979.3 | 1.163× |
At approximately 15% prefill FLOP reduction, ReBound achieves 1.11–1.16× speedup in supported configurations, consistent with the submitted paper's approximately 1.15× result. The shortest batch-one configuration remains slower due to kernel-launch overhead. These results confirm that practical speedup depends on sequence length, batch size, and selective-scan kernel alignment.
We thank the reviewer for his valuable feedback. We will incorporate these additional experiments and profiling results into the revised appendix. Hope we have addressed your concerns. We would appreciate any further feedback on whether these points and the requested experiments sufficiently resolve the open issues. We are happy to clarify anything further.
Q1 — Baseline comparability
Original Table 1 mixed sources: the Mamba-1 1.4B uncompressed row came from Gu and Dao; UTRC rows from Zhan et al.; the 2.8B uncompressed row from our lm-eval run. Therefore the original harness, few-shot and FLOP configurations cannot be asserted to match. The 2.8B baseline difference (68.62 vs. 68.74) changes the UTRC drop to 2.50 points when its own baseline is used.
Additional matched reruns: lm-eval 0.4.11, zero-shot, FP16, identical task splits, answer protection and ~10% prefill-FLOP budget.
These new comparisons supplement the submitted table; the originally published-source UTRC rows remain identified as such.
Q2 — DTP two-stage versus Δ-only and random
These are not the same experiment. DTP prunes at two stages; our Δ-only ablation changes only the token-ranking score while keeping the ReBound boundary. DTP score sees degradation due to early layer pruning. Making pruning layer selection critical.
Q3 — Layer Statistics, Normalization, and Eq. (21)
Pruning-layer statistics. Scores are computed at the full-length calibrated layer before pruning. Next-layer statistics would require additional computation. We evaluated this approximation on Mamba-1 2.8B:
Independent normalization. The bound motivates additive log-space scoring; z-normalization balances component scales but is not derived from the bound. Additional results on 128 WikiText-103 windows (512 tokens, 30% pruning) show lower final-readout KL:
Equation (21). Section 3.3 defines selective-update energy as:
Et=∥qt−D⊙xt∥2=∥Ctht∥2.
Here, qt is the pre-gate selective-scan output, xt its post-convolution input, and D the direct skip parameter. This removes the skip contribution, unlike the full-output norm in Eq. (21). We will correct this inconsistency.
The theoretical missing-update term motivates Et, but does not establish an exact end-to-end error bound. Pruning safety is evaluated through final-readout KL calibration.
Q4 — P-only versus keep-last-K; ablation budget
P-only is exactly keep-last-K under the same token eligibility rules: Δˉk>0 implies τt=∑k>tΔˉk strictly decreases with position. Verified identical keep sets in 20/20 windows.
Submitted ablation (Table 2):
LongBench comparisons match layer, retained-token count and 384-token protected tail. Thus the submitted ablation advantage is reported as specific to its setting, not as a universal advantage over recency.