acceptodds
Under review as a conference paper at ICLR 2027

ReBound: Readout-Aware Boundary Pruning for Efficient Mamba Prefill

Abstract

Mamba language models make autoregressive decoding efficient, but prefill still processes every prompt token through every layer. We propose \methodname, a training-free readout-aware boundary pruning method that runs the selected Mamba layer at full length and then removes low-impact boundary representations before downstream layers. Its score combines channel-mean , selective-update energy, and proximity to the final autoregressive readout, while a calibration-only KL sweep selects the earliest behaviorally safe pruning layer. At 10% FLOP reduction on Mamba-1 2.8B, \methodname reaches 68.0% average lm-evaluation-harness accuracy, only 0.62 points below baseline, and raises WikiText-103 continuation perplexity by only 0.61%. A complete seven-way score ablation shows that the full score is best, and calibration generalizes across architectures, with sharp safe-layer transitions at L44 for Mamba-1 2.8B and L52 for Mamba-2 2.7B. On five LongBench tasks, \methodname preserves long-context quality substantially better than -only DTP on both architectures. Measured hardware performance realizes these savings: at L44 with 50% token keep, latency improves by and throughput by , close to the FLOP-based ceiling. These results show that Mamba token compression should preserve the final readout rather than rely on local token importance alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mamba, SSM, State Space, Pruning, Efficient, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y7YxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReBound, a training-free method for reducing prefill computation in Mamba language models. The method selects a pruning layer using held-out KL divergence, processes the full sequence through that layer, and removes low-impact token representations before subsequent layers. Token importance is estimated by combining the selective timescale parameter (Δ), selective-update energy, and proximity to the final autoregressive readout. The score uses separately normalized terms in log space. The evaluation covers Mamba-1 1.4B and 2.8B, along with Mamba-2 2.7B, using language-model benchmarks, WikiText-103 continuation perplexity, LongBench, component ablations, and H100 measurements. The results suggest a useful trade-off between computational efficiency and prediction quality, although the comparative evidence requires clarification.

Critical strengths
  1. ReBound presents a reasonable readout-aware pruning strategy for causal Mamba models. The seven-way ablation in Table 2 supports its design, with the complete Δ+E+P score increasing perplexity by only 0.69%, compared with 2.52% for Δ alone. The results suggest that combining selective-update energy and readout proximity is more effective than relying solely on local timescale information.

  2. The calibration procedure uses held-out KL divergence without downstream task labels. Figure 4 shows relatively stable safe pruning regions across different pruning ratios and calibration sets, with the selected boundary changing from L44 on Mamba-1 to L52 on Mamba-2. The reported 1.15× H100 speedup further supports its practical value, although it is measured at a different operating point from the main accuracy results.

Critical weaknesses
  1. Table 1 raises concerns about baseline comparability. The uncompressed Mamba-1 1.4B results match Gu and Dao (2024), and the UTRC results match Zhan et al. (2024), while the uncompressed 2.8B baseline differs. Using UTRC's published baseline gives a 2.50-point accuracy reduction rather than the reported 2.38 points. Furthermore, DTP performs worse than random pruning on both models, raising questions about its adaptation from multimodal Mamba. These discrepancies require clarification, particularly given ReBound's reported 0.62-point accuracy reduction.

  2. The theoretical analysis motivates the proposed score but does not fully justify its implementation. Equation (32) analyzes errors in the layer following pruning, while Algorithm 2 uses statistics from the selected layer. The independently normalized score terms also require justification, and Equation (21) appears inconsistent with the skip-adjusted energy definition in Section 3.3. Furthermore, PassageRetrieval-en baseline accuracies are only 0.115 and 0.065, making it difficult to assess whether pruning preserves meaningful long-context retrieval capability.

Questions for authors
  1. Were the uncompressed and UTRC baselines evaluated using the same harness, few-shot settings, and FLOP accounting, and what explains the different Mamba-1 2.8B baseline values?

  2. What exactly does "DTP (Δ)" implement for text-only prefill, and why does it perform worse than random pruning on both models?

  3. How do the selected-layer statistics and independently normalized score components follow from the theoretical bound, and how should Equation (21) be reconciled with Section 3.3?

  4. Since proximity-only pruning effectively retains recent tokens, how does it compare with a keep-last-K baseline, and what pruning ratio and FLOP reduction were used for the Table 2 ablation?

  5. Can the authors provide more informative long-context retrieval results and repeated H100 timing measurements at the same FLOP reduction used for the main accuracy comparison?

5 comments

Q1 — Baseline comparability

Original Table 1 mixed sources: the Mamba-1 1.4B uncompressed row came from Gu and Dao; UTRC rows from Zhan et al.; the 2.8B uncompressed row from our lm-eval run. Therefore the original harness, few-shot and FLOP configurations cannot be asserted to match. The 2.8B baseline difference (68.62 vs. 68.74) changes the UTRC drop to 2.50 points when its own baseline is used.

Additional matched reruns: lm-eval 0.4.11, zero-shot, FP16, identical task splits, answer protection and ~10% prefill-FLOP budget.

ModelUncompressedReBoundUTRCTwo-stage DTP
Mamba-1 1.4B64.9964.7962.3160.49
Mamba-1 2.8B68.6768.2566.3865.80

These new comparisons supplement the submitted table; the originally published-source UTRC rows remain identified as such.

Q2 — DTP two-stage versus Δ\Delta-only and random

ComparisonLayer scheduleWhat it tests
Submitted Table 1 DTPTwo-stage: L15/L45 (2.8B)Effect of the multi-stage Δ\Delta-guided pruning strategy
Δ\Delta-only score ablation (ours)One pruning boundary, ReBound's calibrated layerEffect of replacing ReBound's score with Δ\Delta alone at a matched budget
Random score ablationSame calibrated boundary and budgetRandom-token selection control

These are not the same experiment. DTP prunes at two stages; our Δ\Delta-only ablation changes only the token-ranking score while keeping the ReBound boundary. DTP score sees degradation due to early layer pruning. Making pruning layer selection critical.

Q3 — Layer Statistics, Normalization, and Eq. (21)

Pruning-layer statistics. Scores are computed at the full-length calibrated layer before pruning. Next-layer statistics would require additional computation. We evaluated this approximation on Mamba-1 2.8B:

L44 → L45 agreementResult
Full-score Spearman correlation0.72
Keep-set overlap (30% pruning)87%
Mean Δ correlation0.32
Energy-proxy correlation0.33

Independent normalization. The bound motivates additive log-space scoring; z-normalization balances component scales but is not derived from the bound. Additional results on 128 WikiText-103 windows (512 tokens, 30% pruning) show lower final-readout KL:

LayerZ-normalizedUnit-weight
L200.0550.328
L320.0400.046
L440.00220.0185

Equation (21). Section 3.3 defines selective-update energy as:

Et=∥qt−D⊙xt∥2=∥Ctht∥2.E_t = \|q_t-D\odot x_t\|_2 = \|C_th_t\|_2.

Here, qtq_t is the pre-gate selective-scan output, xtx_t its post-convolution input, and DD the direct skip parameter. This removes the skip contribution, unlike the full-output norm in Eq. (21). We will correct this inconsistency.

The theoretical missing-update term motivates EtE_t, but does not establish an exact end-to-end error bound. Pruning safety is evaluated through final-readout KL calibration.

Q4 — P-only versus keep-last-K; ablation budget

P-only is exactly keep-last-K under the same token eligibility rules: Δˉk>0\bar\Delta_k>0 implies τt=∑k>tΔˉk\tau_t=\sum_{k>t}\bar\Delta_k strictly decreases with position. Verified identical keep sets in 20/20 windows.

Submitted ablation (Table 2):

SettingValue
Model / pruning layerMamba-1 2.8B / L44
Windows80 WikiText-103 windows
Context + continuation1,792 + 256 tokens
Context removed896 tokens = 50% of context
Idealized budget14.8% context-prefill FLOPs, or 13.0% full-window forward FLOPs
P-only PPL increase+2.10%
Δ+E+P\Delta+E+P PPL increase+0.69%

LongBench comparisons match layer, retained-token count and 384-token protected tail. Thus the submitted ablation advantage is reported as specific to its setting, not as a universal advantage over recency.

Q5 — Informative Retrieval and Repeated H100 Timing

Additional RULER S-NIAH-1/2/3 retrieval. We evaluated Mamba-2 2.7B using 300 paired prompts per row. Unpruned accuracy is 82.7% across the selected retrieval settings.

LayerReBoundKeep-last-KDifference (95% CI)
L2083.0%72.0%+11.0 [6.3, 15.7] pp
L3285.0%78.3%+6.7 [3.0, 10.3] pp
L4481.3%76.0%+5.3 [2.0, 9.0] pp
L52 (calibrated)82.0%77.1%−4.9 [1.9, 8.3] pp

At the calibrated boundary, ReBound preserves retrieval accuracy close to the unpruned baseline. Earlier layers also reveal the benefit of score-aware selection over retaining only recent tokens.

Additional H100 prefill profiling (~15% FLOP reduction). We repeated measurements on one NVIDIA H100 80GB GPU using FP16, with 5 warm-ups and 30 interleaved timed runs. Timing includes decoding-state allocation and initialization, score computation, CPU token selection, device synchronization, and gathering. Token counts are scan-aligned, with 14.8% prefill FLOP reduction.

BatchTokensUnpruned (ms)ReBound (ms)Speedup
11,02442.544.10.969×
14,09673.667.41.110×
81,024145.1127.41.147×
82,048284.8246.81.155×
84,096568.3490.51.160×
321,024563.2486.01.160×
322,0481,140.8979.31.163×

At approximately 15% prefill FLOP reduction, ReBound achieves 1.11–1.16× speedup in supported configurations, consistent with the submitted paper's approximately 1.15× result. The shortest batch-one configuration remains slower due to kernel-launch overhead. These results confirm that practical speedup depends on sequence length, batch size, and selective-scan kernel alignment.

We thank the reviewer for his valuable feedback. We will incorporate these additional experiments and profiling results into the revised appendix. Hope we have addressed your concerns. We would appreciate any further feedback on whether these points and the requested experiments sufficiently resolve the open issues. We are happy to clarify anything further.

Reviewer_vdGrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ReBound is a training-free method for reducing Mamba prefill cost. It runs the full prompt through a chosen layer, then removes some token representations before the remaining layers. The last prompt token is always kept. The score combines the average Δ across channels, the SSM output magnitude after subtracting its skip term, and proximity to the final readout. The pruning layer is chosen using the next-token KL divergence on a held-out set. The tests cover Mamba-1 1.4B and 2.8B, with some results also shown for Mamba-2 2.7B. On Mamba-1 2.8B, ReBound reduces FLOPs by 10% while average accuracy drops from 68.62% to 68.00%. WikiText-103 continuation perplexity rises by 0.61% under the same FLOP budget.

Critical strengths
  1. The method is simple and has practical value. The model processes the full prompt before shortening the sequence for later layers. It keeps the final token and preserves the order of the others. There is no retraining, and layer selection does not use downstream task labels.
  2. The paper tests multiple models and tasks, and includes all seven combinations of the three score terms. At the tested ablation setting, the full score increases perplexity by 0.69%, compared with 2.10% for P alone. This supports combining the terms rather than using proximity alone. The H100 results are also useful because they show a measured speedup, not just lower FLOPs.
Critical weaknesses
  1. The cross-layer proof does not yet support the proposed score. The single-layer result in Proposition 1 makes sense under its assumptions. I do not think the extension to the full model works as written. In Eq. (27), later layers only amplify errors from the previous layer. But they also receive a shorter sequence, so missing positions can introduce new errors at each layer. Ordinary Lipschitz continuity for equal-length inputs does not justify ignoring this effect. Running a layer and then gathering tokens is generally different from gathering first and then running the layer. There is another gap between Eqs. (30) and (32). Eq. (30) needs the largest error across all retained positions, but Eq. (32) uses the error at the final position. These are not interchangeable. Errors at earlier positions can still affect the final output through later layers. The bound also uses quantities from layer ℓ⋆+1\ell^\star+1, while the implemented score uses layer ℓ⋆\ell^\star. The paper needs to explain why these are good substitutes. The score may still be a useful heuristic, but the current derivation does not establish the claimed link.
  2. Some key comparisons are hard to judge because the settings are unclear. Table 2 does not state the pruning layer or budget used for the ablation. It reports a 2.52% perplexity increase for Δ-only, while the main 10%-FLOP experiment reports 7.50% for DTP. This could simply reflect different settings, but those settings need to be stated. Because Δ is positive, P-only favors later token representations in exact arithmetic. The paper tests this baseline on continuation perplexity, but not on the main harness tasks or LongBench. I would like to know how much the full score adds over this simple baseline once the pruning layer is fixed. The paper also omits the actual context and continuation lengths and evaluation sample counts. The speed measurements need more detail, especially whether they include state initialization and the full pruning cost. Without these details, I cannot tell how easy it would be to reproduce the quality and speed results.
Questions for authors
  1. How does Eq. (27) apply when the full and pruned sequences have different lengths? Please explain the step from Eq. (30) to Eq. (32). Why should signals from the pruning layer approximate the next-layer quantities used in the bound?
  2. What pruning ratio was used for calibration, and what layer was selected for each model? In Figure 4(a), the 5%-drop curve seems to fall below 0.01 before L44. How does this fit the claim that the earliest safe layer is independent of pruning ratio? Please also show the Mamba-2 calibration results supporting L52.
  3. Were the UTRC results in Table 1 rerun, or taken from Zhan et al. (2024)? They match that paper’s Table 2, including perplexities of 11.46 and 9.53, but the 2.8B unpruned baseline differs. Please clarify whether the checkpoints and evaluation settings match, and how DTP was adapted. How does P-only compare on the harness and LongBench tasks?
  4. Please give the ablation layer and pruning budget, the context and continuation lengths, and the evaluation sample counts. Please also provide the LongBench truncation and generation settings and uncertainty estimates. Which tensors are used to compute E, and what values and numerical handling are used for α and ε?
  5. Do the timings include recurrent and convolution state initialization, score computation, token selection, gathering, and synchronization? Please give absolute timings, precision, the kernels used, and the number of repeats. Also, does L44 use zero-based indexing? Having 19 remaining layers in a 64-layer model fits zero-based indexing, but the mathematical notation starts from 1.
5 comments

We thank the reviewer for thier valuable feedback.

Q1 / Weakness 1 — Cross-layer analysis

Our central contribution separates two complementary problems: (i) identifying a behaviorally safe pruning boundary, and (ii) selecting which token representations to remove at that boundary. The former is addressed through held-out final-readout KL calibration, which directly evaluates the complete model after pruning, including downstream effects. The latter is addressed through a lightweight readout-aware scoring heuristic motivated by the selective-SSM perturbation mechanism characterized in Proposition 1. We do not rely on a tight end-to-end theoretical error bound to determine the safe boundary, nor do we claim that the implemented score is an exact bound on final-readout perturbation.

1. Equation (27): Different-length sequences and downstream errors.

ReBound executes the selected layer ℓ⋆\ell^\star at full length and only then gathers the retained representations. Thus, the retained representations are exact at the pruning boundary, and the first deletion-induced discrepancy appears in layer ℓ⋆+1\ell^\star+1, precisely the setting analyzed by Proposition 1.

The cross-layer propagation statement can be clarified to explicitly account for the additional discrepancy from processing a shortened sequence in each downstream layer. Let RSR_S denote gathering on the retained index set, and define the aligned error em=∥RSYm−Y~m∥∞e_m=\|R_SY^m-\widetilde Y^m\|_\infty. Adding and subtracting Fm(RSYm−1)F_m(R_SY^{m-1}) gives

em≤Γmem−1+dm,e_m\leq \Gamma_m e_{m-1}+d_m,

where

dm=∥RSFm(Ym−1)−Fm(RSYm−1)∥∞.d_m=\|R_SF_m(Y^{m-1})-F_m(R_SY^{m-1})\|_\infty.

The Lipschitz term compares equal-length inputs, while dmd_m captures the effect of missing positions, including the noncommutativity of gathering and applying a Mamba block. The resulting cross-layer error decomposition is

eL≤∑m=ℓ⋆+1L(∏k=m+1LΓk)dm.e_L\leq\sum_{m=\ell^\star+1}^{L}\left(\prod_{k=m+1}^{L}\Gamma_k\right)d_m.

This makes explicit the additional downstream contributions not represented in the simplified presentation of Eq. (27).

2. Equations (30)–(32): Maximum error versus final-position error.

The reviewer correctly distinguishes the maximum error over retained positions from the final-position error. Proposition 1 provides a single-layer, position-dependent attenuation result:

∥et∥≤exp⁡(−∑k=j+1tνk)∥ξj∥.\|e_t\|\leq\exp\left(-\sum_{k=j+1}^{t}\nu_k\right)\|\xi_j\|.

For t=Tt=T, this characterizes how a missing state update is attenuated toward the final readout within the analyzed layer. This final-position expression cannot directly replace the supremum-norm quantity required by Eq. (30). We will clarify this distinction in Appendix B. The local attenuation expression motivates incorporating readout proximity into the scoring heuristic; we do not rely on Eq. (32) as a rigorous end-to-end upper bound on the complete multi-layer model.

Pruning-layer proxy: The decision is made after layer ℓ⋆\ell^\star runs at full length, before ℓ⋆+1\ell^\star+1. Thus we use available ℓ⋆\ell^\star statistics rather than claim that they equal next-layer sensitivities.

Additional neighboring-layer check, 30% dropResult
L44 →\rightarrow L45, full-score Spearman correlationρ=0.72\rho=0.72
L44 →\rightarrow L45, keep-set overlap87%
L44 →\rightarrow L45, individual Δˉ\bar\Delta / KK correlations0.32 / 0.33

The complete-model KL calibration evaluates downstream effects directly. Submitted Figure 4(b) provides a matched-ratio score comparison; submitted Table 2 reports +0.69% continuation-PPL increase for Δ+E+P\Delta+E+P, versus +2.52% for Δ\Delta alone and +2.10% for PP alone.

Q2 — Calibration ratio and selected layers

Calibration removes 30% of eligible non-final tokens on 256 WikiText-103 passages of 128 tokens, with δ=0.01\delta=0.01.

ModelSelected layerCriterion
Mamba-1 1.4BL37Earliest sustained low-KL boundary
Mamba-1 2.8BL44Earliest sustained low-KL boundary
Mamba-2 2.7BL52Earliest sustained low-KL boundary

Figure 4(a): At 5% drop, KL first crosses 0.01 at L20 (0.0081) but rebounds at L21 (0.042). This is not a sustained safe region. We will correct Eq. (3.1) and Algorithm 1 to specify. Even in figure 4(a) we can clearly see the dip in L44. Which is consistent across all pruning ratio.

ℓ⋆=min⁡{ℓ:KLk≤δ for all k≥ℓ}.\ell^\star=\min\{\ell:\mathrm{KL}_k\leq\delta\ \text{for all}\ k\geq\ell\}.

Token drop ratio5%10%30%40%50%60%
First KL crossingL44L44L44L44L44L45
Sustained safe boundaryL44L44L44L44L44L45

Thus our claim is approximate stability over 5–50%, not strict ratio independence. An additional Mamba-2 sweep supports the sustained transition at L52; its KL-versus-layer plot has been added to the updated paper.

Q3 — UTRC, DTP adaptation and P-only

UTRC provenance. The UTRC rows in submitted Table 1 come from Zhan et al. (2024), not our reruns. Their exact harness, few-shot configuration and FLOP accounting cannot be confirmed as identical. The submitted 2.8B baseline is 68.62%, versus 68.74% in Zhan et al.; using their baseline gives a 2.50-point, rather than 2.38-point, UTRC drop. The submitted Table 1 DTP row uses two-stage pruning; the Δ\Delta-only experiment is a separate ReBound score ablation at the calibrated boundary.

Additional matched evaluation (lm-eval 0.4.11; zero-shot; FP16; identical splits; protected answer and last five context tokens; ~10% prefill FLOPs):

ModelUncompressedReBoundUTRCTwo-stage DTPΔ\Delta-onlyRandomP-only
Mamba-1 1.4B64.9964.7962.3160.4963.8562.7563.72

DTP versus Δ\Delta-only: The DTP row in submitted Table 1 is the two-stage pruning strategy (as per original paper), not single-boundary Δ\Delta-only pruning. The two-stage schedule is L15/L45 (1.4B) in the described adaptation. The Δ\Delta-only ablation is our ReBound pruning procedure using only the Δ\Delta score at the calibrated layer, with the same token/FLOP budget. We will distinguish the two-stage baseline from this same-boundary score ablation throughout.

P-only: Because τt,T=∑k>tΔˉk\tau_{t,T}=\sum_{k>t}\bar\Delta_k strictly decreases with tt, P-only is keep-last-K. The submitted Table 2 ablation reports +2.10% PPL for P-only versus +0.69% for the full score. The additional matched harness results above show that the two are comparable at the calibrated boundary. In the additional LongBench run, all ten ReBound–P-only paired 95% confidence intervals include zero (same layer, kept count and 384-token protected tail). We limit the Table 2 claim to its reported setting.

Q4 — Evaluation settings, uncertainty and numerical handling

Score ablation and WikiText continuation:

ExperimentModel / layerContext + continuationWindowsPruning / budget
Submitted Table 2 score ablationMamba-1 2.8B / L441,792 + 25680Remove 896 context tokens (50% of context); ~14.8% context-prefill FLOPs, ~13.0% whole-window forward FLOPs
Submitted Figure 3 PPL sweepMamba-1 2.8B / L44 for ReBound768 + 256278Multiple FLOP budgets (Figure 3)
CalibrationPer-model candidate layers128-token passages25630% non-final-token drop; δ=0.01\delta=0.01

Continuation perplexity is teacher-forced and excludes prompt tokens from the metric. The 43.75% figure is the fraction of the entire 2,048-token window removed (896/2,048), not the context-token pruning ratio.

Harness protocol: lm-eval 0.4.11, zero-shot, FP16, full splits.

BenchmarkSplitSamplesMetric
LAMBADATest5,153Accuracy
HellaSwagValidation10,042Normalized accuracy
PIQAValidation1,838Accuracy
ARC-EasyTest2,376Accuracy
WinoGrande (xl)Validation1,267Accuracy

LongBench: Prompts are left-truncated to the last 2,048 tokens. Decoding is greedy with task-specific stop strings. The submitted Table 2 states 15% / 10% prefill-FLOP targets for Mamba-1 / Mamba-2.

TaskSamplesMax new tokensMetric
Qasper200150F1
GovReport200150ROUGE-L
PassageRetrieval-en20060Accuracy
LCC6148Edit similarity
RepoBench-P7248Edit similarity

Additional uncertainty: LongBench comparisons use 10,000 paired example-level bootstrap resamples for 95% CIs. ReBound–uncompressed intervals include zero on all ten task–model comparisons. Example relative changes: Mamba-1 Qasper −4.0% [−12.2%, +3.7%]; Mamba-2 GovReport −0.4% [−1.3%, +0.6%].

Energy computation (EE). Following Section 3.3, we compute selective-update energy from the full-length execution of the calibrated layer, before token pruning:

qt=Ctht+D⊙xtq_t = C_t h_t + D \odot x_t

Et=∥qt−D⊙xt∥2=∥Ctht∥2E_t = \lVert q_t - D \odot x_t \rVert_2 = \lVert C_t h_t \rVert_2

Here, xtx_t is the post-convolution, post-activation selective-scan input; qtq_t is the scan output before output gating and projection; and DD is the direct SSM skip parameter. Thus, EtE_t isolates the selective-state readout, excluding the direct skip, output gate, and outer residual connection.

Decay coefficient (α\alpha). The readout-proximity term is:

\tau_{t,T} = \sum_{k=t+1}^{T} \bar{\Delta}_k$$ The contraction analysis motivates the layer-dependent rate $\mu = \min_d |A_d|$. The audited contraction-rate estimates are approximately 0.21 for Mamba-1 at L44 and 0.072 for Mamba-2 at L52. We compute $\log P_t = -\alpha \tau_{t,T}$ directly to avoid exponential underflow. Since proximity is independently z-normalized, positive global scaling does not affect its ranking except through numerical safeguards. **Numerical handling ($\epsilon$).** Logarithm inputs are clamped to a minimum of $10^{-12}$, and normalization standard deviations to $10^{-9}$. The three scoring components are normalized independently over non-final positions. Cumulative proximity statistics and final normalized score vectors use FP32.
Reviewer_m5F9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReBound, a training-free method for reducing the computational cost of prefill in Mamba language models. Unlike Transformer-based token pruning, removing token representations from a Mamba sequence can change the recurrent state-transition trajectory and affect subsequent representations; hence, it is not feasible. ReBound addresses this problem through two components. First, it uses calibration-time KL divergence between the original and pruned next-token distributions to identify the earliest acceptable layer boundary for pruning. Second, it ranks token representations using three signals: input-dependent Δ-strength, selective-update energy, and proximity to the final autoregressive readout. Pruning is performed after the selected layer has processed the full sequence, and the final prompt token is always retained. The reported results are promising. At 10% FLOP reduction, ReBound reduces average accuracy by 0.62 percentage points on Mamba-1 2.8B, compared with a 9.06-point reduction for DTP. It also reports a 1.15x latency improvement in a hardware experiment. The paper's central contribution is the combination of calibrated boundary selection and readout- aware token scoring for Mamba prefill.

Critical strengths
  1. The paper identifies an important limitation in efficient Mamba inference
  2. ReBound uses signals specific to Mamba's selective state-space mechanism. Combining update strength, representation energy, and readout proximity is a reasonable approach to estimating the potential effect of removing a token.
  3. The evaluation includes standard language-model benchmarks, continuation perplexity, long-context tasks, ablation studies, and actual GPU measurements. The inclusion of LongBench is particularly useful because token removal may affect the retention of information across long sequences.
  4. The seven-way comparison of scoring components supports the use of the complete ∆ + E + P score. Its reported 0.69% perplexity increase is lower than that of the individual and two-component alternatives.
Critical weaknesses
  1. The experiments cover three Mamba model configurations and a finite set of benchmarks. Although both Mamba-1 and Mamba-2 are included, the method's generality across model sizes, prompt types, and sequence lengths is not fully established.
  2. The reported accuracy and perplexity results are presented primarily as point estimates. The paper provides no confidence intervals or statistical tests for the principal comparisons. 3. The relationship between the theoretical motivation and the proposed scoring function is not fully established. Further justification is needed to explain why these signals jointly estimate the impact of token removal on the final readout.
Questions for authors
  1. Can you comment on the generalizability of the models?
  2. Can you report confidence intervals?
2 comments

We thank the reviewer for understanding the motivation, strength of the work and for the valuable feedback.

Response to Weakness 1 / Q1: Generalizability

We extended our evaluation to seven model configurations across Mamba-1 and Mamba-2, spanning 370M–2.8B parameters. Each model uses its independently calibrated pruning boundary.

ModelBaselineReBoundTwo-stage DTPReBound Δ (95% CI)
Mamba-1 370M56.3755.34 (L39)52.43−1.03 [−2.26, +0.20]
Mamba-1 790M61.1459.81 (L36)57.58−1.33 [−2.54, −0.12]
Mamba-2 370M56.7156.76 (L44)54.20+0.05 [−1.17, +1.27]
Mamba-2 1.3B64.8364.76 (L36)60.95−0.06 [−1.25, +1.12]

All results use 10% prefill FLOP reduction.

ReBound outperforms adapted two-stage DTP by 2.2–3.8 accuracy points across these additional models. The smaller Mamba-1 models exhibit approximately 1-point accuracy degradation, reflecting the more aggressive token removal required at their late safe boundaries.

Beyond model sizes, our evaluations cover language modeling, commonsense reasoning, long-context QA, summarization, retrieval, and code completion. WikiText continuation experiments additionally evaluate multiple context lengths. These results support generalization across the tested model families and task types, while confirming that the safe boundary and achievable compression are model-dependent.

Response to Weakness 2 / Q2: Statistical Confidence

We agree that uncertainty estimates are important and have computed 95% confidence intervals for the additional model-size evaluations, as reported above.

For the main matched 10% FLOP comparison on Mamba-1 2.8B, ReBound achieves 68.25% average accuracy versus 68.67% uncompressed, with an accuracy difference of −0.43 points [−1.59, +0.74].

Against UTRC, ReBound improves average accuracy by +1.87 points [0.70, 3.03] on Mamba-1 2.8B and +2.47 points [1.28, 3.67] on Mamba-1 1.4B.

These intervals support an advantage over UTRC in the matched comparisons, while showing that the small difference from the uncompressed 2.8B baseline is not statistically distinguishable from zero at the 95% level. We will include the uncertainty estimates and their computation protocol in the revised manuscript.

Response to Weakness 3: Theoretical Motivation of the Score

Our perturbation analysis provides a mechanistic motivation for the scoring heuristic rather than an exact end-to-end error certificate.

Proposition 1 identifies two factors influencing single-layer token-deletion error: the missing state-update magnitude and its attenuation through subsequent selective-state transitions. These motivate (i) Δ-strength as an indicator of selective-update behavior, (ii) selective-update energy as an observable magnitude proxy, and (iii) readout proximity as an estimate of attenuation toward the final token.

The components are independently normalized to balance their numerical ranges. Their combined usefulness is empirically supported by the submitted seven-way ablation (Table 2): Δ+E+P increases continuation perplexity by 0.69%, compared with 2.52% for Δ-only, 2.10% for P-only, and 1.20% for E+P.

Importantly, ReBound does not rely on the heuristic as a formal guarantee of safe pruning. Its calibration procedure directly measures the complete model's final-readout KL to identify an acceptable pruning boundary.

We will incorporate these additional experiments and profiling results into the revised appendix. Hope we have addressed your concerns. We would appreciate any further feedback on whether these points and the requested experiments sufficiently resolve the open issues. We are happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.