acceptodds
Under review as a conference paper at ICLR 2027

SURCO: Survivor Compensation for Recurrent-State Pruning of State Space Models

Abstract

State-space language models replace a growing KV cache with a fixed-size recurrent state, but this state can still become a substantial inference cost at scale. Recent state-pruning methods show that the recurrent state dimension can be reduced considerably while retaining useful model quality, making state pruning a direct lever for reducing inference cost. We study an orthogonal question: once a pruning mask and state budget are fixed, can the surviving states be re-optimized to improve accuracy? We introduce SURCO, a training-free framework that augments recurrent-state pruning with survivor compensation and layer-wise state allocation. Using the exact additive decomposition of Mamba-3 outputs, SURCO constructs an output-space Gram matrix from realized state responses, ranks pruning units by conditional necessity, and, after the complete pruning mask is fixed, solves a closed-form least-squares problem that reweights the surviving readouts to reconstruct the dense layer response. The resulting scalar gains are folded into the existing readout parameters and add no decode-time operation. At 50% state removal on Mamba-3 SISO/MIMO, SURCO outperforms by 2-6% accuracy improvement from prior pruning methods. We further evaluate the complete method across Mamba-3 SISO and MIMO models and Mamba-2. When the reduced state is physically materialized, 50% state removal on Mamba-3 SISO-443M yields higher batch-256 throughput and 38% lower peak memory. Thus, SURCO improves the quality retained at a fixed recurrent-state budget without giving up the systems benefits of state pruning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SSM, Mamba-3, Pruning, State Space, State reduction, Efficient ML

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iAeNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SURCO, a post-training method for pruning the recurrent states of SSMs. Instead of directly retaining the original readout gains of surviving states, SURCO adopts a response Gram matrix constructed from a calibration set to refit these gains through closed-form least squares, compensating for the output loss caused by pruning. It also introduces layer-wise pruning budget allocation based on end-to-end KL divergence. Experiments on Mamba-3 SISO/MIMO and Mamba-2 show that SURCO improves accuracy by 2–6% on average over LAST, AIRE-Prune, GHOST, and other baselines at 50% state removal.

Critical strengths
  1. The paper identifies a practical problem overlooked by previous state pruning methods: selecting a pruning mask without optimizing the remaining states' readout gains can lead to unnecessary accuracy loss.
  2. The proposed method is simple and effective. Its training-free, closed-form formulation also makes it practical to implement.
  3. The experimental design is reasonable. Table 4 decouples compensation from mask selection and demonstrates that the mechanism can generalize across different pruning methods.
Critical weaknesses
  1. Limited novelty. Post-pruning refitting of surviving parameters via least-squares reconstruction is a classical technique in neural network compression (e.g., channel pruning, structured sparsity). SURCO's application of this paradigm to SSM recurrent states is not sufficiently differentiated from these prior formulations. Moreover, the proposed conditional necessity criterion, derived from output-space Gram matrices, closely parallels existing importance metrics based on response correlations or Hessian sensitivities, leaving the core methodological innovation unclear.

  2. Narrow experimental scope. Results are restricted to two pruning ratios (37.5% and 50%), with no systematic characterization across the full operating range (e.g., 10%–90%). The largest model evaluated is 1.5B, leaving scalability unverified. More critically, Table 4 shows that SURCO's compensation alone substantially improves masks produced by LAST and AIRE-Prune, suggesting the observed gains may primarily stem from the post-hoc refitting step. The paper must therefore more rigorously establish the incremental value of its full pipeline (joint mask selection + compensation) over simply applying its compensation as a plug-in to existing pruning masks.

Questions for authors
  1. SURCO's compensation mechanism relies heavily on the additive decomposition of the Mamba-3 readout in Eq. (4). Can this approach be generalized to SSMs whose readout does not have a simple additive structure?
  2. Could the authors provide results across a wider range of pruning ratios (e.g., 10%–90%) and direct comparisons of end-to-end inference efficiency against coarse-grained pruning methods such as Mamba-Shedder?
4 comments

We thank the reviewer for their valuable feedback. We added pruning sweeps and comparisons with Mamba-Shedder, including combined accuracy, latency, and memory.

Q1. Generalization beyond Mamba-3

SURCO's exact Gram reconstruction requires an additive decomposition of the monitored state-dependent readout into pruning-unit responses (possibly followed by a fixed linear projection). It applies to compatible additive-readout SSMs beyond Mamba-3. On Mamba-2-780M, SURCO retains 50.40 Avg7 at 50% state removal versus 51.91 dense. For genuinely non-additive readouts, local linearization is a possible extension, but the exact fixed-mask reconstruction guarantee does not carry over; we do not claim to have evaluated that setting. Our contribution is recurrent-state-specific refitting, rotary-safe units, gain folding without decode operations, and KL-based allocation. But addressing the non additive readout SSM to be considered as future work. One possible approach could be to construct a local linear approximation of the readout around calibration trajectories and use the resulting Jacobian-based state-response representations to estimate inter-state dependencies and survivor compensation.

Q2. Pruning Ratio Sweep

Pruning-ratio sweep. On Mamba-3 SISO-443M, we evaluated full SURCO from 0–87.5% state removal; LAST, AIRE-Prune, and GHOST were compared at matched budgets through 75%. FineWeb-Edu PPL (lower is better):

RemovedSURCOLASTAIREGHOST
0%13.57013.57013.57013.570
12.5%13.68813.65113.64913.698
25%13.84613.94713.85014.039
37.5%14.04914.41014.36714.722
50%14.32715.88715.30618.441
62.5%15.09320.00019.36722.509
75%18.25027.06330.77129.384
87.5%22.380NANANA

NOTE: NA in the table means very high PPL (unstable).

LAMBADA accuracy (higher is better):

RemovedSURCOLASTAIREGHOST
0%43.3943.3943.3943.39
12.5%42.7942.3742.6338.81
25%42.6542.3042.5134.66
37.5%42.3542.0242.5731.24
50%40.335.4233.817.58
62.5%30.8022.5719.778.11
75%17.5411.127.923.32

SURCO leads clearly at across the complete sweep window at given pruning budget. Its seven-task average (LM-harness) is 49.99 dense, 49.15 at 50%, 44.45 at 75%, and 42.42 at 87.5%.

Q2. Comparison with Mamba-Shedder: Complementary pruning strategies

Mamba-Shedder and SURCO are complementary pruning methods. Mamba-Shedder removes entire Mamba blocks, reducing model parameters and computation, whereas SURCO reduces recurrent-state dimensions within surviving blocks. Thus, SURCO can be applied on top of Mamba-Shedder to further reduce inference cost. We evaluate their composition in both model accuracy and hardware efficiency.

(a) Accuracy: Dense → Mamba-Shedder → +SURCO

Experimental setup. We reproduced Mamba-Shedder's cumulative, Alpaca-calibrated, PPL-greedy block removal and subsequently applied SURCO to surviving layers. We evaluate Mamba-2-780M and Mamba-3 SISO-443M using the same seven-task LM suite (Avg7).

Mamba-2-780M accuracy (Avg7 ↑):

Blocks removedShedder only+SURCO25+SURCO50
0 (dense)51.9150.6649.11
449.8348.7746.80
846.2045.5543.13
1242.7842.1140.26

Observations: These results show that state pruning remains effective after coarse-grained block removal, with a relatively consistent marginal accuracy cost. Here, we have opperated with uniform allocation, and bf16 evaluation variation reaches approximately 0.33 Avg7 points.

(b) Hardware: Dense → Mamba-Shedder → +SURCO

Experimental setup. On one NVIDIA H100, we physically remove Mamba-3 SISO-443M blocks and materialize SURCO-pruned surviving layers (FoldedMamba3), without hook-simulated decoding. We measure batch-256 throughput and peak memory.

ConfigurationAvg7 ↑Speedup ↑Peak memory ↓
Dense49.991.00×7.92 GB
Shedder: 4 blocks45.571.20×7.83 GB
+SURCO2544.621.55×5.39 GB
+SURCO5043.731.99×3.97 GB
Shedder: 8 blocks40.281.48×7.22 GB
+SURCO2539.721.92×4.53 GB
+SURCO5038.682.47×3.36 GB

Observations: We can observe that SUCO provides real hardware benefits over Mamba-sheddar. The SURCO performance to speedup tradeoff is better than mamba-sheddar

Cross-architecture verification. On Mamba-2-780M, 12-block removal provides 1.33× speedup, SURCO50 alone 1.88×, and their combination 2.51×, reducing peak memory from 12.00 GB (dense) to 5.48 GB (−54%). These Mamba-2 hardware measurements use a materialized state-dimension surrogate; accuracy is measured separately on the actual pruning path.

(c) Matched-state quality against Mamba-Shedder

We also compare SURCO against Mamba-Shedder's coarse SSM removal at identical recurrent-state budgets on Mamba-2-780M.

State removedShedder Avg7SURCO Avg7Shedder PPLSURCO PPL
25%48.6651.6814.58413.658
50%38.6550.4021.71113.850

At matched 50% state removal, SURCO retains +11.75 Avg7 over Shedder's SSM removal, confirming the advantage of fine-grained recurrent-state reduction over eliminating entire SSM components.

Conclusion. These experiments demonstrate two complementary benefits: (i) SURCO preserves substantially more accuracy than coarse SSM removal at matched state budgets; (ii) applying SURCO on top of Mamba-Shedder block removal yields substantial additional memory and latency reductions, with a relatively consistent marginal accuracy cost.

W1. Novelty

Prior state-pruning methods primarily focus on identifying and removing redundant states, while leaving the surviving states unchanged. We in SURCO take a complementary perspective: rather than only deciding which states to remove, it also compensates and re-optimizes the surviving states to recover the lost output response. This motivates our reduced-realization formulation, which combines:

  1. Response geometry: Constructs Gram matrices from realized, post-projection state-to-output contributions, capturing functional overlap between recurrent-state units.
  2. Conditional necessity: Measures the residual response energy of a state unit after projecting it onto the span of other units, using a Schur-complement formulation and normalization by physical state cost (including indivisible rotary pairs).
  3. Survivor compensation: Jointly refits surviving readouts after fixing the complete pruning mask and folds the resulting gains into existing parameters, introducing no additional decode-time operations.
  4. Global allocation: Uses measured end-to-end predictive KL distortion to distribute the state budget across layers, rather than relying solely on local importance scores.

Our contribution is unified, recurrent-state-specific formulation and integration compensating pruned states for selective SSM compression, rather than least-squares refitting in isolation.

W2. Added Experiments

(i) Disentangling compensation from the complete SURCO pipeline

Experimental setup. On Mamba-3 MIMO-1.5B at 50% state removal, we evaluate seven-task LM accuracy (Avg7) and FineWeb-Edu PPL. All runs retain the same total budget (mean 64/128 coordinates per layer). We vary selection, allocation, and compensation independently and refit gains for each mask.

(a) Compensation: fixed mask and allocation.

Selector + allocationNo comp. Avg7+Comp. Avg7Gain
SURCO + KL allocation51.9955.01+3.02
LAST + own allocation49.3351.78+2.45
AIRE-Prune + own allocation48.9752.04+3.07
GHOST + uniform allocation49.5852.59+3.01

Observation: Compensation improves all four fixed masks by 2.45–3.07 points.

(b) Selection: fixed uniform allocation, compensation on.

SelectorFW-Edu PPL ↓Avg7 ↑
SURCO (conditional necessity)11.66152.93
GHOST11.96252.59
LAST12.05352.11
AIRE-Prune12.21351.83

Observation: With allocation and compensation fixed, selectors span 0.80 points; SURCO leads GHOST by 0.34.

(c) Allocation: fixed SURCO selector and compensation. Uniform allocation yields 52.63 Avg7, whereas measured KL-distortion allocation yields 55.01 (+2.38). This isolates the benefit of KL allocation.

Conclusion (Q1). Along a matched-budget controlled path,

GHOST/uniform/no compensation 49.58 → +compensation 52.59 (+3.01) → SURCO selection 52.93 (+0.34) → KL allocation 55.01 (+2.08).

The substantial MIMO gain is driven primarily by survivor compensation and KL-guided allocation with selector having its contribution as well..

(ii) Model scalability.

We evaluated all available released Mamba-3 checkpoints, spanning 187M–1.5B parameters, and additionally evaluated Mamba-2-780M. The 1.5B checkpoint is the largest publicly released Mamba-3 checkpoint used in our study. Evaluating substantially larger models would require access to additional pretrained checkpoints or training resources beyond the computational scope of this work. We therefore leave evaluation at larger scales as an important direction for future work, subject to checkpoint availability and computational resources.

We will incorporate these additional experiments and profiling results into the revised appendix. Hope we have addressed your concerns. We would appreciate any further feedback on whether these points and the requested experiments sufficiently resolve the open issues. We are happy to clarify anything further.

Reviewer_NK33OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Method. This paper proposes SURCO, a training-free method for pruning recurrent states in pretrained SSMs. It constructs a Gram matrix from the observed output contributions of individual state units, uses their conditional necessity to select which units to remove, and fits scalar gains on the surviving units to approximate the dense layer output. The method additionally allocates the state budget across layers using end-to-end KL divergence. The fitted gains can be folded into the existing readout parameters without additional decoding operations.

  2. Experimental setup. The authors evaluate SURCO on Mamba-3 SISO/MIMO models and Mamba-2, comparing against LAST, AIRE-Prune, GHOST, and random pruning. The main language modeling experiments use 50% recurrent-state removal, while retrieval is evaluated on Mamba-3 SISO-1.5B at 37.5% removal. They also measure decoding throughput and memory usage after physically reducing the recurrent state.

  3. Results. SURCO achieves the largest improvements over existing pruning methods on Mamba-3 MIMO. At 50% state removal, MIMO-1.5B retains 55.08 average task accuracy, compared with 55.61 for the dense model and 49.56 for the strongest reported pruning baseline. The paper also shows that survivor compensation improves retrieval performance under fixed pruning masks from several methods. Physical state reduction gives 1.72x higher batch-256 decoding throughput and 38% lower peak memory on Mamba-3 SISO-443M.

Critical strengths
  1. Strong language modeling results, particularly for MIMO. At 50% state removal, SURCO achieves the best average accuracy among the evaluated pruning methods across Mamba-3 SISO and MIMO checkpoints (Tables 1, 5, 6). The improvements are especially substantial for MIMO, where SURCO retains most of the dense model's performance. These results demonstrate that a significant fraction of the recurrent state can be removed while largely preserving language modeling performance.

  2. Ablating the compensation procedure. Table 4 isolates survivor compensation while holding each method's pruning mask and layer allocation fixed. Compensation improves average retrieval performance across LAST, AIRE-Prune, GHOST, and SURCO, showing that its benefits extend beyond the proposed selection method. The fitted gains can also be folded into the existing readout parameters without introducing additional decoding operations.

Critical weaknesses
  1. Limited evaluation of long-context performance. The main language modeling results are encouraging, but it remains unclear how well SURCO preserves longer-context capabilities.

    • Retrieval: Experiments are limited to Mamba-3 SISO-1.5B at 37.5% state removal, using 2K contexts for real-world tasks and 1K–4K for NIAH (Tables 2 and 4). While SURCO generally outperforms existing methods, there is substantial degradation relative to the dense model. For example, NIAH-Single-1 accuracy at 2K drops from 99.3% to 10.7%.
    • Missing evaluation: Performance versus retained-state budget at different context lengths, including long-document language modeling and retrieval on SISO and MIMO, would help characterize this tradeoff. Ideally, the dense checkpoints should have established longer-context capabilities to isolate the effect of pruning.

    These experiments would clarify how much recurrent-state memory can be removed before long-context performance deteriorates.

  2. Limited ablations of the main MIMO improvements. SURCO combines three components: state selection, layer-wise allocation, and survivor compensation. The individual contributions of these components to the main MIMO results are unclear.

    • Existing evidence: Table 4 provides a useful fixed-mask compensation ablation on SISO retrieval, showing that compensation improves several existing pruning methods. However, the main MIMO comparisons (Table 1) change all three components together.
    • Missing ablations: It would be useful to isolate these components on MIMO, particularly at 50% state removal where the gains are largest. Comparing different selectors under the same layer budgets and compensation procedure, and KL-based versus uniform allocation with the selector fixed, would help establish which components account for the improvements.
Questions for authors
  1. What accounts for the large improvements on MIMO at 50% state removal? Table 4 isolates compensation on SISO retrieval, but the contributions of selection, allocation, and compensation to the main MIMO results remain unclear. Could the authors provide the following ablations on Mamba-3 MIMO-1.5B using the main LM benchmarks?

    • Compensation: For each method, hold the pruning mask and layer allocation fixed, and compare performance with and without fitted survivor gains, as in Table 4.
    • Selection: Compare the different selectors under the same per-layer state budgets (e.g., uniform allocation), applying the same compensation procedure to each.
    • Allocation: With SURCO's selector and compensation fixed, compare KL-based allocation against uniform allocation at the same total state budget.

    Compensation should be refitted for each resulting mask. These comparisons would help establish which components account for the improvements over existing methods.

  2. How well does SURCO preserve long-context language modeling and retrieval performance? I would like to understand the tradeoff between recurrent-state size and long-context capabilities. Ideally, this evaluation would start with a checkpoint trained or adapted to longer contexts (e.g., 32K), then prune it without further fine-tuning and compare the dense and pruned models within that context range.

    • Language modeling: Evaluate held-out long-document perplexity, including loss versus token position (as in [1]).
    • Retrieval: Evaluate NIAH, including multi-key, multi-value, and multi-query variants. Selected LongBench tasks would also be useful.
    • Pruning tradeoff: Report performance across several pruning ratios (e.g., 0%, 12.5%, 25%, 37.5%, 50%) and context lengths, comparing SURCO against existing pruning methods at the same state budgets.

    I would be particularly interested in results on both SISO and MIMO. The dense checkpoints should perform meaningfully on the evaluated tasks, so that degradation can be attributed to pruning rather than limitations of the original model. Please also clarify the context length used for pruning calibration.

[1] Arnuv Tandon et al. End-to-End Test-Time Training for Long Context. 2025. https://arxiv.org/abs/2512.23675

4 comments

We thank the reviewer for their detailed and valuable feedback. We report new experiments addressing both concerns.

Q1/W2. What drives MIMO improvements at 50% removal?

Experimental setup. On Mamba-3 MIMO-1.5B at 50% state removal, we evaluate seven-task LM accuracy (Avg7) and FineWeb-Edu PPL. All runs retain the same total budget (50% pruning). We vary selection, allocation, and compensation independently and refit gains for each mask.

(a) Compensation: fixed mask and respective allocation.

MethodsNo comp. Avg7+Comp. Avg7Gain
SURCO51.9955.01+3.02
LAST49.3351.78+2.45
AIRE-Prune48.9752.04+3.07
GHOST49.5852.59+3.01

Observation: Compensation improves all four fixed masks by 2.45–3.07 points.

(b) Selection: fixed uniform allocation, compensation on.

SelectorFW-Edu PPL ↓Avg7 ↑
SURCO11.66152.93
GHOST11.96252.59
LAST12.05352.11
AIRE-Prune12.21351.83

Observation: With allocation and compensation fixed, selectors span 0.80 points; SURCO leads GHOST by 0.34.

(c) Allocation: fixed SURCO selector and compensation. Uniform allocation yields 52.63 Avg7, whereas measured KL-distortion allocation yields 55.01 (+2.38). This isolates the benefit of KL allocation.

Conclusion (Q1/W2). Along a matched-budget controlled path,

GHOST/uniform/no compensation - 49.58 → +compensation - 52.59 (+3.01) → SURCO selection - 52.93 (+0.34) → KL allocation - 55.01 (+2.08).

The substantial MIMO gain is driven primarily by survivor compensation and KL-guided allocation with selector having its contribution as well.

Q2/W1. Long-context language modeling and retrieval

Experimental setup. We adapted Mamba-3 MIMO-1.5B to 32K context using PG19, then applied SURCO, LAST, AIRE-Prune, and GHOST without further fine-tuning. All methods were evaluated at matched state-removal ratios of 12.5%, 25%, 37.5%, and 50%. Calibration used eight 2,048-token FineWeb-Edu sequences (16,384 tokens).

For language modeling, we evaluated PG19 perplexity at 32K. For retrieval, we evaluated single-needle, multi-value, multi-query, and multi-key NIAH at 1K, 2K, 3K, and 4K, using 150 examples per configuration and context length. Multi-* tasks use four needles/queries. All retrieval experiments use the same adapted MIMO-1.5B checkpoint.

(a) Long-context language modeling. The dense checkpoint achieves 16.27 PPL at 32K. Table 1 reports the PPL increase relative to the dense model.

Table 1. PG19 perplexity increase at 32K (ΔPPL ↓).

State removedSURCOGHOSTLASTAIRE-Prune
12.5%0.200.392.902.92
25%0.461.323.846.33
37.5%1.053.506.388.27
50%2.009.3314.4315.49

(b) Long-context retrieval. Tables 2–5 report accuracy (%) at each context length and pruning ratio.

Table 2. Single-needle retrieval (↑).

Method / Removed1K2K3K4K
Dense / 0%90.748.735.316.7
SURCO / 37.5%84.747.329.313.3
LAST / 37.5%31.310.02.70.7
AIRE / 37.5%12.04.00.00.0
GHOST / 37.5%21.38.70.70.7
SURCO / 50%72.734.016.76.7
LAST / 50%≤0.70.00.00.0
AIRE / 50%≤0.70.00.00.0
GHOST / 50%≤0.70.00.00.0

Table 3. Multi-value, multi-query, and multi-key retrieval accuracy (%, ↑).

Method / RemovedMV 1KMV 2KMV 3KMV 4KMQ 1KMQ 2KMQ 3KMQ 4KMK 1KMK 2KMK 3KMK 4K
Dense / 0%38.226.716.38.840.723.517.37.248.731.318.011.3
SURCO / 12.5%38.026.216.79.240.023.517.27.248.731.318.712.0
LAST / 12.5%29.717.310.26.230.217.211.35.839.324.714.09.3
AIRE / 12.5%28.816.87.74.225.314.58.74.237.321.310.08.0
GHOST / 12.5%34.722.211.35.837.320.811.84.042.725.312.06.0
SURCO / 25%36.024.711.87.537.322.713.05.549.327.312.713.3
LAST / 25%24.210.83.72.221.59.75.02.330.014.08.02.7
AIRE / 25%4.02.71.31.04.52.81.51.017.34.74.00.0
GHOST / 25%30.715.54.81.824.313.05.02.734.014.79.33.3
SURCO / 37.5%26.312.06.74.328.213.87.23.340.018.78.78.7
LAST / 37.5%5.31.50.80.52.81.70.80.312.02.02.00.0
AIRE / 37.5%2.01.00.50.51.20.70.80.29.31.30.70.0
GHOST / 37.5%12.02.80.70.511.32.31.30.317.34.72.72.0
SURCO / 50%12.75.54.03.815.05.83.84.526.011.36.74.0
LAST / 50%≤0.20.0≤0.20.0≤0.20.00.00.00.70.00.00.0
AIRE / 50%0.00.00.00.00.00.00.00.00.00.00.00.0
GHOST / 50%0.70.00.00.0≤0.20.00.00.01.30.70.00.0

MV: multi-value; MQ: multi-query; MK: multi-key. Values are percentages.

(c) Aggregate retrieval analysis.

For compact comparison, the following reports the mean over the four context lengths.

Method / RemovedSingleMulti-valueMulti-queryMulti-keyMulti-* Avg
Dense47.822.522.227.324.0
SURCO 12.5%48.522.522.027.724.0
SURCO 25%47.220.019.625.721.8
SURCO 37.5%43.712.313.119.014.8
SURCO 50%32.56.57.312.08.6
GHOST 12.5%39.718.518.521.519.5
GHOST 25%26.313.211.215.313.3
GHOST 37.5%7.84.03.86.74.8
GHOST 50%0.20.20.00.50.2

CONTINUED.....

(d) Loss versus token position.

Following the evaluation protocol suggested by the reviewer, we measured loss across token positions on Mamba-3 SISO-443M up to 32K tokens.

Token-position intervalLoss (nats)
1,024–1,5793.3583
21,247–32,7683.3203

NOTE. Loss does not increase toward later token positions, indicating no observable accumulation of prediction loss over the evaluated 32K sequence positions.

Overall observations and conclusion.

The results establish three consistent trends:

  1. Long-context language modeling: SURCO produces the smallest perplexity degradation at every evaluated pruning ratio, retaining substantially more PG19 performance at 32K, particularly under aggressive state compression.

  2. Retrieval preservation: SURCO maintains near-dense multi-task retrieval performance at 12.5% removal. Its advantage becomes increasingly pronounced at higher pruning ratios and is consistent across single-needle, multi-value, multi-query, and multi-key retrieval.

  3. Compression versus quality: At 37.5% removal, SURCO's multi-* average exceeds every baseline at 25% removal. At 50% removal, SURCO retains measurable retrieval performance across all three multi-* tasks, whereas the baselines are nearly zero.

Overall, these experiments demonstrate that SURCO provides a more favorable tradeoff between recurrent-state compression and retained long-context capabilities. It preserves substantially more language-modeling and retrieval performance than existing pruning methods at matched state budgets, without requiring post-pruning fine-tuning.

We will incorporate these additional experiments and profiling results into the revised appendix. Hope we have addressed your concerns. We would appreciate any further feedback on whether these points and the requested experiments sufficiently resolve the open issues. We are happy to clarify anything further.

Reviewer_8yxhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method called SURCO, which is a training-free framework for recurrent state pruning for linear attention models such as Mamba2 and Mamba3, to compress the memory requirement for linear attention model serving. The main contribution of this work is that instead of simply determining the pruning masks, the method designs a way to compensate the remaining state to recover the performance of the model. By constructing the gram matrix with a small set of calibration data and by conditional necessity to obtain the solution of compensation. Moreover, this paper proposes an end-to-end prediction method for layer-wise pruning budget allocation.

Critical strengths
  1. The motivation is clear. Like the weight pruning in neural networks, the paper not only prunes the state matrix in the linear attention module, but also compensate the remaining state to reduce the pruning errors.

  2. The method is simple, useful and theoretically grounded. The solution is obtained via least-square method. The compensation is fused into the state so no additional cost is added during the inference and serving.

  3. This paper does a comprehensive evaluation across different model architectures and sizes. Moreover, this method is also validated on different tasks such as long-context settings.

Critical weaknesses
  1. The calibration set used in the experiments is too small. This paper only uses 8 sequences from Fineweb-edu dataset as the calibration data, which can cause distribution shift and overfitting in this setting. It is not clear if the performance is overfitted or generalized to all different datasets. The authors are encouraged to provide the ablation for different sample sizes and data sources to demonstrare the robustness of this method.

  2. The analysis of offline computation cost is missing. This work is training-free but the reverting operation of gram matrix adds extra computational cost. The author should provide additional wall-clock time report to make this work more useful.

Questions for authors

See weakness.

4 comments

We thank the reviewer for recognizing SURCO's motivation, theoretical formulation, and practical effectiveness. We address both concerns through new calibration robustness experiments and detailed offline runtime profiling.

W1. Calibration size and distribution robustness

We would like to clarify that the model once calibrated and pruned on FineWeb-Edu is evaluated on other dataset without any further calibration for all the results in the paper and rebuttal response.

Experimental setup. We conduct two controlled ablations on Mamba-3 SISO-443M at 50% recurrent-state removal:

  1. Calibration size: Vary FineWeb-Edu calibration from 4 to 64 sequences (4,096–131,072 tokens), covering a 32× range.
  2. Calibration source: Compare FineWeb-Edu, WikiText, Math, and Code, keeping the calibration budget fixed at 16,384 tokens.

All configurations use the same pruning procedure and downstream evaluation suite.

(a) Calibration size results

SequencesTokensFW-Edu PPL ↓LM Avg. ↑
48,19214.43649.17
8 (default)16,38414.33349.30
1632,76814.38349.12
3265,53614.36149.08
64131,07214.36149.07

Observations.

  • Across 4 - 64 sequences (16× token variation), LM accuracy varies by only 0.23 points, and perplexity by 0.103.
  • Increasing calibration data 8× beyond the default sees saturated accuracy within the noise margin.
  • Performance does not improve monotonically with calibration size.

(b) Calibration source results

SourceFW-Edu PPL ↓LM Avg. ↑
FineWeb-Edu14.33349.30
WikiText14.58648.86
Math14.56548.70
Code14.55348.35

Observations.

  • Across four calibration sources, the maximum variation is 0.95 accuracy points / 0.253 PPL.
  • Even Code calibration achieves 48.35 Avg. / 14.553 PPL, compared with LAST (47.99 / 15.89) and GHOST (45.05 / 18.45) using FineWeb-Edu calibration.

Conclusion (W1): SURCO exhibits limited sensitivity to calibration size and source at 50% pruning. The default 16,384-token budget achieves stable downstream performance, with no systematic benefit from larger calibration sets. Performance remains competitive even with calibration distributions substantially different from FineWeb-Edu. These results directly address the concern that the improvements depend on the original eight calibration sequences.

W2. Offline computational overhead

NOTE: Calibration statistics and KL curves are computed offline once per checkpoint and reused across pruning budgets.

Experimental setup. We profile SURCO on a single NVIDIA H100 GPU across all eight Mamba-3 SISO/MIMO checkpoints (187M–1.5B). We separately measure checkpoint loading, calibration rollout, state scoring, KL-distortion curves, allocation/selection/compensation, and physical materialization where applicable. Reported measurements use a warm Triton kernel cache.

(a) Gram-matrix and compensation overhead

Mamba-3 has 128 physical state coordinates comprising 32 rotary pairs and 64 individual coordinates, yielding 96 pruning units per layer. Therefore, the Gram matrix is 96×96, independent of model width at fixed state dimension.

Measured costs:

  • Gram inversion: 0.287–0.292 ms/layer.
  • Whole-model Gram inversions: 3.5–7.0 ms.
  • Including survivor-compensation solves: 5.5–12.6 ms/model.
  • Combined solves account for 0.008–0.017% of measured pipeline runtime.

Thus, the additional matrix operations contribute negligible offline cost. Their total cost scales with the number of layers, while their per-layer dimension remains fixed for these checkpoints.

(b) End-to-end runtime

ModelCalibrationKL curves
SISO-187M10.0 s21.4 s
SISO-443M18.0 s50.9 s
SISO-893M22.9 s51.8 s
SISO-1.5B23.1 s53.8 s
MIMO-187M39.5 s37.4 s
MIMO-444M44.2 s66.5 s
MIMO-894M50.8 s68.2 s
MIMO-1.5B52.5 s70.6 s

Observations.

  • The dominant additional cost is KL-based layer allocation, which evaluates predictive distortion through repeated forward passes, not Gram inversion.
  • Compared with LAST/AIRE-Prune/GHOST, this incurs 21–70 s additional offline runtime.

Conclusion (W2). We explicitly acknowledge the additional offline cost of KL-based allocation. However, Gram inversion and compensation are negligible contributors, and the principal calibration/allocation cost is amortized across pruning budgets. Furthermore, compensation gains are folded into existing readout parameters, introducing no additional decode-time operations and preserving the inference-efficiency benefits of physical state pruning.

We will incorporate these additional experiments and profiling results into the revised appendix. Hope we have addressed your concerns. We would appreciate any further feedback on whether these points and the requested experiments sufficiently resolve the open issues. We are happy to clarify anything further.

Reviewer_8yxhOpenReview ↗

Response to Authors.

Thank you for your detailed rebuttal and providing additional experiments. My concerns are all addressed. Therefore, I would like to maintain my positive score for this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.