acceptodds
Under review as a conference paper at ICLR 2027

SERESA: Selective Recurrence Restriction for State Space Duality Acceleration

Abstract

Mamba-2's State Space Duality (SSD) enables hardware-efficient chunk-wise recurrence, yet its computational efficiency does not translate proportionally into prefill latency. Despite accounting for less than 5% of FLOPs, SSD consumes nearly 20% of prefill latency, motivating us to examine temporal dependencies across SSD heads. While SSD applies full recurrence uniformly, output-relevant recurrent contributions vary substantially in temporal range across heads. We quantify this heterogeneity by defining the effective recurrence distribution and its expected lag, Effective Recurrence Length (ERL). Our ERL analysis reveals that most heads concentrate their contributions at short temporal lags, whereas long-range contributions are confined to a small subset of heads. The head-wise ERL ordering also remains stable across input distributions, revealing a quasi-static recurrence structure. Building on this, we propose SERESA, a training-free framework that selectively restricts recurrence. SERESA determines the number of restricted heads from the ERL distribution and selects heads that retain the most contribution within a prescribed recurrence length. It reduces their intra- and inter-chunk computation while recovering boundary-crossing contributions through boundary correction, leaving full recurrence for the remaining heads. Without additional training or input-dependent reselection, SERESA restricts 78–83% of heads across Mamba-2 and Nemotron-H while closely matching full-recurrence baseline perplexity and downstream accuracy. It accelerates the SSD core by 1.75–2.02X and achieves up to 1.29X end-to-end prefill speedup with projection quantization. Code will be made publicly available upon acceptance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Mamba-2, State Space Duality, Inference Acceleration, Prefill Acceleration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vifGOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper observes that in Mamba-2 most SSD heads put their output-relevant recurrent contribution at short time lags, and that the ordering of heads by this range is stable across input domains. Based on this, SERESA is a training-free method: a one-time calibration splits the heads into local heads, which keep only the contributions within a lag of R = 64, and global heads, which keep the full recurrence. On Mamba2-1.3B/2.7B/8B and Nemotron-H-8B, restricting 78–83% of the heads gives a 1.75–2.02× speedup of the SSD core with perplexity close to the baseline.

Critical strengths
  1. Clear analysis and a simple, training-free method. The head-wise recurrence length is heavy-tailed in all models tested, and the head partition from a single 2K-token sequence overlaps strongly with the one from 20 sequences.
  2. Informative ablations. Coverage-based head selection clearly beats selecting by decay alone or by a learned gate (perplexity increase of 0.1–0.3% against 0.6–1.7%), and the boundary correction recovers most of the quality lost by skipping inter-chunk recurrence.
  3. Favorable equal-speed comparison and transparent reporting. At about 1.27× SSD speedup on Mamba2-2.7B, SERESA matches the baseline perplexity (9.06 on WikiText-2), while GHOST at 25% pruning gives 9.65 and SSDi8 gives 9.13. The paper also reports the less favorable facts, such as the lower speedup on A100.
Critical weaknesses

W1. The evidence that quality is preserved beyond perplexity is limited, and there are no long-context or retrieval results in the main paper. Table 2 compares SERESA only with the unmodified model. SSDi8 and GHOST, which Table 1 compares against, are missing, so the table cannot show whether SERESA's downstream accuracy is better than theirs. The differences from the baseline are within 1.8 points (the largest drop is 1.0 point), with no sample counts or intervals, so the table neither shows a clear advantage nor a clear absence of degradation. In addition, Table 2 only uses samples of at most 2K tokens (Appendix E), so these long-document tasks do not test long-range behavior. All three tasks are scored by log-likelihood (Appendix E), yet several baseline numbers are near or below chance (Mamba2-8B on the binary Hyperpartisan task: 45.3%; Mamba2-1.3B and 2.7B on the three-class ContractNLI: 13.0% and 17.3%); please report the class balance or a majority-class baseline. Standard zero-shot benchmarks such as ARC-Challenge, ARC-Easy, HellaSwag, PIQA and WinoGrande, and MMLU for the 8B models, would be more informative than the current tasks. The main text contains no long-context or retrieval evaluation at all; it only states that SERESA "maintains long-context performance, as shown in Appendix I". Appendix I covers only Nemotron-H-8B, whose attention layers can handle retrieval, so it does not show that the long-range ability of pure SSM heads is preserved, which is where restricting the recurrence is most likely to hurt. No retrieval test is run on Mamba-2, even within its trained length. Since prefill acceleration matters most for long inputs, these results should be in the main paper and include pure Mamba-2 models.

W2. The claim that the budget rule is "conservative" is shown on only one model. The paper chooses the number of local heads as the number of heads whose recurrence length is below the mean, and says this gives a "conservative" boundary without a performance-based search. The rule is applied to all four models in Table 1, but the evidence that it stays before the point where perplexity starts to rise is a local-head ratio sweep on Mamba2-2.7B only (Figure 4(b); 81.25% of the heads restricted, perplexity increase of 0.1–0.3%). On Mamba-3 (Appendix H, Table 12), the same rule restricts fewer heads (about 73%) but raises perplexity by about 1.0–1.1% on Pile and WikiText-2 (0.4% on C4), several times more than on Mamba-2. It is therefore unclear whether the rule stays on the safe side of that point on other architectures; a similar sweep on Mamba-3 would settle this.

Questions for authors
  1. The abstract reports up to 1.29× end-to-end prefill speedup with projection quantization. According to Sec. 6.2, on Mamba2-2.7B quantization alone gives 1.202× and adding SERESA raises it to 1.287×, so SERESA adds about 7% on top of quantization, while SERESA alone gives about 1.1×. Could the abstract also state the SERESA-only end-to-end speedup?
  2. The paper says the code will be released upon acceptance. Will it include the SERESA kernels (the local-head path with the boundary correction) and the calibration scripts, so that the reported speedups can be reproduced?
  3. Are there speed measurements on Mamba-3? Appendix H reports only perplexity for Mamba-3 (SISO and MIMO, 1.5B). Is the method, in particular the kernel, limited to Mamba-2 models, or does it also apply to Mamba-3, and what speedup does it give there?
9 comments
AuthorsOpenReview ↗

Response #1

Thank you for your careful review and constructive feedback. We appreciate your recognition of the head-wise recurrence analysis underlying SERESA's training-free design and the stability of the head partition even with a single 2K-token calibration sequence. We also appreciate your assessment of the ablations supporting Coverage-based head selection and boundary correction, as well as the baseline comparisons at comparable SSD-core speedups. We address your concerns and questions below. We will include the clarifications and findings from the rebuttal discussion in the final version.

[W1-1] Downstream Accuracy Compared with SSDi8 and GHOST

To assess how SERESA's accuracy compares with that of SSDi8 and GHOST on the downstream tasks in Table 2, we additionally evaluated SSDi8 INT8 and GHOST with 50% state-dimension pruning on the same evaluation examples. The table below reports accuracy (%) for each method. Parentheses give the accuracy difference from the full-recurrence Baseline and its 95% confidence interval, both in percentage points.

ModelTaskBaselineSERESASSDi8 INT8GHOST 50%
Mamba2-1.3BWikiHop25.25%25.00% (−0.25 [−1.74, +1.14])24.50% (−0.75 [−2.58, +0.87])23.50% (−1.75 [−4.52, +0.85])
Mamba2-1.3BHyperpartisan63.50%62.75% (−0.75 [−2.81, +1.16])62.50% (−1.00 [−2.80, +0.49])63.75% (+0.25 [−2.15, +2.68])
Mamba2-1.3BContractNLI13.00%13.25% (+0.25 [−0.95, +1.58])12.00% (−1.00 [−2.80, +0.49])13.25% (+0.25 [−1.31, +1.89])
Mamba2-2.7BWikiHop27.25%27.00% (−0.25 [−1.74, +1.14])28.00% (+0.75 [−1.51, +3.12])26.00% (−1.25 [−4.16, +1.56])
Mamba2-2.7BHyperpartisan63.75%63.75% (0.00 [−1.67, +1.67])64.25% (+0.50 [−0.72, +1.96])64.25% (+0.50 [−1.62, +2.70])
Mamba2-2.7BContractNLI17.25%16.75% (−0.50 [−1.96, +0.72])18.25% (+1.00 [−1.07, +3.24])22.25% (+5.00 [+1.52, +8.69])
Mamba2-8BWikiHop36.00%35.00% (−1.00 [−2.67, +0.28])34.75% (−1.25 [−3.35, +0.60])33.25% (−2.75 [−5.99, +0.32])
Mamba2-8BHyperpartisan45.25%45.75% (+0.50 [−1.37, +2.48])44.75% (−0.50 [−3.87, +2.84])41.75% (−3.50 [−7.64, +0.55])
Mamba2-8BContractNLI48.75%48.75% (0.00 [−0.95, +0.95])47.75% (−1.00 [−2.80, +0.49])49.00% (+0.25 [−0.95, +1.58])
Nemotron-H-8BWikiHop43.25%43.00% (−0.25 [−2.27, +1.72])42.25% (−1.00 [−4.17, +2.11])42.75% (−0.50 [−3.34, +2.30])
Nemotron-H-8BHyperpartisan74.25%73.25% (−1.00 [−3.43, +1.30])72.25% (−2.00 [−4.82, +0.63])75.25% (+1.00 [−1.95, +4.02])
Nemotron-H-8BContractNLI28.50%30.25% (+1.75 [−0.30, +4.08])25.75% (−2.75 [−5.78, +0.07])20.25% (−8.25 [−11.55, −5.54])

Relative to the full-recurrence baseline, SERESA's accuracy differences ranged from −1.00 to +1.75 percentage points (pp), and the 95% confidence intervals included zero for all 12 model–task pairs. The mean absolute differences from Baseline were 0.54 pp for SERESA, 1.13 pp for SSDi8 INT8, and 2.10 pp for GHOST at 50% pruning. The corresponding maximum absolute differences were 1.75, 2.75, and 8.25 pp. SERESA also had the narrowest mean 95% confidence interval for the accuracy difference (3.3 pp, compared with 4.4 pp for SSDi8 and 5.4 pp for GHOST). For GHOST, the 95% confidence interval excluded zero in two pairs, both on ContractNLI: +5.00 pp [1.52, 8.69] for Mamba2-2.7B and −8.25 pp [−11.55, −5.54] for Nemotron-H-8B.

AuthorsOpenReview ↗

Response #2

[W1-2] Evaluation Scope, Class Baselines, and Per-Example Agreement

For each task in Table 2, we report the class distribution of the evaluation sample and the accuracy baseline. We added the total number of eligible examples and the class distributions to Appendix E of the revised manuscript.

TaskModelFinal evaluation sample (n=400n=400)Accuracy baseline
WikiHopAll models2–30 answer candidates per example, with a mean of 10.317.2% (random selection)
HyperpartisanAll models262 Not hyperpartisan / 138 Hyperpartisan65.5% (majority class)
ContractNLIMamba2-1.3B and 2.7B201 / 162 / 3750.25% (majority class)
ContractNLIMamba2-8B196 / 166 / 3849.00% (majority class)
ContractNLINemotron-H-8B199 / 166 / 3549.75% (majority class)

We computed the WikiHop random baseline by averaging the reciprocal of the number of answer candidates across examples. The ContractNLI counts follow the order Not mentioned / Entailment / Contradiction. For ContractNLI, we filtered examples by length using each model's tokenizer before sampling, so the sample composition differs slightly across models with different tokenizers. On WikiHop, both Baseline and SERESA consistently outperformed the random baseline across all four models. On Hyperpartisan, Nemotron-H-8B also exceeded the majority-class baseline, with SERESA maintaining comparable accuracy. Although the remaining model–task pairs exhibited limited baseline performance, SERESA consistently preserved accuracy close to the full-recurrence Baseline, indicating that recurrence restriction introduces minimal additional degradation even in these settings.

To directly assess how closely SERESA preserves Baseline predictions, we also report balanced accuracy and per-example correctness agreement.

  • Balanced accuracy is the mean recall across classes and is unaffected by class imbalance. Under uniform random prediction, its expected value is 0.500 on Hyperpartisan and 0.333 on ContractNLI.
  • Per-example correctness agreement is the fraction of examples that Baseline and SERESA both answered correctly or both answered incorrectly. It shows how closely the model's judgments are preserved at the example level after compression.
TaskModelBaseline balanced accuracySERESA balanced accuracyPer-example correctness agreement
WikiHopMamba2-1.3B0.2530.25098.8%
WikiHopMamba2-2.7B0.2730.27098.8%
WikiHopMamba2-8B0.3600.35098.5%
WikiHopNemotron-H-8B0.4330.43096.8%
HyperpartisanMamba2-1.3B0.4980.49696.8%
HyperpartisanMamba2-2.7B0.4940.49598.0%
HyperpartisanMamba2-8B0.5310.52897.0%
HyperpartisanNemotron-H-8B0.7740.77295.0%
ContractNLIMamba2-1.3B0.3580.36099.3%
ContractNLIMamba2-2.7B0.3500.34699.0%
ContractNLIMamba2-8B0.3390.339100%
ContractNLINemotron-H-8B0.4580.47095.8%

For WikiHop, the answer candidates vary across examples, making conventional class-wise balanced accuracy inapplicable. We therefore report standard accuracy.

The results highlight two findings. First, SERESA preserved Baseline performance in settings where Baseline exceeded chance-level performance. On Hyperpartisan and ContractNLI, both Baseline and SERESA with Nemotron-H-8B exceeded the corresponding chance-level balanced accuracy, with differences of at most 0.012 between the two methods. Second, per-example correctness agreement ranged from 95.0% to 100% across all 12 model–task pairs, including those with low Baseline accuracy. These results indicate that SERESA largely preserves Baseline's per-example correctness patterns, even when the underlying model exhibits limited task performance.

AuthorsOpenReview ↗

Response #3

[W1-3] Additional Benchmark Results

Following the reviewer's suggestion, we compared the full-recurrence Baseline and SERESA on ARC-Challenge, ARC-Easy, HellaSwag, PIQA, WinoGrande, and LAMBADA.

ModelLAMBADA PPL ↓LAMBADA Acc.HellaSwag Norm. Acc.PIQAWinoGrandeARC-EasyARC-Challenge Norm. Acc.Mean accuracy
Mamba2-1.3B5.017 → 4.88365.52 → 66.3159.93 → 60.0073.34 → 73.2961.09 → 60.7764.18 → 64.1433.45 → 33.1959.58 → 59.62
Mamba2-2.7B4.093 → 4.04169.53 → 69.9666.58 → 66.5976.44 → 76.2863.61 → 63.7769.44 → 69.5336.35 → 36.4363.66 → 63.76
Mamba2-8B3.618 → 3.60070.87 → 71.1277.65 → 77.6879.82 → 79.8271.82 → 71.9875.97 → 76.0147.87 → 48.0470.67 → 70.78
Nemotron-H-8B3.829 → 3.78866.68 → 67.0380.45 → 80.5580.03 → 80.0973.09 → 73.0982.37 → 82.3757.42 → 57.4273.34 → 73.42

For all four models, the difference in mean accuracy across the six tasks was at most 0.11 percentage points, and LAMBADA perplexity decreased with SERESA in all four models. We added these results to Appendix G of the revised manuscript. Each entry in the table shows Baseline → SERESA, with accuracy reported in percent.

MMLU. We extended our evaluation to MMLU using Mamba2-8B to assess performance preservation. We used lm-evaluation-harness to perform a 5-shot evaluation across all 57 subjects (14,042 questions). SERESA used the same settings as in the main text (78.2% local heads and R=64R=64). We computed overall MMLU accuracy as the weighted average of subject-level accuracies, using the number of questions in each subject as the weight.

CategoryFull-recurrence BaselineSERESAAccuracy difference
Humanities43.40%42.53%−0.87 pp
Other55.23%55.42%+0.19 pp
Social Sciences53.72%53.01%−0.71 pp
STEM38.31%38.31%0.00 pp
Overall MMLU47.14%46.73%−0.41 pp

SERESA's overall MMLU accuracy was 0.41 percentage points below Baseline. The differences across categories ranged from −0.87 to +0.19 percentage points, and SERESA matched Baseline in STEM. These results show that SERESA largely preserves Baseline performance on short-context knowledge and reasoning evaluations as well.

Reviewer_uB25OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SERESA, which reduces the overall prefill latency of Mamba style models by restricting the recurrence window of certain heads that are predisposed to short-context modeling. By using the introduced Effective Recurrence Length (ERL) metric, the authors identify how many heads to restrict, while individual heads are selected based on their overall coverage, which measures the proportion of the recurrent contribution magnitude retained within a certain recurrence window. Thus, heads with high coverage concentrate overall output contribution on recent tokens and thus are restricted to only attend to a prescribed context window, reducing overall computation and subsequently latency. In the experiments, around 80% of all Mamba heads can be restricted, enabling a 2x speedup in the SSD component and an overall 1.3x speedup when combined with quantization with limited quality degradation. Runs span over pure Mamba and hybrid models from the 1B to 8B scale.

Critical strengths

The paper introduces a training-free methodology that reduces the overall computational load of the SSD algorithm, which comprises a meaningful fraction of the prefill latency. The method, which reduces the overall context window of select heads instead of quantizing or restricting the SSM state, improves the overall speed of the SSD forward pass while degrading perplexity measurements less than compared methods. SERESA can be combined with other methods, such as quantization, to further improve speedups.

The ablations in the paper also help show that the head-selection is generally independent of the dataset used to measure coverage and ERL. Furthermore, the ablations on the various restrictions on SSD’s recurrence helps demonstrate that, as expected, the boundary correction is important for recovering performance as tokens within the receptive field may be cut off by the chunk boundary.

Critical weaknesses

There seems to be some missing related works and comparisons with work on length-extrapolation for Mamba models, which have previously found the specialization of local and global heads. The most notable prior work, LongMamba (https://arxiv.org/abs/2504.16053), also identified the differences in receptive fields between local and global heads and introduced a method for determining token filtering based on the cumulative decay. I believe the paper should compare their ERL metric with those of prior work in more depth. And although the paper includes a decay-only head-selection comparison (which I could not find a description on, please specify the details of this ablation), an additional comparison against a method that restricts the head based on its cumulative-decay, motivated by LongMamba, would strengthen the manuscript.

I also firmly believe that the paper would benefit from a more detailed description of the modified SSD kernel algorithms. It does not seem like the implementation details of the boundary correction algorithm is described anywhere, despite a high-level explanation being provided in Section 5.2. From my understanding, it would prevent the standard chunk-state calculation approach within the standard SSD algorithm and would most likely require a modified intra-chunk approach with a custom decay mask. Also explaining the intra-chunk restriction algorithm would be helpful as the most simple change of adjusting the decay mask to be banded should not change the speed significantly, so I assume there is a bit more at play here.

Questions for authors

What is the general modified SSD algorithm used by SERESA?

How does SERESA perform on other benchmarks, e.g., GSM8K, etc, compared to the baseline and compared methods? How does SERESA actually perform when utilizing KV/state caching? Why is autoregressive answer generation performed without this?

How does SERESA’s ERL + coverage technique for determining recurrence length restriction compare to a simple baseline that restricts heads based on a large A_log value or measured cumulative decay?

What are the gains of SERESA on more current hardware, e.g., H100?

7 comments
AuthorsOpenReview ↗

Response #1

Thank you for your careful review and constructive feedback. We appreciate your recognition that SERESA reduces SSD prefill computation by selectively restricting recurrence without additional training and can be combined with quantization to further reduce prefill latency. We also appreciate your assessment of the ablations showing that head selection is consistent across calibration datasets and that boundary correction helps recover model quality.

We address your concerns and questions below. We will include all clarifications and findings from the rebuttal discussion in the final version.

[W1, Q3] Relationship to LongMamba and comparison with decay-based head-selection criteria

LongMamba and SERESA both build on the observation that the effective context span varies across heads. LongMamba uses cumulative decay over the training sequence length to distinguish local from global heads and applies token filtering to global heads on longer inputs. SERESA measures how much of each SSD head's recurrent contribution is retained when recurrence is restricted to a window of length RR. It constructs a per-head distribution of recurrent contributions that accounts for cumulative decay, BB, CC, and the input XX.

To examine how head selection based on cumulative decay, as used in LongMamba, affects perplexity after recurrence restriction, we compared three decay-based head selection criteria. For each criterion, we scored each head, ranked the heads jointly across all layers by score, and selected local heads under the same local-head budget. We applied the same recurrence length (R=64R=64) and boundary correction to the selected heads and compared the perplexity of each criterion with that of SERESA on Pile, WikiText-2, and C4. Below, let Ah=−exp⁡(Alog⁡,h)A_h=-\exp(A_{\log,h}), and let bhb_h denote the learned Δt\Delta t bias for head hh.

  • Decay-only: We compute the static score sh=exp⁡(Ahsoftplus⁡(bh)d)s_h = \exp(A_h \operatorname{softplus}(b_h) d) using only learned parameters, with d=32d=32 as the fixed lag at which decay is evaluated. The score requires no input data or aggregation across tokens or samples. We select heads in ascending order of shs_h. Because the same positive dd is used for every head, its value does not affect the ranking.
  • Alog⁡A_{\log}: This additional criterion uses only the learned Alog⁡,hA_{\log,h} as the score. We select heads with the largest Alog⁡,hA_{\log,h} first, equivalently those with the largest ∣Ah∣\lvert A_h\rvert. The sole difference from decay-only is the omission of the Δt\Delta t bias bhb_h.
  • Input-dependent cumulative decay (LongMamba-inspired): Using the same five Pile calibration sequences of 2,048 tokens each as Coverage, we compute the actual step sizes Δth,t=softplus⁡(Δth,traw+bh)\Delta t_{h,t} = \operatorname{softplus}(\Delta t_{h,t}^{\mathrm{raw}} + b_h) and the cumulative log-decay Ah∑t=12048Δth,tA_h \sum_{t=1}^{2048} \Delta t_{h,t} over the full training sequence length of 2,048 tokens. We average the values across the five sequences to obtain each head's score and select heads in ascending order, starting with those exhibiting the strongest cumulative decay.

We evaluated each criterion on Mamba2-2.7B with 81.25% of the SSD heads selected as local heads, using R=64R=64 and identical boundary correction. The table below reports relative perplexity increases over Baseline.

Head-selection criterionPileWikiText-2C4
Coverage+0.26%+0.12%+0.08%
Alog⁡A_{\log}+0.60%+0.86%+0.47%
Decay-only+1.32%+1.24%+0.59%
Input-dependent cumulative decay+1.89%+1.58%+0.79%

Under these conditions, Coverage yielded smaller perplexity increases than all three decay-based criteria on every dataset. The criterion using actual input-dependent Δt\Delta t values yielded larger perplexity increases than the static decay-only approximation. This comparison evaluates Coverage's head ranking at a fixed local-head budget. Figure 4(b) assesses sensitivity to the budget by varying the local-head ratio.

In response to your suggestion, we replaced the static decay-only baseline in the head-selection ablation (§6.3, Figure 4(a)) with the LongMamba-inspired input-dependent cumulative decay baseline in the revised manuscript. Appendix M provides the scoring and head-selection procedures for all four criteria and analyzes their results under the same local-head budget.

AuthorsOpenReview ↗

Response #2

[W2, Q1] Modified SSD algorithm and kernel implementation

We added pseudocode for the SERESA forward pass of a single SSD layer to Appendix L of the revised manuscript and referenced it in §5.2. SERESA applies the standard SSD path to global heads and the following restricted path to local heads.

  1. Head reordering and omission of inter-chunk state propagation: We reorder the projection weights offline so that global heads are contiguous within each layer. Only global heads compute chunk states, propagate them across chunks, and apply the C⋅stateC\cdot\mathrm{state} operation that incorporates the propagated states into the output. Local heads skip this path.
  2. Reduced intra-chunk computation: At output position ii of a local head, the kernel sums contributions only from input positions jj in the same chunk that satisfy max⁡(0,i−R+1)≤j≤i\max(0,i-R+1)\le j\le i. The kernel restricts its input-tile loop to tiles that intersect this window.
  3. Boundary correction: A separate kernel restores the omitted inter-chunk contributions that satisfy Q+i−j<RQ+i-j<R, where QQ is the chunk size and ii and jj are the output position in the current chunk and the input position in the preceding chunk, respectively. Since R≤QR\le Q, valid terms lie within a triangular region spanning the last R−1R-1 inputs of the preceding chunk and the first R−1R-1 outputs of the current chunk. The kernel computes these contributions and adds them to the local-head outputs.
AuthorsOpenReview ↗

Response #3

[Q2-1] Additional Benchmark Results

GSM8K. We measured exact-match accuracy (%) under the strict-match filter on 100 examples from GSM8K, a generative reasoning benchmark. On this benchmark, Mamba2-1.3B, 2.7B, and 8B achieved near-zero accuracy even with full recurrence, making it difficult to meaningfully assess the impact of recurrence restriction. We therefore report results for Nemotron-H-8B, comparing the full-recurrence baseline, SERESA, SSDi8, and GHOST in the table below.

ModelBaselineSSDi8GHOSTSERESA
Nemotron-H-8B88.088.080.088.0

SERESA and SSDi8 both matched the Baseline accuracy of 88.0%, while GHOST achieved 80.0%, 8.0 percentage points below Baseline. These results indicate that SERESA's selective recurrence restriction and SSDi8's quantization preserved multistep reasoning accuracy in this setting, while GHOST's state-dimension pruning resulted in a relatively large drop in accuracy.

Additional zero-shot evaluation. We also compared the performance of Baseline, SERESA, SSDi8, and GHOST across four models on six zero-shot benchmarks (LAMBADA, HellaSwag, PIQA, WinoGrande, ARC-Easy, and ARC-Challenge) and added the results to Appendix G of the revised manuscript. Following standard practice, we report length-normalized accuracy for HellaSwag and ARC-Challenge and standard accuracy for the remaining tasks, with all scores expressed as percentages.

ModelMethodLAMBADAHellaSwag (norm.)PIQAWinoGARC-eARC-c (norm.)Mean accuracyΔ vs. Baseline
Mamba2-1.3BBaseline65.559.973.361.164.233.459.58–
SSDi864.659.973.260.564.033.259.22−0.36
GHOST51.558.073.058.664.031.856.14−3.44
SERESA66.360.073.360.864.133.259.62+0.04
Mamba2-2.7BBaseline69.566.676.463.669.436.363.66–
SSDi868.966.576.164.269.236.963.65−0.01
GHOST61.164.575.463.969.036.361.70−1.96
SERESA70.066.676.363.869.536.463.76+0.10
Mamba2-8BBaseline70.977.779.871.876.047.970.67–
SSDi870.777.279.571.376.247.570.41−0.26
GHOST61.274.679.867.575.043.566.92−3.75
SERESA71.177.779.872.076.048.070.78+0.11
Nemotron-H-8BBaseline66.780.580.073.182.457.473.34–
SSDi867.079.978.873.082.956.172.95−0.39
GHOST64.078.578.669.181.456.471.33−2.01
SERESA67.080.680.173.182.457.473.42+0.08

SERESA maintained or slightly improved mean accuracy relative to Baseline across all four models, with differences ranging from +0.04 to +0.11 percentage points. SSDi8 showed decreases in mean accuracy of 0.01–0.39 percentage points, and GHOST showed decreases of 1.96–3.75 percentage points. Averaged across the four models, the differences in mean accuracy relative to Baseline were +0.08 percentage points for SERESA, −0.26 for SSDi8, and −2.79 for GHOST. These results show that SERESA preserves Baseline accuracy better than SSDi8 and GHOST on the evaluated zero-shot benchmarks, and that selectively restricting recurrence has little effect on general language understanding and reasoning performance.

Reviewer_Ma7dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an inefficiency in Mamba-2-style State Space Duality (SSD): all SSD heads perform full recurrence even though their useful temporal ranges may differ substantially. The authors define an effective recurrence distribution from the magnitude of recurrent contributions at different lags and summarize it using Effective Recurrence Length (ERL). Based on the ERL distribution, SERESA divides heads into local and global groups. Local heads retain only contributions within a fixed recurrence length, while global heads preserve full recurrence. The implementation removes unnecessary inter- and intra-chunk computation and restores short cross-chunk contributions through boundary correction. Across three Mamba-2 sizes and Nemotron-H-8B, SERESA restricts recurrence in approximately 78–83% of SSD heads, keeps perplexity close to the full-recurrence baseline, and reports 1.75–2.02x SSD-core speedups on an A5000. Additional experiments cover Mamba-3, long contexts up to 32K, different batch sizes and sequence lengths, and an A100.

Critical strengths
  1. The core insight is clear and useful. The paper shows that long-range recurrent behavior is concentrated in a minority of SSD heads and turns this observation into a simple policy: keep full recurrence for a small set of global heads and use restricted recurrence for the remaining local heads. This is easy to understand and directly connected to the proposed acceleration.
  2. The ablations support the proposed mechanism. At the same local-head ratio, the ERL/coverage criterion performs better than decay-only, learned, magnitude-based, and random selection. The component ablation shows that inter- and intra-chunk restrictions provide the speed gain, while boundary correction recovers most of the quality loss. Varying the recurrence length also gives a reasonable quality–speed trade-off.
  3. The SSD-core results are broad and mostly consistent. The main experiments cover four model configurations and three language-modeling datasets. The method remains effective across batch sizes and sequence lengths, and it achieves a 1.582x SSD-core speedup on an A100. The reported timing includes boundary correction, and the authors also report that the realized speedup depends on the GPU and native kernel.
Critical weaknesses
  1. The practical end-to-end improvement is more limited than the SSD-core result. SERESA gives roughly 2x SSD-core acceleration on the main hardware, but only about 1.1x end-to-end speedup by itself. The reported 1.287x end-to-end result combines SERESA with quantization, while quantization alone already gives 1.202x. Appendix E provides the general kernel-timing environment and measurement procedure, but the paper does not tie the motivation figures—4.74% of FLOPs, 3.81% of memory, and nearly 20% of prefill latency—to a specific model and input configuration or provide a profiling breakdown supporting them.
  2. ERL is not a complete measure of a head’s importance to the final block output. The contribution is measured at the SSD output and normalized separately for each head. This removes absolute scale across heads and does not include the later gated normalization and output projection. The experiments show that the criterion works, but statements that long-range “output-relevant contribution” is concentrated in a few heads should be qualified as conclusions about the proposed normalized statistic.
  3. Several evaluation details needed to judge the small quality differences are missing. The paper does not report the exact splits, number of sequences or tokens, or evaluation stride for the perplexity results. The downstream evaluation keeps only examples that fit within 2K tokens, but the remaining sample counts and class distributions are not given. Table 2 and the passkey experiment also lack uncertainty estimates or trial counts.
  4. The novelty and baseline comparison should be stated more carefully. The high-level local/global-head idea is conceptually related to DuoAttention’s retrieval/streaming-head split. The main new contribution is the recurrence-specific ERL/coverage criterion and efficient SSD implementation with boundary correction. In addition, SSDi8 and GHOST are evaluated at only a few operating points, so the results do not establish a complete quality–latency frontier.
Questions for authors
  1. Can the authors provide exact end-to-end latency for SERESA alone, quantization alone, and their combination, ideally across the main model sizes and both GPUs? Please also identify the exact model, batch size, sequence length, precision, and measurement definitions used for the FLOP, memory, and prefill-latency percentages in the introduction, and provide the corresponding profiling breakdown.
  2. What data splits and numbers of tokens or sequences are used for the perplexity evaluations? After the 2K filter, how many examples remain for each downstream task, and how many trials are used for each passkey result?
  3. Is ERL computed before the Mamba-2 gated normalization and output projection? If so, have the authors tested a criterion that includes the gate, output projection, absolute head scale, or direct loss sensitivity?
  4. Was the default recurrence length R=64R=64 selected on data separate from the reported test results? Which hyperparameters were fixed before the final evaluation?
  5. For the calibration-size experiment, are the n=1n=1 and n=5n=5 samples subsets of the n=20n=20 reference set, and are the reported similarities based on one draw or repeated independent draws?
6 comments
AuthorsOpenReview ↗

Response #1

Thank you for your careful review and constructive feedback. We appreciate your recognition of our central finding that long-range recurrent contributions are concentrated in a small subset of SSD heads, and of the ablation results supporting our head-selection criterion and boundary correction. We address your concerns and questions below. We will reflect all clarifications and findings from the rebuttal discussion in the final version.

[W1, Q1] End-to-end prefill speedup

SSD latency share and end-to-end prefill speedup. To validate the extent to which SSD-core acceleration translates into end-to-end prefill speedup, we evaluated three additional Mamba-2 models on the RTX A5000 at a batch size of 16 and an input length of 2,048 tokens. The arrows indicate the change from the full-recurrence baseline to SERESA alone.

ModelSSD share of baseline prefill latencySSD latency (ms)SSD speedupEnd-to-end prefill latency (ms)End-to-end prefill speedupRelative change in WikiText-2 PPL
130M39.4%66.4 → 33.12.006×168.4 → 138.11.219×+0.75%
370M35.0%177.6 → 86.62.051×508.2 → 431.91.177×+0.46%
780M29.0%268.0 → 131.02.046×924.8 → 821.01.126×+0.12%

Across the three models, SERESA achieved SSD-core speedups of 2.01–2.05×, and end-to-end prefill speedup increased from 1.126× to 1.219× as the SSD share of baseline prefill latency increased from 29.0% to 39.4%. This indicates that the end-to-end gains from SSD-core optimization depend on the share of prefill latency attributable to SSD. We therefore expect SERESA to yield more pronounced end-to-end prefill speedups in future models or workloads where SSD accounts for a greater fraction of prefill latency.

End-to-end prefill latency on the RTX A5000 and A100. To further address the question about end-to-end prefill latency, we measured end-to-end prefill latency for Mamba2-1.3B, Mamba2-2.7B, and Mamba2-8B on the RTX A5000 and A100 at a batch size of 16 and an input length of 2,048 tokens, following the benchmarking setup and timing procedure described in the paper. On each GPU, we used the same input shape and timing procedure for the full-recurrence baseline, SERESA alone, projection quantization alone, and their combination.

GPUModelFull-recurrence baseline (ms)SERESA only (ms)Quantization only (ms)Quantization + SERESA (ms)
RTX A5000Mamba2-1.3B1460.61314.8 (1.111×)1354.9 (1.078×)1260.2 (1.159×)
RTX A5000Mamba2-2.7B2919.92649.6 (1.102×)2431.3 (1.201×)2272.5 (1.285×)
RTX A5000Mamba2-8B5775.55442.3 (1.061×)4438.7 (1.301×)4195.9 (1.376×)
A100 SXM4 80GBMamba2-1.3B535.2464.3 (1.153×)420.6 (1.272×)357.9 (1.495×)
A100 SXM4 80GBMamba2-2.7B1018.9904.6 (1.126×)799.1 (1.275×)685.8 (1.486×)
A100 SXM4 80GBMamba2-8B2040.11918.7 (1.063×)1333.3 (1.530×)1198.4 (1.702×)

SERESA alone reduced end-to-end prefill latency for all three Mamba-2 models on both the RTX A5000 and the A100. Relative to quantization alone, adding SERESA reduced prefill latency by a further 5.5–7.0% on the RTX A5000 and 10.1–14.9% on the A100. These results confirm that SERESA achieves speedups without degradation when combined with quantization of modules outside the SSD, both in the tested RTX A5000 and A100 settings.

AuthorsOpenReview ↗

Response #2

[W1, Q1] Component-wise profiling

Profiling setup and measurement definitions for the Introduction. We profiled the full-recurrence state-spaces/mamba2-2.7b model in FP16 on an NVIDIA RTX A5000 with a batch size of 8, an input length of 2,048 tokens, a chunk size of 256, and num_last_tokens=1.

  • FLOPs (4.74%): We counted FLOPs by operation across all 64 layers. The SSD count includes chunk-state computation, inter-chunk state passing, intra-chunk computation, and the CB⊤CB^\top product. SSD accounted for 4.20 of the model's 88.5 TFLOPs. The input and output projections accounted for approximately 95.2% of total FLOPs.
  • Peak allocated GPU memory (3.81%): We measured peak allocated GPU memory for the baseline forward pass and for a forward pass in which we replaced SSD computation with an output tensor of the same shape. We divided the difference by the baseline peak to estimate the effect of SSD activations and temporary buffers on peak allocated memory.
  • Prefill latency (19.9%): We used CUDA events to time 50 runs after 15 warm-up runs. The end-to-end prefill forward pass averaged 1,415 ms, and the SSD kernels (chunk_cumsum, bmm, chunk_state, state_passing, and chunk_scan) accounted for 282 ms in total.
Selected operations and end-to-end measurementLatency (ms)
Input projection (in_proj)674
Output projection (out_proj)330
SSD kernels (total)282
Normalization (norm)45
1D convolution (conv1d)37
End-to-end prefill forward pass1,415

We added these measurement definitions and component-wise profiling results to Appendix E of the revised manuscript.

AuthorsOpenReview ↗

Response #3

[W2, Q3] Scope of ERL and alternative head-selection criteria

We compute ERL from recurrent contributions within SSD, before Mamba-2's gated normalization and output projection. For each head, we sum the magnitudes of recurrent contributions at each temporal lag and divide by the total contribution magnitude for that head. ERL is the mean lag of the resulting distribution, which allows us to compare the temporal extent of recurrent contributions across heads with different output scales.

Our claim about the concentration of long-range contributions is limited to the temporal distribution of SSD recurrent contributions under per-head normalization. We clarified in the revised manuscript that “output-relevant contribution” refers to recurrent contributions within SSD.

Regarding absolute head scale and direct loss sensitivity, §6.3 presents an ablation study comparing SERESA against two alternative head-selection criteria. The y-magnitude criterion ranks heads by absolute output magnitude. The learning-based criterion uses language-modeling loss to learn a selection gate for each head by minimizing cross-entropy with L1 penalty on the gate values, which pushes heads toward the local mode unless the loss resists. We have evaluated both alternatives at the same local-head ratio (81.25%) and recurrence length (R=64R=64) as SERESA. SERESA yielded smaller perplexity increases than both alternatives on Pile, WikiText-2, and C4.

[W3, Q2] Evaluation protocols, sample sizes, and uncertainty

Based on your comment, we added the data splits, sample sizes, and evaluation protocols to Appendix E of the revised manuscript and specified the number of passkey trials per sequence length in the caption of Table 17.

Perplexity evaluation. For WikiText-2, C4, and Pile, we used non-overlapping windows with a length and stride of 2,048 tokens. We evaluated each window independently after resetting the recurrent state. Baseline and SERESA used identical evaluation data and window boundaries. We computed perplexity by exponentiating the mean negative log-likelihood over all scored tokens. The window and input-token counts below are based on the GPT-NeoX tokenizer used by Mamba2-1.3B and Mamba2-2.7B.

DatasetSplit and document selectionWindowsInput tokens
WikiText-2Entire wikitext-2-raw-v1 test split, with two newline characters between documents140286,720
C4First 1,100 documents from the allenai/c4 English validation shard c4-validation.00000-of-00008256524,288
PileFirst 524,288 tokens after concatenating documents in file order from the monology/pile-uncopyrighted validation file val.jsonl.zst256524,288

Downstream evaluation. For Table 2, given the reported degradation in Mamba's long-context performance [1], we retained only examples whose complete input, including the context, prompt, and longest answer candidate, fit within 2,048 tokens without truncation. The table below reports the number of eligible examples under each model's tokenizer. From this eligible pool, we randomly sampled 400 examples for each model–task pair using a fixed random seed, evaluating Baseline and SERESA on the same examples.

Task and splitExamples before filteringMamba2-1.3B and 2.7BMamba2-8BNemotron-H-8B
WikiHop (QAngaroo train)43,73827,44827,31227,019
Hyperpartisan (SemEval-2019 by-article)645597595592
ContractNLI (SCROLLS validation)1,037457492459

Passkey evaluation. As specified in the revised caption, the accuracies in Table 17 were computed based on two trials, with 300 examples per sequence length in each trial.

[1] Azizi, Seyedarmin, et al. "Mambaextend: A training-free approach to improve long context extension of mamba." International Conference on Learning Representations. Vol. 2025. 2025.

Comments on the paper

3 comments
Area_Chair_AqLSOpenReview ↗

Reminder to reviwers to read author's responses

Dear Reviewers,

Thank you for your efforts in providing thorough reviews.

As the discussion period is underway, I would like to kindly remind you to read the authors' responses to your reviews and provide follow-up comments where appropriate or leave a comment. In particular, please consider whether the responses address your questions and concerns, and update your assessment if the clarifications or additional information affect your initial evaluation.

Your engagement in this discussion is important for ensuring a fair and thorough review process.

Thank you again for your time and contributions!

Best regards, Area Chair

AuthorsOpenReview ↗

General Response #2

3. What ERL measures and why we use Coverage (Ma7d W2/W4/Q3, uB25 W1/Q3)

ERL is the mean lag of the per-head distribution of recurrent contribution magnitudes within SSD, measured before gated normalization and output projection. The revised manuscript clarifies this scope and discusses the relationship to DuoAttention, which assigns KV-cache policies to Transformer heads. On Mamba2-2.7B, with the local-head ratio fixed at 81.25%, a 64-token window, and identical boundary correction, Coverage yielded smaller perplexity increases on Pile, WikiText-2, and C4 than all three decay-based criteria: Alog⁡A_{\log} alone, static decay, and LongMamba-inspired input-dependent cumulative decay. Relative increases across the three datasets were 0.08–0.26% with Coverage and 0.79–1.89% with input-dependent cumulative decay. At the same local-head budget, Coverage also yielded smaller increases than selection by absolute output magnitude or by a gate trained on language-modeling loss. We added forward-pass pseudocode in Appendix L and the decay-based scoring procedures in Appendix M.

4. Mamba-3 and reproducibility (Ma7d Q4/Q5, uB25 Q2, vifG W2/Q2/Q3)

On Mamba3-SISO-1.5B, SERESA achieved a 1.27× SSD-core speedup and a 1.07× end-to-end prefill speedup relative to our full-recurrence SISO implementation under identical conditions, with only a 0.78% relative increase in WikiText-2 perplexity. In a separate sweep, the 72.54% local-head ratio selected by the mean-ERL rule fell on the rising portion of the perplexity curve (Appendix H). In state-cached generation on Mamba2-2.7B and 8B, SERESA reduced prefill latency, and per-token decoding latency differed from the baseline by less than 0.5%. We selected the default 64-token window on a Pile validation subset separate from head calibration and final evaluation, and held each model’s head partition fixed during evaluation. Upon acceptance, we will release the kernels, including the local-head path and boundary correction, along with the calibration and benchmark scripts and run configurations.

Building on the changes already made to the revised manuscript, we will include the additional experiments and clarifications from the rebuttal discussion in the final version.

AuthorsOpenReview ↗

General Response #1

We thank all three reviewers for the time they spent evaluating our work and for their constructive comments. Our head-wise analysis of SSD shows that, under per-head normalization, recurrent contributions from distant tokens are concentrated in a minority of heads. SERESA uses Effective Recurrence Length (ERL) to determine how many heads to restrict and Coverage to select those that retain the largest fraction of their recurrent contribution within the window. Without additional training, SERESA skips inter-chunk state propagation and reduces intra-chunk computation for local heads. Boundary correction restores cross-chunk contributions within the window. In our main experiments, SERESA restricted recurrence in approximately 78–83% of SSD heads and achieved 1.75–2.02× SSD-core speedups on the RTX A5000 while keeping perplexity close to the full-recurrence baseline.

1. End-to-end prefill speedups across GPUs (Ma7d W1/Q1, uB25 Q4, vifG Q1)

SERESA alone accelerated prefill for all six Mamba-2 models on the RTX A5000, H100, and RTX PRO 6000 Blackwell. It also reduced prefill latency for all three models evaluated on the A100.

Mamba-2 modelRTX A5000A100H100RTX PRO 6000 Blackwell
130M1.219×—1.24×1.189×
370M1.177×—1.23×1.163×
780M1.126×—1.19×1.115×
1.3B1.111×1.153×1.15×1.095×
2.7B1.102×1.126×1.17×1.071×
8B1.061×1.063×1.09×1.045×

On the RTX A5000 at batch size 16 and 2,048 input tokens, SERESA alone achieved 1.219×, 1.177×, and 1.126× end-to-end prefill speedups on the 130M, 370M, and 780M models, respectively. SSD-core speedups were similar across the three models (2.01–2.05×). End-to-end gains were larger where SSD accounted for more of baseline prefill latency, with shares of 39.4%, 35.0%, and 29.0%, respectively.

Across the 1.3B, 2.7B, and 8B models, adding SERESA to projection quantization reduced prefill latency by a further 5.5–7.0% on the RTX A5000 and 10.1–14.9% on the A100 compared with projection quantization alone. The revised abstract reports both results for Mamba2-2.7B on the RTX A5000 against the full-recurrence baseline: 1.10× with SERESA alone and 1.29× with SERESA and projection quantization. Appendix E provides the profiling details, and Appendix K reports the H100 and Blackwell results.

2. Zero-shot benchmarks, reasoning, and long-context retrieval (Ma7d W3/W4, uB25 Q2, vifG W1)

We expanded the zero-shot evaluation to LAMBADA, HellaSwag, PIQA, WinoGrande, ARC-Easy, and ARC-Challenge. SERESA’s mean accuracy across the six tasks stayed within 0.11 percentage points of full recurrence on all four models.

ModelFull-recurrence mean accuracySERESA Δ (pp)SSDi8 Δ (pp)GHOST Δ (pp)
Mamba2-1.3B59.58%+0.04−0.36−3.44
Mamba2-2.7B63.66%+0.10−0.01−1.96
Mamba2-8B70.67%+0.11−0.26−3.75
Nemotron-H-8B73.34%+0.08−0.39−2.01
  • 5-shot MMLU. We evaluated Mamba2-8B on 14,042 questions across 57 subjects. Overall accuracy went from 47.14% to 46.73%. Changes across subject categories ranged from −0.87 to +0.19 percentage points. STEM accuracy was 38.31% in both configurations.
  • GSM8K. On 100 examples with Nemotron-H-8B, full recurrence, SERESA, and SSDi8 all achieved 88.0% strict exact-match accuracy, while GHOST achieved 80.0%.
  • Matched downstream comparison. We evaluated all methods on the same 400 examples per model–task pair. Mean absolute accuracy differences from full recurrence were 0.54 percentage points for SERESA, 1.13 for SSDi8 INT8, and 2.10 for GHOST with 50% state-dimension pruning. All 12 confidence intervals for SERESA’s accuracy differences included zero at the 95% confidence level.
  • Long-context retrieval in pure Mamba-2. On RULER NIAH at 4K and 8K, SERESA stayed within 1.0 percentage point of full recurrence on Mamba2-1.3B and 2.7B. On Mamba2-8B at 8K, accuracy was 84.5% with full recurrence, 97.5% with Coverage-based selection, and 0% when the same number of heads was selected at random. All three full-recurrence baselines scored 0% at 16K, so we evaluated contexts beyond 16K using Nemotron-H-8B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.