- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper observes that in Mamba-2 most SSD heads put their output-relevant recurrent contribution at short time lags, and that the ordering of heads by this range is stable across input domains. Based on this, SERESA is a training-free method: a one-time calibration splits the heads into local heads, which keep only the contributions within a lag of R = 64, and global heads, which keep the full recurrence. On Mamba2-1.3B/2.7B/8B and Nemotron-H-8B, restricting 78–83% of the heads gives a 1.75–2.02× speedup of the SSD core with perplexity close to the baseline.
Critical strengths
- Clear analysis and a simple, training-free method. The head-wise recurrence length is heavy-tailed in all models tested, and the head partition from a single 2K-token sequence overlaps strongly with the one from 20 sequences.
- Informative ablations. Coverage-based head selection clearly beats selecting by decay alone or by a learned gate (perplexity increase of 0.1–0.3% against 0.6–1.7%), and the boundary correction recovers most of the quality lost by skipping inter-chunk recurrence.
- Favorable equal-speed comparison and transparent reporting. At about 1.27× SSD speedup on Mamba2-2.7B, SERESA matches the baseline perplexity (9.06 on WikiText-2), while GHOST at 25% pruning gives 9.65 and SSDi8 gives 9.13. The paper also reports the less favorable facts, such as the lower speedup on A100.
Critical weaknesses
W1. The evidence that quality is preserved beyond perplexity is limited, and there are no long-context or retrieval results in the main paper. Table 2 compares SERESA only with the unmodified model. SSDi8 and GHOST, which Table 1 compares against, are missing, so the table cannot show whether SERESA's downstream accuracy is better than theirs. The differences from the baseline are within 1.8 points (the largest drop is 1.0 point), with no sample counts or intervals, so the table neither shows a clear advantage nor a clear absence of degradation. In addition, Table 2 only uses samples of at most 2K tokens (Appendix E), so these long-document tasks do not test long-range behavior. All three tasks are scored by log-likelihood (Appendix E), yet several baseline numbers are near or below chance (Mamba2-8B on the binary Hyperpartisan task: 45.3%; Mamba2-1.3B and 2.7B on the three-class ContractNLI: 13.0% and 17.3%); please report the class balance or a majority-class baseline. Standard zero-shot benchmarks such as ARC-Challenge, ARC-Easy, HellaSwag, PIQA and WinoGrande, and MMLU for the 8B models, would be more informative than the current tasks. The main text contains no long-context or retrieval evaluation at all; it only states that SERESA "maintains long-context performance, as shown in Appendix I". Appendix I covers only Nemotron-H-8B, whose attention layers can handle retrieval, so it does not show that the long-range ability of pure SSM heads is preserved, which is where restricting the recurrence is most likely to hurt. No retrieval test is run on Mamba-2, even within its trained length. Since prefill acceleration matters most for long inputs, these results should be in the main paper and include pure Mamba-2 models.
W2. The claim that the budget rule is "conservative" is shown on only one model. The paper chooses the number of local heads as the number of heads whose recurrence length is below the mean, and says this gives a "conservative" boundary without a performance-based search. The rule is applied to all four models in Table 1, but the evidence that it stays before the point where perplexity starts to rise is a local-head ratio sweep on Mamba2-2.7B only (Figure 4(b); 81.25% of the heads restricted, perplexity increase of 0.1–0.3%). On Mamba-3 (Appendix H, Table 12), the same rule restricts fewer heads (about 73%) but raises perplexity by about 1.0–1.1% on Pile and WikiText-2 (0.4% on C4), several times more than on Mamba-2. It is therefore unclear whether the rule stays on the safe side of that point on other architectures; a similar sweep on Mamba-3 would settle this.
Questions for authors
- The abstract reports up to 1.29× end-to-end prefill speedup with projection quantization. According to Sec. 6.2, on Mamba2-2.7B quantization alone gives 1.202× and adding SERESA raises it to 1.287×, so SERESA adds about 7% on top of quantization, while SERESA alone gives about 1.1×. Could the abstract also state the SERESA-only end-to-end speedup?
- The paper says the code will be released upon acceptance. Will it include the SERESA kernels (the local-head path with the boundary correction) and the calibration scripts, so that the reported speedups can be reproduced?
- Are there speed measurements on Mamba-3? Appendix H reports only perplexity for Mamba-3 (SISO and MIMO, 1.5B). Is the method, in particular the kernel, limited to Mamba-2 models, or does it also apply to Mamba-3, and what speedup does it give there?
9 comments
Response #2
[W1-2] Evaluation Scope, Class Baselines, and Per-Example Agreement
For each task in Table 2, we report the class distribution of the evaluation sample and the accuracy baseline. We added the total number of eligible examples and the class distributions to Appendix E of the revised manuscript.
| Task | Model | Final evaluation sample () | Accuracy baseline |
|---|---|---|---|
| WikiHop | All models | 2–30 answer candidates per example, with a mean of 10.3 | 17.2% (random selection) |
| Hyperpartisan | All models | 262 Not hyperpartisan / 138 Hyperpartisan | 65.5% (majority class) |
| ContractNLI | Mamba2-1.3B and 2.7B | 201 / 162 / 37 | 50.25% (majority class) |
| ContractNLI | Mamba2-8B | 196 / 166 / 38 | 49.00% (majority class) |
| ContractNLI | Nemotron-H-8B | 199 / 166 / 35 | 49.75% (majority class) |
We computed the WikiHop random baseline by averaging the reciprocal of the number of answer candidates across examples. The ContractNLI counts follow the order Not mentioned / Entailment / Contradiction. For ContractNLI, we filtered examples by length using each model's tokenizer before sampling, so the sample composition differs slightly across models with different tokenizers.
On WikiHop, both Baseline and SERESA consistently outperformed the random baseline across all four models. On Hyperpartisan, Nemotron-H-8B also exceeded the majority-class baseline, with SERESA maintaining comparable accuracy. Although the remaining model–task pairs exhibited limited baseline performance, SERESA consistently preserved accuracy close to the full-recurrence Baseline, indicating that recurrence restriction introduces minimal additional degradation even in these settings.
To directly assess how closely SERESA preserves Baseline predictions, we also report balanced accuracy and per-example correctness agreement.
- Balanced accuracy is the mean recall across classes and is unaffected by class imbalance. Under uniform random prediction, its expected value is 0.500 on Hyperpartisan and 0.333 on ContractNLI.
- Per-example correctness agreement is the fraction of examples that Baseline and SERESA both answered correctly or both answered incorrectly. It shows how closely the model's judgments are preserved at the example level after compression.
| Task | Model | Baseline balanced accuracy | SERESA balanced accuracy | Per-example correctness agreement |
|---|---|---|---|---|
| WikiHop | Mamba2-1.3B | 0.253 | 0.250 | 98.8% |
| WikiHop | Mamba2-2.7B | 0.273 | 0.270 | 98.8% |
| WikiHop | Mamba2-8B | 0.360 | 0.350 | 98.5% |
| WikiHop | Nemotron-H-8B | 0.433 | 0.430 | 96.8% |
| Hyperpartisan | Mamba2-1.3B | 0.498 | 0.496 | 96.8% |
| Hyperpartisan | Mamba2-2.7B | 0.494 | 0.495 | 98.0% |
| Hyperpartisan | Mamba2-8B | 0.531 | 0.528 | 97.0% |
| Hyperpartisan | Nemotron-H-8B | 0.774 | 0.772 | 95.0% |
| ContractNLI | Mamba2-1.3B | 0.358 | 0.360 | 99.3% |
| ContractNLI | Mamba2-2.7B | 0.350 | 0.346 | 99.0% |
| ContractNLI | Mamba2-8B | 0.339 | 0.339 | 100% |
| ContractNLI | Nemotron-H-8B | 0.458 | 0.470 | 95.8% |
For WikiHop, the answer candidates vary across examples, making conventional class-wise balanced accuracy inapplicable. We therefore report standard accuracy.
The results highlight two findings. First, SERESA preserved Baseline performance in settings where Baseline exceeded chance-level performance. On Hyperpartisan and ContractNLI, both Baseline and SERESA with Nemotron-H-8B exceeded the corresponding chance-level balanced accuracy, with differences of at most 0.012 between the two methods. Second, per-example correctness agreement ranged from 95.0% to 100% across all 12 model–task pairs, including those with low Baseline accuracy. These results indicate that SERESA largely preserves Baseline's per-example correctness patterns, even when the underlying model exhibits limited task performance.
Response #3
[W1-3] Additional Benchmark Results
Following the reviewer's suggestion, we compared the full-recurrence Baseline and SERESA on ARC-Challenge, ARC-Easy, HellaSwag, PIQA, WinoGrande, and LAMBADA.
| Model | LAMBADA PPL ↓ | LAMBADA Acc. | HellaSwag Norm. Acc. | PIQA | WinoGrande | ARC-Easy | ARC-Challenge Norm. Acc. | Mean accuracy |
|---|---|---|---|---|---|---|---|---|
| Mamba2-1.3B | 5.017 → 4.883 | 65.52 → 66.31 | 59.93 → 60.00 | 73.34 → 73.29 | 61.09 → 60.77 | 64.18 → 64.14 | 33.45 → 33.19 | 59.58 → 59.62 |
| Mamba2-2.7B | 4.093 → 4.041 | 69.53 → 69.96 | 66.58 → 66.59 | 76.44 → 76.28 | 63.61 → 63.77 | 69.44 → 69.53 | 36.35 → 36.43 | 63.66 → 63.76 |
| Mamba2-8B | 3.618 → 3.600 | 70.87 → 71.12 | 77.65 → 77.68 | 79.82 → 79.82 | 71.82 → 71.98 | 75.97 → 76.01 | 47.87 → 48.04 | 70.67 → 70.78 |
| Nemotron-H-8B | 3.829 → 3.788 | 66.68 → 67.03 | 80.45 → 80.55 | 80.03 → 80.09 | 73.09 → 73.09 | 82.37 → 82.37 | 57.42 → 57.42 | 73.34 → 73.42 |
For all four models, the difference in mean accuracy across the six tasks was at most 0.11 percentage points, and LAMBADA perplexity decreased with SERESA in all four models. We added these results to Appendix G of the revised manuscript. Each entry in the table shows Baseline → SERESA, with accuracy reported in percent.
MMLU. We extended our evaluation to MMLU using Mamba2-8B to assess performance preservation. We used lm-evaluation-harness to perform a 5-shot evaluation across all 57 subjects (14,042 questions). SERESA used the same settings as in the main text (78.2% local heads and ). We computed overall MMLU accuracy as the weighted average of subject-level accuracies, using the number of questions in each subject as the weight.
| Category | Full-recurrence Baseline | SERESA | Accuracy difference |
|---|---|---|---|
| Humanities | 43.40% | 42.53% | −0.87 pp |
| Other | 55.23% | 55.42% | +0.19 pp |
| Social Sciences | 53.72% | 53.01% | −0.71 pp |
| STEM | 38.31% | 38.31% | 0.00 pp |
| Overall MMLU | 47.14% | 46.73% | −0.41 pp |
SERESA's overall MMLU accuracy was 0.41 percentage points below Baseline. The differences across categories ranged from −0.87 to +0.19 percentage points, and SERESA matched Baseline in STEM. These results show that SERESA largely preserves Baseline performance on short-context knowledge and reasoning evaluations as well.
Response #1
Thank you for your careful review and constructive feedback. We appreciate your recognition of the head-wise recurrence analysis underlying SERESA's training-free design and the stability of the head partition even with a single 2K-token calibration sequence. We also appreciate your assessment of the ablations supporting Coverage-based head selection and boundary correction, as well as the baseline comparisons at comparable SSD-core speedups. We address your concerns and questions below. We will include the clarifications and findings from the rebuttal discussion in the final version.
[W1-1] Downstream Accuracy Compared with SSDi8 and GHOST
To assess how SERESA's accuracy compares with that of SSDi8 and GHOST on the downstream tasks in Table 2, we additionally evaluated SSDi8 INT8 and GHOST with 50% state-dimension pruning on the same evaluation examples. The table below reports accuracy (%) for each method. Parentheses give the accuracy difference from the full-recurrence Baseline and its 95% confidence interval, both in percentage points.
Relative to the full-recurrence baseline, SERESA's accuracy differences ranged from −1.00 to +1.75 percentage points (pp), and the 95% confidence intervals included zero for all 12 model–task pairs. The mean absolute differences from Baseline were 0.54 pp for SERESA, 1.13 pp for SSDi8 INT8, and 2.10 pp for GHOST at 50% pruning. The corresponding maximum absolute differences were 1.75, 2.75, and 8.25 pp. SERESA also had the narrowest mean 95% confidence interval for the accuracy difference (3.3 pp, compared with 4.4 pp for SSDi8 and 5.4 pp for GHOST). For GHOST, the 95% confidence interval excluded zero in two pairs, both on ContractNLI: +5.00 pp [1.52, 8.69] for Mamba2-2.7B and −8.25 pp [−11.55, −5.54] for Nemotron-H-8B.