- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ESSH, a hybrid of selective spectral mixers and sliding-window attention (SWA) that is trained once and exported at several sizes. The mixer turns each Hankel spectral channel of elastic spectral SSMs (ES-SSM) into an independent recurrent unit with an input-dependent decay and write/read gates, so whole channels can be truncated at export. Training reduces the spectral-channel count and the FFN width at different rates and distills the smaller tiers from the full model. At the 1.5B and 370M scales, the paper reports full-capacity quality comparable to independently trained models, a better quality–size trade-off than MatFormer and MatMamba, retained retrieval in compact tiers, and 2.14–2.80× faster batch-one decoding than Mamba-2 and Mamba-3.
Critical strengths
- Train once, export many, at a meaningful scale. One 1.5B, 100B-token run yields ten tiers, and all ten exports match the shared model bitwise. This makes the deployment story concrete.
- Broad ablations. The paper ablates the selective clock and gates, the initialization and channel order, the mode and channel budgets, the capacity map, sampling and distillation, and also compares against post-hoc elastification. Many natural questions already have at least a partial answer.
- A fast decoding implementation. At matched parameter counts, ESSH has the lowest decode latency at every measured batch size on B300. The paper backs this with kernel counts and a same-architecture comparison of fused and unfused decoding.
Critical weaknesses
W1. The mixer overlaps with gated complex-diagonal recurrences, and the Hankel structure acts mainly as an initialization. Each mode's transition (Sec. 4.1, Eq. 5), an input-dependent decay times a learned fixed rotation, has the same form as the HGRN recurrence [1], and related recurrences such as LRU [2] and HGRN2 [3] are not cited in the related work (Sec. 2). The paper's own ablations suggest that the Hankel spectrum mainly serves as an initialization: the 8-mode fit is poor beyond the leading channels (relative error 0.75 at ; App. L.2, Table 28), S4D-Lin initialization does as well or slightly better (App. L.3, Table 29), shuffling the channel slots barely changes PPL (Table 30), and random or non-oscillatory modes give an equally strong channel-importance ordering once nested training is used (Table 31). The evidence therefore supports "structured initialization plus nested training" rather than the Hankel basis as the source of the ordered elastic axis claimed in the abstract and Sec. 1, and the novelty relative to HGRN-style recurrences combined with MatMamba-style nesting needs to be argued directly.
W2. The comparisons that would isolate the two-rate elastic axis are confounded. The frontier comparison with MatFormer and MatMamba (Sec. 5.2, Fig. 3(a)) changes the backbone (a hybrid with three SWA layers vs. full attention or pure Mamba-2; Table 9) and the recipe (ten sampled capacities with distillation and refinement vs. four capacities with neither; App. D.1, Tables 8–9, App. N.1) at the same time, and each change alone is large: Mamba-3+SWA reaches 14.10 PPL against 16.98 for full MatMamba (Table 10), and removing distillation raises ESSH's T1 PPL from 25.32 to 26.42 (Sec. 5.2, App. M.2). Because truncating spectral channels alone still leaves 91.2% of the parameters (App. M.1), most of the size reduction across tiers comes from FFN width, which is MatFormer's axis. An FFN-only elastic ESSH at matched sizes, and if feasible a head-nested Mamba-3+SWA trained with the same recipe, would show whether the spectral-channel axis of contribution 2 (Sec. 1, Fig. 1(C)) adds anything.
W3. Missing comparison with ES-SSM. The paper's main motivation is that the LTI filters of ES-SSM cannot retain or forget selectively (abstract; Sec. 1; Sec. 3, Eq. 3), yet ES-SSM does not appear in any experiment: it is absent from Table 1 and from the baseline list in Table 8. The selectivity ablations in Table 27 (App. L.1) remove the clock or the gates one at a time and keep the fitted-mode recurrence, so they are not equivalent to ES-SSM. A comparison with an ES-SSM mixer in the same backbone and training recipe is needed to support the motivation.
W4. Retrieval is not tested beyond 2K tokens. NIAH is evaluated only at sequence lengths up to 2048 tokens (App. E.4; Table 18 in App. H.2 covers 512–2048), which equals the SWA window (Table 7), so the key never leaves the attention window; the 2K–16K sweep reports only PPL (Fig. 3(d), App. H.3). Are there NIAH results at longer contexts, where the key has to be kept in the recurrent state?
W5. The decoding speedup over Mamba is not a like-for-like comparison. Only the ESSH decode step is compiled with torch.compile; the baseline steps are not (App. E.5). In the same-architecture A100 control, the unfused ESSH (7.40 ms) is slower than Mamba-2+FFN (4.44 ms), and fusion alone gives a 2.45× gain, larger than ESSH's margin over Mamba-2 (1.45×) (App. P.4, Table 52). The 2.14–2.80× speedup reported in the abstract and Sec. 5.3 (Fig. 4, Table 59) should be re-measured with the baselines compiled or fused comparably, or reported together with these conditions.
Additional feedback (not part of my assessment). Several decisive results, including the S4D-Lin initialization (Table 29), the same-architecture specialists (Table 11), post-hoc elastification (App. N.4–N.6) and Mamba-2 prefill (App. R.3), appear only in the 78-page appendix and deserve a summary table in the main text. Tables 44–45 should also report PPL, numbers for the same control come from different evaluators and checkpoint stages across tables (e.g., Tables 27, 29 and 30) and would benefit from consistent labels, and seed ranges for the 7-task means would show whether one-point differences exceed noise.
References
- [1] Qin et al., Hierarchically Gated Recurrent Neural Network for Sequence Modeling, NeurIPS 2023.
- [2] Orvieto et al., Resurrecting Recurrent Neural Networks for Long Sequences, ICML 2023.
- [3] Qin et al., HGRN2: Gated Linear RNNs with State Expansion, COLM 2024.
Questions for authors
- At matched exported sizes at 370M, what are the T1/T4/T7 PPL and 7-task means of an FFN-only elastic ESSH that keeps all spectral channels, compared with the two-rate ESSH?
- How does a one-shot elastic Mamba-3+SWA with MatMamba-style head nesting, the same FFN map, distillation and refinement compare with the ESSH frontier?
- With
torch.compileor fused step kernels applied to the Mamba-2/3 decode steps under the same CUDA-graph setting, how much of the batch-one latency gap remains?
7 comments
Response to Reviewer bQ5d (Part 2)
3. (Q3, W5) Question / concern
With torch.compile or fused step kernels applied to the Mamba-2/3 decode steps under the same CUDA-graph setting, how much of the batch-one latency gap remains?
Answer. We reran the parameter-matched 1.53B decode benchmarks on all three GPUs with torch.compile for every model and the same CUDA-graph policy. The results below use the trained ESSH attention layout and a 2K cache. ESSH retains a decode advantage on H200 and B300, while batch-one ESSH and Mamba-2 are effectively tied on A100:
| Batch-one decode, ms/token | A100 | H200 | B300 |
|---|---|---|---|
| ESSH | 3.03 | 1.65 | 1.35 |
| Mamba-2 | 3.04 | 1.73 | 1.52 |
| Mamba-3 SISO | 3.48 | 2.13 | 1.94 |
| Mamba-3 MIMO | 3.74 | 2.35 | 2.19 |
The new B300 batch-one speedups are 1.12$\times$ over Mamba-2, 1.43$\times$ over SISO and 1.62$\times$ over MIMO, using unrounded medians. These replace the original timings.
We also remeasured fused and unfused ESSH on A100 at the same 1.531B architecture, with both execution paths compiled:
| Compiled ESSH path | B=1 decode (ms/step) | B=16 decode (ms/step) |
|---|---|---|
| Fused | 3.04 | 3.92 |
| Unfused | 3.64 | 4.58 |
Fusion gives 1.20$\times$ and 1.17$\times$ speedups at batches 1 and 16. ESSH shares one projected value across the modes in each channel and combines projection, state update and readout in the fused path. The paired control shows a measurable execution benefit after compilation is applied to both paths.
We have incorporated this timing interpretation into Section 5.3 and Appendices P.3–P.4. Figure 4 and Appendix R give the updated benchmarks, and Table 66 reports the paired implementation control.
4. (W1) Question / concern
What evidence supports the Hankel basis as the source of the ordered elastic axis, rather than structured initialization followed by nested training? How should the claim account for the 8-mode fit error of 0.75 at , the comparable or better S4D-Lin results, the small effect of shuffling channel slots, and the strong ordering learned by random or non-oscillatory modes?
Answer. First, we agree that the original Tables 29–30 (Tables 34–35 in the revision) could not establish Hankel initialization as the source of the learned predictive ordering. The small PPL change after slot shuffling and the strong ordering learned by random-mode controls show that capacity-mixed training learns useful ordering from different initial arrangements, so we attribute this ordering to the training procedure. The relative fit error of 0.75 at channel shows that eight modes do not closely reproduce every initial filter. The fit initializes trainable modes, while valid channel truncation follows from their independent recurrent states.
Regarding our original claim, we would like to revise it as: In the tested ESSH setting, Hankel-based initialization improves early compact-tier training-token efficiency.
This revised claim is supported by newly added results, which provide training-curve comparisons at 370M on FineWeb-Edu with 3.7B-token capacity-mixed pretraining, budget dropout and full-tier distillation. We compare the tokens needed to reach the same PPL:
| Initialization | Seeds in curves | Tokens to PPL 50 (B) | PPL at 0.39B | PPL at 0.79B |
|---|---|---|---|---|
| Hankel | 2 | 0.40 | 50.4 | 37.3 |
| S4D-Lin | 3 | 0.45 | 54.7 | 38.1 |
| Random modes | 3 | 0.44 | 53.7 | 38.4 |
The training curves show faster early compact-tier optimization with Hankel initialization (Figure 11 and Table 38). Hankel reaches PPL 50 after 0.40B tokens, compared with 0.44–0.45B for the alternatives, and it remains ahead at 0.79B tokens (37.3 versus 38.1–38.4). After full capacity-mixed training, S4D-Lin gives similar compact-tier PPL and slightly lower full-model PPL.
We also added a 370M dense-pretraining control on the same 3.7B FineWeb-Edu tokens: only the full model is pretrained, and compact tiers are introduced during the same 2060-step shared refinement. Across two seeds, Hankel gives PPL 45.53 versus 63.77 for S4D-Lin, while full-model PPL is close (16.06 versus 15.97). Thus initialization affects compact-model recovery even when dense pretraining reaches similar full-model quality.
We have clarified the Hankel attribution in Section 2 and Appendices L.3–L.4. Tables 33–36 retain the original controls, and Figure 11 and Tables 38–39 give new early-training and recovery evidence.
Response to Reviewer bQ5d (Part 3)
5. (W1) Question / concern
What is novel relative to HGRN's gated complex recurrence and related models such as LRU and HGRN2, combined with MatMamba-style nesting? How does the transition differ from HGRN's recurrence? Can the related-work discussion cite and directly position the method against HGRN, LRU and HGRN2?
Answer. Yes. Thank you so much for this comment. LRU, HGRN and HGRN2 laid important foundations for modern recurrent sequence modeling, and Section 2 now cites them and positions ESSH against each. ESSH organizes this recurrence into truncatable spectral-channel groups and restructures its computation for chunked parallel training and fused decoding.
- LRU and HGRN. ESSH shares HGRN's gated complex transition, whereas LRU's transition is input-independent. ESSH learns its write gate independently of retention instead of HGRN's coupling. Both models also gate the readout, HGRN through its output gate and ESSH through the per-mode read gate .
- HGRN2. HGRN2 expands a feature-matrix state and supports multihead partitioning. ESSH groups rotations around a shared value , so each removable channel retains several temporal responses in state values.
- MatMamba and Mamba-2. MatMamba nests Mamba-2 inner dimensions and heads, and each head keeps an input-dependent scalar transition. ESSH instead nests multi-mode spectral channels and reduces FFN width faster than channel count. This keeps more temporal channels while shrinking the FFN weights, which dominate the variable parameter count.
We have incorporated these methodological distinctions into Section 2 of the revised paper.
Response to Reviewer bQ5d (Part 1)
We sincerely appreciate the thorough review process and the constructive comments.
Questions and concerns below are condensed from the review. Q/W labels follow the review's numbering.
1. (Q1, W2) Question / concern
At matched exported sizes at 370M, what are the T1/T4/T7 PPL and seven-task means of an FFN-only elastic ESSH that keeps all spectral channels, compared with the two-rate ESSH?
Answer. The two-rate model reaches smaller sizes and reduces recurrent state and latency. We added the requested FFN-only control in the 370M hybrid, trained on 3.7B FineWeb-Edu tokens with the same capacity-mixed recipe and shared refinement. ESSH values are three-seed means. FFN-only uses seed 197:
FFN-only cannot reach 92.1M because its non-FFN parameters alone occupy 98.4M. Its smallest trained export is 124.9M. Near this size, ESSH gives PPL 21.67 at 118.9M versus 22.25. New batch-one measurements compile both models and use the same 2K cache. Latency is 0.97 versus 1.14 ms on A100 and 0.63 versus 0.75 ms on B300.
Keeping all channels benefits FFN-only at the intermediate tiers. Reducing them instead gives ESSH access to smaller models and lowers recurrent state from 1.09 to 0.27 MB at T1. This is the deployment trade-off behind using two capacity axes.
We have incorporated this interpretation beside the new FFN-only comparison in Appendix M.1, Table 41. Section 5.2 explains the parameter floor.
2. (Q2, W2) Question / concern
How does a one-shot elastic Mamba-3+SWA with MatMamba-style head nesting, the same FFN map, distillation and refinement compare with the ESSH frontier?
Answer. ESSH has better compact-tier PPL and lower recurrent-state cost under the matched recipe. The larger Mamba-3 tiers have slightly better PPL. We added 370M-scale head-nested Mamba-3+SWA runs on 3.7B FineWeb-Edu tokens, matching the SWA layout, depth, residual width, FFN map, capacity sampling, full-tier distillation and shared refinement. Head counts and FFN widths match all four ESSH sizes, down to 92M:
At 92M, ESSH has about 2.1 lower PPL and uses 0.27 versus 1.90 MB of recurrent state. At full size, the states are 1.09 versus 17.70 MB. This difference follows from the smaller bank of two-coordinate modes used for each ESSH value dimension, versus Mamba-3's expanded state. Both retain the same 22.02 MB attention cache.
We have clarified this frontier interpretation in Section 5.2 and Appendix N.2. Table 52 contains the new parameter-aligned results.