acceptodds
Under review as a conference paper at ICLR 2027

Elastic Selective Spectral Hybrids for Train-Once, Export-Many Budgeted Inference

Abstract

Deploying a language model under different computing and latency budgets calls for compact models with different quality-cost trade-offs. Towards this end, elastic spectral state space models provide ordered and temporally decomposed channels that can be truncated, but they are associated with linear time-invariant filters that cannot selectively preserve relevant past information or forget irrelevant information as the context evolves. To address this, we introduce the Elastic Selective Spectral Hybrid (ESSH), which realizes each Hankel spectral channel as an independent recurrent unit using fitted damped rotation modes. It also features an input-dependent decay and write/read gates that make temporal retention and state update input-dependent while preserving channel-wise truncation and a structured recurrence for efficient execution. ESSH combines these selective spectral mixers with sliding-window attention and jointly trains multiple capacities by reducing spectral-channel count and feed-forward width at different rates through a two-rate capacity map with full-model distillation. The resulting models support chunked parallel training and fused recurrent decoding while avoiding computation for discarded channels. At full capacity, ESSH achieves language-modeling quality comparable to similarly sized independently trained models, while smaller exports exhibit a smooth quality-cost trade-off. We validate the effectiveness of the proposed framework using language understanding, retrieval, cross-domain text, and DNA experiments, by assessing quality retention and the trade-off against independently trained and elastic baselines. At the 1.53B model configuration, fused batch-one decoding takes 1.37 ms per token on a B300, providing a - speedup over the tested Mamba-2 and Mamba-3 implementations and over Transformer++ at matched parameter counts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
state space models, Hankel spectral filtering, elastic neural networks, budgeted inference, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bQ5dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ESSH, a hybrid of selective spectral mixers and sliding-window attention (SWA) that is trained once and exported at several sizes. The mixer turns each Hankel spectral channel of elastic spectral SSMs (ES-SSM) into an independent recurrent unit with an input-dependent decay and write/read gates, so whole channels can be truncated at export. Training reduces the spectral-channel count and the FFN width at different rates and distills the smaller tiers from the full model. At the 1.5B and 370M scales, the paper reports full-capacity quality comparable to independently trained models, a better quality–size trade-off than MatFormer and MatMamba, retained retrieval in compact tiers, and 2.14–2.80× faster batch-one decoding than Mamba-2 and Mamba-3.

Critical strengths
  1. Train once, export many, at a meaningful scale. One 1.5B, 100B-token run yields ten tiers, and all ten exports match the shared model bitwise. This makes the deployment story concrete.
  2. Broad ablations. The paper ablates the selective clock and gates, the initialization and channel order, the mode and channel budgets, the capacity map, sampling and distillation, and also compares against post-hoc elastification. Many natural questions already have at least a partial answer.
  3. A fast decoding implementation. At matched parameter counts, ESSH has the lowest decode latency at every measured batch size on B300. The paper backs this with kernel counts and a same-architecture comparison of fused and unfused decoding.
Critical weaknesses

W1. The mixer overlaps with gated complex-diagonal recurrences, and the Hankel structure acts mainly as an initialization. Each mode's transition ρe−Δ(t)R(θ)\rho e^{-\Delta(t)} R(\theta) (Sec. 4.1, Eq. 5), an input-dependent decay times a learned fixed rotation, has the same form as the HGRN recurrence λtexp⁡(iθ)\lambda_t \exp(i\theta) [1], and related recurrences such as LRU [2] and HGRN2 [3] are not cited in the related work (Sec. 2). The paper's own ablations suggest that the Hankel spectrum mainly serves as an initialization: the 8-mode fit is poor beyond the leading channels (relative error 0.75 at k=16k = 16; App. L.2, Table 28), S4D-Lin initialization does as well or slightly better (App. L.3, Table 29), shuffling the channel slots barely changes PPL (Table 30), and random or non-oscillatory modes give an equally strong channel-importance ordering once nested training is used (Table 31). The evidence therefore supports "structured initialization plus nested training" rather than the Hankel basis as the source of the ordered elastic axis claimed in the abstract and Sec. 1, and the novelty relative to HGRN-style recurrences combined with MatMamba-style nesting needs to be argued directly.

W2. The comparisons that would isolate the two-rate elastic axis are confounded. The frontier comparison with MatFormer and MatMamba (Sec. 5.2, Fig. 3(a)) changes the backbone (a hybrid with three SWA layers vs. full attention or pure Mamba-2; Table 9) and the recipe (ten sampled capacities with distillation and refinement vs. four capacities with neither; App. D.1, Tables 8–9, App. N.1) at the same time, and each change alone is large: Mamba-3+SWA reaches 14.10 PPL against 16.98 for full MatMamba (Table 10), and removing distillation raises ESSH's T1 PPL from 25.32 to 26.42 (Sec. 5.2, App. M.2). Because truncating spectral channels alone still leaves 91.2% of the parameters (App. M.1), most of the size reduction across tiers comes from FFN width, which is MatFormer's axis. An FFN-only elastic ESSH at matched sizes, and if feasible a head-nested Mamba-3+SWA trained with the same recipe, would show whether the spectral-channel axis of contribution 2 (Sec. 1, Fig. 1(C)) adds anything.

W3. Missing comparison with ES-SSM. The paper's main motivation is that the LTI filters of ES-SSM cannot retain or forget selectively (abstract; Sec. 1; Sec. 3, Eq. 3), yet ES-SSM does not appear in any experiment: it is absent from Table 1 and from the baseline list in Table 8. The selectivity ablations in Table 27 (App. L.1) remove the clock or the gates one at a time and keep the fitted-mode recurrence, so they are not equivalent to ES-SSM. A comparison with an ES-SSM mixer in the same backbone and training recipe is needed to support the motivation.

W4. Retrieval is not tested beyond 2K tokens. NIAH is evaluated only at sequence lengths up to 2048 tokens (App. E.4; Table 18 in App. H.2 covers 512–2048), which equals the SWA window (Table 7), so the key never leaves the attention window; the 2K–16K sweep reports only PPL (Fig. 3(d), App. H.3). Are there NIAH results at longer contexts, where the key has to be kept in the recurrent state?

W5. The decoding speedup over Mamba is not a like-for-like comparison. Only the ESSH decode step is compiled with torch.compile; the baseline steps are not (App. E.5). In the same-architecture A100 control, the unfused ESSH (7.40 ms) is slower than Mamba-2+FFN (4.44 ms), and fusion alone gives a 2.45× gain, larger than ESSH's margin over Mamba-2 (1.45×) (App. P.4, Table 52). The 2.14–2.80× speedup reported in the abstract and Sec. 5.3 (Fig. 4, Table 59) should be re-measured with the baselines compiled or fused comparably, or reported together with these conditions.

Additional feedback (not part of my assessment). Several decisive results, including the S4D-Lin initialization (Table 29), the same-architecture specialists (Table 11), post-hoc elastification (App. N.4–N.6) and Mamba-2 prefill (App. R.3), appear only in the 78-page appendix and deserve a summary table in the main text. Tables 44–45 should also report PPL, numbers for the same control come from different evaluators and checkpoint stages across tables (e.g., Tables 27, 29 and 30) and would benefit from consistent labels, and seed ranges for the 7-task means would show whether one-point differences exceed noise.

References

  • [1] Qin et al., Hierarchically Gated Recurrent Neural Network for Sequence Modeling, NeurIPS 2023.
  • [2] Orvieto et al., Resurrecting Recurrent Neural Networks for Long Sequences, ICML 2023.
  • [3] Qin et al., HGRN2: Gated Linear RNNs with State Expansion, COLM 2024.
Questions for authors
  1. At matched exported sizes at 370M, what are the T1/T4/T7 PPL and 7-task means of an FFN-only elastic ESSH that keeps all spectral channels, compared with the two-rate ESSH?
  2. How does a one-shot elastic Mamba-3+SWA with MatMamba-style head nesting, the same FFN map, distillation and refinement compare with the ESSH frontier?
  3. With torch.compile or fused step kernels applied to the Mamba-2/3 decode steps under the same CUDA-graph setting, how much of the batch-one latency gap remains?
7 comments
AuthorsOpenReview ↗

Response to Reviewer bQ5d (Part 1)

We sincerely appreciate the thorough review process and the constructive comments.

Questions and concerns below are condensed from the review. Q/W labels follow the review's numbering.

1. (Q1, W2) Question / concern

At matched exported sizes at 370M, what are the T1T_1/T4T_4/T7T_7 PPL and seven-task means of an FFN-only elastic ESSH that keeps all spectral channels, compared with the two-rate ESSH?

Answer. The two-rate model reaches smaller sizes and reduces recurrent state and latency. We added the requested FFN-only control in the 370M hybrid, trained on 3.7B FineWeb-Edu tokens with the same capacity-mixed recipe and shared refinement. ESSH values are three-seed means. FFN-only uses seed 197:

Size target / modelParams (M)PPL ↓\downarrow7-task (%) ↑\uparrow
T1T_1, ESSH92.125.3436.08
T1T_1, FFN-onlyno export––
T4T_4, ESSH185.319.1939.02
T4T_4, FFN-only185.418.7039.51
T7T_7, ESSH274.317.4339.97
T7T_7, FFN-only276.217.1640.33
T10T_{10}, ESSH370.916.2740.82
T10T_{10}, FFN-only370.916.2941.18

FFN-only cannot reach 92.1M because its non-FFN parameters alone occupy 98.4M. Its smallest trained export is 124.9M. Near this size, ESSH gives PPL 21.67 at 118.9M versus 22.25. New batch-one measurements compile both models and use the same 2K cache. Latency is 0.97 versus 1.14 ms on A100 and 0.63 versus 0.75 ms on B300.

Keeping all channels benefits FFN-only at the intermediate tiers. Reducing them instead gives ESSH access to smaller models and lowers recurrent state from 1.09 to 0.27 MB at T1T_1. This is the deployment trade-off behind using two capacity axes.

We have incorporated this interpretation beside the new FFN-only comparison in Appendix M.1, Table 41. Section 5.2 explains the parameter floor.

2. (Q2, W2) Question / concern

How does a one-shot elastic Mamba-3+SWA with MatMamba-style head nesting, the same FFN map, distillation and refinement compare with the ESSH frontier?

Answer. ESSH has better compact-tier PPL and lower recurrent-state cost under the matched recipe. The larger Mamba-3 tiers have slightly better PPL. We added 370M-scale head-nested Mamba-3+SWA runs on 3.7B FineWeb-Edu tokens, matching the SWA layout, depth, residual width, FFN map, capacity sampling, full-tier distillation and shared refinement. Head counts and FFN widths match all four ESSH sizes, down to 92M:

3.7B tokens, after refinementT1T_1 PPL ↓\downarrowT4T_4 PPL ↓\downarrowT7T_7 PPL ↓\downarrowT10T_{10} PPL ↓\downarrow
ESSH, three-seed mean25.3419.1917.4316.27
Head-nested Mamba-3+SWA, two-seed mean27.4319.4417.3116.06

At 92M, ESSH has about 2.1 lower PPL and uses 0.27 versus 1.90 MB of recurrent state. At full size, the states are 1.09 versus 17.70 MB. This difference follows from the smaller bank of two-coordinate modes used for each ESSH value dimension, versus Mamba-3's expanded state. Both retain the same 22.02 MB attention cache.

We have clarified this frontier interpretation in Section 5.2 and Appendix N.2. Table 52 contains the new parameter-aligned results.

AuthorsOpenReview ↗

Response to Reviewer bQ5d (Part 2)

3. (Q3, W5) Question / concern

With torch.compile or fused step kernels applied to the Mamba-2/3 decode steps under the same CUDA-graph setting, how much of the batch-one latency gap remains?

Answer. We reran the parameter-matched 1.53B decode benchmarks on all three GPUs with torch.compile for every model and the same CUDA-graph policy. The results below use the trained ESSH attention layout and a 2K cache. ESSH retains a decode advantage on H200 and B300, while batch-one ESSH and Mamba-2 are effectively tied on A100:

Batch-one decode, ms/token ↓\downarrowA100H200B300
ESSH3.031.651.35
Mamba-23.041.731.52
Mamba-3 SISO3.482.131.94
Mamba-3 MIMO3.742.352.19

The new B300 batch-one speedups are 1.12$\times$ over Mamba-2, 1.43$\times$ over SISO and 1.62$\times$ over MIMO, using unrounded medians. These replace the original timings.

We also remeasured fused and unfused ESSH on A100 at the same 1.531B architecture, with both execution paths compiled:

Compiled ESSH pathB=1 decode (ms/step) ↓\downarrowB=16 decode (ms/step) ↓\downarrow
Fused3.043.92
Unfused3.644.58

Fusion gives 1.20$\times$ and 1.17$\times$ speedups at batches 1 and 16. ESSH shares one projected value across the modes in each channel and combines projection, state update and readout in the fused path. The paired control shows a measurable execution benefit after compilation is applied to both paths.

We have incorporated this timing interpretation into Section 5.3 and Appendices P.3–P.4. Figure 4 and Appendix R give the updated benchmarks, and Table 66 reports the paired implementation control.

4. (W1) Question / concern

What evidence supports the Hankel basis as the source of the ordered elastic axis, rather than structured initialization followed by nested training? How should the claim account for the 8-mode fit error of 0.75 at k=16k=16, the comparable or better S4D-Lin results, the small effect of shuffling channel slots, and the strong ordering learned by random or non-oscillatory modes?

Answer. First, we agree that the original Tables 29–30 (Tables 34–35 in the revision) could not establish Hankel initialization as the source of the learned predictive ordering. The small PPL change after slot shuffling and the strong ordering learned by random-mode controls show that capacity-mixed training learns useful ordering from different initial arrangements, so we attribute this ordering to the training procedure. The relative fit error of 0.75 at channel k=16k=16 shows that eight modes do not closely reproduce every initial filter. The fit initializes trainable modes, while valid channel truncation follows from their independent recurrent states.

Regarding our original claim, we would like to revise it as: In the tested ESSH setting, Hankel-based initialization improves early compact-tier training-token efficiency.

This revised claim is supported by newly added results, which provide training-curve comparisons at 370M on FineWeb-Edu with 3.7B-token capacity-mixed pretraining, budget dropout and full-tier distillation. We compare the tokens needed to reach the same T1T_1 PPL:

InitializationSeeds in curvesTokens to T1T_1 PPL 50 (B) ↓\downarrowT1T_1 PPL at 0.39B ↓\downarrowT1T_1 PPL at 0.79B ↓\downarrow
Hankel20.4050.437.3
S4D-Lin30.4554.738.1
Random modes30.4453.738.4

The training curves show faster early compact-tier optimization with Hankel initialization (Figure 11 and Table 38). Hankel reaches T1T_1 PPL 50 after 0.40B tokens, compared with 0.44–0.45B for the alternatives, and it remains ahead at 0.79B tokens (37.3 versus 38.1–38.4). After full capacity-mixed training, S4D-Lin gives similar compact-tier PPL and slightly lower full-model PPL.

We also added a 370M dense-pretraining control on the same 3.7B FineWeb-Edu tokens: only the full model is pretrained, and compact tiers are introduced during the same 2060-step shared refinement. Across two seeds, Hankel gives T1T_1 PPL 45.53 versus 63.77 for S4D-Lin, while full-model PPL is close (16.06 versus 15.97). Thus initialization affects compact-model recovery even when dense pretraining reaches similar full-model quality.

We have clarified the Hankel attribution in Section 2 and Appendices L.3–L.4. Tables 33–36 retain the original controls, and Figure 11 and Tables 38–39 give new early-training and recovery evidence.

AuthorsOpenReview ↗

Response to Reviewer bQ5d (Part 3)

5. (W1) Question / concern

What is novel relative to HGRN's gated complex recurrence and related models such as LRU and HGRN2, combined with MatMamba-style nesting? How does the transition ρe−Δ(t)R(θ)\rho e^{-\Delta(t)}R(\theta) differ from HGRN's λtexp⁡(iθ)\lambda_t\exp(i\theta) recurrence? Can the related-work discussion cite and directly position the method against HGRN, LRU and HGRN2?

Answer. Yes. Thank you so much for this comment. LRU, HGRN and HGRN2 laid important foundations for modern recurrent sequence modeling, and Section 2 now cites them and positions ESSH against each. ESSH organizes this recurrence into truncatable spectral-channel groups and restructures its computation for chunked parallel training and fused decoding.

  1. LRU and HGRN. ESSH shares HGRN's gated complex transition, whereas LRU's transition is input-independent. ESSH learns its write gate ηw\eta^{\mathrm w} independently of retention instead of HGRN's 1−λt1-\lambda_t coupling. Both models also gate the readout, HGRN through its output gate and ESSH through the per-mode read gate ηr\eta^{\mathrm r}.
  2. HGRN2. HGRN2 expands a feature-matrix state and supports multihead partitioning. ESSH groups mm rotations around a shared value vk(t)∈RPv_k(t)\in\mathbb R^P, so each removable channel retains several temporal responses in 2mP2mP state values.
  3. MatMamba and Mamba-2. MatMamba nests Mamba-2 inner dimensions and heads, and each head keeps an input-dependent scalar transition. ESSH instead nests multi-mode spectral channels and reduces FFN width faster than channel count. This keeps more temporal channels while shrinking the FFN weights, which dominate the variable parameter count.

We have incorporated these methodological distinctions into Section 2 of the revised paper.

Reviewer_t77BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper developed a hybrid (transformer + reccurent attention aka mixer) language model whose recurrent mixer is made of 32 'spectral channels'. Each channel is a small recurrent unit of 8 damped-rotation modes, initialized from 'Hankel spectral filters', with an input-dependent decay and input-dependent write/read gates like SSM. They introduce input-dependent decay and the rotation terms as in Eq. 6 and since the decay and the rotation term are commutative, the model can be trained with a chunked parallel scan and decoded with a fused recurrent step. The model is trained once and exported at different sizes (according to the budget) by keeping fewer channels and a narrower FFN (as in Eq. 7), using capacity sampling, distillation from the full model, and a short refinement stage. Experiments include a 1.53B model trained on 100B tokens and also a 370M study against elastic and non-elastic baselines, retrieval and long-context probes, ablations, and decode benchmarks on A100, H200 and B300, etc..

Critical strengths

The math and method is clearly specified and the derivations both in main text and in the Appendix. The engineering is pretty good! They developed custom kernels and did numerical verification (Table 49). The information is complete, reporting seeds, compute hours. Some honest results are also shown that go against the paper's own story, for example a plain S4D-Lin initialization does slightly better than the Hankel one (Table 30)

Critical weaknesses
  1. The spectral framing, which I believe is the one of the main claim, is not supported by the paper's own ablations.

Table 30 shows that replacing the Hankel initialization with S4D-Lin initialization lowers PPL at both the full tier (−0.11) and the smallest tier (−0.05), and that shuffling the channel order costs only +0.05 and +0.30. Appendix L.3 states that random-mode and shuffled-slot models "also develop strong ordering under budget dropout." So the ordering that makes truncation work comes from the training recipe, not from the Hankel basis. The method is better described as nested selective heads plus FFN nesting with distillation, which is close to MatMamba's head nesting; the paper should position it that way.

  1. The closest prior work is not compared and the selectivity is not isolated

ES-SSM (Song et al. 2026) is the method the abstract says it improves on, but it is not a baseline anywhere.

The two ablations in Table 27 each remove only part of the selectivity (clock or gates, not both), and their effects are modest: +0.34 and +0.63 PPL at full capacity against a three-seed spread of 0.09. An LTI arm in the same hybrid and recipe would settle the question.

  1. The elastic baselines use a weaker recipe

Table 9 and Appendix N.1 show MatFormer and MatMamba were trained without distillation (KL = 0), with four capacities instead of ten, and MatMamba kept one quarter of micro-batches at full width versus ESSH's 0.75. Figure 3(a) therefore cannot separate ESSH's architecture from its recipe.

  1. The quality cost of elasticity is understated

If the paper's selling point is “train one model, get many sizes for free.” The reader needs to know how much quality do you give up compared with just training a separate model at each size? The paper never shows this clearly, I feel.

In Tabl-1a, The 7-task means are ESSH 55.0, Transformer++ 55.4, Mamba-2 55.7, Gated DeltaNet 55.8, Mamba-3 SISO 56.4, Mamba-3 MIMO 57.6. The paper’s text only compares to the two closest ones (55.4 and 55.7) and calls it “comparable.”

Soeme more concerns related to this point are: the main text's strongest ablation, "45.66 vs 25.32" (Sec. 5.2), shrinks the dense model by plain truncation without importance ranking (i.e. without choosing which units to keep), while the properly ranked post-hoc recovery in Table 46 ties ESSH on PPL (17.35 vs 17.38 at T7). The ranked method is not reported at T1, so the 45.66 comparison is not the fair one.

The bolded PPL 10.11 in Table-1a, isn’t comparable. The caption says “PPL uses each source’s validation shard” — meaning ESSH’s perplexity was measured on a different slice of data than the baselines’, so a lower number doesn’t mean a better model. Bolding it as “best” is misleading.

  1. The speed claims are mostly an implementation comparison, and some unfavorable results are left out of the main text

Appendix E.5 says the ESSH decode step is compiled with torch.compile, but the baseline decode steps are not. This matters: in the uncompiled ("eager") column of Table 59, ESSH takes 3.62 ms and Mamba-2 takes 15.18 ms, which tells us the baselines are slow mainly because they launch many small GPU kernels, not because of their architecture. The paper itself explains the speedup by kernel counts (367 for ESSH vs 963 for Mamba-2). So the headline "2.14–3.03× faster decoding" is largely a result of how the code was written, not of the model design.

There are three further points on efficiency. First, Figure 4(d) shows prefill speed only against Transformer++, but Tables 51, 58 and 60 show that Mamba-2 is faster than ESSH at prefill at every context length on every device (for example 222.7 vs 88.0 ktok/s at 128K on B300); this should be in the main text. Second, ESSH also trains more slowly than Transformer++ on A100 (40.66 vs 57.87 ktok/s, Tables 47–48). Third, shrinking the model saves little at inference time on B300: the smallest tier T1 removes 69% of the parameters, yet decode time only drops from 1.37 ms to 1.13 ms, and memory for state plus KV cache only drops from 53.61 MB to 51.15 MB. The main saving from truncation is weight memory, not speed.

  1. Retrieval evidence is too noisy to carry the claims built on it

Three seeds of the same recipe give T4 full-key accuracy of 0.249, 0.030 and 0.134 (Table 29). This is a spread of about 0.22; the caption itself says "retrieval varies substantially across seeds." That spread is larger than the gaps the paper highlights in Table 1(c), ESSH 0.389 vs Transformer++ 0.350 and vs Mamba-3+SWA 0.230, and all of those are single-seed numbers.

At depth 0.1 the 1.5B tiers score only 0.004–0.051 (Table 18) even though the 2048-token attention window covers the whole 2048-token context, so the metric mostly measures recency.

In the synthetic suite, the mixer alone scores at most 0.128 on MQAR while Mamba-3 reaches 1.000 at 8 pairs (Table 21), and the paper states that "mixer-only models remain near zero" (App. I); recall therefore comes from the attention layer, not the proposed mixer.

Questions for authors
  1. Given Tables 29–30, what claim about the Hankel basis do you still make, and which experiment supports it?
  2. Can you add an LTI arm (fixed clock and fixed gates) and, ideally, ES-SSM inside the same hybrid and recipe at 370M, with two seeds?
  3. Can you train MatMamba with the ESSH recipe (ten capacities, f = 0.75, KL = 0.5, shared refinement), or ESSH with the MatMamba recipe?
  4. Please add the 440M and 880M rows from Lahoti et al. to Table 1 and un-bold the PPL.
  5. Please re-time decode with the same compile and graph policy for all models, and add Mamba-2 and Mamba-3 to Figure 4(d).
  6. Why does the two-layer Transformer baseline score 0.022 on 32-pair MQAR at L = 256? Was it tuned?
  7. The paper in general was not an easy read because of too much engineering type details, please work on the flow without distracting into nitty gritty.
5 comments
AuthorsOpenReview ↗

Response to Reviewer t77B (Part 1)

We sincerely appreciate the thorough review process and the constructive comments.

Questions and concerns below are condensed from the review. Q/W labels follow the review's numbering.

1. (Q1, W1) Question / concern

Given Tables 29–30, what claim about the Hankel basis do you still make, and which experiment supports it?

Answer. First, we agree that the original Tables 29–30 (Tables 34–35 in the revision) could not establish a consistent final-quality advantage for Hankel initialization. Instead, these tables support the effectiveness of the architecture and training procedure across multiple initializations.

Regarding our original claim, we would like to revise it as: In the tested ESSH setting, Hankel-based initialization improves early compact-tier training-token efficiency.

This revised claim is supported by newly added results, which provide training-curve comparisons at 370M on FineWeb-Edu with 3.7B-token capacity-mixed pretraining, budget dropout and full-tier distillation. We compare the tokens needed to reach the same T1T_1 PPL:

InitializationSeeds in curvesTokens to T1T_1 PPL 50 (B) ↓\downarrowT1T_1 PPL at 0.39B ↓\downarrowT1T_1 PPL at 0.79B ↓\downarrow
Hankel20.4050.437.3
S4D-Lin30.4554.738.1
Random modes30.4453.738.4

The training curves show faster early compact-tier optimization with Hankel initialization (Figure 11 and Table 38). Hankel reaches T1T_1 PPL 50 after 0.40B tokens, compared with 0.44–0.45B for the alternatives, and it remains ahead at 0.79B tokens (37.3 versus 38.1–38.4). This lowers the training-token cost of reaching the same early compact-tier quality under the ESSH recipe.

We also added a 370M dense-pretraining control on the same 3.7B FineWeb-Edu tokens: only the full model is pretrained, and compact tiers are introduced during the same 2060-step shared refinement. Across two seeds, Hankel gives T1T_1 PPL 45.53 versus 63.77 for S4D-Lin, while full-model PPL is close (16.06 versus 15.97). Thus initialization affects compact-model recovery even when dense pretraining reaches similar full-model quality.

We have incorporated this interpretation into Section 2 and Appendix L.4. Figure 11 and Tables 38–39 give the new evidence, and Tables 34–35 retain the original controls.

2. (Q2, W2) Question / concern

Can you add an LTI arm (fixed clock and fixed gates) inside the same hybrid and recipe at 370M, with two seeds?

Answer. Yes. The new joint LTI control has higher PPL at every tier. This supports the contribution of input-dependent dynamics. We trained two 370M LTI controls on 3.7B FineWeb-Edu tokens, fixing the clock's input dependence and both write/read gates. Hybrid architecture, capacity map, shared refinement and evaluator match ESSH:

After shared refinementT1T_1 PPL ↓\downarrowT4T_4 PPL ↓\downarrowT7T_7 PPL ↓\downarrowT10T_{10} PPL ↓\downarrow
ESSH, three-seed mean25.3419.1917.4316.27
Joint LTI, two-seed mean27.0520.5518.6417.44

Both LTI seeds have higher PPL by 1.1–1.8 at every tier. Fixing the controls removes input-dependent retention, writing and readout.

We have added this discussion to Section 5.2 and Appendix L.4. Table 37 reports the new joint LTI controls.

3. (Q2, W2) Question / concern

Can you also add ES-SSM inside the same hybrid and recipe at 370M, with two seeds?

Answer. We have completed one matched ES-SSM seed. ESSH has lower PPL and higher seven-task and full-key accuracy. For this new comparison, both dense models use 22 layers, residual width 512, the same SWA layout, and 3.7B FineWeb-Edu tokens. FFN widths are adjusted to match total parameters. Results below use seed 197 and raw end-of-pretraining checkpoints:

ModelParams (M)PPL ↓\downarrowSeven-task (%) ↑\uparrowFull-key accuracy ↑\uparrow
ESSH370.816.9540.70.20
ES-SSM370.519.1039.00.13

ESSH's input-dependent clock and separate write/read gates adapt retention and state updates to the input, whereas the original ES-SSM applies fixed temporal filters. The joint LTI comparison above independently supports the value of selectivity.

We have incorporated this explanation into Section 5.2 and Appendix P.5. The new comparison is in Tables 1(c) and 72, with the baseline identified in Table 9.

AuthorsOpenReview ↗

Response to Reviewer t77B (Part 2)

4. (Q3, W3) Question / concern

Can you train MatMamba with the ESSH recipe (ten capacities, f=0.75f=0.75, KL =0.5=0.5, shared refinement), or ESSH with the MatMamba recipe?

Answer. Yes. The ESSH recipe improves MatMamba's smallest and full-size exports, and ESSH remains better at matched small sizes. We added 370M-scale MatMamba runs on 3.7B FineWeb-Edu tokens under its native recipe and under our capacity-sampling and distillation settings. The native recipe uses four capacities, a 25% reserved full-capacity share and no distillation. Our recipe uses ten capacities, a 75% reserved full-capacity share and KL weight 0.5. Both receive the same 2060-step shared refinement. The table gives two-seed MatMamba means and three-seed ESSH means:

Model / training recipeSmallest (M)Smallest PPL ↓\downarrowFull (M)Full PPL ↓\downarrow
MatMamba / native83.360.08369.919.03
MatMamba / ESSH recipe83.333.87369.916.71
ESSH / ESSH recipe92.125.34370.916.27

The recipe directly trains reduced-capacity predictors and supplies full-model distillation targets.

The smallest exports above differ in size. In the new matched-size comparison, ESSH at 92.1M reaches PPL 25.32–25.38 versus 29.47–29.51 for recipe-matched MatMamba at 92.9M. At 274.3M/274.4M, the ranges are 17.38–17.46/18.61–18.64.

We have expanded the recipe-transfer discussion in Section 5.2 and Appendix N.2. Tables 51 and 53–55 contain the new recipe and matched-size results.

5. (Q4, W4) Question / concern

Please add the 440M and 880M rows from Lahoti et al. to Table 1 and un-bold the PPL.

Answer. Yes. In the main manuscript, Table 1(d) now summarizes the comparison of our original compact exports with the published 880M and 440M models, using seven-task means and explicit gaps. The full per-task results are provided in Appendix Table 14. This is an added comparison of existing results, not new training. ESSH trails the independently trained models by 2.1–4.6 percentage points at T4T_4 and 5.8–7.2 at T1T_1. The claim we keep is the train-once trade-off: one run supplies all ten sizes, at a quality cost relative to models trained separately at each size. Among train-once families, the 370M results show a better PPL–size trade-off than MatFormer and MatMamba (Table 1(c), Figure 3(a)). We also removed the bold from the 1.5B PPL column, because each source reports PPL on its own validation shard.

We have incorporated this quality-gap discussion into Sections 5.1–5.2. Table 1(d) and Appendix F.2 compare the original exports with published models, alongside the elastic baselines in Table 1(c).

6. (Q5, W5) Question / concern

Please re-time decode with the same compile and graph policy for all models, and add Mamba-2 and Mamba-3 to Figure 4(d).

Answer. Yes, we reran the main 1.53B decode and prefill benchmarks on A100, H200 and B300 with the trained ESSH attention layout, torch.compile for every model and CUDA graphs for decoding. We also added Mamba-2 and Mamba-3 SISO prefill to Figure 4(d). ESSH retains lower decode latency on H200/B300 and is effectively tied with Mamba-2 at batch one on A100. The models are parameter-matched, with bf16 weights and a 2K decode cache. Columns give decode ms/step and batch-one B300 prefill at 128K in ktok/s:

ModelA100 B1 ↓\downarrowH200 B1 ↓\downarrowB300 B1 ↓\downarrowB300 B128 ↓\downarrowPrefill ↑\uparrow
ESSH3.031.651.354.52113.6
Mamba-23.041.731.525.63226.0
Mamba-3 SISO3.482.131.945.53121.5
Mamba-3 MIMO3.742.352.195.8456.2
Transformer++3.401.991.6221.4326.0

B300 batch-one speedups are 1.12–1.62$\times$ over the Mamba variants and 1.20$\times$ over Transformer++, using unrounded medians. ESSH has the lowest measured decode latency at every tested batch on H200/B300 and from batch 16 onward on A100. Mamba-2 and Mamba-3 SISO prefill faster. ESSH's long-context prefill advantage is relative to full attention.

Under the same compilation policy, the new 92M comparison also favors ESSH: A100/B300 batch-one latency is 0.94/0.71 ms versus 1.23/1.09 for the matched head-nested Mamba-3+SWA. ESSH reuses each channel's projected input across modes and fuses projection, state update and readout. Table 66 separately measures the fusion benefit with both paths compiled.

We have incorporated the common-policy timing analysis into Section 5.3 and Appendices P.4 and R. Figures 2 and 4 and the corresponding tables use the new measurements.

AuthorsOpenReview ↗

Response to Reviewer t77B (Part 3)

7. (Q6) Question / concern

Why does the two-layer Transformer baseline score 0.022 on 32-pair MQAR at L=256L=256? Was it tuned?

Answer. Yes. We tuned the learning rate, and 0.022 is the obtained result. However, after further investigation, we noticed that the low score comes from the narrow model width (hidden dimension 128), not from tuning. We added a width and training-budget comparison for the same two-layer Transformer on this test by extending the width to 256 (inputs of L=256L=256 tokens with 32 key–value pairs; scaled accuracy):

Model widthOriginal budget ↑\uparrow10×10\times training tokens ↑\uparrow
1280.0220.053
2560.9981.000

Ten times more training tokens leave width 128 at 0.053, while width 256 reaches 0.998 under the original budget. The original 0.022 is therefore correct for this narrow setting. Our MQAR claim concerns the hybrid, which stays at 0.971–1.000 with 64 pairs (Table 26), and does not depend on this baseline.

We have incorporated the diagnostic interpretation into Appendix I.1 beside the new width/budget study in Table 27. Tables 25–26 retain the original MQAR results.

8. (Q7) Question / concern

The paper was not an easy read because of too much engineering detail. Can you improve the flow without distracting implementation minutiae?

Answer. Thank you very much for this suggestion. We simplified Section 4.3 to focus on the main computational ideas: processing sequence chunks in parallel, carrying recurrent states between chunks, and using the same recurrence for token-by-token decoding. Figure 2 provides a visual overview. We moved the detailed within-chunk formula to Appendix O.1, while the derivations, backward-pass details, and kernel configurations are provided in Appendices O.1–O.4. We hope this improves the readability of the paper.

9. (W1) Question / concern

How does the method differ from nested selective heads plus FFN nesting with distillation, particularly MatMamba's head nesting? Should the paper position the contribution in those terms rather than attributing the learned ordering to the Hankel basis?

Answer. ESSH shares MatMamba's nested-training principle, with three architectural and execution differences:

  1. Nested unit. MatMamba nests Mamba-2 inner dimensions and heads. ESSH nests spectral channels, each containing mm two-dimensional rotation modes sharing vk(t)v_k(t). Removing a channel reduces its projection and state costs while retaining valid selective dynamics.
  2. Capacity allocation. MatMamba nests the inner width of the Mamba-2 block. FFNs dominate the variable parameter budget, whereas channel count controls recurrent capacity. We reduce FFN width faster than spectral-channel count, removing more tokenwise parameters while retaining a larger fraction of the temporal channels. Budget dropout and full-tier distillation train these models.
  3. Execution. The chunked scan sums mode interactions before multiplying by the shared channel values. The fused decoder reuses each channel's value projection across modes and skips discarded channels, which reduces repeated computation.

On the second question, we agree and have clarified ESSH’s positioning within the nested-training framework. ESSH shares the nested-training principle with MatMamba, while its technical contributions concern the selective spectral-channel recurrence, the two-rate allocation of channel and FFN capacity, and the associated efficient execution.

Consistent with our response to item 1 (Q1, W1), we also clarify that capacity-mixed training encourages useful predictive ordering, as supported by the random-mode and shuffled-slot controls in Table 36.

We have clarified this design rationale in Section 2, with the transition and execution described in Sections 4.1 and 4.3. Table 36 retains the original ordering controls, and Appendix L.4 presents the new initialization evidence.

Reviewer_CtCnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Elastic Selective Spectral Hybrids (ESSH), a hybrid architecture designed for train-once, export-many deployment across compute budgets. ESSH represents each Hankel spectral channel as an independent recurrent unit parameterized by damped planar rotations with input-dependent decay and write/read gates, interleaved with three sliding-window attention (SWA) layers. To support standalone subnetwork exports without post-hoc pruning, the authors jointly vary spectral-channel count and FFN width via a two-rate capacity map (ξT1/2\xi_T^{1/2} for channels, ξT0.8\xi_T^{0.8} for FFN), trained with budget dropout, full-tier KL distillation, and a brief shared-refinement stage. The structured recurrence admits a chunked parallel scan for training and a fused recurrent kernel for decoding. At the 1.53B scale (100B pretraining tokens), a single training run yields ten standalone tiers down to 0.474B parameters, reaching a 55.03% seven-task mean at full capacity and 1.37 ms/token batch-one decoding on a B300 GPU.

Critical strengths

1.Clean formulation of truncatable selective recurrence:Because scalar decay commutes with fixed planar rotation, each mode retains a closed-form multi-step transition product despite input-dependent retention. Keeping value projections, clocks, gates, and state updates channel-local allows discarded channels (k>KTk > K_T) to be cleanly sliced out at export time. 2.Substantial systems engineering:Rather than stopping at parameter counts, the paper provides a chunked parallel scan (Q=64Q=64) with a numerically guarded backward pass and a fused recurrent decoding path that cuts kernel launches per step from 963 (Mamba-2) to 367 on B300. Decode speedups hold across A100, H200, and B300 with CUDA graphs enabled for all baselines, and a same-architecture control on A100 confirms the gain from fusion itself (7.40 to 3.02 ms). 3.Thorough ablations and transparent reporting:The appendices include informative controls—such as independently trained same-shape ESSH specialists, alternative initializations, post-hoc recovery baselines, and seed-variance analyses—and carefully distinguish guarantees about the initial LTI Hankel basis from empirical properties of the trained selective model.

Critical weaknesses
  1. The marginal benefit of spectral truncation for budgeted inference remains unclear. Truncating spectral channels alone leaves the smallest mixer-only export with 91.2% of the full model's parameters (Table 33), meaning nearly all model-size reduction comes from shrinking the FFN. Even when both axes are reduced, moving from T10T_{10} to T1T_1 at 1.5B removes about 69% of the parameters but cuts B300 batch-one latency by only 17.5% (1.37 to 1.13 ms) and persistent state+KV memory by just 4.6% (53.61 to 51.15 MB), since the SWA KV cache stays fixed at 50.33 MB. While the weight-storage reduction is substantial and latency gains are larger on A100, parameter count is not a reliable proxy for every deployment bottleneck.

At the same time, elastic pretraining carries a noticeable quality cost at both ends of the family. In the 370M same-architecture control, capacity-mixed training incurs a full-model PPL penalty of +0.48 after 3.7B tokens and +0.41 after 7.4B tokens prior to shared refinement (Table 37), and the shared T1T_1 export trails its identical-shape specialist in PPL (23.61 vs. 19.79, Table 11), meaning the reported four-model training savings are not quality-matched. While some trade-off is inherent to joint multi-capacity training, it leaves open a central practical question: does spectral-channel elasticity add enough value over FFN-only elasticity within the same hybrid to justify that cost, especially when plotted against measured latency and memory rather than parameter count?

  1. The link between LTI Hankel theory and the learned elastic model is not established empirically. The theoretical motivation relies on the optimality of the Hankel eigenbasis for fixed LTI filters. Once ESSH makes state retention input-dependent and trains the modal parameters end-to-end under nested dropout, that projection argument no longer guarantees that the initial spectral ordering remains preferable in the learned model. Although the appendix appropriately limits the scope of the theory, the main text still leans on a spectral interpretation that the ablations do not bear out.

In particular, S4D-Lin initialization achieves slightly better full-capacity PPL than the Hankel control and matches its compact-tier PPL (Table 29). Likewise, random damped modes and shuffled channel slots still develop strong prefix importance ordering once trained with budget dropout (Table 31). These results indicate that selective recurrence and nested training—not the Hankel initialization itself—drive the elastic behavior. The method should either be framed more modestly around nested selective recurrence, or supported by a final-stage S4D-Lin comparison and a trained, state-matched control fitted with 16 positive real poles.

  1. In the experiment, the comparisons do not isolate the claimed architectural advantage. At full capacity, ESSH is competitive but trails the strongest hybrid and recurrent references: at 370M, it reaches 14.71 PPL versus 14.10 for Mamba-3 + SWA and 14.31 for Gated DeltaNet + SWA, and at 1.5B its 55.0% seven-task mean sits slightly below the published baselines (55.4–57.6%, Table 1). Because elastic training naturally trades off some full-model quality, the case for ESSH rests heavily on its compact-family frontier.

However, the elastic baseline comparisons bundle mixer design with backbone and training-recipe differences. MatFormer is pure attention and MatMamba is purely recurrent; neither shares ESSH's three-layer SWA hybrid backbone, two-axis capacity map, capacity-sampling schedule, full-tier KL distillation, or shared-refinement stage. While ESSH already outperforms both at intermediate sizes before refinement—and the appendix fairly separates baseline exports that fall below their minimum trained width—these comparisons cannot isolate the benefit of the spectral mixer from the hybrid backbone and distillation recipe. Adding a 370M non-spectral hybrid (e.g., Mamba-3 + SWA) trained with the same elastic recipe, and timing the actual trained 1.5B checkpoint layout (SWA at blocks 15, 22, 27 rather than the initialized 9, 19, 27 timing configuration in Appendix R), would make the frontier claims much cleaner. Note also that the 1.5B PPL references in Table 1(a) use different validation shards and should not be treated as a direct ranking.

  1. Retrieval tests do not isolate long-range spectral-state retention, and key ablations need repeated runs. Both the main NIAH probe (L=2048L=2048) and the synthetic recall tasks (L≤1024L \le 1024) fall entirely within the 2048-token SWA window, whereas mixer-only variants perform poorly on MQAR. As a result, these experiments show that the hybrid preserves in-window attention retrieval after truncation, but do not test whether the spectral state can retain keys beyond direct attention access. Even within the 2K window, 1.5B T10T_{10} full-key accuracy drops from 0.633 at depth 0.9 to 0.035 at depth 0.1 (Table 18), and the 2K–16K PPL sweep (Table 19) only measures next-token prediction across different text sets per length rather than explicit long-range recall.

Furthermore, retrieval is highly sensitive to training seeds: in Table 29, T4T_4 full-key accuracy spans 0.030 to 0.249 across three Hankel control seeds. While those raw-checkpoint scores are not directly comparable to the final-stage first-token ablations in Figure 8(b), they show that run-to-run training variance is much larger than item-bootstrap intervals capture. Evaluating needles placed outside the SWA window and repeating the clock/gate controls across matched training seeds (at the same checkpoint stage and metric) would make the mechanism claims much more persuasive.

Questions for authors
  1. Within the same hybrid backbone and training recipe, what concrete quality–latency or quality–memory benefit does two-rate spectral + FFN truncation provide over FFN-only elasticity (keeping all 32 spectral channels fixed)?
  2. Does Hankel initialization retain any measurable advantage over S4D-Lin or state-matched fitted real poles (16 real poles vs. 8 damped rotations) after full pretraining and shared refinement?
  3. Does the compact-family advantage hold up at 370M against a non-spectral recurrent hybrid (such as Mamba-3 + SWA or Gated DeltaNet + SWA) given the same SWA layout, capacity-sampling schedule, and full-tier KL distillation?
  4. How does retrieval behave when the needle is placed beyond the 2048-token SWA window, and do the clock and gate retrieval gains persist across matched training seeds?
4 comments

We sincerely appreciate the thorough review process and the constructive comments.

Questions and concerns below are condensed from the review. Q/W labels follow the review's numbering.

1. (Q1, W1) Question / concern

Within the same hybrid backbone and training recipe, what concrete quality–latency or quality–memory benefit does two-rate spectral + FFN truncation provide over FFN-only elasticity (keeping all 32 spectral channels fixed)?

Answer. The spectral axis extends the reachable size range and reduces recurrent work. We added a 370M FFN-only control on 3.7B FineWeb-Edu tokens, with the same capacity-mixed training and shared refinement. It keeps all 32 channels. Its smallest trained export is 124.9M, and even removing every FFN leaves 98.4M parameters, above the 92.1M T1T_1 target.

Near its smallest trained size, the new comparison gives:

Same recipe and refinementESSH, 118.9MFFN-only, 124.9M
PPL ↓\downarrow21.6722.25
A100 batch-one decode, ms/step ↓\downarrow0.971.14
B300 batch-one decode, ms/step ↓\downarrow0.630.75
State+KV, MB per sequence ↓\downarrow22.3923.11

The latency values are new measurements with both decode paths compiled and the same 2K cache. Both models retain the same 22.02 MB attention cache, but ESSH can reduce recurrent state from 1.09 to 0.27 MB at T1T_1. FFN-only keeps that state fixed. Retaining every channel benefits its intermediate-size quality, by up to 0.5 PPL. The full models are close, at 16.29 for FFN-only versus 16.27 for ESSH. These results explain our design choice: two-rate reduction supports smaller deployments and lower recurrent cost.

We have incorporated this quality–cost discussion beside the new FFN-only results in Appendix M.1, Table 41. Section 5.2 states the parameter-floor result.

2. (Q2, W2) Question / concern

Does Hankel initialization retain any measurable advantage over S4D-Lin or state-matched fitted real poles (16 real poles vs. 8 damped rotations) after full pretraining and shared refinement?

Answer. First, we agree that the original Tables 29–30 (Tables 34–35 in the revision) could not establish a consistent final-quality advantage for Hankel initialization over S4D-Lin. Instead, these tables support the effectiveness of the architecture and training procedure across multiple initializations. Against the state-matched real poles, Hankel retains lower final PPL at every tier.

Regarding our original claim, we would like to revise it as: In the tested ESSH setting, Hankel-based initialization improves early compact-tier training-token efficiency.

This revised claim is supported by newly added results, which provide training-curve comparisons at 370M on FineWeb-Edu with 3.7B-token capacity-mixed pretraining, budget dropout and full-tier distillation. We compare the tokens needed to reach the same T1T_1 PPL:

InitializationSeeds in curvesTokens to T1T_1 PPL 50 (B) ↓\downarrowT1T_1 PPL at 0.39B ↓\downarrowT1T_1 PPL at 0.79B ↓\downarrow
Hankel20.4050.437.3
S4D-Lin30.4554.738.1
Random modes30.4453.738.4
Sixteen fitted real poles10.5160.240.5

The training curves show faster early compact-tier optimization with Hankel initialization (Figure 11 and Table 38). Hankel reaches T1T_1 PPL 50 after 0.40B tokens, compared with 0.44–0.51B for the alternatives, and it remains ahead at 0.79B tokens (37.3 versus 38.1–40.5).

We also added a 370M dense-pretraining control on the same 3.7B FineWeb-Edu tokens: only the full model is pretrained, and compact tiers are introduced during the same 2060-step shared refinement. Across two seeds, Hankel gives T1T_1 PPL 45.53 versus 63.77 for S4D-Lin, while full-model PPL is close (16.06 versus 15.97). Thus initialization affects compact-model recovery even when dense pretraining reaches similar full-model quality.

The sixteen real poles hold the same number of active scalar state values as eight rotations. After the same training and refinement, Hankel's rotation bank has lower PPL at every tier in both seeds, by 0.88 at T1T_1 and 0.41 at T10T_{10} on average. This favors the oscillatory mode bank at a matched state budget.

We have revised the Hankel claim in Section 5.2 and Appendix L.4. Figure 11 and Tables 37–39 present the new training curves, the real-pole control and the dense-pretraining comparison.

3. (Q3, W3) Question / concern

Does the compact-family advantage hold up at 370M against a non-spectral recurrent hybrid (such as Mamba-3 + SWA or Gated DeltaNet + SWA) given the same SWA layout, capacity-sampling schedule, and full-tier KL distillation?

Answer. Yes, at the compact end: ESSH has lower PPL at T1T_1 and T4T_4 and uses less recurrent state. We added 370M-scale Mamba-3+SWA models trained on 3.7B FineWeb-Edu tokens with the same hybrid depth, residual width, SWA layout, FFN reduction rule, capacity sampling, full-tier distillation and shared refinement. Head counts and full-model FFN widths match the four ESSH sizes, including 92M:

3.7B tokens, after refinementT1T_1 PPL ↓\downarrowT4T_4 PPL ↓\downarrowT7T_7 PPL ↓\downarrowT10T_{10} PPL ↓\downarrow
ESSH, three-seed mean25.3419.1917.4316.27
Head-nested Mamba-3+SWA, two-seed mean27.4319.4417.3116.06

At 92M, ESSH improves PPL by about 2.1 and uses 0.27 versus 1.90 MB of recurrent state. Mamba-3 is slightly better in PPL at the larger tiers. The memory difference follows the state parameterization: ESSH uses eight two-coordinate modes per value dimension, whereas this Mamba-3 configuration uses a much larger expanded state. Including the shared 22.02 MB attention cache, full-model persistent memory is 23.1 versus 39.7 MB. Thus the compact-quality and state-memory benefits persist after matching the hybrid and training recipe.

We have incorporated this size-dependent interpretation into Section 5.2 and Appendix N.2. Table 52 presents the new parameter-aligned comparison.

4. (Q4, W4) Question / concern

How does retrieval behave when the needle is placed beyond the 2048-token SWA window?

Answer. Our original NIAH test evaluates retrieval within the 2048-token window, which aligns with the setting the paper claims. In our framework, SWA was added to give exact access to keys inside that window. Retrieval beyond the stacked attention layers' receptive field would rely on the recurrent state alone. Our current framework is not designed to address this challenge, and we anticipate that the performance drop will be significant. Nevertheless, this is an important and interesting problem, and we will address it in our future work.

We have clarified this evaluation scope in Section 5.2 and Appendix H. Tables 20 and 22 retain the original within-window measurements.

5. (Q4, W4) Question / concern

Do the clock and gate retrieval gains persist across matched training seeds?

Answer. Not yet for the clock and gates separately: the original component ablations use one seed. We added a two-seed joint LTI control at 370M. It removes the clock's input dependence and fixes the write/read gates, with 3.7B FineWeb-Edu tokens and the same shared refinement as ESSH:

ModelT4T_4 mean PPL ↓\downarrowT4T_4 2K full-key accuracy range ↑\uparrow
ESSH, three seeds19.190.01–0.26
Joint LTI, two seeds20.550.05–0.10

The overlapping full-key ranges do not establish a retrieval gain across seeds, although PPL consistently favors selective dynamics. Repeating the clock-only and gate-only retrieval ablations across seeds remains future work.

We have clarified the interpretation of the repeated joint control in Appendix L.4. The new results are in Tables 37 and 21, alongside the original component ablations in Table 32.

6. (W1) Question / concern

What practical deployment benefit does truncation provide in weight storage, measured latency, and persistent state plus KV-cache memory? At 1.5B, why does removing approximately 69% of the parameters reduce B300 batch-one latency by only 17.5% (1.37 to 1.13 ms) and state plus KV memory by only 4.6% (53.61 to 51.15 MB), with a fixed 50.33 MB SWA KV cache? Can the trade-off be assessed against the actual deployment bottleneck rather than parameter count alone?

Answer. We remeasured the 1.53B family under the common inference policy, using the trained attention layout, torch.compile and a 2K context/cache, with CUDA graphs for decoding. On B300, the largest benefits remain lower weight storage and faster prefill. The table combines the new timings with the original weight and state/cache counts:

1.5B ESSHFull T10T_{10}Compact T1T_1
Weights (GB) ↓\downarrow3.060.95
B300 prefill at 2K (ktok/s) ↑\uparrow102.0127.6
A100 prefill at 2K (ktok/s) ↑\uparrow34.857.0
A100 decode, B=1 (ms/step) ↓\downarrow3.031.47
B300 decode, B=1 (ms/step) ↓\downarrow1.351.25
B300 decode, B=512 (ms/step) ↓\downarrow12.2110.76
State+KV (MB/sequence) ↓\downarrow53.6151.15

Depth, residual width and attention remain fixed, so reducing channels and FFNs removes only part of each decode step. The 50.33 MB shared attention cache dominates persistent memory. In contrast, weight storage falls from 3.06 to 0.95 GB, and prefill benefits from reducing channel and FFN computation at every token. This makes the relevant deployment benefit depend on whether storage, prefill or batch-one latency is the bottleneck.

We have incorporated this resource-specific interpretation into Section 5.3 and Appendices P.2–P.3 and R. Table 1(b) combines the new B300 timings with the original memory counts, and Tables 65–66 give the A100 tier measurements.

7. (W1, W3) Question / concern

What quality cost does elastic training incur relative to independently trained models at both full and compact capacities? How should the full-model PPL penalties of +0.48 after 3.7B tokens and +0.41 after 7.4B tokens, and the compact export's PPL of 23.61 versus 19.79 for its identical-shape specialist, be reflected in the reported four-model training savings? How should the full-capacity results be characterized relative to the stronger 370M hybrids and the published 1.5B seven-task means of 55.4–57.6%?

Answer. Shared training saves the cost of separate runs but sacrifices some model quality, particularly at T1T_1. The most direct evidence comes from the original results of Table 11 (Table 12 in the revision). They compare identical ESSH deployment shapes trained on 7.4B FineWeb-Edu tokens, with shared refinement included for the shared family:

TierPPL: shared / specialist ↓\downarrowSeven-task %: shared / specialist ↑\uparrow
T1T_123.61 / 19.7937.6 / 38.5
T4T_417.68 / 16.2540.5 / 41.2
T7T_715.91 / 15.1042.0 / 42.4
T10T_{10}14.71 / 14.4442.8 / 42.5

Independent training gives lower PPL at every shape. Seven-task differences stay below one percentage point. The original full-model penalties of 0.48 and 0.41 are measured before shared refinement, after 3.7B and 7.4B tokens. They are therefore not directly comparable with the final-checkpoint gaps above.

For the four ESSH shapes, the original cost accounting gives 53.9 estimated A100-hours for shared training versus 147.4 for four same-architecture specialists. Four independent Transformer++ models cost 101.4 hours. These savings come from sharing training across capacities, with the quality costs shown explicitly.

At full capacity, the original ESSH means are 42.8% at 370M and 55.0% at 1.5B, versus 44.5% for the dedicated 370M Mamba-3+SWA and 55.4–57.6% for the published 1.5B references. We retain these comparisons and now state the full range.

We have clarified this quality–cost discussion in Sections 5.1–5.2 and Appendix N. The original supporting measurements remain in Tables 12, 46 and 61–62.

8. (W3) Question / concern

Can decoding be timed using the actual trained 1.5B checkpoint layout, with SWA at blocks 15, 22, and 27, rather than the initialized timing configuration with SWA at blocks 9, 19, and 27 in Appendix R?

Answer. Yes, we reran the 1.5B decode and prefill benchmarks on A100, H200 and B300 using the trained SWA layout at blocks 15/22/27. All revised comparisons apply torch.compile to every model and use CUDA graphs for decoding. A separate A100 control changes only the SWA positions from 9/19/27 to 15/22/27, with a latency difference below 0.1% at every tested batch size.

We have incorporated this timing interpretation into Section 5.3 and Appendix P.4. Figure 4 and Appendix R give the updated measurements with the trained layout.

9. (W3) Question / concern

Can the 1.5B PPL values in Table 1(a) be used for a direct ranking when the reference models and ESSH are evaluated on different validation shards?

Answer. No. The original 1.5B PPL values use different validation shards and cannot support a direct cross-model ranking. We removed the emphasis on 10.11 and made the source-specific evaluation explicit. The shared seven-task benchmark remains the basis for the published-model quality comparison.

We have incorporated this clarification into Section 5.1 and the Table 1(a) caption, using the original source-specific PPL measurements.

Thank you again for your time and valuable feedback, and we would be happy to address any further questions or discussion. We would be sincerely grateful if you would consider updating your score.

Comments on the paper

1 comment
Area_Chair_vUr4OpenReview ↗

Reminder to reviwers to read author's responses

Dear Reviewers,

Thank you for your efforts in providing thorough reviews.

As the discussion period is underway, I would like to kindly remind you to read the authors' responses to your reviews and provide follow-up comments where appropriate or leave a comment. In particular, please consider whether the responses address your questions and concerns, and update your assessment if the clarifications or additional information affect your initial evaluation.

Your engagement in this discussion is important for ensuring a fair and thorough review process.

Thank you again for your time and contributions!

Best regards,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.