- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper interprets Mamba-3's data-dependent decay-rotation transition, rho*R(theta), as an underdamped second-order linear system and derives the equivalent damped-oscillator equation, with damping set by decay and frequency by rotation. Using this view, it builds a family of transitions parameterized by kappa that share a spectral center and differ only in damping regime, and it compares underdamped, critically damped, and overdamped variants on parity, mod-3, and arithmetic. Underdamped dynamics win clearly.
The paper then studies language modeling in three ways. First, phase removal in pretrained 1.5B Mamba-3 models degrades LAMBADA far more than factual-relation prediction and sharply raises long-context PG-19 perplexity. Second, 187M and 444M models trained from scratch on 100B tokens with and without rotary channels show lower PG-19 perplexity with rotation and LongBench-E gains at 444M. Third, hidden-state analyses show that removing phase increases alignment between retained state and new writes and increases state norm, and write/readout interventions indicate that B and C need consistent phase. The authors conclude that oscillation supports state tracking and long-context modeling by regulating how information accumulates.
Critical strengths
-
The intervention study in pretrained models (Section 5.3). Phase shuffle, removal, and reversal are compared against a decay-permutation control at several context lengths in both SISO and MIMO models. This is a more careful design than typical ablations, and it gives concrete evidence that long-context prediction in pretrained Mamba-3 depends heavily on learned phase and on how phases are assigned to state pairs. This is the most solid and most reusable part of the paper.
-
The controlled pretraining comparison at meaningful scale (two model sizes, 100B tokens each), evaluated on PG-19 up to 64K, on LongBench-E, and on standard benchmarks. Few interpretability-oriented SSM papers include matched from-scratch training, and this comparison is what links the mechanistic story to practical capability.
Critical weaknesses
- The main controlled comparison does not isolate oscillation.
The critically damped and overdamped variants are discretized as exp(Delta*A(kappa)) with real eigenvalues, so all their discrete eigenvalues lie in (0, 1). Grazzi et al. (2025), which the paper cites, show that such positive real spectra cannot implement parity. The parity solution in Fig. 5 uses a rotation of about 180 degrees, and R(pi) = -I is itself a negative real eigenvalue, so the parity result does not distinguish oscillation from access to a non-positive spectrum. Mod-3 does genuinely require complex structure, but the paper does not make this argument.
The variants also differ in ways other than damping regime. In the critically damped variant (kappa = 0), the input-dependent nu_t plays no role, and the transition is a non-normal Jordan block. This may explain why it scores 1.97% on arithmetic, below both the overdamped variant (81.60%) and Mamba-3 without rotation (48.95%), and the paper does not explain this gap. In addition, the persistence analysis in Section 4.2 fixes omega_0, while the experiments fix the spectral center -a. Under the experimental setup, the slow overdamped mode decays more slowly than the underdamped modes, so the theory does not describe the variants that were trained. Finally, the language-modeling experiments compare rotation against scalar decay rather than against the damping regimes, so the second-order framework is tested only on small synthetic tasks. This weakness matters most because the paper's central claim concerns the role of oscillation, and this experiment is the only one designed to isolate it.
- The language-modeling evidence is confounded and statistically thin.
Phase removal in a model trained with phase is far out of distribution, and its damage grows with sequence length (Table 5). LAMBADA passages are much longer than Relation prompts, so the gap in Fig. 2 may reflect input length rather than context dependence. The state-norm and alignment statistics in Table 6 are measured under this same out-of-distribution intervention, not on the matched from-scratch models. The authors note that the from-scratch model without rotation extrapolates stably, which weakens the state-norm explanation.
The from-scratch results come from single runs without variance estimates, and the effects are small or mixed. FineWeb-Edu perplexity differs by about 0.1. At 187M, the rotation model has slightly lower average downstream accuracy and is about 3 points worse on Arc-C. LongBench-E scores near 10 are close to the floor at these scales, and most of the 444M gain comes from four tasks. With two model sizes and one seed each, the data do not support the claim that downstream gains "emerge at larger scale." The evaluation on additional datasets extends only to 4K tokens, twice the training length. This matters because the paper's second and third headline findings depend on these experiments.
Questions for authors
- Is the underdamped variant in Table 2 implemented through exp(Delta*A(kappa)) in companion form, or through Mamba-3's native rotation of B and C? If the latter, how are input injection and readout matched across the variants?
- How is "scaled accuracy" defined, what value of c is used for the overdamped variant, and how many seeds stand behind each number in Tables 2 through 4?
- What explains the critically damped variant scoring far below both the overdamped variant and Mamba-3 without rotation on arithmetic?
- Does removing the rotary channels change the parameter count of the from-scratch models?
Experiments whose results could change my assessment:
- Does a non-oscillatory variant that allows negative real eigenvalues (for example, with gates in [-1, 1] as in Grazzi et al.) match the underdamped model on parity, and how does it perform on mod-3?
- Do the Table 6 statistics (state-write alignment and RMS state norm) differ between the two from-scratch models?
- How large is the run-to-run variance of the PG-19 and LongBench-E gaps at 444M?
- Does the LAMBADA-versus-Relation gap remain when input length is controlled, for example by placing relation queries inside long distractor contexts?
Suggestions for future work (not needed for my assessment):
- Train the damping-regime variants of Eq. 18 at language-model scale.
- Report population-level statistics for the task-aligned phase structure in Fig. 5, rather than relying on hand-selected state pairs.
4 comments
Response to Reviewer iFGy (Part II)
3. Run-to-run variability on PG-19 and LongBench-E at 444M (Q7)
During the rebuttal, we have added two independent training runs per configuration, bringing the 444M comparison to three training seeds each, with and without oscillation. All runs use the same pretraining setup and 100B-token budget. The table summarizes mean ± std across training seeds, with representative context lengths for PG-19.
| Configuration | PG-19 2K PPL ↓ | PG-19 16K PPL ↓ | PG-19 64K PPL ↓ | LongBench-E average ↑ |
|---|---|---|---|---|
| Without oscillation | 27.17 ± 0.21 | 27.97 ± 0.63 | 26.92 ± 0.81 | 11.70 ± 0.82 |
| With oscillation | 26.80 ± 0.12 | 27.05 ± 0.30 | 25.87 ± 0.20 | 12.53 ± 0.64 |
PG-19. Oscillation lowers mean perplexity at all seven evaluated context lengths from 1K to 64K. The reduction increases from 0.37 at 2K to 1.05 at 64K, extending beyond the 2K training context. The oscillatory configuration also exhibits smaller std at every evaluated length across these three runs.
LongBench-E. Oscillation improves the mean aggregate score by 0.83 points. We compute the 13-task average separately for each training seed, then report the mean and std of these averages.
These additional runs support the observed mean improvements while quantifying training variability. Table 3 reports the updated LongBench-E results and Appendix E.4 provides per-task mean and std.
4. LAMBADA versus Relations under matched input lengths (Q8)
We added a length-matched comparison using the official Mamba-3 1.5B SISO and MIMO checkpoints.
We sample as many LAMBADA examples as Relations queries and pair them. Each Relations query is prepended with unrelated text to match its paired LAMBADA input length, preserving the original query and target answer. We repeat this construction with three distractor seeds. The table reports accuracy drops in percentage points following phase removal, averaged equally over all 300 contiguous layer windows. For length-matched Relations, mean ± sample std is computed across the three distractor seeds.
| Model | Relations, original | Relations, length-matched | LAMBADA |
|---|---|---|---|
| SISO | 0.08 | 4.77 ± 0.19 | 11.79 |
| MIMO | −0.19 | 6.32 ± 0.41 | 20.19 |
Longer distractor contexts increase Relations’ sensitivity to phase removal, but its accuracy drops remain substantially smaller than LAMBADA’s in both models. Thus, matching input length does not eliminate the difference between tasks. The greater sensitivity of LAMBADA is consistent with a role for learned phase in context-dependent prediction.
The procedure and results have been reported in Appendix D.2 and Appendix E.2 (Table 13), with discussion in Section 5.1.
5. Implementation of different damping regimes (Q1)
For the damping-regime comparison in Table 2, all three variants use the same companion-form matrix exponential:
\exp\left(\Delta_t A_t(\kappa_t)\right), \quad A_t(\kappa_t)= \begin{pmatrix} 0 & 1 \\\\ -(a_t^2-\kappa_t) & -2a_t \end{pmatrix}.$$ We set $\kappa_t=-\nu_t^2$ for underdamping, $\kappa_t=0$ for critical damping, and $\kappa_t=r_t^2$ for overdamping. In each case, the discretized transition acts **directly on the recurrent hidden state**. The variants share the same state dimension, input-injection and readout rules, input-dependent step-size parameterization, and training protocol. These implementation details have been added in Section 4.1 and Appendix D.1 of the revised manuscript. ## 6. Scaled accuracy, the choice of c, and training seeds (Q2) **Scaled accuracy.** Following [1, 2], we define $$a_{\mathrm{scaled}} = \frac{a-a_{\mathrm{chance}}}{1-a_{\mathrm{chance}}},$$ where $a$ is the measured accuracy and $a_{\mathrm{chance}}$ is the task-specific chance accuracy: $0.5$ for Parity and $1/3$ for Mod-3. This definition is now explicitly provided in Appendix D.1. **Choice of c.** We use $c=1$ in the overdamped parameterization as specified in Appendix D.1. **Training seeds and model selection.** Following the Mamba-3 state-tracking protocol [1], we sweep **eight learning rates** logarithmically spaced between $10^{-4}$ and $10^{-2}$, with **two independent training seeds per learning rate**, yielding **16 runs per model–task combination**. We select the run with the highest validation accuracy and report its held-out test accuracy. Thus, **each Table 2 entry represents a single validation-selected run**.Response to Reviewer iFGy (Part III)
7. Performance of the critically damped variant on arithmetic (Q3)
Our hypothesis is that critical damping limits the flexibility of individual state blocks. Overdamped blocks have two distinct decay modes, , whereas critical blocks have repeated eigenvalues at and lack independent modulation through . All arithmetic variants use three layers, allowing for different decay rates across layers and channels. However, this restriction remains valid within each critical block.
This difference may contribute to the performance gap, but does not fully explain the 1.97% scaled accuracy. The relative contributions of optimization and length generalization remain unclear.
8. Parameter counts across model scales (Q4)
Removing the rotary channels slightly reduces the parameter count. The exact counts are shown below.
| Model size | Without rotation | With rotation | Additional parameters | Relative increase |
|---|---|---|---|---|
| 187M | 186,997,056 | 187,291,968 | 294,912 | 0.158% |
| 444M | 443,408,896 | 444,195,328 | 786,432 | 0.177% |
| 1.5B | 1,495,510,016 | 1,497,082,880 | 1,572,864 | 0.105% |
The relative increase is calculated with respect to the model without rotation. Adding rotation increases the parameter count by less than 0.2% at every evaluated scale, so the configurations remain closely matched in parameter budget.
9. Persistence analysis and learned damping parameters (W1)
Section 4.2 holds fixed to compare persistence across damping regimes. The conclusion that critical damping gives the fastest asymptotic attenuation applies specifically to this comparison.
In the experiments, is input-dependent and learned separately by each variant. The shared spectral center in Section 4.1 describes the common parameterization, and its value is not fixed across regimes.
Our analysis aims to provide an interpretation under a specified comparison condition, while the experiments assess performance when the damping parameters are learned from data.
10. Evaluation lengths on additional datasets (W2)
Our evaluations on CodeParrot, Math-Hard, TriviaQA, and SlimPajama follow the evaluation design in Figure 5 of [2], using input lengths up to 4K tokens in our experiments. We have also added the corresponding results for the 1.5B models with and without oscillation. These new results have been reported in Appendix E.3, Figure 6 of the revised manuscript.
Response to Reviewer iFGy (Part IV)
11. Language modeling under different damping regimes (Future work 1)
Thank you for this suggestion. We compared all three damping regimes using about 50M-parameter models trained on 2B FineWeb-Edu tokens, with the GPT-2 tokenizer, a 2K training context, and otherwise identical architectures and training settings. This scale accommodates the higher computational cost of our explicit second-order implementation.
We report FineWeb-Edu test perplexity at 2K and PG-19 perplexity at context lengths from 2K to 32K. Lower is better.
| Damping regime | FineWeb-Edu test | 2K | 4K | 8K | 16K | 32K |
|---|---|---|---|---|---|---|
| Underdamped | 30.84 | 58.30 | 58.23 | 58.21 | 58.21 | 58.20 |
| Critically damped | 30.89 | 64.79 | 64.88 | 64.93 | 64.93 | 64.94 |
| Overdamped | 30.74 | 58.54 | 58.49 | 58.83 | 59.08 | 59.18 |
The variants achieve similar FineWeb-Edu test perplexities. On the task of length extrapolation, the underdamped model achieves the lowest perplexity at every evaluated length and remains stable through 32K. Its advantage over the overdamped model increases from 0.24 at 2K to 0.98 at 32K.
The training setup and full results have been added in Appendix D.4 and Appendix E.6 (Table 18).
12. Population-level phase statistics and readout ablations (Future work 2)
Thank you for this suggestion. We analyzed all 32 rotary state pairs in each evaluated Parity and Mod-3 model. A pair is classified as task-aligned if its phase increment lies within of for input 0 and of the task-specific target for input 1: for Parity and for Mod-3.
| Statistic | Parity | Mod-3 |
|---|---|---|
| Task-aligned pairs | 2 / 32 | 15 / 32 |
| Proportion | 6.25% | 46.88% |
All identified pairs are concentrated in head 0. Thus, the phase structure illustrated in Figure 5 appears in a subset of rotary pairs, with greater prevalence in Mod-3.
To assess their contribution, we performed two readout ablations on 1,024 newly sampled sequences per task:
- Removing all other rotary pairs yields scaled accuracies of 100% on Parity and 59.28% on Mod-3, compared with 100% and 94.87%, respectively without ablation.
- Removing the task-aligned pairs reduces scaled accuracy to 2.93% on Parity and −2.54% on Mod-3, both near chance.
These results quantify the prevalence of task-aligned phase patterns and show that the identified pairs contribute substantially to performance. The statistics, ablation procedure, and results are reported in Appendix E.1.
References
[1] Lahoti et al. (2026). Mamba-3: Improved Sequence Modeling using State Space Principles.
[2] Grazzi et al. (2025). Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues.
We sincerely thank you again for your careful review and constructive suggestions, and we welcome any further questions or discussion.
Response to Reviewer iFGy (Part I)
We sincerely thank Reviewer iFGy for the considerable time, care, and thought you devoted to reviewing our work. We deeply appreciate your close examination of the theoretical arguments and experimental evidence, as well as your recognition of the phase intervention study and controlled pretraining comparisons. We are particularly grateful for your concrete experimental suggestions and the reasoning behind them. Below we respond to your comments and questions.
1. Negative real eigenvalues and rotational dynamics (Q5)
During the rebuttal, we added an extra rotation-free Mamba-3 variant with signed transition gates by replacing αt=e−atΔt with αt=2e−atΔt−1, allowing for negative real eigenvalues. And the results(scaled Accuracy) are as bellow:
This variant achieves 100% scaled accuracy on Parity, matching the underdamped model, but only has 0.56% accuracy on Mod-3, compared with 95.26% for the underdamped model. As pointed out by the reviewer, the Parity result can be indeed explained by the spectrum: sign alternation can represent Parity’s two-state cycle, consistent with R(π)=−I. By contrast, Mod-3 involves a three-state cycle, naturally represented by a rotation of 2π/3. The Mod-3 result therefore supports the benefit of rotational dynamics beyond access to negative real eigenvalues.
We have added the results to Table 2 and the implementation details to Appendix D.1, highlighting the benefits of rotational dynamics beyond negative real eigenvalues alone.
2. Hidden-state statistics and the role of state norms (Q6)
During the rebuttal, we have added analyses of models trained from scratch and a norm-restoration intervention to examine the contribution of state magnitude.
From-scratch models. Under their native configurations, oscillatory models exhibit lower history–update cosine similarity and RMS state norms at all three scales (187M, 444M, and 1.5B; Table 7). The differences are particularly pronounced at 1.5B: at 64K, cosine similarity decreases from 0.45 to 0.28, and RMS norm from 53.07 to 23.75. Thus, the association between oscillation and reduced alignment also appears in models trained with their respective dynamics.
Growth across layers and heads. Although median RMS norms change little between 16K and 64K, this aggregate statistic can obscure growth in individual layer–head pairs. We define Rℓ,h as the ratio of book-averaged instantaneous state norms at 64K versus 2K for layer ℓ and head h. The proportion exceeding 1.5× growth (Rℓ,h>1.5 in the next table) in the 1.5B from-scratch models decreases from 4.36% without oscillation to 0.26% with oscillation. Such growth is rare in both smaller model sizes (0.00% for 187M and 0.52% to 0.13% for 444M), but is more frequent at the scale of 1.5B. We hypothesis that the growth may be one reason that explains the poor extrapolation of 1.5B model. Phase removal also increases substantial norm growth in the official pretrained models (Appendix E.5, Tables 16–17).
Norm-restoration intervention. For the official 1.5B checkpoints, we rescaled phase-removed states after each update to match native norms at every token, layer, and head, without retraining. The restored states are used for both readout and subsequent recurrence. At 64K:
Norm restoration substantially recovers performance but leaves a gap from the native models. This supports altered state magnitude as one contributor to the degradation under phase removal, while showing that restoring magnitude alone is insufficient. Full procedures appear in Appendix D.5, with results across context lengths in Table 5.