acceptodds
Under review as a conference paper at ICLR 2027

Understanding Mamba-3 through Oscillatory Dynamics

10 views

Abstract

Mamba-3 introduces input-dependent rotations into recurrent state updates, but their contribution beyond state retention remains unclear. We interpret these rotations as **underdamped second-order dynamics**, providing a framework for investigating the role of oscillation in sequence modeling. Controlled comparisons show that underdamped dynamics outperform non-oscillatory damping regimes on state-tracking tasks, while language-model pretraining demonstrates benefits for long-context prediction and length generalization. Interventions in pretrained models further establish the importance of learned phase, and state analysis suggests that rotation regulates how retained information combines with incoming updates. Taken together, these findings identify underdamped dynamics as a useful mechanism for recurrent computation and suggest that oscillation supports long-context processing by regulating how information accumulates.

Primary area:
learning on time series and dynamical systems
Keywords:
state space models, mamba-3, oscillation dynamics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_iFGyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper interprets Mamba-3's data-dependent decay-rotation transition, rho*R(theta), as an underdamped second-order linear system and derives the equivalent damped-oscillator equation, with damping set by decay and frequency by rotation. Using this view, it builds a family of transitions parameterized by kappa that share a spectral center and differ only in damping regime, and it compares underdamped, critically damped, and overdamped variants on parity, mod-3, and arithmetic. Underdamped dynamics win clearly.

The paper then studies language modeling in three ways. First, phase removal in pretrained 1.5B Mamba-3 models degrades LAMBADA far more than factual-relation prediction and sharply raises long-context PG-19 perplexity. Second, 187M and 444M models trained from scratch on 100B tokens with and without rotary channels show lower PG-19 perplexity with rotation and LongBench-E gains at 444M. Third, hidden-state analyses show that removing phase increases alignment between retained state and new writes and increases state norm, and write/readout interventions indicate that B and C need consistent phase. The authors conclude that oscillation supports state tracking and long-context modeling by regulating how information accumulates.

Critical strengths
  1. The intervention study in pretrained models (Section 5.3). Phase shuffle, removal, and reversal are compared against a decay-permutation control at several context lengths in both SISO and MIMO models. This is a more careful design than typical ablations, and it gives concrete evidence that long-context prediction in pretrained Mamba-3 depends heavily on learned phase and on how phases are assigned to state pairs. This is the most solid and most reusable part of the paper.

  2. The controlled pretraining comparison at meaningful scale (two model sizes, 100B tokens each), evaluated on PG-19 up to 64K, on LongBench-E, and on standard benchmarks. Few interpretability-oriented SSM papers include matched from-scratch training, and this comparison is what links the mechanistic story to practical capability.

Critical weaknesses
  1. The main controlled comparison does not isolate oscillation.

The critically damped and overdamped variants are discretized as exp(Delta*A(kappa)) with real eigenvalues, so all their discrete eigenvalues lie in (0, 1). Grazzi et al. (2025), which the paper cites, show that such positive real spectra cannot implement parity. The parity solution in Fig. 5 uses a rotation of about 180 degrees, and R(pi) = -I is itself a negative real eigenvalue, so the parity result does not distinguish oscillation from access to a non-positive spectrum. Mod-3 does genuinely require complex structure, but the paper does not make this argument.

The variants also differ in ways other than damping regime. In the critically damped variant (kappa = 0), the input-dependent nu_t plays no role, and the transition is a non-normal Jordan block. This may explain why it scores 1.97% on arithmetic, below both the overdamped variant (81.60%) and Mamba-3 without rotation (48.95%), and the paper does not explain this gap. In addition, the persistence analysis in Section 4.2 fixes omega_0, while the experiments fix the spectral center -a. Under the experimental setup, the slow overdamped mode decays more slowly than the underdamped modes, so the theory does not describe the variants that were trained. Finally, the language-modeling experiments compare rotation against scalar decay rather than against the damping regimes, so the second-order framework is tested only on small synthetic tasks. This weakness matters most because the paper's central claim concerns the role of oscillation, and this experiment is the only one designed to isolate it.

  1. The language-modeling evidence is confounded and statistically thin.

Phase removal in a model trained with phase is far out of distribution, and its damage grows with sequence length (Table 5). LAMBADA passages are much longer than Relation prompts, so the gap in Fig. 2 may reflect input length rather than context dependence. The state-norm and alignment statistics in Table 6 are measured under this same out-of-distribution intervention, not on the matched from-scratch models. The authors note that the from-scratch model without rotation extrapolates stably, which weakens the state-norm explanation.

The from-scratch results come from single runs without variance estimates, and the effects are small or mixed. FineWeb-Edu perplexity differs by about 0.1. At 187M, the rotation model has slightly lower average downstream accuracy and is about 3 points worse on Arc-C. LongBench-E scores near 10 are close to the floor at these scales, and most of the 444M gain comes from four tasks. With two model sizes and one seed each, the data do not support the claim that downstream gains "emerge at larger scale." The evaluation on additional datasets extends only to 4K tokens, twice the training length. This matters because the paper's second and third headline findings depend on these experiments.

Questions for authors
  • Is the underdamped variant in Table 2 implemented through exp(Delta*A(kappa)) in companion form, or through Mamba-3's native rotation of B and C? If the latter, how are input injection and readout matched across the variants?
  • How is "scaled accuracy" defined, what value of c is used for the overdamped variant, and how many seeds stand behind each number in Tables 2 through 4?
  • What explains the critically damped variant scoring far below both the overdamped variant and Mamba-3 without rotation on arithmetic?
  • Does removing the rotary channels change the parameter count of the from-scratch models?

Experiments whose results could change my assessment:

  • Does a non-oscillatory variant that allows negative real eigenvalues (for example, with gates in [-1, 1] as in Grazzi et al.) match the underdamped model on parity, and how does it perform on mod-3?
  • Do the Table 6 statistics (state-write alignment and RMS state norm) differ between the two from-scratch models?
  • How large is the run-to-run variance of the PG-19 and LongBench-E gaps at 444M?
  • Does the LAMBADA-versus-Relation gap remain when input length is controlled, for example by placing relation queries inside long distractor contexts?

Suggestions for future work (not needed for my assessment):

  • Train the damping-regime variants of Eq. 18 at language-model scale.
  • Report population-level statistics for the task-aligned phase structure in Fig. 5, rather than relying on hand-selected state pairs.
4 comments
AuthorsOpenReview ↗

Response to Reviewer iFGy (Part I)

We sincerely thank Reviewer iFGy\textcolor{green}{\textbf{Reviewer iFGy}} for the considerable time, care, and thought you devoted to reviewing our work. We deeply appreciate your close examination of the theoretical arguments and experimental evidence, as well as your recognition of the phase intervention study and controlled pretraining comparisons. We are particularly grateful for your concrete experimental suggestions and the reasoning behind them. Below we respond to your comments and questions.

1. Negative real eigenvalues and rotational dynamics (Q5)

During the rebuttal, we added an extra rotation-free Mamba-3 variant with signed transition gates by replacing αt=e−atΔt\alpha_t=e^{-a_t\Delta_t} with α~t=2e−atΔt−1\widetilde{\alpha}_t=2e^{-a_t\Delta_t}-1, allowing for negative real eigenvalues. And the results(scaled Accuracy) are as bellow:

ModelParityMod 3Arith. w/o Brackets
Mamba-23.910.0436.76
Mamba-3 w/o rotation3.030.7848.95
Mamba-3 w/ Static rotation4.050.8525.49
Mamba-3 w/o rotation, decay ∈[−1,1]\in [-1,1]100.00.5671.55
Overdamped3.270.5881.60
Critically Damped3.220.321.97
Underdamped (Mamba-3)100.095.2695.68

This variant achieves 100% scaled accuracy on Parity, matching the underdamped model, but only has 0.56% accuracy on Mod-3, compared with 95.26% for the underdamped model. As pointed out by the reviewer, the Parity result can be indeed explained by the spectrum: sign alternation can represent Parity’s two-state cycle, consistent with R(π)=−IR(\pi)=-I. By contrast, Mod-3 involves a three-state cycle, naturally represented by a rotation of 2π/32\pi/3. The Mod-3 result therefore supports the benefit of rotational dynamics beyond access to negative real eigenvalues.

We have added the results to Table 2 and the implementation details to Appendix D.1, highlighting the benefits of rotational dynamics beyond negative real eigenvalues alone.

2. Hidden-state statistics and the role of state norms (Q6)

During the rebuttal, we have added analyses of models trained from scratch and a norm-restoration intervention to examine the contribution of state magnitude.

From-scratch models. Under their native configurations, oscillatory models exhibit lower history–update cosine similarity and RMS state norms at all three scales (187M, 444M, and 1.5B; Table 7). The differences are particularly pronounced at 1.5B: at 64K, cosine similarity decreases from 0.45 to 0.28, and RMS norm from 53.07 to 23.75. Thus, the association between oscillation and reduced alignment also appears in models trained with their respective dynamics.

Growth across layers and heads. Although median RMS norms change little between 16K and 64K, this aggregate statistic can obscure growth in individual layer–head pairs. We define Rℓ,hR_{\ell,h} as the ratio of book-averaged instantaneous state norms at 64K versus 2K for layer ℓ\ell and head hh. The proportion exceeding 1.5× growth (Rℓ,h>1.5R_{\ell,h}>1.5 in the next table) in the 1.5B from-scratch models decreases from 4.36% without oscillation to 0.26% with oscillation. Such growth is rare in both smaller model sizes (0.00% for 187M and 0.52% to 0.13% for 444M), but is more frequent at the scale of 1.5B. We hypothesis that the growth may be one reason that explains the poor extrapolation of 1.5B model. Phase removal also increases substantial norm growth in the official pretrained models (Appendix E.5, Tables 16–17).

ParamsTrainingRℓ,h>1.5R_{\ell,h}>1.5Rℓ,h>2R_{\ell,h}>2
187MWith rotation0.00% (0)0.00% (0)
187MWithout rotation0.00% (0)0.00% (0)
444MWith rotation0.13% (1)0.00% (0)
444MWithout rotation0.52% (4)0.13% (1)
1.5BWith rotation0.26% (4)0.20% (3)
1.5BWithout rotation4.36% (67)1.76% (27)

Norm-restoration intervention. For the official 1.5B checkpoints, we rescaled phase-removed states after each update to match native norms at every token, layer, and head, without retraining. The restored states are used for both readout and subsequent recurrence. At 64K:

ModelNative PPL ↓Phase removalPhase removal + norm restoration
SISO20.60851.4947.58
MIMO20.52309.3665.47

Norm restoration substantially recovers performance but leaves a gap from the native models. This supports altered state magnitude as one contributor to the degradation under phase removal, while showing that restoring magnitude alone is insufficient. Full procedures appear in Appendix D.5, with results across context lengths in Table 5.

AuthorsOpenReview ↗

Response to Reviewer iFGy (Part II)

3. Run-to-run variability on PG-19 and LongBench-E at 444M (Q7)

During the rebuttal, we have added two independent training runs per configuration, bringing the 444M comparison to three training seeds each, with and without oscillation. All runs use the same pretraining setup and 100B-token budget. The table summarizes mean ± std across training seeds, with representative context lengths for PG-19.

ConfigurationPG-19 2K PPL ↓PG-19 16K PPL ↓PG-19 64K PPL ↓LongBench-E average ↑
Without oscillation27.17 ± 0.2127.97 ± 0.6326.92 ± 0.8111.70 ± 0.82
With oscillation26.80 ± 0.1227.05 ± 0.3025.87 ± 0.2012.53 ± 0.64

PG-19. Oscillation lowers mean perplexity at all seven evaluated context lengths from 1K to 64K. The reduction increases from 0.37 at 2K to 1.05 at 64K, extending beyond the 2K training context. The oscillatory configuration also exhibits smaller std at every evaluated length across these three runs.

LongBench-E. Oscillation improves the mean aggregate score by 0.83 points. We compute the 13-task average separately for each training seed, then report the mean and std of these averages.

These additional runs support the observed mean improvements while quantifying training variability. Table 3 reports the updated LongBench-E results and Appendix E.4 provides per-task mean and std.

4. LAMBADA versus Relations under matched input lengths (Q8)

We added a length-matched comparison using the official Mamba-3 1.5B SISO and MIMO checkpoints.

We sample as many LAMBADA examples as Relations queries and pair them. Each Relations query is prepended with unrelated text to match its paired LAMBADA input length, preserving the original query and target answer. We repeat this construction with three distractor seeds. The table reports accuracy drops in percentage points following phase removal, averaged equally over all 300 contiguous layer windows. For length-matched Relations, mean ± sample std is computed across the three distractor seeds.

ModelRelations, originalRelations, length-matchedLAMBADA
SISO0.084.77 ± 0.1911.79
MIMO−0.196.32 ± 0.4120.19

Longer distractor contexts increase Relations’ sensitivity to phase removal, but its accuracy drops remain substantially smaller than LAMBADA’s in both models. Thus, matching input length does not eliminate the difference between tasks. The greater sensitivity of LAMBADA is consistent with a role for learned phase in context-dependent prediction.

The procedure and results have been reported in Appendix D.2 and Appendix E.2 (Table 13), with discussion in Section 5.1.

5. Implementation of different damping regimes (Q1)

For the damping-regime comparison in Table 2, all three variants use the same companion-form matrix exponential:

\exp\left(\Delta_t A_t(\kappa_t)\right), \quad A_t(\kappa_t)= \begin{pmatrix} 0 & 1 \\\\ -(a_t^2-\kappa_t) & -2a_t \end{pmatrix}.$$ We set $\kappa_t=-\nu_t^2$ for underdamping, $\kappa_t=0$ for critical damping, and $\kappa_t=r_t^2$ for overdamping. In each case, the discretized transition acts **directly on the recurrent hidden state**. The variants share the same state dimension, input-injection and readout rules, input-dependent step-size parameterization, and training protocol. These implementation details have been added in Section 4.1 and Appendix D.1 of the revised manuscript. ## 6. Scaled accuracy, the choice of c, and training seeds (Q2) **Scaled accuracy.** Following [1, 2], we define $$a_{\mathrm{scaled}} = \frac{a-a_{\mathrm{chance}}}{1-a_{\mathrm{chance}}},$$ where $a$ is the measured accuracy and $a_{\mathrm{chance}}$ is the task-specific chance accuracy: $0.5$ for Parity and $1/3$ for Mod-3. This definition is now explicitly provided in Appendix D.1. **Choice of c.** We use $c=1$ in the overdamped parameterization as specified in Appendix D.1. **Training seeds and model selection.** Following the Mamba-3 state-tracking protocol [1], we sweep **eight learning rates** logarithmically spaced between $10^{-4}$ and $10^{-2}$, with **two independent training seeds per learning rate**, yielding **16 runs per model–task combination**. We select the run with the highest validation accuracy and report its held-out test accuracy. Thus, **each Table 2 entry represents a single validation-selected run**.
AuthorsOpenReview ↗

Response to Reviewer iFGy (Part III)

7. Performance of the critically damped variant on arithmetic (Q3)

Our hypothesis is that critical damping limits the flexibility of individual state blocks. Overdamped blocks have two distinct decay modes, −a±r-a\pm r, whereas critical blocks have repeated eigenvalues at −a-a and lack independent modulation through νt\nu_t. All arithmetic variants use three layers, allowing for different decay rates across layers and channels. However, this restriction remains valid within each critical block.

This difference may contribute to the performance gap, but does not fully explain the 1.97% scaled accuracy. The relative contributions of optimization and length generalization remain unclear.

8. Parameter counts across model scales (Q4)

Removing the rotary channels slightly reduces the parameter count. The exact counts are shown below.

Model sizeWithout rotationWith rotationAdditional parametersRelative increase
187M186,997,056187,291,968294,9120.158%
444M443,408,896444,195,328786,4320.177%
1.5B1,495,510,0161,497,082,8801,572,8640.105%

The relative increase is calculated with respect to the model without rotation. Adding rotation increases the parameter count by less than 0.2% at every evaluated scale, so the configurations remain closely matched in parameter budget.

9. Persistence analysis and learned damping parameters (W1)

Section 4.2 holds ω0\omega_0 fixed to compare persistence across damping regimes. The conclusion that critical damping gives the fastest asymptotic attenuation applies specifically to this comparison.

In the experiments, ata_t is input-dependent and learned separately by each variant. The shared spectral center −at-a_t in Section 4.1 describes the common parameterization, and its value is not fixed across regimes.

Our analysis aims to provide an interpretation under a specified comparison condition, while the experiments assess performance when the damping parameters are learned from data.

10. Evaluation lengths on additional datasets (W2)

Our evaluations on CodeParrot, Math-Hard, TriviaQA, and SlimPajama follow the evaluation design in Figure 5 of [2], using input lengths up to 4K tokens in our experiments. We have also added the corresponding results for the 1.5B models with and without oscillation. These new results have been reported in Appendix E.3, Figure 6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer iFGy (Part IV)

11. Language modeling under different damping regimes (Future work 1)

Thank you for this suggestion. We compared all three damping regimes using about 50M-parameter models trained on 2B FineWeb-Edu tokens, with the GPT-2 tokenizer, a 2K training context, and otherwise identical architectures and training settings. This scale accommodates the higher computational cost of our explicit second-order implementation.

We report FineWeb-Edu test perplexity at 2K and PG-19 perplexity at context lengths from 2K to 32K. Lower is better.

Damping regimeFineWeb-Edu test2K4K8K16K32K
Underdamped30.8458.3058.2358.2158.2158.20
Critically damped30.8964.7964.8864.9364.9364.94
Overdamped30.7458.5458.4958.8359.0859.18

The variants achieve similar FineWeb-Edu test perplexities. On the task of length extrapolation, the underdamped model achieves the lowest perplexity at every evaluated length and remains stable through 32K. Its advantage over the overdamped model increases from 0.24 at 2K to 0.98 at 32K.

The training setup and full results have been added in Appendix D.4 and Appendix E.6 (Table 18).

12. Population-level phase statistics and readout ablations (Future work 2)

Thank you for this suggestion. We analyzed all 32 rotary state pairs in each evaluated Parity and Mod-3 model. A pair is classified as task-aligned if its phase increment lies within 15∘15^\circ of 0∘0^\circ for input 0 and of the task-specific target for input 1: 180∘180^\circ for Parity and 120∘120^\circ for Mod-3.

StatisticParityMod-3
Task-aligned pairs2 / 3215 / 32
Proportion6.25%46.88%

All identified pairs are concentrated in head 0. Thus, the phase structure illustrated in Figure 5 appears in a subset of rotary pairs, with greater prevalence in Mod-3.

To assess their contribution, we performed two readout ablations on 1,024 newly sampled sequences per task:

  • Removing all other rotary pairs yields scaled accuracies of 100% on Parity and 59.28% on Mod-3, compared with 100% and 94.87%, respectively without ablation.
  • Removing the task-aligned pairs reduces scaled accuracy to 2.93% on Parity and −2.54% on Mod-3, both near chance.

These results quantify the prevalence of task-aligned phase patterns and show that the identified pairs contribute substantially to performance. The statistics, ablation procedure, and results are reported in Appendix E.1.

References

[1] Lahoti et al. (2026). Mamba-3: Improved Sequence Modeling using State Space Principles.

[2] Grazzi et al. (2025). Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues.


We sincerely thank you again for your careful review and constructive suggestions, and we welcome any further questions or discussion.


Reviewer_WKXjOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates rotations in Mamba-3 through second-order underdamped oscillatory dynamics. It provides a thorough study by combining controlled architectural comparisons, state tracking and pretraining, and phase intervention experiments.

Critical strengths
  • The paper is well-written and easy to follow. The motivation and research questions are well formulated.
  • It gives important insights into the inner workings of the Mamba-3 architecture (and beyond in the Appendix).
  • The theoretical contributions and evaluations (state tracking and language modeling) are well connected to each other. The results are presented in a very understandable way. I found the phase interventions in Section 5.3 particularly interesting.
Critical weaknesses
  • I can not point out any critical weaknesses in this paper.
Questions for authors
  • Throughout the paper, results appear to be reported from single runs. Could the authors comment on the variability across multiple training runs where applicable (e.g. Section 5.2)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer WKXj

We thank Reviewer WKXj\textcolor{orange}{\textbf{Reviewer WKXj}} for the positive assessment of our work. We appreciate your recognition of the connection between our theoretical analysis and empirical evaluations, and your interest in the phase interventions in Section 5.3. We address your question below.

1. Variability across training runs (Q1)

During the rebuttal, we have supplemented the pretraining experiments in Section 5.2 with two additional independent runs for each configuration (with and without oscillation) at both 187M and 444M, bringing the total to three training seeds per configuration.

The updated results show higher mean aggregate scores with oscillation at both scales. For example, at 444M, the LongBench-E average increases from 11.70 ± 0.82 to 12.53 ± 0.64, and the average accuracy on standard downstream tasks increases from 50.35 ± 0.30% to 50.86 ± 0.18%. These values are means ± standard deviations across three training seeds.

For the 187M and 444M models, the revised manuscript now includes:

  • Figure 3: Mean PG-19 perplexity with SEM error bars across training seeds for 187M and 444M models.
  • Tables 3 and 4: Mean task scores and mean ± SD for the aggregate scores.
  • Appendix E.4: Per-task means and standard deviations.

The 1.5B results remain based don one single training run per configuration due to the high compute cost. These additions quantify the magnitude and variability of the improvements observed in Section 5.2.


Thank you again for this helpful suggestion. We welcome further questions and discussion.


Thank you for including the multiple seed results. I do not have any further questions.

Thank you for your encouraging feedback and support of our work. We sincerely appreciate your positive assessment and recognition of our contributions, as well as your valuable suggestion to evaluate variability across multiple training seeds.

Reviewer_yEm6OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper interprets the input-dependent rotation mechanism of Mamba-3 as underdamped second-order oscillatory dynamics. Under the assumption of piecewise frozen parameters, the authors derive a mathematical equivalence between Mamba-3's decay-rotation recurrent update and a damped harmonic oscillator. Through controlled experiments comparing underdamped, critically damped, and overdamped regimes, they demonstrate that oscillatory dynamics offer advantages on state-tracking tasks. They further conduct language model pretraining and phase perturbation intervention experiments, which show that the learnable phase can regulate the constructive accumulation of recurrent states and improve length generalization over long contexts. Finally, the authors extend this perspective to various linear attention designs.

Critical strengths
  1. This is a paper with a clear motivation and valuable conclusions. Starting from the classical second-order damped system, the authors provide a reasonable interpretation of Mamba-3's decay-rotation state transition, showing that underdamped dynamics can preserve the two-dimensional state geometry, whereas overdamped and critically damped regimes struggle to maintain it. This is a valuable conceptual contribution to the understanding of state-space models and, in part, linear attention models.

  2. The synthetic state-tracking tasks are well designed. The experiments cover both SISO and MIMO variants, and the empirical evidence is relatively rich.

Critical weaknesses
  1. The current experimental results do not account for training randomness across each variant. Some of the observed gains on certain tasks appear relatively marginal, and reporting means and error bars over multiple runs would help avoid the influence of random fluctuations on the results.

  2. The analysis on additional linear architectures is meaningful, but it remains purely theoretical. Whether these conclusions hold in practice should be further verified, which would substantially strengthen the persuasiveness and value of this work.

  3. Performance gains are not always present, and degradation may occur on some tasks. The authors should analyze such cases in light of the characteristics of the specific tasks.

Questions for authors
  1. The proposed perspective also appears to be related to rotary position embeddings. Could the authors provide at least a brief analysis or discussion of the connection between the two?

  2. As an important class of linear attention architectures, could the proposed perspective also apply to the RWKV family?

5 comments
AuthorsOpenReview ↗

Response to Reviewer yEm6 (Part I)

We sincerely thank Reviewer yEm6\textcolor{blue}{\textbf{Reviewer yEm6}} for the careful review and constructive feedback. We particularly appreciate your suggestions on assessing variability across training runs and extending our analysis and experiments beyond Mamba-3. Below we address your concerns and questions.

1. Variability across training runs (W1)

Thank you for highlighting the importance of accounting for training randomness. We conducted two additional independent training runs for each configuration (with and without oscillation) at both the 187M and 444M scales, bringing the total to three training seeds per configuration. The aggregate results are summarized below as mean ± std across three training seeds.

Model sizeDynamicsLongBench-E averageStandard downstream average accuracy (%)
187MWithout oscillation9.12 ± 0.9543.31 ± 0.22
187MWith oscillation9.45 ± 0.2643.76 ± 0.28
444MWithout oscillation11.70 ± 0.8250.35 ± 0.30
444MWith oscillation12.53 ± 0.6450.86 ± 0.18

In the revised manuscript, Figure 3 reports mean PG-19 perplexity with error bars indicating the standard error of the mean (SEM) across three training seeds for the 187M and 444M models. For the 187M and 444M models, Tables 3 and 4 report mean task scores and the mean ± std of the aggregate scores. Appendix E.4 provides detailed per-task statistics.

2. Applicability Beyond Mamba-3 (W2)

We agree that empirical evidence beyond Mamba-3 would strengthen the theoretical connections discussed in Appendix A. During the rebuttal, we additionally discuss related architectural developments, present our evaluation of publicly released checkpoints, and report a new from-scratch training comparison below.

2.1 Related architectural developments

We find that three concurrent ICLR 2027 submissions similarly choose to explore rotational dynamics in linear recurrent architectures. These works provide complementary architectural connections to our dynamical perspective.

Oscillatory Delta Networks: Phase-Addressed Associative Memory [1]: This work introduces rotation along the key dimension of the recurrent state:

St=ρt(I−βtktkt⊤)Rt(k)St−1+βtktvt⊤.S_t = \rho_t \left(I-\beta_t k_tk_t^\top\right) R_t^{(k)}S_{t-1} + \beta_t k_tv_t^\top.

Here, Rt(k)∈Rdk×dkR_t^{(k)} \in \mathbb{R}^{d_k \times d_k} rotates the retained memory before the key-dependent delta-rule correction, while ρt\rho_t controls scalar retention. The transition therefore controls both the magnitude and orientation of previously stored information.

Rotary-state Gated DeltaNet: Enhancing State Tracking through Rotation [2]: This work applies rotation along the value dimension:

St=ρt(I−βtktkt⊤)St−1(Rt(v))⊤+βtktvt⊤.S_t = \rho_t \left(I-\beta_t k_tk_t^\top\right) S_{t-1}\left(R_t^{(v)}\right)^\top + \beta_t k_tv_t^\top.

Here, Rt(v)∈Rdv×dvR_t^{(v)} \in \mathbb{R}^{d_v \times d_v}. The left-side delta-rule operator performs key-dependent correction, while the right-side rotation transforms the stored value representations.

CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory [3]: CyFA maintains separate, aligned key and value memories organized into relative-time slots. At each step, a learned clock increment δt\delta_t transports both memories before a new key–value pair is written into the age-zero slot:

K_t &= \alpha_t P(\delta_t)K_{t-1} + \beta_t e_0k_t^\top \end{aligned}$$ $$\begin{aligned} V_t &= \alpha_t P(\delta_t)V_{t-1} + \beta_t e_0v_t^\top. \end{aligned}$$ Here, $\alpha_t$ controls forgetting, $\beta_t$ controls writing, and $e_0$ is the one-hot vector selecting the age-zero slot. A unit cyclic shift includes a wraparound connection: $$P(1)= \sum_{i=0}^{m-2} e_{i+1}e_i^\top + e_0e_{m-1}^\top.$$ For an odd number of slots $m$, the fractional cyclic transport admits the real Fourier representation $$P(\tau)=\Phi U(\tau)\Phi^\top,$$ $$U(\tau) = 1\oplus \bigoplus_{j=1}^{(m-1)/2} \operatorname{Rot}\left(\frac{2\pi j\tau}{m}\right),$$ where $\Phi$ is an orthogonal real Fourier basis. Thus, $P(\tau)$ describes transport in slot coordinates, while $U(\tau)$ expresses the same operation through rotation blocks in Fourier coordinates. Combined with the forget gate $\alpha_t$, each nonconstant Fourier mode undergoes decay–rotation dynamics. **We note that all three concurrent submissions report improved performance on language tasks with their respective architectures, which may imply the emerging trend of incorporating rotational dynamics in architecture design, and support the timely contribution of our work.**
AuthorsOpenReview ↗

Response to Reviewer yEm6 (Part II)

2.2 Evaluation of publicly released checkpoints

To complement our theoretical analysis of Complex KDA [4] in Appendix A.8, we additionally evaluated the publicly released 1.3B KDA and Complex KDA checkpoints on LongBench-E and PG-19 during the rebuttal.

LongBench-E. The results across all 13 tasks are summarized below. Higher scores are better.

ModelPCPRGRMNMQAQA2WMHQASSTRTQALCCRBAvg.
KDA3.246.479.1110.3413.325.677.956.8024.7523.0023.6917.9520.0513.26
Complex KDA3.145.898.997.6013.366.768.236.6026.0630.0026.6624.0624.3814.75

PG-19. We additionally evaluated perplexity at context lengths from 2K to 64K. Lower values are better.

Model2K4K8K16K32K64K
KDA14.2814.2114.3414.1813.7114.14
Complex KDA13.7413.7813.1913.6413.3213.72

Complex KDA improves the average LongBench-E score from 13.26 to 14.75. It also achieves lower PG-19 perplexity at all six evaluated context lengths. The consistent improvements across benchmarks demonstrate that oscillations can indeed benefit long-context generalization.

We have added the evaluation setup to Appendix D.7 and the complete results to Appendix E.7, Tables 19 and 20 of the revised manuscript.

2.3 New from-scratch training

Beyond existing works that combine oscillations and sequence models, we additionally conducted a new from-scratch comparison using a hybrid architecture combining Gated DeltaNet-2 (GDN-2) [5] with MLA full attention. We use 10B tokens from Fineweb edu dataset and the model size is about 140M.

ModelParametersTest PPL ↓LAMBADA PPL ↓LAMBADA Acc. ↑HellaSwag ↑PIQA ↑ARC-E ↑ARC-C ↑WinoGrande ↑OBQA ↑Avg. ↑
GDN-2 + MLA140M20.3274.9827.6932.5763.9853.2425.0050.5119.8038.97
GDN-2 + MLA + rotation140M20.1867.8529.2532.6964.0954.5524.5752.2519.2039.51

Adding rotation increases average downstream accuracy from 38.97% to 39.51%. We have added the training configuration to Appendix D.7 and the full results to Appendix E.7, Table 21 of the revised manuscript.

Together, these related architectural developments, pretrained-checkpoint evaluations, and from-scratch experiments provide complementary support for the applicability of our dynamical perspective beyond Mamba-3.

3. Task-dependent benefits (W3)

We agree that the benefits are task-dependent and have examined the updated results in more detail.

Language modeling and length extrapolation. Oscillation lowers FineWeb-Edu validation perplexity at all three scales: 16.33 → 16.17 at 187M, 12.69 → 12.61 at 444M, and 10.34 → 10.24 at 1.5B (Table 4). It also lowers PG-19 perplexity beyond the 2K training context up to 64K, with more pronounced long-context benefits at the larger scales (Figure 3).

Contextual prediction, summarization, and code completion. The table reports changes from adding oscillation, using three-seed means for 187M and 444M and single-run results for 1.5B. LAMBADA, SAMSum, LCC, and RepoBench-P improve at all three scales.

Model sizeLAMBADA PPL ↓LAMBADA Acc. ↑SAMSum ↑LCC ↑RepoBench-P ↑
187M−1.83+1.64+1.21+2.35+2.88
444M−1.56+1.49+4.78+2.17+2.77
1.5B−1.05+2.14+6.14+3.22+3.12

LAMBADA requires contextual word prediction, SAMSum requires integrating dialogue content, and the code-completion tasks require using preceding code and, for RepoBench-P, cross-file dependencies. These improvements are consistent with our interpretation that oscillation supports the retention and integration of contextual information.

Mixed results and degradations. ARC-E and ARC-C show small, mixed differences, while OBQA decreases by 1.60% and 1.20% at 444M and 1.5B, respectively. Since these tasks also depend on factual knowledge and reasoning, the improved recurrent information accumulation alone may not directly translate into higher accuracy. MultiNews likewise decreases at 187M and 444M, indicating that benefits on dialogue summarization do not uniformly extend to multi-document summarization, which additionally requires selecting and combining information across sources.

Taken all the results together, we therefore interpret oscillation as a useful mechanism for contextual information processing whose benefits depend on the demands of the particular tasks.

AuthorsOpenReview ↗

Response to Reviewer yEm6 (Part III)

4. Relationship to rotary positional embeddings (Q1)

RoPE and our perspective share a rotational structure, with a direct connection through recurrent computation.

Relative position through phase. Standard RoPE [6] rotates queries and keys according to their positions. For a two-dimensional channel pair,

q_t^\top R((s-t)\omega)k_s.$$ The attention score therefore depends on token content and relative displacement $s-t$. The rotary factor preserves norms and represents undamped phase evolution along the position coordinate. **Rotation in recurrent dynamics.** With scalar decay and rotation, the recurrent update takes the form $$H_t=\rho_tR(\theta_t)H_{t-1}+k_tv_t^\top, \qquad y_t=H_t^\top q_t.$$ Under piecewise-frozen parameters, the decay–rotation transition admits a second-order oscillatory interpretation. **Decay controls the magnitude of retained history, while rotation controls its orientation relative to incoming updates and readout directions.** Phase therefore regulates how historical contributions reinforce or cancel during accumulation. ## 5. Applicability to the RWKV family (Q2) We have extended our analysis to **RWKV-7** [7] in Appendix A.5 and updated Table 9. Our perspective characterizes both its single-step decay dynamics and the oscillatory dynamics that can emerge across successive updates. **Single-step dynamics.** Under frozen parameters, RWKV-7’s homogeneous transition can be written as $$T=W-L\widehat{\kappa}\widehat{\kappa}^{\top},$$ where $W$ and $L$ are positive diagonal gate matrices and $\|\widehat{\kappa}\|_2=1$. Defining $z=L^{1/2}\widehat{\kappa}$, we obtain $$L^{-1/2}TL^{1/2}=W-zz^\top.$$ This matrix is symmetric, with real eigenvalues appearing in each individual transition. Under the standard gate constraints, these eigenvalues lie in $(-1,1)$. Pairs of distinct positive eigenvalues admit overdamped second-order representations, while negative eigenvalues produce alternating decay. **Oscillatory dynamics across steps.** Products of different RWKV-7 transitions can nevertheless have non-real conjugate eigenvalues. In Appendix A.5, we construct two transitions $$T_i=wI-a\widehat{\kappa}_i\widehat{\kappa}_i^\top$$ with scalar gates satisfying the standard parameter constraints and $a>w$. For directions separated by an angle $\theta$, their product $P=T_2T_1$ has a non-real conjugate pair whenever $$0<\cos^2\theta<\frac{4w(a-w)}{a^2}.$$ Writing its eigenvalues as $\rho e^{\pm i\phi}$, where $\rho=w(a-w)\in(0,1)$ and $\phi\in(0,\pi)$, gives an underdamped realization in real modal coordinates: $$\ddot u+2\gamma\dot u+(\gamma^2+\Omega^2)u=0, \qquad \gamma=-\frac{\log\rho}{2\Delta}>0, \quad \Omega=\frac{\phi}{2\Delta}>0.$$ Here, $\Delta>0$ is the duration of one step. The damping ratio is $\zeta=\gamma/\sqrt{\gamma^2+\Omega^2}<1$. When the transition pair is repeated, this continuous-time system exactly reproduces the discrete states at two-step intervals. Thus, our perspective applies to RWKV-7 at both the individual-transition and composition levels: Mamba-3 directly implements decay–rotation within each update, while RWKV-7 can produce an oscillatory mode through the composition of successive updates. **References** [1] [Oscillatory Delta Networks: Phase-Addressed Associative Memory](https://openreview.net/forum?id=rKJBCTurUb). ICLR 2027 submission. [2] [Rotary-state Gated DeltaNet: Enhancing State Tracking through Rotation](https://openreview.net/forum?id=jYHHPWZOhr). ICLR 2027 submission. [3] [CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory](https://openreview.net/forum?id=MrNCcG3NCj). ICLR 2027 submission. [4] Siems et al. (2026). [Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention](https://arxiv.org/abs/2609.24797). [5] Hatamizadeh et al. (2026). [Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention](https://arxiv.org/abs/2605.22791). [6] Su et al. (2024). [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864). [7] Peng et al. (2025). [RWKV-7 "Goose" with Expressive Dynamic State Evolution](https://arxiv.org/abs/2503.14456). --- We sincerely thank you again for your thoughtful feedback and constructive suggestions. We welcome further discussion and would be happy to address any remaining questions. ---

Comments on the paper

1 comment
AuthorsOpenReview ↗

To All Reviewers: Updated Manuscript and Appendix

We thank all reviewers for their constructive feedback. In addition to our individual responses, we have revised the manuscript and appendix to address the questions and suggestions raised. All revisions are marked in red. The main changes are summarized below:

  1. @Reviewer yEm6\textcolor{blue}{\textbf{Reviewer yEm6}}, @Reviewer WKXj\textcolor{orange}{\textbf{Reviewer WKXj}}, and @Reviewer iFGy\textcolor{green}{\textbf{Reviewer iFGy}}: Variability across training runs. We expanded the 187M and 444M experiments to three seeds per configuration. Tables 3–4 report mean task scores and mean ± std for aggregate scores; Appendix E.4, Tables 14–15 provide per-task mean and std. Figure 3 now additionally includes SEM error bars across these seeds for 187M and 444M model.

  2. @Reviewer yEm6\textcolor{blue}{\textbf{Reviewer yEm6}}: Generalization to other architectures. We added KDA/Complex KDA evaluations and from-scratch training of GDN2 + MLA with and without oscillation. Settings and results appear in Appendices D.7 and E.7 (Tables 19–21).

  3. @Reviewer yEm6\textcolor{blue}{\textbf{Reviewer yEm6}}: Extension to the RWKV family. We added an analysis of RWKV-7’s single-step and multi-step dynamics in Appendix A.5, summarized in Table 9.

  4. @Reviewer iFGy\textcolor{green}{\textbf{Reviewer iFGy}}: Negative real eigenvalues. We added a rotation-free variant with transition gates in [−1,1][-1,1] with results in Table 2 and implementation details in Appendix D.1.

  5. @Reviewer iFGy\textcolor{green}{\textbf{Reviewer iFGy}}: Alignment and state norms. We added from-scratch alignment and norm statistics (Table 7), analyses of substantial norm growth across layers and heads (Appendix E.5, Tables 16–17), and norm-restoration interventions (Table 5). Implementation details are provided in Appendix D.5.

  6. @Reviewer iFGy\textcolor{green}{\textbf{Reviewer iFGy}}: Length-matched comparison of Relations and LAMBADA. We added results in Appendix E.2, Table 13, with implementation details in Appendix D.2.

  7. @Reviewer iFGy\textcolor{green}{\textbf{Reviewer iFGy}}: Larger-scale evaluation. We added 1.5B models with and without oscillation, including standard and LongBench-E evaluations (Tables 3–4), PG-19 length generalization (Figure 3), and evaluations on additional datasets (Appendix E.3, Figure 6).

  8. @Reviewer iFGy\textcolor{green}{\textbf{Reviewer iFGy}}: Language modeling under different damping regimes. We added a comparison of all three damping regimes, with training settings in Appendix D.4 and results in Appendix E.6, Table 18.

  9. @Reviewer iFGy\textcolor{green}{\textbf{Reviewer iFGy}}: Population-level phase statistics and readout ablations. We added statistics across all 32 rotary pairs per model and ablations removing either the task-aligned pairs or the remaining rotary pairs in Appendix E.1.

We appreciate the reviewers’ suggestions, which have helped us clarify the contributions, practical relevance, and scope of the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.