- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
As far as I understand, the authors argue that current linear recurrent language models can only decay their state along the real axis and cannot make it oscillate, based on an analysis that freezes the transition at a single time step. They propose Mira, a recurrent layer built from a stack of learned 2D rotations, so that its state can rotate as well as decay while staying stable. A 0.14B model trained on 3B tokens is reported to beat Mamba, RWKV and attention baselines.
Critical strengths
The lattice-ladder parameterisation with a Schur stability guarantee and a chunkwise scan is a reasonable piece of engineering.
Critical weaknesses
The premise doesn't seem new to me nor accurate as stated: complex-valued recurrences are in S4D-style SSMs and in the LRU (Orvieto et al., 2023, "Resurrecting Recurrent Neural Networks for Long Sequences", https://arxiv.org/abs/2303.06349), and negative eigenvalues are studied by Grazzi et al., 2025 ("Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues", https://arxiv.org/abs/2411.12537), yet neither is discussed or compared against. Moreover, the frozen-system analysis (A.4) says little about time-varying models, since products of symmetric transitions can rotate (Siems et al., 2025, "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products", https://arxiv.org/abs/2502.10297, not mine), and A.4.6 itself concedes that RetNet has complex factors, contradicting Table 1.
The experiments can't support "outperforms strong recurrent baselines": there are no downstream evaluations, and all models are trained with an untuned peak learning rate of 2e-5 (line 418). That is about 30x below what the baselines' own papers use at this scale (6e-4 for Mamba at 125M, Gu and Dao, 2023, App. E.2, https://arxiv.org/abs/2312.00752; 6e-4 for RWKV-7 at 0.1B, Peng et al., 2025, Table 17, https://arxiv.org/abs/2503.14456), so the ranking may reflect sensitivity to the learning rate rather than architecture. Similarly, throughput measured at 0.14B parameters says little about efficiency at the scales where recurrent models matter, and should be measured at larger scales. In Figure 4, K also scales the state size, a confound IMO.
Questions for authors
How was 2e-5 chosen?
4 comments
We've revised the manuscript as per your concerns.
Dear Reviewer gHLx,
We sincerely appreciate your reviewing and assessing for our work.
We've revised the manuscript as per your concerns and suggestions. Specially the appendix now includes the full experimental results you mentioned earlier.
Thank you again.
The authors.
Response to the rebuttal
Apologies for the late reply. Thank you for the detailed response and for the considerable amount of additional work. I appreciate the effort.
That said, the paper has changed a lot: a new title and framing, retrained models with a different recipe and three times the tokens, and a large set of new evaluations. As far as I understand, this is closer to a new submission than to a revision, and I find it difficult to assess it properly within the discussion phase.
Moreover, the responses to me and to the other reviewers do not yet give me confidence in the empirical comparison in particular, since the optimisation does not seem to be carefully tuned:
- You write that 2e-5 followed RWKV's training recipe. If I read it correctly, the RWKV-7 paper uses 6e-4 at 0.1B (Peng et al., 2025, Table 17), so I don't see where 2e-5 comes from.
- The revised runs use a peak learning rate of 2e-4 for all architectures. As far as I can see, there is no sweep, and your response says rates up to 1e-3 now work. Why 2e-4, and was it tuned per architecture?
With a single untuned learning rate and a single run per model, I can't tell whether the ranking reflects the architecture or the recipe. I will therefore keep my score. The revision is clearly a step forward, and I'd encourage you to resubmit once the comparison is tuned and the new material has settled into the paper.
Thanks for your replies.
Reviewer gHLx,
We appreciate your replies.
That said, the paper has changed a lot: a new title and framing, retrained models with a different recipe and three times the tokens, and a large set of new evaluations. As far as I understand, this is closer to a new submission than to a revision, and I find it difficult to assess it properly within the discussion phase.
Thank you for the concern and the assessment. The revision arises from the suggestions of reviewers and the concerns about the evaluation results previously missing.
Moreover, the responses to me and to the other reviewers do not yet give me confidence in the empirical comparison in particular, since the optimisation does not seem to be carefully tuned:
We understand and appreciate your concerns. For limited resources, we are not able to search all optimisation combinations. We can only follow the recipes such as chinchilla rule.
You write that 2e-5 followed RWKV's training recipe. If I read it correctly, the RWKV-7 paper uses 6e-4 at 0.1B (Peng et al., 2025, Table 17), so I don't see where 2e-5 comes from.
We appreciate your concerns. Please see the Start Pre-training section in https://wiki.rwkv.com/advance/pretrain.html#initialize-the-rwkv-model
The revised runs use a peak learning rate of 2e-4 for all architectures. As far as I can see, there is no sweep, and your response says rates up to 1e-3 now work. Why 2e-4, and was it tuned per architecture?
This is for stability under mixed precision training. The learning rate is determined by how many tokens in batch and model size. For a base configuration with model width and training context 512 and batch size 20. We use as base learning rate.
With a single untuned learning rate and a single run per model, I can't tell whether the ranking reflects the architecture or the recipe. I will therefore keep my score. The revision is clearly a step forward, and I'd encourage you to resubmit once the comparison is tuned and the new material has settled into the paper.
We appreciate your concerns.
But we respectfully disagree with you. We don't intent to show engineering perspective in this work, which is off the scope of MIRA. Also training a LLM is very difficult and expensive for all recipes. We're not able exhaust all recipes like doing experiments on toy datasets. We've showed that our results outperform baselines.
We thank the reviewer for the time.
The authors.
Responses to gHLx21 (part 1)
Dear Reviewer gHLx21,
We very much appreciate that you have read and assessed our work.
We will organise our responses point by point to address your concerns.
Before the rebuttal, we want to clarify our core contribution as we articulated in the manuscript at L138.
Core Contribution (see L138)
Regarding
We understand your concerns.
However, this conference paper has a 9-page quota for the first submission. We will include the literature you added into the related work of our final version.
We want to clarify literature doesn't analyze from the perspective of zero-pole; instead, some analyses are conducted fro the spectral perspective.
However, the spectra of transition matrices and the poles of transfer functions are not equal. Moreover, complex spectra do not necessarily imply complex poles. This is because transfer function and transition matrix have different algebraic structure. Generally speaking, complex poles are a subset of the complex eigenvalues.
We didn't claim our contribution is the finding of poles. We used the zero-pole analysis from discrete signal processing domain for motivating this work instead of claiming it as contributions. In contrast, literature such in SSM variants has to discretise from continuous ODE.
Regarding
We understand and appreciate your concerns. Perhaps this term confuses readers. We will change the term to "time-varying" or "adaptive IIR systems".
Suppose input sequence x1:T and output sequence y1:T and their transfer function. Ht(x1:T,y1:T)=b0+∑j=1nbjz−1∑i=1maiz−1 .
The transfer function Ht can be time-invariant and time-varying, depending on analysis context. In fact, "adaptive IIR system" is a standalone research line in applied math/signal processing. For example, you can refer to [1]
I think in the final version, we will include a short material in appendix to introduce some foundations for transfer-function based analysis.
Regarding
We agree and thank the reviewer. We will correct Tab1 and A.4.6. RetNet can have complex factors but RetNet still has pole bias because its pole modulus is positioned between [0.968,1) by default.
We previously included RetNet as a benchmark baseline, but its learning and performance are rather low for first 50k steps. We dropped it to save computation.
Regarding
We appreciate your concerns.
Whenever computational resources are available for us. We will include more results.
Please refer to the responses to reviewer 6P8j22. We have posted: Yes, we had more results, including
Training larger LLMs among with multiple baselines is time-consuming and expensive. We've used 30+ days to complete small-scale experiments.
We have revised our manuscript and included these experiments.