acceptodds
Under review as a conference paper at ICLR 2027

Mira: Mitigating Pole Geometry Bias in Recurrent Language Models

Abstract

We show that the Transformer's recurrent successors — such as selective state-space models (SSMs), Receptance Weighted Key Value (RWKV), and Retentive Network (RetNet) — can be characterized as infinite-impulse-response (IIR) systems. The zero–pole geometries of these IIR systems characterize the temporal modes they can represent. However, to trade parallelism, the algebraic structures of the parameterizations of these recurrent systems often give rise to poles constrained to the real axis, thereby inducing an architectural bias in their recurrent dynamics. To unify these recurrent architectures and eliminate their pole geometry bias, inspired by the lattice–ladder realization of general IIR systems, we present Mira (Modal Adaptive Recurrent Lattice), a unified family of linear-time recurrent architectures for building large language models (LLMs) with a highly parallel algebraic structure. Each Mira block realizes a higher-order IIR system that represents temporal modes from token streams through a lattice–ladder structure, followed by a feed-forward network (FFN). This realization enables the model to represent the temporal modes associated with both real and complex-conjugate poles. We pretrain Mira-S, a 12-layer small-scale LLM with 0.14B parameters composed entirely of Mira blocks. Throughout the pretraining, Mira-S achieves higher next-token prediction accuracy and lower perplexity than strong recurrent baselines, exhibits stronger context-window generalization beyond the training length, and attains higher throughput with a dedicated CUDA kernel.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, LLM Architecture, Language Modeling, Sequence Modeling, Recurrent Neural Networks, Recurrent Dynamics, Long-Context Modeling, Efficient Language Models, Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gHLxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I understand, the authors argue that current linear recurrent language models can only decay their state along the real axis and cannot make it oscillate, based on an analysis that freezes the transition at a single time step. They propose Mira, a recurrent layer built from a stack of learned 2D rotations, so that its state can rotate as well as decay while staying stable. A 0.14B model trained on 3B tokens is reported to beat Mamba, RWKV and attention baselines.

Critical strengths

The lattice-ladder parameterisation with a Schur stability guarantee and a chunkwise scan is a reasonable piece of engineering.

Critical weaknesses

The premise doesn't seem new to me nor accurate as stated: complex-valued recurrences are in S4D-style SSMs and in the LRU (Orvieto et al., 2023, "Resurrecting Recurrent Neural Networks for Long Sequences", https://arxiv.org/abs/2303.06349), and negative eigenvalues are studied by Grazzi et al., 2025 ("Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues", https://arxiv.org/abs/2411.12537), yet neither is discussed or compared against. Moreover, the frozen-system analysis (A.4) says little about time-varying models, since products of symmetric transitions can rotate (Siems et al., 2025, "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products", https://arxiv.org/abs/2502.10297, not mine), and A.4.6 itself concedes that RetNet has complex factors, contradicting Table 1.

The experiments can't support "outperforms strong recurrent baselines": there are no downstream evaluations, and all models are trained with an untuned peak learning rate of 2e-5 (line 418). That is about 30x below what the baselines' own papers use at this scale (6e-4 for Mamba at 125M, Gu and Dao, 2023, App. E.2, https://arxiv.org/abs/2312.00752; 6e-4 for RWKV-7 at 0.1B, Peng et al., 2025, Table 17, https://arxiv.org/abs/2503.14456), so the ranking may reflect sensitivity to the learning rate rather than architecture. Similarly, throughput measured at 0.14B parameters says little about efficiency at the scales where recurrent models matter, and should be measured at larger scales. In Figure 4, K also scales the state size, a confound IMO.

Questions for authors

How was 2e-5 chosen?

4 comments
AuthorsOpenReview ↗

Responses to gHLx21 (part 1)

Dear Reviewer gHLx21,

We very much appreciate that you have read and assessed our work.

We will organise our responses point by point to address your concerns.

Before the rebuttal, we want to clarify our core contribution as we articulated in the manuscript at L138.

Core Contribution (see L138)

A Unified Architectural Family. We introduce MIRA (Modal Adaptive Recurrent Lattice), a unified, positional-encoding-free, linear-time architectural family that realizes higher-order IIR dynamics through an adaptive lattice–ladder topology, enabling the representation of temporal modes associated with both real and complex-conjugate poles.

Regarding

The premise doesn't seem new to me nor accurate as stated: complex-valued recurrences are in S4D-style SSMs and in the LRU [...] and negative eigenvalues are studied by Grazzi et al., 2025 [...], yet neither is discussed or compared against.

We understand your concerns.

However, this conference paper has a 9-page quota for the first submission. We will include the literature you added into the related work of our final version.

We want to clarify literature doesn't analyze from the perspective of zero-pole; instead, some analyses are conducted fro the spectral perspective.

However, the spectra of transition matrices and the poles of transfer functions are not equal. Moreover, complex spectra do not necessarily imply complex poles. This is because transfer function and transition matrix have different algebraic structure. Generally speaking, complex poles are a subset of the complex eigenvalues.

We didn't claim our contribution is the finding of poles. We used the zero-pole analysis from discrete signal processing domain for motivating this work instead of claiming it as contributions. In contrast, literature such in SSM variants has to discretise from continuous ODE.

Regarding

Moreover, the frozen-system analysis (A.4) says little about time-varying models, since products of symmetric transitions can rotate (Siems et al., 2025, [...])

We understand and appreciate your concerns. Perhaps this term confuses readers. We will change the term to "time-varying" or "adaptive IIR systems".

Suppose input sequence x1:Tx_{1:T} and output sequence y1:Ty_{1:T} and their transfer function. Ht(x1:T,y1:T)=∑i=1maiz−1b0+∑j=1nbjz−1H_t(x_{1:T},y_{1:T}) = \frac{\sum_{i=1}^m a_i z^{-1}} {b_0 + \sum_{j=1}^n b_j z^{-1}} .

The transfer function HtH_t can be time-invariant and time-varying, depending on analysis context. In fact, "adaptive IIR system" is a standalone research line in applied math/signal processing. For example, you can refer to [1]

[1] E. W. Kamen, et al. A Transfer-Function Approach to Linear Time-Varying Discrete-Time Systems. SIAM Journal on Control and Optimization, SIAM Journal on Control and Optimization, 1985

I think in the final version, we will include a short material in appendix to introduce some foundations for transfer-function based analysis.

Regarding

and A.4.6 itself concedes that RetNet has complex factors, contradicting Table 1.

We agree and thank the reviewer. We will correct Tab1 and A.4.6. RetNet can have complex factors but RetNet still has pole bias because its pole modulus is positioned between [0.968,1)[0.968,1) by default.

We previously included RetNet as a benchmark baseline, but its learning and performance are rather low for first 50k steps. We dropped it to save computation.

Regarding

The experiments can't support "outperforms strong recurrent baselines": there are no downstream evaluations

We appreciate your concerns.

Whenever computational resources are available for us. We will include more results.

Please refer to the responses to reviewer 6P8j22. We have posted: Yes, we had more results, including

  • Dedicated CUDA Kernel Prefill Performance (newest)
  • Benchmark Pretraining Performance on FineWeb-Edu, Wikitext103, and HellaSwag
  • Zero-Shot & Few-Shot Prompting on MMLU
  • ARC Reasoning
  • NIH/Needle-in-a-Haystack retrieval on FineWeb-Edu, Wikitext103, and HellaSwag
  • RULER-style multi-key retrieval on FineWeb-Edu, Wikitext103, and HellaSwag
  • Multi-query associative recall (MQAR) on FineWeb-Edu, Wikitext103, and HellaSwag
  • Copying on FineWeb-Edu, Wikitext103, and HellaSwag

Training larger LLMs among with multiple baselines is time-consuming and expensive. We've used 30+ days to complete small-scale experiments.

We have revised our manuscript and included these experiments.

AuthorsOpenReview ↗

We've revised the manuscript as per your concerns.

Dear Reviewer gHLx,

We sincerely appreciate your reviewing and assessing for our work.

We've revised the manuscript as per your concerns and suggestions. Specially the appendix now includes the full experimental results you mentioned earlier.

Thank you again.

The authors.

Reviewer_gHLxOpenReview ↗

Response to the rebuttal

Apologies for the late reply. Thank you for the detailed response and for the considerable amount of additional work. I appreciate the effort.

That said, the paper has changed a lot: a new title and framing, retrained models with a different recipe and three times the tokens, and a large set of new evaluations. As far as I understand, this is closer to a new submission than to a revision, and I find it difficult to assess it properly within the discussion phase.

Moreover, the responses to me and to the other reviewers do not yet give me confidence in the empirical comparison in particular, since the optimisation does not seem to be carefully tuned:

  • You write that 2e-5 followed RWKV's training recipe. If I read it correctly, the RWKV-7 paper uses 6e-4 at 0.1B (Peng et al., 2025, Table 17), so I don't see where 2e-5 comes from.
  • The revised runs use a peak learning rate of 2e-4 for all architectures. As far as I can see, there is no sweep, and your response says rates up to 1e-3 now work. Why 2e-4, and was it tuned per architecture?

With a single untuned learning rate and a single run per model, I can't tell whether the ranking reflects the architecture or the recipe. I will therefore keep my score. The revision is clearly a step forward, and I'd encourage you to resubmit once the comparison is tuned and the new material has settled into the paper.

AuthorsOpenReview ↗

Thanks for your replies.

Reviewer gHLx,

We appreciate your replies.

That said, the paper has changed a lot: a new title and framing, retrained models with a different recipe and three times the tokens, and a large set of new evaluations. As far as I understand, this is closer to a new submission than to a revision, and I find it difficult to assess it properly within the discussion phase.

Thank you for the concern and the assessment. The revision arises from the suggestions of reviewers and the concerns about the evaluation results previously missing.

Moreover, the responses to me and to the other reviewers do not yet give me confidence in the empirical comparison in particular, since the optimisation does not seem to be carefully tuned:

We understand and appreciate your concerns. For limited resources, we are not able to search all optimisation combinations. We can only follow the recipes such as chinchilla rule.

You write that 2e-5 followed RWKV's training recipe. If I read it correctly, the RWKV-7 paper uses 6e-4 at 0.1B (Peng et al., 2025, Table 17), so I don't see where 2e-5 comes from.

We appreciate your concerns. Please see the Start Pre-training section in https://wiki.rwkv.com/advance/pretrain.html#initialize-the-rwkv-model

The revised runs use a peak learning rate of 2e-4 for all architectures. As far as I can see, there is no sweep, and your response says rates up to 1e-3 now work. Why 2e-4, and was it tuned per architecture?

This is for stability under mixed precision training. The learning rate is determined by how many tokens in batch and model size. For a base configuration with model width 768768 and training context 512 and batch size 20. We use 2×10−42\times 10^{-4} as base learning rate.

With a single untuned learning rate and a single run per model, I can't tell whether the ranking reflects the architecture or the recipe. I will therefore keep my score. The revision is clearly a step forward, and I'd encourage you to resubmit once the comparison is tuned and the new material has settled into the paper.

We appreciate your concerns.

But we respectfully disagree with you. We don't intent to show engineering perspective in this work, which is off the scope of MIRA. Also training a LLM is very difficult and expensive for all recipes. We're not able exhaust all recipes like doing experiments on toy datasets. We've showed that our results outperform baselines.

We thank the reviewer for the time.

The authors.

Reviewer_6P8jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper finds that existing linear recurrent models have a constraint that poles are restricted to the real axis. To mitigate this constraint, they propose an architecture, MIRA, that is motivated by a lattice-ladder structure. The authors showed that the proposed MIRA model can better learn temporal signals that can be represented by non-real poles, compared to reference models. The model is demonstrated by comparing LM pretraining learning curve, cuda kernel efficiency and length generalization capability to reference models.

Critical strengths

The authors proposed the new method to analyze learning capability of temporal models from the perspective of signal processing system. Based on their observations, the authors desinged a model that removes observed "real pole-bias" and easily parallelizable on modern hardware such as GPUs.

Critical weaknesses

The experimental results are weak to support the effectiveness of the models. The only result related to model performance in context-length generalization shown in Figure 7, which doesn't provide exact values.

In the introduction, the analysis is based on the transfer function. However, the transfer function is defined on a linear time-invariant system. The connection between the transfer-function analysis and the nonlinear time-varying recurrent dynamics is not sufficiently justified.

Questions for authors
  1. In the introduction, "The zero–pole geometry of H(z;xt) governs how temporal modes are retrieved from input sequences: zeros attenuate or cancel specific modes, whereas poles determine persistent or resonant temporal modes". In the reference (Oppenheim & Schafer,2010) the transfer function is defined for linear time-invariant systems only. Does the statement "pole geometry constrained on the real axis therefore induces an architectural bias in recurrent dynamics and delimits the modes that they can represent." hold for a time-variant system also? Does this require any assumptions? Do you have any analysis or reference for this?

  2. Do you have the distribution of poles of pretrained MIRA models as plotted in Figure 1?

  3. Do you have any results on other benchmarks (e.g. zero-shot common-sense reasoning)? Perplexity alone does not fully demonstrate the effectiveness of the proposed model.

10 comments
AuthorsOpenReview ↗

Responses to 6P8j22 (part 1).

Dear Reviewer 6P8j22,

We very much appreciate that you have read and assessed our work. We will organise our responses point by point to address your concerns.

Regarding

In the introduction, "The zero–pole geometry of H(z;xt) governs how temporal modes are retrieved from input sequences: zeros attenuate or cancel specific modes, whereas poles determine persistent or resonant temporal modes". In the reference (Oppenheim & Schafer,2010) the transfer function is defined for linear time-invariant systems only. Does the statement "pole geometry constrained on the real axis therefore induces an architectural bias in recurrent dynamics and delimits the modes that they can represent." hold for a time-variant system also? Does this require any assumptions? Do you have any analysis or reference for this?

We appreciate your concerns.

For TI system, the transfer function is a constant function which doesn't vary wrt input/state/time.

For TV + causal system, the transfer function is no longer a constant function since the coefficients aia_i and bjb_j are input-dependent. There exists a research line named Adapative IIR systems in signal processing and applied mathematics. For example, please refer to [1]

MIRA theoretically falls in the scope of Adapative IIR systems.


[1] E. W. Kamen, et al. A Transfer-Function Approach to Linear Time-Varying Discrete-Time Systems. SIAM Journal on Control and Optimization, SIAM Journal on Control and Optimization, 1985

This holds true for causal IIR systems --- this is the only assumption here (MIRA realises a causal IIR system).

Suppose we have input and output sequences, up to time tt, admitting: x1:t=x1,x2,⋯ ,xtx_{1:t}=x_1,x_2,\cdots, x_t and y1:t=y1,y2,⋯ ,yty_{1:t}=y_1,y_2,\cdots, y_t

We define the transfer function at time tt as H1:t=Z[y1:t]Z[x1:t]=a0(x1:t,y1:t)+∑i=1mai(x1:t,y1:t)z−i∑j=1nbi(x1:t,y1:t)z−jH_{1:t} = \frac{Z[y_{1:t}]}{Z[x_{1:t}]} = \frac{a_0(x_{1:t},y_{1:t})+\sum_{i=1}^m a_i(x_{1:t},y_{1:t}) z^{-i} }{ \sum_{j=1}^n b_i(x_{1:t},y_{1:t}) z^{-j} } where ZZ is zz-transform operator, and the coefficients aa and bb are fully determined by (x1:t,y1:t)(x_{1:t},y_{1:t}).

Now at time point t+1t+1, the corresponding transfer function H1:t+1H_{1:t+1} is fully determined by (x1:t+1,y1:t+1)(x_{1:t+1},y_{1:t+1}).

Regarding

Do you have the distribution of poles of pretrained MIRA models as plotted in Figure 1?

We appreciate your concerns.

Actually Fig. 1's poles are distributions.

They are plotted as point clouds. MIRA's poles converged to ground-truth poles. RWKV and Mamba can't converge to ground-truth poles.

The poles of RWKV and Mamba v1 are the reddish and yellowish dots on the real axis in Fig.1. RWKV and Mamba v1 attempt to fit the map x1:t→y1:tx_{1:t} \to y_{1:t}, which requires complex poles to represent. Because RWKV and Mamba v1 have no complex poles, the fitting can only move their poles along the real axis in search of a set of real poles that approximates the map x1:t→y1:tx_{1:t} \to y_{1:t}.

The poles in the experiments of Fig.~1 are not arbitrary. Because we aim to provably show that a model with pole bias cannot represent the signals associated with complex poles, we construct the ground-truth poles as follows.

How do we construct the input-output sequences in Fig.1

(Please refer to appendix.)

We firstly constructed a random input sequence x1:tx_{1:t} (1≤t≤n1\leq t \leq n), we then constructed a transformer function sequence H1:tH_{1:t}.

By using the ground-truth transfer function sequence H1:tH_{1:t}, at a time point tt, we are able to construct the ground-truth output sequence y1:ty_{1:t} by computing a zz-transform sequence for output y1:ty_{1:t}:

Z[y1:t]=H1:tZ[x1:t]Z[y_{1:t}] = H_{1:t} Z[x_{1:t}]

We then apply inverse zz-transform for computing the actual y1:ty_{1:t}.

Regarding

Do you have any results on other benchmarks (e.g. zero-shot common-sense reasoning)? Perplexity alone does not fully demonstrate the effectiveness of the proposed model.

We appreciate your concerns.

Yes, we had more results, including

  • Dedicated CUDA Kernel Prefill Performance (newest)
  • Benchmark Pretraining Performance on FineWeb-Edu, Wikitext103, and HellaSwag
  • Zero-Shot & Few-Shot Prompting on MMLU
  • ARC Reasoning
  • NIH/Needle-in-a-Haystack retrieval on FineWeb-Edu, Wikitext103, and HellaSwag
  • RULER-style multi-key retrieval on FineWeb-Edu, Wikitext103, and HellaSwag
  • Multi-query associative recall (MQAR) on FineWeb-Edu, Wikitext103, and HellaSwag
  • Copying on FineWeb-Edu, Wikitext103, and HellaSwag

We post the full results below in the form of tables. At the end, we also post a summarisation regarding the results.

We will include the full results into the final version.

AuthorsOpenReview ↗

**Dedicated CUDA Kernel Prefill Performance** (newest)

Dedicated CUDA Kernel Prefill Performance

We benchmark parameter-matched modules in BF16 on one NVIDIA RTX 6000 Ada Generation GPU. At each context length, we maximize the batch size subject to every module fitting within the 40-48 GiB GPU memory budget. All modules process the same number of tokens per batch at a given context length.

Note. The benchmark baselines are determined by the availability of baseline CUDA kernels. We are not able to test all CUDA kernels since their availability.

Prefill throughput (million tokens/s)

Context (tokens)Batch sizeMIRA (CUDA)MHA (FlashAttention)Mamba (S6)RWKV-6RWKV-7
2,0485749.826.953.988.647.94
4,0962919.645.343.858.457.73
8,1921449.353.653.738.317.60
16,384729.092.263.718.127.38
32,768368.101.293.698.006.39

Prefill execution time (ms per batch)

Context (tokens)Batch sizeMIRA (CUDA)MHA (FlashAttention)Mamba (S6)RWKV-6RWKV-7
2,048574119.74169.21295.58136.12148.08
4,096291123.63223.28309.48141.07154.16
8,192144126.20322.96316.02141.95155.18
16,38472129.81522.57317.86145.20159.79
32,76836145.58915.27319.33147.53184.71
AuthorsOpenReview ↗

**Benchmark Pretraining Performance** on FineWeb-Edu, Wikitext103, and HellaSwag

Benchmark Pretraining Performance

Note. The following results are obtained with a peak learning rate 2e-4 and final learning rate 1e-5 by extending training token quota to 9B tokens on FineWeb-Edu.

FineWeb-Edu

ModelSize (M)State (‰)512 Loss512 PPL512 Acc (%)32K Loss32K PPL32K Acc (%)
Transformer (MHA)142.520.0003.40029.9635.317.3171.51e34.58
RWKV-6144.644.4183.46531.9734.523.75242.5920.18
RWKV-7142.663.6343.48532.6334.263.42330.6534.47
Mamba-1148.732.1073.58235.9333.7050.0925.68e211.31
Mamba-2148.4116.8903.46632.0234.7327.5839.53e111.16
Mira-S144.724.0763.32327.7537.033.23125.3038.07

WikiText-103

ModelSize (M)State (‰)512 Loss512 PPL512 Acc (%)32K Loss32K PPL32K Acc (%)
Transformer (MHA)142.520.0003.46832.0836.017.1361.26e36.25
RWKV-6144.644.4183.52533.9435.103.76443.1221.77
RWKV-7142.663.6343.58736.1234.583.51933.7534.16
Mamba-1148.732.1073.56635.3634.6252.0964.22e221.02
Mamba-2148.4116.8903.50133.1435.4327.1796.36e110.75
Mira-S144.724.0763.43531.0336.733.31627.5438.03

HellaSwag

ModelSize (M)State (‰)512 Loss512 PPL512 Acc (%)32K Loss32K PPL32K Acc (%)
Transformer (MHA)142.520.0003.43431.0038.397.1341.25e31.43
RWKV-6144.644.4183.52333.8835.473.81445.3419.21
RWKV-7142.663.6343.64438.2335.113.54434.5934.94
Mamba-1148.732.1073.68039.6334.0546.1621.12e201.02
Mamba-2148.4116.8903.43631.0738.6935.3862.33e150.78
Mira-S144.724.0763.48132.5037.193.35428.6238.48

Validation loss, perplexity (PPL), and next-token accuracy (Acc) are reported at context lengths of 512 and 32,768 tokens. The training context window is 512 tokens. We evaluate FineWeb-Edu on the held-out last 5% of the corpus and WikiText-103 and HellaSwag on their test splits. State (‰) is the ratio of the total recurrent state size across all layers to the model parameter count, expressed in parts per thousand.

Reviewer_jf1hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper interprets linear recurrent models (Mamba, RWKV, RetNet, HGRN2, DeltaNet/GDN) through the lens of digital signal processing: it interprets them as infinite-impulse-response (IIR) systems, highlighting that most modern SSMs confine the poles of the corresponding transfer function to the real axis. They argue this introduces a bias preventing these models from capturing oscillatory temporal modes (associated instead with complex poles), and propose a way to mitigate it. To do so, they introduce MIRA, which replaces the classical first-order recurrence of an SSM with an order-K one, realised through a lattice-ladder filter. At each token, the input is passed through a cascade of K stages, each a scaled 2x2 rotation with input-dependent parameters, and a readout operation combines the states of all stages. By construction, the poles can lie anywhere inside the unit disk, so the recurrence is stable while admitting complex poles. The recurrence is parallelised with a chunked scan implemented in a dedicated CUDA kernel. A 0.14B MIRA model pretrained on 3B FineWeb-Edu tokens is reported to reach lower perplexity than Mamba, RWKV, and a Transformer of similar size, to generalise to contexts of up to 32K tokens, and to be faster than the baselines in forward-pass benchmarks.

Critical strengths
  • The authors draw a neat connection between modern linear RNNs and IIR lattice-filter theory
  • This is leveraged effectively to build a principled framework for general, higher-order linear recurrences, which is stable by construction
  • The provided code, including a fused CUDA kernel, makes adoption straightforward
Critical weaknesses
  • The paper is not properly positioned with respect to similar existing work, impacting justification and novelty
  • The setup for runtime experiments is dubious, and the considered kernels seem suboptimal
  • Key baselines are missing (tied to W1)
Questions for authors

W1: Justification and novelty The main justification behind this work seems to be given by the interest in generalising recursions to include complex poles (L16, L104, the whole discussion around Fig1, and so on). However, linear SSMs that include them already exist: complex-diagonal SSMs (S4D, LRU), oscillatory SSMs (LinOSS), 2x2 block-diagonal SSM with selective rotation matrices (Mamba3), products of Householder (DeltaProduct), and even RetNet, as you yourselves note in App. A.4.6 (L1144–1145), all can produce general complex poles. Even Mamba-1’s implementation allows for complex eigenvalues in their transition matrices, but they were restricted to real on purpose as they were found not to help on discrete modalities such as text (see Sec3.6 in the original Mamba paper) [disclaimer: I’m not an author of any of these works]. This should be made clear in the manuscript: complex poles are not particularly new, and I don’t even believe are the main selling point of the paper. Also the DSP / control-theoretic view of linear recurrences is not really novel (it lies at the core of development of modern SSMs), although I appreciate how you used the connection to their lattice-ladder realisation to build a principled way to generate higher-order recurrences. I believe this is the main contribution of the paper, although its novelty is again tainted by more recent work which sets out to do something similar (Dubinin et al, “Improved state mixing in higher-order and block diagonal linear recurrent networks” [again, not an author]), although with your work you set this up in a much more general and principled way, so it’s definitely a relevant improvement. Regardless, I would encourage you to reframe the paper around these considerations, and to adjust its novelty aspects and its justifications accordingly. Moreover, it would be useful to people in the field if you presented the method in standard linear-RNN notation: the stage / time indexing of Eq. (2) makes it quite hard to properly identify the actual recurrence.

W2: Runtime measurements I was quite excited by the fact that you provided your own CUDA implementation of the method, and by the throughput results you were reporting in Fig5, but I was left a bit confused after looking deeper into the code. My understanding is that the MIRA code works by splitting the sequence length in chunks, but it works sequentially both within a chunk and across chunks: by carefully sizing chunks, you get an overall complexity of T\sqrt{T}. Mamba, by contrast, uses parallel scans within a chunk, and is sequential across chunks (for a complexity of log⁡(L)+T/L\log(L) + T/L). But your runtime experiments in Fig5 show a complexity pretty much linear in TT (btw, a log-log plot with a curve fit would have made this much, much clearer) for all methods considered (bar MHA, which is quadratic, but that’s expected) The code you provided reports your own implementation of Mamba and RWKV, which is purely sequential (matching the complexity we observe in Fig5), and foregoes any within-chunk parallelism. Is this the one you used for your comparisons? Because if so, then the comparison is very much unfair, and should rely on the ones provided in FlashLinearAttention. Regardless, chunk-wise implementations of modern SSMs rely on tensor cores for operations within chunks: if you want the comparison to be meaningful, you should include runtime/throughput results for Mamba2/3 or, even better, GLA/GDN/KDA.

W3: Baselines I don’t like being the one reviewer who just asks for more baselines, but in this case it is relevant, as you’re not really comparing your method against relevant competitors. Consider your Fig1: of course if you ask a model which can only track exponentially decaying functions (as that’s the impulse response of Mamba and other diagonally-gated SSMs) to track sinusoidal ones, it would fail miserably (although I understand you’re showing this for illustrative purposes). The point is, alternatives which track sinusoidal functions (ie, admit complex spectra) do exist, and those are the ones you should be comparing against to really highlight the validity of MIRA. LinOSS would be the bare minimum (block 2x2 transitions, directly inspired by oscillatory dynamics, 2nd order, featuring complex poles), Mamba3 does pretty much the same in a selective fashion, but really Dubinin et al.'s H-LRU is the direct competitor, as they also go for high-order (although your Fig4 already shows that K>2K>2 is not really worth it).

Minor:

  • L15 (and L99): “to trade parallelism, […] poles constrained to the real axis” one can achieve parallelism even with complex poles, so that’s not really the true motivation behind it
  • L97: “Table 1 summarizes the transfer-function and zero-pole characterizations of…“ this is only valid for frozen coefficients: in practice, for most modern RNNs these coefficients vary with time, which is why at the end of the day some of them can still represent pretty complex modes
  • Sec3: You’re using the principle of superposition of effects in linear systems to achieve parallelism via chunk-wise splitting. But the associativity of the operator allows you to employ prefix sum to squeeze extra temporal parallelism also within chunks (a-la Mamba1). Have you tried playing around with that? For certain configurations (very long sequences, small K), it might boost runtime
  • L450: “MIRA consistently achieves lower execution time” MHA wins at T≤4kT\leq4k
  • L465: “Pretraining resumes from an earlier checkpoint … because the baseline losses exhibit spikes”: this doesn’t sound good? What does it mean? Are the results in Fig6 comparable then?
8 comments
AuthorsOpenReview ↗

Responses (part 1)

Dear Reviewer jf1h,

We very much appreciate that you have read and assessed our work. We will organise our responses point by point to address your concerns.

Before the rebuttal, we want to clarify that: COMPLEX eigenvalues of state transition matrices DO NOT necessarily imply COMPLEX POLES.

Regarding

W1: Justification and novelty

The main justification behind this work seems to be given by the interest in generalising recursions to include complex poles (L16, L104, the whole discussion around Fig1, and so on).

We appreciate your concerns. The introduction (L16,L104) part is to introduce the motivation by characteristing recurrent systems from the lens of IIR systems inspired by DSP. To the best of our knowledge, this characterisation for modern recurrent language models doesn't overlap with literature.

Regarding

However, linear SSMs that include them already exist: complex-diagonal SSMs (S4D, LRU), oscillatory SSMs (LinOSS), 2x2 block-diagonal SSM with selective rotation matrices (Mamba3), products of Householder (DeltaProduct), and even RetNet, as you yourselves note in App. A.4.6 (L1144–1145), all can produce general complex poles. Even Mamba-1’s implementation allows for complex eigenvalues in their transition matrices, but they were restricted to real on purpose as they were found not to help on discrete modalities such as text (see Sec3.6 in the original Mamba paper) [disclaimer: I’m not an author of any of these works].

We appreciate time you had to read, understand, and assess our work. We also appreciate the recent literature you mentioned. This is why, we argue that our major contribution is:

(L136-L138) A Unified Architectural Family. We introduce MIRA (Modal Adaptive Recurrent Lattice), a unified, positional-encoding-free, linear-time architectural family that realizes higher-order IIR dynamics through an adaptive lattice–ladder topology, enabling the representation of temporal modes associated with both real and complex-conjugate poles.

We aimed to introduce a unified architectural family which has no pole bias. We do not claim that MIRA is the only LLM architecture that can overcome pole bias.

We wish to introduce a new architectural family that can unify the recurrent systems inspired by IIR systems. We believe our IIR-inspired architecture is a complementary architecture for helping researchers to understand and understand modern recurrent language models.

Regarding:

This should be made clear in the manuscript: complex poles are not particularly new, and I don’t even believe are the main selling point of the paper. Also the DSP / control-theoretic view of linear recurrences is not really novel (it lies at the core of development of modern SSMs), although I appreciate how you used the connection to their lattice-ladder realisation to build a principled way to generate higher-order recurrences.

We appreciate your concerns. Let me simply put.

First, DSP systems are discrete systems; in contrast Mamba and SSM variants (eg S4/S4D) start from continuous equations and then discretise.

Second, DSP or SSM are two separated research lines or views. Mamba is not the first research using SSM equations; likewise, MIRA is not the paper using IIR formulation.

We didn't claim that MIRA is the first LLM inpsired by DSP. What we claim:

MIRA is a unified architecture using the lattice-ladder realization, and the unique algebraic structure that the lattice dynamics endowed to MIRA, which makes it possible to paralelise in modern hardware.

Regarding

I believe this is the main contribution of the paper, although its novelty is again tainted by more recent work which sets out to do something similar (Dubinin et al, “Improved state mixing in higher-order and block diagonal linear recurrent networks” [again, not an author]), although with your work you set this up in a much more general and principled way, so it’s definitely a relevant improvement.

We appreciate your suggestions.

To the best of our knowledge, the discussion from the zero-pole characterisation doesn't overlap with literature. Existing literature often discuss from the perspective of the operator spectra. The spectral theory and zero-pole characterisation in DSP may overlap somehow, however this should not affect the novelty that MIRA is the first paper, as of the submission date, for discussing from the lens of zero--pole characterisation in discrete signal processing.

Regarding

Regardless, I would encourage you to reframe the paper around these considerations, and to adjust its novelty aspects and its justifications accordingly.

We agree with your concerns. In the final version, we will adopt your suggestion to articulate and make clear statements.

AuthorsOpenReview ↗

Responses (part 2)

Regarding

Moreover, it would be useful to people in the field if you presented the method in standard linear-RNN notation:

We appreciate your concerns. However, the notations we adopted from the research in discrete signal processing.

Regarding

the stage / time indexing of Eq. (2) makes it quite hard to properly identify the actual recurrence.

We appreciate your concerns.

MIRA is not a usual linear-RRN systems. So it's impossible to use the conventions. For example, the lattice scanning directions are opposite for U and V signals. U is from K to 1; while V is from 1 to K. The dynamics are one type of lattice dynamics.

Traditional SSMs may use x represent input, h to represent U and y to represent V (3 signals).

However, MIRA has x, U,V, and y (4 signals).

AuthorsOpenReview ↗

Responses (part 3)

Summarisation of the novelty and positioning.

  1. Analysis from the lens of zero-pole geometry in discrete signal processing systems. This differs from the existing literature for starting from state-space equations and spectral analysis. This analysis is unique, to the best of our knowledge.

  2. A unified and principled architecture for LLMs. This is through an realisation of lattice-ladder structure IIR system. To the best of our knowledge, this architectural family is unique.

Regarding

W2: Runtime measurements I was quite excited by the fact that you provided your own CUDA implementation of the method, and by the throughput results you were reporting in Fig5, but I was left a bit confused after looking deeper into the code. My understanding is that the MIRA code works by splitting the sequence length in chunks, but it works sequentially both within a chunk and across chunks: by carefully sizing chunks, you get an overall complexity of .

We agree with you. The O(T)O(\sqrt{T}) is the critical path. This might be only implemented through a tree-scan for merging the states across chunks. In the CUDA version I submitted along with the manuscript, you are correct. The scan for chunks is sequential. But the performance still good as we measured.

After the submission, we continued to optimise the CUDA kernel with a lot of efforts. The current kernel we implemented is through a tree-scan. This tree-scan will attain a shorter execution time and higher throughput (prefill).

We also updated the measuring protocol. The reported results are not maximised since we adopted fixed batches. The newest protocol we adopt max batch by our RTX6000 GPU (~40-48GB vram).

This parallelism is endowed by the unique algebraic structure from our design in the lattice dynamics.

We believe there are more rooms to further improve the parallelism. As of the date of rebuttal submission, we're still working on our CUDA kernel. For example, in your review, you suggested that the in-chunk parallelism can boost run time; planning register use; memory miss/hit, etc. We believe the algebraic structure from lattice-ladder structure may give the potential to further improve the performance. We want to thank you again.

The newest results are:

Prefill throughput (million tokens/s)

Context (tokens)Batch sizeMIRA (CUDA)MHA (FlashAttention)Mamba (S6)RWKV-6RWKV-7
2,0485749.826.953.988.647.94
4,0962919.645.343.858.457.73
8,1921449.353.653.738.317.60
16,384729.092.263.718.127.38
32,768368.101.293.698.006.39

Prefill execution time (ms per batch)

Context (tokens)Batch sizeMIRA (CUDA)MHA (FlashAttention)Mamba (S6)RWKV-6RWKV-7
2,048574119.74169.21295.58136.12148.08
4,096291123.63223.28309.48141.07154.16
8,192144126.20322.96316.02141.95155.18
16,38472129.81522.57317.86145.20159.79
32,76836145.58915.27319.33147.53184.71

Comments on the paper

7 comments
AuthorsOpenReview ↗

Responses to gHLx21 (part 2)

Regarding

and all models are trained with an untuned peak learning rate of 2e-5 (line 418). [...] so the ranking may reflect sensitivity to the learning rate rather than architecture. How was 2e-5 chosen?

We appreciate and agree your concerns regarding peak learning rate. The smaller peak lr was chosen because we wish to use RWKV's training recipe (they have very short warmup steps).

We currently extended warmup steps significantly (>= 5k to 10k steps but still less than the 10% rule). We now can use larger peak learning rate from 2e-4 to 1e-3.

We have revised manuscripts and updated results. The benchmarking results are much better than training with 2e-5.

Regarding

Similarly, throughput measured at 0.14B parameters says little about efficiency at the scales where recurrent models matter, and should be measured at larger scales.

We appreciate and understand your concerns.

Training MIRA is faster but training some baselines (such as Mamba) is slower because of their CUDA kernel. Regarding benchmarking, larger scales may need > 30 days to train on our cluster.

For further work, I'm coordinating GPU cluster with collaborators to train a larger MIRA between 3B-8B on larger dataset. We can compare MIRA with larger LLMs.

Regarding

In Figure 4, K also scales the state size, a confound IMO.

For ablation K, the MIRA module parameter size is matched for K=1,2,4,....

We actually also tested if we do not match MIRA module size. The results remain identical: complex poles can improve performance.

We also tested on The Stack dataset (programming code data), complex poles can significantly improve performance. This may imply MIRA architecture might be good for coding LLMs.

We have revised our manuscript as per your concerns.

We hope these responses and the new results address your concerns.

The authors.

AuthorsOpenReview ↗

Responses to Reviewer jf1h (part 7).

Dear Reviewer jf1h,

We appreciate that you read the responses and appreciate the discussions on the basis of academic disagreements.

Regarding

Positioning (W1)

I want to clarify some points which might be a base point for our responses:

  1. First we agree with you that there is a connection between the discrete form state-space equation (as an example) and transfer function. This is also the base of our paper (see the first sentence in the Abstract).
  • Continuous state-space ODE --> Discrete state-space Equation <--- .... algebraically equivalent.....---> Transfer function.

    This "algebraic equivalence" is the starting point we can analyse recurrent dynamics using DSP theory.

  1. However, algebraic equivalence doesn't automatically endow identical dynamics.

Example. For example (disclaimer, probably not so proper). We can write linear matrix multiplication operation into WxWx, and also we can write convolutional operation into the form W^x\hat{W} x where $\hat{W} $ is a Toeplitz matrix. We can see both linear multiplication and convolutional operation admit the same algebraic form. However, they are different. The key difference here is that W^\hat{W} has its own structure.

Likewise, we can write a IIR filter using the conventions used in DISCRETE state-space systems. But the point is this rewritten doesn't preserve the dynamics inside the transfer function HH.

For example, we can rewrite MIRA's dynamics using discrete state-space conventions. This rewritten can not reflect how the two signals UU and VV propagate on the opposite direction through kk.

I think this is important to judge the novelty and our architecture. MIRA is unique because the realisation of the lattice dynamics.

AuthorsOpenReview ↗

Responses to Reviewer jf1h (part 8).

Regarding

Runtime (W2)

I appreciate the improvements to the kernel. This is a bit unorthodox: generally the chunkwise parallel formulation of modern linear RNNs relies on tensor cores + parallel scan within a chunk, and reduces to a sequential update across chunks, while with the new approach it seems like you’re flipping this view, and going sequential within a chunk and parallel scan across chunks.

We agree with you. In my experiments, the wall from "tensor cores + parallel scan within a chunk" starts take effect for longer context lengths (eg 512k-1M).

For smaller context length (disclaimer: I don't access the hardware to test eg 128K-1M), design elements such as channel-wise parallelism, parallelism of higher-order dynamics, no Conv1D as some SSMs use can enable better parallelism.

Note. In the development of MIRA architecture, we also tested if we add Conv1D to forward signal UtU_t, there is no performance difference. I think this is because the lattice-ladder can efficiently encode local information into the representations.

Regardless, I'm still confused by the results you're posting: execution times are not really comparable with Fig5, since I reckon batch size went up, but throughputs are, and they draw a different picture: at 32k, Fig5 shows MIRA is some ~2.5/2.8x faster than RWKV-6 and RWKV-7; in the new tables this reduces to 1.0/1.3x, whereas the gap to Mamba has grown 1.3->2.2x. What has changed between the two protocols, besides the batch size?

We appreciate your concerns.

This is because the runtime and throughput used different profiling protocols, and we are also optimising our CUDA kernel. Protocols:

  • Runtime. We fix batch size (eg 2), we carefully warmup runs then measure the execution time.
  • Throughput. We do not fix batch size. For each context window, we maximise the GPU vram and measure the maximum throughput.
  • Mixer size. We only measure the performance of the mixer instead of the entire Transformer-style block.
MixerModel widthParametersState ratioKernel
MIRA-S7684,118,8801.19%Dedicated CUDA mixer (mira_dspv2)
MHA1,0084,068,288--PyTorch SDPA with FlashAttention
Mamba-1 (S6)8004,088,0000.63%Upstream selective-scan CUDA with PyTorch causal Conv1d
RWKV-68324,153,3441.28%Upstream wkv6 CUDA
RWKV-77043,964,9281.14%Upstream wkv7 CUDA

In our revised manuscript, we have stated the protocols in appendix.

Also, can you be extra specific about which implementations produced the old and new results in the first place? The code submitted with the paper contains your own re-implementations , which seem to be looping over time with one thread per (batch, channel), while the reply mentions FLA and official code, but that you needed to adapt it: why is this the case? I would expect FLA's chunkwise parallel kernels (including those for GLA, GDN and KDA) and the official Mamba2 kernels to run on your GPU with little or no modification, since it does support tensor cores? Finally, what state sizes did you use in these benchmarks?

The adaption is that my GPU is an old RTX6000 and we have our profiler. We wish to integrate all baselines into one python code. It run as we wish and output the benchmarking plots. We simply download their official code and integrate into code. But they have compiling issues. We need to solve. I've audited the code (Mamba 7 RWKV) again. We used their official implementation.

Regarding GLA/GDN/KDA -- we are working with their CUDA kernel and wish to integrate into our profiler. We will update the results once we had the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.