acceptodds
Under review as a conference paper at ICLR 2027

Non-Abelian State Tracking with Quaternionic State Space Models

14 views

Abstract

Recent developments in state-space model (SSM) design have improved the state tracking and language modeling capabilities of these models through increasingly expressive transition dynamics. In particular, complex-valued diagonal transitions have been shown to be highly effective and computationally efficient. However, their expressivity remains fundamentally limited by the commutativity of diagonal matrices. In this work, we introduce Icosian, a quaternionic SSM with non-commutative state transitions through input-dependent 4D rotations. Icosian exploits the exceptional structure of the 4D rotation group SO(4), parametrized through pairs of unit quaternions. This parametrization enables fast and numerically stable recurrent computation through quaternion arithmetic, without explicitly materializing and composing rotation matrices. We evaluate Icosian on language modeling across scales from 500M to 3B parameters. Across multiple seeds, Icosian consistently outperforms Mamba-3, Gated DeltaNet, and Gated DeltaProduct, while its additional computational cost decreases with scale to only 3% at 3B parameters. We then characterize the state tracking and recall capabilities of Icosian through synthetic experiments. In contrast to complex diagonal SSMs, a single Icosian layer exhibits length generalization on the non-solvable A5 group, while deeper models also track rotations drawn uniformly from continuous groups SO(3) and SO(4). Overall, our results show that Icosian provides expressive non-commutative state dynamics and achieves notable gains in language modeling, while retaining the computational efficiency required for SSMs at scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
state space models, sequence modeling, state tracking, expressivity, linear recurrent neural networks, quaternions, non-commutative dynamics, rotation groups, efficient architectures, language modeling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_tqgPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Use of AI Tools

I used ChatGPT to help understand the paper, check the factual accuracy of my review comments, refine their structure and wording, translate my review from my native language into English, and consider the rating and confidence level.

Summary of Requests

  • “Please translate this into my native language.”
  • “I have written the review comments above. Are any of my points incorrect? Please check their factual accuracy while respecting my own opinions.”
Critical strengths
  • Extending Mamba-3’s complex-valued state transitions to quaternions is a natural design choice, and the proposed non-commutative rotations directly address the motivating expressivity limitation. The reformulation through cumulative quaternion products also enables the use of existing SSD computation, providing a coherent approach to combining increased expressivity with computational efficiency.
  • The paper demonstrates clear improvements over Mamba-3 on non-commutative state-tracking tasks. In particular, a single-layer model achieves 100% accuracy on A5 at sequence lengths beyond those used during training, providing empirical support for the theoretical motivation.
Critical weaknesses
  • The mechanism behind the language-modeling improvements is insufficiently analyzed. On synthetic tasks, the theory and defect analysis of learned rotations explain the effectiveness of non-commutative transitions. For language modeling, however, it remains unclear what the quaternionic blocks learn and how non-commutativity contributes to the perplexity improvements. Without a controlled experiment that restricts non-commutativity, the results do not distinguish improvements due to non-commutativity itself from those due to the additional parameterization.
  • The gains in state tracking are task-dependent. On the finite-state tasks in the Chomsky-hierarchy benchmark, Icosian improves over Mamba-3 but does not consistently outperform other recurrent models. Although Theorem 1 establishes conditions for tracking finite groups, the paper does not explain the performance differences across the tasks in Table 7 or examine which architectural properties, such as the availability of eigenvalue −1 or a discrete transition dictionary, contribute to these differences.
  • The language-modeling evaluation lacks a comparison with Transformers. The baselines are limited to Mamba-3, Gated DeltaNet, and Gated DeltaProduct. Without a Transformer baseline trained with a comparable parameter count and amount of data, Icosian’s position in the trade-off between language-modeling performance and computational efficiency remains insufficiently established.
Questions for authors
  1. What types of linguistic dependencies do the quaternionic non-commutative state transitions help represent? Could an analysis of the learned quaternionic blocks, or a controlled experiment restricting non-commutativity, clarify whether the perplexity improvements are attributable to non-commutativity? Such evidence would help establish the significance of the proposed approach.

  2. How do you explain Icosian’s lower average accuracy compared with GDP, Flash PD-SSM, and the cited LSTM results in Table 7? Does this gap arise from transition expressivity, optimization difficulty, or differences in experimental settings? Please discuss why the substantial improvements on non-commutative group tracking do not translate into consistent advantages on these tasks.

  3. Could you add a comparison with a Transformer trained with a comparable parameter count and amount of data? Comparing language-modeling performance alongside training time, inference speed, and memory usage would clarify Icosian’s position in the performance–efficiency trade-off.

  4. Minor correction: The Table 1 caption states “Five seeds at each scale except 3B,” whereas Section 4.1 states that the 0.5B MIMO configuration also uses three seeds. Please make the descriptions of the seed counts consistent.

4 comments

We thank the Reviewer for their careful reading of the paper, and for highlighting the coherence of our approach and the theoretical and empirical evidence in support of Icosian.

Table numbers below follow the original submission; in the revised PDF, Tables 1, 2, 7 and 8 are Tables 2, 3, 9 and 13.

Following the Reviewer's suggestions, we make the following main changes:

  • Effect of non-commutativity on general tasks (W1): new 3B pretraining on 600B tokens; code gains grow substantially with the number of quaternionic blocks.
  • Table 7 (W2): explanation of the experimental setup, learning-rate sweep of the Chomsky hierarchy tasks; Icosian now solves Parity (99.5 ± 1.1) and Mamba-3's performance on parity is substantially improved.
  • Transformer baseline (W3): 3 seeds of parameter-matched Transformer and Icosian hybrid at 0.5B, together with training-time, inference-speed and memory measurements.

We discuss each point in detail below.


W1 & Q1: Mechanism behind the language-modeling gains

The connection between expressivity and language modeling performance is a very active field of research, with prior literature showing that performance on synthetic tasks correlates with compute-optimal perplexity across architectures [1] and that greater expressivity yields better scaling efficiency [2]. More recently, the DeltaProduct work [3] has demonstrated that increased expressivity induces most substantial gains in the domains of code generation and mathematical reasoning.

In the context of our work, we provide a set of experiments that varies the amount of non-commutativity in Icosian as a control on downstream performance in Table 8. Under a parameter-matched setting, the number of quaternionic blocks ranges from none (Mamba-3) to a quarter of the state, with modest gains as non-commutativity increases. Since the parameter budget is fixed (Icosian adds less than 0.1% parameters over Mamba-3), these gains cannot be attributed to additional parameterization. A mechanistic analysis of which linguistic dependencies the learned quaternionic blocks capture remains open.

We now test this effect for Icosian at large scale, by pretraining Mamba-3, Icosian (1/2,1/321/2, 1/32) and Icosian (1/4,1/41/4, 1/4) at 3B parameters on 600B tokens; the two Icosian variants match in parameters, with (1/4,1/4)(1/4, 1/4) containing more non-commutative structure. Our results show that Icosian substantially improves code generation over Mamba-3, with more modest gains on more general domains, and the code gain grows with the amount of non-commutative structure:

BenchmarkMamba-3Icosian (1/2, 1/32)Icosian (1/4, 1/4)
Code42.846.148.9
HumanEval39.040.244.5
HumanEval+36.039.039.6
MBPP51.657.161.4
MBPP+44.748.150.0
Math & reasoning (GSM8k, MATH, BBH, DROP)39.740.539.2
World knowledge (NQ, TriviaQA)28.428.128.5

Code: pass@1. Others: few-shot exact match (DROP: F1), average scores.

W2 & Q2: Table 7 and task-dependent gains

Unlike the group tracking of Table 2, Table 7 follows prior work cited in the caption exactly for reproducibility, with a single untuned learning rate of 2e-3 and 2-layer models. We now sweep the learning rate of its central models over {10−4,5⋅10−4,10−3}\{10^{-4}, 5\cdot10^{-4}, 10^{-3}\} in addition to 2⋅10−32\cdot10^{-3}:

ModelParityMod. Arith.
Mamba-387.3 ± 14.265.3 ± 11.3
GDP [-1,1], 2 refl.99.7 ± 0.398.2 ± 3.2
Icosian (1/2, 3/32)99.5 ± 1.173.8 ± 6.6

Mean ± sd over 5 seeds at the best learning rate, width 128.

Mamba-3 on Parity improves from 79.8 to 87.3 on the same 5 seeds at width 128 (Table 7 reports 73.2 over ten seeds and the best width); certain seeds reach 100%100\%, but not consistently. The Mamba-3 paper reports 100.0 on Parity as the best over a hyperparameter grid (2 widths, 8 learning rates), with a curriculum raising the maximum training length from 40 to 160, whereas we train on lengths 3 to 40 and report the mean.

The roles of negative eigenvalues [4] and discrete transition dictionaries [5, 6] are analysed in the works that introduce them, as discussed in our related work. We use the more expressive negative-eigenvalue versions of DeltaNet and DeltaProduct, and ablate the number of DeltaProduct factors in additional S5S_5 experiments (see our response to Reviewer fW4X).

W3 & Q3: Transformer baseline

We now add a Llama-style Transformer (RoPE, SwiGLU, RMSNorm, FlashAttention-3) and an Icosian hybrid with attention in 4 of 24 layers (every sixth). Both exactly follow the protocol from Table 1, this being the established FineWeb data-ablation protocol, as detailed in our Section 4.1. Results at 0.5B are reported below:

Model (0.5B)ParamsVal. PPL ↓
Transformer510.8M23.00 ± 0.07
Icosian + attention (4 of 24 layers)504.5M21.83 ± 0.06
Mamba-3514.1M21.76 ± 0.05
Icosian514.6M21.73 ± 0.02

At matched parameters and data, the Transformer trails Mamba-3 and Icosian by over 1.2 perplexity points. By interleaving self-attention layers with the Icosian in a hybrid architecture, we can decrease the gap to the pure Icosian models by over 1 perplexity points.

Training time, inference speed and memory. We measure the parameter-matched models on the same H200 GPU:

TransformerMamba-3Icosian (1/2, 3/32)
Training: step (ms); peak memory (GB)
0.5B210; 40.9252; 50.1304; 53.7
1.1B197; 37.5239; 44.3271; 46.9
3B165; 55.8203; 55.6210; 55.6
Inference at 1.1B: time (s); peak memory (GB)
LL=5126.9; 203.4; 153.7; 16
LL=102419.7; 386.8; 107.3; 10
LL=204864.0; 7413.7; 1014.7; 10
LL=4096OOM27.3; 1029.3; 10
LL=16384OOM105.9; 10118.5; 10

With the highly optimized FlashAttention-3 at sequence length 2048, the Transformer has the cheapest training step. Icosian's overhead decreases with scale, both over the Transformer (1.45× at 0.5B, 1.27× at 3B) and over Mamba-3 (1.20× at 0.5B, 1.04× at 3B).

At inference, Icosian is 1.9–4.4× faster than the Transformer and keeps constant memory, whereas the Transformer's key/value cache grows linearly and exceeds GPU memory from LL=4096 on.

Q4: Seed counts

All language-modeling results now use five seeds per model, as stated in Section 4.1: the MIMO runs always had five seeds and the text has been corrected, and at 3B the two remaining seeds have since finished. The new Transformer and hybrid baselines above use three seeds due to time limitations during the discussion phase; we will also evaluate these models using five seeds.

[1] Poli, Michael, et al. "Mechanistic Design and Scaling of Hybrid Architectures." ICML (2024).

[2] Merrill, William, et al. "Olmo Hybrid: From Theory to Practice and Back." arXiv:2604.03444 (2026).

[3] Siems, Julien, et al. "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products." NeurIPS (2025).

[4] Grazzi, Riccardo, et al. "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues." ICLR (2025).

[5] Terzić, Aleksandar, et al. "Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models." NeurIPS (2025).

[6] Terzić, Aleksandar, et al. "Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models." NeurIPS (2026).

Thank you for the additional experiments and the detailed response. All of my concerns have been addressed. In the revised version, please include these experiments and further deepen the discussion of this method.

We thank the Reviewer for acknowledging our response, and are glad to find that it addressed all their concerns. We remain available for further discussion, and kindly ask the Reviewer to reconsider our score if no more open points remain.

Reviewer_8NSQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Icosian, which adds input-dependent quaternionic SO(4) transitions to Mamba-3. By absorbing accumulated rotations into the input and output transformations, it retains efficient SSD computation while enabling noncommutative state updates. Experiments show strong improvements on group tracking and continuous rotation tasks, alongside small perplexity gains in language modeling. The paper also characterizes finite-group tracking under a specified transition-based definition.

Critical strengths
  1. The paper provides a clean way to combine noncommutative quaternionic transitions with efficient SSD computation. This offers a concrete approach to enriching state dynamics in scalable SSMs.

  2. The state-tracking results are strong. On A5, the reported best-validation accuracy is 100% across five seeds, compared with 7.9% for Mamba-3. The continuous-rotation experiments provide further evidence of improved tracking beyond the training lengths. These capability improvements are valuable even without large gains on general language tasks.

Critical weaknesses
  1. The evidence for broad practical gains is limited, and the statistical reporting needs clarification. Default-layout perplexity improvements over Mamba-3 are approximately 0.14%–0.25%, while zero-shot and recall gains are not consistently significant. At 3B, mean recall accuracy is lower: 30.8 versus 31.5. The experimental setup reports three MIMO seeds, whereas the table caption indicates five. A reported MIMO p-value of 0.0079 is incompatible with the stated exact two-sided rank-sum test if each group contains only three observations; the minimum would be 0.1. This inconsistency needs to be resolved before those significance claims can be assessed.

  2. The scope of the state-tracking theorem should be clearer. It analyzes a finite set of accumulated rotations with an abstract decoder, rather than the complete recurrence and readout. The S5 limitation should be stated within that setting. Separately, the positive construction does not cover normalized inputs. The length-generalization plots also select the best seed at each length, so mean curves and variability would give a clearer picture.

Questions for authors
  1. What are the actual seed counts and per-seed results for the MIMO comparisons? Please reconcile the experimental setup, table caption, and reported statistical tests.
  2. Can you show mean performance and variability across the existing seeds for the length-generalization curves? Does the advantage remain substantial?
  3. Does the S5 limitation apply only to the stated rotation-based definition of tracking? Please clarify how this relates to the full recurrence and implemented readout, and separately whether the positive construction can cover normalized inputs.
3 comments

We thank the Reviewer for the careful consideration and are glad they found the combination of noncommutative transitions with efficient SSD computation clean and the state-tracking results strong.

Following the Reviewer's suggestions, we make the following main changes:

  • Practical gains (W1): New 3B pretraining on 600B tokens with large gains on code generation.
  • Statistical reporting (W1, Q1): Seed counts made consistent in the text, all 3B seeds completed, added per-seed MIMO results.
  • Theory scope and variability (W2, Q2, Q3): mean ± sd for the length-generalization results, extensive experimental measurements of deeper models on S5S_5.

We discuss each point in detail below.


W1 & Q1: Practical gains and statistical reporting

We agree with the Reviewer that the perplexity gains over Mamba-3 at the original scales are modest, and that zero-shot and recall gains are not consistently significant: at 3B, now with all five seeds, mean recall is 31.0 ± 0.4 for Icosian vs. 31.9 ± 1.4 for Mamba-3, a difference that is not significant (p = 0.22).

Following the Reviewer's suggestion for more practical evaluation, we now pretrain Mamba-3 and Icosian models at much larger scales to obtain a larger effect size on their language modeling capabilities. Coupled with the existing statistical evidence, now with five seeds for every model, our results show robust gains for Icosian in specialized domains. Concretely, in our new set of results, we pretrain Mamba-3, Icosian (1/2,1/321/2, 1/32) and Icosian (1/4,1/41/4, 1/4) at 3B parameters on 600B tokens in multiple phases. Icosian notably improves code generation over Mamba-3, with modest gains in general domains, as expected from prior work on expressivity [1, 2], and the code gain grows with the number of quaternionic blocks.

BenchmarkMamba-3Icosian (1/2, 1/32)Icosian (1/4, 1/4)
Code42.846.148.9
HumanEval39.040.244.5
HumanEval+36.039.039.6
MBPP51.657.161.4
MBPP+44.748.150.0
Math & reasoning (GSM8k, MATH, BBH, DROP)39.740.539.2
World knowledge (NQ, TriviaQA)28.428.128.5

Code: pass@1. Others: few-shot exact match (DROP: F1), average scores.

Seeds and p-values. All language-modeling results in the revision use five seeds per model, at every scale (Section 4.1).

  • MIMO (0.5B): these runs always had five seeds per model; the experimental setup of the submitted version incorrectly stated three. We have corrected the text so that setup, captions and tests agree. With five seeds a side, the smallest attainable p of the exact two-sided rank-sum test is 2/(105)=0.00792/\binom{10}{5} = 0.0079, the value reported for Icosian (1/4,1/4)(1/4, 1/4); with three seeds a side it would indeed be 0.1, as the Reviewer correctly notes.
  • 3B: the submitted version reported three seeds per model. The two remaining seeds have since finished, and all 3B numbers in the revision, including the recall averages above and their tests, are over five seeds.

Per-seed MIMO perplexities:

Model (0.5B MIMO)s0s1s2s3s4Mean (sd)p vs. Mamba-3
Mamba-321.3821.3521.3921.3721.3221.36 (0.03)–
Icosian (1/2, 3/32)21.2621.3021.3021.3821.3221.31 (0.04)0.095
Icosian (1/2, 1/4)21.2621.3221.3221.3621.3021.31 (0.04)0.095
Icosian (1/4, 1/4)21.2521.2321.3121.2921.2621.27 (0.03)0.0079

W2 & Q3: Scope of the state-tracking theorem

Our theory currently covers a single-layer Icosian, giving lower and upper bounds on its expressivity. Formally, the S5S_5 limitation applies to the rotation-based definition of tracking, i.e., to the accumulated rotations with an abstract decoder, not to the full recurrence and implemented readout. The positive construction assumes unnormalized layer inputs, while the normalized case is left to future work.

At the same time, we now provide substantial new empirical investigation of the full recurrence and readout with normalized inputs and multi-layer models. We originally reported that a single-layer Icosian reaches 12.3 ± 2.8% on S5S_5 (Table 3). Now we evaluate deeper models, which still do not fully track S5S_5, regardless of the utilized readout (with/without MLP), unlike Gated DeltaProduct (GDP) [2] with negative eigenvalues [3]. Reported below is accuracy in %, mean ± sd over 3 seeds (more in our response to Reviewer fW4X):

ModelDepthno MLPwith MLP
GDP [-1,1], n_h=2273.7 ± 17.855.4 ± 2.9
GDP [-1,1], n_h=4295.8 ± 3.092.3 ± 4.7
GDP [-1,1], n_h=2492.2 ± 4.267.6 ± 14.0
GDP [-1,1], n_h=4493.4 ± 4.893.7 ± 5.4
Icosian448.2 ± 5.947.8 ± 7.6
Icosian647.3 ± 0.743.4 ± 0.7

Based on these experimental measurements, the upper bound on the expressivity of single-layer Icosian seems to extend to multi-layer Icosian with normalized inputs. These results also more clearly differentiate the expressivity of GDP and that of Icosian (see item 2 in the official comment for a deeper discussion).

Q2: Length generalization, mean and variability

In the synthetic tasks evaluated in Table 3 and Figure 4, we already report the mean over seeds (with sd and range). For Figure 3, which the Reviewer might be referring to, the mean ± sd of the best validation accuracy is 58.4 ± 2.1 for Mamba-3 and 85.6 ± 16.8 for Icosian. These numbers are also reported in Appendix B.4. Therefore, the advantage remains substantial on average. We now point more clearly to Appendix B.4 from the main text.

[1] Merrill, William, et al. "Olmo Hybrid: From Theory to Practice and Back." arXiv:2604.03444 (2026).

[2] Siems, Julien, et al. "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products." NeurIPS (2025).

[3] Grazzi, Riccardo, et al. "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues." ICLR (2025).

We once again thank the reviewer for their constructive criticism and hope that we could sufficiently address their concerns. We wonder if they would consider reassessing the work in this final stage of the Author-Reviewer discussion period, taking in mind our added results and clarifications.

Reviewer_fW4XOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces Icosian, a selective state-space model that can track certain non-commutative state transitions using quaternionic parameterizations. This enables non-commutative transition dynamics while retaining efficient computation through quaternion arithmetic, rather than generic matrix operations. The authors compare Icosian against Mamba-3, Gated DeltaNet, and Gated DeltaProduct at model scales ranging from 500M to 3B parameters, reporting improved language modeling performance. They also evaluate Icosian on synthetic state-tracking tasks, showing that a single-layer model can track the non-solvable group (A_5) beyond the training sequence length. Additionally, they demonstrate that deeper Icosian models perform well on continuous state-tracking tasks involving rotations in (SO(3)) and (SO(4)).

The main theoretical result is Theorem 1, which establishes that a single-layer Icosian model with quaternionic-valued transitions and sufficient width can track any finite subgroup of (SO(4)), including (A_5), but cannot track the permutation group (S_5).

Critical strengths

The idea of enabling non-commutative state transitions in linear RNNs while retaining an efficient quaternion-based parameterization (which as far as I could follow remains diagonal) is very interesting. The paper is also well-written and easy to follow.

Theorem 1 provides a useful characterization of the capabilities and limitations of single-layer Icosian models for state tracking.

Critical weaknesses

1- While the idea and results are interesting, as acknowledged in the Limitations section, it remains an open question whether and how increasing the depth of Icosian would improve its expressivity. In particular, the comparison with other models, including DeltaProduct, on the synthetic state-tracking tasks in Table 2 appears to involve only single-layer models, based on the description on line 353. However, the caption of Table 2 does not specify the depth of each model; clarification would be helpful. The DeltaProduct paper shows that, under certain parameter conditions, a model with depth at most four can track (S_5). It is therefore important to compare multi-layer models on these tasks, as this is also the setting relevant to practical applications. Additionally, Table 2 does not make clear the sequence lengths at which each model achieves its best validation accuracy. I encourage the authors to clarify these points and briefly discuss them in the Limitations section. Otherwise, readers might incorrectly conclude that Icosian outperforms DeltaProduct on state-tracking tasks in general, rather than in the specific experimental setting considered.

2- Related to the first point, the range of groups that Icosian can track beyond (SO(4)) and its finite subgroups remains unclear. For comparison, the DeltaProduct paper provides a characterization of the groups that its multi-layer models can track under certain conditions. Without a better understanding of Icosian's expressivity beyond the cases studied in this work, it is difficult to assess the scope and significance of its theoretical contribution.

3- Minor: Some notation could be clarified. For example, at line 147, quaternion multiplication (denoted by (\times)) is used without being explicitly introduced. Although its meaning is clear from the context, defining the operation earlier or providing a simple example would improve readability.

Questions for authors

Questions

  1. Comparing Tables 2 and 7, there seem to be some inconsistencies in the baseline results that would benefit from clarification. Why do both Icosian and Mamba-3 fail on the seemingly simple parity task in Table 7, even though Icosian successfully tracks A5A_5? Similarly, why does Mamba-3 fail on the solvable groups Q8Q_8 and D8D_8 in Table 2? Is the Mamba-3 baseline in these experiments a single-layer model?

  2. Why should improved state-tracking capabilities be expected to translate into better performance on general language modeling tasks? While such improvements seem particularly plausible for tasks involving mathematical reasoning or code generation, I wonder whether the authors considered evaluating Icosian on more targeted benchmarks, such as MathHard or CodeParrot, as explored in some prior work. Such evaluations could help clarify whether the improved state-tracking capabilities translate into gains on tasks that might benefit more directly from them.

  3. What does MIMO refer to in this work? It is mentioned in Section 4.1 without being introduced earlier. Does the Icosian family include both MIMO and SISO variants, and, if so, which variant is used in each experiment?

5 comments

We thank the Reviewer for the careful consideration and are glad they found our work theoretically interesting, well written and easy to follow.

Table numbers below follow the original submission; in the revised PDF, Tables 1, 2, 7 and 8 are Tables 2, 3, 9 and 13.

Following the Reviewer's suggestions, we make the following main changes:

  • Multi-layer models on S5S_5 (W1): GDN (4–8 layers), DeltaProduct (nh∈{2,4}n_h \in \{2, 4\}, 2–4 layers) and Icosian (4–6 layers), with and without MLPs.
  • Table 7 (W1): setup clarified; learning-rate sweep added.
  • 3B pretraining on 600B tokens (Q2): Icosian substantially improves code generation over Mamba-3.
  • Clarifications (W2, W3): The scope of our theory, the cross product, and the MIMO readout inherited from Mamba-3.

We discuss each point in detail below.


W1 & Q1: Model depth and Table 2/7 baselines

We thank the Reviewer for highlighting the important connection between depth, expressivity and learnability, and agree that multi-layer models are more practically relevant than the single-layer models of Table 2. The Reviewer correctly points to the DeltaProduct result [6] that at most 4 layers (nh=1n_h=1) suffice for S5S_5. We now evaluate multi-layer models (and, for DeltaProduct, several nhn_h) on S5S_5, where all single-layer SSMs effectively fail. A summary of the results is below:

ModelDepthno MLP: acc. (%)paramswith MLP: acc. (%)params
GDN [-1,1] (Table 2)19.9 ± 1.07.1M––
GDP [-1,1], n_h=2 (Table 2)141.5 ± 4.27.1M––
Icosian (Table 2)112.3 ± 2.87.1M––
GDN [-1,1]453.4 ± 3.927.7M56.3 ± 5.962.3M
GDN [-1,1]652.4 ± 1.241.4M51.2 ± 2.593.3M
GDN [-1,1]855.8 ± 6.855.1M50.2 ± 2.5124.3M
GDP [-1,1], n_h=2273.7 ± 17.813.9M55.4 ± 2.931.2M
GDP [-1,1], n_h=4295.8 ± 3.021.3M92.3 ± 4.738.6M
GDP [-1,1], n_h=2492.2 ± 4.227.5M67.6 ± 14.062.1M
GDP [-1,1], n_h=4493.4 ± 4.842.4M93.7 ± 5.477.0M
Icosian448.2 ± 5.927.5M47.8 ± 7.662.1M
Icosian647.3 ± 0.741.2M43.4 ± 0.793.1M

Accuracy in %, mean ± sd over 3 seeds.

These results confirm that depth, expressivity and learnability are closely intertwined. Indeed, even at the same number of layers, adding components such as MLPs fundamentally changes the outcome: with MLPs, the accuracy of GDP with nh=2n_h=2 goes from 92.2% to 67.6% at 4 layers (73.7% to 55.4% at 2 layers), while GDN and Icosian change only slightly at every depth, and do not solve the task even at 8 (GDN) and 6 (Icosian) layers. The interplay between multi-layer expressivity and learnability is an active area of research, studied for diagonal SSMs [1], DeltaNet [2] and Transformers [3]. We clarify that the comparison in Table 2 concerns single-layer models, which we now state in the caption.

Regarding Mamba-3 on Q8 and D8: the Mamba-3 baseline of Table 2 is a single-layer model, and a single complex-diagonal layer cannot represent any non-Abelian group [1]. Although multi-layer complex diagonal SSMs can in principle represent solvable groups [1], in practice they are observed to perform considerably worse on non-Abelian solvable groups than on Abelian ones [1, 7]. Q8 and D8 are solvable but non-Abelian, so the results of Mamba-3 in Table 2 are consistent with these prior observations.

As for the inconsistent result between Tables 2 and 7, they use different setups: Table 7 follows the prior work cited in the caption exactly for reproducibility, with a single untuned learning rate and 2-layer models, while Table 2 tunes the learning rate on single-layer models (and now multi-layer ones above). We now clearly state this in the text. We also sweep the learning rate of the Table 7 models over {10−4,5⋅10−4,10−3}\{10^{-4}, 5\cdot10^{-4}, 10^{-3}\} in addition to 2⋅10−32\cdot10^{-3}. The results are summarized below:

ModelParityMod. Arith.
Mamba-387.3 ± 14.265.3 ± 11.3
GDP [-1,1], 2 refl.99.7 ± 0.398.2 ± 3.2
Icosian (1/2, 3/32)99.5 ± 1.173.8 ± 6.6

Mean ± sd over 5 seeds at the best learning rate, width 128. With tuning, Icosian matches GDP on Parity; Mamba-3 improves (79.8 → 87.3 on the same 5 seeds at width 128; Table 7 reports 73.2 over ten seeds and the best width), with only some seeds reaching 100%100\%. The Mamba-3 paper's 100.0 is a best-of-grid result on parity trained on lengths up to 160 (ours: 3–40, mean over seeds), hence the difference.

Regarding sequence lengths in Table 2: the best validation accuracy is not measured at a single length, but is the token accuracy over the entire validation set (lengths 40–256, all positions scored), evaluated at the best checkpoint; training uses lengths 2–40. We now state this in the caption of Table 2.

W2: Expressivity beyond SO(4)

In this work, we provide theoretical guarantees for a single layer, together with extensive empirical results for multi-layer models, both on finite groups (W1) and on continuous rotations in SO(3) and SO(4) (Section 4.2). Multi-layer theory has so far focused on diagonal [1, 4, 5] and Householder-based [2, 6] transitions, and extending it to the quaternionic transitions of Icosian is an exciting direction for future research.

W3 & Q3: Notation (quaternion product, MIMO)

We thank the Reviewer for the suggestions that help clarify our manuscript. Regarding W3, the ×\times operation is in fact defined on line 148, just as the Reviewer suggests: it is the standard cross product of two 3D vectors, which may be a less familiar operation than e.g. the dot product.

All experiments are SISO unless explicitly mentioned, with MIMO referring to the same MIMO design as Mamba-3. We now clarify this aspect in Section 3.

Q2: State tracking and language modeling

As the Reviewer correctly points out, gains from state tracking are more intuitive in code or mathematics than in general language modeling, as also observed for DeltaProduct [6]. However, achieving such capabilities in practice requires notably larger and more sophisticated pretraining. Therefore, we now pretrain Mamba-3, Icosian (1/2,1/321/2, 1/32) and Icosian (1/4,1/41/4, 1/4) at 3B parameters on 600B tokens in multiple phases; the number of quaternionic blocks acts as an expressivity knob, like nhn_h in DeltaProduct.

The experimental results below, on targeted code (HumanEval, MBPP) and math (GSM8k, MATH) benchmarks, confirm the intuition for coding tasks: improved state tracking does not significantly impact general language modeling, but does notably impact code generation. Math and reasoning results remain inconclusive at this scale.

BenchmarkMamba-3Icosian (1/2, 1/32)Icosian (1/4, 1/4)
Code42.846.148.9
HumanEval39.040.244.5
HumanEval+36.039.039.6
MBPP51.657.161.4
MBPP+44.748.150.0
Math & reasoning (GSM8k, MATH, BBH, DROP)39.740.539.2
World knowledge (NQ, TriviaQA)28.428.128.5

Code: pass@1. Others: few-shot exact match (DROP: F1), average scores.

[1] Shakerinava, Mehran, et al. "The Expressive Limits of Diagonal SSMs for State-Tracking." ICLR (2026).

[2] Grazzi, Riccardo, et al. "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues." ICLR (2025).

[3] Li, Belinda Z., et al. "(How) Do Language Models Track State?" ICML (2025).

[4] Merrill, William, et al. "The Illusion of State in State-Space Models." ICML (2024).

[5] Sarrof, Yash, et al. "The Expressive Capacity of State Space Models: A Formal Language Perspective." NeurIPS (2024).

[6] Siems, Julien, et al. "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products." NeurIPS (2025).

[7] Terzić, Aleksandar, et al. "On the Expressiveness and Length Generalization of Selective State Space Models on Regular Languages." AAAI (2025).

I would like to thank the authors for clarifying the points, such as the number of layers for synthetic tasks, explaining more details about the extent of the experiments and also adding new experimental results. It is interesting to see the better performance of Icosian in comparison to Mamba-3 on code-involved tasks. Only as a suggestion, I think this comparison would be more interesting against DeltaProduct, which is also known to be capable of doing more difficult state-tracking tasks than Mamba-3.

Also, I really appreciate the additional results with multi-layer models on synthetic tasks, and it is intriguing to see that Icosian with multiple layers is, in practice, not capable of doing S5, while DeltaProduct solves the task. And by the way, are these results (for example, the new Table 5 in the paper) for the training sequence length, or do they show the results of length extrapolation? It would be nice to add complete information about these details in the captions of the tables to make them self-contained.

It is also interesting to see that adding MLP layers in multi-layer DeltaProduct worsens its results in several cases, including its performance on the S5 or A5 group. I am curious if the authors have any hypothesis for why this could be the case.

Also, one final suggestion: as I find the experimental results in this paper very interesting, I am wondering if the authors would consider including a comparison of best-seed results between different models. I know that the commonly appreciated way of reporting results is by averaging over seeds, and it is great that the authors already did so; but I think for expressivity results, reporting the best observed performance of a model is also pretty valuable and gives a complementary insight into what each model can achieve, like what can we empirically establish as a lower bound of its performance.

Based on the fact that it remains unclear to me whether Icosian provides remarkable advantages over other models such as DeltaProduct, I maintain my score of 3.

Comments on the paper

2 comments

We thank all Reviewers for their thoughtful feedback. The reviews raised critical points on expressivity and practical applicability; we have updated the manuscript accordingly, with changes highlighted in blue.

We highlight the main new contributions below.

  1. We now evaluate the practical gains of the increased expressivity of Icosian at large scale, and find that code generation notably benefits from the non-commutative rotations. Specifically, we pretrain Mamba-3, Icosian (1/2,1/321/2, 1/32) and Icosian (1/4,1/41/4, 1/4) at 3B parameters on 600B tokens with multiple phases. All variants are matched in parameters, with (1/4,1/41/4, 1/4) devoting more of the state to quaternionic (non-commutative) blocks than (1/2,1/321/2, 1/32).
BenchmarkMamba-3Icosian (1/2, 1/32)Icosian (1/4, 1/4)
Code42.846.148.9
HumanEval39.040.244.5
HumanEval+36.039.039.6
MBPP51.657.161.4
MBPP+44.748.150.0
Math & reasoning (GSM8k, MATH, BBH, DROP)39.740.539.2
World knowledge (NQ, TriviaQA)28.428.128.5

Code: pass@1. Others: few-shot exact match (DROP: F1), average scores.

Icosian improves code generation over Mamba-3 by up to 6.1 points on average, and the gain grows with the number of quaternionic blocks, while math, reasoning and world knowledge remain on par. This matches the intuition that state tracking matters most in structured domains such as code, and that greater expressivity yields better scaling [1]. Similar benefits were reported for the DeltaProduct model [2], where increasing expressivity improved code and mathematical domains. We now add these results to the paper.

  1. We now also substantially expand our experimental capability verification: we evaluate GDN (4–8 layers), DeltaProduct [2] (nh∈{2,4}n_h \in \{2, 4\}, 2–4 layers) and Icosian (4–6 layers) on S5S_5, each now with and without MLPs after the SSM layer. GDN and GDP use the negative-eigenvalue range [−1,1][-1,1] [3].
ModelDepthno MLP: acc. (%)paramswith MLP: acc. (%)params
GDN [-1,1] (Table 3)19.9 ± 1.07.1M––
GDP [-1,1], n_h=2 (Table 3)141.5 ± 4.27.1M––
Icosian (Table 3)112.3 ± 2.87.1M––
GDN [-1,1]453.4 ± 3.927.7M56.3 ± 5.962.3M
GDN [-1,1]652.4 ± 1.241.4M51.2 ± 2.593.3M
GDN [-1,1]855.8 ± 6.855.1M50.2 ± 2.5124.3M
GDP [-1,1], n_h=2273.7 ± 17.813.9M55.4 ± 2.931.2M
GDP [-1,1], n_h=4295.8 ± 3.021.3M92.3 ± 4.738.6M
GDP [-1,1], n_h=2492.2 ± 4.227.5M67.6 ± 14.062.1M
GDP [-1,1], n_h=4493.4 ± 4.842.4M93.7 ± 5.477.0M
Icosian448.2 ± 5.927.5M47.8 ± 7.662.1M
Icosian647.3 ± 0.741.2M43.4 ± 0.793.1M

Accuracy in %, mean ± sd over 3 seeds.

These results provide strong empirical evidence that our theoretical upper bound, originally derived for single-layer Icosian, extends to deeper models: at depths 4 and 6, with and without MLPs, Icosian improves substantially over its single-layer accuracy (12.3% → over 43%) but does not fully track S5S_5, as the characterization predicts. DeltaProduct with more reflections, whose transitions are not limited to SO(4)SO(4) blocks, approaches full tracking, confirming that the evaluation is able to separate the two.

  1. We add a parameter-matched Transformer baseline and an Icosian hybrid, and clarify our statistical reporting. We train a Llama-style Transformer (RoPE, SwiGLU, RMSNorm, FlashAttention-3) and an Icosian hybrid with attention in 4 of 24 layers, both following exactly the protocol from Table 2. Results at 0.5B over 3 seeds are below:
Model (0.5B)ParamsVal. PPL ↓
Transformer510.8M23.00 ± 0.07
Icosian + attention (4 of 24 layers)504.5M21.83 ± 0.06
Mamba-3514.1M21.76 ± 0.05
Icosian514.6M21.73 ± 0.02

At matched parameters and data, Icosian improves over the Transformer by 1.27 perplexity points, and the hybrid improves over it by 1.17 points. We also make the seed counts consistent throughout the text: the 0.5B MIMO comparison uses five seeds a side, for which the smallest attainable pp of the exact two-sided rank-sum test is 2/(105)=0.00792/\binom{10}{5} = 0.0079, the value reported. At 3B, the two remaining seeds have since finished, and all 3B results are now over five seeds. We now add the per-seed MIMO perplexities to the appendix.

[1] Merrill, William, et al. "Olmo Hybrid: From Theory to Practice and Back." arXiv:2604.03444 (2026).

[2] Siems, Julien, et al. "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products." NeurIPS (2025).

[3] Grazzi, Riccardo, et al. "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues." ICLR (2025).

To all Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We are deeply grateful to all Reviewers for the interesting and thoughtful discussions over the past two weeks. We would like to individually thank each Reviewer, specifically:

  • Reviewer fW4X for recommending us strongly for acceptance, because of our interesting technical novelty and our demonstration that this novelty does, in fact, translate to concrete advantages in practice;
  • Reviewer tqgP for highlighting our coherent approach and our clear demonstrations of improvements over Mamba3, and for acknowledging that all their concerns have been addressed;
  • Reviewer 8NSQ for recognizing our clear and concrete contribution, especially our strong results. Regrettably, we were unable to connect with the Reviewer, but are convinced that our responses satisfactorily address all their concerns;

We thank once again the Reviewers for the productive discussion, and believe that our work has been much improved during this rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.