- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Use of AI Tools
I used ChatGPT to help understand the paper, check the factual accuracy of my review comments, refine their structure and wording, translate my review from my native language into English, and consider the rating and confidence level.
Summary of Requests
- “Please translate this into my native language.”
- “I have written the review comments above. Are any of my points incorrect? Please check their factual accuracy while respecting my own opinions.”
Critical strengths
- Extending Mamba-3’s complex-valued state transitions to quaternions is a natural design choice, and the proposed non-commutative rotations directly address the motivating expressivity limitation. The reformulation through cumulative quaternion products also enables the use of existing SSD computation, providing a coherent approach to combining increased expressivity with computational efficiency.
- The paper demonstrates clear improvements over Mamba-3 on non-commutative state-tracking tasks. In particular, a single-layer model achieves 100% accuracy on A5 at sequence lengths beyond those used during training, providing empirical support for the theoretical motivation.
Critical weaknesses
- The mechanism behind the language-modeling improvements is insufficiently analyzed. On synthetic tasks, the theory and defect analysis of learned rotations explain the effectiveness of non-commutative transitions. For language modeling, however, it remains unclear what the quaternionic blocks learn and how non-commutativity contributes to the perplexity improvements. Without a controlled experiment that restricts non-commutativity, the results do not distinguish improvements due to non-commutativity itself from those due to the additional parameterization.
- The gains in state tracking are task-dependent. On the finite-state tasks in the Chomsky-hierarchy benchmark, Icosian improves over Mamba-3 but does not consistently outperform other recurrent models. Although Theorem 1 establishes conditions for tracking finite groups, the paper does not explain the performance differences across the tasks in Table 7 or examine which architectural properties, such as the availability of eigenvalue −1 or a discrete transition dictionary, contribute to these differences.
- The language-modeling evaluation lacks a comparison with Transformers. The baselines are limited to Mamba-3, Gated DeltaNet, and Gated DeltaProduct. Without a Transformer baseline trained with a comparable parameter count and amount of data, Icosian’s position in the trade-off between language-modeling performance and computational efficiency remains insufficiently established.
Questions for authors
-
What types of linguistic dependencies do the quaternionic non-commutative state transitions help represent? Could an analysis of the learned quaternionic blocks, or a controlled experiment restricting non-commutativity, clarify whether the perplexity improvements are attributable to non-commutativity? Such evidence would help establish the significance of the proposed approach.
-
How do you explain Icosian’s lower average accuracy compared with GDP, Flash PD-SSM, and the cited LSTM results in Table 7? Does this gap arise from transition expressivity, optimization difficulty, or differences in experimental settings? Please discuss why the substantial improvements on non-commutative group tracking do not translate into consistent advantages on these tasks.
-
Could you add a comparison with a Transformer trained with a comparable parameter count and amount of data? Comparing language-modeling performance alongside training time, inference speed, and memory usage would clarify Icosian’s position in the performance–efficiency trade-off.
-
Minor correction: The Table 1 caption states “Five seeds at each scale except 3B,” whereas Section 4.1 states that the 0.5B MIMO configuration also uses three seeds. Please make the descriptions of the seed counts consistent.
4 comments
W3 & Q3: Transformer baseline
We now add a Llama-style Transformer (RoPE, SwiGLU, RMSNorm, FlashAttention-3) and an Icosian hybrid with attention in 4 of 24 layers (every sixth). Both exactly follow the protocol from Table 1, this being the established FineWeb data-ablation protocol, as detailed in our Section 4.1. Results at 0.5B are reported below:
| Model (0.5B) | Params | Val. PPL ↓ |
|---|---|---|
| Transformer | 510.8M | 23.00 ± 0.07 |
| Icosian + attention (4 of 24 layers) | 504.5M | 21.83 ± 0.06 |
| Mamba-3 | 514.1M | 21.76 ± 0.05 |
| Icosian | 514.6M | 21.73 ± 0.02 |
At matched parameters and data, the Transformer trails Mamba-3 and Icosian by over 1.2 perplexity points. By interleaving self-attention layers with the Icosian in a hybrid architecture, we can decrease the gap to the pure Icosian models by over 1 perplexity points.
Training time, inference speed and memory. We measure the parameter-matched models on the same H200 GPU:
| Transformer | Mamba-3 | Icosian (1/2, 3/32) | |
|---|---|---|---|
| Training: step (ms); peak memory (GB) | |||
| 0.5B | 210; 40.9 | 252; 50.1 | 304; 53.7 |
| 1.1B | 197; 37.5 | 239; 44.3 | 271; 46.9 |
| 3B | 165; 55.8 | 203; 55.6 | 210; 55.6 |
| Inference at 1.1B: time (s); peak memory (GB) | |||
| =512 | 6.9; 20 | 3.4; 15 | 3.7; 16 |
| =1024 | 19.7; 38 | 6.8; 10 | 7.3; 10 |
| =2048 | 64.0; 74 | 13.7; 10 | 14.7; 10 |
| =4096 | OOM | 27.3; 10 | 29.3; 10 |
| =16384 | OOM | 105.9; 10 | 118.5; 10 |
With the highly optimized FlashAttention-3 at sequence length 2048, the Transformer has the cheapest training step. Icosian's overhead decreases with scale, both over the Transformer (1.45× at 0.5B, 1.27× at 3B) and over Mamba-3 (1.20× at 0.5B, 1.04× at 3B).
At inference, Icosian is 1.9–4.4× faster than the Transformer and keeps constant memory, whereas the Transformer's key/value cache grows linearly and exceeds GPU memory from =4096 on.
Q4: Seed counts
All language-modeling results now use five seeds per model, as stated in Section 4.1: the MIMO runs always had five seeds and the text has been corrected, and at 3B the two remaining seeds have since finished. The new Transformer and hybrid baselines above use three seeds due to time limitations during the discussion phase; we will also evaluate these models using five seeds.
[1] Poli, Michael, et al. "Mechanistic Design and Scaling of Hybrid Architectures." ICML (2024).
[2] Merrill, William, et al. "Olmo Hybrid: From Theory to Practice and Back." arXiv:2604.03444 (2026).
[3] Siems, Julien, et al. "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products." NeurIPS (2025).
[4] Grazzi, Riccardo, et al. "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues." ICLR (2025).
[5] Terzić, Aleksandar, et al. "Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models." NeurIPS (2025).
[6] Terzić, Aleksandar, et al. "Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models." NeurIPS (2026).
Thank you for the additional experiments and the detailed response. All of my concerns have been addressed. In the revised version, please include these experiments and further deepen the discussion of this method.
We thank the Reviewer for acknowledging our response, and are glad to find that it addressed all their concerns. We remain available for further discussion, and kindly ask the Reviewer to reconsider our score if no more open points remain.
We thank the Reviewer for their careful reading of the paper, and for highlighting the coherence of our approach and the theoretical and empirical evidence in support of Icosian.
Table numbers below follow the original submission; in the revised PDF, Tables 1, 2, 7 and 8 are Tables 2, 3, 9 and 13.
Following the Reviewer's suggestions, we make the following main changes:
We discuss each point in detail below.
W1 & Q1: Mechanism behind the language-modeling gains
The connection between expressivity and language modeling performance is a very active field of research, with prior literature showing that performance on synthetic tasks correlates with compute-optimal perplexity across architectures [1] and that greater expressivity yields better scaling efficiency [2]. More recently, the DeltaProduct work [3] has demonstrated that increased expressivity induces most substantial gains in the domains of code generation and mathematical reasoning.
In the context of our work, we provide a set of experiments that varies the amount of non-commutativity in Icosian as a control on downstream performance in Table 8. Under a parameter-matched setting, the number of quaternionic blocks ranges from none (Mamba-3) to a quarter of the state, with modest gains as non-commutativity increases. Since the parameter budget is fixed (Icosian adds less than 0.1% parameters over Mamba-3), these gains cannot be attributed to additional parameterization. A mechanistic analysis of which linguistic dependencies the learned quaternionic blocks capture remains open.
We now test this effect for Icosian at large scale, by pretraining Mamba-3, Icosian (1/2,1/32) and Icosian (1/4,1/4) at 3B parameters on 600B tokens; the two Icosian variants match in parameters, with (1/4,1/4) containing more non-commutative structure. Our results show that Icosian substantially improves code generation over Mamba-3, with more modest gains on more general domains, and the code gain grows with the amount of non-commutative structure:
Code: pass@1. Others: few-shot exact match (DROP: F1), average scores.
W2 & Q2: Table 7 and task-dependent gains
Unlike the group tracking of Table 2, Table 7 follows prior work cited in the caption exactly for reproducibility, with a single untuned learning rate of 2e-3 and 2-layer models. We now sweep the learning rate of its central models over {10−4,5⋅10−4,10−3} in addition to 2⋅10−3:
Mean ± sd over 5 seeds at the best learning rate, width 128.
Mamba-3 on Parity improves from 79.8 to 87.3 on the same 5 seeds at width 128 (Table 7 reports 73.2 over ten seeds and the best width); certain seeds reach 100%, but not consistently. The Mamba-3 paper reports 100.0 on Parity as the best over a hyperparameter grid (2 widths, 8 learning rates), with a curriculum raising the maximum training length from 40 to 160, whereas we train on lengths 3 to 40 and report the mean.
The roles of negative eigenvalues [4] and discrete transition dictionaries [5, 6] are analysed in the works that introduce them, as discussed in our related work. We use the more expressive negative-eigenvalue versions of DeltaNet and DeltaProduct, and ablate the number of DeltaProduct factors in additional S5 experiments (see our response to Reviewer fW4X).