- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose Rotary Stage Gated DeltaNetwork, which applies an input-dependent rotation on the state transition of the traditional Gated DeltaNetwork. The authors show that this enables for improved representational capabilites in terms of state tracking while also theoretically being able to track any regular language. Experiments on state-tracking tasks are used to validate this claim while standard language modeling performance remains comparable to traditional models.
Critical strengths
The method is easy to understand and intuitive/sensible. The chunkwise parallelization in Appendix B appears to be correct and Figure 1 would appear to show that the implementation is fairly efficient.
Critical weaknesses
First, I have some concerns about the approach and its novelty. In praticular, the approach appears to be rather similar to [1], which introduces input-dependent learnable rotary embeddings that generalizes RoPE to arbitrary angles, which the authors show with GLA (a related model to GDN) can yield a complex-valued recurrent state that has similar performance gains as those claimed here.
Next, in terms of the formal language analysis (which I am not a full expert on and thus cannot speak directly on the novelty of the theoretical analysis and proofs), I believe that [2] did show similar proofs regarding negative eigenvalues and the ability to generalize to any regular language, which dampens part of the contribution that the authors appear to emphasize. Furthermore, a lot of focus is also on unsolvable groups such as (which is a focus of [4]), which the work doesn't really delve into and thus the comparison against some related works would appear to be missing. Thus while abelian group tracking could be improved by the approach, whether this translates to other algebraic groups or more real world tasks remains an open question.
On the front of real-world tasks, the language modeling results are also fairly modest compared to GDN, while the NIAH performance while it does show some better decaying behaviour still does not completely resolve the issue. Furthermore, performance at the in-distribution training length (4096 tokens) has some concerns would suggest that the models are possibly not trained enough, or alternatively they have reached capacity and thus larger models/datasets would be necessary to derive any additional conclusions.
[1] Selective Rotary Position Embedding. Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher. ICLR. 2026
[2] Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K.H. Franke, Frank Hutter, Massimiliano Pontil. ICLR. 2025
[3] The Illusion of State in State-Space Models. William Merrill, Jackson Petty, Ashish Sabharwal. ICML. 2024
Questions for authors
- Results from Table 2 would appear to contradict some details from the Mamba3 work; I am wondering if this may be perhaps that different models did not recieve the same degree of hyper-parameter tuning and this may lead to different results, which could be an issue with transparancy.
- The reporting of results is slightly inconsistent, with Table 3/Figure 1 reporting Gated DeltaProduct and Transformer while table 4 and 5 do not.
- In Table 4 and 5, if I understand correctly the Gated DeltaNet is one such that the negative eigenvalue property is not the case. Thus would me more representative to have an actual model with such a property, maybe again providing the Gated DeltaProduct from Table 3/Figure 1 or at the very least providing a DeltaNet with negative eigenvalues which was the model proposed in [1]. Likewise, Mamba3 uses complex states (which is another avenue for richer state tracking since it models a data-dependent rotary embedding like [2], but it would be useful to compare against one that only explictly uses negative eigenvalues without the complex state.
- Though a comparison isn't directly necessary, [4] appears to propose a similar type of approach, which I believe the authors can add as a small point of discussion.
- Also not direclty related to a specific weakness, given that Rose-GDN would theoretically still suffer from the same issues of fixed-sized hidden state, could the addition of components such as [5] improve performance, given what may be inconclusive results in Tables 3 and 4?
[4] Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention. Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein. arXiv. 2026
[5] Log Linear Attention. Han Guo, Songlin Yang, Tarushii Goel, Eric P. Xing, Tri Dao, Yoon Kim. ICLR. 2026
9 comments
Response to Reviewer ngPB: Part 2/4
[W2] Next, in terms of the formal language analysis, I believe that [2] did show similar proofs regarding negative eigenvalues and the ability to generalize to any regular language, which dampens part of the contribution that the authors appear to emphasize.
Answer. We agree that our regular-language result (Thm. 2) reaches the same conclusion as [2, Thm. 4] and follows the same Krohn–Rhodes cascade [R3]. What differs is the class of layers for which the conclusion is proved. In [2, Thm. 4], each layer may use a product of up to generalized-Householder (GH) factors per token, where is the number of states of the machine. A Rose-GDN layer instead uses a single GH-rotation factor (Eq. (8)), i.e. the product of one GH matrix and one block rotation matrix.
It is worth noting that the proof of [2, Thm. 4] cannot be used directly for Rose-GDN, and we give an example to illustrate this. Consider recognizing binary numbers divisible by 5 while reading the digits from left to right. It suffices to track the remainder of the number read so far, which a digit turns from into . Reading a 0, for example, leaves the remainder 0 unchanged and sends 1 to 2, 2 to 4, 4 to 3, and 3 to 1. In the proof of [2, Thm. 4], the remainder is stored in one of five coordinates of the memory, and the update for a token is the same whichever coordinate holds it, so the update must rearrange four coordinates at once. With five states, such a rearrangement can take up to four exchanges of two coordinates, and one GH factor makes only one. Their layers make all of them in a single step by multiplying several GH factors per token. A Rose-GDN layer has only one GH factor per token, so this approach does not apply to our proof.
The two constructions also differ in depth. Both proofs first split the machine ( states in the example) into simpler machines (the Krohn–Rhodes cascade). [2, Thm. 4] uses one layer for each of them, hence layers, whereas our proof uses two layers for each plus one rotary layer, hence layers (at most , as stated in Thm. 2). Our construction is therefore deeper, but the total number of GH factors applied per token drops from up to to . We have revised Thm. 2 and added Remark 1 to state this distinction explicitly.
We emphasize that our main theoretical contribution concerns finite group tracking (Thm. 1), which is summarized in Table 1. Specifically, one Rose-GDN layer tracks every finite abelian group (Thm. 1(i)), whereas a single GDN layer provably cannot track , or any group containing an element whose order is not a power of two [2, Thm. 2]. The results in Table 3 are consistent with the theory: with one layer, GDN reaches 0% exact-sequence accuracy on and , while Rose-GDN reaches 98.7–100%.
[W3] Furthermore, a lot of focus is also on unsolvable groups such as (which is a focus of [4]), which the work doesn't really delve into and thus the comparison against some related works would appear to be missing. Thus while abelian group tracking could be improved by the approach, whether this translates to other algebraic groups or more real world tasks remains an open question.
Answer: We agree that the submission focuses on abelian groups and did not study non-solvable groups such as empirically. We have now run this experiment, and we report it below together with what our theory does and does not cover.
On the theory side, solvable non-abelian groups are covered by Thm. 1(ii). The word problem of is a regular language and is therefore covered by Thm. 2.
On the empirical side, we trained five-layer models on state tracking with sequences of length 32 (hidden size 128, 4 heads, 100,000 steps, batch size 512, learning rate ) and evaluated them on 1,024 sequences per length (5, 10, 32 and 60). We observe that exact-sequence accuracy is 0% for every model at every length, so we report token accuracy, i.e., the fraction of positions at which the state is predicted correctly.
Table ngPB.3. Token accuracy (%) on state tracking.
| Model | Length 5 | Length 10 | Length 32 | Length 60 |
|---|---|---|---|---|
| GDN | 41.48 | 21.55 | 7.91 | 4.80 |
| GDN | 41.50 | 21.72 | 7.80 | 4.92 |
| Gated DeltaProduct$_2$ | 41.35 | 20.96 | 7.05 | 4.21 |
| Mamba-3 | 40.84 | 20.98 | 7.11 | 4.11 |
| GDN + Selective RoPE | 41.35 | 21.50 | 7.90 | 4.85 |
| Rose-GDN | 41.43 | 21.71 | 7.87 | 4.95 |
Overall, none of the models successfully tracks beyond the first few steps in this setting, and Rose-GDN is no exception, so we do not claim an empirical advantage on non-solvable groups.
References
[R3] Algebraic theory of machines. i. prime decomposition theorem for finite semigroups and machines. Kenneth Krohn and John Rhodes. Transactions of the American Mathematical Society, 1965.
Response to Reviewer ngPB: Part 3/4
[W4] On the front of real-world tasks, the language modeling results are also fairly modest compared to GDN, while the NIAH performance while it does show some better decaying behaviour still does not completely resolve the issue. Furthermore, performance at the in-distribution training length (4096 tokens) has some concerns would suggest that the models are possibly not trained enough, or alternatively they have reached capacity and thus larger models/datasets would be necessary to derive any additional conclusions.
Answer: We agree with you that the language modeling gains are modest. Our claim is that Rose-GDN improves abelian-group tracking and long-context retrieval over strong recurrent baselines, while remaining competitive on language modeling and commonsense reasoning. We also agree that on the harder NIAH variants, accuracy at 4K is still low for every model. Regarding the 4,096-token setting, we should have been clearer that NIAH is evaluated zero-shot. The models are pretrained on FineWeb-Edu only and never see the retrieval task. 4,096 is the pretraining context length, so the 4K column is within the trained length range but is not in-distribution for the task.
[Q1] Results from Table 2 would appear to contradict some details from the Mamba3 work; I am wondering if this may be perhaps that different models did not recieve the same degree of hyper-parameter tuning and this may lead to different results, which could be an issue with transparancy.
Answer: Thank you for raising this. We clarify that every model in Table 2, including Mamba-3, was trained with the same budget. The discrepancy is due to the training and evaluation protocols, which the submission did not make clear enough. The two protocols are compared below.
Table ngPB.4. Training and evaluation protocols for the formal-language tasks (parity and modular arithmetic) in the Mamba-3 paper and in our Table 2.
| Mamba-3 paper (Table 5b, App. D) | Our Table 2 (protocol of Grazzi et al., 2025) | |
|---|---|---|
| Training lengths | curriculum: minimum 33, maximum raised from 40 to 160 | uniform in 3–40 |
| Test lengths | 256 | 40–256 |
| Learning rates swept | 8 | 4 |
We do not question the Mamba-3 results. Instead, our Table 2 measures how well a model generalizes to longer sequences when it is trained only on lengths up to 40. In this setting, independent work also finds Mamba-3 below perfect accuracy on parity and modular arithmetic (without brackets). For example, [R4, Table 1] also trains on lengths up to 40 and tests on 40–256 (their setup differs from ours in batch size, optimizer, and learning rate selection). They report 76.5 ± 7.8% on parity and 46.8 ± 5.5% on modular arithmetic without brackets, i.e., 0.53 and 0.34 in our scaled accuracy. [R5, Table 2] train with a curriculum up to length 60 and find that, on parity, Mamba-3 keeps 90% accuracy only up to length 200–300, while Gated DeltaNet and Gated DeltaProduct reach the maximum tested length of 1000.
[Q2] The reporting of results is slightly inconsistent, with Table 3/Figure 1 reporting Gated DeltaProduct and Transformer while table 4 and 5 do not.
Answer: Thank you. Gated DeltaProduct$_2$ and Transformer are now included in Tables 4 and 5.
[Q3] In Table 4 and 5, if I understand correctly the Gated DeltaNet is one such that the negative eigenvalue property is not the case. Thus would me more representative to have an actual model with such a property, maybe again providing the Gated DeltaProduct from Table 3/Figure 1 or at the very least providing a DeltaNet with negative eigenvalues which was the model proposed in [1]. Likewise, Mamba3 uses complex states (which is another avenue for richer state tracking since it models a data-dependent rotary embedding like [2], but it would be useful to compare against one that only explictly uses negative eigenvalues without the complex state.
Answer: Thank you for the suggestion. Your reading is right: the Gated DeltaNet in Tables 4–5 uses . We have included both Gated DeltaProduct$_2$ and Gated DeltaNet in Tables 4 and 5.
References
[R4] Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models. Aleksandar Terzic, Francesco S. Carzaniga, Nicolas Menet, Yannick Biehl, Michael Hersche, Thomas Hofmann, Abbas Rahimi. NeurIPS 2026.
[R5] Rethinking State Tracking in Recurrent Models Through Error Control Dynamics. Jiwan Chung, Heechan Choi, Seon Joo Kim. NeurIPS 2026.
Response to Reviewer ngPB: Part 1/4
Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.
[W1] First, I have some concerns about the approach and its novelty. In praticular, the approach appears to be rather similar to [1], which introduces input-dependent learnable rotary embeddings that generalizes RoPE to arbitrary angles, which the authors show with GLA (a related model to GDN) can yield a complex-valued recurrent state that has similar performance gains as those claimed here.
Answer: Compared with Selective RoPE, the distinction is that Rose-GDN erases and writes at different addresses, whereas Selective RoPE uses a single address for both. In the cumulative frame of Eq. (5), Rose-GDN is
S~t=αtS~t−1(I−βtatat⊤)+βtvtbt⊤,at=Φt−1⊤kt,bt=Φt⊤kt=Rt⊤at.
It erases at at, the address of kt before the current token's rotation, and writes at bt, the address after it. Other recent delta-rule models also treat erasing and writing separately, and they all report gains from it. For example, RWKV-7 [R1] uses different keys for removal and replacement. Gated DeltaNet-2 [R2] keeps a single key but replaces the scalar βt with a channel-wise erase gate on the key and a channel-wise write gate on the value.
We also trained GDN + Selective RoPE in the setup of Table 4, so that the two models can be compared directly.
Table ngPB.1. GDN + Selective RoPE and Rose-GDN on language modeling and commonsense reasoning (340M parameters). Lower perplexity and higher accuracy (%) are better.
Table ngPB.2. GDN + Selective RoPE and Rose-GDN on Needle-in-a-Haystack (NIAH). Accuracy at context lengths from 512 to 4K tokens.
In language modeling (Table ngPB.1), Rose-GDN has the lower perplexity on both WikiText and LAMBADA, and the eight accuracy benchmarks are split evenly, with each model ahead on four. On NIAH (Table ngPB.2), Rose-GDN matches or exceeds GDN + Selective RoPE in 11 of the 12 settings; especially, Rose-GDN is clearly ahead on S-NIAH-3. Overall, Rose-GDN is on par with GDN + Selective RoPE in language modeling and ahead in retrieval.
References
[R1] RWKV-7 "Goose" with Expressive Dynamic State Evolution. Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, Christian Zhou-Zheng. COLM 2025.
[R2] Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. Ali Hatamizadeh, Yejin Choi, Jan Kautz. NeurIPS 2026.