acceptodds
Under review as a conference paper at ICLR 2027

Rotary-state Gated DeltaNet: Enhancing State Tracking through Rotation

14 views

Abstract

We introduce **Ro**tary-**s**tat**e** **G**ated **D**elta**N**et (Rose-GDN), which augments Gated DeltaNet with input-dependent block-diagonal rotations. This adds rotational dynamics to the generalized-Householder transition without requiring multiple delta updates per token, while preserving fixed-size memory, linear-time recurrence, and exact chunkwise-parallel computation. We show that, under exact arithmetic, a one-layer Rose-GDN can track any finite abelian group. Increasing depth extends this result to all finite solvable groups, and a finite-depth Rose-GDN recognizes all regular languages. Empirically, Rose-GDN shows gains on state-tracking and long-context retrieval tasks while remaining competitive with strong recurrent baselines on language modeling and commonsense reasoning benchmarks. Together, these results suggest that combining input-dependent rotations with delta-rule memory provides a practical way to broaden the capabilities of efficient recurrent models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
state tracking, linear recurrent models, rotary transitions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ngPBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Rotary Stage Gated DeltaNetwork, which applies an input-dependent rotation on the state transition of the traditional Gated DeltaNetwork. The authors show that this enables for improved representational capabilites in terms of state tracking while also theoretically being able to track any regular language. Experiments on state-tracking tasks are used to validate this claim while standard language modeling performance remains comparable to traditional models.

Critical strengths

The method is easy to understand and intuitive/sensible. The chunkwise parallelization in Appendix B appears to be correct and Figure 1 would appear to show that the implementation is fairly efficient.

Critical weaknesses

First, I have some concerns about the approach and its novelty. In praticular, the approach appears to be rather similar to [1], which introduces input-dependent learnable rotary embeddings that generalizes RoPE to arbitrary angles, which the authors show with GLA (a related model to GDN) can yield a complex-valued recurrent state that has similar performance gains as those claimed here.

Next, in terms of the formal language analysis (which I am not a full expert on and thus cannot speak directly on the novelty of the theoretical analysis and proofs), I believe that [2] did show similar proofs regarding negative eigenvalues and the ability to generalize to any regular language, which dampens part of the contribution that the authors appear to emphasize. Furthermore, a lot of focus is also on unsolvable groups such as S5S_5 (which is a focus of [4]), which the work doesn't really delve into and thus the comparison against some related works would appear to be missing. Thus while abelian group tracking could be improved by the approach, whether this translates to other algebraic groups or more real world tasks remains an open question.

On the front of real-world tasks, the language modeling results are also fairly modest compared to GDN, while the NIAH performance while it does show some better decaying behaviour still does not completely resolve the issue. Furthermore, performance at the in-distribution training length (4096 tokens) has some concerns would suggest that the models are possibly not trained enough, or alternatively they have reached capacity and thus larger models/datasets would be necessary to derive any additional conclusions.


[1] Selective Rotary Position Embedding. Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher. ICLR. 2026

[2] Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K.H. Franke, Frank Hutter, Massimiliano Pontil. ICLR. 2025

[3] The Illusion of State in State-Space Models. William Merrill, Jackson Petty, Ashish Sabharwal. ICML. 2024

Questions for authors
  • Results from Table 2 would appear to contradict some details from the Mamba3 work; I am wondering if this may be perhaps that different models did not recieve the same degree of hyper-parameter tuning and this may lead to different results, which could be an issue with transparancy.
  • The reporting of results is slightly inconsistent, with Table 3/Figure 1 reporting Gated DeltaProduct and Transformer while table 4 and 5 do not.
  • In Table 4 and 5, if I understand correctly the Gated DeltaNet is one such that the negative eigenvalue property is not the case. Thus would me more representative to have an actual model with such a property, maybe again providing the Gated DeltaProduct from Table 3/Figure 1 or at the very least providing a DeltaNet with negative eigenvalues which was the model proposed in [1]. Likewise, Mamba3 uses complex states (which is another avenue for richer state tracking since it models a data-dependent rotary embedding like [2], but it would be useful to compare against one that only explictly uses negative eigenvalues without the complex state.
  • Though a comparison isn't directly necessary, [4] appears to propose a similar type of approach, which I believe the authors can add as a small point of discussion.
  • Also not direclty related to a specific weakness, given that Rose-GDN would theoretically still suffer from the same issues of fixed-sized hidden state, could the addition of components such as [5] improve performance, given what may be inconclusive results in Tables 3 and 4?

[4] Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention. Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein. arXiv. 2026

[5] Log Linear Attention. Han Guo, Songlin Yang, Tarushii Goel, Eric P. Xing, Tri Dao, Yoon Kim. ICLR. 2026

9 comments
AuthorsOpenReview ↗

Response to Reviewer ngPB: Part 1/4

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


[W1] First, I have some concerns about the approach and its novelty. In praticular, the approach appears to be rather similar to [1], which introduces input-dependent learnable rotary embeddings that generalizes RoPE to arbitrary angles, which the authors show with GLA (a related model to GDN) can yield a complex-valued recurrent state that has similar performance gains as those claimed here.

Answer: Compared with Selective RoPE, the distinction is that Rose-GDN erases and writes at different addresses, whereas Selective RoPE uses a single address for both. In the cumulative frame of Eq. (5), Rose-GDN is

S~t=αtS~t−1(I−βtatat⊤)+βtvtbt⊤,at=Φt−1⊤kt,bt=Φt⊤kt=Rt⊤at.\tilde S_t=\alpha_t\tilde S_{t-1}(I-\beta_ta_ta_t^\top)+\beta_tv_tb_t^\top,\qquad a_t=\Phi_{t-1}^\top k_t,\quad b_t=\Phi_t^\top k_t=R_t^\top a_t.

It erases at ata_t, the address of ktk_t before the current token's rotation, and writes at btb_t, the address after it. Other recent delta-rule models also treat erasing and writing separately, and they all report gains from it. For example, RWKV-7 [R1] uses different keys for removal and replacement. Gated DeltaNet-2 [R2] keeps a single key but replaces the scalar βt\beta_t with a channel-wise erase gate on the key and a channel-wise write gate on the value.

We also trained GDN + Selective RoPE in the setup of Table 4, so that the two models can be compared directly.

Table ngPB.1. GDN + Selective RoPE and Rose-GDN on language modeling and commonsense reasoning (340M parameters). Lower perplexity and higher accuracy (%) are better.

ModelWiki. ppl ↓LMB. ppl ↓LMB.PIQAHella.Wino.ARC-eARC-cSIQABoolQ
GDN + Selective RoPE29.4734.9432.8765.8938.9349.6457.0727.9037.0860.58
Rose-GDN28.9534.9033.2265.0239.1250.3656.9027.9936.9560.06

Table ngPB.2. GDN + Selective RoPE and Rose-GDN on Needle-in-a-Haystack (NIAH). Accuracy at context lengths from 512 to 4K tokens.

ModelS-NIAH-1 5121K2K4KS-NIAH-2 5121K2K4KS-NIAH-3 5121K2K4K
GDN + Selective RoPE1.0001.0000.9980.9801.0000.9960.9200.3360.4860.2220.0840.024
Rose-GDN1.0000.9980.9980.9821.0001.0000.9280.3460.6860.4240.2600.048

In language modeling (Table ngPB.1), Rose-GDN has the lower perplexity on both WikiText and LAMBADA, and the eight accuracy benchmarks are split evenly, with each model ahead on four. On NIAH (Table ngPB.2), Rose-GDN matches or exceeds GDN + Selective RoPE in 11 of the 12 settings; especially, Rose-GDN is clearly ahead on S-NIAH-3. Overall, Rose-GDN is on par with GDN + Selective RoPE in language modeling and ahead in retrieval.

References

[R1] RWKV-7 "Goose" with Expressive Dynamic State Evolution. Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, Christian Zhou-Zheng. COLM 2025.

[R2] Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. Ali Hatamizadeh, Yejin Choi, Jan Kautz. NeurIPS 2026.

AuthorsOpenReview ↗

Response to Reviewer ngPB: Part 2/4

[W2] Next, in terms of the formal language analysis, I believe that [2] did show similar proofs regarding negative eigenvalues and the ability to generalize to any regular language, which dampens part of the contribution that the authors appear to emphasize.

Answer. We agree that our regular-language result (Thm. 2) reaches the same conclusion as [2, Thm. 4] and follows the same Krohn–Rhodes cascade [R3]. What differs is the class of layers for which the conclusion is proved. In [2, Thm. 4], each layer may use a product of up to NN generalized-Householder (GH) factors per token, where NN is the number of states of the machine. A Rose-GDN layer instead uses a single GH-rotation factor (Eq. (8)), i.e. the product of one GH matrix and one block rotation matrix.

It is worth noting that the proof of [2, Thm. 4] cannot be used directly for Rose-GDN, and we give an example to illustrate this. Consider recognizing binary numbers divisible by 5 while reading the digits from left to right. It suffices to track the remainder of the number read so far, which a digit bb turns from rr into 2r+b mod 52r+b \bmod 5. Reading a 0, for example, leaves the remainder 0 unchanged and sends 1 to 2, 2 to 4, 4 to 3, and 3 to 1. In the proof of [2, Thm. 4], the remainder is stored in one of five coordinates of the memory, and the update for a token is the same whichever coordinate holds it, so the update must rearrange four coordinates at once. With five states, such a rearrangement can take up to four exchanges of two coordinates, and one GH factor makes only one. Their layers make all of them in a single step by multiplying several GH factors per token. A Rose-GDN layer has only one GH factor per token, so this approach does not apply to our proof.

The two constructions also differ in depth. Both proofs first split the machine (N=5N=5 states in the example) into s≤2Ns\le 2^N simpler machines (the Krohn–Rhodes cascade). [2, Thm. 4] uses one layer for each of them, hence ss layers, whereas our proof uses two layers for each plus one rotary layer, hence 2s+12s+1 layers (at most 2N+1+12^{N+1}+1, as stated in Thm. 2). Our construction is therefore deeper, but the total number of GH factors applied per token drops from up to sNsN to 2s+12s+1. We have revised Thm. 2 and added Remark 1 to state this distinction explicitly.

We emphasize that our main theoretical contribution concerns finite group tracking (Thm. 1), which is summarized in Table 1. Specifically, one Rose-GDN layer tracks every finite abelian group (Thm. 1(i)), whereas a single GDN [−1,1][-1,1] layer provably cannot track Z3\mathbb Z_3, or any group containing an element whose order is not a power of two [2, Thm. 2]. The results in Table 3 are consistent with the theory: with one layer, GDN [−1,1][-1,1] reaches 0% exact-sequence accuracy on Z32\mathbb Z_3^2 and Z33\mathbb Z_3^3, while Rose-GDN reaches 98.7–100%.

[W3] Furthermore, a lot of focus is also on unsolvable groups such as S5S_5 (which is a focus of [4]), which the work doesn't really delve into and thus the comparison against some related works would appear to be missing. Thus while abelian group tracking could be improved by the approach, whether this translates to other algebraic groups or more real world tasks remains an open question.

Answer: We agree that the submission focuses on abelian groups and did not study non-solvable groups such as S5S_5 empirically. We have now run this experiment, and we report it below together with what our theory does and does not cover.

On the theory side, solvable non-abelian groups are covered by Thm. 1(ii). The word problem of S5S_5 is a regular language and is therefore covered by Thm. 2.

On the empirical side, we trained five-layer models on S5S_5 state tracking with sequences of length 32 (hidden size 128, 4 heads, 100,000 steps, batch size 512, learning rate 10−410^{-4}) and evaluated them on 1,024 sequences per length (5, 10, 32 and 60). We observe that exact-sequence accuracy is 0% for every model at every length, so we report token accuracy, i.e., the fraction of positions at which the state is predicted correctly.

Table ngPB.3. Token accuracy (%) on S5S_5 state tracking.

ModelLength 5Length 10Length 32Length 60
GDN [0,1][0,1]41.4821.557.914.80
GDN [−1,1][-1,1]41.5021.727.804.92
Gated DeltaProduct$_2$ [−1,1][-1,1]41.3520.967.054.21
Mamba-340.8420.987.114.11
GDN + Selective RoPE41.3521.507.904.85
Rose-GDN41.4321.717.874.95

Overall, none of the models successfully tracks S5S_5 beyond the first few steps in this setting, and Rose-GDN is no exception, so we do not claim an empirical advantage on non-solvable groups.

References

[R3] Algebraic theory of machines. i. prime decomposition theorem for finite semigroups and machines. Kenneth Krohn and John Rhodes. Transactions of the American Mathematical Society, 1965.

AuthorsOpenReview ↗

Response to Reviewer ngPB: Part 3/4

[W4] On the front of real-world tasks, the language modeling results are also fairly modest compared to GDN, while the NIAH performance while it does show some better decaying behaviour still does not completely resolve the issue. Furthermore, performance at the in-distribution training length (4096 tokens) has some concerns would suggest that the models are possibly not trained enough, or alternatively they have reached capacity and thus larger models/datasets would be necessary to derive any additional conclusions.

Answer: We agree with you that the language modeling gains are modest. Our claim is that Rose-GDN improves abelian-group tracking and long-context retrieval over strong recurrent baselines, while remaining competitive on language modeling and commonsense reasoning. We also agree that on the harder NIAH variants, accuracy at 4K is still low for every model. Regarding the 4,096-token setting, we should have been clearer that NIAH is evaluated zero-shot. The models are pretrained on FineWeb-Edu only and never see the retrieval task. 4,096 is the pretraining context length, so the 4K column is within the trained length range but is not in-distribution for the task.

[Q1] Results from Table 2 would appear to contradict some details from the Mamba3 work; I am wondering if this may be perhaps that different models did not recieve the same degree of hyper-parameter tuning and this may lead to different results, which could be an issue with transparancy.

Answer: Thank you for raising this. We clarify that every model in Table 2, including Mamba-3, was trained with the same budget. The discrepancy is due to the training and evaluation protocols, which the submission did not make clear enough. The two protocols are compared below.

Table ngPB.4. Training and evaluation protocols for the formal-language tasks (parity and modular arithmetic) in the Mamba-3 paper and in our Table 2.

Mamba-3 paper (Table 5b, App. D)Our Table 2 (protocol of Grazzi et al., 2025)
Training lengthscurriculum: minimum 33, maximum raised from 40 to 160uniform in 3–40
Test lengths25640–256
Learning rates swept84

We do not question the Mamba-3 results. Instead, our Table 2 measures how well a model generalizes to longer sequences when it is trained only on lengths up to 40. In this setting, independent work also finds Mamba-3 below perfect accuracy on parity and modular arithmetic (without brackets). For example, [R4, Table 1] also trains on lengths up to 40 and tests on 40–256 (their setup differs from ours in batch size, optimizer, and learning rate selection). They report 76.5 ± 7.8% on parity and 46.8 ± 5.5% on modular arithmetic without brackets, i.e., 0.53 and 0.34 in our scaled accuracy. [R5, Table 2] train with a curriculum up to length 60 and find that, on parity, Mamba-3 keeps 90% accuracy only up to length 200–300, while Gated DeltaNet and Gated DeltaProduct reach the maximum tested length of 1000.

[Q2] The reporting of results is slightly inconsistent, with Table 3/Figure 1 reporting Gated DeltaProduct and Transformer while table 4 and 5 do not.

Answer: Thank you. Gated DeltaProduct$_2$ [−1,1][-1,1] and Transformer are now included in Tables 4 and 5.

[Q3] In Table 4 and 5, if I understand correctly the Gated DeltaNet is one such that the negative eigenvalue property is not the case. Thus would me more representative to have an actual model with such a property, maybe again providing the Gated DeltaProduct from Table 3/Figure 1 or at the very least providing a DeltaNet with negative eigenvalues which was the model proposed in [1]. Likewise, Mamba3 uses complex states (which is another avenue for richer state tracking since it models a data-dependent rotary embedding like [2], but it would be useful to compare against one that only explictly uses negative eigenvalues without the complex state.

Answer: Thank you for the suggestion. Your reading is right: the Gated DeltaNet in Tables 4–5 uses βt∈[0,1]\beta_t\in[0,1]. We have included both Gated DeltaProduct$_2$ [−1,1][-1,1] and Gated DeltaNet [−1,1][-1, 1] in Tables 4 and 5.

References

[R4] Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models. Aleksandar Terzic, Francesco S. Carzaniga, Nicolas Menet, Yannick Biehl, Michael Hersche, Thomas Hofmann, Abbas Rahimi. NeurIPS 2026.

[R5] Rethinking State Tracking in Recurrent Models Through Error Control Dynamics. Jiwan Chung, Heechan Choi, Seon Joo Kim. NeurIPS 2026.

Reviewer_GDsJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes RoseGDN, an extension of Gated DeltaNet (GDN) with rotary matrices. On the theoretical front, the authors prove that this extension allows GDNs increased expressivity (similar to DeltaProduct) while retaining parallelizability (chunkwise-parallel form). Specifically, they show that RoseGDN can simulate the word problem over all soluble groups and, with more (large but still fixed) depth, even simulate all finite automata. Empirically, they compare RoseGDN with other LRNN variants on synthetic, diagnostic tasks involving modular arithmetic and abelian group tracking, finding that RoseGDN's performance, close to 100%, is aligned with what the theory suggests. Further, experiments with a large 340M model show that general language modeling and retrieval performance is preserved.

Critical strengths

The proposed extension of GDN is simple (just one input-dependent rotation matrix added) yet powerful, both theoretically and empirically. In hindsight, it's an intuitive extension. That it preserves the chunkwise parallel form (which I admitted didn't carefully verify) is theoretically interesting and practically valuable.

Empirical results, on most of the synthetic problems tried, are quite strong. RoseGDN achieves 100% or near-100%, while all other RNN variants (including Gated DeltaProduct2 and Mamba-3) fail terribly on one or more tasks. Further, general language modeling and retrieval results are reported for fairly large models, with 340M parameters, showing no performance drop on these general abilities.

Critical weaknesses

I see two main weaknesses:

(1) While the initial part of the paper reads well, the paper becomes increasingly harder to read starting on page 4. Part of it is that some terms are used before being defined (e.g., "derived length d" on page 5) and the main intuition isn't articulated in a useful way. E.g., there isn't any discussion of why / how the derived length d relates to the depth of RoseGDN in Theorem 1 and, perhaps even more importantly, why the depth of RoseGDN grows as roughly 2N2^N when simulating a DFA with NN states. The main paper references the appendix for detailed proofs, but some intuition behind the main results would be really helpful (especially as the appendix isn't very easy to read either).

(2) The results assume exact real arithmetic, rather than using some limited form of arithmetic (e.g., finite precision or log precision or even bounded rationals) like many other papers on architecture expressivity do. This is recognized and acknowledged by the authors, and the empirical results bolster the theoretical claims in any case. Neverthess, this is one weakness of the formal results.

On a smaller note, some key statements should be made more precise. E.g., even in the abstract, a sentence says "a finite-depth RoseGDN recognizes all regular languages": the "all" here should be "any", as no single RoseGDN can recognize all regular languages. More precisely, the claims is that for any regular language, there is a RoseGDN that recognizes it. (This applies to some other claims as well.)

Questions for authors
  • Can you provide intuition for why Theorem 1 uses dd layers?

  • Similar (and perhaps more importantly) why does Theorem 2 need roughly 2N2^N layers?

  • Rotation matrices are naturally defined over (irrational) real numbers. Would you expect the formal results to generalize to rational approximations, and even to log-precision approximations assuming the absolute value of all intermediate numbers is bounded by poly(n)poly(n)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer GDsJ: Part 1/2

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


[W1] While the initial part of the paper reads well, the paper becomes increasingly harder to read starting on page 4. Part of it is that some terms are used before being defined (e.g., "derived length d" on page 5) and the main intuition isn't articulated in a useful way. E.g., there isn't any discussion of why / how the derived length d relates to the depth of RoseGDN in Theorem 1 and, perhaps even more importantly, why the depth of RoseGDN grows as roughly when simulating a DFA with states. The main paper references the appendix for detailed proofs, but some intuition behind the main results would be really helpful (especially as the appendix isn't very easy to read either).

[Q1] Can you provide intuition for why Theorem 1 uses dd layers?

Answer: We agree that Sec. 4 was hard to follow. The derived length is now defined before Theorem 1 (Definition 2), and we explain below why dd layers suffice.

For a group of derived length dd, the derived series (Appendix A.1) is a chain e=Gd◃Gd−1◃⋯◃G1◃G0=G\\{e\\} = G_d \triangleleft G_{d-1} \triangleleft \cdots \triangleleft G_1 \triangleleft G_0 = G whose dd quotients Gi/Gi+1G_i/G_{i+1} are abelian, so one Rose-GDN layer can track each of them (Thm. 1(i)). Intuitively, a quotient Gi/Gi+1G_i/G_{i+1} gives a coarse view of GiG_i: it splits GiG_i into "families". Our proof of Thm. 1(ii) uses dd layers, where each layer ii tracks Gi−1/GiG_{i-1}/G_i. Layer 1 tracks G0/G1G_0/G_1, which tells us the family of the state in GG. Layer 2 tracks G1/G2G_1/G_2, which tells us the "sub-family" within the family given by layer 1. This continues until layer dd tracks Gd−1G_{d-1}, which is abelian itself.

As an example, take S3S_3, the six ways to order three cards A, B, C. Its chain is e=G2◃G1=A3◃G0=S3\\{e\\} = G_2 \triangleleft G_1 = A_3 \triangleleft G_0 = S_3, where A3A_3 consists of the three cyclic shifts (by zero, one, or two places). Here G1=A3G_1 = A_3 because every element a−1b−1aba^{-1}b^{-1}ab of S3S_3 is a cyclic shift, and G2={e}G_2 = \{e\} because cyclic shifts commute, so d=2d=2. The quotient S3/A3S_3/A_3 splits the six orderings into two families: the even orderings (ABC, BCA, CAB) and the odd orderings (BAC, CBA, ACB). Within each family we number the orderings 0, 1, 2 in the order listed. Two Rose-GDN layers are sufficient to track S3S_3:

  • Layer 1 tracks S3/A3≅Z2S_3/A_3 \cong \mathbb{Z}_2, the family. It is a mod-2 counter that flips whenever the token swaps two cards.
  • Layer 2 tracks A3≅Z3A_3 \cong \mathbb{Z}_3, the number within the family. It is a mod-3 counter, and the amount each token adds is determined by the token and by the family reported by layer 1. For instance, the token "move the first card to the end" adds 1 in the even family and 2 in the odd family.

[Q2] Similar (and perhaps more importantly) why does Theorem 2 need roughly 2N2^N layers?

Answer: The exponential term 2N2^N comes from the Krohn–Rhodes theorem [R1], every machine with NN states can be rebuilt as a chain of s≤2Ns \le 2^N simpler machines. Our construction for the proof of Thm. 2 uses two layers for each simple machine and one more rotary layer for the whole chain. This gives 2s+1≤2N+1+12s+1\le 2^{N+1}+1 layers. Here, we emphasize that the bound s≤2Ns \le 2^N is a worst case, and many machines need far fewer layers.

References

[R1] Algebraic theory of machines. i. prime decomposition theorem for finite semigroups and machines. Kenneth Krohn and John Rhodes. Transactions of the American Mathematical Society, 1965.

AuthorsOpenReview ↗

Response to Reviewer GDsJ: Part 2/2

[W2] The results assume exact real arithmetic, rather than using some limited form of arithmetic (e.g., finite precision or log precision or even bounded rationals) like many other papers on architecture expressivity do. This is recognized and acknowledged by the authors, and the empirical results bolster the theoretical claims in any case. Neverthess, this is one weakness of the formal results.

[Q3] Rotation matrices are naturally defined over (irrational) real numbers. Would you expect the formal results to generalize to rational approximations, and even to log-precision approximations assuming the absolute value of all intermediate numbers is bounded by poly(n)poly(n)?

Answer. We expect the results to hold with O(log⁡n)O(\log n) bits, with every intermediate quantity bounded by a constant independent of nn, which is stronger than poly(n)\mathrm{poly}(n). We have added this as a proposition with proof in App. F. A sketch of the proof is given below:

  • (a) Every transition in our constructions is non-expansive, ∥Tt∥2≤1\|T_t\|_2\le1. Rotations are orthogonal, I−βkk⊤I-\beta kk^\top with β∈[0,2]\beta\in[0,2] has eigenvalues 11 and 1−β∈[−1,1]1-\beta\in[-1,1], and α∈{0,1}\alpha\in\{0,1\}. Reachable memories form a finite, bounded set, e.g., St=[ρ(gt)⊤∣U]S_t=[\rho(g_t)^\top|U] in Thm. 1.
  • (b) Rounding parameters and states to pp bits adds an error of at most C2−pC2^{-p} per step. Non-expansive propagation does not amplify these errors, so they add up: ∥S^t−St∥≤C′t 2−p\|\hat S_t-S_t\|\le C' t\,2^{-p} for t≤nt\le n.
  • (c) In exact arithmetic, each layer's output is always one of a fixed, finite set of distinct codes, so its decoder still returns the correct discrete answer for any output error below a fixed margin δ>0\delta>0 that depends on the group or automaton but not on nn.
  • (d) Each layer receives the decoded, discrete outputs of the layers below, so the per-layer guarantee composes by induction over depth.

Hence the output is decoded correctly whenever Cnˊ2−p<δC\'n2^{-p}<\delta, so p=log⁡2n+O(1)p=\log_2 n+O(1) bits suffice. Under the finite-datatype notion of precision of Grazzi et al. [R2, App. A.4], our constructions hold as stated, since their reachable memories form a finite set; this is the sense in which [R2, Thm. 3] and [R3, Thm. 1] track Zm\mathbb Z_m with rotations in one layer. With rounding after every operation, rotations by 2π/m2\pi/m are not exactly representable for m∉1,2,4m\notin\\{1,2,4\\}, so our analysis, like that of [R4], needs O(log⁡n)O(\log n) bits.

[W3] On a smaller note, some key statements should be made more precise. E.g., even in the abstract, a sentence says "a finite-depth RoseGDN recognizes all regular languages": the "all" here should be "any", as no single RoseGDN can recognize all regular languages. More precisely, the claims is that for any regular language, there is a RoseGDN that recognizes it. (This applies to some other claims as well.)

Answer: Thank you for the suggestion. We have corrected them in the revised version of our paper.

References

[R2] Unlocking state-tracking in linear RNNs through negative eigenvalues. Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K.H. Franke, Frank Hutter, and Massimiliano Pontil. ICLR, 2025.

[R3] The expressive limits of diagonal SSMs for state-tracking. Mehran Shakerinava, Behnoush Khavari, Siamak Ravanbakhsh, and Sarath Chandar. ICLR, 2026.

[R4] Bridging expressivity and scalability with adaptive unitary SSMs. Arjun Karuvally, Franz Nowak, T. Anderson Keller, Carmen Amo Alonso, Terrence J. Sejnowski, and Hava T. Siegelmann. NeurIPS, 2025.


We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we could get your further feedback at your earliest convenience.


AuthorsOpenReview ↗

Any Questions from Reviewer GDsJ on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

Reviewer_hTJmOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propse a variant of gated delta net which they term Rose-GDN, where the key dimensions are grouped into pairs. They introduce a rotation R to the GDN update rule which turns each of the pairs by an angle, where the angles are computed from the current token.

Critical strengths

Its a clean design with rose-gdn using one eraser plus a simple plane rotation per token for which they provide a exact fast training form. Generating the text is as fast as GDN, reading costs 1.5x The results are a clear win on the dial-counting tasks, where on Z3,2 and Z3,3 Rose-GDN gets about 99% of sequences fully right, for which GDN and DeltaProduct break down. Also retrieval seems to be better which is a strong evidence that the rotation helps.

Critical weaknesses

As far as I can see the closest existing model is missing as a baseline, and it seems to be very similar. Selective Rope (Movahedi et al) rotates the key and query by an angle and then runs ordinary GDN. This paper cites it but only as a form of positional encoding. So the difference is that Rose-GDN rotates the full memory while Selective Rope turns key and query, so the erase address is shifted by one steps rotation.

The Mamba3 baseline looks wrong, where in Table 2 Mamba-3 scores around 0.3 on parity while even the older Mamba papers report 1.0.

Questions for authors

Please compare against GDN + Selective Rope and report results together with rose-gdn

Explain more clearly why mamba-3 fails parity while older results show that Mamba can solve it.

6 comments
AuthorsOpenReview ↗

Response to Reviewer hTJm: Part 1/2

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


[W1] As far as I can see the closest existing model is missing as a baseline, and it seems to be very similar. Selective Rope (Movahedi et al) rotates the key and query by an angle and then runs ordinary GDN. This paper cites it but only as a form of positional encoding. So the difference is that Rose-GDN rotates the full memory while Selective Rope turns key and query, so the erase address is shifted by one steps rotation.

[Q1] Please compare against GDN + Selective Rope and report results together with rose-gdn

Answer: We agree that Selective RoPE [1] is the closest prior work. We have revised Sec. 3 and Sec. 6 accordingly and added GDN + Selective RoPE as a baseline.

Rose-GDN erases and writes at different addresses, whereas Selective RoPE uses a single address for both. In the cumulative frame of Eq. (5), Rose-GDN is

S~t=αtS~t−1(I−βtatat⊤)+βtvtbt⊤,at=Φt−1⊤kt,bt=Φt⊤kt=Rt⊤at.\tilde S_t=\alpha_t\tilde S_{t-1}(I-\beta_ta_ta_t^\top)+\beta_tv_tb_t^\top,\qquad a_t=\Phi_{t-1}^\top k_t,\quad b_t=\Phi_t^\top k_t=R_t^\top a_t.

It erases at ata_t, the address of ktk_t before the current token's rotation, and writes at btb_t, the address after it, so the two addresses differ by exactly one step's rotation. Selective RoPE erases and writes at btb_t. Other recent delta-rule models also treat erasing and writing separately, and they report gains from it. For example, RWKV-7 [R1] uses different keys for removal and replacement. Gated DeltaNet-2 [R2] keeps a single key but replaces the scalar βt\beta_t with a channel-wise erase gate on the key and a channel-wise write gate on the value.

As requested, we trained GDN + Selective RoPE in the setup of Table 4 and report it together with Rose-GDN.

Table hTJm.1. GDN + Selective RoPE and Rose-GDN on language modeling and commonsense reasoning (340M parameters). Lower perplexity and higher accuracy (%) are better.

ModelWiki. ppl ↓LMB. ppl ↓LMB.PIQAHella.Wino.ARC-eARC-cSIQABoolQ
GDN + Selective RoPE29.4734.9432.8765.8938.9349.6457.0727.9037.0860.58
Rose-GDN28.9534.9033.2265.0239.1250.3656.9027.9936.9560.06

Table hTJm.2. GDN + Selective RoPE and Rose-GDN on Needle-in-a-Haystack (NIAH). Accuracy at context lengths from 512 to 4K tokens.

ModelS-NIAH-1 5121K2K4KS-NIAH-2 5121K2K4KS-NIAH-3 5121K2K4K
GDN + Selective RoPE1.0001.0000.9980.9801.0000.9960.9200.3360.4860.2220.0840.024
Rose-GDN1.0000.9980.9980.9821.0001.0000.9280.3460.6860.4240.2600.048

In language modeling (Table ngPB.1), Rose-GDN has the lower perplexity on both WikiText and LAMBADA, and the eight accuracy benchmarks are split evenly, with each model ahead on four. On NIAH (Table ngPB.2), Rose-GDN matches or exceeds GDN + Selective RoPE in 11 of the 12 settings; especially, Rose-GDN is clearly ahead on S-NIAH-3. Overall, Rose-GDN is on par with GDN + Selective RoPE in language modeling and ahead in retrieval.

References

[R1] RWKV-7 "Goose" with Expressive Dynamic State Evolution. Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, Christian Zhou-Zheng. COLM 2025.

[R2] Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. Ali Hatamizadeh, Yejin Choi, Jan Kautz. NeurIPS 2026.

AuthorsOpenReview ↗

Response to Reviewer hTJm: Part 2/2

[W2] The Mamba3 baseline looks wrong, where in Table 2 Mamba-3 scores around 0.3 on parity while even the older Mamba papers report 1.0.

[Q2] Explain more clearly why mamba-3 fails parity while older results show that Mamba can solve it.

Answer: Thank you for raising this. The discrepancy is due to the different training and evaluation protocols, which the submission did not make clear enough. The two protocols are compared below.

Table hTJm.3. Training and evaluation protocols for the formal-language tasks (parity and modular arithmetic) in the Mamba-3 paper and in our Table 2.

Mamba-3 paper (Table 5b, App. D)Our Table 2 (protocol of Grazzi et al., 2025)
Training lengthscurriculum: minimum 3, maximum raised from 40 to 160uniform in 3–40
Test lengths25640–256
Learning rates swept84

We do not question the Mamba-3 original result. Instead, our Table 2 measures how well a model generalizes to longer sequences when it is trained only on lengths up to 40. In this setting, independent work also finds Mamba-3 below perfect accuracy on parity. For instance, [R3, Table 1] also train on lengths up to 40, test on 40–256 (only differs in batch size, optimizer, and learning-rate selection compared to our protocol), and report 76.5 ± 7.8%, i.e., 0.53 in our scaled accuracy. [R4, Table 2] train with a curriculum up to length 60 and find that Mamba-3 keeps 90% accuracy only up to length 200–300, while Gated DeltaNet and Gated DeltaProduct reach the maximum tested length of 1000.

References

[R3] Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models. Aleksandar Terzic, Francesco S. Carzaniga, Nicolas Menet, Yannick Biehl, Michael Hersche, Thomas Hofmann, Abbas Rahimi. NeurIPS 2026.

[R4] Rethinking State Tracking in Recurrent Models Through Error Control Dynamics. Jiwan Chung, Heechan Choi, Seon Joo Kim. NeurIPS 2026.


We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we could get your further feedback at your earliest convenience.

AuthorsOpenReview ↗

Any Questions from Reviewer hTJm on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Global Response: Part 1/2

Dear AC and Reviewers,

Thank you for your thoughtful reviews and constructive comments, which have helped us improve the paper. We are encouraged by the endorsements that: 1) our design is clean, simple and intuitive (all reviewers), 2) it preserves the chunkwise-parallel form of Gated DeltaNet (GDN) (all reviewers) and runs efficiently (reviewers hTJm, ngPB), 3) the state-tracking results are strong, with Rose-GDN at or near 100% on abelian-group tracking, where other recurrent models fail (reviewers hTJm, GDsJ), and 4) retrieval improves (reviewer hTJm) with no loss in general language modeling at 340M parameters (reviewer GDsJ).

One common question from the reviewers is how Rose-GDN differs from GDN with Selective RoPE [R1] (reviewers hTJm, ngPB). Another is why Mamba-3 scores low on parity in Table 2 of the main text (reviewers hTJm, ngPB). A third concerns the depth bounds, novelty and precision assumption of our theory (reviewers GDsJ, ngPB). We address these questions here.

Relation to Selective RoPE. We agree that GDN with Selective RoPE [R1] is a close baseline, and that the submission did not compare against it empirically. The two models differ in where the memory is erased and written. Selective RoPE rotates the keys and queries and then applies the ordinary delta rule, so it erases and writes at the same rotated key. Rose-GDN rotates the memory itself: in the cumulative frame of Eq. (5), it erases at the address of the key before the current token's rotation and writes at the address after it. Sections 3.1 and 6 now state this, and the empirical comparison is in Global Response (2/2).

Mamba-3 results in Table 2 of the main text. Every model in this table, including Mamba-3, was trained with the same budget. The gap to the Mamba-3 paper comes from the training and evaluation protocol. The original Mamba-3 setup trains with a length curriculum whose maximum grows from 40 to 160, and tests at length 256. We follow the protocol of [R2]: training lengths are sampled uniformly from 3 to 40, and test lengths range from 40 to 256. Our table thus measures generalization from short training sequences to longer ones. We do not question the Mamba-3 result. With the same training and test lengths as ours, Flash PD-SSM [R3, Table 1] also reports Mamba-3 below perfect accuracy on parity (76.5 ± 7.8%, which is 0.53 on our scale).

Theoretical results. Reviewer GDsJ asked why Theorem 1 uses dd layers and Theorem 2 roughly 2N2^N. We have defined the derived length before Theorem 1 (Definition 2). A solvable group of derived length dd has a derived series with dd abelian quotients. One Rose-GDN layer tracks an abelian group, so our construction uses one layer per quotient. In Theorem 2, the exponential term comes from the Krohn–Rhodes theorem [R4], which rebuilds an automaton with NN states as a chain of s≤2Ns \le 2^N simpler automata. We use two layers for each and one more rotary layer, so the bound is now stated as L≤2s+1L \le 2s+1. The value 2N2^N is a worst case.

Reviewer ngPB noted that [R2] already proves a similar result for regular languages. We agree that Theorem 2 reaches the same conclusion as [R2, Theorem 4] with the same cascade. However, a layer in [R2, Theorem 4] may multiply up to NN generalized Householder (GH) factors per token, whereas a Rose-GDN layer has one GH factor and one block rotation, so that proof cannot be reused directly. Our construction is deeper, with 2s+12s+1 layers instead of ss, but it applies 2s+12s+1 GH factors per token in total instead of up to sNsN. We added Remark 1 to state this. Our main theoretical result is Theorem 1: one Rose-GDN layer tracks any finite abelian group, while one GDN [−1,1][-1,1] layer cannot count modulo mm unless mm is a power of two [R2, Theorem 2]. Table 3 is consistent with this: with one layer, GDN [−1,1][-1,1] has 0% exact-sequence accuracy on both groups and Rose-GDN has 98.7–100%.

Reviewer GDsJ also asked whether the results need exact real arithmetic. We added Proposition 1 (Appendix F, summarized in Remark 2), which shows that, with rounding after every operation, log⁡2n+O(1)\log_2 n + O(1) bits per number suffice for inputs of length nn.

References

[R1] Selective Rotary Position Embedding. Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher. ICLR 2026.

[R2] Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K.H. Franke, Frank Hutter, Massimiliano Pontil. ICLR 2025.

[R3] Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models. Aleksandar Terzic, Francesco S. Carzaniga, Nicolas Menet, Yannick Biehl, Michael Hersche, Thomas Hofmann, Abbas Rahimi. NeurIPS 2026.

[R4] Algebraic Theory of Machines. I. Prime Decomposition Theorem for Finite Semigroups and Machines. Kenneth Krohn, John Rhodes. Transactions of the American Mathematical Society, 1965.

AuthorsOpenReview ↗

Global Response: Part 2/2

For the comparison with Selective RoPE, we present in the following tables the results of GDN + Selective RoPE [R1] and Rose-GDN, both trained in the same 340M-parameter setup. The same results are in Tables 4 and 5 of the revised paper.

Table 1: Language modeling perplexity and commonsense reasoning accuracy (%) of GDN + Selective RoPE [R1] and Rose-GDN

ModelWiki. ppl (↓)LMB. ppl (↓)LMB.PIQAHella.Wino.ARC-eARC-cSIQABoolQ
GDN + Selective RoPE [R1]29.4734.9432.8765.8938.9349.6457.0727.9038.0860.58
Rose-GDN (Ours)28.9534.9033.2265.0239.1250.3656.9027.9936.9560.06

We see in Table 1 that the two models are close on language modeling. Rose-GDN has lower perplexity on both WikiText and LAMBADA, and each model is ahead on four of the eight accuracy benchmarks.

Table 2: Needle-in-a-Haystack (NIAH) accuracy of GDN + Selective RoPE [R1] and Rose-GDN at context lengths from 512 to 4K tokens

ModelS-NIAH-1 5121K2K4KS-NIAH-2 5121K2K4KS-NIAH-3 5121K2K4K
GDN + Selective RoPE [R1]1.0001.0000.9980.9801.0000.9960.9200.3360.4860.2220.0840.024
Rose-GDN (Ours)1.0000.9980.9980.9821.0001.0000.9280.3460.6860.4240.2600.048

We see in Table 2 that Rose-GDN matches or exceeds GDN + Selective RoPE in 11 of the 12 settings, with the largest gap on S-NIAH-3.

Additional baselines and experiments. Following the suggestions of reviewer ngPB, Tables 4 and 5 of the revised paper now also include Gated DeltaNet [−1,1][-1,1], Gated DeltaProduct$_2$ [−1,1][-1,1], and a Transformer. All average accuracies on the commonsense reasoning tasks lie within about one point of each other; Rose-GDN has the lowest WikiText perplexity, and it has the highest accuracy among the recurrent models at every length of S-NIAH-3.

Two further experiments are reported in our response to reviewer ngPB. On S5S_5, a non-solvable group, none of the six models we tested, including Rose-GDN, tracks the state successfully, so we do not claim an empirical advantage on non-solvable groups. Combining Rose-GDN with log-linear attention is generally better, with lower perplexity on both WikiText and LAMBADA (34.90 to 30.22) and higher accuracy on five of the eight benchmarks.

References

[R1] Selective Rotary Position Embedding. Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher. ICLR 2026.


We hope that we have cleared your concerns about our work. We are glad to answer any further questions, and we would appreciate it if we could get your further feedback at your earliest convenience.

AuthorsOpenReview ↗

Summary of Revisions

Dear AC and Reviewers,

We thank you for your comments and suggestions. Following them, we summarize here the main changes in the revised paper, which are marked in blue:

  1. We have added GDN + Selective RoPE [R1] as a baseline (Tables 4 and 5, Sections 5.2.1 and 5.2.2). We have also stated in Sections 3.1 and 6 how it differs from Rose-GDN: it erases and writes at the same rotated key, whereas in Rose-GDN the two addresses differ by the current rotation.
  2. We have added Gated DeltaNet [−1,1][-1,1], Gated DeltaProduct$_2$ [−1,1][-1,1] and a Transformer to Tables 4 and 5. The original Gated DeltaNet row is now labeled [0,1][0,1]. Bold marks the best value among the recurrent models, and the Transformer is given as a non-recurrent reference.
  3. We have stated in the caption of Table 5 that NIAH is evaluated zero-shot, and in Section 7 that the gains in group tracking and retrieval are over recurrent baselines.
  4. We have defined the derived length before Theorem 1 (Definition 2, Section 4.1), and recalled that one GDN [−1,1][-1,1] layer cannot count modulo mm unless mm is a power of two [R2].
  5. We have restated the layer bound of Theorem 2 as L≤2s+1L \le 2s+1 with s≤2Ns \le 2^N, followed by a short explanation of where each term comes from (Section 4.2).
  6. We have added Remark 1, which compares our construction with [R2, Theorem 4] in depth and in the number of GH factors per token.
  7. We have added Proposition 1 with its proof (Appendix F, summarized in Remark 2), which shows that Theorems 1 and 2 hold with log⁡2n+O(1)\log_2 n + O(1) bits per number.
  8. We have added to Section 6 a discussion of RWKV-7 [R3], which also uses different keys for removal and replacement, and of Complex KDA [R4].
  9. We have replaced "all" with "any" in the abstract, Section 1, Theorem 2, and Section 6, where the claim is that a suitable Rose-GDN exists for each group or language.

References

[R1] Selective Rotary Position Embedding. Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher. ICLR 2026.

[R2] Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K.H. Franke, Frank Hutter, Massimiliano Pontil. ICLR 2025.

[R3] RWKV-7 "Goose" with Expressive Dynamic State Evolution. Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, Christian Zhou-Zheng. COLM 2025.

[R4] Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention. Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein. arXiv 2026.

AuthorsOpenReview ↗

Additional Experimental Results

Dear AC and Reviewers,

We would like to thank all reviewers again for their thoughtful reviews and valuable feedback. We have obtained additional state-tracking results for GDN + Selective RoPE [R1] and on a non-abelian group (dihedral group D7D_7). We summarize these results below:

(1) Following Reviewer hTJm's suggestion, we ran GDN + Selective RoPE on the formal-language and abelian-group tracking tasks, with the same protocol as the other models.

We report our results below in Tables 1 to 3. We include the results of the other models as well for convenience. The same results can be found in Tables 2 and 3 of our revised manuscript. Rose-GDN matches GDN + Selective RoPE on parity and outperforms it on modular arithmetic and on abelian-group tracking at longer sequence lengths.

Table 1. Formal-language evaluation. We report scaled accuracy, where 1 is perfect and 0 is chance. A higher value reflects better performance.

Model \ TaskParityMod. arithmetic (no brackets)Mod. arithmetic (brackets)
Transformer0.1140.1480.138
Mamba-3 (SISO)0.3300.2450.153
Gated DeltaProduct$_2$ [−1,1][-1,1]0.9160.9990.258
Gated DeltaNet [−1,1][-1,1]0.9921.0000.221
GDN [−1,1][-1,1] + Selective RoPE0.9970.7540.218
Rose-GDN [−1,1][-1,1]1.0000.9980.249

Table 2. Tracking of the abelian group Z32\mathbb{Z}_3^2. We report exact-sequence accuracy / state accuracy (%). Training and evaluation use the same sequence length TT.

Model \ LengthT=20T=20T=40T=40T=60T=60T=80T=80
Transformer88.3 / 99.175.3 / 98.160.8 / 96.651.7 / 95.1
Mamba-3 (SISO)99.0 / 99.90.00 / 37.10.00 / 35.90.00 / 34.6
GDN [−1,1][-1,1]0.00 / 46.080.00 / 31.040.00 / 19.430.00 / 22.81
Gated DeltaProduct$_2$ [−1,1][-1,1]100.00 / 100.0099.90 / 99.99100.00 / 100.000.00 / 38.85
GDN [−1,1][-1,1] + Selective RoPE100.00 / 100.00100.00 / 100.000.29 / 60.260.00 / 53.13
Rose-GDN [−1,1][-1,1]100.00 / 100.00100.00 / 100.00100.00 / 100.0098.93 / 99.97

Table 3. Tracking of the abelian group Z33\mathbb{Z}_3^3, in the same format as Table 2.

Model \ LengthT=20T=20T=40T=40T=60T=60T=80T=80
Transformer90.8 / 98.90.00 / 51.10.00 / 33.10.00 / 38.1
Mamba-3 (SISO)100.0 / 100.00.00 / 36.80.00 / 13.70.00 / 12.2
GDN [−1,1][-1,1]0.00 / 25.420.00 / 14.780.00 / 11.160.00 / 9.23
Gated DeltaProduct$_2$ [−1,1][-1,1]0.00 / 55.410.00 / 39.762.83 / 73.530.00 / 12.51
GDN [−1,1][-1,1] + Selective RoPE100.00 / 100.00100.00 / 100.002.54 / 70.430.10 / 68.72
Rose-GDN [−1,1][-1,1]100.00 / 100.00100.00 / 100.00100.00 / 100.0098.73 / 99.97

(2) From Reviewer ngPB's feedback, it remained open whether our state-tracking results extend beyond abelian groups. Following the reviewer's suggestion, we have also run state tracking on the dihedral group D7D_7, which is non-abelian, using one-layer models trained on sequence lengths up to 32. We report our results in Table 4 below. Rose-GDN learns the task and extrapolates best at the longest lengths. We also include these results in Figure 1 in Section 5.1.3 of our revision.

Table 4. Final-state accuracy (%) when tracking the non-abelian group D7D_7 at different sequence lengths. A higher value reflects better performance. The highest value in each column is in bold and the second highest is underlined.

Model \ Length48163264128256512
Transformer98.0016.6013.286.937.427.477.967.23
Mamba-387.8417.9715.3314.214.392.697.768.35
GDN [−1,1][-1,1]98.29‾\underline{98.29}62.30‾\underline{62.30}23.63‾\underline{23.63}16.1610.066.697.287.57
Gated DeltaProduct$_2$ [−1,1][-1,1]100.00100.00100.00100.0099.9595.8055.47‾\underline{55.47}19.04
GDN [−1,1][-1,1] + Selective RoPE100.00100.00100.0099.85‾\underline{99.85}98.2484.8155.2224.17‾\underline{24.17}
Rose-GDN [−1,1][-1,1]100.00100.00100.00100.0099.22‾\underline{99.22}94.14‾\underline{94.14}67.9739.01

References

[R1] Selective Rotary Position Embedding. Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher. ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.