acceptodds
Under review as a conference paper at ICLR 2027

Rotor Recurrences: What Compact Recurrent States Can Predict

Abstract

Compact recurrent states can preserve order-sensitive computations, but what a state can store need not be what a continuous decoder can recover. For pure rotor transitions associated with rotations in three or four dimensions, we prove a sharp prediction limit for any fixed finite number of heads. Under i.i.d. inputs from a fixed distribution whose target labels generate the group, no fixed continuous strictly positive decoder can asymptotically achieve expected prefix-averaged log loss below (\log 60) when predicting permutations of five objects. The bound is sharp: a parity-tracking recurrence approaches this floor with a decoder that assigns almost all probability mass uniformly within the correct parity class. The same rotor family nevertheless permits arbitrarily small loss when tracking even permutations alone. A general compact-group result characterizes the hidden ambiguity underlying these limits. We then study related affine rotor models with additive writes and decay, which lie outside the theorem. A single rotor layer learns the even-permutation task and generalizes to sequences four times the training length. Across implemented parameterizations of the same ideal rotation family, median sustained solve times among successful runs span roughly a factor of five. These experiments measure task learning without identifying the learned internal representation. Together, the results distinguish what recurrent dynamics can represent, what continuous decoding can recover, and what gradient training discovers.

Primary area:
learning theory
Keywords:
recurrent neural networks, state-space models, recurrent state, group theory, non-commutative dynamics, Clifford algebra, rotor networks, expressivity, predictive limits, learnability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_74SaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes between "what a compact recurrent state can store" and "what a continuous decoder can retrieve from the state," examining the limits of predictability for pure rotor transitions (St=RtSt−1S_t = R_t S_{t-1}, where Rt∈Spin(3)/Spin(4)R_t \in \text{Spin}(3)/\text{Spin}(4)) acting on permutation groups. The core tool is a hidden target fiber JJ: given a compact transition group GG, a finite target group HH, and i.i.d. inputs, "target elements indistinguishable under the same transition" are collapsed into a normal subgroup JJ. The authors prove (Proposition 1) that when observing the full transition, the infimum of the optimal asymptotic prefix-averaged log loss—using a fixed, continuous, strictly positive decoder—is exactly log⁡∣J∣\log|J|; this value remains a lower bound when observing only the state.

Applying this general principle to rotors: in the case of S5S_5, the authors utilize a group-theoretic fact (Lemma 1: no closed subgroup of SU(2)m\text{SU}(2)^m admits a continuous surjective homomorphism to S5S_5) to establish that J⊇A5J \supseteq A_5, implying ∣J∣≥60|J| \ge 60. This yields Theorem 1: for any finite number of Spin(3)/Spin(4)\text{Spin}(3)/\text{Spin}(4) heads, the lower bound for the asymptotic average log loss on S5S_5 is log⁡60\log 60, a bound achieved by a parity tracker (Corollary 1). In contrast, A5A_5 allows for arbitrarily low loss via a binary icosahedral subgroup. The paper also presents an algebraic decomposition for exact tracking (Theorems 2/3, Proposition 3), auxiliary analysis of reflection length, and learning experiments using an affine rotor model (incorporating additive writes and decay, placing it explicitly outside the scope of the theorems): A5A_5 learned the task and generalized to sequences four times longer, whereas S4S_4 and S5S_5 yielded only partial or parity-based solutions; overall, rotors proved significantly weaker than matrix-state baselines in language modeling tasks.

Critical strengths

Theoretically, the most elegant aspect is the clean contrast between S5 and A5: within the same rotor family, one allows for arbitrarily low loss (A5, since 2I ↠ A5), while the other stalls at log 60 (S5, because states cannot distinguish the A5-coset). The authors break down the concepts of "expressivity," "predictability for sequential decoding," and "learnability via optimization" into the three properties listed in Table 1; this framing is valuable in itself and avoids the common pitfall in state-tracking literature of conflating representability with learnability.

The proof structure in Appendix A is clear: it begins by proving Lemma 1 (using induction on m, the swapping of coordinate projection kernels, the fact that the only normal subgroups of S5 are {1}, A5, and S5, and the condition C_{S5}(A5)=1); it then employs Kawada–Itô Cesàro convergence to show the joint walk converges to the Haar measure; finally, it uses Jensen's inequality to establish a lower bound of log|J| and demonstrates achievability via a coset-uniform decoder. Corollary 2 extends the conclusion to fixed suffixes, Theorems 2 and 3 address exact tracking, and Remark 1 provides a counterexample involving a dense free subgroup to explain why the uniform separation condition cannot be dropped—judging by the provided proofs, this logical chain is both self-consistent and well-organized. Furthermore, the authors’ restraint regarding the scope is commendable: they repeatedly clarify that affine models fall outside the scope of the theorems and characterize the S5 plateau merely as "parallel observation," rather than claiming it represents an experimental realization of the theorem.

Critical weaknesses

(1) There is a disconnect between the strongest theoretical result and the full set of experiments—a fact the authors themselves acknowledge. Theorem 1 addresses "pure transitions" (involving neither writes nor decays), whereas all learning experiments utilize the affine model (Eq. 2); the authors repeatedly note in Table 2, Table 5, and Appendix B.6 that these cases fall "outside the theorem." Consequently, the strongest formal claim is not instantiated by any experiment, nor do the experimental failures serve to validate the theorem. In Table 5, the S5 loss values ​​that plateaued are 3.96, 3.98, 4.03, and 4.03—all below log⁡60≈4.094\log 60 \approx 4.094 (compared to 4.71 for the fixed-parity control, and baselines of 4.79 for random and 4.09 for sign-only). This implies that the trained affine models were not only unconstrained by the bound but also outperformed the "sign-only" baseline. While the authors correctly explain that affine models are not subject to this bound, this also demonstrates that the bound imposes no direct constraint on the affine models tested in the paper. It is difficult for a reviewer to view this as a constraint capable of guiding practical application.

(2) The incremental contribution of Theorem 1 relative to Zhang & Li (2026) needs to be articulated more clearly. The paper lists Zhang & Li’s asymptotic accuracy ceiling of 1/∣[G,G]∣1/|[G,G]| in Table 7 (noting its implication of a log-loss bound) and states in the "Point of departure" section that "our theorem does not assume order-blindness." The issue lies in the specifics of this delta: for S5, [S5,S5]=A5[S5,S5]=A5 and ∣A5∣=60|A5|=60, resulting in an accuracy ceiling of 1/601/60, which exactly matches the accuracy of the parity-tracking method in this paper. The authors should explicitly clarify whether Zhang & Li’s log-loss bound also yields log⁡60\log 60 and how the assumptions of the two approaches compare, rather than leaving this for the reader to infer. If the numerical limits are the same, does the primary advantage of Theorem 1 over Zhang & Li lie in its scope of applicability (moving from "order-blind" to "order-aware/state-dependent decoder") or in the strength of its conclusion? While this could indeed be a genuine contribution—the group-theoretic steps in Lemma 1 are non-trivial—the authors need to explicitly explain the role the "order-blind" assumption played in Zhang & Li’s proof, what new tools were required to remove it, and why a broader scope of applicability matters to the community.

(3) The empirical evidence is weak, and in many instances, the results cannot be clearly attributed to specific factors. Achieving success with A5 required 136–192 heads and 1.9–2.4M parameters (Table 3)—an extreme number of heads; S4 stalled at 0.316; A4 solved the task only within the training length but failed to extrapolate; the rows for S5 were largely based on single seeds, and the "learned-parity" row in Table 5 actually represents the best of five runs (Appendix B.7 clarifies that the other four failed to reach a plateau); regarding language modeling, the standalone rotor scored 43.83 versus baselines of 30.22–33.85 (Table 14). More critically, the source of the gain in the hybrid model was not isolated: in Table 15, Rotor+DeltaNet (50.88) performed almost identically to constant rotation (50.81), meaning the current evidence cannot attribute the gain to token-dependent rotation; the paired identity/constant interventions in Table 16 (using a different normalization and evaluation setup, which the authors state is not comparable to the 50M parameter study) also show a difference of only about 0.225% in perplexity between identity and learned constant rotations. The paper itself acknowledges that causal variables such as layer allocation, width, and generic-layer control were not isolated. A more prudent conclusion is that while experiments show gains for certain hybrid configurations, the underlying mechanism remains unidentified; this is insufficient to support strong claims regarding "learned transport."

(4) The headline empirical finding that "parameterization affects solve time by up to 5×" is confounded. Appendix B.4 explicitly acknowledges that changing the rotor parameterization alters the projection structure (four outputs plus bias vs. three outputs without bias), the parameter count (2.10M vs. 2.04M), and random number consumption; consequently, using the same seed does not result in identical initializations. In other words, the 5× disparity reported in Table 4 conflates three distinct factors: parameterization, projection, and initialization. The authors also admit in Section B.4.1 that neither probe accounts for this gap (the single-rotor fit yields a median of 64/55/66, while the ordered product shows a difference of only about 1.5×). Given this, the 5× figure should not be highlighted as a key selling point in the main text.

Questions for authors
  • Does the accuracy ceiling of 1/∣[G,G]∣1/|[G,G]| in Zhang & Li (2026) already encompass the log⁡60\log 60 result for S5? If so, please explicitly state in the main text exactly what additional content Theorem 1 provides once the "order-blindness" assumption is removed (e.g., a stronger conclusion, broader applicability, or a different proof path).
  • Can at least a "partial bridge" be established between Theorem 1 and the affine experiments? For instance, by proving a corresponding lower bound for a specific class of affine models (with finite length and decay), even if only as a finite-length approximation.
  • Regarding the solve-time gap in Table 4, could a paired experiment be conducted? Specifically, by fixing the projection structure, parameter count, and initialization RNG seed while varying only the parameterization, does the 5×5\times difference persist?
  • For the S5 row in Table 5, the loss is already below log⁡60\log 60; how do the authors justify this without leading readers to believe that Theorem 1 imposes no constraints on trainable models? Could you report whether the states of these affine models approximate a parity projection and describe their behavior at longer sequence lengths?
  • Model A5 requires 136–192 heads; is this architecturally acceptable? Could you provide a scaling analysis relating the number of heads to the minimum success rate?
1 comment
AuthorsOpenReview ↗

Novelty, experimental connection and controls

Thank you for the review. We have updated the manuscript; changes are marked in blue. All references below point to the revised version.

The main revision adds pure-rotor experiments and matched controls, and removes claims that the evidence does not isolate. The results now distinguish the theoretical limit from finite-length learning and readout behavior.

  • The S5 numerical floor is the same; the new scope covers order-sensitive rotor states. We now make this explicit in Sections 3.1--3.3 and 5. Zhang and Li's optimal order-blind accuracy ceiling implies the corresponding log⁡60\log 60 log-loss floor by Jensen's inequality; this implication is our derivation. Their result assumes uniform full-group inputs and order-blind prediction. Ours allows order-sensitive transitions under an i.i.d. generating law and fixed continuous strictly positive decoding. Proposition 1 gives the exact full-transition loss characterization and a lower bound for state observations; Lemma 1 supplies the finite-head rotor obstruction. The rotor and abelian floors differ on A5 and S5 x A5.

  • We now test pure-rotor dynamics directly, while retaining the limits of that connection. Section 4.1 and Appendix C add runs with no writes or decay, evaluated to 10,000 tokens with the decoder fixed after training. S5 loss is near the floor within the training horizon, then parity eventually fails (Figure 4). This does not establish asymptotic convergence. The affine models remain outside the theorem, so their below-floor finite-length losses do not contradict it. We provide no finite-length bound for general affine models.

  • The new decay audit explains why scale cancellation fails in the implemented write-free model. With accumulated positive decay cc and RMSNorm parameter ε>0\varepsilon>0,

    = \frac{z}{\sqrt{\operatorname{mean}(z^2)+\frac{\varepsilon}{c^2}}}.$$ Thus the normalized readout still depends on decay. Early-position losses explain the below-floor prefix average of the evaluated write-free S5 checkpoint; this does not explain every affine configuration or establish that its states are parity projections. See Appendix B.7 and Figure 3.
  • The matched study supports an observed slowdown, not the original fivefold headline. The new control matches projection shape, parameter count and initialization draws across five paired seeds per arm. Euler is slower in every seed. The exact within-seed permutation test gives omnibus p=0.0093p=0.0093, but pairwise comparisons are not significant. We removed the fivefold headline and make no equivalence claim. Quaternion normalization retains its small contraction term. See Section 4.3 and Appendix B.4, Figure 2.

  • The original head count is not claimed necessary. A small two-seed sweep succeeds at 64 heads; the smaller tested widths fail within budget (Section 4.2). This is not a minimum-width or success-probability characterization. Separately, retaining a small selected subset of recurrent dynamics improves long-horizon A5 prediction, but frozen heads still supply token-dependent features (Section 4.1, Figure 1). That intervention does not establish successful training of a small network. Appendix C.1 adds accurate quadratic readout probes on selected frozen states; these are post-training diagnostics using heads selected with the group table.

  • We do not attribute the hybrid gain to token-dependent rotation. The constant-rotation control performs similarly, so the mechanism remains unidentified. Section 4.5 and Appendix D retain language modelling as a limitation. Sections 4.4 and 6 likewise keep expanded-family failures limited to the tested configurations and budgets.

Reviewer_RbHfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies recurrent models with hidden state evolving according to St=RtSt−1,Rt∈G,S_t=R_{t}S_{t-1},\qquad R_t\in G, where GG is a compact group, in particular Spin⁡(3)\operatorname{Spin}(3) or Spin⁡(4)\operatorname{Spin}(4). The considered task is finite-group tracking: given inputs x1,…,xtx_1,\ldots,x_t associated with elements π(xi)\pi(x_i) of a finite group HH, the model must predict their running product ht=π(at)⋯π(a1).h_t=\pi(a_t)\cdots\pi(a_1). A continuous decoder p(h∣St)p(h\mid S_t) predicts this targe with asymptotic expected loss L=lim sup⁡T→∞1T∑t=1TE[−log⁡p(ht∣St)].\mathcal L=\limsup_{T\to\infty}\frac1T\sum_{t=1}^T\mathbb E[-\log p(h_t\mid S_t)].

The paper characterizes the optimal loss through a group-theoretic hidden-fibre JJ, obtaining inf⁡L=log⁡∣J∣\inf\mathcal L=\log|J|. For H=S5H=S_5, the paper proves the sharp lower bound L≥log⁡60\mathcal L\geq\log60 for any fixed finite number of pure Spin⁡(3)\operatorname{Spin}(3) or Spin⁡(4)\operatorname{Spin}(4) heads, under i.i.d. inputs and a fixed continuous, strictly positive decoder, showing that the considered recurrences are not expressive enough to learn S5S_5. In contrast, A5A_5 admits arbitrarily small loss.

The experiments study recurrences St=λtRtSt−1+UtS_t=\lambda_tR_tS_{t-1}+U_t, which are outside the main theorem's scope, reporting successful A5A_5 tracking and length extrapolation.

Critical strengths

The main strength of the paper is its theoretical characterization of the limitations of recurrent models to represent noncommutative computations in a concrete setting. In particular, the hidden-fibre result characterizes the optimal asymptotic loss, and the sharp log⁡60\log 60 bound for Spin⁡(3)\operatorname{Spin}(3) recurrences on the S5S_5 target provides a concrete illustration of these limitations. On the other hand, the positive result for A5A_5 demonstrates that the limitation is specific to the target group rather than to noncommutative computations in general. I find these results mathematically interesting.

Critical weaknesses

1. Poor writing and unclear motivation: The paper is rather difficult to follow, especially regarding the motivation for the considered setting and the organization of the results. The introduction does not sufficiently explain why finite group tracking is an interesting benchmark for recurrent models, or clearly introduce the relationship between input tokens, recurrent transitions, hidden states, and target group elements. The motivation for the different experimental tasks (A4A_4, S4S_4, A5A_5, S5S_5) and their relation to the theoretical results is also not sufficiently explained.

2. Lack of relation between theory and experiments: The main theoretical result concerns pure rotor recurrences without sequential inputs, while the experiments on S5S_5 use more general affine recurrences. While the authors acknowledge this distinction, it is unclear why they do not include experiments with the exact recurrent dynamics considered in the theorem.

Questions for authors
  1. What motivates the use of Spin⁡(3)\operatorname{Spin}(3) and Spin⁡(4)\operatorname{Spin}(4) transitions in recurrent models, except the mathematical tractability? Does this model have any relationship to RNNs used in practice?

  2. Why do the authors not include an S5S_5 experiment with the pure rotor recurrence considered in Theorem 1? In particular, why does the no-write ablation in Appendix B.6 still retain learned decay?

  3. What is the purpose of the A4A_4 experiment in Section 4.3? It is unclear what additional insight it provides compared with the A5A_5, S4S_4, and S5S_5 experiments.

1 comment
AuthorsOpenReview ↗

Motivation, pure-rotor experiments and S4

Thank you for the review. We have updated the manuscript; changes are marked in blue. All references below point to the revised version.

The main revision explains the task and model before the theory, then adds experiments with the exact pure-rotor dynamics that were missing from the submission.

  • Group tracking tests whether a recurrent state preserves input order. Each token denotes a group element, and each prefix has an exact running-product label. The model maps tokens to rotations, updates its state sequentially and decodes that label. The theorem's restriction is token-local transitions, not an absence of sequential inputs. We now introduce these objects explicitly in Section 1.

  • Rotors provide a concrete computational property beyond tractability. They give compact, norm-preserving transitions whose products can depend on order, while retaining a linear recurrence compatible with a parallel scan. Sections 1 and 2 relate the commuting complex-phase case to existing recurrent architectures. We do not claim that rotor recurrence itself is new or that these properties alone imply practical superiority.

  • We now remove both writes and decay in the experiment. The new pure-rotor runs use unit rotors and a decoder held fixed after training, with evaluation to 10,000 tokens. Current-token queries and gates fit Corollary 2's fixed-suffix setting. On S5, loss is near log⁡60\log 60 within the training horizon, then parity eventually fails; this does not establish asymptotic convergence. On A5, four of five seeds learn. See Section 4.1, Figure 1 and Appendix C (S5 curves: Figure 4). The earlier no-write ablation retained decay to isolate the effect of removing writes from the affine model; it was not the pure recurrence (Appendix B.6).

  • S4 separates an available construction from successful learning. It is smaller than A5 and has a rotor construction through the binary-octahedral group, yet the tested training configurations fail to find a solution. Its role is a learnability diagnostic, not another impossibility theorem. We now state this purpose explicitly in Section 4.4.

The revised introduction distinguishes the compact-state theorem, the pure-rotor experiments and the separate affine-model studies. It also states that the S5 numerical floor equals the commuting-transition floor; rotor and abelian capabilities differ on A5 and S5 x A5.

Reviewer_1dPROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies recurrent states updated by left multiplication with rotors (unit elements of Spin(n), mainly Cl(3,0) quaternions) and asks what such a compact state lets a continuous decoder predict about a running group product. It has three parts.

  • Theory (§3, App. A). The main tool is a "hidden-fibre" principle (Prop. 1). Take a compact transition group G, a finite target group H, and i.i.d. inputs whose labels generate H. Let K be the closed subgroup of G × H generated by token pairs (T(x), π(x)), and let J = {h : (1,h) ∈ K}. Then the infimum over fixed continuous strictly positive decoders of the asymptotic prefix-averaged log loss is exactly log|J| when the full transition is observed. When only a state is observed, log|J| is still a lower bound. The proof is Cesàro convergence of the joint walk to Haar measure on K (Kawada–Itô) plus Jensen. Lemma 1 shows that no closed subgroup of SU(2)^m has S5 as a continuous quotient. This forces A5 ⊆ J for any finite number of Spin(3) or Spin(4) heads, which gives Theorem 1: a sharp log 60 floor on S5. A parity tracker approaches it (Cor. 1), while A5 admits arbitrarily small loss through the binary icosahedral group. Further results cover fixed-suffix decoders (Cor. 2), an exact-tracking factorization and a uniform-separation quotient theorem (Thms. 2–3, Prop. 3), the dense-free-subgroup counterexample, and reflection-length budgets (App. A.4).
  • Group-tracking experiments (§4.1–4.3, App. B). These use affine rotor models with writes and decay, which are outside the theorem. A one-layer Cl(3,0) model solves A5 at length 128 and holds accuracy 1.000 at 256 and 512 (5/5 runs). Three parameterizations of Spin(3) have median sustained-solve times that differ by up to about 5× (10 seeds each). S4 stalls. On S5, every tested family, including Pin(4) and Spin(5), which can represent S5, settles on a parity-dominated plateau within 30k steps.
  • Language modelling (§4.4, App. C). A standalone rotor LM is much worse than matrix-state baselines (PPL 43.8 vs 30.2–33.9 at 125M). A rotor+DeltaNet hybrid beats a DeltaNet control at 50M, but a token-independent rotation does just as well, and a paired identity-vs-constant intervention slightly favours identity.
Critical strengths
  • Prop. 1 is a tidy, correct statement that turns "which target cosets the closure of the transition walk cannot tell apart" into an exact optimal log loss. It needs neither finite reachability nor a separation margin. The S5 × A5 example (App. A.2), where rotor and abelian floors separate (log 60 vs log 3600, and log 120 for one Spin(3) head), shows that the formula has content beyond the commutator bound.
  • I went through Lemma 1 (the induction uses the commuting coordinate kernels and C_{S5}(A5) = 1), Prop. 1, Theorem 1, Corollaries 1–2, Theorem 3, Prop. 4 (the character-averaging bound dim V ≤ 2b − 1) and Lemma 3 (equal multiplicities of primitive fifth roots for a 5-cycle). I found no errors. The care taken with the left action ρ_L(Spin(n)) versus the induced SO(n) action, and with multi-head products (Remark 2, the (Z2)^3 example), is welcome. Many related papers get this point wrong.
  • It separates representability, continuous predictability and learnability (Table 1). It gives a counterexample showing that separation is needed for the exact-tracking result (dense free subgroup in SU(2)). It states plainly that the trained models are outside the theorem. Negative results, crashed runs and failed seeds are reported.
  • All runs solve A5 at 1.000 to 4× the training length, the write-free and write-variant ablations solve it as well, and a separate routed replication with a commuting fixed-axis control agrees.
  • Protocol tables, per-seed ledgers, archived code snapshots, a frozen pre-registered manifest for the identity/constant intervention, and the ROCm bug notes are all included.
Critical weaknesses
  • The main theorem is the log 60 floor on S5. As §3.1 itself notes, abelian (e.g. diagonal, sign-flip) transitions already force [S5, S5] = A5 ⊆ J and so have the same floor, and a real diagonal recurrence with a −1 eigenvalue already reaches it by tracking parity. So the headline result says that, asymptotically and with continuous decoding, finite-head Spin(3)/Spin(4) state is exactly as limited on S5 as a commuting recurrence. That is a legitimate negative result, but the paper frames it as a rotor-specific insight. The only place rotors beat abelian transitions is A5 (and the S5 × A5 construction), and there the "theory" is the existence of the binary icosahedral lift, which is classical. The framing should be more direct: the new theoretical content is Prop. 1 and Lemma 1, and on S5 the rotor floor coincides with the abelian floor.
  • The bound concerns lim (1/L) Σ_t E[loss] with the decoder fixed as L → ∞. It gives no rate (acknowledged in §3.1 and §6). This matters for several reasons:

At any finite horizon the reachable set is finite, so a continuous decoder can separate it and approach zero loss up to that horizon (compare the dense-free-subgroup tracker in Remark 1). The theorem says nothing about the length-128 or length-512 regime the experiments live in. Equidistribution on K can be arbitrarily slow, for example when the transitions are close to the identity or close to a finite subgroup. So the floor may not be visible at any practical length. Table 5 and Table 13 report losses below log 60 (e.g. 3.96–4.03) at finite length, which the paper rightly says "do not bear on the bound". That also means the experiments cannot confirm or test it. A finite-length version would greatly increase the practical relevance. For example, a spectral-gap rate for transitions with algebraic entries (Bourgain–Gamburd-type results for SU(2)) would give an explicit L-dependent bound. Failing that, an empirical loss-vs-position curve for a pure rotor model at long lengths would help.

  • Every experiment is declared "outside the theorem", so the empirical half documents learnability without testing any theoretical prediction. Two simple connections are missing:
  • No pure-rotor experiment. Neither table trains the model Theorem 1 is about (λ = 1, U = 0). A pure-rotor S5 run would show whether loss approaches log 60 from below as length grows, and a pure-rotor A5 run would show whether training finds the near-zero-loss solution.
  • The write-free ablation may in fact fall under Corollary 2. It keeps sigmoid decay, a token-dependent query and an output gate (App. B.6). The decay is a positive scalar, and the readout is followed by RMSNorm, so up to the normalization epsilon (and fp32 underflow of Π λ_t) it is scale-invariant.
  • The query and gate depend only on the current token, which is the k = 1 fixed-suffix case of Cor. 2. If so, the floor does apply asymptotically to this model, and evaluating it at lengths of 10^3–10^4 would be a real test. The authors should either explain why this reduction fails or run the test.
  • "Median sustained solve times … span roughly a factor of five" compares 14.5k (normalized, 10/10) with 73.5k (Euler, 8/10), conditioned on success. The ranges overlap heavily (10k–93k vs 20k–186k). There is no test (e.g. a log-rank test on the time-to-solve curves, which would also handle censored failures). The arms also differ in projection width and bias (4 outputs with bias vs 3 without), in parameter count, and in RNG consumption, so seeds are not paired. The normalized "rotor" is not unit (q/√(‖q‖² + 10⁻⁶)). App. B.4 acknowledges all of this, but the abstract still presents it as a property of "parameterizations of the same ideal rotation family". The obvious control is to match the projection, i.e. a normalized quaternion without bias, or exp/Euler with a fourth (discarded) output.
  • Each S5 row in Table 5 is one seed at 30k steps, and the versor row is the single run in five that reached the plateau. "Permitted, not learned" for Pin(4)/Spin(5) (Table 6) is therefore a statement about one budget-limited configuration. Construction-informed initialization (as in Siems et al., 2026) would show whether the obstacle is optimization or the readout. The DeltaProduct baseline gets about 7.7% of its published sequence budget, a different batch size and different β₂, and fails. Published DeltaProduct solves A5 in one layer.
  • The paper is careful not to claim sample efficiency, but Table 3 has no working baseline at all, so the * A5 result has no comparative reference. Either reproduce the published recipe or drop the comparison. "Fourfold extrapolation" with 3+2 seeds is modest next to Walker et al. (2025), which the paper cites as extrapolating further.
Questions for authors
  • Can you train and evaluate a pure rotor model (λ_t = 1, U_t = 0) on S5 and A5 and plot loss against position to lengths of 10^3–10^4? Does the S5 loss approach log 60 from below?
  • Does the write-free ablation (decay + RMSNorm + current-token query and gate) reduce to the setting of Corollary 2 up to the normalization epsilon? If not, which component breaks the reduction? If so, does its S5 loss at long lengths respect the floor? *Can you give any finite-L rate for Theorem 1, for example under a spectral-gap assumption on the increment law ν restricted to K? Or can you construct an example where convergence to the floor is arbitrarily slow?
  • Does the theorem extend to stacked pure-rotor layers, where layer-2 transitions depend on layer-1 outputs and are therefore no longer token-local? Is S5 continuously predictable with two such layers?
  • On S5, what does the log-loss floor tell us beyond Zhang & Li's (2026) commutator-based accuracy ceiling, given that rotor and abelian floors coincide there?
  • For the parameterization study, can you run the normalized quaternion without the projection bias (or exp/Euler with matched outputs) and report a log-rank test across the time-to-solve curves, including censored runs?
  • Does construction-informed initialization (e.g. the finite preimage of S5 in Pin(4) or Spin(5)) let the model learn full S5 tracking, and if so does it stay stable during training? That would separate optimization difficulty from readout limitations.
  • Can the DeltaProduct baseline be run on its published budget, or can the local budget be matched in sequences seen, so that Table 3 has a working comparison?
1 comment
AuthorsOpenReview ↗

Pure-rotor experiments, scope and controls

Thank you for the review. We have updated the manuscript; changes are marked in blue. All references below point to the revised version.

The main revision adds experiments with the pure-rotor model and narrows the empirical claims. The remaining theoretical and experimental gaps are stated explicitly.

  • We now train the model covered by the theorem. The new runs use unit rotors, no writes and no decay, with the decoder fixed after training. Current-token queries and gates fit Corollary 2's fixed-suffix setting. We evaluate to 10,000 tokens. On S5, loss is near log⁡60\log 60 within the training horizon, but parity eventually fails at longer horizons; this does not demonstrate asymptotic convergence. On A5, four of five seeds learn. See Section 4.1, Figure 1 and Appendix C (S5 curves: Figure 4).

  • The theoretical advance is the scope of the bound, not a different S5 value. We now explicitly state that the S5 floor equals the order-blind floor. Proposition 1 characterizes the exact optimal asymptotic loss when the full accumulated transition is observed; it gives a lower bound for state-only observations. Lemma 1 extends the obstruction to order-sensitive finite-head rotor states. Zhang and Li assume uniform full-group inputs and order-blind prediction; we allow order-sensitive transitions under an i.i.d. generating law and fixed continuous strictly positive decoding. Their optimal accuracy ceiling implies the order-blind log-loss floor by Jensen's inequality; that implication is our derivation. Rotor and abelian floors differ on A5 and S5 x A5. See Sections 3.1--3.3 and 5.

  • Positive normalization epsilon prevents cancellation of decay. The new audit shows why the implemented write-free model does not reduce to Corollary 2 by scale cancellation. For accumulated positive decay cc,

    = \frac{z}{\sqrt{\operatorname{mean}(z^2)+\frac{\varepsilon}{c^2}}}.$$ With zero epsilon, cancellation holds at nonzero readouts, but the corollary also requires continuity on the relevant compact orbit. The audit records contraction below the normalization threshold. Early-position losses explain the below-floor prefix average of this write-free S5 checkpoint; we do not extend that explanation to all affine models. See Appendix B.7 and Figure 3.
  • The matched control shows an observed Euler slowdown, with limited statistical evidence. Five paired seeds per arm now match projection shape, parameter count and initialization draws. Euler is slower in every seed. The exact within-seed permutation test gives omnibus p=0.0093p=0.0093, but the pairwise comparisons are not significant. We removed the fivefold headline and make no equivalence claim. Quaternion normalization still has a small contraction term. See Section 4.3 and Appendix B.4, Figure 2.

  • The A5 mechanism evidence remains approximate. Keeping selected heads' recurrent dynamics improves extrapolation in three successful seeds, but frozen heads still supply current-token features. This is an intervention on a wide trained model, not proof of exact group representations or successful training at that smaller width. Additional frozen-state readout probes are post-training diagnostics. See Section 4.1, Figure 1 and Appendix C.1.

  • Several requested extensions remain open. We provide no finite-length rate. Stacked pure-rotor layers remain open because higher-layer transitions need not be token-local. Construction-informed Pin(4)/Spin(5) initialization and DeltaProduct at its published budget were not run. Table 3 now contains only rotor measurements; the incomplete baseline reproduction remains documented in Appendix B.1. Expanded-family failures are limited to the tested configurations and budgets. See Sections 4.4 and 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.