- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper distinguishes between "what a compact recurrent state can store" and "what a continuous decoder can retrieve from the state," examining the limits of predictability for pure rotor transitions (, where ) acting on permutation groups. The core tool is a hidden target fiber : given a compact transition group , a finite target group , and i.i.d. inputs, "target elements indistinguishable under the same transition" are collapsed into a normal subgroup . The authors prove (Proposition 1) that when observing the full transition, the infimum of the optimal asymptotic prefix-averaged log loss—using a fixed, continuous, strictly positive decoder—is exactly ; this value remains a lower bound when observing only the state.
Applying this general principle to rotors: in the case of , the authors utilize a group-theoretic fact (Lemma 1: no closed subgroup of admits a continuous surjective homomorphism to ) to establish that , implying . This yields Theorem 1: for any finite number of heads, the lower bound for the asymptotic average log loss on is , a bound achieved by a parity tracker (Corollary 1). In contrast, allows for arbitrarily low loss via a binary icosahedral subgroup. The paper also presents an algebraic decomposition for exact tracking (Theorems 2/3, Proposition 3), auxiliary analysis of reflection length, and learning experiments using an affine rotor model (incorporating additive writes and decay, placing it explicitly outside the scope of the theorems): learned the task and generalized to sequences four times longer, whereas and yielded only partial or parity-based solutions; overall, rotors proved significantly weaker than matrix-state baselines in language modeling tasks.
Critical strengths
Theoretically, the most elegant aspect is the clean contrast between S5 and A5: within the same rotor family, one allows for arbitrarily low loss (A5, since 2I ↠ A5), while the other stalls at log 60 (S5, because states cannot distinguish the A5-coset). The authors break down the concepts of "expressivity," "predictability for sequential decoding," and "learnability via optimization" into the three properties listed in Table 1; this framing is valuable in itself and avoids the common pitfall in state-tracking literature of conflating representability with learnability.
The proof structure in Appendix A is clear: it begins by proving Lemma 1 (using induction on m, the swapping of coordinate projection kernels, the fact that the only normal subgroups of S5 are {1}, A5, and S5, and the condition C_{S5}(A5)=1); it then employs Kawada–Itô Cesàro convergence to show the joint walk converges to the Haar measure; finally, it uses Jensen's inequality to establish a lower bound of log|J| and demonstrates achievability via a coset-uniform decoder. Corollary 2 extends the conclusion to fixed suffixes, Theorems 2 and 3 address exact tracking, and Remark 1 provides a counterexample involving a dense free subgroup to explain why the uniform separation condition cannot be dropped—judging by the provided proofs, this logical chain is both self-consistent and well-organized. Furthermore, the authors’ restraint regarding the scope is commendable: they repeatedly clarify that affine models fall outside the scope of the theorems and characterize the S5 plateau merely as "parallel observation," rather than claiming it represents an experimental realization of the theorem.
Critical weaknesses
(1) There is a disconnect between the strongest theoretical result and the full set of experiments—a fact the authors themselves acknowledge. Theorem 1 addresses "pure transitions" (involving neither writes nor decays), whereas all learning experiments utilize the affine model (Eq. 2); the authors repeatedly note in Table 2, Table 5, and Appendix B.6 that these cases fall "outside the theorem." Consequently, the strongest formal claim is not instantiated by any experiment, nor do the experimental failures serve to validate the theorem. In Table 5, the S5 loss values that plateaued are 3.96, 3.98, 4.03, and 4.03—all below (compared to 4.71 for the fixed-parity control, and baselines of 4.79 for random and 4.09 for sign-only). This implies that the trained affine models were not only unconstrained by the bound but also outperformed the "sign-only" baseline. While the authors correctly explain that affine models are not subject to this bound, this also demonstrates that the bound imposes no direct constraint on the affine models tested in the paper. It is difficult for a reviewer to view this as a constraint capable of guiding practical application.
(2) The incremental contribution of Theorem 1 relative to Zhang & Li (2026) needs to be articulated more clearly. The paper lists Zhang & Li’s asymptotic accuracy ceiling of in Table 7 (noting its implication of a log-loss bound) and states in the "Point of departure" section that "our theorem does not assume order-blindness." The issue lies in the specifics of this delta: for S5, and , resulting in an accuracy ceiling of , which exactly matches the accuracy of the parity-tracking method in this paper. The authors should explicitly clarify whether Zhang & Li’s log-loss bound also yields and how the assumptions of the two approaches compare, rather than leaving this for the reader to infer. If the numerical limits are the same, does the primary advantage of Theorem 1 over Zhang & Li lie in its scope of applicability (moving from "order-blind" to "order-aware/state-dependent decoder") or in the strength of its conclusion? While this could indeed be a genuine contribution—the group-theoretic steps in Lemma 1 are non-trivial—the authors need to explicitly explain the role the "order-blind" assumption played in Zhang & Li’s proof, what new tools were required to remove it, and why a broader scope of applicability matters to the community.
(3) The empirical evidence is weak, and in many instances, the results cannot be clearly attributed to specific factors. Achieving success with A5 required 136–192 heads and 1.9–2.4M parameters (Table 3)—an extreme number of heads; S4 stalled at 0.316; A4 solved the task only within the training length but failed to extrapolate; the rows for S5 were largely based on single seeds, and the "learned-parity" row in Table 5 actually represents the best of five runs (Appendix B.7 clarifies that the other four failed to reach a plateau); regarding language modeling, the standalone rotor scored 43.83 versus baselines of 30.22–33.85 (Table 14). More critically, the source of the gain in the hybrid model was not isolated: in Table 15, Rotor+DeltaNet (50.88) performed almost identically to constant rotation (50.81), meaning the current evidence cannot attribute the gain to token-dependent rotation; the paired identity/constant interventions in Table 16 (using a different normalization and evaluation setup, which the authors state is not comparable to the 50M parameter study) also show a difference of only about 0.225% in perplexity between identity and learned constant rotations. The paper itself acknowledges that causal variables such as layer allocation, width, and generic-layer control were not isolated. A more prudent conclusion is that while experiments show gains for certain hybrid configurations, the underlying mechanism remains unidentified; this is insufficient to support strong claims regarding "learned transport."
(4) The headline empirical finding that "parameterization affects solve time by up to 5×" is confounded. Appendix B.4 explicitly acknowledges that changing the rotor parameterization alters the projection structure (four outputs plus bias vs. three outputs without bias), the parameter count (2.10M vs. 2.04M), and random number consumption; consequently, using the same seed does not result in identical initializations. In other words, the 5× disparity reported in Table 4 conflates three distinct factors: parameterization, projection, and initialization. The authors also admit in Section B.4.1 that neither probe accounts for this gap (the single-rotor fit yields a median of 64/55/66, while the ordered product shows a difference of only about 1.5×). Given this, the 5× figure should not be highlighted as a key selling point in the main text.
Questions for authors
- Does the accuracy ceiling of in Zhang & Li (2026) already encompass the result for S5? If so, please explicitly state in the main text exactly what additional content Theorem 1 provides once the "order-blindness" assumption is removed (e.g., a stronger conclusion, broader applicability, or a different proof path).
- Can at least a "partial bridge" be established between Theorem 1 and the affine experiments? For instance, by proving a corresponding lower bound for a specific class of affine models (with finite length and decay), even if only as a finite-length approximation.
- Regarding the solve-time gap in Table 4, could a paired experiment be conducted? Specifically, by fixing the projection structure, parameter count, and initialization RNG seed while varying only the parameterization, does the difference persist?
- For the S5 row in Table 5, the loss is already below ; how do the authors justify this without leading readers to believe that Theorem 1 imposes no constraints on trainable models? Could you report whether the states of these affine models approximate a parity projection and describe their behavior at longer sequence lengths?
- Model A5 requires 136–192 heads; is this architecturally acceptable? Could you provide a scaling analysis relating the number of heads to the minimum success rate?
Novelty, experimental connection and controls
Thank you for the review. We have updated the manuscript; changes are marked in blue. All references below point to the revised version.
The main revision adds pure-rotor experiments and matched controls, and removes claims that the evidence does not isolate. The results now distinguish the theoretical limit from finite-length learning and readout behavior.
The S5 numerical floor is the same; the new scope covers order-sensitive rotor states. We now make this explicit in Sections 3.1--3.3 and 5. Zhang and Li's optimal order-blind accuracy ceiling implies the corresponding log60 log-loss floor by Jensen's inequality; this implication is our derivation. Their result assumes uniform full-group inputs and order-blind prediction. Ours allows order-sensitive transitions under an i.i.d. generating law and fixed continuous strictly positive decoding. Proposition 1 gives the exact full-transition loss characterization and a lower bound for state observations; Lemma 1 supplies the finite-head rotor obstruction. The rotor and abelian floors differ on A5 and S5 x A5.
We now test pure-rotor dynamics directly, while retaining the limits of that connection. Section 4.1 and Appendix C add runs with no writes or decay, evaluated to 10,000 tokens with the decoder fixed after training. S5 loss is near the floor within the training horizon, then parity eventually fails (Figure 4). This does not establish asymptotic convergence. The affine models remain outside the theorem, so their below-floor finite-length losses do not contradict it. We provide no finite-length bound for general affine models.
The new decay audit explains why scale cancellation fails in the implemented write-free model. With accumulated positive decay c and RMSNorm parameter ε>0,
= \frac{z}{\sqrt{\operatorname{mean}(z^2)+\frac{\varepsilon}{c^2}}}.$$ Thus the normalized readout still depends on decay. Early-position losses explain the below-floor prefix average of the evaluated write-free S5 checkpoint; this does not explain every affine configuration or establish that its states are parity projections. See Appendix B.7 and Figure 3.The matched study supports an observed slowdown, not the original fivefold headline. The new control matches projection shape, parameter count and initialization draws across five paired seeds per arm. Euler is slower in every seed. The exact within-seed permutation test gives omnibus p=0.0093, but pairwise comparisons are not significant. We removed the fivefold headline and make no equivalence claim. Quaternion normalization retains its small contraction term. See Section 4.3 and Appendix B.4, Figure 2.
The original head count is not claimed necessary. A small two-seed sweep succeeds at 64 heads; the smaller tested widths fail within budget (Section 4.2). This is not a minimum-width or success-probability characterization. Separately, retaining a small selected subset of recurrent dynamics improves long-horizon A5 prediction, but frozen heads still supply token-dependent features (Section 4.1, Figure 1). That intervention does not establish successful training of a small network. Appendix C.1 adds accurate quadratic readout probes on selected frozen states; these are post-training diagnostics using heads selected with the group table.
We do not attribute the hybrid gain to token-dependent rotation. The constant-rotation control performs similarly, so the mechanism remains unidentified. Section 4.5 and Appendix D retain language modelling as a limitation. Sections 4.4 and 6 likewise keep expanded-family failures limited to the tested configurations and budgets.