- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper analyzes how channel-wise gating changes the expressivity of Kimi Delta Attention and proposes Complex KDA (CKDA), combining signed gates with an extended delta-rule coefficient. The authors characterize the resulting diagonal-plus-rank-one transitions, establish finite-group tracking constructions and limitations, and derive reduced-depth automata simulations. Experiments evaluate group-word problems, synthetic periodic waveform continuation, language modeling, and scaling behavior. CKDA improves extrapolation over the tested KDA range variants on selected state-tracking tasks while maintaining competitive language-modeling performance and near-KDA recurrent-kernel throughput.
Critical strengths
-
The paper provides a substantive analysis of what signed channel-wise gating adds to delta-rule recurrences. The characterization of orthogonal diagonal-plus-rank-one transitions, the distinction between realization and decoded tracking, and the S5 impossibility result under explicit assumptions provide useful insights beyond the architectural modification itself.
-
The authors connect the analysis to an efficient implementation and a substantial experimental evaluation. The gate-range ablations and learned transition spectra support the proposed mechanism on state-tracking tasks. The language-model scaling experiments and near-KDA kernel throughput also suggest that the added expressivity can be incorporated without a major performance or implementation penalty.
Critical weaknesses
-
The practical learnability of the strongest expressivity results remains unclear. Standard CKDA training fails on A5, whereas the successful theory-initialized experiment also changes the architecture and training schedule, as the authors acknowledge. Figure 17 additionally shows successful A5 extrapolation for DeltaProduct2 without SiLU. Since state-tracking results report the best of three seeds, matched initialization controls, learning curves, and success rates across seeds would better establish when CKDA offers a practical advantage over this closely related baseline.
-
The connection between exact expressivity and robust long-horizon behavior needs further evidence. The constructions use exact arithmetic and endpoint parameters, and S4 accuracy still declines with length. The waveform experiment is a useful controlled demonstration, but it uses one seed and phase shifts of one fixed groove. Precision and perturbation tests, together with evaluation on different periods or waveforms, would clarify how reliably the proposed dynamics generalize beyond these settings.
-
The language-modeling results primarily establish competitiveness with KDA rather than a consistent benefit from the additional expressivity. At 1.3B parameters, recurrent CKDA and the matched KDA control obtain average accuracy of 54.06% and 54.09%, respectively, while the hybrid CKDA model is worse on all three reported recall tasks. Observing complex eigenvalues during training does not establish their functional contribution. Targeted interventions or matched ablations connecting these dynamics to particular capabilities would strengthen the practical motivation.
Questions for authors
See weakness
3 comments
The waveform experiment is a useful controlled demonstration, but it uses one seed and phase shifts of one fixed groove.
Please see our comment to our general response. We extended the experiments to training on many loops of different periods leading to our CKDA based model to in-context continue novel audio loops for long sequences better than S5 and KDA.
I appreciate the authors’ response and their careful answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.
We thank the reviewer for their thoughtful review and address each point below:
We agree that the learnability of A5 for CKDA is poor when using the standard initialization. What we found interesting is that A5 can be solved at all using CKDA and doesn’t require DeltaProduct2. For DeltaProduct2, the construction the model needs to learn is quite straightforward: it needs to learn the rotation group of the dodecahedron in three dimensions. CKDA cannot learn A5 in three dimensions because of the requirements that each rotation (except identity and half-turns) needs to be axis aligned. Only in four dimensions does the quaternion construction enable fitting A5, but this requires a much larger number of recurrent states and significantly more expressive decoder than the 3D icosahedron construction for DeltaProduct2. We added a short sentence to the experimental section (6.1) of the paper explaining this. We also would like to clarify that we did not change the architecture for A5 (just the dimensions), but initialized the recurrence parameters close to the theoretical construction.
Thank you for this detailed feedback. CKDA’s accuracy decays on S4 because linear RNNs, while theoretically capable of perfect length extrapolation on these tasks, still show degradation in finite precision when trained for these tasks. This is a recently established limitation that we already report in the related works (Chung et al. 2026, Dankowiakowski et al 2025).
We agree that our current experiments do not establish a consistent advantage of CKDA over KDA in language modeling. For the recall tasks, we also evaluate our largest scaling ladder models trained for different data budgets showing a rank switching and metrics variance throughout training. Therefore, the scores of Table 2 don’t support the CKDA hybrid being clearly worse consistently on retrieval. We have added a small paragraph in the Appendix H.2 to explain this point. We also show that restricting the beta and alpha ranges degrades performance (see general response). This partly addresses your question on the usefulness of complex eigenvalues for language modeling in CKDA.
References:
Jiwan Chung, Heechan Choi, and Seon Joo Kim. Rethinking state tracking in recurrent models through error control dynamics. NeurIPS 2026
Dankowiakowski, A., & Ronca, A. (2025). Metric Automata Theory: A Unifying Theory of RNNs. In Advances in Neural Information Processing Systems (NeurIPS 2025), volume 38