acceptodds
Under review as a conference paper at ICLR 2027

Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

Abstract

Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition. We show that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate. This requires extending the parameter ranges of KDA by combining two existing range extensions: allowing gates in and the delta-rule coefficient in . We call the resulting model Complex KDA (CKDA). It preserves KDA’s stability and efficiency, with transitions that remain diagonal-plus-rank-one and non-expansive, while reaching the known state-tracking expressivity of DeltaProduct. We characterize the expressivity of CKDA and prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. A single CKDA layer can track every finite group isomorphic to a subgroup of , and many state-tracking results use one fewer layer for CKDA compared to other diagonal-plus-rank-one Linear RNNs. Empirically, combining both extensions yields the strongest length extrapolation among tested KDA range settings on , , and periodic audio continuation. In language modeling, CKDA (and its attention hybrid) is competitive to KDA, outperforms Transformers and other linear RNNs, and shows promising scaling behavior.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Linear RNN, Expressivity, State-tracking, Rotations, State-space models, Linear Attention, Automata, Formal languages, Finite Groups

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VWyXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzes how channel-wise gating changes the expressivity of Kimi Delta Attention and proposes Complex KDA (CKDA), combining signed gates with an extended delta-rule coefficient. The authors characterize the resulting diagonal-plus-rank-one transitions, establish finite-group tracking constructions and limitations, and derive reduced-depth automata simulations. Experiments evaluate group-word problems, synthetic periodic waveform continuation, language modeling, and scaling behavior. CKDA improves extrapolation over the tested KDA range variants on selected state-tracking tasks while maintaining competitive language-modeling performance and near-KDA recurrent-kernel throughput.

Critical strengths
  1. The paper provides a substantive analysis of what signed channel-wise gating adds to delta-rule recurrences. The characterization of orthogonal diagonal-plus-rank-one transitions, the distinction between realization and decoded tracking, and the S5 impossibility result under explicit assumptions provide useful insights beyond the architectural modification itself.

  2. The authors connect the analysis to an efficient implementation and a substantial experimental evaluation. The gate-range ablations and learned transition spectra support the proposed mechanism on state-tracking tasks. The language-model scaling experiments and near-KDA kernel throughput also suggest that the added expressivity can be incorporated without a major performance or implementation penalty.

Critical weaknesses
  1. The practical learnability of the strongest expressivity results remains unclear. Standard CKDA training fails on A5, whereas the successful theory-initialized experiment also changes the architecture and training schedule, as the authors acknowledge. Figure 17 additionally shows successful A5 extrapolation for DeltaProduct2 without SiLU. Since state-tracking results report the best of three seeds, matched initialization controls, learning curves, and success rates across seeds would better establish when CKDA offers a practical advantage over this closely related baseline.

  2. The connection between exact expressivity and robust long-horizon behavior needs further evidence. The constructions use exact arithmetic and endpoint parameters, and S4 accuracy still declines with length. The waveform experiment is a useful controlled demonstration, but it uses one seed and phase shifts of one fixed groove. Precision and perturbation tests, together with evaluation on different periods or waveforms, would clarify how reliably the proposed dynamics generalize beyond these settings.

  3. The language-modeling results primarily establish competitiveness with KDA rather than a consistent benefit from the additional expressivity. At 1.3B parameters, recurrent CKDA and the matched KDA control obtain average accuracy of 54.06% and 54.09%, respectively, while the hybrid CKDA model is worse on all three reported recall tasks. Observing complex eigenvalues during training does not establish their functional contribution. Targeted interventions or matched ablations connecting these dynamics to particular capabilities would strengthen the practical motivation.

Questions for authors

See weakness

3 comments

We thank the reviewer for their thoughtful review and address each point below:

The practical learnability of the strongest expressivity results remains unclear. Standard CKDA training fails on A5, whereas the successful theory-initialized experiment also changes the architecture and training schedule, as the authors acknowledge. Figure 17 additionally shows successful A5 extrapolation for DeltaProduct2 without SiLU. Since state-tracking results report the best of three seeds, matched initialization controls, learning curves, and success rates across seeds would better establish when CKDA offers a practical advantage over this closely related baseline.

We agree that the learnability of A5 for CKDA is poor when using the standard initialization. What we found interesting is that A5 can be solved at all using CKDA and doesn’t require DeltaProduct2. For DeltaProduct2, the construction the model needs to learn is quite straightforward: it needs to learn the rotation group of the dodecahedron in three dimensions. CKDA cannot learn A5 in three dimensions because of the requirements that each rotation (except identity and half-turns) needs to be axis aligned. Only in four dimensions does the quaternion construction enable fitting A5, but this requires a much larger number of recurrent states and significantly more expressive decoder than the 3D icosahedron construction for DeltaProduct2. We added a short sentence to the experimental section (6.1) of the paper explaining this. We also would like to clarify that we did not change the architecture for A5 (just the dimensions), but initialized the recurrence parameters close to the theoretical construction.

The connection between exact expressivity and robust long-horizon behavior needs further evidence. The constructions use exact arithmetic and endpoint parameters, and S4 accuracy still declines with length. The waveform experiment is a useful controlled demonstration, but it uses one seed and phase shifts of one fixed groove. Precision and perturbation tests, together with evaluation on different periods or waveforms, would clarify how reliably the proposed dynamics generalize beyond these settings.

Thank you for this detailed feedback. CKDA’s accuracy decays on S4 because linear RNNs, while theoretically capable of perfect length extrapolation on these tasks, still show degradation in finite precision when trained for these tasks. This is a recently established limitation that we already report in the related works (Chung et al. 2026, Dankowiakowski et al 2025).

The language-modeling results primarily establish competitiveness with KDA rather than a consistent benefit from the additional expressivity. At 1.3B parameters, recurrent CKDA and the matched KDA control obtain average accuracy of 54.06% and 54.09%, respectively, while the hybrid CKDA model is worse on all three reported recall tasks. Observing complex eigenvalues during training does not establish their functional contribution. Targeted interventions or matched ablations connecting these dynamics to particular capabilities would strengthen the practical motivation.

We agree that our current experiments do not establish a consistent advantage of CKDA over KDA in language modeling. For the recall tasks, we also evaluate our largest scaling ladder models trained for different data budgets showing a rank switching and metrics variance throughout training. Therefore, the scores of Table 2 don’t support the CKDA hybrid being clearly worse consistently on retrieval. We have added a small paragraph in the Appendix H.2 to explain this point. We also show that restricting the beta and alpha ranges degrades performance (see general response). This partly addresses your question on the usefulness of complex eigenvalues for language modeling in CKDA.

References:

Jiwan Chung, Heechan Choi, and Seon Joo Kim. Rethinking state tracking in recurrent models through error control dynamics. NeurIPS 2026

Dankowiakowski, A., & Ronca, A. (2025). Metric Automata Theory: A Unifying Theory of RNNs. In Advances in Neural Information Processing Systems (NeurIPS 2025), volume 38

The waveform experiment is a useful controlled demonstration, but it uses one seed and phase shifts of one fixed groove.

Please see our comment to our general response. We extended the experiments to training on many loops of different periods leading to our CKDA based model to in-context continue novel audio loops for long sequences better than S5 and KDA.

I appreciate the authors’ response and their careful answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.

Reviewer_qaMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper comes with various theoretical findings about the expressivity of diagonal-plus-low-rank (DPLR) family of models. It shows that an extension of the KDA model, the complex KDA (CKDA), can express certain kinds of transformations.

In experimental comparisons, no clear advantage of the CKDA over KDA can be seen.

Critical strengths

Novel theoretical findings about the whole diagonal-plus-low-rank (DPLR) family of models, beyond the specific complex KDA model.

Critical weaknesses

Experimental results show that there is no real contribution by the model

Complex KDA doesn't seem to be better than KDA (Table 2). At 1.3B the language-modeling gains over plain KDA are within noise.

Also, Table 2 shows that Transformer is the worst model. But this is misleading. Later is it shown that other Transformer variants such as the Transformer++ are competitive.

Theoretical findings don't have much practical value

Experiments show that learnability is the limiting factor. All the theorems are about expressivity, not learnability.

Theorem 3 (finite-group constructions) is less relevant for the question to decide whether to use CKDA in an LM.

Theorem 4: Not so relevant in practice, nobody trains one-layer models.

Theorem 5 is not convincing that this gives actual practical advantage over Merrill et al.

Theorems (esp Theorem 4) are difficult to verify for me. Lean verifications might make sense.

Questions for authors

Fig 7 shows that there are complex transitions concentrated in the early layers. What is the role of those?

What is the variance per random seed (e.g. Table 7)?

Spread initialization is only used for the 340M models but not for the 1.3B models? Why?

3 comments

We thank the reviewer for their thoughtful comments and for acknowledging the new understanding of the expressivity of DPLR matrices and will address each point raised below.

Complex KDA doesn't seem to be better than KDA (Table 2). At 1.3B the language-modeling gains over plain KDA are within noise. Also, Table 2 shows that Transformer is the worst model. But this is misleading. Later it is shown that other Transformer variants such as the Transformer++ are competitive.

We agree that Complex KDA only shows similar but not better performance than KDA on the language modeling evaluations we conducted. We’d like to clarify that the Transformer results from Table 2 were adopted from GDN-2 [1]. The Transformer++ results shown in our scaling laws were retrained and we show in Figure 15 that both the recurrent and the hybrid models clearly outperform Transformer++ in terms of validation loss across scales and data budgets. In Table 14, we further show average accuracy across the lm eval harness tasks from Table 2 (except the retrieval tasks), also demonstrating a clear advantage of CKDA compared to Transformer++.

Theoretical findings don't have much practical value.

As mentioned in the general response, we agree that the expressivity advantage of CKDA does not translate to improved language modeling performance and further research is required to bridge this gap. In the updated version of the paper, we make this more explicit in the limitation section. However, other applications like audio or time-series might benefit more from the added expressivity of complex KDA.

Given the difficulty of learning state-tracking even in synthetic experiments, we believe that it is valuable to provide improved results on the number of layers needed to solve state-tracking problems. Indeed, the general multi-layer constructions used by [2,3,4] to solve regular languages [2,3], group word problems [3] and WFAs [4] are quite complex and, as far as we know, have never been shown to be actually learned: e.g. no one has shown that S5 (with a single token per transition) can be learned and length-generalize using a linear diagonal plus rank one recurrence with multiple layers. On the other hand, single-layer constructions are generally easier to learn: indeed more expressive single-layer recurrences such as DeltaProduct have been shown to learn S5 [3].

Theorems (esp Theorem 4) are difficult to verify for me. Lean verifications might make sense.

We provide a Lean formalization in the supplementary material (zip) with instructions on how to manually verify the theorems of the Expressivity section. The lean formalization was carried out with several iterations of GPT 6 Astra. Thank you for this suggestion.

Fig 7 shows that there are complex transitions concentrated in the early layers. What is the role of those?

We don’t know what specific function the complex transitions in the early layers fulfill. Our early experiments to understand this better were inconclusive. However, as we show in the general response, we observe degradation in performance when clamping the gate or beta range and hence limit the transitions to have only real eigenvalues.

What is the variance per random seed (e.g. Table 7)?

We only ran a single seed per configuration we report in Table 7, due to limited compute.

Spread initialization is only used for the 340M models but not for the 1.3B models? Why?

Since we did not see a consistent advantage from the spread initialization, we decided to use the normal initialization for the larger scale results, due to limited compute.

References:

[1] Hatamizadeh, Ali, Yejin Choi, and Jan Kautz. "Gated DeltaNet-2: Decoupling erase and write in linear attention." arXiv preprint arXiv:2605.22791 (2026).

[2] Peng, Bo, et al. "Rwkv-7" goose" with expressive dynamic state evolution." arXiv preprint arXiv:2503.14456 (2025).

[3] Siems, Julien, et al. "Deltaproduct: Improving state-tracking in linear rnns via householder products." Advances in Neural Information Processing Systems 38 (2026): 153738-153782.

[4] Merrill, William, et al. "Why Are Linear RNNs More Parallelizable?." arXiv preprint arXiv:2603.03612 (2026).

Thanks a lot for the answer.

Unfortunately I think the weaknesses are still mostly not resolved.

Theoretical findings don't have much practical value.

However, other applications like audio or time-series might benefit more from the added expressivity of complex KDA.

I think this should be demonstrated.

We provide a Lean formalization

Thank you for that. That makes the theoretic part stronger. But I think even a strong theory is not of much value without demonstrating some practical value out of it. And also the mentioned relevance problems remain.

I think this should be demonstrated.

Thank you for the suggestion. Please see our comment to our general response. We extended the experiments to training on many loops of different periods leading to our CKDA based model to in-context continue novel audio loops for long sequences better than S5 and KDA.

Reviewer_Sc61OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors present Complex KDA (CKDA), which improves the expressivity of KDA with some parametric changes that lead to allowing complex eigenvalues in the sequence mixer matrices. The theoretical analysis provided in the paper show how the two main changes to alpha and beta can lead to complex eigenvalues, while its diagonal plus rank one structure assures non-expansivity. As a result, CKDA achieves improved performance on state tracking tasks and length extrapolation on synthetic tasks, while showing competitive results on language modeling settings.

Critical strengths
  • The theoreticla findings and improvements are interesting and the motivation is solid. In particular, how the simple changes to the existing KDA parameterization can lead to imprvoed expressivity was very intersting.
  • As the changes needed to KDA are quite small, CKDA preserves the computational structure of KDA with low extra overhead.
  • The experimental resutls support the improved expressivity claims.
Critical weaknesses
  • Despite the theoretical improvements and the experimental results on synthetic benchmarks, the practical gain on real-world settings (e.g., language modeling) is somewhat unclear. The downstream accuracy is probably within error-bound, and the hybrid versions performing worse than the KDA-based version in both language modeling and recall tasks make me concern if the practical gain is real.
  • There is still a gap between theroetical expressivity and actual learnability, such as CKDA failing to learn A_5 task.
Questions for authors
  • Given that the increased theoretical expressivity does not directly transfer to language modeling performance, which area do the authors expect the theoretical improvements to shine?
  • Coudl the authors provide more insight into the rlatively weak recall performance of CKDA (and weaker language modeling perfomrance of the hybrid version) in Table 2?
1 comment

We would like to thank the reviewer for the thoughtful review and address each point below:

Weaknesses:

Despite the theoretical improvements and the experimental results on synthetic benchmarks, the practical gain on real-world settings (e.g., language modeling) is somewhat unclear. The downstream accuracy is probably within error-bound, and the hybrid versions performing worse than the KDA-based version in both language modeling and recall tasks make me concern if the practical gain is real.

We’d like to clarify that KDA and CKDA perform similarly in the recurrent only model on the language modelling experiments, while CKDA shows larger gains in the hybrids compared to KDA. However, for the recall tasks in Table 2, we also evaluate our largest scaling ladder models trained for different data budgets showing a rank switching between CKDA/KDA. Therefore, the scores of Table 2 don’t support the CKDA hybrid being clearly worse consistently on retrieval. We have added a small paragraph in the Appendix to explain this point (Appendix H.2).

There is still a gap between theroetical expressivity and actual learnability, such as CKDA failing to learn A_5 task.

We agree, we were unable to fit A5 using CKDA using the standard initialization and had to initialize close to the solution to make this experiment work. The single layer construction which the model needs to learn is quite involved compared to S4, since it needs to learn an expressive many to one decoder in four dimensions, compared to the more straightforward construction available to DeltaProduct 2 in three dimensions. We commented on this in the updated version in section 6.

Questions:

Given that the increased theoretical expressivity does not directly transfer to language modeling performance, which area do the authors expect the theoretical improvements to shine?

Concurrent work by Liang & Song (2026) shows that Mamba 3’s rotary dynamics help with long-context evaluations, hence the relatively short context evaluations we made in the paper so far won’t capture all potential advantages. Despite this, the results on audio continuation we conducted suggest that CKDA might be effective to model periodic patterns for e.g. time-series forecasting. Complex transitions were also recently shown to be effective for Mamba-3 based time-series foundation models (Berndt et al. 2026).

Coudl the authors provide more insight into the rlatively weak recall performance of CKDA (and weaker language modeling perfomrance of the hybrid version) in Table 2?

Please see our answer above.

References:

Shutian Liang and Yue Song. "Understanding Mamba-3 Through Oscillatory Dynamics." arXiv preprint arXiv:2610.13341 (2026).

Torben Berndt, Lingyi Yang, Alexander Shears, Benjamin Walker, “Time Flies When You Parallelise: Pareto-efficient Time Series Foundation Models with Linear Recurrences”, NeurIPS 2026

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful comments. We address the main points in this general response. We updated the paper highlighting the main changes in blue.

Lean Proofs. To aid the review of the mathematical results, we add the Lean formalization of the theoretical results with instructions on how to check the results of the main paper and the appendix, to the supplementary material (zip) of our submission. The formalization was made by GPT 6 astra and checked by the authors.

Intervention analysis for the use of complex eigenvalues within the model. To understand whether the extended ranges of alpha and beta are used by the trained model, we cap the alpha and beta of the CKDA and CKDA-hybrid models to their original ranges (either intervention restricts the transition matrix to only real eigenvalues) and find performance degradation across wikitext/lambada/lm eval-harness and recall tasks, showing that the model degrades significantly especially with capped beta range. The only outlier is SQuAD for CKDA recurrent when clamping beta. Table shows improvement over the baseline numbers.

ModelRestrictionWiki. ppl ↓LMB. ppl ↓LMB. acc. ↑PIQA ↑Hella. ↑Wino. ↑ARC-e ↑ARC-c ↑OBQA ↑SIQA ↑BoolQ ↑Avg. ↑SQuAD recall ↑SWDE recall ↑FDA recall ↑
CKDA (recurrent)none (as trained)15.810.151.773.459.359.173.939.028.441.660.554.139.148.731.7
CKDA (recurrent)α → max(0, α)+0.1+1.2-2.7+0.1-0.2-0.5-0.2-0.3-0.2-0.2-1.0-0.6-2.4-4.1-2.2
CKDA (recurrent)β capped at 1+0.9+13.9-16.0-0.7-2.5-2.1-1.9-2.5-0.6-0.6+0.5-2.9+9.7+0.5-5.3
CKDA (recurrent)β capped at 1 + α → max(0, α)+1.1+17.0-18.4-1.0-2.6-2.8-2.4-2.4-0.2-0.7+0.4-3.4+6.6-2.6-8.0
CKDA + attn 3:1none (as trained)15.39.952.673.659.759.972.636.928.242.663.154.444.673.075.7
CKDA + attn 3:1α → max(0, α)+0.0+0.9-2.4-0.1-0.1+0.5-0.2-0.4-0.6-0.2-0.3-0.4-0.8-0.4-0.9
CKDA + attn 3:1β capped at 1+0.8+6.3-10.6-0.9-2.1-1.4-2.1-2.8-2.2-2.0-1.1-2.8-0.3-0.9-7.1
CKDA + attn 3:1β capped at 1 + α → max(0, α)+1.0+7.7-12.4-0.9-2.4-2.4-2.3-3.2-2.0-1.9-1.3-3.2-1.0-1.4-8.7

Gap between expressivity results and language modeling performance. We acknowledge that the difference between CKDA and KDA in language modeling is mild and does not reflect the difference in the theoretical expressivity results. We now explicitly mention this in the limitations of the paper. We believe this is a common issue for this line of work which might require more than just scaling up the models and dataset: for example using different training objectives and the use of state-tracking related data for pre-training, which we leave for future work. State-tracking tasks are notoriously hard to learn even in synthetic setups and they are not very common in natural language. Despite this, the results on audio continuation we conducted suggest that CKDA might be effective to model periodic patterns for e.g. time-series forecasting. Complex transitions were also recently shown to be effective for Mamba-3 based time-series foundation models (Berndt et al. 2026).

Recall Performance Degradation in Table 2 In Hybrid models, Table 2 reports a degraded performance for the recall tasks (SWDE/FDA/SQuAD) for CKDA compared to KDA. We want to point out that this is not a systematic disadvantage. Indeed, for these recall tasks, we also evaluate our largest scaling ladder models trained for different data budgets showing a rank switching and metrics variance throughout training. Therefore, the scores of Table 2 don’t support the CKDA hybrid being clearly worse consistently on retrieval. We have added a small section in the Appendix to explain this point (Appendix H.2). Note also that due to a small bug in batch decoding the recall tasks’ metrics for the hybrids were off, we fixed this in the revision (the ordering of methods stays unchanged).

References:

Torben Berndt, Lingyi Yang, Alexander Shears, Benjamin Walker, “Time Flies When You Parallelise: Pareto-efficient Time Series Foundation Models with Linear Recurrences”, NeurIPS 2026

To demonstrate practical value beyond language modeling (Reviewer qaMj) we extended the audio task to a distribution of musical loops. This also addresses the concern of Reviewer VWyX on the fact that the waveform experiment in the paper uses only one loop.

Training uses 512,000 randomly generated drum/bass/synth loops with random patterns, tempo (100–140 BPM), swing and instrument parameters. Loop periods range from 6 to 32 sixteenth-note steps (with 6 periods held-out), with 64 audio samples per step as in the paper. At test time, the model receives a cue of two periods of a loop never seen in training. It must then continue the waveform from zero inputs in a single causal pass. All models share the same 8-layer (up from 1 layer in the paper experiments), width-256 backbone (6.5–7.8M parameters) and training recipe (16k steps, horizon curriculum up to 448 continuation steps), with learning rates selected per model from a grid ((optimizer: muon,adamw), (lr: 1e-3, 3-e3, 1e-2)). We report continuation SNR on 256 new test loops at the training length (528 steps) and at twice that length.

ModelParamsSNR (dB), 1× lengthSNR (dB), 2× length
CKDA6.5M20.216.9
CKDA, negative gates removed at inference6.5M0.50.6
CKDA, β\beta ≤ 1 at inference6.5M1.51.6
S56.9M17.83.6
KDA6.5M1.81.4

CKDA is the best model at the training length. It is also the only one that retains most of its accuracy at 2×: S5, whose transitions have complex eigenvalues yet lack input selectivity, drops from 17.8 to 3.6 dB. KDA, whose transitions have only real non-negative eigenvalues, does not learn the task (< 2 dB). When removing the complex eigenvalues of the trained CKDA at inference, either by replacing negative gates with a positive decay or by capping β\beta at 1, collapses the SNR to ≈ 1 dB. The learned solution therefore relies on the extended ranges that distinguish CKDA from KDA.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.