acceptodds
Under review as a conference paper at ICLR 2027

Kronecker Delta Rule for Long Context Linear Attention

98 views

Abstract

We propose KronLSTM, a linear-attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations while modifying the memory only along a single direction in matrix space. This minimally invasive memory revision reduces interference with previously stored information compared with delta-rule updates, making it particularly well suited for long-context language modeling. The resulting memory dynamics can be mathematically described as a Markov chain that converges to a steady-state distribution, while the expected norm of the memory matrix remains uniformly bounded throughout the entire input sequence. Hence, KronLSTM ensures stable memory dynamics without requiring additional normalization and stabilization mechanisms for the memory state. Experiments show that KronLSTM is competitive with, and often outperforms, existing linear-attention architectures, including models of the (Gated) DeltaNet family, with particularly strong performance on long context benchmarks. These results establish the Kronecker Delta Rule as an effective and analytically tractable foundation for stable associative memory in recurrent linear-attention architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, xLSTM, linearRNN, LSTM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4vitOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces KronLSTM, a linear attention architecture based on the Kronecker Delta Rule. Unlike traditional delta-rule vector corrections, this method applies scalar association correction along a single direction in the matrix space, aiming to minimize interference with previously stored information for long-context language modeling. The authors provide theoretical proofs for the steady-state Markov chain of the memory dynamics and bounded memory norms, and conduct experiments on language modeling, retrieval, and long-context evaluation.

Critical strengths
  • The paper shifts the linear attention learning objective from full value vector regression (used in the DeltaNet family) to a scalar association correction. This mechanism distinguishes itself from existing works like Kimi Delta Attention (KDA) and establishes a clear orthogonal protection property, avoiding interference if either the keys or values are orthogonal.

  • The authors trained models at a robust 1.5B parameter and 100B token scale. The evaluation is comprehensive, comparing against KDA, GDN-2, and Mamba-3. It assesses not only pure linear models but also hybrid architectures with sliding-window attention, alongside ablation studies and 32k context extensions.

  • The benefits of KronLSTM are not limited to a single task. In Table 1, the pure linear model's average accuracy across five commonsense reasoning tasks and LAMBADA improves from KDA's 58.12 to 59.72. Furthermore, LAMBADA perplexity drops significantly from 12.669 to 11.716, which is a noteworthy result.

Critical weaknesses
  • The theoretical analysis (Theorems 1 and 2) assumes i.i.d. inputs and a fixed step size β\beta. However, the actual model implements an input-dependent forget gate (FtF_t), a time-varying βt\beta_t, and a decoupled write mechanism. The non-expansive nature of the homogeneous state transition is insufficient to guarantee a bounded memory norm when continuous decoupled writing is applied. The original norm bounds need to be re-evaluated since the effective target becomes c~t=ct/βt\tilde{c}_t = c_t / \beta_t after decoupling. The authors must provide derivations applicable to the actual gated updates or narrow their theoretical claims.

  • Vector correction and scalar correction impose entirely different task constraints. While scalar correction preserves more components, it may also inadvertently retain old information that should have been overwritten. Comparing capacity (dkd_k dvd_v scalar associations vs. dkd_k full value vectors) requires caution. The most valuable supplementary experiment would be to fix the network structure and only swap the correction target (scalar vs. vector) to test repeated key overwriting and value co-existence.

  • According to Table A3, after 32k context extension training, KronLSTM's absolute performance on 32k tasks remains quite weak (e.g., S3 accuracy at 8.6, MK at 5.0, MQ at 3.0, and MV at 11.3). Additionally, the pure linear model suffers a severe performance drop on the 16k S1 task, achieving only 11.6 accuracy compared to GDN's 99.2. This warrants a rigorous explanation and cannot be solely dismissed by stating the task is unrealistic for real-world text.

  • Although the paper claims to provide chunkwise-parallel kernels, it fails to report actual training throughput, decoding latency, memory footprint, or practical computational overhead. Mathematical compatibility with the vectorized Delta Rule does not guarantee identical hardware costs. Furthermore, the main language modeling experiments lack variance reporting across different training seeds.

Questions for authors

While the paper claims to provide chunkwise-parallel kernels, it lacks comprehensive end-to-end efficiency tests. The authors must report actual training throughput (e.g., tokens per second), decoding latency, and wall-clock time comparisons against the baselines. Furthermore, there is a strong suspicion that the scalar key-value pair erasure and update mechanism might negatively impact actual hardware training efficiency. To alleviate this concern, the paper needs to report the loss reduction ratio under a strictly matched computational budget (e.g., iso-FLOPs or iso-wall-clock time), proving that the theoretical benefits of the Kronecker Delta Rule translate into practical, efficient training without severe computational overhead.

5 comments
AuthorsOpenReview ↗

Extending Theorem 2 for KronLSTM

We thank the reviewer for their thorough and valuable feedback. We respond to each point below. The revised manuscript includes substantial new material, highlighted in blue. To accommodate the new content within the page limit, we condensed several tables; if any of these changes reduce clarity, we are happy to adjust them in the final version.

W1: The authors must provide derivations applicable to the actual gated updates or narrow their theoretical claims.

Theorems 1–2 are proved for the basic Kronecker Delta Rule (3) with i.i.d. triples, a fixed step size, and no forget gate. Appendix A2.6.7 now extends Theorem 2 to the decoupled write and the time-varying βt\beta_t of KronLSTM (10):

(a) Decoupled write. For fixed β\beta and Ft=IF_t = I, (10) is rule (3) with effective target ct/βc_t/\beta, and Theorem 2 gives E∥St∥F2≤4E[c2]/(β2λmin⁡(Aˉ))\mathbb{E}\Vert S_t \Vert_F^2 \le 4 \mathbb{E}[c^2] / (\beta^2 \lambda_{\min}(\bar{A})). The re-evaluated bound stays independent of sequence length and scales as β−2\beta^{-2}.

(b) Time-varying βt\beta_t. For βt\beta_t drawn jointly i.i.d. with (kt,vt,ct)(k_t, v_t, c_t) and βt≥βmin⁡>0\beta_t \ge \beta_{\min} > 0, the proofs hold with β\beta replaced by βmin⁡\beta_{\min} (Eq. A118). This lower bound on the erase gate, together with non-expansiveness, bounds the memory under decoupled writing.

(c) Forget gate FtF_t. As stated in Section 2.2, the gate only contracts the homogeneous dynamics further.

(d) I.i.d. inputs. Independence of the current input from the memory StS_t lets expectations factorize, as is standard in LMS analysis. The expected-norm bound holds under this assumption.

The revised manuscript separates these guarantees from the empirical claims. We hope this clarifies how the analysis carries over to KronLSTM, and we are willing to further elaborate on any of the points above should any of the points still not be addressed.

AuthorsOpenReview ↗

Comparing the correction target: Kronecker Delta Rule and Delta Rule

W2: The most valuable supplementary experiment would be to fix the network structure and only swap the correction target (scalar vs. vector) to test repeated key overwriting and value co-existence.

We thank the reviewer for the insightful suggestion. We now dedicate Section 3.1 and Figure 4 to investigating the differences between the update rules in small-scale, controlled settings. We would like to highlight that the scalar vs. vector correction target comparison is equivalent to comparing the Kronecker Delta Rule, which corrects the scalar vt⊤Stktv_t^\top S_t k_t towards ctc_t, with the Delta Rule, which corrects the vector StktS_t k_t towards vtv_t.

Setup: In Section 3.1, both the Delta Rule and Kronecker Delta Rule use the same KronLSTM block with the same projections, convolutions, l2-norms and readout, only the memory update rules change. We experiment both with and without decay gates. All models are trained with the same recipe on MQAR-style tasks with 5 seeds (Appendix A4.1 describes all details).

We test on three tasks:

  • Capacity: We test how many distinct KV-pairs the model can recall. We gradually increase the number of KV pairs and track accuracy of all of them being recalled from memory.
  • Interference: How distractor KV-pairs affect recall of a fixed set of target KV-pairs. We keep the targets fixed, grow the number of distractors written after them, and track target recall accuracy. With 4095 keys in dk=64 dimensions per head, distractor keys necessarily overlap the target keys, so every distractor write acts on the key directions that hold the targets. This tests which update rule retains the target associations while other values co-exist under similar keys.
  • Overwrite: New values are continuously being written to the same key, with the latest value being the correct solution, and older values should be erased from memory. We gradually increase the number of new writes and track which update rule can recover the newest value. This aims to answer which update rule holds on to old values that are no longer necessary.
Model (with decay)Capacity N=128 (train)Capacity N=256Interference D=64 (train)Interference D=1024Overwrite R=2 (train)Overwrite R=4
Delta Rule94.755.898.22.3100.099.9
Kronecker Delta Rule92.354.099.999.590.065.7

Results: For each task, we observe the following:

  • Capacity: Both the Kronecker Delta Rule and the Delta Rule have similar capacity. They all solve the task perfectly within the training distribution and then gradually decline as the number of KV-pairs is increased.
  • Interference: This is where the Kronecker Delta Rule's minimally invasive memory updates pay off. Within the training setting of 64 distractors, all update rules solve the task perfectly, however, once extrapolated to more distractors with more key overlap, the Delta Rule gradually decreases in accuracy while the Kronecker Delta Rule maintains a near-perfect score.
  • Overwrite: In the overwriting setting, the Delta Rule's vector target advantage shows. While both update rules solve the task in distribution, only the Delta Rule can show perfect extrapolation, with the Kronecker Delta Rule gradually declining in accuracy with more overwrites.

Conclusion. With the above three settings, we showed that while their raw capacity to retrieve values is similar, the way the Delta Rule and the Kronecker Delta Rule write into the memory differs, showing a trade-off between being robust against distractor KV-pairs and key/value co-existence, or being able to handle overwrites to new values. We thank the reviewer again for this useful experiment, as we believe it improved our manuscript and clearly highlights the benefits of our proposed update rule.

AuthorsOpenReview ↗

Extending the Experiments for Long Context

W3: According to Table A3, after 32k context extension training, KronLSTM's absolute performance on 32k tasks remains quite weak (e.g., S3 accuracy at 8.6, MK at 5.0, MQ at 3.0, and MV at 11.3).

We appreciate the reviewer’s feedback on the overall weak performance at 32k. We present two arguments and additional experiments to alleviate the concerns: (1) on the data-mixture and long-context recipe and (2) overall limitations of the model size and token budget.

(1) The long-context models were first trained on FineWeb-Edu, then extended to 32k context using a different data corpus, ProLong. While this already substantially improved performance, it is not the most optimal data corpus for long-context training.

To address this, we conducted new pretraining experiments on the Olmo-3 pre- and mid-training recipe at the same 100B token budget (details and full results in Appendix A3.5), with the final training phase extending the context to 32k again. In Olmo-3 the datasets were engineered to be compatible and provide better long-context performance. Moreover, we included KDA as an additional baseline.

We evaluate the models on the same language modeling and recall tasks. Additionally, since the new higher-quality training data and the mid-training phase contain curated long-context documents, we can extend our evaluation pipeline to LongBench (English subset), with tasks spanning single-and multi-doc QA, summarization and code. A summary table is given below, with full results in Tables A7–9:

ModelMCQA Avg-6 ↑Recall Avg-6 ↑RULER Avg-6 @4k ↑@8k ↑@16k ↑@32k ↑LMB ppl ↓LongBench Avg-14 ↑
GDN-256.8938.8266.658.134.525.29.18828.6
KDA58.5539.2578.063.039.727.78.00129.8
KronLSTM59.4740.7977.963.746.228.27.92131.7

Summary. This additional pretraining experiment shows the same trends as our initial experiments, namely that KronLSTM leads in the measured benchmarks, with the only exception of RULER at 4k where it follows KDA by 0.1 points on average. On the new LongBench results, KronLSTM leads by 2 points on KDA and over 3 points on GDN-2.

(2) Addressing the overall weak 32k performance. We understand that the above results - while providing improved performance - do still deteriorate over longer contexts. This is a limitation of the model size (1.5B for our original models, 1.7B for the new Olmo-recipe pretrained ones) and the training budget of 100B tokens across all architectures. Larger models and token budgets can alleviate these in general across all architectures; however, we currently do not have the resources to pretrain at an even larger scale.

We thoroughly appreciated the feedback on our long context performance, and we believe that the addition of these new pretraining runs substantially strengthened our paper.

W3: Additionally, the pure linear model suffers a severe performance drop on the 16k S1 task, achieving only 11.6 accuracy compared to GDN's 99.2.

We investigated the drop in the NIAH-S1 scores thoroughly, and we appreciate the reviewers' suggestion to conduct more thorough experiments. We present evidence that the NIAH-S1 result in Table 3 is specific to the KronLSTM pretrained only on FineWeb Edu at 8k context and is not a failure mode of the Kronecker Delta Rule.

In the synthetic setup of Section 3.1, as used above, we built an S1-style haystack. 32 target pairs are placed at a random depth inside D distractor writes that repeat a fixed pool of 4 pairs, akin to the repeated filler sentences of NIAH-S1 in a controlled, small-scale synthetic setting. Models are trained with groups of 4–6 distractors and up to D=64, and evaluated up to D=4096, where each pair is written about 1000 times.

Accuracy (%)D=256D = 256D=1024D = 1024D=4096D = 4096
Delta Rule10010099
Kronecker Delta Rule100100100

Our controlled experiments show that the Kronecker Delta Rule can solve NIAH-S1 style tasks with perfect extrapolation.

Furthermore, our long-context results (Figure 5) and the newly presented pretraining using the Olmo-3 recipe (Table A8, Figure A1, Recipe described in Appendix A3.5) show that KronLSTM performs near-perfectly on NIAH-S1. From a single run, we cannot determine whether the data, the context length, or run-to-run variance causes it.

Reviewer_1fWhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KronLSTM, a linear attention architecture based on a Kronecker Delta Rule that corrects scalar key-value associations through rank-one memory updates. The method combines this update with input-dependent forgetting and separate erase and write gates. The authors analyze interference, capacity, and convergence for the underlying update rule, including expected memory-norm bounds under stochastic assumptions. Experiments compare pure recurrent and sliding-window hybrid models on language modeling, recall, long-context retrieval, and state tracking, with additional context-extension and component-ablation studies.

Critical strengths
  1. The scalar bilinear update provides an interesting alternative to conventional vector-valued delta-rule learning. Its connection to linear regression on Kronecker-product features makes the mechanism interpretable, and the detailed analysis of projection geometry, interference, and stochastic memory dynamics provides useful insight into the proposed associative memory.

  2. The authors evaluate several competitive recurrent architectures under a shared training setup, including both pure and hybrid models. KronLSTM achieves encouraging language-modeling results at two scales and improvements on several retrieval tasks. The write-rule ablations and token-matched context-extension experiments provide useful additional evidence beyond the main benchmark comparisons.

Critical weaknesses
  1. The connection between the theoretical guarantees and the implemented architecture is insufficiently established. Theorems 1-2 assume independent samples and a fixed step size, whereas Equation (10) uses input-dependent gates, channel-wise forgetting, and a decoupled write whose effective target is c_t/beta_t. Bounded write gates do not by themselves bound this effective target, and non-expansiveness of the homogeneous update does not establish the stated bound for the driven memory. The appendices retain the original assumptions. Moreover, an expected state-norm bound does not establish stable training gradients. A result covering the implemented recurrence, or more carefully restricted claims supported by memory and gradient measurements, would strengthen the paper.

  2. The interference and capacity comparisons do not consistently use the same retrieval objective. Equation (5) measures a scalar pair score, whereas DeltaNet is required to recover an entire value vector. Applying the Hebbian update in Equation (1) to the same scalar score also produces a product of key and value overlaps. These arguments therefore do not establish improved vector-retrieval capacity for the actual readout S_t q_t. Although Table A2 provides useful gate and normalization ablations, an update-only comparison within otherwise identical blocks, together with direct retention and overwrite tests, is needed to isolate the proposed mechanism's advantage.

  3. The evidence for practical long-context performance remains limited. The 32k extension compares only with GDN-2, and KronLSTM's accuracy on the five non-S1 tasks is only 3.0-11.3% at 32k despite its relative improvements. The real-world recall suite is evaluated at 2k, so it provides limited evidence for long-document use. Broader long-context baselines and realistic long-document tasks would clarify generalization. In addition, training throughput, decoding latency, and peak-memory measurements are missing, leaving the practical accuracy-efficiency trade-off of the new update insufficiently characterized.

Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Extending Theorem 2 for KronLSTM

We thank the reviewer for their thorough and constructive feedback. We respond to each point below. The revised manuscript includes substantial new material, highlighted in blue. To accommodate the new content within the page limit, we condensed several tables; if any of these changes reduce clarity, we are happy to adjust them in the final version.

W1: The appendices retain the original assumptions. Moreover, an expected state-norm bound does not establish stable training gradients. A result covering the implemented recurrence, or more carefully restricted claims supported by memory and gradient measurements, would strengthen the paper.

We thank the reviewer and agree that the connection between Theorem 2 and the implemented update (10) was not clearly established in the paper. As the reviewer notes, Theorems 1–2 are proved for the basic Kronecker Delta Rule (3), whereas KronLSTM (10) uses input-dependent gates, channel-wise forgetting, and a decoupled write. We now state explicitly which conclusions carry over to KronLSTM (see Appendix A2.6.7 ):

bounded write gates do not by themselves bound this effective target

(a) Decoupled write , c~t=ct/β\tilde c_t = c_t/\beta. For fixed β\beta and Ft=IF_t = I, (10) is rule (3) with target ct/βc_t/\beta, and Theorem 2 gives E∥St∥F2≤4E[c2]/(β2λmin⁡(Aˉ))\mathbb{E}\Vert S_t \Vert_F^2 \le 4 \mathbb{E}[c^2] / (\beta^2 \lambda_{\min}(\bar{A})). The bound is independent of sequence length but scales as β−2\beta^{-2}; a bounded write gate alone does not bound the effective target.

non-expansiveness of the homogeneous update does not establish the stated bound for the driven memory

(b) Input-dependent βt\beta_t. The bound additionally requires a lower bound on the erase gate. If βt\beta_t is drawn jointly i.i.d. with (kt,vt,ct)(k_t, v_t, c_t) and βt≥βmin⁡>0\beta_t \ge \beta_{\min} > 0, the proofs go through with β\beta replaced by βmin⁡\beta_{\min} (Eq. A118). The sigmoid gate does not enforce βmin⁡\beta_{\min} by construction, so this is an assumption.

(c) Channel-wise forget gate. Mentioned in the manuscript (Section 2.2) - the forget gate only contracts the homogeneous dynamics further.

(d) i.i.d. samples. We assume i.i.d. triples so that the current input is independent of the memory StS_t and expectations factorize, as is standard in LMS analysis. Non-expansiveness of each update holds for any input sequence; the expected-norm bound of Theorem 2 is proved only under this independence.

(e) By stable learning we mean that the memory does not grow without bound, which would make the read-out arbitrarily large and cause numerical instabilities during training. Theorem 2 gives the bound for the basic rule (3); for (10) it holds under the conditions in (a) and (b), and the gradient norms in the stress test of Figure 2 are consistent with it.

(f) By gradient flow we mean gradients propagated through the memory over time: the state-to-state Jacobian of (10), (I−βtatat⊤)(Ft⊗I)(I - \beta_t a_t a_t^\top)(F_t \otimes I), has spectral norm at most one for every input sequence and all gate values. These gradients are therefore never amplified, and they are attenuated only by the forget gate and along the pair being rewritten.

Contribution (iii) now separates these two points. We hope this clarifies how the theory carries over to KronLSTM, and are happy to further elaborate on any of the points above.

AuthorsOpenReview ↗

Update-only comparison: the Hebbian Rule, the Kronecker Delta Rule, and the Delta Rule

W2: These arguments therefore do not establish improved vector-retrieval capacity for the actual readout S_t q_t. Although Table A2 provides useful gate and normalization ablations, an update-only comparison within otherwise identical blocks, together with direct retention and overwrite tests, is needed to isolate the proposed mechanism's advantage.

We thank the reviewer for the insightful suggestion. We now dedicate Section 3.1 and Figure 4 to an experiment that can isolate where the Hebbian Rule, the Kronecker Delta Rule, and the Delta Rule differ on small-scale controlled tasks.

Setup. All three update rules (Hebbian, Delta, Kronecker Delta) are placed in the same KronLSTM block with the same projections, convolutions, l2-norms and readout; only the memory update rules change. All models are trained with the same recipe on MQAR-style tasks with 5 seeds (Appendix A4.1 describes all details).

We evaluate three tasks:

  • Capacity: we increase the number of distinct KV-pairs and track the accuracy of recalling all of them.
  • Interference: we add distractor KV-pairs around a fixed set of target pairs and track recall accuracy on the targets.
  • Overwrite: we repeatedly write new values to the same key and track which update rule recovers the most recent one.
Model (with decay)Capacity N=128 (train)Capacity N=256Interference D=64 (train)Interference D=1024Overwrite R=2 (train)Overwrite R=4
Hebbian Rule68.338.098.57.998.637.3
Delta Rule94.755.898.22.3100.099.9
Kronecker Delta Rule92.354.099.999.590.065.7

Results: For each task, we observe the following:

  • Capacity: all rules decline similarly as the number of KV-pairs grows, with only the Hebbian Rule with decay lagging the Kronecker Delta Rule and the Delta Rule.
  • Interference: the Kronecker Delta Rule’s minimally invasive memory updates keep it near perfect even at 64x the 64 training distractors, while the Hebbian Rule and the Delta Rule decline gradually.
  • Overwrite: the Delta Rule’s vector target makes it the only rule to extrapolate perfectly, with the Kronecker Delta Rule declining gradually as overwrites increase.

Conclusion. By testing the update rules on the three settings, we showed that while their raw capacity to retrieve values is similar, the way the Delta Rule and the Kronecker Delta Rule write into the memory differs, showing a tradeoff between being robust against distractor KV-pairs and being able to handle overwrites.

We thank the reviewer again for suggesting this useful experiment, as we believe it improved our manuscript and meaningfully highlights the benefits of the Kronecker update rule.

AuthorsOpenReview ↗

Extending the Experiments for Long Context

W3: The evidence for practical long-context performance remains limited. The 32k extension compares only with GDN-2, and KronLSTM's accuracy on the five non-S1 tasks is only 3.0-11.3% at 32k despite its relative improvements. The real-world recall suite is evaluated at 2k, so it provides limited evidence for long-document use. Broader long-context baselines and realistic long-document tasks would clarify generalization.

We appreciate the reviewer’s feedback on the extension of our long-context experiments. To provide further evidence, we provide new pretraining experiments on the Olmo-3 pre- and mid-training recipe at the same 100B token budget (details and full results in Appendix A3.5 ), with the final training phase extending the context to 32k again. We also included KDA as an additional baseline here.

We evaluate the models on the same language modeling and recall tasks. Additionally, since the new higher-quality training data and the mid-training phase contain curated long-context documents, we can extend our evaluation pipeline to LongBench (English subset), with tasks spanning single-and multi-doc QA, summarization, and code. A summary table is given below, with full results in Tables A7–9:

ModelMCQA Avg-6 ↑Recall Avg-6 ↑RULER Avg-6 @4k ↑@8k ↑@16k ↑@32k ↑LMB ppl ↓LongBench Avg-14 ↑
GDN-256.8938.8266.658.134.525.29.18828.6
KDA58.5539.2578.063.039.727.78.00129.8
KronLSTM59.4740.7977.963.746.228.27.92131.7

This additional pretraining experiment also shows the same trends as our initial experiments, namely that KronLSTM leads in the measured benchmarks, with the only exception of RULER at 4k, where it follows KDA by 0.1 points on average. On the new LongBench results, KronLSTM leads by 1.9 points on KDA and 3.1 points on GDN-2. We hope that this new set of pretraining runs, extended with new evaluations on LongBench, provides further evidence on the generalizability of our long-context claims in the paper.

We appreciate the feedback on broader evaluations for long context, and we believe that the addition of these new pretraining runs substantially strengthened our paper.

Reviewer_ZowyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Kronecker Delta Rule and the resulting KronLSTM architecture. Unlike DeltaNet, which corrects a vector residual toward Sk ≈ v, the proposed rule corrects a scalar residual toward vᵀSk ≈ c, modifying the matrix memory only along the direction vkᵀ. After vectorization, it is equivalent to a delta/LMS update on the Kronecker feature k ⊗ v. The authors analyze single-step error correction, interference with other key–value associations, and stationary distributions and expected memory-norm bounds under assumptions including i.i.d. sampling and a fixed step size. The implemented architecture further introduces a vector forget gate and decoupled erase and write gates.

Experiments cover 400M- and 1.5B-parameter models, with the latter trained on 100B tokens. The authors compare purely linear architectures and hybrids with sliding-window attention, evaluating language modeling, real-world recall, RULER retrieval, context extension to 32K, and state tracking. KronLSTM achieves strong average downstream performance and results on several retrieval tasks, with particularly clear advantages over GDN-2 on some long-range retrieval tasks after context extension. The current manuscript does not report measured comparisons of training or inference efficiency.

Critical strengths
  1. The modeling and retrieval results are promising, with evaluation across several settings. Under the shared training recipe, the purely linear 1.5B/100B KronLSTM achieves an Avg-6 score of 59.72, compared with 58.12 for KDA, the strongest purely linear comparator in Table 1. It also leads the mean real-world recall scores in both the purely linear and hybrid groups in Table 2. After context extension to 32K, its S3, MQ, and MV scores at 16K in Table A3 are 41.8, 21.5, and 30.1, respectively, compared with 7.4, 7.4, and 8.9 for GDN-2. Together with the 400M, hybrid, and state-tracking experiments, these results support the practical potential of the update rule. Its advantages have task-dependent limits, but these positive results deserve recognition.

  2. The update rule has a clear geometric interpretation and an analyzable interference structure. The scalar-association correction in the basic rule admits a relaxed-projection interpretation in matrix space. Equation (5) shows that changes to other scalar associations depend on both key overlap and value overlap. The appendix further analyzes minimal corrections, convergence, and norm bounds for randomized updates, while explicitly distinguishing scalar-association constraints from DeltaNet's vector-retrieval constraints. This offers a useful perspective for understanding and designing matrix memories.

Critical weaknesses
  1. The description of parallel computation for the implemented model and the efficiency evidence remain incomplete. Equation (4) and Equations (A16)–(A17) already provide vectorization and a product expansion for the basic rule and explain its connection to WY/chunkwise techniques. The reproducibility statement also indicates that the supplement provides parallel kernels that were numerically checked against serial reference implementations. However, the main text and appendix do not explicitly present the chunkwise state updates, within-chunk readouts, and state transfers between chunks corresponding to the actual recurrence in Equation (10), particularly how the vector forget gate and decoupled write term enter the computation. The basic-rule expansion and references to existing techniques are not sufficient to understand the implemented parallel algorithm and its computational cost on their own.

    The manuscript also lacks measured comparisons against the main baselines of training throughput, training time, and peak memory, as well as inference prefill time, per-token latency, throughput, and memory usage. The hardware, precision, and state sizes listed in the appendix do not substitute for these measurements. Since the proposed architecture targets long-context linear attention, establishing whether its quality advantages come at an acceptable computation and memory cost is necessary to assess its practical value fully.

  2. The dependence of the retrieval advantages on normalization choices, and their mechanistic attribution, require clarification. Table A2 removes value ℓ2 normalization only within KronLSTM, without showing its effect on applicable baselines. Its effects are also mixed: NIAH@4K is 59.5 with normalization and 39.6 without it, whereas removing it improves real-world Recall and NIAH@8K from 30.3 and 30.5 to 31.3 and 32.7, respectively. The query/key/value normalization settings of each method should therefore be specified, with suitable comparisons across methods to assess how much of KronLSTM's advantage comes from the core update rule and how much is associated with normalization and the accompanying training choices. Such ablations would help establish whether the retrieval gaps and relative rankings persist when applicable baselines use this component.

    The current hybrid uses 2048-token sliding-window attention (SWA). Its local access range and fixed-window cache objective provide context for interpreting the results. A useful supplementary experiment would replace a small number of SWA layers with full causal attention layers and evaluate retrieval, examining complementarity between recurrent memory and global attention, as detailed in Question 4. The authors already identify this direction as future work. It could broaden the evidence for applicability, while accounting for KV-cache and computation costs that grow with context length.

    The scope of the theoretical guarantees and the mechanistic explanation also require clarification: how the assumptions for the basic randomized update apply to the implemented gated architecture, and how reduced interference in scalar associations translates into better retrieval through vector readouts. These issues are addressed after the implementation and experimental questions below.

Questions for authors
  1. Parallel computation formulas: Please provide, in the main text or appendix, the parallel formulation corresponding exactly to the implemented serial recurrence in Equation (10). This should include state updates and output readouts within each chunk, state transfers between chunks, and how the vector forget gate F_t, input-dependent erase gate β_t, and separate write gate c_t enter the computation. Equation (4) and Equation (A17) explain vectorization and expansion of the basic rule, but the complete formulas for the actual architecture remain unclear. Does the implementation explicitly construct the d_k d_v-dimensional Kronecker features, or exploit their structure to avoid materialization? Since numerical checks against a serial reference are already mentioned, please also specify the error ranges and test configurations.

  2. Training and inference efficiency: Please compare the current KronLSTM implementation with the main baselines, such as GDN, KDA, GDN-2, and Mamba-3. Under comparable hardware, precision, model/state sizes, and training batch sizes, the training measurements should include at least tokens/s or step time and peak memory. Inference measurements should cover prefill time, per-token decoding latency or throughput, and actual peak GPU memory across context lengths and batch sizes. Please describe the kernels, degree of optimization, and timing boundaries, distinguishing kernel-level from end-to-end measurements. These results would establish whether the retrieval and modeling gains offer an attractive quality–cost trade-off.

  3. Value ℓ2 normalization across methods: Please specify each baseline's query, key, and value normalization settings. For the main delta-rule baselines where the modification is compatible with the original algorithm, could value ℓ2 normalization be enabled and disabled while controlling other architectural and training choices and specifying the gating or scale settings? Does using this component narrow the retrieval gap to KronLSTM or change the rankings? How do the authors explain its improvement to 4K/16K NIAH but not to real-world Recall or 8K NIAH in Table A2? If a baseline already uses the same component, please clarify this.

  4. Supplementary retrieval experiments with a small number of full attention layers: A useful extension would replace the SWA layers in the current hybrid with full attention at the same positions, initially retaining the existing ratio of linear to attention layers, and compare retrieval across purely linear models, hybrids with SWA, and hybrids with full attention. With the total layer count, comparable parameter scale, and training token budget held fixed, can a small number of global attention layers further improve long-range retrieval and complement KronLSTM's recurrent memory? Applying the same hybrid design to one strong, closely relevant baseline would help distinguish gains from full attention itself from the contribution of the Kronecker update. Please report RULER/recall results at multiple supported context lengths, together with the prefill, decoding latency, and memory changes discussed in Question 2. Full attention introduces a KV cache that grows with context length, and equal token budgets do not imply equal computation costs; this suggestion is intended to supplement the evaluation of retrieval quality versus resource use.

  5. Scope of the theoretical guarantees: Please state the conditions needed to apply Theorem 2 to the implemented Equation (10). How do the i.i.d. sampling and fixed step size in the basic analysis extend to input-dependent β_t, the effective target c_t/β_t associated with decoupled writing, the vector forget gate, and temporally correlated inputs? If the theorems primarily explain an idealized basic rule, please distinguish this from guarantees for the implemented model and qualify the statements about operating in the regime of Theorem 2, stable learning, and gradient behavior accordingly.

  6. Mechanism and failure cases: Could the Kronecker and vector delta updates be compared while keeping projections, convolutions, gates, and normalization fixed, measuring how subsequent writes affect the vector error of the actual readout Sq and task performance? The authors acknowledge that scalar associations and vector retrieval impose different constraints, making this control important for the explanation that reduced interference leads to better retrieval. Is there direct diagnostic evidence for residual accumulation with repeated keys as the explanation for the S1 results? Since some retrieval tasks still have low absolute scores at 32K after context extension, please also clarify which major limitations the current evidence can identify.

Thank you to the author for their efforts. If possible, please complete the answers to the questions above (the vast majority of which are the same as the ‘weakness’ section, so you need only answer them once), and if the results are satisfactory, I will increase my rating.

10 comments
AuthorsOpenReview ↗

Chunkwise/ parallel formulation

We thank the reviewer for their thorough and constructive feedback, and for their willingness to increase their rating in light of our responses. We address each point below; since the questions largely overlap with the weaknesses, we answer each of them once and cross-reference where relevant. The revised manuscript includes substantial new material, highlighted in blue. To accommodate the new content within the page limit, we condensed several tables; if any of these changes reduce clarity, we are happy to adjust them in the final version.

Q1: Please provide, in the main text or appendix, the parallel formulation corresponding exactly to the implemented serial recurrence in Equation (10).

We agree with the reviewer that we should have provided the parallel formulation. The revision adds Section 2.3 in the main text and a complete derivation in Appendix A5, written for the implemented recurrence (see Equation 10 for reference). This contains chunk-local calculations, the state handoff, readout calculations, how the vector forget gate is folded into the key dimension efficiently, and how the gates enter the computation. This addresses Weakness 1 and Q1 simultaneously. We provide a high-level overview here addressing all questions and refer to the manuscript for more detail:

Chunkwise form. Per chunk of CC tokens with incoming memory S0S_0, token rr writes urvrkr⊤u_r v_r k_r^\top, where the memory enters only through the scalar coefficient uru_r. The coefficients within a chunk depend on each other linearly and causally, so all CC of them come from one unit lower triangular solve:

u=(I+diag(β) (Gkk⊙Gvv))−1(c−β⊙p),pr=vr⊤S0 diag(γr) kr,u = \big(I + \mathrm{diag}(\beta)\,(G^{kk}\odot G^{vv})\big)^{-1}\big(c - \beta\odot p\big),\qquad p_r = v_r^\top S_0\,\mathrm{diag}(\gamma_r)\,k_r ,

with GkkG^{kk} the decay-weighted, strictly lower triangular key Gram and Gvv=VV⊤G^{vv} = VV^\top the value Gram.

Gating. As in KDA and GDN-2, the vector forget gate is folded into the keys through the cumulative decays γr=f1⊙⋯⊙fr\gamma_r = f_1 \odot \cdots \odot f_r, so it only appears as decayed keys and queries. The erase gate βt\beta_t scales the system matrix and the coupling to S0S_0, while the decoupled write ctc_t enters only the right-hand side and adds nothing to the solve.

Q1: Equation (4) and Equation (A17) explain vectorization and expansion of the basic rule, but the complete formulas for the actual architecture remain unclear. Does the implementation explicitly construct the dkdvd_k d_v-dimensional Kronecker features, or exploit their structure to avoid materialization?

By the mixed-product property, the Gram of the features kt⊗vtk_t \otimes v_t is Gkk⊙GvvG^{kk} \odot G^{vv}. Two C×CC \times C Grams of dkd_k- and dvd_v-dimensional vectors therefore replace one Gram of dkdvd_k d_v-dimensional vectors, and S0S_0 enters through the CC scalars pp. This avoids materializing the Kronecker features.

State handoff, readouts, and computational cost. Given uu, the state transfer to the next chunk and the within-chunk readouts follow the Delta Rule family (KDA, GDN-2), with diag(u)V\mathrm{diag}(u)V taking the role of the values. Relative to KDA, the only additional computation is the value Gram, so the overall order is unchanged (Appendix A5.5, Table A12).

Q1: Since numerical checks against a serial reference are already mentioned, please also specify the error ranges and test configurations.

Appendix A5.7 also reports error verifications for all major operations in fp64 and bf16, for both the training and inference setups, and shows that our kernels produce errors no larger than KDA and GDN kernels in flash-linear-attention on the same inputs.

AuthorsOpenReview ↗

Training and inference efficency

Q2: Please compare the current KronLSTM implementation with the main baselines, such as GDN, KDA, GDN-2, and Mamba-3. Under comparable hardware, precision, model/state sizes, and training batch sizes, the training measurements should include at least tokens/s or step time and peak memory. Inference measurements should cover prefill time, per-token decoding latency or throughput, and actual peak GPU memory across context lengths and batch sizes. Please describe the kernels, degree of optimization, and timing boundaries, distinguishing kernel-level from end-to-end measurements

We agree that providing kernel efficiency and throughput measurements is necessary, and we therefore extended our manuscript. In the revised version, we dedicate Section 3.6 to hardware efficiency, with Figure 7, and include protocol, peak memory (Figure A2), and a kernel-level breakdown (Table A11) in Appendix A4.5.

Setup: the 1.5B models of Table 1; our Triton kernels built on FLA and FLA’s GDN/KDA/GDN-2 kernels. We report both training and inference-stage metrics of our kernels, measured on the same hardware, precision and model hyperparameters. Training throughput at 2^{17} tokens up to 64k sequence length: KronLSTM is ~10% faster than GDN-2 and ~10% slower than KDA at all lengths. Peak memory is ~5% below GDN-2 and ~15% above GDN/KDA. Prefill (batch 1, 2k–64k): ~15% slower than GDN-2. Decode: within ~4% of GDN-2 at batch sizes 1–128. Table A11 reports kernel-level measurements, showing that the backward kernels for KronLSTM are about 35% faster than GDN-2, while the forward kernels are 15% slower. The source of the slower fwd is the additional value Gram and scalar scoring, but it is offset by a faster backward pass.

KronLSTM therefore stays within the efficiency range of the delta-rule family in both training and inference while delivering the gains on language modeling and recall. We note that the baseline kernels are the latest kernels of flash-linear-attention, which have been tuned over several releases, whereas the KronLSTM kernels are at the first iteration. We are looking forward to further kernel optimization as future work.

Training (k tokens/s per node), sequence length × batch

2k×644k×328k×1616k×832k×464k×2
GDN132.8132.4132.0132.0131.6128.4
KDA112.8113.2112.8112.8112.8109.6
GDN-291.291.291.291.691.289.6
KronLSTM100.8100.4100.4100.499.699.2
Prefill 2kPrefill 8kPrefill 32kDecode B=1Decode B=4Decode B=16Decode B=64
GDN14.255.7221.70.471.796.2318.15
KDA16.166.2240.90.441.695.8717.16
GDN-219.776.2296.60.351.364.8314.75
KronLSTM22.888.6344.70.341.314.6214.07

Prefill: time to first token (ms) at batch size 1, by prompt length (lower is better). Decode: throughput (k tokens/s), by batch size B (higher is better).

Prefill 2kPrefill 8kPrefill 32kDecode B=1Decode B=4Decode B=16Decode B=64Training
GDN2.783.124.492.672.753.054.2847.0
KDA2.673.124.912.532.602.914.1448.6
GDN-23.073.595.652.902.983.284.5157.7
KronLSTM2.863.385.452.692.773.084.3354.8

Peak GPU memory (GB). Prefill: at batch size 1, by prompt length. Decode: by batch size B. Training: allocated memory at 128k tokens.

We thank the reviewer for the questions and are willing to further discuss if there are any points that need to be addressed.

AuthorsOpenReview ↗

Extending Experiments with the KronLSTM Layer Structure

We appreciate the questions about the layer structures and the effect on the models' performance.

Q3: Please specify each baseline's query, key, and value normalization settings + the effect of the ℓ2 normalization

Below, we provide a table outlining each baseline's block structure. KronLSTM and the Delta Rule family models share a similar block structure, with the value l2-norm being the main differentiator.

ModelQK-normV-normShort conv. (q, k, v)SiLU after short conv.Output gateOutput norm placementForget gate parametrization
GDNℓ2\ell_2×✓✓SiLU: RMSNorm(ht)⊙SiLU(Wgxt)\mathrm{RMSNorm}(h_t) \odot \mathrm{SiLU}(W_g x_t)norm → gatescalar: exp⁡(−a,softplus(⋅))\exp(-a, \mathrm{softplus}(\cdot))
Mamba-3 (SISO)RMSNorm on B,CB, C××—SiLU: yt⊙SiLU(Wzxt)y_t \odot \mathrm{SiLU}(W_z x_t)none (gate->norm in hybrids)scalar: exp⁡(softplus(⋅)At)\exp(\mathrm{softplus}(\cdot) A_t)
KDAℓ2\ell_2×✓✓sigmoid: RMSNorm(ht)⊙σ(Wgxt)\mathrm{RMSNorm}(h_t) \odot \sigma(W_g x_t)norm → gatechannel-wise exp⁡(−a,softplus(⋅))\exp(-a, \mathrm{softplus}(\cdot))
GDN-2ℓ2\ell_2×✓✓SiLU: RMSNorm(ht)⊙SiLU(Wgxt)\mathrm{RMSNorm}(h_t) \odot \mathrm{SiLU}(W_g x_t)norm → gatechannel-wise: exp⁡(−a,softplus(⋅))\exp(-a, \mathrm{softplus}(\cdot))
KronLSTMℓ2\ell_2ℓ2\ell_2✓✓sigmoid: RMSNorm(ht)⊙σ(Woxt)\mathrm{RMSNorm}(h_t) \odot \sigma(W_o x_t)norm → gatechannel-wise: σ(⋅)a=exp⁡(−a,softplus(⋅))\sigma(\cdot)^a = \exp(-a, \mathrm{softplus}(\cdot))

We retrained KDA and GDN-2 in the KronLSTM block at the same 1.5B parameter scale on 100B FineWeb tokens so they are comparable to the baselines. We report performance next to their baselines.

ModelMCQA Avg-6 ↑Recall Avg-6 ↑RULER Avg-6 @4k ↑@8k ↑@16k ↑LMB ppl ↓
GDN-257.9429.3952.735.421.212.542
GDN-2 (KronLSTM-block)58.2329.8456.131.319.212.493
KDA58.1229.1743.331.421.712.669
KDA (KronLSTM-block)58.1229.2646.830.223.312.405

The results show that the KronLSTM-block provides minimal gains at best to both GDN-2 and KDA in language modeling and recall. On RULER, the improvements are scattered with no clear direction; for GDN-2 the KronLSTM block structure helps at 4k but hurts performance on 8k and 16k, and for KDA it helps on 4k and 16k, but hurts performance slightly at 8k.

Q3: Does using this component narrow the retrieval gap to KronLSTM or change the rankings?

Rankings remain unchanged between the methods. The results match our ablation on KronLSTM itself (Table A2): the value ℓ2\ell_2-norm improves language modeling but gives a minor degradation in recall, with no consistent effect on RULER-NIAH, where there is no clear direction of improvement. Since the same pattern appears in GDN-2 and KDA, it is a property of the value $\ell_2-norm and not specific to KronLSTM. The retrieval gap is therefore not explained by this component, as adding it to the baselines does not close the gap to KronLSTM.

ModelLen.S1S2S3MKMQMV
GDN-24k100.072.850.226.628.438.1
GDN-2 (KronLSTM block)4k99.897.033.638.227.640.8
GDN-28k99.844.615.417.613.421.4
GDN-2 (KronLSTM block)8k93.045.25.023.07.813.6
GDN-216k99.68.24.27.22.75.1
GDN-2 (KronLSTM block)16k81.013.01.611.61.86.3
KDA4k99.889.412.020.818.419.1
KDA (KronLSTM block)4k100.087.227.022.222.322.2
KDA8k99.035.411.615.813.313.3
KDA (KronLSTM block)8k99.431.88.819.211.710.2
KDA16k93.614.21.611.83.15.7
KDA (KronLSTM block)16k97.219.23.416.20.83.2
KronLSTM4k66.298.273.428.247.243.3
KronLSTM8k29.460.420.423.424.824.6
KronLSTM16k11.616.011.816.012.717.6

Comments on the paper

1 comment

We thank the reviewers for reading our manuscript and for acknowledging the strengths of our method in language modelling. Moreover, we thank the reviewers for pointing out some useful weaknesses, which, we believe, by addressing them strengthened our paper.

We would like to summarize our improvements:

  • Extending Theorem 2 from the Kronecker Delta Rule to KronLSTM (Appendix A2.6.7), mentioned by reviewers 1fWh, 4vit, and Zowy;
  • Providing the chunkwise formulation (Section 2.3, Appendix A5) and doing the inference/training comparisons (Section 3.6, Appendix A4.5), mentioned by reviewers 1fWh, 4vit, and Zowy;
  • Investigating the differences between the Hebbian, Delta, and Kronecker Delta Rule (Section 3.1, Figure 4), mentioned by reviewers 1fWh, 4vit, and Zowy, and the importance of the KronLSTM block structure, mentioned by reviewer Zowy;
  • The thorough investigation of the RULER results and the experimental extension for long context (Appendix A3.5), mentioned by reviewers 1fWh, 4vit, and Zowy;
  • Repeating the 400M parameter experiments with 3 seeds to report variance (Table A1), mentioned by reviewer 4vit.

In addition to the points above, reviewer Zowy proposed extending our method to softmax hybrids (Appendix A3.4), which further highlights the applicability of our proposed linear attention variant. Furthermore, we appreciate reviewer Zowy’s willingness to increase their assessment.

As the rebuttal period is coming to an end, we hope we have addressed all of the reviewers’ concerns, and we remain available to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.