- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces KronLSTM, a linear attention architecture based on the Kronecker Delta Rule. Unlike traditional delta-rule vector corrections, this method applies scalar association correction along a single direction in the matrix space, aiming to minimize interference with previously stored information for long-context language modeling. The authors provide theoretical proofs for the steady-state Markov chain of the memory dynamics and bounded memory norms, and conduct experiments on language modeling, retrieval, and long-context evaluation.
Critical strengths
-
The paper shifts the linear attention learning objective from full value vector regression (used in the DeltaNet family) to a scalar association correction. This mechanism distinguishes itself from existing works like Kimi Delta Attention (KDA) and establishes a clear orthogonal protection property, avoiding interference if either the keys or values are orthogonal.
-
The authors trained models at a robust 1.5B parameter and 100B token scale. The evaluation is comprehensive, comparing against KDA, GDN-2, and Mamba-3. It assesses not only pure linear models but also hybrid architectures with sliding-window attention, alongside ablation studies and 32k context extensions.
-
The benefits of KronLSTM are not limited to a single task. In Table 1, the pure linear model's average accuracy across five commonsense reasoning tasks and LAMBADA improves from KDA's 58.12 to 59.72. Furthermore, LAMBADA perplexity drops significantly from 12.669 to 11.716, which is a noteworthy result.
Critical weaknesses
-
The theoretical analysis (Theorems 1 and 2) assumes i.i.d. inputs and a fixed step size . However, the actual model implements an input-dependent forget gate (), a time-varying , and a decoupled write mechanism. The non-expansive nature of the homogeneous state transition is insufficient to guarantee a bounded memory norm when continuous decoupled writing is applied. The original norm bounds need to be re-evaluated since the effective target becomes after decoupling. The authors must provide derivations applicable to the actual gated updates or narrow their theoretical claims.
-
Vector correction and scalar correction impose entirely different task constraints. While scalar correction preserves more components, it may also inadvertently retain old information that should have been overwritten. Comparing capacity ( scalar associations vs. full value vectors) requires caution. The most valuable supplementary experiment would be to fix the network structure and only swap the correction target (scalar vs. vector) to test repeated key overwriting and value co-existence.
-
According to Table A3, after 32k context extension training, KronLSTM's absolute performance on 32k tasks remains quite weak (e.g., S3 accuracy at 8.6, MK at 5.0, MQ at 3.0, and MV at 11.3). Additionally, the pure linear model suffers a severe performance drop on the 16k S1 task, achieving only 11.6 accuracy compared to GDN's 99.2. This warrants a rigorous explanation and cannot be solely dismissed by stating the task is unrealistic for real-world text.
-
Although the paper claims to provide chunkwise-parallel kernels, it fails to report actual training throughput, decoding latency, memory footprint, or practical computational overhead. Mathematical compatibility with the vectorized Delta Rule does not guarantee identical hardware costs. Furthermore, the main language modeling experiments lack variance reporting across different training seeds.
Questions for authors
While the paper claims to provide chunkwise-parallel kernels, it lacks comprehensive end-to-end efficiency tests. The authors must report actual training throughput (e.g., tokens per second), decoding latency, and wall-clock time comparisons against the baselines. Furthermore, there is a strong suspicion that the scalar key-value pair erasure and update mechanism might negatively impact actual hardware training efficiency. To alleviate this concern, the paper needs to report the loss reduction ratio under a strictly matched computational budget (e.g., iso-FLOPs or iso-wall-clock time), proving that the theoretical benefits of the Kronecker Delta Rule translate into practical, efficient training without severe computational overhead.
5 comments
Comparing the correction target: Kronecker Delta Rule and Delta Rule
W2: The most valuable supplementary experiment would be to fix the network structure and only swap the correction target (scalar vs. vector) to test repeated key overwriting and value co-existence.
We thank the reviewer for the insightful suggestion. We now dedicate Section 3.1 and Figure 4 to investigating the differences between the update rules in small-scale, controlled settings. We would like to highlight that the scalar vs. vector correction target comparison is equivalent to comparing the Kronecker Delta Rule, which corrects the scalar towards , with the Delta Rule, which corrects the vector towards .
Setup: In Section 3.1, both the Delta Rule and Kronecker Delta Rule use the same KronLSTM block with the same projections, convolutions, l2-norms and readout, only the memory update rules change. We experiment both with and without decay gates. All models are trained with the same recipe on MQAR-style tasks with 5 seeds (Appendix A4.1 describes all details).
We test on three tasks:
- Capacity: We test how many distinct KV-pairs the model can recall. We gradually increase the number of KV pairs and track accuracy of all of them being recalled from memory.
- Interference: How distractor KV-pairs affect recall of a fixed set of target KV-pairs. We keep the targets fixed, grow the number of distractors written after them, and track target recall accuracy. With 4095 keys in dk=64 dimensions per head, distractor keys necessarily overlap the target keys, so every distractor write acts on the key directions that hold the targets. This tests which update rule retains the target associations while other values co-exist under similar keys.
- Overwrite: New values are continuously being written to the same key, with the latest value being the correct solution, and older values should be erased from memory. We gradually increase the number of new writes and track which update rule can recover the newest value. This aims to answer which update rule holds on to old values that are no longer necessary.
| Model (with decay) | Capacity N=128 (train) | Capacity N=256 | Interference D=64 (train) | Interference D=1024 | Overwrite R=2 (train) | Overwrite R=4 |
|---|---|---|---|---|---|---|
| Delta Rule | 94.7 | 55.8 | 98.2 | 2.3 | 100.0 | 99.9 |
| Kronecker Delta Rule | 92.3 | 54.0 | 99.9 | 99.5 | 90.0 | 65.7 |
Results: For each task, we observe the following:
- Capacity: Both the Kronecker Delta Rule and the Delta Rule have similar capacity. They all solve the task perfectly within the training distribution and then gradually decline as the number of KV-pairs is increased.
- Interference: This is where the Kronecker Delta Rule's minimally invasive memory updates pay off. Within the training setting of 64 distractors, all update rules solve the task perfectly, however, once extrapolated to more distractors with more key overlap, the Delta Rule gradually decreases in accuracy while the Kronecker Delta Rule maintains a near-perfect score.
- Overwrite: In the overwriting setting, the Delta Rule's vector target advantage shows. While both update rules solve the task in distribution, only the Delta Rule can show perfect extrapolation, with the Kronecker Delta Rule gradually declining in accuracy with more overwrites.
Conclusion. With the above three settings, we showed that while their raw capacity to retrieve values is similar, the way the Delta Rule and the Kronecker Delta Rule write into the memory differs, showing a trade-off between being robust against distractor KV-pairs and key/value co-existence, or being able to handle overwrites to new values. We thank the reviewer again for this useful experiment, as we believe it improved our manuscript and clearly highlights the benefits of our proposed update rule.
Extending the Experiments for Long Context
W3: According to Table A3, after 32k context extension training, KronLSTM's absolute performance on 32k tasks remains quite weak (e.g., S3 accuracy at 8.6, MK at 5.0, MQ at 3.0, and MV at 11.3).
We appreciate the reviewer’s feedback on the overall weak performance at 32k. We present two arguments and additional experiments to alleviate the concerns: (1) on the data-mixture and long-context recipe and (2) overall limitations of the model size and token budget.
(1) The long-context models were first trained on FineWeb-Edu, then extended to 32k context using a different data corpus, ProLong. While this already substantially improved performance, it is not the most optimal data corpus for long-context training.
To address this, we conducted new pretraining experiments on the Olmo-3 pre- and mid-training recipe at the same 100B token budget (details and full results in Appendix A3.5), with the final training phase extending the context to 32k again. In Olmo-3 the datasets were engineered to be compatible and provide better long-context performance. Moreover, we included KDA as an additional baseline.
We evaluate the models on the same language modeling and recall tasks. Additionally, since the new higher-quality training data and the mid-training phase contain curated long-context documents, we can extend our evaluation pipeline to LongBench (English subset), with tasks spanning single-and multi-doc QA, summarization and code. A summary table is given below, with full results in Tables A7–9:
| Model | MCQA Avg-6 ↑ | Recall Avg-6 ↑ | RULER Avg-6 @4k ↑ | @8k ↑ | @16k ↑ | @32k ↑ | LMB ppl ↓ | LongBench Avg-14 ↑ |
|---|---|---|---|---|---|---|---|---|
| GDN-2 | 56.89 | 38.82 | 66.6 | 58.1 | 34.5 | 25.2 | 9.188 | 28.6 |
| KDA | 58.55 | 39.25 | 78.0 | 63.0 | 39.7 | 27.7 | 8.001 | 29.8 |
| KronLSTM | 59.47 | 40.79 | 77.9 | 63.7 | 46.2 | 28.2 | 7.921 | 31.7 |
Summary. This additional pretraining experiment shows the same trends as our initial experiments, namely that KronLSTM leads in the measured benchmarks, with the only exception of RULER at 4k where it follows KDA by 0.1 points on average. On the new LongBench results, KronLSTM leads by 2 points on KDA and over 3 points on GDN-2.
(2) Addressing the overall weak 32k performance. We understand that the above results - while providing improved performance - do still deteriorate over longer contexts. This is a limitation of the model size (1.5B for our original models, 1.7B for the new Olmo-recipe pretrained ones) and the training budget of 100B tokens across all architectures. Larger models and token budgets can alleviate these in general across all architectures; however, we currently do not have the resources to pretrain at an even larger scale.
We thoroughly appreciated the feedback on our long context performance, and we believe that the addition of these new pretraining runs substantially strengthened our paper.
W3: Additionally, the pure linear model suffers a severe performance drop on the 16k S1 task, achieving only 11.6 accuracy compared to GDN's 99.2.
We investigated the drop in the NIAH-S1 scores thoroughly, and we appreciate the reviewers' suggestion to conduct more thorough experiments. We present evidence that the NIAH-S1 result in Table 3 is specific to the KronLSTM pretrained only on FineWeb Edu at 8k context and is not a failure mode of the Kronecker Delta Rule.
In the synthetic setup of Section 3.1, as used above, we built an S1-style haystack. 32 target pairs are placed at a random depth inside D distractor writes that repeat a fixed pool of 4 pairs, akin to the repeated filler sentences of NIAH-S1 in a controlled, small-scale synthetic setting. Models are trained with groups of 4–6 distractors and up to D=64, and evaluated up to D=4096, where each pair is written about 1000 times.
| Accuracy (%) | |||
|---|---|---|---|
| Delta Rule | 100 | 100 | 99 |
| Kronecker Delta Rule | 100 | 100 | 100 |
Our controlled experiments show that the Kronecker Delta Rule can solve NIAH-S1 style tasks with perfect extrapolation.
Furthermore, our long-context results (Figure 5) and the newly presented pretraining using the Olmo-3 recipe (Table A8, Figure A1, Recipe described in Appendix A3.5) show that KronLSTM performs near-perfectly on NIAH-S1. From a single run, we cannot determine whether the data, the context length, or run-to-run variance causes it.
Extending Theorem 2 for KronLSTM
We thank the reviewer for their thorough and valuable feedback. We respond to each point below. The revised manuscript includes substantial new material, highlighted in blue. To accommodate the new content within the page limit, we condensed several tables; if any of these changes reduce clarity, we are happy to adjust them in the final version.
Theorems 1–2 are proved for the basic Kronecker Delta Rule (3) with i.i.d. triples, a fixed step size, and no forget gate. Appendix A2.6.7 now extends Theorem 2 to the decoupled write and the time-varying βt of KronLSTM (10):
(a) Decoupled write. For fixed β and Ft=I, (10) is rule (3) with effective target ct/β, and Theorem 2 gives E∥St∥F2≤4E[c2]/(β2λmin(Aˉ)). The re-evaluated bound stays independent of sequence length and scales as β−2.
(b) Time-varying βt. For βt drawn jointly i.i.d. with (kt,vt,ct) and βt≥βmin>0, the proofs hold with β replaced by βmin (Eq. A118). This lower bound on the erase gate, together with non-expansiveness, bounds the memory under decoupled writing.
(c) Forget gate Ft. As stated in Section 2.2, the gate only contracts the homogeneous dynamics further.
(d) I.i.d. inputs. Independence of the current input from the memory St lets expectations factorize, as is standard in LMS analysis. The expected-norm bound holds under this assumption.
The revised manuscript separates these guarantees from the empirical claims. We hope this clarifies how the analysis carries over to KronLSTM, and we are willing to further elaborate on any of the points above should any of the points still not be addressed.