- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Kalman Delta Networks (KDNs), which modify delta-rule memory updates by explicitly tracking uncertainty. Instead of directly predicting a write gate, KDN derives the update gain from accumulated memory uncertainty and learned observation noise. The authors develop diagonal and isotropic approximations with efficient chunkwise implementations, and evaluate them through controlled pretraining at 750M and 1.3B parameters.
Critical strengths
-
Could you report RULER results for the existing μ = 1 and μ = d_k ablations, alongside KDA? How much do the retrieval gains depend on information scaling?
-
How does KDN compare with Preconditioned DeltaNet under the same training setup, particularly in retrieval accuracy and computational cost?
Critical weaknesses
-
The source of the retrieval gains is not fully clear. Information scaling noticeably affects performance, but its ablation does not include RULER, where some of the strongest improvements are reported.
-
The advantage over closely related methods remains unclear. The paper discusses Preconditioned DeltaNet but does not directly compare against it, leaving uncertainty about the benefit of the proposed uncertainty recursion over other history-based preconditioning rules.
Questions for authors
-
Could you report RULER results for the existing μ = 1 and μ = d_k ablations, alongside KDA? How much do the retrieval gains depend on information scaling?
-
How does KDN compare with Preconditioned DeltaNet under the same training setup, particularly in retrieval accuracy and computational cost?
2 comments
Thank you!
Dear Reviewer xaJW,
We would like to kindly follow up on our rebuttal and ask whether our responses and additional experiments have adequately addressed your questions. We would greatly appreciate any further feedback and would be happy to answer any follow-up questions you may have.
Thank you again for your time and thoughtful review!
Response to Reviewer xaJW
We thank the reviewer for the thoughtful review and constructive feedback. We have revised the paper accordingly, with all changes marked in blue, and address each concern below. Table, figure and equation numbers refer to the revised PDF.
W1 + Q1. RULER results for the information-scale ablation
The retrieval gains depend little on information scaling. At μ=1, Diagonal KDN reaches 69.0 on the 14-cell NIAH mean, compared with 70.3 at μ=dk and 62.1 for KDA.
We evaluated the μ=1 checkpoint from Table 5 with the RULER protocol of Table 2 (14 cells, 500 samples each):
Each task column is the mean over context lengths.
Additionally, we have added an experiment and discussion of the uncertainty propagation ablation (Section 5.3) to clarify its contribution to the observed performance gains.
W2 + Q2. Comparison with Preconditioned DeltaNet
We added Preconditioned KDA (Tumma et al., 2026) under the same training setup. Diagonal KDN leads it by 8.1 NIAH points, with lower perplexity and comparable accuracy, at 0.92x its training throughput.
We trained Preconditioned KDA at 750M/50B with the same backbone, recipe and data order, using its released implementation. We additionally compare with GKA (Gated KalmaNet), another history-based preconditioning rule, which keeps a dense dk×dk Gram matrix of past keys per head. We also added a dedicated section comparing KDN with GKA and KLA to Appendix B (Additional Related Work).
Throughput is a full training step at 4K on one H200; memory is peak memory at 8K (Table 10).
The methods differ in how the gain is formed. Preconditioned DeltaNet combines a diagonal curvature estimate with token-predicted gains, and GKA solves a ridge regression over past keys, whereas KDN derives the gain from propagated uncertainty. Replacing KDN's gain with a token-predicted one lowers NIAH from 70.3 to 63.7 (Table 4).
We thank the reviewer again for the detailed feedback. Please let us know whether we have addressed your concerns adequately; we are happy to answer any follow-up questions.