acceptodds
Under review as a conference paper at ICLR 2027

Kalman Delta Networks: Uncertainty-aware Associative Memory

Abstract

Linear attention enables efficient long-context inference by compressing token history into a fixed-size recurrent memory. This compression makes each update a trade-off between incorporating new information and preserving useful associations. Models such as DeltaNet, Gated DeltaNet, and KDA predict write strength from the current token representation, without explicitly tracking uncertainty in the stored memory. Yet this uncertainty matters: a new observation should have greater influence when the existing association is uncertain and less when it is already well supported. We introduce Kalman Delta Networks (KDNs), a family of linear-attention models that explicitly track memory uncertainty to guide each update. By formulating associative memory as a linear–Gaussian state-space model, KDNs propagate both the memory estimate and its uncertainty, using the Kalman gain to balance accumulated evidence against the reliability of new observations. This formulation also recovers standard delta-rule updates by replacing tracked covariance with a token-predicted isotropic surrogate. To support hardware-efficient training and inference, we derive Diagonal KDN and Isotropic KDN, which retain one uncertainty value per key channel and per head, respectively. Their uncertainty updates admit associative scans with logarithmic parallel depth, requiring only and auxiliary state per head. Across controlled pretraining at 750M and 1.3B parameters, both variants consistently improve perplexity and mean downstream accuracy over the evaluated state-of-the-art linear-attention baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, state-space models, kalman filter, associative memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xaJWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Kalman Delta Networks (KDNs), which modify delta-rule memory updates by explicitly tracking uncertainty. Instead of directly predicting a write gate, KDN derives the update gain from accumulated memory uncertainty and learned observation noise. The authors develop diagonal and isotropic approximations with efficient chunkwise implementations, and evaluate them through controlled pretraining at 750M and 1.3B parameters.

Critical strengths
  1. Could you report RULER results for the existing μ = 1 and μ = d_k ablations, alongside KDA? How much do the retrieval gains depend on information scaling?

  2. How does KDN compare with Preconditioned DeltaNet under the same training setup, particularly in retrieval accuracy and computational cost?

Critical weaknesses
  1. The source of the retrieval gains is not fully clear. Information scaling noticeably affects performance, but its ablation does not include RULER, where some of the strongest improvements are reported.

  2. The advantage over closely related methods remains unclear. The paper discusses Preconditioned DeltaNet but does not directly compare against it, leaving uncertainty about the benefit of the proposed uncertainty recursion over other history-based preconditioning rules.

Questions for authors
  1. Could you report RULER results for the existing μ = 1 and μ = d_k ablations, alongside KDA? How much do the retrieval gains depend on information scaling?

  2. How does KDN compare with Preconditioned DeltaNet under the same training setup, particularly in retrieval accuracy and computational cost?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xaJW

We thank the reviewer for the thoughtful review and constructive feedback. We have revised the paper accordingly, with all changes marked in blue, and address each concern below. Table, figure and equation numbers refer to the revised PDF.

W1 + Q1. RULER results for the information-scale ablation

The retrieval gains depend little on information scaling. At μ=1\mu=1, Diagonal KDN reaches 69.0 on the 14-cell NIAH mean, compared with 70.3 at μ=dk\mu=d_k and 62.1 for KDA.

We evaluated the μ=1\mu=1 checkpoint from Table 5 with the RULER protocol of Table 2 (14 cells, 500 samples each):

ModelS-NIAH-1S-NIAH-2S-NIAH-3MK-NIAH-1Mean
KDA95.468.942.028.962.1
Diagonal KDN, μ=1\mu=199.875.559.229.369.0
Diagonal KDN, μ=dk\mu=d_k100.072.662.535.770.3

Each task column is the mean over context lengths.

Additionally, we have added an experiment and discussion of the uncertainty propagation ablation (Section 5.3) to clarify its contribution to the observed performance gains.

W2 + Q2. Comparison with Preconditioned DeltaNet

We added Preconditioned KDA (Tumma et al., 2026) under the same training setup. Diagonal KDN leads it by 8.1 NIAH points, with lower perplexity and comparable accuracy, at 0.92x its training throughput.

We trained Preconditioned KDA at 750M/50B with the same backbone, recipe and data order, using its released implementation. We additionally compare with GKA (Gated KalmaNet), another history-based preconditioning rule, which keeps a dense dk×dkd_k\times d_k Gram matrix of past keys per head. We also added a dedicated section comparing KDN with GKA and KLA to Appendix B (Additional Related Work).

ModelWiki. PPLLMB. PPLAvg.NIAHThroughput (K tok/s)Memory (GiB)
Preconditioned KDA18.7715.0754.8562.257.186
GKA19.4718.2653.0260.555.680
Isotropic KDN18.4214.6854.4162.465.972
Diagonal KDN18.6414.1554.9770.352.6102

Throughput is a full training step at 4K on one H200; memory is peak memory at 8K (Table 10).

  • Retrieval. Diagonal KDN leads GKA by 9.8 NIAH points. Its gap to Preconditioned KDA is largest at long context: S-NIAH-1 at 8K is 100.0 vs. 67.4, and S-NIAH-3 at 4K is 32.2 vs. 18.6.
  • Language modeling. Both KDN variants have lower perplexity than both baselines. Average accuracy is comparable to Preconditioned KDA's and higher than GKA's.
  • Cost. Isotropic KDN trains 1.15x faster than Preconditioned KDA and 1.19x faster than GKA, with less memory. Diagonal KDN trains at 0.92x and 0.95x their throughput and uses more memory.

The methods differ in how the gain is formed. Preconditioned DeltaNet combines a diagonal curvature estimate with token-predicted gains, and GKA solves a ridge regression over past keys, whereas KDN derives the gain from propagated uncertainty. Replacing KDN's gain with a token-predicted one lowers NIAH from 70.3 to 63.7 (Table 4).

We thank the reviewer again for the detailed feedback. Please let us know whether we have addressed your concerns adequately; we are happy to answer any follow-up questions.

AuthorsOpenReview ↗

Thank you!

Dear Reviewer xaJW,

We would like to kindly follow up on our rebuttal and ask whether our responses and additional experiments have adequately addressed your questions. We would greatly appreciate any further feedback and would be happy to answer any follow-up questions you may have.

Thank you again for your time and thoughtful review!

Reviewer_wiiCOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Kalman Delta Networks (KDNs), a family of uncertainty-aware recurrent linear-attention models. The key idea is to reinterpret delta-rule associative memory as a linear-Gaussian state-space model, where the recurrent memory is treated as a latent key-value map and each token provides a noisy observation of that map. Under this view, the Kalman gain naturally determines how strongly new information should update the memory based on both the current observation noise and the accumulated uncertainty of the stored association.

Since exact Kalman filtering requires maintaining a dense covariance matrix, the authors derive two efficient approximations: Diagonal KDN, which tracks one uncertainty value per key channel, and Isotropic KDN, which tracks a single uncertainty scalar per head. Both variants are designed to remain compatible with associative scans and efficient parallel training. The models are evaluated at 750M and 1.3B parameter scales on language modeling, zero-shot reasoning, and retrieval tasks, where they consistently improve over several recent recurrent and linear-attention baselines.

Critical strengths
  1. The paper provides a principled and useful new perspective on delta-rule associative memory. Recasting the residual write as a Kalman update makes memory uncertainty an explicit state variable and gives a coherent explanation for how write strength should depend on accumulated evidence rather than only on the current token.

  2. The paper closes the gap between the probabilistic formulation and a practical large-scale architecture. The diagonal and isotropic approximations preserve scan-compatible training with only O(dk)O(d_k) or O(1)O(1) additional uncertainty state, and the resulting models show consistent gains over strong recent recurrent baselines at both 750M and 1.3B scales.

Critical weaknesses
  1. The practical KDN variants are fairly strong approximations to the exact Kalman formulation. In particular, Diagonal KDN discards cross-channel covariance and requires an additional information-scaling factor to compensate for excessive overwrite. It is therefore not entirely clear how much of the empirical gain comes specifically from principled uncertainty propagation versus the approximation and calibration choices introduced to make it practical.

  2. The positioning against the closest contemporaneous Bayesian/Kalman sequence models could be stronger. KLA and Bayesian Layers are discussed qualitatively, but are not included in the main empirical comparisons. This makes it somewhat difficult to assess the empirical advantage of KDN relative to the most closely related uncertainty-aware sequence models.

Questions for authors
  1. Can the authors provide a more direct ablation isolating the benefit of propagated memory uncertainty, for example by replacing the covariance-derived gain with a matched token-predicted or fixed-covariance gain while keeping the architecture and state budget unchanged?

  2. Could the authors clarify the practical and empirical differences from KLA and Bayesian Layers, ideally with a matched comparison if feasible? In particular, how do these approaches compare in accuracy, auxiliary-state cost, and throughput under similar model scales?

  3. The variational interpretation is exact for the diagonal update with μ=1\mu=1, whereas the practical model uses information scaling μ>1\mu>1, which changes the covariance update without changing the current Kalman gain. Is there a coherent probabilistic interpretation of this scaled update, or should the resulting covariance state be viewed primarily as a memory-protection statistic rather than posterior uncertainty?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wiiC [1/2]

We thank the reviewer for the thoughtful review and constructive feedback. We have revised the paper accordingly, with all changes marked in blue, and address each concern below. Table, figure and equation numbers refer to the revised PDF.

W1 + Q1. Isolating the benefit of propagated uncertainty

We added this ablation (new Table 4, 750M/50B). With the architecture and state budget unchanged, replacing the propagated covariance with a fixed one, or the Kalman gain with a token-predicted gate, degrades every metric.

  • No carry keeps Diagonal KDN's architecture and the networks that predict ωt\omega_t and rtr_t, but drops the history term of the covariance prediction p^t=αt2⊙pt−1+ωt\widehat p_t=\alpha_t^2\odot p_{t-1}+\omega_t, setting p^t=ωt\widehat p_t=\omega_t. The gain keeps its Kalman form without accumulated evidence: κt=ωt⊙kt/(rt+∑iωt,ikt,i2)\kappa_t=\omega_t\odot k_t/(r_t+\sum_i\omega_{t,i}k_{t,i}^2).
  • Sigmoid gate replaces the gain with a GDN-2-style channel-wise gate, κt=bt⊙kt\kappa_t=b_t\odot k_t, where bt∈(0,1)dkb_t\in(0,1)^{d_k} is a sigmoid of the same projections that produce ωt\omega_t and rtr_t.
ModelWiki. PPLLMB. PPLAvg.NIAH
Diagonal KDN18.6414.1554.9770.3
no carry (reset)18.7414.3254.5066.9
sigmoid gate18.7314.6154.5863.7

The reset model is on par with the sigmoid gate on WikiText and accuracy, so the improvement comes from propagated uncertainty rather than from the gain's functional form.

For Isotropic KDN, KDA is the corresponding control: it has the same transition but uses a token gate instead of the propagated-uncertainty gain. Isotropic KDN improves on it on all four metrics:

ModelWiki. PPLLMB. PPLAvg.NIAH
Isotropic KDN18.4214.6854.4162.4
KDA18.8515.0653.8762.1

Approximation and calibration. Table 4 holds the architecture and diagonal state fixed, so the drop isolates propagation rather than the approximation. The gains also do not come from the calibration: at =1, the exact reverse-KL projection without information scaling, Diagonal KDN still improves on KDA (WikiText 18.69 vs. 18.85, accuracy 54.12 vs. 53.87, NIAH 69.0 vs. 62.1; Table 5).

W2 + Q2. Comparison with KLA and Bayesian Layers

We added a matched comparison with KLA, in which Isotropic KDN is better on perplexity, average accuracy and NIAH. A matched comparison with Bayesian Layers is not practical at our scale. We also added a dedicated section comparing KDN with KLA and GKA to Appendix B (Additional Related Work).

KLA (new Table 9). KLA's released kernel is too slow to train at our scale: at 750M it is about 11x slower than Isotropic KDN (below). We therefore compared the two at 220M/20B, with matched backbone, recipe, data, recurrent memory (65,536 entries per layer) and parameters (219.85M vs. 219.88M).

ModelWiki. PPLLMB. PPLAvg.
KLA32.8645.5344.4
Isotropic KDN29.0237.7644.9

Isotropic KDN is also more accurate in every NIAH cell except S-NIAH-3 at 2K and 4K, where both models score zero. For example, S-NIAH-1 at 4K is 75.4 vs. 18.0.

Throughput (new Table 10). Full 750M training step on one H200, 32K tokens per step, in K tokens/s:

Model4K32KPeak memory at 8K (GiB)
KLA5.84.9129
Isotropic KDN65.963.472
Diagonal KDN52.645.1102

Isotropic KDN trains 11-13x faster than KLA with 44% less memory, and Diagonal KDN 9x faster.

The two differ in three ways:

  • Modeling. KLA filters each state entry independently under input-independent dynamics. KDN places the uncertainty on the key–value map and predicts dynamics and noise per token.
  • Auxiliary state. KLA keeps one precision per state entry, as many as the memory has. KDN adds only dkd_k scalars per head (Diagonal) or 1 (Isotropic).
  • Kernel. KLA runs its per-entry scan in fp32 without tensor cores.

Bayesian Layers. They propagate a dense dk×dkd_k\times d_k covariance per head, as large as the memory, and compute their gains with a sequential loop within each chunk. The largest model they train from scratch has 14–17M non-embedding parameters (520M WikiText-103 tokens). At 340M, they only replace 4 of the 24 mixers of a pretrained Gated DeltaNet through distillation. They report no throughput and list the quadratic cost in the key dimension as a limitation.

GKA. We added GKA as the closest baseline with a dense dk×dkd_k\times d_k state. At 750M, both KDN variants outperform it on perplexity, accuracy and NIAH:

ModelWiki. PPLLMB. PPLAvg.NIAH
GKA19.4718.2653.0260.5
Isotropic KDN18.4214.6854.4162.4
Diagonal KDN18.6414.1554.9770.3

Isotropic KDN also trains 1.19x faster than GKA.

AuthorsOpenReview ↗

Response to Reviewer wiiC [2/2]

Q3. Probabilistic interpretation of the scaled update

Yes. The scaled update is the exact optimum of a weighted variational objective, which reduces to the ELBO only at μ=1\mu=1. We therefore read ptp_t as a calibrated uncertainty that controls future writes.

Weighted ELBO. With predictive prior π^t\widehat\pi_t and q=N(m,diag(p))q=\mathcal N(m,\mathrm{diag}(p)), the ELBO is Eq[log⁡p(v∣S)]−KL(q ∥ π^t)\mathbb E_q[\log p(v\mid S)]-\mathrm{KL}(q\,\|\,\widehat\pi_t). Its expected log-likelihood splits into the fit at the mean and a predictive-variance penalty, log⁡N(v; k⊤m, r)−V(q)\log\mathcal N(v;\,k^\top m,\,r)-V(q) with V(q)=k⊤diag(p)k/(2r)V(q)=k^\top\mathrm{diag}(p)k/(2r). Weighting the penalty by μ\mu gives Lμ(q)=log⁡N(v; k⊤m, r)−μ V(q)−KL(q ∥ π^t).\mathcal L_\mu(q)=\log\mathcal N(v;\,k^\top m,\,r)-\mu\,V(q)-\mathrm{KL}(q\,\|\,\widehat\pi_t). Its unique maximizer is the Kalman mean with the gain computed from rr, and 1/pi=1/p^i+μki2/r1/p_i=1/\widehat p_i+\mu k_i^2/r, which is exactly the scaled update.

  • For μ=1\mu=1, L1\mathcal L_1 is the ELBO, and its maximizer is the variational update of Proposition 4.1.
  • For μ≠1\mu\neq1, Lμ\mathcal L_\mu is not an ELBO: a Gaussian likelihood with any noise level gives the fit and the variance penalty the same weight. Tempering the likelihood (r→r/μr\to r/\mu) scales both and therefore also changes the gain.

The extra weight only penalizes the variance passed to later tokens, so the current write is unchanged.

Why μ=dk\mu=d_k. The next write to a key depends on its directional variance k⊤Pkk^\top Pk (Eq. 46), which the μ=1\mu=1 projection overstates for keys spread over several channels (Eq. 45). With μ=m\mu=m, the diagonal state matches the exact directional variance for equal-energy keys on mm channels (Eq. 48), so μ=dk\mu=d_k is the calibration for dense keys.

We added this result to Appendix D.

We thank the reviewer again for the detailed feedback. Please let us know whether we have addressed your concerns adequately; we are happy to answer any follow-up questions.

Thank you for the detailed rebuttal. I have two follow-up questions:

  1. Have you observed whether the propagated uncertainty reflects the actual reliability of stored associations, beyond its role in controlling memory updates?

  2. The controlled recall study shows that Diagonal KDN is sensitive to channel overlap even when key inner products are preserved. Do you expect the model to learn a favorable key representation during training, or could this coordinate dependence remain a limitation in practice?

Reviewer_ouwjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper formulates delta-rule associative memory as a linear-Gaussian filtering problem, then derives diagonal and isotropic approximations that retain a small uncertainty state. A Möbius scan computes the gains, followed by a chunkwise delta-rule memory update. An information-scaling parameter compensates for uncertainty lost by the diagonal projection. Parameter-matched language-model pretraining at 750M/50B and 1.3B/100B shows improved perplexity and generally stronger retrieval, especially for Diagonal KDN.

Critical strengths
  1. The derivation is clear and technically coherent. The paper distinguishes the exact Kalman filter from the projected approximation, preserves the one-step posterior mean conditional on the approximate prior, and explains why the covariance update admits an associative scan (Sections 3-4; Appendices C, E, and F). This is a useful connection between probabilistic filtering and practical delta-rule mixers.
  2. The empirical comparison is substantial and reasonably controlled: recurrent models share a backbone, non-embedding parameter counts are matched within 0.03%, and Mamba-3 state sizes are matched (Appendix A.1, p. 13). Diagonal KDN improves WikiText perplexity from 15.40 to 15.04 against KDA at 1.3B and makes sizeable gains in several retrieval settings, including S-NIAH-1 at 8K and multi-key retrieval (Tables 1-3, pp. 7-8).
  3. The paper goes beyond reporting aggregate quality by analyzing projection-induced overwrite, studying information scaling, and giving an implementable two-stage chunkwise algorithm. The length-extrapolation and noise ablations in the appendix also help characterize the method (Table 4; Figure 5; Tables 7-8; Appendix F).
Critical weaknesses
  1. The evidence does not yet isolate how much improvement comes specifically from propagating historical uncertainty. Tables 4 and 7 vary information scaling and learned noise, while Figure 2 and Table 8 measure effective write strength; none compares the same gain parameterization with its uncertainty state reset or replaced by an input-only control. Moreover, the practical default mu = d_k is no longer the reverse-KL projection in Proposition 4.1, and its directional-variance justification assumes an equal-energy key with no intervening covariance evolution (Appendix D, p. 18, Eqs. 44-49). A focused control would better support the central interpretation that accumulated confidence protects useful associations, rather than the gains mainly reflecting a different gate parameterization.
  2. Some headline quality margins are small relative to the unreported variability. At 1.3B, the six-task averages are 60.45 for Diagonal KDN, 60.35 for Isotropic KDN, and 60.28 for KDA; the hybrid Diagonal KDN advantage is likewise 60.11 versus 59.94 (Table 1, p. 7). Appendix A.1 states a common data seed but provides no across-run variability, and there are no paired uncertainty intervals for these averages. The stronger perplexity and retrieval results remain meaningful evidence, but the claim of consistent downstream superiority should be qualified until the small aggregate differences are better characterized.
  3. The practical efficiency trade-off is only partly established. Figure 4 (p. 9) measures mixer-layer throughput on one H200; Diagonal KDN is visibly slower than KDA, while its additional state and gain computation are motivated as hardware efficient. End-to-end training or prefill/decode measurements and peak-memory figures would show whether the quality gain is attractive at comparable wall-clock cost. Appendix F.1 (p. 22) also clarifies that the implemented carry pass is serial across gain chunks, so logarithmic scan depth describes an available parallel formulation rather than the entire measured implementation.
Questions for authors
  1. Can a focused ablation retain the same noise/gain networks while removing access to historical uncertainty, for example by resetting the covariance before each token? A small controlled recall study with repeated, interfering, and orthogonal keys would also help test whether the default information scale improves retention without suppressing unrelated writes.
  2. For the small Table 1 differences, can the authors provide paired evaluation intervals and any available training-run variability, separating evaluation uncertainty from training variation? This need not require repeating every large-scale experiment.
  3. What exactly is included in the Figure 4 timing, and what are the end-to-end throughput and peak-memory costs for one representative matched KDA/Diagonal KDN configuration? Please distinguish the theoretical associative scan from the serial inter-chunk carry used by the current kernel.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ouwj [1/2]

We thank the reviewer for the thoughtful review and constructive feedback. We have revised the paper accordingly, with all changes marked in blue, and address each concern below. Table, figure and equation numbers refer to the revised PDF.

W1 + Q1 (first part). Isolating the benefit of propagated uncertainty

We added the requested control. Resetting the covariance before every token, while keeping the same noise networks, degrades Diagonal KDN on all four metrics (new Table 4, 750M/50B).

  • No carry keeps Diagonal KDN's architecture and the networks that predict ωt\omega_t and rtr_t, but drops the history term of the covariance prediction p^t=αt2⊙pt−1+ωt\widehat p_t=\alpha_t^2\odot p_{t-1}+\omega_t, setting p^t=ωt\widehat p_t=\omega_t. The gain keeps its Kalman form without accumulated evidence: κt=ωt⊙kt/(rt+∑iωt,ikt,i2)\kappa_t=\omega_t\odot k_t/(r_t+\sum_i\omega_{t,i}k_{t,i}^2).
  • Sigmoid gate replaces the gain with a GDN-2-style channel-wise gate, κt=bt⊙kt\kappa_t=b_t\odot k_t, where bt∈(0,1)dkb_t\in(0,1)^{d_k} is a sigmoid of the same projections that produce ωt\omega_t and rtr_t.
ModelWiki. PPLLMB. PPLAvg.NIAH
Diagonal KDN18.6414.1554.9770.3
no carry (reset)18.7414.3254.5066.9
sigmoid gate18.7314.6154.5863.7

The reset model is on par with the sigmoid gate on WikiText and accuracy, so the improvement comes from propagated uncertainty rather than from the gain's functional form.

For Isotropic KDN, KDA is the corresponding control: it has the same transition but uses a token gate instead of the propagated-uncertainty gain. Isotropic KDN improves on it on all four metrics:

ModelWiki. PPLLMB. PPLAvg.NIAH
Isotropic KDN18.4214.6854.4162.4
KDA18.8515.0653.8762.1

W1 + Q1 (second part). Controlled recall study and the default information scale

We added the requested study (Appendix D.1, Figure 7). When stored and new keys occupy different channels, μ=dk\mu=d_k improves retention without suppressing unrelated writes.

Setup. Synthetic streams (dk=128d_k=128) interleave 8 targets written 8 times with noisy values, 4 interfering keys (cosine 0.6 to a target) and 4 orthogonal keys written once. All methods share the delta-rule update and differ only in the gain: a fixed gain κ=βk\kappa=\beta k (DeltaNet, and GDN/KDA with α=1\alpha=1; the reset control also reduces to it), Diagonal KDN, and the exact Kalman filter. Since Eq. (50) ties write suppression to channel overlap rather than to inner products, we place keys on disjoint 8-channel supports and rotate them to raise the overlap while preserving all inner products. Methods are compared at the tuned fixed gain's operating point (512 streams, 2 seeds).

OverlapTarget err. at matched new-key err.<br>Fixed / μ=1\mu{=}1 / μ=dk\mu{=}d_k / ExactNew-key err. at matched target err.<br>Fixed / μ=1\mu{=}1 / μ=dk\mu{=}d_k / Exact
0.000.153 / 0.066 / 0.051 / 0.0320.369 / 0.277 / 0.222 / 0.223
0.480.153 / 0.082 / 0.070 / 0.0320.369 / 0.300 / 0.317 / 0.223
0.980.153 / 0.111 / 0.150 / 0.0320.369 / 0.341 / 0.364 / 0.223
  • Separate channels. At equal new-key error, μ=dk\mu=d_k cuts the target error 3x (0.051 vs. 0.153). At equal target error, its new-key error matches the exact filter's (0.222 vs. 0.223).
  • Dense keys. μ=dk\mu=d_k matches the fixed gain, while the exact filter is unaffected, so the gap comes from the diagonal projection rather than from uncertainty propagation.

On μ=dk\mu=d_k. We agree that it is neither the reverse-KL projection nor a universal calibration. Consistent with Eq. (48), the best scale tracks the keys' support size: for 8-channel keys, μ=dk≈11\mu=\sqrt{d_k}\approx 11, the value closest to m=8m=8, retains best. Appendix D now also gives a probabilistic reading of the scaled update (see our response to Reviewer wiiC, Q3).

AuthorsOpenReview ↗

Response to Reviewer ouwj [2/2]

W2 + Q2. Training variability

We have softened the claim in the abstract, introduction, results, and conclusion: KDN improves perplexity and generally improves mean downstream accuracy.

  • Evaluation noise. A binomial approximation over the six test sets (about 22K items) gives a standard error of about 0.43 points for each model's average. The 1.3B differences are therefore within noise: 60.45 / 60.35 / 60.28, and 60.11 vs. 59.94 for the hybrids. Due to an infra issue, we do not have access to our current checkpoints to perform paired bootstrap interval comparison. We report the stderr of the controlled checkpoints we just trained here for reference of the value range. We note that these unpaired estimates are expected to be more conservative than paired confidence intervals.
ModelWiki pplLMB pplLMB accPIQAHellaWinoARC-eARC-cAvg
Diagogal KDN with Sigmoid Gate18.7314.61 ± 0.4645.2 ± 0.770.3 ± 1.152.3 ± 0.557.2 ± 1.468.0 ± 1.034.4 ± 1.454.6 ± 0.4
Diagonal KDN with NoCarry18.7414.32 ± 0.4545.0 ± 0.771.0 ± 1.152.0 ± 0.556.4 ± 1.467.7 ± 1.034.9 ± 1.454.5 ± 0.4
  • Retrieval. The retrieval margins exceed evaluation noise. With 500 samples per NIAH cell, the standard error is at most 2.2 points per cell. At 1.3B, the 14-cell mean is 76.5 vs. 64.3 for KDA, and S-NIAH-1 at 8K is 99.8 vs. 30.2.
  • Training variation. We follow the protocol of GDN-2, whose recipe and architectures we adopt, where each configuration is trained once, which is common practice for pretraining at this scale.

W3 + Q3. Figure 5 timing, end-to-end cost, and the serial carry

Figure 5 (formerly Figure 4) times the forward and backward pass of a single mixer layer. End to end, Isotropic KDN trains at 1.02x KDA's throughput with less memory, and Diagonal KDN at 0.90x the throughput of GDN-2, its closest channel-wise counterpart (new Table 10).

Figure 5. It times one mixer layer at the 750M width, covering the projections, core operator and output projection. Runs use bf16 at batch 4 and report the median of 30 runs. The MLP, embeddings, loss and optimizer are excluded. We now state this in Section 5.3.

End-to-end training. We timed a full 750M training step on one H200 at 32K tokens per step, with 4K sequences:

ModelThroughput (K tok/s)Peak memory at 8K (GiB)
KDA64.374
Isotropic KDN65.972
GDN-258.588
Diagonal KDN52.6102
GKA55.680
Preconditioned KDA57.186

Diagonal KDN reaches 0.90x GDN-2's throughput at the training length and 0.79x at 32K, with 16% more peak memory. It is within 8% of GKA and Preconditioned KDA.

Kernel precision. Our Diagonal KDN kernel can be optimized further by storing intermediates in BF16, as GDN-2's kernel does. The resulting mixer layer trains at 1.02x GDN-2's throughput at 4K and 1.08x at 32K, with the same memory. However, BF16 storage may change the training dynamics, so Figure 5 and Table 10 report the FP32-storage kernel for consistency.

Inference.

  • Prefill. In forward-only mode (same benchmark as Figure 5, 4K), Isotropic KDN matches KDA (6.76M vs. 6.69M tok/s). Diagonal KDN reaches 4.28M tok/s, which is 0.64x KDA and 0.88x GDN-2.
  • Decode. Each token updates the same dk×dvd_k\times d_v memory as KDA. Diagonal KDN adds 128 scalars per head (0.8% more state) and O(dk)O(d_k) work for its gain.

Associative scan vs. serial carry. We agree with this distinction.

  • The gain kernel scans its 2×22\times2 Möbius summaries serially across chunks (16 steps at 4K with Cg=256C_g=256) and replays the chunks in parallel.
  • The memory scan is serial across chunks, as in KDA.

Logarithmic depth therefore describes the associative formulation, not the measured kernel. Section 4.1 now states this clearly. Thank the reviewer for pointing this out.

We thank the reviewer again for the detailed feedback. Please let us know whether we have addressed your concerns adequately; we are happy to answer any follow-up questions.

Reviewer_ouwjOpenReview ↗

Follow-up on the rebuttal

Thank you for the detailed response and additional experiments. The no-carry control in Table 4 directly addresses my main attribution concern: removing covariance history while retaining the architecture and noise-prediction networks reduces NIAH from 70.3 to 66.9 and average accuracy from 54.97 to 54.50, while worsening both perplexities. This substantially strengthens the evidence that propagated uncertainty contributes to the gains. The controlled recall study is also informative. Together with the follow-up discussion, it clarifies that the default information scale is a write-control approximation whose benefit depends on channel overlap, rather than a universally calibrated posterior uncertainty estimate.

The full-model measurements in Table 10 and the distinction between the associative formulation and the implemented serial inter-chunk carry largely resolve my efficiency question. They establish an explicit trade-off: Diagonal KDN achieves 52.6 versus 64.3 K tokens/s for KDA at 4K, and uses 102 versus 74 GiB peak memory at 8K. Isotropic KDN offers a lower-cost alternative. I also appreciate the additional matched comparisons with related methods.

My remaining question is the original W2/Q2 about uncertainty in the small Table 1 average-accuracy differences. I could not find paired evaluation intervals or language-model training-run variability in the response or revised manuscript; the two seeds in the synthetic recall study address a different issue. Could you provide intervals from existing predictions and any available run-variability information? If unavailable, please identify the single-run comparisons and qualify the claims consistently. In particular, the abstract's accuracy claim should reflect the added Preconditioned KDA result, whose 750M average exceeds Isotropic KDN's. This does not require repeating all large-scale runs.

Overall, the added controls strengthen the paper, and I maintain my rating of 3 (weak acceptance).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.