acceptodds
Under review as a conference paper at ICLR 2027

Interference-Aware DeltaNet for Streaming Data

Abstract

Transformers excel at sequence modeling through attention, but their quadratic compute and growing key-value cache make them costly for long-range streaming tasks. Linear attention variants, such as DeltaNet, address this by reformulating attention as a recurrent memory equipped with an error-corrective update rule. However, we show that DeltaNet's update rule induces unintended interference on past-key responses, as the correction signal leaks into every past key that is not orthogonal to the current one. We propose Interference-Aware DeltaNet (IA-DeltaNet), which generalizes the delta rule by decoupling the key used for error evaluation from the direction along which the memory is updated. We formulate the choice of write direction as a constrained quadratic program that minimizes the total squared interference on past-key responses while exactly preserving DeltaNet's current-key correction, and derive its closed-form solution as a preconditioned key obtained from the inverse uncentered covariance of past keys. To ensure scalability, we introduce a diagonal approximation of the past-key covariance, preserving the per-step efficiency and chunk-parallelizability of the original architecture. On multi-query associative recall, interference-aware writing improves retention under high associative load. It also improves over recurrent-memory baselines as a bounded memory in a streaming VideoLLM, and substantially mitigates DeltaNet's degradation under prolonged recurrent-state propagation in language modeling. These results indicate that explicitly accounting for past-key geometry offers an effective design principle for associative memory in linear-attention architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear attention, associative memory, streaming video understanding, long-context sequence modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cSHzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines interference in DeltaNet’s fixed-size recurrent memory: updating the response at the current key also changes responses at overlapping past keys. It proposes Interference-Aware DeltaNet (IA-DeltaNet), which separates the key used to evaluate prediction error from the direction used to write the correction. The authors formulate a constrained quadratic optimization problem that minimizes interference while preserving DeltaNet’s current-key correction. Its exact solution uses the inverse covariance of past keys to adjust the write direction, recovering DeltaNet under isotropic key geometry. For practical implementation, the paper introduces a diagonal approximation based on accumulated squared keys. This adds no trainable parameters, maintains a fixed-size recurrent state, and supports chunk-parallel computation. Evaluation covers multi-query associative recall, text and video needle retrieval, long-document language modeling, and streaming video QA. The results show improved recall under high associative load, gains on video understanding benchmarks, and substantially reduced degradation relative to DeltaNet during prolonged state propagation. Analyses on synthetic and learned video keys provide evidence of reduced interference, while comparisons with the exact variant characterize the effect of the diagonal approximation.

Critical strengths

1. A clear and principled formulation. The paper identifies how DeltaNet’s write direction perturbs past-key responses and formulates a constrained objective for reducing this interference. Preserving the current-key correction makes the proposed change easy to interpret, while the exact solution provides a useful reference for evaluating the approximation.

2. Evidence across multiple settings. The improvements extend from synthetic recall to text retrieval and video understanding. For example, at model width 128 with 256 associations, diagonal IA-DeltaNet improves MQAR accuracy from 81.7% to 95.5%. The interference measurements on synthetic and learned video keys provide additional support for the method beyond benchmark scores.

3. A modification that can be reused beyond the base architecture. The diagonal update introduces no trainable parameters and remains compatible with chunk-parallel computation. Appendix A.2 also shows improvements when the same write rule is applied to Gated DeltaNet on MQAR. This suggests that interference-aware writing can complement gated forgetting, increasing its potential usefulness beyond DeltaNet.

Critical weaknesses

1. The interpretation of the memory improvements remains partly unresolved. Figure 3 and Appendix D provide evidence of reduced interference, but interference and errors on stored associations are not tracked together under a shared input stream. The gap between diagonal and exact updates is also not quantified on learned video keys. Meanwhile, several MQAR baselines exhibit substantial variance or near-zero accuracy despite the three-seed evaluation and learning-rate sweep. These gaps make it difficult to separate improved preservation of learned associations from more reliable convergence, and to determine how consistently the proposed interference mechanism explains the observed gains.

2. The streaming benefits and scalability are not fully characterized. The key statistics accumulate without decay, but their behavior under a distribution change within a stream is not explicitly tested. Appendix C.3 reports timing and peak memory for the memory module rather than the full system. Furthermore, the language-model experiments use a single scale of approximately 374M parameters for IA-DeltaNet. The 4B video backbone provides a useful application setting, but the proposed updates are confined to two added memory blocks. These results leave open how the method’s adaptation, accuracy, and resource requirements behave in larger IA-DeltaNet language models and complete streaming systems.

Questions for authors

1. Additional analyses on interference and retention. Figure 3 establishes reduced interference, while Table 1 reports improved recall. Under a shared key–value stream and initial state, does lower interference also lead to lower error on previously stored associations as updates accumulate? A small controlled experiment would help connect these observations.

2. Clarification and additional analysis on approximation on learned keys. Figure 3(c) compares DeltaNet and diagonal IA-DeltaNet on real video keys. Are the write rules evaluated on identical key sequences? How close is the diagonal direction to the exact solution on those same keys, when both are evaluated using the full covariance? An analysis on selected heads or sequences would be sufficient.

3. Robustness on MQAR. Do the high-variance MQAR results reflect runs that fail to converge, or different performance levels among converged runs? Per-seed learning curves and sensitivity to the learning rates already explored would help. Additional seeds on a representative difficult setting would further strengthen this assessment.

4. Distribution change. How does the cumulative statistic behave when the dominant key dimensions change partway through a stream? A controlled distribution-switch experiment, measuring retention of old associations and acquisition of new ones, would address this concern. Comparing cumulative and decayed statistics would help identify whether stale statistics affect adaptation.

5. System efficiency. Could the authors extend Appendix C.3 to report full-model frame-processing latency, question-to-first-token latency, and peak GPU memory across stream lengths? Comparisons with the DeltaNet, Gated DeltaNet, and attention variants should use the same backbone, hardware, precision, and visible-window budget. Please distinguish processing performed before the question arrives from work included in TTFT.

6. Model Scale. Do the recall and language-modeling gains persist at a larger IA-DeltaNet model scale? The current language-model evaluation uses approximately 374M parameters, while the Qwen3.5-4B experiments apply IA-DeltaNet through added memory blocks. A representative comparison with DeltaNet at a second, larger scale, using matched parameter counts and training budgets, would help establish whether the gains extend beyond the current configuration.

6 comments

1. Interference and retention under sequential writes.

We conducted a controlled sequential-write experiment using identical key-value streams and initial memory states for DeltaNet and IA-DeltaNet. We used dk=dv=128d_k=d_v=128, keys with target effective rank 64 and covariance rotation θ=π/16\theta = \pi/16, and independently sampled unit-norm values, averaging over 15 seeds.

We measure cumulative interference IT\mathcal{I}_T and mean squared retrieval error ET\mathcal{E}_T after TT writes as:

\mathcal{E}_T = \frac{1}{T} \sum_{j=1}^{T} \lVert S_T k_j - v_j \rVert_2^2.$$ Table R5. Relative reductions (%) in cumulative interference and retrieval error compared with DeltaNet, computed from the means over 15 seeds. | Writes ($T$) | $\mathcal{I}_T$ reduction (%) | $\mathcal{E}_T$ reduction (%) | |---:|---:|---:| | 32 | 16.4 | 12.8 | | 64 | 22.2 | 14.6 | | 128 | 28.3 | 13.8 | Lower interference is accompanied by lower retrieval error at every stream length: IA-DeltaNet reduces cumulative interference by 16.4-28.3% and the retrieval error of the stored associations by 12.8-14.6% after 32-128 sequential writes. At 128 writes, the retrieval error decreases from 0.879 to 0.758. Because both rules replay identical streams from the same initial state without any training, this difference is attributable to the write direction alone. **2. Evaluating the diagonal approximation on learned keys.** In Figure 3(c), DeltaNet and IA-DeltaNet use their own learned key representations, so the comparison reflects differences in both representations and write rules. To disentangle these effects, we extracted key sequences from each trained model and evaluated three write directions on the same keys: $w_t=k_t$, $w_t=u_t^\text{diag}$, and $w_t=u_t^\text{exact}$, corresponding to DeltaNet, IA-DeltaNet, and exact IA-DeltaNet, respectively. For each write direction $w_t$, we computed interference using the same full past-key covariance: $$I_t(w_t)=w_t^\top\Sigma_{t-1}w_t,\qquad \Sigma_{t-1}=I+\sum_{j<t}k_jk_j^\top.$$ We sum interference across sequences, heads, and token positions, then normalize by the total for $w_t=k_t$: $$R(w)=\frac{\sum_{h,t}I_t(w_t)}{\sum_{h,t}I_t(k_t)}.$$ Table R6. Mean interference per write on learned video keys. Parentheses show values normalized by $w_t=k_t$ within each key source. Lower is better. | Write direction | DeltaNet keys | IA-DeltaNet keys | |---|---:|---:| | $w_t=k_t$ | 1,769.4 (1.000) | 1,513.8 (1.000) | | $w_t=u_t^{\mathrm{diag}}$ | 484.6 (0.274) | 358.8 (0.237) | | $w_t=u_t^{\mathrm{exact}}$ | **51.8 (0.029)** | **36.4 (0.024)** | Under $w_t=k_t$, IA-DeltaNet-trained keys exhibit 14.4% lower interference than DeltaNet-trained keys, indicating differences in their learned key representations. On identical keys, the diagonal write direction reduces interference to 27.4% and 23.7% of the corresponding reference values on DeltaNet- and IA-DeltaNet-trained keys, respectively. The exact direction further reduces interference to 2.9% and 2.4%, with diagonal-exact direction cosine similarities of 0.612 and 0.638. The diagonal approximation thus achieves about three-quarters of the reduction of the exact direction (75% and 78%), while a clear gap to the exact solution remains. These results show that the write rule itself substantially reduces interference on both key representations, beyond differences induced by training. **3. Robustness on MQAR.** The high variance of Gated DeltaNet in Table 1 arises from two distinct behaviors: convergence failures at $d_{\mathrm{model}}=128$ and varying accuracy levels among converged runs at $d_{\mathrm{model}}=64, K=64$. To verify that these observations were not specific to the original seeds, we ran three additional Gated DeltaNet seeds (3-5) in each of the four settings, using the same training configurations and learning-rate sweep. At $d_{\mathrm{model}}=128$, 3/6, 5/6, and 6/6 seeds failed to converge at $K=64, 128$, and $256$, respectively, across the original and additional seeds. At $d_{\mathrm{model}}=64, K=64$, all six Gated DeltaNet runs converged, but their accuracies varied. The per-seed results are shown in Table R7. Table R7. Per-seed MQAR accuracy (%) at $d_{\text{model}}=64, K=64$. Seeds 0-2 correspond to Table 1 of the main paper, while seeds 3-5 are additional runs. | Model | Seeds | Accuracy (%) | Mean $\pm$ std (%) | |---|---|---|---:| | Gated DeltaNet | 0-2 | 83.46 / 98.52 / 95.85 | 92.6 $\pm$ 8.0 | | Gated DeltaNet | 3-5 | 97.39 / 79.27 / 92.51 | 89.7 $\pm$ 9.4 | | IA-DeltaNet | 0-2 | 94.39 / 94.79 / 96.12 | 95.1 $\pm$ 0.9 | | Exact IA-DeltaNet | 0-2 | 98.00 / 97.75 / 98.65 | 98.2 $\pm$ 0.5 | The additional runs confirm the high variability of Gated DeltaNet, while both IA-DeltaNet variants achieve higher mean accuracy with substantially lower seed-to-seed variance in the reported setting.

4. Distribution change.

To examine how cumulative key statistics behave under distribution change, we conducted a controlled two-phase experiment. In the switch setting, the dominant key dimensions shift from 1-32 in the first 32 writes (Phase A) to 17-48 in the next 32 writes (Phase B). In the stationary control, the dominant dimensions remain 1-32 throughout both phases.

We compared DeltaNet, IA-DeltaNet with cumulative key statistics, and IA-DeltaNet with exponentially decayed key statistics (half-life H=16H=16). All methods used identical key–value streams and initial memory states within each setting, with results averaged over 15 seeds.

For each phase G∈A,BG \in \\{A,B\\}, we measure the mean squared retrieval error of its stored associations after all 64 writes: EG=1∣G∣∑j∈G∥S64kj−vj∥22.\mathcal{E}_G = \frac{1}{|G|}\sum_{j\in G}\lVert S_{64}k_j-v_j\rVert_2^2.

Table R8. Retrieval MSE on associations stored in Phase A (first 32 writes) and Phase B (last 32 writes), evaluated after 64 writes and averaged over 15 seeds. Lower is better.

DistributionModelPhase A MSEPhase B MSE
SwitchDeltaNet0.9430.644
IA-DeltaNet (cumulative)0.8060.657
IA-DeltaNet (decayed)0.8920.636
StationaryDeltaNet1.4100.674
IA-DeltaNet (cumulative)1.3820.646
IA-DeltaNet (decayed)1.4040.653

In the switch setting, cumulative statistics achieve the lowest retrieval error on Phase A associations (0.806, compared with 0.892 for decayed statistics and 0.943 for DeltaNet), while decayed statistics achieve the lowest error on Phase B associations (0.636, compared with 0.657 for cumulative statistics and 0.644 for DeltaNet). In the stationary setting, the cumulative variant achieves lower error in both phases. These results illustrate a retention-adaptation trade-off, with the benefit of decay depending on the key distribution.

5. System efficiency.

The efficiency evaluation in our manuscript measured GPU kernel time and memory usage only for the memory modules. Following your suggestion, we additionally evaluated full-model streaming efficiency for DeltaNet, Gated DeltaNet, IA-DeltaNet, and full Attention, using the same Qwen3.5-4B backbone, hardware, precision, and visible window (W=8W=8).

We measured (i) online frame-processing latency, including preprocessing, vision encoding, and memory updates, (ii) question-to-first-token latency (TTFT), and (iii) full-model peak GPU memory. Frame processing is performed before question arrival and excluded from TTFT. We evaluated stream lengths of 64, 256, 1024, and 2048 frames.

Table R9. Full-model streaming efficiency at 64 and 2048 sampled frames. Latencies are medians over ten runs, and GPU memory is the maximum observed peak allocation.

ModelFramesAvg. processing (ms/frame)TTFT (ms)Peak GPU memory (GiB)
DeltaNet6421.1759.409.030
204820.0558.869.034
Gated DeltaNet6421.3457.989.130
204820.6358.369.137
IA-DeltaNet6422.2258.089.034
204821.9659.479.037
Full Attention6418.6256.979.084
204826.1860.5212.353

As shown in Table R9, IA-DeltaNet maintains nearly constant processing latency and GPU memory across stream lengths, with a modest overhead over DeltaNet. While full Attention is faster for shorter streams, its processing latency and memory usage increase with stream length. TTFT remains comparable across models under the same fixed visible-window budget. Full results across all four stream lengths are provided in Appendix C.3.

6. Experiments on Larger Models.

We are training 1B-scale DeltaNet and IA-DeltaNet language models from scratch with matched parameter counts and training budgets. We will report the results once the experiments are complete.

Dear Reviewer cSHz,

Thank you again for your feedback. As the discussion period progresses, we would appreciate hearing whether our responses have addressed your main concerns. Please let us know if there are any remaining questions or points we could clarify.

Reviewer_vxmJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeltaNet formulates its correction based on the ongoing key, which subsequently affects any previous key that shares an overlap with it. In this study, the current key is utilized to calculate the error, while a different direction is employed to reduce the interference from past keys. The final direction achieved is referred to as a whitened key, utilizing a cost-effective diagonal approximation in practical applications. The experiments conducted include MQAR, S-NIAH, and PG-19, involving approximately 374 million LMs, as well as streaming video QA on Qwen3.5-4B.

Critical strengths
  1. The concept is straightforward, and the derivation is straightforward as well. It simplifies to DeltaNet in cases where the key covariance is isotropic, without introducing any additional parameters.

  2. The diagonal variant is economical and maintains the chunk-parallel structure. Including the precise version along with the rotated-key verification (Fig. 3b) clearly indicates the limitations of the approximation.

  3. The improvements remain consistent across MQAR, video QA, and PG-19. The result from PG-19 was particularly persuasive: DeltaNet reaches approximately 90 people at 64-128K, whereas IA-DeltaNet hovers around 25.

Critical weaknesses
  1. Preconditioned delta-rule methods (e.g. Tumma et al. 2026, Zhou et al. 2026) cannot be compared directly, although the diagonal rule resembles an Adagrad-style preconditioner. In the absence of this, it remains unclear if the gains arise from the interference perspective or from adequate preconditioning.

  2. The crucial statistic d_t consistently increases and remains unaffected by decay even in the gated version, leading to outdated keys continuously influencing the writing process. Additionally, ||u_t|| lacks constraints, and there is no examination of its magnitude or stability.

  3. Evidence for the LM is limited. Only one model at 374M scale exists, there are no established LM or commonsense benchmarks, and in some configurations, GDN outperforms (e.g. S-NIAH-3 at 2K).

Questions for authors

See weaknesses

3 comments

1. Comparison with preconditioned delta-rule methods.

Although IA-DeltaNet, OSDN, and PDN all employ preconditioning, they differ in their derivations and update rules. IA-DeltaNet derives its write direction by minimizing interference with previously stored associations while preserving DeltaNet's current-key correction (ut⊤kt=1u_t^\top k_t=1), without introducing additional trainable parameters. OSDN (Zhou et al. 2026) adapts diagonal scaling through online hypergradient feedback, while PDN (Tumma et al. 2026) derives preconditioning from online least-squares optimization.

We additionally compare IA-DeltaNet with OSDN and PDN on MQAR.

Table R2. MQAR accuracy (%), mean ±\pm standard deviation over three seeds, with KK key-value pairs and sequence length L=8KL=8K. IA-DeltaNet$^{\dagger}$ uses the exact write direction.

dmodeld_{\mathrm{model}}KKLLDeltaNetGated DeltaNetOSDNPDNIA-DeltaNetIA-DeltaNet$^{\dagger}$
643225699.1±0.199.1\pm0.196.6±3.296.6\pm3.299.2±0.1\mathbf{99.2}\pm0.199.2±0.2\mathbf{99.2}\pm0.299.1±0.299.1\pm0.299.2±0.1\mathbf{99.2}\pm0.1
646451294.2±2.094.2\pm2.092.6±8.092.6\pm8.093.4±5.193.4\pm5.185.4±21.785.4\pm21.795.1±0.995.1\pm0.998.2±0.5\mathbf{98.2}\pm0.5
64128102450.7±23.850.7\pm23.826.2±23.326.2\pm23.354.3±6.154.3\pm6.158.3±12.658.3\pm12.664.0±0.564.0\pm0.570.0±5.7\mathbf{70.0}\pm5.7
1286451299.3±0.299.3\pm0.266.2±57.366.2\pm57.399.3±0.299.3\pm0.299.1±0.199.1\pm0.199.2±0.199.2\pm0.199.4±0.3\mathbf{99.4}\pm0.3
128128102499.1±0.199.1\pm0.132.8±56.832.8\pm56.899.2±0.199.2\pm0.198.7±0.598.7\pm0.599.1±0.199.1\pm0.199.6±0.2\mathbf{99.6}\pm0.2
128256204881.7±14.981.7\pm14.90.0±0.00.0\pm0.054.8±19.254.8\pm19.294.3±2.494.3\pm2.495.5±1.195.5\pm1.197.5±2.0\mathbf{97.5}\pm2.0

At low associative load, most methods achieve near-saturated accuracy. Under high associative load, IA-DeltaNet achieves higher mean accuracy than both OSDN and PDN, although PDN closely approaches its performance at the larger model width. Exact IA-DeltaNet achieves the highest mean accuracy in every setting. This suggests that the gains stem from the interference-aware choice of the write direction rather than from preconditioning alone.

2. Cumulative key statistics and write-direction stability.

We agree that decaying historical key statistics is a reasonable design choice, particularly when the recurrent state already incorporates forgetting. However, the benefit of decay may depend on the task, as discounting historical keys can affect the balance between retaining past associations and adapting to new ones.

We examined the write-direction norm ∥ut∥2\lVert u_t\rVert_2 under document-level state carry on PG-19. The median head-wise norm increased from 1.25 at 0-2K to 1.59 at 64-128K, with little further growth beyond 32K. Although the norm has no explicit upper bound, all writes and memory states remained finite up to 128K tokens despite training with only 2K contexts, and perplexity under state carry increased only from 20.60 to 25.51 over this range, compared with 20.30 to 90.84 for DeltaNet.

We also compared cumulative key statistics with an exponentially decayed variant on MQAR, using a half-life of H=L/4H=L/4, where L=8KL=8K is the sequence length. Decay was applied only to the key statistic, leaving the recurrent memory update unchanged.

Table R3. MQAR accuracy (%), mean ±\pm standard deviation over three seeds, comparing cumulative and exponentially decayed key statistics with a half-life of H=L/4H=L/4, where L=8KL=8K is the sequence length.

dmodeld_{\text{model}}KKCumulativeDecayed
643299.1 ± 0.299.2 ± 0.1
6495.1 ± 0.996.3 ± 1.2
12864.0 ± 0.560.0 ± 3.6
1286499.2 ± 0.199.2 ± 0.2
12899.1 ± 0.199.0 ± 0.1
25695.5 ± 1.185.3 ± 13.8

The two variants achieve comparable accuracy at low associative load. At high associative load, (dmodel,K)=(64,128)(d_{\mathrm{model}},K)=(64,128) and (128,256)(128,256), cumulative statistics achieve higher mean accuracy, with lower seed-to-seed variability in both setting.

We further examine decay under stationary and distribution-switch settings (see also our response to Reviewer cSHz, Question 4). In a controlled two-phase experiment, we compare a stationary stream with one in which the dominant key dimensions change midway through the stream. Under the stationary setting, cumulative statistics achieve lower retrieval error on associations from both phases. Under the distribution switch, cumulative statistics better preserve earlier associations (MSE: 0.806 vs. 0.892), whereas decay achieves lower error on later associations (0.636 vs. 0.657).

Together, these results indicate that the benefit of decay depends on the task and key distribution, reflecting a trade-off between retention and adaptation. Adapting decay based on the importance of historical information is an interesting direction for future work.

3. Language understanding and commonsense benchmarks.

We additionally evaluated the 374M DeltaNet and IA-DeltaNet language models on language understanding and commonsense benchmarks.

Table R4. Accuracy (%) on language understanding and commonsense benchmarks.

ModelARC-EARC-CHellaSwagLAMBADAPIQAWinoGrandeBoolQSciQAvg.
DeltaNet45.9622.9534.3430.5862.7351.0759.8278.6048.26
IA-DeltaNet45.8323.6333.9731.9863.4951.5461.6578.0048.76

IA-DeltaNet achieves slightly higher average accuracy (48.76% vs. 48.26%), with broadly comparable performance across individual benchmarks. These results complement the paper's main evaluations, as the relatively short input sequences in these benchmarks place less emphasis on long-range memory retention, whereas our main evaluations more directly examine memory preservation under sequential updates and prolonged state propagation.

Regarding Gated DeltaNet, its forget gate and interference-aware writing act on different parts of the memory update and are complementary. Applying the interference-aware write direction to Gated DeltaNet substantially improves its mean MQAR accuracy in all six evaluated settings (Appendix A.2).

We acknowledge that larger models or models trained with longer contexts may learn better representations and memory dynamics, potentially reducing the practical impact of interference. However, increasing model scale or training context length does not inherently eliminate the underlying mechanism by which new writes perturb responses at overlapping past keys. Thus, we believe that the motivation for interference-aware writing remains relevant to long-horizon sequence modeling.

Dear Reviewer vxmJ,

Thank you again for your feedback. As the discussion period progresses, we would appreciate hearing whether our responses have addressed your main concerns. Please let us know if there are any remaining questions or points we could clarify.

Reviewer_3bDAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Interference-Aware DeltaNet (IA-DeltaNet), which separates the key used to compute the current prediction error from the direction used to write the correction into recurrent memory. The authors formulate the write direction as a constrained optimization problem: minimize the total squared change to past-key responses while preserving DeltaNet's correction on the current key. The resulting exact solution uses the inverse covariance of past keys. To retain linear-time efficiency, the paper approximates this covariance with a diagonal state that accumulates per-feature key energy. The experiments cover multi-query associative recall, needle-in-a-haystack retrieval, long-document language modeling, and streaming video understanding. The paper reports that IA-DeltaNet improves recall under high associative load, reduces DeltaNet's perplexity degradation during long state propagation, and improves results in a Qwen3.5-4B streaming-video setup. The exact covariance version performs better than the diagonal approximation in the reported MQAR settings, while the diagonal version is substantially cheaper.

Critical strengths

The paper presents a clear intervention on the DeltaNet update rule, derives an interpretable solution, and evaluates it across several memory-intensive settings. Clear motivation and derivation. The paper identifies how DeltaNet's correction at the current key can alter responses at similar past keys. The proposed constrained objective is easy to connect to the update rule, and the exact solution has a useful geometric interpretation as covariance preconditioning. An efficient approximation with practical relevance. The diagonal state reduces the extra cost from maintaining and applying a full covariance matrix to per-feature operations. It preserves the current-key correction constraint and is compatible with chunk-parallel scans, which makes the proposal more practical than the exact solution. Evaluation beyond a single synthetic task. The paper includes associative recall, long-document language modeling, and streaming-video experiments. In particular, the Qwen3.5-4B video setup and PG-19 evaluation with state carried across chunks provide useful evidence that the method can affect downstream tasks beyond synthetic recall.

Critical weaknesses

The interference objective may penalize useful generalization. The method counts changes to all past-key responses as interference, independent of whether those changes harm the task. In natural data, similar keys may correspond to similar values, so a write that also shifts responses at nearby keys could provide useful smoothing or generalization. The exact-retention and needle-style tasks emphasize preserving distinct associations, where such leakage is especially costly. The paper does not directly test settings where key similarity and value similarity are correlated, so it remains unclear when reducing past-response changes improves task performance and when it may suppress useful transfer. The language-model evidence is limited in scale and scope. The paper does evaluate language modeling: its language models are trained from scratch at roughly 374M parameters, and the streaming-video setup uses Qwen3.5-4B. These are useful experiments, but they do not establish how the method behaves in larger language models or common long-context language-model workloads using pretrained models. The evidence supports the tested settings more strongly than a broad claim about practical LLM deployment. A larger-scale language-model evaluation, or a more explicit limitation of the claims, would strengthen the paper. The diagonal approximation has not been evaluated across key dimensions. Its fidelity depends on the off-diagonal structure of the past-key covariance, not on d_k alone. The synthetic covariance analysis varies anisotropy and rotation but fixes d_k=160. The MQAR results also show that the exact covariance variant improves on the diagonal approximation in the reported settings. Without a controlled sweep over d_k, covariance structure, and effective rank, it is difficult to tell whether the approximation remains reliable as key dimensions increase. The paper establishes favorable asymptotic cost, but not how approximation quality scales with dimension.

Questions for authors

When is reduced interference beneficial? Could the authors evaluate controlled settings where similar keys map to either similar or conflicting values? This would help determine whether IA-DeltaNet's gains come from preventing harmful overwrites, and whether it loses useful generalization when nearby keys should receive similar updates. How far do the language-model results generalize? The paper includes a 374M language-model study and a Qwen3.5-4B streaming-video study. Could the authors clarify what conclusions they believe these results support for larger pretrained LLMs and long-context language modeling? If feasible, an evaluation on a larger pretrained language model would help establish the practical scope. How does diagonal fidelity change with d_k? Could the authors report a controlled sweep over key dimensions and covariance geometries, comparing the diagonal approximation with the exact solution? Metrics such as the ratio of their total interference, direction similarity, runtime, and memory would show whether the efficiency gain continues to justify the approximation at larger dimensions.

5 comments

1. When does reducing interference help?

We clarify when reducing interference improves prediction accuracy through an analytical decomposition and a controlled experiment, now included in the revised Appendix D.1.3 (“When Does Interference Reduction Help?”).

For an arbitrary query qq with target y(q)y(q), the change in squared prediction error after a memory write is: ∥S2q−y(q)∥22−∥S1q−y(q)∥22=∥(S2−S1)q∥22+2(S1q−y(q))⊤(S2−S1)q.\lVert S_2q-y(q)\rVert_2^2-\lVert S_1q-y(q)\rVert_2^2=\lVert (S_2-S_1)q\rVert_2^2+2(S_1q-y(q))^\top(S_2-S_1)q.

If the memory already retrieves the target exactly, S1q=y(q)S_1q = y(q), any nonzero response change increases prediction error. Otherwise, a new write can reduce the existing error by moving the response closer to the target.

To examine these effects, we use non-orthogonal keys with k1⊤k2=1/2k_1^\top k_2=1/\sqrt{2}, vary the value similarity η=v1⊤v2\eta=v_1^\top v_2 from −1 to 1, and control the pre-write prediction error through β1\beta_1, the write strength for the first association (k1,v1)(k_1,v_1). We compare DeltaNet and IA-DeltaNet after writing the second association, averaging over β2∈[0,1]\beta_2\in[0,1].

As shown in Figure D.2, IA-DeltaNet generally achieves lower prediction error when the existing association is accurately retrieved, or the new value has low similarity to the stored value. Conversely, when the existing response is inaccurate and the new value is highly similar, DeltaNet's larger response change can help correct the remaining error. These results clarify when reducing interference helps preserve existing associations and when additional response changes can provide beneficial corrections.

2. Generalization to larger pretrained language models.

Our language-model experiments show that IA-DeltaNet substantially mitigates DeltaNet's degradation when recurrent states are propagated beyond the 2K training context. The Qwen3.5-4B experiments provide complementary evidence that interference-aware writing improves streaming video understanding when integrated as a memory module into a larger pretrained multimodal model.

We acknowledge that larger models or models trained with longer contexts may learn better representations and memory dynamics, potentially reducing the practical impact of interference. However, increasing model scale or training context length does not inherently eliminate the underlying mechanism by which new writes perturb responses at overlapping past keys. Thus, we believe that the motivation for interference-aware writing remains relevant to long-horizon sequence modeling. We will revise the manuscript accordingly.

3. Fidelity and efficiency of the diagonal approximation across key dimensions.

To examine how approximation fidelity changes with key dimension, we conducted a controlled sweep over dk∈64,128,160,256,512d_k \in \\{64, 128, 160, 256, 512\\}, target relative effective ranks reff/dk∈0.1,0.5,1.0r_{eff}/d_k \in \\{0.1, 0.5, 1.0\\}, and covariance rotations θ∈0,π/16,π/4\theta \in \\{0, \pi/16, \pi/4\\}. We used the same key streams for diagonal and exact IA-DeltaNet, with T=32dkT=32d_k writes per stream. The target effective ranks refer to the Gaussian generator before key normalization. Results are averaged over 15 independent trials per condition.

We evaluate the ratio of interference accumulated over the evaluated writes (diagonal/exact), using It(w)=w⊤Σt−1w\mathcal{I}_t(w)=w^\top\Sigma_{t-1}w with Σ0=I\Sigma_0=I, and the mean cosine similarity between the two write directions. Table R1 reports the results at target reff/dk=0.1r_{\mathrm{eff}}/d_k=0.1; the complete 45-condition sweep is provided in Appendix D.1.4.

Table R1. Diagonal-to-exact interference ratio and write-direction cosine similarity across key dimensions and covariance rotations at target reff/dk=0.1r_{eff}/d_k =0.1, averaged over 15 trials. Values closer to 1 indicate higher approximation fidelity.

Metricdkd_kθ=0\theta=0θ=π/16\theta=\pi/16θ=π/4\theta=\pi/4
Interference ratio641.0131.6092.015
1281.0141.7612.323
1601.0141.7822.382
2561.0141.8262.496
5121.0141.8742.622
Direction similarity640.9890.4490.368
1280.9900.4460.355
1600.9910.4470.354
2560.9910.4470.351
5120.9910.4460.346

With unrotated keys (θ=0\theta=0), the diagonal approximation remains close to the exact solution across all evaluated key dimensions. Under rotated, low-rank covariance, the interference ratio increases gradually with dk (from 1.61 to 1.87 at θ=π/16\theta=\pi/16 and from 2.02 to 2.62 at θ=π/4\theta=\pi/4), while the direction similarity changes little. Overall, approximation fidelity depends more strongly on covariance geometry than on key dimension, and the gap diminishes as the covariance approaches isotropy. In every condition, the diagonal direction accumulates no more interference than DeltaNet's write direction wt=ktw_t=k_t (the two nearly coincide for isotropic keys); at dk=512d_k=512 and reff/dk=0.1r_{eff}/d_k=0.1, it accumulates 0.35, 0.64, and 0.90 times DeltaNet's interference at θ=0\theta=0, π/16\pi/16, and π/4\pi/4, respectively.

From dk=64d_k=64 to dk=512d_k=512, the time per write increases from 5.77 to 6.44 μ\mus for diagonal IA-DeltaNet and from 8.47 to 11.00 μ\mus for exact IA-DeltaNet, while the key-statistic storage per head grows from 0.25 KiB to 2 KiB and from 16 KiB to 1 MiB, respectively (dkd_k versus dk2d_k^2 entries).

Overall, these results characterize the fidelity-efficiency trade-off of the diagonal approximation, demonstrating its efficiency benefits while identifying the covariance geometries where approximation fidelity degrades.

Dear Reviewer 3bDA,

Thank you again for your feedback. As the discussion period progresses, we would appreciate hearing whether our responses have addressed your main concerns. Please let us know if there are any remaining questions or points we could clarify.

Comments on the paper

1 comment

We thank the Area Chair and reviewers for their constructive feedback. We have uploaded a revised manuscript with changes highlighted in blue. Below, we summarize the additional experiments and analyses addressing the main concerns.

  • Interference reduction, generalization, and write-rule attribution [3bDA, cSHz]. We clarify when reducing interference improves prediction accuracy and when larger response changes can be beneficial. Controlled sequential-write experiments show that lower interference is accompanied by lower retrieval error. We further evaluate retrieval and generalization under smooth key-value mappings, and compare write directions on keys extracted from each model to distinguish the effects of learned representations and the write rule.
  • Diagonal approximation fidelity and efficiency [3bDA]. We examine the fidelity of the diagonal approximation across key dimensions and covariance geometries, along with its computational and storage costs.
  • Historical statistics and distribution changes [vxmJ, cSHz]. We examine write-direction stability under prolonged state propagation and compare cumulative and decayed key statistics under different associative loads and key distributions.
  • Comparisons and broader evaluation [vxmJ]. We add MQAR comparisons with OSDN and PDN and report language understanding and commonsense results.
  • MQAR robustness [cSHz]. We report additional Gated DeltaNet runs with per-seed results to clarify its performance on MQAR.
  • Full-model streaming efficiency [cSHz]. We extend the efficiency analysis to full-model frame-processing latency, time-to-first-token, and GPU memory usage across stream lengths.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.