- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines interference in DeltaNet’s fixed-size recurrent memory: updating the response at the current key also changes responses at overlapping past keys. It proposes Interference-Aware DeltaNet (IA-DeltaNet), which separates the key used to evaluate prediction error from the direction used to write the correction. The authors formulate a constrained quadratic optimization problem that minimizes interference while preserving DeltaNet’s current-key correction. Its exact solution uses the inverse covariance of past keys to adjust the write direction, recovering DeltaNet under isotropic key geometry. For practical implementation, the paper introduces a diagonal approximation based on accumulated squared keys. This adds no trainable parameters, maintains a fixed-size recurrent state, and supports chunk-parallel computation. Evaluation covers multi-query associative recall, text and video needle retrieval, long-document language modeling, and streaming video QA. The results show improved recall under high associative load, gains on video understanding benchmarks, and substantially reduced degradation relative to DeltaNet during prolonged state propagation. Analyses on synthetic and learned video keys provide evidence of reduced interference, while comparisons with the exact variant characterize the effect of the diagonal approximation.
Critical strengths
1. A clear and principled formulation. The paper identifies how DeltaNet’s write direction perturbs past-key responses and formulates a constrained objective for reducing this interference. Preserving the current-key correction makes the proposed change easy to interpret, while the exact solution provides a useful reference for evaluating the approximation.
2. Evidence across multiple settings. The improvements extend from synthetic recall to text retrieval and video understanding. For example, at model width 128 with 256 associations, diagonal IA-DeltaNet improves MQAR accuracy from 81.7% to 95.5%. The interference measurements on synthetic and learned video keys provide additional support for the method beyond benchmark scores.
3. A modification that can be reused beyond the base architecture. The diagonal update introduces no trainable parameters and remains compatible with chunk-parallel computation. Appendix A.2 also shows improvements when the same write rule is applied to Gated DeltaNet on MQAR. This suggests that interference-aware writing can complement gated forgetting, increasing its potential usefulness beyond DeltaNet.
Critical weaknesses
1. The interpretation of the memory improvements remains partly unresolved. Figure 3 and Appendix D provide evidence of reduced interference, but interference and errors on stored associations are not tracked together under a shared input stream. The gap between diagonal and exact updates is also not quantified on learned video keys. Meanwhile, several MQAR baselines exhibit substantial variance or near-zero accuracy despite the three-seed evaluation and learning-rate sweep. These gaps make it difficult to separate improved preservation of learned associations from more reliable convergence, and to determine how consistently the proposed interference mechanism explains the observed gains.
2. The streaming benefits and scalability are not fully characterized. The key statistics accumulate without decay, but their behavior under a distribution change within a stream is not explicitly tested. Appendix C.3 reports timing and peak memory for the memory module rather than the full system. Furthermore, the language-model experiments use a single scale of approximately 374M parameters for IA-DeltaNet. The 4B video backbone provides a useful application setting, but the proposed updates are confined to two added memory blocks. These results leave open how the method’s adaptation, accuracy, and resource requirements behave in larger IA-DeltaNet language models and complete streaming systems.
Questions for authors
1. Additional analyses on interference and retention. Figure 3 establishes reduced interference, while Table 1 reports improved recall. Under a shared key–value stream and initial state, does lower interference also lead to lower error on previously stored associations as updates accumulate? A small controlled experiment would help connect these observations.
2. Clarification and additional analysis on approximation on learned keys. Figure 3(c) compares DeltaNet and diagonal IA-DeltaNet on real video keys. Are the write rules evaluated on identical key sequences? How close is the diagonal direction to the exact solution on those same keys, when both are evaluated using the full covariance? An analysis on selected heads or sequences would be sufficient.
3. Robustness on MQAR. Do the high-variance MQAR results reflect runs that fail to converge, or different performance levels among converged runs? Per-seed learning curves and sensitivity to the learning rates already explored would help. Additional seeds on a representative difficult setting would further strengthen this assessment.
4. Distribution change. How does the cumulative statistic behave when the dominant key dimensions change partway through a stream? A controlled distribution-switch experiment, measuring retention of old associations and acquisition of new ones, would address this concern. Comparing cumulative and decayed statistics would help identify whether stale statistics affect adaptation.
5. System efficiency. Could the authors extend Appendix C.3 to report full-model frame-processing latency, question-to-first-token latency, and peak GPU memory across stream lengths? Comparisons with the DeltaNet, Gated DeltaNet, and attention variants should use the same backbone, hardware, precision, and visible-window budget. Please distinguish processing performed before the question arrives from work included in TTFT.
6. Model Scale. Do the recall and language-modeling gains persist at a larger IA-DeltaNet model scale? The current language-model evaluation uses approximately 374M parameters, while the Qwen3.5-4B experiments apply IA-DeltaNet through added memory blocks. A representative comparison with DeltaNet at a second, larger scale, using matched parameter counts and training budgets, would help establish whether the gains extend beyond the current configuration.
6 comments
4. Distribution change.
To examine how cumulative key statistics behave under distribution change, we conducted a controlled two-phase experiment. In the switch setting, the dominant key dimensions shift from 1-32 in the first 32 writes (Phase A) to 17-48 in the next 32 writes (Phase B). In the stationary control, the dominant dimensions remain 1-32 throughout both phases.
We compared DeltaNet, IA-DeltaNet with cumulative key statistics, and IA-DeltaNet with exponentially decayed key statistics (half-life ). All methods used identical key–value streams and initial memory states within each setting, with results averaged over 15 seeds.
For each phase , we measure the mean squared retrieval error of its stored associations after all 64 writes:
Table R8. Retrieval MSE on associations stored in Phase A (first 32 writes) and Phase B (last 32 writes), evaluated after 64 writes and averaged over 15 seeds. Lower is better.
| Distribution | Model | Phase A MSE | Phase B MSE |
|---|---|---|---|
| Switch | DeltaNet | 0.943 | 0.644 |
| IA-DeltaNet (cumulative) | 0.806 | 0.657 | |
| IA-DeltaNet (decayed) | 0.892 | 0.636 | |
| Stationary | DeltaNet | 1.410 | 0.674 |
| IA-DeltaNet (cumulative) | 1.382 | 0.646 | |
| IA-DeltaNet (decayed) | 1.404 | 0.653 |
In the switch setting, cumulative statistics achieve the lowest retrieval error on Phase A associations (0.806, compared with 0.892 for decayed statistics and 0.943 for DeltaNet), while decayed statistics achieve the lowest error on Phase B associations (0.636, compared with 0.657 for cumulative statistics and 0.644 for DeltaNet). In the stationary setting, the cumulative variant achieves lower error in both phases. These results illustrate a retention-adaptation trade-off, with the benefit of decay depending on the key distribution.
5. System efficiency.
The efficiency evaluation in our manuscript measured GPU kernel time and memory usage only for the memory modules. Following your suggestion, we additionally evaluated full-model streaming efficiency for DeltaNet, Gated DeltaNet, IA-DeltaNet, and full Attention, using the same Qwen3.5-4B backbone, hardware, precision, and visible window ().
We measured (i) online frame-processing latency, including preprocessing, vision encoding, and memory updates, (ii) question-to-first-token latency (TTFT), and (iii) full-model peak GPU memory. Frame processing is performed before question arrival and excluded from TTFT. We evaluated stream lengths of 64, 256, 1024, and 2048 frames.
Table R9. Full-model streaming efficiency at 64 and 2048 sampled frames. Latencies are medians over ten runs, and GPU memory is the maximum observed peak allocation.
| Model | Frames | Avg. processing (ms/frame) | TTFT (ms) | Peak GPU memory (GiB) |
|---|---|---|---|---|
| DeltaNet | 64 | 21.17 | 59.40 | 9.030 |
| 2048 | 20.05 | 58.86 | 9.034 | |
| Gated DeltaNet | 64 | 21.34 | 57.98 | 9.130 |
| 2048 | 20.63 | 58.36 | 9.137 | |
| IA-DeltaNet | 64 | 22.22 | 58.08 | 9.034 |
| 2048 | 21.96 | 59.47 | 9.037 | |
| Full Attention | 64 | 18.62 | 56.97 | 9.084 |
| 2048 | 26.18 | 60.52 | 12.353 |
As shown in Table R9, IA-DeltaNet maintains nearly constant processing latency and GPU memory across stream lengths, with a modest overhead over DeltaNet. While full Attention is faster for shorter streams, its processing latency and memory usage increase with stream length. TTFT remains comparable across models under the same fixed visible-window budget. Full results across all four stream lengths are provided in Appendix C.3.
6. Experiments on Larger Models.
We are training 1B-scale DeltaNet and IA-DeltaNet language models from scratch with matched parameter counts and training budgets. We will report the results once the experiments are complete.
Dear Reviewer cSHz,
Thank you again for your feedback. As the discussion period progresses, we would appreciate hearing whether our responses have addressed your main concerns. Please let us know if there are any remaining questions or points we could clarify.
1. Interference and retention under sequential writes.
We conducted a controlled sequential-write experiment using identical key-value streams and initial memory states for DeltaNet and IA-DeltaNet. We used dk=dv=128, keys with target effective rank 64 and covariance rotation θ=π/16, and independently sampled unit-norm values, averaging over 15 seeds.
We measure cumulative interference IT and mean squared retrieval error ET after T writes as:
\mathcal{E}_T = \frac{1}{T} \sum_{j=1}^{T} \lVert S_T k_j - v_j \rVert_2^2.$$ Table R5. Relative reductions (%) in cumulative interference and retrieval error compared with DeltaNet, computed from the means over 15 seeds. | Writes ($T$) | $\mathcal{I}_T$ reduction (%) | $\mathcal{E}_T$ reduction (%) | |---:|---:|---:| | 32 | 16.4 | 12.8 | | 64 | 22.2 | 14.6 | | 128 | 28.3 | 13.8 | Lower interference is accompanied by lower retrieval error at every stream length: IA-DeltaNet reduces cumulative interference by 16.4-28.3% and the retrieval error of the stored associations by 12.8-14.6% after 32-128 sequential writes. At 128 writes, the retrieval error decreases from 0.879 to 0.758. Because both rules replay identical streams from the same initial state without any training, this difference is attributable to the write direction alone. **2. Evaluating the diagonal approximation on learned keys.** In Figure 3(c), DeltaNet and IA-DeltaNet use their own learned key representations, so the comparison reflects differences in both representations and write rules. To disentangle these effects, we extracted key sequences from each trained model and evaluated three write directions on the same keys: $w_t=k_t$, $w_t=u_t^\text{diag}$, and $w_t=u_t^\text{exact}$, corresponding to DeltaNet, IA-DeltaNet, and exact IA-DeltaNet, respectively. For each write direction $w_t$, we computed interference using the same full past-key covariance: $$I_t(w_t)=w_t^\top\Sigma_{t-1}w_t,\qquad \Sigma_{t-1}=I+\sum_{j<t}k_jk_j^\top.$$ We sum interference across sequences, heads, and token positions, then normalize by the total for $w_t=k_t$: $$R(w)=\frac{\sum_{h,t}I_t(w_t)}{\sum_{h,t}I_t(k_t)}.$$ Table R6. Mean interference per write on learned video keys. Parentheses show values normalized by $w_t=k_t$ within each key source. Lower is better. | Write direction | DeltaNet keys | IA-DeltaNet keys | |---|---:|---:| | $w_t=k_t$ | 1,769.4 (1.000) | 1,513.8 (1.000) | | $w_t=u_t^{\mathrm{diag}}$ | 484.6 (0.274) | 358.8 (0.237) | | $w_t=u_t^{\mathrm{exact}}$ | **51.8 (0.029)** | **36.4 (0.024)** | Under $w_t=k_t$, IA-DeltaNet-trained keys exhibit 14.4% lower interference than DeltaNet-trained keys, indicating differences in their learned key representations. On identical keys, the diagonal write direction reduces interference to 27.4% and 23.7% of the corresponding reference values on DeltaNet- and IA-DeltaNet-trained keys, respectively. The exact direction further reduces interference to 2.9% and 2.4%, with diagonal-exact direction cosine similarities of 0.612 and 0.638. The diagonal approximation thus achieves about three-quarters of the reduction of the exact direction (75% and 78%), while a clear gap to the exact solution remains. These results show that the write rule itself substantially reduces interference on both key representations, beyond differences induced by training. **3. Robustness on MQAR.** The high variance of Gated DeltaNet in Table 1 arises from two distinct behaviors: convergence failures at $d_{\mathrm{model}}=128$ and varying accuracy levels among converged runs at $d_{\mathrm{model}}=64, K=64$. To verify that these observations were not specific to the original seeds, we ran three additional Gated DeltaNet seeds (3-5) in each of the four settings, using the same training configurations and learning-rate sweep. At $d_{\mathrm{model}}=128$, 3/6, 5/6, and 6/6 seeds failed to converge at $K=64, 128$, and $256$, respectively, across the original and additional seeds. At $d_{\mathrm{model}}=64, K=64$, all six Gated DeltaNet runs converged, but their accuracies varied. The per-seed results are shown in Table R7. Table R7. Per-seed MQAR accuracy (%) at $d_{\text{model}}=64, K=64$. Seeds 0-2 correspond to Table 1 of the main paper, while seeds 3-5 are additional runs. | Model | Seeds | Accuracy (%) | Mean $\pm$ std (%) | |---|---|---|---:| | Gated DeltaNet | 0-2 | 83.46 / 98.52 / 95.85 | 92.6 $\pm$ 8.0 | | Gated DeltaNet | 3-5 | 97.39 / 79.27 / 92.51 | 89.7 $\pm$ 9.4 | | IA-DeltaNet | 0-2 | 94.39 / 94.79 / 96.12 | 95.1 $\pm$ 0.9 | | Exact IA-DeltaNet | 0-2 | 98.00 / 97.75 / 98.65 | 98.2 $\pm$ 0.5 | The additional runs confirm the high variability of Gated DeltaNet, while both IA-DeltaNet variants achieve higher mean accuracy with substantially lower seed-to-seed variance in the reported setting.