- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
By pairing the fast-moving delta state with a slow-moving long-term memory state, Integral DeltaNet can alleviate forgetting induced by decay-based delta methods. This is a straightforward addition on top of previous delta models, leveraging a data-dependent moving average of past states and query-dependent mixing of short- and long-term information. The authors implement a fast kernel, showing comparable training throughput though with double the activation memory. Overall this method demonstrates strong performance on common sense reasoning, recall, and length extrapolation tasks.
Critical strengths
-
Simple, Strong Method: The proposed update rule is easy to implement and yields strong empirical results on downstream language modeling tasks. The idea is simple and well-executed with principled design choices (channel-wise mixing) and kernel implementation.
-
Presentation Clarity: This work was very straightforward and nice to read. The work was presented in the same template or format as others such as Gated DeltaNet, which is helpful for readers with a background in this space.
Critical weaknesses
- As a proposed fix to memory collisions, it seems mildly concerning that there isn’t a significant improvement in S-NIAH performance (and a slight decrease from GDN-2 SN1). Multikey settings show a bigger increase. Perhaps this setting could be further expanded upon (e.g. categorizing failures like retrieving a different needle, retrieving a number that doesn't appear in context, etc.) to help understand why this auxiliary state is helping more here compared to the other settings.
2 It is somewhat unclear if the performance gain is significant enough to warrant doubling the state size.
Questions for authors
- How important is channel-wise mixing as opposed to scalar mixing? I imagine this idea has been attempted by researchers in the past, but wonder if the design choice of channel-wise mixing is the main performance differentiator. It would be nice if a small 340M (or even 100-200M) ablation can be ran comparing all 4 combinations from channel-wise vs. scalar gamma / lambda.
- How does the overall VRAM cost compare between models when training? I understand each state now has double the activation footprint, but how does this translate to the overall model size?
- How does the task performance compare to using KDA / GDN-2 with equivalent or near-equivalent state sizes (i.e. doubling expand_v)? Of course FLOPs should be kept somewhat similar.
- Do you have any comments on how this idea in principle is related to momentum-based update rules?
4 comments
Q4. Comparison to Momentum-based Update Rules
Thank you for raising this connection. Momentum-based updates have also been explored in linear attention, for example in Momentum DeltaNet (MDN) [1]. Momentum combines current and past error corrections. This can help reinforce new information over several steps, even when later related tokens require only small corrections.
Our method instead preserves memory states at different timescales and adaptively mixes their readouts. The key distinction is that MDN uses its auxiliary state to guide memory updates, whereas our integral state stores a longer-term summary of past memory states that can be directly queried alongside the delta state. Our framework can augment chunkwise-trainable linear-attention backbones and can therefore, in principle, be combined with momentum-based methods such as MDN. This combination may offer complementary benefits, but whether it yields further improvements remains open.
Thanks for your time and consideration.
Best regards,
Authors
Reference
[1] Huang et al. MDN: Parallelizing Stepwise Momentum for Delta Linear Attention. ICML, 2026.
Thank you for these added experiments, this will strengthen the paper.
For W2, Q3) could you also evaluate these 3 models on S-NIAH-1/2/3? This should be straightforward as the models are already trained, and this is where I would imagine a larger improvement from state size would be, between KDA and KDA State x2.
Lastly, as a comment for future work, is Integral DeltaNet needed at every layer in the model for improved performance? With the larger compute costs for this layer, would we see any significant performance benefit when only using few Integral layers. For example, an 8-layer hybrid model could be: KKIKKIKK for K=KDA, I=Integral KDA. Though, I imagine certain layers benefit more from an integral state than others. I believe future readers will have similar questions to this. It would save time and resources for the community if you could run an experiment like this.
Thank you for your follow-up and helpful suggestions. Following your suggestion, we also evaluated the models compared in our response above—KDA, Integral KDA, and KDA State×2—on S-NIAH. Integral KDA outperforms both baselines across all evaluated settings, suggesting benefits beyond simply increasing state capacity.
| Model | S-NIAH 1, 1k | S-NIAH 1, 2k | S-NIAH 1, 4k | S-NIAH 1, 8k | S-NIAH 2, 1k | S-NIAH 2, 2k | S-NIAH 2, 4k | S-NIAH 2, 8k | S-NIAH 3, 1k | S-NIAH 3, 2k | S-NIAH 3, 4k |
|---|---|---|---|---|---|---|---|---|---|---|---|
| KDA | 87.6 | 42.6 | 38.4 | 16.0 | 98.4 | 45.8 | 31.6 | 13.6 | 40.4 | 7.8 | 0.4 |
| Integral KDA | 99.8 | 100.0 | 100.0 | 98.4 | 99.8 | 97.6 | 53.6 | 14.4 | 55.0 | 21.8 | 6.8 |
| KDA State×2 | 97.8 | 90.2 | 69.0 | 34.4 | 99.2 | 73.2 | 13.0 | 4.8 | 50.8 | 14.8 | 0.0 |
Thank you for suggesting hybrid layer configurations as a direction to explore. We agree that this would be useful for the community. Our limited compute budget makes this experiment difficult to complete during the rebuttal period, but we will consider including it in the next revision.
Thank you again for your time and consideration. Please feel free to let us know if you have any further questions or suggestions.
Best regards,
Authors
We would like to express our gratitude for your valuable comments. In the following, we address the points raised by the reviewer.
W1. Analysis on MK-NIAH
Thank you for suggesting this analysis. MK-NIAH requires retrieving the value for a queried key among multiple key–value pairs. Across the four models and tested lengths on MK-NIAH-1, most of failures involve retrieving another needle. Both Integral variants reduce these errors at every tested length.
Each cell reports all failures / failures from retrieving another needle's value out of 500 samples per setting, with the latter's share of all failures in parentheses.
These results suggest that Integral helps more with distinguishing the target needle from competing needles than with recognizing needles in the context. This may explain the larger gains on MK-NIAH than on S-NIAH.
W2, Q3. Comparison at Matched State Capacity
We compared Integral KDA with KDA State×2 at the 340M scale, using the same parameter budget and 15B-token training recipe. State×2 doubles the value dimension to match Integral KDA's total recurrent-state size. Integral KDA achieves higher average commonsense and retrieval accuracy, suggesting that our memory design is more effective for these tasks than simply enlarging the state.
Language Modeling (LM) and Commonsense Reasoning (CS)
Real-world Retrieval
Q1. Scalar versus Channel-wise Gates
Thank you for suggesting this ablation. We evaluated all four scalar/channel-wise combinations of the readout (γ) and integration (λ) gates using the KDA-based 340M model trained on 15B tokens. Channel-wise gating improves overall performance, and using it for both gates yields strong results across language modeling, commonsense reasoning, and retrieval. We have added the full ablation results for all four gate combinations to Appendix B.1 of the revised manuscript (see Table 6).
Q2. VRAM Cost During Training
Thank you for suggesting this comparison. We have added the training throughput and activation-memory comparisons to Section 4 of the revised manuscript (see Figure 4). The table uses the paper's throughput benchmark setup: 1.3B models on one B300, batch size 1, and a 32K-token sequence, without model-level activation checkpointing. KDA uses the FLA kernel; T and M denote throughput-first and memory-first.
Static memory changes little because the recurrent-state buffers are small. T saves intermediate activations to avoid recomputation, using more VRAM for higher throughput. M recomputes them during backward, reducing VRAM at the cost of lower throughput. We used T for the main experiments in the paper.