acceptodds
Under review as a conference paper at ICLR 2027

Integral DeltaNet: Improved Linear Attention with Adaptive Memory Integration

26 views

Abstract

Linear attention replaces softmax attention with a linear formulation, enabling linear-time computation with a fixed-size recurrent state instead of a growing key–value cache. These computational and memory advantages have attracted growing interest in linear attention for long-context language modeling. However, compressing past information into a fixed-size state causes *memory collisions* that hinder long-context understanding. Although DeltaNet mitigates memory collisions through delta-rule updates and Gated DeltaNet extends this approach with gating, these mechanisms may also discard information relevant to future predictions. To address this limitation, we introduce *Integral DeltaNet*, which maintains two complementary recurrent states: a fast-updating *delta state* that captures current key–value associations and an *integral state* that preserves a weighted history of the delta state. At readout, the model adaptively combines these states to access information across different timescales. We further derive a chunkwise parallel formulation that enables hardware-efficient training. We evaluate Integral DeltaNet using 340M and 1.3B parameter models trained from scratch on 15B and 100B FineWeb-Edu tokens, respectively. Comparisons with strong recurrent baselines show overall improvements in language modeling, commonsense reasoning, real-world and synthetic retrieval, long-context understanding, and length extrapolation. Training throughput remains close to that of the backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
subquadratic model, linear attention, delta rule, gating, Gated DeltaNet, long context

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_5cp2OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

By pairing the fast-moving delta state with a slow-moving long-term memory state, Integral DeltaNet can alleviate forgetting induced by decay-based delta methods. This is a straightforward addition on top of previous delta models, leveraging a data-dependent moving average of past states and query-dependent mixing of short- and long-term information. The authors implement a fast kernel, showing comparable training throughput though with double the activation memory. Overall this method demonstrates strong performance on common sense reasoning, recall, and length extrapolation tasks.

Critical strengths
  1. Simple, Strong Method: The proposed update rule is easy to implement and yields strong empirical results on downstream language modeling tasks. The idea is simple and well-executed with principled design choices (channel-wise mixing) and kernel implementation.

  2. Presentation Clarity: This work was very straightforward and nice to read. The work was presented in the same template or format as others such as Gated DeltaNet, which is helpful for readers with a background in this space.

Critical weaknesses
  1. As a proposed fix to memory collisions, it seems mildly concerning that there isn’t a significant improvement in S-NIAH performance (and a slight decrease from GDN-2 SN1). Multikey settings show a bigger increase. Perhaps this setting could be further expanded upon (e.g. categorizing failures like retrieving a different needle, retrieving a number that doesn't appear in context, etc.) to help understand why this auxiliary state is helping more here compared to the other settings.

2 It is somewhat unclear if the performance gain is significant enough to warrant doubling the state size.

Questions for authors
  1. How important is channel-wise mixing as opposed to scalar mixing? I imagine this idea has been attempted by researchers in the past, but wonder if the design choice of channel-wise mixing is the main performance differentiator. It would be nice if a small 340M (or even 100-200M) ablation can be ran comparing all 4 combinations from channel-wise vs. scalar gamma / lambda.
  2. How does the overall VRAM cost compare between models when training? I understand each state now has double the activation footprint, but how does this translate to the overall model size?
  3. How does the task performance compare to using KDA / GDN-2 with equivalent or near-equivalent state sizes (i.e. doubling expand_v)? Of course FLOPs should be kept somewhat similar.
  4. Do you have any comments on how this idea in principle is related to momentum-based update rules?
4 comments

We would like to express our gratitude for your valuable comments. In the following, we address the points raised by the reviewer.

W1. Analysis on MK-NIAH

Thank you for suggesting this analysis. MK-NIAH requires retrieving the value for a queried key among multiple key–value pairs. Across the four models and tested lengths on MK-NIAH-1, most of failures involve retrieving another needle. Both Integral variants reduce these errors at every tested length.

Each cell reports all failures / failures from retrieving another needle's value out of 500 samples per setting, with the latter's share of all failures in parentheses.

MK-NIAH-1KDAIntegral KDAGDN-2Integral GDN-2
1K264 / 258 (97.7%)230 / 229 (99.6%)274 / 272 (99.3%)118 / 112 (94.9%)
2K315 / 307 (97.5%)248 / 240 (96.8%)336 / 326 (97.0%)164 / 159 (97.0%)
4K396 / 329 (83.1%)322 / 309 (96.0%)381 / 346 (90.8%)301 / 281 (93.4%)

These results suggest that Integral helps more with distinguishing the target needle from competing needles than with recognizing needles in the context. This may explain the larger gains on MK-NIAH than on S-NIAH.

W2, Q3. Comparison at Matched State Capacity

We compared Integral KDA with KDA State×2 at the 340M scale, using the same parameter budget and 15B-token training recipe. State×2 doubles the value dimension to match Integral KDA's total recurrent-state size. Integral KDA achieves higher average commonsense and retrieval accuracy, suggesting that our memory design is more effective for these tasks than simply enlarging the state.

Language Modeling (LM) and Commonsense Reasoning (CS)

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA acc ↑Hella. accₙ ↑Wino. acc ↑ARC-e acc ↑ARC-c accₙ ↑OBQA accₙ ↑SIQA acc ↑BoolQ acc ↑CS Avg. ↑
KDA24.8926.1636.466.941.253.459.427.333.639.057.946.1
Integral KDA24.9724.9936.968.440.752.958.228.034.838.859.546.5
KDA State×224.9524.4037.466.741.353.361.727.533.638.750.345.6

Real-world Retrieval

ModelSWDE ↑SQuAD ↑FDA ↑TriviaQA ↑NQ ↑DROP ↑Retr. Avg. ↑
KDA10.228.68.147.814.417.721.1
Integral KDA18.130.69.648.915.017.323.3
KDA State×215.130.57.646.814.019.122.2

Q1. Scalar versus Channel-wise Gates

Thank you for suggesting this ablation. We evaluated all four scalar/channel-wise combinations of the readout (γ\gamma) and integration (λ\lambda) gates using the KDA-based 340M model trained on 15B tokens. Channel-wise gating improves overall performance, and using it for both gates yields strong results across language modeling, commonsense reasoning, and retrieval. We have added the full ablation results for all four gate combinations to Appendix B.1 of the revised manuscript (see Table 6).

Q2. VRAM Cost During Training

Thank you for suggesting this comparison. We have added the training throughput and activation-memory comparisons to Section 4 of the revised manuscript (see Figure 4). The table uses the paper's throughput benchmark setup: 1.3B models on one B300, batch size 1, and a 32K-token sequence, without model-level activation checkpointing. KDA uses the FLA kernel; T and M denote throughput-first and memory-first.

ModelStatic memory (GiB)Saved for backward (GiB) ↓Peak training VRAM (GiB) ↓
KDA21.6186.5103.4
Integral KDA (T)21.61164.0180.9
Integral KDA (M)21.6150.171.4

Static memory changes little because the recurrent-state buffers are small. T saves intermediate activations to avoid recomputation, using more VRAM for higher throughput. M recomputes them during backward, reducing VRAM at the cost of lower throughput. We used T for the main experiments in the paper.

Q4. Comparison to Momentum-based Update Rules

Thank you for raising this connection. Momentum-based updates have also been explored in linear attention, for example in Momentum DeltaNet (MDN) [1]. Momentum combines current and past error corrections. This can help reinforce new information over several steps, even when later related tokens require only small corrections.

Our method instead preserves memory states at different timescales and adaptively mixes their readouts. The key distinction is that MDN uses its auxiliary state to guide memory updates, whereas our integral state stores a longer-term summary of past memory states that can be directly queried alongside the delta state. Our framework can augment chunkwise-trainable linear-attention backbones and can therefore, in principle, be combined with momentum-based methods such as MDN. This combination may offer complementary benefits, but whether it yields further improvements remains open.


Thanks for your time and consideration.

Best regards,

Authors


Reference

[1] Huang et al. MDN: Parallelizing Stepwise Momentum for Delta Linear Attention. ICML, 2026.

Thank you for these added experiments, this will strengthen the paper.

For W2, Q3) could you also evaluate these 3 models on S-NIAH-1/2/3? This should be straightforward as the models are already trained, and this is where I would imagine a larger improvement from state size would be, between KDA and KDA State x2.

Lastly, as a comment for future work, is Integral DeltaNet needed at every layer in the model for improved performance? With the larger compute costs for this layer, would we see any significant performance benefit when only using few Integral layers. For example, an 8-layer hybrid model could be: KKIKKIKK for K=KDA, I=Integral KDA. Though, I imagine certain layers benefit more from an integral state than others. I believe future readers will have similar questions to this. It would save time and resources for the community if you could run an experiment like this.

Thank you for your follow-up and helpful suggestions. Following your suggestion, we also evaluated the models compared in our response above—KDA, Integral KDA, and KDA State×2—on S-NIAH. Integral KDA outperforms both baselines across all evaluated settings, suggesting benefits beyond simply increasing state capacity.

ModelS-NIAH 1, 1kS-NIAH 1, 2kS-NIAH 1, 4kS-NIAH 1, 8kS-NIAH 2, 1kS-NIAH 2, 2kS-NIAH 2, 4kS-NIAH 2, 8kS-NIAH 3, 1kS-NIAH 3, 2kS-NIAH 3, 4k
KDA87.642.638.416.098.445.831.613.640.47.80.4
Integral KDA99.8100.0100.098.499.897.653.614.455.021.86.8
KDA State×297.890.269.034.499.273.213.04.850.814.80.0

Thank you for suggesting hybrid layer configurations as a direction to explore. We agree that this would be useful for the community. Our limited compute budget makes this experiment difficult to complete during the rebuttal period, but we will consider including it in the next revision.


Thank you again for your time and consideration. Please feel free to let us know if you have any further questions or suggestions.

Best regards,

Authors

Reviewer_msQxOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the forgetting mechanisms in modern linear attention, decay in Mamba-2 and targeted overwriting in DeltaNet and its gated variants, reduce memory collisions but can also erase associations that later queries still need. It proposes Integral DeltaNet. This keeps a fast delta state updated with the KDA or GDN-2 rule and adds an integral state, a channel-wise exponential moving average of past delta states, with an input-dependent channel-wise gate mixing reads from the two at output time. A three-key example in two dimensions shows that any single-state linear readout has retrieval risk of at least 1/3, while the two-state model with suitable gates reaches zero. The authors derive a chunkwise parallel form built on retention-weighted prefix operators and report 87 to 94% of backbone training throughput. Integral KDA and Integral GDN-2 are trained from scratch at 340M on 15B tokens and at 1.3B on 100B tokens of FineWeb-Edu, then compared with Mamba-2, DeltaNet, GDN, KDA, GDN-2 and Transformer++ on perplexity, zero-shot commonsense, real-world retrieval, RULER NIAH, LongBench and length extrapolation. The paper reports improvements over the matching backbones on most of these, with the clearest gains for Integral KDA at 1.3B.

Critical strengths
  1. Simple mechanism that fits existing kernels. The integral state is just a channel-wise EMA of the delta state, and the two are combined only at readout. The KDA or GDN-2 update and its WY/UT machinery therefore stay untouched. The chunkwise derivation through retention-weighted prefix operators is clean, and training throughput stays at 87 to 94% of the backbones, which makes this a realistic drop-in for existing linear attention stacks.

  2. Gains show up where the motivation predicts. On MK-NIAH-1, which stresses many competing associations, both variants improve at every length (Integral GDN-2 goes from 45.2 to 76.4 at 1K and from 32.8 to 67.2 at 2K). Figure 3 shows both variants with lower perplexity than their backbones at every length beyond 4K on all six extrapolation datasets. Integral KDA at 1.3B is also consistently better on WikiText and LAMBADA perplexity, the commonsense average (+1.5) and the retrieval average (+1.9). The toy example is correct as stated, and the Appendix A derivations check out down to the -23/512 margin that lets X_3 recover the first answer.

Critical weaknesses
  1. The mechanism is not separated from added capacity. Each head now carries two d_v × d_k matrices, so the recurrent state is doubled, yet every baseline keeps a single state and decoding memory and latency are never reported. This matters because the largest retrieval jumps are on SWDE and FDA, the tasks with the biggest gap to Transformer++ and the ones most likely to benefit from extra state. The toy example has the same issue. The separation in Proposition 2 comes from a query-dependent γ choosing between two different linear maps, and even a plain linear attention state run alongside the GDN state gives the correct answer for k1 there (m1 = -1). There are no ablations on the gates (query-dependent vs. constant γ, channel-wise vs. scalar λ, an independently decayed slow state in place of the EMA), no analysis of the learned timescales, and no seeds or error bars.

  2. Long-context claims go further than the tables. On LongBench the 1.1-point gain for Integral KDA comes almost entirely from TREC (29.5 to 52.5). Over the remaining 13 tasks it is slightly below KDA, losing about 6 points on each code task, while Integral GDN-2 is only 0.2 above GDN-2 and many scores sit near floor at this scale. On NIAH, Integral GDN-2 falls from 97.8 to 85.8 on S-NIAH-1 at 8K and from 74.2 to 65.2 on S-NIAH-2 at 4K, Integral KDA drops about 10 points on S-NIAH-3 at 2K, and at 340M both Integral variants have worse WikiText perplexity than their backbones. With per-task swings this large and no variance estimates, the abstract’s claim of overall improvements in long-context understanding should be toned down or backed with multiple seeds. Also in general, it seems like adding this method on top of KDA benefits more than adding on top of GDN-2. It's not clear why.

Questions for authors
  1. Can you add a backbone with a matched recurrent state, for example GDN-2 with twice the key dimension or twice the heads at equal parameter count, and report decoding memory and per-token latency for all models? A comparison with log-linear attention (Guo et al., 2025), which also keeps extra states covering different spans of the past on top of Gated DeltaNet, would help position the method.

  2. How does performance change if γ is a learned constant instead of input-dependent, if λ is one scalar per head, or if X_t is replaced by an independently updated slow state such as a second KDA or plain linear attention state?

  3. What do the learned λ and γ look like after training? Do some heads or channels settle on long timescales and read mostly from X_t, or does the readout stay close to S_t as at initialization? How sensitive are results to the 1/1024 initialization?

  4. What explains the 23-point TREC jump for Integral KDA alongside its drops on both code tasks, and the S-NIAH-1 and S-NIAH-2 regressions for Integral GDN-2? Are any of these differences larger than seed-to-seed variance?

  5. How are P̄ and H̄ computed in the kernel without materializing all prefix matrices? How do you keep the within-chunk retention products numerically stable, and where does the roughly 2x activation memory at 32K come from?

8 comments

We express our gratitude for your valuable comments. In the following, we address the points raised by the reviewer.

W1, Q1. State Matching and decoding

Matched-state comparison. Thank you for suggesting this comparison. We compare Integral KDA with KDA State×2 at 340M, using the same parameter budget and 15B-token training recipe. State×2 doubles the value dimension to match Integral KDA's total recurrent-state size. Integral KDA achieves higher average commonsense and retrieval accuracy, suggesting that our memory design is more effective for these tasks than simply enlarging the state.

Language Modeling (LM) and Commonsense Reasoning (CS)

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA ↑Hella. accₙ ↑Wino. ↑ARC-e ↑ARC-c accₙ ↑OBQA accₙ ↑SIQA ↑BoolQ ↑CS Avg. ↑
KDA24.8926.1636.466.941.253.459.427.333.639.057.946.1
Integral KDA24.9724.9936.968.440.752.958.228.034.838.859.546.5
KDA State×224.9524.4037.466.741.353.361.727.533.638.750.345.6

Real-world Retrieval

ModelSWDE ↑SQuAD ↑FDA ↑TriviaQA ↑NQ ↑DROP ↑Retrieval Avg. ↑
KDA10.228.68.147.814.417.721.1
Integral KDA18.130.69.648.915.017.323.3
KDA State×215.130.57.646.814.019.122.2

Decoding memory and latency. Thank you for suggesting these comparisons. We have added the decoding memory and per-token latency results, together with the measurement setup, to Section 4 of the revised manuscript (see Table 5). The additional state occupies only a small fraction of total memory, so the Integral variants have similar decoding memory to their backbones, with some additional latency. Both costs remain nearly constant as context length grows, whereas those of Transformer++ increase.

W1, Q2. Gating ablation and an independently updated slow state

Thank you for suggesting these ablations. We have added the ablation results for all four scalar/channel-wise gate combinations, including per-head scalar λ\lambda, to Appendix B.1 of the revised manuscript (see Table 6). The channel-wise/channel-wise configuration achieves strong overall performance.

Constant readout gate. We removed input dependence from the readout gate by setting γt=γˉ\gamma_t=\bar{\gamma} for all tt, with γˉ\bar{\gamma} learned per channel and shared across tokens, leaving the integral update and its input-dependent λt\lambda_t unchanged. We trained this variant at 340M on 15B tokens with the same recipe. The constant gate retains a modest retrieval gain over KDA but underperforms input-dependent readout on both averages, highlighting the importance of adaptive readout.

Language Modeling (LM) and Commonsense Reasoning (CS)

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA ↑Hella. accₙ ↑Wino. ↑ARC-e ↑ARC-c accₙ ↑OBQA accₙ ↑SIQA ↑BoolQ ↑CS Avg. ↑
KDA24.8926.1636.466.941.253.459.427.333.639.057.946.1
Integral KDA24.9724.9936.968.440.752.958.228.034.838.859.546.5
Integral KDA, constant readout gate25.2625.0436.767.140.750.559.027.733.839.058.545.9

Real-world Retrieval

ModelSWDE ↑SQuAD ↑FDA ↑TriviaQA ↑NQ ↑DROP ↑Retrieval Avg. ↑
KDA10.228.68.147.814.417.721.1
Integral KDA18.130.69.648.915.017.323.3
Integral KDA, constant readout gate12.728.99.646.913.219.521.8

Independent slow state. We replaced the EMA with a second KDA state with its own decay and erase/write gates, sharing queries, keys, and values with the first state. At 340M/15B tokens with the same training recipe and matched parameter budgets, Integral KDA achieves lower perplexities and higher commonsense and retrieval averages, supporting the benefit of integrating the delta-state history.

Language Modeling (LM) and Commonsense Reasoning (CS)

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA ↑Hella. accₙ ↑Wino. ↑ARC-e ↑ARC-c accₙ ↑OBQA accₙ ↑SIQA ↑BoolQ ↑CS Avg. ↑
KDA24.8926.1636.466.941.253.459.427.333.639.057.946.1
Integral KDA24.9724.9936.968.440.752.958.228.034.838.859.546.5
Two KDA states with adaptive readout25.1025.4036.365.940.852.057.927.134.639.461.746.2

Real-world Retrieval

ModelSWDE ↑SQuAD ↑FDA ↑TriviaQA ↑NQ ↑DROP ↑Retrieval Avg. ↑
KDA10.228.68.147.814.417.721.1
Integral KDA18.130.69.648.915.017.323.3
Two KDA states with adaptive readout13.029.77.648.614.918.222.0

W1, Q3. Learned Timescales and Readout

Thank you for suggesting these analyses. We analyze both gates in Integral KDA 1.3B on S-NIAH-2 with a 4K context.

Learned timescales. The λ\lambda distribution, averaged over 64 held-out prompts, shows that the model learns a broad range of retention times:

λ\lambda percentile25th50th75th
λ\lambda0.000140.001070.00710
Corresponding timescale 1/λ1/\lambda (tokens)7,155939142

The median nevertheless remains close to initialization (939 vs. 1,024 tokens). We therefore believe that initialization should reflect training context length, with longer contexts potentially benefiting from a longer initial timescale.

Learned readout behavior. The readout gate γ\gamma departs from its initialization near one: distant needles shift the readout toward the integral state XtX_t, while recent needles favor the delta state StS_t.

Below, we report the mean answer-position readout gate γ\gamma for distant and recent needles, averaging over samples within each group, heads, key channels, and all 18 layers. We have added this position-dependent readout analysis, including the detailed evaluation setup and layer-wise results, to Appendix B.2 of the revised manuscript (see Figure 5). This shift toward the integral state for distant needles supports our motivation of adaptively accessing historical information.

Needle groupTokens before the answerMean answer-position γ\gamma, all layers
Distant needle3,122–3,9220.465
Recent needle44–1320.534

W1, W2, Q4. Performance and No seed

Overall performance. Thank you for pointing this out. We have toned down the abstract in the revised manuscript to emphasize overall improvements. We agree that gains vary across tasks, while the consistent gains on MK-NIAH-1 and length extrapolation align with our motivation of preserving useful information across timescales.

Seed variability. We use greedy decoding during evaluation, so there is no token-sampling variance. We use a single training seed, as is conventional in this line of work given the cost of large-scale pretraining. Due to the time constraints of the rebuttal period, we cannot add multi-seed results at this stage, but will consider them for the final revision.

Integral KDA on LongBench. TREC is a few-shot classification task in which preceding demonstrations provide question–label associations. Better retention of these associations may partly explain the large gain, although this remains a hypothesis and the gain may vary across training seeds.

Code completion and S-NIAH. Code completion requires precise syntax and identifier use, so retaining historical states may offer limited additional benefit. S-NIAH-1/2 retrieve a single number and often achieve high baseline accuracy at shorter lengths, leaving less room for improvement. We believe these characteristics may partly explain the lack of improvement on these tasks.

Differences between backbones. GDN-2 appears to benefit less overall than KDA. We suspect that decoupling erase and write already helps preserve useful information during updates, reducing the additional benefit of the integral state.

Q1. Comparison to Log-Linear Attention

Thank you for suggesting this comparison. Log-Linear Attention (Guo et al., 2025) addresses memory limitations by maintaining a set of recurrent states whose number grows logarithmically with context length. In contrast, Integral DeltaNet maintains fixed-size states, keeping its recurrent memory footprint independent of context length. Within this fixed memory budget, the two states capture information at different timescales, and an input-dependent gate adaptively combines their readouts to access information relevant to the current query.

Reviewer_oe8iOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper augments DeltaNet-like state with an additional state branch, which is a moving average of history state. They further propose to introduce a new input-dependent gate to aggregate the information of two recurrent states at readout. The authors develop the corresponding parallel formulation for their method. They train 340M/1.3B models on 15B/100B tokens and evaluate their method across LM, retrieval, and length extrapolation tasks.

Critical strengths
  1. The proposed method is easy to follow, and they provide a clear chunkwise formulation for implementation.
  2. They provided broad evaluation. Two scales with two token budgets, and their evaluation covers comprehensive sets of tasks for language models.
Critical weaknesses
  1. The technical novelty is quite limited. The main contribution is an EMA state over recurrent state history. They mentioned the structural similarity to previous work, but overall the distinction is narrow. The ablation is not enough to validate the other design; it is unclear whether a simple EMA state could improve for the performance of DeltaNet or not. Similarly, alternatives such as EMA over raw input state, vtkt⊤v_tk_t^\top, are also not tested (without the deltarule).

  2. The improvement is uneven and not significant for most tasks. The integral variants are relatively stable but sometimes regress, e.g. Integral GDN vs GDN in Table 2.

  3. Another main concern is that the mechanism is not technically supported. "Fast" and "slow" are asserted narratively, not measured. It is unclear how distinct the 'fast' and 'slow' states are, like the overlap between what SS and XX retrieve (cosine similarity of readouts, or mutual information), how much of XX is redundant with SS versus genuinely recovering erased information. There are no technical or empirical supports to these designs.

  4. Is the comparison fair for vanilla Deltanet variants? e.g., will direct doubling the state size or head number outperform integral variants?

Questions for authors
  1. Does a plain EMA branch, without the input-dependent gate, already improve DeltaNet?

  2. How does an EMA over the raw outer products vtkt⊤v_t k_t^\top (no delta rule) compare?

  3. How the gate distributes weight across tasks and positions?

  4. Natural controls would be DeltaNet/GDN baselines with doubled state or more heads. It is necessary to report parameter count, state size, and throughput for all models so the comparison is matched?

9 comments

Thank you for your valuable comments. In the following, we address the points raised by the reviewer.

W1. Technical Novelty and Comparison to Previous Work

While prior work has explored enlarging recurrent states and using memory modules at different timescales, our contribution is to extend efficiently parallelizable linear attention with adaptive readout across timescales.

Our chunkwise parallel derivation enables efficient, scalable training with throughput close to the backbones. Consistent gains in multi-key retrieval and length extrapolation, together with position-dependent readout weights, provide empirical support for the benefits of accessing information across different timescales.

W1, Q1. A constant readout gate, without input dependence

We interpreted Q1 as removing input dependence only from the readout gate by setting γt=γˉ\gamma_t = \bar{\gamma} for all tt, with γˉ\bar{\gamma} learned per channel and shared across tokens, while leaving the integral update, including input-dependent λt\lambda_t, unchanged. We trained this variant at 340M on 15B tokens.

The constant gate retains a modest retrieval gain over KDA but underperforms input-dependent readout on average commonsense and retrieval accuracy, highlighting the importance of adaptive readout. We would appreciate clarification if you intended a different control.

Language Modeling (LM) and Commonsense Reasoning (CS)

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA acc ↑Hella. accₙ ↑Wino. acc ↑ARC-e acc ↑ARC-c accₙ ↑OBQA accₙ ↑SIQA acc ↑BoolQ acc ↑CS Avg. ↑
KDA24.8926.1636.466.941.253.459.427.333.639.057.946.1
Integral KDA24.9724.9936.968.440.752.958.228.034.838.859.546.5
Integral KDA, constant readout gate25.2625.0436.767.140.750.559.027.733.839.058.545.9

Real-world Retrieval

ModelSWDE ↑SQuAD ↑FDA ↑TriviaQA ↑NQ ↑DROP ↑Retr. Avg. ↑
KDA10.228.68.147.814.417.721.1
Integral KDA18.130.69.648.915.017.323.3
Integral KDA, constant readout gate12.728.99.646.913.219.521.8

W1, Q2. EMA over the raw outer products, without the delta rule

We interpreted Q2 as replacing the delta-state history with raw outer products only in the EMA update:

\text{Original:}\quad X_t &= \mathrm{diag}(1-\lambda_t)X_{t-1}+\mathrm{diag}(\lambda_t)S_t,\\\\ \text{Control:}\quad X_t &= \mathrm{diag}(1-\lambda_t)X_{t-1}+\mathrm{diag}(\lambda_t)k_t v_t^{\top}. \end{aligned}$$ We trained this variant with the same 340M, 15B-token setup, keeping the delta-state update and both gates unchanged. The raw-outer-product variant does not improve average commonsense or retrieval accuracy over KDA, whereas Integral KDA improves both, highlighting the benefit of integrating delta-state history rather than raw key–value outer products. We would appreciate clarification if you intended a different control. **Language Modeling (LM) and Commonsense Reasoning (CS)** | **Model** | **Wiki. ppl ↓** | **LMB. ppl ↓** | **LMB. acc ↑** | **PIQA acc ↑** | **Hella. accₙ ↑** | **Wino. acc ↑** | **ARC-e acc ↑** | **ARC-c accₙ ↑** | **OBQA accₙ ↑** | **SIQA acc ↑** | **BoolQ acc ↑** | **CS Avg. ↑** | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | KDA | **24.89** | 26.16 | 36.4 | 66.9 | **41.2** | **53.4** | **59.4** | 27.3 | 33.6 | **39.0** | 57.9 | 46.1 | | Integral KDA | 24.97 | **24.99** | **36.9** | **68.4** | 40.7 | 52.9 | 58.2 | **28.0** | **34.8** | 38.8 | 59.5 | **46.5** | | Integral KDA, EMA of $k_t v_t^{\top}$ | 25.37 | 25.78 | 36.3 | 66.4 | 41.1 | 51.9 | 58.0 | 27.0 | 34.2 | 38.1 | **59.9** | 45.9 | **Real-world Retrieval** | **Model** | **SWDE ↑** | **SQuAD ↑** | **FDA ↑** | **TriviaQA ↑** | **NQ ↑** | **DROP ↑** | **Retr. Avg. ↑** | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | KDA | 10.2 | 28.6 | 8.1 | 47.8 | 14.4 | **17.7** | 21.1 | | Integral KDA | **18.1** | **30.6** | 9.6 | **48.9** | **15.0** | 17.3 | **23.3** | | Integral KDA, EMA of $k_t v_t^{\top}$ | 10.9 | 27.8 | **10.4** | 46.9 | 14.1 | 15.5 | 20.9 |

W2. Performance Improvement

We acknowledge that the improvements are not uniform across all benchmarks. However, our models improve performance on most tasks, while the decreases on some tasks are generally modest, yielding overall gains over their corresponding backbones.

While our LongBench results are mixed across tasks, we observe consistent gains on MK-NIAH-1 and length extrapolation, which evaluate multi-key retrieval and language modeling beyond the training context length, respectively. Both Integral variants outperform their backbones on MK-NIAH-1 across all evaluated context lengths. In length extrapolation, both variants achieve lower perplexity than their backbones across all six datasets beyond 4K context length.

These results support our motivation of retaining useful information over long contexts, with the most consistent evidence coming from multi-key retrieval and length extrapolation.

W3, Q3. Empirical analysis of adaptive readout

We examined whether the learned readout weights adapt to the distance of the information being retrieved. Our design suggests lower γ\gamma for distant needles, giving relatively more mixing weight to the integral state XX, and higher γ\gamma for nearby needles, shifting weight toward the delta state SS.

We analyzed Integral KDA 1.3B on S-NIAH-2 at 4K, where it improves accuracy over KDA from 70.0% to 99.6%. The table summarizes γ\gamma at the answer position, averaged over all 18 layers, heads, and key channels, for distant and nearby needles. We have added the detailed setup and layer-wise results of this analysis to Appendix B.2 of the revised manuscript (see Figure 5).

Needle groupDistance to answer position (tokens)Mean γ\gamma
Distant3,122–3,9220.465
Recent44–1320.534

On average, the gate follows the expected trend: distant needles receive relatively more mixing weight from XX, while nearby needles shift weight toward SS. The layer-wise results in the appendix show the same trend in most layers, with a clearer separation in deeper layers, providing empirical support for position-dependent readout.

W4, Q4. Comparison at matched parameter and state budgets

The efficiency results newly added to Section 4 of the revised manuscript (see Figure 4 and Table 5) show that the integral state adds manageable training and decoding overhead. Thus, matching recurrent-state size need not be a prerequisite for a fair comparison.

Still, matching state size is valuable for distinguishing the benefit of our memory design from that of added capacity. We therefore compare Integral KDA with KDA State×2, which doubles the value dimension to match Integral KDA's total recurrent-state size. All three models use the same parameter budget and 15B-token training recipe.

Integral KDA achieves higher average commonsense and retrieval accuracy than KDA State×2 at matched parameter and state budgets. These results suggest that our memory design is more effective for these tasks than simply doubling the recurrent state.

Language Modeling (LM) and Commonsense Reasoning (CS)

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA acc ↑Hella. accₙ ↑Wino. acc ↑ARC-e acc ↑ARC-c accₙ ↑OBQA accₙ ↑SIQA acc ↑BoolQ acc ↑CS Avg. ↑
KDA24.8926.1636.466.941.253.459.427.333.639.057.946.1
Integral KDA24.9724.9936.968.440.752.958.228.034.838.859.546.5
KDA State×224.9524.4037.466.741.353.361.727.533.638.750.345.6

Real-world Retrieval

ModelSWDE ↑SQuAD ↑FDA ↑TriviaQA ↑NQ ↑DROP ↑Retr. Avg. ↑
KDA10.228.68.147.814.417.721.1
Integral KDA18.130.69.648.915.017.323.3
KDA State×215.130.57.646.814.019.122.2

We hope our response helps to resolve any concerns.

Best regards,

Authors

Thank you for the additional experiments. However, the new results do not address my original concern and, in fact, raise further questions about the fairness of your comparisons.

I am concerned that the experiments still do not properly account for the role of key and value dimensions in determining memory capacity. In your new experiments, you doubled the value dimension dvd_v while leaving the key dimension dkd_k unchanged. However, it is widely understood in the linear attention community that memory capacity is primarily constrained by the key dimension, as discussed in the DeltaRule paper [1]:

"The dot product will attend to more than one key and return a linear combination of values. With keys embedded in a d_dot space, there cannot be more than d_dot orthogonal vectors."

In other words, the key dimension fundamentally limits the number of distinguishable key representations and, consequently, the capacity for storing and retrieving key-value associations. Increasing dvd_v alone does not address this limitation.

Your method increased both the key and value dimensions, while the new experiments only increased the value dimension. Neither setting provides a convincing comparison that properly isolates the effect of the proposed method under a comparable memory capacity.

Furthermore, the fact that simply doubling the value dimension already yields a substantial improvement raises another concern. Given that the gains from your proposed method under the original setting are relatively marginal, these results call into question whether the proposed method is necessary or offers meaningful benefits over straightforward architectural scaling.

[1] Linear Transformers Are Secretly Fast Weight Programmers, Schlag et al.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. We have revised the manuscript with the following additions:

  • Training throughput and activation-memory comparisons, along with decoding memory and per-token latency (Section 4).
  • Ablations of the integration and readout gates (Appendix B.1).
  • Analysis of position-dependent readout (Appendix B.2).

Thank you again for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.