acceptodds
Under review as a conference paper at ICLR 2027

Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention

Abstract

Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active correction is still anchored to that same write address. As a result, stale information stored at a different address cannot be actively removed before new content is written elsewhere. We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write. The key insight is that recurrent memory models should not only correct the current write, but also selectively suppress outdated memory at an independently chosen address. Concretely, our method first applies a targeted erase step along a learned erase direction, and then performs the standard delta-style corrective write along the current write direction. This preserves the corrective behavior of delta-rule updates while expanding their memory-management capacity. Language-model pretraining experiments across dense 2.5B and MoE 25B-A2.8B model families show that EDA performs best in both settings. The gain persists after 80B-token long-context midtraining of the MoE models, where EDA also performs best in long-context evaluations from 4k to 128k contexts. A compact update analysis and memory-state probes suggest why: EDA keeps the delta-rule corrective write intact while allocating an additional cleanup path most strongly when passive decay is weak. These results suggest that recurrent memory models should decide not only what to write, but also what stale information to erase and where.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, delta rule, recurrent memory, efficient language models, hybrid model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3QrXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EDA modifies KDA by inserting an independently addressed erase factor between the diagonal decay and the standard delta correction/write. The goal is to let the recurrent state remove stale content more effectively.

Critical strengths
  • The authors train models at a scale few academic studies reach (400B tokens, up to a 25B-A2.8B MoE, plus 80B-token long-context midtraining).
Critical weaknesses
  • The novelty claim misses prior memory architectures with decoupled erasure. It is repeatedly presented that targeted erasure at an independent address as a "missing memory-management capability." This erase-then-write mechanism dates back to the Neural Turing Machine (Graves et al., 2014).
  • EDA appears to be a constrained special case of an existing approach that the paper argues is "orthogonal." Eq. (8) is gated DeltaProduct with n_h = 2 and the first value set to zero, and Appendix C even implements it the same way (two sub-steps per token).
  • The improvement is very close to GDN-2 and roughly indistinguishable from noise. At the dense 2.5B scale, EDA and GDN-2 are each better on 3 out of 6 benchmarks. EDA's +0.31 average lead (28.45 vs. 28.14) comes almost entirely from GSM8K (+2.5). Excluding GSM8K, GDN-2 averages higher (29.49 vs. 29.35).
  • More severely, GDN-2, the closest baseline in the dense setting, is missing from all MoE experiments.
  • An architecture paper should show how the proposed change behaves across model sizes under controlled budgets. There is no scaling analysis, and the two scales are not comparable.
  • Another point about a proper architecture paper: only hybrid models are evaluated. This is a serious confound, given that the paper's claim concerns how the recurrent state manages stale content.
  • No ablations isolate the proposed mechanism, and several confounds are left open.
  • The motivating scenario is never tested and is arguably already handled by the delta rule.
  • The related work coverage is insufficient. The related work section consists of two short paragraphs. There are many papers missing (too many to name here), and a proper research paper should adequately credit prior work.
  • About the writing: the presentation is highly repetitive. The core idea of "erase at an independent address, then perform the standard delta write" is restated around 20 times across nine pages, sometimes near-verbatim. For example, "stale information stored at a different address cannot be [actively/explicitly] removed/erased before new content is written elsewhere" appears at L015–017, L133–136, and L145–147. "This coupling is the limitation we target" (L145) reappears as "This coupling is the core limitation we address" (L191). The GDN-2 distinction is repeated five times. Eq. (5) and Eq. (7) are identical. Meanwhile the ablations listed above are absent. Condensing the repeated material would have left room for the missing evidence.

It is a clear reject now. Some of the changes, like a proper scaling analysis and better positioning of the paper within the literature, require a major revision.

Questions for authors

Check weaknesses

Reviewer_F8c6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Delta-rule linear attention is anchored to that same write address, so stale content stored elsewhere cannot be actively cleared. This paper proposes Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write. EDA first erases along a learned direction and then applies the standard delta-style corrective write along the current write direction, keeping the delta rule’s corrective behavior while giving the memory more room to manage itself. EDA performs best across dense 2.5B and MoE 25B-A2.8B model families, and update analysis and memory-state probes indicate that EDA preserves the delta-rule write while adding a cleanup path that is strongest when passive decay is weak.

Critical strengths
  1. The consistency between the problem and solution is clear, and the derivation of the erase operation is interesting (especially the part that views DeltaNet as a reconstruction loss) and well-reasoned.
  2. It addresses a clear problem: delta-rule linear attention is anchored to the same write address.
  3. Effectiveness is demonstrated through experiments on 400B tokens across both dense and MoE model families.
  4. Overall, writing is easy to understand.
Critical weaknesses
  1. There are no ablations. Variants such as reversed update order, training with e=k, and erase rank substitution experiments are absent, leaving the validation of "correct erase" insufficient.
  2. The claims are overclaimed. The abstract states it is best across 4k-128k, but in Table 3, EDA ranks first only at 4k-16k, and baseline1 outperforms it >32k length.
  3. Raw write-key recall seems like another important metric, but no numbers can be found for it. (Please let me know if it exists.)
  4. Despite being closely related work mentioned in the paper, comparisons with GDN-2 and a Transformer baseline (the nearest competitors in the delta-rule line) are missing, as are comparisons with delta-style variants DeltaProduct, RWKV-7, and Comba.
  5. In Table 1, no explanation is given for the inferior values in several cases, and only the average is prioritized.The fact that dense MMLU & MoE GSM8K rank last, and post mid-training GSM8K·EvalPlus scores fall below KDA is not addressed.
Questions for authors
  • How does eₜ locate stale content?

  • Is the gain over DeltaProduct due to the erase operation, or because there are two rank-1 factors?

Reviewer_VTrKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on channel-wise gated delta-rule linear attention (the KDA-style setting) and argues that the active correction in these models is always tied to the current write key. As a result, stale content stored somewhere else can only fade through decay; it cannot be removed directly. The proposed Erase-then-Delta Attention (EDA) adds an erase step before the usual delta write. The erase step uses its own learned direction and gate (from a low-rank projection) to suppress what the memory currently returns along that direction. The standard delta correction-and-write at the current key then follows unchanged. A bounded "safe gate" for the decay is also introduced, and the two-step update is implemented by interleaving each token into two sub-steps, so existing DPLR chunk-wise kernels can be reused. Experiments train 3:1 hybrid models at two scales, 400B tokens each: a dense 2.5B model compared against Transformer, GDN, GDN-2, and KDA, and an MoE 25B-A2.8B model compared against GDN and KDA. MoE results are also reported after 80B tokens of 32k midtraining, plus RULER up to 128k. Appendix probes look at how contraction strength is split between decay, delta correction and erase, and at what happens when the learned erase direction is swapped out at inference time.

Critical strengths
  1. A clean, well-motivated idea that is easy to plug in. The point that delta-rule models can only actively clean up at the address they are currently writing to is stated clearly, and the fix is minimal: one extra erase step in front of an otherwise unchanged delta write. The kernel trick of splitting each token into an erase sub-step and a write sub-step means EDA can reuse existing chunk-wise kernels, which makes it practical to adopt.
  2. The evaluation is at a scale most linear-attention papers don't reach. Training dense 2.5B and MoE 25B-A2.8B hybrids for 400B tokens, then doing 32k midtraining and RULER up to 128k, is expensive. EDA has the best overall score in all three tables. The gateallocation probe is also a nice touch: it suggests the erase path is used mostly in heads where decay is weak, which fits the intended "cleanup when nothing else forgets" story.
Critical weaknesses
  1. The contribution feels like an incremental extension of existing delta-rule variants. EDA is KDA with one extra rank-one contraction applied before the write. Viewed that way, each token gets two Householder-style steps, where the first one only erases. That looks very close to a gated DeltaProduct with two steps whose first step writes nothing. The paper even implements it the same way, by doubling the sequence into sub-steps. Yet DeltaProduct is described as an orthogonal "transition enrichment" line and isn't compared against. Similarly, RWKV-7 already uses a removal key that differs from its write key, which overlaps directly with the claim that existing methods only erase at the current write address. RWKV-7 is mentioned only as having a "richer transition parameterization." GDN-2 already separates erase and write strengths, so what remains new here is mainly giving the erase step its own learned direction. That may well be a useful design choice, but as presented it reads as a modest variation within a crowded family rather than a new capability. I'd want the paper to position itself much more directly against DeltaProduct and RWKV-7, ideally with at least one of them as a baseline.

  2. There's no end-to-end ablation showing that decoupling the address is what helps. The central claim is that erasing at an independently chosen address is better than erasing at the write address. But all the evidence for this is post-hoc: the appendix takes a trained EDA model and, at replay time, swaps the learned erase direction for a random one, a shuffled one, or the write key, then measures readout perturbation. That shows the trained model relies on its learned direction. It doesn't show that a model trained with the erase direction tied to the write key would do worse. The missing comparison is a model trained end to end with the same extra erase gate, but with the erase address forced to equal the write key, against the full EDA. This matters because EDA also adds parameters (an extra projection and gate) and an extra contraction per step, so the gains in Table 1 could come from "more contraction capacity" rather than "a different address." The benchmark gains are also fairly concentrated:

  • In the dense setting, most of the lead comes from GSM8K. On the other five tasks, EDA is roughly on par with or slightly behind GDN-2 (for example, the lowest MMLU in the group).
  • After midtraining, the lead over KDA is largely MATH, while EvalPlus drops.
  • On RULER, EDA is ahead only from 4k to 16k.

Without the tied-address ablation, I find it hard to attribute these gains to the mechanism the paper is about.

Questions for authors
  1. Could you train a tied-address variant? Specifically, the same EDA layer with the same extra erase gate and decay, but with the erase direction set equal to the write key, trained end to end under the same budget, at least at the dense 2.5B scale. This is the comparison that would tell us whether the independent address matters, as opposed to simply having an additional contraction step. If the tied version comes close, the story changes quite a bit.

  2. How does EDA relate to a two-step gated DeltaProduct, and to RWKV-7's separate removal key? Is there anything EDA can express that a two-step gated DeltaProduct, with the first step's write turned off, cannot? If not, would you be open to framing EDA as a constrained case of it and comparing the two directly? A short discussion of how EDA's independently projected erase direction differs from RWKV-7's removal key would also help.

  3. Can you show the effect on tasks that actually stress memory overwriting? The motivation is about variable reassignment, fact correction and context shifts, but the evaluation is mostly knowledge and reasoning benchmarks. Results on synthetic recall or state-tracking tasks with overwrites (e.g., MQAR-style recall with key updates), or language-modeling loss on long documents, would connect the mechanism to the gains much more directly. Also a small thing: GDN's EvalPlus score is exactly 50.09 both before and after midtraining, while all its other scores change. Is that correct?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.