- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
EDA modifies KDA by inserting an independently addressed erase factor between the diagonal decay and the standard delta correction/write. The goal is to let the recurrent state remove stale content more effectively.
Critical strengths
- The authors train models at a scale few academic studies reach (400B tokens, up to a 25B-A2.8B MoE, plus 80B-token long-context midtraining).
Critical weaknesses
- The novelty claim misses prior memory architectures with decoupled erasure. It is repeatedly presented that targeted erasure at an independent address as a "missing memory-management capability." This erase-then-write mechanism dates back to the Neural Turing Machine (Graves et al., 2014).
- EDA appears to be a constrained special case of an existing approach that the paper argues is "orthogonal." Eq. (8) is gated DeltaProduct with n_h = 2 and the first value set to zero, and Appendix C even implements it the same way (two sub-steps per token).
- The improvement is very close to GDN-2 and roughly indistinguishable from noise. At the dense 2.5B scale, EDA and GDN-2 are each better on 3 out of 6 benchmarks. EDA's +0.31 average lead (28.45 vs. 28.14) comes almost entirely from GSM8K (+2.5). Excluding GSM8K, GDN-2 averages higher (29.49 vs. 29.35).
- More severely, GDN-2, the closest baseline in the dense setting, is missing from all MoE experiments.
- An architecture paper should show how the proposed change behaves across model sizes under controlled budgets. There is no scaling analysis, and the two scales are not comparable.
- Another point about a proper architecture paper: only hybrid models are evaluated. This is a serious confound, given that the paper's claim concerns how the recurrent state manages stale content.
- No ablations isolate the proposed mechanism, and several confounds are left open.
- The motivating scenario is never tested and is arguably already handled by the delta rule.
- The related work coverage is insufficient. The related work section consists of two short paragraphs. There are many papers missing (too many to name here), and a proper research paper should adequately credit prior work.
- About the writing: the presentation is highly repetitive. The core idea of "erase at an independent address, then perform the standard delta write" is restated around 20 times across nine pages, sometimes near-verbatim. For example, "stale information stored at a different address cannot be [actively/explicitly] removed/erased before new content is written elsewhere" appears at L015–017, L133–136, and L145–147. "This coupling is the limitation we target" (L145) reappears as "This coupling is the core limitation we address" (L191). The GDN-2 distinction is repeated five times. Eq. (5) and Eq. (7) are identical. Meanwhile the ablations listed above are absent. Condensing the repeated material would have left room for the missing evidence.
It is a clear reject now. Some of the changes, like a proper scaling analysis and better positioning of the paper within the literature, require a major revision.
Questions for authors
Check weaknesses