- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
The paper introduces Oscillatory Delta Networks (ODN), which apply volume-preserving state transport (rotations or special-linear flows) before each delta-rule write to address key interference in DeltaNet-family models.
-
Core contributions: (i) proof that oscillation-based addressing reaches the theoretical memory-capacity limit even when all writes share one key; (ii) two architectures — ODN-SO(2) (rotational orbits) and ODN-SL(2) (learned orbit geometry); (iii) empirical gains on synthetic tasks and 340M-parameter language modeling.
Critical strengths
-
Rigorous theoretical characterization. Propositions 1–5 form a complete chain: from problem diagnosis (key correlation → interference) to mechanism design (oscillatory addressing) to capacity guarantees (exact recall of d values under one key). This elevates the contribution from an architectural tweak to a principled understanding of associative memory capacity in linear attention.
-
Novel and broadly applicable mechanism. A shared, content-dependent clock driving a spectrum of state-space oscillators is conceptually fresh and not limited to ODN; it provides a qualitatively different design axis in the crowded delta-rule literature.
Critical weaknesses
-
Lack of discussion of related work like LRU [1] and MDN [2]. Its complex-valued diagonal transitions are conceptually identical to ODN-SO(2) block-wise rotations; without explaining how a shared real-valued clock differs from independently learned complex eigenvalues, the contribution appears incremental.
-
Experiments lack 1B+ parameter results, it is impossible to tell whether theoretical capacity gains strengthen or wash out at scale. Lack of analysis for the perplexity–accuracy trade-off. ODN-SO(2) achieves worse WikiText-2 perplexity (34.02 vs. 31.52) than Gated DeltaNet yet better downstream accuracy; without explaining whether this is a fundamental tension or a training artifact, the mechanism's practical reliability is unclear.
-
Lack of a "mechanism-only" ablation. Without a control adding comparable parameters but no oscillatory transport, gains may stem from extra capacity rather than the mechanism itself.
-
Lack of justification for the hand-designed hyperbolic guard. Equation 6 uses a manually constructed decay rate; without explaining why this must be hand-engineered rather than learned, the core claim of "learning orbit geometry" is internally inconsistent.
-
The citation on lines 48–49 is broken across a line break in a misleading way.
-
The paper introduces ODN-SO(2) and ODN-SL(2) without explaining the group-theoretic suffixes for readers outside the Lie-algebra audience.
[1] Resurrecting recurrent neural networks for long sequences. ICML, 2023.
[2] MDN: Parallelizing Stepwise Momentum for Delta Linear Attention. ICML, 2026.
Questions for authors
see weaknesses
8 comments
Reply to Reviewer rMAD - W1 - Part 2
Response to Reviewer 3 — W1, continued
Lack of discussion of related work like LRU [1] and MDN [2]. Its complex-valued diagonal transitions are conceptually identical to ODN-SO(2) block-wise rotations; without explaining how a shared real-valued clock differs from independently learned complex eigenvalues, the contribution appears incremental.
MDN.
MDN accumulates the delta-rule gradient in a momentum state which then updates the memory as: If we regroup momentum and state vectors of a -dimensional MDN we can compare it with a -dimensional ODN:
| ODN-SO(2) | MDN | |
|---|---|---|
| state | ||
| transport | ||
| delta | ||
| readout |
The two models differ in the following ways:
(i)
For each output coordinate MDN doubles the recurrent state by adding the momentum variable. The dimensional keys and queries in MDNs are effectively expanded to the latent space introducing a predefined coupling between those two dimensions and the keys and queries. MDN then also only reads half of the memory state via read direction . In contrast, ODN learns -dimensional keys and queries that can flexibly address and read every state coordinate.
(ii)
In MDN the write direction , and the residual direction are different as the delta rule acts through the momentum term. In contrast in ODN the delta rule directly acts on the memory state providing a more consistent memory management mechanism.
(iii)
MDN's transition is upper triangular with real eigenvalues and , so it does not rotate. Complex eigenvalues can only arise indirectly through the delta-rule erasing memory along the current key. In contrast, ODN's transition is a rotation with eigenvalues , which turns every stored address regardless of the current key. This separates memory transform from memory update.
(iv)
ODNs learns each 2x2 block's frequency independently, while MDN provides the same transformation at each block. This provides ODNs with a more fine grained control of the memory transformation, including adjusting the spectrum to facilitate or suppress memory interferences as outlined in Propositions 3 - 5.
In summary, MDN and ODN are related but organize their recurrent states differently. In the matched-state comparison above, both models store the same number of recurrent-state elements, but MDN's keys and queries have only half as many coordinates as ODN's: MDN reserves the other half of its state for momentum, which its queries cannot read directly. ODN's full-width keys and queries, together with its learned rotations, provide a flexible mechanism for addressing and separating stored memories. The approximately orthogonal memory addresses observed in the Lag-Indexed Overwrite Recall Task are consistent with this mechanism (see also our reply to Reviewer 1).
Reply to Reviewer rMAD - W2a
Reviewer 3: W2a.
Experiments lack 1B+ parameter results, it is impossible to tell whether theoretical capacity gains strengthen or wash out at scale.
The revised manuscript now includes 1B-parameter results in Sections 5.4–5.5. On short-context zero-shot accuracy, ODN-SO(2) is comparable to Gated DeltaNet and KDA (Table 4: 59.19, 59.19, and 59.64, respectively). The clearest gains appear on long-context benchmarks, where memory load accumulates.
The revised manuscript now includes 1B-parameter results in Sections 5.4–5.5. On short-context zero-shot accuracy, ODN-SO(2) is comparable to Gated DeltaNet and KDA (Table 4: 59.19, 59.19, and 59.64, respectively). Intriguingly, ODN achieves significant gains on long-context benchmarks, where memory load accumulates.
The new manuscript Tables 5–6 are reproduced below and show PG-19 prediction under continuous processing and 2K-token state resets; Table 6 reports LongBench scores.
Under continuous processing, ODN-SO(2) maintains a 64–128K PG-19 perplexity of 14.25, compared with 37.55 for Gated DeltaNet and 75.11 for KDA. Resetting the state every 2K tokens improves both baselines but increases ODN-SO(2)'s perplexity to 15.44. These results support greater stability as memory load accumulates and useful retention beyond 2K tokens.
On LongBench, ODN-SO(2) leads on 12 of 14 tasks and achieves the highest average score (14.38, versus 12.44 for Gated DeltaNet and 8.99 for KDA).
Thus, the gains remain apparent at 1B parameters and are strongest when memories accumulate over long contexts, consistent with ODN's design principle of using learned memory transport to reorganize stored information and manage memory interference over time. We now state this distinction explicitly in Section 5.4.
Manuscript Table 5. PG-19 token perplexity () on the same 25 books: continuous processing versus state resets every 2K tokens. Columns show 2K position intervals and the 64–128K aggregate (K = 1,024).
| Model | – | – | – | – | – | – | – (all) |
|---|---|---|---|---|---|---|---|
| Continuous | |||||||
| Gated DeltaNet | |||||||
| KDA | |||||||
| ODN-SO(2) | |||||||
| ODN-SO(2)-IDCR | |||||||
| Reset every 2K | |||||||
| Gated DeltaNet | |||||||
| KDA | |||||||
| ODN-SO(2) | |||||||
| ODN-SO(2)-IDCR |
Manuscript Table 6. Scores (%) on 14 tasks from LongBench (Bai et al., 2023): Narrative QA, QasperQA, MultiField QA, HotpotQA, 2WikiMulti QA, Musique, GovReport, QMSum, MultiNews, TRec, Trivia QA, SamSum, LCC, and RepoBench-P, in column order.
Column groups: Single-Doc QA (NQA, QQA, MFQ); Multi-Doc QA (HQA, 2WM, Mus); Summarization (GvR, QMS, MNs); Few-shot (TRC, TQA, SSM); Code (LCC, RBP).
| Model | NQA | QQA | MFQ | HQA | 2WM | Mus | GvR | QMS | MNs | TRC | TQA | SSM | LCC | RBP | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GDN | |||||||||||||||
| KDA | |||||||||||||||
| ODN-SO(2) | |||||||||||||||
| ODN-SO(2)-IDCR |
Reply to Reviewer rMAD - W1 - Part 1
Response to Reviewer 3
We thank the reviewer for the insightful comments and highlighting the theoretical link from key interference through oscillatory addressing to memory-capacity guarantees. We also appreciate their recognition of the shared, content-dependent clock as a novel mechanism with potential beyond ODN.
We now have thoroughly addressed the reviewers concerns in new version of the manuscript and provide detailed point-by-point replies below:
Reviewer 3: W1.
Thank you for pointing out these two closely related works. We have added both to the related work (Section 2 and Appendix B). Below we discuss the relationship in mathematical detail:
LRU.
The LRU recurrence is
\Lambda=\operatorname{diag}(\lambda_j),\qquad \lambda_j=e^{-\nu_j+i\theta_j}.$$ In real coordinates, each complex eigenvalue is the $2\times2$ block $e^{-\nu_j}R(\theta_j)$, which has the same form as an ODN-SO(2) block $\rho_tR(\omega_r\Delta_t)$. The difference lies in what is rotated and what drives the rotation. **(a) What is rotated?** LRU *rotates a vector hidden state that receives* additive input. ODN rotates the *key side of a matrix associative memory* $S_t\in\mathbb R^{d\times d_v}$ before a delta-rule write. The rotation in ODNs therefore moves the addresses of all stored associations: at read time $t$, association $i$ is stored under $k_iR(H_t-H_i)^{\top}$, where $H_t=\sum_{j\le t}\Delta_j$, is the total internal time up to token $t$. Two writes overlap by $G_{ij}=k_iR(H_i-H_j)k_j^{\top}$ (Eq. 8) where the rotation can separate two correlated or identical keys. The delta-rule then acts on these separated addresses. This produces the ‘memory orthogonalization’ and capacity results of Section 4, which have no analogue in the LRU, since it has no key–value read or erase. **(b) What drives the rotation?** Each LRU channel advances by its own constant angle $\theta_j$ per step. ODN advances all oscillators of a head by one input-dependent internal time $\Delta_t$, so oscillator $r$ has phase $\omega_rH_t$ where $\omega_r$ is its learned frequency. The phase separation between any two writes is then set by one scalar, $H_i-H_j$, across all oscillators. LRU-style transport with independently learned eigenvalues corresponds to ODN with $\Delta_t\equiv1$, which we ablate in Table 7. The constant step clock lowers average accuracy from 48.24 to 47.46 and raises LAMBADA perplexity from 32.46 to 34.27; with the spectrum also frozen, accuracy drops to 46.25. Giving every oscillator its own clock lowers accuracy to 48.14 and raises LAMBADA perplexity to 35.30, despite adding 4.5M parameters.