acceptodds
Under review as a conference paper at ICLR 2027

Oscillatory Delta Networks: Phase-Addressed Associative Memory

Abstract

Linear attention and its delta-rule variants, such as Gated DeltaNet, act as implicit associative memories with a fixed-size matrix state. Such a memory reaches full capacity only with orthogonal keys, yet these models have no explicit inductive bias toward them. Our analysis shows that trained DeltaNets generate highly correlated keys and therefore overwrite one another. To overcome such interference, we introduce Oscillatory Delta Networks (ODN), with an inductive bias decorrelate keys by transporting the memory along volume-preserving orbits before each delta-rule write. For rotation based ODNs (ODN-SO(2)) one input-dependent clock per head advances a spectrum of oscillators. In this framework each key-value association is addressed by both its key and its internal write time. We prove that under a suitable spectrum and clock, this oscillation-based content addressing attains the memory's theoretical capacity limit even when all writes share one key, whereas static content addressing retains only the most recent value. Extending memory transports to special linear transformations (ODN-SL(2)) additionally learns each orbit's geometry under the same clock. In language modeling, ODN-SO(2) and ODN-SL(2) outperforms Gated DeltaNet, Selective Rope, and Kimi Delta Attention (KDA). Experiments show this route to be also more effective and parameter-efficient than the channel-wise gating (KDA, Selective Rope). Our approach, inspired by oscillatory phase-dependent short-term memory encoding in biological brain circuits, reduces the overlap between addresses of repeated tokens and enables efficient modeling of correlated long-range interactions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Oscillatory Dyanmics, State Space Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rMADOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper introduces Oscillatory Delta Networks (ODN), which apply volume-preserving state transport (rotations or special-linear flows) before each delta-rule write to address key interference in DeltaNet-family models.

  • Core contributions: (i) proof that oscillation-based addressing reaches the theoretical memory-capacity limit even when all writes share one key; (ii) two architectures — ODN-SO(2) (rotational orbits) and ODN-SL(2) (learned orbit geometry); (iii) empirical gains on synthetic tasks and 340M-parameter language modeling.

Critical strengths
  • Rigorous theoretical characterization. Propositions 1–5 form a complete chain: from problem diagnosis (key correlation → interference) to mechanism design (oscillatory addressing) to capacity guarantees (exact recall of d values under one key). This elevates the contribution from an architectural tweak to a principled understanding of associative memory capacity in linear attention.

  • Novel and broadly applicable mechanism. A shared, content-dependent clock driving a spectrum of state-space oscillators is conceptually fresh and not limited to ODN; it provides a qualitatively different design axis in the crowded delta-rule literature.

Critical weaknesses
  • Lack of discussion of related work like LRU [1] and MDN [2]. Its complex-valued diagonal transitions are conceptually identical to ODN-SO(2) block-wise rotations; without explaining how a shared real-valued clock differs from independently learned complex eigenvalues, the contribution appears incremental.

  • Experiments lack 1B+ parameter results, it is impossible to tell whether theoretical capacity gains strengthen or wash out at scale. Lack of analysis for the perplexity–accuracy trade-off. ODN-SO(2) achieves worse WikiText-2 perplexity (34.02 vs. 31.52) than Gated DeltaNet yet better downstream accuracy; without explaining whether this is a fundamental tension or a training artifact, the mechanism's practical reliability is unclear.

  • Lack of a "mechanism-only" ablation. Without a control adding comparable parameters but no oscillatory transport, gains may stem from extra capacity rather than the mechanism itself.

  • Lack of justification for the hand-designed hyperbolic guard. Equation 6 uses a manually constructed decay rate; without explaining why this must be hand-engineered rather than learned, the core claim of "learning orbit geometry" is internally inconsistent.

  • The citation on lines 48–49 is broken across a line break in a misleading way.

  • The paper introduces ODN-SO(2) and ODN-SL(2) without explaining the group-theoretic suffixes for readers outside the Lie-algebra audience.

[1] Resurrecting recurrent neural networks for long sequences. ICML, 2023.

[2] MDN: Parallelizing Stepwise Momentum for Delta Linear Attention. ICML, 2026.

Questions for authors

see weaknesses

8 comments
AuthorsOpenReview ↗

Reply to Reviewer rMAD - W1 - Part 1

Response to Reviewer 3

We thank the reviewer for the insightful comments and highlighting the theoretical link from key interference through oscillatory addressing to memory-capacity guarantees. We also appreciate their recognition of the shared, content-dependent clock as a novel mechanism with potential beyond ODN.

We now have thoroughly addressed the reviewers concerns in new version of the manuscript and provide detailed point-by-point replies below:

Reviewer 3: W1.

Lack of discussion of related work like LRU [1] and MDN [2]. Its complex-valued diagonal transitions are conceptually identical to ODN-SO(2) block-wise rotations; without explaining how a shared real-valued clock differs from independently learned complex eigenvalues, the contribution appears incremental.

Thank you for pointing out these two closely related works. We have added both to the related work (Section 2 and Appendix B). Below we discuss the relationship in mathematical detail:

LRU.

The LRU recurrence is

\Lambda=\operatorname{diag}(\lambda_j),\qquad \lambda_j=e^{-\nu_j+i\theta_j}.$$ In real coordinates, each complex eigenvalue is the $2\times2$ block $e^{-\nu_j}R(\theta_j)$, which has the same form as an ODN-SO(2) block $\rho_tR(\omega_r\Delta_t)$. The difference lies in what is rotated and what drives the rotation. **(a) What is rotated?** LRU *rotates a vector hidden state that receives* additive input. ODN rotates the *key side of a matrix associative memory* $S_t\in\mathbb R^{d\times d_v}$ before a delta-rule write. The rotation in ODNs therefore moves the addresses of all stored associations: at read time $t$, association $i$ is stored under $k_iR(H_t-H_i)^{\top}$, where $H_t=\sum_{j\le t}\Delta_j$, is the total internal time up to token $t$. Two writes overlap by $G_{ij}=k_iR(H_i-H_j)k_j^{\top}$ (Eq. 8) where the rotation can separate two correlated or identical keys. The delta-rule then acts on these separated addresses. This produces the ‘memory orthogonalization’ and capacity results of Section 4, which have no analogue in the LRU, since it has no key–value read or erase. **(b) What drives the rotation?** Each LRU channel advances by its own constant angle $\theta_j$ per step. ODN advances all oscillators of a head by one input-dependent internal time $\Delta_t$, so oscillator $r$ has phase $\omega_rH_t$ where $\omega_r$ is its learned frequency. The phase separation between any two writes is then set by one scalar, $H_i-H_j$, across all oscillators. LRU-style transport with independently learned eigenvalues corresponds to ODN with $\Delta_t\equiv1$, which we ablate in Table 7. The constant step clock lowers average accuracy from 48.24 to 47.46 and raises LAMBADA perplexity from 32.46 to 34.27; with the spectrum also frozen, accuracy drops to 46.25. Giving every oscillator its own clock lowers accuracy to 48.14 and raises LAMBADA perplexity to 35.30, despite adding 4.5M parameters.
AuthorsOpenReview ↗

Reply to Reviewer rMAD - W1 - Part 2

Response to Reviewer 3 — W1, continued

Lack of discussion of related work like LRU [1] and MDN [2]. Its complex-valued diagonal transitions are conceptually identical to ODN-SO(2) block-wise rotations; without explaining how a shared real-valued clock differs from independently learned complex eigenvalues, the contribution appears incremental.

MDN.

MDN accumulates the delta-rule gradient in a momentum state Mt∈Rd~×dvM_t\in\mathbb{R}^{\tilde d\times d_v} which then updates the memory St∈Rd~×dvS_t\in\mathbb{R}^{\tilde d\times d_v} as: Mt=μtMt−1+ηt∇S∥αtktSt−1−vt∥2,St=αtSt−1−βtMtM_t=\mu_tM_{t-1}+\eta_t\nabla_S \left\| \alpha_t k_tS_{t-1} -v_t \right\|^2, \qquad S_t=\alpha_tS_{t-1}-\beta_tM_t If we regroup momentum and state vectors of a d~=d/2\tilde d = d/ 2-dimensional MDN we can compare it with a dd-dimensional ODN:

ODN-SO(2)MDN
statezt,r=[zt,r,1 zt,r,2]∈R2×dv,z_{t,r}=\begin{bmatrix}z_{t,r,1}\\\ z_{t,r,2}\end{bmatrix}\in\mathbb R^{2\times d_v},zt,r=[st,r mt,r]∈R2×dvz_{t,r}=\begin{bmatrix}s_{t,r}\\\ m_{t,r}\end{bmatrix}\in\mathbb R^{2\times d_v}
transportzˉt,r=ρt[cos⁡(ωr,tΔt)−sin⁡(ωr,tΔt) sin⁡(ωr,tΔt)cos⁡(ωr,tΔt)]zt−1,r,\bar z_{t,r}=\rho_t\begin{bmatrix}\cos\left(\omega_{r,t}\Delta_t\right)&-\sin\left(\omega_{r,t}\Delta_t\right)\\\ \sin\left(\omega_{r,t}\Delta_t\right)&\hspace{8pt}\cos\left(\omega_{r,t}\Delta_t\right)\end{bmatrix}z_{t-1,r},zˉt,j=[αt−βtμt 0μt]zt−1,j\bar z_{t,j}=\begin{bmatrix}\alpha_t&-\beta_t\mu_t\\\ 0&\mu_t\end{bmatrix}z_{t-1,j}
deltazt,r=zˉt,r+βt[kt,r,1 kt,r,2](vt−∑s[kt,s,1,kt,s,2]zˉt,s)z_{t,r}=\bar z_{t,r}+\beta_t\begin{bmatrix}k_{t,r,1}\\\ k_{t,r,2}\end{bmatrix}\Bigl(v_t-\sum_s[k_{t,s,1},k_{t,s,2}]\bar z_{t,s}\Bigr)zt,r=zˉt,r+ηt[βtkt,r −kt,r](vt−∑s[kt,s,βtkt,s]zˉt,s)z_{t,r}=\bar z_{t,r}+\eta_t\begin{bmatrix}\beta_t k_{t,r}\\\ -k_{t,r}\end{bmatrix}\Bigl(v_t-\sum_s[k_{t,s},\beta_t k_{t,s}]\bar z_{t,s}\Bigr)
readoutot=∑r[qt,r,1,qt,r,2]zt,ro_t=\sum_r[q_{t,r,1},q_{t,r,2}]z_{t,r}ot=∑r[qt,r,0]zt,r,o_t=\sum_r[q_{t,r},0]z_{t,r},

The two models differ in the following ways:

(i)

For each output coordinate MDN doubles the recurrent state by adding the momentum variable. The d~\tilde{d} dimensional keys and queries in MDNs are effectively expanded to the latent space introducing a predefined coupling between those two dimensions and the keys and queries. MDN then also only reads half of the memory state via read direction qt,j[1,0]q_{t,j}[1,0]. In contrast, ODN learns dd-dimensional keys and queries that can flexibly address and read every state coordinate.

(ii)

In MDN the write direction (βt,1)kt,i(\beta_t,1)k_{t,i}, and the residual direction (1,−βt)kt,i(1,-\beta_t)k_{t,i} are different as the delta rule acts through the momentum term. In contrast in ODN the delta rule directly acts on the memory state providing a more consistent memory management mechanism.

(iii)

MDN's transition is upper triangular with real eigenvalues αt\alpha_t and μt\mu_t, so it does not rotate. Complex eigenvalues can only arise indirectly through the delta-rule erasing memory along the current key. In contrast, ODN's transition is a rotation with eigenvalues ρte±iωrΔt\rho_te^{\pm i\omega_r\Delta_t}, which turns every stored address regardless of the current key. This separates memory transform from memory update.

(iv)

ODNs learns each 2x2 block's frequency ωr\omega_r independently, while MDN provides the same transformation at each block. This provides ODNs with a more fine grained control of the memory transformation, including adjusting the spectrum to facilitate or suppress memory interferences as outlined in Propositions 3 - 5.

In summary, MDN and ODN are related but organize their recurrent states differently. In the matched-state comparison above, both models store the same number of recurrent-state elements, but MDN's keys and queries have only half as many coordinates as ODN's: MDN reserves the other half of its state for momentum, which its queries cannot read directly. ODN's full-width keys and queries, together with its learned rotations, provide a flexible mechanism for addressing and separating stored memories. The approximately orthogonal memory addresses observed in the Lag-Indexed Overwrite Recall Task are consistent with this mechanism (see also our reply to Reviewer 1).

AuthorsOpenReview ↗

Reply to Reviewer rMAD - W2a

Reviewer 3: W2a.

Experiments lack 1B+ parameter results, it is impossible to tell whether theoretical capacity gains strengthen or wash out at scale.

The revised manuscript now includes 1B-parameter results in Sections 5.4–5.5. On short-context zero-shot accuracy, ODN-SO(2) is comparable to Gated DeltaNet and KDA (Table 4: 59.19, 59.19, and 59.64, respectively). The clearest gains appear on long-context benchmarks, where memory load accumulates.

The revised manuscript now includes 1B-parameter results in Sections 5.4–5.5. On short-context zero-shot accuracy, ODN-SO(2) is comparable to Gated DeltaNet and KDA (Table 4: 59.19, 59.19, and 59.64, respectively). Intriguingly, ODN achieves significant gains on long-context benchmarks, where memory load accumulates.

The new manuscript Tables 5–6 are reproduced below and show PG-19 prediction under continuous processing and 2K-token state resets; Table 6 reports LongBench scores.

Under continuous processing, ODN-SO(2) maintains a 64–128K PG-19 perplexity of 14.25, compared with 37.55 for Gated DeltaNet and 75.11 for KDA. Resetting the state every 2K tokens improves both baselines but increases ODN-SO(2)'s perplexity to 15.44. These results support greater stability as memory load accumulates and useful retention beyond 2K tokens.

On LongBench, ODN-SO(2) leads on 12 of 14 tasks and achieves the highest average score (14.38, versus 12.44 for Gated DeltaNet and 8.99 for KDA).

Thus, the gains remain apparent at 1B parameters and are strongest when memories accumulate over long contexts, consistent with ODN's design principle of using learned memory transport to reorganize stored information and manage memory interference over time. We now state this distinction explicitly in Section 5.4.

Manuscript Table 5. PG-19 token perplexity (↓\downarrow) on the same 25 books: continuous processing versus state resets every 2K tokens. Columns show 2K position intervals and the 64–128K aggregate (K = 1,024).

Model22–4K4\mathrm{K}66–8K8\mathrm{K}1414–16K16\mathrm{K}6262–64K64\mathrm{K}9494–96K96\mathrm{K}126126–128K128\mathrm{K}6464–128K128\mathrm{K} (all)
Continuous
Gated DeltaNet13.7913.7919.84‾\underline{19.84}28.78‾\underline{28.78}36.46‾\underline{36.46}39.14‾\underline{39.14}37.03‾\underline{37.03}37.55‾\underline{37.55}
KDA13.36‾\underline{13.36}23.5623.5669.6169.6172.5072.5078.5278.5274.3474.3475.1175.11
ODN-SO(2)14.0814.0818.78\mathbf{18.78}14.29\mathbf{14.29}14.24\mathbf{14.24}14.49\mathbf{14.49}13.76\mathbf{13.76}14.25\mathbf{14.25}
ODN-SO(2)-IDCR12.56\mathbf{12.56}49.0849.08412.59412.591278.711278.71415.91415.9195.6895.68118.02118.02
Reset every 2K
Gated DeltaNet15.0815.0815.3515.3515.3415.3415.3915.3915.6515.6514.8714.8715.4315.43
KDA14.41‾\underline{14.41}14.93‾\underline{14.93}14.44‾\underline{14.44}14.72‾\underline{14.72}15.03‾\underline{15.03}14.24‾\underline{14.24}14.77‾\underline{14.77}
ODN-SO(2)15.1815.1815.4015.4015.3715.3715.3115.3115.6715.6714.8914.8915.4415.44
ODN-SO(2)-IDCR13.62\mathbf{13.62}14.27\mathbf{14.27}13.67\mathbf{13.67}14.08\mathbf{14.08}14.23\mathbf{14.23}13.72\mathbf{13.72}14.04\mathbf{14.04}

Manuscript Table 6. Scores (%) on 14 tasks from LongBench (Bai et al., 2023): Narrative QA, QasperQA, MultiField QA, HotpotQA, 2WikiMulti QA, Musique, GovReport, QMSum, MultiNews, TRec, Trivia QA, SamSum, LCC, and RepoBench-P, in column order.

Column groups: Single-Doc QA (NQA, QQA, MFQ); Multi-Doc QA (HQA, 2WM, Mus); Summarization (GvR, QMS, MNs); Few-shot (TRC, TQA, SSM); Code (LCC, RBP).

ModelNQAQQAMFQHQA2WMMusGvRQMSMNsTRCTQASSMLCCRBPAvg.
GDN0.890.895.49\mathbf{5.49}12.20‾\underline{12.20}4.02‾\underline{4.02}7.36‾\underline{7.36}2.162.168.18‾\underline{8.18}14.29‾\underline{14.29}11.3511.3545.00\mathbf{45.00}9.719.7116.75‾\underline{16.75}17.7317.7318.97‾\underline{18.97}12.44‾\underline{12.44}
KDA1.49‾\underline{1.49}4.884.889.299.292.652.656.636.631.111.116.496.4911.0811.0811.65‾\underline{11.65}17.0017.0010.63‾\underline{10.63}7.627.6218.33‾\underline{18.33}17.0017.008.998.99
ODN-SO(2)2.35\mathbf{2.35}5.02‾\underline{5.02}13.79\mathbf{13.79}5.28\mathbf{5.28}8.34\mathbf{8.34}3.63\mathbf{3.63}10.45\mathbf{10.45}18.21\mathbf{18.21}12.12\mathbf{12.12}22.50‾\underline{22.50}25.61\mathbf{25.61}28.35\mathbf{28.35}22.77\mathbf{22.77}22.82\mathbf{22.82}14.38\mathbf{14.38}
ODN-SO(2)-IDCR0.690.692.932.939.569.563.923.924.954.953.45‾\underline{3.45}3.993.999.809.809.049.0414.0014.0010.4710.4710.2710.2716.5116.5113.7513.758.098.09
Reviewer_CQJYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents an approach to enhance the memory mechanism of delta rule models, tacking a limitation of how they update their model’s state. The limitation is that the way Delta-rule models implement memory retention inherently has a geometric flaw: they are optimal only under orthogonality conditions, yet during training those conditions are almost never met.

The authors propose a new architecture (ODN) that helps the memory mechanism in those models. Specifically, they propose two version ODN-SO(2) and ODN-SL(2), and show that they outperform classical delta rule architectures on selected tasks.

Critical strengths

The observation that trained keys are correlated and that this limits memory capabilities is interesting.

Grounding a proposed architecture is an observed failure is well motivated.

The experiments are comprehensive.

Critical weaknesses

The main weakness is the presentation: I did not understand the rationale behind the solution proposed.

For example, could the authors explain more clearly why they propose to couple pairs of latent dimensions and make them rotate? why not triplets of latent dimensions? Also, why coupling consecutive latent dimensions?

Additionally, could you explain more clearly why preserving volume is interesting?

The writing felt confusing, often too technical: Figure 1 exemplifies the same issue, in a figure: it is also hard to understand because it uses notations that have not been introduced on page 2.

Some typos, eg inductive bias decorrelate keys -> inductive bias to decorrelate keys ODN-SO(2) and ODN-SL(2) outperforms -> outperform memory retention(Schlag: missing a space (Kohonen, 1972; Schlag et al., 2021) (Kohonen, 1972; Schlag et al., 2021) (appears twice) d = 128. over 192 layer–heads -> remove period. a input-dependent -> an input-dependent

Questions for authors

See my answer in critical weaknesses

1 comment
AuthorsOpenReview ↗

Reply to Reviewers CQJY Comments

Reply to Reviewer CQJY

We thank the reviewer for the helpful comments. We have revised the manuscript to make it more accessible and clarify our rationale.

Reviewer 2: Could the authors explain more clearly why they propose to couple pairs of latent dimensions and make them rotate? Why not triplets ? Why couple consecutive latent dimensions?

We agree that transport could act on larger subspaces or even the entire latent space. We chose the minimal 2×22\times2 blocks required for rotational transforms for three reasons:

(i) For dd latent dimensions, 2×22\times2 diagonal blocks reduce the flow's computational demands from O(d2)O(d^2) to O(d)O(d), making the approach scalable to LLMs. This is in line with previous literature that reduces rotational operators to block-diagonal forms [1, 2].

(ii) Coupling rotational modes or oscillators makes it more difficult to ensure the overall stability of the system, as eigenvalues are more challenging to calculate for larger matrices. Thus, controlling gradients so that they do not explode (or vanish) becomes more computationally demanding. Keeping each oscillation or flow isolated helps keep the dynamics stable. This is also in line with previous studies that have integrated oscillatory dynamics into neural state-space models [2], [3] and recurrent networks [4].

(iii) Mathematically, any rotation matrix has a normal form identical in structure to our 2×22\times 2 block-diagonal matrices. That is, we can find a linear basis transformation in which a generic rotation is split into its “eigenmodes”. This argument, however, does not generalize to SL(N)SL(N), where NN is the matrix dimension: its generators can have Jordan blocks of any size as shears are added to the rotational content (KAN decomposition).

We note that choosing consecutive coordinates to form the blocks in our representation is arbitrary and plays no critical role in our model. Keys, queries, and values undergo learned linear transformations that can reorder latent dimensions arbitrarily. In fact, those maps can contribute to the basis transformation that demixes the rotational eigenmodes, as explained in (iii).

In summary, our choice is mainly motivated by computational efficiency and stability. We agree with the reviewer that exploring larger subspaces to enable the system to learn more complex shear flows could be an interesting direction. Allowing rotational modes to mix by letting the network select from a set of rotation matrices (and/or basis transformations) could be another interesting direction to study, though different attention heads can already account for this to some extent. Given the time and compute limitations, we defer these interesting aspects to future work.

We now clearly motivate our choice of ‘structured flows’ in more detail in the manuscript when introducing our model.

Reviewer 2: Could you explain more clearly why preserving volume is interesting?

Our transport flows have unit determinant and thus preserve state-space volumes. This property translates to ‘information preservation’: consider a cloud of latent network states representing different input histories. The flow may deform this cloud, but its volume does not shrink, and thus these different histories remain largely recoverable. This information-preserving role also motivates Hamiltonian, symplectic, and unitary neural networks [4-6]. In ODNs, it lets memories move through time without imposing “memory decay” through the flow; the additional retention (ρ\rho) and write (β\beta) gates control forgetting and writing of the stored memory. In this way, ODNs keep the mechanisms for memory deformation, forgetting, and writing separate. For ODN-SL(2), an additional guard limits hyperbolic stretching to ensure numerical stability (see also our response to Reviewer 3).

We now discuss these points in the revised manuscript. Given our clarifications and new results we hope we could address the reviewers concerns, and would greatly appreciate it if you could consider revisiting your rating.

References

[1] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. RoFormer: Enhanced Transformer with Rotary Position Embedding Neurocomputing 568, 127063 (2024).

[2] Gupta, A., Gu, A., and Berant, J. Diagonal State Spaces Are as Effective as Structured State Spaces. NeurIPS 35, 2022.

[3] Rusch, T. K., and Rus, D. Oscillatory State-Space Models (LinOSS). ICLR, 2025.

[4] Rusch, T. K., and Mishra, S. UnICORNN: A Recurrent Model for Learning Very Long Time Dependencies ICML, 2021.

[4] Greydanus, S., Dzamba, M., and Yosinski, J. Hamiltonian Neural Networks NeurIPS, 2019.

[5] Zhong, Y. D., Dey, B., and Chakraborty, A. Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control ICLR, 2020.

[6] Arjovsky, M., Shah, A., and Bengio, Y. Unitary Evolution Recurrent Neural Networks ICML, 2016.

Reviewer_TYHZOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes Oscillatory Delta Networks (ODN), which transform recurrent memory before each delta-rule write. The transformation uses internal time to separate similar keys and reduce interference. The authors prove exact recall under idealized conditions with matched queries and evaluate ODN on associative recall, state tracking, and language modeling.

Critical strengths

S1. ODN reduces interference without increasing the recurrent state size or asymptotic computational cost. Measurements in trained models support the proposed mechanism: transport reduces address overlap between repeated occurrences of the same token.

S2. ODN clearly outperforms the tested baselines on lag-indexed overwrite recall. Stopping the clock sharply reduces accuracy, providing direct evidence that transport matters for this task.

Critical weaknesses

W1. The exact-recall result explains how a matched query can retrieve a stored value. The lag-indexed task also shows that the trained model can recover the requested value. It remains unclear how closely the learned queries align with the target phase addresses, which limits how directly the theory explains this result.

W2. The language-model gains are modest and vary by task. ODN-SL(2) improves mean accuracy over KDA by 0.59 percentage points and achieves the best LAMBADA perplexity, while Selective RoPE leads on WikiText-2. A single pretraining seed leaves the reproducibility of the average gain unclear.

Questions for authors

Q1. For the same sequence of writes, how does changing the requested lag affect the query’s alignment with the target and other stored addresses?

Q2. Equation (1) computes the residual from St−1S_{t-1}, while Eq. (2) with Tt=IT_t=I uses ρtSt−1\rho_t S_{t-1}. Which update did the Gated DeltaNet baseline use? Does stopping the ODN clock recover that update exactly?

Q3. Do you have results across pretraining seeds? For the reported checkpoints, paired uncertainty estimates would also help quantify how much the measured gains depend on the evaluation examples.

7 comments
AuthorsOpenReview ↗

Response to Reviewer TYHZ - Q1 - Part 1

Response to Reviewer 1

We thank the reviewer for the valuable feedback and insightful comments. Reviewer 1 highlights that ODNs reduce memory interference and that ODN outperforms on lag-indexed overwrite recall elucidating the role of transport. Below we respond to the reviewer's questions.

W1

W1. ...It remains unclear how closely the learned queries align with the target phase addresses, which limits how directly the theory explains this result.

Q1. For the same sequence of writes, how does changing the requested lag affect the query's alignment with the target and other stored addresses?

We thank the reviewer for this question. We now have investigated in detail the underlying mechanisms of memory retrieval in ODNs including in the functional Lag-Indexed Overwrite Recall task (manuscript Figure 9, Appendix F.3), as well as for a pretrained 1B model on RULER (Figure 10).

Our analysis shows that highly correlated keys are nearly orthogonalized through the oscillatory rotations in ODNs allowing different values to be stored at distinct addresses in the latent state memory without destroying or interfering with others. Queries are then learned to directly address these distinct memories using the requested lag or key.

Overall, these findings suggest that our models have an inductive bias toward orthogonalizing memory addresses in gated delta attention and that learned queries align with the target phase addresses, consistent with the initial motivation and theoretical results of this work.

Memory formation and retrieval in the Lag-Indexed Overwrite Recall task

For the Lag-Indexed Overwrite Recall task, we find that ODN learns a rotation-driven memory shift register that works as follows.

Let tt and ss be positive integer time indices with s≤ts\leq t. Let dd be the key and state dimension, and dvd_v the value dimension. The memory St∈Rd×dvS_t\in\mathbb R^{d\times d_v} is a matrix; the key ks∈R1×dk_s\in\mathbb R^{1\times d}, query qt∈R1×dq_t\in\mathbb R^{1\times d}, and value vs∈R1×dvv_s\in\mathbb R^{1\times d_v} are row vectors. Superscript ⊤\top denotes transpose, IdI_d is the d×dd\times d identity matrix, βs\beta_s is the scalar write strength, ρs\rho_s is the scalar retention factor, and Ts∈Rd×dT_s\in\mathbb R^{d\times d} is the block-diagonal rotation at time ss. Define the state-update matrix As=(Id−βsks⊤ks)ρsTsA_s=(I_d-\beta_s k_s^\top k_s)\rho_sT_s and the transport matrix Et←s=AtAt−1⋯As+1E_{t\leftarrow s}=A_tA_{t-1}\cdots A_{s+1}, with Et←t=IdE_{t\leftarrow t}=I_d. With zero initial memory S0=0S_0=0, the recurrence unrolls as

S_t &= \sum_{s=1}^{t}E_{t\leftarrow s}\,\beta_s k_s^\top v_s, \\ E_{t\leftarrow s} &= A_tA_{t-1}\cdots A_{s+1}, \\ A_s &= (I_d-\beta_s k_s^\top k_s)\rho_sT_s. \end{aligned}$$ Here $E_{t\leftarrow s}$ transports the write made at time $s$ through the subsequent state updates up to time $t$. The readout $o_t\in\mathbb R^{1\times d_v}$ and scalar read weight $w_{t,s}$ are $$o_t=q_tS_t=\sum_{s=1}^{t}w_{t,s}v_s, \qquad w_{t,s}=q_tE_{t\leftarrow s}\,\beta_s k_s^\top.$$ The weight $w_{t,s}$ measures how strongly the query reads write $s$ at time $t$. For any target memory length $d'\leq d$, choose a unit key $k\in\mathbb R^{1\times d}$ fixed across time and oscillator frequencies and key energies such that, for a constant time increment $\Delta_s=\tau$, the block rotation $T_s=R(\tau)$ satisfies $$kR(j\tau)k^\top=0\quad (1\leq j<d'), \qquad R(d'\tau)=-I_d.$$ Here $\Delta_s$ is the time increment at step $s$, $\tau$ is its constant value, $R(x)$ denotes the rotation through increment $x$, and $j$ is an integer phase-step index. The first condition makes the first $d'-1$ rotated addresses orthogonal to the key; the second specifies the rotation after $d'$ steps. This can always be achieved by a construction as in Proposition 3. For SO(2) and $\rho_s=\beta_s=1$, the update matrix is constant: $A=(I_d-k^\top k)R(\tau)$. The above conditions imply $A^jk^\top=R(j\tau)k^\top$ for $0\leq j<d'$ and $A^{d'}k^\top=0$, where the superscript on $A$ denotes repeated matrix multiplication. The memory therefore becomes $$S_t= \sum_{s=\max(1,t-d'+1)}^{t} R\bigl((t-s)\tau\bigr)k^\top v_s.$$ The most recent $d'$ writes occupy orthogonal phase addresses, and a write is erased when it reaches age $d'$. Let $D$ denote the age of the desired value at read time, with $0\leq D<\min(t,d')$. The phase-matched row query $$q_t(D)=kR(-D\tau)$$ selects that value exactly: $$\begin{aligned} o_t(D) &= q_t(D)S_t \\ &= \sum_{s=\max(1,t-d'+1)}^{t} kR\bigl((t-s-D)\tau\bigr)k^\top v_s \\ &= v_{t-D}. \end{aligned}$$ Here $o_t(D)$ is the output for requested age $D$, and $\mathbf 1\{s=t-D\}$ equals one when $s=t-D$ and zero otherwise. Rotation advances stored values through distinct phase addresses, while the delta update removes the oldest value and makes room for a new one. This gives a $d'$-slot shift-register mechanism even when all writes use the same content key.
AuthorsOpenReview ↗

Response to Reviewer TYHZ - Q1 - Part 2

Response to Reviewer 1 — W1, continued

Answer, continued.

Figure 9 in the revised manuscript shows that ODN indeed learned the shift-register mechanism. We find that the keys for the values to be stored are nearly identical (Figure 9a), that the rotation advances by a fixed τ\tau (Figure 9b), which orthogonalizes the keys (Figure 9c). Indeed, the learned approximate shift register has period d′=30>Dd'=30>D (not shown). Queries of the form qt(D)=kR(−Dτ)q_t(D)=kR(-D\tau) are then learned to recall the correct value at delay DD (Figure 9d).

This finding is further confirmed in Figure 9e, where the query read weight wt,sw_{t,s} on the delay token strongly peaks at the desired delay t−Dt-D value. This becomes even more pronounced with the output share, defined as

ηt,s=⟨wt,svs, ot⟩∥ot∥2.\eta_{t,s}=\frac{\langle w_{t,s}v_s,\,o_t\rangle}{\lVert o_t\rVert^2}.

Here ηt,s\eta_{t,s} is the share of output oto_t attributable to write ss; ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle is the Euclidean inner product on row values, and ∥⋅∥\lVert\cdot\rVert is the corresponding Euclidean norm. This is shown in Figure 9f.

In summary, these data show that ODN has learned a d′d' shift register making use of the rotational mechanisms that motivated our work. Rotational ablation prevents ODA from learning this task (Table 3) similarly to Gated DeltaNet and KDA. Thus, ODNs have a clear inductive bias to learn organized memory registers.

AuthorsOpenReview ↗

Response to Reviewer TYHZ - Q2

Response to Reviewer 1 — Q2

Q2. Equation (1) computes the residual from St−1S_{t-1}, while Eq. (2) with Tt=IT_t=I uses ρtSt−1\rho_tS_{t-1}. Which update did the Gated DeltaNet baseline use? Does stopping the ODN clock recover that update exactly?

Thank you for catching this. The Gated DeltaNet baseline applies retention before computing the residual:

St=ρtSt−1+βtkt⊤(vt−ktρtSt−1).S_t=\rho_tS_{t-1}+\beta_tk_t^{\top}\bigl(v_t-k_t\rho_tS_{t-1}\bigr).

Here t−1t-1 is the preceding time step, Tt=IdT_t=I_d means no state rotation, and the other symbols retain their definitions above. The equation adds a delta correction to the retained prior memory, using the difference between the new value and the value read from that retained memory.

Eq. (1) in the initial submission accidentally omitted ρt\rho_t inside the residual; we have corrected it in the revised manuscript. With the corrected Eq. (1), stopping the clock in ODNs (Δt=0\Delta_t=0) recovers this update exactly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.