Oscillatory Delta Networks: Phase-Addressed Associative Memory

Linear attention and its delta-rule variants, such as Gated DeltaNet, act as implicit associative memories with a fixed-size matrix state. Such a memory reaches full capacity only with orthogonal keys, yet these models have no explicit inductive bias toward them. Our analysis shows that trained DeltaNets generate highly correlated keys and therefore overwrite one another. To overcome such interference, we introduce Oscillatory Delta Networks (ODN), with an inductive bias decorrelate keys by transporting the memory along volume-preserving orbits before each delta-rule write. For rotation based ODNs (ODN-SO(2)) one input-dependent clock per head advances a spectrum of oscillators. In this framework each key-value association is addressed by both its key and its internal write time. We prove that under a suitable spectrum and clock, this oscillation-based content addressing attains the memory's theoretical capacity limit even when all writes share one key, whereas static content addressing retains only the most recent value. Extending memory transports to special linear transformations (ODN-SL(2)) additionally learns each orbit's geometry under the same clock. In language modeling, ODN-SO(2) and ODN-SL(2) outperforms Gated DeltaNet, Selective Rope, and Kimi Delta Attention (KDA). Experiments show this route to be also more effective and parameter-efficient than the channel-wise gating (KDA, Selective Rope). Our approach, inspired by oscillatory phase-dependent short-term memory encoding in biological brain circuits, reduces the overlap between addresses of repeated tokens and enables efficient modeling of correlated long-range interactions.