acceptodds
Under review as a conference paper at ICLR 2027

HACANN: Gated DeltaNet as the Native Dynamics of Continuous Attractor Neural Networks

Abstract

Attention mechanisms like Gated DeltaNet drive modern large language models, yet whether such associative computations can natively emerge from biological neural circuits remains an open question. Here, we introduce Hierarchical Adaptive Continuous Attractor Neural Networks (HACANN), developing the hypothesis that continuous attractors serve as canonical building blocks for relational memory. In HACANN, coupled recurrent rate populations encode association components, enabling cues to retrieve predictions and steer error-driven corrections as new evidence arrives. Theoretically, we prove that the projected dynamics of these biologically plausible rate circuits yield a generalized gated Delta rule, exactly recovering the canonical Gated DeltaNet recurrence in a defined timing regime. Empirically, full spatial population simulations with over 4.4 million rate variables sustain both stable prediction and ongoing learning. On WikiText-2, HACANN achieves natural language prediction performance comparable to matched Gated DeltaNet models. Further semantic and visual experiments demonstrate selective revision of stored relations and activity-driven feedback calibration that restores representation learning. Together, our results provide a formal circuit foundation for modern linear attention and map out a pathway toward building large-scale language models entirely from neural population dynamics.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Continuous Attractor Neural Networks,Gated DeltaNet,Associative Memory,Neural Population Dynamics,Linear Attention

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_DNgtOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Hierarchical Adaptive Continuous Attractor Neural Networks (HACANNs) and connects this continuous attractor model to the Gated DeltaNet linear attention model. Specifically, they show that an adaptive neural population can maintain a continuous memory coordinate, and that coupling many of these coordinates produces an associative memory matrix. Under a number of assumptions, this reduces to the Gated DeltaNet update. The authors analyze two implementations of this, one of which uses the full spatial field in the forward pass whereas the other ("projected") implementation evolves Z directly without using the spatial field. They show that these two implementations are close to one another in performance during continued learning. They also study the effects of perturbing different components of the model on prediction. On WikiText-2, they show that HACANN has comparable performance to Gated DeltaNet at small scale. They do further experiments showing that relations can be selectively revised in HACANN and that performance is again similar to Gated DeltaNet. Finally, they show that a repaired feedback pathway in HACANN can learn about as well as an undamaged / intact feedback pathway.

Critical strengths

The paper has a few different strengths.

First, the conceptual link it presents between population dynamics / continuous attractors and a particular kind of linear attention is interesting and the theoretical construction is strong.

Second, the perturbation experiment in Figure 2 is interesting and shows that changing Q while preserving Z has little effect on predictions whereas changing Z while preserving Q has a larger effect.

Third, the comparisons with Gated DeltaNet also clearly show that at small scale these models perform similarly, which bolsters the main argument of the paper.

Critical weaknesses

The paper has a few different weaknesses.

First, it is very confusingly written and difficult to read, largely due to the fact that much of the text appears to be AI generated. The figures also appear to be AI generated and are often not the clearest or strongest visual representation of the results. One specific example of the latter is that in Figure 5, the "Earlier" classes bleed into the "Introduced" section.

Second, the all experiments are performed at < 1M parameter scale and the only comparison is with Gated DeltaNet, which makes the results more limited than they otherwise might be.

Third, the framing of "Gated DeltaNet as the native dynamics" is too strong. The authors require additional assumptions (such as unit keys among other things) for the correspondence to hold, and so should adjust how they frame their method.

Fourth, the authors discuss the relationship with prior work in a few different sections, but there is no dedicated related work section for direct comparison with prior work and analysis of novelty. Some aspects of the paper (e.g. using reverse replay for credit assignment [1]) seem to already appear in uncited prior work. A more thorough and comprehensive discussion of related work is needed.

[1] Huzi Cheng, Joshua W. Brown; Replay as a Basis for Backpropagation Through Time in the Brain. Neural Comput 2025; 37 (3): 403–436. doi: https://doi.org/10.1162/neco_a_01735

Questions for authors

The "Matched continuation" section is a bit confusing. When you say "Learning uses the translation dynamics evaluated along this physical trajectory. Credit spans the whole window. Its calculation retains the influence of a coordinate shift and sets derivatives of the remaining spatial deformation to zero", does this mean that you do not backpropagate through all the spatial variables and instead use the projected model Z?

Did you investigate how performance and spatial/projected agreement change as the assumptions are relaxed (e.g. slower prediction, stronger deformation, and incomplete evidence reset)?

Can you report memory and computational costs compared to Gated DeltaNet at different param counts / widths?

Reviewer_gU8GOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attemps at connecting attention mechanisms with continuous attractor neural networks, specifically by demonstrating that its proposed biologically plausible architecture (Hierarchically Adaptive Continuous Attractor Neural Network - HACANN) is theoretically equivalent to the Gated DeltaNet (GDN) in certain limits.

Continuous Attractor Neural Networks (CANNs) are recurrent neural networks that have stable, translation-invariant equilibrium solutions in the form of a bump of activity U(x)U(x) around a position xx. CANNs therefore store scalars in the form of xx that persist when no input is present. The paper uses the adaptive version of CANNs given by the coupled PDE prescribing the dynamics of $\dot{U}(x,t) $ (activity), V˙(x,t)\dot{V}(x,t) (adaptation), and g˙(x,t)\dot{g}(x,t) (filtered drive), presented in Equation 4 in the main text, and derive equations for the dynamics of its relaxation to an equilibrium position ZZ after a perturbation (shown in Equation 5).

The authors construct a network of coupled CANNs. Then they assign each of the CANNs ZijZ_{ij} to an entry MijM_{ij} in a memory matrix MM, and couple their dynamics through the ODE system shown in Equation 6. By assuming all the other variables constant over an integration invertal Δ\Delta between two inputs, the authors profit from the resulting linearity to integrate Equation 6 analytically to obtain the value of MM after the interval Δ\Delta (called M+M_+). The main claim of the paper comes at this point by showing that the expression for M+M_+ corresponds exactly to the GDN update rule ( M+=αM(I−βkkT)+βvkTM_+ = \alpha M (I- \beta k k^T ) + \beta v k^T ) in Equation 3, specifically in the limit of relative prediction response time θ→0\theta \to 0 and evidence u=0u=0.

The paper presents four experiments (E1 to E4) to illustrate the accuracy achieved by the projection to a scalar coordinate, and demonstrate and compare its performance with respect to GDN.

Critical strengths
  • The paper pursues a theoretical appealing goal, which is connecting the succesful attention mechanism powering modern machine learning systems, to a plausible implementation in biological neural circuits. The authors choose the elegant pathway, which is showing a direct analytical correspondence between the selected attention mechanism's (Gated DeltaNet) update rule and the dynamics of equilibria in a network constructed with multiple modules of a popular model for biological circuits (adaptive Continuous Attractor Neural Networks), and they complement the analytical derivations with four sets of experiments.
Critical weaknesses
  • The main claim of the paper is that Gated DeltaNet is recovered from HACANN equations for Zi,jZ_{i,j} (renamed as Mi,jM_{i,j}, and under the limits and asumptions specified in the text), when Equation 6 is integrated over the time interval between inputs. It is presented as an emergent property due to CANN itself. However, the Gated DeltaNet rule is already prescribed in Equation 6, when the authors design the specific form for the coupling between MM components. In other words, the argument appears to be circular, since GDN is recovered in Equations 7 and 8 because it is imposed in Equation 6 by construction of the coupling between coordinates. The title uses wording ("native") that may suggest association between CANNs and GDN even stronger than claimed in the text.

  • The previous point may reveal a weaker point, which is the independence of the result on ZZ being the equilibrium of a CANN. In fact, it appears that any integrator with similar dynamical characteristics (a line attractor, for instance) could replace the role of CANN because the equivalence to GDN depends on their coupling.

  • The authors do not offer explanation on why they present results with two or three seeds, which makes their statistics really thin.

  • The poor presentation obscures the potential merits of the paper and amplifies the negative impact of the previous weaknesses. The whole text is written in short declarative sentences without connecting reasoning, which makes it extremely difficult to follow. Everyday words are repeatedly used with technical meanings without warning, consistency and rigurosity (e.g. unstructured interchanged use of "relation", "association" and "memory" for MM, the word "energy" in 4.1 onwards). The word "write" and the semicolon are persistently used in unortodox ways, affected the intended meaning. While visually appealing, illustrations in the figures might be misleading, since the HACANN they presented uses 1-D PDEs for U(x,t)U(x,t), V(x,t)V(x,t) and g(x,t)g(x,t). Furthermore, symbols are inconsistent in very plot. The text and visual negative aspects of the manuscript conspire against its potential contribution and should be addressed for clarity and fidelity to the work.

Questions for authors
  • The authors may clarify the main claim with respect the 1st and 2nd Critical Weakness, by addressing the question on what is added by the CANN attractor beyond integration. Do the rest of the results hold if changed by other integrators?

  • How large, if exist, is the gap between the trained HACANN model and the GDN? The authors may report values with the trained models (θ=0.25\theta = 0.25) and explore what happens towards the limit θ→0\theta \to 0.

  • The authors could improve the statistics in the results reported with two or three seeds.

Reviewer_VcGXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HACANN, a continuous attractor neural network framework for associative memory. By coupling neural population coordinates through prediction-error-driven updates, the authors derive a generalized gated Delta rule that recovers Gated DeltaNet under specific conditions. Experiments involving neural population simulations, language modeling, semantic memory updates, and visual learning demonstrate the feasibility of the proposed construction.

Critical strengths
  1. The paper explores an interesting connection between continuous attractor dynamics and modern associative memory mechanisms.
  2. The theoretical construction is supported by experiments across neural population simulations, language modeling, and memory updating tasks.
Critical weaknesses
  1. Limited fundamental novelty. The proposed population coupling explicitly incorporates key–value associations and prediction-error-driven updates closely resembling the target Delta rule. Therefore, the derivation primarily establishes that a specifically constructed CANN can implement Gated DeltaNet, rather than demonstrating that this computation naturally emerges from general attractor dynamics.

  2. Limited independent computational contribution. The experiments mainly demonstrate performance comparable to Gated DeltaNet, without establishing clear advantages in memory capacity, learning efficiency, or computational cost. Consequently, the significance of the substantially more complex neural population implementation remains unclear.

Questions for authors
  1. Which specific properties of continuous attractor dynamics are essential for deriving the gated Delta update, beyond implementing the prescribed population coupling? Could a simpler dynamical integrator yield the same recurrence?

  2. What computational or learning capabilities does HACANN provide beyond the equivalent Gated DeltaNet formulation, and which experimental results directly demonstrate these benefits?

Reviewer_4pEfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an associative memory model that integrates two lines of research: efficient large-context language modeling with gated delta nets, and continuous-time topographically organized neural fields common in computational neuroscience. After introduction and background, it first presents a fairly complex continuous-time neural model that involves spatial convolution, recurrent inhibition, and time lags, among other components. Each entry (i,j) in an associative connection matrix is modeled with an entire population of neurons governed by this model. Next, the paper demonstrates a close relationship between the proposed model and the gated delta net forward pass from prior work. Building on this relationship, the empirical experiments go on to demonstrate comparable performance between the proposed model and gated delta net on the WikiText-2 language modeling benchmark. Several additional experiments are included to further probe the properties of the model, including some tests with CIFAR-10 image data.

Critical strengths
  • The derived equivalence between gated delta nets and a continuous-time, topographically organized neural model may be of significant interest to computational neuroscientists.

  • The paper appears to do a good job on reproducibility: +/- standard deviations are reported for performance metrics, and the submission provides the full code needed to reproduce the results even without GPU (although I did not try running this code).

Critical weaknesses

Aside from possible scientific interest to computational neuroscientists, it was difficult to identify the concrete motivations for this work. Since it performs similarly to gated delta nets with a more complex model, is there any engineering benefit to the proposed approach? Related to that, is the proposed approach more or less computationally demanding than gated delta nets? I did not see empirical results comparing run time, memory usage, or similar metrics.

I found the paper rather difficult to read and understand; both the natural language descriptions and the mathematical equations. Possibly this is due to jargon and notation that would be more familiar to computational neuroscientists. Regardless, I feel much of the writing is unnecessarily vague and imprecise. This seems to be a common deficiency in LLM-assisted scientific writing that I've observed repeatedly while reviewing for this year's ICLR. It also makes it more difficult to check the mathematics for soundness. Some concrete examples (not exhaustive):

  • Lines 107-109: "Translating the pattern preserves their balance. It can retain a value and integrate an input that moves it." What does "their" refer to: the neurons? the interactions? What does "it" refer to, the pattern? Then what does it mean to "move" a pattern? There are many examples like this of pronouns whose referents are not clear, and the reader has to reread surrounding text multiple times to recover the intended meaning.

  • Equation 5: It is stated that angle brackets denote integration, but in this equation there are two comma-separated terms between the angle brackets, as opposed to a single integrand. What does this notation mean? Is it like a functional dot product (integral of a point-wise product between two functions)?

  • Line 112: What is a "preferred coordinate"? It seems it might mean a point in 2d space, but then it should be "coordinates" (plural). Coordinate (singular) sounds like it refers to one dimension, i.e. one index in a multidimensional vector.

  • Section 3.1 defines several quantities (UU, rr, etc) that I believe to be functions (or arrays after discretization), without specifying their domains (or shapes). Are they 2d grids? vectors? functions of the continuous 2d plane, in which case is it a bounded or unbounded domain? something else?

  • It is counterintuitive that a single (i,j) entry in an associative weight matrix is represented by an entire population of neurons; this would benefit from additional motivation, explanation, clarification when introduced in section 3.2.

  • Figure 2 caption mentions a "sham" but this term is not defined anywhere in the main text

  • The appendix is over 50 pages of what is likely AI-generated slop-bloat. For a venue like ICLR with a tight page limit, the goal is to distill advanced ideas into a digestible form, so that important results can be widely disseminated and understood. Submitting gigantic, poorly written LLM theses like this is counterproductive to that goal.

Questions for authors
  • Did I understand correctly that an entire neural population is needed to represent each (i,j) connection in the associative connection matrix, and if so, why? More conventionally this would represent a synaptic connection between two neurons (or neural populations). Can the authors elaborate on this?

  • Can the authors address my confusion above about the angle bracket notation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.