- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces Hierarchical Adaptive Continuous Attractor Neural Networks (HACANNs) and connects this continuous attractor model to the Gated DeltaNet linear attention model. Specifically, they show that an adaptive neural population can maintain a continuous memory coordinate, and that coupling many of these coordinates produces an associative memory matrix. Under a number of assumptions, this reduces to the Gated DeltaNet update. The authors analyze two implementations of this, one of which uses the full spatial field in the forward pass whereas the other ("projected") implementation evolves Z directly without using the spatial field. They show that these two implementations are close to one another in performance during continued learning. They also study the effects of perturbing different components of the model on prediction. On WikiText-2, they show that HACANN has comparable performance to Gated DeltaNet at small scale. They do further experiments showing that relations can be selectively revised in HACANN and that performance is again similar to Gated DeltaNet. Finally, they show that a repaired feedback pathway in HACANN can learn about as well as an undamaged / intact feedback pathway.
Critical strengths
The paper has a few different strengths.
First, the conceptual link it presents between population dynamics / continuous attractors and a particular kind of linear attention is interesting and the theoretical construction is strong.
Second, the perturbation experiment in Figure 2 is interesting and shows that changing Q while preserving Z has little effect on predictions whereas changing Z while preserving Q has a larger effect.
Third, the comparisons with Gated DeltaNet also clearly show that at small scale these models perform similarly, which bolsters the main argument of the paper.
Critical weaknesses
The paper has a few different weaknesses.
First, it is very confusingly written and difficult to read, largely due to the fact that much of the text appears to be AI generated. The figures also appear to be AI generated and are often not the clearest or strongest visual representation of the results. One specific example of the latter is that in Figure 5, the "Earlier" classes bleed into the "Introduced" section.
Second, the all experiments are performed at < 1M parameter scale and the only comparison is with Gated DeltaNet, which makes the results more limited than they otherwise might be.
Third, the framing of "Gated DeltaNet as the native dynamics" is too strong. The authors require additional assumptions (such as unit keys among other things) for the correspondence to hold, and so should adjust how they frame their method.
Fourth, the authors discuss the relationship with prior work in a few different sections, but there is no dedicated related work section for direct comparison with prior work and analysis of novelty. Some aspects of the paper (e.g. using reverse replay for credit assignment [1]) seem to already appear in uncited prior work. A more thorough and comprehensive discussion of related work is needed.
[1] Huzi Cheng, Joshua W. Brown; Replay as a Basis for Backpropagation Through Time in the Brain. Neural Comput 2025; 37 (3): 403–436. doi: https://doi.org/10.1162/neco_a_01735
Questions for authors
The "Matched continuation" section is a bit confusing. When you say "Learning uses the translation dynamics evaluated along this physical trajectory. Credit spans the whole window. Its calculation retains the influence of a coordinate shift and sets derivatives of the remaining spatial deformation to zero", does this mean that you do not backpropagate through all the spatial variables and instead use the projected model Z?
Did you investigate how performance and spatial/projected agreement change as the assumptions are relaxed (e.g. slower prediction, stronger deformation, and incomplete evidence reset)?
Can you report memory and computational costs compared to Gated DeltaNet at different param counts / widths?