- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper “Dynamical memory retrieval in attractor networks as a denoising diffusion process” analyzes a memory retrieval process based on the temperature / noise annealing process that is typical of diffusion models (DMs). Despite several studies in the literature have already investigated the formal analogies between modern hopfield networks and DMs (e.g. Hoover, 2023; Raya & Ambrogioni, 2024), no-one had clearly studied the noise reduction schedule that is typical of memorizing DMs in the context of associative memories. This work first establishes a formal equivalence between a class of continuous attractor neural networks (CANNs) and variance exploding DMs, then applies the annealing routine in real experiments.
Critical strengths
The analogy between CANNs and DMs is indeed interesting. This type of associative memory is very much adopted by theoretical neuroscientists to study spatial memory recall, and this study might in practice foster biological implementations of diffusion models. The paper is simply written and instructive, the story line is clear.
Critical weaknesses
My impression as a Reviewer is that, once the analogy between CANNs and DMs is established, the rest is equivalent to memorization in DMs, which is a broadly studied phenomenon that does not necessarily need a new demonstration, as the one in Section 4. I think that the paper would be effectively novel and fundamental if a more biologically plausible model was employed to parametrize the CANN. This would have answered a question sounding similar to “Does the Hippocampus learn a diffusion model?”.
Questions for authors
I will now present my comments and questions with the goal of improving the quality of the presented work. I ask the Authors to address my points extensively.
Q1a: The gaussian mixture expression of the mean bump position depends on the structure of I_ext but also on the fact that J is chosen to be symmetric Gaussian. Then authors decide to choose the width of the interaction a to vary in time. Why so? Isn't it more biologically accurate to assume the interactions among place cells to be quenched and dependent on the data (i.e. fixed) and then imagine the memory retrieval to be driven by a time dependent external stimulus that regulates the temperature of the neural dynamics ?
Q1b: what would be the interpretation, in terms of neural activity, of a decreasing bump width? Would this be observed from physiological data, where neural dynamics is usually captured at the equilibrium?
Q1c: The choice of is very specific. What is its physical / biological interpretation ?
Q2: Works such as Battista & Monasson (2020) or Rosay & Monasson (2013) are based on graphical models of neurons, as in more classic Hopfield-like associative memories. On the other hand, dense associative memories (Krotov & Hopfield, 2017) do not have a direct interactive counterpart. Can you imagine a mapping between a neural network and the CANN studied in this work? Since we still have not discovered any diffusion model in the brain, if I wanted to actually measure whether a biological neural network is trained in the same way as generative diffusion, how would I do it ? This question is already treated in the conclusions but I would like the authors to expand on this.
Q3: Deep networks that parametrize diffusion models are usually trained by denoising score matching. After training, when the model generates a configuration, even one that coincides with a training point if the model is in the memorization regime as it is in the paper, the training points are not seen again. Conversely, each memory retrieval “annealing scheme” as described in paper relies on presenting all training points to the model, which is costful, and possibly not very biologically plausible. I invite the Authors to discuss this computational issue.
Q4: Since CANNs were involved I asked many questions about biological plausibility, but I realize that Authors have chosen CANNs as a memory model to demonstrate the true efficacy of this annealing retrieval routine. I hence invite them to discuss the novelty of this work with respect to memorization in DMs.
Q5: Authors should definitely cite Lucibello & Mézard (2024). "Exponential capacity of dense associative memories". This seminal paper shows that dense hopfield with spherical constraints have infinite capacity, and this is shown by Raya & Ambrogioni (2024) to be equivalent to a diffusion model in the variance exploding set-up which is also the dynamic setting of the analyzed CANN. More generally I feel like the paper lacks citations both in the matter of modern hopfield networks and the very few biological implementations (see Krotov & Hopfield, 2023; Kafraj et al., 2025). The paper also lacks references regarding memorization in diffusion models, e.g. Biroli & Mézard (2024), Raya & Ambrogioni (2024).
5 comments
Rebuttal
I thank the Authors for addressing my questions. However, I believe that answers might go a bit deeper.
-
The answer to Q1 is useful. It is evident that there might be several biological mechanisms giving origin to this form of memory retrieval, and I thank the Authors for providing examples. Nevertheless, it is still not clear to me why internal network modulation (either in terms of STP and neural activity) in MEC should be the favourite candidate, instead of a modulation of . If the aim of the work is also motivating future neuro-physiological experiments, the paper should me more clear and explicit about this. The Authors still do not expand around the biological interpretation of as the mixture of Gaussians in Eq. (4).
-
Q2: Where are the neuron-neuron interactions in both J and ? As in a dense associative memory (Krotov & Hopfield, 2017), pairwise (as in Battista & Monasson (2020) and Rosay & Monasson (2013) ), and even multi-body interactions across neurons (Garnder, "Multiconnected neural network models" (1987)) are not explicit (there is not an effective graph underneath), and this makes the model less biologically plausible. Please expand.
-
Q3: I understand your answer. Still, as for Q2, the paper is missing the "biological" plausibility: you state "the learning process is carried out through the formation of intercortical synaptic connections". Where are the (learned) synaptic connections in the model?
We thank the reviewer for these further questions.
First, we would like to clarify that both the Gaussian recurrent connectivity and the Gaussian external inputs are widely adopted in established CANN models (e.g., Wu et al., 2008; Fung et al., 2010), and they could both be derived from the neuronal level. Specifically, could be explained by Hebbian learning (e.g., Shen and McNaughton, 1996, Page 686). can also be derived in the same way through Hebbian learning. For example, let denote the spatial coordinate of MEC neurons and the indexing coordinate of HPC neurons. For memory , let denote the associated HPC firing pattern, and we assume the spatial projection of this memory onto MEC as
-\frac{|\mathbf{s}-\mathbf{x}_n|^2}{2b^2} \right).$$ (We approximate $\phi_n$ as a Gaussian because it provides a simple, mathematically convenient description of localized neural activity while capturing the general statistical features of spatially distributed representations.) The Hebbian-connectivity can then be $$W(\mathbf{s},\mathbf{s}')=\eta\sum_{n=1}^{N}\phi_n(\mathbf{s})r_{H,n}(\mathbf{s}').$$ Given the current HPC firing rates $r_H(\mathbf{s}',t)$, the external input received by MEC is $$I_{ext}= \rho_H\int W(\mathbf{s},\mathbf{s}')r_H(\mathbf{s}',t) ,d\mathbf{s}' =\sum_{n=1}^{N} \alpha_n(t) \exp\left( -\frac{|\mathbf{s}-\mathbf{x}_n|^2}{2b^2} \right),$$ where $$\alpha_n(t)= \eta\rho_H\int r_{H,n}(\mathbf{s}')r_H(\mathbf{s}',t) ,d\mathbf{s}'.$$ This recovers the Gaussian mixture input adopted in our model, and this is the "effective graph underneath". Besides, we've always agreed the possibility of the modulation of $I_{ext}$. We wouldn't like to say "memory retrieval is implemented merely through internal network modulations". What our work suggests is **the mechanism, the effectiveness, and the possibility** of such internal modulation in retrieval tasks. (Perhaps internal+external modulation works better, but that's not what we do in this paper.) In this work, we control the external modulation as almost-fixed (except for the slight alteration in $\alpha$) for a better analysis of the dynamical properties of the internal modulation itself, but we're not claiming "internal network modulation in MEC is the favourite candidate". Moreover, such internal modulation doesn't cost significantly more than external modulations.
We thank the reviewer for the comments and questions. First of all, we would like to clarify our modelling background. Our CANN is modelling the activities of grid cells in MEC, where recurrent connections are considered to be homogeneous. (There are substantial experimental evidence suggesting that grid cell populations in the MEC form an internally maintained continuous manifold [1], where the recurrent connection J is required to be homogeneous to form this continuous attractor.) Information about the memories is transmitted from HPC to MEC, and this is the biological interpretation of the external inputs Iext. The bump in the MEC travels along the attractor landscape under the attraction of these inputs and carries out the retrieval process.
It is better to interpret the decrease in a as a descriptive process. Our claim is that MEC may achieve memory retrieval through the modulation of its own network (that is, the variation of the bump width) under a fixed memory configuration, but we also clearly know that there are biological mechanisms underlying this process. In fact, we proposed another model (which is not in this paper) where we gradually increased the firing threshold of the grid cells, and this model also successfully achieved the retrieval dynamics we proposed in our paper. Besides, short-term plasticity may also regulate the recurrent width in order to achieve this. We may observe the width modulation directly from large-scale population recordings during the transition, or indirectly by comparing the bump widths across different stable states, or by finding intercortical feedbacks that could modulate this.
As for Q2, we would like to summarize the mapping between a neural network and our CANN, hoping that you can form a clearer picture in your mind. The CANN units represent MEC grid cells, with x corresponding to grid phase on the latent manifold. The recurrent kernel J represents approximately homogeneous interactions within MEC, while Iext represents memory-related input transmitted from the hippocampus to MEC.
We concluded that CANN can implement the reverse ODE of DDPM. However, we did not indicate that the learning/training process of CANN and DDPM are also the same. From our perspective, in real neural systems, the learning process is carried out through the formation of intercortical synaptic connections between HPC and MEC. After learning a stable connection, it would not be costful for HPC to transmit memory-related information to MEC; the neural activities of HPC would naturally produce proper external inputs for MEC to conduct memory retrieval. Under this picture, Iext is not “presenting all training points”. HPC is activated under a certain retrieval context; these neural activities in HPC are then naturally encoded into external inputs to the MEC through HPC-MEC connections.
As for Q4, we would like to clarify that the CANN is not introduced merely as a memory model for demonstrating the efficacy of an already established annealing retrieval routine. Our primary motivation is neuroscientific: we aim to identify a biologically interpretable network mechanism for memory retrieval that provides new computational capabilities and, at the same time, suggests testable neural mechanisms. In particular, dynamically regulating the interaction scale allows the same recurrent network to transition from a regime with a large capture range to one with high retrieval resolution. The connection to diffusion models then provides a theoretical characterization of this neural retrieval process, showing that the resulting coarse-to-fine dynamics can formally implement a denoising diffusion process.
Finally, thank you for pointing out the important related works in Q5. We will incorporate these works into the manuscript shortly and provide appropriate discussion at relevant places to better situate our work within the related literature.
[1] Toroidal Topology of Population Activity in Grid Cells - Gardner et al., 2022