- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
In this paper the authors study how the attention weighting rule, temperature, and memory correlations affect retrieval in modern Hopfield networks. Their setting consists of fixed vectors in , interpreted as stored patterns, and a query vector . Similarities between the query and these patterns determine attention weights, and retrieval updates the query toward their weighted combination.
The authors introduce a partition function formulation and a Fenchel-conjugate free energy. Their "generalized (relaxed) model" in equation 3.5 combines a quadratic query penalty with a Tsallis entropy regularization. Different choices of the Tsallis parameter recover softmax, entmax, and sparsemax attention for the distribution of memory patterns. Within this model, the authors analyze when competing retrieval solutions have equal energy and in particular when a sparsemax solution ceases to be a locally stable minimum, relating the latter condition to correlations among the memories receiving nonzero weight.
In the main numerical experiments the authors present, they investigate the retrieval solutions obtained when changing the inverse temperature or number of patterns. In the appendices the authors also discuss fluctuating queries and attention weights (which are kept fixed in the main setting presented), they test the analytical conditions numerically, and also train simple cross-attention models on two tasks: recovering the value associated with one target memory and averaging the values of a relevant group of memories.
Critical strengths
I personally find the question of how correlated memories and sparse attention weights affect retrieval to be interesting and scientifically relevant. In particular, in the proposition 4.2 the authors provide an explicit local stability condition using the Gram matrix of the memories receiving nonzero weight. As long as this collection of memories remains unchanged, the condition identifies when a query solution remains a locally stable minimum of the retrieval energy. I found this to be the clearest theoretical result of the paper, since here they make the dependence of local stability on memory correlations explicit.
I also appreciate the numerical effort, much detailed in the appendix. Exploring different initializations and gradually increasing or decreasing the parameters helps investigate competing retrieval solutions and dependence on the previous state. The appendix includes numerical checks of the analytical formulas and trained cross-attention experiments. I also find the comparison between recovering one target value and averaging several relevant values useful, since it illustrates why concentrating attention can help one task while being detrimental to another.
Critical weaknesses
Overall I personally found the paper difficult to follow, particularly when trying to identify its genuinely new results from the already literature. Below I list the main weaknesses I personally found, which in order concern the novelty and significance of the results beyond previous work, the motivation and interpretation of the main theoretical findings, and some inconsistencies in the free-energy construction.
I found that a substantial part of the main text concerns established connections between optimization with entropy regularization, attention weights, and Hopfield retrieval energies. The variational characterization of softmax, sparsemax, and entmax and the broader Fenchel formulation are already available in the cited literature. The authors themselves describe proposition 3.3 as a "re-derivation" and state that equation 3.5 recovers an existing energy form. I personally find these connections elegant and interesting, but recovering the corresponding attention weights after choosing the Tsallis regularizer does not, in my opinion, constitute a new derivation of attention from first principles.
It seems to me that propositions 4.1 and 4.2, together with the numerical investigation of competing retrieval solutions, are the principal additional contributions of this work. I had however difficulty in understanding their significance relative to the broad claims that are present in the abstract and introduction. Proposition 4.1 compares the energies of different retrieval solutions, while proposition 4.2 provides a local stability condition for sparsemax retrieval. These results are useful, but I think the authors should better explain which new conclusions follow from them, and in particular how they establish the claimed transitions and retrieval reliability. In the current form of the manuscript I am not convinced that these additional results sufficiently support the broad contribution claimed in the paper.
I also think the organization of the manuscript contributes to this difficulty. Much of the early main text is devoted to statistical mechanical and convex analysis preliminaries (which could have maybe be shifted in the appendix), while important details needed to understand the specific results and experiments are deferred to the appendices. I think the authors should move some of these preliminary concepts to the appendix, and instead better motivate propositions 4.1 and 4.2, which seem to be the genuinely novel results, explain their implications, and present the main numerical setting more clearly.
I also find the title and abstract broader than the theoretical setting actually studied. The main analysis presented concerns iterative retrieval with fixed memories in modern Hopfield networks, rather than a general transformer attention mechanism. In self-attention, queries, keys, and values depend on the input and learned projections (, , and ). These quantities can indeed be considered fixed for a given input, and I also recognize the established connection between attention and Hopfield retrieval. However, I don't think that recovering the same attention weights and applying them to arbitrary values automatically extend the stability and coexistence results to a transformer layer with separate keys and values. I think the authors should be more explicit about the scope of their conclusions and how they connect to transformer architectures. In my opinion, a title explicitly mentioning modern Hopfield networks would better reflect the actual contribution of the manuscript.
I have two technical concerns regarding the free energy construction presented in the first part of the main text. First, definition 2.3 defines as the global Fenchel conjugate of , and remark 2.2 states that this function is convex. However, the expression obtained in theorem 3.1, equation 3.4, is generally nonconvex. I do not understand how these two expressions can represent the same free energy under the definitions provided. I think the authors should clarify whether an additional approximation or a different definition is being introduced. equation 3.5 can of course be studied as a separate energy per-se, but its connection to the exact free-energy construction of the initial main text remains unclear to me.
In theorem 2.1, differentiating the stated "deformed log-partition function" gives , with a factor . The expectation in the theorem is normalized, whereas the derivative contains this additional factor. Therefore, it seems to me that the identification of the conjugate variable with the normalized expectation in equation 2.4 does not follow in general. I would appreciate clarification of this step. I don't think these two issues invalidate the subsequent calculations of the authors, but I think they are important since the statistical mechanical derivation is presented as a central contribution of the paper.
Regarding the main analytical results, I think the interpretation of propositions 4.1 and 4.2 should also be more precise. The equality of energies in proposition 4.1 does not establish alone the coexistence of two locally stable retrieval solutions, I think. For proposition 4.2, the stability condition is derived while the set of memories receiving nonzero weight remains fixed. I don't see that this condition alone determines the temperature at which an actual discontinuous jump occurs. I think the distinction between equal energy solutions, loss of local stability, and actual discontinuities in the retrieved state should be explained better, especially since these are central to the claimed phase transitions, and seem to me the genuinely novel results.
I find that some of the interpretations of the numerical results are stronger than what the experiments directly establish. Figures 2 and 3 concern Hopfield retrieval rather than transformer training, and I think their experimental setting should be described more clearly in the main text. In particular, the maximal overlap measures alignment with some stored pattern, but does not identify whether a specific target memory was correctly recovered. It seems to me that a small variance of this observable does not necessarily establish retrieval of the same memory. Similarly observing no hysteresis in the softmax scans does not establish that only one stable retrieval solution exists.
I also could not understand where the claimed transitions "between glassy and robust-inference phases" are actually characterized. The experiments show competing retrieval solutions, hysteresis, and changes in attention concentration, but I did not find a precise criterion distinguishing a glassy phase from the coexistence of several local minima. I appreciate that appendix H also includes experiments with actual trained cross-attention models, but these mainly illustrate the different effects of sparse/dense attention on selective retrieval and aggregation tasks. I do not see how they directly validate the claimed glassy/robust transitions, or how their results are connected to the thresholds derived in propositions 4.1 and 4.2. I think this connection is particularly important given the claims made in the abstract.
In the end for all the weaknesses stated above, I lean towards rejection. I find the overall framework interesting, but from what I understand to be the genuinely new contributions, I am not yet convinced that these results alone justify acceptance, particularly in view of the unclear connection to the broader claims about transformer attention and the technical concerns discussed above. I am happy to reconsider my score if the authors resolve all of my issues.
Questions for authors
I list below my questions for the authors, please equally refer to more detailed weaknesses section.
-
Can the authors clarify what they consider to be the genuinely new theoretical contributions of this work, beyond the established connections between Hopfield attention with entropy regularizatio? In particular, are propositions 4.1 and 4.2 the principal new results? What is a more concrete explanation of their significance and of what new conclusions about retrieval they provide relative to the existing literature?
-
Can the authors clarify the free-energy construction in sections 2 and 3? How does the convex Fenchel conjugate free-energy in definition 2.3 lead to the nonconvex expression in theorem 3.1? Can they also check the normalization factor in theorem 2.1, and explain if it affects the macroscopic variable in equation 2.4 and what it implies?
-
For propositions 4.1 and 4.2, can the authors explain more precisely when these conditions correspond to actual phase transitions in retrieval?
-
Can the authors clarify what they mean by the "glassy" and "robust-inference" phases claimed in the abstract, and which theoretical or experimental results establish their distinction? I would also like to understand more precisely which conclusions obtained for iterative Hopfield retrieval translate to ordinary transformer attention, and whether their trained cross-attention experiments test any of the phase transitions predicted in the main text.
8 comments
Rebuttal 2: single-step attn, value-independence of attn weights, nonconvexity of
W4. The main analysis presented concerns iterative retrieval with fixed memories in modern Hopfield networks, rather than a general transformer attention mechanism. I don't think that recovering the same attention weights and applying them to arbitrary values automatically extend the stability and coexistence results to a transformer layer with separate keys and values.
Sorry for any confusion caused. To clarify,
- our analysis considers only single-step attention in standard Transformer.
- Importantly, our phase analysis focuses on how attention weights are distributed over fixed keys and when that configuration is stable.
- Since attention weights are independent of values, applying to arbitrary values does not change this energy landscape and hence our analysis.
To be more concrete:
-
“The main analysis presented concerns iterative retrieval with fixed memories in modern Hopfield networks, rather than a general transformer attention mechanism.”
- To clarify, the attention-Hopfield correspondence links two objects
- (i) the fixed-point dynamics of the retrieval update and
- (ii) local minima of the free-energy landscape of the statistical-mechanical system.
- Further, [1, Corollary 3.1.2] and [5, Thm 5] show, theoretically and numerically, that a single update can reach a retrieval fixed point under realistic settings.
- Thus, our framework considers a one-step attention in Transformer due to the Hopfield-attention dual.
- Our phase analysis then asks: how stable attention configurations compete as temperature changes, when one loses stability, and when enters a robust-inference regime. We study this through the local minima of the fixed-key energy landscape.
For more details, please also see our response to kcrx W3
- To clarify, the attention-Hopfield correspondence links two objects
-
“ I don't think that recovering the same attention weights and applying them to arbitrary values automatically extend the stability and coexistence results to a transformer layer with separate keys and values.”
-
The stationary equations involve the keys and their attention weights. With , they are
-
Here, the keys stay fixed while the stationary query and its weights vary. Values do not enter these equations. Applying the weights through changes the output, but leaves this landscape unchanged. Feeding that output into a different Transformer recurrence requires its own stability analysis.
-
For a stability of the full Transformer layer, including LayerNorm, feed-forward networks, residual updates operations, we deem it as an important future direction. However, it requires substantial additional analysis on statistical mechanics correspondence for each operation, rather than only Hopfield-attention dual. They are beyond the scope of this draft.
-
[1] On Sparse Modern Hopfield Model, Hu et al., NeurIPS 2023.
[5] Hopfield Networks is All You Need, Ramsauer et al., ICLR 2021.
W5. First, Def 2.3 defines as the global Fenchel conjugate of , and Rmk 2.2 states that this function is convex. However, the expression obtained in theorem 3.1, equation 3.4, is generally nonconvex. I do not understand how these two expressions can represent the same free energy under the definitions provided. I think the authors should clarify whether an additional approximation or a different definition is being introduced. equation 3.5 can of course be studied as a separate energy per-se, but its connection to the exact free-energy construction of the initial main text remains unclear to me.
You are right that these are different objects. The global Fenchel conjugate in Def 2.3 is convex. On the other hand, the energies in Eqs. (3.4) and (3.5) are reduced free energies obtained by minimizing over the attention weights at each fixed query .
Eq. (3.4) minimizes the joint Hamiltonian over . Eq. (3.5) includes the entropy regularizer. For the latter, with on the probability simplex , we have
This is a convex quadratic minus another convex function, so the reduced energy need not be convex in . For more details, we invite the reviewer to have a look on our response to KCRJ. We will make the distinction between the exact Fenchel-conjugate free energy and this reduced energy explicit.
Rebuttal 3: Thm 2.1 normalization factor typo (thanks!), Prop 4.1 & 4.2 clarifications
W6. In theorem 2.1, differentiating the stated "deformed log-partition function" gives , with a factor . The expectation in the theorem is normalized, whereas the derivative contains this additional factor. Therefore, it seems to me that the identification of the conjugate variable with the normalized expectation in equation 2.4 does not follow in general. I would appreciate clarification of this step. I don't think these two issues invalidate the subsequent calculations of the authors, but I think they are important since the statistical mechanical derivation is presented as a central contribution of the paper.
Thank you for your careful reading.
You are absolutely correct - the factor is missing. The conjugate variable is the unnormalized escort moment, , rather than the normalized expectation alone.
Here is the calculation: we differentiate the objective in
For , the two derivative identities are
and
Combining them, at an interior maximizer where these derivatives exist,
Thus,
The free energy then simplifies to
However, this still recovers the forms in Thm 3.1 (Free Energy of MHM) and Eq. (3.5) (modified free energy), since for a collapsing distribution . Hence, the following statistical mechanics interpretation and stability analysis remain unchanged.
W7. The equality of energies in Prop 4.1 does not establish alone the coexistence of two locally stable retrieval solutions. For Prop 4.2, the stability condition is derived while the set of memories receiving nonzero weight remains fixed. I don't see that this condition alone determines the temperature at which an actual discontinuous jump occurs. I think the distinction between equal energy solutions, loss of local stability, and actual discontinuities in the retrieved state should be explained better, especially since these are central to the claimed phase transitions, and seem to me the genuinely novel results.
Sorry for any confusion caused. Yes, equal energy, loss of stability, and an abrupt change in attention weights are different conditions. Specifically:
- Equal energy. Def 4.2 and Prop 4.1 compare stable attention configurations, represented by the local minima (Def 4.1). It means neither configuration is energetically preferred over the other. It is an additional condition, rather than a proof that those minima exist. Both may remain locally stable at this point.
- Spinodal and support change. Prop 4.2 gives the stability threshold while the solution varies smoothly and the set of keys with non-zero weight stays fixed. On the other hand, a local minimum can switch to another local minimum by support change.
- Abrupt change in attention weights. If the sweep switches to another local minimum, the query and attention weights may change abruptly. However, neither loss of stability nor a support change alone establishes a jump in maximal overlap. That jump must be checked separately from energy equality or a support change.
Rebuttal 1: what's new, Prop 4.1 & 4.2 significance, shorten prelims
To clarify, in the main text, only Prop 2.1 & 3.3 come from prior work, and we cite them explicitly. All other results are genuinely new. Please also see our response to kcrx W1 for related discussions.
As for Prop 3.3, Yes, Prop 3.3 recovers known attention maps [1,2], and we cite them explicitly. We include it to show how those known results arise within our stats-mech framework.
The point is - the route from a Hamiltonian to attention, and how this route lets us study attention through statistical mechanics:
We remark that - these steps connect microscopic variables to macroscopic observables. They are the same treatments in stats-mech textbooks for ideal gases, Fermi/Bose gases, and the Ising model [3-6]. So the role of prop 3.3 (and other things to prepare it) is to see/confirm how attn emerge from standard stats-mech.
[1] On Sparse Modern Hopfield Model, Hu et al., NeurIPS 2023.
[2] Sparse sequence-to-sequence models, Peters et al., 2019.
[3] Introduction to Modern Statistical Mechanics, David Chandler, 1987
[4] Statistical Physics of Particles (& Fields), Mehran Kardar, 2007
[5] Statistical Mechanics 4th, R.K. Pathria, 2021
[6] Statistical Mechanics, Kerson Huang, 1987
[7] Possible generalization of Boltzmann-Gibbs statistics, Tsallis, 1988.
[8] Deformed exponentials and logarithms in generalized thermostatistics, Naudts, 2002.
The significance of Prop 4.1 and 4.2 is - they establish phase transitions between Transformer robust-inference and glassy phases.
Specifically:
Together, they characterize how and under what conditions does one local minimum switch to another one. This leads to the phase transition between the glassy and robust-inference phases for Transformers. For the detailed connection between local minima and inference robustness, please see our responses to below W9 and Q4.
Fig. 2 shows why both Prop 4.1 & 4.2 matter, for our analysis of glassy and robust-inference regimes:
Notably, an abrupt transition in the weights need not occur at the energy crossing.
Lastly, we have new numerical results confirming that Prop 4.1 & 4.2 characterize phase transitions of attention inference. Please see our response to 88rq Q1.
Thank you for your suggestions. We will further shorten our prelims sec. However, we also remark that - the prelim overview the standard stats-mech procedures. They provide the conceptual framework for us to derive attention from free energy formulation rigorously. For example, to see the physical meaning of the derivations, we need the ideas of partition function, Fenchel conjugate, and equation of motions.
We will also state the numerical setting alongside Figs. 2 & 3 if the space allows. For instance, we will include how each temperature sweep keeps the keys fixed and warm-starts each step from the previous solution in the main text. We hope this would make the narrative more easier to follow.