acceptodds
Under review as a conference paper at ICLR 2027

On the Statistical Mechanics of Attention

Abstract

We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention Mechanism, Statistical Mechanics, Transformer, Free Energy, Phase Transition

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_X3M8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

In this paper the authors study how the attention weighting rule, temperature, and memory correlations affect retrieval in modern Hopfield networks. Their setting consists of MM fixed vectors in Rd\mathbb R^d, interpreted as stored patterns, and a query vector xx. Similarities between the query and these patterns determine attention weights, and retrieval updates the query toward their weighted combination.

The authors introduce a partition function formulation and a Fenchel-conjugate free energy. Their "generalized (relaxed) model" in equation 3.5 combines a quadratic query penalty with a Tsallis entropy regularization. Different choices of the Tsallis parameter α\alpha recover softmax, entmax, and sparsemax attention for the distribution of memory patterns. Within this model, the authors analyze when competing retrieval solutions have equal energy and in particular when a sparsemax solution ceases to be a locally stable minimum, relating the latter condition to correlations among the memories receiving nonzero weight.

In the main numerical experiments the authors present, they investigate the retrieval solutions obtained when changing the inverse temperature or number of patterns. In the appendices the authors also discuss fluctuating queries and attention weights (which are kept fixed in the main setting presented), they test the analytical conditions numerically, and also train simple cross-attention models on two tasks: recovering the value associated with one target memory and averaging the values of a relevant group of memories.

Critical strengths

I personally find the question of how correlated memories and sparse attention weights affect retrieval to be interesting and scientifically relevant. In particular, in the proposition 4.2 the authors provide an explicit local stability condition using the Gram matrix of the memories receiving nonzero weight. As long as this collection of memories remains unchanged, the condition identifies when a query solution remains a locally stable minimum of the retrieval energy. I found this to be the clearest theoretical result of the paper, since here they make the dependence of local stability on memory correlations explicit.

I also appreciate the numerical effort, much detailed in the appendix. Exploring different initializations and gradually increasing or decreasing the parameters helps investigate competing retrieval solutions and dependence on the previous state. The appendix includes numerical checks of the analytical formulas and trained cross-attention experiments. I also find the comparison between recovering one target value and averaging several relevant values useful, since it illustrates why concentrating attention can help one task while being detrimental to another.

Critical weaknesses

Overall I personally found the paper difficult to follow, particularly when trying to identify its genuinely new results from the already literature. Below I list the main weaknesses I personally found, which in order concern the novelty and significance of the results beyond previous work, the motivation and interpretation of the main theoretical findings, and some inconsistencies in the free-energy construction.

I found that a substantial part of the main text concerns established connections between optimization with entropy regularization, attention weights, and Hopfield retrieval energies. The variational characterization of softmax, sparsemax, and entmax and the broader Fenchel formulation are already available in the cited literature. The authors themselves describe proposition 3.3 as a "re-derivation" and state that equation 3.5 recovers an existing energy form. I personally find these connections elegant and interesting, but recovering the corresponding attention weights after choosing the Tsallis regularizer does not, in my opinion, constitute a new derivation of attention from first principles.

It seems to me that propositions 4.1 and 4.2, together with the numerical investigation of competing retrieval solutions, are the principal additional contributions of this work. I had however difficulty in understanding their significance relative to the broad claims that are present in the abstract and introduction. Proposition 4.1 compares the energies of different retrieval solutions, while proposition 4.2 provides a local stability condition for sparsemax retrieval. These results are useful, but I think the authors should better explain which new conclusions follow from them, and in particular how they establish the claimed transitions and retrieval reliability. In the current form of the manuscript I am not convinced that these additional results sufficiently support the broad contribution claimed in the paper.

I also think the organization of the manuscript contributes to this difficulty. Much of the early main text is devoted to statistical mechanical and convex analysis preliminaries (which could have maybe be shifted in the appendix), while important details needed to understand the specific results and experiments are deferred to the appendices. I think the authors should move some of these preliminary concepts to the appendix, and instead better motivate propositions 4.1 and 4.2, which seem to be the genuinely novel results, explain their implications, and present the main numerical setting more clearly.

I also find the title and abstract broader than the theoretical setting actually studied. The main analysis presented concerns iterative retrieval with fixed memories in modern Hopfield networks, rather than a general transformer attention mechanism. In self-attention, queries, keys, and values depend on the input and learned projections (Q=XWQQ=XW_Q, K=XWKK=XW_K, and V=XWVV=XW_V). These quantities can indeed be considered fixed for a given input, and I also recognize the established connection between attention and Hopfield retrieval. However, I don't think that recovering the same attention weights and applying them to arbitrary values automatically extend the stability and coexistence results to a transformer layer with separate keys and values. I think the authors should be more explicit about the scope of their conclusions and how they connect to transformer architectures. In my opinion, a title explicitly mentioning modern Hopfield networks would better reflect the actual contribution of the manuscript.

I have two technical concerns regarding the free energy construction presented in the first part of the main text. First, definition 2.3 defines Γ\Gamma as the global Fenchel conjugate of WW, and remark 2.2 states that this function is convex. However, the expression obtained in theorem 3.1, equation 3.4, is generally nonconvex. I do not understand how these two expressions can represent the same free energy under the definitions provided. I think the authors should clarify whether an additional approximation or a different definition is being introduced. equation 3.5 can of course be studied as a separate energy per-se, but its connection to the exact free-energy construction of the initial main text remains unclear to me.

In theorem 2.1, differentiating the stated "deformed log-partition function" gives ∇JWα(J)=Cα(J)EpJ(α)[x]\nabla_J W_\alpha(J)=C_\alpha(J)\mathbb E_{p_J}^{(\alpha)}[x], with a factor Cα(J)=∫XpJ(x)α dμ(x)C_\alpha(J)=\int_{\mathcal X}p_J(x)^\alpha\,d\mu(x). The expectation in the theorem is normalized, whereas the derivative contains this additional factor. Therefore, it seems to me that the identification of the conjugate variable φ\varphi with the normalized expectation in equation 2.4 does not follow in general. I would appreciate clarification of this step. I don't think these two issues invalidate the subsequent calculations of the authors, but I think they are important since the statistical mechanical derivation is presented as a central contribution of the paper.

Regarding the main analytical results, I think the interpretation of propositions 4.1 and 4.2 should also be more precise. The equality of energies in proposition 4.1 does not establish alone the coexistence of two locally stable retrieval solutions, I think. For proposition 4.2, the stability condition is derived while the set of memories receiving nonzero weight remains fixed. I don't see that this condition alone determines the temperature at which an actual discontinuous jump occurs. I think the distinction between equal energy solutions, loss of local stability, and actual discontinuities in the retrieved state should be explained better, especially since these are central to the claimed phase transitions, and seem to me the genuinely novel results.

I find that some of the interpretations of the numerical results are stronger than what the experiments directly establish. Figures 2 and 3 concern Hopfield retrieval rather than transformer training, and I think their experimental setting should be described more clearly in the main text. In particular, the maximal overlap measures alignment with some stored pattern, but does not identify whether a specific target memory was correctly recovered. It seems to me that a small variance of this observable does not necessarily establish retrieval of the same memory. Similarly observing no hysteresis in the softmax scans does not establish that only one stable retrieval solution exists.

I also could not understand where the claimed transitions "between glassy and robust-inference phases" are actually characterized. The experiments show competing retrieval solutions, hysteresis, and changes in attention concentration, but I did not find a precise criterion distinguishing a glassy phase from the coexistence of several local minima. I appreciate that appendix H also includes experiments with actual trained cross-attention models, but these mainly illustrate the different effects of sparse/dense attention on selective retrieval and aggregation tasks. I do not see how they directly validate the claimed glassy/robust transitions, or how their results are connected to the thresholds derived in propositions 4.1 and 4.2. I think this connection is particularly important given the claims made in the abstract.

In the end for all the weaknesses stated above, I lean towards rejection. I find the overall framework interesting, but from what I understand to be the genuinely new contributions, I am not yet convinced that these results alone justify acceptance, particularly in view of the unclear connection to the broader claims about transformer attention and the technical concerns discussed above. I am happy to reconsider my score if the authors resolve all of my issues.

Questions for authors

I list below my questions for the authors, please equally refer to more detailed weaknesses section.

  1. Can the authors clarify what they consider to be the genuinely new theoretical contributions of this work, beyond the established connections between Hopfield attention with entropy regularizatio? In particular, are propositions 4.1 and 4.2 the principal new results? What is a more concrete explanation of their significance and of what new conclusions about retrieval they provide relative to the existing literature?

  2. Can the authors clarify the free-energy construction in sections 2 and 3? How does the convex Fenchel conjugate free-energy in definition 2.3 lead to the nonconvex expression in theorem 3.1? Can they also check the normalization factor in theorem 2.1, and explain if it affects the macroscopic variable in equation 2.4 and what it implies?

  3. For propositions 4.1 and 4.2, can the authors explain more precisely when these conditions correspond to actual phase transitions in retrieval?

  4. Can the authors clarify what they mean by the "glassy" and "robust-inference" phases claimed in the abstract, and which theoretical or experimental results establish their distinction? I would also like to understand more precisely which conclusions obtained for iterative Hopfield retrieval translate to ordinary transformer attention, and whether their trained cross-attention experiments test any of the phase transitions predicted in the main text.

8 comments
AuthorsOpenReview ↗

Rebuttal 1: what's new, Prop 4.1 & 4.2 significance, shorten prelims

W1. ...substantial part of the main text concerns established connections...Recovering the attention weights (Prop 3.3) after choosing the Tsallis regularizer does not constitute a new derivation of attention from first principles....

To clarify, in the main text, only Prop 2.1 & 3.3 come from prior work, and we cite them explicitly. All other results are genuinely new. Please also see our response to kcrx W1 for related discussions.

As for Prop 3.3, Yes, Prop 3.3 recovers known attention maps [1,2], and we cite them explicitly. We include it to show how those known results arise within our stats-mech framework.

The point is - the route from a Hamiltonian to attention, and how this route lets us study attention through statistical mechanics:

  • Hamiltonian formulation (Def 3.1).
  • Partition function through a Gibbs average (Def 2.1).
  • Free energy via Fenchel conjugation (Def 2.3 and Thm 3.1).
  • Attention via free-energy minimization (Prop 3.2 and Prop 3.3).

We remark that - these steps connect microscopic variables to macroscopic observables. They are the same treatments in stats-mech textbooks for ideal gases, Fermi/Bose gases, and the Ising model [3-6]. So the role of prop 3.3 (and other things to prepare it) is to see/confirm how attn emerge from standard stats-mech.

[1] On Sparse Modern Hopfield Model, Hu et al., NeurIPS 2023.

[2] Sparse sequence-to-sequence models, Peters et al., 2019.

[3] Introduction to Modern Statistical Mechanics, David Chandler, 1987

[4] Statistical Physics of Particles (& Fields), Mehran Kardar, 2007

[5] Statistical Mechanics 4th, R.K. Pathria, 2021

[6] Statistical Mechanics, Kerson Huang, 1987

[7] Possible generalization of Boltzmann-Gibbs statistics, Tsallis, 1988.

[8] Deformed exponentials and logarithms in generalized thermostatistics, Naudts, 2002.


W2. I had difficulty in understanding Prop 4.1 and 4.2 significance relative to the broad claims that are present in the abstract and introduction. I think the authors should better explain which new conclusions follow from them, and in particular how they establish the claimed transitions and retrieval reliability.

The significance of Prop 4.1 and 4.2 is - they establish phase transitions between Transformer robust-inference and glassy phases.

Specifically:

  • Prop 4.1: compares the energies of two locally stable attention configurations.
  • Prop 4.2: gives the stability threshold. A transition of a local minimum can occur in two ways: one is by support-change events, and the other is by vanishing curvature. Our Prop 4.2 gives the formula for those vanishing curvature points.

Together, they characterize how and under what conditions does one local minimum switch to another one. This leads to the phase transition between the glassy and robust-inference phases for Transformers. For the detailed connection between local minima and inference robustness, please see our responses to below W9 and Q4.

Fig. 2 shows why both Prop 4.1 & 4.2 matter, for our analysis of glassy and robust-inference regimes:

  • At intermediate β\beta, several stable configurations compete, and opposite sweeps may give different attention weights.
  • At large β\beta, the maximal query-key overlap z_\max^\star saturates across seeds and corresponds to concentrated attention.

Notably, an abrupt transition in the weights need not occur at the energy crossing.

Lastly, we have new numerical results confirming that Prop 4.1 & 4.2 characterize phase transitions of attention inference. Please see our response to 88rq Q1.


W3. I think the authors should move some of the preliminary concepts to the appendix, and instead better motivate propositions 4.1 and 4.2, which seem to be the genuinely novel results, explain their implications, and present the main numerical setting more clearly.

Thank you for your suggestions. We will further shorten our prelims sec. However, we also remark that - the prelim overview the standard stats-mech procedures. They provide the conceptual framework for us to derive attention from free energy formulation rigorously. For example, to see the physical meaning of the derivations, we need the ideas of partition function, Fenchel conjugate, and equation of motions.

We will also state the numerical setting alongside Figs. 2 & 3 if the space allows. For instance, we will include how each temperature sweep keeps the keys fixed and warm-starts each step from the previous solution in the main text. We hope this would make the narrative more easier to follow.

AuthorsOpenReview ↗

Rebuttal 2: single-step attn, value-independence of attn weights, nonconvexity of

W4. The main analysis presented concerns iterative retrieval with fixed memories in modern Hopfield networks, rather than a general transformer attention mechanism. I don't think that recovering the same attention weights and applying them to arbitrary values automatically extend the stability and coexistence results to a transformer layer with separate keys and values.

Sorry for any confusion caused. To clarify,

  • our analysis considers only single-step attention in standard Transformer.
  • Importantly, our phase analysis focuses on how attention weights are distributed over fixed keys and when that configuration is stable.
  • Since attention weights pp are independent of values, applying pp to arbitrary values does not change this energy landscape and hence our analysis.

To be more concrete:

  1. “The main analysis presented concerns iterative retrieval with fixed memories in modern Hopfield networks, rather than a general transformer attention mechanism.”

    • To clarify, the attention-Hopfield correspondence links two objects
      • (i) the fixed-point dynamics of the retrieval update and
      • (ii) local minima of the free-energy landscape of the statistical-mechanical system.
    • Further, [1, Corollary 3.1.2] and [5, Thm 5] show, theoretically and numerically, that a single update can reach a retrieval fixed point under realistic settings.
    • Thus, our framework considers a one-step attention in Transformer due to the Hopfield-attention dual.
    • Our phase analysis then asks: how stable attention configurations compete as temperature changes, when one loses stability, and when enters a robust-inference regime. We study this through the local minima of the fixed-key energy landscape.

    For more details, please also see our response to kcrx W3

  2. “ I don't think that recovering the same attention weights and applying them to arbitrary values automatically extend the stability and coexistence results to a transformer layer with separate keys and values.”

    • The stationary equations involve the keys and their attention weights. With Gμν=(ξμ)⊤ξν/dG_{\mu\nu}=(\xi^\mu)^\top\xi^\nu/d, they are

      x⋆=1d∑μ=1Mξμpμ⋆,zμ⋆=∑ν=1MGμνpν⋆.x^\star=\frac{1}{\sqrt d}\sum_{\mu=1}^{M}\xi^\mu p_\mu^\star,\quad z_\mu^\star=\sum_{\nu=1}^{M}G_{\mu\nu}p_\nu^\star.

    • Here, the keys stay fixed while the stationary query and its weights vary. Values VV do not enter these equations. Applying the weights through Attn(Q,K,V)=p(Q,K)V\text{Attn}(Q,K,V)=p(Q,K)V changes the output, but leaves this landscape unchanged. Feeding that output into a different Transformer recurrence requires its own stability analysis.

    • For a stability of the full Transformer layer, including LayerNorm, feed-forward networks, residual updates operations, we deem it as an important future direction. However, it requires substantial additional analysis on statistical mechanics correspondence for each operation, rather than only Hopfield-attention dual. They are beyond the scope of this draft.

[1] On Sparse Modern Hopfield Model, Hu et al., NeurIPS 2023.

[5] Hopfield Networks is All You Need, Ramsauer et al., ICLR 2021.


W5. First, Def 2.3 defines Γ\Gamma as the global Fenchel conjugate of WW, and Rmk 2.2 states that this function is convex. However, the expression obtained in theorem 3.1, equation 3.4, is generally nonconvex. I do not understand how these two expressions can represent the same free energy under the definitions provided. I think the authors should clarify whether an additional approximation or a different definition is being introduced. equation 3.5 can of course be studied as a separate energy per-se, but its connection to the exact free-energy construction of the initial main text remains unclear to me.

You are right that these are different objects. The global Fenchel conjugate in Def 2.3 is convex. On the other hand, the energies in Eqs. (3.4) and (3.5) are reduced free energies obtained by minimizing over the attention weights at each fixed query xx.

Eq. (3.4) minimizes the joint Hamiltonian over pp. Eq. (3.5) includes the entropy regularizer. For the latter, with pp on the probability simplex ΔM\Delta^M, we have

Γα(mod)[x,ξ]=inf⁡p∈ΔM{βH~[x,p,ξ]−Ψα[p]}=β2∑i=1dxi2−sup⁡p∈ΔM{β∑μ=1Mzμpμ+Ψα[p]}.\Gamma^{(\text{mod})}_\alpha[x,\xi]=\inf_{p\in\Delta^M}\{\beta\tilde{\mathcal{H}}[x,p,\xi]-\Psi_\alpha[p]\}=\frac{\beta}{2}\sum_{i=1}^dx_i^2-\sup_{p\in\Delta^M}\{\beta\sum_{\mu=1}^Mz_\mu p_\mu+\Psi_\alpha[p]\}.

This is a convex quadratic minus another convex function, so the reduced energy need not be convex in xx. For more details, we invite the reviewer to have a look on our response to KCRJ. We will make the distinction between the exact Fenchel-conjugate free energy and this reduced energy explicit.

AuthorsOpenReview ↗

Rebuttal 3: Thm 2.1 normalization factor typo (thanks!), Prop 4.1 & 4.2 clarifications

W6. In theorem 2.1, differentiating the stated "deformed log-partition function" gives ∇JWα(J)=Cα(J)EpJ(α)[x]\nabla_J W_\alpha(J)=C_\alpha(J)\mathbb{E}_{p_J}^{(\alpha)}[x], with a factor Cα(J)=∫XpJα(x)C_\alpha(J)=\int_{\mathcal{X}}p_J^\alpha(x). The expectation in the theorem is normalized, whereas the derivative contains this additional factor. Therefore, it seems to me that the identification of the conjugate variable φ\varphi with the normalized expectation in equation 2.4 does not follow in general. I would appreciate clarification of this step. I don't think these two issues invalidate the subsequent calculations of the authors, but I think they are important since the statistical mechanical derivation is presented as a central contribution of the paper.

Thank you for your careful reading.

You are absolutely correct - the factor Cα(J)=∫XpJα(x)dμ(x)C_\alpha(J)=\int_{\mathcal{X}}p^\alpha_J(x)\mathrm{d}\mu(x) is missing. The conjugate variable is the unnormalized escort moment, φ=Cα(J⋆)EpJ⋆(α)[x]\varphi=C_\alpha(J^\star)\mathbb{E}^{(\alpha)}_{p_{J^\star}}[x], rather than the normalized expectation alone.

Here is the calculation: we differentiate the objective in

sup⁡J{∑i=1dJi(φi−Ex∼pJ(α)[xi])+βEx∼pJ(α)[H[x]]−Ψα[pJ]Zαα−1[J]Cα[J]}.\sup_{J}\{\sum_{i=1}^dJ_i(\varphi_i-\mathbb{E}_{x\sim p_J}^{(\alpha)}[x_i])+\beta\mathbb{E}_{x\sim p_J}^{(\alpha)}[\mathcal{H}[x]]-\frac{\Psi_\alpha[p_J]}{Z_\alpha^{\alpha-1}[J]C_\alpha[J]}\}.

For α≠1\alpha\ne1, the two derivative identities are

β∂∂JkEx∼pJ(α)[H[x]]−∑i=1dJi∂∂JkEx∼pJ(α)[xi]=−∂JkCα(α−1)Zαα−1Cα2\beta\frac{\partial}{\partial J_k}\mathbb{E}^{(\alpha)}_{x\sim p_J}[\mathcal{H}[x]]-\sum_{i=1}^dJ_i\frac{\partial}{\partial J_k}\mathbb{E}_{x\sim p_J}^{(\alpha)}[x_i]=-\frac{\partial_{J_k}C_\alpha}{(\alpha-1)Z_\alpha^{\alpha-1}C_\alpha^2}

and

∂∂Jk(Ψα[pJ]Zαα−1Cα)=(Cα−1)EpJ(α)[xk]−∂JkCα(α−1)Zαα−1Cα2.\frac{\partial}{\partial J_k}(\frac{\Psi_\alpha[p_J]}{Z_\alpha^{\alpha-1}C_\alpha})=(C_\alpha-1)\mathbb{E}_{p_J}^{(\alpha)}[x_k]-\frac{\partial_{J_k}C_\alpha}{(\alpha-1)Z_\alpha^{\alpha-1}C_\alpha^2}.

Combining them, at an interior maximizer J⋆J^\star where these derivatives exist,

∂∂Jk{∑i=1dJi(φi−Ex∼pJ(α)[xi])+βEx∼pJ(α)[H[x]]−Ψα[pJ]Zαα−1[J]Cα[J]}∣J=J⋆=φk−Cα[J⋆]Ex∼pJ⋆(α)[xk]=0.\frac{\partial}{\partial J_k}\{\sum_{i=1}^dJ_i(\varphi_i-\mathbb{E}_{x\sim p_J}^{(\alpha)}[x_i])+\beta\mathbb{E}_{x\sim p_J}^{(\alpha)}[\mathcal{H}[x]]-\frac{\Psi_\alpha[p_J]}{Z_\alpha^{\alpha-1}[J]C_\alpha[J]}\}|_{J=J^\star}=\varphi_k-C_\alpha[J^\star]\mathbb{E}_{x\sim p_{J^\star}}^{(\alpha)}[x_k]=0.

Thus,

φ=Cα[J⋆]Ex∼pJ⋆(α)[x]=∫XxpJ⋆α(x)dμ(x).\varphi=C_\alpha[J^\star]\mathbb{E}_{x\sim p_{J^\star}}^{(\alpha)}[x]=\int_{\mathcal{X}}xp^\alpha_{J^\star}(x)\mathrm{d}\mu(x).

The free energy then simplifies to

Γα[φ]=βCα[J⋆]Ex∼pJ⋆(α)[H[x]]−Ψα[pJ⋆].\Gamma_\alpha[\varphi]=\beta C_\alpha[J^\star]\mathbb{E}_{x\sim p_{J^\star}}^{(\alpha)}[\mathcal{H}[x]]-\Psi_\alpha[p_{J^\star}].

However, this still recovers the forms in Thm 3.1 (Free Energy of MHM) and Eq. (3.5) (modified free energy), since Cα[J]=1C_\alpha[J]=1 for a collapsing distribution ρJ(x,p)\rho_J(x,p). Hence, the following statistical mechanics interpretation and stability analysis remain unchanged.


W7. The equality of energies in Prop 4.1 does not establish alone the coexistence of two locally stable retrieval solutions. For Prop 4.2, the stability condition is derived while the set of memories receiving nonzero weight remains fixed. I don't see that this condition alone determines the temperature at which an actual discontinuous jump occurs. I think the distinction between equal energy solutions, loss of local stability, and actual discontinuities in the retrieved state should be explained better, especially since these are central to the claimed phase transitions, and seem to me the genuinely novel results.

Sorry for any confusion caused. Yes, equal energy, loss of stability, and an abrupt change in attention weights are different conditions. Specifically:

  • Equal energy. Def 4.2 and Prop 4.1 compare stable attention configurations, represented by the local minima (Def 4.1). It means neither configuration is energetically preferred over the other. It is an additional condition, rather than a proof that those minima exist. Both may remain locally stable at this point.
  • Spinodal and support change. Prop 4.2 gives the stability threshold while the solution varies smoothly and the set of keys with non-zero weight stays fixed. On the other hand, a local minimum can switch to another local minimum by support change.
  • Abrupt change in attention weights. If the sweep switches to another local minimum, the query and attention weights may change abruptly. However, neither loss of stability nor a support change alone establishes a jump in maximal overlap. That jump must be checked separately from energy equality or a support change.
Reviewer_KCRJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies attention as a modern Hopfield model. Section 2 introduces the standard free-energy formalism: a partition function with a source term, its logarithm, and the Legendre transform of the logarithm, extended to Tsallis entropies. Section 3 assumes that the query and the attention weights have no randomness. Under this assumption the free energy reduces to β times the Hopfield energy. A Tsallis entropy term is then added, and the result is the energy function of Hu et al. (2023). Softmax, sparsemax and entmax are obtained as its gradient. Section 4 defines coexistence points and spinodal points of the local minima of this energy as β varies, and derives two conditions for the sparsemax case. Section 5 presents numerical scans of the maximal overlap against β and against the memory load. The paper reports jumps and hysteresis for sparsemax and smooth behavior without hysteresis for softmax. This summary covers the main text only.

Critical strengths

None.

Critical weaknesses
  1. The paper contains no statistical mechanics. The formalism of Section 2 rests on averages over a Gibbs measure. Section 3 sets the variance to zero, which is the zero-temperature limit of that measure: the free energy collapses to the Hopfield energy itself (the paper states that (3.5) is exactly the energy of Hu et al., 2023), and the only surviving β is the kernel sharpness inside the log-sum-exp, not a temperature. A genuine treatment keeps kernel sharpness and bath temperature distinct, includes an entropy term, averages over random patterns, and yields a phase diagram in the load-temperature plane, as in Amit, Gutfreund and Sompolinsky (1985, 1987), cited by the paper, and Petrova et al. (ICML 2026, PMLR 306:98488–98498). What the paper actually does is a bifurcation analysis of a fixed deterministic energy as the kernel sharpness varies. "Temperature", "phase transition", "glassy phase" and "rigorous statistical-mechanics framework" do not apply.

  2. The main claim is contradicted by the paper's own results. The abstract announces first-order phase transitions in attention between a glassy and a robust-inference phase. Transformer attention is softmax, and for softmax the paper finds no transition: Section 4 calls discontinuities rare, Section 5 reports no hysteresis, and the conclusion states that softmax attention is "a mixture of all patterns". This also contradicts work the paper cites: Ramsauer et al. (2020) prove that well-separated patterns are stable fixed points at large β, and Koulischer et al. (2023) show a transition in β for softmax in the same equidistant-pattern setting as Proposition 4.1. The glassy and robust phases are never defined, and no order parameter is computed. The sparsemax theory does not repair this: the only worked example of Proposition 4.1 compares a minimum with a saddle, and a single-pattern state can never satisfy the spinodal condition of Proposition 4.2.

Questions for authors
  1. Which β is the temperature: the one in the Gibbs weight of Definition 2.1, or the one inside the log-sum-exp of (3.5)? If the former is sent to zero by the zero-variance assumption, what part of the analysis remains statistical, and on what grounds is the energy function of Hu et al. called a free energy?

  2. Please define the glassy phase and the robust-inference phase, and specify the order parameter that distinguishes them. Which figure shows a glassy phase?

13 comments
AuthorsOpenReview ↗

Rebuttal 1: W1 - stats-mech concerns

W1. The paper contains no statistical mechanics. The formalism of Section 2 rests on averages over a Gibbs measure. Section 3 sets the variance to zero, which is the zero-temperature limit of that measure: the free energy collapses to the Hopfield energy itself (the paper states that (3.5) is exactly the energy of Hu et al., 2023), and the only surviving β is the kernel sharpness inside the log-sum-exp, not a temperature. A genuine treatment keeps kernel sharpness and bath temperature distinct, includes an entropy term, averages over random patterns, and yields a phase diagram in the load-temperature plane, as in Amit, Gutfreund and Sompolinsky (1985, 1987), cited by the paper, and Petrova et al. (ICML 2026, PMLR 306:98488–98498).

Thank you for raising these points. We believe there are some oversights.

To clarify, our work follows standard statistical-mechanical steps [1-4]:

  • A Hamiltonian for the microscopic system (lines 95 and 234).
  • Gibbs averages and a partition function to obtain macroscopic observables (Def 2.1).
  • Free energy through the Fenchel conjugate (Def 2.3, Thm 3.1, and Eq. (3.5)).
  • Attention as an equation of motion from free-energy minimization (Prop 3.2 and 3.3).

These steps connect microscopic variables to macroscopic observables. They also appear in stats-mech textbook treatments of ideal gases, Fermi/Bose gases, and the Ising model [1-4]. This is why statistical mechanics is central to our work. It gives a way to discuss entropy, query/key fluctuations, and effective dynamics beyond the update rule itself.

  1. Section 3 sets the variance to zero, which is the zero-temperature limit of that measure

    Our deterministic-query assumption is a modeling choice (no stochasticity in query), not a zero-temperature limit. We treat queries as controlled during memory retrieval. We do not take a thermodynamic limit or average over disorder either. Please see the setup in lines 239-245. App E gives the extension with stochasticity in queries. Please also see our response below Q1.

  2. the only surviving β is the kernel sharpness inside the log-sum-exp, not a temperature

    We may be reading this point differently. Do you mean that β\beta in the free energy, Eq. (3.5), differs from β\beta in the log-partition function, Eq. (2.2)?

    We use the same inverse-temperature β\beta in both. We derive the free energy from the log-partition function through the Fenchel conjugate. We do not introduce another β\beta or take a zero-temperature limit in this step. Please also see our response below Q1.

    If we misunderstand your concern, please let us know.

  3. A genuine treatment keeps kernel sharpness and bath temperature distinct, includes an entropy term, averages over random patterns, and yields a phase diagram in the load-temperature plane

    • Our setup uses a finite-dimensional system with clustered patterns and finite inverse-temperature β\beta. Eq. (3.5) lets us study entropy effects and the phase structure in this setting. We do not assume two separate temperatures or random patterns.

    • The deterministic-query setting and the extension with fluctuations in App E are two parts of this framework. Neither requires us to introduce a new β\beta. We hope this clarifies the statistical-mechanical setting we study.

    • Lastly, we find it hard to interpret "A genuine treatment keeps kernel sharpness and bath temperature distinct". We do not see how this is genuine in standard stats-mech. Can you kindly elaborate more if you deem this important. Many thanks!

[1] Introduction to Modern Statistical Mechanics, David Chandler, 1987

[2] Statistical Physics of Particles (& Fields), Mehran Kardar, 2007

[3] Statistical Mechanics 4th, R.K. Pathria, 2021

[4] Statistical Mechanics, Kerson Huang, 1987

AuthorsOpenReview ↗

Rebuttal 2: W2 - more than bifurcation analysis

W2. What the paper actually does is a bifurcation analysis of a fixed deterministic energy as the kernel sharpness varies. "Temperature", "phase transition", "glassy phase" and "rigorous statistical-mechanics framework" do not apply.

We disagree. Our work covers much more than bifurcation analysis of fixed points:

  1. What the formulation adds beyond bifurcation analysis?

    • Our free energy assigns a value to each retrieval branch (Eq. (3.5)). This lets us compare their energetic preference, not just their local stability. We also evaluate coexistence points, where two competing branches have equal energy (Def 4.2 and Prop 4.1).

    • The same formulation extends to query stochasticity in App E. It also brings statistical-mechanical tools to the study of Transformer attention, including phase analysis and connections to annealing and many-body methods [5,6]. This perspective can inform energy-based architectures as well [7,8].

    • These aspects are not captured by the bifurcation analysis alone.

  2. What we mean by the terms in W2?

    • Temperature. We introduce β\beta in the partition function (Def 2.1) to set the weight of each input xx. We keep the same β\beta throughout the free-energy derivation. In Eq. (3.5), it controls the balance between the Hamiltonian and entropy.

    • Phase transition. We study discontinuous changes between stable retrieval states as β\beta or the memory load κ\kappa varies (Sec 4). The order parameter is the maximal overlap z^\star_\max. We also track attention support and branch energies to examine retrieval selectivity. Fig 2 shows hysteresis and the transition associated with loss of stability.

    • Robust and glassy phases. By “robust,” we mean that attention concentrates on the keys most relevant to the query. In Fig 2, this corresponds to the concentrated selection in the large-β\beta regime. By “glassy,” we mean that attention also aggregates irrelevant keys. The intermediate-β\beta regime shows aggregation of distinct keys, with the retrieved state depending on the sweep direction.

    • Statistical mechanics. We specify a Hamiltonian, construct a partition function and macroscopic observables, and introduce free energy through the Fenchel conjugate. These are standard textbook steps [1-4]. Our setup requires neither a thermodynamic limit nor a zero-temperature limit.

[1] Introduction to Modern Statistical Mechanics, David Chandler, 1987

[2] Statistical Physics of Particles, Mehran Kardar, 2007

[3] Statistical Mechanics 4th, R.K. Pathria, 2021

[4] Statistical Mechanics, Kerson Huang, 1987

[5] Dynamical Properties of Dense Associative Memory, Mimura et al., ICLR 2026

[6] Statistical Advantage of Softmax Attention: Insights from Single-Location Regression, Duranthon et al., ICLR 2026

[7] Large Associative Memory Problem in Neurobiology and Machine Learning, Krotov and Hopfield, ICLR 2021

[8] Energy-Based Transformers are Scalable Learners and Thinkers, Gladstone et al., ICLR 2026

AuthorsOpenReview ↗

Rebuttal 3: W3 - no real contradiction

W3. The main claim is contradicted by the paper's own results. The abstract announces first-order phase transitions in attention between a glassy and a robust-inference phase. Transformer attention is softmax, and for softmax the paper finds no transition: Section 4 calls discontinuities rare, Section 5 reports no hysteresis, and the conclusion states that softmax attention is "a mixture of all patterns". This also contradicts work the paper cites: Ramsauer et al. (2020) prove that well-separated patterns are stable fixed points at large β, and Koulischer et al. (2023) show a transition in β for softmax in the same equidistant-pattern setting as Proposition 4.1. The glassy and robust phases are never defined, and no order parameter is computed. The sparsemax theory does not repair this: the only worked example of Proposition 4.1 compares a minimum with a saddle, and a single-pattern state can never satisfy the spinodal condition of Proposition 4.2.

Sorry for the confusion. The claimed contradictions do not hold. We clarify three points:

  1. Softmax

    There is no contradiction with Ramsauer's or Koulischer's work. The absence of hysteresis in our experiment does not imply that softmax cannot have multiple attractors. Positive memory weights can still concentrate strongly near individual memories.

  2. Order parameter, robust phase, and glassy phase

    As discussed in W2, our order parameter is the maximal overlap z^\star_\max. By “robust,” we mean that Transformer attention aggregates the keys most relevant to the query during inference. By “glassy,” we mean that attention also aggregates some irrelevant keys.

  3. Single-pattern stability and the two-key example

    A single-pattern state has no finite interior spinodal. Its Hessian is ∇x2Γα=2(mod)=β1d\nabla^2_x\Gamma_{\alpha=2}^{(\text{mod})}=\beta\mathbb{1}_d. This does not contradict Prop 4.2. A mixed branch can lose stability at finite β\beta, as in the symmetric two-key example.

    For the saddle-point question, our claim is that the two-key example has degenerate global minima at β=2/(1−ϵ)\beta=2/(1-\epsilon), not a saddle. Substituting this value into Eq. (3.5) gives

    Γα=2(mod)[x,ξ]=−β2,at  β=21−ϵ.\Gamma_{\alpha=2}^{(\text{mod})}[x,\xi]=-\frac{\beta}{2},\quad\text{at }~\beta=\frac{2}{1-\epsilon}.

We hope this clarifies the three points. If we misunderstand your concerns, please let us know.

Reviewer_88rqOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper starts with a mathematical comparison between modern Hopfield networks and attention, where attention corresponds to the time evolution of an input in a Hopfield network through minimization of the free energy. It introduces a statistical mechanics formalism of a generating function and a free energy functional, and uses the generalized Tsallis entropy as a regularizer to go beyond the standard mapping of Shannon entropy to softmax/Gibbs distributions. The Tsallis entropy has a single parameter that interpolates between the Shannon case, which recovers softmax, and sparse alternatives such as sparsemax and entmax. The paper then analyzes branching and first-order phase transitions of the model as the temperature and the number of stored memories vary, comparing sparse and softmax attention. It derives conditions for these transitions in simplified cases and supports them numerically, reporting hysteresis and discontinuous jumps in the sparse case and smooth behavior in the softmax case. The appendix adds small-scale Transformer experiments comparing softmax and sparse attention on retrieval synthetic tasks.

Critical strengths

The paper reads well overall, and the main mathematical statements are clearly formulated. The motivation and the gaps in the existing literature are well articulated. The analysis of how the retrieval states of modern Hopfield networks reorganize as the temperature and memory load vary, through branching, coexistence and spinodal points, and changes in the support of the attention weights, offers an interesting new angle on these models, particularly in the contrast it draws between sparse and dense attention. This direction should be extended both experimentally, beyond synthetic inputs, and analytically, through a disordered statistical-mechanics treatment.

Critical weaknesses

The paper presents an elaborate statistical-mechanics formalism (generating functional, Fenchel-conjugate free energy, escort averages, deformed exponentials). However, most of this machinery is not used in deriving the main results. A statistical-mechanics treatment of associative memory typically considers random (disordered) patterns and averages over the microscopic degrees of freedom, so that sharp thresholds emerge in a thermodynamic limit. Here, in contrast, the authors explicitly restrict to a deterministic setting (e.g., line 243 and footnote 3), in which the query, the attention weights, and the memory patterns are all fixed, so the analysis is conditioned on a specific realization of the memories. In this setting the free energy reduces to the energy itself (Theorem 3.1).

As far as I can tell, after integrating out the query xx, the main results follow from minimizing a quadratic energy in the attention weights with an entropic regularizer, min⁡p∈ΔM[−12pTGp−1βΨα(p)]\min_{p \in \Delta^M} [ - \frac{1}{2} p^T G p - \frac{1}{\beta} \Psi_\alpha(p) ], where Gμν=1d∑iξiμξiνG_{\mu\nu} = \frac{1}{d} \sum_i \xi_i^\mu \xi_i^\nu is the Gram matrix of the memories and Ψα(p)=1−∑μpμαα−1\Psi_\alpha(p) = \frac{1 - \sum_\mu p_\mu^\alpha}{\alpha - 1} is the Tsallis entropy. The effect of this regularizer on retrieval is interesting, but the corresponding sparse modern Hopfield model and its retrieval properties were introduced and analyzed in prior work (Hu et al., 2023; Wu et al., 2023; Santos et al., 2024, 2025). Moreover, a disordered statistical-mechanics analysis of dense associative memories at exponential load, with random patterns, a thermodynamic limit, a phase diagram, and exact retrieval thresholds, was carried out by Lucibello and Mézard (2024). It should be cited and discussed; extending such an analysis to the sparse case would be a genuine contribution if it is analytically tractable.

In addition, the critical values of β\beta derived in Section 4 are verified only on constructed examples (Appendix H.3) and are not compared with the transitions observed in the main numerical results. In fact, the authors note (Remark G.4) that most of the observed jumps come from changes in the support of the attention weights rather than from the spinodal mechanism of Proposition 4.2. Finally, since dd and MM are fixed and no scaling with dd is studied, the reported transitions are bifurcations of local minima rather than phase transitions in the statistical-mechanics sense, which correspond to non-analytic points of the free energy in the thermodynamic limit.

Despite the motivation from Transformer attention, the only experiments involving an actual attention layer (Appendix H) are very small in scale (a single-layer, four-head cross-attention model with model dimension 96) and use synthetic random inputs rather than realistic data, even for a qualitative comparison. Moreover, these experiments do not test the paper's main theoretical predictions: they report accuracy and effective support size, but not the hysteresis, support-change transitions, or critical values of β\beta derived in Section 4.

Minor note: the maximal overlap zmax⁡∗z^*_{\max} shown in Figures 2-3 (and similarly in the appendix) is not normalized by the dimension dd properly, so it is not clear from the figures what value corresponds to perfect retrieval. This should be fixed such that perfect retrieval corresponds to 1, making the sparse and dense regimes easier to compare.

Questions for authors

(1) For the branches tracked in Figure 2, could the authors compute the predicted spinodal values β=2/λmax⁡(PGP)\beta = 2/\lambda_{\max}(PGP) from the actual Gram matrix and support of each branch, and overlay them on the observed jumps?

(2) Could the authors repeat the β\beta sweeps of Figure 2 for increasing dimensions (e.g., d=32,64,128d = 32, 64, 128) at fixed κ=log⁡M/d\kappa = \log M / d, and report how the hysteresis width, the number of coexisting branches, and the jump locations change with dd? This would indicate whether the transitions sharpen or disappear as dd grows.

(3) Could the authors measure retrieval of a specific target by initializing the dynamics from a distorted version of a chosen pattern and reporting the fraction of runs that converge to it (top-1 and top-5, for example) as a function of β\beta and κ\kappa, for both α=1\alpha = 1 and α=2\alpha = 2?

(4) Could the authors test for the predicted phenomena in an actual attention model, for example by iterating a trained attention layer for several steps using key and query representations extracted from a pretrained language model, and checking whether hysteresis under β\beta sweeps or support-change jumps appear?

8 comments
AuthorsOpenReview ↗

Rebuttal 1: transition data & support change before spinodal

W1. ...Moreover, a disordered statistical-mechanics analysis of dense associative memories at exponential load, with random patterns, a thermodynamic limit, a phase diagram, and exact retrieval thresholds, was carried out by Lucibello and Mézard (2024). It should be cited and discussed.

Thanks for pointing out the related references and we will cite and discuss them. To clarify, we go beyond their results by at least 2 points:

  • More ML-realistic: We also do not require the thermodynamic limit or random patterns as assumptions.
  • New perspective (stat-mech dual of Transformer attn), new techniques (spinodal, coexist...etc analysis) and new results (robust inference phase transition and characterization): Our analysis complements the prior work by studying branches and their stability, including the support changes in sparse attention. This allows us to characterize how model configurations (like α/β/κ\alpha/\beta/\kappa) determines robust attn forward pass. So we provide a new perspective allowing new migrations of stats-mech/physics machineries to frontier ML problems and leads to new understanding and useful guidances for practitioners (via α/β/κ\alpha/\beta/\kappa ...etc).

We will explain those distinctions in the Related Work.


W2. In addition, the critical values of β\beta derived in Section 4 are verified only on constructed examples (Appendix H.3) and are not compared with the transitions observed in the main numerical results. In fact, the authors note (Remark G.4) that most of the observed jumps come from changes in the support of the attention weights rather than from the spinodal mechanism of Proposition 4.2.

Thank you for your careful reading and raising this question.

  • “Not compared with the transitions observed in the main numerical results.”

    We agree our theory becomes stronger if we provide the transition data in the main-text experiments.
    In response, we conducted new exps with the requested transition data table in accordance to your Q1. We invite the reviewer to take a look.

    In our final version, we will include these theoretical spinodal points, observed spinodal points, and the support change events in real experiments. We hope this establishes better clarification and validation.

  • “Most of the observed jumps come from changes in the support of the attention weights rather than from the spinodal mechanism of Proposition 4.2.”

    In fact, most of the support change events occur before spinodal are expected. Specifically, we express the stationary as (1k−β2PG)p=1k1(\mathbb{1}_k-\frac{\beta}{2}PG)p=\frac{1}{k}\mathbf{1}, where 1\mathbf{1} is a vector full of 1. Then by projecting on a top eigenvector (PGP)v=\lambda_\max v, we have

    v⊤(p−1k1)=βv⊤PG1/2k1−βλmax⁡/2.v^\top(p-\frac{1}{k}\mathbf{1})=\frac{\beta v^\top PG\mathbf{1}/2k}{1-\beta\lambda_{\max}/2}.

    Hence, if the numerator is non-zero and pp in non-uniform, then as β\beta approaches to βjump=2/λmax⁡\beta_{\text{jump}} = 2/\lambda_{\max}, this would violate 0<p<10<p<1 and the simplex condition. In this case, the branch should encounter a support change event before spinodal.

    We will include this argument in the main text to support our theory. Please also see our response to W2 & Q5 of kcrx.

AuthorsOpenReview ↗

Rebuttal 2: stats-mech, thermodynamic limit, beyond bifurcations analysis, scaling

W3. Finally, since dd and MM are fixed and no scaling with dd is studied, the reported transitions are bifurcations of local minima rather than phase transitions in the statistical-mechanics sense, which correspond to non-analytic points of the free energy in the thermodynamic limit.

Sorry for any confusion caused. We respectfully disagree. Here are our point-to-point clarifications:

  • “The reported transitions are not phase transitions in the statistical-mechanics sense.”

    We clarify that our phase transition analysis follows the textbook-standard statis-mech treatments [1-4]. It invites abundant physics intuition, and goes beyond bifurcation analysis. Specifically, our use of “phase transition” considers the free energy minimization principle from a stats-mech point of view. Please also see our W1 response to KCRJ.

  • “It corresponds to non-analytic points of the free energy in the thermodynamic limit.”

    • Our results do not assume any thermodynamic limit nor zero-temperature assumption.

    • Please see our W1 & W2 responses to KCRJ for similar concerns.

    • Hence, our temperature-sweeps inherit statistical-mechanical phase structure in a finite-size, finite-temperature physical system. In particular, we interpret attention as an equation of motion from free energy minimization principle, and it guides the path toward a local minimum. This local minimum is meta-stable in the intermediate-β\beta regime in the sparse case, and leads to a superfreezing-like first-order phase transition with order parameter z^\star_\max.

  • “The reported transitions are bifurcations of local minima.”

    Our stats-mech perspective provides much more than just bifurcation analysis:

    • Free Energy. Our free energy assigns a value to each retrieval branch (Eq. (3.5)). This lets us compare their energetic preference, not just their local stability. We also evaluate coexistence points, where two competing branches have equal energy (Def 4.2 and Prop 4.1).
    • Stochasticity. The formulation extends to fluctuations in App E. It also brings statistical-mechanical tools to the study of Transformer attention, including phase analysis and connections to annealing and many-body methods [5,6]. This perspective can inform energy-based architectures as well [7,8].

    Please also see our W2 response to KCRJ for the same concern.

  • “dd and MM are fixed and no scaling with dd is studied.”

    We clarify that we also do the memory-load experiment with κ=log⁡(M)/d\kappa=\log(M)/d in Fig 3. Hence, we do study the scaling with number of memory patterns and their dimension.

If we misunderstand anything, please feel free to let us know.

for the rest of your concerns, we are working on the asked new numerical results. will post them shortly (in 1 or 2 days).

[1] Introduction to Modern Statistical Mechanics, David Chandler, 1987

[2] Statistical Physics of Particles, Mehran Kardar, 2007

[3] Statistical Mechanics 4th, R.K. Pathria, 2021

[4] Statistical Mechanics, Kerson Huang, 1987

[5] Dynamical Properties of Dense Associative Memory, Mimura et al., ICLR 2026

[6] Statistical Advantage of Softmax Attention: Insights from Single-Location Regression, Duranthon et al., ICLR 2026

[7] Large Associative Memory Problem in Neurobiology and Machine Learning, Krotov and Hopfield, ICLR 2021

[8] Energy-Based Transformers are Scalable Learners and Thinkers, Gladstone et al., ICLR 2026

AuthorsOpenReview ↗

Rebuttal 3: Q1-Q2

Q1. For the branches tracked in Figure 2, could the authors compute the predicted spinodal values β=2/λmax(PGP)\beta=2/\lambda_{\text{max}(PGP)} from the actual Gram matrix and support of each branch, and overlay them on the observed jumps?

Yes, we can, and thank you for your suggestion that could support our theory more. We examine this theoretical validation in the figures: https://imgur.com/a/auWRfpW, https://imgur.com/a/qbAWb1r. In our additional numerical test, we compare two events:

  • The theoretical points predicted by spinodal formula (Prop 4.2) with the observed spinodal points.
  • The theoretical predicted support change events with the observed support change events.

We provide the transition data tables below.

  • Spinodal Events (Increasing β\beta)
SeedPre-Jump Memory IndicesPredicted Spinodal β\betaObserved Spinodal β\beta Interval
0{61, 84}296.217027[296.164298, 296.258361]
1{89, 95}262.760030[262.677676, 262.771739]
2{18, 25}259.517861[259.479515, 259.573579]
3{76, 80}300.989950[300.961538, 301.055602]
4{32, 90}359.254134[359.186873, 359.280936]
5{2, 14}306.852161[306.793478, 306.887542]
6{3, 92}293.229475[293.154264, 293.248328]
6{3, 62}311.296323[311.214465, 311.308528]
7{7, 26}235.801222[235.775502, 235.869565]
8{5, 82}263.390362[263.336120, 263.430184]
9{50, 97}287.059903[287.040134, 287.134197]
  • Support-Change Jumps (Decreasing β\beta)
SeedSupport Size Before → AfterPredicted Support-Change β\betaObserved Jump β\beta Interval
01 → 2296.217027[296.164298, 296.258361]
11 → 2262.760030[262.677676, 262.771739]
21 → 2259.517861[259.479515, 259.573579]
31 → 2300.989950[300.961538, 301.055602]
41 → 2359.254134[359.186873, 359.280936]
51 → 2306.852161[306.793478, 306.887542]
63 → 4188.552655[188.461538, 188.555602]
61 → 2311.296323[311.214465, 311.308528]
71 → 2235.801222[235.775502, 235.869565]
81 → 2263.390362[263.336120, 263.430184]
91 → 2287.059903[287.040134, 287.134197]

This validates our spinodal formula (Prop 4.2) and the support-change events.


Q2. Could the authors repeat the β\beta sweeps of Figure 2 for increasing dimensions (e.g., d=32,64,128d=32,64,128) at fixed κ=log⁡(M)/d\kappa=\log(M)/d, and report how the hysteresis width, the number of coexisting branches, and the jump locations change with dd? This would indicate whether the transitions sharpen or disappear as dd grows.

We agree that the scaling dd experiment would show the complementary scaling behavior. We examine the scaling results in these figures: https://imgur.com/a/PEjy2Qd , https://imgur.com/a/I4mONM8. Please take a look. Specifically, we repeat the sweeps at d=32,64,128d = 32, 64, 128 with fixed κ=log⁡(100)/64\kappa = \log(100) / 64 and three seeds. The jumps persist across all experiments, and hysteresis appears at d=128d = 128. At β/d=4.5\beta/d = 4.5, we discover on average 8,21.3,258, 21.3, 25 stable states, respectively.

Reviewer_kcrxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies Transformer attention through modern Hopfield models using a free-energy view. Attention weights come from minimizing a free energy with a Tsallis entropy term, where α = 1 gives softmax, α = 2 gives sparsemax, and values in between give entmax. Keeping the stored patterns fixed, the authors track the local minima of this free energy as the inverse temperature β or the memory load κ = log M/d changes, and define coexistence points (two stable states with equal free energy) and spinodal points (a state losing stability). Simulations show that sparsemax retrieval has jumps and hysteresis while softmax changes smoothly, and that storing more memories lowers retrieval reliability in both. The appendix also trains a small one-layer cross-attention Transformer on synthetic data, where sparsemax works best for selecting one memory and softmax for averaging many.

Critical strengths
  • The question is relevant: when attention retrieval is reliable and when it suddenly switches. The main finding is clear and intuitive. Sparsity changes the behavior: sparsemax shows jumps and hysteresis, while softmax changes smoothly. Table 1 and the appendix tutorial make the physics accessible to an ML audience.
  • The closed-form thresholds for sparsemax are simple and interpretable (more similar keys need sharper attention to separate them), and they match numerical calculations very closely on two-key and general k-support systems.
Critical weaknesses
  • The novelty and depth of the theory are limited. Deriving softmax, entmax and sparsemax as stationary points of an entropy-regularized objective is already known and the paper itself notes that its free energy recovers Hu et al. exactly. The main analysis sets all fluctuations to zero, so the free energy becomes the deterministic energy, and the “phase transitions” are jumps between local minima of a function in finite dimension rather than thermodynamic transitions. The closed forms exist only for sparsemax under strong assumptions (uniform weights, equal correlation between all keys, support of 1 vs 2), with nothing for softmax. And the paper’s own experiments say most observed jumps come from support changes, which the theory does not predict.
  • The link to real Transformers is weak and the experiments are small. Real attention does a single lookup per layer with learned Q, K and V; it does not iterate a query to a fixed point, so it is unclear whether multiple stable states and hysteresis matter in practice. The Hopfield simulations are synthetic (M = 100, d = 48–64, clustered random patterns). The only trained model is a one-layer cross-attention network on synthetic data, which compares the three attention maps but does not test any phase transition, hysteresis or predicted threshold; its main conclusion, that sparse attention suits selection and dense attention suits averaging, is largely expected. There is no real data and no pretrained model.
Questions for authors
  • Does the hysteresis or multiple-stable-state behavior appear in a trained Transformer? For example, an up/down β sweep on the App. H model or a pretrained model, together with β_coexist and β_jump computed from the learned keys’ Gram matrix, would directly test the main claim.
  • In Fig. 17c (matched β), the aggregation error of sparsemax and entmax jumps sharply at high β. Is this one of the predicted transitions, and can the formulas predict where it happens? In Fig. 16c (frozen β), is the roughly 20× error increase at high β a phase transition or simply a mismatch with the training temperature?
  • In App. H, the models are trained with M ≤ 64 but tested up to M = 4096 (Figs. 14–15). Is the accuracy drop really a capacity–reliability trade-off, or out-of-distribution generalization? Training with larger M would separate the two.
  • Standard attention is a single step, not an iteration to a fixed point. How do the phase results transfer to single-step attention, or to energy-based Transformers that do iterate the update?
  • Most observed jumps come from support changes rather than spinodal points. Can the theory predict where support changes happen?
  • How do the results change for realistic keys with unequal correlations, and how sensitive is β_jump = 2/λ_max(PGP) to the Gram matrix structure?
  • The β values (100–500) and overlaps (about 1/d) come from the paper’s normalization. How do they map to the usual attention temperature 1/√d_k? Reporting the overlap relative to its 1/d maximum would make the plots easier to read.
  • How many seeds are used in App. H? The paper is hard to follow and has many typos (e.g., “dingle”, “overpal”, “fromulas”, “protocals”, “mimimum”, “Fencehl”). There are also notation clashes: θ denotes both the parameter vector and the sparsemax threshold, µ is both the memory index and the measure. A careful proofreading pass would help.
4 comments
AuthorsOpenReview ↗

Rebuttal 1: W1 - novelty & W2 - phase transitions

W1. The novelty and depth of the theory are limited.

Thank you for raising this. Only Prop 2.1 and Prop 3.3 come from prior work, and we cite them explicitly.

All other results (Thm 2.1, Prop 3.1, Thm 3.1, Prop 3.2, Prop 4.1, Prop 4.2) are new. These include the unified free-energy formulation and the stability analysis.

We discuss these differences in Related Work (lines 769-778):

  • Prior work on the Transformer-MHM correspondence [1,2,3] starts from the free energy and studies the retrieval rule. We derive the free energy from a Hamiltonian (Def 3.1). This gives the statistical-mechanical formulation in Sec 3 and its extension with fluctuations in App E.
  • The classical Hopfield analysis in [4] assumes infinite-dimensional, i.i.d. random patterns. Our phase analysis (Sec 4) and simulations (Sec 5) study finite-dimensional MHMs with clustered patterns.
  • [4] distinguishes phases of the classical Hopfield model. It does not identify additional transitions within the retrieval phase. We study a new discontinuous transition within this phase in the sparse regime (Figs 2-3). We discuss this further in our response to W2.

We invite the reviewer to have a look.

[1] Hopfield networks is all you need, Ramsauer et al., ICLR 2021

[2] On Sparse Modern Hopfield Model, NeurIPS’23

[3] From softmax to sparsemax: A sparse model of attention and multi-label classification, PMLR’16

[4] Storing infinite numbers of patterns in a spin-glass model of neural networks, Amit et al., 1985


W2. The “phase transitions” are jumps between local minima of a function in finite dimension rather than thermodynamic transitions. The closed forms exist only for sparsemax under strong assumptions (uniform weights, equal correlation between all keys, support of 1 vs 2), with nothing for softmax. And the paper’s own experiments say most observed jumps come from support changes, which the theory does not predict.

Sorry for any confusion caused. We believe there are some potential oversights. To clarify:

  1. The “phase transitions” are jumps between local minima of a function in finite dimension rather than thermodynamic transitions.

    We do NOT take a thermodynamic limit in our phase analysis [5]. Our setting is finite-dimensional. We study metastability and changes between retrieval states in this setting. They are, by def, phase transitions in stats mech.

    Please let us know if we misunderstand your concern about thermodynamic transitions. In particular, we are not sure what do you mean by "thermodynamic transitions."

  2. The closed forms exist only for sparsemax under strong assumptions (uniform weights, equal correlation between all keys, support of 1 vs 2), with nothing for softmax.

    The spinodal formula 2/λmax⁡(PGP)2/\lambda_{\max}(PGP) does NOT require uniform attention weights, equal pairwise correlations, or support size two. The formula βjump=2/(1−ϵ)\beta_{\mathrm{jump}}=2/(1-\epsilon) is the simplest example under the symmetry assumptions, not the general result.

    For softmax, Fig 2 shows only one retrieval state in the reported experiment. We therefore do not include a softmax spinodal formula. But it's certainly doable. We can derive and add it if the reviewer finds this useful.

  3. The paper's own experiments say most observed jumps come from support changes, which the theory does not predict.

    Our theory also identifies support changes. We give the calculation in below Q5. Under the conditions stated there, a branch encounters a support boundary before reaching the spinodal point.

Our results therefore cover finite-dimensional stability, including the sparsemax spinodal threshold and support boundaries. In this final version, we will use the additional page to explain these mechanisms in the main text. We hope this addresses your concerns.

[5] Dynamical Properties of Dense Associative Memory, Mimura et al., ICLR 2026

AuthorsOpenReview ↗

Rebuttal 2: W3 - fixed points, multiple stable states and hysteresis, Hopfield simulations

W3. Real attention does a single lookup per layer with learned Q, K and V; it does not iterate a query to a fixed point, so it is unclear whether multiple stable states and hysteresis matter in practice. The Hopfield simulations are synthetic (M = 100, d = 48–64, clustered random patterns). The only trained model is a one-layer cross-attention network on synthetic data, which compares the three attention maps but does not test any phase transition, hysteresis or predicted threshold; its main conclusion, that sparse attention suits selection and dense attention suits averaging, is largely expected. There is no real data and no pretrained model.

Thank you for raising these. Here are clarifications:

  1. Real attention does a single lookup per layer with learned Q, K and V; it does not iterate a query to a fixed point, so it is unclear whether multiple stable states and hysteresis matter in practice.

    Sorry for the confusion caused. Our setting does NOT iterate toward fixed points. We consider standard Transformer attention, with only one forward pass at inference.

    • To clarify, the attention-Hopfield correspondence links two objects:

      • (i) the fixed-point dynamics of the retrieval update and
      • (ii) local minima of the free-energy landscape of the statistical-mechanical system.
    • Further, [1, Thm 5] and [2, Corollary 3.1.2] show, theoretically and numerically, that a single update can approximate a retrieval fixed point under realistic settings. Our analysis takes this as its premise, matching the practical attention setting you described.

    • So our main results do not require iterative retrieval updates (i.e., looped attention inference). We focus on retrieval/inference robustness as these local minima evolve.

    • Through this correspondence (fixed points <-> local minima),

      • the hysteresis of local minima determines the robustness of Transformer inference.
      • In particular, Fig 2 shows that opposite β\beta-sweeps select distinct local minima in the intermediate-β\beta regime. This indicates multiple competing local minima.
      • Moreover, through the Transformer-Hopfield correspondence, multiple distinct fixed points involving relevant and irrelevant keys.
      • Thus, we call this a glassy-inference phase of Transformer attention (glass = involvement of irrelevant keys/representations). Together, our phase analysis characterizes the robustness (regimes and conditions) of Transformer inference.
    • To further clarify, we used gradient descent to locate local minima on the energy landscape (Sec G.2, Experimental Settings). These gradient-descent trajectories are not the iterative-update trajectories in the Transformer framework. We will clarify this in the final version.

  2. (Hopfield simulations) The only trained model is a one-layer cross-attention network on synthetic data, which compares the three attention maps but does not test any phase transition, hysteresis or predicted threshold...

    • The mentioned (Hopfield simulations) (App H.1 & H.2) aims to test how sparsity affects Transformer performance on tasks.

    • Hence, this is a different question from our phase analysis idea in the main text.

    • Our phase transition and hysteresis analysis are about Transformer inference robustness.

    • We test these phenomena (phase transition, hysteresis or predicted threshold...etc) in Sec 5, and validate the threshold formulas in App H.3.

    We invite the reviewer to have a look and hope this clarify your confusion.

  3. its main conclusion, that sparse attention suits selection and dense attention suits averaging, is largely expected.

    • It is true - the selection-versus-aggregation result is expected. But here, it's an illustration of consequences (that, the derived insights and consequences align with practice and known facts). We will revise the practical insights wording to make this clear.

We hope these clarifications suffice. We will make highlight them more clearly in our final version. thank you again for pointing these out!

[1] Hopfield Networks is All You Need, Hubert Ramsauer et al., ICLR 2021.

[2] On Sparse Modern Hopfield Model, Jerry Yao-Chieh Hu et al., NeurIPS 2023.

AuthorsOpenReview ↗

Rebuttal 3: Q1-3

Q1. Does the hysteresis or multiple-stable-state behavior appear in a trained Transformer? For example, an up/down β sweep on the App. H model or a pretrained model, together with β_coexist and β_jump computed from the learned keys’ Gram matrix, would directly test the main claim.

No. What we characterize here are the "regimes (or conditions/thresholds)" of robust inference. These regimes are defined by varying model configurations (like β/α/κ\beta/\alpha/\kappa). They do not change during training. So it's out of our scope - we are not studying training regimes but configuration regimes.

So our experiments address two related questions:

  1. Robust attn inference phases. The main-text simulations track retrieval branches (i.e., local minima) as the inverse-temperature β\beta and memory load κ\kappa vary.
  2. Learned attention (Hopfield simulations). App H.1 and H.2 use learned queries, keys, and values to study how sparsity affects selective retrieval and aggregation.

So the suggested " β_coexist and β_jump" test is not really applicable in our considered scenario.

Yet, since you mentioned, we have ongoing work developing a separate Hamiltonian formulation for evolving keys at a fixed query and for joint query-key dynamics. This requires substantial additional analysis. So we decided to leave them to another project and remains outside the present work.

We will clarify above scope and the role of App H in the final version. We hope this addresses your question.


Q2. In Fig. 17c (matched β), the aggregation error of sparsemax and entmax jumps sharply at high β. Is this one of the predicted transitions, and can the formulas predict where it happens? In Fig. 16c (frozen β), is the roughly 20× error increase at high β a phase transition or simply a mismatch with the training temperature?

Thanks for the question. To clarify, just like clarified above, our results do not identify these error increases as the predicted transitions. Our characterizations are about configuration regimes, instead of training regimes:

  1. Fig 17c (matched β\beta). Each point comes from a separately trained model (with difference configuration), so the learned keys change across the curve. Our fixed-memory spinodal formulas do not directly predict this curve.

  2. Fig 16c (frozen β\beta). The model is trained at β=1\beta=1. Increasing β\beta at inference concentrates attention and creates a mismatch with the aggregation task. The error increase alone does not demonstrate a phase transition.

We will make these distinctions explicit in the final version. Thank you for looking closely at these experiments.


Q3. In App. H, the models are trained with M ≤ 64 but tested up to M = 4096 (Figs. 14–15). Is the accuracy drop really a capacity–reliability trade-off, or out-of-distribution generalization? Training with larger M would separate the two.

Thank you for pointing this out. Training and testing at matched larger MM helps separate these effects. We examined this with below new experiments: https://imgur.com/a/K6xRzM6

For selective retrieval, training directly at M=4096M=4096 does not learn useful retrieval within the original 2500 steps. We therefore make two copies of each model trained with M≤64M\leq64 and give each copy 500 additional training steps:

  1. One continues with M≤64M\leq64.
  2. The other uses M=4096M=4096.

At test M=4096M=4096, training at the larger load improves accuracy for all three attention types and all three seeds. Accuracy still decreases as MM grows, consistent with the observed capacity-reliability trade-off. But the improvement also shows that the training-test mismatch matters.

We will distinguish this generalization effect in our discussion of the experiments. Thank you for the suggestion.

AuthorsOpenReview ↗

Rebuttal 4: Q4-Q8

Q4. Standard attention is a single step, not an iteration to a fixed point. How do the phase results transfer to single-step attention, or to energy-based Transformers that do iterate the update?

Thank you for raising this. By above discussion, there are two settings to distinguish:

  1. Single-step attention

    Sorry for the confusion caused. As explained in above W3, our consideration is the single step standard transformer attention. So our phase results ask how local minima, and hence Transformer inference robustness changes with β/α/κ\beta/\alpha/\kappa.

  2. Energy-based Transformers

    The phase analysis applies only when the energy-based models satisfy the same coincidence between the updated fixed points and the local minima on energy landscape. An energy-minimization principle alone does not establish that the same stability result holds for a different update. We deem exploring our framework on energy transformers an interesting future direction.


Q5. Most observed jumps come from support changes rather than spinodal points. Can the theory predict where support changes happen?

Yes. We characterized the support changes by

  1. Where the support changes (Sec D.7 or line 1495 of the submitted version)

    For a sparsemax branch with active support SS of size kk, the threshold is

    θ=1k(∑μ∈Szμ−2β).\theta=\frac{1}{k}\left(\sum_{\mu\in S}z_\mu-\frac{2}{\beta}\right).

    A support boundary occurs when zμ=θz_\mu=\theta. This is a different condition from βjump=2/λmax⁡(PGP)\beta_{\mathrm{jump}}=2/\lambda_{\max}(PGP), so the two values need not coincide.

  2. Why a support boundary can occur first (will be included into the final version)

    We write the stationary equation as

    (1k−β2PG)p=1k1,(\mathbb{1}_k-\frac{\beta}{2}PG)p=\frac{1}{k}\mathbf{1},

    where 1\mathbf{1} is the all-ones vector. Projecting onto a top eigenvector (PGP)v=λmax⁡v(PGP)v=\lambda_{\max}v gives

    v⊤(p−1k1)=βv⊤PG1/2k1−βλmax⁡/2.v^\top(p-\frac{1}{k}\mathbf{1})=\frac{\beta v^\top PG\mathbf{1}/2k}{1-\beta\lambda_{\max}/2}.

    If the numerator is non-zero, the right-hand side diverges as β\beta approaches 2/λmax⁡2/\lambda_{\max}. But a probability vector on the simplex stays bounded. A feasible branch continued toward this value must therefore encounter a support boundary first.

We will use the additional page to include this argument in the main text and distinguish it from the spinodal condition. Thank you for raising this question.


Q6. How do the results change for realistic keys with unequal correlations, and how sensitive is β_jump = 2/λ_max(PGP) to the Gram matrix structure?

Our results already covered unequal correlations. Specifically, Prop 4.2 allows unequal key correlations and nonuniform weights.

As for the sensitivity, recall the threshold is

βjump=2λmax⁡(PGP).\beta_{\mathrm{jump}}=\frac{2}{\lambda_{\max}(PGP)}.

For a fixed support, a perturbation δG\delta G changes the projected Gram matrix by PδGPP\delta GP. Thus the threshold is insensitive when this perturbation is small relative to the positive eigenvalue λmax⁡(PGP)\lambda_{\max}(PGP). It becomes more sensitive when that eigenvalue is small.

This comparison assumes the support stays fixed. A support change requires evaluating the corresponding branch separately. We hope this clarifies the dependence on the Gram matrix.


Q7. The β values (100–500) and overlaps (about 1/d) come from the paper’s normalization. How do they map to the usual attention temperature 1/√d_k? Reporting the overlap relative to its 1/d maximum would make the plots easier to read.

Our overlap is zμ=x⊤ξμ/dz_\mu=x^\top\xi_\mu/\sqrt{d}. Thus the logit βzμ\beta z_\mu has the usual scaled dot-product form when d=dkd=d_k and β=1\beta=1, using the same vector scaling.

We will state the normalization explicitly and report overlap relative to its 1/d1/d maximum to make the plots easier to read. Thank you for the suggestion.


Q8. How many seeds are used in App. H? The paper is hard to follow and has many typos (e.g., “dingle”, “overpal”, “fromulas”, “protocals”, “mimimum”, “Fencehl”). There are also notation clashes: θ denotes both the parameter vector and the sparsemax threshold, µ is both the memory index and the measure. A careful proofreading pass would help.

We use 5 seeds for Figs 14-16 and three for Fig 17. In the final version, we will state these counts, perform 3 more rounds of proofreadings, correct the typos, and separate the notation for the parameter vector, sparsemax threshold, memory index, and measure as suggested.

thank you again for attention to the details!


Thanks again for the review. We hope our responses clarify your questions and give a clearer picture of our contributions. We will incorporate these clarifications into the paper and update the PDF in the next few days. Please let us know if anything remains unclear. We would be happy to continue the discussion!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Rebbutal Response

Dear Reviewers and Program Chairs,

We thank all reviewers for the careful reviews and follow-up discussions. The comments helped us sharpen both the main contribution and the scope of our paper. In response to the reviewer's suggestions, we clarify terminologies, correct the missing normalization factor, and add numerical data table in our revision. The modified parts are highlighted in blue for your convenience.

We summarize the key contributions, reviewer-specific clarifications, and major revisions below.


Summary of Contributions

The main contribution is not only the established Hopfield-attention correspondence or the recovery of existing attention maps. Rather, we use the free-energy landscape to ask a different question: how stable attention-inference configurations compete, change support, and lose stability as the attention parameters vary. In particular:

  • Statistical-Mechanical Formulation.

    • From a Hamiltonian, we interpret the connection between entropy and free-energy landscape from the first principle, recovering the attention family as an equation of motion.
    • This enables meta-stability, phase structure, and stochastic analysis.
  • Phase Analysis for Attention Inference.

    • We distinguish three different criticalities and mechanisms: (1) energetic coexistence (2) spinodal (3) support change.
    • We discuss them in Sec 4. Importantly, energy crossing, loss of stability, support change, and an actual jump need not occur at the same parameter value.
  • Finite-Dimensional, Structured-Keys Phase Analysis.

    • Our formulation and theory consider the fixed, finite-dimensional, structured keys, not i.i.d. random keys.
    • This allows the stability characterization of a concrete attention-inference instance.
    • We validate the criticalities and mechanisms in Sec 5, Sec F and Sec G.4 numerically.

Distinguishment from Prior Work and Reviewer-Specific Clarifications

  • To reviewer kcrx, we clarify that our phase analysis considers single-step attention in Transformers, not iterative Hopfield retrieval. We also make it clear by referring to the correspondence between (i) local minima on free-energy landscape and (ii) fixed-points of retrieval updates based on Ramsauer’s and Hu’s works.

  • To reviewer 88rq, we distinguish our transition from thermodynamic criticality of an exact finite-system Gibbs free energy. The latter is analytic at finite size and develops singularities through a thermodynamic limit. In contrast, our object is the local stable branches. Even when branches are smooth, minimization over competing branches can produce a non-differentiable lower envelope. Hence, we study finite-dimensional variational phase transitions rather than thermodynamic-limit criticality.

  • To reviewer KCRJ, we discuss the distinction between the phase transitions in our work, and the phase transitions from thermal fluctuation and activation. Importantly, we emphasize that we apply a statistical-mechanics approach with a single β\beta throughout the derivation, rather than introduce an additional bath temperature.

  • To reviewer X3M8, we clarify and correct the mathematical details. The global Fenchel conjugate free energy in Sec 2 is convex, while Eq. (3.5) is non-convex due to the minimization over weights pp. We also correct the missing escort normalization Cα[J]C_\alpha[J] in Thm 2.1. Thanks for the detailed review!

Common Concern: Only a Bifurcation Analysis?

No. Without a statistical-mechanics framework, one would not know why to use regularizer as entropy, how temperature shapes the free-energy landscape, and where stochasticity arises from. Additionally, our free-energy formulation allows identifying coexistence, metastability, energy preference, and phase structure via the free-energy minimization principle. We hope this resolves the reviewer’s concerns.

AuthorsOpenReview ↗

Revision Details

Major revisions include:


  • Clarification of robust/glassy inference phase.

    • line 458-465 Define robust/glassy inference phases in Sec 4.

    • line 476-489 State which regimes of Fig 2 and 3 correspond to glassy and robuste-inference phases.

  • Better distinction between Transformer experiments and the main text.

    • line 3212-3216 State the purpose of Transformer experiments explicitly at the beginning of the section: test how parameters affect Transformer performance and support on tasks.
  • Correction of normalization factor.

    • line 1375-1416 Include missing normalization factor Cα[J]C_\alpha[J] in the proof.

    • line 209-212 Revise Thm 2.1.

  • Clarification of non-convexity.

    • line 313-318 Explain why the minimization may be non-convex after Rmk 3.1.
  • Include spinodal verification with a data table.

    • line 3740-3751 Include a data table in Appendix to compare (1) theoretical spinodal points (Prop 4.2), and (2) observed spinodal points of Fig 2.

    • line 3751-3760 Include a data table in Appendix to compare (1) predicted support-change points, and (2) observed support-change points of Fig 2.

  • Include the derivation of why support-change events mainly occur before spinodal points.

    • line 1751-1801 Include the derivation of when support-change events occur before spinodal points.

    • line 425-431 Include the results in Sec 4.

  • Related work.

    • line 787-799 Include the discussion of the paper: Petrova et al. (2026).
    • line 800-806 Include the discussion of the paper: Lucibello & Mézard (2024).

We hope these modifications make our draft better. Thank you again for your constructive suggestions and attention to details!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.