acceptodds
Under review as a conference paper at ICLR 2027

When Is Softmax Attention a Gradient? Exact Characterization and Conditioning

Abstract

Energy and potential representations provide structural explanations for the fixed points and dynamics of tied or specially structured attention maps. Fixed-memory softmax attention can be a gradient even when its keys and effective values are untied, but this depends on the additive geometry of the keys. For , we give a finite necessary-and-sufficient criterion for a global potential by grouping equal key-pair sums. Generic memories force a common scaling and translation of the keys; binary Cartesian memories admit exactly independent coordinate scalings. We characterize constant-metric convex representations and derive two-sided finite-query bounds on the distance to the conservative value space from measured Jacobian asymmetry, with explicit conditioning and observation-error dependence. A near-Cartesian family shows why conditioning is essential: curl tends to zero while both the parameter distance and the best field approximation within the same-key conservative class remain bounded away from zero. Deterministic numerical checks validate the exact characterization and finite-query certificates, and illustrate the predicted loss of conditioning near pair-sum collisions. Finally, an analytic bifurcation construction shows that arbitrarily small incompatible perturbations can produce oscillation near a reference map's stability boundary. These results distinguish exact compatibility, reliable approximate certification, and dynamical stability.

Primary area:
learning theory
Keywords:
Softmax Attention, Conservative Vector Fields, Memory Geometry, Finite-Query Certification, Numerical Conditioning, Dynamical Stability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3pKDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Final Justification after rebuttal.

Thank you for the detailed response. My observations about Hopfield reparameterizations still hold, but the rebuttal clarifies the distinct value of the characterization

broader practical benefits and the relevance of exact Cartesian geometry remain open concerns. Nevertheless, the clarified theoretical significance, together with the conditioning results, sufficiently addresses my main concern

therefore I will reassess my score


This paper studies when a fixed-memory softmax attention map F(q)=U⊤softmax⁡(βKq)F(q) = U^\top \operatorname{softmax}(\beta Kq) admits a global scalar potential. In particular, the authors claim (lines 12–14) that fixed-memory softmax attention with untied keys and effective values can be a gradient, depending on the additive geometry of the keys (e.g., binary Cartesian memories in Theorem 3).

In other words, the authors extend the analysis beyond the tied key–value assumption considered in Ramsauer et al., 2021.

To characterize this condition, the authors derive a finite, query-independent criterion Ba(K,U)=0B_a(K,U)=0 for all a∈AKa\in\mathcal A_K. Beyond the Euclidean gradient setting, the authors consider a constant metric H=H⊤≻0H=H^\top\succ0 with F=H−1∇ΦF=H^{-1}\nabla\Phi, where Φ∈C2(Rd)\Phi\in C^2(\mathbb R^d) is convex, and provide necessary and sufficient conditions for such a representation under generic key configurations.

The authors further provide finite-query bounds on the distance to the conservative value space CK=ker⁡LK\mathcal C_K=\ker\mathcal L_K, depending on the conditioning of the measurement operator. However, through a near-Cartesian construction (i.e., not exactly Cartesian), they show that even when the Jacobian asymmetry approaches zero (i.e., lim⁡ϵ→0+sup⁡∣∣DFϵ(q)−DFϵ(q)⊤∣∣F=O(ϵ)→0\lim_{\epsilon\to0^+}\sup ||DF_\epsilon(q)-DF_\epsilon(q)^\top||_F=\mathcal O(\epsilon)\to0), the distance to the same-key Euclidean conservative class can remain bounded away from zero (e.g., approaching 2\sqrt{2} in the case of Eq. (20)).


In short, the contributions as I understand them are:

  • Provided a finite, query-independent criterion Ba(K,U)=0B_a(K,U)=0 for characterizing conservative softmax attention.
  • Characterized how the geometry of the keys determines the conservative value space, including generic restrictions and exceptions for structured memories.
  • Provided finite-query quantitative bounds on the distance to the conservative class and demonstrated their dependence on conditioning.
Critical strengths

1. Well-Motivated Problem Setup

The paper addresses a fundamental question "when can untied softmax attention admit a global potential?" I find the problem well-motivated.


2. Interesting Characterization

Theorem 1

Theorem 1 provides an elegant, query-independent criterion for the existence of a global potential, reducing Jacobian symmetry to finite algebraic conditions Ba(K,U)=0B_a(K,U)=0.

Theorem 3

Theorem 3 identifies nontrivial conservative solutions for Cartesian memories, allowing coordinatewise scaling

U=Kdiag⁡(a)+1b⊤.U=K\operatorname{diag}(a)+\mathbf1b^\top.

This highlights the role of key geometry in gradient compatibility. In particular, the theorem provides a complete characterization of conservative effective values for binary Cartesian memories, rather than merely giving a sufficient condition.

Theorem 5 & 6

Theorem 5 extends the exact characterization to approximate compatibility by providing finite-query bounds on the distance to the conservative value space. In particular, it establishes finite-query identifiability and provides quantitative bounds that account for the conditioning of the measurement operator and observation errors.

Theorem 6 provides an explicit near-Cartesian construction showing that small Jacobian asymmetry does not necessarily imply proximity to the same-key Euclidean conservative class. The result is further strengthened by showing that the discrepancy can persist at the level of the attention maps themselves, rather than only in the parameter space.

Critical weaknesses

1. Limited Scientific Insight

Corollary 2

In Corollary 2, conservative attention requires U=αK+1b⊤U=\alpha K + \boldsymbol 1b^\top. Let K~=U=αK+1b⊤\widetilde K=U=\alpha K+\mathbf1b^\top and β~=βα\widetilde\beta=\frac{\beta}{\alpha}. Then for α>0\alpha>0:

β~K~q=βα(αK+1b⊤)q=βKq+βα(b⊤q)1.\widetilde\beta\widetilde Kq=\frac{\beta}{\alpha}(\alpha K+\mathbf1b^\top)q=\beta Kq+\frac{\beta}{\alpha}(b^\top q)\mathbf1.

Since softmax⁡(z+c1)=softmax⁡(z)\operatorname{softmax}(z+c\mathbf1)=\operatorname{softmax}(z), we have:

softmax⁡(β~K~q)=softmax⁡(βKq),\operatorname{softmax}(\widetilde\beta\widetilde Kq)=\operatorname{softmax}(\beta Kq),

which yields:

F(q)=U⊤softmax⁡(βKq)=K~⊤softmax⁡(β~K~q).F(q)=U^\top\operatorname{softmax}(\beta Kq)=\boxed{\widetilde K^\top\operatorname{softmax}(\widetilde\beta\widetilde Kq)}.

This result yields exactly the same form as the already-known tied-attention formulation (where the key and value spaces coincide; Ramsauer et al., 2021). Therefore, at first glance, Corollary 2’s practical implication seems unclear to me.

Theorem 4

In Theorem 4 (and lines 86–88), the authors introduce H=H⊤≻0H=H^\top \succ 0 for F=H−1∇ΦF=H^{-1}\nabla\Phi. In my understanding, the motivation for considering H≠IH\ne I is to find a meaningful interpretation beyond Euclidean space (i.e., by defining ⟨v1,v2⟩H\langle v_1,v_2\rangle_H). However, only the inner product space changes, and the conclusion in the transformed space remains the same.

For simplicity, I will ignore PcP_c, since it is just a centering matrix (i.e., the following derivation assumes U=AU=A and K=BK=B in line 191). The conditions presented in Theorem 4 are:

  • F=H−1∇ΦF=H^{-1}\nabla\Phi, for some H=H⊤≻0H=H^\top\succ 0 with convex Φ∈C2\Phi\in C^2.
  • H=H⊤≻0,withUH=KH=H^\top\succ0,\quad\text{with}\quad UH=K.

Then, the attention map F(q)=U⊤softmax⁡(βKq)F(q)=U^\top\operatorname{softmax}(\beta Kq) becomes:

F(q)=U⊤softmax⁡(βKq)=(KH−1)⊤softmax⁡(βKq)=H−1K⊤softmax⁡(βKq).F(q)=U^\top\operatorname{softmax}(\beta Kq)=(KH^{-1})^\top\operatorname{softmax}(\beta Kq)=\boxed{H^{-1}K^\top\operatorname{softmax}(\beta Kq)}.

Therefore, in the HH-metric space,

HF(q)=K⊤softmax⁡(βKq),HF(q)=K^\top\operatorname{softmax}(\beta Kq),

which reduces to the conventional tied attention formulation.

While I appreciate the mathematical rigor of these characterizations, I remain unconvinced that the paper sufficiently establishes their scientific significance beyond the formal classification itself.


2. Limited Practical Implications

Theorem 3

At first glance, I found Theorem 3 particularly interesting, as it identifies non-trivial conservative solutions beyond the conventional tied key–value formulation. However, upon closer consideration, I am not sure how meaningful this result is beyond its mathematical characterization.

In particular, the additional flexibility is restricted to coordinate-wise scaling under an exact binary Cartesian key geometry (i.e., U=Kdiag⁡(a1,…,ad)+1b⊤U=K\operatorname{diag}(a_1,\dots,a_d)+\boldsymbol1b^\top). Thus, the additional freedom seems to amount to simple rescaling along the coordinate axes. Moreover, when all scaling factors are strictly positive, this can be interpreted as an SPD coordinate transformation, under which the attention map reduces to the conventional tied Hopfield formulation.

Beyond its mathematical characterization, its practical and conceptual implications remain unclear to me. For example, can such an exact Cartesian structure naturally emerge or be induced during training? Even if such a structure is obtained, what additional behavior or insight does it provide beyond the existing Hopfield interpretation?

Questions for authors

My main concern, as outlined in the Weaknesses section, is that the characterizations are mathematically nontrivial, but the resulting compatible dynamics appear to be either conventional tied Hopfield updates or their reparameterizations. Consequently, I remain uncertain about the additional scientific insight provided by these results beyond the characterization itself.

Could the authors clarify whether I have misunderstood any aspect of the results or their implications? In particular, if the exact characterization itself constitutes a substantial theoretical contribution, I would appreciate a clearer explanation of its significance, including what new understanding it provides beyond existing Hopfield formulations.

I am open to the possibility that I have underestimated the value of the characterization. A convincing clarification of its significance in the rebuttal could positively affect my assessment, and I would be willing to reconsider my score accordingly.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3pKD

We thank the reviewer for the careful and constructive evaluation of our work, and for recognizing the motivation and mathematical contributions of the characterization. We particularly appreciate the insightful observations regarding its relationship to existing Hopfield formulations and the questions concerning its broader scientific significance. These comments help us identify where the distinction between known sufficient constructions and our complete compatibility characterization should be made clearer. We address the reviewer's concerns in detail below.

Response to Comment 1: Scientific Significance

Corollary 2: Necessity versus sufficiency. For α>0\alpha>0, the reviewer's reparameterization indeed recovers the tied Hopfield update of Ramsauer et al. (2021). The known construction establishes sufficiency, whereas Corollary 2 establishes necessity. For fixed generic KK, every Euclidean-conservative UU must satisfy

U=αK+1b⊤.U=\alpha K+\mathbf{1}b^\top.

Common scaling and translation therefore exhaust the admissible value freedom. This gives an exact compatibility test for a specified map and a closed-form minimum Frobenius-norm effective-value correction to the same-key convex-gradient class (Eq. 15).

Theorem 4: Metric existence versus metric reparameterization. We agree that the coordinate equivalence holds, including centering. Writing A=PcUA=P_cU and B=PcKB=P_cK, the condition AH=BAH=B implies UH=K+1b⊤UH=K+\mathbf{1}b^\top. With x=H1/2qx=H^{1/2}q and K~=UH1/2\widetilde{K}=UH^{1/2}, the transformed iteration is tied attention because the common logit shift cancels.

The substantive question, however, is whether such an SPD metric exists for a prescribed (K,U)(K,U). It cannot always be constructed. For example, under the generic-key assumptions and for a nonconstant map, equivalently PcU≠0P_cU\ne0, if ki≠kjk_i\ne k_j but ui=uju_i=u_j, subtracting the corresponding rows of UH=K+1b⊤UH=K+\mathbf{1}b^\top yields a contradiction. When the certificate succeeds, it also guarantees energy descent for the original iteration (Eq. 13). Thus, our result complements existing constructions using SPD outer projections (Santos et al., 2025) by providing a compatibility test for specified keys and values.

Together, these results make the gradient interpretation of a prescribed attention map testable. Memory geometry determines the admissible conservative values and their sensitivity to perturbations (Theorems 1, 3, and 6). Metric certificates determine when a convex-gradient interpretation can be recovered, while Proposition 7 exhibits oscillatory dynamics outside the constant-SPD gradient class arbitrarily close to a critical tied reference. The scientific insight is therefore a connection between memory geometry, recoverable gradient structure, and the dynamical limits of the resulting energy interpretation.

Response to Comment 2: Practical and Conceptual Implications

We agree that strictly positive coordinate scales in the standard Cartesian family yield a tied Hopfield update after a change of coordinates. Exact key geometry can be enforced through a structured parameterization of the full sign grid or its invertible affine images (Appendix A.5). The explicit full-grid construction contains 2d2^d keys. The paper does not establish that such geometry emerges under unconstrained training.

Together, Theorems 3 and 6 establish how exact additive relations among the keys govern conservative value freedoms and their sensitivity to perturbations. In the binary square, the two diagonal pairs share the same key-pair sum, allowing their skew contributions to cancel for every attainable softmax query. This permits unequal coordinate scales that generic keys do not admit.

Theorem 6 demonstrates what happens when this relation is broken. Moving one corner removes the pair-sum collision while retaining the construction Uϵ=Kϵdiag⁡(1,2)U_{\epsilon}=K_{\epsilon}\operatorname{diag}(1,2). For fixed β,R>0\beta,R>0, as ϵ↓0\epsilon\downarrow0, Jacobian asymmetry vanishes uniformly on [−R,R]2[-R,R]^2, yet the minimum Frobenius-norm same-key Euclidean repair approaches 2\sqrt{2}. The best conservative attention map with the perturbed keys also has a strictly positive limiting L2(μR)L^2(\mu_R) approximation error, where μR\mu_R is uniform probability measure on that square.

Both the exact and perturbed maps admit a convex-gradient representation in the same constant metric H=diag⁡(1,1/2)H=\operatorname{diag}(1,1/2), and the state transformation x=H1/2qx=H^{1/2}q yields a tied Hopfield update for each. The comparison establishes that a Hopfield representation and arbitrarily small Jacobian asymmetry can coexist with a nonvanishing same-key Euclidean repair cost. The Cartesian family thus provides a controlled example of how key geometry governs exact compatibility and why near-symmetry alone cannot certify a small effective-value repair with the original keys and query coordinates fixed.

AuthorsOpenReview ↗

Response to Reviewer 3pKD

Response to Questions

Thank you for the opportunity to clarify the scientific significance. A concrete use is to determine whether a trained attention denoiser's residual is the gradient of an explicit regularizer in the original pixel coordinates. When such a regularizer exists, it can be combined with a data-fidelity term to define an image reconstruction objective.

First, checking whether the prescribed update admits a scalar objective. Theorem 1 gives a finite necessary-and-sufficient test through Ba(K,U)=0B_a(K,U)=0 for whether q−F(q)q-F(q) is the Euclidean gradient of a scalar objective. As explained in Comment 1, Corollary 2 makes this test explicit under its generic-key assumptions. Values outside the family U=αK+1b⊤U=\alpha K+\mathbf{1}b^\top rule out every scalar objective with this prescribed gradient. This determines whether the given iteration can be interpreted as q+=q−∇R(q)q^+=q-\nabla R(q) in the original coordinates. A Hopfield representation after a metric change does not by itself establish this Euclidean objective interpretation.

Second, quantifying the correction and assessing whether observations suffice. Fixing the keys, temperature, and query coordinates preserves the attention-weight function. Projection onto CK=ker⁡LK\mathcal{C}_K=\ker\mathcal{L}_K gives the smallest effective-value correction in Frobenius norm needed for Euclidean conservativeness. Corollary 2 and Theorem 3 give the admissible conservative values for generic and Cartesian memories. When only Jacobian observations are available and the keys are known, Theorem 5 gives distance bounds under its identifiability or validated-certificate conditions, with explicit dependence on conditioning and observation errors.

Theorem 6 shows why a small observed asymmetry alone is insufficient. Its near-collision family has Jacobian asymmetry tending uniformly to zero on fixed compact squares, while both the minimum same-key repair distance and the best same-key conservative field-approximation error remain bounded away from zero. A constant-SPD convex-gradient representation remains available throughout this family. Thus, neither small observed skew nor metric recoverability alone justifies a small Euclidean repair. The quantitative results specify the coverage and conditioning checks needed to support that inference.

Third, checking the conditions for dynamical guarantees. Under its generic-key and nonconstant-map assumptions, Theorem 4 tests whether a constant positive-definite metric supplies a convex-potential representation and the associated energy-descent guarantee for the original iteration. Transferring convergence via Corollary 11 additionally requires its positive contraction margin, derivative-defect bound, and regularity and invariant-domain assumptions. Proposition 7 demonstrates why structural proximity alone cannot replace these conditions near criticality. Its softmax family undergoes a supercritical Neimark–Sacker bifurcation arbitrarily close to a critical tied reference; the nonreal fixed-point multipliers also exclude a constant-SPD gradient representation.

Application of theory to image denoising. We trained a single-head, 128-slot attention denoiser on 8×88\times 8 MNIST with additive Gaussian noise (σ=0.2\sigma=0.2). Across five paired seeds, we evaluated both approaches on the same 2,000 held-out images.

First, we tested whether a trained denoiser could provide an explicit regularizer with ∇R(q)=q−D(q)\nabla R(q)=q-D(q) while retaining its attention weights. All five trained unrestricted denoisers had generic keys, and their effective values were outside the conservative family characterized by Corollary 2. We froze the learned keys and temperature and refit the effective values by least squares over the full conservative family U=αK+1b⊤U=\alpha K+\mathbf{1}b^\top. Test MSE increased from 0.008730 to 0.010129. A matched unrestricted refit with the same frozen keys and fitting data achieved 0.007919.

Second, we learned K,α,bK,\alpha,b jointly from scratch under the same constraint. This guaranteed an explicit regularizer throughout training and achieved a test MSE of 0.008686, compared with 0.008730 for unrestricted training under the same budget.

The positive-scale models admit the tied Hopfield reparameterization noted by the reviewer. Corollary 2 gives the complete set of compatible values for the original generic keys, which lets us quantify the cost of preserving the learned attention weights. The joint-training result shows that, when the keys can adapt to this constraint, the model can provide an exact regularizer while retaining denoising quality in this setting.

We will clarify the scientific significance of the exact characterization, strengthen the discussion of its relationship to existing Hopfield formulations, and explain the implications of memory geometry, same-key repair, and metric compatibility more explicitly in the manuscript.

Reviewer_cGjMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the conditions under which a fixed-memory Softmax attention mechanism can be expressed as the gradient of a scalar potential function. For a mapping of the form F(q)=U⊤softmax⁡(βKq)F(q) = U^\top \operatorname{softmax}(\beta Kq), the authors leverage the symmetry of the Jacobian, the exponential structure of the Softmax function, and the linear independence of exponential functions to transform the condition of continuous symmetry into a finite set of algebraic constraints grouped by key-pair sums. Based on this, they characterize the space of conservative mappings for both "general-position" memory and binary Cartesian memory. Furthermore, the paper examines representations via convex potential functions under a constant positive-definite metric, derives certificates for approximate conservativeness given a finite set of queries, and analyzes the degradation of the condition number near "near-collision" scenarios. Finally, the authors employ a Neimark–Sacker bifurcation analysis to characterize oscillatory behavior as the system approaches the stability boundary.

Critical strengths

The paper translates the query-dependent Jacobian symmetry condition into a finite set of necessary and sufficient algebraic constraints. This framework applies to any fixed set of keys (including scenarios where key-pair sums collide) and offers a clear, broadly applicable theoretical formulation.

The authors demonstrate that, for keys in a general position, "effective values" typically require a uniform scaling and translation structure, whereas binary Cartesian memories allow for coordinate-wise independent scaling. These findings shed light on why specific memory structures can support a richer variety of gradient fields.

Furthermore, the paper analyzes the identifiability and condition numbers associated with constant metric representations and approximate compatibility. It also distinguishes between the structure of potential functions and dynamical stability, cautioning against the oversimplified equation of gradient structures with iterative convergence.

The mathematical proofs in the paper are correct.

Critical weaknesses
  1. The foundation of this paper lies in the classical equivalence between Jacobian symmetry and conservative vector fields. Existing literature, such as Smart et al. (2025), has already derived the query-dependent symmetry conditions for Softmax attention. Consequently, the primary contribution of this work is the refinement of these conditions into a finite algebraic criterion based on groupings of (ki+kj)(k_i + k_j), which allows for a precise classification across various memory geometries.

However, the proof of the main theorem relies primarily on covariance expansion, the grouping of exponential terms, and the linear independence of distinct exponential functions. While this derivation is both correct and useful, the technical approach is relatively straightforward; the paper does not fully articulate the extent to which this represents a significant theoretical breakthrough beyond the existing query-dependent conditions. In particular, given that the link between the finite algebraic characterization and the classical linear independence of exponential functions is quite direct, the authors need to clarify why this result constitutes a theoretically valuable contribution in its own right, rather than merely an algebraic reformulation of established criteria.

  1. Theorem 5 provides a distance certificate based on a finite number of query points. It states that, under assumptions such as independent sampling of query points, the distribution is absolutely continuous with respect to the Lebesgue measure of the ambient space and the satisfaction of certain rank conditions, the common kernel of the space of conservative values ​​can be recovered almost surely. In practical models, however, query points may lie near a low-dimensional manifold or be heavily constrained by data distribution, normalization, and network architecture; consequently, these sampling assumptions may not necessarily hold.

While the characterization based on finite algebras is theoretically precise, the construction process requires enumerating key-pair sums and establishing corresponding linear constraints. In the worst case, involves O(P2)O(P^2) such pair sums. For large-scale memory systems, the explicit construction of the constraint matrix and the computation of its null space can entail significant computational and storage costs.

  1. The analysis of Neimark–Sacker bifurcations suggests that proximity to a convex-gradient reference map is insufficient to guarantee dynamical stability near the stability boundary. However, this conclusion is derived from a specific family of low-dimensional maps and cannot be directly generalized to arbitrary attention architectures. Furthermore, the model-based feedback and synthetic retrieval experiments presented in the paper primarily serve to illustrate the scope of the theory; they do not yet fully demonstrate the practical benefits of using finite-algebraic characterizations for the analysis, stability diagnosis, or structural design of large-scale models.
Questions for authors
  • Beyond existing query-dependent symmetry conditions, what is the core theoretical contribution of this paper—one that cannot be dismissed as a mere algebraic expansion of those existing conditions?
  • Could you provide a more systematic set of experiments regarding computational complexity or scale, and explain the practical advantages of this method compared to directly checking the symmetry of the Jacobian?
3 comments
AuthorsOpenReview ↗

Response to Reviewer cGjM

We thank the reviewer for the careful and constructive assessment of our work, and for recognizing the correctness of the mathematical results and the value of the finite algebraic characterization. We appreciate the concerns regarding theoretical novelty, sampling assumptions, computational scalability, and practical relevance. We address these points below with further theoretical clarification and additional experimental evidence.

W1. Theoretical contribution beyond query-dependent symmetry. Smart et al. (2025, Appendix I.2) derive a query-dependent symmetry condition for softmax attention. Our contribution is to characterize, for a fixed key memory, the entire space of effective values UU that yield a global Euclidean potential. Theorem 1 gives an exact finite necessary-and-sufficient criterion for the actual softmax query family. Antisymmetric contributions from pairs sharing the same sum ki+kjk_i+k_j can cancel, and each distinct pair-sum group must vanish separately.

Solving these constraints yields two exact classifications. For distinct, noncollinear keys with distinct pair sums, Corollary 2 proves that conservativeness is equivalent to U=αK+1b⊤U=\alpha K+\mathbf{1}b^{\top}, giving a d+1d+1-dimensional space. For a full binary Cartesian memory, Theorem 3 characterizes the entire space as U=Kdiag⁡(a1,…,ad)+1b⊤U=K\operatorname{diag}(a_1,\ldots,a_d)+\mathbf{1}b^{\top}, giving 2d2d dimensions. The four-key square exhibits the mechanism. Its diagonals share a pair sum, allowing their antisymmetric contributions to cancel with unequal coordinate scales. In the perturbed-square family of Theorem 6, moving one corner by an arbitrarily small positive displacement breaks this equality and reduces the conservative value space from four dimensions to three.

W2. Sampling assumptions and computational scalability. The almost-sure recovery statement in Theorem 5 assumes independent probes with laws absolutely continuous in the ambient space and the stated sufficient probe count. For a fixed probe set, its deterministic certificate gives two-sided distance bounds when the reconstruction condition ∥ΠK−JAN∥2<1\lVert \Pi_K-JA_N\rVert_2 < 1 and the observation-error bound are verified. Here ΠK\Pi_K projects onto the orthogonal complement of the conservative value space determined by the known keys. This provides a check for constrained query sets, with the resulting precision controlled by reconstruction conditioning and observation error. The algebraic characterization in Theorem 1 requires no sampling assumption.

The general construction processes (P2)\binom{P}{2} key pairs and entails additional constraint-assembly and null-space costs. Once the key geometry is established, Corollary 2 and Theorem 3 provide explicit conservative families that permit direct verification and projection. Verifying the generic-key assumptions can still require enumerating the key pairs. Our experiments on constrained queries and the separate P,d,NP,d,N scale sweeps are reported in Q2 below.

W3. Scope of the bifurcation result and practical diagnosis. Proposition 7 gives a four-key counterexample to a stability guarantee based solely on proximity to a convex-gradient reference near criticality. Arbitrarily small nonresonant rotations of the effective values yield a supercritical Neimark–Sacker bifurcation at β=2\beta=2, where the tied reference has zero linear stability margin at the origin.

To quantify the intervention required to impose Euclidean conservativeness on trained maps, we computed the minimum same-key repair in 72 frozen GPT-2 cases with returns linearized at the anchor. The median repair was 99.4% of the centered effective-value norm. For dynamical guarantees, Corollary 11 compares the contraction margin mm of a same-key convex-gradient reference with a uniform derivative-defect bound δ\delta obtained from the key and residual-value diameters. On a closed convex domain invariant under the iterated map, δ<m\delta < m guarantees convergence to a unique fixed point.

For the imposed GPT-2 recurrence with nonlinear feedback, we evaluated the full return Jacobian, including LayerNorm and the query and output projections. All 72 original feedback anchors were locally stable at g=1g=1, and both the original and tied-control Jacobians had nonreal eigenvalues in every case. For structural design, tied values V=KV=K together with a symmetric positive-definite return C=LL⊤C=LL^{\top} guarantee energy descent in our synthetic retrieval study for each fixed episode memory. All three variants converged on the tested episodes, and each had lower mean long-horizon accuracy than its initial readout. We also used Corollary 2 to train a fixed-memory MNIST denoiser with an explicit regularizer and comparable test MSE under the same training budget (see Application of theory to image denoising in our response to Reviewer 3pKD).

AuthorsOpenReview ↗

Response to Reviewer cGjM

Q1. Core theoretical contribution. For a prescribed fixed-memory attention map, our results determine whether a potential interpretation is justified and what modification it requires. The finite criterion tests Euclidean gradient compatibility, while the generic and Cartesian classifications explain how memory geometry restricts the admissible values (Theorem 1, Corollary 2, and Theorem 3). This enables a researcher to verify an energy assumption and compute the minimum effective-value correction while preserving the attention-weight function. Under the assumptions of Theorem 4, constructive certificates additionally determine whether the map admits a convex potential in a constant positive-definite metric without modifying its values.

The quantitative results determine whether finite Jacobian observations justify a small-repair conclusion, accounting for probe coverage, conditioning, and observation error (Theorem 5). Theorem 6 shows why small asymmetry alone is insufficient. It can tend to zero while both the same-key repair distance and the best same-key conservative field-approximation error remain bounded away from zero. For convergence, Corollary 11 provides a sufficient check by comparing a uniform derivative defect with a positive contraction margin under its regularity and invariant-domain assumptions. Proposition 7 identifies the limit of a proximity-only argument through oscillations arbitrarily close to a convex-gradient reference near its stability boundary.

Q2. Comparison with direct Jacobian checks and scaling. We evaluated 30 controlled cases, comparing direct symmetry screening with validated repair-distance bounds from the same sampled Jacobian skews and error allowances. Twenty-one cases passed reconstruction validation.

With an additional RMS observation error allowance of 10−610^{-6}, direct screening ruled out exact conservativeness for two value choices on the same generic memory. The distance certificate distinguished a small same-key repair, approximately [2.64,3.65]×10−4[2.64,3.65]\times10^{-4}, from a much larger one, approximately [3.14236,3.14246][3.14236,3.14246]. These bounds quantify the change required to impose conservativeness.

Query coverage also affected the conclusions. In a four-key example with repair distance 2, all 25 probes on the line q1=0q_1=0 missed the nonconservative component. With 25 probes and an additional RMS error allowance of 10−610^{-6}, narrowing the strip from ∣q1∣≤10−4|q_1|\le10^{-4} to ∣q1∣≤10−6|q_1|\le10^{-6} widened the certified interval from approximately [1.96,2.04][1.96,2.04] to [0,5.93][0,5.93].

We varied P,d,NP,d,N separately. At d=2d=2 and N=144N=144, increasing PP from 8 to 128 increased the number of nonzero constraint coefficients from 112 to 32,512. At P=128P=128, the 144 scalar observations fell short of the target rank of 253. At P=16P=16 and N=64N=64, increasing dd from 2 to 16 increased the number of nonzero coefficients from 480 to 57,600. In the probe-count sweep with target rank 29, N=9N=9 and N=25N=25 were insufficient, and N=36N=36 and N=100N=100 passed reconstruction validation. The (P,d)=(100,64)(P,d)=(100,64) case reached the certificate-construction budget, and the (P,d)=(512,64)(P,d)=(512,64) case reached the pair-grouping budget. The benchmark used analytic conservative spaces after geometry verification. The additional certificate setup quantifies the minimum repair and can be reused for fixed keys and a fixed probe operator.

We will incorporate the additional experiments and scaling results into the manuscript, clarify the theoretical contributions relative to prior work, and make the sampling assumptions, computational limitations, and scope of the stability analysis more explicit.

Thanks to the authors for the rebuttal, which resolved most of my questions. I am maintaining my score and recommending acceptance.

Reviewer_2QxNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the conditions under which softmax attention is equal to the gradient of a function (the "potential"). This is of interest, because Ramsauer et al 2021 derived a special form softmax attention as gradient w.r.t. queries and argued the forward pass was minimizing the potential, similar to Modern Hopfield Networks and Dense-Associative Memories. Hoover et al 2023, Energy Transformer (ET), does a more general version for the full transformer, more on it later. The paper uses smart and simple idea: if F(q)=softmax(Kq)=∇ϕF(q)=\mathrm{softmax}(K q) = \nabla \phi then its Jacobian DF(q)DF(q) will be a Hessian and therefore needs to be symmetric (using ideas from Im et al 2016). Their main contributions, in my opinion, are the condition Ba(K,U)=0B_a(K, U)=0 from eq 3 and Theorem 1. They also derive the potential Φ=∑iailog⁡cosh⁡(qi)+bTq\Phi = \sum_i a_i \log \cosh(q_i) + b^T q for "cartesian memory" (keys being on a binary hypercube), and derive similar conditions for a more general form F(q)=H−1∇ϕF(q) = H^{-1} \nabla \phi. Finally, they derive approximate conditions for when Ba(K,U)B_a(K,U) is not exactly zero ("Near collision").

Critical strengths
  1. The idea of Jacobian symmetry to check for gradients in attention is such a simple and nice method. to my knowledge it was missing from classic works like Ramsauer 2021 and Hoover 2023.
  2. The "cartesian memory" case could be useful for quantized models, or maybe BitNet.
  3. The simple integral method eq 6 for deriving the potential is practical.
  4. the metric case with HH could become a basis for further generalization (see questions).
Critical weaknesses

Not critical flaws, but:

  1. The authors could have used real trained models (even a tiny one trained on shakespeare on a laptop for a few minutes) to check whether in real models attention is close to being a gradient or if some version of the near-collisions is satified.
  2. The paper spends a substantial part of the work on the "cartesian memory" scenario, with P=2dP=2^d (all possible vectors), which is very hard to relate to a practical case, even for quantized models or BitNets. Instead devoting some efforts to empirical studies could have made the paper much more interesting to the community.
Questions for authors
  1. Theorem 3, when all aia_i are the same, it becomes a special case of the logsumexp potential from Ramsauer 2021, right?
  2. I am curious how your results would generalize to the energy transformer (ET) (Hoover 2023). The Layernorm provides one layer of extra complexity, but even if we ignore it, the main difference between ET and Ramsauer is that in ET the derivative hit both keys and queries, with F(g)j=−∇gjϕ(g),ϕ(g)=−∑j∈Mlog⁡∑i∉Mexp⁡[giTWQTWLKgj].F(g)_j = -\nabla_{{g_j}} \phi(g), \qquad \phi(g) = -\sum_{j\in \mathcal{M}} \log \sum_{i\not\in \mathcal{M}} \exp [g_i^T W_Q^TW_LK g_j].
  3. Regarding HH: a followup to ET, called NRGPT (ICLR 2026) uses attention F(q)i=−Π∇giEi(q)F(q)_i = - \Pi \nabla_{g_i} E_i(q). Their per-token potential complicates things, but ignoring that part, they showed their model learned better with an unconstrained preconditioner Π\Pi. Also, to get descent on the potential, only the symmetric part of Π\Pi needs to be positive def, i.e. Π=H+(A−AT)\Pi = H+ (A-A^T) with pos def HH and arbitrary AA. So I think instead of the strict F=H−1∇ϕF= H^{-1} \nabla \phi, if you use F=Π∇ϕF=\Pi \nabla \phi with pos def sym(Π)\mathrm{sym}(\Pi) a much larger class of softmax attentions could be effectively "descending" on the potential. Note that for the Associative memory picture we only need attention to be an effective descent, not a strict gradient.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2QxN

We thank the reviewer for the thoughtful and constructive assessment of our work, particularly for recognizing the value of the exact characterization. We also appreciate the suggestions on empirical validation using trained models and the connections to broader energy-based formulations. We address the concerns and questions individually below.

W1. How close is pretrained attention to a gradient?

We analyzed 72 cases from GPT-2 small, using 12 selected heads and six fixed 100-token contexts. For each case, we froze the keys and values and linearized the return map from the head output to the query at the anchor. The resulting query-coordinate map FF has effective values UU and the same Jacobian at the anchor query q0q_0 as the nonlinear frozen-head feedback map.

The relative Jacobian asymmetry,

∥DF(q0)−DF(q0)⊤∥F∥DF(q0)∥F,\frac{\lVert DF(q_0)-DF(q_0)^\top\rVert _F}{\lVert DF(q_0)\rVert _F},

had a median of 1.374 and ranged from 1.145 to 1.423. All 72 key sets satisfied the generic-position conditions of Corollary 2. We therefore computed the minimum relative effective-value change required for exact Euclidean gradient compatibility with the same keys as

ρ(K,U)=min⁡α∈R,b∈Rdh∥U−αK−1b⊤∥F∥U−1uˉ⊤∥F,\rho(K,U)= \frac{ \min_{\alpha\in\mathbb R,\\ b\in\mathbb R^{d_h}} \lVert U-\alpha K-\mathbf1b^\top\rVert _F }{ \lVert U-\mathbf1\bar u^\top\rVert _F },

where uˉ\bar u is the mean effective-value vector. The median was 99.40%, with a minimum of 90.30%. These maps exhibit substantial local Jacobian asymmetry and require large effective-value changes to become exact Euclidean gradients with the same keys.

All key-pair sums were distinct. The smallest pair-sum separation across the cases was 1.27% of the corresponding key diameter. The effect of approximate collisions on gradient-compatibility diagnostics in these cases remains unresolved.

W2. What does the Cartesian example tell us about practical memories?

The full 2d2^d-vertex cube imposes a restrictive memory geometry. It provides a completely solvable structured case showing how pair-sum collisions relax generic rigidity. The mechanism already appears in the four-key square, whose diagonals have the same pair sum and allow cancellations in the antisymmetric Jacobian terms. This construction identifies how memory geometry changes the effective-value structures admitting a potential. The GPT-2 analysis in W1 applies the general compatibility criterion to trained attention with generic key sets.

AuthorsOpenReview ↗

Response to Reviewer 2QxN

Q1. Equal scales and the Hopfield potential.

Yes. For the full binary cube,

Z(q)=∏j2cosh⁡(βqj).Z(q)=\prod_j 2\cosh(\beta q_j).

When all aj=aa_j=a, Eq. (10) reduces to aβ−1log⁡Z(q)+b⊤qa\beta^{-1}\log Z(q)+b^\top q, up to an additive constant. Setting a=1a=1 and b=0b=0 recovers the log-sum-exp potential underlying the tied Hopfield update of Ramsauer et al. (2021).

Theorem 3 provides a complete classification in the original key and query coordinates. Every conservative effective-value matrix has the form

U=Kdiag⁡(a1,…,ad)+1b⊤,U=K\operatorname{diag}(a_1,\ldots,a_d)+\mathbf1b^\top,

allowing independent coordinatewise scaling. For β>0\beta>0, the resulting potential is convex if and only if all aj≥0a_j\geq0.

Q2. Relation to Energy Transformer.

Setting LayerNorm aside, let AA denote the bilinear score matrix in the reviewer's masked attention energy, and let sijs_{ij} be the softmax weight of unmasked key token ii for masked query jj. Differentiation gives

−∇gjϕ=∑i∉MsijA⊤gi(j∈M),-\nabla_{g_j}\phi =\sum_{i\notin\mathcal M}s_{ij}A^\top g_i \qquad (j\in\mathcal M),

−∇giϕ=∑j∈MsijAgj(i∉M).-\nabla_{g_i}\phi =\sum_{j\in\mathcal M}s_{ij}Ag_j \qquad (i\notin\mathcal M).

Freezing the unmasked tokens gives the tied case of our framework, K⊤softmax⁡(Kgj)K^\top\operatorname{softmax}(Kg_j), with fixed effective keys ki=A⊤gik_i=A^\top g_i. When both groups follow the joint negative-gradient update, the second expression supplies the key-side contribution. In ET's full attention energy, a token can contribute through both roles, and the joint attention field in gg-coordinates is conservative by construction (Hoover et al., 2023).

Our classification asks when a given fixed-memory attention map with independently specified effective values admits a potential. Extending it to general joint updates with evolving keys requires symmetry of the full joint Jacobian, including its cross-token blocks. A finite characterization must also handle state-dependent keys, since Theorem 1 groups fixed key-pair sums.

Q3. Nonsymmetric preconditioners and energy descent.

For a fixed constant preconditioner Π\Pi with sym⁡(Π)≻0\operatorname{sym}(\Pi)\succ0, Theorem 1 gives

F=Π∇Φ⟺Ba(K,UΠ−⊤)=0for every key-pair sum a.F=\Pi\nabla\Phi \quad\Longleftrightarrow\quad B_a(K,U\Pi^{-\top})=0 \quad\text{for every key-pair sum }a.

The normalization-free descent calculation discussed in NRGPT (Dehmamy et al., 2026, Proposition 2.2) applies to the flow q˙=−Π∇E(q)\dot q=-\Pi\nabla E(q), for which

E˙=−∇E(q)⊤sym⁡(Π)∇E(q)≤0.\dot E =-\nabla E(q)^\top\operatorname{sym}(\Pi)\nabla E(q) \leq0.

For the discrete residual update q+=q−ηΠ∇E(q)q^+=q-\eta\Pi\nabla E(q), a descent guarantee additionally requires control of the step size relative to the smoothness of EE.

Section 4 establishes a one-step descent bound for q+=F(q)q^+=F(q) when F=H−1∇ΦF=H^{-1}\nabla\Phi, with constant H=H⊤≻0H=H^\top\succ0 and convex Φ\Phi. Defining EH(q)=12q⊤Hq−Φ(q)E_H(q)=\frac12q^\top Hq-\Phi(q), Eq. (13) gives

EH(F(q))−EH(q)≤−12∥F(q)−q∥H2.E_H(F(q))-E_H(q) \leq-\frac12\lVert F(q)-q\rVert _H^2.

Our four-key construction shows that the broader preconditioned representation permits different discrete dynamics. At θ=0.6\theta=0.6,

Fβ,θ=Rθ∇Φβ,sym⁡(Rθ)=cos⁡(θ)I≻0,F_{\beta,\theta}=R_\theta\nabla\Phi_\beta, \qquad \operatorname{sym}(R_\theta)=\cos(\theta)I\succ0,

with convex Φβ\Phi_\beta. Proposition 7 establishes a supercritical Neimark–Sacker bifurcation at β=2\beta=2, yielding an attracting invariant closed curve without fixed points for β>2\beta>2 sufficiently close to 22. Any continuous energy attains a minimum on this compact curve. The next iterate remains on the curve, so strict decrease at every nonfixed point is impossible. Trajectories on the curve also cannot converge to a fixed point.

We will incorporate the GPT-2 compatibility results into the manuscript, condense the Cartesian discussion around Theorem 3 and the four-key example, and clarify the connections to ET and nonsymmetric preconditioning.

Thank you for the new derivations for energy transformer and NRGPT. I also like the elegant way the preconditioned descent and Newton's update generalize the results. Of course, GPT-2 being far away from gradient update is important and warrants future studies. Or maybe the preconditioned version of the results change the GPT-2 results. I am satisfied with your answers and further argue for the paper to be accepted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.