- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
Final Justification after rebuttal.
Thank you for the detailed response. My observations about Hopfield reparameterizations still hold, but the rebuttal clarifies the distinct value of the characterization
broader practical benefits and the relevance of exact Cartesian geometry remain open concerns. Nevertheless, the clarified theoretical significance, together with the conditioning results, sufficiently addresses my main concern
therefore I will reassess my score
This paper studies when a fixed-memory softmax attention map admits a global scalar potential. In particular, the authors claim (lines 12–14) that fixed-memory softmax attention with untied keys and effective values can be a gradient, depending on the additive geometry of the keys (e.g., binary Cartesian memories in Theorem 3).
In other words, the authors extend the analysis beyond the tied key–value assumption considered in Ramsauer et al., 2021.
To characterize this condition, the authors derive a finite, query-independent criterion for all . Beyond the Euclidean gradient setting, the authors consider a constant metric with , where is convex, and provide necessary and sufficient conditions for such a representation under generic key configurations.
The authors further provide finite-query bounds on the distance to the conservative value space , depending on the conditioning of the measurement operator. However, through a near-Cartesian construction (i.e., not exactly Cartesian), they show that even when the Jacobian asymmetry approaches zero (i.e., ), the distance to the same-key Euclidean conservative class can remain bounded away from zero (e.g., approaching in the case of Eq. (20)).
In short, the contributions as I understand them are:
- Provided a finite, query-independent criterion for characterizing conservative softmax attention.
- Characterized how the geometry of the keys determines the conservative value space, including generic restrictions and exceptions for structured memories.
- Provided finite-query quantitative bounds on the distance to the conservative class and demonstrated their dependence on conditioning.
Critical strengths
1. Well-Motivated Problem Setup
The paper addresses a fundamental question "when can untied softmax attention admit a global potential?" I find the problem well-motivated.
2. Interesting Characterization
Theorem 1
Theorem 1 provides an elegant, query-independent criterion for the existence of a global potential, reducing Jacobian symmetry to finite algebraic conditions .
Theorem 3
Theorem 3 identifies nontrivial conservative solutions for Cartesian memories, allowing coordinatewise scaling
This highlights the role of key geometry in gradient compatibility. In particular, the theorem provides a complete characterization of conservative effective values for binary Cartesian memories, rather than merely giving a sufficient condition.
Theorem 5 & 6
Theorem 5 extends the exact characterization to approximate compatibility by providing finite-query bounds on the distance to the conservative value space. In particular, it establishes finite-query identifiability and provides quantitative bounds that account for the conditioning of the measurement operator and observation errors.
Theorem 6 provides an explicit near-Cartesian construction showing that small Jacobian asymmetry does not necessarily imply proximity to the same-key Euclidean conservative class. The result is further strengthened by showing that the discrepancy can persist at the level of the attention maps themselves, rather than only in the parameter space.
Critical weaknesses
1. Limited Scientific Insight
Corollary 2
In Corollary 2, conservative attention requires . Let and . Then for :
Since , we have:
which yields:
This result yields exactly the same form as the already-known tied-attention formulation (where the key and value spaces coincide; Ramsauer et al., 2021). Therefore, at first glance, Corollary 2’s practical implication seems unclear to me.
Theorem 4
In Theorem 4 (and lines 86–88), the authors introduce for . In my understanding, the motivation for considering is to find a meaningful interpretation beyond Euclidean space (i.e., by defining ). However, only the inner product space changes, and the conclusion in the transformed space remains the same.
For simplicity, I will ignore , since it is just a centering matrix (i.e., the following derivation assumes and in line 191). The conditions presented in Theorem 4 are:
- , for some with convex .
- .
Then, the attention map becomes:
Therefore, in the -metric space,
which reduces to the conventional tied attention formulation.
While I appreciate the mathematical rigor of these characterizations, I remain unconvinced that the paper sufficiently establishes their scientific significance beyond the formal classification itself.
2. Limited Practical Implications
Theorem 3
At first glance, I found Theorem 3 particularly interesting, as it identifies non-trivial conservative solutions beyond the conventional tied key–value formulation. However, upon closer consideration, I am not sure how meaningful this result is beyond its mathematical characterization.
In particular, the additional flexibility is restricted to coordinate-wise scaling under an exact binary Cartesian key geometry (i.e., ). Thus, the additional freedom seems to amount to simple rescaling along the coordinate axes. Moreover, when all scaling factors are strictly positive, this can be interpreted as an SPD coordinate transformation, under which the attention map reduces to the conventional tied Hopfield formulation.
Beyond its mathematical characterization, its practical and conceptual implications remain unclear to me. For example, can such an exact Cartesian structure naturally emerge or be induced during training? Even if such a structure is obtained, what additional behavior or insight does it provide beyond the existing Hopfield interpretation?
Questions for authors
My main concern, as outlined in the Weaknesses section, is that the characterizations are mathematically nontrivial, but the resulting compatible dynamics appear to be either conventional tied Hopfield updates or their reparameterizations. Consequently, I remain uncertain about the additional scientific insight provided by these results beyond the characterization itself.
Could the authors clarify whether I have misunderstood any aspect of the results or their implications? In particular, if the exact characterization itself constitutes a substantial theoretical contribution, I would appreciate a clearer explanation of its significance, including what new understanding it provides beyond existing Hopfield formulations.
I am open to the possibility that I have underestimated the value of the characterization. A convincing clarification of its significance in the rebuttal could positively affect my assessment, and I would be willing to reconsider my score accordingly.
2 comments
Response to Reviewer 3pKD
Response to Questions
Thank you for the opportunity to clarify the scientific significance. A concrete use is to determine whether a trained attention denoiser's residual is the gradient of an explicit regularizer in the original pixel coordinates. When such a regularizer exists, it can be combined with a data-fidelity term to define an image reconstruction objective.
First, checking whether the prescribed update admits a scalar objective. Theorem 1 gives a finite necessary-and-sufficient test through for whether is the Euclidean gradient of a scalar objective. As explained in Comment 1, Corollary 2 makes this test explicit under its generic-key assumptions. Values outside the family rule out every scalar objective with this prescribed gradient. This determines whether the given iteration can be interpreted as in the original coordinates. A Hopfield representation after a metric change does not by itself establish this Euclidean objective interpretation.
Second, quantifying the correction and assessing whether observations suffice. Fixing the keys, temperature, and query coordinates preserves the attention-weight function. Projection onto gives the smallest effective-value correction in Frobenius norm needed for Euclidean conservativeness. Corollary 2 and Theorem 3 give the admissible conservative values for generic and Cartesian memories. When only Jacobian observations are available and the keys are known, Theorem 5 gives distance bounds under its identifiability or validated-certificate conditions, with explicit dependence on conditioning and observation errors.
Theorem 6 shows why a small observed asymmetry alone is insufficient. Its near-collision family has Jacobian asymmetry tending uniformly to zero on fixed compact squares, while both the minimum same-key repair distance and the best same-key conservative field-approximation error remain bounded away from zero. A constant-SPD convex-gradient representation remains available throughout this family. Thus, neither small observed skew nor metric recoverability alone justifies a small Euclidean repair. The quantitative results specify the coverage and conditioning checks needed to support that inference.
Third, checking the conditions for dynamical guarantees. Under its generic-key and nonconstant-map assumptions, Theorem 4 tests whether a constant positive-definite metric supplies a convex-potential representation and the associated energy-descent guarantee for the original iteration. Transferring convergence via Corollary 11 additionally requires its positive contraction margin, derivative-defect bound, and regularity and invariant-domain assumptions. Proposition 7 demonstrates why structural proximity alone cannot replace these conditions near criticality. Its softmax family undergoes a supercritical Neimark–Sacker bifurcation arbitrarily close to a critical tied reference; the nonreal fixed-point multipliers also exclude a constant-SPD gradient representation.
Application of theory to image denoising. We trained a single-head, 128-slot attention denoiser on MNIST with additive Gaussian noise (). Across five paired seeds, we evaluated both approaches on the same 2,000 held-out images.
First, we tested whether a trained denoiser could provide an explicit regularizer with while retaining its attention weights. All five trained unrestricted denoisers had generic keys, and their effective values were outside the conservative family characterized by Corollary 2. We froze the learned keys and temperature and refit the effective values by least squares over the full conservative family . Test MSE increased from 0.008730 to 0.010129. A matched unrestricted refit with the same frozen keys and fitting data achieved 0.007919.
Second, we learned jointly from scratch under the same constraint. This guaranteed an explicit regularizer throughout training and achieved a test MSE of 0.008686, compared with 0.008730 for unrestricted training under the same budget.
The positive-scale models admit the tied Hopfield reparameterization noted by the reviewer. Corollary 2 gives the complete set of compatible values for the original generic keys, which lets us quantify the cost of preserving the learned attention weights. The joint-training result shows that, when the keys can adapt to this constraint, the model can provide an exact regularizer while retaining denoising quality in this setting.
We will clarify the scientific significance of the exact characterization, strengthen the discussion of its relationship to existing Hopfield formulations, and explain the implications of memory geometry, same-key repair, and metric compatibility more explicitly in the manuscript.
Response to Reviewer 3pKD
We thank the reviewer for the careful and constructive evaluation of our work, and for recognizing the motivation and mathematical contributions of the characterization. We particularly appreciate the insightful observations regarding its relationship to existing Hopfield formulations and the questions concerning its broader scientific significance. These comments help us identify where the distinction between known sufficient constructions and our complete compatibility characterization should be made clearer. We address the reviewer's concerns in detail below.
Response to Comment 1: Scientific Significance
Corollary 2: Necessity versus sufficiency. For α>0, the reviewer's reparameterization indeed recovers the tied Hopfield update of Ramsauer et al. (2021). The known construction establishes sufficiency, whereas Corollary 2 establishes necessity. For fixed generic K, every Euclidean-conservative U must satisfy
U=αK+1b⊤.
Common scaling and translation therefore exhaust the admissible value freedom. This gives an exact compatibility test for a specified map and a closed-form minimum Frobenius-norm effective-value correction to the same-key convex-gradient class (Eq. 15).
Theorem 4: Metric existence versus metric reparameterization. We agree that the coordinate equivalence holds, including centering. Writing A=PcU and B=PcK, the condition AH=B implies UH=K+1b⊤. With x=H1/2q and K=UH1/2, the transformed iteration is tied attention because the common logit shift cancels.
The substantive question, however, is whether such an SPD metric exists for a prescribed (K,U). It cannot always be constructed. For example, under the generic-key assumptions and for a nonconstant map, equivalently PcU=0, if ki=kj but ui=uj, subtracting the corresponding rows of UH=K+1b⊤ yields a contradiction. When the certificate succeeds, it also guarantees energy descent for the original iteration (Eq. 13). Thus, our result complements existing constructions using SPD outer projections (Santos et al., 2025) by providing a compatibility test for specified keys and values.
Together, these results make the gradient interpretation of a prescribed attention map testable. Memory geometry determines the admissible conservative values and their sensitivity to perturbations (Theorems 1, 3, and 6). Metric certificates determine when a convex-gradient interpretation can be recovered, while Proposition 7 exhibits oscillatory dynamics outside the constant-SPD gradient class arbitrarily close to a critical tied reference. The scientific insight is therefore a connection between memory geometry, recoverable gradient structure, and the dynamical limits of the resulting energy interpretation.
Response to Comment 2: Practical and Conceptual Implications
We agree that strictly positive coordinate scales in the standard Cartesian family yield a tied Hopfield update after a change of coordinates. Exact key geometry can be enforced through a structured parameterization of the full sign grid or its invertible affine images (Appendix A.5). The explicit full-grid construction contains 2d keys. The paper does not establish that such geometry emerges under unconstrained training.
Together, Theorems 3 and 6 establish how exact additive relations among the keys govern conservative value freedoms and their sensitivity to perturbations. In the binary square, the two diagonal pairs share the same key-pair sum, allowing their skew contributions to cancel for every attainable softmax query. This permits unequal coordinate scales that generic keys do not admit.
Theorem 6 demonstrates what happens when this relation is broken. Moving one corner removes the pair-sum collision while retaining the construction Uϵ=Kϵdiag(1,2). For fixed β,R>0, as ϵ↓0, Jacobian asymmetry vanishes uniformly on [−R,R]2, yet the minimum Frobenius-norm same-key Euclidean repair approaches 2. The best conservative attention map with the perturbed keys also has a strictly positive limiting L2(μR) approximation error, where μR is uniform probability measure on that square.
Both the exact and perturbed maps admit a convex-gradient representation in the same constant metric H=diag(1,1/2), and the state transformation x=H1/2q yields a tied Hopfield update for each. The comparison establishes that a Hopfield representation and arbitrarily small Jacobian asymmetry can coexist with a nonvanishing same-key Euclidean repair cost. The Cartesian family thus provides a controlled example of how key geometry governs exact compatibility and why near-symmetry alone cannot certify a small effective-value repair with the original keys and query coordinates fixed.