- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies repulsive mean-field self-attention on the sphere with symmetric positive-definite interactions. Assuming uniform density and Sobolev bounds, it proves an energy-gap estimate. It then studies simultaneous token evolution and parameter training, and derives a learning-rate schedule and a bound on the average squared training-loss gradient.
Critical strengths
This is the first paper I have seen that gives mathematical results on the training dynamics of this mean-field transformer model, although the results are conditional (see weaknesses).
The relation between the inner dissipation exponent and the learning-rate schedule is the part I found most interesting. It shows how token relaxation affects the training estimate. The fact that the proof is short is a positive aspect.
I found the rotating-anisotropy example useful because the parameter affects the loss through the equilibrium distribution. The authors check the estimates in (14) without an explicit formula for this distribution.
Critical weaknesses
The main issue is Assumption 3.2. The paper does not verify the density and Sobolev bounds for a nonstationary class of initial conditions. The uniform positive density bound is particularly strong: together with convergence, it implies , while even full support remains conjectural for general positive-definite (as the authors acknowledge). Appendix E checks (14) for the chosen losses, but the density and Sobolev bounds remain assumed during training.
The results do not directly apply to empirical token distributions. For , , already in the isotropic case. Figure 2 uses a harmonic cutoff and removes diagonal terms. It therefore does not measure the full norm of the empirical discrepancy, and the paper gives no error estimate relating this metric to the usual continuum norm.
Questions for authors
-
At least in the isotropic case, and at least locally, for initial densities in a neighborhood of , can the positive density be proved?
-
How should the particle number and harmonic cutoff increase in Figure 2 to approximate the continuum Sobolev norm? What is the effect of removing the diagonal terms?
-
A clearer definition of when introduced in the main text would be helpful
2 comments
New result addressing Assumption 3.2
We would like to share a new result addressing your concern about Assumption 3.2. For fixed isotropic parameters, we now have an explicit nonstationary example in which both bounds in the assumption hold for all time: the density stays bounded away from zero, and the negative Sobolev discrepancy from equilibrium remains bounded.
The proof was generated by LLMs and reviewed by the authors and by Astra (see detailed answer).
We have posted the precise statement, a concrete initialization, and a detailed proof sketch with references to Appendix B in our follow-up response to Reviewer wenE.
We hope this additional result helps clarify the scope and validity of the assumption.
We thank the reviewer for their positive evaluation, constructive comments, and helpful suggestions. We are glad that the analysis of the training dynamics, the link between the inner dissipation exponent and learning rates, and the rotating-anisotropy example were found useful. Below, we address your questions regarding the density bounds, the continuum norm estimation, and the gradient representation I(μ,θ).
Q1. On Proving Positive Density and Sobolev Bounds (Assumption 3.2)
We fully agree that Assumption 3.2 is a strong and structural condition. To address concerns regarding whether these bounds can be rigorously verified beyond a pure assumption, we have established new analytical results (detailed in Appendix B of the revised manuscript):
For further details on these proofs and initializations, we kindly refer to our detailed response to Reviewer 1.
Q2. Continuum Norm Estimation & Effect of Diagonal Terms (Figure 2)
On empirical measures, the harmonic cutoff, and diagonal removal
We agree with the reviewer that the full Sobolev discrepancy of an empirical token distribution is infinite for d≥3, already when the reference measure is uniform. Assumption 3.2 concerns the continuum distribution, and Figure 2 should not be interpreted as evaluating the full Sobolev norm of the empirical measure.
What Figure 2 measures. The figure estimates the truncated continuum discrepancy HM, which retains spherical harmonics up to degree M. It therefore probes the discrepancy at the resolved harmonic degrees.
Effect of removing the diagonal terms. For independent i.i.d. samples, the off-diagonal estimator is unbiased for the truncated population discrepancy, as in standard kernel discrepancy estimation. Since particles within our simulations interact, this unbiasedness statement does not automatically apply to our setting. Removing diagonal terms eliminates self-interactions, but does not eliminate errors caused by particle dependence.
Its effect can nevertheless be bounded deterministically. On S2, the truncated kernel has constant diagonal
=2\log M+O(1).$$ Let $H_M^{\mathrm{emp}}$ denote the truncated discrepancy computed from the two empirical measures with all diagonal terms retained, and let $\widehat H_M$ denote our off-diagonal estimator. For $N,N_{\mathrm{ref}}>1$, $$0\leq H_M^{\mathrm{emp}}-\widehat H_M \leq C_M\left(\frac{1}{N-1}+\frac{1}{N_{\mathrm{ref}}-1}\right).$$ This bound holds for arbitrary particle configurations. The corrected estimator can be negative and is not itself a squared norm. How to increase the particle number and cutoff. At a fixed time and fixed cutoff $M$, convergence of the empirical measures to their continuum targets implies convergence of the truncated discrepancy, because $K_M$ is bounded and continuous. One can then let $M$ increase to recover the full continuum Sobolev discrepancy, provided it is finite. For a joint limit, the condition $$\log M\left(\frac{1}{N}+\frac{1}{N_{\mathrm{ref}}}\right)\longrightarrow0$$ makes the diagonal correction negligible. This condition alone does not establish consistency for the full norm: one must also control the particle approximation, the equilibrium-reference error, and the unresolved spectral tail. Our current analysis does not provide these joint error bounds. We will clarify that Figure 2 provides numerical evidence at the tested spectral resolutions, rather than a verification of the full, uniform-in-time Sobolev assumption. --- ### Q3. Clarifying the Gradient Representation $I(\mu, \theta)$ To make the presentation more self-contained, we will clarify the definition of $\mathcal{I}(\mu,\theta)$ when it is first introduced in Section 4. In particular, $\mathcal{I}(\mu,\theta)$ is a computable surrogate for the outer gradient, which recovers the true gradient at equilibrium: $$\nabla \mathcal{L}(\theta) = \mathcal{I}(\mu_\theta^*,\theta).$$ When evaluated at a non-equilibrium token distribution $\mu$, it provides an approximation to this gradient. As verified in Appendix F for our experimental objectives, this approximation satisfies the following coupling error bound: $\Vert \mathcal{I}(\mu, \theta) - \mathcal{I}(\mu^{\ast}_{\theta}, \theta) \Vert_{2} \le C_{\mathcal{I}} \sqrt{h(\mu, \theta)}$ This bound directly relates the gradient approximation error caused by using a non-equilibrium distribution to the corresponding inner energy gap.