acceptodds
Under review as a conference paper at ICLR 2027

Convergence of Mean-Field Transformers and Bilevel Training Dynamics in the Repulsive Regime

Abstract

We study token convergence and parameter training in continuous-depth, mean-field self-attention on the sphere, in the repulsive regime with parameters shared across depth. For fixed symmetric positive-definite interactions, we investigate a Polyak–ojasiewicz-type inequality relating energy dissipation to a power of the energy gap. Under uniform regularity and non-degeneracy bounds along the flow, we establish this inequality with exponent , yielding an energy-gap bound. We then study training as a bilevel optimization problem: the token distribution evolves toward equilibrium, while the parameters are adjusted to minimize a loss defined at that equilibrium. Tokens and parameters evolve simultaneously, without requiring equilibration between parameter updates. Under suitable regularity and gradient-approximation assumptions, a dissipation exponent determines a learning-rate schedule yielding an bound on the time-averaged squared training-loss gradient. In particular, quadratic dissipation gives an training bound. Numerical experiments support the predicted convergence rates.

Primary area:
optimization
Keywords:
Transformers, Self-attention, Mean-field limits

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yt8sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies repulsive mean-field self-attention on the sphere with symmetric positive-definite interactions. Assuming uniform density and Sobolev bounds, it proves an O(t−1)O(t^{-1}) energy-gap estimate. It then studies simultaneous token evolution and parameter training, and derives a learning-rate schedule and a bound on the average squared training-loss gradient.

Critical strengths

This is the first paper I have seen that gives mathematical results on the training dynamics of this mean-field transformer model, although the results are conditional (see weaknesses).

The relation between the inner dissipation exponent and the learning-rate schedule is the part I found most interesting. It shows how token relaxation affects the training estimate. The fact that the proof is short is a positive aspect.

I found the rotating-anisotropy example useful because the parameter affects the loss through the equilibrium distribution. The authors check the estimates in (14) without an explicit formula for this distribution.

Critical weaknesses

The main issue is Assumption 3.2. The paper does not verify the density and Sobolev bounds for a nonstationary class of initial conditions. The uniform positive density bound is particularly strong: together with convergence, it implies μD⋆≥c0ν\mu_D^\star\geq c_0\nu, while even full support remains conjectural for general positive-definite DD (as the authors acknowledge). Appendix E checks (14) for the chosen losses, but the density and Sobolev bounds remain assumed during training.

The results do not directly apply to empirical token distributions. For d≥3d\geq3, ∥μN−ν∥H˙−1(ν)=∞\|\mu_N-\nu\|_{\dot H^{-1}(\nu)}=\infty, already in the isotropic case. Figure 2 uses a harmonic cutoff and removes diagonal terms. It therefore does not measure the full norm of the empirical discrepancy, and the paper gives no error estimate relating this metric to the usual continuum norm.

Questions for authors
  • At least in the isotropic case, and at least locally, for initial densities in a neighborhood of ν\nu, can the positive density be proved?

  • How should the particle number and harmonic cutoff increase in Figure 2 to approximate the continuum Sobolev norm? What is the effect of removing the diagonal terms?

  • A clearer definition of I(μ,θ)I(\mu,\theta) when introduced in the main text would be helpful

2 comments

We thank the reviewer for their positive evaluation, constructive comments, and helpful suggestions. We are glad that the analysis of the training dynamics, the link between the inner dissipation exponent and learning rates, and the rotating-anisotropy example were found useful. Below, we address your questions regarding the density bounds, the continuum norm estimation, and the gradient representation I(μ,θ)I(\mu, \theta).


Q1. On Proving Positive Density and Sobolev Bounds (Assumption 3.2)

We fully agree that Assumption 3.2 is a strong and structural condition. To address concerns regarding whether these bounds can be rigorously verified beyond a pure assumption, we have established new analytical results (detailed in Appendix B of the revised manuscript):

  • Finite-Time Local Bound: In the isotropic case, we prove via a Lagrangian Grönwall estimate on the flow map that if min⁡ρ0>0\min \rho_0 > 0, the density remains strictly positive on any finite time horizon (min⁡ρt≥ρmin⁡(0)e−C0t\min \rho_t \ge \rho_{\min}(0)e^{-C_0 t}).

For further details on these proofs and initializations, we kindly refer to our detailed response to Reviewer 1.


Q2. Continuum Norm Estimation & Effect of Diagonal Terms (Figure 2)

On empirical measures, the harmonic cutoff, and diagonal removal

We agree with the reviewer that the full Sobolev discrepancy of an empirical token distribution is infinite for d≥3d\geq3, already when the reference measure is uniform. Assumption 3.2 concerns the continuum distribution, and Figure 2 should not be interpreted as evaluating the full Sobolev norm of the empirical measure.

What Figure 2 measures. The figure estimates the truncated continuum discrepancy HMH_M, which retains spherical harmonics up to degree MM. It therefore probes the discrepancy at the resolved harmonic degrees.

Effect of removing the diagonal terms. For independent i.i.d. samples, the off-diagonal estimator is unbiased for the truncated population discrepancy, as in standard kernel discrepancy estimation. Since particles within our simulations interact, this unbiasedness statement does not automatically apply to our setting. Removing diagonal terms eliminates self-interactions, but does not eliminate errors caused by particle dependence.

Its effect can nevertheless be bounded deterministically. On S2\mathbb S^2, the truncated kernel has constant diagonal

=2\log M+O(1).$$ Let $H_M^{\mathrm{emp}}$ denote the truncated discrepancy computed from the two empirical measures with all diagonal terms retained, and let $\widehat H_M$ denote our off-diagonal estimator. For $N,N_{\mathrm{ref}}>1$, $$0\leq H_M^{\mathrm{emp}}-\widehat H_M \leq C_M\left(\frac{1}{N-1}+\frac{1}{N_{\mathrm{ref}}-1}\right).$$ This bound holds for arbitrary particle configurations. The corrected estimator can be negative and is not itself a squared norm. How to increase the particle number and cutoff. At a fixed time and fixed cutoff $M$, convergence of the empirical measures to their continuum targets implies convergence of the truncated discrepancy, because $K_M$ is bounded and continuous. One can then let $M$ increase to recover the full continuum Sobolev discrepancy, provided it is finite. For a joint limit, the condition $$\log M\left(\frac{1}{N}+\frac{1}{N_{\mathrm{ref}}}\right)\longrightarrow0$$ makes the diagonal correction negligible. This condition alone does not establish consistency for the full norm: one must also control the particle approximation, the equilibrium-reference error, and the unresolved spectral tail. Our current analysis does not provide these joint error bounds. We will clarify that Figure 2 provides numerical evidence at the tested spectral resolutions, rather than a verification of the full, uniform-in-time Sobolev assumption. --- ### Q3. Clarifying the Gradient Representation $I(\mu, \theta)$ To make the presentation more self-contained, we will clarify the definition of $\mathcal{I}(\mu,\theta)$ when it is first introduced in Section 4. In particular, $\mathcal{I}(\mu,\theta)$ is a computable surrogate for the outer gradient, which recovers the true gradient at equilibrium: $$\nabla \mathcal{L}(\theta) = \mathcal{I}(\mu_\theta^*,\theta).$$ When evaluated at a non-equilibrium token distribution $\mu$, it provides an approximation to this gradient. As verified in Appendix F for our experimental objectives, this approximation satisfies the following coupling error bound: $\Vert \mathcal{I}(\mu, \theta) - \mathcal{I}(\mu^{\ast}_{\theta}, \theta) \Vert_{2} \le C_{\mathcal{I}} \sqrt{h(\mu, \theta)}$ This bound directly relates the gradient approximation error caused by using a non-equilibrium distribution to the corresponding inner energy gap.
AuthorsOpenReview ↗

New result addressing Assumption 3.2

We would like to share a new result addressing your concern about Assumption 3.2. For fixed isotropic parameters, we now have an explicit nonstationary example in which both bounds in the assumption hold for all time: the density stays bounded away from zero, and the negative Sobolev discrepancy from equilibrium remains bounded.

The proof was generated by LLMs and reviewed by the authors and by Astra (see detailed answer).

We have posted the precise statement, a concrete initialization, and a detailed proof sketch with references to Appendix B in our follow-up response to Reviewer wenE.

We hope this additional result helps clarify the scope and validity of the assumption.

Reviewer_tjFvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies mean-field self-attention on the sphere in the repulsive regime, with parameters shared across depth. In this regime, the authors prove a bound on the energy gap between the distribution of tokens at a given tt and the equilibrium distribution, The authors also consider a model in which the token dynamics and the training dynamics are connected by the same continuous-time variable. Under this setting, they show that a learning rate ηt∝(1+t)−1/2\eta_t\propto(1+t)^{-1/2} yields an O(log⁡T/T)O(\log T/\sqrt T) bound on the time-averaged squared gradient of the equilibrium training loss.

Critical strengths

None

Critical weaknesses
  1. My biggest issue with the paper is the assumption of coupled dynamics, which seems essential for the second result. In Section 4, Eq. (11) uses tt both for the representation of depth, as used in many token dynamics papers, and for continuous optimization time. Although the learning rate controls their relative speeds, the use of tt for both is a nontrivial choice and is justified only vaguely in the discussion surrounding this equation.
  2. The main contributions and the presentation are very hard to understand from the way the paper is written. Terms and definitions are introduced very briefly, and the assumptions lack an informative introduction.
Questions for authors
  1. Could you better explain why the coupling of network depth and optimization time makes sense?

  2. The contributions of the work are not clearly explained throughout the paper. Could you add a contributions paragraph that highlights the main results and refers to their corresponding theorems?

  3. Could you introduce the assumptions in the paper and explain why they are reasonable? This applies to all assumptions, but specifically, Assumption 3.2 is a bit counterintuitive to me: it is unclear why the density should remain uniformly bounded below for all time. An example demonstrating this could also help.

  4. Could you explain more clearly the practical implications of your analysis? What are the practical conclusions from both the energy-gap bound and the training-rate balance?

  5. Beyond the existing particle simulations, a good way to justify the setting would be an experiment with a transformer showing whether the results provide useful guidance in practice. Could the authors provide such an experiment?

6 comments

We thank the reviewer for their constructive feedback and helpful suggestions on presentation, clarity, and practical implications. We appreciate the opportunity to clarify the coupling of timescales, detail our contributions, and connect our findings to practical transformer architectures. Below is our point-by-point response.


Q2 and Q4. Clarifying Contributions and Presentation, Practical Implications

We will add a dedicated contributions paragraph with explicit pointers to the results and their interpretation. Our analysis provides actionable theoretical guidance for practitioners working with parameter-tied networks:

  • Fixed parameters: convergence of the token representations, from finite to infinite depth — Proposition 3.3: Under Assumption 3.2, the energy gap ete_t satisfies a dissipation inequality of the form −e˙t≥κinetβ-\dot e_t \geq \kappa_{\mathrm{in}} e_t^\beta with exponent β=2\beta=2, yielding an O(t−1)O(t^{-1}) energy-gap bound.

    Interpretation: Applying successive attention layers changes the token representations. In our continuous-depth, mean-field model with fixed shared parameters, this bound quantifies how (fast) increasing depth brings their distribution closer to the infinite-depth equilibrium in interaction energy. It provides a sufficient continuous depth to achieve a prescribed energy accuracy ensuring that tokens are sufficiently equilibrated.

  • Convergence rates for transformer parameter training — Theorem 4.2: Under Assumption 4.1, we bound the time-averaged squared gradient of the equilibrium training loss in terms of the inner dissipation exponent β\beta. In particular, for β=2\beta=2, the schedule ηt=η0(1+t)−1/2\eta_t=\eta_0(1+t)^{-1/2} yields an O(log⁡T/T)O(\log T/\sqrt{T}) bound.

    Interpretation: This result explains how finite-depth attention computation can be used to train an objective defined at infinite depth. In the coupled dynamics, parameters are updated using the current token representations before they have fully equilibrated. The learning-rate schedule accounts for this incomplete relaxation and yields the stated training convergence bound. The coupling result (Theorem 4.2) dictates how aggressively one can update parameters without waiting for full inner-loop convergence. Specifically, it proves that one does not need to fully equilibrate the token dynamics at every parameter step (which would be computationally prohibitive); instead, a decaying learning rate schedule (δ=(β−1)/β\delta = (\beta-1)/\beta) guarantees convergence of the training loss while saving substantial computation.

Q1. Why Coupling Network Depth and Optimization Time Makes Sense

The use of the same continuous variable tt for both token depth and parameter optimization represents the relative speed at which parameters evolve compared to the internal token representations. Consider a sequence of parameter iterates θs\theta_s. Between two parameter updates, the current token representations are propagated through an additional depth ΔTs\Delta T_s. If asa_s denotes the parameter step size, the relative rate between parameter update and token transport is

ηs=asΔTs.\eta_s=\frac{a_s}{\Delta T_s}.

This gives the following interpretation:

  • With a fixed parameter step size, decreasing ηs\eta_s means increasing the depth between parameter updates: more layers are applied to the current tokens before the parameters are updated again. Hence for ηs=η0(1+s)−δ\eta_s = \eta_0 (1+s)^{-\delta}, a decreasing learning rate (δ>0\delta > 0) means that as training progresses, parameters change increasingly slowly relative to the token depth, giving the inner token distribution more time to track its evolving equilibrium. This suggests that, within our framework, the effective depth of the transformer should scale up as training stabilizes, to ensure the tokens remain close to equilibrium.
  • With a fixed parameter step size, constant ηs\eta_s allows this depth to remain constant.

Thus, one can jointly choose a sequence of parameter updates and a sequence of depths. In the continuous-time formulation, ηt\eta_t controls how quickly parameters change relative to depth.

Q3. Justifying Assumption 3.2 (Uniform Boundedness Away from Zero)

We agree that Assumption 3.2 is substantive, particularly because its density and Sobolev bounds must remain uniform over the entire trajectory. The revised manuscript now includes Appendix B, which establishes an explicit positivity guarantee on finite depth horizons.

For the isotropic interaction D=αIdD=\alpha I_d, Proposition B.2 proves that any classical solution starting from a density bounded away from zero satisfies

min⁡xρt(x)≥ρmin⁡(0)e−C0t,\min_x \rho_t(x) \geq \rho_{\min}(0)e^{-C_0t},

at every time at which the solution is defined, with an explicit constant C0=C0(α,d)C_0=C_0(\alpha,d). The proof combines the Lagrangian representation of the density with a uniform bound on the divergence of the attention velocity.

Thus, on every finite interval [0,T][0,T] of existence, the density remains bounded below by the positive constant ρmin⁡(0)e−C0T\rho_{\min}(0)e^{-C_0T}. For example, starting from

ρ0(x)=1+εx1,0<∣ε∣<1,\rho_0(x)=1+\varepsilon x_1, \qquad 0 < |\varepsilon| < 1,

the estimate gives ρt≥(1−∣ε∣)e−C0T\rho_t \geq (1-|\varepsilon|)e^{-C_0T} throughout that interval. This argument requires no smallness or analyticity assumption beyond the stated classical-solution framework and strict positivity at initialization.

We emphasize the distinction between this finite-horizon result and Assumption 3.2: the lower bound deteriorates with TT, so it does not establish a positive lower bound uniform over all time. A nonstationary class satisfying the full assumption for infinite time remains to be established.

For the Sobolev condition, Figure 2 provides complementary numerical evidence through decreasing truncated discrepancies across the tested anisotropies and harmonic cutoffs. These observations concern the resolved harmonic degrees; they do not establish boundedness of the full continuum Sobolev norm. We will clarify this interpretation in the experimental discussion.


Q5. Connection to Real Transformers and Practical Guidance

Our framework studies a simplified self-attention model, but its assumption of shared parameters across depth has concrete architectural precedents.

Parameter-tied architectures such as Looped Transformers [4,5] and ALBERT [3] reuse the same parameters across multiple layers or iterations. In particular, Looped Transformers iteratively update token representations by repeatedly applying a shared transformer block, making their evolution over an increasing number of iterations a natural object of study.

Our analysis provides a simplified continuous-depth, mean-field perspective on these dynamics, allowing us to investigate their asymptotic behavior as the number of iterations grows. In particular, we study the relaxation of continuous token densities in the mean-field limit under fixed attention parameters, as well as the interplay between this relaxation and parameter optimization.

We emphasize that our model isolates the self-attention dynamics rather than reproducing a complete Transformer architecture, which also includes components such as feed-forward networks and normalization layers. Our experiments implement the corresponding finite-token softmax attention dynamics, including residual updates and spherical normalization, and focus on validating the predicted relaxation and training mechanisms. Extending these insights to full Transformer architectures and downstream tasks remains an important direction for future work.

References

[1] Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. The emergence of clusters in self-attention dynamics. NeurIPS, 2023.

[2] Bruno, G., Pasqualotto, F., and Agazzi, A. Emergence of meta-stable clustering in mean-field transformer models. ICLR, 2025.

[3] Lan, Z., et al. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. ICLR, 2020.

[4] Yang, L., Lee, K., Nowak, R., and Papailiopoulos, D. Looped Transformers are Better at Learning Learning Algorithms. ICLR, 2024.

[5] Saunshi, N., Dikkala, N., Li, Z., Kumar, S., and Reddi, S. J. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR, 2025.

Reviewer_tjFvOpenReview ↗

Response to the author clarifications

Thank you for your response. I appreciate the clarifications but I still have some things that are not clear.

  1. "We will add a dedicated contributions paragraph with explicit pointers to the results and their interpretation", If possible add those changes to the rebuttal version, and please color the changes from the previous version.

Q1

I still don't understand your answer, why does " increasing the depth between parameter updates" is something sensible, can you give an example to when that happens in practice? when is this coupled procedure used in practice, and how does it relate to standard Transformer training?

Q2 Q4

Still this interpretation and conclusion sits on the assumption that the training dynamics and the depth dynamics are coupled, I don't understand when it happens in practice

Q3

What do you mean by that? Are you saying the example you provide doesn't actually adhere to the assumption? "We emphasize the distinction between this finite-horizon result and Assumption 3.2: the lower bound deteriorates with, so it does not establish a positive lower bound uniform over all time. A nonstationary class satisfying the full assumption for infinite time remains to be established."

Q5

Ok

Reviewer_wenEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper contributes to the theoretical analysis of transformers, in the limit where the number of layers goes to infinity. More precisely, it investigates the continuous-depth, mean-field limit of self-attention on the sphere specifically within the parameter-tied, repulsive regime where the value, key and query matrices satisfies a specific connection. Under suitable assumptions, the authors prove an energy dissipation inequality and then propose a bilevel optimization framework for training transformers.

Critical strengths

(1) The mathematical framework is clear.

(2) The setting (repulsive vs attractive) has received less attention so far in this literature.

(3) The theoretical analysis seems rigorous.

Critical weaknesses

(1) The assumptions are very restrictive (in particular Assumption 3.2 which involves the whole flow).

(2) The main condition on the architecture is very struct (V=−DV = -D with D=QKTD= Q K^T); this is certainly convenient for the analysis but it is not really clear how far this assumption is from being satisfied in practice.

(3) The experiments are very limited in terms of scope (not on real transformers as far as I understand).

Questions for authors

(1) Could you please provide a concrete example satisfying all the assumptions?

(2) Could you please explain how likely it is that the structural assumption is satisfied in practice? If it is not, then are your results likely to extend to situations encountered in transformers in practice?

(3) This might be challenging, but could you provide experiments on real transformers and explain how they connect to your theory?

5 comments

We thank the reviewer for their careful reading and constructive feedback. In response, we have added a finite-horizon density-positivity result and clarified the motivation for the repulsive regime, the role of parameter sharing, and the scope of our experiments.

Q1. Assumption 3.2 and concrete examples

We agree that Assumption 3.2 is substantive, particularly because its density and Sobolev bounds must remain uniform over the entire trajectory. The revised manuscript now includes Appendix B, which establishes an explicit positivity guarantee on finite depth horizons.

For the isotropic interaction D=αIdD=\alpha I_d, Proposition B.2 proves that any classical solution starting from a density bounded away from zero satisfies

min⁡xρt(x)≥ρmin⁡(0)e−C0t,\min_x \rho_t(x) \geq \rho_{\min}(0)e^{-C_0t},

at every time at which the solution is defined, with an explicit constant C0=C0(α,d)C_0=C_0(\alpha,d). The proof combines the Lagrangian representation of the density with a uniform bound on the divergence of the attention velocity.

Thus, on every finite interval [0,T][0,T] of existence, the density remains bounded below by the positive constant ρmin⁡(0)e−C0T\rho_{\min}(0)e^{-C_0T}. For example, starting from

ρ0(x)=1+εx1,0<∣ε∣<1,\rho_0(x)=1+\varepsilon x_1, \qquad 0 < |\varepsilon| < 1,

the estimate gives ρt≥(1−∣ε∣)e−C0T\rho_t \geq (1-|\varepsilon|)e^{-C_0T} throughout that interval. This argument requires no smallness or analyticity assumption beyond the stated classical-solution framework and strict positivity at initialization.

We emphasize the distinction between this finite-horizon result and Assumption 3.2: the lower bound deteriorates with TT, so it does not establish a positive lower bound uniform over all time. A nonstationary class satisfying the full assumption for infinite time remains to be established.

For the Sobolev condition, Figure 2 provides complementary numerical evidence through decreasing truncated discrepancies across the tested anisotropies and harmonic cutoffs. These observations concern the resolved harmonic degrees; they do not establish boundedness of the full continuum Sobolev norm. We will clarify this interpretation in the experimental discussion.

Q2. On the Structural Assumption (V=−DV = -D)

As you note among the strengths, the repulsive regime has received comparatively little attention. The restriction V=−DV=-D, with symmetric positive-definite D=Q⊤KD=Q^\top K, isolates a repulsive regime complementary to the extensively studied attractive setting (V=DV=D), including the clustering analyses of Geshkovski et al. [1] and Bruno, Pasqualotto, and Agazzi [2].

Both regimes capture important mechanisms: attractive attention promotes token clustering, while repulsive attention favors dispersion. In particular, the isotropic repulsive model has the uniform distribution as its unique energy minimizer. Understanding this complementary behavior is a central motivation for our work.

Although restrictive, this structural assumption gives the dynamics an interaction-energy gradient-flow formulation. It enables our quantitative convergence analysis under the stated regularity assumptions and, crucially, connects attention relaxation to simultaneous parameter learning.

Q3. Connection to transformer architectures and scope of the experiments

Our framework studies a simplified self-attention model, but its assumption of shared parameters across depth has concrete architectural precedents.

Parameter-tied architectures such as Looped Transformers [4,5] and ALBERT [3] reuse the same parameters across multiple layers or iterations. In particular, Looped Transformers iteratively update token representations by repeatedly applying a shared transformer block, making their evolution over an increasing number of iterations a natural object of study.

Our analysis provides a simplified continuous-depth, mean-field perspective on these dynamics, allowing us to investigate their asymptotic behavior as the number of iterations grows. In particular, we study the relaxation of continuous token densities in the mean-field limit under fixed attention parameters, as well as the interplay between this relaxation and parameter optimization.

We emphasize that our model isolates the self-attention dynamics rather than reproducing a complete Transformer architecture, which also includes components such as feed-forward networks and normalization layers. Our experiments implement the corresponding finite-token softmax attention dynamics, including residual updates and spherical normalization, and focus on validating the predicted relaxation and training mechanisms. Extending these insights to full Transformer architectures and downstream tasks remains an important direction for future work.

References

[1] Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. The emergence of clusters in self-attention dynamics. NeurIPS, 2023.

[2] Bruno, G., Pasqualotto, F., and Agazzi, A. Emergence of meta-stable clustering in mean-field transformer models. ICLR, 2025.

[3] Lan, Z., et al. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. ICLR, 2020.

[4] Yang, L., Lee, K., Nowak, R., and Papailiopoulos, D. Looped Transformers are Better at Learning Learning Algorithms. ICLR, 2024.

[5] Saunshi, N., Dikkala, N., Li, Z., Kumar, S., and Reddi, S. J. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR, 2025.

I would like to thank the authors for the detailed response. Some points are clearer but it seems my Q1 has not been addressed: I was expecting an example satisfying all the assumptions at once (a precise example and a precise check for each assumption). Perhaps I missed it but for now I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.