acceptodds
Under review as a conference paper at ICLR 2027

Nonasymptotic P Theory for Neural Networks with Applications to Small-to-Large Generalization

Abstract

Maximal Update Parameterization (P) provides a principled framework for scaling neural networks while preserving non-degenerate feature-learning dynamics in the width limit. However, existing theory for P is largely asymptotic, relying on the infinite-width limit and leaving its behavior in finite-width regimes poorly understood. In this paper, we use the Tensor Program (TP) framework to provide a non-asymptotic characterization of gradient descent dynamics in canonical scalar-input, width- shallow settings: two-layer linear and quadratic networks, and three-layer linear networks. Our analysis provides high-probability guarantees for the finite-width approximation over each fixed training horizon. As an application, we analyze small-to-large generalization for KL-regularized domain reweighting under two-layer square-loss dynamics, showing that the learned domain weights along the width- training trajectory remain close to their infinite-width counterparts with discrepancy . Finally, synthetic and real-world experiments provide qualitative evidence of cross-width domain-weight stability.

Primary area:
optimization
Keywords:
Nonasymptotic, maximal update parameterization, neural networks, small-to-large generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5J4wOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper develops finite-width concentration estimates for gradient descent under maximal update parameterization. The analysis covers two-layer linear and quadratic networks and three-layer linear networks with scalar inputs.

Critical strengths

The paper derives high-probability finite-width concentration estimates for several shallow networks under μP. Theorems 4.3 and 4.4 distinguish unstable square-loss dynamics from a specific stable cross-entropy setting.

Critical weaknesses

[Note added Nov 14, 2026: I withdraw the struck-through criticism below regarding BTB_T. I misread line 367: Theorem 6.1 already includes the square root in defining BTB_T, matching Appendix G, so the bound follows directly from Lemma G.3. My remaining weaknesses are unchanged.]

Theorem 6.1 defines BTB_T as the maximum infinite-width domain loss, BT=max⁡t,iℓi,t(∞)B_T=\max_{t,i}\ell_{i,t}^{(\infty)}, and states the domain-weight discrepancy bound as 2λ(BT+ε)ε2\lambda(B_T+\varepsilon)\varepsilon. Appendix G instead defines BTB_T as the maximum root-mean-square residual. These are different quantities.

For the definition in Theorem 6.1, the Cauchy–Schwarz argument in Lemma G.3 yields λ(2BTε+ε2)\lambda(2\sqrt{B_T}\varepsilon+\varepsilon^2). The displayed bound in the main theorem does not follow from this argument for arbitrary positive BTB_T. The authors must correct the definition or the inequality and verify the consequences for the cross-width stability statement. The explicit theorem is not consistently stated.

Theorem 4.1 the displayed estimate does not provide a meaningful quantitative prescription for selecting proxy-model widths over ordinary training durations.

Theorem 4.3 establishes that exponentially amplified fluctuations are possible under the square-loss assumptions. The time-uniform result in Theorem 4.4 concerns a special data configuration and learning rate for which the infinite-width dynamics reach a stable fixed point after one update. It does not establish comparable stability for general cross-entropy training.

The application to domain reweighting consequently remains limited. Theorem 6.1 controls best-response domain weights along jointly trained two-layer trajectories. It does not establish that weights learned by a smaller proxy model improve the training of a separate larger model. Moreover, the GPT experiments use exponentiated-gradient updates and MuAdamW rather than the exact best-response gradient-descent dynamics analyzed in the theorem. These experiments do not validate the stated finite-width guarantees for that training procedure.

Questions for authors
  1. Which definition of BTB_T is intended in Theorem 6.1? The main theorem uses the maximum squared loss, while Appendix G uses the maximum root-mean-square residual. Can the authors provide the corrected bound and confirm that all dependent results remain valid?
  2. Can the authors evaluate the concentration bound in Theorem 4.1 for representative training horizons and failure probabilities, including the constants CTC_T? At what model widths does the resulting bound become smaller than a useful error tolerance?
  3. What additional assumptions would yield a non-vacuous cross-width guarantee for training horizons that grow with width? Theorem 4.3 rules out uniform control under the present assumptions in general.
  4. How does Theorem 6.1 apply to the iterative domain-weight updates and MuAdamW optimizer used in the GPT experiments? Is there a bound on the difference between those updates and the exact KL best responses required by the theorem?
11 comments

Q1. Theorem 6.1 defines BTB_T as the maximum infinite-width domain loss, BT=max⁡t,iℓi,t(∞)B_T=\max_{t,i}\ell_{i,t}^{(\infty)}, and states the domain-weight discrepancy bound as 2λ(BT+ε)ε2\lambda(B_T+\varepsilon)\varepsilon. Appendix G instead defines (BT)(B_T) as the maximum root-mean-square residual. These are different quantities. For the definition in Theorem 6.1, the Cauchy–Schwarz argument in Lemma G.3 yields λ(2BTε+ε2)\lambda(2\sqrt{B_T}\varepsilon+\varepsilon^2). The displayed bound in the main theorem does not follow from this argument for arbitrary positive BTB_T. The authors must correct the definition or the inequality and verify the consequences for the cross-width stability statement. The explicit theorem is not consistently stated. Theorem 4.1 the displayed estimate does not provide a meaningful quantitative prescription for selecting proxy-model widths over ordinary training durations. Theorem 4.3 establishes that exponentially amplified fluctuations are possible under the square-loss assumptions. The time-uniform result in Theorem 4.4 concerns a special data configuration and learning rate for which the infinite-width dynamics reach a stable fixed point after one update. It does not establish comparable stability for general cross-entropy training. The application to domain reweighting consequently remains limited. Theorem 6.1 controls best-response domain weights along jointly trained two-layer trajectories. It does not establish that weights learned by a smaller proxy model improve the training of a separate larger model. Moreover, the GPT experiments use exponentiated-gradient updates and MuAdamW rather than the exact best-response gradient-descent dynamics analyzed in the theorem. These experiments do not validate the stated finite-width guarantees for that training procedure.

A1. The alleged inconsistency in BTB_T is not present in the supplied manuscript. Theorem 6.1 already defines BTB_T using ℓi,t(∞)\sqrt{\ell_{i,t}^{(\infty)}}, exactly as Appendix G does. With this definition, the displayed inequality follows from Lemma G.3; we give the calculation in A2 below.

The other points concern the scope or tightness of guarantees, not a contradiction in their proofs. Our contribution is a finite-width training-dynamics theorem around the μ\muP limit, not a universal theorem for every architecture, optimizer, or training duration. Theorem 4.3 establishes a genuine obstruction to a general polynomial-time bound, while Theorem 4.4 demonstrates a stability mechanism that removes horizon dependence in its stated setting. Proposition I.1 provides an additional explicit sensitivity-based bound.

The domain-weight result is also more than post-processing a nonadaptive output theorem. The weights change the subsequent training trajectory. Appendix G controls this feedback through the coefficient-error estimate in Lemma G.3 and the adaptive bootstraps in Lemmas G.4--G.5. Theorem 6.1 then provides a quantitative cross-width guarantee for these jointly trained trajectories. A performance guarantee for a fresh target trained with frozen proxy weights is a different result, explicitly excluded by Remark 6.2.

Finally, Section 7.2 identifies the GPT procedure as a practical variant beyond the assumptions, not an instantiation of the proved GD theorem. We will keep that distinction prominent. Given the actual definition of BTB_T and the explicitly stated scope, we respectfully ask that the soundness assessment be reconsidered separately from requests for broader or sharper results.

Q2. Which definition of BTB_T is intended in Theorem 6.1? The main theorem uses the maximum squared loss, while Appendix G uses the maximum root-mean-square residual. Can the authors provide the corrected bound and confirm that all dependent results remain valid?

A2. The main theorem already uses the maximum root-mean-square residual, not the maximum squared loss. In the supplied PDF, Theorem 6.1 reads

$ B_T:= \max_{0\le t\le T}\max_{i\in[K]} \sqrt{\ell_{i,t}^{(\infty)}}<\infty. $

Appendix G uses the same definition. Thus, no change to the theorem is required on this point.

Q3. Can the authors evaluate the concentration bound in Theorem 4.1 for representative training horizons and failure probabilities, including the constants CTC_T? At what model widths does the resulting bound become smaller than a useful error tolerance?

A3. Theorem 4.1 establishes

\le C_Tm^{-1/2} \left(1+\log\frac{2m(T+1)}{\delta}\right)^{a_T}, \qquad a_T=2\cdot7^{T+2}+1.$$ Although the proof establishes that $C_T$ is finite, it does not numerically calibrate this constant. We therefore conduct a simulation to estimate its empirical counterpart. We consider the two-layer linear $\mu$P model under square loss, with Gaussian initialization, repeated training data $(\xi,y)=(1,1)$, and learning rate $\eta=0.01$. For each width $m\in\{128,512,2048,8192\}$, we run $20{,}000$ independent simulations and measure \$ E_{m,T}=\max_{0\le t\le T} |g_t^{(m)}(1)-\bar g_t(1)|. \$ We estimate the effective concentration prefactor by \$ \widehat K_{T,\delta} =\max_m\sqrt m\, \widehat Q_{1-\delta}(E_{m,T}), \$ where $\widehat Q_{1-\delta}$ is the empirical quantile. The results are: | Training horizon $T$ | $\delta=0.05$ | $\delta=0.01$ | |---:|---:|---:| | 1 | 1.970 | 2.636 | | 10 | 1.971 | 2.636 | | 100 | 1.972 | 2.636 | | 500 | 1.972 | 2.636 | For tolerance $\tau=0.1$, the empirical error is below $\tau$ at width $m=512$ for 95% confidence and $m=2048$ for 99% confidence, among the tested widths. The effective prefactor remains nearly constant with $T$ in this stable example. We also estimate the constant using the exact logarithmic factor in Theorem 4.1: \$ \widehat C_{T,\delta} =\max_m \frac{\sqrt m\,\widehat Q_{1-\delta}(E_{m,T})} {\left(1+\log\frac{2m(T+1)}{\delta}\right)^{a_T}}. \$ For $\delta=0.05$, this gives $\log_{10}\widehat C_{1,\delta}\approx-693.6$ and $\log_{10}\widehat C_{2,\delta}\approx-4932.0$. These extremely small values illustrate the conservatism of the theorem's logarithmic exponent. These estimates are empirical, not certified values of the universal $C_T$. The experiment demonstrates finite-width accuracy at practical widths, while obtaining rigorous numerical width thresholds from Theorem 4.1 requires sharper explicit control of its proof constants. **Q4. What additional assumptions would yield a non-vacuous cross-width guarantee for training horizons that grow with width? Theorem 4.3 rules out uniform control under the present assumptions in general.** **A4.** The key additional requirement is stability of the training dynamics, which prevents finite-width errors from being amplified over time. Theorem 4.3 shows that such stability cannot be guaranteed under the general square-loss assumptions. Theorem 4.4 already establishes sufficient conditions: under its specified data, label-smoothed cross-entropy loss, and learning rate, the output direction is locally contractive, while the drift in the remaining neutral direction is summable. Gaussian initialization concentration and a local bootstrap then yield \$ \sup_{t\ge0}\max_i |g_t^{(m)}(\xi_i)-\bar g_t(\xi_i)| \le C\sqrt{\frac{\log(4/\delta)}{m}}, \$ for $m\ge C_0\log(4/\delta)$, with constants independent of T. Applying this bound to two widths and using the triangle inequality gives \$ \sup_{t\ge0}\max_i |g_t^{(m_s)}(\xi_i)-g_t^{(m_l)}(\xi_i)| \le C\sqrt{\log(8/\delta)} \left(m_s^{-1/2}+m_l^{-1/2}\right) \$ with probability at least $1-\delta$. Thus, Theorem 4.4 already provides a non-vacuous cross-width guarantee for arbitrarily growing training horizons.

Q5. How does Theorem 6.1 apply to the iterative domain-weight updates and MuAdamW optimizer used in the GPT experiments? Is there a bound on the difference between those updates and the exact KL best responses required by the theorem?

A5. We clarify that adaptive domain reweighting is already an essential part of Theorem 6.1, rather than a departure from the theory. In Algorithm 1, the domain weights are updated at every training step through the exact KL-regularized best response $ \pi_{i,t}^{*} =\frac{\exp(\lambda\ell_{i,t})} {\sum_j\exp(\lambda\ell_{j,t})}, $ and the resulting weights determine the subsequent μ\muP gradient-descent update. Theorem 6.1 explicitly controls this coupling between domain weights and network training.

The GPT experiments in Section 7.2 follow the same KL-regularized reweighting objective but use exponentiated-gradient ascent to update the weights iteratively. For a uniform domain prior, the update takes the form $ \pi_{i,t+1}\propto \pi_{i,t}^{1-\eta_\pi/\lambda} \exp(\eta_\pi\ell_{i,t}). $ For fixed domain losses, its stationary point coincides with the exact best response in Theorem 6.1. Thus, the theoretical and experimental weight updates share the same optimization objective, although the theorem does not provide a finite-step tracking bound for their discrepancy.

Similarly, our theoretical guarantees apply to μ\muP gradient descent, whereas we use MuAdamW in the GPT experiments because AdamW-style optimizers are standard for Transformer training. Extending the guarantees to MuAdamW is beyond the current analysis.

Importantly, Section 7.1 provides more directly comparable synthetic experiments using Algorithm 1. For two-layer linear networks with input dimension d=500d=500, the learned domain weights closely agree across widths, achieving DKL<2×10−8\mathrm{D}_{\mathrm{KL}}<2\times10^{-8} for student widths m≥500m\ge500 after 500500 training steps. The ReLU experiments and the additional quadratic-activation comparison further examine this phenomenon. Although the multidimensional and ReLU settings extend beyond the assumptions of Theorem 6.1, these experiments directly test its predicted cross-width stability under joint domain reweighting and μ\muP training.

Therefore, the synthetic experiments provide controlled empirical support for the theoretical mechanism, while the GPT experiments demonstrate that similar cross-width domain-weight agreement persists with practical Transformer architectures and MuAdamW. We distinguish this empirical extension from a formal finite-width guarantee for the GPT training procedure.

Reviewer_WmqeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies nonasymptotic finite-width concentration of discrete-time GD trajectories around their infinite-width Maximal Update Parameterization (μ\muP) limits (Yang & Hu, 2021) over a fixed horizon TT. For three scalar-input (ξ∈R\xi \in \mathbb{R}) architectures of width mm under square loss, Theorem 4.1 establishes high-probability output-trajectory concentration bounds scaling as CTm−1/2(1+log⁡2m(T+1)δ)aTC_T m^{-1/2}(1 + \log\frac{2m(T+1)}{\delta})^{a_T} with aT=2⋅7T+2+1a_T = 2\cdot 7^{T+2}+1. Complementing the upper bounds, it exhibits a scalar two-layer linear instance under square loss where the asymptotic variance prefactor grows as Θ(9T)\Theta(9^T), while Theorem 4.4 exhibits a linearly separable binary classification setting under label-smoothed cross-entropy where finite-width trajectory deviations remain uniformly bounded for sufficiently large width. It then extends the scalar two-layer linear and quadratic analyses to KL-regularized domain reweighting with exact softmax best responses. It also reports empirical transfer results on synthetic ReLU MLPs and a 4-layer GPT-style model trained on SlimPajama-6B.

Critical strengths
  • The exact two-dimensional empirical moment closure (qt(m),st(m))(q_t^{(m)},s_t^{(m)}) makes the two-layer linear examples particularly transparent. In Theorem 4.3, an unstable limiting fixed point forces a fixed-confidence m−1/2m^{-1/2} fluctuation prefactor of order 3T3^T. In Theorem 4.4, the output direction is contractive while the other state direction is neutral; the proof controls its drift by summing the decaying output errors. This yields time-uniform O(m−1/2log⁡(4/δ))O(m^{-1/2}\sqrt{\log(4/\delta)}) control for m≥m0(δ)m\ge m_0(\delta).
  • Proposition I.1 extends the covariance analysis to vector inputs x∈Rdx\in\mathbb R^d. With h=(d+1)log⁡9+log⁡(2/δ)h=(d+1)\log9+\log(2/\delta) and m≥64hm\ge64h, its bound is 4XDT(h/m+h/m)4XD_T(\sqrt{h/m}+h/m) for evaluation inputs of norm at most XX. The logarithmic powers are independent of TT and there is no log⁡m\log m factor; time and learning-rate dependence remain explicit in the sensitivity factor DTD_T.
Critical weaknesses
  • In the three-layer linear analysis (Appendix F), unrolling the discrete-time rank-one weight updates Wt+1=W0−∑r=0tρrm(mvr)ur⊤W_{t+1} = W_0 - \sum_{r=0}^t \frac{\rho_r}{m}(m\mathbf{v}_r)u_r^\top into words in {W0,W0⊤}\{W_0, W_0^\top\} acting on the initial Gaussian vectors {u0,v0}\{u_0, \mathbf{v}_0\} follows the initialization-word expansion structure of Tensor Programs (Yang & Hu, 2021) and Chizat, Colombo, Fernández-Real, & Figalli (2024, Section 3). Because the input is scalar and initialization is Gaussian, every observable at finite step TT is a polynomial of fixed degree qT=4⋅7T+2q_T = 4\cdot 7^{T+2} in standard Gaussians; upgrading O(m−1)O(m^{-1}) Gaussian Poincaré variance bounds to tail bounds via Wiener-chaos hypercontractivity (Beckner, 1989) and expanding expectations via Wick pairings is a direct Gaussian-polynomial argument that produces the aT=2⋅7T+2+1a_T = 2\cdot 7^{T+2}+1 exponent, while identifying the limit recursion with the μ\muP limit (Lemmas D.4 and F.15) still invokes the convergence theorem of Yang & Hu (2021) as a black box.
    • All three models in Theorem 4.1 and both models in Theorem 6.1 have scalar input ξ∈R\xi\in\mathbb R. For every width mm, their predictors have the form gθt(m)(ξ)=ct(m)ξg_{\theta_t}^{(m)}(\xi)=c_t^{(m)}\xi in the linear cases or ct(m)ξ2c_t^{(m)}\xi^2 in the quadratic case. The internal parameter updates remain coupled and nonlinear, but the predictor class is one-dimensional and fixed across widths. Consequently, these main theorems do not capture learning new input-space directions or multidimensional representations. The abstract, Section 5.1, and Appendix A explicitly acknowledge this scope; its narrowness nevertheless limits the connection to the motivating feature-learning applications.
Questions for authors
  1. Could the linear case of Theorem 6.1 be strengthened via the exact closure (qt(m),st(m))(q_t^{(m)},s_t^{(m)})? The domain-weighted coefficient At(m)A_t^{(m)} is a smooth deterministic function of qt(m)q_t^{(m)}, so the same two-dimensional state still closes. Proposition I.1 suggests separating initialization concentration from deterministic sensitivity, with an explicit width condition and horizon-dependent constants. Are there additional obstacles to carrying this out for the adaptive update?

  2. Lemma D.2 uses ∣χt∣≤CtMt3|\chi_t|\le C_tM_t^3 to obtain Mt+1≤CtMt5M_{t+1}\le C_tM_t^5, leading to βT=3⋅5T\beta_T=3\cdot5^T in Theorem D.3. In a sufficiently-large-width regime where Δt=∣gθt(m)−gtref∣≤1\Delta_t=|g_{\theta_t}^{(m)}-g_t^{\mathrm{ref}}|\le1 and εt≤1\varepsilon_t\le1 are maintained inductively, one instead has ∣χt∣≤∣χtref∣+1|\chi_t|\le|\chi_t^{\mathrm{ref}}|+1 and MtM_t bounded by a constant times the reference particle maximum plus εt\varepsilon_t. Since that maximum is bounded by CtLm,δ2tC_tL_{m,\delta}^{2^t}, could an explicit width condition and a closed local bootstrap reduce the logarithmic exponents? Here Lm,δ=1+log⁡(2m/δ)L_{m,\delta}=1+\log(2m/\delta), and the constants may depend on the fixed horizon.

  3. The uniqueness-of-limits argument in Lemma F.15 identifies the word-closed limit with the Tensor Program limit. For a more concrete comparison of the two descriptions, could the authors also give a direct algebraic identification of the deterministic recursion in Proposition F.14, based on the closure in Lemma F.9 and the initial Wick calculations in Lemmas F.12–F.13, with the Tensor Program equations for the three-layer linear network?

6 comments

Q1. In the three-layer linear analysis (Appendix F), unrolling the discrete-time rank-one weight updates Wt+1=W0−∑r=0tρrm(mvr)ur⊤W_{t+1}=W_0-\sum_{r=0}^{t}\frac{\rho_r}{m}(mv_r)u_r^{\top} into words in W0,W0⊤W_0,W_0^{\top} acting on the initial Gaussian vectors u0,v0u_0,v_0 follows the initialization-word expansion structure of Tensor Programs (Yang & Hu, 2021) and Chizat, Colombo, Fernández-Real, & Figalli (2024, Section 3). Because the input is scalar and initialization is Gaussian, every observable at finite step TT is a polynomial of fixed degree qT=4⋅7T+2q_T=4\cdot7^{T+2} in standard Gaussians; upgrading O(m−1)O(m^{-1}) Gaussian Poincaré variance bounds to tail bounds via Wiener-chaos hypercontractivity (Beckner, 1989) and expanding expectations via Wick pairings is a direct Gaussian-polynomial argument that produces the aT=2⋅7T+2+1a_T=2\cdot7^{T+2}+1 exponent, while identifying the limit recursion with the μ\muP limit (Lemmas D.4 and F.15) still invokes the convergence theorem of Yang & Hu (2021) as a black box.

A1. We would like to clarify that the main difference between our work and Yang and Hu (2021) is asymptotic convergence versus quantitative finite-width guarantees. Yang and Hu establish the existence and characterization of infinite-width μ\muP training dynamics using Tensor Programs. However, their convergence theorem does not directly quantify how accurately a finite-width network approximates this limit at a given width mm and confidence level δ\delta. Our work addresses precisely this question by establishing non-asymptotic, high-probability bounds for the exact discrete-time training trajectories.

Importantly, these finite-width estimates are derived independently of the Tensor Program convergence theorem. Lemma F.15 invokes Yang and Hu's result only to identify our independently constructed deterministic reference with the standard μ\muP infinite-width limit. Thus, our contribution is not a new Gaussian inequality or the m−1/2m^{-1/2} exponent itself, but a rigorous non-asymptotic characterization of finite-width training dynamics.

Q2. All three models in Theorem 4.1 and both models in Theorem 6.1 have scalar input ξ∈R\xi\in\mathbb R. For every width m, their predictors have the form gθt(m)(ξ)=ct(m)ξg_{\theta_t}^{(m)}(\xi)=c_t^{(m)}\xi in the linear cases or ct(m)ξ2c_t^{(m)}\xi^2 in the quadratic case. The internal parameter updates remain coupled and nonlinear, but the predictor class is one-dimensional and fixed across widths. Consequently, these main theorems do not capture learning new input-space directions or multidimensional representations. The abstract, Section 5.1, and Appendix A explicitly acknowledge this scope; its narrowness nevertheless limits the connection to the motivating feature-learning applications.

A2. The nonasymptotic μ\muP theory is complicated even in one-dimensional case, and we mainly use one-dimension to illustrate that nonasymptotic guarantee is feasible. Nevertheless, we also provide guarantees for multidimensional data in certain cases. For example, Proposition I.1 already extends our analysis to two-layer linear networks with multidimensional inputs x∈Rdx\in\mathbb{R}^d. Specifically, for h=(d+1)log⁡9+log⁡(2/δ)h=(d+1)\log 9+\log(2/\delta) and m≥64hm\ge 64h, we establish with probability at least 1−δ1-\delta that $ \max_{0\le t\le T}\sup_{|x|_2\le X} |g_t^{(m)}(x)-\bar g_t(x)| \le 4XD_T\left(\sqrt{\frac{h}{m}}+\frac{h}{m}\right), $ where DTD_T explicitly captures the dependence on training dynamics and the learning rate. For deeper and nonlinear networks, the natural extension is to track additional matrix-valued and cross-time observables, combining this covariance-based approach with the Gaussian concentration and polynomial techniques developed in Appendix F. Although these extensions require further analysis, our results provide a concrete foundation beyond the scalar-input setting.

Q3. Could the linear case of Theorem 6.1 be strengthened via the exact closure (qt(m),st(m))(q_t^{(m)},s_t^{(m)})? The domain-weighted coefficient At(m)A_t^{(m)} is a smooth deterministic function of qt(m)q_t^{(m)}, so the same two-dimensional state still closes. Proposition I.1 suggests separating initialization concentration from deterministic sensitivity, with an explicit width condition and horizon-dependent constants. Are there additional obstacles to carrying this out for the adaptive update?

A3. Yes. The exact (qt,st)(q_t,s_t) closure allows us to extend the stability argument of Theorem 4.4 to adaptive domain reweighting.

Specifically, with b(q)=η∑iπi(q)Li′(q)b(q)=\eta\sum_i\pi_i(q)L_i'(q), the exact dynamics satisfy $ q_{t+1}=(1+b(q_t)^2)q_t-2b(q_t)s_t, \quad s_{t+1}=(1+b(q_t)^2)s_t-2b(q_t)q_t. $

We provide a concrete time-uniform example. Consider two domains with $ L_1(q)=q^2,\quad L_2(q)=(q-1)^2,\quad \lambda=1,\quad \eta=\frac{1+e}{8e}. $ The infinite-width trajectory starts at (0,1)(0,1) and reaches the stationary point (q⋆,s⋆)=(1/2,17/16)(q_\star,s_\star)=(1/2,17/16) after one update. Direct calculation gives the output-direction eigenvalue $ 1-2\eta s_\star A'(q_\star) =\frac{13-51/e}{64}\approx-0.09, $ where A(q)=∑iπi(q)Li′(q)A(q)=\sum_i\pi_i(q)L_i'(q).

The proof follows Theorem 4.4: the output error contracts geometrically, while the drift in sts_t is bounded by the summable output errors. Specifically, in a sufficiently small neighborhood of the stationary point, $ |q_{t+1}^{(m)}-q_\star| \le \tfrac15|q_t^{(m)}-q_\star|, \quad |s_{t+1}^{(m)}-s_t^{(m)}| \le \tfrac35|q_t^{(m)}-q_\star|. $ Gaussian initialization concentration and smoothness of the first update place the finite-width trajectory in this stable neighborhood for sufficiently large m. Thus $ \sup_{t\ge0}|q_t^{(m)}-\bar q_t| \le C\sqrt{\frac{\log(4/\delta)}{m}}. $ Since the softmax map is Lipschitz, this implies $ \sup_{t\ge0} |\pi_t^{(m)}-\pi_t^{(\infty)}|_\infty \le C'\sqrt{\frac{\log(4/\delta)}{m}}, $ with constants independent of TT.

Thus, time-uniform adaptive domain-weight stability can be rigorously established using the same contraction-and-summable-drift mechanism as Theorem 4.4. This is a new special-case extension under the assumptions of Theorem 6.1.

Q4. Lemma D.2 uses ∣χt∣≤CtMt3|\chi_t|\le C_tM_t^3 to obtain Mt+1≤CtMt5M_{t+1}\le C_tM_t^5, leading to βT=3⋅5T\beta_T=3\cdot5^T in Theorem D.3. In a sufficiently-large-width regime where Δt=∣gθt(m)−gtref∣≤1\Delta_t=|g_{\theta_t}^{(m)}-g_t^{\mathrm{ref}}|\le1 and εt≤1\varepsilon_t\le1 are maintained inductively, one instead has ∣χt∣≤∣χtref∣+1|\chi_t|\le|\chi_t^{\mathrm{ref}}|+1 and MtM_t bounded by a constant times the reference particle maximum plus εt\varepsilon_t. Since that maximum is bounded by CtLm,δ2tC_tL_{m,\delta}^{2^t}, could an explicit width condition and a closed local bootstrap reduce the logarithmic exponents? Here Lm,δ=1+log⁡(2m/δ)L_{m,\delta}=1+\log(2m/\delta), and the constants may depend on the fixed horizon.

A4. Yes. The local argument improves the sufficient logarithmic exponent at sufficiently large width, but it does not remove exponential dependence on TT.

Set L=1+log⁡(2m(T+1)/δ)L=1+\log(2m(T+1)/\delta) and work on the simultaneous reference concentration event from Lemma D.1. While εt≤1\varepsilon_t\le1, reference-particle comparison gives Mt≤cTL2tM_t\le c_TL^{2^t}. The inequalities in Lemma D.2 imply

$ \Delta_t \le c_Tm^{-1/2}L^{3\cdot2^t} + c_TL^{2\cdot2^t}\varepsilon_t, \varepsilon_{t+1} \le c_TL^{4\cdot2^t}\varepsilon_t + c_Tm^{-1/2}L^{5\cdot2^t}. $

Since ε0=0\varepsilon_0=0, the first coordinate-error exponent is p1=5p_1=5, and

$p_{t+1}= \max\{p_t+4\cdot2^t,,5\cdot2^t\}, p_t=4\cdot2^t-3. $

Consequently, for 1≤t≤T1\le t\le T,

$ \varepsilon_t \le C_T^{\mathrm{loc}}m^{-1/2}L^{4\cdot2^t-3}, \quad \Delta_t \le C_T^{\mathrm{loc}}m^{-1/2}L^{6\cdot2^t-3}. $

For T≥1T\ge1, choose CTlocC_T^{\mathrm{loc}} to dominate the finitely many induction constants and the initialization bound, and impose

$ C_T^{\mathrm{loc}}m^{-1/2} \left( 1+\log\frac{2m(T+1)}\delta \right)^{6\cdot2^T-3} \le\frac12. $ At each step, the coordinate estimate first establishes εt+1≤1/2\varepsilon_{t+1}\le1/2, which justifies the reference-particle bound at that next step; the output estimate then gives Δt+1≤1/2\Delta_{t+1}\le1/2. This closes the induction without assuming its conclusion.

Thus 6⋅2T−36\cdot2^T-3 replaces 3⋅5T3\cdot5^T as a sufficient output exponent in this local regime. The constant may differ from the original one, and the width condition is implicit in m. This is an additional refinement of sharpness, not a polynomial-time or time-uniform result.

Q5. The uniqueness-of-limits argument in Lemma F.15 identifies the word-closed limit with the Tensor Program limit. For a more concrete comparison of the two descriptions, could the authors also give a direct algebraic identification of the deterministic recursion in Proposition F.14, based on the closure in Lemma F.9 and the initial Wick calculations in Lemmas F.12–F.13, with the Tensor Program equations for the three-layer linear network?

A5. Thanks for this suggestion. The word-closed recursion in Appendix F can be directly identified with the Tensor Program (TP) recursion by matching their corresponding scalar observables.

Specifically, writing at=mvta_t=mv_t, the correspondence is $ (G_{t,r}^{xv})^\infty =\mathbb E[Z^{z_t}Z^{a_r}],\quad (K_{r,t}^{xx})^\infty =\mathbb E[Z^{u_r}Z^{u_t}],\quad (K_{t,r}^{vv})^\infty =\mathbb E[Z^{a_t}Z^{a_r}]. $ Consequently, both descriptions give the same output and update coefficient: $ \bar g_t(\xi)=\xi(G_{t,t}^{xv})^\infty, \quad \bar\rho_t =\xi_t\bigl(\xi_t(G_{t,t}^{xv})^\infty-y(\xi_t)\bigr). $

As a concrete comparison, consider the first update. The corresponding TP coordinate equations give $ Z^{a_1}=Z^{a_0}-\bar\rho_0 Z^{z_0}, \quad Z^{z_1}=Z^{z_0} -\bar\rho_0\bigl(Z^{W_0d_0}+Z^{a_0}\bigr), $ where the initial Gaussian contractions satisfy $ \mathbb E[Z^{a_0}Z^{W_0d_0}]=1,\quad \mathbb E[(Z^{W_0d_0})^2]=2,\quad \mathbb E[(Z^{z_0})^2]=1. $ Hence the TP equations yield $ \mathbb E[Z^{a_1}Z^{z_1}]=-3\bar\rho_0, \quad \mathbb E[(Z^{z_1})^2]=1+5\bar\rho_0^2. $ These agree exactly with the word-closed quantities $ (G_{1,1}^{xv})^\infty=-3\bar\rho_0, \quad (Q_1^z)^\infty=1+5\bar\rho_0^2 $ obtained from the initialization calculations in Proposition F.14.

Importantly, the nonzero correlation E[Za0ZW0d0]=1\mathbb E[Z^{a_0}Z^{W_0d_0}]=1 captures the dependence arising from reusing W0W_0 and W0⊤W_0^\top. Thus, the two formulations retain the same training-induced correlations rather than introducing independent matrix copies.

More generally, the corresponding TP moments satisfy the same polynomial update relations PtlinP_t^{\mathrm{lin}} constructed in Lemma F.9, with matching initialization. This provides a direct algebraic comparison of the two deterministic descriptions, while Lemma F.15 gives their formal identification through uniqueness of limits.

Reviewer_ETrvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper establishes the convergence rate of finite-width training trajectories toward their infinite-width limits for two-layer linear/quadratic and three-layer linear networks. It also provides empirical experiments to support cross-width domain-weight stability.

Critical strengths

The paper extends asymptotic µP theory to quantitative finite-width, high-probability guarantees, addressing an important limitation of existing theory. It handles training-induced dependencies between neurons, including the more challenging three-layer linear setting, using Gaussian concentration and polynomial representations.

Critical weaknesses
  1. Most guarantees consider scalar-input shallow networks with linear or quadratic activations and square loss. Their applicability to modern Transformer architectures remains unclear.
  2. Theorem 4.1 establishes finite-width convergence only for fixed training horizons, with an exponentially growing logarithmic exponent. These bounds may be vacuous even for modest numbers of training steps. Although Theorem 4.3 demonstrates that exponential error amplification is unavoidable in certain unstable settings, and Theorem 4.4 establishes time-uniform concentration in a special stable case, the paper does not characterize the practically relevant regime between these extremes. It remains unclear whether the results provide meaningful guarantees for realistic training durations
Questions for authors

Can the authors establish polynomial or time-uniform dependence on (T) under more general training-stability assumptions, or empirically quantify the finite-width approximation error as a function of both width and training horizon?

1 comment

Q1. Most guarantees consider scalar-input shallow networks with linear or quadratic activations and square loss. Their applicability to modern Transformer architectures remains unclear.

A1. Thank you for your suggestion. We would like to emphasize that deriving non-asymptotic, high-probability guarantees for exact discrete-time training under μ\muP is already nontrivial, even for shallow linear networks. Note that our paper is the first one to provide a nonasymptotic quantitative control of finite-width training trajectories around their μ\muP limits.

The simplified architectures isolate the central difficulty: training couples initially independent neurons through the empirical residual, and the three-layer matrix creates additional cross-coordinate and cross-time dependencies. Sections 5.1--5.3 explain how the analysis controls these dependencies. These difficulties do not disappear merely because the input is scalar or the activation is linear.

Moreover, the paper is not exclusively scalar-input: Proposition I.1 establishes a vector-input two-layer linear result. Theorem 4.4 also supplies a time-uniform cross-entropy example. These are concrete extensions, although not a general Transformer theorem. Section 7.2 explicitly presents GPT-style experiments as qualitative evidence beyond the proved setting. That being said, the applicability to transformers is empirically justified in Section 7.2, but deriving the transformer learning dynamics is a highly challenging task and we leave it to future work.

Q2. Theorem 4.1 establishes finite-width convergence only for fixed training horizons, with an exponentially growing logarithmic exponent. These bounds may be vacuous even for modest numbers of training steps. Although Theorem 4.3 demonstrates that exponential error amplification is unavoidable in certain unstable settings, and Theorem 4.4 establishes time-uniform concentration in a special stable case, the paper does not characterize the practically relevant regime between these extremes. It remains unclear whether the results provide meaningful guarantees for realistic training durations.

A2. A fixed-horizon theorem is not merely an infinite-width convergence statement. Theorem 4.1 does not conceal polynomial width losses. For every fixed T and δ\delta, the bound tends to zero with width, even though its worst-case constants can be too large for practical calibration.

The additional results do more than restate this upper bound. Theorem 4.3 proves that an Ω(3T)\Omega(3^T) fluctuation prefactor is necessary under the square-loss setting. Thus, a general polynomial-in-T guarantee cannot be satisfied under the same assumptions. This concerns the fluctuation prefactor, not optimality of the logarithmic exponent. Conversely, Theorem 4.4 proves time-uniform concentration in a stable cross-entropy example. Note that Theorem 4.4 is indeed inside a practically relevant regime since we use cross-entropy loss instead of square loss, which is the default choice for classification problems in deep learning.

Q3. Can the authors establish polynomial or time-uniform dependence on T under more general training-stability assumptions, or empirically quantify the finite-width approximation error as a function of both width and training horizon?

A3. Our Theorem 4.4 already establishes a time-uniform finite-width concentration bound under stable cross-entropy training: $ \sup_{t\ge0}\max_i |g_t^{(m)}(\xi_i)-\bar g_t(\xi_i)| \le C\sqrt{\frac{\log(4/\delta)}{m}}, $ where both C and the required width threshold are independent of T. The key stability conditions are contraction in the output direction and summable drift in the neutral direction.

Our experiments further investigate cross-width stability beyond the theoretical setting. Figures 1 and 2 study the effect of width, while Figure 6 examines training horizons T∈[100,5000]T\in[100,5000]. Note that Figure 6 shows that the KL discrepancy between small and large models only slightly increases in terms of T, which suggests that the the cross-width domain-weight stability only has a mild dependence on the time horizon T. These experiments complement our theoretical results, while extending general time-uniform guarantees remains an important direction.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of manuscript revision and rebuttal results

Dear Area Chair and Reviewers,

Thank you for your detailed feedback. We have added Appendix J to the revised manuscript and reported additional theoretical clarifications and experimental results in our discussion replies.

New time-uniform result in Appendix J. Appendix J is marked in blue in the revised PDF. Theorem J.1 extends the original collinear, one-step example in Theorem 4.4 to spanning inputs and a range of fixed learning rates under binary label-smoothed cross-entropy. Corollary J.2 gives a sufficient proxy width that controls the cross-width logit gap at every training step for a prescribed tolerance and confidence level. The sufficient width is independent of the training horizon.

Theoretical questions addressed in the rebuttal. We clarified the worst-case exponential amplification under square loss in Theorem 4.3 and explained how Appendix J controls the evolving layer moments to establish stability throughout training. We also explained how Appendix F derives quantitative finite-width estimates and how Tensor Program convergence identifies the reference trajectory. Our replies address adaptive moment closure, the local bootstrap, and the definition of BTB_T in Theorem 6.1.

Reweighting and additional experiments. We clarified that Theorem 6.1 concerns agreement between domain weights learned adaptively at different widths over a fixed horizon. In the discussion, we reported synthetic diagnostics showing that teacher weights can be close to uniform at λ=1\lambda=1 and that cross-width agreement can also occur with nonuniform teacher weights at λ=10\lambda=10. We also reported GPT runs across five widths and three seeds. These results quantify agreement with the finite width-1024 reference, variability across seeds, and the differences that remain across widths.

As the discussion period comes to a close today, we would appreciate any further feedback on the revised manuscript or our responses. Thank you again for your time and consideration.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.