- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper develops finite-width concentration estimates for gradient descent under maximal update parameterization. The analysis covers two-layer linear and quadratic networks and three-layer linear networks with scalar inputs.
Critical strengths
The paper derives high-probability finite-width concentration estimates for several shallow networks under μP. Theorems 4.3 and 4.4 distinguish unstable square-loss dynamics from a specific stable cross-entropy setting.
Critical weaknesses
[Note added Nov 14, 2026: I withdraw the struck-through criticism below regarding . I misread line 367: Theorem 6.1 already includes the square root in defining , matching Appendix G, so the bound follows directly from Lemma G.3. My remaining weaknesses are unchanged.]
Theorem 6.1 defines as the maximum infinite-width domain loss, , and states the domain-weight discrepancy bound as . Appendix G instead defines as the maximum root-mean-square residual. These are different quantities.
For the definition in Theorem 6.1, the Cauchy–Schwarz argument in Lemma G.3 yields . The displayed bound in the main theorem does not follow from this argument for arbitrary positive . The authors must correct the definition or the inequality and verify the consequences for the cross-width stability statement. The explicit theorem is not consistently stated.
Theorem 4.1 the displayed estimate does not provide a meaningful quantitative prescription for selecting proxy-model widths over ordinary training durations.
Theorem 4.3 establishes that exponentially amplified fluctuations are possible under the square-loss assumptions. The time-uniform result in Theorem 4.4 concerns a special data configuration and learning rate for which the infinite-width dynamics reach a stable fixed point after one update. It does not establish comparable stability for general cross-entropy training.
The application to domain reweighting consequently remains limited. Theorem 6.1 controls best-response domain weights along jointly trained two-layer trajectories. It does not establish that weights learned by a smaller proxy model improve the training of a separate larger model. Moreover, the GPT experiments use exponentiated-gradient updates and MuAdamW rather than the exact best-response gradient-descent dynamics analyzed in the theorem. These experiments do not validate the stated finite-width guarantees for that training procedure.
Questions for authors
- Which definition of is intended in Theorem 6.1? The main theorem uses the maximum squared loss, while Appendix G uses the maximum root-mean-square residual. Can the authors provide the corrected bound and confirm that all dependent results remain valid?
- Can the authors evaluate the concentration bound in Theorem 4.1 for representative training horizons and failure probabilities, including the constants ? At what model widths does the resulting bound become smaller than a useful error tolerance?
- What additional assumptions would yield a non-vacuous cross-width guarantee for training horizons that grow with width? Theorem 4.3 rules out uniform control under the present assumptions in general.
- How does Theorem 6.1 apply to the iterative domain-weight updates and MuAdamW optimizer used in the GPT experiments? Is there a bound on the difference between those updates and the exact KL best responses required by the theorem?
11 comments
Q3. Can the authors evaluate the concentration bound in Theorem 4.1 for representative training horizons and failure probabilities, including the constants ? At what model widths does the resulting bound become smaller than a useful error tolerance?
A3. Theorem 4.1 establishes
\le C_Tm^{-1/2} \left(1+\log\frac{2m(T+1)}{\delta}\right)^{a_T}, \qquad a_T=2\cdot7^{T+2}+1.$$ Although the proof establishes that $C_T$ is finite, it does not numerically calibrate this constant. We therefore conduct a simulation to estimate its empirical counterpart. We consider the two-layer linear $\mu$P model under square loss, with Gaussian initialization, repeated training data $(\xi,y)=(1,1)$, and learning rate $\eta=0.01$. For each width $m\in\{128,512,2048,8192\}$, we run $20{,}000$ independent simulations and measure \$ E_{m,T}=\max_{0\le t\le T} |g_t^{(m)}(1)-\bar g_t(1)|. \$ We estimate the effective concentration prefactor by \$ \widehat K_{T,\delta} =\max_m\sqrt m\, \widehat Q_{1-\delta}(E_{m,T}), \$ where $\widehat Q_{1-\delta}$ is the empirical quantile. The results are: | Training horizon $T$ | $\delta=0.05$ | $\delta=0.01$ | |---:|---:|---:| | 1 | 1.970 | 2.636 | | 10 | 1.971 | 2.636 | | 100 | 1.972 | 2.636 | | 500 | 1.972 | 2.636 | For tolerance $\tau=0.1$, the empirical error is below $\tau$ at width $m=512$ for 95% confidence and $m=2048$ for 99% confidence, among the tested widths. The effective prefactor remains nearly constant with $T$ in this stable example. We also estimate the constant using the exact logarithmic factor in Theorem 4.1: \$ \widehat C_{T,\delta} =\max_m \frac{\sqrt m\,\widehat Q_{1-\delta}(E_{m,T})} {\left(1+\log\frac{2m(T+1)}{\delta}\right)^{a_T}}. \$ For $\delta=0.05$, this gives $\log_{10}\widehat C_{1,\delta}\approx-693.6$ and $\log_{10}\widehat C_{2,\delta}\approx-4932.0$. These extremely small values illustrate the conservatism of the theorem's logarithmic exponent. These estimates are empirical, not certified values of the universal $C_T$. The experiment demonstrates finite-width accuracy at practical widths, while obtaining rigorous numerical width thresholds from Theorem 4.1 requires sharper explicit control of its proof constants. **Q4. What additional assumptions would yield a non-vacuous cross-width guarantee for training horizons that grow with width? Theorem 4.3 rules out uniform control under the present assumptions in general.** **A4.** The key additional requirement is stability of the training dynamics, which prevents finite-width errors from being amplified over time. Theorem 4.3 shows that such stability cannot be guaranteed under the general square-loss assumptions. Theorem 4.4 already establishes sufficient conditions: under its specified data, label-smoothed cross-entropy loss, and learning rate, the output direction is locally contractive, while the drift in the remaining neutral direction is summable. Gaussian initialization concentration and a local bootstrap then yield \$ \sup_{t\ge0}\max_i |g_t^{(m)}(\xi_i)-\bar g_t(\xi_i)| \le C\sqrt{\frac{\log(4/\delta)}{m}}, \$ for $m\ge C_0\log(4/\delta)$, with constants independent of T. Applying this bound to two widths and using the triangle inequality gives \$ \sup_{t\ge0}\max_i |g_t^{(m_s)}(\xi_i)-g_t^{(m_l)}(\xi_i)| \le C\sqrt{\log(8/\delta)} \left(m_s^{-1/2}+m_l^{-1/2}\right) \$ with probability at least $1-\delta$. Thus, Theorem 4.4 already provides a non-vacuous cross-width guarantee for arbitrarily growing training horizons.Q5. How does Theorem 6.1 apply to the iterative domain-weight updates and MuAdamW optimizer used in the GPT experiments? Is there a bound on the difference between those updates and the exact KL best responses required by the theorem?
A5. We clarify that adaptive domain reweighting is already an essential part of Theorem 6.1, rather than a departure from the theory. In Algorithm 1, the domain weights are updated at every training step through the exact KL-regularized best response $ \pi_{i,t}^{*} =\frac{\exp(\lambda\ell_{i,t})} {\sum_j\exp(\lambda\ell_{j,t})}, $ and the resulting weights determine the subsequent P gradient-descent update. Theorem 6.1 explicitly controls this coupling between domain weights and network training.
The GPT experiments in Section 7.2 follow the same KL-regularized reweighting objective but use exponentiated-gradient ascent to update the weights iteratively. For a uniform domain prior, the update takes the form $ \pi_{i,t+1}\propto \pi_{i,t}^{1-\eta_\pi/\lambda} \exp(\eta_\pi\ell_{i,t}). $ For fixed domain losses, its stationary point coincides with the exact best response in Theorem 6.1. Thus, the theoretical and experimental weight updates share the same optimization objective, although the theorem does not provide a finite-step tracking bound for their discrepancy.
Similarly, our theoretical guarantees apply to P gradient descent, whereas we use MuAdamW in the GPT experiments because AdamW-style optimizers are standard for Transformer training. Extending the guarantees to MuAdamW is beyond the current analysis.
Importantly, Section 7.1 provides more directly comparable synthetic experiments using Algorithm 1. For two-layer linear networks with input dimension , the learned domain weights closely agree across widths, achieving for student widths after training steps. The ReLU experiments and the additional quadratic-activation comparison further examine this phenomenon. Although the multidimensional and ReLU settings extend beyond the assumptions of Theorem 6.1, these experiments directly test its predicted cross-width stability under joint domain reweighting and P training.
Therefore, the synthetic experiments provide controlled empirical support for the theoretical mechanism, while the GPT experiments demonstrate that similar cross-width domain-weight agreement persists with practical Transformer architectures and MuAdamW. We distinguish this empirical extension from a formal finite-width guarantee for the GPT training procedure.
Q1. Theorem 6.1 defines BT as the maximum infinite-width domain loss, BT=maxt,iℓi,t(∞), and states the domain-weight discrepancy bound as 2λ(BT+ε)ε. Appendix G instead defines (BT) as the maximum root-mean-square residual. These are different quantities. For the definition in Theorem 6.1, the Cauchy–Schwarz argument in Lemma G.3 yields λ(2BTε+ε2). The displayed bound in the main theorem does not follow from this argument for arbitrary positive BT. The authors must correct the definition or the inequality and verify the consequences for the cross-width stability statement. The explicit theorem is not consistently stated. Theorem 4.1 the displayed estimate does not provide a meaningful quantitative prescription for selecting proxy-model widths over ordinary training durations. Theorem 4.3 establishes that exponentially amplified fluctuations are possible under the square-loss assumptions. The time-uniform result in Theorem 4.4 concerns a special data configuration and learning rate for which the infinite-width dynamics reach a stable fixed point after one update. It does not establish comparable stability for general cross-entropy training. The application to domain reweighting consequently remains limited. Theorem 6.1 controls best-response domain weights along jointly trained two-layer trajectories. It does not establish that weights learned by a smaller proxy model improve the training of a separate larger model. Moreover, the GPT experiments use exponentiated-gradient updates and MuAdamW rather than the exact best-response gradient-descent dynamics analyzed in the theorem. These experiments do not validate the stated finite-width guarantees for that training procedure.
A1. The alleged inconsistency in BT is not present in the supplied manuscript. Theorem 6.1 already defines BT using ℓi,t(∞), exactly as Appendix G does. With this definition, the displayed inequality follows from Lemma G.3; we give the calculation in A2 below.
The other points concern the scope or tightness of guarantees, not a contradiction in their proofs. Our contribution is a finite-width training-dynamics theorem around the μP limit, not a universal theorem for every architecture, optimizer, or training duration. Theorem 4.3 establishes a genuine obstruction to a general polynomial-time bound, while Theorem 4.4 demonstrates a stability mechanism that removes horizon dependence in its stated setting. Proposition I.1 provides an additional explicit sensitivity-based bound.
The domain-weight result is also more than post-processing a nonadaptive output theorem. The weights change the subsequent training trajectory. Appendix G controls this feedback through the coefficient-error estimate in Lemma G.3 and the adaptive bootstraps in Lemmas G.4--G.5. Theorem 6.1 then provides a quantitative cross-width guarantee for these jointly trained trajectories. A performance guarantee for a fresh target trained with frozen proxy weights is a different result, explicitly excluded by Remark 6.2.
Finally, Section 7.2 identifies the GPT procedure as a practical variant beyond the assumptions, not an instantiation of the proved GD theorem. We will keep that distinction prominent. Given the actual definition of BT and the explicitly stated scope, we respectfully ask that the soundness assessment be reconsidered separately from requests for broader or sharper results.
Q2. Which definition of BT is intended in Theorem 6.1? The main theorem uses the maximum squared loss, while Appendix G uses the maximum root-mean-square residual. Can the authors provide the corrected bound and confirm that all dependent results remain valid?
A2. The main theorem already uses the maximum root-mean-square residual, not the maximum squared loss. In the supplied PDF, Theorem 6.1 reads
$ B_T:= \max_{0\le t\le T}\max_{i\in[K]} \sqrt{\ell_{i,t}^{(\infty)}}<\infty. $
Appendix G uses the same definition. Thus, no change to the theorem is required on this point.