acceptodds
Under review as a conference paper at ICLR 2027

Adam as Magnitude-Weighted Sign Descent: Dimension-Free Convergence under Weak Assumptions

Abstract

We develop a convergence analysis of fully bias-corrected Adam through its *magnitude-weighted stochastic sign* representation. The key step controls the accumulated wrong-sign loss while retaining the joint randomness of Adam's update magnitude, momentum, and gradient noise. Under -smoothness and conditional -affine second moments, we prove that the expected average gradient norm is at most , where has no dependence on explicit dimension factor and the rate has no extra ; under global point-pair -smoothness, a stopped analysis yields the same exponent with high probability. We further instantiate the theory for fan-in-scaled LayerNorm/RMSNorm networks with fixed depth and increasing hidden width. For a concrete fixed-affine family, explicit width–horizon conditions guarantee that the Adam iterates remain in a width-uniform parameter region throughout training, yielding a convergence constant independent of the total parameter count. To our knowledge, this is the first Adam convergence-rate result establishing dimension-free optimization for an explicit growing family of normalized networks.

Primary area:
optimization
Keywords:
Adam, Sign-like Descent, Dimensio-Free Convergence, Weak Assumptions

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GZDWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes Adam with full bias correction by writing its update as xt+1=xt−γut∘Sign(mt)x_{t+1}=x_t-\gamma u_t\circ\mathrm{Sign}(m_t) with ut=∣m^t∣/(v^t+ϵ)u_t=|\hat m_t|/(\sqrt{\hat v_t}+\epsilon). Under LL-smoothness and p-affine variance, with 1−β1=T−1/21-\beta_1=T^{-1/2}, γ=C2/T\gamma=C_2/\sqrt T and any β2\beta_2, Corollary 4.3 gives 1T∑tE∥∇F(xt)∥2≤CT−1/4\frac1T\sum_t E\Vert\nabla F(x_t)\Vert_2\le CT^{-1/4}, where CC has no explicit factor of dd and no log⁡T\log T. Corollary 4.4 gives a high-probability version under global (L0,L1,q)(L_0,L_1,q)-smoothness. Section 5 shows that for fan-in-scaled LayerNorm/RMSNorm networks of growing width the problem constants stay bounded (Theorem 5.1). If the width grows with the horizon, the iterates stay in this region (Theorem 5.2). The paper has no experiments.

Critical strengths

S1. The proofs are correct. I checked the main chain in Appendices C, D, E and F.6 and found no error. Unlike many earlier analyses, the paper keeps both bias corrections of Adam, as in the standard implementation. The final step to the ℓ2\ell_2 norm (Appendix D, eqs. (49) to (54)) is done carefully and does not hide a factor d\sqrt d.

S2. Bounded gradients are removed under affine-type noise. Earlier analyses in the same regime bound the denominator of Adam pathwise and so need bounded gradients. Here a second-moment bound on the largest denominator, eq. (53), is enough. The paper is also careful about its scope. It says that its noise improvement "should not be read as a uniform relaxation of every assumption" (page 4) and that the bounds are "not uniform as ε→0" (page 8). The constants in Section 5 are explicit.

Critical weaknesses

W1. The analysis covers momentum SGD with a bounded preconditioner, not the adaptive regime of Adam, and its main claims are already known. The step size condition (10) gives γ≤ϵ/(32LT)\gamma\le\epsilon/(32L\sqrt T). So every coordinate moves with an effective step of at most 1/(32LT)1/(32L\sqrt T), which is the standard SGDM step. The proof uses only three facts about the denominator v^t+ϵ\sqrt{\hat v_t}+\epsilon. It is at least ϵ\epsilon, its largest entry bounds all entries, and that largest entry has bounded expectation. The structure of v^t\hat v_t and the value of β2\beta_2 enter only through eq. (52), which uses that v^t\hat v_t is an average of past gk2g_k^2. This is why any β2\beta_2 is allowed. The sign representation of Lemma 4.1 is the identity ut∘Sign(mt)=m^t/(v^t+ϵ)u_t\circ\mathrm{Sign}(m_t)=\hat m_t/(\sqrt{\hat v_t}+\epsilon). The same proof goes through with ⟨∇F,m^t/(v^t+ϵ)⟩\langle\nabla F,\hat m_t/(\sqrt{\hat v_t}+\epsilon)\rangle in place of T1,t−T2,tT_{1,t}-T_{2,t}.

The price is in the constant. From (10) to (14), CW≥64LΔ/ϵC_W\ge64L\Delta/\epsilon and CC grows like ϵ−1/2\epsilon^{-1/2}. With L=Δ=σ0=1L=\Delta=\sigma_0=1 and the largest admissible C2C_2, I get C≈54C\approx54 at ϵ=1\epsilon=1, C≈1.4⋅103C\approx1.4\cdot10^3 at ϵ=10−3\epsilon=10^{-3} and C≈4⋅105C\approx4\cdot10^5 at the default ϵ=10−8\epsilon=10^{-8}. The admissible step is then γT≈3⋅10−10\gamma\sqrt T\approx3\cdot10^{-10}, while practical learning rates are about 10−310^{-3}. Earlier bounds such as that of Défossez et al. pay dln⁡(1/ϵ)d\ln(1/\epsilon) but allow learning rates that do not scale with ϵ\epsilon. So "no explicit dd" is bought by moving to a regime where Adam behaves like SGD.

The headline features are also in prior work. Theorem 4.1 of "Convergence of Adam under Relaxed Assumptions" (Li, Rakhlin, Jadbabaie) covers bias-corrected Adam with no condition on β2\beta_2, no dimension factor and no logarithm. Its complexity grows as O(λ−2)O(\lambda^{-2}), where their λ\lambda plays the role of ϵ\epsilon. Table 1 lists only their Theorem 4.2 with (ln⁡T)13/16(\ln T)^{13/16}. "A Novel Convergence Analysis for Algorithms of the Adam Family" (Guo, Xu, Yin, Jin, Yang) treats Adam as momentum SGD with a denominator between two bounds, under affine variance and without a dimension factor. It is not cited. The same holds for "Momentum Improves Normalized SGD" (Cutkosky, Mehta), whose proof has the same shape as Theorem 4.2. After these comparisons, the new part of Section 4 is the removal of bounded gradients via eq. (53). This is correct but narrow.

W2. The network result holds only in a lazy regime with bounded gradients. On the region Θ\Theta, Theorem 5.1 bounds the gradient by G∗G_*, eq. (19), and the proof of Theorem 5.2 uses this bound, eq. (100). So the main advantage of Section 4, no bounded gradients, does not apply to the network result. With bounded gradients, the results of Li et al. and Guo et al. already apply on Θ\Theta with the width-free constants of Theorem 5.1. The claim to be first therefore rests on the containment argument in Appendix F.6. That argument needs nmin⁡≥cdet⁡Tn_{\min}\ge c_{\det}T, eq. (104), or nmin⁡≥chpT/ρn_{\min}\ge c_{hp}\sqrt T/\rho. The total displacement is O(T)O(\sqrt T) while leaving the region needs rnmin⁡r\sqrt{n_{\min}}. So the hidden width must grow with the horizon, and the scaled weights stay close to initialization. This is an NTK-type regime. In the fan-in-scaled parameterization the per-coordinate gradients shrink with width, so at fixed ϵ\epsilon we get v^t≪ϵ\sqrt{\hat v_t}\ll\epsilon and Adam again reduces to SGDM. The sign-like regime where Adam is used in practice, with ∥ut∥2\Vert u_t\Vert_2 of order d\sqrt d, is not covered. Theorem 5.2 also fixes the affine normalization parameters, has no residual connections or attention, uses no weight decay, and ties β1=1−T−1/2\beta_1=1-T^{-1/2} to the horizon.

Questions for authors

Q1. Regarding W1, which step of the proof needs the sign representation or the structure of v^t\hat v_t? Would the same result follow from a standard preconditioned SGDM analysis with a diagonal preconditioner between 1/max⁡jDt(j)1/\max_jD_t^{(j)} and 1/ϵ1/\epsilon?

Q2. Regarding W1, please state the explicit dependence of CC on ϵ\epsilon and compare the result with Theorem 4.1 of Li et al. and Theorem 1 of Guo et al. What does Corollary 4.3 give at ϵ=10−8\epsilon=10^{-8} and a learning rate of 10−310^{-3}?

Q3. Regarding W2, can the containment in Theorem 5.2 hold when the width does not grow with TT, or when Adam is in the sign-like regime v^t≫ϵ\sqrt{\hat v_t}\gg\epsilon?

Q4. A suggestion that would raise my score. The paper has no experiments. A small study on a normalized MLP or a small Transformer could check three things. First, whether the bound is informative at standard ϵ\epsilon and learning rates. Second, how far the scaled weights move from initialization when nmin⁡≥Tn_{\min}\ge T. Third, whether the measured constant is independent of width in the sign-like regime.

Q5. Clarification. In eq. (45), p-affine variance seems to reduce to affine variance with modified constants. Does p<2p<2 add anything to Theorem 4.2? Also, global (L0,L1,q)(L_0,L_1,q)-smoothness for all pairs allows at most linear growth of the gradient. Which standard examples of generalized smoothness does Corollary 4.4 cover?

6 comments
AuthorsOpenReview ↗

Responses to Reviewer GZDW (Part I)

Thank you for carefully reading the paper and checking the proofs, and suggesting the normalized-MLP experiment. We ran 135 trajectories across seven widths. The effective denominator improved the step-size estimate by factors of 296 to 22,660, although the estimate remained below the learning rate used in every run. Across the same widths, the empirical stationarity constant fell from 0.28151 to 0.11373.

1. Response to W1, Q1, and Q2: Preconditioned SGDM and the dependence on ϵ\epsilon

We agree that preconditioned SGDM can establish convergence without explicit dependence on dd, as demonstrated by Guo et al. and Li et al. However, their proofs use worst-case bounds on the diagonal preconditioner, whereas ours retains the fine-grained structure of ut=∣m^t∣/(v^t+ϵ)u_t=|\hat m_t|/(\sqrt{\hat v_t}+\epsilon). In Eq. (39), the same cumulative update energy ∑tE∥ut∥22\sum_t\mathbb E\|u_t\|_2^2 controls gradient drift and descent errors, yielding weaker dependence on ϵ\epsilon.

Table 1. Comparison of learning-rate restrictions and convergence bounds.

Analysis   Learning rate  Average ℓ2\ell_2 gradient bound 
Guo et al., Thm. 1O(min⁡{ϵ2,ϵ3/2T−1/2})O(\min\{\epsilon^2,\epsilon^{3/2}T^{-1/2}\})O(ϵ−3/4T−1/4)O(\epsilon^{-3/4}T^{-1/4})
Li et al., Thm. 4.1O(ϵ3/2T−1/2)O(\epsilon^{3/2}T^{-1/2})O(ϵ−3/4T−1/4)O(\epsilon^{-3/4}T^{-1/4})
Ours (submitted)O(ϵT−1/2)O(\epsilon T^{-1/2})O(ϵ−1/2T−1/4)O(\epsilon^{-1/2}T^{-1/4})
Ours (refined)O(ρT−1/2)O(\rho T^{-1/2})O(ρ−1/2T−1/4)O(\rho^{-1/2}T^{-1/4})

The comparison fixes 1−β1=T−1/21-\beta_1=T^{-1/2} and other problem constants, considering sufficiently large TT. The prior bounds are matched-schedule specializations, not optimal guarantees over all parameter choices. Guo et al. require bounded preconditioners, while Li et al. assume bounded noise and provide a high-probability result.

Further improvement through the effective denominator. Our original proof conservatively uses the minimum denominator ϵ\epsilon. We improve this by introducing the update-energy-weighted average

ρ=∑tE∑j(v^t,j+ϵ)ut,j2∑tE∥ut∥22≥ϵ.\rho=\frac{\sum_t\mathbb E\sum_j(\sqrt{\hat v_{t,j}}+\epsilon)u_{t,j}^2}{\sum_t\mathbb E\|u_t\|_2^2}\ge\epsilon.

This relaxes the sufficient learning-rate condition from γT≤ϵ/(LA)\gamma\sqrt T\le\epsilon/(LA) to γT≤ρ/(LA)\gamma\sqrt T\le\rho/(LA), where A=max⁡(32,60pσ12)A=\max(32,60p\sigma_1^2).

Crucially, this refinement cannot be directly applied to the preconditioned SGDM, since their worst-case preconditioner bounds discard the dependence on utu_t. Moreover, ρ\rho is an energy-weighted average, not a coordinatewise lower bound. Recovering this improvement requires retaining the coupled update structure, as in our analysis.

To evaluate this refinement, we independently trained each width 15 times using different minibatch sampling seeds, with initialization fixed. The reported ρ^\hat\rho is the pooled update-energy-weighted estimate across these 15 runs, rather than the arithmetic mean of individual ratios.

Table 2. Empirical improvement from ϵ\epsilon to ρ\rho (15 training runs per width).

Width ρ^/ϵ\hat\rho/\epsilon   ϵ\epsilon  ρ^\hat\rho 
25622,660×10−810^{-8}2.266×10−42.266\times10^{-4}
51213,496×10−810^{-8}1.350×10−41.350\times10^{-4}
1,0246,886×10−810^{-8}6.886×10−56.886\times10^{-5}
2,0483,029×10−810^{-8}3.029×10−53.029\times10^{-5}
4,0961,434×10−810^{-8}1.434×10−51.434\times10^{-5}
8,192512×10−810^{-8}5.115×10−65.115\times10^{-6}
16,384296×10−810^{-8}2.958×10−62.958\times10^{-6}

The effective denominators improve the denominator-dependent learning-rate allowance by 296–22,660× across the tested widths. These are empirical estimates, not certified lower bounds. We have included the refined proof and clarify the comparison in the revision.

Additionally, We thank the reviewer for suggesting A Novel Convergence Analysis for Algorithms of the Adam Family (Guo et al.) and Momentum Improves Normalized SGD (Cutkosky and Mehta). We have added both references to the revised manuscript. Cutkosky and Mehta analyze momentum with globally normalized SGD updates, while our work considers fully bias-corrected Adam with coordinatewise, history-dependent second-moment normalization. Although both analyses exploit momentum tracking, their method does not address the stochastic coupling between Adam's adaptive update magnitudes and tracking errors, which is central to our analysis and the ρ\rho refinement. We have clarified these distinctions in the revision.

AuthorsOpenReview ↗

Responses to Reviewer GZDW (Part II)

2. Response to W1 and Q4(a): Practical Learning Rates and the Theory–Practice Gap

We agree that the submitted learning-rate condition is conservative. To investigate this, we conducted normalized-MLP experiments with ϵ=10−8\epsilon=10^{-8} and T=4096T=4096. A preliminary search over {10−5,2×10−5,5×10−5,10−4,2×10−4,5×10−4,10−3,2×10−3,5×10−3}\{10^{-5},2\times10^{-5},5\times10^{-5},10^{-4},2\times10^{-4},5\times10^{-4},10^{-3},2\times10^{-3},5\times10^{-3}\} identified γ=10−4\gamma=10^{-4} as the best-performing setting. We fixed this learning rate across seven widths, with 15 independent minibatch-sampling runs per width.

As discussed in our response to Q1–Q2, our effective-denominator refinement relaxes the sufficient condition to γ≤ρ/(LAT)\gamma\le\rho/(LA\sqrt T), where A=max⁡(32,60pσ12)A=\max(32,60p\sigma_1^2). Table 3 evaluates this condition using empirical ρ^\hat\rho, local curvature L^\hat L, and A=32A=32.

Table 3. Practical learning rates and refined step-size diagnostics.

WidthRefined LR cap γ^max⁡\hat\gamma_{\max}Actual LR / capMeasured mean gradient norm
2569.126e-7109.60.035189
5125.053e-7197.90.026414
1,0242.683e-7372.70.020634
2,0489.669e-81,034.30.017932
4,0965.614e-81,781.30.017944
8,1922.086e-84,794.60.015140
16,3849.679e-910,331.60.014216

The refined cap is γ^max⁡=ρ^/(32L^T)\hat\gamma_{\max}=\hat\rho/(32\hat L\sqrt T). The empirical denominators and local curvature estimates do not certify the global theorem conditions.

Comparison with prior analyses. With ϵ=10−8\epsilon=10^{-8} and 1−β1=1/641-\beta_1=1/64, the preconditioner-endpoint restrictions in Guo et al. and Li et al. involve terms as small as O(ϵ2)O(\epsilon^2) and O(ϵ3/2(1−β1))O(\epsilon^{3/2}(1-\beta_1)), corresponding to scales of 10−1610^{-16} and 1.56×10−141.56\times10^{-14} before other constants. Compared with a 10−1410^{-14} reference scale, our refined empirical caps of 9.7×10−99.7\times10^{-9} to 9.1×10−79.1\times10^{-7} are approximately 10610^6–10810^8 times larger. Thus, the nominal gap to the practical learning rate is substantially reduced, although this is an indicative scale comparison rather than a certified numerical comparison across different theorems.

Nevertheless, our refined cap remains 110–10,332× smaller than the practical learning rate. The refined stationarity expressions also remain much larger than the observed gradient norms. We therefore investigated where the remaining conservatism arises.

Table 4. Quantifying conservatism in curvature and momentum-error estimates.

WidthLocal / directional curvatureDrift utilization (envelope factor)Descent margin: observed / conservativeDescent-cost inflation
25618.0×0.697 (1.43×)0.860 / 0.5822.99×
51217.2×0.598 (1.67×)0.801 / 0.4132.95×
1,02416.4×0.460 (2.18×)0.719 / 0.1413.05×
2,04824.9×0.382 (2.62×)0.648 / −0.3693.89×
4,09621.9×0.280 (3.57×)0.557 / −1.0084.53×
8,19237.8×0.231 (4.32×)0.500 / −2.3816.76×
16,38441.0×0.176 (5.70×)0.402 / −4.2388.76×

Curvature compares local estimates with realized positive-remainder curvature. Drift utilization is the ratio of actual vector drift to its scalar norm envelope; the parenthesized factor is its inverse. Descent-cost inflation is (1−Mconservative)/(1−Mobserved)(1-M_{\rm conservative})/(1-M_{\rm observed}), calculated from the reported mean margins. These are distinct diagnostics, not independent multipliers of the admissible learning rate.

Why does the gap remain? First, the global smoothness estimate is conservative along actual update directions: the measured directional curvature is 16–41× smaller than the local curvature estimate. Second, triangle and Cauchy–Schwarz inequalities discard momentum-drift cancellations, with scalar envelopes overestimating realized drift by 1.4–5.7×.

Together, these conservative substitutions inflate the measured normalized descent cost by approximately 3–9×. At width 16,384, the observed descent margin is 0.402, whereas the conservative estimate gives −4.238. The latter cannot establish descent despite the positive observed margin.

These diagnostics identify substantial additional slack beyond the denominator bound, but their factors cannot simply be multiplied to close the remaining learning-rate gap. Our analysis has significantly narrowed the gap relative to earlier worst-case preconditioner approaches, while a fully practical learning-rate guarantee remains open. We have clarified these findings and limitations in the revision.

AuthorsOpenReview ↗

Responses to Reviewer GZDW (Part III)

3. Response to W2, Q3, and Q4(b–c): Network Containment and the Sign-like Regime

We agree that Theorem 5.1 establishes bounded gradients within a specified region. Our contribution is to derive width-independent smoothness and noise constants for normalized networks and establish trajectory containment in Theorem 5.2.

Importantly, our analysis does not require the weights to remain at initialization. The region is defined by

∥Wℓ,t−Wℓ,0∥opnℓ−1<r,ℓ≥2,\frac{\|W_{\ell,t}-W_{\ell,0}\|_{\mathrm{op}}}{\sqrt{n _{\ell-1} }}<r,\quad \ell\ge2,

where r>0r>0 is fixed and does not shrink with width. Thus, the condition allows nonzero scaled weight movement rather than requiring r=0r=0. We acknowledge that this does not establish a non-lazy infinite-width limit. Theorem 5.2 provides finite-horizon guarantees when nmin⁡≥cdetTn_{\min}\ge c_{\mathrm{det}}T, or nmin⁡≥chpT/αn_{\min}\ge c_{\mathrm{hp}}\sqrt T/\alpha with high probability. It does not guarantee containment for arbitrarily large TT at fixed width.

Regarding the sign-like regime in Q3, we agree that v^t,j≪ϵ\sqrt{\hat v_{t,j}}\ll\epsilon may arise at extremely large widths with fixed ϵ\epsilon, causing Adam's coordinatewise normalization to become negligible. However, this behavior is not observed in our tested width range.

Weight displacement and empirical stationarity. To measure how far the weights move from initialization, we define the maximum scaled spectral displacement as

Dmax⁡=max⁡t∈Smax⁡2≤ℓ≤K∣Wℓ,t−Wℓ,0∣opnℓ−1,D_{\max}=\max_{t\in\mathcal S}\max_{2\le\ell\le K}\frac{|W_{\ell,t}-W_{\ell,0}|{\mathrm{op}}}{\sqrt{n_{\ell-1}}},

where S\mathcal S denotes the sampled checkpoints and Wℓ,0W_{\ell,0} is the initial weight matrix. This directly corresponds to the region used in Theorem 5.2.

We conducted 135 normalized-MLP training runs across seven widths, including 15 paired minibatch-sampling runs per width, with ϵ=10−8\epsilon=10^{-8}, γ=10−4\gamma=10^{-4}, and T=4096T=4096.

Table 5. Scaled weight displacement and empirical stationarity across widths.

Widthnmin⁡/Tn_{\min}/TMax. scaled spectral displacement(Dmax⁡D_{\max})Linearization error (%)CempC_{\mathrm{emp}}Update energy with v^t,j>10ϵ\sqrt{\hat v_{t,j}}>10\epsilon (%)
2560.06250.53440.870.2815199.9999
5120.1250.51335.740.2113199.9999
1,0240.250.48932.280.1650799.9996
2,0480.50.56330.810.1434699.9990
4,09610.58929.040.1435599.9971
8,19220.70831.350.1211299.9904
16,38440.61527.420.1137399.9568

All values are means over 15 runs at a fixed initialization. Spectral displacement is the maximum over internal/output layers and sampled checkpoints. Linearization error measures the relative residual of the first-order output expansion around initialization. Cemp=T1/4T−1∑t∥∇F(Wt)∥2C_{\mathrm{emp}}=T^{1/4}T^{-1}\sum_t\|\nabla F(W_t)\|_2 uses full objective gradients without dimension normalization. Update energy refers to the fraction of cumulative squared normalized-update magnitudes satisfying the stated condition.

First, the weights exhibit non-negligible movement. At widths 8,192 and 16,384, the most-displaced internal layers move by 70.8% and 61.5% of their initial spectral norms, respectively. The initialization linearization errors remain 31.35% and 27.42%, indicating appreciable nonlinear changes during training. These finite-width observations are not consistent with weights remaining effectively unchanged, although they do not establish a non-lazy asymptotic limit.

Second, Adam remains in a second-moment-dominated regime. Even at width 16,384, 99.9568% of cumulative update energy comes from coordinates with v^t,j>10ϵ\sqrt{\hat v_{t,j}}>10\epsilon, and this fraction remains 99.8287% in the final quarter of training. Thus, the observed behavior is not explained by ϵ\epsilon dominating Adam's denominator.

For perspective, the publicly released configurations of DeepSeek-V4-Pro and Kimi-K3 both specify a residual hidden dimension of 7,168, whereas our experiment reaches 16,384. This provides practical context for the tested width range, although these MoE architectures are not covered directly by our MLP theorem.

Finally, the measured stationarity constant does not deteriorate with width. CempC_{\mathrm{emp}} decreases from 0.28151 to 0.11373, with a widest-to-narrowest ratio of 0.404 and a multiplicity-adjusted paired-bootstrap interval of [0.395, 0.412]. Additional initializations reproduce the downward trend.

Overall, the experiments support finite-width optimization without deterioration of the measured stationarity constant in a second-moment-dominated regime, while the theoretical guarantees remain limited to the stated network family and width–horizon conditions. We have clarified this distinction in the revision.

Reviewer_SNiqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops a convergence analysis of fully bias-corrected Adam through interpreting its updates as magnitude-weighted sign descent. The main technical contribution is a coupled analysis of update magnitudes, momentum tracking errors, and incorrect gradient signs, avoiding independence assumptions between these quantities. Under standard LL-smoothness and conditional pp-affine second-moment noise assumptions, the authors establish an O(T−1/4)O(T^{-1/4}) convergence rate for the expected average ℓ2\ell_2 gradient norm, without explicit dimension dependence or additional logarithmic factors. This analysis accommodates arbitrary fixed β2∈[0,1)\beta_2 \in [0,1) without requiring a coupling condition between the momentum coefficients. A high-probability extension is further developed under a global generalized smoothness assumption.

This paper, as well, investigates whether the problem-dependent constants remain bounded as neural network width increases. For a family of fan-in-scaled LayerNorm/RMSNorm networks with fixed depth, the authors establish width-uniform bounds on the relevant optimization quantities. For a more restricted fixed-affine network family, explicit width–iteration-horizon conditions ensure trajectory containment and dimension-independent convergence constants. The results are validated through mathematical analysis and proofs, rather than empirical experiments.

Critical strengths
  1. The main strength is the treatment of Adam's correlated update magnitudes and momentum tracking errors via the magnitude-weighted sign formulation. The coupled error analysis in Theorem 4.2 and Corollary 4.3 establishes an O(T−1/4)O(T^{-1/4}) expected ℓ2\ell_2 stationarity guarantee under conditional pp-affine second moments, without explicit dimension factors or additional log⁡T\log T terms. Importantly, the analysis retains full bias correction and does not impose a coupling condition between β1\beta_1 and β2\beta_2. These properties constitute a meaningful theoretical contribution relative to several existing Adam convergence analyses.

  2. A worth-noticing contribution is the attempt to go beyond removing explicit dimension factors from a general convergence bound. Theorems 5.1 and 5.2 analyze the behavior of the problem-dependent constants for a concrete family of normalized networks and establish conditions under which these constants remain independent of the total parameter count. This distinction is important because a formally dimension-free rate can still conceal substantial dimension dependence through smoothness or variance parameters. Providing an explicit architectural setting where such dependence can be controlled strengthens the theoretical significance of the work.

Critical weaknesses
  1. Despite the fact that dimension-free convergence under model scaling is a central motivation, the strongest guarantees apply to a relatively specific network family with fixed depth, regularized normalization, smooth activations, and fixed affine normalization parameters. More importantly, Theorem 5.2 requires the minimum hidden width to grow at least linearly with the iteration horizon for deterministic trajectory containment, or proportionally to T/ρ\sqrt{T}/\rho for high-probability containment. These requirements substantially restrict the regimes in which the dimension-independent guarantee applies. In particular, the theorem does not establish uniform convergence guarantees for arbitrarily long training horizons at a fixed network width. This limits the extent to which the results explain optimization behavior in practical large-scale neural networks.

  2. The paper is totoally theoretical and does not contain numerical experiments to examine the tightness or practical relevance of the derived bounds. In addition, the main convergence guarantee requires horizon-dependent choices 1−β1=T−1/21-\beta_1=T^{-1/2} and γ=O(T−1/2)\gamma=O(T^{-1/2}), with a learning-rate constant constrained by the numerical-stability parameter ϵ\epsilon. These choices differ from common Adam training configurations, where momentum coefficients are generally fixed independently of the training horizon. While such parameter schedules are reasonable for theoretical analysis, the absence of empirical evidence makes it difficult to assess whether the proposed guarantees meaningfully characterize practical Adam behavior or offer advantages beyond existing analyses under comparable settings.

Questions for authors

1. Necessity of the width–horizon conditions (Theorem 5.2). Are the requirements nmin⁡=Ω(T)n_{\min}=\Omega(T) and nmin⁡=Ω(T/ρ)n_{\min}=\Omega(\sqrt{T}/\rho) mainly consequences of the current trajectory-containment argument, or are they fundamental to obtaining width-uniform convergence guarantees? Could the authors provide intuition or a tighter characterization of these conditions? Clarifying this point would help assess the scope of the main network-scaling contribution.

2. Dependence on the optimizer hyperparameters (Theorem 4.2). To what extent can the convergence analysis accommodate conventional fixed choices of β1\beta_1 and a learning-rate schedule that does not require advance knowledge of TT? Particularly, is the condition 1−β1=T−1/21-\beta_1=T^{-1/2} essential for controlling the accumulated momentum tracking error, or could a related guarantee be obtained under a more practical parameter schedule?

3. Empirical relevance of the dimension-free constants. Can the authors offer a small-scale numerical study across increasing network widths to examine the behavior of gradient norms, optimization trajectories, and the quantities entering the theoretical convergence constants? Such evidence would help establish whether the width-uniform theoretical bounds are informative in realistic optimization regimes, rather than being primarily worst-case guarantees.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SNiq (Part I)

We thank the reviewer for recognizing the contributions of our sign-based analysis of Adam and for asking how closely the theory reflects practical training. We clarify why the width–horizon conditions are sufficient for trajectory containment and how the choice 1−β1=T−1/21-\beta_1=T^{-1/2} controls noise in the momentum estimate. We also report 135 normalized-MLP training runs across a 64-fold width range. At a common learning rate selected by grid search, the empirical stationarity constant was smaller at the widest model than at the narrowest, while the theoretical learning-rate bound remained conservative.

1. Response to W1 and Q1: Width–Horizon Conditions

We thank the reviewer for raising this important question. The width–horizon conditions in Theorem 5.2 are merely convenient and tractable sufficient conditions, rather than fundamental requirements for dimension-free convergence.

More precisely, Theorem 5.1 establishes width-independent smoothness and noise constants within a specified parameter region. Theorem 5.2 further provides a concrete way to guarantee that Adam's iterates remain in this region, defined by

∥Wℓ,t−Wℓ,0∥opnℓ−1<r,ℓ≥2,\frac{\|W _{\ell,t}-W _{\ell,0}\| _{\mathrm{op}}}{\sqrt{n _{\ell-1}}}<r,\quad \ell\ge2,

where r>0r>0 is fixed and independent of width, rather than r=0r=0. Thus, our analysis does not require the weights to remain at initialization.

The conditions nmin⁡≥cdetTn_{\min}\ge c_{\mathrm{det}}T and nmin⁡≥chpT/αn_{\min}\ge c_{\mathrm{hp}}\sqrt T/\alpha follow from relatively conservative bounds on cumulative parameter displacement and update energy. They represent only one way to establish trajectory containment. If alternative, potentially weaker conditions can guarantee the same containment property, these specific width–horizon requirements are unnecessary. The essential requirement is that the iterates remain in a region where the problem-dependent constants are width-independent, not that network width must scale with TT.

Our experiments further support this distinction. As shown in Table 1, even at width 256, far below T=4096T=4096, the scaled spectral displacement remains moderate (Dmax⁡=0.534D_{\max}=0.534). Across widths 256–16,384, the empirical stationarity constant CempC_{\mathrm{emp}} decreases from 0.28151 to 0.11373. These observations suggest that the sufficient width–horizon conditions may be considerably more restrictive than necessary, although they do not establish an alternative theoretical guarantee.

We have clarified that Theorem 5.2 establishes one provably dimension-free regime, rather than characterizing all possible regimes or imposing necessary width–horizon conditions in the revision.

AuthorsOpenReview ↗

Response to Reviewer SNiq (Part II)

2. Response to W1 and Q1: Width–Horizon Conditions

We agree that practitioners typically choose β1\beta_1 independently of the training horizon. In our theorem, β1\beta_1 stays fixed throughout each run, while 1−β1=T−1/21-\beta_1=T^{-1/2} determines how much noise the momentum estimate averages out. Even under bounded noise, closely related analyses shrink 1−β11-\beta_1 as the target accuracy improves. Examples include Guo et al., A Novel Convergence Analysis for Algorithms of the Adam Family (Theorem 1), and Li et al., Convergence of Adam Under Relaxed Assumptions (Theorem 4.1). Our choice yields an O(T−1/4)O(T^{-1/4}) stationarity bound under conditional pp-affine variance, with both Adam bias corrections, arbitrary β2\beta_2, and no explicit dimension factor.

A fixed β1\beta_1 can still be compatible with approximate stationarity. For example, under the assumptions of Guo et al., a fixed β1\beta_1 allows a noise-dependent stationarity tolerance of order O(σ1−β1)O(\sigma\sqrt{1-\beta_1}), up to preconditioner constants. Our Eq. (36) gives the same scaling for the noise component of the momentum estimate. In practice, β1\beta_1 is often chosen close to one. If minibatch gradient variance also becomes small late in training, this residual can fall below the accuracy needed in practice. Establishing the corresponding fixed-β1\beta_1 stationarity bound under our assumptions remains open.

AuthorsOpenReview ↗

Response to Reviewer SNiq (Part III)

3. Response to W2 and Q3: Empirical Relevance of Dimension-Free Convergence

We appreciate the suggestion to include experiments. We conducted a normalized-MLP study with hidden widths from 256 to 16,384, covering a 64-fold range. We selected the learning rate by grid search over {10−5,2×10−5,5×10−5,10−4,2×10−4,5×10−4,10−3,2×10−3,5×10−3}\{10^{-5}, 2\times10^{-5}, 5\times10^{-5}, 10^{-4}, 2\times10^{-4}, 5\times10^{-4}, 10^{-3}, 2\times10^{-3}, 5\times10^{-3}\}. The best-performing learning rate for the normalized MLP was γ=10−4\gamma=10^{-4}, which we used at every width. All models were trained for T=4096T=4096 iterations using bias-corrected Adam with ϵ=10−8\epsilon=10^{-8}, β1=1−T−1/2\beta_1=1-T^{-1/2}, and β2=0.999\beta_2=0.999. Each width was trained 15 times with different minibatch sampling seeds. Additional initializations bring the total to 135 training runs.

We measure the empirical stationarity constant

Cemp=T1/41T∑t=0T−1∥∇F(Wt)∥2,C_{\mathrm{emp}}=T^{1/4}\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(W_t)\|_2,

using full training-objective gradients without dimension normalization. We also measure the maximum scaled spectral displacement

Dmax⁡=max⁡t∈Smax⁡2≤ℓ≤K∥Wℓ,t−Wℓ,0∥opnℓ−1,D_{\max}=\max_{t\in\mathcal S}\max_{2\le\ell\le K}\frac{\|W_{\ell,t}-W_{\ell,0}\|_{\mathrm{op}}}{\sqrt{n_{\ell-1}}},

where S\mathcal S is the set of sampled checkpoints.

Table 1. Empirical stationarity and weight displacement across network widths.

Width nmin⁡/Tn_{\min}/T  CempC_{\mathrm{emp}}  Dmax⁡D_{\max}  Local curvature L^\hat L 
2560.06250.28151 ± 0.000960.5340.1212
5120.1250.21131 ± 0.001680.5130.1304
1,0240.250.16507 ± 0.003300.4890.1253
2,0480.50.14346 ± 0.003340.5630.1530
4,09610.14355 ± 0.005130.5890.1247
8,19220.12112 ± 0.003180.7080.1198
16,38440.11373 ± 0.003530.6150.1492

At each width, CempC_{\mathrm{emp}} is the mean ± standard deviation over 15 runs at a fixed initialization. The Dmax⁡D_{\max} column reports the mean of the per-run maxima over internal and output layers at the sampled checkpoints. Local curvature L^\hat L is estimated along the training trajectories.

First, the measured stationarity constant shows no deterioration as width increases. It decreases overall from 0.28151 to 0.11373, with a widest-to-narrowest ratio of 0.404 and a multiplicity-adjusted paired-bootstrap interval of [0.395, 0.412]. Additional initializations show the same trend.

Second, the scaled weights move appreciably from initialization. At widths 8,192 and 16,384, the mean maximum scaled spectral displacements are 0.708 and 0.615, respectively. The corresponding relative errors of the first-order linearization at initialization are 31.35% and 27.42%.

Third, the local curvature estimates remain within 0.120–0.153 across widths. Even at width 16,384, an average of 99.96% of the cumulative squared update energy comes from coordinates satisfying v^t,j>10ϵ\sqrt{\hat v_{t,j}}>10\epsilon. Adam's second-moment estimate therefore dominates ϵ\epsilon on the coordinates carrying nearly all of the update energy.

These observations support the predicted width dependence over the tested range at T=4096T=4096. The absolute learning-rate condition remains conservative. Even after substituting the measured effective denominator and local curvature, the grid-selected learning rate exceeds the resulting step-size estimate at every width. Establishing asymptotic dimension independence requires the theorem's trajectory-containment and learning-rate conditions.

Closing Remarks

We sincerely thank the reviewer for the constructive and insightful comments. They motivated us to clarify the scope of the width–horizon guarantees, explain the role of the horizon-dependent momentum schedule, and conduct a substantial empirical study. These additions have significantly strengthened the presentation and practical relevance of our work.

Dear Reviewer SNiq,

We sincerely appreciate the time and effort you have already devoted to reviewing our paper. We just kindly remind you that the reviewer-author discussion period is scheduled to close in about 3 hours. We would be very grateful if you could take a moment to review our rebuttal and share your response before the deadline, should your time permit.

Best,

Authors of Paper #36631

Reviewer_DB94OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors establish convergence guarantees for Adam, both in expectation and with high probability, without explicit dependence on the number of model parameters in the convergence bounds. These results are obtained by analyzing Adam as a magnitude-weighted stochastic sign descent method. To further establish that the problem-dependent constants in the convergence bounds can remain independent of model dimensionality, the authors study a family of fan-in-scaled LayerNorm/RMSNorm networks. They show that, under certain architectural, initialization, and width conditions, Adam's iterates remain in a parameter region where the relevant problem-dependent constants are bounded independently of network width.

Critical strengths
  • The paper is mostly self-contained and easy to follow. The authors present the theoretical results in an intuitive manner, and the analysis is well-structured.
  • The theoretical results appear significant, as the authors establish convergence guarantees for Adam without explicit dependence on the model parameter dimension. Furthermore, the analysis of a family of LayerNorm/RMSNorm networks helps establish that the problem-dependent constants in the convergence bounds can remain independent of network width.
  • The analysis technique used to establish the convergence guarantees for Adam appears novel and interesting.
Critical weaknesses
  • The convergence guarantees require specific choices of the learning rate and momentum parameter based on the total number of training iterations. These choices differ from the commonly used settings for Adam, which may limit the practical applicability of the theoretical results.
  • The convergence guarantees in Theorem 5.2 require the minimum hidden-layer width to be sufficiently large relative to the number of training iterations. It is unclear why such conditions are necessary and whether similar guarantees can be established for smaller network widths. Some additional discussion on the motivation behind these requirements and whether they can be relaxed would be helpful.
Questions for authors
  • Can the authors discuss whether the convergence guarantees can be extended to more commonly used learning rate and momentum parameter settings for Adam?
  • Can the authors explain why the minimum hidden-layer width needs to scale with the number of training iterations in Theorem 5.2, and whether this requirement can be relaxed?
  • Can the authors clarify whether the width-independent constants established in Section 5 could also be used with the convergence guarantees of Li et al. (2023), and what additional benefits the proposed analysis provides in this setting?
4 comments
AuthorsOpenReview ↗

Response to Reviewer DB94 (Part I)

We sincerely thank the reviewer for the positive assessment of our theoretical contributions and the constructive suggestions. We address three main points: the dependence on horizon-specific hyperparameters, the role of width–horizon conditions in Theorem 5.2, and the relationship between our network analysis and Li et al. (2023). We also present new results from 135 normalized-MLP training runs to examine the practical relevance of our dimension-free analysis.

1. Response to W1 and Q1: Learning Rate and Momentum Settings

We agree that our theoretical choices of β1\beta_1 and γ\gamma depend on the training horizon. However, the horizon-dependent learning-rate scaling does not require practitioners to know the theoretical constant C2C_2.

Our theorem sets γ=C2T−1/2\gamma=C_2T^{-1/2}, where C2C_2 depends on problem-specific constants that are generally unknown in practice. Therefore, as commonly done in model training, we directly tune the learning rate through grid search rather than estimating C2C_2. In our experiments, we searched over {10−5,2×10−5,5×10−5,10−4,2×10−4,5×10−4,10−3,2×10−3,5×10−3}\{10^{-5},2\times10^{-5},5\times10^{-5},10^{-4},2\times10^{-4},5\times10^{-4},10^{-3},2\times10^{-3},5\times10^{-3}\} and selected γ=10−4\gamma=10^{-4}.

Moreover, the theoretical T−1/2T^{-1/2} scaling provides potential practical guidance when changing the training horizon. For example, if a suitable learning rate γ1\gamma_1 has been found for T1T_1 iterations, one can use γ1T1/T2\gamma_1\sqrt{T_1/T_2} as a starting point for a narrower search at T2T_2 iterations, rather than repeating the full grid search. This is a heuristic motivated by our theoretical scaling, not a proven relationship between optimal learning rates.

Our analysis also provides practical guidance for choosing β1\beta_1. The choice 1−β1=T−1/21-\beta_1=T^{-1/2} ensures that momentum-tracking noise vanishes at the desired rate. With fixed β1\beta_1, the noise contribution remains of order O(σ1−β1)O(\sigma\sqrt{1-\beta_1}), suggesting a noise-dependent stationary neighborhood under suitable conditions. This suggests that choosing a relatively large fixed β1\beta_1, together with potentially smaller gradient variance σ\sigma during later training stages, can make the residual sufficiently small for practical purposes. Thus, beyond establishing a vanishing O(T−1/4)O(T^{-1/4}) guarantee under weak noise assumptions, our analysis offers guidance on both learning-rate scaling and the practical trade-off between momentum and gradient noise.

AuthorsOpenReview ↗

Response to Reviewer DB94 (Part III)

3. Response to Q3: Relationship to Li et al. (2023)

We appreciate this insightful question. Yes, the width-independent problem constants established in Theorem 5.1 could, in principle, also benefit the convergence analysis of Li et al. (2023). These architectural bounds are not specific to our sign-based proof.

In particular, Theorem 5.1 establishes width-independent smoothness, gradient, and noise bounds within the specified network region. For the fixed-affine network family, the bounded sample gradients also imply bounded gradient noise within this region, making Li et al.'s bounded-noise setting applicable in principle. However, because our architectural bounds hold regionally rather than globally, applying their convergence theorem would still require a compatible trajectory-containment argument. Their original theorem does not directly provide the width–horizon certification established in our Theorem 5.2.

Our analysis provides two additional contributions.

First, our general convergence guarantee requires only conditional pp-affine second moments, whereas Li et al.'s Theorem 4.1 assumes almost-surely bounded gradient noise, and their Theorem 4.2 assumes sub-Gaussian noise. Our analysis therefore applies to a broader noise class, although the particular bounded network family also satisfies stronger noise conditions within its admissible region.

Second, by preserving Adam's actual normalized update magnitude utu_t, our analysis establishes a cumulative update-energy bound UT=O(T)U_T=O(\sqrt T). This bound is crucial for the high-probability trajectory-containment argument, yielding an exit probability of order O(T/nmin⁡)O(\sqrt T/n_{\min}) and consequently the sufficient condition nmin⁡=Ω(T/α)n_{\min}=\Omega(\sqrt T/\alpha). Thus, our sign-based analysis provides a concrete way to connect Adam's update dynamics with dimension-free convergence under model scaling.

We agree that width-independent architectural constants could be combined with other Adam convergence frameworks. Our contribution is not that such constants are exclusive to our method, but that we combine them with a finer analysis of Adam's updates to establish explicit trajectory containment and dimension-free convergence guarantees. We have clarified this distinction and the relationship to Li et al. in the revision.

Closing Remarks

We sincerely thank the reviewer for the thoughtful and constructive comments. They have helped us clarify the role of horizon-dependent hyperparameters, the sufficient nature of the width–horizon conditions, and the relationship between our analysis and prior work. Together with the new empirical results, these clarifications have substantially improved the quality and presentation of our paper. We greatly appreciate the reviewer's careful reading and valuable feedback.

Thank you for the detailed response, clarifications, and the new empirical results to support your claims. I will maintain my positive score.

Thank you for your strong support throughout the review process. Your encouragement means a great deal to us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.