- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes Adam with full bias correction by writing its update as with . Under -smoothness and p-affine variance, with , and any , Corollary 4.3 gives , where has no explicit factor of and no . Corollary 4.4 gives a high-probability version under global -smoothness. Section 5 shows that for fan-in-scaled LayerNorm/RMSNorm networks of growing width the problem constants stay bounded (Theorem 5.1). If the width grows with the horizon, the iterates stay in this region (Theorem 5.2). The paper has no experiments.
Critical strengths
S1. The proofs are correct. I checked the main chain in Appendices C, D, E and F.6 and found no error. Unlike many earlier analyses, the paper keeps both bias corrections of Adam, as in the standard implementation. The final step to the norm (Appendix D, eqs. (49) to (54)) is done carefully and does not hide a factor .
S2. Bounded gradients are removed under affine-type noise. Earlier analyses in the same regime bound the denominator of Adam pathwise and so need bounded gradients. Here a second-moment bound on the largest denominator, eq. (53), is enough. The paper is also careful about its scope. It says that its noise improvement "should not be read as a uniform relaxation of every assumption" (page 4) and that the bounds are "not uniform as ε→0" (page 8). The constants in Section 5 are explicit.
Critical weaknesses
W1. The analysis covers momentum SGD with a bounded preconditioner, not the adaptive regime of Adam, and its main claims are already known. The step size condition (10) gives . So every coordinate moves with an effective step of at most , which is the standard SGDM step. The proof uses only three facts about the denominator . It is at least , its largest entry bounds all entries, and that largest entry has bounded expectation. The structure of and the value of enter only through eq. (52), which uses that is an average of past . This is why any is allowed. The sign representation of Lemma 4.1 is the identity . The same proof goes through with in place of .
The price is in the constant. From (10) to (14), and grows like . With and the largest admissible , I get at , at and at the default . The admissible step is then , while practical learning rates are about . Earlier bounds such as that of Défossez et al. pay but allow learning rates that do not scale with . So "no explicit " is bought by moving to a regime where Adam behaves like SGD.
The headline features are also in prior work. Theorem 4.1 of "Convergence of Adam under Relaxed Assumptions" (Li, Rakhlin, Jadbabaie) covers bias-corrected Adam with no condition on , no dimension factor and no logarithm. Its complexity grows as , where their plays the role of . Table 1 lists only their Theorem 4.2 with . "A Novel Convergence Analysis for Algorithms of the Adam Family" (Guo, Xu, Yin, Jin, Yang) treats Adam as momentum SGD with a denominator between two bounds, under affine variance and without a dimension factor. It is not cited. The same holds for "Momentum Improves Normalized SGD" (Cutkosky, Mehta), whose proof has the same shape as Theorem 4.2. After these comparisons, the new part of Section 4 is the removal of bounded gradients via eq. (53). This is correct but narrow.
W2. The network result holds only in a lazy regime with bounded gradients. On the region , Theorem 5.1 bounds the gradient by , eq. (19), and the proof of Theorem 5.2 uses this bound, eq. (100). So the main advantage of Section 4, no bounded gradients, does not apply to the network result. With bounded gradients, the results of Li et al. and Guo et al. already apply on with the width-free constants of Theorem 5.1. The claim to be first therefore rests on the containment argument in Appendix F.6. That argument needs , eq. (104), or . The total displacement is while leaving the region needs . So the hidden width must grow with the horizon, and the scaled weights stay close to initialization. This is an NTK-type regime. In the fan-in-scaled parameterization the per-coordinate gradients shrink with width, so at fixed we get and Adam again reduces to SGDM. The sign-like regime where Adam is used in practice, with of order , is not covered. Theorem 5.2 also fixes the affine normalization parameters, has no residual connections or attention, uses no weight decay, and ties to the horizon.
Questions for authors
Q1. Regarding W1, which step of the proof needs the sign representation or the structure of ? Would the same result follow from a standard preconditioned SGDM analysis with a diagonal preconditioner between and ?
Q2. Regarding W1, please state the explicit dependence of on and compare the result with Theorem 4.1 of Li et al. and Theorem 1 of Guo et al. What does Corollary 4.3 give at and a learning rate of ?
Q3. Regarding W2, can the containment in Theorem 5.2 hold when the width does not grow with , or when Adam is in the sign-like regime ?
Q4. A suggestion that would raise my score. The paper has no experiments. A small study on a normalized MLP or a small Transformer could check three things. First, whether the bound is informative at standard and learning rates. Second, how far the scaled weights move from initialization when . Third, whether the measured constant is independent of width in the sign-like regime.
Q5. Clarification. In eq. (45), p-affine variance seems to reduce to affine variance with modified constants. Does add anything to Theorem 4.2? Also, global -smoothness for all pairs allows at most linear growth of the gradient. Which standard examples of generalized smoothness does Corollary 4.4 cover?
6 comments
Responses to Reviewer GZDW (Part II)
2. Response to W1 and Q4(a): Practical Learning Rates and the Theory–Practice Gap
We agree that the submitted learning-rate condition is conservative. To investigate this, we conducted normalized-MLP experiments with and . A preliminary search over identified as the best-performing setting. We fixed this learning rate across seven widths, with 15 independent minibatch-sampling runs per width.
As discussed in our response to Q1–Q2, our effective-denominator refinement relaxes the sufficient condition to , where . Table 3 evaluates this condition using empirical , local curvature , and .
Table 3. Practical learning rates and refined step-size diagnostics.
| Width | Refined LR cap | Actual LR / cap | Measured mean gradient norm |
|---|---|---|---|
| 256 | 9.126e-7 | 109.6 | 0.035189 |
| 512 | 5.053e-7 | 197.9 | 0.026414 |
| 1,024 | 2.683e-7 | 372.7 | 0.020634 |
| 2,048 | 9.669e-8 | 1,034.3 | 0.017932 |
| 4,096 | 5.614e-8 | 1,781.3 | 0.017944 |
| 8,192 | 2.086e-8 | 4,794.6 | 0.015140 |
| 16,384 | 9.679e-9 | 10,331.6 | 0.014216 |
The refined cap is . The empirical denominators and local curvature estimates do not certify the global theorem conditions.
Comparison with prior analyses. With and , the preconditioner-endpoint restrictions in Guo et al. and Li et al. involve terms as small as and , corresponding to scales of and before other constants. Compared with a reference scale, our refined empirical caps of to are approximately – times larger. Thus, the nominal gap to the practical learning rate is substantially reduced, although this is an indicative scale comparison rather than a certified numerical comparison across different theorems.
Nevertheless, our refined cap remains 110–10,332× smaller than the practical learning rate. The refined stationarity expressions also remain much larger than the observed gradient norms. We therefore investigated where the remaining conservatism arises.
Table 4. Quantifying conservatism in curvature and momentum-error estimates.
| Width | Local / directional curvature | Drift utilization (envelope factor) | Descent margin: observed / conservative | Descent-cost inflation |
|---|---|---|---|---|
| 256 | 18.0× | 0.697 (1.43×) | 0.860 / 0.582 | 2.99× |
| 512 | 17.2× | 0.598 (1.67×) | 0.801 / 0.413 | 2.95× |
| 1,024 | 16.4× | 0.460 (2.18×) | 0.719 / 0.141 | 3.05× |
| 2,048 | 24.9× | 0.382 (2.62×) | 0.648 / −0.369 | 3.89× |
| 4,096 | 21.9× | 0.280 (3.57×) | 0.557 / −1.008 | 4.53× |
| 8,192 | 37.8× | 0.231 (4.32×) | 0.500 / −2.381 | 6.76× |
| 16,384 | 41.0× | 0.176 (5.70×) | 0.402 / −4.238 | 8.76× |
Curvature compares local estimates with realized positive-remainder curvature. Drift utilization is the ratio of actual vector drift to its scalar norm envelope; the parenthesized factor is its inverse. Descent-cost inflation is , calculated from the reported mean margins. These are distinct diagnostics, not independent multipliers of the admissible learning rate.
Why does the gap remain? First, the global smoothness estimate is conservative along actual update directions: the measured directional curvature is 16–41× smaller than the local curvature estimate. Second, triangle and Cauchy–Schwarz inequalities discard momentum-drift cancellations, with scalar envelopes overestimating realized drift by 1.4–5.7×.
Together, these conservative substitutions inflate the measured normalized descent cost by approximately 3–9×. At width 16,384, the observed descent margin is 0.402, whereas the conservative estimate gives −4.238. The latter cannot establish descent despite the positive observed margin.
These diagnostics identify substantial additional slack beyond the denominator bound, but their factors cannot simply be multiplied to close the remaining learning-rate gap. Our analysis has significantly narrowed the gap relative to earlier worst-case preconditioner approaches, while a fully practical learning-rate guarantee remains open. We have clarified these findings and limitations in the revision.
Responses to Reviewer GZDW (Part III)
3. Response to W2, Q3, and Q4(b–c): Network Containment and the Sign-like Regime
We agree that Theorem 5.1 establishes bounded gradients within a specified region. Our contribution is to derive width-independent smoothness and noise constants for normalized networks and establish trajectory containment in Theorem 5.2.
Importantly, our analysis does not require the weights to remain at initialization. The region is defined by
where is fixed and does not shrink with width. Thus, the condition allows nonzero scaled weight movement rather than requiring . We acknowledge that this does not establish a non-lazy infinite-width limit. Theorem 5.2 provides finite-horizon guarantees when , or with high probability. It does not guarantee containment for arbitrarily large at fixed width.
Regarding the sign-like regime in Q3, we agree that may arise at extremely large widths with fixed , causing Adam's coordinatewise normalization to become negligible. However, this behavior is not observed in our tested width range.
Weight displacement and empirical stationarity. To measure how far the weights move from initialization, we define the maximum scaled spectral displacement as
where denotes the sampled checkpoints and is the initial weight matrix. This directly corresponds to the region used in Theorem 5.2.
We conducted 135 normalized-MLP training runs across seven widths, including 15 paired minibatch-sampling runs per width, with , , and .
Table 5. Scaled weight displacement and empirical stationarity across widths.
| Width | Max. scaled spectral displacement() | Linearization error (%) | Update energy with (%) | ||
|---|---|---|---|---|---|
| 256 | 0.0625 | 0.534 | 40.87 | 0.28151 | 99.9999 |
| 512 | 0.125 | 0.513 | 35.74 | 0.21131 | 99.9999 |
| 1,024 | 0.25 | 0.489 | 32.28 | 0.16507 | 99.9996 |
| 2,048 | 0.5 | 0.563 | 30.81 | 0.14346 | 99.9990 |
| 4,096 | 1 | 0.589 | 29.04 | 0.14355 | 99.9971 |
| 8,192 | 2 | 0.708 | 31.35 | 0.12112 | 99.9904 |
| 16,384 | 4 | 0.615 | 27.42 | 0.11373 | 99.9568 |
All values are means over 15 runs at a fixed initialization. Spectral displacement is the maximum over internal/output layers and sampled checkpoints. Linearization error measures the relative residual of the first-order output expansion around initialization. uses full objective gradients without dimension normalization. Update energy refers to the fraction of cumulative squared normalized-update magnitudes satisfying the stated condition.
First, the weights exhibit non-negligible movement. At widths 8,192 and 16,384, the most-displaced internal layers move by 70.8% and 61.5% of their initial spectral norms, respectively. The initialization linearization errors remain 31.35% and 27.42%, indicating appreciable nonlinear changes during training. These finite-width observations are not consistent with weights remaining effectively unchanged, although they do not establish a non-lazy asymptotic limit.
Second, Adam remains in a second-moment-dominated regime. Even at width 16,384, 99.9568% of cumulative update energy comes from coordinates with , and this fraction remains 99.8287% in the final quarter of training. Thus, the observed behavior is not explained by dominating Adam's denominator.
For perspective, the publicly released configurations of DeepSeek-V4-Pro and Kimi-K3 both specify a residual hidden dimension of 7,168, whereas our experiment reaches 16,384. This provides practical context for the tested width range, although these MoE architectures are not covered directly by our MLP theorem.
Finally, the measured stationarity constant does not deteriorate with width. decreases from 0.28151 to 0.11373, with a widest-to-narrowest ratio of 0.404 and a multiplicity-adjusted paired-bootstrap interval of [0.395, 0.412]. Additional initializations reproduce the downward trend.
Overall, the experiments support finite-width optimization without deterioration of the measured stationarity constant in a second-moment-dominated regime, while the theoretical guarantees remain limited to the stated network family and width–horizon conditions. We have clarified this distinction in the revision.
Responses to Reviewer GZDW (Part I)
Thank you for carefully reading the paper and checking the proofs, and suggesting the normalized-MLP experiment. We ran 135 trajectories across seven widths. The effective denominator improved the step-size estimate by factors of 296 to 22,660, although the estimate remained below the learning rate used in every run. Across the same widths, the empirical stationarity constant fell from 0.28151 to 0.11373.
1. Response to W1, Q1, and Q2: Preconditioned SGDM and the dependence on ϵ
We agree that preconditioned SGDM can establish convergence without explicit dependence on d, as demonstrated by Guo et al. and Li et al. However, their proofs use worst-case bounds on the diagonal preconditioner, whereas ours retains the fine-grained structure of ut=∣m^t∣/(v^t+ϵ). In Eq. (39), the same cumulative update energy ∑tE∥ut∥22 controls gradient drift and descent errors, yielding weaker dependence on ϵ.
Table 1. Comparison of learning-rate restrictions and convergence bounds.
The comparison fixes 1−β1=T−1/2 and other problem constants, considering sufficiently large T. The prior bounds are matched-schedule specializations, not optimal guarantees over all parameter choices. Guo et al. require bounded preconditioners, while Li et al. assume bounded noise and provide a high-probability result.
Further improvement through the effective denominator. Our original proof conservatively uses the minimum denominator ϵ. We improve this by introducing the update-energy-weighted average
ρ=∑tE∥ut∥22∑tE∑j(v^t,j+ϵ)ut,j2≥ϵ.
This relaxes the sufficient learning-rate condition from γT≤ϵ/(LA) to γT≤ρ/(LA), where A=max(32,60pσ12).
Crucially, this refinement cannot be directly applied to the preconditioned SGDM, since their worst-case preconditioner bounds discard the dependence on ut. Moreover, ρ is an energy-weighted average, not a coordinatewise lower bound. Recovering this improvement requires retaining the coupled update structure, as in our analysis.
To evaluate this refinement, we independently trained each width 15 times using different minibatch sampling seeds, with initialization fixed. The reported ρ^ is the pooled update-energy-weighted estimate across these 15 runs, rather than the arithmetic mean of individual ratios.
Table 2. Empirical improvement from ϵ to ρ (15 training runs per width).
The effective denominators improve the denominator-dependent learning-rate allowance by 296–22,660× across the tested widths. These are empirical estimates, not certified lower bounds. We have included the refined proof and clarify the comparison in the revision.
Additionally, We thank the reviewer for suggesting A Novel Convergence Analysis for Algorithms of the Adam Family (Guo et al.) and Momentum Improves Normalized SGD (Cutkosky and Mehta). We have added both references to the revised manuscript. Cutkosky and Mehta analyze momentum with globally normalized SGD updates, while our work considers fully bias-corrected Adam with coordinatewise, history-dependent second-moment normalization. Although both analyses exploit momentum tracking, their method does not address the stochastic coupling between Adam's adaptive update magnitudes and tracking errors, which is central to our analysis and the ρ refinement. We have clarified these distinctions in the revision.