- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper analyzes Adam under heavy-tailed gradient noise and compares two choices of the stabilization parameter : a coordinatewise and a single shared . The main positive result shows that coordinatewise , tuned as $ \epsilon_i \propto \max (\sigma_i,,(L_i/L)\Sigma), $ attains the sharp convergence rate. I.e., the upper bound relies on coordinatewise knowledge of noise and curvature. The paper also proves a lower bound for Adam with a shared , showing that no dimension-uniform guarantee with the same exponent is possible over arbitrarily heterogeneous coordinate scales.
Finally, the paper assigns a problem-dependent role, which I believe is disconnected from both practical usage and the broader theoretical goal of reducing hyperparameter tuning [1–3].
[1] Orabona, Francesco. "Research: Parameter-Free Machine Learning." https://francesco.orabona.com/research.html.
[2] Defazio, Aaron, et al. "The road less scheduled." Advances in Neural Information Processing Systems 37 (2024): 9974-10007.
[3] Attia, Amit, and Tomer Koren. "How free is parameter-free stochastic optimization?." arXiv preprint arXiv:2402.03126 (2024).
Critical strengths
- The analysis is technically clear and transparent about different settings, for example Table 1.
- The paper addresses a clearly defined theoretical question about the interaction between Adam's coordinatewise second-moment and the denominator stabilization parameter.
Critical weaknesses
On the motivation of the paper.
The two settings studied in this paper are, respectively, impractical (point A) and at odds with how Adam is used in practice (point B).
(A) Coordinatewise is not a realistic setting. Choosing according to coordinatewise noise and curvature bounds requires tuning problem-dependent hyperparameters. For deep models, this means tuning millions or billions of hyperparameters, which is infeasible. I do not see the purpose of analyzing Adam under such a setting.
(B) The analysis of shared runs against practical usage. Some well-known facts are relevant here:
- Scaling with the noise and curvature bounds greatly simplifies the convergence analysis of Adam, and it also effectively reduces the method to SGD with momentum somewhat. In practice, is meant to be a tiny numerical constant. Empirical evidence, notably the Adam-atan2 study [1], suggests that should be kept negligible, and that methods which remove altogether perform better.
- In short, both practice and theory point to the same question: can Adam's convergence be established without relying on tuning ? I therefore disagree with the paper's claim that "this points to a distinction that is easy to miss in adaptive methods."
- I believe the role of as a tuned optimization parameter is a distinction that should be set aside, not emphasized.
[1] Everett, Katie, et al. "Scaling Exponents Across Parameterizations and Optimizers." ICML 2024.
Questions for authors
- Does your analysis work under condition setting as zero and tuning arbitrarily?
- Can the analysis be extended to the regime where is a small numerical constant rather than a problem-dependent parameter?
- Why do some parts use , while others use
4 comments
3. On the Adam-atan2 citation
The cited reference does not support the reviewer’s objection. Instead, it supports our central theoretical point that has a substantive, scale-dependent role in Adam. Its abstract explicitly states that “must be scaled correctly to avoid gradient underflow” [3].
Section 4.3 reports comparable improvements from a small constant , per-layer , and Adam-atan2 in the mean-field experiments. It also reports instability when is too small. These findings reinforce the importance of understanding how interacts with gradient scales.
If were truly negligible in Adam's optimization dynamics, there would be no reason to develop a new optimizer specifically to eliminate its effects. Everett et al. [3] introduce Adam-atan2 precisely because breaks scale invariance and can adversely affect optimization performance. The very motivation for eliminating demonstrates that it has a substantive effect on optimization. The cited work therefore directly supports our motivation for studying the role of , rather than the reviewer's suggestion that this role should be set aside.
4. On using a tiny constant
Theorem 4.2 allows the learning rate, momentum parameters, and shared to depend on past observations. This includes choosing as a tiny constant or setting .
For , this class is subject to a dimension-uniform lower bound of . Making the shared arbitrarily small, or setting it to zero, still cannot attain the coordinatewise benchmark uniformly over our problem class. The setting suggested by the reviewer is already covered by our analysis.
5. On setting
Under Adam's second-moment recursion,
with and standard zero initialization, necessarily implies , and consequently the numerator . Thus, this coordinate receives no updates, regardless of the choice of positive .
The remaining coordinates can continue updating normally. The irrelevance of in a coordinate with identically zero stochastic gradients provides no information about its role in the active coordinates. If holds for every coordinate throughout training, then all stochastic gradients and momentum terms vanish, and the algorithm makes no updates at all under any well-defined update rule.
6. On
Both main theorems apply at . At this endpoint, the upper and lower time exponents both equal , so the strict exponent separation disappears. We therefore use for the theorems and when stating the strict separation between the two parameter settings. This distinction reflects a more mathematically precise formulation of our results.
References
[3] Katie E. Everett et al. Scaling Exponents Across Parameterizations and Optimizers. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:12666-12700, 2024.
I have reviewed the rebuttal and decided to keep my score.
The concerns remain:
- Existing work such as (Liu, 2026b, already cited in this work) has resolved the problem with a better setting: small , retaining bias-corrections, heavy-tailed noise;
- The author claims the difference as "The setting considered here uses the expected ℓ1 norm of the true gradient, coordinatewise smoothness, and centered coordinatewise noise moments.". I just checked the paper (Liu, 2026b), it also use the coordinatewise version assumptions. I don't see much difference since the (λ, ε)-L1 stationary point in (Liu, 2026b) under nonsmooth setting can be extended to general first order stationary point in smooth case.
- The motivation again, "the role of as a tuned optimization parameter is a distinction that should be set aside, not emphasized."
The reviewer's response overlooks several substantive mathematical differences between our work and that of Liu [1], while providing no evidence to support the claim that should be set aside.
1. On the comparison with Liu's work
The reviewer states, "I don't see much difference." However, the differences between the two works are very clear:
-
Different assumptions. Their work requires the true gradient to be coordinatewise bounded, whereas we assume coordinatewise smoothness without requiring bounded gradients. Even the simplest quadratic function satisfies our assumptions but violates theirs, since is unbounded over the entire space. This demonstrates a substantive difference between the problem classes studied by the two papers.
-
Different convergence guarantees. Their work establishes a - stationarity guarantee for an exponential moving average of the iterates, whereas we directly control at the original iterates. The reviewer claims that the former can be extended to ordinary first-order stationarity. However, even assuming such a conversion could be achieved without any loss, the resulting time exponent would still be smaller than ours.
-
Different momentum parameter restrictions. Their abstract and contribution statements explicitly emphasize that the main theoretical results are established under . Their final explicit heavy-tailed convergence rate still relies on this restriction. In contrast, our upper bound does not require .
-
Different time exponents. Their time exponent is under and , whereas ours is for . In particular, at , the former gives , while ours gives .
-
Different theoretical conclusions. Their work provides convergence upper bounds, and the author explicitly acknowledges in the conclusion that "our finite-time rates, for both known and unknown , are clearly suboptimal" [1]. In contrast, we establish both upper and lower bounds and explain why a shared cannot achieve the dimension-uniform optimal rate.
2. On the significance of
The reviewer repeatedly argues that is unimportant, but provides neither a substantive theoretical justification nor any supporting reference for this judgment. In fact, both existing literature and our theoretical results directly contradict this claim.
Moreover, even the paper by Everett et al. [2], cited by the reviewer, demonstrates that breaks Adam's scale invariance. Our work further establishes that the structure of the parameter directly determines whether the algorithm can achieve the optimal convergence rate.
A “motivation” is not evidence.
References
[1] Zijian Liu. Adam Converges in Nonsmooth Nonconvex Optimization. arXiv preprint arXiv:2606.22326, 2026.
[2] Katie E. Everett, Lechao Xiao, Mitchell Wortsman, et al. Scaling Exponents Across Parameterizations and Optimizers. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:12666–12700, 2024.
Our motivation is to close the gap in Adam's convergence theory under heavy-tailed noise. The key difficulty is to retain the stochastic denominator and turn weighted descent estimates into a convergence guarantee for the true gradient. Yu et al. [2] explicitly discuss this obstacle. We resolve it under coordinatewise smoothness and finite p-th centered noise moments.
The denominator itself has been a technical obstacle in prior analyses. Zhang et al. [1] explain that their surrogate-denominator argument requires bounded gradients when the stabilizer is outside the square root. Without this assumption, that proof route does not directly close. They replace vt+ϵ with vt+ϵ to control the error. This is a technical compromise made to complete the proof. We directly handle the stochastic denominator under the heavy-tailed setting above. This direct analysis of Adam's denominator under heavy-tailed noise is a central contribution of our paper.
1. On hyperparameters
Coordinatewise ϵi is a theoretical parameter setting used to study Adam’s convergence capabilities. Our theorem determines these values through the unified formula
\epsilon_i=\rho_2^{-\frac{2-p}{2p}}\max\left\\{\sigma_i,\frac{L_i}{L}\Sigma\right\\}.
Once the problem scales and global parameters are specified, all ϵi values are determined, without independently tuning each coordinate.
We deliberately compare coordinatewise and shared-scalar stabilization to reveal the role of ϵ under heterogeneous coordinate scales. We fix total curvature L=∑iLi and total noise scale Σ=∑iσi, while allowing the dimension and coordinate-scale allocation to vary. This comparison isolates how the choice of stabilization affects Adam’s convergence guarantees.
Together, Theorems 4.1 and 4.2 establish the convergence-rate separation between coordinatewise and shared-scalar ϵ, which is a central contribution of our paper.
Regarding practical feasibility, please refer to our response to Reviewer tCY8, where we present additional experiments on both vision and language modeling tasks. Our implementation estimates coordinatewise noise scales online from observed gradients and determines ϵi through a unified rule. It requires no independent tuning of d hyperparameters.
2. On the claimed reduction to momentum SGD
The reviewer's claim rests on a misunderstanding of our parameter construction. The reviewer appears to assume that we choose ϵi sufficiently large to dominate vt,i, thereby reducing Adam to momentum SGD. However, our choice of ϵi is derived from the balance between heavy-tailed momentum noise and gradient drift in Theorem 4.1. In particular, the factor ρ2−(2−p)/(2p) reflects the natural heavy-tailed noise scale of the second-moment estimator. Our theorem neither requires nor establishes vt,i≪ϵi. The stochastic denominator remains an essential part of the algorithm and the convergence proof. Indeed, our analysis exploits its self-normalization to control heavy-tailed fluctuations that cannot be handled by simply replacing the denominator with a constant. Our convergence analysis does not rely on any reduction of Adam to momentum SGD in any sense, or on the simplifications such a reduction would entail.
References
[1] Qi Zhang, Yi Zhou, and Shaofeng Zou. Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance. Transactions on Machine Learning Research, 2025.
[2] Dingzhi Yu, Hongyi Tao, Yuanyu Wan, Luo Luo, and Lijun Zhang. Open Problem: Is AdamW Effective Under Heavy-Tailed Noise? arXiv:2606.23676, 2026.