acceptodds
Under review as a conference paper at ICLR 2027

Coordinatewise Epsilon for Adam under Heavy-Tailed Noise

Abstract

Adam estimates second moments coordinatewise, but typically adds the same scalar to every denominator. We study how this shared affects convergence when noise and curvature vary across coordinates. Under coordinatewise finite- noise moments, , choosing from the noise and curvature bounds for each coordinate attains the heavy-tailed -stationarity benchmark , matching the known optimal time exponent for smooth nonconvex stochastic optimization. For , Adam with a shared admits a lower bound of order . This lower bound holds for every predictable scalar choice of , , , and , precluding a dimension-uniform benchmark rate for this class.

Primary area:
optimization
Keywords:
Adam, optimization, stochastic optimization, heavy-tailed noise

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uxuTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes Adam under heavy-tailed gradient noise and compares two choices of the stabilization parameter ϵ\epsilon: a coordinatewise ϵi\epsilon_i and a single shared ϵ\epsilon. The main positive result shows that coordinatewise ϵ\epsilon, tuned as $ \epsilon_i \propto \max (\sigma_i,,(L_i/L)\Sigma), $ attains the sharp convergence rate. I.e., the upper bound relies on coordinatewise knowledge of noise and curvature. The paper also proves a lower bound for Adam with a shared ϵt\epsilon_t, showing that no dimension-uniform guarantee with the same exponent is possible over arbitrarily heterogeneous coordinate scales.

Finally, the paper assigns ϵ\epsilon a problem-dependent role, which I believe is disconnected from both practical usage and the broader theoretical goal of reducing hyperparameter tuning [1–3].


[1] Orabona, Francesco. "Research: Parameter-Free Machine Learning." https://francesco.orabona.com/research.html.

[2] Defazio, Aaron, et al. "The road less scheduled." Advances in Neural Information Processing Systems 37 (2024): 9974-10007.

[3] Attia, Amit, and Tomer Koren. "How free is parameter-free stochastic optimization?." arXiv preprint arXiv:2402.03126 (2024).

Critical strengths
  • The analysis is technically clear and transparent about different settings, for example Table 1.
  • The paper addresses a clearly defined theoretical question about the interaction between Adam's coordinatewise second-moment and the denominator stabilization parameter.
Critical weaknesses

On the motivation of the paper.

The two settings studied in this paper are, respectively, impractical (point A) and at odds with how Adam is used in practice (point B).

(A) Coordinatewise ϵi\epsilon_i is not a realistic setting. Choosing ϵi\epsilon_i according to coordinatewise noise and curvature bounds requires tuning dd problem-dependent hyperparameters. For deep models, this means tuning millions or billions of hyperparameters, which is infeasible. I do not see the purpose of analyzing Adam under such a setting.

(B) The analysis of shared ϵ\epsilon runs against practical usage. Some well-known facts are relevant here:

  • Scaling ϵ\epsilon with the noise and curvature bounds greatly simplifies the convergence analysis of Adam, and it also effectively reduces the method to SGD with momentum somewhat. In practice, ϵ\epsilon is meant to be a tiny numerical constant. Empirical evidence, notably the Adam-atan2 study [1], suggests that ϵ\epsilon should be kept negligible, and that methods which remove ϵ\epsilon altogether perform better.
  • In short, both practice and theory point to the same question: can Adam's convergence be established without relying on tuning ϵ\epsilon? I therefore disagree with the paper's claim that "this points to a distinction that is easy to miss in adaptive methods."
  • I believe the role of ϵ\epsilon as a tuned optimization parameter is a distinction that should be set aside, not emphasized.

[1] Everett, Katie, et al. "Scaling Exponents Across Parameterizations and Optimizers." ICML 2024.

Questions for authors
  • Does your analysis work under condition setting vt,iv_{t,i} as zero and tuning ϵ\epsilon arbitrarily?
  • Can the analysis be extended to the regime where ϵ\epsilon is a small numerical constant rather than a problem-dependent parameter?
  • Why do some parts use 1<p<21 < p < 2, while others use 1<p≤21 < p \le 2
4 comments

Our motivation is to close the gap in Adam's convergence theory under heavy-tailed noise. The key difficulty is to retain the stochastic denominator and turn weighted descent estimates into a convergence guarantee for the true gradient. Yu et al. [2] explicitly discuss this obstacle. We resolve it under coordinatewise smoothness and finite pp-th centered noise moments.

The denominator itself has been a technical obstacle in prior analyses. Zhang et al. [1] explain that their surrogate-denominator argument requires bounded gradients when the stabilizer is outside the square root. Without this assumption, that proof route does not directly close. They replace vt+ϵ\sqrt{v_t}+\epsilon with vt+ϵ\sqrt{v_t+\epsilon} to control the error. This is a technical compromise made to complete the proof. We directly handle the stochastic denominator under the heavy-tailed setting above. This direct analysis of Adam's denominator under heavy-tailed noise is a central contribution of our paper.

1. On hyperparameters

Coordinatewise ϵi\epsilon_i is a theoretical parameter setting used to study Adam’s convergence capabilities. Our theorem determines these values through the unified formula

\epsilon_i=\rho_2^{-\frac{2-p}{2p}}\max\left\\{\sigma_i,\frac{L_i}{L}\Sigma\right\\}.

Once the problem scales and global parameters are specified, all ϵi\epsilon_i values are determined, without independently tuning each coordinate.

We deliberately compare coordinatewise and shared-scalar stabilization to reveal the role of ϵ\epsilon under heterogeneous coordinate scales. We fix total curvature L=∑iLiL=\sum_iL_i and total noise scale Σ=∑iσi\Sigma=\sum_i\sigma_i, while allowing the dimension and coordinate-scale allocation to vary. This comparison isolates how the choice of stabilization affects Adam’s convergence guarantees.

Together, Theorems 4.1 and 4.2 establish the convergence-rate separation between coordinatewise and shared-scalar ϵ\epsilon, which is a central contribution of our paper.

Regarding practical feasibility, please refer to our response to Reviewer tCY8, where we present additional experiments on both vision and language modeling tasks. Our implementation estimates coordinatewise noise scales online from observed gradients and determines ϵi\epsilon_i through a unified rule. It requires no independent tuning of dd hyperparameters.

2. On the claimed reduction to momentum SGD

The reviewer's claim rests on a misunderstanding of our parameter construction. The reviewer appears to assume that we choose ϵi\epsilon_i sufficiently large to dominate vt,i\sqrt{v_{t,i}}, thereby reducing Adam to momentum SGD. However, our choice of ϵi\epsilon_i is derived from the balance between heavy-tailed momentum noise and gradient drift in Theorem 4.1. In particular, the factor ρ2−(2−p)/(2p)\rho_2^{-(2-p)/(2p)} reflects the natural heavy-tailed noise scale of the second-moment estimator. Our theorem neither requires nor establishes vt,i≪ϵi\sqrt{v_{t,i}}\ll\epsilon_i. The stochastic denominator remains an essential part of the algorithm and the convergence proof. Indeed, our analysis exploits its self-normalization to control heavy-tailed fluctuations that cannot be handled by simply replacing the denominator with a constant. Our convergence analysis does not rely on any reduction of Adam to momentum SGD in any sense, or on the simplifications such a reduction would entail.

References

[1] Qi Zhang, Yi Zhou, and Shaofeng Zou. Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance. Transactions on Machine Learning Research, 2025.

[2] Dingzhi Yu, Hongyi Tao, Yuanyu Wan, Luo Luo, and Lijun Zhang. Open Problem: Is AdamW Effective Under Heavy-Tailed Noise? arXiv:2606.23676, 2026.

3. On the Adam-atan2 citation

The cited reference does not support the reviewer’s objection. Instead, it supports our central theoretical point that ϵ\epsilon has a substantive, scale-dependent role in Adam. Its abstract explicitly states that ϵ\epsilon “must be scaled correctly to avoid gradient underflow” [3].

Section 4.3 reports comparable improvements from a small constant ϵ\epsilon, per-layer ϵ\epsilon, and Adam-atan2 in the mean-field experiments. It also reports instability when ϵ\epsilon is too small. These findings reinforce the importance of understanding how ϵ\epsilon interacts with gradient scales.

If ϵ\epsilon were truly negligible in Adam's optimization dynamics, there would be no reason to develop a new optimizer specifically to eliminate its effects. Everett et al. [3] introduce Adam-atan2 precisely because ϵ\epsilon breaks scale invariance and can adversely affect optimization performance. The very motivation for eliminating ϵ\epsilon demonstrates that it has a substantive effect on optimization. The cited work therefore directly supports our motivation for studying the role of ϵ\epsilon, rather than the reviewer's suggestion that this role should be set aside.

4. On using a tiny constant ϵ\epsilon

Theorem 4.2 allows the learning rate, momentum parameters, and shared ϵt\epsilon_t to depend on past observations. This includes choosing ϵt\epsilon_t as a tiny constant or setting ϵt=0\epsilon_t=0.

For 1<p<21<p<2, this class is subject to a dimension-uniform lower bound of Ω(T−p−12p)\Omega(T^{-\frac{p-1}{2p}}). Making the shared ϵ\epsilon arbitrarily small, or setting it to zero, still cannot attain the coordinatewise benchmark uniformly over our problem class. The setting suggested by the reviewer is already covered by our analysis.

5. On setting vt,i=0v_{t,i}=0

Under Adam's second-moment recursion,

vt,i=β2vt−1,i+ρ2gt,i2,v_{t,i}=\beta_2v_{t-1,i}+\rho_2g_{t,i}^2,

with ρ2>0\rho_2>0 and standard zero initialization, vt,i≡0v_{t,i}\equiv0 necessarily implies gt,i≡0g_{t,i}\equiv0, and consequently the numerator mt,i≡0m_{t,i}\equiv0. Thus, this coordinate receives no updates, regardless of the choice of positive ϵi\epsilon_i.

The remaining coordinates can continue updating normally. The irrelevance of ϵi\epsilon_i in a coordinate with identically zero stochastic gradients provides no information about its role in the active coordinates. If vt,i=0v_{t,i}=0 holds for every coordinate throughout training, then all stochastic gradients and momentum terms vanish, and the algorithm makes no updates at all under any well-defined update rule.

6. On 1<p≤21 < p\leq2

Both main theorems apply at p=2p=2. At this endpoint, the upper and lower time exponents both equal 1/41/4, so the strict exponent separation disappears. We therefore use 1<p≤21 < p\leq2 for the theorems and 1<p<21 < p < 2 when stating the strict separation between the two parameter settings. This distinction reflects a more mathematically precise formulation of our results.

References

[3] Katie E. Everett et al. Scaling Exponents Across Parameterizations and Optimizers. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:12666-12700, 2024.

I have reviewed the rebuttal and decided to keep my score.

The concerns remain:

  • Existing work such as (Liu, 2026b, already cited in this work) has resolved the problem with a better setting: small ϵ\epsilon, retaining bias-corrections, heavy-tailed noise;
    • The author claims the difference as "The setting considered here uses the expected ℓ1 norm of the true gradient, coordinatewise smoothness, and centered coordinatewise noise moments.". I just checked the paper (Liu, 2026b), it also use the coordinatewise version assumptions. I don't see much difference since the (λ, ε)-L1 stationary point in (Liu, 2026b) under nonsmooth setting can be extended to general first order stationary point in smooth case.
  • The motivation again, "the role of ϵ\epsilon as a tuned optimization parameter is a distinction that should be set aside, not emphasized."

The reviewer's response overlooks several substantive mathematical differences between our work and that of Liu [1], while providing no evidence to support the claim that ϵ\epsilon should be set aside.

1. On the comparison with Liu's work

The reviewer states, "I don't see much difference." However, the differences between the two works are very clear:

  1. Different assumptions. Their work requires the true gradient to be coordinatewise bounded, whereas we assume coordinatewise smoothness without requiring bounded gradients. Even the simplest quadratic function f(x)=12∥x∥22f(x)=\frac12\lVert x\rVert_2^2 satisfies our assumptions but violates theirs, since ∇f(x)=x\nabla f(x)=x is unbounded over the entire space. This demonstrates a substantive difference between the problem classes studied by the two papers.

  2. Different convergence guarantees. Their work establishes a (λ,ϵ)(\lambda,\epsilon)-L1L_1 stationarity guarantee for an exponential moving average of the iterates, whereas we directly control 1T∑t=1TE∥∇f(xt)∥1\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\lVert\nabla f(x_t)\rVert_1 at the original iterates. The reviewer claims that the former can be extended to ordinary first-order stationarity. However, even assuming such a conversion could be achieved without any loss, the resulting time exponent would still be smaller than ours.

  3. Different momentum parameter restrictions. Their abstract and contribution statements explicitly emphasize that the main theoretical results are established under β1=β2\beta_1=\beta_2. Their final explicit heavy-tailed convergence rate still relies on this restriction. In contrast, our upper bound does not require β1=β2\beta_1=\beta_2.

  4. Different time exponents. Their time exponent is 2(3p−4)19p−12\frac{2(3p-4)}{19p-12} under β1=β2\beta_1=\beta_2 and 4/3<p≤24/3<p\leq2, whereas ours is p−13p−2\frac{p-1}{3p-2} for 1<p≤21<p\leq2. In particular, at p=2p=2, the former gives 2/132/13, while ours gives 1/41/4.

  5. Different theoretical conclusions. Their work provides convergence upper bounds, and the author explicitly acknowledges in the conclusion that "our finite-time rates, for both known and unknown pp, are clearly suboptimal" [1]. In contrast, we establish both upper and lower bounds and explain why a shared ϵ\epsilon cannot achieve the dimension-uniform optimal rate.

2. On the significance of ϵ\epsilon

The reviewer repeatedly argues that ϵ\epsilon is unimportant, but provides neither a substantive theoretical justification nor any supporting reference for this judgment. In fact, both existing literature and our theoretical results directly contradict this claim.

Moreover, even the paper by Everett et al. [2], cited by the reviewer, demonstrates that ϵ\epsilon breaks Adam's scale invariance. Our work further establishes that the structure of the parameter directly determines whether the algorithm can achieve the optimal convergence rate.

A “motivation” is not evidence.

References

[1] Zijian Liu. Adam Converges in Nonsmooth Nonconvex Optimization. arXiv preprint arXiv:2606.22326, 2026.

[2] Katie E. Everett, Lechao Xiao, Mitchell Wortsman, et al. Scaling Exponents Across Parameterizations and Optimizers. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:12666–12700, 2024.

Reviewer_7nihOpenReview ↗
Rating
4
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the role of the stabilization parameter in Adam under coordinatewise smoothness and heavy-tailed stochastic-gradient noise with finite moments. The main positive result shows that choosing a separate epsilon-i according to the noise and curvature scales of each coordinate achieves the sharp heavy-tailed l1 stationarity rate. The main negative result shows that Adam with a single shared epsilon cannot achieve this rate uniformly over dimension and coordinate scales.

Critical strengths

Very well written paper. It is a bit on the technical end, but for the most part was written fine and understandable. Also, the positive and negative results complement eachother well.

Critical weaknesses

None

Questions for authors

How tight is the dimension dependence in the shared epsilon upper bound? Remark 4.4 gives an O(d) prefactor while the lower bound establishes that some dimension dependence is necessary for the dimension-uniform comparison. Can the authors say whether a linear dependence on d is necessary, or whether substantially better dependence is possible?

Reviewer_tCY8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Controlling momentum noise and gradient drift is important for Adam under heavy-tailed noise. This motivates the use of coordinatewise stabilization parameters εi\varepsilon_i. This paper establishes a dimension-uniform convergence separation between coordinatewise- and shared-ε\varepsilon Adam. The authors show that coordinatewise stabilization achieves the optimal convergence rate, whereas shared stabilization cannot guarantee the same rate uniformly over dimensions and coordinate heterogeneity.

Critical strengths
  • The coordinatewise-ε\varepsilon analysis achieves the optimal convergence rate $ O\left(T^{-\frac{p-1}{3p-2}}\right), $ while the lower bound for shared-ε\varepsilon Adam shows that this rate cannot be achieved uniformly across dimensions and heterogeneous coordinate scales.

  • The theory connects the role of the stabilization parameter directly to momentum noise and gradient drift, providing theoretical insight into an implementation detail of Adam.

Critical weaknesses
  • In the paper, the significance and implications of coordinate-wise ϵ\epsilon are not sufficiently emphasized in the Abstract, Introduction, and Contributions sections. Reader is hard to infer the overall conclusions from separate theorems and remarks.

  • Given the practical effectiveness of Adam, it remain unclear whether a shared ϵ\epsilon parameter is indeed a critical bottleneck in real applications. The paper provides no numerical experiments to evaluate the proposed coordinatewise modification. Given the strong theoretical claims, it is necessary to have convincing experiments to testify the significance of the problem studied in this work.

  • The theoretical guarantees rely on some problem-specific parameters. It will be better to have further discussion of the feasibility of these assumptions and practical parameter choices.

Questions for authors
  • Can the authors compare the proposed coordinatewise-Adam with the shared-Adam to validate the theoretical claims across different standard deep learning models and datasets?

  • For each main result. It will be better to have further discussion of the feasibility of these assumptions and practical parameter choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.