acceptodds
Under review as a conference paper at ICLR 2027

Sharp Heterogeneity Barriers for Adam: Basis-Aware Convergence and High-Dimensional Divergence

Abstract

Convergence proofs for Adam on finite sums often assume strong growth, . The constant is at least , with equality exactly when all component gradients agree, yet the known cyclic counterexamples have . We ask how much heterogeneity Adam tolerates at fixed memory , and the answer depends on the dimension and on the coordinate system. In one dimension the threshold is : a sharp inequality for Adam's averaged normalized update gives convergence below it, for strongly convex averages with Lipschitz component gradients, and divergent instances exist at . In higher dimensions, positive-definite quadratics with a common minimizer can make Adam diverge with arbitrarily close to one, under cyclic order, IID minibatches and random reshuffling; in the random cases the probability of divergence tends to one for distant starting points. An outward divergent ray needs at condition number , and this is exact under cyclic order. Bounded conditioning gives a dimension-independent convergence region, which is nonempty exactly when ; for this limit is sharp, because at every larger trajectories that keep turning can diverge as . The threshold returns under coordinatewise growth, for separable averages and with a single shared second moment, also under arbitrary epoch orders and IID sampling, with non-asymptotic bounds for cyclic order. Since a quadratic average is separable in its eigenbasis, a change of basis alone can turn divergence into convergence. All results allow denominator regularization and bias correction, and interval arithmetic certifies cyclic and IID divergence at the default . In a small Transformer, function-preserving rotations consistently increase test loss, and spectral bounds on Adam's frozen updates separate two bases with identical aggregate growth.

Primary area:
optimization
Keywords:
Adam, optimization theory, convergence and divergence, gradient heterogeneity, coordinatewise adaptivity, basis dependence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Wh9NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how gradient heterogeneity and coordinate geometry affect Adam’s convergence under fixed memory parameters. It establishes a sharp stationary drift inequality in one dimension, yielding convergence below the normalized heterogeneity threshold under strong convexity, together with asymptotically matching divergence examples. In higher dimensions, it constructs divergent positive-definite quadratic problems with a common minimizer and C arbitrarily close to one.

Critical strengths
  1. The scalar inequality (Theorem 1) drives the multi-dimensional theory. The paper builds its multi-dimensional results on it: the vector pairing (Prop. 1), coordinatewise and separable-average convergence (Thm. 7), the shared-second-moment guarantee (Thm. 20), and the conditioning lower bound for divergent rays (Thm. 5).

  2. The counterexamples sit where convergence was widely expected. The instances have positive-definite components, a common minimizer, diminishing steps, vanishing relative gradient variance and default β's. They settle the question in Remark 7 of Zhang et al. (2026), and the answer depends on dimension.

  3. The counterexamples are not just constructions but come with a mechanism and necessary conditions. Coordinatewise normalization amplifies many small, disagreeing gradient coordinates. Theorem 5 and Proposition 7 show that every outward ray requires κ ≳ 1/(C−1) and p ≳ 1/(C−1). Shared second-moment normalization and AMSGrad-style running maxima both remove the mechanism, which pins the cause on coordinatewise normalization.

Critical weaknesses
  1. The multi-dimensional convergence guarantees are weak. The coordinatewise condition D<2n is very strong. in the paper’s own Transformer experiment it is of order 10¹⁰.

  2. A complete convergence/divergence picture is given only in three cases: one dimension, ray trajectories, and the memoryless setting. Non-ray trajectories remain open.

  3. The results are numerous and fragmented.

  • The central question is clear: which values of C guarantee convergence at fixed memory. However, it is diluted by roughly 21 theorems across the main text and appendices.
  • The paper uses several notions of “sharpness” with different meanings, and the rate results (Thm. 8) read as add-ons.
  • The Transformer experiment tests none of the thresholds; it only confirms the known rotation sensitivity of Adam. A single phase diagram over C, κ and memory, marking proven-convergent, proven-divergent and open regions, would make the contribution much easier to follow.
Questions for authors

See Weakness.

Reviewer_wepHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how gradient heterogeneity affects Adam at fixed memory parameters. It gives an asymptotically sharp threshold of D1=2nD_1=2n in one dimension, while showing that Adam can diverge in higher dimensions even when D1/nD_1/n is arbitrarily close to one. The paper also gives convergence guarantees under coordinatewise growth, separable averages, and shared second moments. A change of basis can turn divergence into convergence on the same quadratic problem. Synthetic instances and a small Transformer experiment illustrate these results and Adam's basis sensitivity.

My recommendation is Weak Accept. I find the problem meaningful and the main theoretical results technically interesting. The scalar threshold and high-dimensional counterexamples explain why aggregate heterogeneity alone is insufficient to characterize Adam's behavior. The exact threshold for divergent rays also holds for all admissible memory parameters. My reservations concern the scope of the general-trajectory conditioning result and the practical interpretation of the Transformer diagnostics. These are relatively minor limitations, and I think the main theoretical contribution is strong enough to support acceptance.

Critical strengths

I find the scalar inequality and its sharpness construction technically interesting. The analysis connects the stationary update to actual Adam iterates and covers zero initial moments, denominator regularization, and bias correction.

The high-dimensional counterexamples give a useful distinction between aggregate heterogeneity and coordinate geometry. The components are positive definite and share a minimizer, yet Adam can diverge with arbitrarily small relative gradient variance. The result that a change of basis can restore convergence is also compelling.

The Transformer experiments use careful controls, including identical initial functions and shared batches. The independent replication cohort and rotation-equivariant optimizer controls make the observed basis sensitivity more convincing.

Critical weaknesses

The conditioning cutoff for general trajectories is only proved sharp without memory. Theorem 5 gives an exact threshold for regular outward rays for all admissible memory parameters. However, Theorem 6 establishes divergence arbitrarily close to minimal heterogeneity for κ>3+22\kappa>3+2\sqrt{2} only when β1=β2=0\beta_1=\beta_2=0. The corresponding cutoff remains unresolved for Adam with memory. The paper states this restriction clearly, but it limits what the result tells us about commonly used Adam settings. A brief discussion of the obstacles to extending the non-ray construction to nonzero memory would be helpful.

The practical interpretation of the frozen-update diagnostics is somewhat limited. The matched-state diagnostics in Section 6 convincingly show that aggregate heterogeneity misses differences between the frozen updates in the two bases. Their evaluation at training endpoints using stationary moments, however, leaves open how informative they are about actual updates during training. The paper does not claim to establish a causal explanation of the loss gap. Still, measurements at intermediate checkpoints, compared with actual gradient-update alignment, would help connect the diagnostics to optimization behavior.

Questions for authors

No additional comments. Just address the weaknesses mentioned above.

Reviewer_EF2MOpenReview ↗
Rating
2(+1)
Confidence
2(+1)
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper aims to understand the connection between the heterogeneity of gradient evaluations in the stochastic finite-sum optimization and the convergence of Adam. The heterogeneity is modeled by D1D_1 in the strong growth condition. The manuscripts then presents a series of theoretical results named "Sharp stationary drift inequality", "Cyclic convergence below 2n2n", "Divergence just above 2n2n", "Sharp conditioning threshold for divergent rays". The manuscript also considers coordinate-wise strong growth condition and presents the theoretical analysis there.

Critical strengths

The paper tackles an important problem of analyzing the convergence of Adam and constructing geometry-dependent counter examples.

Critical weaknesses
  1. The writing is difficult to follow. There are too many new terminologies. For example, in line 160, the title says frozen fields and rays. None of the terminology is explained clearly. In line 177, expectation is not defined. dtd_t is not defined either. The manuscript reads like an unpolished report from GPT.

  2. The major contribution is unclear. The significance of the three contributions listed from line 59 is unclear compared with the results from Zhang et al. (2026). Previous works have already constructed non-converging counter-examples, why is the scalar threshold important? The notion of minimal heterogeneity is difficult to understand. The practical significance of geometry-aware convergence is also abstruse.

Questions for authors
  1. Can authors use simple language to describe the novelty of the theoretical analysis and the new insights?
2 comments
AuthorsOpenReview ↗

Where the revision addresses your two points

Edited on 18 Nov so that the line numbers match the PDF uploaded that day.

As the discussion closes soon, here is where the revised PDF addresses your two points, in case a quick check is easier than rereading our reply. Line numbers refer to the current PDF.

  1. Undefined terms (your lines 160 and 177). Each term is now explained in words where it is first used: mm, vv and E\mathbb E already in the introduction (line 066), then dtd_t at line 162, the frozen field at line 182 and E\mathbb E in general at line 192. Outward rays are defined at line 258, before the first theorem that uses them. The appendix opens with a glossary (page 13) that explains each of these terms, and the other main terms, in one or two plain sentences.

  2. Contribution relative to Zhang et al. (2026). From line 054 the introduction asks one question, how large C=D1/nC=D_1/n can be when (β1,β2)(\beta_1,\beta_2) are fixed, and gives three answers. In one dimension C<2C<2 is safe and the constant 2 cannot be improved (line 060). In higher dimensions no C>1C>1 is safe by itself (line 071). The threshold 2 returns when the disagreement is bounded coordinate by coordinate, when the average is separable, or when Adam uses one shared second moment (line 086). Lines 093–100 compare these results with Zhang et al. (2026) point by point and answer their Remark 7, and the paragraph from line 102 explains why a threshold matters and what the results mean in practice. A table on page 3 lists all results in one place, and page 14 sets them beside those of Zhang et al. (2022; 2026).

We would be grateful to know, before the discussion closes, whether this addresses your concerns.

AuthorsOpenReview ↗

The paper in one paragraph

Edited on 18 Nov so that the line numbers match the PDF uploaded that day.

Following up on our earlier note, here is the paper in one paragraph, in case it helps before the discussion closes at the end of 18 Nov (AoE).

We ask how much the gradients of the individual training examples may disagree before Adam, with fixed settings such as (β1,β2)=(0.9,0.999)(\beta_1,\beta_2)=(0.9,0.999), can fail to converge. Disagreement is measured by C=D1/nC=D_1/n, whose smallest possible value is 1. In one dimension C<2C<2 is safe, and 2 cannot be improved. In higher dimensions no C>1C>1 is safe by itself, because Adam rescales each coordinate separately; the threshold 2 returns when the disagreement is bounded coordinate by coordinate, when the average is separable, or when Adam uses one shared second moment. This answers Remark 7 of Zhang et al. (2026): Adam can fail in higher dimensions even as CC approaches 1, while in one dimension no such example exists below C=2C=2 (lines 054–107).

Page 13 explains each term in one or two plain sentences, and page 14 compares our results with Zhang et al. (2022; 2026) in a short table.

If this resolves your two points, we would be grateful if your final review could reflect it. If anything is still unclear, a pointer to the sentence is enough, and we will clarify it right away.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.