- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how gradient heterogeneity and coordinate geometry affect Adam’s convergence under fixed memory parameters. It establishes a sharp stationary drift inequality in one dimension, yielding convergence below the normalized heterogeneity threshold under strong convexity, together with asymptotically matching divergence examples. In higher dimensions, it constructs divergent positive-definite quadratic problems with a common minimizer and C arbitrarily close to one.
Critical strengths
-
The scalar inequality (Theorem 1) drives the multi-dimensional theory. The paper builds its multi-dimensional results on it: the vector pairing (Prop. 1), coordinatewise and separable-average convergence (Thm. 7), the shared-second-moment guarantee (Thm. 20), and the conditioning lower bound for divergent rays (Thm. 5).
-
The counterexamples sit where convergence was widely expected. The instances have positive-definite components, a common minimizer, diminishing steps, vanishing relative gradient variance and default β's. They settle the question in Remark 7 of Zhang et al. (2026), and the answer depends on dimension.
-
The counterexamples are not just constructions but come with a mechanism and necessary conditions. Coordinatewise normalization amplifies many small, disagreeing gradient coordinates. Theorem 5 and Proposition 7 show that every outward ray requires κ ≳ 1/(C−1) and p ≳ 1/(C−1). Shared second-moment normalization and AMSGrad-style running maxima both remove the mechanism, which pins the cause on coordinatewise normalization.
Critical weaknesses
-
The multi-dimensional convergence guarantees are weak. The coordinatewise condition D<2n is very strong. in the paper’s own Transformer experiment it is of order 10¹⁰.
-
A complete convergence/divergence picture is given only in three cases: one dimension, ray trajectories, and the memoryless setting. Non-ray trajectories remain open.
-
The results are numerous and fragmented.
- The central question is clear: which values of C guarantee convergence at fixed memory. However, it is diluted by roughly 21 theorems across the main text and appendices.
- The paper uses several notions of “sharpness” with different meanings, and the rate results (Thm. 8) read as add-ons.
- The Transformer experiment tests none of the thresholds; it only confirms the known rotation sensitivity of Adam. A single phase diagram over C, κ and memory, marking proven-convergent, proven-divergent and open regions, would make the contribution much easier to follow.
Questions for authors
See Weakness.
Where the revision addresses your two points
Edited on 18 Nov so that the line numbers match the PDF uploaded that day.
As the discussion closes soon, here is where the revised PDF addresses your two points, in case a quick check is easier than rereading our reply. Line numbers refer to the current PDF.
Undefined terms (your lines 160 and 177). Each term is now explained in words where it is first used: m, v and E already in the introduction (line 066), then dt at line 162, the frozen field at line 182 and E in general at line 192. Outward rays are defined at line 258, before the first theorem that uses them. The appendix opens with a glossary (page 13) that explains each of these terms, and the other main terms, in one or two plain sentences.
Contribution relative to Zhang et al. (2026). From line 054 the introduction asks one question, how large C=D1/n can be when (β1,β2) are fixed, and gives three answers. In one dimension C<2 is safe and the constant 2 cannot be improved (line 060). In higher dimensions no C>1 is safe by itself (line 071). The threshold 2 returns when the disagreement is bounded coordinate by coordinate, when the average is separable, or when Adam uses one shared second moment (line 086). Lines 093–100 compare these results with Zhang et al. (2026) point by point and answer their Remark 7, and the paragraph from line 102 explains why a threshold matters and what the results mean in practice. A table on page 3 lists all results in one place, and page 14 sets them beside those of Zhang et al. (2022; 2026).
We would be grateful to know, before the discussion closes, whether this addresses your concerns.