acceptodds
Under review as a conference paper at ICLR 2027

Clock Phases and Signed Feedback in Adaptive Optimizer Memory

Abstract

Adam computes its first and second moments from the same minibatch. We show that this detail governs how Adam responds to the order of its training data. Linearizing a step, the adaptive contribution to order dispersion splits into a non-negative preconditioner term and a numerator–denominator covariance . We prove that under same-sample coupling is negative whenever momentum agrees in sign with the current gradient, and that it outweighs when is close to one; an independent second-moment sample makes vanish, so decoupled memory can only amplify order dispersion. Coupling also gives Adam a per-coordinate step bound, a known consequence of Cauchy–Schwarz that we show holds if and only if the second-moment input dominates the numerator input. We test both consequences by changing the pairing and nothing else. In a small adapter, manipulating the alignment between numerator and second-moment fluctuations flips the sign of the adaptive correction in all 18 units. At mature ResNet-18 checkpoints, replacing each second-moment sample by an independent draw with the same task and marginal law increases order dispersion on three task pairs, at every checkpoint for parameter and prediction variance. On pretrained Pythia-160M the same replacement drives a small set of coordinates – times past the step bound at the first update and the model diverges within a few updates, even when fifteen of sixteen sequences in the second-moment minibatch are shared.

Primary area:
optimization
Keywords:
adaptive optimizers, AdamW, training-order effects, optimizer memory, second moments, batch-size scaling, optimizer state

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_HeJYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript focuses on the training effect by using the same mini-batch. The authors study the coupling relationship between update, data order. They also provide somehow a little bit "formula-style" theorectical results and corresponding supportive experiments.

Critical strengths

The pipeline design is clean. The authors have acknowledged about the limitations of their work.

Critical weaknesses

My main concern is the theorecitical novelty since it is already known that Adam update dynamics is bounded (Kingma & Ba, 2015, Zou et al., 2019, Defossez et al., 2022). That is because Adam optimizer uses the second-moment of gradient as the denominator. So, if the gradient is large, then the update should be should, but the second-moment of gradients are larger than the numerator making the update magnitude small, that is why these "large updates" are bounded. That is why Adafactor and StableAdamW use additional mechanisms to control this thing. I do not understand what new understanding here and what "different" pratical benefits this manuscript can offer.

Another concern is Theorem 1 only consider one-step first order effect. This means it is unclear why and whether "this work" can explain the long traning runs behavior. Especially, it is unclear whether it work in large models or not.

Questions for authors

Can the authors add experiments that controls for update size? e.g., in ResNet-18 setting? If we do not control this, we can not tell whether the increased order dispersion comes from which part, e.g., covariance term C or large update?

Does the Pythia divergence occur under the gradient clipping and even under realistic \epsilon? The problem is that cipping can rescure the decoupled arm, if so, what these results can tell us beyond previous existing methods?

In Theorem 1, damping requires the formula [Consequently: (i) on page 3] per coordinate. To do an alabtion experiemnt, I think the authors can measure how many coordinates actually satidfy the consition during training?

Reviewer_tJBnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper dives deep into the necessity of coupling of the gradient sequence in the two averagers in Adam optimizers, i.e., the first and the second order moment estimators. It is previously known since the first advent of Adam optimizer that this coincidence is somewhat important to produce a stable training dynamics. The paper presents a series of theory and controlled experiments on ResNet (ImageNet), Pythia-160M (Github/Wikipedia) to demonstrate this necessity.

Critical strengths
  • Consolidating the commonly known practical wisdom is a great way to contribute to the community, however simple the fact it is. I value the motivation of this research.
  • Up to my knowledge, theoretical justification on the necessity of numerator-denominator alignment is new.
  • I do appreciate how this paper maintains short length and focusing on a single particular research question without extraneous arguments that are outside the central question.
Critical weaknesses
  • One downside of this paper is that the “wisdom” it proves is somewhat trivial. Gradients are not a stationary signal. The model is meant to innovate during training and so does the gradients it produces. Changing the order of the collection of the statistics therefore will break down overall structure, which we can easily infer from common knowledge. However, if we can quantify how much will a misalignment contribute to the overall stability of the training, then we have a solid knowledge about this phenomenon. Such contribution will be acknowledged.
  • The experiments can be improved. For example, I see that Theorem 1 is the central claim. The experiments using Pythia diverges and does not test this theorem. The authors honestly mention that in lines 310-312 by delegating this job to some future work. However, I believe it should not be easily delegated.
  • Likewise, the authors honestly mention that their ResNet experiments do not separate the changing step size in the cause of decoupling. It is good to have such level of honesty, but it is better to have more carefully draw experiments. I believe that the direction of this research is acknowledgeable, but the evidences in both theoretical and empirical sides can be improved.
Questions for authors
  • In real practices we use larger epsilon (~1e-6) than the current setting (1e-8). Is the divergence of Transformer finetuning occurring in this regime, too?
Reviewer_kdvLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how same-sample coupling in Adam affects training-order dispersion and stability. It derives local damping conditions from a signed covariance decomposition and analyzes a per-coordinate adaptive-step bound. AdamW experiments test the sign predictions in a small adapter and show increased ResNet-18 endpoint dispersion after replacing the second-moment input with an independent same-task minibatch. On Pythia-160M, this intervention causes step-bound violations and divergence, leaving order-dispersion suppression untested.

Critical strengths

The paper provides a clear theoretical analysis connecting same-sample coupling, a commonly taken-for-granted property of Adam in which the first- and second-moment estimates are updated using the same gradient, to training-order dispersion. The analysis offers a convincing explanation of how same-sample coupling contributes to reducing training-order variability. The experimental setups are clearly described, and the results effectively support the authors' theoretical predictions.

Critical weaknesses

Theorem 1 and the experiments in Section 5.2 do not sufficiently address the paper's central research question, "When does Adam's adaptive preconditioner amplify or suppress training-order variability?", in a general setting. Theorem 1 assumes a single-step update from a shared optimizer state, where variations arise solely from differences in the current gradients. This assumption limits the theoretical analysis to a local setting rather than the full training trajectory. The experiments in Section 5.2 investigate the effects of same-sample coupling on training-order dispersion over longer optimization horizons using controlled minibatch inputs. However, as the authors themselves acknowledge, removing same-sample coupling can also change the effective update magnitudes. Without additional experiments that control for these differences, the results do not provide sufficient evidence that the covariance between the first- and second-moment estimates is the primary mechanism responsible for suppressing training-order dispersion over extended optimization trajectories.

Questions for authors

I acknowledge the difficulty of analyzing and controlling the optimization dynamics of Adam. However, to extend the paper's claims to longer training trajectories and large language models, the step-matched control experiment discussed in Section 5.2 is necessary. Could the authors conduct this experiment, including on LLMs, to isolate the effect of same-sample coupling from differences in update magnitudes and provide more convincing empirical evidence of its role in reducing training-order dispersion over extended optimization horizons and in large-scale language model training?

Section 6 connects the findings to update clipping in Adafactor and StableAdamW. Could the authors clarify the scope of this connection, particularly for StableAdamW, which retains same-sample moment updates?

Reviewer_fRCdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the sensitivity of the Adam optimizer to the order of minibatches during training. The authors demonstrate that when the same mini-batch is used to update both the first- and second-order moments, the coupling between the numerator and the denominator can reduce the variance of the updates. Furthermore, the authors derive an upper bound on the update magnitude for individual coordinates and validate these theoretical findings through some small-scale experiments.

Critical strengths

This paper investigates an interesting mechanism within Adam and provides a theoretical explanation for it.

Critical weaknesses

Theorem 1 in this paper is based on a local first-order approximation. And, upper bounds on the magnitude of single-coordinate updates in the Adam algorithm have already been explored in existing literature. And, the paper’s experiments fail to adequately disentangle the effects of momentum coupling from those resulting from changes in the effective step size. Overall, the new theoretical contributions remain limited.

Questions for authors

Could the authors conduct some experiments to better verify the independent effect of coupling on the variance resulting from training order?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.