- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how a perturbation of a mini-batch gradient can affect the training loss in the future, through the optimizer state, especially the 1st- and 2nd-moment estimates of Adam. The paper presents a theoretical framework to treat the loss dynamics as a linear dynamic system, and provides theoretical and empirical evidence that the approximation can work well even with non-linear deep neural networks. In addition, the paper also tries to find and investigate some metrics that are consistent across any (random) future sampling of the training data.
Critical strengths
The paper has its value in documenting the difference between the empirical optimizer dynamics and its linear approximation. Knowing that the linear approximation works, especially knowing the magnitude of the Normalized Root Mean Square Error (NRMSR), at least within a horizon of training steps for quadratic models, training steps for CIFAR-10 CNNs and MLPs, and training steps for pre-trained language models, could be useful for related study.
Critical weaknesses
The theoretical part (Section 1--3) of the paper is well-written, but the theory is a bit mundane in the sense that it does not bring much empirical insights. To elaborate:
-
First of all, the effect of a mini-batch perturbation is not confined to the step at which it is observed, even for SGD: because the perturbation will change the model parameters, which is also a state. Yet, this paper starts by "With adaptive optimizers such as AdamW, however, ...", and especially focuses on adaptive optimizers such as AdamW, which feels weird to me. In other words, this theoretical discussion does not seem to point out anything specific to adaptive optimizers.
-
Of course, the existence of optimizer-specific states, such as mu and nu (the 1st- and 2nd-moment estimates) for Adam, indeed makes the dynamics much more complicated. The value of this paper is showing that, despite the complexity, one can still apply linear approximation and convert the dynamics into a linear dynamic system. Moreover, the climax is Proposition 3.3, where the authors show that with this simple linear dynamic system, one can find examples such that the response gets amplified as the dynamic system evolves, even when one assumes that the transition matrix of the dynamic system is a constant, contraction map. However, it is not clear at all if this hypothetical example has any connections with empirical settings. The transition matrix depends on the random training example, so it is not a constant matrix even with a quadratic loss function, under stochastic training. Rather than "a perturbation shock gets delayed amplified response", the empirical picture is more like "some noise gets into the dynamic system at each step".
-
In particular, the paper explicitly tries to connect the theoretical framework to loss spikes (on Page 5). Currently, this connection is purely speculative: A delayed response that surpasses a threshold does not necessarily mean a sudden, peaky surge that one observes as loss spikes in practice -- it could be also be a smooth and gradual fluctuation. There is no figure in the paper that actually illustrates how the hypothetical linear dynamic system evolves -- Does it look spiky?
What makes the situation worse, is that the Experiments part (Section 4) of the paper is quite unclear. It even feels like AI-generated without sufficient human supervision, because there are so many phrases, seemingly indicating detailed experiment settings, that are unexplained. For example:
-
In Section 4.1, which measures the empirical error of the linear dynamic system, it seems that different metrics are used for different model sizes (quadratic models, CIFAR-10 CNNs and MLPs, and pre-trained LMs)? -- Because the column titles in Table 1, 2, 3 are completely different. Yet it is unclear what the differences are, and why different metrics are used. For example, the paragraph below Table 3, it says: "At this scale we estimate the end-to-end ISO directional response numerically using centered finite differences of the probe logits, followed by the exact cross-entropy differential; we denote this quantity by ISO TANGENT (FD)." So what does "centered finite differences of the probe logits, followed by the exact cross-entropy differential" mean exactly? Why is this the reasonable choice "at this scale"? How do you abbreviate "ISO TANGENT" into "FD"? And what is the difference between "ISO-FD" in Table 3 and "Full ISO" in Table 2?
-
In Section 4.2, it is supposed to investigate some metrics that are consistent across any random future sampling of the training data; Equation (6) proposes such a metric. Then, the paper says: "This is a protocol-specific variance ratio, not an information-theoretic fraction of predictable risk. We also report the median Spearman correlation between each branch ranking and the conditional-mean ranking. To test whether the structure can be extracted without observing any sampled future branch, ..." I don't understand what it is saying at all. What is a "protocol-specific variance ratio"? What is an "information-theoretic fraction of predictable risk"? What is a "branch ranking"? What branches? Equation (6) proposes a metric, but why is it called "structure"? What structure are you talking about?
Overall, I feel the paper should be rated slightly below "Weak acceptance". Currently, I don't think it should be accepted by the conference. I give it a "3" rating only in case I can get all my concerns addressed by the authors; and because it does feel better than some other papers I reviewed.
Questions for authors
-
The horizon becomes smaller as the model size gets larger: Is this due to the computation cost, or because the empirical error of the linear approximation generally gets larger as the model sizes grow?
-
Does the error increase rapidly as increases?
-
How does the linear dynamic system evolve empirically? Do you have figures to illustrate?
-
What is brought by the linear dynamic system approximation? I think the experiments in Section 4.2 can be done without mentioning any linear dynamic system approximation -- So what is the point?
2 comments
Thanks for the response
The paper is improved by the revision: the overall motivation, and the experiments, now become clearer. New figures are added into the paper to illustrate the behavior of both the exact probe-loss response and the analytic tangent proposed by the paper.
But I feel the more essential Critical Weaknesses that I raised, like both SGD and Adam are state-dependent so why only focusing on Adam, what is so special in Adam; and does the linear approximation actually bring any non-trivial insights into practice -- these are acknowledged but not straightforwardly answered. Therefore, I would like to keep my score, while still maintaining the assessment that the paper is slightly below "Weak acceptance". This is a decent paper which does what it wants to do; but it is still slightly below because I feel it does not establish enough impact regarding why these experiments matter.
Thank you for the questions about optimizer memory, stochastic futures, and the experiments. We revised these explanations and added ViT-20M trajectories and horizon-dependent errors in Appendix F.2.4.
The introduction now explicitly notes that SGD propagates an initial parameter displacement. AdamW additionally stores the perturbation in two moment states. Our new full-state and parameter-only branches start with identical post-shock parameters and immediate probe losses. The parameter-only branch restores the moments to their control values once, and both branches subsequently run normal AdamW. Retaining the perturbed moments increases the 32-step absolute peak in all six trials. Figures 2 and 3 show this comparison and the signed channel responses.
Both trajectories use the same future minibatches and randomness. Their state difference evolves under the same random update map evaluated at two states. Future randomness changes the propagation of the initiating perturbation, while pairing isolates its effect. The full tangent uses the changing Jacobians and probe readouts along this continuation.
Proposition 3.3 gives a tractable memory-feedback calculation: delayed amplification can occur when the frozen matrix is asymptotically stable, with spectral radius below one. This condition does not require Euclidean-norm contraction. Section 3.5 now distinguishes this frozen result from the time-varying experimental dynamics.
The threshold relation in §3.4 characterizes a loss-threshold exceedance. It does not determine whether that exceedance is abrupt, narrow, or followed by rapid recovery. Figures 3–4 show signed paired probe-loss responses; identifying the mechanism of spontaneous training-loss spikes requires additional evidence about the actual training-loss trajectory and its temporal shape.
Analytic Full ISO propagates the parameter–moment tangent and applies the probe-loss differential. ISO-FD means finite-difference estimation: in Pythia, we estimate the probe-logit derivative by centered differences and apply the cross-entropy differential at the control logits (Eqs. 77–78). This avoids directly subtracting nearly equal scalar losses. Both target the same local directional response, with different computational implementations.
NRMSE measures trajectory error divided by the norm of the exact trajectory; cosine measures trajectory alignment; Spearman correlation measures candidate ordering by peak magnitude. The tables report these complementary properties. The FD consistency test separately checks sensitivity to the difference step.
The reported windows are evaluation horizons, rather than estimates of the longest accurate horizon. The cross-family protocols do not isolate model-size effects from other protocol and computational differences. Pythia uses H = 8 at all three model sizes; Table 14 shows no systematic deterioration of local NRMSE with size in those conditions.
The new ViT experiment evaluates nested prefixes of the same 32-step continuation. From H = 8 to 32, median NRMSE rises from 0.163 to 0.564 on CIFAR-10 and from 0.108 to 0.125 on synthetic quadrants (Table 16). At H = 32, the three CIFAR-10 values are 19.868, 0.047, and 0.564. Figure 4 overlays the exact response and analytic tangent for the median-error seed; Figure 3 shows all six state-intervention trials. These results show substantial seed dependence. Theorem 3.4 explains why a correct local derivative can lose finite-shock accuracy: subsequent propagators carry and amplify the remainders generated along the trajectory.
A branch is one sampled future minibatch continuation, shared across candidates. Its ranking orders candidates by their peak on that future; the mean ranking averages peaks over 32 futures. Equation (6) divides the variance across candidate means by its sum with the average within-candidate variance across futures.
These descriptive quantities come from exact paired runs and can indeed be computed without a tangent model. They establish whether candidate-specific effects persist across futures. The linear model enters in the present-frozen and reference-surrogate ISO scores, which use current information to estimate that mean ordering without observing the sampled futures. Table 4 illustrates the distinction: matched quadratic rankings are stable across futures (branch correlation 0.909), yet present-frozen ISO has correlation -0.066 with the mean ordering; matched MLP–GELU gives 0.941. Stable candidate differences and their recovery by a present-time model are therefore separate empirical questions.