acceptodds
Under review as a conference paper at ICLR 2027

Finite-Horizon Input–Output Dynamics of Minibatch Perturbations in AdamW

19 views

Abstract

A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input–state–output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations.

Primary area:
interpretability and explainable AI
Keywords:
mini-batch, loss spike, finite-horizon control, loss propogation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hLaiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how a perturbation of a mini-batch gradient can affect the training loss in the future, through the optimizer state, especially the 1st- and 2nd-moment estimates of Adam. The paper presents a theoretical framework to treat the loss dynamics as a linear dynamic system, and provides theoretical and empirical evidence that the approximation can work well even with non-linear deep neural networks. In addition, the paper also tries to find and investigate some metrics that are consistent across any (random) future sampling of the training data.

Critical strengths

The paper has its value in documenting the difference between the empirical optimizer dynamics and its linear approximation. Knowing that the linear approximation works, especially knowing the magnitude of the Normalized Root Mean Square Error (NRMSR), at least within a horizon of H=32H=32 training steps for quadratic models, H=12H=12 training steps for CIFAR-10 CNNs and MLPs, and H=8H=8 training steps for pre-trained language models, could be useful for related study.

Critical weaknesses

The theoretical part (Section 1--3) of the paper is well-written, but the theory is a bit mundane in the sense that it does not bring much empirical insights. To elaborate:

  1. First of all, the effect of a mini-batch perturbation is not confined to the step at which it is observed, even for SGD: because the perturbation will change the model parameters, which is also a state. Yet, this paper starts by "With adaptive optimizers such as AdamW, however, ...", and especially focuses on adaptive optimizers such as AdamW, which feels weird to me. In other words, this theoretical discussion does not seem to point out anything specific to adaptive optimizers.

  2. Of course, the existence of optimizer-specific states, such as mu and nu (the 1st- and 2nd-moment estimates) for Adam, indeed makes the dynamics much more complicated. The value of this paper is showing that, despite the complexity, one can still apply linear approximation and convert the dynamics into a linear dynamic system. Moreover, the climax is Proposition 3.3, where the authors show that with this simple linear dynamic system, one can find examples such that the response gets amplified as the dynamic system evolves, even when one assumes that the transition matrix of the dynamic system is a constant, contraction map. However, it is not clear at all if this hypothetical example has any connections with empirical settings. The transition matrix depends on the random training example, so it is not a constant matrix even with a quadratic loss function, under stochastic training. Rather than "a perturbation shock gets delayed amplified response", the empirical picture is more like "some noise gets into the dynamic system at each step".

  3. In particular, the paper explicitly tries to connect the theoretical framework to loss spikes (on Page 5). Currently, this connection is purely speculative: A delayed response that surpasses a threshold does not necessarily mean a sudden, peaky surge that one observes as loss spikes in practice -- it could be also be a smooth and gradual fluctuation. There is no figure in the paper that actually illustrates how the hypothetical linear dynamic system evolves -- Does it look spiky?

What makes the situation worse, is that the Experiments part (Section 4) of the paper is quite unclear. It even feels like AI-generated without sufficient human supervision, because there are so many phrases, seemingly indicating detailed experiment settings, that are unexplained. For example:

  1. In Section 4.1, which measures the empirical error of the linear dynamic system, it seems that different metrics are used for different model sizes (quadratic models, CIFAR-10 CNNs and MLPs, and pre-trained LMs)? -- Because the column titles in Table 1, 2, 3 are completely different. Yet it is unclear what the differences are, and why different metrics are used. For example, the paragraph below Table 3, it says: "At this scale we estimate the end-to-end ISO directional response numerically using centered finite differences of the probe logits, followed by the exact cross-entropy differential; we denote this quantity by ISO TANGENT (FD)." So what does "centered finite differences of the probe logits, followed by the exact cross-entropy differential" mean exactly? Why is this the reasonable choice "at this scale"? How do you abbreviate "ISO TANGENT" into "FD"? And what is the difference between "ISO-FD" in Table 3 and "Full ISO" in Table 2?

  2. In Section 4.2, it is supposed to investigate some metrics that are consistent across any random future sampling of the training data; Equation (6) proposes such a metric. Then, the paper says: "This is a protocol-specific variance ratio, not an information-theoretic fraction of predictable risk. We also report the median Spearman correlation between each branch ranking and the conditional-mean ranking. To test whether the structure can be extracted without observing any sampled future branch, ..." I don't understand what it is saying at all. What is a "protocol-specific variance ratio"? What is an "information-theoretic fraction of predictable risk"? What is a "branch ranking"? What branches? Equation (6) proposes a metric, but why is it called "structure"? What structure are you talking about?

Overall, I feel the paper should be rated slightly below "Weak acceptance". Currently, I don't think it should be accepted by the conference. I give it a "3" rating only in case I can get all my concerns addressed by the authors; and because it does feel better than some other papers I reviewed.

Questions for authors
  1. The horizon HH becomes smaller as the model size gets larger: Is this due to the computation cost, or because the empirical error of the linear approximation generally gets larger as the model sizes grow?

  2. Does the error increase rapidly as HH increases?

  3. How does the linear dynamic system evolve empirically? Do you have figures to illustrate?

  4. What is brought by the linear dynamic system approximation? I think the experiments in Section 4.2 can be done without mentioning any linear dynamic system approximation -- So what is the point?

2 comments

Thank you for the questions about optimizer memory, stochastic futures, and the experiments. We revised these explanations and added ViT-20M trajectories and horizon-dependent errors in Appendix F.2.4.

  1. What is specific to optimizer memory?

The introduction now explicitly notes that SGD propagates an initial parameter displacement. AdamW additionally stores the perturbation in two moment states. Our new full-state and parameter-only branches start with identical post-shock parameters and immediate probe losses. The parameter-only branch restores the moments to their control values once, and both branches subsequently run normal AdamW. Retaining the perturbed moments increases the 32-step absolute peak in all six trials. Figures 2 and 3 show this comparison and the signed channel responses.

Both trajectories use the same future minibatches and randomness. Their state difference evolves under the same random update map evaluated at two states. Future randomness changes the propagation of the initiating perturbation, while pairing isolates its effect. The full tangent uses the changing Jacobians and probe readouts along this continuation.

  1. Frozen dynamics and loss spikes.

Proposition 3.3 gives a tractable memory-feedback calculation: delayed amplification can occur when the frozen matrix is asymptotically stable, with spectral radius below one. This condition does not require Euclidean-norm contraction. Section 3.5 now distinguishes this frozen result from the time-varying experimental dynamics.

The threshold relation in §3.4 characterizes a loss-threshold exceedance. It does not determine whether that exceedance is abrupt, narrow, or followed by rapid recovery. Figures 3–4 show signed paired probe-loss responses; identifying the mechanism of spontaneous training-loss spikes requires additional evidence about the actual training-loss trajectory and its temporal shape.

  1. What do Full ISO, ISO-FD, and the metrics measure?

Analytic Full ISO propagates the parameter–moment tangent and applies the probe-loss differential. ISO-FD means finite-difference estimation: in Pythia, we estimate the probe-logit derivative by centered differences and apply the cross-entropy differential at the control logits (Eqs. 77–78). This avoids directly subtracting nearly equal scalar losses. Both target the same local directional response, with different computational implementations.

NRMSE measures trajectory error divided by the norm of the exact trajectory; cosine measures trajectory alignment; Spearman correlation measures candidate ordering by peak magnitude. The tables report these complementary properties. The FD consistency test separately checks sensitivity to the difference step.

  1. Horizon choice and empirical evolution (Q6–Q8).

The reported windows are evaluation horizons, rather than estimates of the longest accurate horizon. The cross-family protocols do not isolate model-size effects from other protocol and computational differences. Pythia uses H = 8 at all three model sizes; Table 14 shows no systematic deterioration of local NRMSE with size in those conditions.

The new ViT experiment evaluates nested prefixes of the same 32-step continuation. From H = 8 to 32, median NRMSE rises from 0.163 to 0.564 on CIFAR-10 and from 0.108 to 0.125 on synthetic quadrants (Table 16). At H = 32, the three CIFAR-10 values are 19.868, 0.047, and 0.564. Figure 4 overlays the exact response and analytic tangent for the median-error seed; Figure 3 shows all six state-intervention trials. These results show substantial seed dependence. Theorem 3.4 explains why a correct local derivative can lose finite-shock accuracy: subsequent propagators carry and amplify the remainders generated along the trajectory.

  1. What does the linear approximation contribute to Experiment 2 (Q9)?

A branch is one sampled future minibatch continuation, shared across candidates. Its ranking orders candidates by their peak on that future; the mean ranking averages peaks over 32 futures. Equation (6) divides the variance across candidate means by its sum with the average within-candidate variance across futures.

These descriptive quantities come from exact paired runs and can indeed be computed without a tangent model. They establish whether candidate-specific effects persist across futures. The linear model enters in the present-frozen and reference-surrogate ISO scores, which use current information to estimate that mean ordering without observing the sampled futures. Table 4 illustrates the distinction: matched quadratic rankings are stable across futures (branch correlation 0.909), yet present-frozen ISO has correlation -0.066 with the mean ordering; matched MLP–GELU gives 0.941. Stable candidate differences and their recovery by a present-time model are therefore separate empirical questions.

Reviewer_hLaiOpenReview ↗

Thanks for the response

The paper is improved by the revision: the overall motivation, and the experiments, now become clearer. New figures are added into the paper to illustrate the behavior of both the exact probe-loss response and the analytic tangent proposed by the paper.

But I feel the more essential Critical Weaknesses that I raised, like both SGD and Adam are state-dependent so why only focusing on Adam, what is so special in Adam; and does the linear approximation actually bring any non-trivial insights into practice -- these are acknowledged but not straightforwardly answered. Therefore, I would like to keep my score, while still maintaining the assessment that the paper is slightly below "Weak acceptance". This is a decent paper which does what it wants to do; but it is still slightly below because I feel it does not establish enough impact regarding why these experiments matter.

Reviewer_D8bSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the delayed effect of minibatch perturbations in AdamW by comparing paired training trajectories that differ only at one update and share the same subsequent training sequence. In particular, the authors formulate AdamW as a finite-horizon input–state–output (ISO) system, with the model parameters and optimizer moments forming the joint state. Using this formulation, the authors derive the directional derivatives of the joint state and a probe loss that explicitly depend on the training dynamics of AdamW. They further derive an exact multistep error decomposition and establish fixed-horizon quadratic approximation error under local smoothness.

The authors examine the accuracy of these approximations, the effects of changing individual optimizer-state components, and whether differences between gradient perturbations persist across different future minibatch sequences through their experiments. Across quadratic losses, CIFAR-10 networks, and pretrained language (Pythia) models, the approximations along the realized training trajectory generally recover the relative ordering of perturbations, with high median Spearman correlations between predicted and observed maximum absolute loss effects.

Critical strengths
  1. The authors construct perturbations with equal immediate absolute loss changes and show that their later effects can nevertheless differ substantially. Separately changing the parameters and optimizer moments also reveals different response times, while varying moment persistence with the first future parameter displacement held fixed helps distinguish memory effects from differences in initial update magnitude. These comparisons support the motivation for studying influence over multiple updates rather than relying only on the immediate loss change.

  2. The authors repeat each perturbation under 32 independently sampled future minibatch sequences, separating differences in average influence between perturbations from variation caused by the future training data. They then test whether scores computed from current information recover the ordering of average influence. These scores perform well in the CIFAR-10 experiments but poorly for the matched quadratic perturbations. This is informative as it distinguishes stable differences in influence from approximations, providing evidence of both the usefulness and limitations of the approach.

Critical weaknesses

W1. [1] derives the propagation of a gradient perturbation through AdamW’s joint parameter and moment states (cf. Appendix B.3). Although the authors acknowledge this prior work, the distinction drawn between training-data attribution and tracking delayed minibatch effects does not adequately explain the mathematical novelty. Additionally, the error analysis, while useful and interesting, requires additional clarification and details. For example, the authors prove that a sufficient condition for which the aggregate switching defect is quadratic is an activation margin condition. However it is unclear in which settings and models this condition or the aggregate switching defect hold true. The authors should clarify what new theoretical insight these results provide beyond the existing joint-state sensitivity analysis and for which situation these results generalize to.

W2. There are some issues with clarity in the experimental section that I would like the authors to address. First, Experiment 1 measures differences in a fixed evaluation loss between perturbed and control runs, reporting both approximation errors and Spearman correlations for rankings based on the largest absolute future loss difference. These do not seem to directly establish the role of optimizer memory in delayed effects. The authors additionally construct perturbations with equal immediate loss-change magnitudes to examine differences in later effects, and change individual optimizer-state components to examine their influence. These are informative, but their purpose and connection to the theoretical claims should be explained more directly.

[1] Deng, Junwei, et al. "How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines." arXiv preprint arXiv:2605.18814 (2026).

Questions for authors

Q1. Could the authors provide some more clarification distinguishing between the theoretical results established in the manuscript and that of [1]? In particular, how does formulating AdamW as an ISO operator differ mathematically from the joint-state sensitivity calculation of [1]? If there is no difference, do the main theoretical contributions lie in the memory analysis and approximation-error guarantees?

Q2. The authors prove that a sufficient condition for which the aggregate switching defect is quadratic is an activation margin condition. However it is unclear in which settings and models this condition or the aggregate switching defect hold true. Could the authors clarify which models and training settings satisfy these assumptions, or satisfy the quadratic bound directly? Do they hold in the experiments, particularly for the ReLU network when activations switch?

Q3. Could the authors more directly explain what perturbations with equal immediate loss-change magnitudes and separate parameter/moment interventions establish about delayed optimizer-memory effects beyond approximation and ranking accuracy? How do these conclusions differ from Experiment 2’s findings on ranking stability across futures and prediction using current information?

[1] Deng, Junwei, et al. "How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines." arXiv preprint arXiv:2605.18814 (2026).

3 comments

Thank you for the questions about the theoretical contribution, activation switching, and the interpretation of the experiments. We revised the contribution statement and experimental explanations, and added ViT-20M controls in §4.1 and Appendix F.2.4.

  1. Theoretical contribution relative to Deng et al. (W1, Q1).

The joint AdamW derivative is shared machinery, as now stated in §2. Deng et al. also analyze approximation error and construct an error proxy. Our analysis builds on this foundation by examining how optimizer memory shapes the temporal response and how finite-horizon approximation defects propagate.

Proposition 3.3 separates the first- and second-moment channels and derives conditions for their response to grow over successive horizons even when the frozen joint dynamics are asymptotically stable. The response depends on the coupled feedback terms, beyond the individual moment decay rates. Theorem 3.4 gives an exact error identity in the complete parameter–moment state, separating nonlinear shock write-in, smooth propagation remainders, and activation-switching defects. It then bounds the output error under the stated probe regularity. This exact joint-state decomposition and its conditional guarantee differ from the approximations used to construct the error proxy in Deng et al., §4.3 and Appendix F. A central implication is that the propagators carrying the desired response also can amplify approximation defects, connecting transient response growth to the limits of tangent fidelity.

  1. Conditions for ReLU networks (W1, Q2).

Corollary 3.5 requires local regularity of the AdamW transitions and probe, together with a quadratic switching-defect bound. For smooth networks, the switching defect is zero. For a finite ReLU control trajectory strictly inside all relevant training and probe activation regions, sufficiently small perturbations preserve those regions, so switching defects also vanish locally.

When gates switch, Appendix E.4 requires both the per-switch linear defect bound in Eq. (27) and the weighted activation-margin condition in Eq. (29). These are additional assumptions; continuity of the ReLU forward map does not establish them because parameter gradients can jump. The CNN–ReLU experiments do not verify these assumptions: the fitted error exponent decreases from 2.017 at horizon 1 to 1.120 at horizon 12 over the tested scales, alongside activation-pattern differences. The revised §3.6 and Appendix E.4 clarify this scope. The state-error identity in Theorem 3.4 retains switching defects explicitly, while its output bound separately requires the stated Lipschitz-gradient condition on the probe.

  1. What the controls establish (W2, Q3).

Trajectory fidelity tests the approximation itself. Exact first-step matching tests whether immediate loss magnitude determines the later peak. State interventions test the effect of the initial optimizer-memory deviation at fixed post-shock parameters.

The new ViT-20M intervention compares full-state and parameter-only branches with identical post-shock parameters and shared future minibatches. The parameter-only branch restores both moments to their control values once; both branches subsequently run normal AdamW. Retaining the perturbed moments produces a larger 32-step absolute peak in all six trials, covering three seeds each on CIFAR-10 and synthetic quadrants. Figures 2 and 3 show the paired peaks and signed trajectories.

The natural-direction control preserves each gradient-difference direction and calibrates only its positive scalar amplitude to match the exact first-step absolute loss change. Across the six runs, the largest-to-smallest later peak ratio among matched candidates ranges from 4.73 to 26.05 (Table 17). Thus, immediate loss magnitude alone does not determine the future peak; the separate state intervention identifies the contribution of the initial moment deviations.

These controls fix one future per run. Experiment 2 resamples 32 futures to measure candidate-ranking stability and then tests whether present-only ISO scores recover the mean ordering. Its matched family uses the construction in Eq. (74); the new natural-direction experiment establishes fixed-future peak heterogeneity. This separates approximation fidelity, optimizer-memory effects, and prospective recoverability.

I would like to thank the authors for their detailed rebuttal and revisions. Here is what the rebuttal has resolved for me:

  • I now have a better understanding of the authors' intended theoretical contributions beyond [1], particularly the characterization of optimizer-memory responses and the finite-horizon approximation-error analysis. I also appreciate the clarification that the quadratic switching-defect condition is not generally guaranteed for ReLU networks.

  • I believe that the revisions have improved the experimental presentation. In particular, the new ViT-20M experiments provide stronger evidence that the optimizer's moment states can significantly influence future loss responses even when the immediate parameter states are identical. I also better understand the distinct purposes of Experiments 1 and 2.

There are a few unresolved concerns that I have even with the revisions that the authors provide.

  • While I understand the authors' intended contributions, I still find the theoretical results beyond [1] to be relatively incremental. In particular, Proposition 3.3 considers a linear system and derives conditions for transient amplification by examining the first few powers of the transition matrix. However, this analysis looks to be the impulse-response coefficients of a discrete-time linear system [2] (Eq. 5), and the observation that asymptotically stable systems can exhibit transient amplification is well established in the literature on dynamical systems [3]. Similarly, Theorem 3.4 appears to be a direct application of [4] (Theorem 3.17), where the multistep decomposition follows directly from recursively propagating the local approximation errors. While I agree that applying these techniques to AdamW provides useful interpretations, it is unclear to me what substantially new theoretical insights these results establish beyond the existing literature as many of these results rely on the initial development of [1].

  • I appreciate the authors' clarification regarding Corollary 3.5, but it remains unclear in which practically relevant settings its assumptions would hold. For smooth networks, or ReLU networks where perturbations preserve the activation regions, the quadratic approximation error seems to follow from a standard smooth Taylor expansion. However, when activation patterns change, the required quadratic switching-defect bound is not generally guaranteed and is not verified in the experiments. Furthermore, Corollary 3.5 only establishes a local, fixed-horizon guarantee with constants that may depend on the horizon and control trajectory. Consequently, it is difficult to understand how informative this result is for finite perturbations over longer horizons.

Given these concerns, particularly the first point, I am currently keeping my score unchanged. I do agree that the revisions have improved the clarity of the paper and that the experiments provide useful insights into the delayed effects of optimizer memory. However, I remain unconvinced that the theoretical results represent a sufficiently substantial advance beyond existing literature and the incremental theoretical results over [1].

[1] Deng, Junwei, et al. "How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines." arXiv preprint arXiv:2605.18814 (2026).

[2] De Schutter, B. (2000). Minimal state-space realization in linear system theory: An overview. Journal of Computational and Applied Mathematics, 121(1–2), 331–354.

[3] Schmid, P. J. (2007). Nonmodal stability theory. Annual Review of Fluid Mechanics, 39, 129–162.

[4] Elaydi, S. (2005). An introduction to difference equations (3rd ed.). Springer.

Thank you for sharpening the comparison. Deng et al. derive the joint-state AdamW sensitivity and develop an attribution-error proxy. We now cite these contributions directly in §3.4 and §3.6. The impulse-response calculation in Proposition 3.3 uses standard tools and gives a specific result for AdamW: in the frozen model, even when the perturbation starts in only one moment, both moment-feedback terms enter the fourth-step parameter response. The decay rate of the initially perturbed moment alone therefore cannot determine whether the response grows. The ViT-20M intervention tests the role of the moment state under time-varying updates: retaining the perturbed moments increased the later peak in all six trials at identical post-shock parameters. Corollary 3.5 gives a local, fixed-horizon guarantee, and Table 18 separately measures how finite-perturbation accuracy changes with scale.

Reviewer_toSUOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a gradient perturbation introduced at a single AdamW update affects subsequent losses. It compares two training runs. The runs start from the same state and use the same future minibatches and randomness. They differ only in the gradient used at the perturbed step. The paper includes the model parameters and both optimizer moments in a joint state. It derives a linear response model with three parts: pertubation injection, state propagation, and loss readout. It also derives a multistep error decomposition and a conditional accuracy results. A simplified model explains how optimizer memory can delay or amplify a response. The experiments use quadratic systems, small neural networks, and pretrained Pythia models. Further experiments resample future minibatches. They test whether each candidate has a consistent effect across different futures. They also test whether scores based on current information can estimate this effect.

Critical strengths
  • Sharing the future realization between control and perturbed trajectories isolates the pathwise effect of the initiating intervention. The paper explicitly acknowledges that the the exact pathwise operator is not itself an online predictor. It also correctly distinguishes a larger absolute loss response from an adverse loss increase or an actual threshold-crossing spike.
  • The derivation accounts for first and second-momentum perturbations, bias corrections, and decoupled weight decay. It also includes the effect of parameter changes on later gradients. The joint Jacobian and error recursion appear consistent with these updates.
  • The small-scale experiments include useful controls. The authors vary the perturbation size, match the immediate loss response, intervene on optimizer-state components, and vary the momentum coefficients.
Critical weaknesses
  • The paper need a clearer comparison with the closest prior work. Deng et al (2026) derive an AdamW sensitivity model over the parameters and both optimizer models. They already give the joint-state derivative before making those approximations. The authors should state what is new after the above connection is made.
  • The language-model results do not show a clear advantage over the one-step baseline. The Pythia experiments use centered finite differences of probe logits. They estimate the derivative of the complete training trajectory. They do not propagate the derived parameter-moment tangent analytically. This is a reasonable way to estimate a local response. However, it does not independently check the explicit block decomposition at this scale. It also does not show a computational advantage over rerunning several perturbed trajectories.
  • The ReLU experiments do not directly test the switching assumptions. Corollary 3.5 assumes a quadratic switching defect. Appendix E.4 also assumes that each gate's contribution to the switching defect is at most linear in the state displacement. These conditions do not follow from continuity of the ReLU forward map. Parameter gradients can jump at activation boundaries. AdamW updates depend on these gradients. For example. consider l(\theta)=1/2(ReLU(\theta)1)^2, Its gradient jumps at zero. It shows that the switching assumptions exclude some standard boundary crossings. [1] Deng et al "How faithful is trajectory-based data attribution? Error sources, Remedies, and practical guidelines." (arxiv: 2605.18814)
Questions for authors
  1. How does the joint AdamW derivative differ from Deng et al's formulation?
  2. Can you compare the analytic tangent with ISO-FD on a Transformer?
  3. Do the results hold when matching preserves natural gradient directions?
1 comment

Thank you for the questions about the relation to Deng et al., Transformer validation, and natural-direction matching. The revision adds ViT-20M analytic-tangent checks and controls in §4.1 and Appendix F.2.4.

  1. Relation to Deng et al. (Q1).

As clarified in §2, our joint-state derivative shares the sensitivity machinery of Deng et al. Their work also studies approximation error. Our analysis develops the memory-channel response timing and transient amplification conditions in Proposition 3.3, and an exact finite-horizon error decomposition in Theorem 3.4. The latter operates on the complete parameter–moment state and separates shock write-in, smooth propagation remainders, and activation-switching defects, followed by a conditional output-error bound. These are the specific additions beyond the shared derivative and the approximate error proxy developed in Deng et al., §4.3 and Appendix F.

  1. Analytic tangent and finite differences on a Transformer (Q2).

For ViT-20M, we analytically propagate the joint-state tangent using training-loss Hessian–vector products and read it out with the probe gradient. We compare it with centered differences of complete paired trajectories, evaluated directly on the probe loss. On CIFAR-10 seed 2027, we additionally checked the three calibrated directions with the largest absolute analytic response at horizon 32. As the multiplier step decreased from 1e-2 to 1e-6, the finite differences converged to the signed analytic values; relative errors were below 5.4e-6 at the smallest step. This verifies the directional response along the tested directions at that horizon.

Pythia ISO-FD instead differences probe logits and applies the cross-entropy differential at the control logits, avoiding direct subtraction of nearly equal scalar losses. By the chain rule, both procedures target the same local loss derivative, although their finite-step numerical estimators differ. The Pythia study measures end-to-end local-response fidelity; it does not separately validate the internal Jacobian blocks or establish computational savings. Table 15 reports the one-step comparison directly: both methods have median Spearman correlation 0.762 across nine conditions, with ISO-FD higher in five and exact one-step higher in four.

  1. Matching along natural gradient directions (Q3).

Each candidate retains its original gradient difference, and only a positive scalar multiplier is calibrated to match the exact first-step absolute probe-loss change. Calibration uses only the first-step response, with relative tolerance 5e-8. Reference, probe, and future minibatches are shared within each run. Matching succeeds for 48/48 CIFAR-10 and 44/48 synthetic-quadrant candidates; Table 17 reports all attempted candidates and computes response statistics over successful matches.

Across six runs, the largest-to-smallest exact peak ratio over horizons 2–32 ranges from 4.73 to 26.05, and 89/92 matched candidates attain a larger absolute response after step 1. Thus, later peak differences persist when matching preserves natural directions. This new control concerns fixed-future responses; Experiment 2 retains the matched construction specified in Eq. (74).

Tangent and exact peak rankings agree closely in five runs, with correlations 0.982–1.000. CIFAR-10 seed 2027 has correlation 0.176 and median peak error 1900% at the original matching scale. Table 18 separates local derivative accuracy from finite-shock fidelity: rematching to a first-step target reduced to 1/16 of the original target gives correlation 0.791 but still 511.58% median peak error; uniformly rescaling the original matched shocks by 2^-16 gives correlation 1.000 and 0.51% error, with first-step magnitudes then free to vary. The differential checks support the local derivative, while the scale sweep exposes a narrow accurate regime for this seed. The exact matched-trajectory evidence for later peak heterogeneity remains valid independently of tangent fidelity.

  1. ReLU switching assumptions.

We clarify the conditional guarantee in §3.6 and Appendix E.4. The sufficient condition requires both a per-switch linear defect bound, Eq. (27), and a weighted activation-margin bound, Eq. (29). ReLU forward continuity alone does not imply these conditions. If sufficiently small perturbations preserve the relevant activation regions, switching defects vanish. For the CNN–ReLU runs with switching, the fitted error exponent falls from 2.017 at horizon 1 to 1.120 at horizon 12; these experiments do not establish a quadratic switching bound. Theorem 3.4 retains the switching defect in its exact state-error identity, while the output-error bound additionally requires the stated probe-gradient regularity.

Comments on the paper

1 comment

We thank all the valuable comments from three reviewers. This paper asks what a single minibatch perturbation does after the update where it is introduced. It makes two contributions. First, we explain how a perturbation propagates along a realized AdamW trajectory. We trace its signed loss response across updates, characterize how the moment states can delay and amplify it, and establish when the tangent approximation is accurate. Second, we ask how much of that delayed effect can be anticipated before the future minibatches are known. We use repeated futures to measure whether candidates retain different typical effects, then test whether scores computed from present information recover their ordering. The two contributions follow the same question from mechanism to prediction: how does a disturbance affect later loss on a given trajectory, and what can we learn about that effect before the trajectory unfolds?

Optimizer memory at fixed parameters (D8bS). The reviewer asked whether the delayed response could be attributed to the initial parameter displacement. In the new ViT-20M intervention, both branches start with identical post-shock parameters and process the same future minibatches. The full-state branch retains the perturbed first and second moments, and the parameter-only branch restores them to their control values. Retaining the perturbed moments increased the 32-step peak absolute probe-loss response in all six trials across CIFAR-10 and synthetic quadrants. The full-state peaks were 1.08–2.63× the corresponding parameter-only peaks (Figures 2–3). SGD can propagate a disturbance through the parameters. This intervention isolates the additional contribution of AdamW’s moment states.

Matching along natural directions (toSU). The reviewer asked whether differences in later peaks depended on how the matched perturbations were constructed. We preserved each candidate’s original gradient-difference direction and adjusted only its scale to match the exact first-step absolute loss change. Across the six runs, the ratio of the largest to the smallest later peak among matched candidates ranged from 4.73× to 26.05×, and 89 of 92 matched candidates reached a larger absolute response after step 1 (Table 17). These exact paired trajectories show that matching the immediate loss magnitude leaves substantial differences in later responses. Table 18 separately examines how accurately the tangent predicts those peaks at different perturbation scales. Both new controls are in §4.1 and Appendix F.2.4.

What the theory adds (D8bS and toSU). Deng et al. derive joint-state sensitivity for AdamW, including decoupled weight decay, and study approximation error. Building on that derivative, Proposition 3.3 characterizes when perturbations stored in either moment state first affect the parameter response and how that response can grow even when the frozen dynamics are asymptotically stable. Theorem 3.4 gives an exact joint-state error decomposition that tracks how errors from shock injection, smooth propagation, and activation switching enter and propagate across updates. We clarified the distinction from prior work in §2 and the conditions for first-order accuracy in §3.6 and Appendix E.4.

How the response evolves and what persists across futures (hLai). Figure 4 overlays the exact signed response and analytic tangent for CIFAR-10 seed 2029, and Table 16 quantifies how agreement changes across horizons and seeds. We also clarified in §4.2 that exact paired trajectories measure the persistence of candidate rankings across sampled futures, while present-only ISO scores test whether their mean ordering can be recovered in advance. In the matched standard quadratic setting, the median Spearman correlation between a single-future ranking and the mean ranking is 0.909. In the matched CIFAR-10 MLP–GELU setting, present-frozen ISO reaches 0.941 against the mean ranking (Table 4).

The revision thus provides direct evidence for both parts of the paper’s claim. On a realized path, AdamW’s moment states can amplify delayed responses even when the initial parameter displacement is fixed, and equal immediate loss magnitudes can conceal large differences in later peaks. Across sampled futures, those delayed differences can retain a consistent ordering, which scores computed from present information can recover in some settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.