acceptodds
Under review as a conference paper at ICLR 2027

What Ages a Gradient? When Optimizer Time and Gradient-Field Relevance Diverge

Abstract

AdamW assigns age by counting optimizer updates, but equal update counts can correspond to very different amounts of learning-state change. We ask whether elapsed optimizer time is sufficient to predict how well a historical gradient direction remains aligned with the current gradient field. Fixed held-out probes keep text identity constant, isolating gradient-field evolution from minibatch variation. On Pythia-160M, elapsed steps and accumulated learning rate leave a reproducible predictive gap: OBJECTIVE, a loss-normalized gradient–update work diagnostic, reduces held-out prediction error by 9.87% and 7.48% in two environments and transfers between them without refitting. The gap survives a new seed and both shorter and longer first-moment memory; under the longer-memory setting, a scale-normalized parameter-motion diagnostic reaches +16.00%. A post-discovery decomposition shows that learning state captures substantially more of the residual: a four-variable state summary raises held-out from 0.588/0.621 to 0.741/0.731. Parameter motion and training progress explain part of this signal, while temporal misalignment erodes the remainder. For actual stochastic gradients, relevance also reflects alignment at gradient birth: unconditional state transfer is weak, while a prespecified birth-conditioned comparison is positive. Together, these results show that fixed-timescale optimizer memory compresses training histories that can have different present-day gradient relevance because they traverse different learning states.

Primary area:
optimization
Keywords:
optimization, AdamW, gradient dynamics, optimizer memory, gradient alignment, learning dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UpMXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the relevance of historical gradients in AdamW can be measured by elapsed optimizer steps alone. The authors argue that even when two gradients have the same age in terms of update steps, the model may have undergone different amounts of change, leading to different levels of alignment with the current gradient. To study this, they use fixed held-out text samples to measure gradient alignment across training checkpoints, avoiding variation caused by different minibatches. They also introduce two diagnostic measures, OBJECTIVE and PATH, which capture gradient–update interactions and relative parameter changes, respectively.

Experiments are mainly conducted on Pythia-160M, with additional tests on OPT-125M. The results show that OBJECTIVE reduces prediction error by 9.87% and 7.48% in two training environments and shows some transferability across environments. Further analysis finds that combining information about gradient norms, update magnitudes, gradient–update alignment, and training loss improves the prediction of historical gradient relevance. The authors also examine different training settings, temporal misalignment, and actual stochastic gradients to test the robustness and limits of these findings.

Overall, the paper shows that elapsed optimizer steps alone do not fully capture the relevance of historical gradients. Changes in the model's learning state provide additional information, which may help better understand optimizer memory and inform future state-aware optimization methods.

Critical strengths

S1. The paper addresses a meaningful and interesting research question. It examines whether the relevance of historical gradients can be measured by elapsed optimizer steps alone and how changes in the learning state affect gradient relevance. This helps improve our understanding of optimizer memory and may inform the design of more flexible gradient memory strategies.

S2. The experimental design is relatively rigorous. The authors use fixed held-out samples to control for variation across minibatches and test their findings under different random seeds, learning rate schedules, and momentum settings. Cross-environment and cross-architecture experiments also provide some evidence of the robustness of the results.

S3. The paper provides a detailed analysis of the experimental findings. Beyond comparing different measures for predicting historical gradient relevance, the authors examine the roles of gradient norms, parameter updates, and training loss. In particular, Section 5 considers alternative simple explanations, such as the capacity of the time-based baseline, parameter motion, and training progress. The temporal misalignment experiments further help assess the role of learning-state information and the limits of the findings.

Critical weaknesses

W1. Limited experimental scale and generalizability. The experiments mainly focus on Pythia-160M, with additional cross-architecture tests on OPT-125M. However, both models are relatively small, and training lasts only about 1,500 optimizer steps. It remains unclear whether the findings generalize to larger models, longer training runs, or different architectures and tasks.

W2. Limited transfer from fixed probes to actual training gradients. Using fixed held-out samples helps isolate the effects of learning-state changes from minibatch variation, but it also differs from real training conditions. Section 6 shows that the proposed measures have limited predictive value for actual stochastic gradients without conditioning on their alignment at birth. This raises questions about how well the main findings apply to practical optimizer settings.

W3. Unclear practical value of the proposed diagnostic measures. The paper shows that OBJECTIVE, PATH, and learning-state features can better predict historical gradient relevance, but it does not demonstrate whether they improve training efficiency or final model performance. Moreover, gradient alignment does not necessarily reflect how useful a historical gradient is for optimization, since momentum also helps reduce noise and stabilize updates. The additional computation and storage costs of these measures are also unclear, making their practical benefits difficult to assess.

Questions for authors

Q1. Could the authors provide some additional evidence, even on a small scale, to help assess whether the findings generalize to larger models or longer training runs? (see W1)

Q2. How can the proposed measures be made more effective for actual stochastic gradients without relying on birth-alignment information? (see W2)

Q3. To what extent can the proposed measures improve practical optimization performance, and what are their additional computational and storage costs? (see W3)

8 comments
AuthorsOpenReview ↗

Response to Reviewer UpMX

Thank you for your positive assessment of our research question, controlled experimental design, and the confounding-factor and temporal-misalignment analyses in Section 5. We address each of your weaknesses and questions below.

W1 & Q1. Experimental scale and generalizability to larger models and longer runs

We agree that extending the evaluation beyond Pythia-160M / OPT-125M and the 1,5001{,}500-update horizon is important to assess generalizability. We are currently running two controlled validation experiments that preserve the original Rich Step-LR baseline specification, probe/split assignments, diagnostic definitions, and six-part consistency criteria:

  1. Longer horizon on Pythia-160M: Four matched schedules of 6,000 updates each, to test whether the predictive gap persists into later training stages.

  2. Larger model with extended horizon: Pythia-410M with the same four schedules and 6,000-update horizon.

The learning-rate schedule is extended to 6,000 updates, measurement-window starts are scaled by four, and the reference preconditioner is taken at step 3,000. The 64-checkpoint windows and historical lags remain unchanged. Objective and Path are separate prespecified candidates, and the State block is a supporting secondary. This tests later training stages and model size within Pythia.

W2 & Q2. Transfer from fixed probes to actual training gradients without birth-alignment information

In Section 6, the unconditional actual-gradient comparison yields −0.89-0.89\\%, whereas conditioning on birth alignment (Bk,p,ebirthPB_{k,p,e}^{\mathrm{birthP}}, Eq. 20) yields +4.83+4.83\\% (4/44/4 positive test positions, 6/76/7 positive groups), showing positive state predictivity conditional on birth alignment. Whether minibatch sampling noise explains the weak unconditional result is the hypothesis tested by the follow-up.

To test whether reducing birth variance makes the learning-state features directly effective without relying on held-out birth-alignment information (birthP), we are currently conducting a batch-averaging ablation on actual training gradients. Specifically, under the exact same unconditional Rich STEP-LR baseline and evaluation protocol, we compare gradient estimates formed at the same historical parameter checkpoint, averaging raw gradients before one global clipping operation and leaving the original training trajectory unchanged:

  • A single minibatch gradient (1×1\times).
  • An average of 4 minibatches (4×4\times).
  • An average of 16 minibatches (16×16\times).

This directly tests the hypothesis that suppressing stochastic minibatch noise at birth allows the State block signal to emerge unconditionally. Methodologically, we also note that batch averaging modifies the measured historical vector itself, so we treat this as a controlled diagnostic of whether initial gradient stability affects state predictability rather than claiming an immediate modification to standard training utility.

W3 & Q3. Practical optimization performance, computational costs, and storage overhead

We address the computational/storage costs of the diagnostics and their relationship to practical optimization separately:

  1. Direct profiling of computation and storage overhead (currently underway): OBJECTIVE\text{OBJECTIVE}, PATH\text{PATH}, and the State summaries use gradients, parameters, optimizer moments, learning rates, and losses available during training. Computing the norms, dot products, and Path parameter-group norms requires additional reductions. To provide a concrete quantitative assessment of their overhead, we are currently profiling:

(i) the exact per-update wall-clock time required to compute OBJECTIVE\text{OBJECTIVE}, PATH\text{PATH}, and the State block; and

(ii) the additional memory and storage footprint compared against vanilla AdamW training.

We distinguish scalar-statistics overhead from full-observer, probe-measurement, historical-gradient-storage, and checkpoint costs. The results will quantify the additional cost of each mode.

  1. Scope regarding optimization performance: As noted in Section 8 and Appendix E, our study is designed as an upstream measurement investigation of gradient aging rather than an optimizer intervention paper. We fully agree that momentum in AdamW serves both to accumulate consistent directions and to smooth stochastic noise. Our findings establish that chronological step count is an incomplete coordinate for directional relevance; testing whether state-modulated forgetting rules translate this diagnostic signal into faster training convergence requires a dedicated intervention study and is a natural next step.
Reviewer_UpMXOpenReview ↗

Thanks for your response.

Thank you for your detailed response and for your efforts to address my concerns.

I would be happy to review any additional experimental results or analyses if they become available during the rebuttal period. I will also take into account the existing evidence and the other reviewers' perspectives, particularly regarding actual-gradient relevance and cross-architecture generalizability, when finalizing my assessment.

AuthorsOpenReview ↗

Follow-up: revised PDF and new results

Thank you for your thoughtful and constructive review. Your three questions defined most of the new experiments, and we believe they have made the paper substantially stronger. The revised PDF is now uploaded, with new and rewritten text in blue.

W1 / Q1. Larger model and longer training

We trained Pythia-160M and Pythia-410M on four matched 6,000-update schedules each, four times the original horizon. The Rich Step–LR baseline, probes, splits, lags and six consistency criteria are unchanged, and the window start steps are scaled by four.

ModelPath (prespecified candidate)State block (prespecified supporting comparison)
Pythia-160M+5.15%, 6/6 criteria+41.66% test, +46.81% confirmation, 6/6
Pythia-410M+4.06%, 6/6 criteria+63.46% test, +63.75% confirmation, 6/6

For the State block, all four test positions are positive at both sizes. The predictive gap left by elapsed steps and accumulated learning rate therefore persists into later training and at a larger model size (Sec. 5.2, Fig. 6, Table 3).

W2 / Q2. Actual gradients without birth alignment

At the same historical checkpoints, we estimated each historical gradient from M=1,4,16M = 1, 4, 16 training minibatches, evaluated at the same parameter point and globally clipped once after averaging, with the training trajectory unchanged. Without any birth-alignment information, the State block gives test gains of −0.89%, +2.26% and +4.39%. For M=4M=4 and M=16M=16, the 90% stability intervals are [1.35%, 3.17%] and [0.06%, 8.72%], the confirmation gains are +3.70% and +8.18%, and both pass all six criteria. To answer Q2 directly: the measures become effective for stochastic gradients once the historical estimate averages a few minibatches. For reference, with β1=0.9\beta_1 = 0.9 the first moment itself averages roughly ten recent minibatch gradients (Sec. 6, Fig. 7).

W3 / Q3. Computational and storage cost

On Pythia-160M (RTX 3090, 45 synchronized timed runs), computing Objective, Path or all four State components adds 3.13%, 3.38% or 3.51% to the median time per update, with unchanged peak memory and 69–231 bytes of log per update; our full research observer adds 12.39% (Sec. 8, Table 4). Probe gradients, stored historical gradients and checkpoints are needed only to measure relevance in this study, not to compute the diagnostics. We agree that testing a state-dependent memory rule on convergence and final loss is the natural next step, and these results show that its inputs would be cheap to compute.

All results above appear in the revised PDF (Sec. 5.2, Sec. 6, Sec. 8 and App. G). Your three questions, on scale and training length, on actual gradients without birth alignment, and on practical cost, each led to a dedicated experiment, and we think the paper is considerably stronger for them. We would value your view on whether these results answer the questions as you intended, and whether you see any further analysis that would make the evidence more convincing. If you feel the revision has addressed the weaknesses noted in your review, we would be grateful if this could be reflected in your final assessment. Thank you again for your careful, constructive and encouraging review.

Reviewer_Jg1eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate an interesting research question on whether optimiser memory with a fixed timescale is sufficient to characterise the correlation between historical gradients and the current gradient field. AdamW exponentially decays historical gradients according to the number of update steps, but the same gap in update steps may correspond to different degrees of change in the learning state. To investigate this issue, the authors measure gradient-direction consistency between different training checkpoints using fixed text probes and construct baselines based on the number of update steps and the cumulative learning rate.

Critical strengths

S1: The observation that identical gaps in update steps may correspond to different degrees of change in the gradient field offers a promising perspective for investigating the timescale of optimiser memory. S2: The paper examines different training environments, momentum decay coefficients, parameter-motion diagnostics, and an alternative model architecture. It also reports negative results from experiments involving actual gradients and identifies several limitations on generalisation.

Critical weaknesses

W1: The actual-gradient experiments in the paper show that unconditionally incorporating learning-state features does not yield positive predictive gains. Therefore, the current results are insufficient to support stronger claims about gradient effective age or the underlying optimizer memory mechanism. W2: The substantial predictive gains from learning-state summaries in the Pythia experiments arise from an exploratory decomposition, whereas the prespecified main experiment on learning-state features using OPT-125M did not satisfy all consistency criteria.

Questions for authors

Q1: What is the precise relationship between historical–current gradient directional consistency measured on fixed probes and the effectiveness of historical gradients in AdamW’s first-moment estimate during actual training? Q2: Given that the unconditional actual-gradient experiments did not yield positive gains, what conclusions about optimiser memory do the authors believe are supported by the results presented in this paper? Q3: Could the authors clarify the rationale for the Adam geometry metric adopted in the paper and its relationship to the preconditioning operator used in actual AdamW updates? Would the choice of a different metric affect the main conclusions? Q4: Learning-state summaries yield substantial predictive gains on Pythia, but the prespecified main experiment on OPT-125M did not satisfy all consistency criteria. Could the authors provide further independent validation to establish whether this learning-state signal can be robustly reproduced across architectures?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Jg1e:W1-Q1

Thank you for recognizing the value of our research question, the controlled fixed-probe design, and our thorough reporting of both positive and negative results across settings. We address your concerns and questions below.

W1. Unconditional actual-gradient results

We agree that the unconditional actual-gradient result is weak (−0.89-0.89\\%), whereas the prespecified birth-conditioned comparison yields +4.83+4.83\\% (4/44/4 positive test positions, 6/76/7 positive evidence groups, and 2/52/5 positive survival levels). An actual stochastic gradient g~k\tilde{g}_k varies with the minibatch at birth (step kk). The contrast shows additional state predictivity conditional on birth alignment; whether sampling noise explains the weak unconditional result is the hypothesis tested below.

To directly test whether minibatch sampling noise at birth is the primary driver of the weak unconditional transfer, we are currently running a controlled batch-averaging ablation. Specifically, at the same historical parameter checkpoint, we compare a single-minibatch gradient with averages of 44 and 1616 raw minibatch gradients, applying global clipping once after averaging and leaving the training trajectory unchanged, under the same unconditional evaluation protocol to measure whether stabilizing the historical gradient estimate makes the State block signal easier to detect without conditioning on birthP.

W2. Post-discovery decomposition and OPT consistency

Our manuscript explicitly separates the post-discovery Pythia state decomposition from the prespecified OPT-125M architecture test. In Appendix E.1 (Table 10), the prespecified OPT State primary yields a large positive mean gain (+14.80+14.80\\% on test and +13.69+13.69\\% on confirmation, with 4/44/4 confirmation positions and 5/75/7 test evidence groups positive), but meets 4/64/6 criteria: the 9090\\% finite-design SI is [−3.82,33.41][-3.82, 33.41]\\%, and only 2/52/5 survival levels are positive. Meanwhile, the prespecified PATH\text{PATH} secondary achieves +4.81+4.81\\% (9090\\% SI [+0.15,+9.48][+0.15, +9.48]\\%) and satisfies all 6/66/6 consistency criteria. Thus, the cross-architecture evidence in the current paper is carried by PATH\text{PATH}, rather than claiming that the four-variable Pythia state parameterization transfers unchanged to OPT. We will make this boundary even clearer in the revised discussion.

Q1. Relationship between fixed-probe consistency and actual AdamW momentum

Fixed-probe consistency measures the directional alignment between historical and current gradients evaluated on the same held-out text (gp,kg_{p,k} vs. gp,tg_{p,t}), whereas AdamW's first moment

mt+1=(1−β1)∑k=0tβ1t−kg~km_{t+1} = (1-\beta_1)\sum_{k=0}^{t}\beta_1^{t-k}\tilde{g}_k

accumulates clipped stochastic gradients g~k\tilde{g}_k from different training minibatches, with m0=0m_0=0. Fixed probes track the gradient field for the same held-out text along the realized trajectory. The difference between a clipped training gradient and that fixed-probe gradient includes both minibatch differences and clipping; the fixed probe is not the population gradient. The bridge therefore measures cross-time directional alignment of an actual historical gradient, rather than the optimization benefit of the full accumulated first moment. The prospective bridge experiment (Section 6, Eqs. 18 and 20) directly connects these two quantities by showing that once initial birth alignment (Bk,p,ebirthPB_{k,p,e}^{\mathrm{birthP}} between g~k(e)\tilde{g}_k^{(e)} and gp,k(e)g_{p,k}^{(e)} in Pk(e)P_k^{(e)} geometry) is conditioned on, adding the learning-state block reduces held-out alignment-prediction error by 4.834.83\\% relative to Rich Step-LR + birthP. The descriptive middle birth-alignment stratum has a 10.8610.86\\% gain using the same fitted models (Table 9).

AuthorsOpenReview ↗

Response to Reviewer Jg1e:Q2-Q4

Q2. Conclusions supported about optimizer memory

The results support three concrete conclusions regarding optimizer memory under the tested AdamW settings:

  1. Insufficiency of chronological step count: Elapsed optimizer steps and accumulated learning rate (Rich STEP-LR) do not fully capture fixed-probe historical-gradient relevance; the predictive gap recurs across trajectories and memory timescales (β1∈0.85,0.90,0.95\beta_1 \in \\{0.85, 0.90, 0.95\\}), with the most stable scalar varying by setting and PATH\text{PATH} carrying the OPT-125M cross-architecture result.

  2. State dependence of field evolution: On Pythia, the four-variable State block, including gradient-update alignment Scos⁡S_{\cos} and inverse loss S1/LS_{1/L}, raises held-out R2R^2 from 0.588/0.6210.588/0.621 to 0.741/0.7310.741/0.731. With Path and absolute endpoint progress additionally included in the baseline, the residual gain is 12.9412.94\\%/4.12\\% and becomes −0.34-0.34\\%/-4.16\\% under half-phase temporal misalignment (ρ=0.50\rho = 0.50).

  3. Joint dependence on birth alignment and state traversal for stochastic gradients: For actual minibatch gradients, present relevance reflects both how the gradient aligns with the field at birth and the learning state traversed afterward. We will emphasize these exact boundaries in the revised manuscript.

Q3. Adam geometry and preconditioning

In AdamW, historical gradients accumulated in mtm_t are preconditioned by the inverse square root of the bias-corrected second moment,

Pt=diag((v^t+ϵadam)−1),P_t = \mathrm{diag}\left((\sqrt{\hat{v}_t} + \epsilon_{\mathrm{adam}})^{-1}\right),

to produce the loss-driven parameter update, together with the learning rate and first-moment bias correction. In Eq. (4), gp,t⊤Ptgp,kg_{p,t}^{\top}P_tg_{p,k} is the local first-order interaction with a historical direction preconditioned by PtP_t. The PtP_t-weighted normalized inner product corresponds to the geometry induced by Pt1/2P_t^{1/2}; it is not the Euclidean cosine between Ptgp,tP_tg_{p,t} and Ptgp,kP_tg_{p,k}, which would involve Pt2P_t^2.

The positive directional State signal in Original/Second persists across the three tested geometries. Figure 4(a) explicitly compares current-P (PtP_t) against Raw Euclidean geometry (II) and a Fixed Reference preconditioner (PrefP_{\mathrm{ref}}, frozen at step 750750), showing that directional State block gains remain strongly positive across all three geometries. Furthermore, we report here an export of the existing stored OBJECTIVE\text{OBJECTIVE} results using the original richer-baseline aggregation. Across PtP_t, II, and PrefP_{\mathrm{ref}}, the gains are 9.879.87\\%, 8.778.77\\%, and 8.038.03\\% in Original, and 7.487.48\\%, 5.305.30\\%, and 6.256.25\\% in Second. All six Original/Second Objective comparisons meet the original six-part criteria. We will include this consolidated table in the revised text so these comparisons are immediately accessible.

Q4. Independent validation across architectures

For additional validation, we are currently running four matched training schedules for each of Pythia-160M and Pythia-410M, with 6,000 updates per run. These extensions preserve the diagnostic definitions, Rich Step-LR baseline specification, probe/split assignments, and six-part consistency criteria. The learning-rate schedule is extended to 6,000 updates; measurement-window starts are scaled by four, and the reference preconditioner is taken at step 3,000. The historical lags and 64-checkpoint window length remain unchanged.

  1. Longer horizon on Pythia-160M: Testing the diagnostics at later training stages within the original architecture.
  2. Pythia-410M with the same extended horizon: Testing a larger model under the matched extended design.

Objective and Path are separate prespecified candidates; the State block is a supporting secondary. These experiments test scale and later training within the Pythia family. The cross-architecture State result remains the OPT-125M comparison above, which meets 4/6 criteria.

AuthorsOpenReview ↗

Follow-up: revised PDF and new results

Thank you for your positive assessment and for the clear questions, which directly guided the new experiments. The revised PDF is now uploaded, with new and rewritten text in blue. Below we report the results promised in our first reply.

W1 / Q2. Actual-gradient results

At the same historical checkpoints, we estimated each historical gradient from M=1,4,16M = 1, 4, 16 training minibatches. All constituent gradients are evaluated at the same parameter point, and the averaged raw gradient is globally clipped once; the training trajectory is unchanged.

Without birth conditioning, the State block gives test gains of −0.89% (M=1M=1), +2.26% (M=4M=4) and +4.39% (M=16M=16). For M=4M=4 and M=16M=16, the 90% stability intervals are [1.35%, 3.17%] and [0.06%, 8.72%], the confirmation gains are +3.70% and +8.18%, and both pass all six consistency criteria. With birth alignment in the baseline, the gains are +5.65% and +6.14%. For your Q2, this means that learning-state features predict the alignment of stochastic gradient estimates unconditionally once single-minibatch noise is reduced, and not only after conditioning on birth alignment.

W2 / Q4. The State result at larger scale and longer training

We extended the fixed-probe study to 6,000 updates at Pythia-160M and Pythia-410M, with the baseline, probes, splits and criteria unchanged. In these runs the four-variable State block was specified before the outcomes. It reaches +41.66% at 160M and +63.46% at 410M (confirmation +46.81% and +63.75%) and passes all six criteria at both sizes. This addresses your concern that the Pythia state result came only from an exploratory decomposition. Path also passes all six criteria at both sizes (+5.15% and +4.06%). For cross-architecture evidence, the OPT-125M Path comparison continues to pass all six criteria.

Q3. Adam geometry

Table 1 now repeats the Objective and Path analyses in Euclidean and frozen-reference geometry. All 18 mean gains remain positive, so the conclusions do not depend on using the current Adam preconditioner.

Q1. Relation to the first moment

Sec. 6 now states the link explicitly: a historical gradient g~k\tilde g_k contributes wk,t gp,t⊤Ptg~kw_{k,t}\, g_{p,t}^\top P_t \tilde g_k to gp,t⊤Ptmtg_{p,t}^\top P_t m_t. The step count fixes the weight wk,tw_{k,t}, and our response measures the second factor.

All results above appear in the revised PDF (Sec. 2.1, Sec. 5.2, Sec. 6 and App. G). In your review, the two points that limited the strength of the conclusions were the exploratory status of the Pythia state result and the absence of an unconditional gain for actual gradients. In the new experiments, the State block is now a prespecified comparison that passes all six criteria at two model sizes, and the unconditional actual-gradient comparison passes all six criteria once the historical gradient is estimated from a few minibatches. We would be glad to hear whether you find that these results resolve those two limitations, or whether there is a remaining aspect you would like us to clarify before the discussion closes. If the new evidence does change your view of how well the claims are supported, we would be grateful if your final evaluation could take it into account. Thank you again for your helpful and careful review.

Reviewer_rG7cOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Summary: This paper looks at how historical gradients are weighted in the Adam optimizer. It observes that there is implicitly a fixed forgetting schedule according to number of optimizer steps. However, the actual relevance of old gradients could potentially decay quicker or slower than number of optimization steps suggests. It defines a metric for gradient relevance based on Adam-geometry inner gradient inner product, and then it looks for other indicators besides timesteps that could be predictive of gradient relevance. Two proposed indicators “Objective” and “Path” are found to provide improved prediction of gradient relevance.

Critical strengths

S1: The premise that optimizer steps is a poor indicator of gradient relevance is correct.

S2: Exploring other indicators of gradient relevance is an interesting exploratory intellectual exercise.

S3: Two of the proposed indicators do indeed improve the prediction of gradient relevance.

S4: It’s interesting that the indicator=>relevance regression has some cross-environment transferability.

Critical weaknesses

W1: Limited conceptual novelty/unsurprising. The paper writing implies that gradient significance being dependent on other factors than timestep is a new idea/direction of inquiry. However, this is not the case: Fixed timestep dependent gradient accumulation is an assumption well known to be sub-optimal and strong. There are a lot of optimizers that use different strategies to adaptively weight historical/new gradients. Besides the adaptive-memory optimizers mentioned in Related Work section, AdaBelief and L-BFGS use other indicators than number of steps. Furthermore, learned neural optimizers (EG: classic GD^2 NeurIPS’16, Velo arXiv:2211.09760) are all learning how to optimally integrate gradients over time.

W2: The framing of the paper seems to be all about Adam’s momentum term. However, the actual AdamW update depends on bias correction, clipping, v_t, etc. This limits how much insight the analysis provides about the big picture on Adam’s efficacy.

W3: The proposed gradient alignment metric (Eq 4) depends on the Adam-estimated optimizer geometry (Eq 3). (1) Measured alignment will change under different geometries. It’s not unambiguously justified why the Adam-estimated geometry is the ideal one to use here (it feels somewhat circular). (2) Measured alignment is normalized inner product. Do we really want gradients with dramatically different magnitudes but same direction to be considered equally relevant?

W4: Section 6 on actual gradients rather than probe gradients is the more realistic case, but has weaker result. Even after the “birth alignment” step the result is much weaker than the previous sections (+4% vs +20-40%), and less consistent. This undermines the significance of the previous sections.

W5: Overall while there are some interesting observations (albeit of debatable novelty and strength, see previous weaknesses). The whole thing is very far from being operationalized to improve anything in optimization. Unlike for example some of the optimizers mentioned in W1.

W6: The writing is very(!) difficult to parse. Everything is introduced abruptly without motivation or scaffolding. To give a few examples: (1) New jargon “OBJECTIVE” and “PATH” are introduced on Pg1/intro with no context about what they are, or what’s the intuition for why they are relevant/important. (2) The experimental protocol is not explained in a systematic organized way. Lot’s of terminology specific to this paper are introduced abruptly without context or explanation (eg: “survival levels” “burst positions” “test position” “stability interval”, “Original is a post-discovery richer-baseline follow-up” ) (3) The narrative is structured very much like research chronology rather than to follow scientific logic. (“original experiment” “second experiment” post-discovery” “prospective followup” , etc ) Similarly, it lacks a clear separation and structure between scientific conclusions, diagnostics, ablations, and explorations. (4) Lack of clear explanation on important concepts/results throughout. EG: Section 4 is one of the most important results, but the experiment is mentioned almost in passing without any unpacking about why.

Questions for authors

In Eq 6 l (in the sumnation) is undefined.

6 comments
AuthorsOpenReview ↗

Response to Reviewer rG7c:W1-W2

Thank you for recognizing the soundness of our core premise, the value of investigating alternative relevance indicators, and the predictive gains and cross-environment transferability achieved by OBJECTIVE\text{OBJECTIVE} and PATH\text{PATH}. While we fully agree with your critique regarding the manuscript's presentation and narrative structure (W6) and are substantially reorganizing the paper, several technical concerns (W1–W4) stem from existing mathematical formulations and controls that were obscured by our layout. Below, we clarify these existing results and outline the targeted experiments currently underway.

W1. Limited conceptual novelty

We agree that the general idea of using information beyond the timestep count to adjust how gradients are processed is not novel, and we discuss related work in the introduction. However, comparing our work directly with adaptive or learned optimizers (AdaBelief, L-BFGS, GD2\text{GD}^2, and VeLO) conflates designing an update heuristic with measuring the coordinate of gradient aging. Such optimizers modify how gradients are scaled or combined to lower training loss, whereas we hold AdamW fixed and ask a more basic measurement question: is elapsed optimizer time (t−kt-k) sufficient to predict how well a historical gradient remains directionally aligned with the current gradient field? This is exactly the question in the title, “What Ages a Gradient?”

Using a fixed probe protocol, we evaluate the same held-out text at different checkpoints and find that this question can be measured directly. The observed predictive gap not only exceeds a flexible Step–LR baseline, but also transfers across environments without refitting and persists after the first-moment timescale changes. Our work therefore provides a quantitative diagnostic of how optimizer gradients are updated from a new angle. To be clear, this is a diagnostic paper: it deliberately moves beyond the traditional framework of searching for new optimizers and instead measures and diagnoses a higher-level question. This is a point we emphasize repeatedly in the paper. The ICLR 2027 Reviewer Guidelines ask reviewers to be rigorous, open-minded, and concise. In particular, the guidelines state: “not all work needs to compete on an established leaderboard, or work in an established theoretical framework.” Based on our diagnostic results and the comparison with related research questions in Section 7, we are confident that this diagnostic work has its own novelty, and that the diagnostic perspective of moving beyond directly inventing new optimizers reflects distinct value. We therefore believe the novelty of this work should be reassessed appropriately.

W2. AdamW update depends on bias correction, clipping, and the second moment

Our measurement framework already incorporates the second moment (vtv_t), bias correction, gradient clipping, and decoupled weight decay rather than analyzing mtm_t in isolation:

Second moment (vtv_t) and bias correction: Equation (3) directly evaluates alignment under the current bias-corrected second-moment preconditioner:

Pt=diag((v^t+ϵadam)−1).P_t = \mathrm{diag}\left((\sqrt{\hat{v}_t} + \epsilon_{\mathrm{adam}})^{-1}\right).

Moreover, as given in Equation (19) (Appendix F), the loss-driven increment uilossu_i^{\mathrm{loss}} used in PATH\text{PATH} (Eq. 6) and OBJECTIVE\text{OBJECTIVE} (Eq. 7) is reconstructed directly from the bias-corrected moments and learning rate, while cleanly separating decoupled weight decay −ηiλθi-\eta_i \lambda \theta_i:

-\frac{\eta_i}{1-\beta_1^{i+1}} \frac{m_{i+1}}{\sqrt{v_{i+1}/(1-\beta_2^{i+1})} + \epsilon_{\mathrm{adam}}}.$$ Gradient clipping: Appendix B (Table 5) explicitly tests clipping controls: clip exposure alone adds almost no predictive value (1.33% and 1.29%, with 90% SI spanning zero), whereas recomputing $\text{OBJECTIVE}$ with post-clipping gradients (Clipped Objective) preserves the signal (10.80% in Original and 7.00% in Second). Section 6 likewise evaluates the actual clipped gradients $\tilde{g}_k$ entering AdamW. We will move Equation (19) and the clipping controls from Appendices B and F into Section 2 so that the full AdamW mechanics are immediately clear in the main text.
AuthorsOpenReview ↗

Response to Reviewer rG7c:W3-W4

W3. Choice of Adam geometry and normalized alignment

Alternative-geometry state-block comparisons and gradient-magnitude comparisons are already included in Figure 4 (Section 5); Equation (21) (Appendix F) defines the magnitude response:

  • Why PtP_t is natural, and why the findings do not depend on it: The numerator gp,t⊤Ptgp,kg_{p,t}^{\top}P_tg_{p,k} measures the local first-order interaction between the current probe gradient and a historical direction preconditioned by the current AdamW state. To test dependence on this geometry, Figure 4(a) compares current-P (PtP_t) against Raw Euclidean geometry (II) and a Fixed Reference preconditioner (PrefP_{\mathrm{ref}}, frozen at step 750750). Directional state-block gains remain strongly positive across all three geometries. Moreover, we report here an export of the existing stored OBJECTIVE\text{OBJECTIVE} results, using the original richer-baseline aggregation, rather than the state-block aggregation in Figure 4. Under PtP_t, II, and PrefP_{\mathrm{ref}}, the gains are 9.879.87\\%, 8.778.77\\%, and 8.038.03\\% in Original, and 7.487.48\\%, 5.305.30\\%, and 6.256.25\\% in Second. All six Original/Second Objective comparisons meet the original six-part criteria. The positive directional signal in these two environments therefore also appears in unpreconditioned Euclidean space.

  • Direction vs. Magnitude: Equation (4) deliberately isolates directional alignment from magnitude:

\frac{g_{p,t}^{\top} P_t g_{p,k}} {\\|g_{p,t}\\|_{P_t}\\|g_{p,k}\\|_{P_t} + \epsilon_{\mathrm{metric}}}.$$ Meanwhile, magnitude log-ratios, $$\log \frac{\\|g_{p,k}\\|_P + \epsilon_{\mathrm{metric}}} {\\|g_{p,t}\\|_P + \epsilon_{\mathrm{metric}}}, \quad P \in \\{I, P_{\mathrm{ref}}, P_t\\},$$ are evaluated separately in Figure 4(b) (Eq. 21). Figure 4 shows that learning-state features strongly predict *directional* evolution ($+16.69\\%$ to $+37.16\\%$), whereas *magnitude* gains are small, mixed, or interval-unstable across the three geometries. We will surface these Euclidean/Reference-$P$ and magnitude results prominently in Section 2.1. **W4. Weaker results on actual gradients** Rather than undermining earlier sections, Section 6 separates two distinct sources of variation in actual stochastic gradients: (1) **minibatch sampling variation at gradient birth** (step $k$), and (2) **gradient-field evolution** between steps $k$ and $t$. Fixed probes hold text identity constant to isolate (2), whereas an actual single-minibatch gradient $\tilde{g}_k$ also varies with minibatch identity. Section 6 reports $-0.89\\%$ unconditional gain vs. $+4.83\\%$ ($4/4$ test positions and $6/7$ parameter groups positive, $2/5$ survival levels) after birth alignment is added to the regression baseline (birthP, Eq. 20). Birth noise is a possible explanation for this contrast, to be tested by the ablation below. To test directly whether minibatch noise at birth is the primary reason unconditional state transfer is weaker, we are currently running a controlled **batch-averaging ablation**. Specifically, at the same historical parameter checkpoint, we compare a single-minibatch gradient with averages of $4$ and $16$ raw minibatch gradients, applying global clipping once after averaging and leaving the training trajectory unchanged, under the same unconditional evaluation protocol to measure how stabilizing the historical gradient estimate affects the predictive gain of the State block.
AuthorsOpenReview ↗

Response to Reviewer rG7c:W5-Q1

W5. Limited practical value

In empirical deep learning theory and optimization diagnostics, rigorously demonstrating that chronological optimizer time is an incomplete coordinate for fixed-probe gradient relevance is a contribution in its own right. Jumping directly to a modified optimizer without first establishing a clean measurement framework confounds diagnostic validity with hyperparameter tuning. Our work provides a quantitative benchmark showing what information the chronological Step-LR baseline misses (OBJECTIVE\text{OBJECTIVE}, PATH\text{PATH}, and the four-variable State summary) and where it matters (directional evolution versus magnitude), with Objective/Path checks across first-moment memory timescales. This gives a concrete empirical basis for evaluating state-aware memory rules.

Furthermore, OBJECTIVE\text{OBJECTIVE}, PATH\text{PATH}, and the State components can be computed from gradients, parameters, optimizer moments, learning rates, and scalar losses available during AdamW training. For a fixed diagnostic set, each completed interval is represented by O(1)O(1) scalars, independent of model size. Computing these summaries requires additional norm and dot-product reductions, including the parameter-group norms used by Path. We are currently profiling the wall-clock, memory, and storage overhead of the diagnostics to report concrete numbers in the revision.

W6. Presentation and readability

Thank you for your direct and detailed feedback on the writing. We agree that the manuscript is difficult to parse in several places—specifically the abrupt introduction of OBJECTIVE\text{OBJECTIVE} and PATH\text{PATH}, the dense protocol terminology (burst positions, survival levels, stability interval), and structuring the narrative around our audit chronology (Original, Second, post-discovery, prospective follow-up) rather than scientific logic.

We are preparing a substantially rewritten revision that addresses each of your four points:

  1. Clear motivation and intuition upfront: We will introduce OBJECTIVE\text{OBJECTIVE} (loss-normalized gradient-update work, Eq. 7) and PATH\text{PATH} (scale-normalized parameter motion, Eq. 6) in Sections 1 and 2 with clear optimization intuition before defining the regression baselines.

  2. Systematic protocol organization and de-jargoning: We will replace internal protocol jargon (burst positions, survival levels) with self-explanatory terminology in the main text and provide a clean schematic overview of the experimental design.

  3. Scientific structure over research chronology: We will reorganize Sections 3–5 around scientific questions rather than chronological audit stages—separating core findings, state decompositions, and confounder ablations—while moving the pre-registration/post-discovery audit ledger to Appendices A and E.

  4. Unpacking key mechanisms: We will expand Sections 4 and 5 to thoroughly explain why inverse loss S1/L(k,t)S_{1/L}(k,t) and gradient-update alignment Scos⁡(k,t)S_{\cos}(k,t) capture gradient-field evolution.

Q. Undefined summation index in Equation (6)

Thank you for catching this omission. In Equation (6):

\sqrt{ \sum_{\ell} \frac{ \\|u_i^{\mathrm{loss},\ell}\\|_2^2 }{ \\|\theta_i^{\ell}\\|_2^2 + \epsilon_{\mathrm{path}} } }$$ the index $\ell$ denotes the fixed parameter groups used for $\text{PATH}$. For Pythia-160M these are the embedding, each of the 12 Transformer layers, the final LayerNorm, and the output head (15 groups). Within each group, squared update norms and squared parameter norms are summed over its tensors before forming the ratio. The outer sum runs over these groups. We will explicitly define $\ell$ right before Equation (6) in the revised manuscript.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Additional Existing-Log Audits: Geometry and Signed-Work Checks

1. Objective and Path across three gradient geometries

These tables consolidate existing fits using the original richer-baseline aggregation and complement Figure 4's State-block comparison.

Geometry denotes the alignment-response metric: Raw is Euclidean; Ref-P is frozen at completed update 750 of the 1.0x LR run; Current-P uses the current endpoint. Objective and Path definitions are unchanged.

Gain is the reduction in held-out prediction error relative to Rich Step-LR. SI is the recorded 90% finite-design stability interval. Gains, SI bounds, and confirmation gains are in percent.

Test, Surv., Groups, and C.sign count positive gains across test positions, survival levels, evidence groups, and confirmation positions. Conf. is the mean confirmation gain; Met counts criteria satisfied.

Objective: gains and SI

Environment / geometryGainSI lowSI high
Original / Raw8.76824.982512.5539
Original / Ref-P8.02814.527611.5285
Original / Current-P9.87456.507313.2417
Second / Raw5.30413.57056.0024
Second / Ref-P6.24835.58456.3048
Second / Current-P7.48137.05267.5885
Third / Raw2.5972-2.08555.8164
Third / Ref-P4.50130.75926.2468
Third / Current-P4.66180.25996.7595

Objective: consistency audit

Environment / geometryTestSurv.GroupsConf.C.signMet
Original / Raw4/45/57/78.72134/46/6
Original / Ref-P4/44/57/78.02703/46/6
Original / Current-P4/44/57/710.31914/46/6
Second / Raw3/45/56/710.94274/46/6
Second / Ref-P4/45/57/710.25553/46/6
Second / Current-P4/45/57/712.91953/46/6
Third / Raw1/43/52/75.44543/43/6
Third / Ref-P3/43/53/73.94363/45/6
Third / Current-P2/43/54/73.93213/45/6

Path: gains and SI

Environment / geometryGainSI lowSI high
Original / Raw3.70640.76386.6491
Original / Ref-P4.71211.91027.5139
Original / Current-P5.20652.23968.1734
Second / Raw2.2482-1.03525.5317
Second / Ref-P3.57140.39806.7447
Second / Current-P4.04170.32957.7540
Third / Raw3.2272-0.11586.5701
Third / Ref-P4.43550.66378.2074
Third / Current-P5.00720.78179.2327

Path: consistency audit

Environment / geometryTestSurv.GroupsConf.C.signMet
Original / Raw4/44/57/74.97964/46/6
Original / Ref-P4/45/57/76.31314/46/6
Original / Current-P4/45/57/76.58034/46/6
Second / Raw2/43/57/72.59252/43/6
Second / Ref-P3/44/57/72.76692/45/6
Second / Current-P3/44/57/72.93002/45/6
Third / Raw3/45/57/73.41603/45/6
Third / Ref-P4/45/57/74.33363/46/6
Third / Current-P4/45/57/75.04093/46/6

The six criteria are: positive SI lower bound; at least 3/4 positive test positions, 3/5 survival levels, and 4/7 groups; positive confirmation mean; and at least 3/4 positive confirmation positions. Each metric/geometry is evaluated separately.

Objective meets 6/6 in all geometries in Original and Second. In Third, Path meets 6/6 under Current-P and Ref-P, and 5/6 under Raw; Objective meets 3/6 to 5/6 despite positive means.

2. Signed descent work and absolute Objective

Appendix C.1, Eq. (17), reports the identity for Original/Second. Here we consolidate the same existing-log check for all three environments. The signed increment is

-\frac{(g_i^{\mathrm{raw}})^{\top}u_i^{\mathrm{loss}}} {L_i+\epsilon_{\mathrm{obj}}}.$$ Each row pools four original 1,500-update runs. Signs lists positive/zero/negative update counts. Rows counts stored interval rows. Max. diff. is the maximum absolute difference between the signed interval sum and stored absolute Objective. | Environment | Updates | Signs | Rows | Max. diff. | |:---|---:|:---:|---:|---:| | Original | 6,000 | 6,000/0/0 | 99,008 | 0 | | Second | 6,000 | 6,000/0/0 | 99,008 | 0 | | Third | 6,000 | 6,000/0/0 | 99,008 | 0 | All signed increments are positive, and their stored interval sums equal absolute Objective exactly. On these trajectories, the positive-work component equals Objective and the negative component is zero. Signed work supplies the same predictor; these logs contain no negative-work subset for a separate comparison.

Dear Reviewers,

The authors have finished uploading their responses to reviewers, and the discussion phase is now underway. If you haven't already, please carefully review the authors' responses to understand their perspectives. Please engage in thoughtful and constructive discussions with the authors, and update your review or rating if necessary.

The discussion period lasts until November 18th AoE, which means you have less than five days left to interact with the authors. Please start engaging well before the period ends, so that the authors will have sufficient time to respond to any follow-up questions/comments.

Your active participation and contribution to the ongoing discussion are highly encouraged. Thank you very much for your contribution to ICLR.

Best regards,

AC

AuthorsOpenReview ↗

General Response: Summary of Revisions and New Results

We thank the AC and all three reviewers for their time and feedback. For convenience, we summarize the revision and the new results in one place. New and rewritten text in the PDF is marked in blue.

New experiments

  • Longer training and a larger model. Pythia-160M and Pythia-410M, four matched 6,000-update schedules each. Path (+5.15% / +4.06%) and the prespecified State block (+41.66% / +63.46%) pass all six consistency criteria at both sizes (Sec. 5.2, Fig. 6).
  • Actual gradients without birth conditioning. With the historical gradient estimated from M=4M = 4 or 1616 minibatches at the same checkpoint, the unconditional State gain is +2.26% and +4.39%, passing all six criteria (Sec. 6, Fig. 7).
  • Geometry. Objective and Path in Euclidean, frozen-reference and current Adam geometry: all 18 mean gains are positive (Table 1).
  • Cost. Computing the diagnostics online adds 3.1–3.5% per update with unchanged peak memory (Sec. 8, Table 4).
  • Training stage. Training-loss curves and per-schedule window gains (App. G.5). Objective does not carry over to the 6,000-update horizon; the abstract and conclusion now state this, while the predictive gap, Path and the State block do carry over.

Presentation. Following Reviewer rG7c's comments, the paper was rewritten: terms are explained before use, the protocol is described in one place with a schematic, Sections 3–5 follow the scientific questions, and the full AdamW update and clipping controls are in the main text.

We are especially grateful to Reviewer UpMX, who engaged with every round of the discussion and whose questions on scale, actual gradients, cost and training stage each led directly to one of the new experiments above. Their careful and constructive feedback made the paper substantially stronger, and we appreciate their note that the responses have largely addressed their concerns. We thank all reviewers again and remain happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.