- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the relevance of historical gradients in AdamW can be measured by elapsed optimizer steps alone. The authors argue that even when two gradients have the same age in terms of update steps, the model may have undergone different amounts of change, leading to different levels of alignment with the current gradient. To study this, they use fixed held-out text samples to measure gradient alignment across training checkpoints, avoiding variation caused by different minibatches. They also introduce two diagnostic measures, OBJECTIVE and PATH, which capture gradient–update interactions and relative parameter changes, respectively.
Experiments are mainly conducted on Pythia-160M, with additional tests on OPT-125M. The results show that OBJECTIVE reduces prediction error by 9.87% and 7.48% in two training environments and shows some transferability across environments. Further analysis finds that combining information about gradient norms, update magnitudes, gradient–update alignment, and training loss improves the prediction of historical gradient relevance. The authors also examine different training settings, temporal misalignment, and actual stochastic gradients to test the robustness and limits of these findings.
Overall, the paper shows that elapsed optimizer steps alone do not fully capture the relevance of historical gradients. Changes in the model's learning state provide additional information, which may help better understand optimizer memory and inform future state-aware optimization methods.
Critical strengths
S1. The paper addresses a meaningful and interesting research question. It examines whether the relevance of historical gradients can be measured by elapsed optimizer steps alone and how changes in the learning state affect gradient relevance. This helps improve our understanding of optimizer memory and may inform the design of more flexible gradient memory strategies.
S2. The experimental design is relatively rigorous. The authors use fixed held-out samples to control for variation across minibatches and test their findings under different random seeds, learning rate schedules, and momentum settings. Cross-environment and cross-architecture experiments also provide some evidence of the robustness of the results.
S3. The paper provides a detailed analysis of the experimental findings. Beyond comparing different measures for predicting historical gradient relevance, the authors examine the roles of gradient norms, parameter updates, and training loss. In particular, Section 5 considers alternative simple explanations, such as the capacity of the time-based baseline, parameter motion, and training progress. The temporal misalignment experiments further help assess the role of learning-state information and the limits of the findings.
Critical weaknesses
W1. Limited experimental scale and generalizability. The experiments mainly focus on Pythia-160M, with additional cross-architecture tests on OPT-125M. However, both models are relatively small, and training lasts only about 1,500 optimizer steps. It remains unclear whether the findings generalize to larger models, longer training runs, or different architectures and tasks.
W2. Limited transfer from fixed probes to actual training gradients. Using fixed held-out samples helps isolate the effects of learning-state changes from minibatch variation, but it also differs from real training conditions. Section 6 shows that the proposed measures have limited predictive value for actual stochastic gradients without conditioning on their alignment at birth. This raises questions about how well the main findings apply to practical optimizer settings.
W3. Unclear practical value of the proposed diagnostic measures. The paper shows that OBJECTIVE, PATH, and learning-state features can better predict historical gradient relevance, but it does not demonstrate whether they improve training efficiency or final model performance. Moreover, gradient alignment does not necessarily reflect how useful a historical gradient is for optimization, since momentum also helps reduce noise and stabilize updates. The additional computation and storage costs of these measures are also unclear, making their practical benefits difficult to assess.
Questions for authors
Q1. Could the authors provide some additional evidence, even on a small scale, to help assess whether the findings generalize to larger models or longer training runs? (see W1)
Q2. How can the proposed measures be made more effective for actual stochastic gradients without relying on birth-alignment information? (see W2)
Q3. To what extent can the proposed measures improve practical optimization performance, and what are their additional computational and storage costs? (see W3)
8 comments
Thanks for your response.
Thank you for your detailed response and for your efforts to address my concerns.
I would be happy to review any additional experimental results or analyses if they become available during the rebuttal period. I will also take into account the existing evidence and the other reviewers' perspectives, particularly regarding actual-gradient relevance and cross-architecture generalizability, when finalizing my assessment.
Follow-up: revised PDF and new results
Thank you for your thoughtful and constructive review. Your three questions defined most of the new experiments, and we believe they have made the paper substantially stronger. The revised PDF is now uploaded, with new and rewritten text in blue.
W1 / Q1. Larger model and longer training
We trained Pythia-160M and Pythia-410M on four matched 6,000-update schedules each, four times the original horizon. The Rich Step–LR baseline, probes, splits, lags and six consistency criteria are unchanged, and the window start steps are scaled by four.
| Model | Path (prespecified candidate) | State block (prespecified supporting comparison) |
|---|---|---|
| Pythia-160M | +5.15%, 6/6 criteria | +41.66% test, +46.81% confirmation, 6/6 |
| Pythia-410M | +4.06%, 6/6 criteria | +63.46% test, +63.75% confirmation, 6/6 |
For the State block, all four test positions are positive at both sizes. The predictive gap left by elapsed steps and accumulated learning rate therefore persists into later training and at a larger model size (Sec. 5.2, Fig. 6, Table 3).
W2 / Q2. Actual gradients without birth alignment
At the same historical checkpoints, we estimated each historical gradient from training minibatches, evaluated at the same parameter point and globally clipped once after averaging, with the training trajectory unchanged. Without any birth-alignment information, the State block gives test gains of −0.89%, +2.26% and +4.39%. For and , the 90% stability intervals are [1.35%, 3.17%] and [0.06%, 8.72%], the confirmation gains are +3.70% and +8.18%, and both pass all six criteria. To answer Q2 directly: the measures become effective for stochastic gradients once the historical estimate averages a few minibatches. For reference, with the first moment itself averages roughly ten recent minibatch gradients (Sec. 6, Fig. 7).
W3 / Q3. Computational and storage cost
On Pythia-160M (RTX 3090, 45 synchronized timed runs), computing Objective, Path or all four State components adds 3.13%, 3.38% or 3.51% to the median time per update, with unchanged peak memory and 69–231 bytes of log per update; our full research observer adds 12.39% (Sec. 8, Table 4). Probe gradients, stored historical gradients and checkpoints are needed only to measure relevance in this study, not to compute the diagnostics. We agree that testing a state-dependent memory rule on convergence and final loss is the natural next step, and these results show that its inputs would be cheap to compute.
All results above appear in the revised PDF (Sec. 5.2, Sec. 6, Sec. 8 and App. G). Your three questions, on scale and training length, on actual gradients without birth alignment, and on practical cost, each led to a dedicated experiment, and we think the paper is considerably stronger for them. We would value your view on whether these results answer the questions as you intended, and whether you see any further analysis that would make the evidence more convincing. If you feel the revision has addressed the weaknesses noted in your review, we would be grateful if this could be reflected in your final assessment. Thank you again for your careful, constructive and encouraging review.
Response to Reviewer UpMX
Thank you for your positive assessment of our research question, controlled experimental design, and the confounding-factor and temporal-misalignment analyses in Section 5. We address each of your weaknesses and questions below.
W1 & Q1. Experimental scale and generalizability to larger models and longer runs
We agree that extending the evaluation beyond Pythia-160M / OPT-125M and the 1,500-update horizon is important to assess generalizability. We are currently running two controlled validation experiments that preserve the original Rich Step-LR baseline specification, probe/split assignments, diagnostic definitions, and six-part consistency criteria:
Longer horizon on Pythia-160M: Four matched schedules of 6,000 updates each, to test whether the predictive gap persists into later training stages.
Larger model with extended horizon: Pythia-410M with the same four schedules and 6,000-update horizon.
The learning-rate schedule is extended to 6,000 updates, measurement-window starts are scaled by four, and the reference preconditioner is taken at step 3,000. The 64-checkpoint windows and historical lags remain unchanged. Objective and Path are separate prespecified candidates, and the State block is a supporting secondary. This tests later training stages and model size within Pythia.
W2 & Q2. Transfer from fixed probes to actual training gradients without birth-alignment information
In Section 6, the unconditional actual-gradient comparison yields −0.89, whereas conditioning on birth alignment (Bk,p,ebirthP, Eq. 20) yields +4.83 (4/4 positive test positions, 6/7 positive groups), showing positive state predictivity conditional on birth alignment. Whether minibatch sampling noise explains the weak unconditional result is the hypothesis tested by the follow-up.
To test whether reducing birth variance makes the learning-state features directly effective without relying on held-out birth-alignment information (birthP), we are currently conducting a batch-averaging ablation on actual training gradients. Specifically, under the exact same unconditional Rich STEP-LR baseline and evaluation protocol, we compare gradient estimates formed at the same historical parameter checkpoint, averaging raw gradients before one global clipping operation and leaving the original training trajectory unchanged:
This directly tests the hypothesis that suppressing stochastic minibatch noise at birth allows the State block signal to emerge unconditionally. Methodologically, we also note that batch averaging modifies the measured historical vector itself, so we treat this as a controlled diagnostic of whether initial gradient stability affects state predictability rather than claiming an immediate modification to standard training utility.
W3 & Q3. Practical optimization performance, computational costs, and storage overhead
We address the computational/storage costs of the diagnostics and their relationship to practical optimization separately:
(i) the exact per-update wall-clock time required to compute OBJECTIVE, PATH, and the State block; and
(ii) the additional memory and storage footprint compared against vanilla AdamW training.
We distinguish scalar-statistics overhead from full-observer, probe-measurement, historical-gradient-storage, and checkpoint costs. The results will quantify the additional cost of each mode.