acceptodds
Under review as a conference paper at ICLR 2027

Optimizer Memory: Persistence and Intervention Value in Language Model Adaptation

Abstract

Optimizer moments preserve gradient history, but a persistent causal effect need not identify a useful intervention. Controlled language-model adaptation separates intervention at a reached checkpoint from planning at a common warm origin. Across twelve allocations, late second-moment replacement reduces target loss by 674.54 micro-NLL while increasing reference loss by 15.60; the complete procedure nevertheless loses to higher-rate retained-state training. A separate comparison finds a small refresh benefit of 34.02 micro-NLL, but additional ordinary updates reduce target loss below an old-rich intervention when its training-time allowance is reinvested. On an unedited native trajectory, recent-history reweighting worsens both domains in all eight confirmation allocations after symmetric finite-menu suffix selection. Native graft comparisons clarify this failure: both the magnitude change and the direction change raise target loss, although their reference effects differ. Actual adaptation to WikiText-103 retains the recent-direction target penalty in one further setting. These paired interventions distinguish causal persistence, conditional marginal gain and useful continuation without establishing a general preference for older statistics or practical optimizer superiority.

Primary area:
optimization
Keywords:
language model adaptation, optimizer state, second moments, adaptive optimization, training dynamics, retention-adaptation trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_17RgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the distinction between causal persistence and useful intervention value of optimizer memory in language model adaptation. Focusing on Adam's second-moment states, the authors conduct a series of rigorously controlled experiments on Qwen2.5-1.5B, adapting from C4 to FineMath and WikiText-103. They systematically compare four scenarios: (1) late second-moment replacement at a reached checkpoint, (2) warm-origin planning with fixed higher-rate training, (3) source refresh via gradient probes, and (4) native recent-history reweighting. Findings reveal that while second-moment edits causally persist (reducing target loss by up to 674.54 micro-NLL in one setting), they are consistently dominated by simpler retained-state continuation or additional ordinary updates funded by the intervention's training-time allowance. Native recent reweighting harms both domains in all eight confirmation allocations, and source refresh yields only a small, regime-specific gain (34.02 micro-NLL). The authors conclude that optimizer-memory interventions should not be judged by persistence alone; instead, their value depends on the specified decision point, objective, and feasible comparator—a framing that clarifies when such interventions are genuinely useful versus merely consequential.

Critical strengths
  1. Conceptual clarity and rigorous framing.​ The paper cleanly separates "optimizer state persists" from "optimizer state intervention is useful," and systematically contrasts four question types (reached-checkpoint continuation, warm-origin planning, marginal refresh, native reweighting) via Table 1. This avoids the common "can change loss → should use it" fallacy in many optimizer papers.
  2. Exceptionally controlled experimental design.​ Forking from identical weights, RNG, data cursors, and future tapes while replacing only vv; using shadow moments, source probes, native accumulators, magnitude/direction grafts, and training-time reinvestment to peel back mechanisms layer by layer. Allocation-level bootstrap uncertainty and the explicit warning that "shared allocations are not independent replications" are handled with commendable rigor.
  3. Honest, conservative conclusions.​ The authors do not overclaim a "new optimizer" or SOTA; they explicitly state "small NLL gains have no demonstrated downstream value," "unresolved intervals are not equivalence tests," and disclose AI use, ethics, and reproducibility limits. This intellectual honesty is a strength.
Critical weaknesses
  1. Narrow scale and limited generalizability.​ Essentially only Qwen2.5-1.5B with C4/FineMath/WikiText-103. No multi-family, multi-scale, or alternative optimizer (SGD, Adafactor, Shampoo, Adam-mini) validation. The findings risk being perceived as "fine-grained phenomena on one 1.5B trajectory" rather than general training dynamics principles.
  2. Extremely small effect sizes with unclear practical value.​ Primary results are in micro-NLL (10⁻⁶ NLL): 34, 674, 307 appear large in tables but represent ~10⁻⁴ relative to total NLL. The paper itself admits no downstream task/utility/security/privacy evidence, so the question of "is it actually useful?" remains unanswered.
  3. Your results hold on Qwen2.5-1.5B. If you scaled to 7B/13B or switched to SGD-momentum / Shampoo / Adam-mini, would you still predict that "late second-moment edits persist but do not dominate"? Have you run any pilot studies? Micro-NLL gains are nearly imperceptible in perplexity. Do you believe the applicable boundary of optimizer-memory interventions should be "training cost / recovery cost / continual-learning state management" rather than "final loss"? If so, can you articulate a concrete deployment scenario where this value proposition holds? RECENT_HALF harmed both domains, but your symmetric LR/STOP menu selected rates favoring B while constraining A. If you used Pareto / multi-objective / looser reference constraints, could native recent-history reweighting still have conditional value in some allocations? In the training-time reinvestment experiment, the additional ordinary updates (PP₁₄₂₁) consumed more target-domain tokens, so it is not a strictly matched FLOP/data comparison. How do you view the causal separation between "the value of old-rich editing" and "the value of simply training a few more steps"?Heavy writing burden that buries the core message.​ The main text + appendices pile on allocation dependencies, menu selection, interval families, replay details, GPU hours, and deleted checkpoints. The core claim could be stated in three sentences, but readers must excavate it from 22+ pages. For ICLR, the communication cost is high and risks reviewers remembering only "they did many experiments but drew no conclusion."
Questions for authors
  1. Your results hold on Qwen2.5-1.5B. If you scaled to 7B/13B or switched to SGD-momentum / Shampoo / Adam-mini, would you still predict that "late second-moment edits persist but do not dominate"? Have you run any pilot studies?
  2. Micro-NLL gains are nearly imperceptible in perplexity. Do you believe the applicable boundary of optimizer-memory interventions should be "training cost / recovery cost / continual-learning state management" rather than "final loss"? If so, can you articulate a concrete deployment scenario where this value proposition holds?
  3. RECENT_HALF harmed both domains, but your symmetric LR/STOP menu selected rates favoring B while constraining A. If you used Pareto / multi-objective / looser reference constraints, could native recent-history reweighting still have conditional value in some allocations?
  4. In the training-time reinvestment experiment, the additional ordinary updates (PP₁₄₂₁) consumed more target-domain tokens, so it is not a strictly matched FLOP/data comparison. How do you view the causal separation between "the value of old-rich editing" and "the value of simply training a few more steps"?
Reviewer_ayxrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether the gradient history stored in Adam is worth actively modifying during model adaptation. Starting from the same checkpoint, the authors keep the weights, momentum, and subsequent training data fixed, and only replace or reweight the second moment. They compare this against controls such as adjusting the learning rate or training for more steps. They find that these historical statistics do keep influencing later training, but modifying them usually brings only small gains, which ordinary learning rate tuning or extra training can match or exceed. In the tested settings, putting more weight on recent gradient statistics actually hurts performance on both the target domain and the original domain. The main takeaway is that a lasting influence of optimizer history does not mean modifying it is useful in practice. The evidence so far comes mainly from adaptation experiments on Qwen2.5-1.5B, and downstream task gains have not been shown.

Critical strengths

(1) A clear distinction between two claims. The paper separates “optimizer state has a lasting causal effect” from “modifying that state is useful in practice.” It also separates continued-training decisions at the current checkpoint from planning training from a shared starting point, which helps explain results that would otherwise seem contradictory.

(2) Tightly controlled interventions. At each branch point, the authors modify only the second moment, keeping the weights, momentum, step count, and random state unchanged, and compare runs on the same subsequent training data.

(3) Baselines go beyond keeping the original optimizer state. The authors also consider learning rate choices, and spending the time cost of the intervention on extra training instead. This helps judge whether the observed gains are worth the cost of intervening.

(4) Careful handling of uncertainty and dependence between runs. The paper keeps development-stage choices separate from confirmation experiments, reports intervals that cover variation in both training data allocation and evaluation documents, and avoids treating repeated measurements as independent replicates.

Critical weaknesses

(1) The experiments in the paper are narrow. For example, almost all the experiments based on the same model, Qwen25-1.5B, which is adapted from C4 to FineMath dataset. So It's not clear whether the proposed method can scale to larger models and other experiment settings.

(2) It's not clear whether the proposed method can work well in the real applications. Some of the gains are only a few dozen micro-NLL, which is tiny compared with roughly 0.13 NLL performance gains from the adaptation itself.

(3) The interventions do not isolate the role of temporal history. Reweighting the second moment also changes its scale, how concentrated the estimate is, and its coordinate-wise geometry. The extra controls help characterize these effects, but they still cannot establish whether the results come from how recent the statistics are.

(4) The continued-training baseline is not fully tuned. Even with a wider learning rate search, the baseline that keeps the original state still picks the upper end of the search range. Searching further over learning rates, schedules, and intervention strength would help show whether the reported ranking of methods is stable.

(5) Given how small the effects are, numerical robustness needs more checking. Replaying the same execution path supports reproducibility under the current implementation, but the check on the maximum coordinate-wise difference between full-batch and micro-batch accumulation did not pass. So it remains unclear how sensitive the results are to implementation changes.

Questions for authors

N/A

Reviewer_VUcVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is extremely hard to read that it is almost impossible to fully understand.

The paper studies how to reset the momentum of Adam and how does this influence the effect of continual pretraining, in terms of both adaption and forgetting. As the experiment designs are complicated and hard to fully parse, it is unclear what the final conclusion is. However, as the maximal change in this paper is smaller than 1e-3 in pretraining loss, it seems reasonable to conclude that none of the intervention fundamentally changes the learning dynamics.

Critical strengths

None

Critical weaknesses
  1. The paper is poorly written and structured

  2. The experiment results seems to be all within noise (less than 1e-3 loss difference)

Questions for authors
  1. Is this purely AI generated articles and have you actually fact checked the results before submission?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.