- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines the distinction between causal persistence and useful intervention value of optimizer memory in language model adaptation. Focusing on Adam's second-moment states, the authors conduct a series of rigorously controlled experiments on Qwen2.5-1.5B, adapting from C4 to FineMath and WikiText-103. They systematically compare four scenarios: (1) late second-moment replacement at a reached checkpoint, (2) warm-origin planning with fixed higher-rate training, (3) source refresh via gradient probes, and (4) native recent-history reweighting. Findings reveal that while second-moment edits causally persist (reducing target loss by up to 674.54 micro-NLL in one setting), they are consistently dominated by simpler retained-state continuation or additional ordinary updates funded by the intervention's training-time allowance. Native recent reweighting harms both domains in all eight confirmation allocations, and source refresh yields only a small, regime-specific gain (34.02 micro-NLL). The authors conclude that optimizer-memory interventions should not be judged by persistence alone; instead, their value depends on the specified decision point, objective, and feasible comparator—a framing that clarifies when such interventions are genuinely useful versus merely consequential.
Critical strengths
- Conceptual clarity and rigorous framing. The paper cleanly separates "optimizer state persists" from "optimizer state intervention is useful," and systematically contrasts four question types (reached-checkpoint continuation, warm-origin planning, marginal refresh, native reweighting) via Table 1. This avoids the common "can change loss → should use it" fallacy in many optimizer papers.
- Exceptionally controlled experimental design. Forking from identical weights, RNG, data cursors, and future tapes while replacing only ; using shadow moments, source probes, native accumulators, magnitude/direction grafts, and training-time reinvestment to peel back mechanisms layer by layer. Allocation-level bootstrap uncertainty and the explicit warning that "shared allocations are not independent replications" are handled with commendable rigor.
- Honest, conservative conclusions. The authors do not overclaim a "new optimizer" or SOTA; they explicitly state "small NLL gains have no demonstrated downstream value," "unresolved intervals are not equivalence tests," and disclose AI use, ethics, and reproducibility limits. This intellectual honesty is a strength.
Critical weaknesses
- Narrow scale and limited generalizability. Essentially only Qwen2.5-1.5B with C4/FineMath/WikiText-103. No multi-family, multi-scale, or alternative optimizer (SGD, Adafactor, Shampoo, Adam-mini) validation. The findings risk being perceived as "fine-grained phenomena on one 1.5B trajectory" rather than general training dynamics principles.
- Extremely small effect sizes with unclear practical value. Primary results are in micro-NLL (10⁻⁶ NLL): 34, 674, 307 appear large in tables but represent ~10⁻⁴ relative to total NLL. The paper itself admits no downstream task/utility/security/privacy evidence, so the question of "is it actually useful?" remains unanswered.
- Your results hold on Qwen2.5-1.5B. If you scaled to 7B/13B or switched to SGD-momentum / Shampoo / Adam-mini, would you still predict that "late second-moment edits persist but do not dominate"? Have you run any pilot studies? Micro-NLL gains are nearly imperceptible in perplexity. Do you believe the applicable boundary of optimizer-memory interventions should be "training cost / recovery cost / continual-learning state management" rather than "final loss"? If so, can you articulate a concrete deployment scenario where this value proposition holds? RECENT_HALF harmed both domains, but your symmetric LR/STOP menu selected rates favoring B while constraining A. If you used Pareto / multi-objective / looser reference constraints, could native recent-history reweighting still have conditional value in some allocations? In the training-time reinvestment experiment, the additional ordinary updates (PP₁₄₂₁) consumed more target-domain tokens, so it is not a strictly matched FLOP/data comparison. How do you view the causal separation between "the value of old-rich editing" and "the value of simply training a few more steps"?Heavy writing burden that buries the core message. The main text + appendices pile on allocation dependencies, menu selection, interval families, replay details, GPU hours, and deleted checkpoints. The core claim could be stated in three sentences, but readers must excavate it from 22+ pages. For ICLR, the communication cost is high and risks reviewers remembering only "they did many experiments but drew no conclusion."
Questions for authors
- Your results hold on Qwen2.5-1.5B. If you scaled to 7B/13B or switched to SGD-momentum / Shampoo / Adam-mini, would you still predict that "late second-moment edits persist but do not dominate"? Have you run any pilot studies?
- Micro-NLL gains are nearly imperceptible in perplexity. Do you believe the applicable boundary of optimizer-memory interventions should be "training cost / recovery cost / continual-learning state management" rather than "final loss"? If so, can you articulate a concrete deployment scenario where this value proposition holds?
- RECENT_HALF harmed both domains, but your symmetric LR/STOP menu selected rates favoring B while constraining A. If you used Pareto / multi-objective / looser reference constraints, could native recent-history reweighting still have conditional value in some allocations?
- In the training-time reinvestment experiment, the additional ordinary updates (PP₁₄₂₁) consumed more target-domain tokens, so it is not a strictly matched FLOP/data comparison. How do you view the causal separation between "the value of old-rich editing" and "the value of simply training a few more steps"?