- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies what optimizer state must survive when a low-rank optimizer refreshes its gradient subspace. Its main contribution is to separate two effects that are usually conflated in reset ablations: losing the historical second moment (v), and applying an inappropriate bias-correction age to a newly rebuilt (v). The authors derive the resulting update amplification, show in matched GaLore experiments from 60M to 1B parameters that local-age reconstruction can recover near-carry quality even after discarding all pre-refresh (v) history, and identify a state-dependent failure mode in which oversized updates lead to attention concentration and intermittent gradients. They validate these claims with analytical predictions, matched multi-scale language-model runs, experiments with LDAdam, PLUMAGE, and Fira, a sweep over and refresh interval, and targeted interventions on update magnitude, attention logits, and projector sequences.
Critical strengths
-
It makes partial progress on the right question. The local-age experiments show that, in the tested settings, preserving the complete pre-refresh (v) history is not necessary for carry-level quality. This is a useful empirical constraint on optimizer-state design.
-
The confounding is analyzed carefully. The paper compares branches with identical (v)-history loss and different correction ages, and the closed-form amplification prediction agrees with measured update ratios.
Critical weaknesses
-
The central prescription is natural rather than novel. Restarting the bias-correction age after resetting an EMA is the direct application of Adam's standard correction formula. The paper gives a useful quantitative analysis of this point, but does not introduce a new optimizer mechanism or a non-obvious reconstruction rule.
-
The practical failure is not shown to occur in the native optimizers. The original GaLore and Fira implementations retain their moments, LDAdam transports them, and PLUMAGE realigns them. The severe global-age results come from adding a
reset_vintervention while keeping the global counter. Thus the paper exposes a weakness of a diagnostic reset protocol, rather than a demonstrated defect in the existing low-rank optimizers.
Questions for authors
Clarification requests
-
Do any native GaLore, LDAdam, PLUMAGE, or Fira implementations actually clear (v) at a refresh while retaining a global bias-correction counter? If not, what existing reset procedure is the global-age
reset_vcondition intended to represent? -
In the cross-optimizer experiments, were the reset interventions applied only after each method's native state transition, and do the reported local-age results use exactly the same native transition and hyperparameters? This affects whether the results support a claim about the methods or only about the added reset protocol.
Suggestions for future work
None.
5 comments
Response to Reviewer R8Pj (2/3)
3. But this age mismatch does exist in published methods
The mismatch that our intervention isolates (moments that have just been rebuilt, but are corrected with an already old counter) does, however, exist in published methods. SPAM lets us test it directly; ReLoRA provides a second published example of the same general pattern.
SPAM (ICLR 2025) does not use a low-rank subspace. In its native pretraining procedure, it periodically resets both Adam moments to zero, keeps the global counter, and applies a warmup after each reset. We tested whether correcting the ages of the rebuilt moments directly can replace this compensating warmup.
Comparisons 1→2, 2→3 and 3→4 each change a single factor. Row 5 is our own intervention, the one studied in the paper, applied to SPAM.
| Condition | Rebuilt moments | Correction ages | Post-reset warmup | Learning rate | Final PPL |
|---|---|---|---|---|---|
| 1. Published recipe | global | yes | 1e-3 | 30.6 | |
| 2. No warmup | global | no | 1e-3 | 36.9 | |
| 3. Calibrated ages | local | no | 1e-3 | 31.8 | |
| 4. Calibrated ages, adjusted LR | local | no | 2e-3 | 30.3 | |
| 5. Our intervention: reset of only, calibrated age | local | no | 2e-3 | 30.1 |
All rows: mean over three matched realizations.
Removing the warmup degrades the native recipe (row 2), consistent with the SPAM authors' own ablation. Restarting the ages alone recovers most of that loss with no warmup (row 3), and the rest is recovered at twice the learning rate (row 4), in line with the roughly 1.8× larger updates we measure under the global age, which follow our prediction within 5–10%. Our response to Reviewer aiXW walks through the table step by step.
Row 5 is our intervention: keeping and rebuilding only with a local age reaches the lowest perplexity in the table (30.1) with no warmup, in all three matched realizations. This is not SPAM's native procedure; the row shows that the mechanism isolated by our intervention also appears outside the low-rank setting.
In the grid reported by the SPAM authors, 1e-3 is the highest learning rate. We additionally evaluated their native recipe at 1.5e-3 and 2e-3: it is optimal at 1e-3 in this extended grid and degrades above.
We do not claim that SPAM implements our selective intervention: SPAM resets both moments. Nor do we present this as a defect that makes SPAM fail; its recipe works. SPAM provides a native, published instance of the broader phenomenon our analysis isolates: young statistics corrected under an advanced counter and compensated by a warmup.
ReLoRA offers a second example of this general pattern. Its official implementation prunes the two Adam moments almost entirely without resetting the step counter, and the method uses a short warmup after each restart. The authors report that a naive reset makes training diverge. We do not claim that ReLoRA performs our reset-v intervention. It does show that rebuilding optimizer state under an already advanced clock, and then compensating with a warmup, is a real practice. We tested it at 60M, at the learning rate given by its authors' rule: without the restart warmup, training diverges at the first reset in all three matched realizations, whereas restarting the step counter of the reset parameters, still without warmup, keeps it stable, at a perplexity equivalent to that of the published recipe (41.10 against 40.97).
The severe GaLore intervention therefore remains an identification experiment, while the SPAM result establishes that this calibration problem exists in a native reset procedure, with a measurable practical consequence.
Response to Reviewer R8Pj (3/3)
4. Where the interventions are applied
Yes: in every cross-optimizer experiment, the method first performs its native state transition, and the intervention is applied only afterwards to the resulting state, before the current gradient is accumulated. All matched variants share the initialization, the data order, the learning-rate schedule, the hyperparameters, the initial projector construction and the native transition rule. The local-age and global-age branches differ only in the age used for the rebuilt .
More precisely:
- GaLore performs its native refresh while retaining its projected moments, then the intervention is applied.
- PLUMAGE performs its native realignment, then the intervention is applied to the realigned state. The local-age result therefore directly tests whether the history of realigned by PLUMAGE is empirically necessary in the tested runs.
- Fira performs its projector refresh while retaining its residual path; that path and all native mechanisms remain unchanged across interventions.
- LDAdam first applies its native transport and error feedback. Because LDAdam refreshes at every update, the asymmetry experiment applies the interventions at every update, whereas the check of the age mechanism uses a single reset at update 2,000, so that can rebuild. We do not present this second protocol as an alternative to LDAdam's native carry.
Appendix A and the description of the experimental setup have been extended to make this ordering explicit.
5. What we claim, and what we do not
Two statements are now kept apart in the paper:
- the catastrophic global-age results characterize the added diagnostic reset protocol, not a defect of the native GaLore, LDAdam, PLUMAGE or Fira algorithms;
- because the interventions are applied after each method's native transition, the local-age experiments do provide a method-specific empirical statement: the complete history of produced by that native transition can be discarded, in the tested configurations, without reducing final quality.
In short, we fully accept the distinction you raise: our global-age reset-v is a diagnostic ablation, not a failure of the four native methods. The empirical conclusions nonetheless remain substantial. In the tested configurations, the entire history of can be deleted at every refresh while recovering carry-level quality up to 1B; deleting as well is far less damaging than deleting alone in the configurations of Table 2; and SPAM shows that the mismatch between the actual age of rebuilt statistics and the counter used to correct them exists in a published procedure. In our SPAM experiment, correcting this age directly removes the need for the warmup without degrading final quality.
Response to Reviewer R8Pj (1/3)
Thank you for the careful review. You are right on two important points: restarting the age of a rebuilt statistic is a standard application of Adam's correction, and none of the four methods we evaluate natively performs the global-age
reset-vthat we use as an intervention.The revision now makes both points explicit (changes in blue). They clarify what our intervention is; they do not summarize what the experiments establish.
1. What the ablation actually establishes
Keeping the global age after a reset, as our
reset-vbranch does on purpose, is the miscalibrated choice; restarting it is the standard one. We do not present this restart as a contribution: it is our experimental control. The two branches erase exactly the same history, and comparing them is what lets us tell the loss of history apart from the miscalibration of the rebuilt statistic. The non-obvious results are what this comparison establishes:reset-allis less damaging thanreset-vin the four optimizers of Table 2, although it removes more state. At GaLore-60M, the final-perplexity ratios are 1.36× forreset-alland 11.82× forreset-v.The intended contribution is thus an identification and mechanism result about optimizer-state reconstruction, not a new optimizer mechanism.
2. None of the four methods performs this reset natively
No, and we now say so directly. The selective
reset-vbranch is not meant to reproduce any native implementation of GaLore, LDAdam, PLUMAGE or Fira, and we do not claim that these methods contain this defect. It is a controlled intervention, designed to answer a narrower causal question:The global-age and local-age branches hold fixed the native transition, the retained m, the deleted history of v, the refresh times, the hyperparameters, the data and the learning-rate schedule. They differ only in the correction age assigned to the rebuilt v. This is what makes the history-loss question identifiable. The revised paper calls
reset-va diagnostic ablation, not a proposed training rule or a failure of a native optimizer.