acceptodds
Under review as a conference paper at ICLR 2027

What Must Survive a Subspace Refresh? Optimizer Memory, Calibration, and Instability

Abstract

Refresh-based low-rank optimizers periodically change the low-dimensional representation in which they maintain gradient statistics. When this representation changes, must the accumulated statistics be preserved, or can they be rebuilt from new gradients? We vary the statistical age used in Adam's standard second-moment bias correction. At each refresh, we erase the second-moment statistic while retaining the first moment , which smooths gradients. We compare two reconstructions that discard the same pre-refresh history and differ only in whether the correction age is global or restarted. The global-age reconstruction raises final perplexity to roughly the no-reset baseline on average across three GaLore-1B runs. Using the rebuilt statistic's own age recovers carry-level quality across GaLore scales up to 1B parameters, and in PLUMAGE and Fira runs at 60M, despite discarding all pre-refresh history at every refresh. The failure of the global-age ablation therefore cannot be attributed to history loss alone: preserving pre-refresh history is not necessary to recover carry-level quality in these settings. We also show that calibrated reconstruction can become hazardous from some training states, and trace one mechanism by which this occurs. In GaLore-60M, repeated oversized updates can drive attention concentration and highly intermittent gradients, causing a freshly rebuilt second moment to misrepresent the scale of retained momentum. Targeted interventions support this feedback mechanism. A late switch to correctly aged reconstruction progressively improves the gradient regime and training quality. A correct statistical age is therefore sufficient along the healthy trajectories tested, but calibration alone does not make reconstruction state-invariant: safe reconstruction also depends on the state training has reached.

Primary area:
optimization
Keywords:
Low-rank optimization, Adaptive optimizers, Optimizer state, Optimizer-state reconstruction, Subspace refresh, Training stability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_R8PjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what optimizer state must survive when a low-rank optimizer refreshes its gradient subspace. Its main contribution is to separate two effects that are usually conflated in reset ablations: losing the historical second moment (v), and applying an inappropriate bias-correction age to a newly rebuilt (v). The authors derive the resulting update amplification, show in matched GaLore experiments from 60M to 1B parameters that local-age reconstruction can recover near-carry quality even after discarding all pre-refresh (v) history, and identify a state-dependent failure mode in which oversized updates lead to attention concentration and intermittent gradients. They validate these claims with analytical predictions, matched multi-scale language-model runs, experiments with LDAdam, PLUMAGE, and Fira, a sweep over β2\beta_2 and refresh interval, and targeted interventions on update magnitude, attention logits, and projector sequences.

Critical strengths
  1. It makes partial progress on the right question. The local-age experiments show that, in the tested settings, preserving the complete pre-refresh (v) history is not necessary for carry-level quality. This is a useful empirical constraint on optimizer-state design.

  2. The confounding is analyzed carefully. The paper compares branches with identical (v)-history loss and different correction ages, and the closed-form amplification prediction agrees with measured update ratios.

Critical weaknesses
  1. The central prescription is natural rather than novel. Restarting the bias-correction age after resetting an EMA is the direct application of Adam's standard correction formula. The paper gives a useful quantitative analysis of this point, but does not introduce a new optimizer mechanism or a non-obvious reconstruction rule.

  2. The practical failure is not shown to occur in the native optimizers. The original GaLore and Fira implementations retain their moments, LDAdam transports them, and PLUMAGE realigns them. The severe global-age results come from adding a reset_v intervention while keeping the global counter. Thus the paper exposes a weakness of a diagnostic reset protocol, rather than a demonstrated defect in the existing low-rank optimizers.

Questions for authors

Clarification requests

  1. Do any native GaLore, LDAdam, PLUMAGE, or Fira implementations actually clear (v) at a refresh while retaining a global bias-correction counter? If not, what existing reset procedure is the global-age reset_v condition intended to represent?

  2. In the cross-optimizer experiments, were the reset interventions applied only after each method's native state transition, and do the reported local-age results use exactly the same native transition and hyperparameters? This affects whether the results support a claim about the methods or only about the added reset protocol.

Suggestions for future work

None.

5 comments
AuthorsOpenReview ↗

Response to Reviewer R8Pj (1/3)

Thank you for the careful review. You are right on two important points: restarting the age of a rebuilt statistic is a standard application of Adam's correction, and none of the four methods we evaluate natively performs the global-age reset-v that we use as an intervention.

The revision now makes both points explicit (changes in blue). They clarify what our intervention is; they do not summarize what the experiments establish.

1. What the ablation actually establishes

Keeping the global age after a reset, as our reset-v branch does on purpose, is the miscalibrated choice; restarting it is the standard one. We do not present this restart as a contribution: it is our experimental control. The two branches erase exactly the same history, and comparing them is what lets us tell the loss of history apart from the miscalibration of the rebuilt statistic. The non-obvious results are what this comparison establishes:

  1. Deleting the entire history of vv can preserve final quality: in the tested configurations, the entire pre-refresh history of vv can be discarded at every refresh while recovering the quality obtained by keeping the state (carry). At 1B, the three matched 100,000-step runs end within 0.6% of carry.
  2. The amount of deleted history does not explain the failure: reset-all is less damaging than reset-v in the four optimizers of Table 2, although it removes more state. At GaLore-60M, the final-perplexity ratios are 1.36× for reset-all and 11.82× for reset-v.
  3. Correcting the age does not restore the deleted history: the global-age and local-age branches erase exactly the same history of vv and retain exactly the same mm. Restarting the age repairs the normalization, but it does not give back to the denominator the missing history for gradients still represented in mm. Proposition 2 shows that the worst-case history-compatibility factor remains unbounded in both branches. Yet this reconstruction recovers carry-level quality up to 1B. That result does not simply follow from Adam's correction formula.
  4. This correction does not make reconstruction safe from every state: in the configuration we study, once excessive updates have changed the attention and gradient regime, the same correctly calibrated reconstruction can become hazardous again.

The intended contribution is thus an identification and mechanism result about optimizer-state reconstruction, not a new optimizer mechanism.

2. None of the four methods performs this reset natively

No, and we now say so directly. The selective reset-v branch is not meant to reproduce any native implementation of GaLore, LDAdam, PLUMAGE or Fira, and we do not claim that these methods contain this defect. It is a controlled intervention, designed to answer a narrower causal question:

When training fails after vv is removed, does the failure establish that its historical contents were necessary, or can it come from how the replacement statistic is normalized?

The global-age and local-age branches hold fixed the native transition, the retained mm, the deleted history of vv, the refresh times, the hyperparameters, the data and the learning-rate schedule. They differ only in the correction age assigned to the rebuilt vv. This is what makes the history-loss question identifiable. The revised paper calls reset-v a diagnostic ablation, not a proposed training rule or a failure of a native optimizer.

AuthorsOpenReview ↗

Response to Reviewer R8Pj (2/3)

3. But this age mismatch does exist in published methods

The mismatch that our intervention isolates (moments that have just been rebuilt, but are corrected with an already old counter) does, however, exist in published methods. SPAM lets us test it directly; ReLoRA provides a second published example of the same general pattern.

SPAM (ICLR 2025) does not use a low-rank subspace. In its native pretraining procedure, it periodically resets both Adam moments to zero, keeps the global counter, and applies a warmup after each reset. We tested whether correcting the ages of the rebuilt moments directly can replace this compensating warmup.

Comparisons 1→2, 2→3 and 3→4 each change a single factor. Row 5 is our own intervention, the one studied in the paper, applied to SPAM.

ConditionRebuilt momentsCorrection agesPost-reset warmupLearning rateFinal PPL
1. Published recipem,vm, vglobalyes1e-330.6
2. No warmupm,vm, vglobalno1e-336.9
3. Calibrated agesm,vm, vlocalno1e-331.8
4. Calibrated ages, adjusted LRm,vm, vlocalno2e-330.3
5. Our intervention: reset of vv only, calibrated agevvlocalno2e-330.1

All rows: mean over three matched realizations.

Removing the warmup degrades the native recipe (row 2), consistent with the SPAM authors' own ablation. Restarting the ages alone recovers most of that loss with no warmup (row 3), and the rest is recovered at twice the learning rate (row 4), in line with the roughly 1.8× larger updates we measure under the global age, which follow our prediction within 5–10%. Our response to Reviewer aiXW walks through the table step by step.

Row 5 is our intervention: keeping mm and rebuilding only vv with a local age reaches the lowest perplexity in the table (30.1) with no warmup, in all three matched realizations. This is not SPAM's native procedure; the row shows that the mechanism isolated by our intervention also appears outside the low-rank setting.

In the grid reported by the SPAM authors, 1e-3 is the highest learning rate. We additionally evaluated their native recipe at 1.5e-3 and 2e-3: it is optimal at 1e-3 in this extended grid and degrades above.

We do not claim that SPAM implements our selective intervention: SPAM resets both moments. Nor do we present this as a defect that makes SPAM fail; its recipe works. SPAM provides a native, published instance of the broader phenomenon our analysis isolates: young statistics corrected under an advanced counter and compensated by a warmup.

ReLoRA offers a second example of this general pattern. Its official implementation prunes the two Adam moments almost entirely without resetting the step counter, and the method uses a short warmup after each restart. The authors report that a naive reset makes training diverge. We do not claim that ReLoRA performs our reset-v intervention. It does show that rebuilding optimizer state under an already advanced clock, and then compensating with a warmup, is a real practice. We tested it at 60M, at the learning rate given by its authors' rule: without the restart warmup, training diverges at the first reset in all three matched realizations, whereas restarting the step counter of the reset parameters, still without warmup, keeps it stable, at a perplexity equivalent to that of the published recipe (41.10 against 40.97).

The severe GaLore intervention therefore remains an identification experiment, while the SPAM result establishes that this calibration problem exists in a native reset procedure, with a measurable practical consequence.

AuthorsOpenReview ↗

Response to Reviewer R8Pj (3/3)

4. Where the interventions are applied

Yes: in every cross-optimizer experiment, the method first performs its native state transition, and the intervention is applied only afterwards to the resulting state, before the current gradient is accumulated. All matched variants share the initialization, the data order, the learning-rate schedule, the hyperparameters, the initial projector construction and the native transition rule. The local-age and global-age branches differ only in the age used for the rebuilt vv.

More precisely:

  • GaLore performs its native refresh while retaining its projected moments, then the intervention is applied.
  • PLUMAGE performs its native realignment, then the intervention is applied to the realigned state. The local-age result therefore directly tests whether the history of vv realigned by PLUMAGE is empirically necessary in the tested runs.
  • Fira performs its projector refresh while retaining its residual path; that path and all native mechanisms remain unchanged across interventions.
  • LDAdam first applies its native transport and error feedback. Because LDAdam refreshes at every update, the asymmetry experiment applies the interventions at every update, whereas the check of the age mechanism uses a single reset at update 2,000, so that vv can rebuild. We do not present this second protocol as an alternative to LDAdam's native carry.

Appendix A and the description of the experimental setup have been extended to make this ordering explicit.

5. What we claim, and what we do not

Two statements are now kept apart in the paper:

  • the catastrophic global-age results characterize the added diagnostic reset protocol, not a defect of the native GaLore, LDAdam, PLUMAGE or Fira algorithms;
  • because the interventions are applied after each method's native transition, the local-age experiments do provide a method-specific empirical statement: the complete history of vv produced by that native transition can be discarded, in the tested configurations, without reducing final quality.

In short, we fully accept the distinction you raise: our global-age reset-v is a diagnostic ablation, not a failure of the four native methods. The empirical conclusions nonetheless remain substantial. In the tested configurations, the entire history of vv can be deleted at every refresh while recovering carry-level quality up to 1B; deleting mm as well is far less damaging than deleting vv alone in the configurations of Table 2; and SPAM shows that the mismatch between the actual age of rebuilt statistics and the counter used to correct them exists in a published procedure. In our SPAM experiment, correcting this age directly removes the need for the warmup without degrading final quality.

Reviewer_ytebOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at refresh-based low-rank optimizers such as GaLore, where the subspace Adam works in is periodically replaced, and asks what has to happen to the optimizer state at a refresh. The authors show that the standard way of testing whether the second-moment history matters, resetting v and watching whether training degrades, is confounded: since the bias-correction step count is not restarted, the first update after a reset is far too large, so the failure says little about the history itself. If the rebuilt v is corrected for its own age instead, discarding the history at every refresh costs essentially nothing in the settings tested. Interestingly, resetting both moments is much less harmful than resetting v alone. The second part of the paper shows the calibration is not always safe: in some configurations training still degrades badly, and the authors trace one such case to a feedback loop involving a few attention heads in the first layer. The validation covers four optimizers, scales up to 1B parameters, and matched seeds.

Critical strengths
  1. The confound is clean and exactly checkable, and the asymmetry is a real surprise. The amplification factor is closed-form, it follows from the two ages, and the matched-state ratios reproduce it. But the stronger result is empirical: reset-all is much less damaging than reset-v alone (1.36x vs 11.82x at GaLore-60M, and the ordering holds in all four optimizers), which the usual "the history was lost" explanation cannot account for. In my view this is the paper's best result, and the abstract buries it.

  2. The scale and the matching. The main claim is demonstrated at 1B parameters on a 100k-step schedule, three realizations sharing initialization, data order and schedule, with the 1B local-age endpoints within 0.6% of carry. Most analysis work in this area stops at 130M. The mechanism study is also more careful than usual: several different interventions pointing the same way, rather than one correlation.

Critical weaknesses
  1. The paper corrects an ablation, it does not propose a method, and it is not clear anybody made the mistake being corrected. The authors are honest about the first part (Section 1): local-age reset matches carry in memory and in quality, so there is no practical reason to adopt it over carrying state. The paper's value is then the warning, plus the mechanism. the warning is the weaker leg. Line 63 says reset ablations are "commonly justified by removing state and observing whether training degrades", without a citation. Of the cited methods papers I checked (GaLore, LDAdam, PLUMAGE), none contains an erase-v / keep-m / global-clock ablation or draws a necessity conclusion from one; I did not check Fira, COAP, Flora, Q-GaLore or SubTrack. The one real reset failure cited, ReLoRA diverging after a "naive optimizer reset" and being fixed by a jagged learning-rate schedule, is mentioned only in passing (lines 107-108) and never connected to the calibration story. ReLoRA prunes 99% of both moments, so it is the paper's reset-all case, and if its step counter is not restarted, eq. 2 predicts a sizeable post-reset shock, which is a natural explanation for the divergence ReLoRA reports. As written, Sections 4-5 correct a mistake the paper has not shown anyone made. This is the point that most determines my assessment.

  2. The mechanism is shown in one narrow setting and may be configuration-specific. All the mechanistic evidence is GaLore-60M at (beta2, T) = (0.999, 200), layer-0 Q/K coordinates, three heads selected by an energy rule, three seeds. Nothing mechanistic is reported at 130M or 1B, and the 1B global-age failure (64x) is not shown to share the route. The training configuration (Table 5: weight decay 0, gradient clipping 0, no QK-norm) is one known to be susceptible to attention-logit growth; Wortsman et al. (2024), which the paper cites, show QK-layernorm removes that growth. Whether the Section 7 regime exists with weight decay, clipping or QK-norm is untested, which limits how far the "state-dependence" conclusion generalizes.

Smaller points, none decisive on its own:

  • The consistent local-age advantage is left uninterpreted. Local-age ends below carry in 12 of 14 matched pairs (all nine GaLore pairs, all three PLUMAGE pairs; Fira is the exception) and the paper says "we do not interpret this gap". given the direct matched-state measurement (lines 757-759) puts local-age's first post-refresh step at 1.23x carry's, i.e. roughly step-size neutral, the gap is not a step-size artefact and deserves at least a hypothesis, e.g. along the lines of the Thomas (2026) result the paper cites on down-weighting old gradients.
  • The reset sensitivity R (Tables 4 and 10) is defined against the"own" or"replay" run's own next step, which at a refresh boundary is itself a globally-aged, roughly 2.3x inflated update (line 321). Readers will misread R = 0.25 and 0.16 as "calibrated reset shrinks the update relative to carry". It does not. The paper should say so where R is introduced.
  • The headline gaps are small next to the disclosed noise. Carry for realization 42 is 34.57 in Table 6 and 35.74 in Table 9, a 3.4% difference attributed to hardware alone, while the carry-vs-local-age gap called "carry-level" is 1-1.5%. Paired same-hardware comparisons survive, but the equivalence statements should be framed against this noise floor. Also, "the local-age endpoints lie slightly below carry" (line 160) is inaccurate for Fira, where local-age is slightly worse on both seeds (table 6).
  • The T grid {20, 50, 52, 53, 55, 75, 100, 200} is irregular and unexplained. The 50-55 cluster sits far from the severe/non-severe transition (between 75 and 100 at beta2 = 0.998), so it does not look like boundary refinement. Partial retention (table 9) uses a single realization on different hardware from the rest of the paper. And PLUMAGE realization 123: the carry-family runs (carry 42.40, reset-m 42.35, reset-all 58.20) are all shifted up while local-age (33.52) is not. This is disclosed but not investigated; if carry became unstable on that seed and local-age escaped it, that is relevant to the uninterpreted gap above, not just a footnote.
  • The cross-optimizer claims are GaLore-flavored in magnitude. Reset-all is 4.4x and 4.6x worse than baseline for LDAdam and fira (table 2), against 1.4x for GaLore. LDAdam appears only in a single-reset check (Table 8); the paper is explicit that this is a mechanism check and not a carry alternative, so it is not an overclaim, but LDAdam then adds little.
  • Presentation. The abstract leads with 64x, the failure of the confounded ablation the paper argues against. The informative numbers are the 1B local-age gap and the reset-all/reset-v asymmetry. The text is dense with coined terms (OWN, REPLAY, R, Psi, "carry-level", "local v age") and heavy hedging, which will cost it readers outside the subspace-optimizer niche. The "within 0.1%" agreement should be presented as what the paper itself calls it, an implementation check of an algebraic identity (eq. 9), not as evidence for a hypothesis.
Questions for authors
  1. Can you name a prior paper that drew a history-necessity conclusion from a global-clock reset ablation? If not, would you reframe Sections 4-5 around ReLoRA: is its divergence after a naive reset explained by eq. 1, and does local-age correction remove the need for ReLoRA's jagged schedule?
  2. How was the T grid chosen? Were 52, 53 and 55 added after observing 50 and 75?
  3. Local-age ends below carry in 12 of 14 matched pairs, while its first post-refresh step is 1.23x carry's. Do you have a hypothesis, for example along the lines of Thomas (2026), for why discarding the v history slightly helps?
  4. Do the global-age failures at 130M and 1B show the same signatures (attention entropy collapse, large Psi, high Top-5)? even a single checkpoint diagnostic would help.
  5. Does the section 7 regime appear with nonzero weight decay, gradient clipping, or QK-norm?
  6. Of the severe runs in figure 2, what fraction show attention collapse in layer 0?
  7. For PLUMAGE realization 123, has the carry run been repeated on the same hardware as the rest?
4 comments
AuthorsOpenReview ↗

Response to Reviewer yteb (1/3)

Thank you for this careful and constructive review. You identified the reset-all/reset-v asymmetry as the paper's best result more clearly than our abstract did: the revised abstract now opens with this asymmetry and with the matched 1B result. Changes are marked in blue in the revised version.

In brief: the revision adds three pieces of evidence prompted by your review. (i) SPAM and ReLoRA show the calibration problem in published reset procedures, with a measurable consequence. (ii) At 1B, attention concentration precedes the sustained loss departure, and the degraded state shows the Ψ and Top-5 signatures. (iii) QK-norm, and a configuration combining weight decay with gradient clipping, separate the age-induced shock, which remains, from the catastrophic loss runaway, which no longer develops in the tested configurations.

We first address the two points that weighed most in your assessment, then your other questions.

1. Whether the problem exists in a published procedure (Q1):

We cannot name a prior paper that draws a history-necessity conclusion from a global-clock reset ablation, and we have removed the sentence at line 63. The global-age reset-v is a diagnostic ablation, not a reproduction of a published practice.

The mismatch this ablation isolates (moments that have just been rebuilt, corrected with an already old counter) does exist in two published procedures, which complement each other: ReLoRA, which you suggest, shows it in a low-rank procedure, and SPAM gives the cleanest test. This mismatch is therefore not specific to a subspace refresh. We start with SPAM: it periodically resets both Adam moments to zero, keeps the global step counter, and applies a warmup after each reset, with no subspace change and no parameter reinitialization. This is the reset-all case. We tested it directly at 60M:

ConditionRebuilt momentsCorrection agesPost-reset warmupLearning rateFinal PPL
1. Published recipem,vm, vglobalyes1e-330.6
2. No warmupm,vm, vglobalno1e-336.9
3. Calibrated agesm,vm, vlocalno1e-331.8
4. Calibrated ages, adjusted LRm,vm, vlocalno2e-330.3
5. Our intervention: reset of vv only, calibrated agevvlocalno2e-330.1

All rows: mean over three matched realizations.

Removing the warmup degrades the native recipe (row 2), consistent with the SPAM authors' own ablation. Restarting the ages alone recovers most of the observed perplexity degradation, with no warmup (row 3). The remaining gap closes once the learning rate is adjusted (row 4), in line with the roughly 1.8× larger updates we measure under the global age, which follow our prediction within 5–10%. The native recipe, for its part, is best at 1e-3 and degrades at 1.5e-3 and 2e-3. Row 5 is our intervention, the one studied in the paper: keep mm and rebuild only vv, with a local age. It reaches the lowest perplexity in the table, with no warmup, in all three realizations.

For SPAM, correcting the ages thus removes the need for the post-reset warmup in this configuration (30.3 with no warmup, at the adjusted learning rate, against 30.6 with it).

The ReLoRA test you suggested gives the same answer: at 2e-3, removing the restart warmup makes training diverge at the first reset in all three matched realizations, whereas restarting the ages alone keeps it stable, at a perplexity equivalent to that of the published recipe. Details are in the next comment.

AuthorsOpenReview ↗

Response to Reviewer yteb (2/3)

1. Whether the problem exists in a published procedure (Q1), continued:

ReLoRA: as for SPAM, correcting the ages removes the need for the restart warmup. We tested ReLoRA at 60M with the schedule and reset period of the official code, at the learning rate given by the authors' rule for this model size (2e-3), then at 4e-3. Final perplexities:

2e-34e-3
Published recipe (restart warmup)40.97diverges (561)
No restart warmupdiverges (68 to 430)diverges (779)
No restart warmup, local ages41.1040.95

2e-3: mean over three matched realizations (range for the diverged runs). 4e-3: one realization.

At 2e-3, without the restart warmup, training diverges at the first reset in all three realizations, which reproduces the authors' own ablation (their Table 6, at 130M). Restarting the step counter of the reset parameters is enough to avoid it, still without warmup, at a perplexity equivalent to that of the published recipe: in this 60M test, the divergence of the naive reset is thus explained by the age mismatch. At 4e-3 (one realization), the published recipe diverges despite its warmup, and only the local-age run remains stable. The mismatch can be measured in the published recipe itself: over the 300 updates that follow a reset, its updates are 2.2 to 2.3 times larger than with local ages, for 2.3 predicted by Eq. 2. One clarification: ReLoRA reinitializes its low-rank factors at each reset and prunes 99.9% of both moment buffers, which refer to parameters that no longer exist, while leaving the step counter untouched. We therefore restart the age of both moments: this is the near-joint reset of Eq. 2, not our selective-vv intervention.

Sections 4–5 now present reset-v as an identification experiment, point to SPAM as the native instance, and connect ReLoRA to Eq. 2.

2. How far the mechanism extends (Q4, Q5, Q6):

We added diagnostics at 1B and 130M, tested the safeguards you list, and probed every run of Figure 2.

At 1B (Q4): we re-ran the first 6,000 updates of the global-age reset-v configuration and its matched carry (one realization), saving a checkpoint every 1,000 updates, and probed all 24 layers and 768 attention heads. The reset-v training loss reaches its minimum near update 3,170 and begins a sustained departure around update 3,800.

UpdateMeasurereset-vcarry
2,000Concentrated heads, layer 023 / 320 / 32
2,000Concentrated heads, whole network112 / 76824 / 768
3,000Concentrated heads, layer 032 / 320 / 32
3,000Concentrated heads, whole network410 / 76834 / 768
3,000Mean entropy, layer 00.010.66
5,200Ψ, layer-0 Q/K28.10.39
5,200Top-5, layer-0 Q/K56%9.5%

In the reset-v and carry columns, the "concentrated heads" rows give the number of concentrated heads / total number of heads. Concentrated head: normalized entropy below 0.2.

The attention signature is in place before the sustained departure: at update 3,000, all of layer 0 is concentrated and its Q/K weight norms are 2.3× those of carry, the largest ratio across the 24 layers. The concentration itself is not confined to layer 0 or to a few heads: by update 3,000, at least 25 of 32 heads are concentrated in each of layers 0 to 9. After the departure, at update 5,200, the other two signatures you list are present as well. The 1B failure thus shows the three signatures of your question, and it is consistent with the early-layer attention route of Section 7.

At 130M (Q4): at late checkpoints, all 144 heads have normalized entropy at most 0.002, Top-5 is 54–87% against 25–26% for carry, and layer-0 Q/K norms are 500 to 6,000 times larger than in the matched carry run, against 3 to 10 times in deeper layers. Ψ requires the gradients of the 200 preceding updates, which we recorded only at 1B.

AuthorsOpenReview ↗

Response to Reviewer yteb (3/3)

2. How far the mechanism extends (Q5, Q6), continued:

Safeguards (Q5): we re-ran the 60M reference configuration with QK-norm, and then with weight decay 0.1 and gradient clipping 1.0 enabled together. In both cases, the first update after a refresh remains, in median, about 15 times that of carry, and the final reset-v perplexity stays 3.8 above carry. The loss runaway, however, disappears (final perplexity from 408.9 to 38.8 in both cases), and under QK-norm the local-age run stays at carry level. The two safeguards do not act in the same way: QK-norm suppresses the growth of the logits, whereas weight decay and clipping bound the loss without preventing the concentration (all 8 layer-0 heads). In the safeguarded configurations we tested, the age-induced shock remains, but it no longer develops into a catastrophic loss runaway. Section 7 and the limitations now say so.

Severe runs of Figure 2 (Q6): we probed the final checkpoints of the 192 sweep runs and of their baselines. All 86 severe runs have all 8 layer-0 heads below a normalized entropy of 0.2, whereas none of the 192 baselines has more than 3. Among the 106 non-severe reset-v runs, 81 have at least 4: within this sweep, layer-0 concentration accompanies every severe outcome but is not sufficient to produce one. The threshold does not carry the result: the median layer-0 mean entropy is 0.00 for severe runs, 0.22 for non-severe runs and 0.64 for baselines.

3. The TT grid (Q2):

Yes, 52, 53 and 55 were added afterwards, to examine a benign-looking pocket near T=50T=50, which turned out to depend on the realization. We now say so, and we present Figure 2 as an exploratory susceptibility map. The native values are T=200T=200 and β2=0.999\beta_2=0.999; the GaLore paper reports little difference between T=50T=50 and T=1000T=1000.

4. Why local age ends slightly below carry (Q3):

Our hypothesis is the following. In GaLore, carry keeps a vv accumulated in the coordinates of earlier subspaces, whereas a rebuilt vv reflects only the current subspace. This is in line with Thomas (2026), who finds that a shorter second-moment memory helps refreshing optimizers. It is not the whole explanation: PLUMAGE already realigns its vv and shows the same ordering.

5. PLUMAGE, realization 123 (Q7):

Yes. We repeated carry and local-age on 4 L40S GPUs; both runs reproduce the reported values exactly (42.40 and 33.52), with identical losses over the 10,000 updates. The gap is therefore not an execution artifact. Carry destabilizes at update 1,015, fifteen updates after the refresh at update 1,000 and at the end of the learning-rate warmup: the update norm rises from 0.11 to 1.15, the loss from 4.5 to 10.3, and the run does not recover. Local-age goes through the same refresh without incident. As you suggest, this is relevant to the gap discussed in point 4: on this trajectory, carry becomes unstable after a refresh and local-age escapes it.

6. Smaller points:

We have incorporated your other remarks. RR is now defined as a comparison with the next update of the same run, not with carry. Equivalence statements are framed as paired, same-hardware comparisons. The sentence at line 160 is corrected for Fira, Table 9 is flagged as a single realization on different hardware, cross-optimizer magnitudes are reported method by method, and the 0.1% agreement is described as an implementation check. On presentation, we trimmed the hedging, now gathered in the limitations section, and defined OWN, REPLAY, RR and Ψ at first use.

Thank you again: your questions led directly to the SPAM and ReLoRA experiments, the 1B diagnostics and the sweep-wide attention probe. We hope these additions address the two points that weighed most in your assessment, while making the scope of the mechanism more precise.

AuthorsOpenReview ↗

Follow-up to Reviewer yteb

Thank you again for your review and for suggesting the ReLoRA test. If anything in our response or in the revised paper is unclear, we would be glad to clarify it or to answer any other question.

Reviewer_aiXWOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper discusses an issue with low-rank optimizers where they investigate what happens with a subspace refresh. The authors show that test for the second moment history is confounded with other variables, making them inaccurate as is. The authors validate their approach over four optimizers with models up to 1B parameters.

Critical strengths

The main argument presented by the authors, that there is a confounding variable makes sense and easily checked.

The authors also show this arises in models up to 1B in parameters, showing some scaling can be useful

Critical weaknesses
  1. One of the main issues is that this work is its presentation. It is difficult to read or really tease out what exactly the authors are proposing or what their experiments are meant to show. For example, Table 2 is only referenced in the writing in Appendix B. Table 4, which contains 7 different rows, is only referenced as evidence to support a single point in the main writing. It also isn't completely clear what this "reference run" refers to exactly

  2. This work does not appear to propose a method per se, but rather a minor correction to what seems like a narrow application. This is exemplified by the fact that they do not have a traditional method section, but rather begin with their experimental setup. The authors should be clearer about their claims and contributions

  3. The authors do not show that any of these issues provide meaningful differences in the final representation. Section 6's title seems to be aimed at getting to this point, but Figure 2, which seems to be what they rely on, but the authors provide little guidance on the justification for the hyperparameters they test (i.e., examples of cases where they are used).

  4. The related work section is narrow, missing many related works, e.g., [A,B,C,D]. Note that this is a non-exhaustive list, but simply what a few minute search comes up with.

[A] GoRA: Gradient-driven Adaptive Low Rank Adaptation. NeurIPS 2025

[B] Unbiased Gradient Low-Rank Projection, 2025

[C] Low Rank Gradients and Where to Find Them. NeurIPS 2025

[D] Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-tuning. AAAI 2026

In summary- this paper was difficult to follow and requires significant revision to be ready to be published at ICLR 2027. The discussion of related work should be more extensive, and the experimental setup needs to be better justified. I would not be opposed to raising my score based on the author response and the other reviewer's comments, but it would require significant changes.

Questions for authors

see weaknesses

6 comments
AuthorsOpenReview ↗

Response to Reviewer aiXW (1/3)

Thank you for your careful reading and precise remarks. You are right: the scientific question and the role of each experiment were not visible enough in the submitted version. We restructured the paper, broadened the related work, and added an experiment on a published reset procedure. Changes are marked in blue in the revised version.

In brief: restarting Adam's bias-correction age after a reset is standard, and we use it only as a control. What is new is that, in the tested settings, the entire history of vv can be deleted at every refresh with no loss of final quality, up to 1B parameters; that erasing both moments is far less damaging than erasing vv alone in all four optimizers we test; and that SPAM, a published optimizer, shows the same age mismatch in its native reset procedure.

1. Presentation and organization of the experiments

We corrected the three presentation problems you identified.

  • Table 2 is now introduced and interpreted in Section 4, where the asymmetry between reset-all (both Adam moments reset to zero) and reset-v (the second moment only) is established.
  • Table 4 is now discussed diagnostic by diagnostic: the text explains the role of each row and the contrasts between its columns, then ties these results to the interventions of Section 7.
  • "Reference run" is defined at first use: GaLore-60M, realization (seed) 123, with a global-age reset-v at (β2,T)=(0.999,200)(\beta_2, T) = (0.999, 200), used for the mechanistic study of Section 7.

We also added a roadmap before the main experiments:

  • Table 2 establishes the asymmetry between interventions;
  • Table 3 tests whether the history of vv is needed, up to 1B;
  • Figure 2 maps the conditions under which a repeated perturbation stays tolerable or becomes persistent;
  • Table 4 and the interventions of Section 7 test the proposed mechanism.

2. Nature of the contribution

We indeed propose no new optimizer. The intended contribution is an empirical and mechanistic study about optimizer-state reconstruction: the paper asks what information must survive a change of representation, and when rebuilt optimizer state remains safe. As you point out, the submitted version did not say this clearly enough; the revision does.

The starting idea is simple: when an optimizer erases a statistic and rebuilds it, two things change at once. Its history is lost, and the statistic becomes "young" again. Adam's bias correction depends on that age: the number of steps over which the statistic has been accumulated. If vv is erased while the step counter keeps running from the start of training (global age), Adam treats a brand-new statistic as an old one. If steps are counted from the reset (local age), the correction is matched to what the rebuilt statistic has actually accumulated.

Restarting this age after a reset is simply what Adam's bias correction prescribes; keeping the global age, as our reset-v branch does on purpose, is the miscalibrated choice. We do not present this restart as a contribution: it is our experimental control. The two branches erase exactly the same history, and comparing them is what lets us tell the loss of history apart from the miscalibration of the rebuilt statistic. Three results follow.

  1. Necessity of history: In the tested configurations, the entire pre-refresh history of vv can be discarded at every refresh while recovering the quality obtained by keeping the state (which we call carry). At 1B, the three matched 100,000-step runs end within 0.6% of carry: in the tested settings, this history is not needed to recover carry-level final quality.
  2. Separating history loss from calibration: Two branches erase exactly the same pre-refresh history of vv, but produce radically different trajectories depending on the age used to correct the rebuilt statistic. A second observation points the same way: in the configurations of Table 2, reset-all is less damaging than reset-v in all four optimizers, although it removes more state. At GaLore-60M, the final-perplexity ratios are 1.36× for reset-all and 11.82× for reset-v. In other words, the amount of deleted information alone does not explain the damage; the interaction between the first moment, which is kept, and the second moment, which is rebuilt, is decisive.
  3. Dependence on the state reached: In the configuration we study, complementary interventions on update magnitude, attention logits and projector sequences identify a route by which initially excessive updates transform the attention and gradient regime, from which even a correctly calibrated reconstruction can become hazardous. Whether a reset is safe therefore also depends on the state training has reached.

The question, the interventions and what each controls now come before the experimental details.

AuthorsOpenReview ↗

Response to Reviewer aiXW (2/3)

3. Practical scope beyond subspace changes

To address your concern that the application may be narrow, we added an experiment on SPAM, an optimizer published at ICLR 2025 that does not rely on a low-rank subspace.

Native procedure: In its pretraining configuration, SPAM periodically resets both Adam moments to zero and applies a warmup after each reset. The moments are therefore rebuilt from zero, while the counter used for their bias correction remains global. This is exactly the situation described above: young statistics corrected as if they were old. Our analysis predicts that this age mismatch amplifies the updates relative to a reconstruction that uses the actual age of the rebuilt statistics.

Question tested: To what extent does the post-reset warmup compensate for this mismatch? If calibration explains a large part of it, correcting the ages of the two moments directly should reduce or remove the need for a warmup.

The table reads from top to bottom: comparisons 1→2, 2→3 and 3→4 each change a single factor, the warmup, then the correction ages, then the learning rate. Row 5 is our own intervention, the one studied in the paper, applied to SPAM: keep mm and rebuild only vv, with a local age.

ConditionRebuilt momentsCorrection agesPost-reset warmupLearning rateFinal PPL
1. Published recipem,vm, vglobalyes1e-330.6
2. No warmupm,vm, vglobalno1e-336.9
3. Calibrated agesm,vm, vlocalno1e-331.8
4. Calibrated ages, adjusted LRm,vm, vlocalno2e-330.3
5. Our intervention: reset of vv only, calibrated agevvlocalno2e-330.1

All rows: mean over three matched realizations.

  • From 1 to 2, removing the warmup strongly degrades the native recipe: perplexity goes from 30.6 to 36.9, consistent with the ablation reported by the SPAM authors. So the native recipe does depend on its warmup.
  • From 2 to 3, changing only the correction ages of the two rebuilt moments recovers most of the observed perplexity degradation, with no warmup: final perplexity improves from 36.9 to 31.8. This shows that the age mismatch accounts for a large part of the degradation observed when the warmup is removed.
  • From 3 to 4, the remaining gap is recovered in the tested configuration by adjusting the learning rate, consistent with the measured difference in effective update scale. Under the global age, the global/local ratio of update norms is about 1.8 on average over the cycle and follows the analytical prediction within 5–10%; the published learning rate was selected under these native dynamics, which include this difference in scale. At 2e-3, the calibrated variant reaches 30.3 with no warmup, on par with the published recipe.
  • Row 5, our intervention: keeping mm and rebuilding only vv with a local age, exactly as in the rest of our paper, reaches the lowest perplexity in the table (30.1) with no warmup, with the same ordering in all three matched realizations. This is not SPAM's native procedure: this row shows that the mechanism isolated by our intervention also appears outside the low-rank setting.

In the grid reported by the SPAM authors, 1e-3 is the highest learning rate. We additionally evaluated their native recipe at 1.5e-3 and 2e-3: it is optimal at 1e-3 in this extended grid and degrades above. The calibrated variants were evaluated at 1e-3 and 2e-3 (row 5 also at 1.5e-3); 2e-3 is the best value in both cases.

These results show that the problem we study is not specific to subspace changes: a published reset procedure does rebuild moments under a global counter, and calibrating their ages directly can, in this configuration, replace the compensating warmup with no loss in final quality. ReLoRA, a published low-rank training method, gives the same result: at the learning rate given by its authors' rule, removing its restart warmup makes training diverge in all three matched realizations, whereas restarting the ages alone keeps it stable, at a perplexity equivalent to that of the published recipe (41.10 against 40.97). The full protocol, hyperparameters and per-realization results are in a new appendix.

AuthorsOpenReview ↗

Response to Reviewer aiXW (3/3)

4. Effects on the final result, and justification of Figure 2

We understand your remark as asking whether the effect persists up to the final trained model, rather than being limited to the first update after the reset. Tables 2–3 and Figure 2 report final validation perplexity after complete training runs. The 9.5× to 35× gaps in Table 2 are endpoint differences, not transient one-step spikes: a ratio of 11.82× means that the trained model has a perplexity nearly twelve times higher than its matched baseline. For GaLore, repeated local-age reconstruction stays within 1.5% of carry at 60M and 130M, and within 0.6% at 1B.

Section 7 separately shows that the perturbation can durably change the state reached by training: attention concentration, gradient intermittency, and mismatch between the retained momentum and the rebuilt denominator. In particular, a temporary protection of the update magnitude remains effective well after it is removed.

You are right that the grid of Figure 2 needed justification. We now present it as an exploratory susceptibility map. Its two axes follow from the analysis:

  • β2\beta_2 sets the memory length of the second moment (on the order of 1/(1−β2)1/(1-\beta_2) steps), hence the initial size and the duration of the reconstruction shock;
  • TT is the interval between two reconstructions, hence the time a shock has to dissipate before the next one.

The sweep includes the native values of the evaluated methods (β2=0.999\beta_2 = 0.999 for GaLore, PLUMAGE and Fira; 0.990.99 for LDAdam). T=200T = 200 is GaLore's pretraining configuration, and its authors report that values between 50 and 1000 make little difference to their results. Intermediate values of β2\beta_2 are there to resolve the transition between regimes.

5. Related work

Thank you for references [A–D]. We added them and restructured the section, rather than only extending the list of citations. It now distinguishes: subspace selection and evolution; transport, retention or realignment of optimizer state; biases introduced by low-rank projections; low-rank structure of gradients; reset or reconstruction of optimizer state; and magnitude control or warmup after reconstruction.

[A] addresses rank allocation and initialization in low-rank adaptation; [C] the structural origin of low-rank gradients; and [D] a low-rank approximation of curvature for zeroth-order optimization. [B] is particularly relevant context: it restarts momentum at each projection period, which shows that handling optimizer state is an active design choice when the representation changes. Our question is complementary to [A–D]: none of them separates statistical age from the deletion of second-moment history, which is what our matched full-training runs up to 1B test.

We hope these substantial revisions address the points behind your initial assessment and make the paper's question, and what each experiment is for, easy to identify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to the reviewers and the Area Chair

We thank the three reviewers for their detailed reports. Two questions come up in all three reports and explain most of the disagreement. What is the contribution beyond Adam's standard correction? And does the problem matter in practice, in an existing procedure? The revision answers both and adds several experiments. Details are in the individual responses, and changes are marked in blue in the revised PDF. To stay within the page limit, we also removed a few sentences that repeated points made elsewhere; no result was removed.

1. What the paper establishes:

The paper does not propose a new optimizer. Its contribution is an identification result and a mechanistic analysis of optimizer-state reconstruction. Restarting the correction age of a rebuilt statistic is simply what Adam's bias correction prescribes. We now say explicitly that it is our experimental control, not our contribution. It lets us tell the loss of history apart from the calibration of the statistic that replaces it, and three results follow:

  • in the tested configurations, the entire history of vv can be discarded at every refresh while recovering the quality obtained by keeping the state: at 1B, the three matched runs end within 0.6% of carry;
  • in the configurations of Table 2, resetting both moments is far less damaging than resetting vv alone, although more state is removed (1.36× against 11.82× at GaLore-60M). The amount of deleted information is therefore not enough to explain the failure;
  • whether a correctly calibrated reconstruction is safe also depends on the state training has reached: Section 7 establishes one such route, replicated on three realizations.

2. The age mismatch exists in published procedures:

Our global-age reset-v is a diagnostic ablation. None of the four methods we evaluate performs it natively, and we do not present its results as a defect of GaLore, LDAdam, PLUMAGE or Fira. The mismatch it isolates (statistics that have just been rebuilt, corrected with an already old counter) does exist in SPAM (ICLR 2025), which periodically resets both moments to zero, keeps the global counter, and applies a warmup after each reset. We tested it at 60M:

ConditionRebuilt momentsCorrection agesPost-reset warmupLearning rateFinal PPL
1. Published recipem, vglobalyes1e-330.6
2. No warmupm, vglobalno1e-336.9
3. Calibrated agesm, vlocalno1e-331.8
4. Calibrated ages, adjusted LRm, vlocalno2e-330.3
5. Our intervention: reset of v only, calibrated agevlocalno2e-330.1

All rows: mean over three matched realizations.

At the same learning rate and with no warmup, restarting the ages alone takes perplexity from 36.9 to 31.8; at the adjusted learning rate, the calibrated variant reaches 30.3 with no warmup, against 30.6 for the published recipe. In this configuration, calibrating the ages can thus replace the warmup with no loss in final quality, and the intervention studied in the paper reaches 30.1 with no warmup (row 5). ReLoRA, a low-rank method, gives the same result: at the learning rate given by its authors' rule, removing its restart warmup makes training diverge in all three matched realizations, whereas restarting the ages alone keeps it stable, at a perplexity equivalent to that of the published recipe (41.10 against 40.97).

3. Other additions in the revision:

  • Mechanism at larger scale (yteb): on a matched 1B trajectory, attention concentration precedes the sustained loss departure, and the degraded state shows the Ψ and Top-5 signatures. Late 130M checkpoints also show attention concentration. The causal interventions remain established at 60M.
  • Safeguards (yteb): with QK-norm, or with weight decay and gradient clipping combined, the age-induced shock remains, but it no longer develops into a catastrophic loss runaway in the tested configurations.
  • Figure 2 sweep (yteb): all 86 severe runs have all 8 layer-0 heads below the entropy threshold, whereas none of the 192 baselines has more than 3. At least 4 heads are also below the threshold in 81 of the 106 non-severe runs: within the sweep, concentration accompanies every severe outcome but is not sufficient to produce one.
  • Presentation (aiXW, yteb, R8Pj): the abstract is rewritten and opens with the asymmetry and the 1B result; a roadmap precedes the experiments; Tables 2 and 4 are discussed in the text; the related work is restructured and includes the suggested references; the order between native transition and intervention is stated for each method.

We hope the revision makes clear what is standard here (restarting the age) and what is new (the three results above), and that the calibration problem has a measurable consequence in a published procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.