acceptodds
Under review as a conference paper at ICLR 2027

PLUMAGE: A Probabilistic Low-rank Unbiased Min-Variance Gradient Estimator for Memory Efficient Large Model Training

Abstract

Accelerator memory and communication bandwidth are major bottlenecks in large-scale training, motivating low-rank gradient estimators (LGEs) that compress both gradients and optimizer states. Existing LGEs trade bias for variance: deterministic top-k estimators are biased, whereas randomized estimators are unbiased but may incur high variance. We propose PLUMAGE, a probabilistic low-rank unbiased minimum-variance gradient estimator for a given rank budget. PLUMAGE uses fixed-rank sampling without replacement, preserving the asymptotic memory footprint (with only k additional scale factors per layer) of prior one-sided low-rank methods. Additionally, we introduce a numerically robust second-moment alignment procedure for alternating subspaces in stateful optimizers. Across pre-training and fine-tuning, PLUMAGE reduces the optimization gap relative to full-rank training by 36% on average versus the top-k LGE used by GALORE, at essentially the same runtime (within ∼ 1%). Moreover, PLUMAGE reaches GALORE’s terminal loss in fewer steps, including a 30% reduction in 1B-scale pre-training while reusing the full-rank learning rate. Beyond the Adam-based LGE setting, PLUMAGE’s subspace-sampling rule also transfers to the Muon-style SUMO optimizer, reducing its perplexity gap to full-rank Muon by  40% at 130M and 350M scales in our experiments.

Primary area:
optimization
Keywords:
optimization, low-rank, gradient-estimator

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gCxVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PLUMAGE proposes a randomized low-rank gradient optimizer, that given the current exact singular basis of a gradient matrix, deterministically retains some leading components and samples the remaining components without replacement using inclusion probabilities proportional to their singular values, yielding a conditionally unbiased estimator. The paper also uses first- and second-moment realignment rules for Adam moments will get misaligned after changing subspaces.

Critical strengths

1- The paper clearly identifies the truncation bias introduced by top-k gradient projection and proposes a same budget alternative while preserving the dominant directions.

2- The authors explicitly state that fixed-target conditional unbiasedness an a specific iteration does not imply trajectory-level unbiasedness after a random projector is reused.

3- The experiments include pretraining, GLUE fine-tuning, and 8B supervised fine-tuning. Also state-realignment ablations, a full-rank lm_head control, and a transfer experiment with SUMO is meaningful. Overall, paper tries to explain source of differences via structured experiments.

Critical weaknesses

1- The proposed method still pays the O(m2n)O(m^2n) decomposition cost as it needs the full decomposition of the batch gradient, which leads to higher computation time compared to more efficient implementations of GaLore [1]. This is while there are more recent works, like APOLLO [2] or SubTrack++ [3], that are proposed to solve the high computation requirement of SVD. Specifically, APOLLO [2] is a highly relevant baseline to be included as they already outperforms GaLore with randomization, and although they are rescaling the discarded gradient, they provide same memory saving while incurring less computation.

2- As the paper itself acknowledges, the subspace and inclusion probabilities are refreshed every τ iterations, while subsequent gradients depend on earlier updates obtained using the sampled projector; therefore, the practical method is not trajectory-level unbiased, while being unbiased is the main claim of the paper, also stated in the title. A comparison against efficient randomized methods such as APOLLO [2] or simple random projections could help clarify whether the practical gains arise from conditional unbiasedness, stochastic tail coverage, moment realignment, or another aspect of the method, especially as figure 3-b shows small gap between uniform and Plumage sampling.

3- The gap between GaLorempGaLore_{mp} and PlumagempPlumage_{mp} in figure 1-b is very close, and it leads to questioning whether the claim that PLUMAGE reaches GaLore’s terminal loss in fewer steps, is really related to the proposed method, or could be achieved by smaller changes, like realigning the rotated moments in GaLore, as it is done in earlier works like LDAdam [4] or rescaling the gradient residual like done in earlier works [2-5].

4- Since the implementation differs materially from the original GaLore setup, using GaLore’s previously recommended hyperparameters may not guarantee that it is near its optimum in the new setup. A matched sweep for both methods would make the comparison more conclusive.

5- Tail components with small singular values can receive small inclusion probabilities and therefore large 1/pi1/p_i scaling when selected. It would strengthen the paper to report distributions of minimum inclusion probabilities and inverse scales or number of spikes solved by gradient clipping across layers and training. This is especially important as r∗r^* differs per layer.

6- As the learning dynamics can change in larger models, experiments on models with 7B+ parameters are very much helpful and appreciated.


References

[1] Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML 2024.

[2] Zhu, H., Zhang, Z., Cong, W., Liu, X., Park, S., Chandra, V., Long, B., Pan, D. Z., Wang, Z., and Lee, J. APOLLO: SGD-like Memory, AdamW-level Performance. MLSys 2025.

[3] Rajabi, S., Nonta, N., and Rambhatla, S. SubTrack++: Gradient Subspace Tracking for Scalable LLM Training. 2025.

[4] Robert, T., Safaryan, M., Modoranu, I.-V., and Alistarh, D. LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. ICLR 2025.

[5] Chen, X., Feng, K., Li, C., Lai, X., Yue, X., Yuan, Y., and Wang, G. FiRA: Can We Achieve Full-Rank Training of LLMs under Low-Rank Constraint? 2024.

Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Authors Response Part 1/2

We thank the reviewer for valuing structured experiments; their question about where gains arise shaped our controls (⋆ = new).

W1. SVD cost; APOLLO, SubTrack++. PLUMAGE needs the full spectrum at each refresh: O(m2n)O(m^2n) for an m×nm\times n gradient against O(mnk)O(mnk) for top-kk. As our rank grows with the width (m/km/k is 3–4), the two are of the same order, and top-kk is cheaper by a constant factor, not at a faster rate. The paper claims no cost parity (§3.4), and all reported runs, GaLore's included, use the same dense SVD. With optimized SVD backends (⋆) and the refresh amortized over τ=200\tau=200 steps, a PLUMAGE step costs 9.0% / 1.0% more than full-rank Adam at 350M / 1B, against 7.2% / 0.5% for GaLore's fastest variant, at equal memory.

APOLLO is a memory-efficient, SVD-free optimizer, not an estimator: it rescales the full-rank gradient by a low-rank version of Adam's rule in a random projection instead of top-kk (§2.2). Its Table 2 puts most of the gain in the full-rank path: at 350M its SVD variant gives a perplexity of 16.67 against 16.85 with the random projection, while Fira's full-rank residual takes GaLore from 18.95 to 17.03. PLUMAGE works with any optimizer, but it matters where the low-rank gradient determines the update direction, as in the widely used Adam and Muon (SUMO). In APOLLO, the projection estimates only Adam's scaling factors, so a gap between PLUMAGE and APOLLO measures the full-rank update, not the estimator, and APOLLO cannot serve as the control W2 asks for; we add no such run. SubTrack++ replaces the SVD with rank-1 tracking of a deterministic subspace; the revision discusses it in §2.2.

W2. Unbiasedness, the title, source of the gains. The title's claim concerns the estimator, unbiased by construction at every refresh; as the abstract states, it does not extend to the trajectory, since reusing a sample breaks it for any randomized estimator (§3.4). We claim no unbiased update. The gain comes from which directions are updated: GaLore's top-kk discards the tail directions at every refresh, so its error accumulates there. In contrast, a new unbiased projection at each refresh decorrelates successive errors.

Possible sourceWhat the experiments show
Moment realignment (S/MP: both of Adam's moments realigned at a refresh)⋆ GaLore with / without S/MP: perplexity 30.22±0.06 / 30.26±0.05 at 130M, 23.32±0.02 / 23.30±0.01 at 350M: no gain. PLUMAGE with/without S/MP: 28.73 / 29.57±0.03 (130M); with S/MP: 21.82 (350M).
Stochastic tail coverage: uniform exact-kk sampling in place of top-kk, equal in expected error to a rescaled random projection (Table 8)In SUMO, a Muon-style optimizer: perplexity falls from 30.32 to 28.67 at 130M and, at 350M (single runs), from 23.75 to 22.32: about 80% of the way to PLUMAGE (next row). FLORA, a random projection, has a mean GLUE score of 84.32 against PLUMAGE's 85.66 (Table 2).
PLUMAGE's probabilities (Eq. 7) in place of uniformSUMO: further to 28.29 and 21.91, the remaining 20% or so. Adam (Fig. 3b): small mean gap, but PLUMAGE's spread over seeds is 9× smaller.
The 1/pi1/p_i scaleNot a source under Adam (its normalization cancels it up to ϵ\epsilon) or SUMO (which omits it); there the gain comes from the sampled directions. It acts under SGD with momentum (W3).

W3. Fewer steps; smaller changes. In Fig. 1b (SGD with momentum, where the 1/pi1/p_i-scaled estimator enters the update linearly) the curves separate late: terminal evaluation loss is 5.05–5.07 for GaLore, 4.83–4.86 for PLUMAGE and 4.79–4.83 for full rank (2 runs each, 3rd seed pending; values in the revised caption). The fewer-steps claim rests on the Adam runs (§4.3). Against GaLore's new 350M runs (the submitted run reset Adam's moments at each refresh), each PLUMAGE run reaches GaLore's terminal loss after 65–66% of the steps (interpolated; was 50%). Smaller changes do not achieve this: realigning GaLore's moments (S/MP) does not change its result (table under W2), and with a Fira-style residual already present (SUMO), replacing top-kk by PLUMAGE still closes 40.8% of the gap to full-rank Muon at 130M.

W4. Hyperparameters. GaLore's recommended learning rate, 10−210^{-2}, is the best of a new sweep at 130M (⋆): perplexity 30.51 / 30.26±0.05 / 30.32 at 0.005 / 0.01 / 0.02 (single seeds except at 0.01). PLUMAGE reuses full-rank Adam's 10−310^{-3}, near the best of PLUMAGE's own sweep (App. I.2); the larger scales reuse both learning rates. We also ran both methods in GaLore's reference code, which reproduces its published 350M result. With lm_head also low-rank there (single runs; Table 12), PLUMAGE closes 65% of GaLore's gap to full rank, against 35% in our setup. On GLUE, the sweep is matched: the learning rate is tuned per task and method (App. I.3).

W5 and W6 follow in a second comment.

AuthorsOpenReview ↗

Authors Response Part 2/2

Reply to Reviewer gCxV, continued (⋆ = new).

W5. pip_i and 1/pi1/p_i. Under Eq. 7, every sampled tail component is rescaled to one magnitude at a refresh, σi/pi=c≤σr⋆\sigma_i/p_i=c\le\sigma_{r^\star}, where σr⋆\sigma_{r^\star} is the smallest singular value that is always kept. The 1/pi1/p_i scale is applied before Adam's moments (the submitted Eq. 18 and Alg. 1 wrote it at the back-projection), and Adam's normalization absorbs it: the step is Adam's with ϵ=10−6\epsilon=10^{-6} replaced by piϵp_i\epsilon, which matters only where V\sqrt V is near zero. Statistics over layers and training (⋆):

Median over layers130M350M
smallest pip_i (worst layer)3.3⋅10−43.3\cdot10^{-4} (4.6⋅10−64.6\cdot10^{-6})1.7⋅10−41.7\cdot10^{-4} (6.2⋅10−76.2\cdot10^{-7})
largest sampled 1/pi1/p_i (share of layers above 100)25 (7%)37 (17%), stable over training
r⋆/kr^\star/k (share of the rank always kept)0.250.15

Empirically, there is no instability relative to GaLore, with no gradient clipping: over five 350M seeds per method, the largest rise of the training loss in any 2,000-step window after step 2,000 is 0.010–0.014 for PLUMAGE and 0.010–0.012 for GaLore.

W6. Larger-scale results. 7B+ pre-training is beyond our compute. PLUMAGE stays ahead at every scale we trained, and in Llama-3.1-8B fine-tuning, it lowers GaLore's training loss from 1.05 to 0.97 (Table 3).

Reviewer_qpshOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PLUMAGE, which replaces deterministic top-k gradient projection with importance sampling of singular components. For a fixed gradient and its current exact singular basis, the estimator is unbiased and minimizes Frobenius estimation variance within the stated comparison class. The paper also proposes second-moment realignment and evaluates the method in language-model pretraining, fine-tuning, and a SUMO-based Muon setting.

Critical strengths
  1. The design of the method originates from a good theoretical motivation. Minimizing gradient-estimation variance while retaining support on tail directions is an interesting approach to improve low-rank optimization.

  2. The experiment settings and baselines are reasonable and the empirical results look promising. The pretraining and fine-tuning experiments report useful improvements over GaLore in loss and perplexity across the tested settings. The paper also clearly states the limits of its unbiased guarantees that only apply to limited cases.

Critical weaknesses
  1. The connection between estimator theory and the practical optimizer remains incomplete. The projected Adam update in Eq. (18) is not generally equivalent to full-space Adam applied to the unbiased reconstructed gradient. Thus, minimum gradient-estimation variance does not directly imply better Adam updates, and it is unknown that whether the variance-minimized operation may introduce further bias in the settings or not, making the method's effectiveness hard to explain. The authors acknowledge this. This is okay for me since it should work for optimizers like Muon, but still somewhat limits the paper's scope.
  2. Closely related work is missing. Golore [1] studies GaLore's possible non-convergence and random subspace selection. GUM [2] directly studies debiasing low-rank optimization with Muon. These papers should be discussed, especially GUM directly study the same unbiased low-rank projection, which is at the same motivation and starting point as the paper.
Questions for authors
  1. Could the authors include the discussion with Golore and GUM?
2 comments
AuthorsOpenReview ↗

Authors Response

We thank the reviewer for crediting the theoretical motivation and the clearly stated limits of the guarantee.

W1. From estimator to optimizer. As the review notes, the projected Adam update is not full-space Adam on the unbiased estimate. The guarantee is on the gradient estimate, and we do not claim an unbiased update: Adam's second moment rescales the step, so its update is biased for any stochastic gradient, full-rank included. The gain comes from which directions are updated: GaLore's top-kk discards the tail directions at every refresh, so its error accumulates there, whereas PLUMAGE draws a new unbiased projection at each refresh, so successive errors decorrelate. The scale D−1D^{-1}, with D=diag(pi)D=\mathrm{diag}(p_i) and pip_i the inclusion probabilities, is applied to the projected gradient before the moments: at each refresh, Adam's first moment receives an unbiased estimate. Applied after the normalization, it would multiply the step of a rarely sampled direction by 1/pi1/p_i; applied before, Adam's normalization absorbs it (here MM and VV are Adam's moments of the unscaled projected gradient),

D−1MD−2V+ϵ=MV+Dϵ,\frac{D^{-1}M}{\sqrt{D^{-2}V}+\epsilon}=\frac{M}{\sqrt{V}+D\epsilon},

exactly up to ϵ\epsilon while the sample is fixed, and approximately across a refresh (derivation in the new App. A.5), so the reweighting adds no further bias to the step. Eqs. 16–18 and Alg. 1 are updated to reflect the implementation (the submitted Eq. 18 wrote the scale at the back-projection). So under Adam the estimator acts through the directions it samples. The minimum-variance result enters the theory only when a new sample is drawn at every step, where it adds to the gradient noise in the SGD rate (Eq. 24); we claim no such bound for projected Adam. On scope, the paper already goes beyond Adam with SUMO, a Muon-style optimizer, where PLUMAGE's selection closes 40.8% of top-kk's gap to full-rank Muon at 130M (5 seeds) and 47.3% at 350M (single run) (Table 8; SUMO omits the 1/pi1/p_i scale, so this tests the selection). Uniform sampling of the tail gives about 80% of that gain; Eq. 7's spectrum-weighted probabilities add the remaining 20% over uniform (refresh every 200 steps).

W2, Q1. GoLore and GUM. The revision cites both in §2.2 and discusses them in the related work (App. D). PLUMAGE is a strict low-rank gradient estimator: its update and optimizer state use only the projected k×nk\times n gradient, GaLore's budget (§2.2). The two relate to it as follows:

  • GoLore (He et al., 2025) shows that under gradient noise, GaLore's SVD subspace can capture the noise, so that GaLore need not converge. Its remedy is a random projection, similar to FLORA's, resampled at GaLore's refresh period in place of the top-kk one, with an O(1/T)O(1/\sqrt{T}) rate for momentum SGD. Its authors note that applying it directly "may not be ideal", since early in training GaLore's SVD subspaces retain more gradient information, and use it only for the last 20–50% of training, after GaLore. PLUMAGE has both properties throughout training: it samples from the gradient's spectrum, so it is unbiased at every refresh while favoring the dominant directions, and its error is never above that of an unbiased random projection of the same rank (Eqs. 6–7; App. D). The paper compares against random projections of this kind: FLORA (Table 2) and uniform sampling (Table 8).
  • GUM (Pan et al., 2025) debiases GaLore under Muon by layer-wise sampling: a random subset of layers receives a compensated full-rank update, so that the expected update matches full-rank Muon. A sampled layer needs the full-rank gradient and optimizer state. So GUM's unbiasedness comes from these full-rank layer updates, not from the low-rank projection: it exceeds the fixed per-layer budget and belongs with the methods that retain full-rank gradient information, which §2.2 separates from strict low-rank gradient estimators such as PLUMAGE.

Thank you for the clarification and for including the related works in the discussions. I think the limitation is reasonable, and I will keep my rating at 3 in favor of the theoretical motivation of the paper.

Reviewer_UzdYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces \textbf{PLUMAGE}, a probabilistic low-rank gradient estimator for memory-efficient training of large neural networks. Unlike the deterministic top-kk approximation used in GaLore, PLUMAGE samples exactly kk singular components without replacement, using inclusion probabilities that minimize conditional Frobenius mean-squared error within a specified class of unbiased estimators.

The authors implement the estimator through a compact one-sided projection and propose a procedure for realigning Adam's first- and second-moment statistics when the projection subspace changes. The method is evaluated on Llama pre-training at scales from 130M to 1B parameters, RoBERTa and Llama 3.1-8B fine-tuning, and through integration of the subspace-sampling rule into SUMO. The results demonstrate improvements over GaLore with similar runtime under the authors' matched dense-SVD backend.

Critical strengths
  1. The paper addresses a limitation of deterministic top-kk gradient approximation: the systematic exclusion of lower-energy singular components. PLUMAGE offers a bias-variance tradeoff under a fixed, predictable representation budget. Its one-sided formulation is compatible with existing memory-efficient low-rank optimization frameworks.

  2. The authors derive optimal marginal inclusion probabilities and establish minimum conditional Frobenius error among unbiased estimators selecting exactly kk components from the current singular basis. Importantly, they do not claim that the guarantee automatically extends to trajectories with one-sided projectors.

  3. The experiments include pre-training and fine-tuning, multiple model scales, ablations, comparisons with uniform sampling, and a transfer experiment involving SUMO. The results consistently favor PLUMAGE over the matched GaLore baseline in the reported C4 experiments. The ablations offer useful evidence for both the sampling rule and the moment-realignment procedure.

Critical weaknesses
  1. The optimal inclusion probabilities derived in Section 3.1 are essentially the same as those obtained in ATOMO (Wang et al., 2018). Hence, the main advantage is a deterministic representation budget, not a strictly smaller variance with identical marginals. This construction is useful, but the paper could position its novelty more precisely relative to these established results.

  2. The unbiasedness and minimum-error results apply to a fixed gradient and its current exact singular basis. The experiments reuse a sampled projector for 200 optimization steps and employ nonlinear adaptive scaling through Adam. Conditional unbiasedness with respect to the current optimization history is not guaranteed in that regime. The authors acknowledge this limitation, but it substantially restricts to what extent the theory explains the observed training improvements.

  3. Since the method introduces additional randomness into subspace selection, the lack of uncertainty estimates at larger scales limits confidence in the robustness and magnitude of the gains.

  4. PLUMAGE requires a full-spectrum SVD at projection refresh steps. The runtime comparison uses a matched dense-SVD GaLore backend, whereas optimized GaLore implementations may employ cheaper partial or approximate decompositions. Moreover, Table~10 reports higher GPU-hours for both low-rank methods than for full-rank Adam at the same training budget. This should be clearer studied and reported

Questions for authors
  1. Could the authors compare PLUMAGE with top-kk GaLore using exactly the same first- and second-moment realignment procedure? Such a control would better isolate the contribution of probabilistic subspace selection, which appears to be the central to novelty.

  2. How robust are the improvements on Llama 350M across independent random seeds? A small number of additional runs at this scale would be valuable given that the headline 350M and 1B results currently rely on single runs.

  3. How does PLUMAGE compare with GaLore using an optimized partial or randomized SVD backend when evaluated in end-to-end wall-clock time and peak GPU memory? A breakdown of the cost of SVD refresh would also be informative.

3 comments
AuthorsOpenReview ↗

Authors Response Part 1/2

We thank the reviewer for crediting the fixed, predictable representation budget and the ablations; the three analyses they ask for are now in the revision (Q1–Q3).

W1. Inclusion probabilities and novelty relative to ATOMO. The submission's introduction motivates an exact fixed-rank budget and a one-sided projection by the variable rank of unbiased alternatives such as ATOMO (§1), and §2.1 credits ATOMO with the minimum-variance unbiased estimator. The inclusion probabilities of §3.1 (Eq. 7) coincide with ATOMO's sampling probabilities, and the revision now states this explicitly. As the review notes, the advantage is the deterministic budget at equal error: our contribution is to attain this minimum within the fixed budget (Alg. 4), so memory stays at GaLore's k×nk\times n. A stale SVD can be reused only one-sidedly: its two-sided atoms uivi⊤u_iv_i^\top are biased even with fresh sampling, so ATOMO must re-decompose every gradient, while a freshly sampled one-sided projection stays unbiased (App. A.4, revised).

W2. Scope of the theory. As the review says, the theory explains the gains only in part; the rest is shown empirically. The part it explains: GaLore's top-kk discards the tail directions at every refresh, so its error can accumulate, while with a new unbiased projection drawn at each refresh the errors are expected to decorrelate. The estimator with its 1/pi1/p_i scale is tested directly where the update is linear in the gradient: under SGD with momentum at 350M (Fig. 1b), terminal evaluation loss is 5.05–5.07 for top-kk, 4.83–4.86 for PLUMAGE and 4.79–4.83 for full rank (2 runs each; values in the revised caption). The Q1 table (second comment) isolates the estimator: PLUMAGE keeps its gain when GaLore is given the same realignment. Within its scope, the theory adds one link to optimization: when a new sample is drawn at every step, the estimator's variance enters the SGD rate as extra gradient noise (Eq. 24).

W3, Q2. Seeds. Table 4 now reports mean ± standard deviation at 350M, as at 130M (⋆ = new; standard deviation over 5 seeds):

C4 perplexityfull rankPLUMAGEGaLore
130M26.06±0.0328.73±0.0430.26±0.05
⋆ 350M (full rank: 3 seeds)19.04±0.0221.82±0.0323.30±0.01
1B (single run)14.316.2917.03

The added seeds showed that the submitted 350M GaLore run (reported 24.08) reset Adam's moments at every refresh. Table 4's other runs are unaffected: the 130M and 1B GaLore runs keep the moments, and the 350M PLUMAGE and full-rank results reproduce. PLUMAGE now closes 35% of GaLore's gap to full rank at 350M (was 45%) and 33% on average over the three scales (abstract; was 36%), and reaches GaLore's terminal loss in 66% of the steps (§4.3; was 50%). We keep one seed at 1B: a 1B run took about 690 GPU-hours (dense SVD, 8 GPUs), and the PLUMAGE–GaLore perplexity gap there is more than 10× the largest standard deviation over seeds at the smaller scales.

W4, Q3. SVD cost. With optimized SVD backends (⋆), a PLUMAGE step costs at most 2 percentage points more than GaLore's fastest randomized variant, at the same peak memory. Step times are means with the refresh amortized over τ=200\tau=200 steps (4× A100-80GB); one refresh of all layers is timed on one A100-40GB (the dense SVD of all reported runs: 15.3 / 62.0 s). Each cell is 350M / 1B.

Backendstep time relative to Adampeak memory relative to Adam (GiB)one refresh (seconds)
GaLore, randomized top-kk (torch.svd_lowrank, q=kq=k, niter=2)+8.2% / +1.2%−2.47 / −6.583.4 / 9.1
PLUMAGE, complete basis (float64 Gram + eigh)+9.0% / +1.0%−2.46 / −6.552.5 / 7.5

Top-kk costs O(mnk)O(mnk) against O(m2n)O(m^2n) for the full spectrum of an m×nm\times n gradient; as our rank grows with the width (m/km/k is 3–4), that is a constant factor, not a faster rate (App. H.2, revised). Randomized top-kk backends are approximate: the singular-value error of svd_lowrank reaches 1.6% of σ1\sigma_1, against 2.4⋅10−102.4\cdot10^{-10} for PLUMAGE's Gram route. They also return no tail directions, which PLUMAGE needs to sample. GaLore's fastest randomized variant (in the revised Table 10) reaches +7.2% / +0.5%.

The submitted Table 10 overstated the overhead at 1B (+47%): its Adam entry came from an older run at global batch 512, mislabelled as 1024, and not from the Table 4 run. Its GPU-hours also counted duplicated work: all 8 GPUs repeated the same dense SVD (22–23% of a step at 350M, 15% at 1B); the Gram route is 5–8× faster. Table 10 now lists the measured step times and the GPU-hours they imply (step time × steps × 4 GPUs, Gram for both low-rank methods): 34.7 / 37.5 / 37.9 at 350M and 476 / 480 / 481 at 1B for Adam / GaLore / PLUMAGE.

Q1 follows in a second comment.

AuthorsOpenReview ↗

Authors Response Part 2/2

Reply to Reviewer UzdY, continued (⋆ = new).

Q1. Same realignment. With the same realignment for both, PLUMAGE keeps its lead (1.5 perplexity points at 130M and 350M), and the realignment does not change GaLore's result at either scale. Each row below fixes the optimizer and the realignment and swaps the estimator. S/MP realigns both of Adam's moments at a refresh, MP only the first. Values are C4 perplexity; in the Adam rows, learning rates are those of the Table 4 runs (App. I.2: GaLore's recommended 10−210^{-2}; full-rank Adam's 10−310^{-3} for PLUMAGE).

top-kkPLUMAGE
⋆ Adam, S/MP for both, 130M30.22±0.0628.73±0.04
⋆ Adam, S/MP for both, 350M23.32±0.0221.82±0.03
⋆ Adam, no realignment, 130M30.26±0.0529.57±0.03
SUMO (Muon-style, MP), 130M (Table 8)30.32±0.0428.29±0.04

PLUMAGE is also ahead when neither method realigns (0.7 perplexity points).

I thank the authors for their response and clarifications. While these addressed some points and improved my understanding in places, they did not substantially change my overall assessment of the paper. I therefore prefer to retain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers and the area chair. Each reviewer has a self-contained reply. The reviews raise four main concerns; each is stated below as a question, with our answer:

  • Does the unbiasedness guarantee cover a projector reused for 200 steps, or Adam? (all reviewers) No: PLUMAGE is an unbiased gradient estimator at GaLore's memory; we do not claim an unbiased update. The gain comes from which directions are updated: GaLore's top-kk discards the tail directions at every refresh, so its error can accumulate there; PLUMAGE draws a new unbiased projection at each refresh, so successive errors are expected to decorrelate and not accumulate in directions that top-kk may never visit. The estimator with its inverse-probability scale is tested directly under SGD with momentum, where the update is linear in the gradient (Fig. 1b): terminal evaluation loss is 4.83–4.86, against 5.05–5.07 for top-kk and 4.79–4.83 for full rank.
  • Does the gain come from the moment realignment and not from the estimator? (UzdY, gCxV) Not from the realignment alone: given PLUMAGE's moment realignment, GaLore does not improve meaningfully, while PLUMAGE leads by 1.5 perplexity points with it (130M and 350M) and by 0.7 without it (130M). With SUMO, a Muon-style optimizer, at a 200-step refresh, uniform sampling of the discarded (tail) directions gives about 80% of the gain from top-kk, while PLUMAGE's spectrum-weighted probabilities add the remaining 20% over that uniform choice.
  • Do the larger-scale results hold over seeds? (UzdY) We added seeded 350M experiments with Adam, five seeds per method (three for full rank): PLUMAGE closes 35% of GaLore's gap to full-rank training. 1B remains a single run.
  • How much more does PLUMAGE's full decomposition cost than GaLore's? (UzdY, gCxV) The full decomposition cost is O(m2n)O(m^2n) per refresh for an m×nm\times n gradient (m≤nm\le n), against O(mnk)O(mnk) for top-kk. The rank scales with the width (m/km/k = 3 or 4, at every model scale), so the two are of the same asymptotic complexity order, and top-kk is cheaper by a constant factor for the given rank. Only the smaller side of each gradient is decomposed, and one refresh is amortized over τ=200\tau=200 steps: with newly reported optimized decompositions, a training step costs 1–9% more than full-rank Adam on average, up to 2 percentage points above GaLore's fastest measured variant, at equal memory. The Gram-matrix route (with float64 eigen-decomposition) is fast and accurate, to 2.4⋅10−102.4\cdot10^{-10} of σ1\sigma_1; randomized SVD with power iterations may not be as accurate (singular-value error up to 1.6% of σ1\sigma_1 in our measurement, using 2 power iterations).

Corrections. The revision corrects three errors in the submission: a 350M baseline run that reset Adam's moments at each refresh (PLUMAGE's share of the gap is 35%, not 45%; the abstract's average is 33%, not 36%; GaLore's terminal loss is reached in 66% of the steps, not 50%); the Adam reference entry of the runtime table, which had overstated the overhead; and Eq. 18, which placed a scaling factor differently from the code (no result changes). No major result or conclusion is withdrawn.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.