- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PLUMAGE proposes a randomized low-rank gradient optimizer, that given the current exact singular basis of a gradient matrix, deterministically retains some leading components and samples the remaining components without replacement using inclusion probabilities proportional to their singular values, yielding a conditionally unbiased estimator. The paper also uses first- and second-moment realignment rules for Adam moments will get misaligned after changing subspaces.
Critical strengths
1- The paper clearly identifies the truncation bias introduced by top-k gradient projection and proposes a same budget alternative while preserving the dominant directions.
2- The authors explicitly state that fixed-target conditional unbiasedness an a specific iteration does not imply trajectory-level unbiasedness after a random projector is reused.
3- The experiments include pretraining, GLUE fine-tuning, and 8B supervised fine-tuning. Also state-realignment ablations, a full-rank lm_head control, and a transfer experiment with SUMO is meaningful. Overall, paper tries to explain source of differences via structured experiments.
Critical weaknesses
1- The proposed method still pays the decomposition cost as it needs the full decomposition of the batch gradient, which leads to higher computation time compared to more efficient implementations of GaLore [1]. This is while there are more recent works, like APOLLO [2] or SubTrack++ [3], that are proposed to solve the high computation requirement of SVD. Specifically, APOLLO [2] is a highly relevant baseline to be included as they already outperforms GaLore with randomization, and although they are rescaling the discarded gradient, they provide same memory saving while incurring less computation.
2- As the paper itself acknowledges, the subspace and inclusion probabilities are refreshed every τ iterations, while subsequent gradients depend on earlier updates obtained using the sampled projector; therefore, the practical method is not trajectory-level unbiased, while being unbiased is the main claim of the paper, also stated in the title. A comparison against efficient randomized methods such as APOLLO [2] or simple random projections could help clarify whether the practical gains arise from conditional unbiasedness, stochastic tail coverage, moment realignment, or another aspect of the method, especially as figure 3-b shows small gap between uniform and Plumage sampling.
3- The gap between and in figure 1-b is very close, and it leads to questioning whether the claim that PLUMAGE reaches GaLore’s terminal loss in fewer steps, is really related to the proposed method, or could be achieved by smaller changes, like realigning the rotated moments in GaLore, as it is done in earlier works like LDAdam [4] or rescaling the gradient residual like done in earlier works [2-5].
4- Since the implementation differs materially from the original GaLore setup, using GaLore’s previously recommended hyperparameters may not guarantee that it is near its optimum in the new setup. A matched sweep for both methods would make the comparison more conclusive.
5- Tail components with small singular values can receive small inclusion probabilities and therefore large scaling when selected. It would strengthen the paper to report distributions of minimum inclusion probabilities and inverse scales or number of spikes solved by gradient clipping across layers and training. This is especially important as differs per layer.
6- As the learning dynamics can change in larger models, experiments on models with 7B+ parameters are very much helpful and appreciated.
References
[1] Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML 2024.
[2] Zhu, H., Zhang, Z., Cong, W., Liu, X., Park, S., Chandra, V., Long, B., Pan, D. Z., Wang, Z., and Lee, J. APOLLO: SGD-like Memory, AdamW-level Performance. MLSys 2025.
[3] Rajabi, S., Nonta, N., and Rambhatla, S. SubTrack++: Gradient Subspace Tracking for Scalable LLM Training. 2025.
[4] Robert, T., Safaryan, M., Modoranu, I.-V., and Alistarh, D. LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. ICLR 2025.
[5] Chen, X., Feng, K., Li, C., Lai, X., Yue, X., Yuan, Y., and Wang, G. FiRA: Can We Achieve Full-Rank Training of LLMs under Low-Rank Constraint? 2024.
Questions for authors
Please refer to the weaknesses.
2 comments
Authors Response Part 2/2
Reply to Reviewer gCxV, continued (⋆ = new).
W5. and . Under Eq. 7, every sampled tail component is rescaled to one magnitude at a refresh, , where is the smallest singular value that is always kept. The scale is applied before Adam's moments (the submitted Eq. 18 and Alg. 1 wrote it at the back-projection), and Adam's normalization absorbs it: the step is Adam's with replaced by , which matters only where is near zero. Statistics over layers and training (⋆):
| Median over layers | 130M | 350M |
|---|---|---|
| smallest (worst layer) | () | () |
| largest sampled (share of layers above 100) | 25 (7%) | 37 (17%), stable over training |
| (share of the rank always kept) | 0.25 | 0.15 |
Empirically, there is no instability relative to GaLore, with no gradient clipping: over five 350M seeds per method, the largest rise of the training loss in any 2,000-step window after step 2,000 is 0.010–0.014 for PLUMAGE and 0.010–0.012 for GaLore.
W6. Larger-scale results. 7B+ pre-training is beyond our compute. PLUMAGE stays ahead at every scale we trained, and in Llama-3.1-8B fine-tuning, it lowers GaLore's training loss from 1.05 to 0.97 (Table 3).
Authors Response Part 1/2
We thank the reviewer for valuing structured experiments; their question about where gains arise shaped our controls (⋆ = new).
W1. SVD cost; APOLLO, SubTrack++. PLUMAGE needs the full spectrum at each refresh: O(m2n) for an m×n gradient against O(mnk) for top-k. As our rank grows with the width (m/k is 3–4), the two are of the same order, and top-k is cheaper by a constant factor, not at a faster rate. The paper claims no cost parity (§3.4), and all reported runs, GaLore's included, use the same dense SVD. With optimized SVD backends (⋆) and the refresh amortized over τ=200 steps, a PLUMAGE step costs 9.0% / 1.0% more than full-rank Adam at 350M / 1B, against 7.2% / 0.5% for GaLore's fastest variant, at equal memory.
APOLLO is a memory-efficient, SVD-free optimizer, not an estimator: it rescales the full-rank gradient by a low-rank version of Adam's rule in a random projection instead of top-k (§2.2). Its Table 2 puts most of the gain in the full-rank path: at 350M its SVD variant gives a perplexity of 16.67 against 16.85 with the random projection, while Fira's full-rank residual takes GaLore from 18.95 to 17.03. PLUMAGE works with any optimizer, but it matters where the low-rank gradient determines the update direction, as in the widely used Adam and Muon (SUMO). In APOLLO, the projection estimates only Adam's scaling factors, so a gap between PLUMAGE and APOLLO measures the full-rank update, not the estimator, and APOLLO cannot serve as the control W2 asks for; we add no such run. SubTrack++ replaces the SVD with rank-1 tracking of a deterministic subspace; the revision discusses it in §2.2.
W2. Unbiasedness, the title, source of the gains. The title's claim concerns the estimator, unbiased by construction at every refresh; as the abstract states, it does not extend to the trajectory, since reusing a sample breaks it for any randomized estimator (§3.4). We claim no unbiased update. The gain comes from which directions are updated: GaLore's top-k discards the tail directions at every refresh, so its error accumulates there. In contrast, a new unbiased projection at each refresh decorrelates successive errors.
W3. Fewer steps; smaller changes. In Fig. 1b (SGD with momentum, where the 1/pi-scaled estimator enters the update linearly) the curves separate late: terminal evaluation loss is 5.05–5.07 for GaLore, 4.83–4.86 for PLUMAGE and 4.79–4.83 for full rank (2 runs each, 3rd seed pending; values in the revised caption). The fewer-steps claim rests on the Adam runs (§4.3). Against GaLore's new 350M runs (the submitted run reset Adam's moments at each refresh), each PLUMAGE run reaches GaLore's terminal loss after 65–66% of the steps (interpolated; was 50%). Smaller changes do not achieve this: realigning GaLore's moments (S/MP) does not change its result (table under W2), and with a Fira-style residual already present (SUMO), replacing top-k by PLUMAGE still closes 40.8% of the gap to full-rank Muon at 130M.
W4. Hyperparameters. GaLore's recommended learning rate, 10−2, is the best of a new sweep at 130M (⋆): perplexity 30.51 / 30.26±0.05 / 30.32 at 0.005 / 0.01 / 0.02 (single seeds except at 0.01). PLUMAGE reuses full-rank Adam's 10−3, near the best of PLUMAGE's own sweep (App. I.2); the larger scales reuse both learning rates. We also ran both methods in GaLore's reference code, which reproduces its published 350M result. With
lm_headalso low-rank there (single runs; Table 12), PLUMAGE closes 65% of GaLore's gap to full rank, against 35% in our setup. On GLUE, the sweep is matched: the learning rate is tuned per task and method (App. I.3).W5 and W6 follow in a second comment.