- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OrthVMS combines adaptive optimization in a low-rank subspace with an orthogonal residual correction using column-wise second-moment statistics. It preserves full-rank updates with little additional optimizer state, potentially improving the accuracy–memory trade-off of low-rank training.
Critical strengths
- The residual correction requires only a small additional vector of statistics and preserves orthogonality to the coarse update.
- Experiments cover both Transformer and GLA models. This is the first time I have seen GLA models in this kind of paper.
Critical weaknesses
- Transformer experiments reach only 350M parameters, while the largest experiment uses a roughly 1.4B-parameter GLA model. For an optimizer motivated by memory-efficient LLM pretraining, this leaves its convergence and practical efficiency at larger Transformer (3B+) scales insufficiently established.
- Column-wise scaling preserves orthogonality under the chosen left projection, but this does not establish its advantage over a right-projection variant with row-wise scaling. Given existing row/column normalization methods such as NorMuon and SCALE, an ablation would help clarify whether the chosen orientation materially affects performance.
Questions for authors
- Does OrthVMS use FiRA’s norm-growth limiter or any similar update-norm control? Please clarify this for both OrthVMS and the FiRA baseline. If such a mechanism is used in OrthVMS, an ablation without it would help separate its contribution from the proposed residual preconditioning.
- What is the measured runtime overhead? Please report optimizer-step time and total training time under matched settings, including the amortized cost of SVD refreshes. The complexity analysis alone does not establish practical training efficiency.
4 comments
Thank you for the additional experiments. The orientation ablation and runtime measurements address part of my concerns, and I am willing to raise my rating from 1 to 2.
However, the 7B experiment remains too short to establish sustained pretraining gains. With a global batch size of 16, sequence length 4096, and 10K updates, the run covers at most approximately 0.655B tokens. This provides useful early-training evidence, but does not substitute for a sufficiently long pretraining run. If resources are limited, a longer run on a 3B Transformer would also be a reasonable way to address this concern.
I also remain uncertain about the timing comparison. In my own reproductions, APOLLO has consistently been faster than GaLore across the settings I tested, whereas your table shows the opposite at 60M and 130M. This discrepancy does not necessarily imply an error, but please clarify the APOLLO projection mode, implementation, and timing protocol, including whether projection refresh costs are included. If the difference results from a particular experimental configuration, that should be made explicit.
Thank you for raising your rating and for sharing your APOLLO measurements. For each baseline, we follow its original paper for the optimizer and projection method. We use the official APOLLO implementation with Gaussian random projection. APOLLO source
In our previous response, we divided the end-to-end C4 training time on eight A100-80GB GPUs by the number of completed optimizer updates. That measurement includes data loading, forward and backward passes, optimizer updates, evaluation, and checkpointing. Variation in data loading and other training operations can affect the end-to-end comparison.
We measured optimizer-step time on a single A100-80GB using BF16, C4 batches of 16 sequences, and a sequence length of 1024. After 100 warm-up updates, we timed 400 consecutive calls to optimizer.step(). The projection-based optimizers used rank 256 and refreshed every 200 updates. APOLLO resampled its random projection, while GaLore, FiRA, and OrthVMS used SVD. Each mean for these methods includes both refreshes.
LLaMA-60M, 130M, and 350M optimizer step times (ms)
| Optimizer | 60M | 130M | 350M |
|---|---|---|---|
| AdamW | 2.03 | 4.23 | 10.74 |
| APOLLO | 15.78 | 23.84 | 46.66 |
| GaLore | 15.98 | 35.74 | 98.80 |
| FiRA | 26.97 | 52.37 | 133.07 |
| OrthVMS | 23.03 | 46.37 | 123.10 |
Consistent with your reproductions, APOLLO is faster than GaLore once refresh costs are included. The two are nearly tied at 60M, and APOLLO is 1.5× and 2.1× faster at 130M and 350M. OrthVMS had lower mean optimizer-step times than FiRA at all three model sizes.
Thank you for the helpful suggestion. We followed your recommendation and pretrained a LLaMA model with approximately 3 billion parameters on C4 using GaLore and OrthVMS. Given the limited time available, each run completed 30,000 updates with a global batch size of 32 and a sequence length of 4096. This amounts to a budget of 3.93B token slots per run, six times the corresponding budget of our earlier 7B run. Both methods used projection rank 64 and the same training setup.
OrthVMS achieved lower validation loss at every evaluation, with a final validation loss of 3.5381 compared with 4.3232 for GaLore. Appendix C.4 of the revised manuscript provides the training curves and experimental details.
Response to tnGF
We thank the reviewer for the careful review and thoughtful feedback.
W1: Transformer scale
Thank you for your concern. During the rebuttal period, we ran LLaMA-7B pretraining on C4 with GaLore, FiRA, and OrthVMS. All three methods complete 10K updates with a global batch size of 16, a maximum sequence length of 4096, and projection rank 64. Final validation results are reported below.
OrthVMS reduces PPL by 41.1% relative to GaLore and 62.0% relative to FiRA. These results demonstrate the scalability of OrthVMS: its advantage over GaLore and FiRA carries over to LLaMA-7B pretraining. The training loss and validation PPL curves are shown in Figure 3 (Appendix C.3) of the revised paper.
W2: Projection orientation and residual scaling
Thank you for your suggestion. We ran the requested orientation ablation on GLA-60M by reversing the projection direction and matching the residual-scaling axis. On wide matrices, the comparison is left projection with column-wise scaling versus right projection with row-wise scaling. The roles reverse on tall matrices. Both variants preserve residual orthogonality and use rank 128, a refresh interval of 200, and β₂ = β₃ = 0.999. The results are shown below.
The standard policy performs better at all three learning rates. Selecting the best result for each policy gives 46.55 versus 67.38. This shows that the joint choice of projection direction and scaling axis materially affects performance and supports the policy used by OrthVMS in this setting.
Q1: Norm-growth limiter
Thank you for your question. OrthVMS does not use a norm-growth limiter such as the one used in FiRA. Its residual correction is derived through a rigorous mathematical analogy with VMS theory, yielding a closed-form solution under our scalar closure.
Q2: Training time
Thank you for your question. The results are shown below.
From the table, OrthVMS takes 3–9% more time per step than GaLore, which also performs periodic SVD. Compared with FiRA, OrthVMS is faster per step at all scales.