acceptodds
Under review as a conference paper at ICLR 2027

OrthVMS: Orthogonal Variational Multiscale Correction for Full-Rank Updates with Low-Rank Optimizer States

Abstract

Training large language models (LLMs) incurs substantial memory overhead, motivating methods that compress optimizer states via low-rank gradient projection. However, projection-based optimizers restrict each update to a compact subspace, discarding the orthogonal gradient component, while maintaining full-size moment states for this residual remains memory-prohibitive. To harness this orthogonal information without sacrificing efficiency, we propose **OrthVMS**, an **Orth**ogonal **V**ariational **M**ulti**S**cale correction framework. Drawing inspiration from variational multiscale modeling (VMS) in computational mechanics, OrthVMS treats the projected space and its orthogonal complement as coarse and fine spaces, respectively. It constructs a full-space preconditioner from two decoupled components: the coarse component maintains compact adaptive moments within the projected space, while the fine component preconditions the orthogonal residual using its historical energy statistics. This residual-driven correction requires only a single persistent vector of additional states. Extensive experiments on LLM benchmarks demonstrate that OrthVMS consistently outperforms existing memory-efficient training methods. Our code is available [here](https://anonymous.4open.science/r/Orthogonal-VMS-C26C).

Primary area:
optimization
Keywords:
Large Language Models, Memory-Efficient Training, Low-Rank Optimization, Variational Multiscale Methods, Orthogonal Gradient Correction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tnGFOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

OrthVMS combines adaptive optimization in a low-rank subspace with an orthogonal residual correction using column-wise second-moment statistics. It preserves full-rank updates with little additional optimizer state, potentially improving the accuracy–memory trade-off of low-rank training.

Critical strengths
  1. The residual correction requires only a small additional vector of statistics and preserves orthogonality to the coarse update.
  2. Experiments cover both Transformer and GLA models. This is the first time I have seen GLA models in this kind of paper.
Critical weaknesses
  1. Transformer experiments reach only 350M parameters, while the largest experiment uses a roughly 1.4B-parameter GLA model. For an optimizer motivated by memory-efficient LLM pretraining, this leaves its convergence and practical efficiency at larger Transformer (3B+) scales insufficiently established.
  2. Column-wise scaling preserves orthogonality under the chosen left projection, but this does not establish its advantage over a right-projection variant with row-wise scaling. Given existing row/column normalization methods such as NorMuon and SCALE, an ablation would help clarify whether the chosen orientation materially affects performance.
Questions for authors
  1. Does OrthVMS use FiRA’s norm-growth limiter or any similar update-norm control? Please clarify this for both OrthVMS and the FiRA baseline. If such a mechanism is used in OrthVMS, an ablation without it would help separate its contribution from the proposed residual preconditioning.
  2. What is the measured runtime overhead? Please report optimizer-step time and total training time under matched settings, including the amortized cost of SVD refreshes. The complexity analysis alone does not establish practical training efficiency.
4 comments
AuthorsOpenReview ↗

Response to tnGF

We thank the reviewer for the careful review and thoughtful feedback.

W1: Transformer scale

Thank you for your concern. During the rebuttal period, we ran LLaMA-7B pretraining on C4 with GaLore, FiRA, and OrthVMS. All three methods complete 10K updates with a global batch size of 16, a maximum sequence length of 4096, and projection rank 64. Final validation results are reported below.

MethodValidation lossPPL
GaLore4.8400126.46
FiRA5.2796196.29
OrthVMS4.311174.53

OrthVMS reduces PPL by 41.1% relative to GaLore and 62.0% relative to FiRA. These results demonstrate the scalability of OrthVMS: its advantage over GaLore and FiRA carries over to LLaMA-7B pretraining. The training loss and validation PPL curves are shown in Figure 3 (Appendix C.3) of the revised paper.

W2: Projection orientation and residual scaling

Thank you for your suggestion. We ran the requested orientation ablation on GLA-60M by reversing the projection direction and matching the residual-scaling axis. On wide matrices, the comparison is left projection with column-wise scaling versus right projection with row-wise scaling. The roles reverse on tall matrices. Both variants preserve residual orthogonality and use rank 128, a refresh interval of 200, and β₂ = β₃ = 0.999. The results are shown below.

Projection and scaling policyLR 0.01LR 0.015LR 0.02Best
OrthVMS standard policy46.5549.5146.6446.55
Reversed policy68.9567.3873.1867.38

The standard policy performs better at all three learning rates. Selecting the best result for each policy gives 46.55 versus 67.38. This shows that the joint choice of projection direction and scaling axis materially affects performance and supports the policy used by OrthVMS in this setting.

Q1: Norm-growth limiter

Thank you for your question. OrthVMS does not use a norm-growth limiter such as the one used in FiRA. Its residual correction is derived through a rigorous mathematical analogy with VMS theory, yielding a closed-form solution under our scalar closure.

Q2: Training time

Thank you for your question. The results are shown below.

Method60M (ms/step)60M total (min)130M (ms/step)130M total (min)350M (ms/step)350M total (min)
AdamW83.8113.97147.0249.01754.22754.22
GaLore100.2016.70171.1657.05849.69849.69
APOLLO113.1118.85175.2258.41810.01810.01
FiRA130.9521.82217.9772.66884.72884.72
OrthVMS106.7117.79185.9361.98876.35876.35

From the table, OrthVMS takes 3–9% more time per step than GaLore, which also performs periodic SVD. Compared with FiRA, OrthVMS is faster per step at all scales.

Thank you for the additional experiments. The orientation ablation and runtime measurements address part of my concerns, and I am willing to raise my rating from 1 to 2.

However, the 7B experiment remains too short to establish sustained pretraining gains. With a global batch size of 16, sequence length 4096, and 10K updates, the run covers at most approximately 0.655B tokens. This provides useful early-training evidence, but does not substitute for a sufficiently long pretraining run. If resources are limited, a longer run on a 3B Transformer would also be a reasonable way to address this concern.

I also remain uncertain about the timing comparison. In my own reproductions, APOLLO has consistently been faster than GaLore across the settings I tested, whereas your table shows the opposite at 60M and 130M. This discrepancy does not necessarily imply an error, but please clarify the APOLLO projection mode, implementation, and timing protocol, including whether projection refresh costs are included. If the difference results from a particular experimental configuration, that should be made explicit.

Thank you for raising your rating and for sharing your APOLLO measurements. For each baseline, we follow its original paper for the optimizer and projection method. We use the official APOLLO implementation with Gaussian random projection. APOLLO source

In our previous response, we divided the end-to-end C4 training time on eight A100-80GB GPUs by the number of completed optimizer updates. That measurement includes data loading, forward and backward passes, optimizer updates, evaluation, and checkpointing. Variation in data loading and other training operations can affect the end-to-end comparison.

We measured optimizer-step time on a single A100-80GB using BF16, C4 batches of 16 sequences, and a sequence length of 1024. After 100 warm-up updates, we timed 400 consecutive calls to optimizer.step(). The projection-based optimizers used rank 256 and refreshed every 200 updates. APOLLO resampled its random projection, while GaLore, FiRA, and OrthVMS used SVD. Each mean for these methods includes both refreshes.

LLaMA-60M, 130M, and 350M optimizer step times (ms)

Optimizer60M130M350M
AdamW2.034.2310.74
APOLLO15.7823.8446.66
GaLore15.9835.7498.80
FiRA26.9752.37133.07
OrthVMS23.0346.37123.10

Consistent with your reproductions, APOLLO is faster than GaLore once refresh costs are included. The two are nearly tied at 60M, and APOLLO is 1.5× and 2.1× faster at 130M and 350M. OrthVMS had lower mean optimizer-step times than FiRA at all three model sizes.

Thank you for the helpful suggestion. We followed your recommendation and pretrained a LLaMA model with approximately 3 billion parameters on C4 using GaLore and OrthVMS. Given the limited time available, each run completed 30,000 updates with a global batch size of 32 and a sequence length of 4096. This amounts to a budget of 3.93B token slots per run, six times the corresponding budget of our earlier 7B run. Both methods used projection rank 64 and the same training setup.

OrthVMS achieved lower validation loss at every evaluation, with a final validation loss of 3.5381 compared with 4.3232 for GaLore. Appendix C.4 of the revised manuscript provides the training curves and experimental details.

Reviewer_mubKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors analyze the existing memory efficient optimizers, such as GaLore, APOLLO, and FiRA, showing that they do not sufficiently make use of the information in the orthogonal complement. Therefore, in this paper, the authors propose Orthogonal Variational MultiScale (OrthVMS). The authors give two separate updates, coarse-space update and fine-space update. They remain orthogonal since the fine correction preserves the orthogonal-complement structure.

Critical strengths

The authors analyze a very important problem in LLM pretraining: reducing the optimizer memory without sacrificing model performance. This paper successfully addresses the limitations of prior works.

I think this paper is straightforward and easy to understand. The decomposition ensures that the coarse and fine updates are orthogonal. It is a useful structural property and makes the algorithm easy to interpret.

Critical weaknesses

However, my major concern about this paper is that the novelty seems limited. FiRA already introduces an orthogonal residual correction to recover gradient information. I think the main difference is how the residual is scaled. For example, FiRA uses the projected gradient for estimating norm ratio, whereas OrthVMS uses a column-wise exponential moving average of the squared orthogonal residual norm.

In addition, I believe that the downstream improvements are relatively small. For example, the LLaMA-350M shows an average of 43.94% for OrthVMS, and it has 43.75% for APOLLO. In addition, it does not seem that OrthVMS can consistently outperform APOLLO across different tasks.

It would be better if the authors may consider including the ablation study. For example, it would be better to compare GaLore with GaLore plus unscaled residual, and with OrthVMS with β3=0\beta_3 = 0, and full OrthVMS.

Questions for authors

Please see weaknesses.

1 comment
AuthorsOpenReview ↗

Response to mubK

We thank the reviewer for the careful review.

W1: Comparison with FiRA

Thank you for your concern. We agree that both methods add back the orthogonal residual, but they are fundamentally different in viewpoint and in how the residual is preconditioned:

  • Viewpoint. To our knowledge, OrthVMS is the first to bring the variational multiscale (VMS) viewpoint from computational mechanics into memory-efficient LLM pretraining: it treats the projected subspace and its orthogonal complement as the coarse and fine scales and preconditions them separately, which yields a simple algorithm with a convergence guarantee. In contrast, FiRA is motivated by an empirical observation and does not provide a convergence guarantee.
  • Residual preconditioning. Following the VMS view, OrthVMS preconditions the fine scale with its own statistics: each residual column is scaled by its own historical RMS, which requires only one extra vector per matrix. FiRA instead rescales the residual by the norm ratio between the projected Adam update and the projected gradient.

To conclude, the key differences are summarized below:

AspectFiRAOrthVMS
MotivationEmpirical observationVMS modeling
Convergence guaranteeNot providedYes (Section 5)
Residual scaleTransferred from projected spaceResidual's own historical RMS

As shown in W3, this difference in residual scaling is exactly what drives the gain.

W2: Downstream improvements

Thank you for your concern. OrthVMS achieves the highest average downstream score and outperforms APOLLO on six of the ten tasks. The improvements over APOLLO include 2.80 percentage points on SciQ, 2.13 on ARC-Challenge, and 2.10 on ARC-Easy. Furthermore, OrthVMS achieves a pretraining perplexity of 19.45 on LLaMA-350M, compared with 20.17 for APOLLO and 21.65 for FiRA, at approximately the same reported persistent storage of 1.389 GB.

Taken together, the lower pretraining perplexity, the improvements on most downstream tasks, and the highest average score demonstrate the effectiveness of OrthVMS.

W3: Suggested ablation

Thank you for your suggestion. Keeping the coarse update fixed, we ran the suggested ablation on GLA-60M, from GaLore to full OrthVMS. Best PPL over lr ∈ {0.01, 0.015, 0.02} (GaLore: {0.015, 0.02, 0.03}):

MethodResidual scalingPPL
GaLoreNo residual75.43
GaLore + unscaled residualNone78.28
OrthVMS (β₃ = 0)Per-column instantaneous RMS54.12
OrthVMSPer-column historical RMS46.55

From the experiment, we can see that adding the residual without scaling does not improve over GaLore (75.43 → 78.28). Per-column RMS scaling reduces PPL to 54.12, and historical RMS further reduces it to 46.55. Therefore, the gain comes from how the residual is scaled rather than from adding the residual itself.

Reviewer_vszdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OrthVMS, a memory-efficient optimizer that combines low-rank adaptive updates with orthogonal residual corrections. The residual is preconditioned using historical column-wise gradient energy, requiring only a small additional optimizer state. Experiments on LLaMA and GLA pretraining demonstrate lower perplexity over existing memory-efficient optimizers.

Critical strengths

1. Memory-efficient full-space updates. The method incorporates orthogonal gradient information while introducing only a small additional vector of optimizer states.

2. Theoretical analysis. The paper provides convergence guarantees for the combined coarse- and fine-space updates under explicit assumptions.

Critical weaknesses

1. Questionable experimental baselines. In the long-sequence GLA experiments, GaLore and several other baselines perform substantially worse than AdamW. In particular, GaLore-60M achieves its best result at the upper boundary of a restricted learning-rate search. Moreover, the paper does not specify the β1 and β2 values used, raising concerns about whether the baselines are sufficiently tuned.

2. Insufficient validation of the core mechanism. The proposed fine-space preconditioning essentially performs historical column-wise RMS normalization. The paper lacks controlled ablations demonstrating its advantages over simpler residual normalization and properly scaled residual updates.

Questions for authors
  1. Could the authors clarify the β1 and β2 values used for AdamW, GaLore, and OrthVMS? Given that β2 = 0.95 can outperform 0.999 in LLM pretraining, could the authors evaluate different β2 settings to ensure fair comparisons?

  2. Could the authors compare historical RMS preconditioning with instantaneous RMS normalization and properly scaled residual updates to isolate the benefits of historical energy statistics?

  3. Could the authors report end-to-end training throughput and wall-clock efficiency, considering the additional computation and periodic SVD operations?

3 comments
AuthorsOpenReview ↗

Response to vszd

We thank the reviewer for the careful review.

W1 / Q1: β values and baseline tuning

Thank you for your concern. All optimizers in our paper (AdamW, GaLore, FiRA, APOLLO, OrthVMS) use β₁ = 0.9 and β₂ = 0.999, following the default settings of FiRA, GaLore, and APOLLO. During the rebuttal period, we additionally ran GLA-60M with β₁ = 0.9 and β₂ = 0.95.

β₂ = 0.95 (GLA-60M). Best learning rate and final PPL:

Methodβ₂ = 0.95 learning ratesβ₂ = 0.95 best lr (PPL)
AdamW{0.003, 0.006, 0.01}0.006 (47.89)
GaLore{0.015, 0.02, 0.03}0.02 (75.85)
OrthVMS{0.01, 0.015, 0.02}0.01 (47.26)

Under β₁ = 0.9 and β₂ = 0.95, OrthVMS still achieves the lowest PPL among the three methods.

W2 / Q2: Historical RMS vs. simpler residual normalization

Thank you for your question. Keeping the coarse update fixed, we ablated how the residual is normalized on GLA-60M: per column vs. per matrix, and historical vs. instantaneous RMS. Best PPL over lr ∈ {0.01, 0.015, 0.02}:

Residual scalingInstantaneous RMSHistorical RMS
Per column54.1246.55 (OrthVMS)
One scale per matrix50.0464.90

From the experiment, we can see that OrthVMS performs best. Historical RMS improves per-column scaling (54.12 → 46.55) but hurts per-matrix scaling (50.04 → 64.90), so the gain comes from combining per-column scaling with historical RMS.

Q3: End-to-end training time

Thank you for your question. We measured the end-to-end training time of LLaMA pretraining on C4 with 8 × A100 GPUs. The total time covers training, periodic SVD refreshes, evaluation, and checkpointing. Total time (min):

Method60M (10K steps)130M (20K steps)350M (60K steps)
AdamW13.9749.01754.22
GaLore16.7057.05849.69
APOLLO18.8558.41810.01
FiRA21.8272.66884.72
OrthVMS17.7961.98876.35

From the table, OrthVMS takes 3–9% more time than GaLore, which also performs periodic SVD. Compared with FiRA, OrthVMS is faster at all scales.

I thank the authors for the rebuttal and the additional experiments. The wall-clock results address Q3.

My remaining concern is the learning-rate setup:

  1. Different search ranges. AdamW, GaLore, and OrthVMS are tuned over different learning-rate ranges ({0.003, 0.006, 0.01}, {0.015, 0.02, 0.03}, and {0.01, 0.015, 0.02}). Why were they not tuned over the same range? Notably, OrthVMS's best learning rate (0.01) lies at the lower boundary of its range.
  2. Scale factor. Do GaLore and OrthVMS use a scale factor, as in GaLore, so that the embedding and LM head (updated by AdamW) effectively receive a different learning rate from the hidden layers? I suggest following Moonlight and matching the update RMS of all parameter groups to 0.2, so that all methods can share one learning-rate range and the comparison is fair.

I therefore maintain my original score.

Thank you for the follow-up.

1. Learning-rate ranges. We tuned the learning rate separately for each optimizer because different optimizers can perform best at different learning rates. The best tested rates for AdamW and GaLore were 0.006 and 0.02, respectively, with both lower and higher rates tested in each sweep. In our GLA-60M runs with β₂ = 0.95, at the shared learning rate of 0.01, AdamW, GaLore, and OrthVMS achieved validation PPLs of 48.15, 79.08, and 47.26, respectively.

2. Scale factor. Yes. We follow the official default configurations of the baselines: GaLore and FiRA scale the projected update by 0.25, and APOLLO and APOLLO-Mini use their own default scales (1 and 128, respectively). OrthVMS uses the same factor of 0.25 as GaLore for its projected update. The embedding and LM head are updated by standard AdamW at the same base learning rate, so their update RMS can differ from that of the projected layers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.