- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes the dominant directions of the gradient after singular value decomposition during supervised fine-tuning, and ask whether these directions change depending based on energy overlap for input/output side (right vs. left singular vectors). The authors measure this by looking at the energy overlap and comparing the current gradient's top singular directions with an earlier subspace. The experiments show that input-side overlap exceeds output-side significantly and the asymmetry is observed at all model-average measurement points. Input-side is also more stable than output-side and based on this insight, the authors propose OG-GaLore, which always projects on the input side.
Critical strengths
The paper demonstrates the asymmetry across many SFT settings, to show that input-side gradient subspaces keep more energy than output-side subspaces at all 223 model-level measurement points. This is evaluated across eight datasets and model families. The observed asymmetry provides a useful empirical finding that can guide the design of subspace-based memory-efficient optimizers.
Critical weaknesses
-
The paper provides limited explanation of the observed asymmetry beyond an empirical account. For example, activation geometry is a natural candidate explanation for input-side persistence, since the gradient’s input-side directions depend jointly on input activations and backpropagated errors. However, the errors can shift which directions dominate, and the activation subspace itself may drift, so this relationship alone does not establish temporal stability. The paper does not directly investigate these factors, limiting its explanatory contribution beyond documenting the pattern.
-
The proposed optimizer is an existing GaLore setting, so presenting it as a new memory-efficient optimizer overstates the contribution. The empirical support is also insufficient to demonstrate a strong practical advantage. The comparisons only cover one dataset, reports validation loss only with no downstream task evaluation.
Questions for authors
-
Is OG-GaLore the same as using projection right side, based on GaLore? If it's identical, then OG-GaLore should be presented as an evaluation of that config rather than a new optimizer.
-
How much of the input-side subspace is explained by the principal subspace of the layer's input activations?
4 comments
Reply to weakness 1 & question 2 (2/2)
(3) Results across training steps, diagnostic ranks, and optimizers.
The results cover 196 parameter matrices across 28 transformer layers, including the Q/K/V/O and gate/up/down projections. Measurements are taken at steps 1, 40, 80, 120, 160, 200, 240, and 280. Local comparisons use step 1 as the reference for step 40 and the preceding recorded checkpoint thereafter.
All values are percentages, averaged equally over the 196 matrices. These are per-checkpoint matrix averages. The four carrier-to-gradient columns report full-gradient energy retention.
AdamW — Qwen3-1.7B, PrOntoQA
| Step | Local | Local | Same-checkpoint | Same-checkpoint | Local | Local | |
|---|---|---|---|---|---|---|---|
| 40 | 8 | 92.36 | 27.93 | 61.05 | 43.13 | 55.52 | 18.21 |
| 40 | 16 | 92.06 | 35.21 | 68.89 | 55.97 | 62.15 | 24.83 |
| 40 | 32 | 91.85 | 42.31 | 76.32 | 67.57 | 68.68 | 32.72 |
| 80 | 8 | 98.75 | 34.76 | 63.31 | 64.97 | 62.58 | 24.73 |
| 80 | 16 | 98.28 | 41.95 | 70.96 | 75.94 | 69.98 | 34.10 |
| 80 | 32 | 98.25 | 49.50 | 78.09 | 84.95 | 77.04 | 43.58 |
| 120 | 8 | 99.30 | 32.17 | 63.07 | 63.16 | 62.61 | 23.99 |
| 120 | 16 | 98.93 | 38.02 | 70.48 | 73.85 | 69.68 | 31.24 |
| 120 | 32 | 98.82 | 44.61 | 77.47 | 82.99 | 76.52 | 39.12 |
| 160 | 8 | 99.53 | 34.49 | 60.33 | 75.15 | 60.04 | 27.59 |
| 160 | 16 | 99.24 | 41.72 | 68.28 | 83.76 | 67.79 | 35.50 |
| 160 | 32 | 99.08 | 47.71 | 75.39 | 91.24 | 75.12 | 43.00 |
| 200 | 8 | 99.70 | 17.18 | 60.01 | 64.48 | 59.96 | 13.51 |
| 200 | 16 | 99.41 | 21.25 | 67.23 | 75.62 | 67.45 | 18.59 |
| 200 | 32 | 99.32 | 26.16 | 76.06 | 84.68 | 75.94 | 24.35 |
| 240 | 8 | 99.71 | 22.16 | 60.95 | 65.86 | 61.03 | 15.94 |
| 240 | 16 | 99.38 | 28.35 | 68.36 | 76.28 | 68.60 | 22.82 |
| 240 | 32 | 99.26 | 36.29 | 76.51 | 84.82 | 76.34 | 31.09 |
| 280 | 8 | 99.67 | 28.79 | 62.48 | 64.06 | 62.22 | 21.48 |
| 280 | 16 | 99.31 | 34.21 | 69.79 | 74.98 | 69.53 | 28.36 |
| 280 | 32 | 99.17 | 40.65 | 76.59 | 84.19 | 76.30 | 36.15 |
Muon — Qwen3-1.7B, PrOntoQA
| Step | Local | Local | Same-checkpoint | Same-checkpoint | Local | Local | |
|---|---|---|---|---|---|---|---|
| 40 | 8 | 86.79 | 26.48 | 61.84 | 52.40 | 53.95 | 15.34 |
| 40 | 16 | 86.50 | 33.14 | 69.64 | 63.36 | 60.11 | 22.62 |
| 40 | 32 | 86.53 | 40.62 | 77.43 | 73.27 | 66.90 | 31.67 |
| 80 | 8 | 93.19 | 31.99 | 61.14 | 69.08 | 57.39 | 23.89 |
| 80 | 16 | 92.85 | 38.90 | 68.87 | 79.81 | 64.44 | 32.25 |
| 80 | 32 | 93.01 | 45.88 | 77.00 | 88.34 | 71.55 | 40.29 |
| 120 | 8 | 95.97 | 27.80 | 67.75 | 57.88 | 64.69 | 22.07 |
| 120 | 16 | 95.77 | 33.18 | 74.27 | 69.48 | 70.96 | 29.91 |
| 120 | 32 | 95.91 | 39.51 | 81.09 | 79.97 | 77.61 | 38.39 |
| 160 | 8 | 97.92 | 33.27 | 65.43 | 63.54 | 63.95 | 28.16 |
| 160 | 16 | 97.79 | 39.13 | 72.68 | 74.85 | 71.04 | 35.87 |
| 160 | 32 | 97.69 | 46.56 | 80.36 | 83.75 | 78.60 | 45.11 |
| 200 | 8 | 99.08 | 22.15 | 63.34 | 76.74 | 62.94 | 18.43 |
| 200 | 16 | 98.98 | 30.78 | 70.04 | 86.15 | 69.51 | 27.76 |
| 200 | 32 | 98.93 | 40.01 | 77.93 | 93.05 | 77.15 | 37.67 |
| 240 | 8 | 99.51 | 9.92 | 59.07 | 73.72 | 58.97 | 8.61 |
| 240 | 16 | 99.28 | 14.09 | 67.04 | 82.61 | 67.40 | 12.64 |
| 240 | 32 | 99.17 | 19.46 | 75.57 | 91.07 | 75.67 | 17.89 |
| 280 | 8 | 99.50 | 14.11 | 59.78 | 74.32 | 59.15 | 11.03 |
| 280 | 16 | 99.27 | 19.29 | 68.83 | 83.23 | 67.73 | 16.06 |
| 280 | 32 | 99.10 | 26.47 | 78.29 | 91.60 | 77.21 | 23.25 |
(4) Interpretation of the results.
Activation directions remain more stable across checkpoints. At every reported step and rank under both optimizers, local activation energy overlap exceeds local error energy overlap. At step 280 and , these values are 99.17% versus 40.65% under AdamW and 99.10% versus 26.47% under Muon.
Previously recorded activation directions retain nearly the same gradient energy as current activation directions. At step 280 and , same-checkpoint activation directions retain 76.59% and 78.29% of the full gradient energy under AdamW and Muon. Using activation directions from step 240 retains 76.30% and 77.21%, respectively.
Error directions show strong same-checkpoint alignment but substantially weaker temporal persistence. At step 280 and , current error directions retain 84.19% under AdamW and 91.60% under Muon. Using error directions from step 240 reduces retention to 36.15% and 23.25%, respectively.
These results provide empirical evidence connecting the observed asymmetry to the temporal persistence of activation directions and their continued ability to capture subsequent gradient energy.
References:
[1] Decoupled Weight Decay Regularization. In ICLR 2019.
[2] Muon: An Optimizer for Hidden Layers in Neural Networks. Technical blog, 2024.
Reply to weakness 2 & question 1 (1/2)
For weakness 2: The proposed optimizer is an existing GaLore setting, so presenting it as a new memory-efficient optimizer overstates the contribution. The empirical support is also insufficient to demonstrate a strong practical advantage. The comparisons only cover one dataset, reports validation loss only with no downstream task evaluation.
& For question 1: Is OG-Galore the same as using projection right side, based on GaLore? If it's identical, then OG-Galore should be presented as an evaluation of that config rather than a new optimizer.
A2 / Q1: Thank you for your constructive comments.
(1) Clarification of our contribution.
OG-GaLore uses GaLore’s [1] existing right projection option. We name it Overlap-Guided Gradient Low-Rank Projection (OG-GaLore) to emphasize our motivation: allowing gradients to retain more of their original information. We consider this an innovative application arising naturally from our main empirical findings.
The manuscript is intended to present OG-GaLore as an application of our findings, rather than a new optimizer. Nevertheless, we recognize that phrases such as “we propose a memory-efficient optimizer” in the abstract and “we introduce a memory-efficient optimizer” in the introduction may cause misunderstanding. We have revised these two statements to explicitly describe OG-GaLore as an optimization strategy for memory-efficient optimizers such as GaLore.
GaLore [1] and its variants, including SubTrack++ [2], GoLore [6], and LDAdam [7], all use the shape-based std strategy. In contrast, OG-GaLore consistently uses the input-side right strategy across all target matrices.
**More importantly, although right is available in the code, GaLore [1], SubTrack++ [2], GoLore [6], and LDAdam [7] have not used it as a uniform projection policy across all target matrices, instead adopting the shape-based std strategy. They neither discuss this uniform input-side policy nor report experiments comparing it with alternative projection-side selection strategies. They also do not compare the relative merits of the four strategies—std, reverse_std, left, and right. Our in-depth investigation of these strategies therefore constitutes a novel contribution in its own right. Even though right already exists in GaLore’s code, showing that it achieves the best performance in most settings and approaches the best performance in a small number of settings is itself a contribution. Using right as the default can reduce the projection-side search cost by approximately 75%.
To the best of our knowledge, we are the first to systematically compare four projection-side selection strategies and demonstrate the overall advantage of uniform right projection in LLM supervised fine-tuning. We connect this advantage to the temporal input–output asymmetry of gradient subspaces.
Our main empirical contribution is to reveal the temporal dynamics of gradient subspaces and their input/output asymmetry: historical input-side subspaces generally retain more of the current gradient’s dominant energy than historical output-side subspaces. OG-GaLore, a projection-strategy improvement for GaLore, is a natural application of this main finding. Our novelty therefore extends well beyond proposing a strategy for improving an optimizer. We therefore believe that our work still has substantial novelty.
(2) Additional downstream evaluations across models, tasks, ranks, and seeds.
We expand the evaluation beyond validation loss. We evaluate downstream accuracy on PrOntoQA [3] and ProsQA [4], comparing four projection strategies within the same GaLore framework (std, reverse_std, left, and right).
For Qwen3-1.7B and Llama-3.2-1B-Instruct, we evaluate projection ranks . Values below are accuracy percentages, reported as mean ± sample standard deviation over three seeds. The projection rank is the optimizer’s rank, distinct from the diagnostic rank used in the subspace analysis. Bold indicates the highest mean accuracy in each row.
Qwen3-1.7B — three seeds
| Dataset | Rank | std | reverse_std | left | right |
|---|---|---|---|---|---|
| PrOntoQA | 4 | 31.87 ± 0.81 | 42.47 ± 0.81 | 34.60 ± 1.44 | 49.00 ± 1.11 |
| PrOntoQA | 8 | 29.53 ± 1.14 | 44.67 ± 1.50 | 32.40 ± 1.11 | 51.80 ± 0.92 |
| PrOntoQA | 16 | 28.47 ± 1.10 | 47.53 ± 2.19 | 32.00 ± 1.78 | 54.80 ± 1.39 |
| PrOntoQA | 32 | 22.53 ± 0.95 | 52.07 ± 1.81 | 31.00 ± 0.35 | 55.53 ± 1.75 |
| ProsQA | 4 | 40.33 ± 0.81 | 55.93 ± 0.31 | 32.00 ± 1.20 | 58.07 ± 0.61 |
| ProsQA | 8 | 43.07 ± 0.70 | 55.80 ± 1.78 | 31.73 ± 0.50 | 58.87 ± 0.12 |
| ProsQA | 16 | 44.33 ± 0.31 | 58.00 ± 1.44 | 34.67 ± 0.83 | 60.33 ± 1.14 |
| ProsQA | 32 | 48.20 ± 0.72 | 58.40 ± 0.53 | 38.93 ± 0.64 | 62.07 ± 1.21 |
Reply to weakness 2 & question 1 (2/2)
Llama-3.2-1B-Instruct — three seeds
| Dataset | Rank | std | reverse_std | left | right |
|---|---|---|---|---|---|
| PrOntoQA | 4 | 1.93 ± 0.42 | 32.87 ± 4.28 | 1.60 ± 0.40 | 33.27 ± 3.33 |
| PrOntoQA | 8 | 3.53 ± 0.64 | 37.67 ± 3.90 | 2.47 ± 0.83 | 37.53 ± 3.10 |
| PrOntoQA | 16 | 12.20 ± 0.92 | 41.93 ± 1.94 | 9.27 ± 0.58 | 44.60 ± 2.96 |
| PrOntoQA | 32 | 30.93 ± 1.80 | 47.40 ± 1.93 | 25.60 ± 1.06 | 48.93 ± 0.83 |
| ProsQA | 4 | 53.20 ± 1.22 | 57.87 ± 0.90 | 52.40 ± 1.25 | 59.80 ± 0.53 |
| ProsQA | 8 | 54.20 ± 1.40 | 58.60 ± 0.92 | 53.33 ± 1.22 | 59.40 ± 0.20 |
| ProsQA | 16 | 56.47 ± 0.81 | 61.07 ± 1.68 | 55.80 ± 1.31 | 61.53 ± 1.85 |
| ProsQA | 32 | 58.80 ± 1.60 | 62.67 ± 0.31 | 59.27 ± 0.61 | 63.27 ± 1.01 |
Across these two models, fixed right projection outperforms std in all 16 model–task–rank settings and achieves the highest mean accuracy in 15 of the 16 settings. The exception is Llama-3.2-1B-Instruct on PrOntoQA at rank 8, where reverse_std exceeds right by 0.14 percentage points.
Qwen3-0.6B — three seeds
We additionally evaluate the four projection strategies on Qwen3-0.6B at projection ranks . Values are accuracy percentages, reported as mean ± sample standard deviation over three seeds. Bold indicates the highest mean accuracy in each row.
| Dataset | Rank | std | reverse_std | left | right |
|---|---|---|---|---|---|
| PrOntoQA | 16 | 17.53 ± 0.61 | 17.60 ± 1.51 | 28.07 ± 0.90 | 22.27 ± 2.05 |
| PrOntoQA | 32 | 26.27 ± 0.42 | 13.53 ± 1.70 | 23.53 ± 1.45 | 29.67 ± 0.50 |
| ProsQA | 16 | 59.40 ± 1.00 | 30.00 ± 1.59 | 35.33 ± 0.99 | 60.40 ± 0.72 |
| ProsQA | 32 | 61.33 ± 1.40 | 36.27 ± 0.50 | 33.33 ± 1.33 | 61.67 ± 0.61 |
right performs best in 18 of the 20 settings. In the remaining two, it trails the best strategy by 0.14 and 5.80 percentage points, respectively. The largest accuracy gaps from the best strategy across the 20 settings are 34.14, 30.40, and 35.33 percentage points for std, reverse_std, and left, respectively, compared with 5.80 percentage points for right. Thus, right is the best-performing overall strategy in these experiments.
References:
[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. In ICML 2024.
[2] SubTrack++: Gradient Subspace Tracking for Scalable LLM Training. In NeurIPS 2025.
[3] Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. In ICLR 2023.
[4] Training Large Language Models to Reason in a Continuous Latent Space. In COLM 2025.
[6] Subspace Optimization for Large Language Models with Convergence Guarantees. In ICML 2025.
[7] LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. In ICLR 2025.
Reply to weakness 1 & question 2 (1/2)
For weakness 1: The paper provides limited explanation of the observed asymmetry beyond an empirical account. For example, activation geometry is a natural candidate explanation for input-side persistence, since the gradient’s input-side directions depend jointly on input activations and backpropagated errors. However, the errors can shift which directions dominate, and the activation subspace itself may drift, so this relationship alone does not establish temporal stability. The paper does not directly investigate these factors, limiting its explanatory contribution beyond documenting the pattern.
& For question 2: How much of the input-side subspace is explained by the principal subspace of the layer’s input activations?
A1 / Q2: Thank you for your constructive comments. Our additional experiments show that activation subspaces exhibit greater temporal stability than backpropagated-error subspaces, and activation directions from the previous recorded checkpoint retain more subsequent gradient energy than the corresponding error directions. This pattern holds across diagnostic ranks k=8,16,32 in the completed Qwen3-1.7B–PrOntoQA runs with both AdamW [1] and Muon [2].
For example, at step 280 and k=32, activation directions from step 240 retain 76.30% and 77.21% of the complete current gradient energy under AdamW and Muon, respectively, whereas error directions from the same checkpoint retain 36.15% and 23.25%.
(1) Gradient factorization and matrix shapes.
For a linear layer Yt=WtXt, let Wt∈Rdout×din, Xt∈Rdin×N, and Δt=∂L/∂Yt∈Rdout×N, where N is the number of token positions included in the batch and loss-normalization factors are absorbed into Δt. The gradient is
Gt=ΔtXt⊤=UtΣtVt⊤∈Rdout×din.
Its input-side and output-side Gram matrices satisfy
Gt⊤Gt=XtΔt⊤ΔtXt⊤,
GtGt⊤=ΔtXt⊤XtΔt⊤.
These identities characterize how input activations and backpropagated errors jointly determine the gradient subspaces. Our experiments measure the temporal stability of their principal directions and how much current gradient energy these directions retain.
(2) Temporal energy overlap and full-gradient energy retention.
We follow the energy-overlap notation in the manuscript. Let QX,t∈Rdin×k and QΔ,t∈Rdout×k contain the leading k left singular vectors of Xt and Δt, respectively. Equivalently, these are the leading eigenvectors of XtXt⊤ and ΔtΔt⊤.
For C∈{X,Δ}, define the normalized top-k energy matrix as
DC,t=diag(aC,t,1,…,aC,t,k),aC,t,i=∑j=1kσC,t,j2σC,t,i2.
The temporal energy overlap is
\mathcal{O}_{C}^{E}(s,t)=\left\\|Q_{C,s}^{\top}Q_{C,t}D_{C,t}^{1/2}\right\\|_{F}^{2}.
For local comparisons, s denotes the previous recorded checkpoint, following the manuscript.
We additionally measure how much of the complete current gradient energy is retained by an activation or error reference subspace. Let Gt=UtΣtVt⊤ be the full thin SVD, with Ut∈Rdout×r, Vt∈Rdin×r, and r=min(dout,din). Define
DG,tfull=diag(bt,1,…,bt,r),bt,i=∑j=1rσG,t,j2σG,t,i2.
The full-gradient energy retention is
Rin(s,t)=QX,s⊤Vt(DG,tfull)1/2F2,
Rout(s,t)=QΔ,s⊤Ut(DG,tfull)1/2F2.
Both products inside the Frobenius norms have shape k×r: the reference contains k directions, while the current gradient retains its full spectrum. We report same-checkpoint retention with s=t and local retention with s equal to the previous recorded checkpoint.