acceptodds
Under review as a conference paper at ICLR 2027

Gradient Subspace Dynamics During Supervised Fine-Tuning of LLMs: Input–Output Asymmetry

Abstract

The gradient dynamics of supervised fine-tuning (SFT) in large language models (LLMs) remain poorly understood. For a layer , we call the side of the weight gradient corresponding to the input features the input side, and the side corresponding to the output features the output side. To unveil the gradient dynamics of SFT, we conduct seventeen full-parameter SFT runs using AdamW, spanning eight datasets and models from the Qwen3, Llama, and MiniCPM families. Fifteen runs use diagnostic rank , and two additional runs use . We identify four main findings: input-side energy overlap generally remains high during fine-tuning; output-side energy overlap is generally low, especially outside query/key projections; input-side energy overlap is consistently higher than output-side overlap in model averages and is higher in the vast majority of individual-matrix comparisons; and reversals are concentrated in query/key projections and embedding tables. Across the fifteen runs, input-side energy overlap exceeds output-side overlap in 93.58% of individual-matrix comparisons with the initial subspace and 92.98% of comparisons with the subspace from the previous measurement. As an application of our findings, we propose a memory-efficient optimizer, Overlap-Guided Gradient Low-Rank Projection (), which always projects gradients onto input-side subspaces. It achieves the lowest or near-lowest validation loss among the compared projection rules across the different model–rank configurations. Our code is available at https://anonymous.4open.science/r/gradient_subspace_dynamics-3DD6/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Gradient Dynamics, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XdF1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the dominant directions of the gradient after singular value decomposition during supervised fine-tuning, and ask whether these directions change depending based on energy overlap for input/output side (right vs. left singular vectors). The authors measure this by looking at the energy overlap and comparing the current gradient's top kk singular directions with an earlier subspace. The experiments show that input-side overlap exceeds output-side significantly and the asymmetry is observed at all model-average measurement points. Input-side is also more stable than output-side and based on this insight, the authors propose OG-GaLore, which always projects on the input side.

Critical strengths

The paper demonstrates the asymmetry across many SFT settings, to show that input-side gradient subspaces keep more energy than output-side subspaces at all 223 model-level measurement points. This is evaluated across eight datasets and model families. The observed asymmetry provides a useful empirical finding that can guide the design of subspace-based memory-efficient optimizers.

Critical weaknesses
  1. The paper provides limited explanation of the observed asymmetry beyond an empirical account. For example, activation geometry is a natural candidate explanation for input-side persistence, since the gradient’s input-side directions depend jointly on input activations and backpropagated errors. However, the errors can shift which directions dominate, and the activation subspace itself may drift, so this relationship alone does not establish temporal stability. The paper does not directly investigate these factors, limiting its explanatory contribution beyond documenting the pattern.

  2. The proposed optimizer is an existing GaLore setting, so presenting it as a new memory-efficient optimizer overstates the contribution. The empirical support is also insufficient to demonstrate a strong practical advantage. The comparisons only cover one dataset, reports validation loss only with no downstream task evaluation.

Questions for authors
  1. Is OG-GaLore the same as using projection right side, based on GaLore? If it's identical, then OG-GaLore should be presented as an evaluation of that config rather than a new optimizer.

  2. How much of the input-side subspace is explained by the principal subspace of the layer's input activations?

4 comments
AuthorsOpenReview ↗

Reply to weakness 1 & question 2 (1/2)

For weakness 1: The paper provides limited explanation of the observed asymmetry beyond an empirical account. For example, activation geometry is a natural candidate explanation for input-side persistence, since the gradient’s input-side directions depend jointly on input activations and backpropagated errors. However, the errors can shift which directions dominate, and the activation subspace itself may drift, so this relationship alone does not establish temporal stability. The paper does not directly investigate these factors, limiting its explanatory contribution beyond documenting the pattern.

& For question 2: How much of the input-side subspace is explained by the principal subspace of the layer’s input activations?

A1 / Q2: Thank you for your constructive comments. Our additional experiments show that activation subspaces exhibit greater temporal stability than backpropagated-error subspaces, and activation directions from the previous recorded checkpoint retain more subsequent gradient energy than the corresponding error directions. This pattern holds across diagnostic ranks k=8,16,32k=8,16,32 in the completed Qwen3-1.7B–PrOntoQA runs with both AdamW [1] and Muon [2].

For example, at step 280 and k=32k=32, activation directions from step 240 retain 76.30% and 77.21% of the complete current gradient energy under AdamW and Muon, respectively, whereas error directions from the same checkpoint retain 36.15% and 23.25%.

(1) Gradient factorization and matrix shapes.

For a linear layer Yt=WtXtY_t=W_tX_t, let Wt∈Rdout×dinW_t\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}, Xt∈Rdin×NX_t\in\mathbb{R}^{d_{\mathrm{in}}\times N}, and Δt=∂L/∂Yt∈Rdout×N\Delta_t=\partial\mathcal{L}/\partial Y_t\in\mathbb{R}^{d_{\mathrm{out}}\times N}, where NN is the number of token positions included in the batch and loss-normalization factors are absorbed into Δt\Delta_t. The gradient is

Gt=ΔtXt⊤=UtΣtVt⊤∈Rdout×din.G_t=\Delta_tX_t^{\top}=U_t\Sigma_tV_t^{\top}\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}.

Its input-side and output-side Gram matrices satisfy

Gt⊤Gt=XtΔt⊤ΔtXt⊤,G_t^{\top}G_t=X_t\Delta_t^{\top}\Delta_tX_t^{\top},

GtGt⊤=ΔtXt⊤XtΔt⊤.G_tG_t^{\top}=\Delta_tX_t^{\top}X_t\Delta_t^{\top}.

These identities characterize how input activations and backpropagated errors jointly determine the gradient subspaces. Our experiments measure the temporal stability of their principal directions and how much current gradient energy these directions retain.

(2) Temporal energy overlap and full-gradient energy retention.

We follow the energy-overlap notation in the manuscript. Let QX,t∈Rdin×kQ_{X,t}\in\mathbb{R}^{d_{\mathrm{in}}\times k} and QΔ,t∈Rdout×kQ_{\Delta,t}\in\mathbb{R}^{d_{\mathrm{out}}\times k} contain the leading kk left singular vectors of XtX_t and Δt\Delta_t, respectively. Equivalently, these are the leading eigenvectors of XtXt⊤X_tX_t^{\top} and ΔtΔt⊤\Delta_t\Delta_t^{\top}.

For C∈{X,Δ}C\in\{X,\Delta\}, define the normalized top-kk energy matrix as

DC,t=diag⁡(aC,t,1,…,aC,t,k),aC,t,i=σC,t,i2∑j=1kσC,t,j2.D_{C,t}=\operatorname{diag}(a_{C,t,1},\ldots,a_{C,t,k}),\qquad a_{C,t,i}=\frac{\sigma_{C,t,i}^{2}}{\sum_{j=1}^{k}\sigma_{C,t,j}^{2}}.

The temporal energy overlap is

\mathcal{O}_{C}^{E}(s,t)=\left\\|Q_{C,s}^{\top}Q_{C,t}D_{C,t}^{1/2}\right\\|_{F}^{2}.

For local comparisons, ss denotes the previous recorded checkpoint, following the manuscript.

We additionally measure how much of the complete current gradient energy is retained by an activation or error reference subspace. Let Gt=UtΣtVt⊤G_t=U_t\Sigma_tV_t^{\top} be the full thin SVD, with Ut∈Rdout×rU_t\in\mathbb{R}^{d_{\mathrm{out}}\times r}, Vt∈Rdin×rV_t\in\mathbb{R}^{d_{\mathrm{in}}\times r}, and r=min⁡(dout,din)r=\min(d_{\mathrm{out}},d_{\mathrm{in}}). Define

DG,tfull=diag⁡(bt,1,…,bt,r),bt,i=σG,t,i2∑j=1rσG,t,j2.D_{G,t}^{\mathrm{full}}=\operatorname{diag}(b_{t,1},\ldots,b_{t,r}),\qquad b_{t,i}=\frac{\sigma_{G,t,i}^{2}}{\sum_{j=1}^{r}\sigma_{G,t,j}^{2}}.

The full-gradient energy retention is

Rin(s,t)=∥QX,s⊤Vt(DG,tfull)1/2∥F2,R_{\mathrm{in}}(s,t)=\left\|Q_{X,s}^{\top}V_t\left(D_{G,t}^{\mathrm{full}}\right)^{1/2}\right\|_{F}^{2},

Rout(s,t)=∥QΔ,s⊤Ut(DG,tfull)1/2∥F2.R_{\mathrm{out}}(s,t)=\left\|Q_{\Delta,s}^{\top}U_t\left(D_{G,t}^{\mathrm{full}}\right)^{1/2}\right\|_{F}^{2}.

Both products inside the Frobenius norms have shape k×rk\times r: the reference contains kk directions, while the current gradient retains its full spectrum. We report same-checkpoint retention with s=ts=t and local retention with ss equal to the previous recorded checkpoint.

AuthorsOpenReview ↗

Reply to weakness 1 & question 2 (2/2)

(3) Results across training steps, diagnostic ranks, and optimizers.

The results cover 196 parameter matrices across 28 transformer layers, including the Q/K/V/O and gate/up/down projections. Measurements are taken at steps 1, 40, 80, 120, 160, 200, 240, and 280. Local comparisons use step 1 as the reference for step 40 and the preceding recorded checkpoint thereafter.

All values are percentages, averaged equally over the 196 matrices. These are per-checkpoint matrix averages. The four carrier-to-gradient columns report full-gradient energy retention.

AdamW — Qwen3-1.7B, PrOntoQA

StepkkLocal OXE\mathcal O_X^ELocal OΔE\mathcal O_\Delta^ESame-checkpoint Xt→GtX_t\to G_tSame-checkpoint Δt→Gt\Delta_t\to G_tLocal Xs→GtX_s\to G_tLocal Δs→Gt\Delta_s\to G_t
40892.3627.9361.0543.1355.5218.21
401692.0635.2168.8955.9762.1524.83
403291.8542.3176.3267.5768.6832.72
80898.7534.7663.3164.9762.5824.73
801698.2841.9570.9675.9469.9834.10
803298.2549.5078.0984.9577.0443.58
120899.3032.1763.0763.1662.6123.99
1201698.9338.0270.4873.8569.6831.24
1203298.8244.6177.4782.9976.5239.12
160899.5334.4960.3375.1560.0427.59
1601699.2441.7268.2883.7667.7935.50
1603299.0847.7175.3991.2475.1243.00
200899.7017.1860.0164.4859.9613.51
2001699.4121.2567.2375.6267.4518.59
2003299.3226.1676.0684.6875.9424.35
240899.7122.1660.9565.8661.0315.94
2401699.3828.3568.3676.2868.6022.82
2403299.2636.2976.5184.8276.3431.09
280899.6728.7962.4864.0662.2221.48
2801699.3134.2169.7974.9869.5328.36
2803299.1740.6576.5984.1976.3036.15

Muon — Qwen3-1.7B, PrOntoQA

StepkkLocal OXE\mathcal O_X^ELocal OΔE\mathcal O_\Delta^ESame-checkpoint Xt→GtX_t\to G_tSame-checkpoint Δt→Gt\Delta_t\to G_tLocal Xs→GtX_s\to G_tLocal Δs→Gt\Delta_s\to G_t
40886.7926.4861.8452.4053.9515.34
401686.5033.1469.6463.3660.1122.62
403286.5340.6277.4373.2766.9031.67
80893.1931.9961.1469.0857.3923.89
801692.8538.9068.8779.8164.4432.25
803293.0145.8877.0088.3471.5540.29
120895.9727.8067.7557.8864.6922.07
1201695.7733.1874.2769.4870.9629.91
1203295.9139.5181.0979.9777.6138.39
160897.9233.2765.4363.5463.9528.16
1601697.7939.1372.6874.8571.0435.87
1603297.6946.5680.3683.7578.6045.11
200899.0822.1563.3476.7462.9418.43
2001698.9830.7870.0486.1569.5127.76
2003298.9340.0177.9393.0577.1537.67
240899.519.9259.0773.7258.978.61
2401699.2814.0967.0482.6167.4012.64
2403299.1719.4675.5791.0775.6717.89
280899.5014.1159.7874.3259.1511.03
2801699.2719.2968.8383.2367.7316.06
2803299.1026.4778.2991.6077.2123.25

(4) Interpretation of the results.

Activation directions remain more stable across checkpoints. At every reported step and rank under both optimizers, local activation energy overlap exceeds local error energy overlap. At step 280 and k=32k=32, these values are 99.17% versus 40.65% under AdamW and 99.10% versus 26.47% under Muon.

Previously recorded activation directions retain nearly the same gradient energy as current activation directions. At step 280 and k=32k=32, same-checkpoint activation directions retain 76.59% and 78.29% of the full gradient energy under AdamW and Muon. Using activation directions from step 240 retains 76.30% and 77.21%, respectively.

Error directions show strong same-checkpoint alignment but substantially weaker temporal persistence. At step 280 and k=32k=32, current error directions retain 84.19% under AdamW and 91.60% under Muon. Using error directions from step 240 reduces retention to 36.15% and 23.25%, respectively.

These results provide empirical evidence connecting the observed asymmetry to the temporal persistence of activation directions and their continued ability to capture subsequent gradient energy.

References:

[1] Decoupled Weight Decay Regularization. In ICLR 2019.

[2] Muon: An Optimizer for Hidden Layers in Neural Networks. Technical blog, 2024.

AuthorsOpenReview ↗

Reply to weakness 2 & question 1 (1/2)

For weakness 2: The proposed optimizer is an existing GaLore setting, so presenting it as a new memory-efficient optimizer overstates the contribution. The empirical support is also insufficient to demonstrate a strong practical advantage. The comparisons only cover one dataset, reports validation loss only with no downstream task evaluation.

& For question 1: Is OG-Galore the same as using projection right side, based on GaLore? If it's identical, then OG-Galore should be presented as an evaluation of that config rather than a new optimizer.

A2 / Q1: Thank you for your constructive comments.

(1) Clarification of our contribution.

OG-GaLore uses GaLore’s [1] existing right projection option. We name it Overlap-Guided Gradient Low-Rank Projection (OG-GaLore) to emphasize our motivation: allowing gradients to retain more of their original information. We consider this an innovative application arising naturally from our main empirical findings.

The manuscript is intended to present OG-GaLore as an application of our findings, rather than a new optimizer. Nevertheless, we recognize that phrases such as “we propose a memory-efficient optimizer” in the abstract and “we introduce a memory-efficient optimizer” in the introduction may cause misunderstanding. We have revised these two statements to explicitly describe OG-GaLore as an optimization strategy for memory-efficient optimizers such as GaLore.

GaLore [1] and its variants, including SubTrack++ [2], GoLore [6], and LDAdam [7], all use the shape-based std strategy. In contrast, OG-GaLore consistently uses the input-side right strategy across all target matrices.

**More importantly, although right is available in the code, GaLore [1], SubTrack++ [2], GoLore [6], and LDAdam [7] have not used it as a uniform projection policy across all target matrices, instead adopting the shape-based std strategy. They neither discuss this uniform input-side policy nor report experiments comparing it with alternative projection-side selection strategies. They also do not compare the relative merits of the four strategies—std, reverse_std, left, and right. Our in-depth investigation of these strategies therefore constitutes a novel contribution in its own right. Even though right already exists in GaLore’s code, showing that it achieves the best performance in most settings and approaches the best performance in a small number of settings is itself a contribution. Using right as the default can reduce the projection-side search cost by approximately 75%.

To the best of our knowledge, we are the first to systematically compare four projection-side selection strategies and demonstrate the overall advantage of uniform right projection in LLM supervised fine-tuning. We connect this advantage to the temporal input–output asymmetry of gradient subspaces.

Our main empirical contribution is to reveal the temporal dynamics of gradient subspaces and their input/output asymmetry: historical input-side subspaces generally retain more of the current gradient’s dominant energy than historical output-side subspaces. OG-GaLore, a projection-strategy improvement for GaLore, is a natural application of this main finding. Our novelty therefore extends well beyond proposing a strategy for improving an optimizer. We therefore believe that our work still has substantial novelty.

(2) Additional downstream evaluations across models, tasks, ranks, and seeds.

We expand the evaluation beyond validation loss. We evaluate downstream accuracy on PrOntoQA [3] and ProsQA [4], comparing four projection strategies within the same GaLore framework (std, reverse_std, left, and right).

For Qwen3-1.7B and Llama-3.2-1B-Instruct, we evaluate projection ranks r=4,8,16,32r=4,8,16,32. Values below are accuracy percentages, reported as mean ± sample standard deviation over three seeds. The projection rank rr is the optimizer’s rank, distinct from the diagnostic rank kk used in the subspace analysis. Bold indicates the highest mean accuracy in each row.

Qwen3-1.7B — three seeds

DatasetRankstdreverse_stdleftright
PrOntoQA431.87 ± 0.8142.47 ± 0.8134.60 ± 1.4449.00 ± 1.11
PrOntoQA829.53 ± 1.1444.67 ± 1.5032.40 ± 1.1151.80 ± 0.92
PrOntoQA1628.47 ± 1.1047.53 ± 2.1932.00 ± 1.7854.80 ± 1.39
PrOntoQA3222.53 ± 0.9552.07 ± 1.8131.00 ± 0.3555.53 ± 1.75
ProsQA440.33 ± 0.8155.93 ± 0.3132.00 ± 1.2058.07 ± 0.61
ProsQA843.07 ± 0.7055.80 ± 1.7831.73 ± 0.5058.87 ± 0.12
ProsQA1644.33 ± 0.3158.00 ± 1.4434.67 ± 0.8360.33 ± 1.14
ProsQA3248.20 ± 0.7258.40 ± 0.5338.93 ± 0.6462.07 ± 1.21
AuthorsOpenReview ↗

Reply to weakness 2 & question 1 (2/2)

Llama-3.2-1B-Instruct — three seeds

DatasetRankstdreverse_stdleftright
PrOntoQA41.93 ± 0.4232.87 ± 4.281.60 ± 0.4033.27 ± 3.33
PrOntoQA83.53 ± 0.6437.67 ± 3.902.47 ± 0.8337.53 ± 3.10
PrOntoQA1612.20 ± 0.9241.93 ± 1.949.27 ± 0.5844.60 ± 2.96
PrOntoQA3230.93 ± 1.8047.40 ± 1.9325.60 ± 1.0648.93 ± 0.83
ProsQA453.20 ± 1.2257.87 ± 0.9052.40 ± 1.2559.80 ± 0.53
ProsQA854.20 ± 1.4058.60 ± 0.9253.33 ± 1.2259.40 ± 0.20
ProsQA1656.47 ± 0.8161.07 ± 1.6855.80 ± 1.3161.53 ± 1.85
ProsQA3258.80 ± 1.6062.67 ± 0.3159.27 ± 0.6163.27 ± 1.01

Across these two models, fixed right projection outperforms std in all 16 model–task–rank settings and achieves the highest mean accuracy in 15 of the 16 settings. The exception is Llama-3.2-1B-Instruct on PrOntoQA at rank 8, where reverse_std exceeds right by 0.14 percentage points.

Qwen3-0.6B — three seeds

We additionally evaluate the four projection strategies on Qwen3-0.6B at projection ranks r=16,32r=16,32. Values are accuracy percentages, reported as mean ± sample standard deviation over three seeds. Bold indicates the highest mean accuracy in each row.

DatasetRankstdreverse_stdleftright
PrOntoQA1617.53 ± 0.6117.60 ± 1.5128.07 ± 0.9022.27 ± 2.05
PrOntoQA3226.27 ± 0.4213.53 ± 1.7023.53 ± 1.4529.67 ± 0.50
ProsQA1659.40 ± 1.0030.00 ± 1.5935.33 ± 0.9960.40 ± 0.72
ProsQA3261.33 ± 1.4036.27 ± 0.5033.33 ± 1.3361.67 ± 0.61

right performs best in 18 of the 20 settings. In the remaining two, it trails the best strategy by 0.14 and 5.80 percentage points, respectively. The largest accuracy gaps from the best strategy across the 20 settings are 34.14, 30.40, and 35.33 percentage points for std, reverse_std, and left, respectively, compared with 5.80 percentage points for right. Thus, right is the best-performing overall strategy in these experiments.

References:

[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. In ICML 2024.

[2] SubTrack++: Gradient Subspace Tracking for Scalable LLM Training. In NeurIPS 2025.

[3] Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. In ICLR 2023.

[4] Training Large Language Models to Reason in a Continuous Latent Space. In COLM 2025.

[6] Subspace Optimization for Large Language Models with Convergence Guarantees. In ICML 2025.

[7] LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. In ICLR 2025.

Reviewer_v8W2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies gradient subspace dynamics during supervised fine-tuning of LLMs. The authors analyze the energy overlap between current and historical gradient subspaces across different models and datasets. They find that input-side subspaces generally retain more gradient energy than output-side subspaces, with exceptions mainly in query/key projections and embedding layers. Based on these findings, they propose OG-GaLore, which always projects gradients onto input-side subspaces. Experiments show that OG-GaLore achieves competitive validation loss compared with other GaLore projection strategies.

Critical strengths

The paper provides an interesting empirical study of gradient dynamics during LLM fine-tuning. The experiments cover multiple model families and eight datasets, making the observed input–output asymmetry interesting and potentially useful. I also appreciate the detailed analysis of different parameter matrices and the exceptions in query/key projections. The proposed OG-GaLore method provides a simple practical application of these findings.

Critical weaknesses

1. Limited generality of the empirical findings. Most experiments use a fixed diagnostic rank of 8 and only one training run per model–dataset setting. All subspace analyses are also based on AdamW training. Although the appendix includes additional rank-16 experiments, it remains unclear whether the observed asymmetry holds across different ranks, optimizers, and training settings.

2. Limited novelty of the proposed optimization method. OG-GaLore always selects input-side projection rather than dynamically using the measured energy overlap. The paper also shows that some query/key projections and embedding layers favor the output side. It is unclear whether a more adaptive or layer-specific selection strategy would perform better. The proposed method therefore seems relatively simple compared with the broader findings of the paper.

3. Insufficient validation of OG-GaLore. The optimizer experiments are limited to OpenThoughts fine-tuning with a single random seed and a short training budget. The results mainly report validation loss, without downstream task performance or multi-seed evaluations. The paper also lacks direct measurements of memory usage and runtime for the different projection strategies. This makes it difficult to assess the robustness and practical efficiency of OG-GaLore.

Questions for authors
  1. Does the observed input–output asymmetry remain consistent across different projection ranks, optimizers, and random seeds? Can the authors provide additional evidence beyond the current AdamW settings?

  2. Since some query/key projections and embedding layers show higher output-side overlap, would a layer-specific or dynamically selected projection strategy outperform the always-input approach used by OG-GaLore?

  3. Can the authors provide multi-seed experiments and additional downstream task evaluations for OG-GaLore? It would also be helpful to report actual GPU memory usage and training time compared with standard GaLore.

8 comments
AuthorsOpenReview ↗

Reply to weakness 1 & question 1

For weakness 1: Limited generality of the empirical findings. Most experiments use a fixed diagnostic rank of 8 and only one training run per model–dataset setting. All subspace analyses are also based on AdamW training. Although the appendix includes additional rank-16 experiments, it remains unclear whether the observed asymmetry holds across different ranks, optimizers, and training settings.

& For question 1: Does the observed input–output asymmetry remain consistent across different projection ranks, optimizers, and random seeds? Can the authors provide additional evidence beyond the current AdamW settings?

A1 / Q1: Thank you for your constructive comments. Our additional experiments show higher model-level input-side than output-side energy overlap at the final recorded checkpoint across both AdamW [1] and Muon [2], two models, two datasets, and three seeds.

(1) Direct gradient analysis across optimizers and seeds.

We conduct 24 full-parameter training runs covering Qwen3-1.7B and Llama-3.2-1B-Instruct, PrOntoQA [3] and ProsQA [4], AdamW [1] and Muon [2], and three seeds.

The following table reports gradient energy overlap at diagnostic rank k=16k=16 and step 280. Initial-reference comparisons use the first recorded gradient, whereas local comparisons use the previous recorded checkpoint. Within each run, we use the manuscript’s model-level energy-weighted aggregation. We then report mean ± sample standard deviation across three seeds. All values are percentages.

ModelDatasetOptimizerInitial inputInitial outputLocal inputLocal output
Qwen3-1.7BPrOntoQAAdamW63.88 ± 0.5114.29 ± 2.3075.27 ± 1.2716.37 ± 0.79
Qwen3-1.7BPrOntoQAMuon58.35 ± 5.0713.98 ± 4.1678.20 ± 10.0125.78 ± 10.12
Qwen3-1.7BProsQAAdamW57.57 ± 2.0410.81 ± 1.7384.04 ± 1.0826.17 ± 1.86
Qwen3-1.7BProsQAMuon50.07 ± 1.359.16 ± 1.5483.19 ± 0.4227.21 ± 1.07
Llama-3.2-1B-InstructPrOntoQAAdamW36.87 ± 0.5513.35 ± 1.5966.99 ± 2.0532.32 ± 3.17
Llama-3.2-1B-InstructPrOntoQAMuon32.54 ± 2.4912.70 ± 0.1876.21 ± 2.8836.99 ± 7.57
Llama-3.2-1B-InstructProsQAAdamW33.95 ± 0.2214.32 ± 0.6373.62 ± 1.1933.36 ± 0.47
Llama-3.2-1B-InstructProsQAMuon29.68 ± 0.8011.23 ± 0.9075.69 ± 1.5332.05 ± 1.31

In all 24 individual runs, input-side overlap exceeds output-side overlap at this checkpoint for both reference choices.

(2) Comparisons across diagnostic ranks.

We distinguish diagnostic rank kk from optimizer projection rank rr. In an additional full-parameter AdamW cohort, we evaluate double-energy overlap at k=8,16,32k=8,16,32, weighting both historical and current directions by their squared singular values. This is a separate robustness metric, not a replacement for the original current-energy overlap.

The following results combine all seven valid local checkpoint comparisons from steps 40 through 280 and give equal weight to each matrix–checkpoint pair. They cover V/O and gate/up/down matrices; Q/K are separated because of their documented exceptions, and embeddings/lm_head were not collected in this cohort. All values are percentages.

ModelDatasetkkInput meanOutput meanInput higher
Qwen3-0.6BPrOntoQA880.416.7499.80
Qwen3-0.6BPrOntoQA1680.676.9899.80
Qwen3-0.6BPrOntoQA3280.827.1099.80
Qwen3-0.6BProsQA885.994.3399.80
Qwen3-0.6BProsQA1686.154.7299.80
Qwen3-0.6BProsQA3286.294.9599.80
Qwen3-1.7BPrOntoQA876.265.7099.80
Qwen3-1.7BPrOntoQA1676.455.8799.80
Qwen3-1.7BPrOntoQA3276.565.9699.80
Qwen3-1.7BProsQA884.964.8699.39
Qwen3-1.7BProsQA1685.125.2499.39
Qwen3-1.7BProsQA3285.265.4599.39
Llama-3.2-1B-InstructPrOntoQA857.188.3296.25
Llama-3.2-1B-InstructPrOntoQA1657.648.6796.25
Llama-3.2-1B-InstructPrOntoQA3257.898.8396.25
Llama-3.2-1B-InstructProsQA873.4710.2091.79
Llama-3.2-1B-InstructProsQA1673.8110.8791.79
Llama-3.2-1B-InstructProsQA3274.1311.2491.79

The input-side advantage persists across these three diagnostic ranks in the reported module types.

References:

[1] Decoupled Weight Decay Regularization. In ICLR 2019.

[2] Muon: An Optimizer for Hidden Layers in Neural Networks. Technical report, 2024.

[3] Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. In ICLR 2023.

[4] Training Large Language Models to Reason in a Continuous Latent Space. In COLM 2025.

AuthorsOpenReview ↗

Reply to weakness 2 & question 2 (1/2)

For weakness 2: Limited novelty of the proposed optimization method. OG-GaLore always selects input-side projection rather than dynamically using the measured energy overlap. The paper also shows that some query/key projections and embedding layers favor the output side. It is unclear whether a more adaptive or layer-specific selection strategy would perform better. The proposed method therefore seems relatively simple compared with the broader findings of the paper.

& For question 2: Since some query/key projections and embedding layers show higher output-side overlap, would a layer-specific or dynamically selected projection strategy outperform the always-input approach used by OG-GaLore?

A2 / Q2: Thank you for your constructive comments.

(1) Clarification of our contribution.

OG-GaLore uses GaLore’s [1] existing right projection option. We name it Overlap-Guided Gradient Low-Rank Projection (OG-GaLore) to emphasize our motivation: allowing gradients to retain more of their original information. We consider this an innovative application arising naturally from our main empirical findings.

GaLore [1] and its variants, including SubTrack++ [2], GoLore [6], and LDAdam [7], all use the shape-based std strategy. In contrast, OG-GaLore consistently uses the input-side right strategy across all target matrices.

Although fixed right projection is available in the code of GaLore and its subsequent variants, these works—including GaLore [1], SubTrack++ [2], GoLore [3], and LDAdam [4]—do not systematically investigate the temporal advantage of input-side gradient subspaces. More importantly, these works neither discuss uniform right projection across all target matrices as a projection policy nor report experiments comparing it with other projection-side selection rules. They also do not compare the relative merits of the four strategies—std, reverse_std, left, and right. Our in-depth investigation of these strategies therefore constitutes a novel contribution in its own right.

To the best of our knowledge, we are the first to systematically compare four projection-side selection strategies and demonstrate the overall advantage of uniform right projection in LLM supervised fine-tuning.

Our main empirical contribution is to reveal the temporal dynamics of gradient subspaces and their input/output asymmetry. OG-GaLore, a projection-strategy improvement for GaLore, is a natural application of this main finding. Our novelty therefore extends well beyond proposing a strategy for improving an optimizer. We therefore believe that our work still has substantial novelty.

(2) Additional experiments on per-matrix projection-side selection.

We compare fixed input-side projection with overlap-based per-matrix selection on Llama-3.2-1B-Instruct, covering PrOntoQA and ProsQA, projection ranks r=4,8,16,32r=4,8,16,32, and three seeds, for a total of 48 completed training runs.

Both policies undergo the same short full-parameter AdamW trial using the original training-data order. We measure temporal gradient energy overlap across the first update that actually changes the parameters, using the gradients immediately before that update and on the following effective training batch. The overlap-based policy selects the side with higher current-energy overlap independently for each target matrix; the fixed-input control always selects right.

After the trial, we restore the initial model parameters, formal optimizer and scheduler states, random-number-generator states, and training-data order, discard the trial optimizer state, and begin formal training from the initial weights. Thus, trial updates do not carry over into formal training. During formal training, each matrix uses its selected input-side or output-side projection, and its projection basis is recomputed from the current gradient every 40 optimizer updates, following GaLore. The specific projection subspace therefore evolves during training. Embeddings and the language-model head are not projected and retain full-dimensional AdamW updates.

All accuracy values are percentages, reported as mean ± sample standard deviation across three seeds. The difference is overlap-based selection minus fixed right, computed within each seed and reported as mean ± sample standard deviation in percentage points.

DatasetRankFixed rightOverlap-based selectionPaired difference (pp)
PrOntoQA433.40 ± 3.3334.47 ± 2.66+1.07 ± 5.08
PrOntoQA840.20 ± 1.9137.87 ± 2.69−2.33 ± 3.65
PrOntoQA1644.07 ± 0.9545.87 ± 0.92+1.80 ± 1.71
PrOntoQA3247.93 ± 1.4548.60 ± 1.11+0.67 ± 2.50
ProsQA459.60 ± 2.4258.93 ± 1.14−0.67 ± 1.80
ProsQA858.93 ± 0.8158.53 ± 0.42−0.40 ± 0.60
ProsQA1662.53 ± 0.9961.80 ± 1.71−0.73 ± 0.99
ProsQA3263.47 ± 0.9063.47 ± 0.500.00 ± 0.69
AuthorsOpenReview ↗

Reply to weakness 2 & question 2 (2/2)

Overlap-based per-matrix selection does not consistently outperform fixed input-side projection in this experiment. It improves mean accuracy in three of the eight task–rank settings, lowers it in four, and gives the same mean accuracy in one. On PrOntoQA at rank 16, the mean improvement is 1.80 percentage points; on PrOntoQA at rank 8, the mean difference is −2.33 percentage points.

These results show that per-matrix selection based on the initial trial can improve performance in some settings, but does not provide a consistent downstream advantage over fixed input-side projection. In future work, we will further investigate strategies that exploit the differences in query/key projections and embedding layers to improve performance.

References:

[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. In ICML 2024.

[2] SubTrack++: Gradient Subspace Tracking for Scalable LLM Training. In NeurIPS 2025.

[3] Subspace Optimization for Large Language Models with Convergence Guarantees. In ICML 2025.

[4] LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. In ICLR 2025.

Reviewer_dB7WOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper is an analysis-based paper that investigates the input-output asymmetry present in gradient matrices for LLM training. The authors conducted a comprehensive study across various datasets and models, tracking the energy overlap during the training process and comparing between the input side and output side. The key observation is that input-side energy overlap is generally higher than that of the output side. Based on this insight, the authors introduce OG-GaLore, a simple update to GaLore that always uses the input-side projection, outperforming GaLore on validation loss in most of the test scenarios.

Critical strengths
  • This paper conducts a comprehensive and in-depth analysis of an important pattern.
  • The proposed method is well-supported by observed evidence, and outperforms the baseline while remaining simple.
Critical weaknesses
  • More justification is needed as to why "energy overlap" and "weighted mean energy overlap" are used as the main comparison metrics. This is important as it affects the overall technical soundness.
Questions for authors
  • For layers with different input/output dimensions, does this dimension difference itself bias the energy overlap patterns?
  • The energy overlap formulation on its own does not seem to take into consideration of the portion of information lost by choosing the top-k singular components, which could be different for different parts of the training. Would this affect the observations?
  • The energy overlap always uses singular values of the latter measurement, would its soundness be impacted if the distribution of singular value magnitudes is substantially different for the historical measurement?
  • Is layer dimensionality on its own a factor that affects the Frobenius norm in equation (4)?
  • Do authors plan to improve the presentation of figures 1, 2 and table 1?
  • Do authors plan to compare/combine OG-GaLore with newer gradient projection works (or discuss OG-GaLore's potential to be combined with them), such as the works cited in Section 2, "Memory-efficient gradient projection"?
10 comments
AuthorsOpenReview ↗

Reply to weakness 1

For weakness 1: More justification is needed as to why “energy overlap” and “weighted mean energy overlap” are used as the main comparison metrics. This is important as it affects the overall technical soundness.

A1: Thank you for your constructive comment. We use energy overlap to account for the unequal contributions of singular directions within each gradient matrix, and weighted mean energy overlap to summarize the overall trend across all measured modules of an LLM.

(1) Energy overlap accounts for the different contributions of singular directions to the gradient matrix.

For a gradient matrix Gt=∑iσt,iut,ivt,i⊤G_t=\sum_i\sigma_{t,i}u_{t,i}v_{t,i}^{\top}, each singular component contributes σt,i2\sigma_{t,i}^2 to its squared Frobenius norm. An unweighted subspace overlap treats all retained directions equally, even when their energy contributions differ substantially. Equation (4) in the manuscript accounts for these differences:

=\left\|(Q_{\ell,m,s}^{z})^\top Q_{\ell,m,t}^{z}D_{\ell,m,t}^{1/2}\right\|_F^2,$$ Here, $s<t$ denote the historical and current measurement steps, respectively. **We use the initial measurement as the historical reference for initial-reference comparisons and the previous recorded measurement for local-reference comparisons.** The indices $\ell,m$ identify the layer and module, and $z$ denotes the input or output side. The columns of $Q_{\ell,m,\tau}^{z}$ are the top-$k$ singular vectors on side $z$ at measurement $\tau$, and $$D_{\ell,m,t} =\operatorname{diag}(a_{\ell,m,t,1},\ldots,a_{\ell,m,t,k}), \qquad a_{\ell,m,t,i} =\frac{\sigma_{\ell,m,t,i}^{2}} {\sum_{j=1}^{k}\sigma_{\ell,m,t,j}^{2}}.$$ **Expanding the squared Frobenius norm in Equation (4) column by column gives the equivalent expression:** $$\mathcal O_{\ell,m,z}^{E}(s,t) =\sum_{i=1}^{k}a_{\ell,m,t,i} \left\|(Q_{\ell,m,s}^{z})^\top q_{\ell,m,t,i}^{z}\right\|_2^2.$$ Thus, Equation (4) weights the retention of each current singular direction by its share of the current top-$k$ energy. **For example**, suppose two retained singular directions contribute 90% and 10% of the current top-$k$ energy. If one historical subspace preserves only the first direction and another preserves only the second, their unweighted overlaps are both 0.5. However, their energy overlaps are **0.9 and 0.1**, respectively. Energy overlap distinguishes these cases because preserving the first direction retains much more of the gradient’s dominant energy. Thus, energy overlap measures alignment according to each direction’s contribution to the current gradient’s top-$k$ energy. Here, importance refers specifically to gradient-energy contribution; it does not imply that singular values alone determine downstream performance. **(2) Weighted mean energy overlap summarizes the overall trend across the LLM’s measured modules.** An LLM contains many parameter matrices across layers and module types. Individual-matrix measurements reveal local behavior, while a model-level summary makes it possible to examine the overall trend in input-side energy overlap and output-side energy overlap during training. We therefore aggregate the energy overlaps across all measured matrices: $$\overline O_z^E(s,t) =\frac{\sum_{(\ell,m)\in\mathcal M}w_{\ell,m}(t)O_{\ell,m,z}^E(s,t)} {\sum_{(\ell,m)\in\mathcal M}w_{\ell,m}(t)}, \qquad w_{\ell,m}(t)=N_{\ell,m}E_{\ell,m}^{(k)}(t).$$ Here, $\mathcal M$ denotes the set of all measured parameter matrices. The parameter-count factor $N_{\ell,m}$ accounts for differences in block size, while the top-$k$ energy fraction $E_{\ell,m}^{(k)}$ accounts for how much of each matrix’s gradient the selected components represent. **The purpose is to characterize the overall trend across modules**, complemented by per-matrix results that expose variation and exceptions. This statistic is a weighted average of per-matrix overlap scores, rather than the fraction of the model’s summed gradient energy retained. **(3) The observed trend also holds without the model-level weighting.** To check whether the aggregation determines the finding, we recomputed equal-weight statistics from the existing fifteen $k=8$ runs. Across 45,827 paired matrix–step observations, the results are: | Comparison | Mean input-side energy overlap | Mean output-side energy overlap | Pairs with input > output | Pairs with input > output excluding Q/K/embedding | |:---|:---|:---|:---|:---| | Initial reference | 0.6733 | 0.1225 | 93.58% | 99.97% | | Local reference | 0.7744 | 0.1501 | 92.98% | 99.98% | The last column reports the proportion of pairs with input-side energy overlap exceeding output-side energy overlap after excluding query/key projections and embedding matrices. The input-side advantage therefore persists under equal weighting and is present in the large majority of individual comparisons.
AuthorsOpenReview ↗

Reply to question 1

Q1. Does the difference between input and output dimensions bias the overlap patterns?

For question 1: For layers with different input/output dimensions, does this dimension difference itself bias the energy overlap patterns?

A1: Thank you for this important question. The observed energy overlap asymmetry cannot be explained by the dimension difference alone. Below, we provide specific matrix examples covering m < n, m = n, and m > n from additional experiments with three models on PrOntoQA and ProsQA. Here, m and n denote the output and input dimensions, respectively.

All examples use layer 8 and diagnostic rank k = 16. We report the local energy overlap at step 280, using step 240 as the reference.

Shape regimeModelMatrix within model.layers.8Matrix shape: m × nPrOntoQA (input / output)ProsQA (input / output)
m < nLlama-3.2-1B-Instructmlp.down_proj.weight2048 × 81920.5679 / 0.16930.7474 / 0.2601
m < nQwen3-0.6Bmlp.down_proj.weight1024 × 30720.6476 / 0.19420.6939 / 0.2514
m < nQwen3-1.7Bmlp.down_proj.weight2048 × 61440.6914 / 0.13090.7470 / 0.1933
m = nLlama-3.2-1B-Instructself_attn.o_proj.weight2048 × 20480.6952 / 0.22530.8494 / 0.2020
m = nQwen3-0.6Bself_attn.v_proj.weight1024 × 10240.8866 / 0.08620.9225 / 0.1389
m = nQwen3-1.7Bself_attn.o_proj.weight2048 × 20480.8640 / 0.06980.9281 / 0.0844
m > nLlama-3.2-1B-Instructmlp.up_proj.weight8192 × 20480.6537 / 0.16340.8139 / 0.2167
m > nQwen3-0.6Bmlp.up_proj.weight3072 × 10240.8034 / 0.12530.8422 / 0.1646
m > nQwen3-1.7Bmlp.up_proj.weight6144 × 20480.8352 / 0.08800.8738 / 0.1268

For Qwen3-0.6B, we use v_proj as the square-matrix example because its o_proj is rectangular.

Input-side energy overlap is higher than output-side energy overlap in all examples shown, covering m < n, m = n, and m > n.

AuthorsOpenReview ↗

Reply to question 2 (1/2)

For question 2: The energy overlap formulation on its own does not seem to take into consideration the portion of information lost by choosing the top-k singular components, which could be different for different parts of the training. Would this affect the observations?

A2: Thank you for your question. The original energy overlap measures retention of the current gradient's top-kk energy. To examine whether excluding the remaining components affects the observations, we compare it with full-gradient energy retention and a bilaterally energy-weighted top-k–full similarity.

(1) Original energy overlap: historical top-k versus current top-k.

Equation (4) in the manuscript is

OzE(s,t)=∥(Qs,kz)⊤Qt,kzDt,k1/2∥F2.\mathcal{O}_z^E(s,t)=\Vert(Q_{s,k}^{z})^{\top}Q_{t,k}^{z}D_{t,k}^{1/2}\Vert_F^2.

Here z∈in,outz\in\\{\mathrm{in},\mathrm{out}\\}.

Here,

Qτ,kin=Vτ,k,Qτ,kout=Uτ,k,Q_{\tau,k}^{\mathrm{in}}=V_{\tau,k},\qquad Q_{\tau,k}^{\mathrm{out}}=U_{\tau,k},

Dτ,k=diag(aτ,1,…,aτ,k),D_{\tau,k}=\mathrm{diag}(a_{\tau,1},\ldots,a_{\tau,k}),

aτ,i=στ,i2∑j=1kστ,j2.a_{\tau,i}=\frac{\sigma_{\tau,i}^{2}}{\sum_{j=1}^{k}\sigma_{\tau,j}^{2}}.

This metric weights the current top-kk directions by their energy contributions and measures how much of that energy the historical top-kk subspace retains.

(2) Full-gradient energy retention: historical top-k versus the full current gradient.

To include the components excluded by current top-kk truncation, we compute

Cin(s,t)=∥GtVs,k∥F2∥Gt∥F2,C_{\mathrm{in}}(s,t)=\frac{\Vert G_tV_{s,k}\Vert_F^2}{\Vert G_t\Vert_F^2},

Cout(s,t)=∥Us,k⊤Gt∥F2∥Gt∥F2.C_{\mathrm{out}}(s,t)=\frac{\Vert U_{s,k}^{\top}G_t\Vert_F^2}{\Vert G_t\Vert_F^2}.

These quantities measure retention of the full current gradient's energy, including its spectral tail, by the historical top-kk subspace.

(3) Bilaterally energy-weighted top-k–full similarity.

We additionally incorporate historical energy weights while retaining the full current gradient spectrum:

Szk,full(s,t)=∥Ds,k1/2(Qs,kz)⊤Qt,rzDt,full1/2∥F2tr(Ds,k2)tr(Dt,full2).S_z^{k,\mathrm{full}}(s,t)=\frac{\Vert D_{s,k}^{1/2}(Q_{s,k}^{z})^{\top}Q_{t,r}^{z}D_{t,\mathrm{full}}^{1/2}\Vert_F^2}{\sqrt{\mathrm{tr}(D_{s,k}^2)\mathrm{tr}(D_{t,\mathrm{full}}^2)}}.

Here r=min⁡(m,n)r=\min(m,n) for Gt∈Rm×nG_t\in\mathbb{R}^{m\times n}, Qt,rin=Vt,rQ_{t,r}^{\mathrm{in}}=V_{t,r}, Qt,rout=Ut,rQ_{t,r}^{\mathrm{out}}=U_{t,r}, and

Dt,full=diag(bt,1,…,bt,r),D_{t,\mathrm{full}}=\mathrm{diag}(b_{t,1},\ldots,b_{t,r}),

bt,i=σt,i2∑j=1rσt,j2.b_{t,i}=\frac{\sigma_{t,i}^{2}}{\sum_{j=1}^{r}\sigma_{t,j}^{2}}.

Only the historical gradient is truncated in this comparison. Directions are weighted by their energy contributions in both measurements.

(4) Results.

We evaluate all three metrics during full-parameter AdamW training on PrOntoQA and ProsQA, with diagnostic rank k=16k=16. The results cover all six model–dataset runs for Qwen3-0.6B, Qwen3-1.7B, and Llama-3.2-1B-Instruct on both tasks. We aggregate local comparisons at steps 40–280, recorded every 40 steps, using the previous recorded checkpoint as the reference. Each matrix–checkpoint pair receives equal weight.

Mean scores are reported as percentages. Each proportion entry reports (a, b), where a uses all measured matrices and b excludes Q/K/embedding. Embeddings and the language-model head are absent from this diagnostic cohort.

Table 1. Mean original energy overlap (%).

ModelDatasetInputOutput
Qwen3-0.6BPrOntoQA79.1625.42
Qwen3-0.6BProsQA83.1825.91
Qwen3-1.7BPrOntoQA81.5327.27
Qwen3-1.7BProsQA82.8525.24
Llama-3.2-1B-InstructPrOntoQA63.4134.14
Llama-3.2-1B-InstructProsQA69.2835.82

Table 2. Percentage of pairs with input-side scores exceeding output-side scores for original energy overlap.

ModelPrOntoQAProsQA
Qwen3-0.6B(98.69%, 98.98%)(98.54%, 100.00%)
Qwen3-1.7B(98.03%, 98.67%)(97.74%, 100.00%)
Llama-3.2-1B-Instruct(77.81%, 93.39%)(77.55%, 95.18%)

Table 3. Mean full-gradient energy retention (%).

ModelDatasetInputOutput
Qwen3-0.6BPrOntoQA76.2325.04
Qwen3-0.6BProsQA77.1724.91
Qwen3-1.7BPrOntoQA78.0826.82
Qwen3-1.7BProsQA76.1124.14
Llama-3.2-1B-InstructPrOntoQA59.8133.30
Llama-3.2-1B-InstructProsQA62.1933.82

Table 4. Percentage of pairs with input-side scores exceeding output-side scores for full-gradient energy retention.

ModelPrOntoQAProsQA
Qwen3-0.6B(98.69%, 98.98%)(98.54%, 100.00%)
Qwen3-1.7B(98.10%, 98.67%)(97.81%, 100.00%)
Llama-3.2-1B-Instruct(77.81%, 93.39%)(77.68%, 95.18%)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.