Gradient Subspace Dynamics During Supervised Fine-Tuning of LLMs: Input–Output Asymmetry

The gradient dynamics of supervised fine-tuning (SFT) in large language models (LLMs) remain poorly understood. For a layer $y=Wx$, we call the side of the weight gradient corresponding to the input features $x$ the input side, and the side corresponding to the output features $y$ the output side. To unveil the gradient dynamics of SFT, we conduct seventeen full-parameter SFT runs using AdamW, spanning eight datasets and models from the Qwen3, Llama, and MiniCPM families. Fifteen runs use diagnostic rank $k=8$, and two additional runs use $k=16$. We identify four main findings: input-side energy overlap generally remains high during fine-tuning; output-side energy overlap is generally low, especially outside query/key projections; input-side energy overlap is consistently higher than output-side overlap in model averages and is higher in the vast majority of individual-matrix comparisons; and reversals are concentrated in query/key projections and embedding tables. Across the fifteen $k=8$ runs, input-side energy overlap exceeds output-side overlap in 93.58% of individual-matrix comparisons with the initial subspace and 92.98% of comparisons with the subspace from the previous measurement. As an application of our findings, we propose a memory-efficient optimizer, Overlap-Guided Gradient Low-Rank Projection (), which always projects gradients onto input-side subspaces. It achieves the lowest or near-lowest validation loss among the compared projection rules across the different model–rank configurations. Our code is available at https://anonymous.4open.science/r/gradient_subspace_dynamics-3DD6/.