Curvature-Aware LoRA Initialization in Function Space

Parameter-efficient fine-tuning methods such as LoRA enable efficient adaptation of large pretrained models, but often lag behind full fine-tuning in both convergence speed and final performance. Recent approaches aim to reduce this gap by aligning LoRA parameter updates with those of full fine-tuning, but such parameter-space alignment only indirectly controls model predictions. Instead, we adopt a function-space perspective and formulate the prediction alignment problem, with the objective of matching the outputs of LoRA fine-tuning to those of full fine-tuning, both after the first training step and at convergence. We show that these two alignment objectives are governed by the curvature of the loss: first-step alignment leads to a curvature-colored gradient, whereas alignment at convergence leads to a Newton-like, curvature-whitened gradient. Building on these characterizations, we introduce Curvature-Guided LoRA (CG-LoRA), with two variants that initialize the adapter column spaces using the dominant singular subspaces of the colored and whitened gradients, respectively. We further derive the initialization spectrum from conditioning considerations, eliminating the need for scale tuning. Across natural language understanding, mathematical reasoning, and code generation benchmarks, CG-LoRA matches or exceeds state-of-the-art non-zero-initialization LoRA variants at a lower initialization cost.