How Much Adaptation Space Does LoRA Need?

Low-rank adaptation (LoRA) and its variants provide memory- and compute-efficient alternatives to full-parameter fine-tuning. Still, it remains unclear how aggressively we can restrict their adaptation space without sacrificing performance and generalization. We study this question through structured extensions of LoRA—Cheap LoRA (cLA), which trains a single low-rank factor while fixing the other in a compute-efficient way, and $c^3$LA, a chained-circulant variant of cLA. We interpret cLA as a structured form of asymmetric LoRA and, more broadly, as a controlled column-subspace restriction of full fine-tuning. We derive information-theoretic generalization error bounds for these restricted adaptation classes along with their base variants, providing a theoretical perspective on their generalization behavior. Empirically, we evaluate **11 fine-tuning methods** across **10 pre-trained models and 14 datasets** using downstream performance, generalization, loss-landscape, and spectral analyses. Across tasks, cLA and $c^3$LA remain competitive with baselines matched on trainable parameters while additionally reducing training time relative to LoRA by up to 14.4% and peak GPU memory by up to 11%. Our results suggest that substantially restricting the adaptation subspace can preserve effective transfer while reducing fine-tuning cost as an additional benefit.