GrabLoRA: A Grassmannian Framework for Tracking Gradient Subspace Rotation in LoRA

Low-rank adaptation (LoRA) is a widely used approach in parameter-efficient fine-tuning, and a recent line of work augments it by initializing LoRA with gradient-informed subspaces. Such one-time placement, however, overlooks the evolution of the gradient space during training. In this work, we investigate two questions: Does the latent gradient subspace rotate as LoRA training proceeds? If yes, then how to keep track of the rotating subspace? To study these problems in a geometry-aware setting, we introduce GrabLoRA, a Bayesian Grassmannian LoRA framework that treats the low-rank subspace as an evolving state rather than a fixed initialization. To realize this, GrabLoRA inherits the information-rich early gradient subspace as a prior, realigns the left and right subspaces on the Grassmann manifold using incoming gradient evidence, and applies an uncertainty-aware preconditioner that balances historical geometric uncertainty with newly observed gradients. Across natural language understanding and generation tasks, GrabLoRA achieves consistent improvements over standard LoRA and gradient-initialized baselines. We also provide theoretical insights for gradient-space coverage and convergence of GrabLoRA.