Similar Factors, Different Updates: Rethinking What to Share in Federated LoRA

Low-rank adaptation (LoRA) enables efficient fine-tuning of foundation models in federated learning. However, determining what to share across clients remains challenging under data heterogeneity. Factor similarity provides an intuitive basis for selective sharing, but similar factors may reflect common initialization and accumulated server history rather than shared knowledge. We show that a single gradient step can produce perfectly aligned factors with opposite effective updates, revealing a fundamental limitation of similarity-based interpretations. Motivated by this finding, we propose GReP-LoRA, which uses task gradients to guide shared and personalized adaptation. Our method initializes a shared adapter from the population gradient and selects frozen private spectral bases from client residual gradients. Clients jointly learn shared tangent increments and compact private cores that rescale and combine the selected directions, while only shared parameters are aggregated. Our theoretical analysis connects captured residual energy to personalized objective improvement under quadratic assumptions and establishes nonconvex stationarity bounds with explicit optimization errors. We evaluate our method on vision and language tasks. Held-out gradient prediction and private-branch transfer experiments demonstrate client-specific specialization, while controlled comparisons support the usefulness of a full trainable core.