Federated LoRA under Client Heterogeneity: Factor Representation and Convergence

Low-rank adaptation (LoRA) offers a parameter-efficient approach to fine-tuning foundation models. In federated learning, LoRA can reduce local training and communication costs by restricting updates to the low-rank factors. However, the role of the low-rank parameterization in heterogeneous local training and convergence is not yet fully understood. In this paper, we develop a non-asymptotic analysis of full-batch federated LoRA with full client participation, simultaneous updates of both factors, and factor averaging at the server. We relate weight-space client heterogeneity to local drift in the LoRA factors, deriving a one-round descent bound that tracks factor norms, local steps, and stepsize. From this bound, we establish convergence rates for the minimum squared factor-gradient norm over $T$ communication rounds. With harmonic stepsizes, we obtain an $O(1/\log T)$ rate by controlling trajectory growth, without assuming uniformly bounded factor norms. Under additional uniform trajectory bounds, a stepsize chosen according to $T$ and held constant within each run yields an $O(T^{-2/3})$ rate. Experiments illustrate how client heterogeneity and learning-rate selection affect training dynamics, suggesting that factor and gradient magnitudes can inform local stepsize selection.