FedCSO-LoRA: Coordinated Subspace Optimization for Heterogeneous Federated LoRA

Low-Rank Adaptation (LoRA) enables an efficient federated fine-tuning of large language models. In practice, however, clients with heterogeneous resources may have different rank budgets, constraining the dimensions of their active latent subspaces. Some heterogeneous-rank methods randomly select the active components of a shared LoRA adapter, which may miss the informative combinations of components that could be learned even with the same rank budget. Learning subspaces from the local gradients can capture these combinations, but independently selected client subspaces may yield poorly aligned updates when data are heterogeneous. The challenge is therefore to preserve locally the informative directions under different rank budgets, while coordinating the resulting client updates towards the global objective. To this end, we propose FedCSO-LoRA, a coordinated subspace optimization framework for heterogeneous federated LoRA. Given its rank budget, each client learns the combinations of shared adapter coordinates that maximize the retained energy of its estimated local factor gradients. The coordination step aligns the selected subspace with a global update reference, while preserving a prescribed fraction of the initially retained gradient energy. The server maps the client updates into the common factor coordinates, and filters their aggregate according to the subspace coverage. We provide a nonconvex convergence analysis under the heterogeneous rank budgets that characterizes the effects of local training and server aggregation. Experiments across multiple datasets and language models demonstrate the effectiveness of FedCSO-LoRA compared with the various baselines.