FedCoDi: Directional Coherence for Personalized Federated Low-Rank Adaptation

Federated low-rank adaptation enables efficient fine-tuning of pretrained language models on decentralized data. Under heterogeneous data, however, client updates can point in conflicting directions and partially cancel when averaged. We characterize this directional cancellation through pairwise directional coherence and observe it in client adapters during federated training. Mitigating cancellation must also preserve client-specific adaptations, rather than simply enforce uniform directions. To address this tension, we propose FedCoDi, a personalized federated fine-tuning framework that coordinates how clients incorporate global directions and form subsequent local updates. Building on a magnitude–direction decomposition, FedCoDi retains trainable weight magnitudes locally and shares both low-rank factors. A row-wise coherence gate blends broadcast and previous local factors, retaining more local state where their directions disagree. During local training, a complementary regularizer penalizes angular disagreement between the local low-rank product and the unblended global product, which serves as a shared reference across clients. Both mechanisms operate on clients, while the server retains standard factor-wise averaging. Experiments with RoBERTa-large and Llama-3-8B on four GLUE tasks, GSM8K, and Dolly-15K show consistent gains over the evaluated federated LoRA baselines, achieving substantial improvements over the strongest compared baseline in each setting.