FedCASA: Federated Consensus Alignment via Selective Aggregation in low-rank spaces

The rapid evolution of Large Language Models (LLMs) has driven widespread adoption of LoRA-based Federated Learning (FL), accompanied by increasingly diverse application demands. While some scenarios require a generalized model endowed with broad domain knowledge, others demand highly specialized models tailored to specific local tasks. However, existing federated LoRA frameworks struggle to simultaneously achieve both robust global generalization and effective local personalization. To address this gap, we propose FedCASA (Federated Consensus Alignment via Selective Aggregation). By exploiting the inherent rank independence and directional discrepancy of LoRA, our approach decouples the client-side LoRA updates into a global consensus component and a local personalized component. We evaluate both the derived global generalized model and the client-specific personalized models on multi-task instruction-tuning datasets across varying degrees of data heterogeneity. Extensive experiments demonstrate that FedCASA consistently outperforms existing baselines in both global and local performance, while simultaneously maintaining low communication costs. Our code is available at https://anonymous.4open.science/r/Phrolova-72.