FLORETS: Weaving Cross-Client Low-Rank Adapters into an Aligned Subspace for Federated LLMs

Federated fine-tuning of Large Language Models with low-rank adapters faces a fundamental challenge: updates from client adapters trained on non-IID data lie in misaligned representational bases. Conventional schemes aggregate these adapters via weighted averaging without aligning their bases, leading to destructive parameter interference. We propose a novel federated framework, FLORETS (Federated LOw-Rank adaptation with Election and Truncation for Subspace alignment) for aligning client updates within a shared basis entirely inside the low-rank adapter subspace, without ever reconstructing full-dimensional matrices. FLORETS uses a block-diagonal stacking mechanism %to handle heterogeneous client ranks to uncover the shared basis through a distributed Householder QR decomposition that parallelizes %orthogonalization construction of the shared subspace amongst multiple clients. In contrast to simple weighted averaging, FLORETS aggregates the aligned updates using a coordinate-wise, consensus-driven sign election followed by rank selection. Extensive evaluations across Llama-3.2 (1B and 3B) on Dolly and GSM8K under severe Dirichlet non-IID settings demonstrate that FLORETS consistently outperforms existing federated LoRA baselines in downstream accuracy and computational overhead.