ProLoRA: Bidirectional Communication-Efficient Federated Fine-Tuning with LoRA-Aware Error Feedback

Federated fine-tuning of large language models (LLMs) using low-rank adaptation (LoRA) addresses data scarcity while preserving data privacy. However, the frequent exchange of LoRA updates leads to bidirectional communication bottlenecks. Existing solutions either reduce communication by restricting the trainable LoRA factors, leading to performance degradation, or compress LoRA updates, resulting in high downlink communication overhead or additional server-side compression. To overcome these challenges, we propose ProLoRA by designing novel and efficient random orthogonal projection (ROP) compressor and LoRA-aware error feedback (LA-EF), which enables efficient bidirectional communication while maintaining high model performance. Specifically, ROP enables direct aggregation of compressed LoRA updates through its linearity, while LoRA-aware error feedback exploits shared LoRA geometry to regulate historical compression residuals and improve model recovery. We theoretically prove that ProLoRA achieves the same $\mathcal{O}(1/\sqrt{T})$ convergence order as full-precision FedAvg. We conduct extensive experiments on multiple datasets and LLMs to demonstrate that ProLoRA reduces total communication costs by up to 96.9% without sacrificing model performance when compared with state-of-the-art (SOTA) solutions.