SeFoRA: Sketch-Aggregated Federated LoRA with Heterogeneous Client Ranks

We consider federated parameter-efficient fine-tuning of foundation models, e.g., LLMs and LVMs, using low-rank adaptation (LoRA;  Hu et al. 2022). Combining LoRA with federated learning introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose $\text{SeFoRA}$, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local $\Delta \mathbf{W}$ updates, enabling direct aggregation at the federator. As a result, $\text{SeFoRA}$ overcomes the rank heterogeneity challenge and alleviates the bilinear mismatch. We establish convergence to a neighborhood of a first-order stationary point at rate $\mathcal{O}(T^{-1/2})$. Extensive numerical experiments on language and vision models showcase the performance of our algorithm compared to existing methods.