Same Updates, Different Aggregates: Rethinking Federated LoRA through Shared Subspaces

Federated LoRA communicates low-rank factors to adapt models across decentralized clients. Equivalent factorizations preserve each local model, yet can produce different server aggregates under factor averaging. We formulate *gauge invariance* as an aggregation principle: equivalent representations of the same client updates should yield the same server update. Controlled audits of ordinary training uploads show that product-preserving refactorization can change predictions after factor averaging. Guided by this principle, we introduce GLoRA, a **G**auge-invariant aggregation framework for federated **LoRA** that separates client subspace geometry from update coordinates. Invariant client projectors define a shared reference space, giving client updates a common coordinate frame for aggregation. The resulting aggregate is gauge-invariant at finite server rank and recovers the exact update average under sufficient subspace coverage. A common low-rank server state supplies adapters for heterogeneous client ranks. Extensive experiments on GLUE and SuperNI cover homogeneous-rank, heterogeneous-rank, and task-heterogeneous settings, including sparse participation and unseen-task generalization. Compared with the strongest baselines in each setting, GLoRA improves homogeneous-rank GLUE averages by 0.58–0.93 accuracy points and unseen-task ROUGE-L by an average of 1.52 points across three SuperNI rank distributions, while reducing server aggregation time in the tested low-rank regimes.