FLoSER: Spectral Experts and Response-Aware Routing for Federated Sparse LoRA Adaptation

Federated fine-tuning adapts large language models to distributed private data, but training and communicating task-specific parameters remain costly for resource-constrained clients. A recent resource-efficient paradigm reuses pretrained LoRA adapters as sparse rank-wise experts by freezing their rank-one components and training and aggregating only a lightweight token router. Although this enables token-level composition with few trainable and communicated parameters, it assumes that LoRA factor coordinates define stable expert identities. We show that equivalent factorizations preserve the same dense update but can induce different rank-one expert banks and sparse-routing families. Moreover, a stable expert bank alone does not ensure effective token-wise selection. We therefore propose Federated LoRA with Spectral Experts and Response-Aware Routing (FLoSER). FLoSER constructs an ordered spectral expert bank from the complete LoRA update—rank-one canonical for distinct singular values and block-canonical for repeated values—and augments learned routing with a signed, token-dependent correction derived from each expert's instantaneous response magnitude. This correction provides a learnable salience signal without changing the active expert budget; during federation, the backbone and expert bank remain frozen, and only router parameters are optimized and communicated. Mechanism analyses show that spectral construction removes factor-coordinate dependence and admits better sparse solutions, although learned routing does not automatically recover them. Experimental results show that FLoSER-S improves matched Raw rank-wise routing by up to 16.40 percentage points, while full FLoSER yields additional setting-dependent gains. Additional transfer and cross-task studies delineate backbone- and scale-sensitivity boundaries, and grouped controls contextualize the benefit of fine-grained learned routing under the primary heterogeneous protocol.