One PEFT Does Not Fit All: Heterogeneous PEFT Mixtures for Federated LLM Fine-Tuning

Parameter-efficient fine-tuning (PEFT) enables efficient federated adaptation of large language models, yet most federated PEFT (FedPEFT) methods adopt a single PEFT type for all clients. We observe that the strongest PEFT type varies across client tasks, while naively stacking multiple types does not consistently outperform the best single choice. Learning heterogeneous PEFT compositions introduces a selection-aggregation coupling: local module selection determines both which modules are trained and which clients contribute to their aggregation. To address this coupling, we propose **Hermes**, a heterogeneous FedPEFT framework that moves beyond a globally fixed PEFT choice to learn adaptive, client-specific PEFT compositions. A unified block-level interface makes structurally different PEFT modules composable while preserving their original internal computations. Building on this interface, Hermes coordinates sparse routing with usage-aware aggregation, allowing clients to learn distinct PEFT compositions while collaboratively training a shared heterogeneous PEFT pool. Extensive experiments under task-homogeneous and task-heterogeneous settings demonstrate consistent improvements over strong FedPEFT baselines.