FedORS: Sample-Efficient Personalized Federated Learning via Orthogonal Residual Sparse Components

Personalized federated learning (PFL) often combines a shared representation with client-specific prediction heads. However, these heads cannot recover predictive information absent from the shared representation, limiting performance when clients exhibit heterogeneous structures. We propose FedORS, which augments this architecture with client-specific residual components to capture information beyond the shared representation. In its linear formulation, these components are modeled as sparse and orthogonal to the shared representation subspace, enabling local corrections without increasing the dimension of the shared representation. Under structural and sparsity assumptions, we establish theoretical guarantees on convergence and sample complexity, and further characterize adaptation to previously unseen clients with limited local data. Our analysis highlights how low-dimensional shared structure and residual sparsity can support sample-efficient PFL. Experiments on synthetic and real-world datasets demonstrate improved performance across heterogeneous client distributions and effective adaptation to unseen clients.