DistillPFL: Distilling Foundation Model Knowledge for Personalized Federated Learning via Semantic Proxies

Personalized federated learning (PFL) aims to learn models adapted to heterogeneous data distributions across clients. In practical settings, substantial data heterogeneity and limited local samples can cause personalized models to overfit scarce local data and generalize poorly. Foundation models offer external knowledge that can enhance PFL. However, existing approaches that leverage foundation models for PFL often rely on a shared public dataset or require clients to execute the foundation model, limiting their applicability when suitable public data are unavailable or client resources are constrained. To tackle these challenges, we propose DistillPFL, a PFL framework that uses semantic proxies to construct predictive anchors from a server-side foundation model to distill its knowledge to client models. DistillPFL supports effective personalization under data heterogeneity across clients and limited local data, without requiring a shared public dataset or client-side execution of the foundation model. Extensive experiments across three benchmark datasets show consistent improvements over representative baselines under heterogeneous data distributions and limited local data. White-box reconstruction attacks further show that the transmitted semantic proxies resist recovery of raw samples even under a strong adversary.