acceptodds
Withdrawn from ICLR 2027

DistillPFL: Distilling Foundation Model Knowledge for Personalized Federated Learning via Semantic Proxies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalized federated learning (PFL) aims to learn models adapted to heterogeneous data distributions across clients. In practical settings, substantial data heterogeneity and limited local samples can cause personalized models to overfit scarce local data and generalize poorly. Foundation models offer external knowledge that can enhance PFL. However, existing approaches that leverage foundation models for PFL often rely on a shared public dataset or require clients to execute the foundation model, limiting their applicability when suitable public data are unavailable or client resources are constrained. To tackle these challenges, we propose DistillPFL, a PFL framework that uses semantic proxies to construct predictive anchors from a server-side foundation model to distill its knowledge to client models. DistillPFL supports effective personalization under data heterogeneity across clients and limited local data, without requiring a shared public dataset or client-side execution of the foundation model. Extensive experiments across three benchmark datasets show consistent improvements over representative baselines under heterogeneous data distributions and limited local data. White-box reconstruction attacks further show that the transmitted semantic proxies resist recovery of raw samples even under a strong adversary.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Personalized federated learning, Knowledge distillation, Foundation model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CYCfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a personalized federated learning method named DistillPFL, aiming to address three issues, namely overfitting caused by insufficient client data and non-IID distributions, the reliance of some foundation-model-assisted knowledge transfer methods on public datasets and the requirement for clients to deploy large pre-trained models. DistillPFL addresses client-side computational resource constraints by deploying the larger DINOv2 model on the server. Additionally, it constructs multiple predictive anchors for each class and selects the best-matching anchor based on the client's current prediction, thereby enhancing model accuracy. DistillPFL was compared against seven baselines across three visual datasets, demonstrating the model's performance. Additionally, the model's privacy-preserving capabilities were validated through two types of privacy experiments.

Critical strengths

a. DistillPFL enables the transfer of knowledge from a server-side foundation model to small client-side models without requiring public data or large models on the client side.

b. DistillPFL constructs multiple predictive anchors for each class and selects the best-matching anchor based on the client's current prediction, thereby enhancing model accuracy.

Critical weaknesses

a. The proposed model is essentially a combination of several existing, well-established federated learning techniques, such as semantic proxy construction, multiple predictive anchors, and clustering. However, there is no theoretical proof demonstrating the necessity and effectiveness of such a combination.

b. Apart from FedSA, the baselines used in the paper represent earlier research, so the lack of comparison with state-of-the-art methods casts doubt on the DistillPFL's effectiveness. The method proposed in the paper employs a large DINOv2 ViT-L/14 model on the server side. However, the paper does not specify whether the baselines have access to the same foundation model knowledge, making the attribution of the performance gains inconclusive.

Questions for authors

a. Could the authors clarify whether the baselines have access to the same pretrained DINOv2? If not, how can the performance gains be distinguished from the benefits of additional pretrained knowledge?

b. Could the authors theoretically demonstrate the necessity of the components?

c. The semantic transformation module is trained using features from the local client model, while the resulting proxies are processed by a different foundation model on the server. How do the authors ensure that the semantic information preserved in the local representation space remains informative to the server-side foundation model? Could the authors provide empirical evidence supporting this relationship?

d. Could the authors provide a comparison of computational costs, communication overhead, and total training time with the baselines, including the additional costs introduced by DINOv2 and Stage I?

e. While CIFAR-100 is commonly used in federated learning research, GTSRB and PatternNet are less commonly evaluated in PFL studies. Could the authors provide additional results on more widely used benchmarks, such as Tiny-ImageNet or DomainNet, to further validate the generalizability of DistillPFL?

Reviewer_RF1qOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DistillPFL, a personalized FL method with DINOv2 kept on the server. In Stage I, clients send noisy averaged "semantic proxies" of their data, and the server turns them into a few soft-label anchors per class using DINOv2. In Stage II, clients run FedRep-style training with a KD loss toward these anchors. Experiments on CIFAR-100, GTSRB and PatternNet show 1.3–4.5 point gains over baselines. The paper still lacks theory or a clear justification for why the method works. The numerical study also needs a fairer setup, since only DistillPFL uses a foundation model and gets per-setting tuning.

Critical strengths
  • Practical setting. The foundation model stays only on the server and no public dataset is needed, which fits weak clients. Client compute are the same as FedRep, so the method is cheap to run.

  • Consistent results. DistillPFL beats all baselines in all 12 dataset/split/architecture settings, with margins clearly above the std.

Critical weaknesses

1. It's not shown that the foundation model is what helps. In Table 3, "w/o KD" gives almost the same numbers as FedRep in Table 1 (e.g. 55.87 vs 55.56, 59.25 vs 59.19), so all the gain comes from the KD term. But the clients only receive a few soft-label vectors per class, and soft targets alone (e.g. label smoothing) usually help when local data is this small. So the gain might come from this regularization effect and not from DINOv2's knowledge. What happens if DINOv2 is replaced by a random or small network, or if FedRep is simply trained with label smoothing? Also, the paper strongly suffers from a lack of theory or justification for why DINOv2 should give meaningful features for proxies that are transformed by a client-specific MLP, averaged and noised. The reconstructions in App.

2. Server-side Foundation model DistillPFL seems it is the only method using a 300M-parameter FM, and none of the FM-based FL methods the paper cites (FedPromo, Wu et al. 2023, Shi et al. 2024) are compared, not even as reference points.

3. Hyperparameters. DistillPFL's 5 hyperparameters are grid-searched separately for each dataset, split and architecture (App. C, Table 4), while the baselines use the recommended settings from their original papers, only "adjusted when necessary" (App. B.2). The paper also doesn't say which data the grid search was done on, since no validation split is described, so tuning on test data can't be ruled out.

4. Writing and clarity. The method stacks many parts together: local pretraining, the ψ module, two losses, grouping, averaging, noise, quotas, a server-side linear classifier, k-means, min-KL anchor selection, and alternating updates. The paper doesn't explain why each part is needed or how they interact. At the same time, basic details are missing: the input/output size of ψ (App. B.2 only says it is a two-layer MLP), the image resolution fed to DINOv2, and how the server assigns quotas (Sec. 3.2 and Alg. 1 only say "accordingly"). Instead, the same motivation sentence ("without relying on public datasets or client-side execution of the foundation model") is repeated around 7 times across the abstract, introduction, related work and conclusion.

5. DistillPFL clients also do 10 epochs of local pretraining plus 3 epochs of ψ training before federated training (App. C, Alg. 1 lines 3–4), which the baselines seem don't get.

6. The privacy part is also thin: one optimization attack, 5 clients, an SSIM < 0.5 threshold taken from other papers, and no positive control to show the attack works at all. The examples use σ²=1e-4 while the main config uses 1e-2, and there’s no utility vs. noise result.

7. Scalability is claimed but not tested. All experiments use 100 clients. The paper claims the proxy budget makes the method scale to large client populations (Sec. 3.2), but this is never shown. Stage I also requires all N clients to take part (Alg. 1, line 1), while Stage II samples only 10%. How does Stage I work with thousands of clients, many of them offline? And with only B = 20 proxies per class, how representative are the anchors when most clients don't contribute? The paper should also report the communication and compute cost of Stage I.

Questions for authors

Please also see the questions raised in the Weaknesses section above. Additionally:

  1. Which data was used for the grid search of DistillPFL's hyperparameters (Table 4)? Were the baselines tuned with the same effort?

  2. Is the locally pretrained model (especially the personalized head) reused in Stage II, and how is the shared feature extractor initialized? Do the baselines get the same pretraining?

  3. What are the input/output sizes of ψ, and what image resolution goes into DINOv2? Can you show some example proxies?

  4. How does the server assign quotas? And how does Stage I work when only part of the clients are online, e.g. with thousands of clients?

  5. What happens if DINOv2 is replaced by a random or small network, or if FedRep is simply trained with label smoothing? If these give similar gains, the foundation model is not what drives the improvement.

  6. How does DistillPFL compare with FedPromo, which also keeps DINOv2 ViT-L/14 on the server?

  7. For privacy: does the attack work in an easy case (one image per proxy, no noise), to show the attack itself is strong enough? And what is the accuracy for different noise levels (σ² = 0, 1e-4, 1e-2)?

Reviewer_UCDhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DistillPFL, which extracts knowledge from a server-side foundation model through semantic proxies and employs multiple predictive anchors for knowledge distillation to improve personalized federated learning under data heterogeneity.

Critical strengths
  • The method leverages external knowledge from a server-side foundation model without requiring public datasets or deploying the foundation model on clients.

  • Experiments are conducted on multiple datasets under different data heterogeneity settings, with ablation studies validating the effectiveness of the main components.

Critical weaknesses
  • The method mainly combines existing techniques, including semantic proxies, knowledge distillation, and multi-anchor learning, resulting in limited novelty.

  • None of the baseline methods utilize the same foundation model knowledge, making it difficult to distinguish whether the performance improvements come from the foundation model or the proposed distillation mechanism.

  • The computational costs of semantic proxy construction and server-side foundation model inference are not sufficiently analyzed, while the privacy evaluation is mainly limited to image reconstruction attacks.

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.