- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a personalized federated learning method named DistillPFL, aiming to address three issues, namely overfitting caused by insufficient client data and non-IID distributions, the reliance of some foundation-model-assisted knowledge transfer methods on public datasets and the requirement for clients to deploy large pre-trained models. DistillPFL addresses client-side computational resource constraints by deploying the larger DINOv2 model on the server. Additionally, it constructs multiple predictive anchors for each class and selects the best-matching anchor based on the client's current prediction, thereby enhancing model accuracy. DistillPFL was compared against seven baselines across three visual datasets, demonstrating the model's performance. Additionally, the model's privacy-preserving capabilities were validated through two types of privacy experiments.
Critical strengths
a. DistillPFL enables the transfer of knowledge from a server-side foundation model to small client-side models without requiring public data or large models on the client side.
b. DistillPFL constructs multiple predictive anchors for each class and selects the best-matching anchor based on the client's current prediction, thereby enhancing model accuracy.
Critical weaknesses
a. The proposed model is essentially a combination of several existing, well-established federated learning techniques, such as semantic proxy construction, multiple predictive anchors, and clustering. However, there is no theoretical proof demonstrating the necessity and effectiveness of such a combination.
b. Apart from FedSA, the baselines used in the paper represent earlier research, so the lack of comparison with state-of-the-art methods casts doubt on the DistillPFL's effectiveness. The method proposed in the paper employs a large DINOv2 ViT-L/14 model on the server side. However, the paper does not specify whether the baselines have access to the same foundation model knowledge, making the attribution of the performance gains inconclusive.
Questions for authors
a. Could the authors clarify whether the baselines have access to the same pretrained DINOv2? If not, how can the performance gains be distinguished from the benefits of additional pretrained knowledge?
b. Could the authors theoretically demonstrate the necessity of the components?
c. The semantic transformation module is trained using features from the local client model, while the resulting proxies are processed by a different foundation model on the server. How do the authors ensure that the semantic information preserved in the local representation space remains informative to the server-side foundation model? Could the authors provide empirical evidence supporting this relationship?
d. Could the authors provide a comparison of computational costs, communication overhead, and total training time with the baselines, including the additional costs introduced by DINOv2 and Stage I?
e. While CIFAR-100 is commonly used in federated learning research, GTSRB and PatternNet are less commonly evaluated in PFL studies. Could the authors provide additional results on more widely used benchmarks, such as Tiny-ImageNet or DomainNet, to further validate the generalizability of DistillPFL?