- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ParaDa (Parameters as Data), a method for federated few-shot learning that leverages pretrained classifier weights as an additional source of supervision. Specifically, the method trains a text-conditioned MLP to predict classifier weights from class embeddings, enabling classifier construction for new target classes without accessing source images. It further incorporates LLM-generated class descriptions for semantic refinement and learns classifier residuals from limited client data through federated aggregation. The resulting classifiers can also be used to initialize or regularize existing federated learning methods. Extensive experiments across diverse datasets, backbones, and few-shot settings demonstrate competitive classification performance with substantially reduced computation and communication costs compared to many federated baselines.
Critical strengths
- The paper introduces an interesting Parameters as Data perspective for federated few-shot learning, treating pretrained classifier weights as an additional source of supervision rather than merely as transferable model parameters. This provides a promising direction for addressing data scarcity in federated learning.
- The paper evaluates the proposed method across diverse datasets, model backbones, and few-shot settings, with comparisons against different federated learning baselines. The results demonstrate the effectiveness and practical potential of the proposed approach.
- The framework is simple and practical. By combining text-conditioned classifier construction with lightweight residual adaptation while keeping backbones frozen, it avoids expensive full-model updates during target adaptation and makes the weight predictor easy to reuse across tasks.
- The proposed method offers good efficiency and high modularity. Beyond low computation and communication costs, parameter-derived supervision can be easily plugged into existing federated learning pipelines via simple initialization or regularization.
Critical weaknesses
- ParaDa leverages a CLIP text encoder pretrained on large-scale image-text data, whereas most federated learning baselines do not have access to the same additional semantic knowledge. Although the visual backbones and target support budgets are same, the pretrained knowledge available to different methods is not fully controlled. This makes it difficult to determine whether the reported improvements stem from the proposed method or from the additional pretrained knowledge.
- The MLP predictor is trained on classifier weights from ImageNet-21K, while several downstream datasets exhibit substantial overlap with the source label space. Consequently, the predictor may benefit from previously observed class–weight relationships rather than demonstrating genuine generalization to unseen classes.
- The text-side refinement relies on LLM (such as GPT-4o) to generate class descriptions and VLM (such as CLIP) to extract semantic embeddings. These pretrained models already encode substantial semantic knowledge that may support strong zero-shot classification. It is therefore unclear to what extent the performance improvements can be attributed specifically to the proposed Parameters as Data mechanism, rather than to the powerful pretrained components.
Questions for authors
-
Could the authors provide comparisons with baselines using the same pretrained semantic knowledge to ensure a fair evaluation?
-
How does the source–target label overlap affect generalization? Could the authors evaluate the method after excluding overlapping classes from MLP training?
4 comments
Official Comment by Authors (2/2)
W2&Q2. Several downstream datasets exhibit substantial overlap with the source label space. How does the source–target label overlap affect generalization?
-
We provide a class-overlap analysis in 📍Sec. D.6, showing that ArtBench-10, Camelyon17, and OCT2017 have no class overlap with ImageNet-21K. We further evaluate the effect of removing overlapping classes from MLP training, averaging the results over five runs. The table below reports the accuracy differences before and after removing overlapping classes (Δ = Overlap-removed − Full training (%). Negative values indicate accuracy drops.).
Δ (pp) CIFAR-10 CIFAR-100 Places365 RESISC45 EuroSAT DTD ArtBench Waterbirds Avg. K=0 -0.57 -1.65 -4.98 -1.32 -0.94 -1.35 -0.87 +1.10 -1.32 K=1 -0.03 -0.34 -0.15 -1.55 -0.57 -0.07 -0.02 -0.02 -0.34
As shown, removing overlapping classes leads to only a modest decrease in accuracy, particularly in the one-shot setting, while the rankings relative to the baselines remain nearly unchanged.
-
We agree that our current method depends on the coverage of pretrained models. While this dependence is a limitation, it also offers an opportunity for extensibility: our framework can leverage classifier parameters from additional pretrained models to construct richer supervision.
-
we extend the ImageNet-21K pretrained ConvNeXt by incorporating classifier weights from another ConvNeXt pretrained on 📍EU-Common-707 (a European bird dataset) to construct the more (
class text embedding,classifier parameter) pairs as data. Following the model-stitching approach of [R3], we align two frozen ConvNeXt encoders using public images by learning a linear mapping such that , then transform model B's classifier weights as . We combine the weights from both pretrained models to construct the training pairs and train the MLP using the same pipeline. -
As shown in Table 2, this leads to improvements on CUB-2011 and Waterbirds, further addressing the reviewer's concern in W2.1 regarding performance on such datasets. We are conducting further experiments with pretrained parameters from different architectures and additional benchmarks for inclusion in the revision. Given the limited rebuttal period and academic computational resources, we currently report only the ConvNeXt results.
Table 2 Accuracy (%) CIFAR-100 CUB EuroSAT DTD Waterbirds Avg. K=0 ImageNet-21K 77.1 54.9 26.0 40.8 77.9 55.3 K=0 ImageNet-21K+ EU-Common-707 78.2↑ 75.3↑ 26.1↑ 38.9⭣ 85.6↑ 60.8 K=5 ImageNet-21K 82.5 86.6 87.8 71.2 90.9 83.8 K=5 ImageNet-21K+ EU-Common-707 85.9↑ 88.4↑ 87.5⭣ 71.1⭣ 92.4↑ 85.1
-
From another perspective, our approach aim to distills the knowledge encoded in these pretrained parameters into a lightweight MLP with only 2.17M parameters. Clients only need to deploy this simple MLP to support a wide range of few-shot tasks, without requiring separate adaptation for each task.
[R1]: Image-Free Classifier Injection for Zero-Shot Classification, ICCV 2023.
[R2]: Federated Text-Driven Prompt Generation for Vision-Language Models. ICLR 2024.
[R3]: Revisiting Model Stitching to Compare Neural Representations. NeurIPS 2021.
Official Comment by Reviewer VSTd
Thank you for the detailed responses. I will retain my positive score.
We sincerely thank you for your positive feedback and continued support of our work. We greatly appreciate your time and constructive comments, which have helped us improve our paper.
Official Comment by Authors (1/2)
We sincerely thank the reviewer for the thoughtful and constructive comments and the time devoted to reviewing our manuscript. In the following, W denotes Weakness, Q denotes Question, and R denotes Reference.
To address the reviewer's concern, we report the zero-shot performance and 5-shot linear probing results of CLIP models in Table 1. In addition, we include other CLIP-based baselines and [R2], which also leverages CLIP's semantic knowledge, to provide a more comprehensive comparison under the same pretrained semantic knowledge. For linear probing, we search over learning rates of {0.001, 0.003, 0.01}.
As shown, ParaDa achieves competitive performance with minimal additional overhead. From another perspective, our method effectively distills the knowledge encoded in pretrained models into a lightweight MLP, enabling efficient transfer of pretrained knowledge to downstream few-shot tasks. Its performance can be further improved as the coverage of pretrained models expands, as demonstrated by the additional experimental results presented in the sunsequent Table 2.