- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses two key challenges in federated few-shot CLIP learning. First, it is difficult to incorporate client-specific class-level visual evidence into collaborative adaptation while keeping it local. Second, sparse or unreliable visual priors may harm generalization. To address these challenges, the authors propose pFedVSA, which separates private visual personalization from shared semantic adaptation. Each client maintains a personalized image adapter and a Local Visual Prior Memory (LVPM). A shared Prototype-to-Semantic Transfer (PST) module maps visual priors into semantic pseudo-tokens. The Semantic Alignment in CLIP Space (SACS) module calibrates predictions using bounded residuals based on the number of support samples and image-prior consistency. During federated training, only the PST module and text adapter are aggregated. For base classes without local visual support and all novel classes, predictions fall back to the standard adapted semantic branch. The paper evaluates pFedVSA on 11 benchmarks, covering base-to-novel generalization, label heterogeneity, and feature heterogeneity. In the main 16-shot experiments on seven datasets using ViT-B/16, the harmonic mean (HM) calculated from the average Local, Base, and Novel accuracies reaches 90.13%. This is 5.58 percentage points higher than pFedMMA. Additional experiments with different shot numbers, a second backbone, and several ablation studies further support the effectiveness of the proposed components.
Critical strengths
- The separation between private visual adaptation and shared semantic learning is clear and well-motivated. The paper adopts an asymmetric private-visual and shared-semantic design. Personalized image adapters and class-level visual priors remain on the clients, while the shared PST module and text adapter enable collaborative learning across clients. This design provides a clear way to leverage client-specific visual evidence. It also balances local personalization with transferable recognition. This is one of the most valuable contributions of the paper.
- The control of visual prior reliability directly addresses the challenges of few-shot learning. SACS considers both support sufficiency and image–prior consistency. It uses reliability-controlled residual calibration to limit the influence of local evidence. LVPM is rebuilt from unique training samples in each round and remains fixed within that round. This avoids repeated counting and stale representations across rounds. For classes without local visual support, the model retains the standard semantic branch. These designs allow the method to adjust predictions based on the availability and reliability of visual evidence. They also provide a clear motivation for the proposed mechanisms.
- The experiments are relatively comprehensive and provide evidence for the training-time transfer mechanism. The experiments cover 11 benchmarks, two CLIP backbones, and different few-shot settings. The main results are averaged over three independent runs. Beyond component ablations, the paper also evaluates shuffled visual priors, a fixed semantic token, a simple visual cache, and different module-sharing strategies. Novel-class inference does not use local visual priors. Therefore, these comparisons provide concrete evidence that training-time visual-to-semantic transfer improves generalization to novel classes.
Critical weaknesses
- Insufficient comparisons with the closest related methods limit the assessment of the novel contributions. The related work section explicitly discusses FedMVP, which leverages visual context, and FedSPA, which uses semantic–visual prototype alignment. However, neither method is included in the main experiments. The existing component ablations and simple visual cache comparisons help demonstrate the role of individual modules. Nevertheless, they are insufficient to establish the additional value of the shared visual-to-semantic mapping over these closely related complete methods. This limits the assessment of the method's novelty and the sources of its performance gains.
- The specific replacement rules in a key ablation are unclear, making it difficult to attribute the performance gains to the proposed mechanism. In Table 3, removing SACS (w/o SACS) reduces Base accuracy from 91.20% to 53.45%. However, the paper does not clearly explain how final predictions are computed in this variant. Does it retain the standard adapted semantic branch? How is the prototype-conditioned branch used? How are classes without local visual support handled? The appendix provides comparisons with fixed and learnable gates, but the mathematical definition of this key ablation remains unclear. Without this clarification, the large performance drop cannot be directly attributed to the reliability gating mechanism alone.
- The substantial degradation of some baselines lacks explanation, and statistical uncertainty is not sufficiently reported. In Table 2, FedOTP achieves Base, Novel, and harmonic mean H_{BN} accuracies of 9.97%, 16.74%, and 12.50%, respectively, under the ViT-B/32, 16-shot setting. These results are substantially worse than those of zero-shot CLIP. Such degradation may be related to protocol adaptation or training behavior. More implementation details are needed to explain these results. Although the paper reports averages over three independent runs, the main results do not include standard deviations or confidence intervals. This makes it difficult to assess result stability and the significance of smaller performance differences.
Questions for authors
- What are the new mechanisms and advantages of pFedVSA compared with the closest related methods? Please clarify the specific differences between pFedVSA and the related methods FedMVP and FedSPA in terms of how visual evidence is used, what information is shared, and how semantic adaptation is performed. Are there any direct comparisons under the same data splits, backbones, and evaluation label spaces? If there are difficulties in obtaining or implementing these methods, please explain them. This information will affect my assessment of the method's novelty and the completeness of the experimental comparisons.
- How is "w/o SACS" defined, and what exactly causes its performance degradation? Please provide the final prediction formula used when SACS is removed. Does this variant retain the standard adapted semantic branch, the prototype-conditioned branch, and the fallback rule for unsupported classes? How does it differ from the fixed gate baseline in the appendix? These details are important for determining whether the large performance drop in Table 3 can be attributed to the reliability control mechanism.
- How can the substantial degradation of some baselines be explained, and are the main performance gains stable? Please explain why FedOTP performs substantially worse than zero-shot CLIP in Table 2. Have the authors verified its implementation, training convergence, and inference procedure under the global Base/Novel label space? Could the authors provide the results from all three runs, along with standard deviations or confidence intervals, for the main experiments and key ablations? This information will help me assess the reliability of the baseline comparisons and the stability of the reported performance gains.
3 comments
Thank you for the detailed response. The definition of w/o SACS and the cross-evaluation of training and inference gates resolve my concern about the interpretation of that ablation. The same-checkpoint FedOTP diagnostic also provides a convincing explanation for much of its poor performance under the Base/Novel evaluation protocol. I appreciate the addition of direct comparisons with FedMVP and FedSPA.
My remaining question concerns the reported margin on DTD. The 3.60-point gain is relative to the strongest baseline listed in the rebuttal table. Using the Base and Novel accuracies reported in the original Table 1, I calculate an of approximately 68.89% for FedCLIP-LoRA, compared with 69.89% for pFedVSA under the same 16-shot ViT-B/16 setting. This puts the margin at approximately 1.0 percentage point when that baseline is included.
Could you include FedCLIP-LoRA alongside FedMVP and FedSPA in the revised comparison table and distinguish the gain over the newly added methods from the gain over the strongest baseline overall?
Rebuttal for sb5L
We sincerely thank the reviewer for the careful follow-up and for acknowledging our clarifications and additional experiments. You are correct that the 3.60-point gain on DTD is relative to FedMVP, the strongest newly added method. Including FedCLIP-LoRA from the original Table 1, its Base/Novel accuracies of 81.94%/59.42% yield an of 68.89%, compared with 69.89% for pFedVSA, giving a gain of approximately 1.00 percentage point over the strongest baseline overall. We apologize for the ambiguity and will include FedCLIP-LoRA alongside FedMVP and FedSPA in the revised comparison table, explicitly distinguishing these two comparison scopes in the accompanying discussion.
Rebuttal for sb5L
We thank the reviewer for recognizing our private-visual/shared-semantic design and reliability control. We address the three concerns below. All accuracies are percentages; HBN denotes the Base/Novel harmonic mean.
[Q1] Comparisons and mechanisms relative to FedMVP/FedSPA.
[A1] We add both methods under the same 16-shot ViT-B/16 protocol, splits and evaluation label spaces. FedMVP uses visual/attribute information to generate visual prompts; FedSPA communicates class-level prototypes for semantic–visual alignment. pFedVSA instead keeps visual priors and image adapters private, shares the PST mapping/text adapter, and transfers visual evidence into semantic adaptation during training. Unsupported and Novel classes use the standard semantic branch.
pFedVSA improves HBN over the strongest comparator by 3.33/3.60 points on SUN397/DTD, supporting the value of private visual-to-semantic transfer beyond the component ablations.
[Q2] Exact definition and interpretation of w/o SACS.
[A2] It sets the supported-class gate to 1 during both training and inference. With standard/prototype-conditioned similarities sca,scp and locally supported classes Ci, its logits are
ℓc={αscp,αsca,c∈Ci,c∈/Ci.
Both branches, PST, anchoring and distillation remain; unsupported Base classes and all Novel classes retain semantic fallback. Full SACS uses ℓc=α[(1−gc)sca+gcscp] on supported classes, with adaptive 0≤gc≤0.5. The appendix's fixed-gate baseline uses a tuned constant mixture, whereas w/o SACS forces prototype-only supported-class prediction.
We cross-evaluate training/inference gates on the four Table 3 datasets (16-shot ViT-B/16); A denotes adaptive gating.
Removing inference calibration lowers Base by 22.22 points; restoring it to the ungated-training checkpoint raises Base by 32.67 points. With adaptive inference fixed, adaptive training adds 5.08 Base/3.76 Novel points. Thus, the original drop combines training effects and supported/unsupported score calibration, rather than isolating one effect.
[Q3] FedOTP degradation and statistical uncertainty.
[A3] Same-checkpoint diagnostics identify strong local-class prediction bias. FedOTP's original evaluation uses client-private test distributions; our Base/Novel tests additionally require cross-client and unseen-class transfer. Its local prompt still participates through OT for these classes. The following uses ViT-B/32, 16-shot, matching Table 2. L/B/N denote Local/Base/Novel; R is the percentage of full-Base predictions assigned to the client's local classes.
Removing the local prompt substantially restores Base/Novel accuracy and reduces local-class concentration. This supports a contribution from local-prompt bias; it does not attribute the entire CLIP gap to that prompt. Global-only inference is a diagnostic, not a replacement baseline. We will detail the implementation, training curves and global Base/Novel vocabulary construction in the revision.
Below are representative three-seed results. Each seed entry is B/N/HBN; summaries are mean ± sample SD. HBN is computed per seed before aggregation.
These examples show consistent gains across all three seeds. The revision will provide per-seed results and standard deviations for every dataset in the main experiments and key ablations.