acceptodds
Under review as a conference paper at ICLR 2027

pFedVSA: Personalized Federated Visual-to-Semantic Alignment for CLIP Adaptation

Abstract

Federated few-shot CLIP learning aims to exploit client-specific visual evidence for personalization while preserving transferable recognition over global base classes and unseen novel classes. A key challenge is how to use client-specific class-level visual evidence for collaborative adaptation without directly sharing such evidence, especially when local visual priors can be sparse or unreliable. We propose Personalized Federated Visual-to-Semantic Alignment (pFedVSA), which separates private visual personalization from shared semantic adaptation. Each client maintains a personalized image adapter and a Local Visual Prior Memory (LVPM) that remain local throughout federated training. A shared Prototype-to-Semantic Transfer (PST) module learns to map locally available visual priors into CLIP-compatible semantic pseudo-tokens, enabling collaborative visual-to-semantic adaptation without aggregating the priors themselves. Semantic Alignment in CLIP Space (SACS) further incorporates prototype-conditioned evidence through reliability-controlled residual calibration based on prior support and image–prior consistency. For classes without local visual support, prediction falls back to the standard adapted semantic branch. Extensive experiments on eleven datasets under label and feature heterogeneity show that pFedVSA consistently improves the balance among client-specific recognition, global base-class transfer, and novel-class generalization.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
personalized federated learning, vision-language models, prompt learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sb5LOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses two key challenges in federated few-shot CLIP learning. First, it is difficult to incorporate client-specific class-level visual evidence into collaborative adaptation while keeping it local. Second, sparse or unreliable visual priors may harm generalization. To address these challenges, the authors propose pFedVSA, which separates private visual personalization from shared semantic adaptation. Each client maintains a personalized image adapter and a Local Visual Prior Memory (LVPM). A shared Prototype-to-Semantic Transfer (PST) module maps visual priors into semantic pseudo-tokens. The Semantic Alignment in CLIP Space (SACS) module calibrates predictions using bounded residuals based on the number of support samples and image-prior consistency. During federated training, only the PST module and text adapter are aggregated. For base classes without local visual support and all novel classes, predictions fall back to the standard adapted semantic branch. The paper evaluates pFedVSA on 11 benchmarks, covering base-to-novel generalization, label heterogeneity, and feature heterogeneity. In the main 16-shot experiments on seven datasets using ViT-B/16, the harmonic mean (HM) calculated from the average Local, Base, and Novel accuracies reaches 90.13%. This is 5.58 percentage points higher than pFedMMA. Additional experiments with different shot numbers, a second backbone, and several ablation studies further support the effectiveness of the proposed components.

Critical strengths
  1. The separation between private visual adaptation and shared semantic learning is clear and well-motivated. The paper adopts an asymmetric private-visual and shared-semantic design. Personalized image adapters and class-level visual priors remain on the clients, while the shared PST module and text adapter enable collaborative learning across clients. This design provides a clear way to leverage client-specific visual evidence. It also balances local personalization with transferable recognition. This is one of the most valuable contributions of the paper.
  2. The control of visual prior reliability directly addresses the challenges of few-shot learning. SACS considers both support sufficiency and image–prior consistency. It uses reliability-controlled residual calibration to limit the influence of local evidence. LVPM is rebuilt from unique training samples in each round and remains fixed within that round. This avoids repeated counting and stale representations across rounds. For classes without local visual support, the model retains the standard semantic branch. These designs allow the method to adjust predictions based on the availability and reliability of visual evidence. They also provide a clear motivation for the proposed mechanisms.
  3. The experiments are relatively comprehensive and provide evidence for the training-time transfer mechanism. The experiments cover 11 benchmarks, two CLIP backbones, and different few-shot settings. The main results are averaged over three independent runs. Beyond component ablations, the paper also evaluates shuffled visual priors, a fixed semantic token, a simple visual cache, and different module-sharing strategies. Novel-class inference does not use local visual priors. Therefore, these comparisons provide concrete evidence that training-time visual-to-semantic transfer improves generalization to novel classes.
Critical weaknesses
  1. Insufficient comparisons with the closest related methods limit the assessment of the novel contributions. The related work section explicitly discusses FedMVP, which leverages visual context, and FedSPA, which uses semantic–visual prototype alignment. However, neither method is included in the main experiments. The existing component ablations and simple visual cache comparisons help demonstrate the role of individual modules. Nevertheless, they are insufficient to establish the additional value of the shared visual-to-semantic mapping over these closely related complete methods. This limits the assessment of the method's novelty and the sources of its performance gains.
  2. The specific replacement rules in a key ablation are unclear, making it difficult to attribute the performance gains to the proposed mechanism. In Table 3, removing SACS (w/o SACS) reduces Base accuracy from 91.20% to 53.45%. However, the paper does not clearly explain how final predictions are computed in this variant. Does it retain the standard adapted semantic branch? How is the prototype-conditioned branch used? How are classes without local visual support handled? The appendix provides comparisons with fixed and learnable gates, but the mathematical definition of this key ablation remains unclear. Without this clarification, the large performance drop cannot be directly attributed to the reliability gating mechanism alone.
  3. The substantial degradation of some baselines lacks explanation, and statistical uncertainty is not sufficiently reported. In Table 2, FedOTP achieves Base, Novel, and harmonic mean H_{BN} accuracies of 9.97%, 16.74%, and 12.50%, respectively, under the ViT-B/32, 16-shot setting. These results are substantially worse than those of zero-shot CLIP. Such degradation may be related to protocol adaptation or training behavior. More implementation details are needed to explain these results. Although the paper reports averages over three independent runs, the main results do not include standard deviations or confidence intervals. This makes it difficult to assess result stability and the significance of smaller performance differences.
Questions for authors
  1. What are the new mechanisms and advantages of pFedVSA compared with the closest related methods? Please clarify the specific differences between pFedVSA and the related methods FedMVP and FedSPA in terms of how visual evidence is used, what information is shared, and how semantic adaptation is performed. Are there any direct comparisons under the same data splits, backbones, and evaluation label spaces? If there are difficulties in obtaining or implementing these methods, please explain them. This information will affect my assessment of the method's novelty and the completeness of the experimental comparisons.
  2. How is "w/o SACS" defined, and what exactly causes its performance degradation? Please provide the final prediction formula used when SACS is removed. Does this variant retain the standard adapted semantic branch, the prototype-conditioned branch, and the fallback rule for unsupported classes? How does it differ from the fixed gate baseline in the appendix? These details are important for determining whether the large performance drop in Table 3 can be attributed to the reliability control mechanism.
  3. How can the substantial degradation of some baselines be explained, and are the main performance gains stable? Please explain why FedOTP performs substantially worse than zero-shot CLIP in Table 2. Have the authors verified its implementation, training convergence, and inference procedure under the global Base/Novel label space? Could the authors provide the results from all three runs, along with standard deviations or confidence intervals, for the main experiments and key ablations? This information will help me assess the reliability of the baseline comparisons and the stability of the reported performance gains.
3 comments
AuthorsOpenReview ↗

Rebuttal for sb5L

We thank the reviewer for recognizing our private-visual/shared-semantic design and reliability control. We address the three concerns below. All accuracies are percentages; HBN denotes the Base/Novel harmonic mean.

[Q1] Comparisons and mechanisms relative to FedMVP/FedSPA.

[A1] We add both methods under the same 16-shot ViT-B/16 protocol, splits and evaluation label spaces. FedMVP uses visual/attribute information to generate visual prompts; FedSPA communicates class-level prototypes for semantic–visual alignment. pFedVSA instead keeps visual priors and image adapters private, shares the PST mapping/text adapter, and transfers visual evidence into semantic adaptation during training. Unsupported and Novel classes use the standard semantic branch.

MethodSUN BaseNovelHBNDTD BaseNovelHBN
pFedMMA70.9976.3773.5855.4461.5558.34
FedMVP74.3477.8976.0768.9063.8766.29
FedSPA76.9875.4576.2172.1760.4465.79
pFedVSA78.8280.2879.5475.3365.1969.89

pFedVSA improves HBN over the strongest comparator by 3.33/3.60 points on SUN397/DTD, supporting the value of private visual-to-semantic transfer beyond the component ablations.

[Q2] Exact definition and interpretation of w/o SACS.

[A2] It sets the supported-class gate to 1 during both training and inference. With standard/prototype-conditioned similarities sca,scps_c^a,s_c^p and locally supported classes Ci\mathcal C_i, its logits are

ℓc={αscp,c∈Ci,αsca,c∉Ci.\ell_c=\begin{cases}\alpha s_c^p,&c\in\mathcal C_i,\\\alpha s_c^a,&c\notin\mathcal C_i.\end{cases}

Both branches, PST, anchoring and distillation remain; unsupported Base classes and all Novel classes retain semantic fallback. Full SACS uses ℓc=α[(1−gc)sca+gcscp]\ell_c=\alpha[(1-g_c)s_c^a+g_cs_c^p] on supported classes, with adaptive 0≤gc≤0.50\leq g_c\leq0.5. The appendix's fixed-gate baseline uses a tuned constant mixture, whereas w/o SACS forces prototype-only supported-class prediction.

We cross-evaluate training/inference gates on the four Table 3 datasets (16-shot ViT-B/16); A denotes adaptive gating.

Train/Test gateLocalBaseNovel
A/A99.8791.2085.67
A/199.7768.9885.67
1/A99.7186.1281.91
1/199.6053.4581.91

Removing inference calibration lowers Base by 22.22 points; restoring it to the ungated-training checkpoint raises Base by 32.67 points. With adaptive inference fixed, adaptive training adds 5.08 Base/3.76 Novel points. Thus, the original drop combines training effects and supported/unsupported score calibration, rather than isolating one effect.

[Q3] FedOTP degradation and statistical uncertainty.

[A3] Same-checkpoint diagnostics identify strong local-class prediction bias. FedOTP's original evaluation uses client-private test distributions; our Base/Novel tests additionally require cross-client and unseen-class transfer. Its local prompt still participates through OT for these classes. The following uses ViT-B/32, 16-shot, matching Table 2. L/B/N denote Local/Base/Novel; R is the percentage of full-Base predictions assigned to the client's local classes.

DatasetInferenceLBNR
SUN397CLIP69.8169.7872.99—
SUN397FedOTP, global+local90.069.035.7997.87
SUN397Same checkpoint, global only84.6665.3159.8817.56
DTDCLIP52.2753.1354.47—
DTDFedOTP, global+local97.7010.6818.4896.82
DTDSame checkpoint, global only93.6947.1942.5719.29

Removing the local prompt substantially restores Base/Novel accuracy and reduces local-class concentration. This supports a contribution from local-prompt bias; it does not attribute the entire CLIP gap to that prompt. Global-only inference is a diagnostic, not a replacement baseline. We will detail the implementation, training curves and global Base/Novel vocabulary construction in the revision.

Below are representative three-seed results. Each seed entry is B/N/HBN; summaries are mean ± sample SD. HBN is computed per seed before aggregation.

Dataset/MethodSeed 1Seed 2Seed 3B ± SDN ± SDHBN ± SD
SUN/FedOTP8.87/5.83/7.049.15/5.68/7.019.07/5.86/7.129.03±0.145.79±0.107.05±0.06
SUN/pFedMMA70.62/73.78/72.1770.57/73.87/72.1870.76/73.66/72.1870.65±0.1073.77±0.1172.18±0.01
SUN/Ours78.79/75.87/77.3078.81/75.87/77.3178.86/75.90/77.3578.82±0.0475.88±0.0277.32±0.03
DTD/FedOTP10.17/18.53/13.1310.87/18.42/13.6711.00/18.49/13.7910.68±0.4518.48±0.0613.53±0.35
DTD/pFedMMA54.35/53.88/54.1154.78/53.90/54.3454.73/54.01/54.3754.62±0.2453.93±0.0754.27±0.14
DTD/Ours66.65/57.47/61.7266.72/57.42/61.7266.67/57.46/61.7266.68±0.0457.45±0.0361.72±0.001

These examples show consistent gains across all three seeds. The revision will provide per-seed results and standard deviations for every dataset in the main experiments and key ablations.

Thank you for the detailed response. The definition of w/o SACS and the cross-evaluation of training and inference gates resolve my concern about the interpretation of that ablation. The same-checkpoint FedOTP diagnostic also provides a convincing explanation for much of its poor performance under the Base/Novel evaluation protocol. I appreciate the addition of direct comparisons with FedMVP and FedSPA.

My remaining question concerns the reported margin on DTD. The 3.60-point gain is relative to the strongest baseline listed in the rebuttal table. Using the Base and Novel accuracies reported in the original Table 1, I calculate an HBNH_{BN} of approximately 68.89% for FedCLIP-LoRA, compared with 69.89% for pFedVSA under the same 16-shot ViT-B/16 setting. This puts the margin at approximately 1.0 percentage point when that baseline is included.

Could you include FedCLIP-LoRA alongside FedMVP and FedSPA in the revised comparison table and distinguish the gain over the newly added methods from the gain over the strongest baseline overall?

AuthorsOpenReview ↗

Rebuttal for sb5L

We sincerely thank the reviewer for the careful follow-up and for acknowledging our clarifications and additional experiments. You are correct that the 3.60-point gain on DTD is relative to FedMVP, the strongest newly added method. Including FedCLIP-LoRA from the original Table 1, its Base/Novel accuracies of 81.94%/59.42% yield an HBNH_{BN} of 68.89%, compared with 69.89% for pFedVSA, giving a gain of approximately 1.00 percentage point over the strongest baseline overall. We apologize for the ambiguity and will include FedCLIP-LoRA alongside FedMVP and FedSPA in the revised comparison table, explicitly distinguishing these two comparison scopes in the accompanying discussion.

Reviewer_CgyzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents pFedVSA, a new method for personalized federated CLIP learning. It keeps each client's image adapter and visual prototypes private while sharing semantic modules across clients. The method uses Prototype-to-Semantic Transfer (PST) to turn local visual information into semantic features. It also uses Semantic Alignment in CLIP Space (SACS) to control the effect of this information on predictions.

Critical strengths
  1. The proposed pFedVSA framework focuses on an important problem in federated learning and may improve collaborative model training.
  2. The method may improve communication efficiency, model performance, and adaptation to different types of client data.
  3. This work studies an important research topic and may be useful for privacy-preserving distributed learning applications.
Critical weaknesses
  1. The difference between pFedVSA and existing personalized federated learning methods should be explained more clearly.
  2. More experiments are needed to compare the proposed method with state-of-the-art methods. Ablation studies should also be included.
  3. The assumptions, computational cost, and limitations of the proposed method should be discussed in more detail.
Questions for authors
  1. What are the main technical contributions of pFedVSA compared with existing federated learning methods?
  2. How does pFedVSA handle differences in data among clients?
  3. How does pFedVSA compare with recent state-of-the-art methods?
  4. What are the computational cost and communication overhead of the proposed method?
  5. Have the authors conducted ablation studies to evaluate the contribution of each main component?
  6. What are the main limitations of pFedVSA, and how can they be addressed in future work?
4 comments
AuthorsOpenReview ↗

Rebuttal for Cgyz

We thank the reviewer for recognizing the importance of pFedVSA. We address the six questions below and will include the clarifications and results in the revision.

[Q1] What are the main technical contributions?

[A1] pFedVSA enables private class-level visual evidence to supervise a shared semantic transformation. Beyond separating global/local parameters: (i) LVPM and the image adapter capture client-specific visual evidence locally; (ii) shared PST converts it into semantic pseudo-tokens without aggregating the priors; (iii) SACS controls its contribution using support sufficiency and image–prior consistency, with semantic fallback for unsupported classes.

Compared with FedMVP's image/attribute-conditioned visual prompts and FedSPA's uploaded visual prototypes, we share the visual-to-semantic mapping and text adapter while keeping visual states local. At equal shared capacity, PST achieves 88.35% Base–Novel harmonic mean versus 84.59% for a text-side residual adapter, supporting the value of this mapping beyond additional parameters.

[Q2] How are client data differences handled?

[A2] Different mechanisms address feature, label and support heterogeneity. Private image adapters and LVPM preserve client-specific appearance; shared PST/text adaptation transfers knowledge across clients with disjoint class assignments. SACS limits unreliable local evidence, and unsupported Base classes and all Novel classes use the standard semantic branch.

Figures 2–3 evaluate label skew and cross-domain feature shift. Table 5 varies class support while keeping the average sample budget fixed: combining support and consistency reaches 73.98% Base–Novel harmonic mean, versus 69.36% for fixed gating. These experiments connect each heterogeneity source to its corresponding mechanism.

[Q3] How does pFedVSA compare with recent methods?

[A3] We add FedMVP and FedSPA on SUN397 and DTD, using our 16-shot ViT-B/16 protocol, data splits and Base/Novel label spaces. HBN denotes the harmonic mean of Base and Novel accuracy; all accuracies are percentages.

MethodSUN BaseSUN NovelSUN HBNDTD BaseDTD NovelDTD HBN
pFedMMA70.9976.3773.5855.4461.5558.34
FedMVP74.3477.8976.0768.9063.8766.29
FedSPA76.9875.4576.2172.1760.4465.79
pFedVSA78.8280.2879.5475.3365.1969.89

pFedVSA exceeds the strongest comparator in HBN by 3.33 points on SUN397 and 3.60 on DTD. These comparisons complement the original seven-dataset results, where pFedVSA achieves 90.13% Local/Base/Novel harmonic mean versus 84.55% for pFedMMA.

[Q4] What are the computational and communication costs?

[A4] We provide a controlled per-client cost benchmark: A100 40GB, ViT-B/16, FP32, batch size 32, 80 images (5 classes × 16 shots), and two local epochs per round.

MethodTime/round (s)Peak memory (GiB)Params./direction
pFedMMA2.75.283,072
pFedVSA, PST-163.25.3219,456
pFedVSA, PST-323.25.3235,840

The full model adds 0.5 s and 0.04 GiB relative to pFedMMA in this benchmark. Only PST/text-adapter parameters are communicated; the image adapter and LVPM remain local. The locally trainable parameter count is 186,368. PST-16 reduces communication by 45.71% and achieves 86.76% HBN versus 88.35% for PST-32 in the four-dataset capacity study. This quantifies the accuracy–communication trade-off; timing uses a fixed workload rather than dataset-dependent client sizes.

[Q5] Have the main components been ablated?

[A5] Yes. Table 3 evaluates image-adapter personalization, LVPM, PST, text adaptation, SACS and both regularizers. Removing LVPM/PST/text adaptation reduces the four-dataset Local/Base/Novel harmonic mean from 91.88% to 87.14%/86.39%/87.40%. Tables 4–5 examine transfer and reliability; Tables A3–A10 cover sharing, memory lifecycle, gates, cache alternatives and loss weights.

Additional matched-input controls improve Novel accuracy from 83.90% with text features to 85.67% with correct visual priors. Cross-checkpoint gating separates SACS effects: removing inference calibration lowers Base from 91.20% to 68.98%; restoring it to the ungated-training model raises Base from 53.45% to 86.12%.

[Q6] What are the assumptions and limitations?

[A6] We assume a common pretrained architecture and shared class vocabulary. Classes without local visual support use the standard semantic branch and benefit from adaptation indirectly. The main Base-to-Novel protocol uses ten fully participating clients. Future work will study heterogeneous architectures, incomplete vocabularies and larger, partially participating federations. We will explicitly describe these scope conditions and measured resource trade-offs.

Thank you so much for the detailed responses to my concerns and also providing additional experiments. I carefully checked the responses to my comment. For Q3, could you please add a reference in front of each method? Based on the new results that you have provided, I can see a substantial impact of inference calibration. Could the authors explain how sensitive is the proposed SACS to noisy and/or misleading local visual prototypes when we have domain shifts or imbalanced client data? Finally, I encourage the authors to add these new results and clarifications into the revised version of the paper.

AuthorsOpenReview ↗

Rebuttal for Cgyz

We sincerely thank the reviewer for carefully examining our responses and additional experiments, and for highlighting prototype reliability under domain shifts and client imbalance. Your follow-up motivated a direct sensitivity experiment. We address both points below.

[Q1] References for the methods compared in Q3.

[A1] We will label the baselines [1] pFedMMA, [2] FedMVP and [3] FedSPA in the comparison table. Their full references are:

[1] Sajjad Ghiasvand, Mahnoosh Alizadeh, and Ramtin Pedarsani. pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models. International Conference on Learning Representations (ICLR), 2026. https://arxiv.org/abs/2507.05394

[2] Mainak Singha, Subhankar Roy, Sarthak Mehrotra, Ankit Jha, Moloud Abdar, Biplab Banerjee, and Elisa Ricci. FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17869–17878, 2025. https://doi.org/10.1109/ICCV51701.2025.01660

[3] Jiarong Yang and Yuan Liu. Beyond Description: Federated Adaptation via Semantic-Visual Prototype Alignment. Proceedings of the 43rd International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, vol. 306, pp. 145444–145463, 2026. https://proceedings.mlr.press/v306/yang26at.html

[Q2] SACS sensitivity to noisy or misleading prototypes.

[A2] We add controlled inference-time prototype perturbations under both support imbalance and domain shifts.

Protocol. The support-imbalance setting follows Table 5: class support counts are {4,8,16,24,28}\{4,8,16,24,28\}, averaging 16 images per class; we report mean Base accuracy over SUN397/DTD. DomainNet follows Figure 3's domain partition and Dirichlet parameter β=0.5\beta=0.5; we report mean test accuracy across domains.

The fixed-gate variant and full SACS use their respective trained checkpoints. We freeze all parameters and retain the test images, labels, support counts and candidate vocabularies. Only the inference-time prototypes are perturbed; no retraining is performed.

Perturbations. (i) For each normalized local prototype pcp_c, sample a Gaussian vector zcz_c and normalize its direction:

\tilde p_c=\frac{p_c+\eta u_c}{\|p_c+\eta u_c\|_2}.$$ We use $\eta=0.5,1.0$ with matched directions. Here $\eta$ is the added vector's norm before renormalization. (ii) For approximately 25% of eligible client–class pairs, replace the prototype with its most cosine-similar *different-class* prototype from the same client, retaining the original class name and support count. This tests visually similar but semantically misleading priors. Each condition starts from clean prototypes and remains fixed throughout evaluation. Perturbed prototypes feed both PST and SACS consistency computation. The variants use matched perturbation directions and class selections. **Results.** Accuracy is in %. Drop denotes full SACS's clean accuracy minus its perturbed accuracy. Mean $g$ summarizes its true-class gates on the corresponding affected-class test samples. |Setting|Perturbation|Strength|Fixed gate|Full SACS|SACS drop (pp)|Mean $g$| |---|---|---|---:|---:|---:|---:| |Imbalanced SUN397/DTD|Clean|—|72.12|77.02|0.00|0.304| |Imbalanced SUN397/DTD|Directional noise|0.5|68.44|75.32|1.70|0.261| |Imbalanced SUN397/DTD|Directional noise|1.0|62.19|72.99|4.03|0.205| |Imbalanced SUN397/DTD|Similar wrong prototype|25%|65.70|74.77|2.25|0.223| |DomainNet|Clean|—|51.76|54.81|0.00|0.286| |DomainNet|Directional noise|0.5|48.12|53.19|1.62|0.252| |DomainNet|Directional noise|1.0|44.16|51.88|2.93|0.219| |DomainNet|Similar wrong prototype|25%|46.68|53.02|1.79|0.242| **Full SACS exhibits smaller accuracy losses under every tested perturbation.** At $\eta=1.0$, fixed gating loses 9.93/7.60 points on imbalanced SUN397/DTD and DomainNet, versus **4.03/2.93** for SACS. Mean gates decrease from 0.304 to 0.205 and from 0.286 to 0.219, consistent with reduced reliance on inconsistent priors. For similar wrong prototypes, SACS loses **2.25/1.79** points, versus 6.42/5.08 for fixed gating. Thus, the full variant is less sensitive under the tested misleading-prior conditions. These comparisons assess each complete variant using its own checkpoint. Together with the existing gate ablations, they support the role of support- and consistency-aware calibration in limiting unreliable prototype contributions while preserving the standard semantic branch. We will incorporate the references, experimental protocol, results and clarifications into the revision.
AuthorsOpenReview ↗

Rebuttal for Cgyz

Dear Reviewer Cgyz, thank you again for your thoughtful feedback and continued consideration of our work. In our latest response, we provided the requested references and additional experiments examining SACS’s sensitivity to noisy and misleading prototypes under client imbalance and domain shifts. We would appreciate hearing whether these results and clarifications address your concerns. Should you have any remaining or new questions, we would be happy to provide further explanations. We will incorporate these additions into the revised manuscript. If you feel that your concerns have been satisfactorily addressed, we would be grateful if you would consider updating your assessment and score in light of the additional evidence. Thank you for your time and constructive guidance.

Reviewer_K4NzOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a key challenge in federated VLM adaptation is how to use client-specific class-level visual evidence for collaborative adaptation without directly sharing such evidence, especially when local visual priors can be sparse or unreliable. To solve it, the paper proposes pFedVSA, a personalized federated CLIP adaptation framework that separates private visual personalization from collaboratively learned semantic adaptation.

Critical strengths
  1. The evaluation is broad. Eleven benchmarks, five shot settings, two backbones, label and feature heterogeneity, and ablations on module sharing, LVPM lifecycle, gate design, and loss weights constitute a thorough empirical package.

  2. The reliability gate is well designed and properly ablated. Eq. 4 combines support sufficiency with image-prior consistency, and Table 5 separates fixed, support-only, similarity-only, learned, and full gates. This shows the two factors are complementary rather than redundant, and it is more informative than reporting only the full model.

Critical weaknesses
  1. Communication cost is higher, and the trade-off is acknowledged but not justified. Table A8 reports 35,840 shared parameters per round, higher than PromptFL, FedPGP, and pFedMMA, and the paper explicitly says it does not claim communication efficiency. That honesty is welcome, but the paper does not ask whether the same accuracy could be obtained with a smaller shared module (e.g., a lower-dimensional PST or a shared adapter without PST). The cost–benefit of PST's capacity is not analyzed.

  2. The privacy motivation is never tested. The paper's central justification is that visual priors stay local and are never uploaded. But PST is trained on those priors, and its parameters are aggregated, so the shared module can leak local class statistics. No leakage analysis is provided. Since privacy is the reason the architecture is asymmetric, this is the most serious gap.

  3. The novel-class gain is indirect and not causally isolated. Novel classes never use LVPM or the prototype-conditioned branch at inference, so their improvement must come from PST and the text adapter being shaped by visual priors during training. Table 4 is suggestive, but a non-visual auxiliary control of comparable capacity is missing, so it remains unclear whether the gain comes from visual-to-semantic transfer or from regularizing the shared modules in general.

  4. The large SACS ablation is not decomposed. Removing SACS drops Base accuracy from 91.20% to 53.45%. Since SACS only modifies supported-class logits at inference, this must involve more than the residual term, removing SACS also removes the gradient path that lets prototype-conditioned prompts shape the text adapter. The two effects are not separated, so SACS's actual role remains ambiguous.

Questions for authors

Please see the weakness above.

1 comment
AuthorsOpenReview ↗

Rebuttal for K4Nz

We thank the reviewer for the positive assessment of our evaluation and reliability gate. Below, all accuracies are four-dataset averages (DTD, Caltech101, Flowers102, OxfordPets), using 16-shot ViT-B/16. HBN denotes the Base/Novel harmonic mean; HM includes Local.

[Q1] Is PST's extra capacity justified?

[A1] We test smaller PSTs and an equal-budget text adapter. The latter replaces PST with a bias-free 512→32→512 residual text-feature adapter, retains the original text/image adapters and corresponding losses, and uses no visual-prior branch.

Shared moduleBottleneckParams./directionBaseNovelHBN
Text adapter3235,84087.2182.1384.59
PST811,26487.8782.1984.94
PST1619,45689.2184.4486.76
Full PST3235,84091.2085.6788.35

PST improves HBN by 3.76 points at equal communication. PST-16 reduces communication by 45.71% while exceeding the larger text-adapter control by 2.17 points. PST-32 gains another 1.59 points over PST-16, quantifying the capacity–accuracy trade-off.

[Q2] Can shared updates leak private statistics?

[A2] We complement local retention with an explicit private-update extension. Images, individual features, LVPM and image adapters stay local; only PST/text-adapter updates are shared. This avoids direct prototype transmission, while shared updates can still contain data-dependent information.

Against an honest-but-curious server, we jointly clip the complete uploaded update to norm C and add client-side Gaussian noise:

\tilde\Delta=\bar\Delta+\mathcal N(0,4C^2\sigma^2I).$$ Replacing one client's dataset has sensitivity at most 2C, conditional on fixed public vocabulary/aggregation metadata. With fixed C, independent noise, 50 fully participating rounds and $\delta=10^{-5}$, Gaussian composition gives $$\rho=25/\sigma^2,\quad \epsilon_b=\rho+2\sqrt{\rho\ln(10^5)}.$$ |Setting|$\sigma$|$\epsilon_b$|Base|Novel|HBN| |---|---:|---:|---:|---:|---:| |Original|0|$\infty$|91.20|85.67|88.35| |Clipping only|0|$\infty$|90.81|85.29|87.96| |Clip + noise|0.5|167.86|90.54|84.43|87.38| |Clip + noise|1.0|58.93|88.60|81.91|85.12| |Clip + noise|2.0|23.22|85.18|77.90|81.38| At $\sigma=0.5$, HBN decreases by **0.97 points**. Stronger noise yields smaller conservative privacy bounds at the reported utility cost. The guarantee covers both uploaded modules over all 50 rounds, without sampling amplification, and applies to the perturbed extension, not the original method. We will add the mechanism and accounting details. **[Q3] Visual transfer or generic regularization?** **[A3] We add capacity-matched non-visual transfer-input controls.** All use the same PST, 35,840 shared parameters, visual memory, gate formula, losses and training budget; only PST's input changes. |PST input|Base|Novel|HBN| |---|---:|---:|---:| |Fixed class-specific random vector|87.89|80.17|83.85| |Frozen class-text feature|89.12|83.90|86.43| |Shuffled visual prior|84.44|79.45|81.87| |Class-matched visual prior|91.20|85.67|88.35| Correct visual priors improve Novel accuracy by **5.50/1.77/6.22 points**, respectively. Extra capacity or non-visual auxiliary processing does not explain the full gain. During training, visual-conditioned prediction updates the shared text adapter and private image adapter, shaping the representations later used by the standard Novel branch; no Novel prior is required. **[Q4] What does w/o SACS remove?** **[A4] It fixes the supported-class gate to 1 during training and inference.** Specifically, $\ell_c=\alpha s_c^p$ for supported classes and $\ell_c=\alpha s_c^a$ otherwise. PST, both branches and anchoring/distillation remain active: the prototype-conditioned gradient path is retained. Full SACS uses adaptive $g\leq0.5$, preserving at least half of the standard score. We cross-evaluate the existing checkpoints; A denotes adaptive gating: |Train/Test gate|Local|Base|Novel|HM| |---|---:|---:|---:|---:| |A/A (Full)|99.87|91.20|85.67|91.88| |A/1|99.77|68.98|85.67|82.89| |1/A|99.71|86.12|81.91|88.63| |1/1 (w/o SACS)|99.60|53.45|81.91|73.25| Removing inference calibration lowers Base by **22.22 points**; restoring it raises Base by **32.67 points**. Local recognition barely changes, supporting calibration's role in global competition between supported/unsupported classes. With adaptive inference fixed, adaptive training adds **5.08 Base/3.76 Novel points**. Thus, the original ablation combines training and inference effects; the existing fixed/single-cue/learned-gate controls further isolate reliability design. We will add this definition and decomposition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.