- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces FedMUSE, a method for adapting black-box vision-language models in a federated setting, with access limited to continuous prompt inputs and class-score outputs, without gradients or internal features. The core idea is to split the problem in two. First, it searches for text and visual prompts using a mean–contrast parameterization and block-wise zeroth-order estimation. Then it freezes the prompt-based predictor, aggregates sufficient statistics from the clients, solves for a global residual correction in closed form, and picks the correction strength on a validation set. Because the decision parameters can be solved for directly, they never get pulled into the expensive black-box search. Across 13 classification benchmarks, FedMUSE reaches an average accuracy of 70.15%.
Critical strengths
- A useful contribution is the separation of query-based prompt adaptation from global decision learning. This design focuses the limited black-box search budget on prompt adaptation without expanding the search space to include decision parameters that can be solved directly.
Critical weaknesses
-
Section 4.3 reports that applying the same global residual refinement directly to zero-shot CLIP achieves 69.16%, only 0.99 percentage points below FedMUSE. This suggests that most of the improvement over zero-shot CLIP can be obtained without the first-stage prompt search, which accounts for most of the query cost. The paper should provide stronger evidence that this additional gain justifies the cost of prompt search. Table 2 also reports 65.16% for this setting, and this inconsistency needs to be resolved.
-
The practical applicability of global residual refinement is limited by its sensitivity to client coverage. Appendix B.9 shows that its average gain over the fixed prompt predictor drops from 5.16 percentage points with all 20 clients to just 0.32 points with 10 clients. This dependence on broad client coverage raises concerns about whether the reported gains can be retained if some clients cannot upload their statistics.
Questions for authors
-
Does the checkpoint with the highest prompt-only validation accuracy also perform best after global residual refinement? Could the authors examine whether selecting the checkpoint before fitting the residual limits the final predictor’s performance?
-
Could the authors clarify whether the comparison with INTERLEAVED isolates the benefit of separating prompt search from residual refinement? Appendix A.8 states that INTERLEAVED optimizes prompts using squared loss on corrected predictions and fixes γ = 1, whereas FedMUSE uses cross-entropy for prompt search and selects γ on validation data. Table 2 shows that fixing γ = 1 alone reduces FedMUSE’s accuracy from 70.15% to 68.36%, suggesting that strength selection materially affects this comparison. Could the authors give INTERLEAVED the same validation-based strength selection and provide a comparison with matched prompt-training objectives under a comparable query budget?
6 comments
Response to Reviewer k3qE part.2
All results below use the same three additional seeds described in Part 1. Accuracy is in %, reported as mean±sample SD.
3. Does the best prompt-only checkpoint remain best after refinement?
The rankings can differ. We evaluated fixed-round checkpoints, fitting a residual and selecting γ on validation at each listed round. Round 0 is zero-shot plus residual refinement. Calls count prompt search plus reference queries.
| Dataset | Round | Calls | Val. before | Val. after | Test after |
|---|---|---|---|---|---|
| DTD | 0 | 80 | 44.15±0.92 | 45.74±1.06 | 43.89±2.03 |
| DTD | 6 | 6080 | 44.15±2.66 | 53.19±2.81 | 54.51±0.78 |
| DTD | 12 | 12080 | 44.86±2.15 | 58.16±3.21 | 57.39±1.69 |
| DTD | 18 | 18080 | 45.21±0.53 | 59.04±2.76 | 57.60±2.39 |
| DTD | 24 | 24080 | 45.92±0.81 | 59.57±2.32 | 57.58±1.14 |
| EuroSAT | 0 | 40 | 48.33±8.04 | 62.50±8.66 | 61.96±0.38 |
| EuroSAT | 6 | 6040 | 55.83±11.27 | 73.33±3.82 | 66.95±2.46 |
| EuroSAT | 12 | 12040 | 64.17±10.10 | 74.17±3.82 | 65.71±4.54 |
| EuroSAT | 18 | 18040 | 66.67±8.78 | 73.33±1.44 | 67.58±6.21 |
| EuroSAT | 24 | 24040 | 70.00±9.01 | 73.33±6.29 | 66.95±5.41 |
| CLEVR | 0 | 40 | 12.50±0.00 | 25.00±3.13 | 26.03±0.85 |
| CLEVR | 6 | 6040 | 26.04±1.80 | 29.17±4.77 | 27.39±5.88 |
| CLEVR | 12 | 12040 | 30.21±1.80 | 30.21±1.80 | 26.57±3.19 |
| CLEVR | 18 | 18040 | 31.25±0.00 | 31.25±0.00 | 26.56±1.92 |
| CLEVR | 24 | 24040 | 31.25±0.00 | 31.25±0.00 | 26.56±1.92 |
On EuroSAT, prompt-only validation accuracy peaks at round 24, whereas refined validation accuracy peaks at round 12. These are rankings of the means across three seeds; evaluating a revised selection rule requires choosing the checkpoint within each seed.
FedMUSE fits the residual once after selecting the prompt checkpoint. Selecting by refined validation accuracy would require fitting a residual and collecting reference statistics for each candidate checkpoint.
4. Matching objectives and strength selection in INTERLEAVED
The original comparison changes the prompt objective, residual-fitting schedule, and strength selection together. We added controls for end-round refitting, selecting γ only at evaluation, feeding the selected γ into subsequent training, and changing the prompt objective. The last variant below uses the same raw-logit cross-entropy (CE) as FedMUSE; its residual head does not enter prompt training.
| Variant | Prompt loss | γ rule | DTD | EuroSAT | CLEVR |
|---|---|---|---|---|---|
| FedMUSE | Raw CE | Select after search | 57.58±1.14 | 66.95±5.41 | 26.56±1.92 |
| Interleaved | Corrected square | 1 | 56.48±1.03 | 66.16±1.55 | 28.68±3.32 |
| End-round refit | Corrected square | 1 | 56.74±1.04 | 67.92±2.28 | 30.87±1.82 |
| Eval-only selection | Corrected square | 1 in training; select at eval | 56.74±1.04 | 67.92±2.28 | 30.87±1.82 |
| Selection feedback | Corrected square | Select; feed into training | 56.74±1.04 | 66.74±2.17 | 31.22±1.20 |
| Corrected CE + feedback | Corrected CE | Select; feed into training | 56.32±0.66 | 64.51±2.78 | 31.20±3.39 |
| Raw CE + per-round refit | Raw CE | Select at eval | 56.50±0.98 | 64.42±6.28 | 26.98±1.80 |
The corrected-CE variant retains the original loss scaling.
End-round refitting improves the mean accuracy of the original interleaved method on all three tasks. Evaluation-only strength selection gives the same reported results, while feeding the selected strength into training has mixed effects. With the raw-CE objective, FedMUSE achieves higher mean accuracy on DTD and EuroSAT, while per-round refitting is slightly better on CLEVR. FedMUSE fits the residual head once after checkpoint selection, requiring one round of reference collection and statistic upload.
Response to Reviewer k3qE regarding the revised manuscript
Thank you for your detailed suggestions. We have updated the revised manuscript which incorporating four updates:
- Search contribution and cost: Section 4.3 (pp. 7–8) corrects the no-search result to 65.16%, a 4.99 pp gap, and adds a budget comparison: on DTD, 12 versus 24 rounds achieve 57.39% versus 57.58% with approximately half the search-plus-reference calls.
- Missing uploads: Section 4.4 (p. 10, Figure 4) and Appendix B.10 (pp. 32–34, Tables 11–14) add participation experiments across four partitions. With 18/20 clients, losses are 0.40–0.56 pp under IID/Dirichlet and 1.68 pp under pathological skew; larger dropout remains consequential.
- Checkpoint selection: Section 4.3 (p. 8) and Appendix B.6 (pp. 29–30, Figure 5) show that checkpoint rankings can change after refinement. These comparisons do not establish an improved checkpoint-selection rule.
- INTERLEAVED controls: Section 4.3 (p. 8) and Appendix B.5 (pp. 28–29, Table 9) add validation-selected strength and matched raw-CE objective controls. Results are task-dependent.
We hope these corrections, additional controls, and clarified limitations support a favorable reassessment of the paper.
Response to Reviewer k3qE part.1
Thank you for the questions on search cost, client participation, checkpoint selection, and interleaved optimization. We address the first two here and the latter two in Part 2. All new experiments use three additional seeds, separate from the submission; accuracy is in %, with mean±sample SD unless otherwise stated.
1. Contribution and cost of prompt search
The correct original no-search result is 65.16% in Table 2, rather than 69.16% in Section 4.3. Its gap to FedMUSE (70.15%) is therefore 4.99 pp. We will correct the text and the associated 0.99 pp statement.
We also repeated the comparison with three additional seeds. The no-search configuration applies the same global residual procedure directly to zero-shot CLIP scores, with identical training references, pooling, regularization, and validation-based strength selection. Δ below is full minus no search, computed within each seed.
Adding prompt search improves mean accuracy by 4.73 pp over zero-shot CLIP with the same residual refinement.
On DTD, 12 search rounds give 57.39±1.69 accuracy, compared with 57.58±1.14 at 24 rounds, using 12,080 versus 24,080 search-plus-reference calls. The shorter run reaches a similar mean accuracy with about half the calls. Part 2 reports the fixed-round results.
2. Missing client uploads
To separate participation effects from changes in prompt training, we hold the predictor and reference scores fixed, repartition the reference samples, and recompute pooled statistics at different participation levels. The following table summarizes all 13 tasks.
Values are approximate, equally weighted averages across the 13 tasks, computed from the rounded per-task means. Each subsampled condition averages 10 client masks per training seed. Full participation pools the same reference observations under every partition and therefore has the same result.
With 18 of 20 clients, the loss is 0.40–0.56 pp under the Dirichlet and IID partitions, compared with 1.68 pp under the pathological partition. With only 10 clients, the loss grows to 1.92–4.45 pp. Missing classes accompany the largest loss, but the Dirichlet and IID results show that fewer reference observations also reduce accuracy even with near-complete class coverage.
Collecting delayed uploads or accumulating statistics across participation rounds could increase reference coverage while keeping the predictor fixed, without repeating prompt search.