acceptodds
Under review as a conference paper at ICLR 2027

SEARCH WHAT YOU MUST, SOLVE WHAT YOU CAN: FEDERATED CONTINUOUS PROMPT ADAPTATION FOR BLACK-BOX VLMS

Abstract

Federated adaptation of pretrained vision–language models (VLMs) commonly relies on lightweight prompts, yet existing approaches typically require gradient or feature access to the frozen backbone. Such white-box access may be unavailable when VLMs are exposed through restricted inference interfaces or opaque runtimes, while distributed users still need to adapt them to private local data. We therefore study federated adaptation of black-box VLMs, where clients can interact with the model through continuous multimodal prompts and class-score feedback but cannot access model internals. Rather than treating black-box adaptation as a single optimization problem, we ask a simpler question: what must be searched, and what can be solved? We propose FedMUSE, which decomposes federated black-box VLM adaptation into two complementary components: multimodal prompt search for what must be optimized through queries, and global decision refinement for what can be solved explicitly from the returned score space. FedMUSE first introduces mean–contrast prompt search, which parameterizes textual and visual prompts through common and differential latent kernels and performs block-wise zeroth-order estimation to reduce cross-block interference. It then freezes the selected prompt predictor and performs global residual refinement in the resulting shared score space. Clients communicate additive sufficient statistics rather than locally fitted heads, allowing the server to recover the pooled residual solution exactly without collecting individual scores or labels. Across 13 classification benchmarks, FedMUSE achieves 70.15% mean accuracy, improving its prompt-only predictor by 5.16 percentage points while adding less than 0.34% training queries, with compact communication and single-call VLM inference. Code is available at https://anonymous.4open.science/r/FedMUSE-04C1.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Prompt Learning, Vision-Language Model, Blackbox Tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k3qEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces FedMUSE, a method for adapting black-box vision-language models in a federated setting, with access limited to continuous prompt inputs and class-score outputs, without gradients or internal features. The core idea is to split the problem in two. First, it searches for text and visual prompts using a mean–contrast parameterization and block-wise zeroth-order estimation. Then it freezes the prompt-based predictor, aggregates sufficient statistics from the clients, solves for a global residual correction in closed form, and picks the correction strength on a validation set. Because the decision parameters can be solved for directly, they never get pulled into the expensive black-box search. Across 13 classification benchmarks, FedMUSE reaches an average accuracy of 70.15%.

Critical strengths
  1. A useful contribution is the separation of query-based prompt adaptation from global decision learning. This design focuses the limited black-box search budget on prompt adaptation without expanding the search space to include decision parameters that can be solved directly.
Critical weaknesses
  1. Section 4.3 reports that applying the same global residual refinement directly to zero-shot CLIP achieves 69.16%, only 0.99 percentage points below FedMUSE. This suggests that most of the improvement over zero-shot CLIP can be obtained without the first-stage prompt search, which accounts for most of the query cost. The paper should provide stronger evidence that this additional gain justifies the cost of prompt search. Table 2 also reports 65.16% for this setting, and this inconsistency needs to be resolved.

  2. The practical applicability of global residual refinement is limited by its sensitivity to client coverage. Appendix B.9 shows that its average gain over the fixed prompt predictor drops from 5.16 percentage points with all 20 clients to just 0.32 points with 10 clients. This dependence on broad client coverage raises concerns about whether the reported gains can be retained if some clients cannot upload their statistics.

Questions for authors
  1. Does the checkpoint with the highest prompt-only validation accuracy also perform best after global residual refinement? Could the authors examine whether selecting the checkpoint before fitting the residual limits the final predictor’s performance?

  2. Could the authors clarify whether the comparison with INTERLEAVED isolates the benefit of separating prompt search from residual refinement? Appendix A.8 states that INTERLEAVED optimizes prompts using squared loss on corrected predictions and fixes γ = 1, whereas FedMUSE uses cross-entropy for prompt search and selects γ on validation data. Table 2 shows that fixing γ = 1 alone reduces FedMUSE’s accuracy from 70.15% to 68.36%, suggesting that strength selection materially affects this comparison. Could the authors give INTERLEAVED the same validation-based strength selection and provide a comparison with matched prompt-training objectives under a comparable query budget?

6 comments
AuthorsOpenReview ↗

Response to Reviewer k3qE part.1

Thank you for the questions on search cost, client participation, checkpoint selection, and interleaved optimization. We address the first two here and the latter two in Part 2. All new experiments use three additional seeds, separate from the submission; accuracy is in %, with mean±sample SD unless otherwise stated.

1. Contribution and cost of prompt search

The correct original no-search result is 65.16% in Table 2, rather than 69.16% in Section 4.3. Its gap to FedMUSE (70.15%) is therefore 4.99 pp. We will correct the text and the associated 0.99 pp statement.

We also repeated the comparison with three additional seeds. The no-search configuration applies the same global residual procedure directly to zero-shot CLIP scores, with identical training references, pooling, regularization, and validation-based strength selection. Δ below is full minus no search, computed within each seed.

DatasetNo search + residualFedMUSEΔ (pp)
DTD43.89±2.0357.58±1.1413.69±3.13
EuroSAT61.96±0.3866.95±5.415.00±5.58
CLEVR26.03±0.8526.56±1.920.53±1.38
Flowers10272.09±0.8685.06±0.9112.97±0.05
RESISC4561.95±0.5575.69±1.2313.74±1.44
Aircraft26.17±0.4831.31±0.455.14±0.08
Pets91.82±0.6692.61±0.690.79±0.69
Caltech10192.21±0.3992.60±0.570.39±0.68
Cars65.44±0.8669.62±0.604.18±0.43
Mini-ImageNet86.82±0.5187.78±1.120.96±0.87
SUN39767.23±0.1768.53±0.301.31±0.28
Food10186.23±0.2186.32±0.250.10±0.17
UCF10170.35±1.2873.04±0.852.69±0.80
13-task mean65.55±0.1070.28±0.574.73±0.62

Adding prompt search improves mean accuracy by 4.73 pp over zero-shot CLIP with the same residual refinement.

On DTD, 12 search rounds give 57.39±1.69 accuracy, compared with 57.58±1.14 at 24 rounds, using 12,080 versus 24,080 search-plus-reference calls. The shorter run reaches a similar mean accuracy with about half the calls. Part 2 reports the fixed-round results.

2. Missing client uploads

To separate participation effects from changes in prompt training, we hold the predictor and reference scores fixed, repartition the reference samples, and recompute pooled statistics at different participation levels. The following table summarizes all 13 tasks.

PartitionClientsClass coverage (%)Accuracy (%)Δ vs. full (pp)
Pathological10/2065.5865.83-4.45
Pathological18/2094.4768.61-1.68
Pathological19/2097.2269.34-0.95
Dirichlet 0.510/2099.9967.92-2.36
Dirichlet 0.518/20100.0069.75-0.53
Dirichlet 0.519/20100.0070.02-0.26
Dirichlet 110/20100.0068.36-1.92
Dirichlet 118/20100.0069.88-0.40
Dirichlet 119/20100.0070.00-0.28
IID10/2099.9968.20-2.08
IID18/20100.0069.72-0.56
IID19/20100.0069.92-0.36
All partitions20/20100.0070.280.00

Values are approximate, equally weighted averages across the 13 tasks, computed from the rounded per-task means. Each subsampled condition averages 10 client masks per training seed. Full participation pools the same reference observations under every partition and therefore has the same result.

With 18 of 20 clients, the loss is 0.40–0.56 pp under the Dirichlet and IID partitions, compared with 1.68 pp under the pathological partition. With only 10 clients, the loss grows to 1.92–4.45 pp. Missing classes accompany the largest loss, but the Dirichlet and IID results show that fewer reference observations also reduce accuracy even with near-complete class coverage.

Collecting delayed uploads or accumulating statistics across participation rounds could increase reference coverage while keeping the predictor fixed, without repeating prompt search.

AuthorsOpenReview ↗

Response to Reviewer k3qE part.2

All results below use the same three additional seeds described in Part 1. Accuracy is in %, reported as mean±sample SD.

3. Does the best prompt-only checkpoint remain best after refinement?

The rankings can differ. We evaluated fixed-round checkpoints, fitting a residual and selecting γ on validation at each listed round. Round 0 is zero-shot plus residual refinement. Calls count prompt search plus reference queries.

DatasetRoundCallsVal. beforeVal. afterTest after
DTD08044.15±0.9245.74±1.0643.89±2.03
DTD6608044.15±2.6653.19±2.8154.51±0.78
DTD121208044.86±2.1558.16±3.2157.39±1.69
DTD181808045.21±0.5359.04±2.7657.60±2.39
DTD242408045.92±0.8159.57±2.3257.58±1.14
EuroSAT04048.33±8.0462.50±8.6661.96±0.38
EuroSAT6604055.83±11.2773.33±3.8266.95±2.46
EuroSAT121204064.17±10.1074.17±3.8265.71±4.54
EuroSAT181804066.67±8.7873.33±1.4467.58±6.21
EuroSAT242404070.00±9.0173.33±6.2966.95±5.41
CLEVR04012.50±0.0025.00±3.1326.03±0.85
CLEVR6604026.04±1.8029.17±4.7727.39±5.88
CLEVR121204030.21±1.8030.21±1.8026.57±3.19
CLEVR181804031.25±0.0031.25±0.0026.56±1.92
CLEVR242404031.25±0.0031.25±0.0026.56±1.92

On EuroSAT, prompt-only validation accuracy peaks at round 24, whereas refined validation accuracy peaks at round 12. These are rankings of the means across three seeds; evaluating a revised selection rule requires choosing the checkpoint within each seed.

FedMUSE fits the residual once after selecting the prompt checkpoint. Selecting by refined validation accuracy would require fitting a residual and collecting reference statistics for each candidate checkpoint.

4. Matching objectives and strength selection in INTERLEAVED

The original comparison changes the prompt objective, residual-fitting schedule, and strength selection together. We added controls for end-round refitting, selecting γ only at evaluation, feeding the selected γ into subsequent training, and changing the prompt objective. The last variant below uses the same raw-logit cross-entropy (CE) as FedMUSE; its residual head does not enter prompt training.

VariantPrompt lossγ ruleDTDEuroSATCLEVR
FedMUSERaw CESelect after search57.58±1.1466.95±5.4126.56±1.92
InterleavedCorrected square156.48±1.0366.16±1.5528.68±3.32
End-round refitCorrected square156.74±1.0467.92±2.2830.87±1.82
Eval-only selectionCorrected square1 in training; select at eval56.74±1.0467.92±2.2830.87±1.82
Selection feedbackCorrected squareSelect; feed into training56.74±1.0466.74±2.1731.22±1.20
Corrected CE + feedbackCorrected CESelect; feed into training56.32±0.6664.51±2.7831.20±3.39
Raw CE + per-round refitRaw CESelect at eval56.50±0.9864.42±6.2826.98±1.80

The corrected-CE variant retains the original loss scaling.

End-round refitting improves the mean accuracy of the original interleaved method on all three tasks. Evaluation-only strength selection gives the same reported results, while feeding the selected strength into training has mixed effects. With the raw-CE objective, FedMUSE achieves higher mean accuracy on DTD and EuroSAT, while per-round refitting is slightly better on CLEVR. FedMUSE fits the residual head once after checkpoint selection, requiring one round of reference collection and statistic upload.

AuthorsOpenReview ↗

Response to Reviewer k3qE regarding the revised manuscript

Thank you for your detailed suggestions. We have updated the revised manuscript which incorporating four updates:

  • Search contribution and cost: Section 4.3 (pp. 7–8) corrects the no-search result to 65.16%, a 4.99 pp gap, and adds a budget comparison: on DTD, 12 versus 24 rounds achieve 57.39% versus 57.58% with approximately half the search-plus-reference calls.
  • Missing uploads: Section 4.4 (p. 10, Figure 4) and Appendix B.10 (pp. 32–34, Tables 11–14) add participation experiments across four partitions. With 18/20 clients, losses are 0.40–0.56 pp under IID/Dirichlet and 1.68 pp under pathological skew; larger dropout remains consequential.
  • Checkpoint selection: Section 4.3 (p. 8) and Appendix B.6 (pp. 29–30, Figure 5) show that checkpoint rankings can change after refinement. These comparisons do not establish an improved checkpoint-selection rule.
  • INTERLEAVED controls: Section 4.3 (p. 8) and Appendix B.5 (pp. 28–29, Table 9) add validation-selected strength and matched raw-CE objective controls. Results are task-dependent.

We hope these corrections, additional controls, and clarified limitations support a favorable reassessment of the paper.

Reviewer_5zSsOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the challenge of adapting pretrained vision-language models (VLMs) in a black-box, federated setup—where clients can only pass continuous prompts and receive full class probability scores, without access to gradients or internal embeddings. The key insight of the paper is to decouple adaptation into what requires derivative-free search and what can be solved analytically. To this end, the authors propose FedMUSE, combining two stages: (1) a mean-contrast prompt search that optimizes text/visual prompts via block-wise SPSA to lower estimation variance, and (2) a global residual refinement stage where clients send aggregated objective statistics from a frozen score space so the server can solve the optimal linear residual head in closed form. Across 13 benchmarks, the approach achieves a strong 70.15% average accuracy, outperforming prompt-only baselines with minimal extra query overhead and low communication cost.

Critical strengths

The conceptual distinction between searching non-convex prompt spaces via zeroth-order queries and analytically solving the downstream classification head is well-reasoned and highly effective. In federated black-box learning, blindly applying zeroth-order optimization to all parameters (including classification heads) leads to severe noise and heterogeneity issues. By freezing the prompt space and pooling additive sufficient statistics, the server computes a globally optimal ridge residual without the loss of quality typical of local head averaging.

The paper is exceptionally thorough. The authors don't just present heuristics; they provide explicit error bounds for block estimation, a precise formulation of the heterogeneity gap, and PAC-Bayes validation bounds. Empirically, testing across 13 distinct vision benchmarks with detailed ablations makes the claims very convincing.

Critical weaknesses

While the paper operates under a "black-box" setting, it actually assumes that the VLM can return a full, dense score vector across all target classes. In practice, many commercial APIs or deployed models often provide only Top-KK predictions or plain text. It would be very helpful if the authors could briefly discuss how the proposed method might be extended or adapted to handle such truncated or sparse output interfaces.

The global residual refinement and strength tuning rely quite heavily on having a well-balanced local validation set (e.g., four examples per class). In extreme non-IID or streaming federated environments, however, local validation data can be noisy or even missing certain classes. This might make the validation gating less robust or potentially lead to local overfitting. Adding some discussion or insights on this limitation would strengthen the paper.

Questions for authors

Please see Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5zSs

Thank you for the suggestions on practical output interfaces and validation availability. All additional experiments use three new seeds, separate from the submission. Results are accuracy (%) as mean±sample SD unless stated otherwise.

1. Top-k and text-only interfaces

For an interface returning Top-k class identities and numerical logits, the residual stage can use a consistent score-completion rule. We freeze the predictor and fill each unreturned class with the kth returned logit minus 1, using the same rule for training references, validation, and testing. We then fit the residual and select its strength as usual.

The table reports the resulting test accuracy. K denotes the number of classes; CLEVR has K=8, so Top-min(10,K) uses its full scores.

DatasetTop-1Top-3Top-5Top-min(10,K)Full
DTD44.21±1.4946.51±0.9447.58±1.9550.43±0.6557.58±1.14
EuroSAT61.96±0.3861.96±0.3861.96±0.3866.95±5.4166.95±5.41
CLEVR26.56±1.9226.56±1.9226.56±1.9226.56±1.9226.56±1.92
Flowers10272.09±0.8674.92±1.0774.85±1.4175.31±1.0785.06±0.91
RESISC4561.95±0.5567.75±0.9270.32±0.5471.74±0.8375.69±1.23
Aircraft26.17±0.4827.64±0.9927.54±1.3028.35±1.4631.31±0.45
Pets91.82±0.6692.27±0.6192.18±0.5192.45±0.2892.61±0.69
Caltech10192.21±0.3992.40±0.3892.21±0.3992.39±0.3792.60±0.57
Cars65.44±0.8667.15±0.7068.13±0.6968.35±0.5269.62±0.60
Mini-ImageNet86.82±0.5187.30±0.7687.43±0.8787.09±0.5187.78±1.12
SUN39767.23±0.1767.57±0.1568.01±0.2468.06±0.1868.53±0.30
Food10186.23±0.2186.33±0.2586.27±0.2186.25±0.2086.32±0.25
UCF10170.35±1.2872.16±1.2372.44±0.6872.42±1.0473.04±0.85

Retaining more classes generally brings accuracy closer to the full-score result. This experiment evaluates residual refinement with a frozen predictor; prompt search through a Top-k-only interface would also require a compatible search objective. Text-only outputs would need to be mapped to comparable class scores before fitting the residual head.

2. Limited or incomplete validation data

The residual head is fitted from training-reference statistics. Validation selects the prompt checkpoint and one global strength γ. Thus, “four examples per class” describes the global validation setting; every client's local validation set need not contain every class.

To assess strength selection with less validation information, we fix the prompt checkpoint and residual head, reduce examples per class or randomly omit validation classes, and reselect γ. For subsampled conditions, we average 10 draws within each training seed before reporting the mean and sample SD across the three seeds.

Validation condition13-task meanEuroSATEuroSAT γ=0 (%)
4/class, all classes70.28±0.5766.95±5.4133.33±57.74
2/class, all classes70.04±0.2164.81±3.2163.33±32.15
1/class, all classes69.55±0.0762.58±0.6490.00±10.00
~75% classes, 4/class70.21±0.3765.58±3.5650.00±34.64
~50% classes, 4/class69.88±0.1863.56±1.3076.67±11.55

Reducing validation data to one example per class lowers mean accuracy from 70.28% to 69.55%. EuroSAT is more sensitive, falling from 66.95% to 62.58%, with γ=0 selected in 90% of the subsampling runs. Selecting γ=0 retains the reference predictor. In extreme non-IID or streaming settings, changes in the validation distribution can affect both checkpoint and strength selection.

AuthorsOpenReview ↗

Response to Reviewer 5zSs regarding the revised manuscript

Thank you for your positive assessment and practical suggestions. We have incorporated the following updates:

  • Restricted outputs: Sections 3.1 (p. 4) and 5 (p. 10) clarify the interface assumptions. Appendix B.15 (pp. 39–40, Figure 11) adds Top-k score-completion experiments across all 13 tasks. These evaluate frozen-predictor refinement; prompt search and text-only outputs require further adaptation.
  • Validation availability: Section 4.1 (p. 6) clarifies that the validation budget is global. Section 4.4 (p. 10) and Appendix B.16 (pp. 40–41, Figure 12 and Table 17) add reduced-sample and missing-class experiments. With one example per class, mean accuracy changes from 70.28% to 69.55%.

We hope these additions address your concerns and reinforce your positive assessment.

AuthorsOpenReview ↗

Follow Up Author Response to Reviewer 5zSs

Dear Reviewer 5zSs,

Thank you for raising these practical questions. We revised the paper to explain how refinement works with top-k scores and what a text-only interface would require. We also examined strength selection when validation examples are scarce or some classes are missing.

As the discussion draws to a close, we hope these additions have addressed your concerns. We would be grateful to hear your thoughts and would be happy to clarify anything further.

Reviewer_dnZFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FedMUSE first searches for multimodal prompts using black-box class scores, then freezes the predictor and fits a residual classifier from pooled client statistics. Validation determines how much of the correction to apply. The method reaches 70.15% mean accuracy on 13 CLIP benchmarks, compared with 64.99% for the prompt predictor alone.

Critical strengths
  1. The pooled residual fit is the most useful part of the paper. Averaging local heads loses 4.12 points, and the derivation explains why: fitting locally and averaging does not solve the pooled problem. The same refinement also improves FedBPT from 62.56% to 68.28%, so this benefit is not tied to the authors' prompt optimizer.
  2. The experimental reporting is thorough. Results span 13 datasets and several client partitions, with three seeds and detailed query and communication costs. I also appreciate the explicit caveats: transmitting statistics is not a privacy guarantee, and validation gating does not guarantee that test accuracy will improve.
Critical weaknesses
  1. Section 4.3 says removing prompt search gives 69.16%, but Table 2 says 65.16%. This is an important discrepancy, since it changes the contribution of prompt search from 0.99 to 4.99 points. I cannot tell which conclusion to take from this ablation.
  2. The case for mean–contrast coordinates is less convincing. On the four tasks in Appendix B.4, they trail independent coordinates before refinement (48.97% versus 49.78%) and lead by 0.93 points afterward. I would regard pooled score-space fitting as the main contribution; the current evidence supports only a conditional benefit from the proposed search coordinates.
Questions for authors
  1. Which no-prompt-search number is correct? Please reconcile Section 4.3 and Table 2, and confirm that the residual-fitting protocol is otherwise identical.
  2. What is the paired uncertainty on the coordinate comparison in Appendix B.4? Are comparable results available for the remaining tasks?
4 comments
AuthorsOpenReview ↗

Response to Reviewer dnZF

Thank you for highlighting the reporting discrepancy and the role of the search coordinates. All new experiments below use three additional seeds, separate from the submission. Accuracy is in %, reported as mean±sample SD; differences are in percentage points (pp).

1. No-prompt-search result and contribution of prompt adaptation

The correct original result is 65.16% in Table 2, giving a 4.99 pp gap to FedMUSE (70.15%). The 69.16% in Section 4.3 arose from unsynchronized text and table files. We will correct the text and its interpretation.

Without prompt search, we apply global residual refinement directly to zero-shot CLIP scores. The training references, statistic pooling, regularization, and validation-based strength selection are identical to those of the full method. The additional-seed comparison is below; Δ is full minus no search, computed within each seed.

DatasetNo search + residualFedMUSEΔ (pp)
DTD43.89±2.0357.58±1.1413.69±3.13
EuroSAT61.96±0.3866.95±5.415.00±5.58
CLEVR26.03±0.8526.56±1.920.53±1.38
Flowers10272.09±0.8685.06±0.9112.97±0.05
RESISC4561.95±0.5575.69±1.2313.74±1.44
Aircraft26.17±0.4831.31±0.455.14±0.08
Pets91.82±0.6692.61±0.690.79±0.69
Caltech10192.21±0.3992.60±0.570.39±0.68
Cars65.44±0.8669.62±0.604.18±0.43
Mini-ImageNet86.82±0.5187.78±1.120.96±0.87
SUN39767.23±0.1768.53±0.301.31±0.28
Food10186.23±0.2186.32±0.250.10±0.17
UCF10170.35±1.2873.04±0.852.69±0.80
13-task mean65.55±0.1070.28±0.574.73±0.62

Adding prompt search improves mean accuracy by 4.73 pp over zero-shot CLIP with the same residual refinement.

2. Coordinate comparison and paired uncertainty

We repeated the four-task comparison with three additional seeds. Both coordinate systems use block search; Δ is MC minus independent, calculated within each seed and summarized as mean±SD.

StageDatasetMC/blockIndependent/blockΔ (pp)
PromptCLEVR26.56±1.9226.30±1.300.26±2.95
PromptDTD44.13±2.2643.34±2.270.79±3.61
PromptEuroSAT61.55±0.9461.10±1.730.45±1.39
PromptUCF10170.43±1.0569.82±0.580.61±1.44
Prompt4-task mean50.67±0.8950.14±1.350.53±1.86
RefinedCLEVR26.56±1.9227.60±1.41-1.04±2.83
RefinedDTD57.58±1.1457.53±1.370.06±0.97
RefinedEuroSAT66.95±5.4165.89±3.911.07±3.95
RefinedUCF10173.04±0.8572.56±0.650.48±1.49
Refined4-task mean56.03±1.4755.90±0.870.14±1.64

Across the four tasks, the paired mean difference is 0.53±1.86 pp before refinement and 0.14±1.64 pp afterward. The coordinate effect is small relative to the variation across seeds in these runs. We have not yet completed the comparison on the other nine tasks.

MC uses common and differential directions for prompt search. The invertible map ZT=C+DZ_T=C+D, ZV=C−DZ_V=C-D preserves the latent prompt family. For latent modality gradients gT,gVg_T,g_V,

gC=gT+gV,gD=gT−gV,g_C=g_T+g_V,\quad g_D=g_T-g_V,

∥gC∥2−∥gD∥2=4⟨gT,gV⟩,∥gC∥2+∥gD∥2=2(∥gT∥2+∥gV∥2).\|g_C\|^2-\|g_D\|^2=4\langle g_T,g_V\rangle,\quad \|g_C\|^2+\|g_D\|^2=2(\|g_T\|^2+\|g_V\|^2).

When the modality-gradient inner product is small relative to their total energy, MC gives nearly balanced block energies, even when one modality has a much larger gradient norm. Under the locally linear model, the optimal continuous query allocation is mb∝db−1∥gb∥m_b\propto\sqrt{d_b-1}\|g_b\|. With equal dimensions and balanced energies, allocating three directions to one block and two to the other is optimal for a budget of five directions. We use this fixed split without estimating gradient norms.

Block SPSA has local-linear error E∥g^b−gb∥2=(db−1)∥gb∥2/mb\mathbb E\|\hat g_b-g_b\|^2=(d_b-1)\|g_b\|^2/m_b. Perturbing one block at a time removes the other block's additive gradient-energy term from the joint-perturbation estimator. This explains the choice of MC with block search when the block energies are balanced.

AuthorsOpenReview ↗

Response to Reviewer dnZF regarding the revised manuscript

Thank you for your feedback. The revised manuscript corrects the discrepancy and clarifies the contribution of prompt search.

  • No-search result: Section 4.3 (p. 7) now reports 65.16%, consistent with Table 2, giving a 4.99 pp gap to FedMUSE. It explicitly confirms that the same global residual procedure is applied to zero-shot scores.
  • Coordinate contribution: Theoretically, mean–contrast coordinates support balanced query allocation under weak modality-gradient correlation, while block-wise estimation removes cross-block interference in the local-linear model. Empirical benefits remain conditional (Appendix B.4). Time constraints prevented extending the comparison to all 13 datasets during discussion. We will complete this evaluation with paired uncertainty estimates and further analyze query efficiency and convergence stability to clarify when these theoretical advantages translate into practical gains.

We hope these corrections and clarifications support a favorable reassessment of our contribution.

Thank you for resolving the no-search discrepancy and providing additional paired experiments. These results address my main concern and support the contribution of prompt search. The benefit of mean–contrast coordinates remains uncertain, but I find the overall approach and the global residual-fitting contribution sufficiently supported. I am raising my rating from 2 to 3.

AuthorsOpenReview ↗

We Thank Reviewer dnZF For The Valuable Feedback

Thank you for your thoughtful reassessment and recognition of our contributions. We greatly appreciate your constructive feedback and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.