acceptodds
Under review as a conference paper at ICLR 2027

When Prompt Experts Disagree: Diagnosing Open-World Decisions in Federated Vision–Language Models

Abstract

Personalized federated prompt learning adapts frozen vision–language models (VLMs) by exchanging lightweight prompts across clients, yet prediction accuracy alone reveals little about the internal behavior of the resulting prompt experts, such as whether they agree with one another, whether each is individually confident, and whether a given expert mixture is safe. To make these questions answerable explicitly, we introduce an open-world diagnostic framework (pFedOPEN). First, pFedOPEN (i) retains a frozen zero-shot VLM as a semantic anchor along with a geometry regularizer to avoid excessive representation drift, (ii) routes experts via class-aware evidence to prevent unbounded expert misadaptation, and (iii) mixes temperature-controlled expert probabilities instead of logits to make each expert's contribution to the final prediction explicit. On top of this design, pFedOPEN records within-expert uncertainty, inter-expert disagreement, and routing ambiguity as three separate and normalized scores. These diagnostic scores are subsequently used by a client-wise held-out calibration to determine whether to accept the mixture, fall back to the zero-shot expert, or abstain, thereby enhancing transparency of expert assembly. We additionally offer theoretical guarantees. Extensive experiments demonstrate pFedOPEN's powerful diagnostic capabilities with strong far-OOD discrimination in open-world settings. The code will be released.

Primary area:
interpretability and explainable AI
Keywords:
Personalized federated prompt learning, Open-world diagnostic, Auditable expert mixtures, Held-out calibration, Uncertainty decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_3js2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates personalized federated prompt learning with frozen CLIP, where each client combines its own prompt with prompts obtained from other clients. The authors argue that existing approaches primarily focus on accuracy, without examining expert agreement, prediction reliability, or when a model should defer its decisions. The proposed pFedOPEN retains a frozen zero-shot CLIP expert as a reference with a minimum mixture weight, while introducing a geometry regularizer to constrain prompt drift. It uses class-aware evidence to route experts and combine their predictions, and separates predictive uncertainty into within-expert ambiguity, inter-expert disagreement, and routing entropy. A client-specific conformal threshold on a combined uncertainty score determines whether to accept the mixture, fall back to zero-shot, or abstain. The method reports substantial improvements in base-to-novel harmonic mean and strong far-OOD AUROC. However, the paper's own analysis reveals difficulties in combining these components, reduced accuracy from zero-shot fallback, and failures of coverage under domain shift.

Critical strengths
  1. The paper addresses a relevant problem. Existing federated prompt learning methods mainly focus on prediction accuracy, while understanding the reliability of mixed expert predictions is also important for practical deployment.

  2. Maintaining a frozen zero-shot CLIP expert as a fixed reference provides a straightforward and reasonable mechanism for detecting and controlling prompt drift.

  3. The uncertainty decomposition is mathematically exact. Predictive entropy is separated into A + D, where D corresponds to mutual information. Reporting A, D, and R individually also makes it possible to examine different sources of uncertainty.

  4. The evaluation covers a broad range of scenarios, including closed-set recognition, base-to-novel generalization, far-OOD detection, and domain shift. The main comparisons also include paired bootstrap significance tests with Holm correction.

  5. The authors are transparent about several negative findings, including the detrimental effect of R, accuracy degradation caused by zero-shot fallback, and the failure of conformal coverage under distribution shift.

Critical weaknesses
  1. The claimed strength in far-OOD discrimination is not well supported by comparisons with existing methods. In Table 11, pFedMoAP's raw entropy, MSP, and energy scores all outperform U on average (approximately 97.9, 96.8, and 97.8 versus 94.01), with raw entropy performing better on every dataset (e.g., DTD: 92.01 versus 75.51). Table 10 further shows that pFedOPEN performs significantly worse on NLL, Brier, ECE, and AURC. Its reported +3.31 AUROC improvement is only relative to temperature-scaled entropy, which is the weakest baseline score. These results make it difficult to establish a clear advantage for the proposed uncertainty measure.

  2. The base-to-novel results are difficult to interpret because the reproduced baselines are substantially weaker than their originally reported performance. For example, pFedMMA reports 77.40 base and 81.49 novel accuracy in its original paper, whereas the reproduction achieves only 44.35 and 54.89, despite comparable local accuracy (97.17 versus 97.35). Moreover, this paper computes H.M. using only base and novel accuracy, while pFedMMA also includes local accuracy in its harmonic mean. This difference obscures pFedOPEN's approximately 18-point reduction in local base accuracy. Consequently, the claimed +24.72 H.M. improvement depends on considerably weaker reproduced baselines and a different evaluation metric.

  3. Several design choices are difficult to justify given the paper's own experimental findings. The routing entropy R is negatively correlated with OOD detection performance (AUROC 41.94), yet it is still included in U with a weight of 0.20. Similarly, zero-shot fallback consistently decreases accuracy across all datasets, while conformal coverage drops to 59.19% on DomainNet against an intended 80% target. The authors themselves conclude that "useful components ≠ useful combination ≠ safe action," which suggests that the proposed components do not form a consistently effective system. This negative finding is difficult to reconcile with the paper's broader claims about pFedOPEN's diagnostic capabilities.

  4. The methodological novelty and theoretical contributions are relatively limited. The entropy and mutual-information decomposition, temperature scaling, and split conformal calibration are established techniques, as the authors acknowledge. The four theoretical propositions also appear to follow directly from these standard formulations rather than providing substantially new insights.

  5. The ablation results contain inconsistencies, and the statistical evidence is somewhat limited. In Tables 7 and 14, the λd = 0, λr = 0, λ_a = 0, and Fixed threshold variants modify only the selection score or decision threshold, so their Acc/ECE/NLL should remain identical to the Full model (78.78 / 9.83 / 1.468). However, these variants instead report exactly the same values as "No router entropy floor" (78.96 / 10.38 / 1.420), suggesting that they may have been evaluated using a different underlying model. In addition, conformal calibration uses only 8 examples per client, producing very coarse thresholds, and several ablations rely on only 2 random seeds.

Questions for authors
  1. Why do the reproduced pFedMMA and pFedMoAP baselines perform substantially worse than their originally reported results under the same protocol? Could you also report H.M. using pFedMMA's original three-term definition?

  2. In Table 7, why do the λ, fixed-threshold, and output-temperature ablations report the same Acc/ECE/NLL as "No router entropy floor" rather than the Full model? Which underlying trained model was used for these experiments?

  3. Given that pFedMoAP's raw entropy, MSP, and energy scores outperform U on far-OOD detection, what specific benefit does the proposed uncertainty score provide?

  4. Since R is negatively correlated with OOD detection and zero-shot fallback consistently reduces accuracy, what motivates retaining these components in the final framework, and what practical benefits should users expect from them?

4 comments
AuthorsOpenReview ↗

Follow-up Message to Reviewer 3js2

Thank you for your review. We provide the complete OOD baseline comparisons, report the three-term harmonic mean including local accuracy, and test how the candidate vocabulary affects baseline results. We also correct the score-only ablations using fixed main checkpoints and report three-seed evaluations with larger calibration sets. Our response clarifies the propositions' design implications and the conditions under which calibration and fallback are justified. We really appreciate that you can consider a more favorable recommendation based on the evidence presented.

Reviewer_3js2OpenReview ↗

My major concern remains

Thank you for your detailed response and additional experiments. However, my major concern remains:

  1. The number U (92.16) is below raw entropy, MSP and energy on far-OOD, so the "strong far-OOD discrimination" claim does not hold.

  2. pFedOPEN is only 1.45 points above zero-shot CLIP on H2 and lower on novel accuracy (72.18 vs 74.29). The headline gain mostly reflects baselines collapsing under full-vocabulary evaluation.

Besides, the manuscrippt is not revised. Many of the questionable claims are still in the paper. Therefore, I keep my original rating.

AuthorsOpenReview ↗

Clarifying Claims and Planned Manuscript Revisions

Dear Reviewer 3js2,

Thank you for clarifying your remaining concerns. We agree that the additional results require narrower claims, and that these qualifications must appear in the manuscript itself.

  1. Far-OOD detection. The unscreened comparison gives AUROC 92.16% for UU, versus 97.31% for raw entropy, 96.21% for MSP, and 96.96% for energy. We will remove the unqualified “strong far-OOD discrimination” headline and explicitly state that the combined score does not outperform these controls. The supported contribution is making uncertainty sources and their effects on decisions inspectable.

  2. Transfer performance. The matched zero-shot comparison shows H2=73.53%H_2=73.53\% versus 72.08%72.08\%, a 1.45-point gain, while novel accuracy is lower: 72.18% versus 74.29%. We will present this as a transfer trade-off and make the zero-shot comparison central. The larger gaps to the reproduced baselines will be reported with their protocol dependence, rather than attributed to the proposed diagnostics.

You are also correct that the submitted manuscript has not yet incorporated these corrections. If accepted, we will merge the rebuttal’s corrected results, evaluation details, and qualified claims into the camera-ready paper, including the abstract, introduction, experiments, and conclusion. The revised paper will consistently present pFedOPEN as an auditable federated prompt-learning framework, with its empirical advantages and limits stated explicitly.

AuthorsOpenReview ↗

Rebuttal summary for Reviewer 3js2

Dear Reviewer 3js2,

Thank you for your careful review and follow-up comments. Our rebuttal addresses your concerns through the following completed experiments and clarifications:

  1. Complete OOD comparisons. We report all four pFedMoAP scoring controls and evaluate OOD inputs sampled without CLIP-score screening. On the unscreened data, UU achieves 92.16% mean AUROC, compared with 97.31% for raw entropy, 96.21% for MSP, and 96.96% for energy. These results support reporting the measured discrimination while narrowing the claim about the combined score’s comparative advantage.

  2. Transfer metrics and baseline protocols. Recomputing the submitted runs with the three-term harmonic mean, including local accuracy, gives H3=74.53%H_3=74.53\% for pFedOPEN, 57.12% for pFedMMA, and 54.17% for pFedMoAP. The matched zero-shot control gives H2=72.08%H_2=72.08\% and H3=70.38%H_3=70.38\%, versus 73.53% and 74.53% for pFedOPEN. Zero-shot CLIP retains higher novel accuracy, at 74.29% versus 72.18%. We also show that restricting candidate labels substantially changes baseline performance, while clarifying that this diagnostic does not establish full equivalence with the published protocols.

  3. Corrected ablations and larger calibration sets. We traced the original ablation inconsistency to different trained predictors. Reevaluation with fixed main checkpoints preserves Food101 accuracy, ECE, and NLL at 78.78%, 9.83%, and 1.468 across score and threshold variants. Three-seed calibration evaluations on all five main datasets yield coverage of 81.50%, 91.05%, and 95.82% at targets of 80%, 90%, and 95%. Zero-shot replacement remains detrimental at these settings, supporting its treatment as an action requiring validation.

  4. Methodological and theoretical scope. We clarify how the propositions connect uncertainty decomposition, reference influence, acceptance coverage, and conditional fallback benefit. Their role is to make the framework’s behavior and operating assumptions explicit. The empirical contribution is balanced transfer and component-level diagnosis, rather than uniformly superior OOD detection or independently large gains from every component.

If accepted, we will incorporate all substantive rebuttal material into the camera-ready paper. This includes correcting the ablation results, reporting the matched controls and protocol differences, replacing the unqualified OOD headline, and presenting the zero-shot transfer comparison prominently. We will revise the abstract, introduction, experiments, and conclusion consistently so that the manuscript reflects the evidence and qualifications stated in our responses.

Reviewer_gzwkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies personalized federated prompt learning for CLIP and asks whether a mixture of prompt experts can tell when its prediction should be trusted. The proposed framework pFedOPEN keeps a frozen zero shot CLIP expert as a semantic anchor with a minimum routing mass, routes experts with class aware log sum exp evidence, mixes expert probabilities instead of logits, and adds a geometry regularizer in Eq. 12. From the mixture it records three normalized scores in Eq. 8 to Eq. 11, namely within expert ambiguity A, expert disagreement D and routing ambiguity R. A per client conformal threshold in Eq. 14 then decides whether to accept the mixture, fall back to the zero shot expert, or abstain. The theory in Appendix B gives the entropy identity H equal to A plus D, an anchor radius bound, split conformal coverage under exchangeability, and a condition for when fallback helps. Experiments cover base to novel transfer on seven datasets in Table 1, five CLIP benchmarks with screened far OOD sets in Table 2, Table 3 and Table 5, feature shift on DomainNet and Caltech10 in Table 4, and ablations in Table 6 to Table 8. The headline result is a base to novel harmonic mean of 73.53 against 48.81 for pFedMMA. The paper also reports several negative findings. Zero shot fallback lowers accuracy on every dataset, R is anti correlated with OOD, and calibration does not transfer across domains.

Critical strengths
  1. The paper is unusually transparent about its own failure modes. Table 2, Table 3 and Table 5 show that fallback hurts accuracy on all five datasets, that R has an OOD AUROC of 41.94, and that A alone beats the combined score U. Table 10 also reports that the proper scoring rules and AURC favor the baseline. This honest component level audit is the main reason the paper is informative to me. The message that a useful component does not give a useful combination or a safe action is relevant for the federated prompt learning community.
  2. The experimental protocol is carefully registered. Splits, OOD manifests and seeds are fixed and hashed as described in Appendix C.1 and Appendix C.3, both baselines are rerun on the same partitions, and the primary comparison uses paired bootstrap intervals with Holm correction in Table 10. This makes the reported numbers easier to trust and to reproduce. It raises my assessment of the clarity of the evidence, even though I disagree with some of the conclusions drawn from it.
Critical weaknesses
  1. The main empirical claims are not supported by the baseline controls reported in the paper. The abstract and the contributions claim strong far OOD discrimination. Table 11 shows that pFedMoAP with raw entropy gives a mean AUROC of about 97.87, with MSP about 96.77 and with energy about 97.82, while U gives 94.01. On DTD the gap is 92.01 for raw entropy against 75.51 for U. The positive OOD result in Table 10 holds only against temperature scaled entropy, which has the weakest mean of 90.70 among the four controls. At the same time Table 3 and Table 10 show that pFedOPEN is clearly worse on ECE, NLL, Brier and AURC. Risk at 90 percent coverage is 17.13 for pFedOPEN against 1.93 for pFedMoAP with temperature scaling. In the base to novel track, local base accuracy falls from 97.35 for pFedMMA to 79.03 for pFedOPEN in Table 1. Table 1 also compares pFedOPEN with abstention against baselines in their raw settings. It is unclear how abstained samples enter the accuracy, and no zero shot CLIP row is given. The novel accuracies of the baselines, 47.70 and 54.89, look far below what zero shot CLIP usually reaches on these datasets. So the 73.53 against 48.81 gain may mostly reflect baseline collapse and the anchor, not the proposed diagnostics. This weakness determines my recommendation because it affects every headline number.
  2. The calibration and decision layer is evaluated in a regime too small to support the claims. The main study uses only eight calibration examples per client, as stated in Appendix C.1. With this budget the conformal rank is fixed by n and the realized coverage of 88.83 in Table 3 is simply eight out of nine. Table 12 shows that target levels of 0.90 and 0.95 accept every test example and trigger no action. The output temperature fit on eight samples also seems unstable, since removing it gives a lower NLL of 0.839 against 1.468 in Table 7. The only calibration size sweep is on CIFAR100 and not on the five main datasets. The theory in Appendix B is standard, as the authors admit in Appendix A, so the contribution rests on this empirical decision layer. In its current form two of the three actions are not supported. Fallback is harmful on real data and abstention is tested only at a degenerate calibration size.
Questions for authors
  1. In Table 1, how is accuracy computed for pFedOPEN with abstention. Please clarify whether abstained samples count as errors or are removed. Please also add zero shot CLIP and pFedOPEN without abstention under the same protocol. If the gain over pFedMMA largely disappears in this setting, my soundness score would not change. If it remains, I would raise it.
  2. Why does Table 10 use temperature scaled pFedMoAP entropy as the primary comparator. Please report the same paired test against raw entropy, MSP and energy from Table 11. The OOD claim in the abstract depends on this answer.
  3. The default Food101 setting gives three different results. Table 7 reports 78.78 accuracy and 9.83 ECE for the full model. The rows with one uncertainty weight set to zero report 78.96 and 10.38, although these weights do not change the classifier. Table 12 and Table 13 report 78.41 and 9.37 for the default configuration. Please clarify which runs these rows come from.
  4. The five dataset track uses a ResNet50 backbone for pFedOPEN and pFedMoAP but the ViT backbone for pFedMMA, as stated in Appendix C.1. Please clarify which numbers in Table 3 and Table 11 are backbone matched.
  5. The OOD manifests in Table 16 are selected automatically by low RN50 CLIP similarity and receive no manual review. The scores of pFedOPEN are also computed from CLIP experts. Please clarify whether this screening favors CLIP based scores, and report results on unscreened or near OOD data if possible.
  6. Please report the main calibration and selective results on the five datasets with a larger calibration set per client. The current conclusion that abstention is preferable rests on eight samples.
2 comments
AuthorsOpenReview ↗

Follow-up Message to Reviewer gzwk

Thank you for reviewing our paper. We clarified how abstained examples are counted and added a matched zero-shot control: pFedOPEN retains a three-term harmonic mean of 74.53% versus 70.38% for zero-shot CLIP when all test examples are included. We also corrected the ablation comparisons using fixed checkpoints, evaluated larger calibration sets on all five main datasets, and tested OOD performance without CLIP-based sample selection. These results clarify the transfer benefit and the scope of our diagnostic claims. We really appreciate that you can consider a more favorable recommendation based on this evidence.

AuthorsOpenReview ↗

Rebuttal summary for Reviewer gzwk

Thank you for your careful review and constructive suggestions. Our rebuttal addresses your concerns through the following completed experiments and clarifications:

  1. Accuracy accounting and zero-shot comparison. All test examples, including those flagged for abstention, contribute to the reported classifier accuracy. Disabling abstention therefore leaves these results unchanged. Against matched zero-shot CLIP, pFedOPEN improves H2H_2 from 72.08% to 73.53% and H3H_3 from 70.38% to 74.53%. Novel accuracy is lower, at 72.18% versus 74.29%, making the transfer trade-off explicit.

  2. Corrected ablation comparisons. We identified that the original score-only ablations used different trained predictors. Reevaluation with fixed main checkpoints preserves Food101 accuracy, ECE, and NLL at 78.78%, 9.83%, and 1.468 across score and threshold variants, isolating their effects on selection.

  3. Larger calibration sets. Three-seed evaluations on all five main datasets use up to 20–50 calibration examples per client. Mean NLL decreases from 1.178 to 0.749 and ECE from 9.87% to 6.41%. Realized coverage reaches 81.50%, 91.05%, and 95.82% at the respective 80%, 90%, and 95% targets. Zero-shot replacement still reduces overall accuracy by 3.51 percentage points at the 80% target.

  4. Expanded OOD controls. We completed paired comparisons against raw entropy, MSP, and energy, clarified backbone matching, and evaluated uniformly sampled OOD inputs without CLIP-score screening. pFedOPEN’s mean AUROC changes from 94.01% to 92.16%. The stronger baseline scores remain explicitly reported, qualifying the scope of our OOD claim.

If accepted, we will incorporate all substantive rebuttal material into the camera-ready paper, including the additional experiments, corrected results, protocol clarifications, and qualified claims. We will revise the abstract, introduction, experiments, and conclusion consistently around the supported contribution: an auditable federated prompt-learning framework with clearly stated empirical benefits and limitations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary for Reviewer piMW

Dear Reviewer piMW,

Thank you for your careful review and follow-up comments. Our rebuttal clarifies the component contributions and decision policy through the following completed evaluations:

  1. Uncertainty-component comparisons. Using fixed predictors, we compared the combined score UU with its individual components. On unscreened OOD data, within-expert ambiguity AA achieves 93.13% mean AUROC versus 92.16% for UU, while both have mean ID AURC of 10.20. These results support component-level diagnosis but do not establish a consistent performance advantage for the fixed combination.

  2. Calibration and decision policies. Three-seed evaluations with larger calibration sets on all five main datasets reduce mean NLL from 1.178 to 0.749 and ECE from 9.87% to 6.41%. Coverage reaches 81.50%, 91.05%, and 95.82% at targets of 80%, 90%, and 95%. At the 80% target, zero-shot replacement reduces overall accuracy by 3.51 percentage points. We clarify that fallback or abstention is configured separately from the acceptance threshold and that coverage requires exchangeability.

  3. Transfer performance and component effects. Under full-vocabulary evaluation, pFedOPEN achieves H3=74.53%H_3=74.53\%, compared with 70.38% for zero-shot CLIP and 54.24% for the pFedMoAP rebuttal rerun. Its lower local accuracy accompanies higher other-base and novel accuracy than pFedMoAP. Removing the zero-shot expert reduces mean H2H_2 by 0.45 points on Food101 and 1.10 points on DTD; removing geometry regularization reduces it by 0.11 and 0.08 points. We report these as descriptive component effects within the full model, without inferring statistical significance from three-seed summaries.

The supported contribution is an auditable framework that exposes expert behavior and tests the consequences of uncertainty aggregation and prediction replacement. The evidence does not establish that every component independently produces a substantial accuracy gain.

If accepted, we will incorporate all substantive rebuttal material into the camera-ready paper, including the completed experiments, corrected comparisons, clarified component roles, and qualified claims. We will revise the abstract, introduction, experiments, and conclusion consistently to reflect the evidence and the scope of the diagnostic contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.