When Prompt Experts Disagree: Diagnosing Open-World Decisions in Federated Vision–Language Models

Personalized federated prompt learning adapts frozen vision–language models (VLMs) by exchanging lightweight prompts across clients, yet prediction accuracy alone reveals little about the internal behavior of the resulting prompt experts, such as whether they agree with one another, whether each is individually confident, and whether a given expert mixture is safe. To make these questions answerable explicitly, we introduce an open-world diagnostic framework (pFedOPEN). First, pFedOPEN (i) retains a frozen zero-shot VLM as a semantic anchor along with a geometry regularizer to avoid excessive representation drift, (ii) routes experts via class-aware evidence to prevent unbounded expert misadaptation, and (iii) mixes temperature-controlled expert probabilities instead of logits to make each expert's contribution to the final prediction explicit. On top of this design, pFedOPEN records within-expert uncertainty, inter-expert disagreement, and routing ambiguity as three separate and normalized scores. These diagnostic scores are subsequently used by a client-wise held-out calibration to determine whether to accept the mixture, fall back to the zero-shot expert, or abstain, thereby enhancing transparency of expert assembly. We additionally offer theoretical guarantees. Extensive experiments demonstrate pFedOPEN's powerful diagnostic capabilities with strong far-OOD discrimination in open-world settings. The code will be released.