- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates personalized federated prompt learning with frozen CLIP, where each client combines its own prompt with prompts obtained from other clients. The authors argue that existing approaches primarily focus on accuracy, without examining expert agreement, prediction reliability, or when a model should defer its decisions. The proposed pFedOPEN retains a frozen zero-shot CLIP expert as a reference with a minimum mixture weight, while introducing a geometry regularizer to constrain prompt drift. It uses class-aware evidence to route experts and combine their predictions, and separates predictive uncertainty into within-expert ambiguity, inter-expert disagreement, and routing entropy. A client-specific conformal threshold on a combined uncertainty score determines whether to accept the mixture, fall back to zero-shot, or abstain. The method reports substantial improvements in base-to-novel harmonic mean and strong far-OOD AUROC. However, the paper's own analysis reveals difficulties in combining these components, reduced accuracy from zero-shot fallback, and failures of coverage under domain shift.
Critical strengths
-
The paper addresses a relevant problem. Existing federated prompt learning methods mainly focus on prediction accuracy, while understanding the reliability of mixed expert predictions is also important for practical deployment.
-
Maintaining a frozen zero-shot CLIP expert as a fixed reference provides a straightforward and reasonable mechanism for detecting and controlling prompt drift.
-
The uncertainty decomposition is mathematically exact. Predictive entropy is separated into A + D, where D corresponds to mutual information. Reporting A, D, and R individually also makes it possible to examine different sources of uncertainty.
-
The evaluation covers a broad range of scenarios, including closed-set recognition, base-to-novel generalization, far-OOD detection, and domain shift. The main comparisons also include paired bootstrap significance tests with Holm correction.
-
The authors are transparent about several negative findings, including the detrimental effect of R, accuracy degradation caused by zero-shot fallback, and the failure of conformal coverage under distribution shift.
Critical weaknesses
-
The claimed strength in far-OOD discrimination is not well supported by comparisons with existing methods. In Table 11, pFedMoAP's raw entropy, MSP, and energy scores all outperform U on average (approximately 97.9, 96.8, and 97.8 versus 94.01), with raw entropy performing better on every dataset (e.g., DTD: 92.01 versus 75.51). Table 10 further shows that pFedOPEN performs significantly worse on NLL, Brier, ECE, and AURC. Its reported +3.31 AUROC improvement is only relative to temperature-scaled entropy, which is the weakest baseline score. These results make it difficult to establish a clear advantage for the proposed uncertainty measure.
-
The base-to-novel results are difficult to interpret because the reproduced baselines are substantially weaker than their originally reported performance. For example, pFedMMA reports 77.40 base and 81.49 novel accuracy in its original paper, whereas the reproduction achieves only 44.35 and 54.89, despite comparable local accuracy (97.17 versus 97.35). Moreover, this paper computes H.M. using only base and novel accuracy, while pFedMMA also includes local accuracy in its harmonic mean. This difference obscures pFedOPEN's approximately 18-point reduction in local base accuracy. Consequently, the claimed +24.72 H.M. improvement depends on considerably weaker reproduced baselines and a different evaluation metric.
-
Several design choices are difficult to justify given the paper's own experimental findings. The routing entropy R is negatively correlated with OOD detection performance (AUROC 41.94), yet it is still included in U with a weight of 0.20. Similarly, zero-shot fallback consistently decreases accuracy across all datasets, while conformal coverage drops to 59.19% on DomainNet against an intended 80% target. The authors themselves conclude that "useful components ≠ useful combination ≠ safe action," which suggests that the proposed components do not form a consistently effective system. This negative finding is difficult to reconcile with the paper's broader claims about pFedOPEN's diagnostic capabilities.
-
The methodological novelty and theoretical contributions are relatively limited. The entropy and mutual-information decomposition, temperature scaling, and split conformal calibration are established techniques, as the authors acknowledge. The four theoretical propositions also appear to follow directly from these standard formulations rather than providing substantially new insights.
-
The ablation results contain inconsistencies, and the statistical evidence is somewhat limited. In Tables 7 and 14, the λd = 0, λr = 0, λ_a = 0, and Fixed threshold variants modify only the selection score or decision threshold, so their Acc/ECE/NLL should remain identical to the Full model (78.78 / 9.83 / 1.468). However, these variants instead report exactly the same values as "No router entropy floor" (78.96 / 10.38 / 1.420), suggesting that they may have been evaluated using a different underlying model. In addition, conformal calibration uses only 8 examples per client, producing very coarse thresholds, and several ablations rely on only 2 random seeds.
Questions for authors
-
Why do the reproduced pFedMMA and pFedMoAP baselines perform substantially worse than their originally reported results under the same protocol? Could you also report H.M. using pFedMMA's original three-term definition?
-
In Table 7, why do the λ, fixed-threshold, and output-temperature ablations report the same Acc/ECE/NLL as "No router entropy floor" rather than the Full model? Which underlying trained model was used for these experiments?
-
Given that pFedMoAP's raw entropy, MSP, and energy scores outperform U on far-OOD detection, what specific benefit does the proposed uncertainty score provide?
-
Since R is negatively correlated with OOD detection and zero-shot fallback consistently reduces accuracy, what motivates retaining these components in the final framework, and what practical benefits should users expect from them?
4 comments
My major concern remains
Thank you for your detailed response and additional experiments. However, my major concern remains:
-
The number U (92.16) is below raw entropy, MSP and energy on far-OOD, so the "strong far-OOD discrimination" claim does not hold.
-
pFedOPEN is only 1.45 points above zero-shot CLIP on H2 and lower on novel accuracy (72.18 vs 74.29). The headline gain mostly reflects baselines collapsing under full-vocabulary evaluation.
Besides, the manuscrippt is not revised. Many of the questionable claims are still in the paper. Therefore, I keep my original rating.
Clarifying Claims and Planned Manuscript Revisions
Dear Reviewer 3js2,
Thank you for clarifying your remaining concerns. We agree that the additional results require narrower claims, and that these qualifications must appear in the manuscript itself.
-
Far-OOD detection. The unscreened comparison gives AUROC 92.16% for , versus 97.31% for raw entropy, 96.21% for MSP, and 96.96% for energy. We will remove the unqualified “strong far-OOD discrimination” headline and explicitly state that the combined score does not outperform these controls. The supported contribution is making uncertainty sources and their effects on decisions inspectable.
-
Transfer performance. The matched zero-shot comparison shows versus , a 1.45-point gain, while novel accuracy is lower: 72.18% versus 74.29%. We will present this as a transfer trade-off and make the zero-shot comparison central. The larger gaps to the reproduced baselines will be reported with their protocol dependence, rather than attributed to the proposed diagnostics.
You are also correct that the submitted manuscript has not yet incorporated these corrections. If accepted, we will merge the rebuttal’s corrected results, evaluation details, and qualified claims into the camera-ready paper, including the abstract, introduction, experiments, and conclusion. The revised paper will consistently present pFedOPEN as an auditable federated prompt-learning framework, with its empirical advantages and limits stated explicitly.
Rebuttal summary for Reviewer 3js2
Dear Reviewer 3js2,
Thank you for your careful review and follow-up comments. Our rebuttal addresses your concerns through the following completed experiments and clarifications:
-
Complete OOD comparisons. We report all four pFedMoAP scoring controls and evaluate OOD inputs sampled without CLIP-score screening. On the unscreened data, achieves 92.16% mean AUROC, compared with 97.31% for raw entropy, 96.21% for MSP, and 96.96% for energy. These results support reporting the measured discrimination while narrowing the claim about the combined score’s comparative advantage.
-
Transfer metrics and baseline protocols. Recomputing the submitted runs with the three-term harmonic mean, including local accuracy, gives for pFedOPEN, 57.12% for pFedMMA, and 54.17% for pFedMoAP. The matched zero-shot control gives and , versus 73.53% and 74.53% for pFedOPEN. Zero-shot CLIP retains higher novel accuracy, at 74.29% versus 72.18%. We also show that restricting candidate labels substantially changes baseline performance, while clarifying that this diagnostic does not establish full equivalence with the published protocols.
-
Corrected ablations and larger calibration sets. We traced the original ablation inconsistency to different trained predictors. Reevaluation with fixed main checkpoints preserves Food101 accuracy, ECE, and NLL at 78.78%, 9.83%, and 1.468 across score and threshold variants. Three-seed calibration evaluations on all five main datasets yield coverage of 81.50%, 91.05%, and 95.82% at targets of 80%, 90%, and 95%. Zero-shot replacement remains detrimental at these settings, supporting its treatment as an action requiring validation.
-
Methodological and theoretical scope. We clarify how the propositions connect uncertainty decomposition, reference influence, acceptance coverage, and conditional fallback benefit. Their role is to make the framework’s behavior and operating assumptions explicit. The empirical contribution is balanced transfer and component-level diagnosis, rather than uniformly superior OOD detection or independently large gains from every component.
If accepted, we will incorporate all substantive rebuttal material into the camera-ready paper. This includes correcting the ablation results, reporting the matched controls and protocol differences, replacing the unqualified OOD headline, and presenting the zero-shot transfer comparison prominently. We will revise the abstract, introduction, experiments, and conclusion consistently so that the manuscript reflects the evidence and qualifications stated in our responses.
Follow-up Message to Reviewer 3js2
Thank you for your review. We provide the complete OOD baseline comparisons, report the three-term harmonic mean including local accuracy, and test how the candidate vocabulary affects baseline results. We also correct the score-only ablations using fixed main checkpoints and report three-seed evaluations with larger calibration sets. Our response clarifies the propositions' design implications and the conditions under which calibration and fallback are justified. We really appreciate that you can consider a more favorable recommendation based on the evidence presented.