acceptodds
Under review as a conference paper at ICLR 2027

FOCUS:UNKNOWN-AWARE PROMPTING AND SE- MANTIC INJECTION FOR OPEN-SET DOMAIN GENER- ALIZATION IN FEDERATED SETTINGS

Abstract

We introduce Federated Open-Set Domain Generalization (FODG), a novel paradigm that requires decentralized models to reconcile domain shifts while identifying novel categories. While prompt-based federated learning has advanced domain generalization, existing methods predominantly operate under a closed-set assumption and falter in data-scarce regimes, leading to catastrophic misclassification of unknown samples. To address these challenges, we propose FOCUS (Federated Open-Set Cross-Domain Unknown Separation), an innovative federated prompt-learning framework with two core breakthroughs. Specifically, FOCUS reformulates the task as a unified prompt learning problem by introducing an additional unknown-aware prompt, enabling federated vision-language models to learn transferable decision boundaries between known and unseen categories across decentralized domains. Furthermore, to combat severe domain shifts in data-scarce federated settings, we develop a cross-attention-based attribute-semantic injection mechanism that dynamically grounds textual attributes into instance-specific visual features. By synergizing this mechanism with a hierarchical multi-modal prompt learner and a class-diversity regularization that enforces orthogonal semantic representations, our framework facilitates robust domain-invariant adaptation and strictly prevents feature entanglement between known and novel categories. Extensive experiments on five benchmark datasets demonstrate that FOCUS consistently outperforms state-of-the-art methods under challenging few-shot federated settings. Code is available at https://anonymous.4open.science/r/FOCUS-2104.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Federated Prompt Learning, Vision-Language Model, Domain Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_daiMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a federated prompt-learning framework that uses unknown-aware prompts and attribute-semantic injection to recognize known classes in unseen domains and identify unknown classes when clients have limited labeled data.

Critical strengths

The paper considers a timely novel problem: open-set domain generalization, and the performance is slightly higher than the benchmarks.

Critical weaknesses

The technical contributions of the paper are very limited; specifically, for the first contribution, the problem already exists, you discussed it, and highlighting it in a federated setting is not a contribution. Secondly, all of the components are adopted from the OSLoPrompt and slightly change the used term: X-attention from OSLoPrompt à ASI in the current paper, the use of LLM to generate the attribute for attention, the two prompt approaches one for classification and one for domain-specific (they change the terms to Shallow and Deep), and even the loss alignment between features. The only new things here compared to the OSLOPrompt paper are the residual prompt (more things to train), the class-diversity loss, and the federated learning setting, while the rest of the components of the system are adopted. However, they do not put references in the equations and in sections 4.2- 4.4, which makes reviewers and readers misunderstand that this is their proposal, I would mark this as a highly misleading paper and strongly oppose the acceptance of the paper.

Questions for authors

Please refer to my critical weaknesses.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer daiM

Thank you for your time and the constructive feedback on our manuscript. We thank the reviewer for the direct critique.

Response to the question of contribution

We thank the reviewer for the direct critique and agree that "highlighting" the FODG problem is not itself a technical contribution. We note, however, that the reviewer lists the federated learning setting among the new elements relative to OSLoPrompt, so we respectfully disagree that federation contributes nothing—what we concede is the current wording, not the setting. Concretely, FODG imposes three constraints that do not co-occur in OSLoPrompt: heterogeneous and only partially overlapping label spaces across clients, a lightweight prompt as the only shareable object, and instance-level visual-semantic grounding that must remain local and non-aggregatable. These constraints jointly determine the method structure rather than wrapping a centralized method in FedAvg, and the revised contribution list will be built around them. If the reviewer has a specific prior work in mind that already formalizes FODG, we would be grateful for the reference and will cite it and clarify the difference; to our knowledge, a formulation combining heterogeneous partial label spaces with a prompt-only communication constraint has not been systematically established.

Clarificaiton for the components

We clarify that X-attention is not our innovation, and the revision will cite OSLoPrompt in Eqs.(1)-(10) and Sections 4.2-4.4. Our contribution is how these components are adapted under federation\textbf{how these components are adapted under federation}. Building on centralized ODG, we found that cross-attention (our ASI) is unsuitable for aggregation: with scarce, heterogeneous client data, aggregated cross-attention parameters cancel out and damage local grounding. This motivated the prompt-only\textbf{prompt-only} protocol-ASI stays local, only prompts are aggregated-a design derived from the federated constraint, not an engineering simplification. Consequently, LalignL_{align} changes meaning: it aligns the globally aggregated prompt with locally diverged ASI modules, not two jointly optimized prompts. aunk\mathbf{a}_{unk} is the negative mean of known attributes, requiring no synthesized open-set images and forming an active rejection boundary locally; Table 2 ablates it against four alternatives. Shallow/Deep prompts are adapted and will be explicitly marked, with the residual form in Eq.(2) as our addition. In short, federation changes which objects are aggregatable\textbf{which objects are aggregatable}, and correspondingly the alignment semantics, the unknown anchor, and the regularization design.

Reviewer_daiMOpenReview ↗

Response to author

Thank you for the response. However, I remain unconvinced by the claimed technical contributions, as most components are adopted from OSLoPrompt with limited modifications, and applying them to a federated setting does not sufficiently establish methodological novelty; therefore, I maintain my original score.

Reviewer_Nek9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FOCUS, a federated prompt learning framework for Federated Open-Set Domain Generalization (FODG), where decentralized source clients with heterogeneous distributions train a model that must classify known classes and reject novel classes in an unseen target domain. The motivation is that existing federated domain generalization methods rely on a closed-set assumption and misclassify unknown samples, especially in data-scarce settings. The main contributions for this paper are (1) define a new problem, (2) an unknown-aware prompt that reformulates the task as a unified ∣C∣+1|C| + 1 prompt learning problem, and (3) an Attribute-Semantic Injection module that employs cross-attention to fuse visual features with textual attributes. Experiments on PACS, VLCS, Office-Home, Multi-Dataset, and Mini-DomainNet in a two-shot setting compare FOCUS against six federated prompt learning and adapted open-set baselines. FOCUS reaches an average H-score of 78.81%, 12.09% above the strongest baseline, while transmitting merely 24K parameters per round.

Critical strengths
  • The experiments are comprehensive. They cover 5 benchmark datasets and 6 baselines, including centralized prompt tuning, federated prompt learning, and adapted open-set methods. All baselines use the same Stable Diffusion pseudo-open samples for a fair comparison. The ablations cover each component, the unknown anchor design, the loss weight, and the number of shots.
Critical weaknesses

Several reported numbers and references in the text do not match the tables. (1) Section 5.1 discusses the main results but cites Table 7, which reports the ViT-B/16 results in Appendix C. (2) the paper also reports that PromptFL yields a 0.0 H-score on PACS (in line 357), but Table 1 reports 73.26% and Table 7 reports 69.99%.

Questions for authors

Can you explain the mismatches between the text and the tables listed in the weaknesses?

2 comments
AuthorsOpenReview ↗

Reply to review Nek9

Thank you for your time and the constructive feedback on our manuscript.

Clarificaiton for inconsistency

Table 7 presents the supplementary ViT-B/16 results in Appendix C. In addition, the “0.0” H-score for PromptFL was an unfilled placeholder, not an actual experimental result. The correct H-scores are 73.26% in Table 1 and 69.99% in Table 7. We have corrected these issues and carefully re-verified all numerical references throughout the manuscript. Thank you for your time and the constructive feedback on our manuscript.

Thanks for your reply. I will maintain my score.

Reviewer_XFLtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FOCUS, a framework that employs an unknown-aware prompt to enable federated VLMs to learn transferable decision boundaries between known and unseen categories across decentralized domains. It further adds a cross-attention-based attribute-semantic injection mechanism to prevent feature entanglement between known and novel categories. It performs extensive experiments across five benchmark datasets under challenging, data-scarce federated settings. Empirical results demonstrate that FOCUS establishes a new SOTA, outperforming the strongest baseline by a substantial margin of 12.09% in average H-score, while drastically reducing communication overhead to merely 24K parameters per round.

Critical strengths
  1. Practical significance of the problem setting: This paper introduces FODG, addressing domain shifts and the identification of novel categories in unseen target domains. It overcomes the closed-set assumption of existing federated domain generalization methods.

  2. Experimental performance: The paper conducts extensive experiments across five benchmark datasets. FOCUS outperforms the strongest baseline by 12.09 percentage points in average H-score, while reducing communication overhead to merely 24K parameters per round. Ablation studies further validate the effectiveness of the key components.

Critical weaknesses
  1. Limited novelty: ODG-CLIP (CVPR 2024) has already employed unknown prompts for open-set recognition, while OSLoPrompt (CVPR 2025) combines cross-attention with attribute-semantic enhancement for open-set domain generalization. Therefore, FOCUS mainly integrates existing techniques into a federated setting, with limited methodological novelty.

  2. Inconsistencies in experimental results: In Table 7 of Appendix C, the average H-scores of FOCUS and FedOSLoPROMPT should be 79.84% and 73.50%, respectively, rather than the reported 78.81% and 66.72%. Moreover, the Average column in Table 7 is identical to that in Table 1. The authors should verify and correct these results.

Questions for authors
  1. Section 5 uses Stable Diffusion to generate pseudo-open samples, but does not clearly describe how the image generation prompts are constructed. Could the authors clarify how the class names and semantic information in these prompts are determined? Are any labels or class names of novel categories in the target domain used?

  2. Section 4.4.2 states that the ASI module is kept local to each client, while the server only aggregates prompt parameters. Could the authors clarify whether the ASI module is used during inference on unseen target domains?

  3. In Table 7 of Appendix C, the Average column is inconsistent with the results across the five datasets. Could the authors verify these values and clarify whether the corrections affect the performance conclusions?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer XFLt

We appreciate reviewer XFLt's detailed evaluation and constructive feedback.

Below, we address your concerns point by point and outline the corresponding revisions made to the manuscript.

1.Clarification for novelty

The core innovation of FOCUS lies not in any individual component, but in redesigning the relationships among components under the constraints of FODG: how the unknown anchor is constructed, where the ASI module is placed, and what should and should not be transmitted. None of these three considerations arise in centralized ODG, yet all of them are unavoidable in the federated setting — and our ablations demonstrate that each choice is necessary.

2.Clarification for inconsistency

The Average column in Table 7 was inadvertently left unchanged when the ViT-B/16 results were entered, resulting in the values from Table 1 being carried over. We have corrected the Average column in Table 7 accordingly, and the overall conclusions remain unchanged. We sincerely appreciate the reviewer’s careful attention to this issue.

3.Usage of Stable Diffusion

Pseudo-open samples are generated locally with Stable Diffusion using LLM-synthesized class names. For each source domain, the positive prompt is "A [source domain] image of a [pseudo-novel class]", where the pseudo-novel class name is generated by GPT and verified to differ from all known classes. The negative prompt lists the known class names to steer generation away from known semantics. No target-domain novel class names, labels, or target-domain information are used.

4.Question of ASI

During inference on unseen target domains, the ASI module does not participate in computation. The ASI module is a training-time, client-local enhancement/regularization mechanism that helps each client absorb instance-specific visual semantics and mitigate prompt drift. At test time, the target domain therefore requires no local training or adaptation. Inference relies solely on the globally shared prompts, which is consistent with the FODG setting where the target domain is unlabeled and unseen.

Thank you for the clarifications and for correcting the results. Your responses have addressed most of my concerns about Stable Diffusion, the use of ASI during inference, and the inconsistencies in Table 7.

However, I still have some concerns about the novelty of the method. I understand why these components are combined in the federated setting, but most of them are based on existing techniques, and I don't see enough new ideas beyond their combination.

For this reason, I will keep my rating at 2.

Comments on the paper

1 comment
Area_Chair_KBYGOpenReview ↗

Encouraging Reviewer Feedback on Author Responses

Dear Reviewers,

Thank you all for your time and effort in reviewing and responding. Just a gentle reminder to reviewers to take a look at the authors' responses and share any updated thoughts on whether your main concerns have been addressed and, if not, to indicate which points are not well addressed. The author–reviewer discussion period ends on November 18 (AoE), so I would greatly appreciate your input before then. Thank you again for your time and contributions!

Kind Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.