acceptodds
Withdrawn from ICLR 2027

Visual Structure-Guided Textual Semantic Space Adaptation for Domain-Shifted Generalized Category Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Domain-Shifted Generalized Category Discovery (DS-GCD) requires categorizing an unlabeled target domain containing both source-seen and target-private categories under domain shift. Existing VLM-based approaches commonly cluster target visual features and represent discovered categories with anonymous visual prototypes, leaving the textual space unused as the target assignment coordinates. We investigate an alternative division of roles: target visual geometry provides unlabeled structural evidence, whereas text embeddings define the coordinates of the target assignment space. We propose Visual Structure Guided Textual Semantic Space Adaptation (ViSTA), which uses target visual structure to search a predefined noun vocabulary for textual anchors and then adapts CLIP within the resulting textual semantic space. In Stage 1, ViSTA filters candidate nouns and greedily selects anchors according to the compactness and separation of the target partitions they induce, under a given target category budget. In Stage 2, the selected words remain fixed while source supervision and target regularization adapt both CLIP encoders. Across four DS-GCD benchmarks and three seen/unseen splits, ViSTA attains a 75.76% average H-Score over 12 settings. These results support textual anchors as category coordinates while retaining target visual geometry as selection evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
category discovery, domain adaptation, vision-language learning, semantic representations, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EHJqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes ViSTA for Domain-Shifted Generalised Category Discovery (DS-GCD). It’s a two-stage approach. Stage 1 constructs a textual semantic space by searching for unseen textual anchors from a WordNet noun pool, selecting the anchors whose induced target partitions are most compact and separated in the visual feature space. Stage 2 keeps the selected words fixed and adapts CLIP with LoRA to refine the target assignments. The proposed approach is evaluated on four openly available DS-GCD benchmark datasets.

Critical strengths
  1. The idea is sound: it separates the roles of the two modalities, with visual structure as evidence and text as category coordinates.
  2. Code is attached for reproducibility.
Critical weaknesses
  1. The evaluation is not convincing enough. Most baselines were published between 2021 and 2023; only CROW (2024) and an adapted SpectralGCD (2026, VisDA only) are more recent. Given the fast-moving research, the authors should include more recent work.
  2. Very narrow design choice on the validation. All ablations and analyses (Tables 3–6, Figure 3) are on VisDA 6/6, which is too thin to support the claims.
  3. It’s unclear how much of the gain comes from the proposed search and how much from CLIP’s zero-shot knowledge of the target categories. The WordNet vocabulary contains the target class names.
Questions for authors
  1. How did the authors obtain the numbers for DANCE, OVANet, UniOT, GLC and CROW? If by re-running, please state and provide details.
  2. How are the initial anchors Ainit\mathcal{A}_{init} in Table 5 constructed? The other three variants improve the H-Score by 27 to 40 points over this V00 baseline.
  3. Can the authors compare the proposed approach against state-of-the-art approaches published in 2024–26, such as the DS-GCD methods cited in the introduction, for a comprehensive evaluation?
Reviewer_W8BfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ViSTA, which is used for generalized category discovery under domain shift conditions. The method first uses the visual features of the target image to filter out candidate words from WordNet, and then selects the text anchors based on the compactness and separation degree of the image groups formed by these text anchors. After fixing the selected words, it uses the labeled source domain images and unlabeled target domain images to adapt the two encoders of CLIP through LoRA. The experiments cover four benchmarks and three different seen/unseen category divisions.

Critical strengths

1.The category discovery under domain shift studied in this paper is of practical significance. The core idea is clear and intuitive. The method uses the visual structure of target images to select textual anchors and introduces the textual knowledge of CLIP into category discovery. At the same time, it does not require the names of unseen categories as input.

2.The experiments involve four benchmarks and three different class splits. The scope of the experiment is quite comprehensive. Moreover, the ablation study clearly shows the specific effects of candidate filtering, anchor search, and model adaptation. The paper also compares anchor search and model adaptation separately and in combination, which helps to understand the method design.

Critical weaknesses

1.This paper lacks some methods: HiLo (ICLR 2025) and FREE (NeurIPS 2025). These methods are category discovery approaches under domain shift, and they also use CLIP for experiments. It is necessary to provide a comparison of these methods so as to conduct a more comprehensive evaluation of this method.

2.The computational cost of this method still needs to be explained. This method includes two stages: anchor point search and model adaptation. However, the paper does not report the time and memory required for these two stages. The author is hoped to supplement the comparative experiments with SpectralGCD under the same hardware conditions, especially for the DomainNet dataset. At the same time, please report the computational costs of these two stages separately, which will help evaluate the practical application value of this method.

Questions for authors

I recommend weak rejection. I will also read the other reviewers' comments to see whether I have overlooked any important points. Please refer to the weaknesses above for specific questions.

Reviewer_GgrhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ViSTA for domain-shifted generalized category discovery (DS-GCD), where labeled source images cover seen classes, while unlabeled target images contain both seen and unseen classes under domain shift. ViSTA first constructs a target textual semantic space using source class names and candidate nouns from WordNet. It filters candidates using visual evidence and performs greedy semantic search guided by a silhouette-based score to select textual anchors for unseen categories. It then adapts CLIP’s visual and textual encoders using LoRA, combining source classification, target information maximization, and prediction-level contrastive consistency. Experiments on Office, OfficeHome, VisDA, and DomainNet demonstrate improved average H-score over the compared baselines.

Critical strengths

The paper is clearly written and easy to follow, with detailed descriptions of each stage and component. Figure 2 provides a helpful overview of the ViSTA framework.

Critical weaknesses
  1. The proposed method ViSTA is not novel enough. Representing categories with textual prototypes is standard in vision-language models, and ViSTA combines established filtering, clustering-quality scoring, search, and adaptation objectives. The contribution of visual-structure-guided anchor selection requires clearer differentiation from prior concept-selection methods such as SpectralGCD [1]. Describing textual anchors as “assignment coordinates” does not, by itself, establish a new mechanism.

  2. Stage 1 requires similarity computations across an initial WordNet noun pool, followed by greedy semantic search. However, the paper does not clearly report the initial vocabulary size or provide runtime and memory measurements.

  3. Except for SpectralGCD-SC, all compared methods were published before 2024. However, the authors do not report SpectralGCD-SC results across all datasets. For example, Table 1 lacks its results on Office, OfficeHome, and DomainNet.

  4. The results about CROW shown in this paper didn’t match the original one. For example, in Table 1, the reported Office H-scores for CROW [2] (71.6, 74.7, and 79.7) appear substantially lower than the original results identified by the reviewer (84.7, 84.9, and 85.6). The paper should clarify whether differences in the backbone, evaluation protocol, or implementation explain this discrepancy before conclusions about comparative performance can be assessed.

  5. ViSTA does not consistently outperform competing methods on Office, particularly under the 21/10 split. The paper provides limited analysis of why its benefits vary across datasets and category splits.

  6. Table 3 shows that PCL alone achieves 93.1% seen accuracy, compared with 85.6% for the full method. Although the authors acknowledge that PCL favors seen recognition, they do not explain the mechanism behind this substantial difference or analyze how the additional objectives affect seen-class predictions.

[1] Caselli, Lorenzo, et al. "SpectralGCD: Spectral concept selection and cross-modal representation learning for generalized category discovery." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Wen, Shuo, and Maria Brbic. "Cross-domain open-world discovery." arXiv preprint arXiv:2406.11422 (2024).

Questions for authors

Question:

  1. What is the key methodological distinction between ViSTA’s anchor selection and prior concept-selection approaches such as SpectralGCD? Could you compare against a simpler baseline that clusters image features and retrieves a textual prototype for each cluster?

  2. How many nouns are included in the initial and filtered vocabularies from wordnet? Could you report Stage-1 and Stage-2 runtime, peak GPU memory, and how semantic search scales with vocabulary and dataset size?

  3. Why is SpectralGCD-SC evaluated only on VisDA? Could you provide its results on the other three benchmarks or explain the constraints preventing these evaluations?

  4. Could you reconcile the CROW results with those in the original paper and specify any differences in backbone, splits, metrics, or training protocol?

  5. Why does ViSTA underperform the strongest baselines on some Office splits? Are these cases associated with poor anchor selection, domain shift, or reduced seen-class accuracy?

  6. Why does PCL-only achieve substantially higher seen accuracy than the full objective? Could you analyze which predictions change when IM and source CE are introduced?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.