acceptodds
Under review as a conference paper at ICLR 2027

Vocabulary-Incomplete Unsupervised Vision-Language Model Adaptation

11 views

Abstract

In certain real-world scenarios, obtaining labeled data is excessively expensive, motivating increasing interest in adapting models without annotations. Vision-language models (VLMs) provide pseudo-labels for unlabeled images through image-text similarity, enabling adaptation without manual labeling. The previous unsupervised vision-language model adaption methods assume access to an adaptation vocabulary containing all class names. However, a complete vocabulary is difficult to construct in real-world scenarios, as unlabeled data may contain classes that are overlooked or fall outside the current task scope. Thus, we propose **V**ocabulary-**I**ncomplete **U**nsupervised **V**ision-language model **A**daptation (VI-UVA), where only a subset of class names are available during adaptation. The vocabulary incompleteness induces an adaptation bias toward provided classes, which degrades omitted-class recognition. To address this issue, we propose **L**abel **S**pace **R**econstruction (LSR), which reconstructs a latent label space from the unlabeled data and incorporates it into prompt tuning, thereby extending adaptation beyond the incomplete vocabulary and mitigating the adaptation bias. Furthermore, we derive an omitted-class semantic discrepancy bound for LSR and identify the key factors affecting the bound. Extensive experiments across eight benchmarks demonstrate that LSR outperforms existing state-of-the-art methods. **Code is available in the supplementary materials.**

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unsupervised Vision-Language Model Adaptation, Vocabulary-Incomplete

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HYLqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces vocabulary-incomplete unsupervised VLM adaptation and proposes Label Space Reconstruction, which reconstructs a latent pseudo-vocabulary from unlabeled images to guide prompt tuning and preserve omitted-class recognition. It provides a theoretical bound on semantic discrepancy and validates the method through baseline comparisons and ablations on eight datasets at three vocabulary omission ratios.

Critical strengths

The paper finds a relevant limitation of unsupervised VLM adaptation, incomplete class vocabularies impairing recognition of omitted classes. The strength for the proposed LSR is reconstructing a pseudo-vocabulary from visual features through cross-modal projection and semantic anchor selection, providing prompt-tuning supervision beyond the available class names. Extensive experiments provide evidence that reconstructing a latent label space helps mitigate this bias.

Critical weaknesses

The paper does not specify how omitted classes are selected or report results across multiple vocabulary partitions. Omitting classes semantically similar to the provided classes may pose a different reconstruction difficulty from omitting semantically distant classes.

Questions for authors
  1. The reported seeds vary optimization, but the selection of missing classes is unclear. How were these classes selected, and did the authors test different vocabulary partitions?

2.What changes did the authors make to baselines originally using visual adapters or full fine-tuning when adapting them to textual prompt tuning?

  1. Table 10 shows that adding LR improves omitted-class accuracy, but the separate roles of LP and LR remain unclear. A comparison of each loss alone and their combination, reporting provided-class accuracy, omitted-class accuracy, and OGR, would help distinguish improved classification from reduced bias toward provided classes.

4.Figure 2 shows that LSR mostly matches zero-shot CLIP on omitted-class recognition and falls slightly below it in some settings. This leaves limited evidence that the reconstructed label space improves recognition of omitted classes beyond the pretrained model’s existing ability.

5 comments

Vocabulary partitions (W1 & Q1)

In our original experiments, the first 1−γ1-\gamma proportion of classes in the class-name list were selected as provided classes, while the remaining classes were treated as omitted classes, where γ\gamma denotes the omission ratio.

To further examine the effect of vocabulary partitions, we repeat the experiment three times with different random class partitions. For each partition, a proportion γ\gamma of classes is randomly selected as omitted classes, and the results are averaged across three partitions.

As shown in Table R1, LSR consistently outperforms all baselines across eight datasets under random vocabulary partitions, achieving an average accuracy of 57.69%. These results demonstrate that LSR remains effective across different vocabulary partitions, supporting its robustness to the selection of omitted classes. We have clarified the vocabulary partition in Appendix G.3 and added the experiments in Appendix H.5, with the new content highlighted in blue.

Table R1 All-class accuracy (%) under random vocabulary partitions (omission ratio = 0.3).

MethodsFlw.FA.CUB.DTD.Pets.Cars.RES.CIFAR.Avg.
CLIP63.6317.6451.7743.1984.5158.9354.4864.1954.79
FPL64.0417.2150.3643.1786.6556.5257.4764.5854.99
GRIP57.1914.6642.3941.0975.5947.0160.3563.0950.17
CPL58.7915.8145.0341.5375.9945.6560.7360.8050.54
MCM59.1414.7545.1938.6882.1151.0351.7464.6750.91
UEO61.996.5045.1141.4284.4653.0760.5653.5550.83
M-Tuning64.3017.2950.2144.2686.4956.6058.2664.9755.30
BGAT59.0117.2347.4040.2182.6754.9752.5964.9852.38
Ours66.5719.2652.4345.3287.5359.4364.8766.0757.69

Baseline adaptation details (Q2)

To ensure a consistent adaptation setting, we adopt textual prompt tuning for all baselines except UEO.

  • FPL, GRIP, CPL, CAP, and AiR are methods developed for unsupervised VLM adaptation (UVA). We adopt their textual prompt-tuning implementations.
  • MCM and BGAT are originally training-free methods. We incorporate their respective scoring and filtering mechanisms into pseudo-label selection, followed by textual prompt tuning.
  • For UEO, its original fine-tuning strategy involves not only textual prompt tuning but also the optimization of channel-wise affine parameters in the visual encoder. We retain this broader fine-tuning strategy in our experiments rather than restricting it to textual prompt tuning.
  • M-Tuning is designed to mitigate overfitting in prompt learning, making it compatible with our textual prompt-tuning setting.

We have clarified the implementation details of these baselines in Appendix G.3, with the new content highlighted in blue.

More detailed ablation study (Q3)

We conduct a more detailed ablation study to investigate the individual and combined effects of LP\mathcal{L}_P and LR\mathcal{L}_R, reporting omitted-class accuracy (OCA), provided-class accuracy (PCA), and omitted-group recall (OGR).

As shown in Table R2, R3 and R4, LP\mathcal{L}_P improves provided-class accuracy but reduces OCA and OGR, indicating increased adaptation bias toward provided classes. In contrast, LR\mathcal{L}_R preserves omitted-class recognition, while providing smaller gains in PCA. Combining both losses achieves high PCA and improves omitted-class recognition over using LP\mathcal{L}_P alone. These results demonstrate their complementary roles in improving PCA while mitigating adaptation bias.

Table R2 Ablation of LP\mathcal{L}_P and LR\mathcal{L}_R on the DTD dataset.

LossOCAPCAOGR
None45.7542.5055.00
LP\mathcal{L}_P42.7546.0853.25
LR\mathcal{L}_R46.2545.5056.50
Both47.5046.1555.75

Table R3 Ablation of LP\mathcal{L}_P and LR\mathcal{L}_R on the OxfordPets dataset.

LossOCAPCAOGR
None64.1964.6167.25
LP\mathcal{L}_P63.3467.2266.54
LR\mathcal{L}_R64.0265.4267.21
Both66.8067.4970.90

Table R4 Ablation of LP\mathcal{L}_P and LR\mathcal{L}_R on the CIFAR dataset.

LossOCAPCAOGR
None92.6382.2993.27
LP\mathcal{L}_P91.7586.0492.41
LR\mathcal{L}_R93.5284.9394.03
Both93.5386.7194.04

Limitation of omitted-class recognition (Q4)

Our primary goal is to mitigate vocabulary overfitting during adaptation while improving overall recognition performance. We acknowledge that LSR does not consistently outperform zero-shot CLIP on omitted classes. One contributing factor is that most of our benchmarks involve fine-grained categories with subtle inter-class differences (e.g., CUB and Flowers102). Preserving omitted-class discrimination in these datasets is particularly challenging without ground-truth labels. Nevertheless, on the relatively coarse-grained like CIFAR-100 dataset, LSR outperforms both zero-shot CLIP and other adaptation methods in omitted-class recognition (Table R7), demonstrating its potential when class distinctions are more apparent.

Table R5 Omitted-class accuracy (OCA) and Omitted-class recall (OCR) on the CIFAR-100 dataset (omission ratio = 0.4).

metricsCLIPFPLGRIPCPLAiRMCMUEOM-TuningBGATLSR (Ours)
OCA65.4064.6859.5049.0558.3863.1561.0065.3565.2867.38 (+1.98)
OGR79.5278.6273.7264.3072.9379.0379.5079.4179.1281.55 (+2.03)

Further improving omitted-class recognition on fine-grained datasets beyond the pretrained model remains a promising direction for future research. We have added a limitation section in Appendix J, with the new content highlighted in blue.

Dear Reviewer HYLq,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 3 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_eaRbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers unsupervised adaptation of CLIP-style models when the adaptation vocabulary is incomplete: the unlabeled data contain images from classes whose names are not given during adaptation, and these names only appear in the evaluation vocabulary. The authors call this setting VI-UVA and argue that standard pseudo-labeling pushes omitted-class images toward the provided classes, so the tuned prompts overfit to the given vocabulary and omitted-class accuracy drops.

To handle this, they propose Label Space Reconstruction (LSR). A linear projector (CMSP) is first trained on high-confidence pseudo-labeled images of the provided classes to map image features into pseudo-token sequences for the frozen text encoder, using an L1 feature regression loss plus a classification loss. All unlabeled images are then projected, the resulting text embeddings are clustered with K-means, clusters with too few images are removed, and the pseudo-token sequences closest to the remaining centers form a pseudo-vocabulary. Shared context tokens are then tuned on two pseudo-labeled sets, one built from this pseudo-vocabulary and one from the provided class names. The paper also derives a bound on the omitted-class semantic discrepancy, which depends on projection quality, cross-modal recomposition and representation approximation.

Experiments are run on eight classification datasets (DTD, FGVC-Aircraft, CUB, Flowers102, Oxford Pets, Stanford Cars, RESISC45, CIFAR-100) with omission ratios such as 0.2 and 0.3. Against ten baselines (including FPL, GRIP, CPL, CAP and BGAT) with the same CLIP backbone and prompt tuning, LSR reports the best all-class accuracy, and the best omitted-class accuracy and omitted-group recall among the adaptation methods, together with ablations on the CMSP losses and an analysis of how many anchors come from omitted classes.

Critical strengths

The paper is well written and logically coherent.

The experimental results are relatively sufficient.

Critical weaknesses

I think the biggest problem with this paper is that the proposed "VI-UVA" lacks practical significance and overlaps heavily with existing settings.

The proposed setting requires that the names of the omitted classes are unavailable during adaptation but available at test time. This is unusual, and its practical realism is not convincingly established. The paper does not provide strong justification for the reasonableness of this setting. In addition, under this condition, since the names are obtained later, why not re-adapt once with the full vocabulary?

What exactly is the "new" scientific problem this paper aims to solve? Section 2.3 and Appendix A (Table 7) compare VI-UVA with open-set recognition, OOD detection and Generalized Category Discovery, but the paper does not discuss its relationship to base-to-new generalization (e.g., CoCoOp), Universal Domain Adaptation, or Text-assisted Image Clustering. Are there methods in these directions that could be borrowed? The authors should discuss these lines of work as well.

Although the authors provide Tables 1-2 to demonstrate the advantages of their method, they do not consider the related lines of work mentioned above (in particular text-assisted image clustering). Therefore, it is unclear whether a combination of existing methods/ideas could effectively solve VI-UVA.

What is the relationship between the proposed CMSP and the TES component of GET[a]? GET is cited in Section 2.3, but only as a GCD method. Both CMSP and TES map visual embeddings into token inputs of the frozen CLIP text encoder, so the authors should explain how CMSP differs from TES and clarify the novelty.

[a] GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery

Questions for authors

See Weaknesses.

7 comments

We appreciate the reviewer's insightful comments and respond to each point below.

Practical significance of VI-UVA (W1)

VI-UVA reflects a practical scenario where the adaptation vocabulary is limited by the current task, while the recognition requirements may expand over time. It aims to improve provided-class accuracy and preserve omitted-class recognition capability during adaptation.

For example, in wildlife monitoring, camera traps collect wildlife data, while researchers often focus on a limited set of species of interest [1, 2]. In such scenarios, researchers may adapt a VLM using unlabeled camera-trap images and a vocabulary containing species of interest, even though these images also contain species beyond the adaptation vocabulary. As the monitoring scope expands, researchers need to recognize some previously omitted species [3, 4].

Moreover, the suggestion to re-adapt the model once assumes that the vocabulary remains fixed after a single expansion or rarely changes. In practical applications, recognition requirements evolve repeatedly, making frequent re-adaptation and deployment costly and time-consuming [5, 6]. BioCLIP [7] also emphasizes the importance of recognizing species beyond the adaptation vocabulary without repeated training. VI-UVA aims to preserve omitted-class recognition during adaptation, allowing their names to be incorporated at inference without re-adaptation. In our evaluation, making all omitted-class names available at once serves as a simplified simulation of future recognition requirements in real-world applications. We have clarified the practical significance of VI-UVA in Appendix A, with the new content highlighted in blue.

[1] ECCV'24-Insect Identification in the Wild: The AMI Dataset

[2] CVPR'25-MammAlps: A Multi-view Video Behavior Monitoring Dataset of Wild Mammals in the Swiss Alps

[3] WACV'25-CATALOG: A Camera Trap Language-Guided Contrastive Learning Model

[4] IJCV'24-WildCLIP: Scene and Animal Attribute Retrieval from Camera Trap Data with Domain-Adapted Vision-Language Models

[5] NeurIPS'24-A Practitioner's Guide to Real-World Continual Multimodal Pretraining

[6] ICCV'25-Adapting In-Domain Few-Shot Segmentation to New Domains without Source Domain Retraining

[7] CVPR'24-BioCLIP: A Vision Foundation Model for the Tree of Life

Relation to existing paradigms (W2)

In the original manuscript, we compare VI-UVA with open-set recognition, OOD detection, and generalized category discovery (GCD) because these paradigms also address classes beyond the available category knowledge. However, open-set recognition and OOD detection primarily focus on rejecting unknown samples, while GCD typically relies on labeled known-class data to discover novel categories. In contrast, VI-UVA requires fine-grained recognition of all classes without ground-truth annotations during adaptation.

We further clarify the relationships with the suggested directions:

  • Base-to-new Generalization. These methods aim to improve generalization from labeled base classes to unseen new classes. During adaptation, only labeled base-class images are used, while new-class images are excluded from training. In contrast, VI-UVA uses entirely unlabeled images containing both provided and omitted classes, while only provided-class names are available. This introduces an additional challenge, as omitted-class images may be incorrectly assigned to provided classes during pseudo-labeling, further increasing adaptation bias. Moreover, VI-UVA requires joint classification over provided and omitted classes using the complete evaluation vocabulary, rather than evaluating base and new classes separately.

  • Universal Domain Adaptation. This paradigm addresses domain and category shifts between a labeled source domain and an unlabeled target domain. It aims to classify samples from shared classes while identifying target-private samples as unknown. In contrast, VI-UVA does not require a labeled source domain or an explicit domain shift, and its adaptation vocabulary contains only provided-class names. More importantly, VI-UVA requires fine-grained recognition of omitted classes rather than treating them as a single unknown category.

  • Text-assisted Image Clustering. This direction exploits external textual semantics (e.g., WordNet) to improve the clustering of unlabeled images. These approaches primarily aim to discover discriminative visual clusters through cross-modal information. In contrast, VI-UVA aims to improve VLM adaptation rather than clustering performance itself.

Overall, VI-UVA differs from these paradigms in both its supervision requirements and evaluation objectives. Although some paradigms address related challenges, they do not fully capture the combined difficulties of limited supervision and demanding recognition requirements in practical applications. We have added the discussion of these related learning paradigms in Section 2.3 and Appendix B, with the new content highlighted in blue.

Experimental comparison with related paradigms (W3)

We have added six baselines covering base-to-new generalization (DPC [1] and KgCoOp [2]), universal domain adaptation (TASC [3] and EGRS [4]), and text-assisted image clustering (TAC [5] and GradNorm [6]). We adapt all baselines to VI-UVA and the details are as followed:

  • DPC: We retain dual-prompt collaboration and hard-negative optimization, replacing labeled base-class supervision with provided-class pseudo-labels.

  • KgCoOp: Provided-class pseudo-labels are used for training, with a regularizer that aligns learned text features with original CLIP counterparts.

  • TASC: We adopt WordNet-based semantic search, information maximization and optimize textual prompts instead of the encoder.

  • EGRS: We train a source prompt with provided-class pseudo-labels, then combine frozen CLIP knowledge with feature perturbations and dynamic prototype correction to guide adaptation.

  • TAC: Relevant WordNet nouns are selected to represent unlabeled images, with cross-modal neighborhood consistency and cluster distillation guiding prompt tuning.

  • GradNorm: We train a classifier on unsupervised visual clusters, select WordNet nouns through gradient norms, and exploit joint visual-textual clustering to generate auxiliary targets for prompt tuning.

As shown in Table R1 and R2, LSR outperforms all six baselines on each dataset, achieving average accuracies of 58.52% and 57.72% at omission ratios of 0.2 and 0.3. Other methods do not address the adaptation bias caused by vocabulary incompleteness. DPC and KgCoOp rely on provided-class supervision, while TASC and EGRS remain sensitive to the limited semantic information. TAC and GradNorm exploit additional semantic structures, but do not reconstruct a pseudo-vocabulary to mitigate adaptation bias during prompt tuning. In contrast, LSR incorporates reconstructed latent semantics into prompt tuning, providing additional supervision beyond the provided vocabulary. We have added the comparison experiments with these related learning paradigms in Appendix H.4, with the new content highlighted in blue.

Table R1 All-class accuracy (%) comparison with other baselines (Omission ratio = 0.2).

MethodsFlw.FA.CUB.DTD.Pets.Cars.RES.CIFAR.Avg.
DPC66.6117.5250.8145.6988.0857.2159.3464.5356.22
KgCoOp66.5517.9448.1346.4887.6358.3262.8466.3056.77
TASC65.3417.7651.0143.6182.5057.2555.4457.7553.83
EGRS66.5118.7552.0846.0186.4858.4359.4866.2256.75
TAC66.2916.7150.8445.8587.7357.1558.2665.4356.03
GradNorm66.5717.6751.7945.1586.9157.1458.3065.2556.10
Ours66.8919.3453.2747.0288.4960.1565.8267.1458.52

Table R2 All-class accuracy (%) comparison with other baselines (Omission ratio = 0.3).

MethodsFlw.FA.CUB.DTD.Pets.Cars.RES.CIFAR.Avg.
DPC62.2017.3150.5642.5584.2156.0256.7064.5854.27
KgCoOp64.4917.2250.7944.5187.0657.5662.3366.0456.25
TASC63.4017.2250.1540.2173.0955.0856.2555.4351.35
EGRS65.0918.7851.6743.2986.0457.9657.8866.0255.84
TAC64.1818.1249.4442.8782.7256.3958.2365.0954.63
GradNorm65.4017.1949.8443.2984.8456.1257.2365.0954.88
Ours65.5018.9253.0045.3287.5359.6865.2666.5357.72

[1] CVPR'25-DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models

[2] CVPR'23-Visual-Language Prompt Tuning With Knowledge-Guided Context Optimization

[3] AAAI'25-Target Semantics Clustering via Text Representations for Robust Universal Domain Adaptation

[4] TMM'26-Exploring Generic Knowledge and Reactivating Source Model for Source-Free Universal Domain Adaptation

[5] ICML'24-Image Clustering with External Guidance

[6] ICCV'25-On the Provable Importance of Gradients for Autonomous Language-Assisted Image Clustering

Reviewer_7qTJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies vocabulary-incomplete unsupervised vision-language model adaptation (VI-UVA), where unlabeled adaptation images contain classes whose names are unavailable during adaptation. The authors propose Label Space Reconstruction (LSR) to mitigate the bias toward provided classes. LSR first trains a cross-modal semantic projector to map image features into pseudo-token sequences. These embeddings are clustered to discover latent semantic anchors, and representative pseudo-token sequences are retrieved to construct a pseudo-vocabulary. Shared context tokens are then optimized using supervision from both the reconstructed pseudo-vocabulary and the provided class names. The paper evaluates the method on eight datasets, reporting improved overall accuracy over competing adaptation methods while preserving zero-shot CLIP’s omitted-class recognition.

Critical strengths
  1. The paper provides a broad empirical evaluation across eight datasets and ten baselines, including recent methods such as BGAT (2026).
  2. The paper provides a theoretical perspective through an omitted-class semantic discrepancy bound, offering an interpretation of the method’s design.
Critical weaknesses
  1. CMSP is trained using provided-class prototypes as supervision. It remains unclear why this mapping should preserve or improve distinctions among omitted classes, particularly when omitted-class images may receive incorrect provided-class pseudo-labels.

  2. A cluster may contain images from different classes, such as dogs and wolves. Using a single representative for this mixed cluster could encourage the model to overlook meaningful class distinctions. Although the paper give the number of clusters, the effect of cluster number purity on downstream performance remains unclear.

  3. The bound identifies three relevant terms, εproj\varepsilon_{\mathrm{proj}}, bob_o, and ror_o, but the experiments do not directly measure whether the proposed components reduce these quantities. This leaves the connection between the theoretical analysis and empirical improvements insufficiently established.

  4. In Section 4.4, omitted-group recall and omitted-class accuracy do not consistently surpass zero-shot CLIP and sometimes fall below it. These results provide limited evidence that reconstruction improves omitted-class discrimination beyond the pretrained model.

Questions for authors
  1. The setting assumes omitted-class names are unavailable during adaptation but available at inference. Could the authors clarify a concrete deployment scenario that requires this separation? If the complete vocabulary can be obtained during inference, why would one adapt using an incomplete vocabulary?

  2. Could you compare clustering original CLIP image features with clustering CMSP-generated embeddings, particularly in terms of omitted-class separation?

  3. How frequently do retained clusters mix different ground-truth classes, especially provided and omitted classes? Could you report cluster purity and analyze how mixed clusters affect recognition performance? The initial cluster number is 2∣Cp∣2|C_p |, could you do the parameter sensitivity analysis on cluster number?

  4. Could you measure εproj\varepsilon_{\mathrm{proj}}, bob_o, and ror_o, or suitable empirical proxies, across relevant ablations to validate the proposed theoretical explanation?

  5. Given the limited gains over zero-shot CLIP on omitted-class metrics, should the contribution primarily be interpreted as preventing adaptation-induced degradation? Under what conditions does LSR consistently improve omitted-class recognition beyond CLIP?

8 comments

We appreciate the reviewer's insightful comments and respond to each point below.

CMSP preserves distinctions (W1)

Although CMSP is trained with provided-class prototypes, the learned mapping can generalize to omitted classes through shared visual attributes [1, 2]. For example, different bird species share semantic attributes such as feather colors, wing shapes, and beak structures. Learning to map these attributes into the textual space can also benefit omitted bird species, even without their class names.

Incorrect pseudo-labels for omitted-class images are inevitable. However, high-confidence sample selection helps reduce such noise, limiting its influence on CMSP training. Moreover, CMSP learns a shared cross-modal mapping rather than a classifier restricted to provided classes. When applied to all unlabeled images, it can still capture semantic differences beyond the provided vocabulary, as supported by the omitted-class coverage results in Table 4 of the original manuscript.

[1] CVPR'25-Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning

[2] CVPR'24-Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning

Effect of mixed clusters and cluster number (W2 & Q3)

In unsupervised fine-grained classification, clusters containing multiple ground-truth classes are difficult to avoid. However, LSR does not require clusters to correspond one-to-one with ground-truth classes. Instead, it exploits semantic distinctions among different clusters to provide auxiliary supervision through LR\mathcal{L}_R, preventing the model from being adapted solely toward provided classes. Meanwhile, LP\mathcal{L}_P maintains provided-class discrimination.

  • To quantify the mixing between provided and omitted classes, we define the provided/omitted mixing ratio ($\downarrow $) for each retained cluster as: MixingRatiok=min⁡(nkp,nko)nkp+nko,\mathrm{MixingRatio}_k=\frac{\min(n_k^p,n_k^o)}{n_k^p+n_k^o}, where nkpn_k^p and nkon_k^o denote the numbers of provided- and omitted-class samples in cluster kk, respectively. We average this ratio over all retained clusters to obtain the overall mixing ratio (i.e., MixingRatio\mathrm{MixingRatio}). As shown in Table R1, MixingRatio\mathrm{MixingRatio} remains relatively low, with average values of 13.47%, 17.79%, and 21.04% at omission ratios of 0.2, 0.3, and 0.4, respectively. These results suggest that the clusters can still maintain a certain degree of separation between provided and omitted classes, even with an increasingly incomplete adaptation vocabulary .

    Table R1 MixingRatio\mathrm{MixingRatio} ($\downarrow $) at differnet omission ratios (cluster number = 2∣Cp∣2|\mathcal{C}_p|).

    Dataset0.20.30.4
    FA.16.8520.7525.64
    CUB.8.3412.0416.53
    Pets.14.5816.0218.05
    CIFAR.12.9916.3519.63
    Avg.13.4717.7821.04
  • We further analyze the effects of cluster number on cluster purity ($\uparrow $) and $\mathrm{MixingRatio} $ ($\downarrow $). As shown in Tables R2 and R3, increasing the number of clusters generally improves cluster purity while reducing the mixing ratio. These results suggest that our over-clustering strategy helps preserve finer-grained semantic distinctions and improves the separation between provided and omitted classes.

    Table R2 MixingRatio\mathrm{MixingRatio} ($\downarrow $) at differnet cluster ratios (cluster number / ∣Cp∣|\mathcal{C}_p|) (omission ratios = 0.4).

    Dataset0.10.21.02.03.0
    DTD.37.6033.7630.2029.4726.86
    Flw.30.6428.8618.0815.2514.15
    Cars.29.1329.7022.3319.4419.14
    RES.37.2538.2328.0424.3325.13

    Table R3 Cluster purity ($\uparrow $) at differnet cluster ratios (cluster number / ∣Cp∣|\mathcal{C}_p|) (omission ratios = 0.4).

    Dataset0.10.21.02.03.0
    DTD.5.4722.2441.7243.3745.95
    Flw.11.0916.4061.9263.4966.11
    Cars.8.4615.6643.8747.5848.58
    RES.7.9118.9549.2754.9153.70
  • Finally, we investigate the impact of cluster number and purity on overall recognition performance. As shown in Table R4, together with the purity results in Table R3, increasing the cluster number generally improves classification performance by enhancing cluster purity. However, further increasing the cluster number (i.e. cluster number > 2∣Cp∣2|\mathcal{C}_p|) yields limited improvement.

    Table R4 All-class accuracy ($\uparrow $) of LSR at differnet cluster ratios (cluster number / ∣Cp∣|\mathcal{C}_p|) (omission ratios = 0.4).

    Dataset0.10.21.02.03.0
    DTD.44.0945.3745.3146.8645.53
    Flw.63.2163.6663.3564.5365.19
    Cars.57.7658.5858.5059.2457.97
    RES.62.0263.2463.5564.5263.77

Empirical Validation of the Theoretical Analysis (W3 & Q4)

Since bob_o primarily characterizes the dataset-dependent cross-modal structural compatibility between provided and omitted classes, we focus empirical validation on projection error εproj\varepsilon_{\mathrm{proj}} and representation approximation error ror_o.

Projection error $\varepsilon_{\mathrm{proj}} $ (↓\downarrow). Since εproj\varepsilon_{\mathrm{proj}} involves a global smoothness constant that is difficult to estimate, we measure the empirical proxy:

$\varepsilon_{\mathrm{proj}} =\max_{j\in C_p}|F(\bar{x}_j)-c_j|_2,$

where xjx_j is the mean visual representation of the selected samples assigned pseudo-label jj, and cjc_j is the corresponding provided-class textual prototype. FF represents the complete CMSP mapping. This proxy measures the center-to-prototype discrepancy controlled by εproj\varepsilon_{\mathrm{proj}} in Lemma 2.

Representation approximation error ror_o (↓\downarrow). We directly compute ror_o according to its theoretical definition:

$r_o= \sqrt{\frac{1}{N_o}\sum_{i\in I_o} \min_{a\in T_R}|t_i-a|_2^2}.$

ti=F(xi)t_i=F(x_i) denotes the generated textual representation of xix_i, IoI_o indexes omitted-class samples, and No=∣Io∣N_o=|I_o|. TRT_R contains the retrieved textual representatives.

Table R5 shows that the complete CMSP loss produces lower εproj\varepsilon_{\mathrm{proj}} and ror_o than individual loss variant. We further investigate the effect of cluster number on ror_o. As shown in Table R6, over-clustering effectively reduces ror_o. We have added the empirical validation in Appendix H.6, with the new content highlighted in blue.

Table R5 Complete CMSP loss acheives lower εproj\varepsilon_{\mathrm{proj}} and ror_o.

Datasetεproj\varepsilon_{\mathrm{proj}} - Individualεproj\varepsilon_{\mathrm{proj}} - CMSPror_o - Individualror_o - CMSP
Flw.1.100.740.720.55
FA.1.170.910.640.37
CUB.1.331.070.710.54
DTD.1.160.860.650.41
Pets.0.860.580.520.33
Cars.1.160.690.600.43
RES.1.110.850.770.47
CIFAR.1.100.930.580.40

Table R6 ror_o at different cluster ratios (cluster number / ∣Cp∣|\mathcal{C}_p|).

Dataset0.10.21.02.03.0
Flw.0.830.760.590.550.56
FA.0.570.470.380.370.37
CUB.0.730.670.570.540.54
DTD.0.770.710.420.410.40
Pets0.940.770.400.330.32
Cars0.760.660.460.430.42
RES.0.820.700.500.470.45
CIFAR.0.770.700.410.400.43

Comments on the paper

2 comments

Dear Reviewers,

We are currently in the discussion phase. The authors have responded to the comments, and I would greatly appreciate it if you could take a moment to check the rebuttal and leave any follow-up comments or questions. If the response has addressed your concerns, please also consider updating your review accordingly.

Your feedback is very valuable to both the authors and the decision process. Thank you again for your help.

Best regards, AC

Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely thank the reviewers for their thoughtful comments and valuable questions, which have significantly strengthened our paper. We also deeply appreciate the Area Chair's time and careful handling of the review process.

To facilitate your assessment of our submission, we offer a brief summary of the review process and the improvements made during the rebuttal phase.

Summary of Rebuttal

Reviewer 7qTJ (Original Score: 2): The reviewer acknowledged our extensive experimental evaluation and theoretical analysis, but raised concerns about reconstruction reliability, cluster mixing, and omitted-class recognition. We addressed these concerns through additional clustering analyses, parameter sensitivity experiments, and empirical validation of the theoretical bound. We also clarified the deployment scenario and the limitations of VI-UVA. Although the reviewer did not engage in discussion, we believe these additions address the reviewer's concerns and justify reconsideration of the original score.

Reviewer eaRb (Original Score: 2): The reviewer acknowledged our clear presentation and sufficient experimental results, but questioned the practical significance and novelty of VI-UVA. We addressed these concerns by clarifying the practical motivation, expanding comparisons with related learning paradigms, and introducing six additional baselines. We also clarified the differences between CMSP and the TES component in GET. Although the reviewer did not engage in discussion, we believe our additional experiments and clarifications support a more favorable evaluation of our work.

Reviewer HYLq (Original Score: 3): The reviewer recognized the importance of addressing vocabulary incompleteness and the effectiveness of LSR, but raised concerns about vocabulary partitioning, baseline fairness, and omitted-class recognition. We addressed these concerns through random vocabulary partition experiments and additional ablations. We also clarified the baseline implementations and the limitations of VI-UVA. The reviewer maintained a positive score, reflecting a favorable assessment of our work.

Key Revisions

The key revisions are summarized below:

Practical motivation and related paradigms (Sec.2.3; Apps.A-B, Tab.7): We clarify the practical motivation of VI-UVA and its distinctions from base-to-new generalization, universal domain adaptation, and text-assisted image clustering.

Methodological novelty (Sec. 3.1): We clarify the relationship between CMSP and the TES component in GET, emphasizing the role of CMSP in latent label space reconstruction.

Additional baseline comparisons (App.H.4, Tabs.12-13): We introduce six baselines from related learning paradigms. LSR outperforms all six methods across eight datasets, achieving average accuracies of 58.52% and 57.72% at omission ratios of 0.2 and 0.3, respectively.

Implementation details (App.G.3): We clarify how existing baselines are adapted to the VI-UVA setting, including the use of textual prompt tuning and the integration of scoring and filtering mechanisms into pseudo-label selection.

Robustness to vocabulary partitions (App.H.5, Tab.14): We conduct experiments with three random vocabulary partitions. LSR achieves 57.69% average accuracy, outperforming the strongest baseline at 55.30%.

Empirical theoretical validation (App.H.6, Tabs.15-16): We measure empirical projection and representation approximation errors, demonstrating the benefits of the complete CMSP objective and over-clustering.

Limitations (App.J): We explicitly discuss the challenges of surpassing zero-shot CLIP on omitted classes, particularly on fine-grained datasets.

Detailed responses to individual concerns are provided in the corresponding reviewer discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.