- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces vocabulary-incomplete unsupervised VLM adaptation and proposes Label Space Reconstruction, which reconstructs a latent pseudo-vocabulary from unlabeled images to guide prompt tuning and preserve omitted-class recognition. It provides a theoretical bound on semantic discrepancy and validates the method through baseline comparisons and ablations on eight datasets at three vocabulary omission ratios.
Critical strengths
The paper finds a relevant limitation of unsupervised VLM adaptation, incomplete class vocabularies impairing recognition of omitted classes. The strength for the proposed LSR is reconstructing a pseudo-vocabulary from visual features through cross-modal projection and semantic anchor selection, providing prompt-tuning supervision beyond the available class names. Extensive experiments provide evidence that reconstructing a latent label space helps mitigate this bias.
Critical weaknesses
The paper does not specify how omitted classes are selected or report results across multiple vocabulary partitions. Omitting classes semantically similar to the provided classes may pose a different reconstruction difficulty from omitting semantically distant classes.
Questions for authors
- The reported seeds vary optimization, but the selection of missing classes is unclear. How were these classes selected, and did the authors test different vocabulary partitions?
2.What changes did the authors make to baselines originally using visual adapters or full fine-tuning when adapting them to textual prompt tuning?
- Table 10 shows that adding LR improves omitted-class accuracy, but the separate roles of LP and LR remain unclear. A comparison of each loss alone and their combination, reporting provided-class accuracy, omitted-class accuracy, and OGR, would help distinguish improved classification from reduced bias toward provided classes.
4.Figure 2 shows that LSR mostly matches zero-shot CLIP on omitted-class recognition and falls slightly below it in some settings. This leaves limited evidence that the reconstructed label space improves recognition of omitted classes beyond the pretrained model’s existing ability.
5 comments
More detailed ablation study (Q3)
We conduct a more detailed ablation study to investigate the individual and combined effects of and , reporting omitted-class accuracy (OCA), provided-class accuracy (PCA), and omitted-group recall (OGR).
As shown in Table R2, R3 and R4, improves provided-class accuracy but reduces OCA and OGR, indicating increased adaptation bias toward provided classes. In contrast, preserves omitted-class recognition, while providing smaller gains in PCA. Combining both losses achieves high PCA and improves omitted-class recognition over using alone. These results demonstrate their complementary roles in improving PCA while mitigating adaptation bias.
Table R2 Ablation of and on the DTD dataset.
| Loss | OCA | PCA | OGR |
|---|---|---|---|
| None | 45.75 | 42.50 | 55.00 |
| 42.75 | 46.08 | 53.25 | |
| 46.25 | 45.50 | 56.50 | |
| Both | 47.50 | 46.15 | 55.75 |
Table R3 Ablation of and on the OxfordPets dataset.
| Loss | OCA | PCA | OGR |
|---|---|---|---|
| None | 64.19 | 64.61 | 67.25 |
| 63.34 | 67.22 | 66.54 | |
| 64.02 | 65.42 | 67.21 | |
| Both | 66.80 | 67.49 | 70.90 |
Table R4 Ablation of and on the CIFAR dataset.
| Loss | OCA | PCA | OGR |
|---|---|---|---|
| None | 92.63 | 82.29 | 93.27 |
| 91.75 | 86.04 | 92.41 | |
| 93.52 | 84.93 | 94.03 | |
| Both | 93.53 | 86.71 | 94.04 |
Limitation of omitted-class recognition (Q4)
Our primary goal is to mitigate vocabulary overfitting during adaptation while improving overall recognition performance. We acknowledge that LSR does not consistently outperform zero-shot CLIP on omitted classes. One contributing factor is that most of our benchmarks involve fine-grained categories with subtle inter-class differences (e.g., CUB and Flowers102). Preserving omitted-class discrimination in these datasets is particularly challenging without ground-truth labels. Nevertheless, on the relatively coarse-grained like CIFAR-100 dataset, LSR outperforms both zero-shot CLIP and other adaptation methods in omitted-class recognition (Table R7), demonstrating its potential when class distinctions are more apparent.
Table R5 Omitted-class accuracy (OCA) and Omitted-class recall (OCR) on the CIFAR-100 dataset (omission ratio = 0.4).
| metrics | CLIP | FPL | GRIP | CPL | AiR | MCM | UEO | M-Tuning | BGAT | LSR (Ours) |
|---|---|---|---|---|---|---|---|---|---|---|
| OCA | 65.40 | 64.68 | 59.50 | 49.05 | 58.38 | 63.15 | 61.00 | 65.35 | 65.28 | 67.38 (+1.98) |
| OGR | 79.52 | 78.62 | 73.72 | 64.30 | 72.93 | 79.03 | 79.50 | 79.41 | 79.12 | 81.55 (+2.03) |
Further improving omitted-class recognition on fine-grained datasets beyond the pretrained model remains a promising direction for future research. We have added a limitation section in Appendix J, with the new content highlighted in blue.
Dear Reviewer HYLq,
Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 3 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.
Regards from the authors.
Vocabulary partitions (W1 & Q1)
In our original experiments, the first 1−γ proportion of classes in the class-name list were selected as provided classes, while the remaining classes were treated as omitted classes, where γ denotes the omission ratio.
To further examine the effect of vocabulary partitions, we repeat the experiment three times with different random class partitions. For each partition, a proportion γ of classes is randomly selected as omitted classes, and the results are averaged across three partitions.
As shown in Table R1, LSR consistently outperforms all baselines across eight datasets under random vocabulary partitions, achieving an average accuracy of 57.69%. These results demonstrate that LSR remains effective across different vocabulary partitions, supporting its robustness to the selection of omitted classes. We have clarified the vocabulary partition in Appendix G.3 and added the experiments in Appendix H.5, with the new content highlighted in blue.
Table R1 All-class accuracy (%) under random vocabulary partitions (omission ratio = 0.3).
Baseline adaptation details (Q2)
To ensure a consistent adaptation setting, we adopt textual prompt tuning for all baselines except UEO.
We have clarified the implementation details of these baselines in Appendix G.3, with the new content highlighted in blue.