- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tackles the problem of CLIP-based few-shot classification. The proposed method, named TCOC (for text-conditioned orthogonal compensation), consists of 2 main elements. The first is the computation of a text-conditioned residual which is generated by a learned network that takes as input the text prototypes projected onto the text-coordinate basis formed by singular vectors. The residual is not directly used, it is projected onto the orthogonal complement of the principal text-classifier subspace. The projected residual is then added to the original zero-shot classifier. Experiments span few-shot classification, domain generalization, base-to-new generalization and cross-dataset generalization, and the proposed method exhibits better performance than previous methods.
Critical strengths
Motivating the insufficiency of few-shot adaptation methods to cover the visual features directions from a modality-gap perspective is interesting. This discussion makes the "orthogonal compensation" part of the method not merely an arbitrary design choice. On the other hand, the experiments are comprehensive and include different aspects of adaptation (few-shot classification, domain generalization, cross-dataset generalization, base-to-new generalization). The performance appears to be competitive to multiple state-of-the-art methods. I also find the complementarity to CLIP-SVD results very interesting (Table 3).
Critical weaknesses
One of my main concerns about the paper is related to the experimental protocol and the reported results. Concerning the experimental setting, the authors claim using the validation sets of each dataset to perform hyperparameter selection. While this has been done in earlier methods, there is a tendency in the literature to adopt a "few-shot validation sets" [1] and a "validation-free" [2,3,4] settings. This is so important because the few-shot adaptation setting is motivated by data-scarcity in some domains in real-world applications. So supposing having access to validation sets where the number of samples significantly exceeds the number of support set samples may inflate the gains.
For the base-to-new generalization results, some methods are missing from Table 6. For instance, it is unclear what is the practical advantage of the proposed method compared to MMRL [5] and VaMP [6] which outperform it on base-to-new generalization. Moreover, the paper uses specific prompt templates for each dataset, how does this affect the results compared to using a single template? It is not clear whether all the methods used for comparison adopt the same template; this is crucial for comparison fairness. For instance, [4] achieves 79.08% (harmonic mean) on base-to-new generalization, which is not very far from the paper's reported result (80.29%), while using a single template "a photo of {}." for all datasets. What part of this difference is due to the difference in prompt templates?
For the ablations, I believe Table 7 does not give enough evidence on the role of the text-conditioned generator and the orthogonal compensation, since it is confined to cross-dataset generalization which is tested with a single architecture under a single few-shot setting (ImageNet 16-shot). The same ablation is necessary for few-shot classification, domain generalization and base-to-new generalization to support the advantage of including both elements.
[1] Huang et al., LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP. CVPR 2024
[2] Silva-Rodríguez et al., A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models. CVPR 2024
[3] Zanella and Ben Ayed, Low-Rank Few-Shot Adaptation of Vision-Language Models. CVPRW 2024
[4] Fahes et al., CLIP’s Visual Embedding Projector is a Few-shot Cornucopia. WACV 2026
[5] Guo and Gu, MMRL: Multi-Modal Representation Learning for Vision-Language Models. CVPR 2025
[6] Cheng and Han, VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models. NeurIPS 2025
Questions for authors
-
In Section 3.2, the authors state: "When text features are used as classifier weights, the subspace spanned by the text classifier may not fully cover the discriminative directions of downstream visual features." (Lines 137-139). Isn't this problem also possible when there's no modality gap?
-
What makes TCOC complementary to CLIP-SVD? CLIP-SVD tunes parameters in the text branch, so I would intuitively suppose the complementarity would be stronger for methods that only tune parameters in the visual encoder for instance. Does this complementarity hold for other methods?
Suggestions
- It would be great to include the standard deviations in the tables.
7 comments
Response to W1 — Validation Budget and Hyperparameter Selection (2/5)
Thank you for raising the question of how many additional labeled images are used for validation in a few-shot setting. Our statement that hyperparameters were selected “on the corresponding validation sets” did not specify the validation budget or explain how we handled ImageNet. We built our few-shot experiments on the official TaskRes implementation [1].
On the 10 datasets other than ImageNet, each (K)-shot experiment uses (K) labeled training images per class to fit TCOC. We sample at most (\min(K,4)) additional labeled images per class from a separate validation split to select the initial learning rate from ({10^{-3}, 5\times10^{-4}, 10^{-4}}). Thus, at every shot count, the validation set is at most as large as the training support set. On ImageNet, we follow the TaskRes supplementary material and reserve the 50,000 images in the official ImageNet validation split for final testing. We select hyperparameters using only the (K)-shot training support set for that run, so the number of additional validation labels used for tuning is 0.
| Maximum number of additional validation images per class | 1-shot | 2-shot | 4-shot | 8-shot | 16-shot |
|---|---|---|---|---|---|
| 10 datasets other than ImageNet | 1 | 2 | 4 | 4 | 4 |
| ImageNet | 0 | 0 | 0 | 0 | 0 |
Every TCOC experiment uses the weights from the final training epoch. We set the number of epochs in advance for each configuration. Where validation data are used, they serve only to compare learning rates. We will revise Section 5.1 and Appendix C.3 to state both the validation budgets and the checkpoint selection rule explicitly. Our reported gains therefore do not rely on access to an oversized validation set, and we hope this clarification addresses your concern.
[1] Tao Yu et al. Task Residual for Tuning Vision-Language Models. CVPR 2023.
Response to W2 — The Effect of Prompt Templates and the Practical Advantages of TCOC (3/5)
Thank you for pointing out that the choice of prompt templates affects how we compare the TCOC results in Table 6 with those reported for ProLIP. We also agree that we should compare TCOC with recent methods such as MMRL and VaMP and clarify the conditions under which TCOC offers a practical advantage.
1. TCOC remains effective with the same single template as ProLIP
Following the base-to-novel protocol in Table 6, we retrained TCOC with three random seeds using a photo of a {}. as the sole template across all 11 datasets. The table below compares TCOC and ProLIP under the same single-template setting.
| Method and template | Base (%) | Novel (%) | HM (%) |
|---|---|---|---|
| TCOC · Single generic template | 84.32 | 75.64 | 79.74 |
| ProLIP · Same generic template (published result) | 83.85 | 74.78 | 79.08 |
With the template held fixed, TCOC still exceeds ProLIP by 0.66 percentage points in the harmonic mean (HM) of base and novel accuracies. Its base and novel accuracies are higher by 0.47 and 0.86 percentage points, respectively.
2. TCOC learns classifier-side corrections that transfer to novel classes
We acknowledge the stronger base-to-novel performance of MMRL and VaMP. TCOC’s architectural contribution is that it learns category-flexible corrections entirely on the classifier side. Using features from a few labeled images, we train a shared function that maps each class’s text embedding to a residual for its classifier weight. When the test classes change, the same function generates corrections from the new class text embeddings without requiring any parameters to be retrained.
This design is particularly useful when access to the upstream model is restricted. For example, a deployment may require both the image and text encoders to remain frozen, or it may expose only their output features for security reasons. In either setting, TCOC can use the extracted features for few-shot adaptation and construct classifiers for novel classes. Training updates only the shared generator and requires no backpropagation through either encoder.
3. TCOC can further improve classifiers after upstream adaptation
As you observed from the CLIP-SVD results, upstream representation adaptation and classifier-side correction can complement each other. Even after an upstream method improves the image or text representations, the principal text subspace may still underrepresent some visual directions needed to distinguish downstream classes. TCOC uses the adapted features directly to learn orthogonal classifier residuals that compensate for this remaining mismatch.
To test this complementarity, we combined TCOC separately with MMRL and ProLIP. MMRL adapts both visual and text representations, and ProLIP directly tunes the final projection layer of the visual encoder. For each method, we froze the trained upstream model and compared its base-to-novel performance with and without TCOC.
| Method | Base (%) | Novel (%) | HM (%) |
|---|---|---|---|
| MMRL | 85.64 | 76.93 | 81.05 |
| MMRL + TCOC | 86.05 | 77.20 | 81.39 |
| ProLIP | 83.79 | 74.76 | 79.02 |
| ProLIP + TCOC | 84.55 | 75.68 | 79.87 |
In these comparisons, adding TCOC increased HM by 0.34 percentage points for MMRL and 0.85 percentage points for ProLIP. Novel-class accuracy increased by 0.27 and 0.92 percentage points, respectively. These experiments show that TCOC’s classifier-side correction can be applied after either multimodal representation adaptation or visual projection-layer tuning.
VaMP adapts upstream representations through multimodal prompt learning, while TCOC operates on the downstream classifier. We therefore expect the same two-stage strategy to apply. We would first train and freeze VaMP, then use its image and class text embeddings to learn TCOC’s classifier corrections in a separate stage. We could not locate public training code for VaMP while preparing this response, so we leave its reproduction and evaluation in combination with TCOC to future work.
We will include the single-template comparison and the experiments combining TCOC with MMRL and ProLIP in the revised manuscript. We will also cite MMRL, ProLIP, and VaMP and discuss how each relates to TCOC.
Overview of Our Response to Reviewer BY8A (1/5)
Thank you for your careful review and for recognizing the geometric motivation behind TCOC and its complementarity with CLIP-SVD. We agree that both the number of labeled examples used for validation and the choice of prompt templates affect comparisons between methods. We also agree that extending our original cross-dataset ablation would help clarify what each of TCOC’s two core components contributes in other settings. In response, we clarify the experimental protocol, provide direct comparisons, and explain the connection between the modality gap and orthogonal compensation.
a photo of a {}.as the sole template. The harmonic mean (HM) of base and novel accuracies was 79.74%, 0.66 percentage points above the 79.08% reported for ProLIP with the same template. We agree that comparisons with strong base-to-novel methods such as MMRL and VaMP are important. TCOC offers a practical alternative that achieves competitive performance when resources are limited or training conditions are more restrictive. Using only features from frozen encoders, it learns classifier corrections that transfer to novel classes. This design also allows TCOC to complement existing methods and further improve their base-to-novel performance, as our experiments with MMRL and ProLIP show.The following four posts address W1–W3 and Q1–Q2 in turn.