acceptodds
Under review as a conference paper at ICLR 2027

Text-Conditioned Orthogonal Compensation for CLIP-based Few-Shot Adaptation

Abstract

CLIP exhibits strong transferability in few-shot adaptation, yet two key challenges remain. First, existing adaptation methods are structurally diverse, while a unified geometric understanding of their underlying connections is still lacking. Second, effective adaptation should capture task-specific information while preserving the inherent category flexibility of CLIP, which is often weakened by label-space-specific adaptation. To address these challenges, we introduce a modality-gap-inspired geometric framework that interprets major CLIP adaptation strategies through the visual–classifier mismatch, and our subspace analysis further motivates complementary classifier directions beyond the principal text-classifier subspace. Based on this insight, we propose Text-Conditioned Orthogonal Compensation (TCOC), which uses a shared text-conditioned generator to produce class-wise residuals and projects them onto the orthogonal complement of the principal text-classifier subspace. This design retains the original text classifier as a semantic anchor while introducing complementary task-specific directions. Since the generator is shared across classes, TCOC can naturally extend to new label spaces. Experiments on standard few-shot classification and multiple generalization settings demonstrate its effectiveness and category flexibility. The code will be publicly released upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Orthogonal Compensation, Modality Gap, Category Flexibility, CLIP Adaptation, Few-Shot Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BY8AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of CLIP-based few-shot classification. The proposed method, named TCOC (for text-conditioned orthogonal compensation), consists of 2 main elements. The first is the computation of a text-conditioned residual which is generated by a learned network that takes as input the text prototypes projected onto the text-coordinate basis formed by singular vectors. The residual is not directly used, it is projected onto the orthogonal complement of the principal text-classifier subspace. The projected residual is then added to the original zero-shot classifier. Experiments span few-shot classification, domain generalization, base-to-new generalization and cross-dataset generalization, and the proposed method exhibits better performance than previous methods.

Critical strengths

Motivating the insufficiency of few-shot adaptation methods to cover the visual features directions from a modality-gap perspective is interesting. This discussion makes the "orthogonal compensation" part of the method not merely an arbitrary design choice. On the other hand, the experiments are comprehensive and include different aspects of adaptation (few-shot classification, domain generalization, cross-dataset generalization, base-to-new generalization). The performance appears to be competitive to multiple state-of-the-art methods. I also find the complementarity to CLIP-SVD results very interesting (Table 3).

Critical weaknesses

One of my main concerns about the paper is related to the experimental protocol and the reported results. Concerning the experimental setting, the authors claim using the validation sets of each dataset to perform hyperparameter selection. While this has been done in earlier methods, there is a tendency in the literature to adopt a "few-shot validation sets" [1] and a "validation-free" [2,3,4] settings. This is so important because the few-shot adaptation setting is motivated by data-scarcity in some domains in real-world applications. So supposing having access to validation sets where the number of samples significantly exceeds the number of support set samples may inflate the gains.

For the base-to-new generalization results, some methods are missing from Table 6. For instance, it is unclear what is the practical advantage of the proposed method compared to MMRL [5] and VaMP [6] which outperform it on base-to-new generalization. Moreover, the paper uses specific prompt templates for each dataset, how does this affect the results compared to using a single template? It is not clear whether all the methods used for comparison adopt the same template; this is crucial for comparison fairness. For instance, [4] achieves 79.08% (harmonic mean) on base-to-new generalization, which is not very far from the paper's reported result (80.29%), while using a single template "a photo of {}." for all datasets. What part of this difference is due to the difference in prompt templates?

For the ablations, I believe Table 7 does not give enough evidence on the role of the text-conditioned generator and the orthogonal compensation, since it is confined to cross-dataset generalization which is tested with a single architecture under a single few-shot setting (ImageNet 16-shot). The same ablation is necessary for few-shot classification, domain generalization and base-to-new generalization to support the advantage of including both elements.

[1] Huang et al., LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP. CVPR 2024
[2] Silva-Rodríguez et al., A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models. CVPR 2024
[3] Zanella and Ben Ayed, Low-Rank Few-Shot Adaptation of Vision-Language Models. CVPRW 2024
[4] Fahes et al., CLIP’s Visual Embedding Projector is a Few-shot Cornucopia. WACV 2026
[5] Guo and Gu, MMRL: Multi-Modal Representation Learning for Vision-Language Models. CVPR 2025
[6] Cheng and Han, VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models. NeurIPS 2025

Questions for authors
  • In Section 3.2, the authors state: "When text features are used as classifier weights, the subspace spanned by the text classifier may not fully cover the discriminative directions of downstream visual features." (Lines 137-139). Isn't this problem also possible when there's no modality gap?

  • What makes TCOC complementary to CLIP-SVD? CLIP-SVD tunes parameters in the text branch, so I would intuitively suppose the complementarity would be stronger for methods that only tune parameters in the visual encoder for instance. Does this complementarity hold for other methods?

Suggestions

  • It would be great to include the standard deviations in the tables.
7 comments
AuthorsOpenReview ↗

Overview of Our Response to Reviewer BY8A (1/5)

Thank you for your careful review and for recognizing the geometric motivation behind TCOC and its complementarity with CLIP-SVD. We agree that both the number of labeled examples used for validation and the choice of prompt templates affect comparisons between methods. We also agree that extending our original cross-dataset ablation would help clarify what each of TCOC’s two core components contributes in other settings. In response, we clarify the experimental protocol, provide direct comparisons, and explain the connection between the modality gap and orthogonal compensation.

Reviewer questionOur response and evidence
W1. How much additional labeled data is used for validation?We apologize for the confusion about the validation budget. When constructing the few-shot validation split for a setting with (K) training images per class, we use at most (\min(K,4)) additional labeled images per class for hyperparameter selection. We will state this limit explicitly to address your concern about the number of labels used for tuning.
W2. How much do prompt templates affect base-to-novel performance, and what practical advantage does TCOC offer compared with strong alternatives?We retrained TCOC on each of the 11 datasets using a photo of a {}. as the sole template. The harmonic mean (HM) of base and novel accuracies was 79.74%, 0.66 percentage points above the 79.08% reported for ProLIP with the same template. We agree that comparisons with strong base-to-novel methods such as MMRL and VaMP are important. TCOC offers a practical alternative that achieves competitive performance when resources are limited or training conditions are more restrictive. Using only features from frozen encoders, it learns classifier corrections that transfer to novel classes. This design also allows TCOC to complement existing methods and further improve their base-to-novel performance, as our experiments with MMRL and ProLIP show.
W3. What do the shared text-conditioned generator and orthogonal compensation each contribute?We extended the ablation in Table 7 to five shot counts on 11 datasets, base-to-novel generalization, and domain generalization from 16-shot ImageNet to four ImageNet variants. We compare class-wise residuals, a shared generator without the orthogonal constraint, and full TCOC. The first comparison examines what the shared generator adds, particularly its ability to produce corrections for unseen classes. Comparing Free with full TCOC then isolates the contribution of orthogonal projection. Following your suggestion, we also report standard deviations for experiments run with three random seeds.
Q1. Can the text subspace miss visual discriminative directions even in the absence of a modality gap?Even without systematic separation between image and text representations, the principal text subspace may miss visual directions needed to distinguish downstream classes. In CLIP, class text embeddings serve directly as classifier weights. When systematic image–text misalignment limits how well these weights capture task-relevant visual distinctions, the resulting visual–classifier mismatch is a task-level manifestation of the modality gap. TCOC uses a few labeled images to learn classifier corrections in the orthogonal complement of the principal text subspace
Q2. Why is TCOC complementary to CLIP-SVD, and does this extend to other adaptation methods?CLIP-SVD adapts the representations produced by the encoders. TCOC can then use the labeled support images to correct the classifier weights constructed from those representations. In addition to our original CLIP-SVD results, we report plug-in comparisons with TCP, CoCoOp, MaPLe, MMRL, and ProLIP. ProLIP is particularly relevant to your question because it adapts only the visual projection layer, allowing us to examine whether classifier-side compensation also helps after vision-only adaptation.

The following four posts address W1–W3 and Q1–Q2 in turn.

AuthorsOpenReview ↗

Response to W1 — Validation Budget and Hyperparameter Selection (2/5)

Thank you for raising the question of how many additional labeled images are used for validation in a few-shot setting. Our statement that hyperparameters were selected “on the corresponding validation sets” did not specify the validation budget or explain how we handled ImageNet. We built our few-shot experiments on the official TaskRes implementation [1].

On the 10 datasets other than ImageNet, each (K)-shot experiment uses (K) labeled training images per class to fit TCOC. We sample at most (\min(K,4)) additional labeled images per class from a separate validation split to select the initial learning rate from ({10^{-3}, 5\times10^{-4}, 10^{-4}}). Thus, at every shot count, the validation set is at most as large as the training support set. On ImageNet, we follow the TaskRes supplementary material and reserve the 50,000 images in the official ImageNet validation split for final testing. We select hyperparameters using only the (K)-shot training support set for that run, so the number of additional validation labels used for tuning is 0.

Maximum number of additional validation images per class1-shot2-shot4-shot8-shot16-shot
10 datasets other than ImageNet12444
ImageNet00000

Every TCOC experiment uses the weights from the final training epoch. We set the number of epochs in advance for each configuration. Where validation data are used, they serve only to compare learning rates. We will revise Section 5.1 and Appendix C.3 to state both the validation budgets and the checkpoint selection rule explicitly. Our reported gains therefore do not rely on access to an oversized validation set, and we hope this clarification addresses your concern.

[1] Tao Yu et al. Task Residual for Tuning Vision-Language Models. CVPR 2023.

AuthorsOpenReview ↗

Response to W2 — The Effect of Prompt Templates and the Practical Advantages of TCOC (3/5)

Thank you for pointing out that the choice of prompt templates affects how we compare the TCOC results in Table 6 with those reported for ProLIP. We also agree that we should compare TCOC with recent methods such as MMRL and VaMP and clarify the conditions under which TCOC offers a practical advantage.

1. TCOC remains effective with the same single template as ProLIP

Following the base-to-novel protocol in Table 6, we retrained TCOC with three random seeds using a photo of a {}. as the sole template across all 11 datasets. The table below compares TCOC and ProLIP under the same single-template setting.

Method and templateBase (%)Novel (%)HM (%)
TCOC · Single generic template84.3275.6479.74
ProLIP · Same generic template (published result)83.8574.7879.08

With the template held fixed, TCOC still exceeds ProLIP by 0.66 percentage points in the harmonic mean (HM) of base and novel accuracies. Its base and novel accuracies are higher by 0.47 and 0.86 percentage points, respectively.

2. TCOC learns classifier-side corrections that transfer to novel classes

We acknowledge the stronger base-to-novel performance of MMRL and VaMP. TCOC’s architectural contribution is that it learns category-flexible corrections entirely on the classifier side. Using features from a few labeled images, we train a shared function that maps each class’s text embedding to a residual for its classifier weight. When the test classes change, the same function generates corrections from the new class text embeddings without requiring any parameters to be retrained.

This design is particularly useful when access to the upstream model is restricted. For example, a deployment may require both the image and text encoders to remain frozen, or it may expose only their output features for security reasons. In either setting, TCOC can use the extracted features for few-shot adaptation and construct classifiers for novel classes. Training updates only the shared generator and requires no backpropagation through either encoder.

3. TCOC can further improve classifiers after upstream adaptation

As you observed from the CLIP-SVD results, upstream representation adaptation and classifier-side correction can complement each other. Even after an upstream method improves the image or text representations, the principal text subspace may still underrepresent some visual directions needed to distinguish downstream classes. TCOC uses the adapted features directly to learn orthogonal classifier residuals that compensate for this remaining mismatch.

To test this complementarity, we combined TCOC separately with MMRL and ProLIP. MMRL adapts both visual and text representations, and ProLIP directly tunes the final projection layer of the visual encoder. For each method, we froze the trained upstream model and compared its base-to-novel performance with and without TCOC.

MethodBase (%)Novel (%)HM (%)
MMRL85.6476.9381.05
MMRL + TCOC86.0577.2081.39
ProLIP83.7974.7679.02
ProLIP + TCOC84.5575.6879.87

In these comparisons, adding TCOC increased HM by 0.34 percentage points for MMRL and 0.85 percentage points for ProLIP. Novel-class accuracy increased by 0.27 and 0.92 percentage points, respectively. These experiments show that TCOC’s classifier-side correction can be applied after either multimodal representation adaptation or visual projection-layer tuning.

VaMP adapts upstream representations through multimodal prompt learning, while TCOC operates on the downstream classifier. We therefore expect the same two-stage strategy to apply. We would first train and freeze VaMP, then use its image and class text embeddings to learn TCOC’s classifier corrections in a separate stage. We could not locate public training code for VaMP while preparing this response, so we leave its reproduction and evaluation in combination with TCOC to future work.

We will include the single-template comparison and the experiments combining TCOC with MMRL and ProLIP in the revised manuscript. We will also cite MMRL, ProLIP, and VaMP and discuss how each relates to TCOC.

Reviewer_19YEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a modality gap inspired geometric framework, which analyzes how existing adaptation methods sacrifice CLIP's inherent category flexibility. To address this, the author introduces TCOC method. Instead of optimizing a class-indexed residual matrix, TCOC employs a shared text-conditioned generator to produce classifier corrections. Crucially, these learned residuals are orthogonally projected away from the principal text-classifier subspace. This allows the model to capture complementary downstream visual discriminative features while maintaining the original text representation as a semantic anchor.

Critical strengths
  1. The geometric interpretation of the visual-classifier mismatch is elegant. Characterizing different branches of CLIP adaptation within a unified subspace framework and proving the necessity of exploring the orthogonal complement via SVD, provides deep insights into the field.

  2. The design of TCOC directly addresses the category flexibility problem. By utilizing a shared generator on mapped text-coordinate bases rather than learning task-specific logits, the methodology inherently ensures zero-shot reusability on novel classes and distinct datasets.

Critical weaknesses
  1. TCOC leans hard on prompt ensembling, even getting a bigger boost from it than the zero-shot baseline. This kind of suggests the method is super picky about having high-quality, diverse text inputs. That could be a real bottleneck in niche domains where you just don't have a ton of rich prompts to work with.

  2. The ablation study makes it pretty clear that TCOC really needs prompt ensembling to shine. It seems highly sensitive to the quality and variety of the text prompts, which is going to be a problem for specialized fields.

Questions for authors

In Table 14, TCOC exhibits a significant dependency on prompt ensembling. How does the model perform if the novel classes during cross-dataset transfer have access only to a single, poorly descriptive prompt? Does the orthogonal compensation degenerate or suffer from mode collapse?

7 comments
AuthorsOpenReview ↗

Overview of Our Response to Reviewer 19YE (1/5)

Thank you for recognizing the value of our geometric analysis and TCOC’s ability to preserve CLIP’s category flexibility. We understand that W1, W2, and Q raise a common concern about how much TCOC’s performance depends on the quality and diversity of class descriptions. As you observed in Table 14, prompt ensembling yields a larger gain for TCOC in the few-shot setting than it does for zero-shot CLIP. You also point out that writing several high-quality descriptions for each novel class can be difficult in specialized domains. We address this concern from four perspectives.

1. Does orthogonal compensation still transfer when each novel target class has only one simple description? We test this directly by keeping the ImageNet-1K 16-shot model fixed and changing the target prompts. With a single template, TCOC achieves 66.41% mean accuracy across the ten target datasets, 1.26 percentage points higher than zero-shot CLIP using the same template, with gains on 9 of the 10 targets.

2. Does single-template TCOC maintain state-of-the-art performance? We compare single-template TCOC with leading category-flexible methods in cross-dataset transfer, standard few-shot classification, and base-to-novel generalization. TCOC achieves the highest mean transfer accuracy among the methods compared. It also reaches 76.25% mean accuracy across the five shot counts and a 79.74% harmonic mean of base and novel accuracies. Single-template TCOC thus achieves state-of-the-art performance on all three aggregate metrics among the category-flexible methods compared.

3. How much do different methods benefit from the same set of enhanced templates? We compare the effects of prompt ensembling in both standard few-shot classification and cross-dataset transfer. In the few-shot setting, TCP’s mean accuracy changes very little, TaskRes improves slightly, MaPLe’s accuracy decreases, and TCOC improves at all five shot counts. We discuss these differences in light of how TCOC uses class text embeddings to form the initial classifier, provide inputs to the residual generator, and determine the principal text directions used for orthogonal compensation.

4. Can class representations be learned from data when rich descriptions are difficult to write by hand? When a few labeled images are available, an upstream prompt learner can learn class representations from the support set. We can then freeze the upstream model and let TCOC learn classifier corrections from those representations. We tested this approach with CoCoOp, TCP, and MaPLe. Adding TCOC improves all three methods at all five shot counts, showing that text representations learned through prompt learning can also provide an effective basis for TCOC’s classifier corrections.

AuthorsOpenReview ↗

Response to W1, W2, and Q — Does Orthogonal Compensation Still Transfer with a Single Target Template? (2/5)

Reviewer asked whether orthogonal compensation learned in the source domain remains effective when each novel target class has only one simple description a photo of a {} . To test this directly, we keep the model trained on ImageNet-1K with 16 shots per class fixed and change the target prompts. For each of the ten target datasets, we encode the class names using either a single template or a prompt ensemble. In each setting, we compare TCOC with zero-shot CLIP using the same target descriptions. The table reports accuracy (%) and the gain from compensation in percentage points (pp).

Target datasetCLIP (single)TCOC (single)Gain (pp)CLIP (ensemble)TCOC (ensemble)Gain (pp)
Caltech10192.9894.02+1.0493.6794.05+0.38
OxfordPets89.1389.81+0.6889.7589.91+0.16
StanfordCars65.2965.76+0.4766.1665.98−0.18
OxfordFlowers71.3071.25−0.0571.5071.25−0.25
Food10186.1186.72+0.6186.2286.81+0.59
FGVC Aircraft24.9025.35+0.4525.3525.55+0.20
SUN39762.5966.96+4.3763.4267.23+3.81
DTD44.5645.73+1.1744.6245.21+0.59
EuroSAT47.8450.92+3.0848.4052.30+3.90
UCF10166.8367.58+0.7567.4468.80+1.36
Mean over 10 targets65.1566.41+1.2665.6566.71+1.06

Replacing the target prompt ensemble with a single template reduces mean accuracy by 0.30 percentage points for TCOC and 0.50 percentage points for zero-shot CLIP. With the same single template, TCOC improves on CLIP on 9 of the 10 target datasets, raising mean accuracy from 65.15% to 66.41%, a gain of 1.26 percentage points. We therefore observe no collapse in transfer performance under the single-template condition tested. The source-trained compensation continues to improve recognition on most targets.

AuthorsOpenReview ↗

Response to W1, W2, and Q — Single-Template TCOC Compared with Other Category-Flexible Methods (3/5)

We next examine whether TCOC remains competitive with leading category-flexible methods when only one prompt template is available. We compare performance in cross-dataset transfer, standard few-shot classification, and base-to-novel generalization.

1. Cross-dataset transfer

We compare TCOC with TCP and MaPLe, two representative category-flexible methods from Table 5 of the main paper. We report ImageNet-1K source accuracy, mean accuracy over the ten transfer targets, and mean accuracy over all 11 datasets. Both means give equal weight to each dataset included.

MethodImageNet-1K source (%)Mean over 10 targets (%)Mean over all 11 datasets (%)
TCP71.4066.3066.76
MaPLe70.7266.3066.69
TCOC (single template)73.0166.4167.01

With one description per target class, TCOC achieves the highest mean accuracy among the three methods both across the ten targets (66.41%) and across all 11 datasets (67.01%).

2. Standard few-shot classification

We compare the single-template TCOC results from Table 14 of our submission with the single-template results for TCP and MaPLe. Each shot column reports mean accuracy (%) across 11 datasets.

Method1-shot2-shot4-shot8-shot16-shotMean over shots
TCP70.6373.5976.0778.3980.9875.93
MaPLe69.2771.5374.3377.6580.2774.61
TCOC71.0473.7576.2678.8581.3576.25

Single-template TCOC outperforms both baselines at all five shot counts, from 1 to 16 shots per class.

3. Base-to-novel generalization

We use a photo of a {}. as TCOC’s sole template on all 11 datasets. The model learns the compensation from the few labeled examples of the base classes and then applies the same generator to the text representations of the novel classes.

MethodBase (%)Novel (%)HM (%)
TCP84.1375.3679.51
ProLIP83.8574.7879.08
DeKgCoOp82.5974.9378.57
MaPLe82.2875.1478.55
TCOC (single template)84.3275.6479.74

Among the leading category-flexible methods compared, single-template TCOC achieves state-of-the-art performance on all three aggregate metrics. It reaches 66.41% mean accuracy across the ten transfer targets, 76.25% mean accuracy across the five few-shot settings, and 79.74% harmonic mean of base and novel accuracy. We hope these comparisons directly address your concern about performance when only one template is available.

Reviewer_H7BUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Text-Conditioned Orthogonal Compensation (TCOC) for CLIP-based few-shot adaptation. The authors introduce a modality-gap inspired geometric perspective on existing CLIP adaptation methods, identifying the subspace mismatch between visual features and text classifiers as an important issue. Based on this analysis, TCOC employs a shared text-conditioned generator to produce class-wise classifier residuals and projects them onto the orthogonal complement of the principal text-classifier subspace. The method aims to introduce complementary discriminative information while preserving CLIP's original semantic prior and category flexibility. Experiments on standard few-shot classification, domain generalization, and cross-category and cross-dataset transfer demonstrate improvements over several existing methods.

Critical strengths
  1. Practically relevant problem and lightweight design.

The paper addresses the meaningful problem of improving CLIP few-shot adaptation while preserving generalization to unseen categories. The proposed method keeps the pretrained encoders frozen and avoids class specified parameters, making it lightweight and applicable to different label spaces.

  1. Reasonably comprehensive empirical evaluation.

The experiments cover multiple few-shot classification and generalization benchmarks, reporting relatively consistent improvements. The paper also includes component ablations, evaluations across different backbones, and additional analyses of prompt ensembles, which provide some empirical support for the method's effectiveness.

Critical weaknesses
  1. The motivation and distinctive contribution of the problem formulation are insufficiently established.

The authors emphasize the limited category flexibility of existing class-specific adaptation methods. However, standard few-shot adaptation typically involves a fixed label space, where class-specific classifiers or caches are reasonable design choices. Their inability to transfer directly to unseen categories does not necessarily constitute a limitation under the original task setting. While cross-category generalization is a valuable research direction, it has already been extensively studied by methods such as CoCoOp, MaPLe, and TCP. Although TCOC reports improvements over these approaches, the paper does not sufficiently establish what fundamentally new capability or insight it introduces beyond existing category-flexible adaptation methods.

  1. The key methodological choices lack sufficient motivation and validation.

First, the necessity of BGB_G is unclear. A shared text-conditioned generator can directly operate on class text features and generate residuals for unseen categories without an SVD-based coordinate transformation. Under the full-rank ImageNet setting, BGB_G can be absorbed into the generator's input-layer weights without changing its representational capacity. Although the paper investigates different coordinate dimensions and reference vocabularies, it lacks a direct comparison with raw text features and, more importantly, an explanation of what the SVD transformation contributes to learning or generalization.

Second, the choice of BPB_P is insufficiently justified. The principal singular directions of text features do not necessarily correspond to discriminative information that should be preserved, and their relationship with the proposed visual classifier mismatch remains unclear.

Questions for authors
  1. On the necessity and role of BGB_G:

Since a shared generator can directly take class text features as input and support unseen categories, why is the additional SVD-based coordinate transformation necessary? Could the authors directly compare hθ(T)h_\theta(T) and hθ(TBG)h_\theta(TB_G) under matched settings? Beyond performance comparisons, a more insightful analysis is needed to explain what the SVD coordinates actually contribute to optimization or cross-category generalization, particularly given that the full-rank transformation does not change representational capacity.

  1. On the geometric motivation for BPB_P:

Why is orthogonal projection onto the complement of the principal text subspace necessary, and how does it address the proposed visual–classifier mismatch? The authors should provide stronger justification and deeper insights beyond empirical performance improvements.

  1. On the distinctive contribution relative to existing methods:

Given that approaches such as CoCoOp, MaPLe, and TCP already support cross-category generalization, what is the central contribution of TCOC beyond these methods? Apart from empirical performance improvements, can the authors clarify what new insight or previously unresolved limitation is addressed by the proposed residual generation and orthogonal compensation?

I would be willing to reconsider my rating if these concerns are convincingly addressed.

6 comments
AuthorsOpenReview ↗

Overview of Our Response to Reviewer H7BU (1/4)

Thank you for your careful review and for recognizing the practical value of preserving CLIP’s category flexibility during few-shot adaptation. Your questions helped us clarify the roles of the two text-derived bases and how TCOC complements existing prompt-learning methods.

Reviewer questionOur response and evidence
Q1. What does the input basis BGB_G contribute?We agree that a full-rank BGB_G preserves the generator’s representational capacity. Its benefit is that each class is expressed in coordinates defined by the reference text features. Providing these axes before training can make it easier for a high-dimensional generator to learn useful classifier corrections from few labeled images. SVD also lets us construct a lower-dimensional reference space by retaining only the leading directions. Therefore, we directly compared raw text features, a random orthogonal rotation, and text SVD coordinates under matched training settings. The SVD coordinates yielded the highest mean accuracy across 11 datasets at all five shot counts. We also examined how truncating the basis affects transfer from ImageNet to ten target datasets.
Q2. Why constrain the residual using BPB_P?Our experiments revealed that a significant portion of the unconstrained residual energy is distributed along the principal text directions. Projecting the residual onto their orthogonal complement forces the correction to act outside those dominant directions. Few-shot visual supervision can then capture class distinctions that the principal text subspace underrepresents, directly addressing the visual–classifier mismatch. We trained the same generator with three choices: no projection, projection away from a random subspace, and projection away from the principal text subspace of the same rank. The text-based projection reduced the induced gap and increased orthogonal separation, showing that the learned corrections better cover visual directions.
Q3. What does TCOC add beyond methods such as CoCoOp, MaPLe, and TCP?TCOC brings category flexibility to classifier-side few-shot adaptation. A shared generator produces a correction to each classifier weight from the corresponding class text, so it can also generate corrections for new classes. CoCoOp, MaPLe, and TCP achieve flexibility through upstream prompt learning. TCOC can act on the classifier weights produced by those learned prompts, and we tested this compatibility directly by applying it to trained CoCoOp, MaPLe, and TCP models.

The following three posts address Q1, Q2, and Q3 in order.

AuthorsOpenReview ↗

Response to W2/Q1 — The Role of the Text-Coordinate Basis (2/4)

We agree that a shared generator hθ(T)h_\theta(T) can produce residuals for unseen classes directly from their text features. We also agree that a full-rank BGB_G can be absorbed into the generator’s input weights. We argue that an explicit BGB_G offers two benefits in practice.

1. Text-derived coordinates guide learning when labeled images are scarce.

With few labeled images per class, many mappings in a high-dimensional parameter space can fit the same support set yet generalize differently. We therefore use the task’s class texts to define the input coordinates before training. For T=UΣV⊤T=U\Sigma V^\top, setting BG=VB_G=V gives TBG=UΣTB_G=U\Sigma, the projections of each class text onto directions ordered by singular value. The generator learns to map these coordinates to discriminative classifier residuals in the original CLIP embedding space. In this basis, adjusting one weight changes a first-layer unit’s response along a single principal text direction. With raw or randomly rotated inputs, the same adjustment generally requires changing several weights together. Adam scales each weight update using its own gradient history, so even at full rank the generators can follow different optimization paths from the same initial function.

We directly compared the reviewer’s proposed hθ(T)h_\theta(T) with hθ(TQ)h_\theta(TQ) for a random orthogonal rotation QQ and with our text-derived hθ(TBG)h_\theta(TB_G) under matched training settings. The table reports accuracy (%) averaged over 11 datasets. Differences are in percentage points (pp).

ShotsA. Raw text hθ(T)h_\theta(T)B. Random rotation hθ(TQ)h_\theta(TQ)C. Full-dimensional text SVD hθ(TBG)h_\theta(TB_G)C − A (pp)C − B (pp)
170.87 ± 0.3070.87 ± 0.3272.00 ± 0.35+1.13+1.13
273.76 ± 0.3473.74 ± 0.3374.34 ± 0.24+0.58+0.60
476.26 ± 0.1876.25 ± 0.1876.85 ± 0.18+0.59+0.60
879.06 ± 0.2379.04 ± 0.2279.87 ± 0.21+0.81+0.83
1681.45 ± 0.1781.46 ± 0.1582.09 ± 0.16+0.64+0.63

Text SVD gives the highest mean accuracy in all five settings. These results suggest that defining the input axes BGB_G from the task’s class texts before training makes it easier for the generator to learn the mapping from text coordinates to discriminative classifier corrections, particularly when only a few labeled images per class are available to guide optimization.

2. An explicit basis gives direct control over the rank of the reference space.

With SVD, we can choose any number rgr_g of leading text directions to retain in BGB_G. Using the raw text features directly in hθ(T)h_\theta(T) does not offer this choice. Our standard ImageNet configuration uses the full 512-dimensional basis. To examine the effect of truncation, we trained on ImageNet with 16 shots per class at ranks 128, 256, and 512 and kept each resulting source basis and generator fixed across all ten target datasets.

Datasetrank 128rank 256rank 512
ImageNet (source)72.2372.8173.12
Caltech10194.1194.0794.05
OxfordPets90.2190.1989.91
StanfordCars66.0265.9965.98
OxfordFlowers71.8271.7171.25
Food10186.8386.8086.81
FGVCAircraft25.8725.9625.55
SUN39767.2467.2867.23
DTD45.5945.4445.21
EuroSAT51.6651.7152.30
UCF10169.0268.8868.80
Mean over ten targets66.8466.8066.71

Reducing the rank from 512 to 128 lowers ImageNet few-shot accuracy by 0.89 pp and raises mean transfer accuracy from 66.71% to 66.84%. These results illustrate the trade-off that an explicit reference rank makes available in TCOC, and suggest that reducing the rank of BGB_G may improve generalization to new label spaces.

We thank the reviewer again for raising this question. We will incorporate this discussion into the revised manuscript to clarify why TCOC explicitly constructs BGB_G and how this design benefits few-shot adaptation and transfer.

AuthorsOpenReview ↗

Response to W2/Q2 — The Role of Orthogonal Projection (3/4)

1. Why project the residual onto the complement of the principal text subspace?

Orthogonal projection is how TCOC turns its modality-gap motivation into a classifier correction. A direction that separates downstream visual classes can be weakly represented in the principal subspace of the text classifier TT. We use the leading right singular vectors BPB_P of TT because they identify the directions that carry the most energy in the existing classifier. Their role is to characterize what the text classifier already represents. Few-shot visual supervision determines which additional directions are useful for discrimination.

Writing PT=BPBP⊤P_T=B_PB_P^\top, Proposition 1 shows that restricting an ideal downstream classifier W⋆W^\star to the principal text subspace leaves its component W⋆(I−PT)W^\star(I-P_T) unrepresented. TCOC therefore applies the complementary projector to the learned residual, giving R⊥=R(I−PT)R_\perp=R(I-P_T). The rows of R⊥R_\perp can span a decision subspace orthogonal to the principal text directions. This allows labeled images to guide the correction toward class distinctions that the text classifier underrepresents.

2. What changes when we constrain the residual in this way?

Section 5.5 reports that, on ImageNet under 16-shot training, 60.82% of the unconstrained residual energy lies in just the top eight text directions. Table 8 further compares the unprojected generator (second row, 72.58% accuracy) and TCOC (73.12%). TCOC also yields the lowest induced gap (0.6397) and the highest orthogonal separation (0.1235). To test whether the choice of the projected subspace matters, we added a control that learns the residual while projecting it away from a random subspace of the same rank.

The table reports averages over 11 datasets at each shot count. Each triplet follows the order Free / Random / TCOC. Free learns an unconstrained residual. Random and TCOC project the learned residual away from a random subspace and the principal text subspace, respectively. Both use rank 8 on ImageNet and rank 4 on the other ten datasets.

  • Accuracy is top-1 classification accuracy.
  • Induced gap measures how well the joint text-and-residual decision subspace covers the principal visual directions. A smaller value indicates better coverage.
  • Orth. Sep. measures how much more closely each residual aligns with its own visual class prototype than with other classes’ prototypes on average.
  • Free residual energy in the principal text subspace is the fraction of the unconstrained residual’s squared norm captured by BPB_P.
Shots per classAccuracy ↑ (%)Induced gap ↓Orth. Sep. ↑Free residual energy in the principal text subspace
171.75 / 71.74 / 72.000.8662 / 0.8663 / 0.83250.0843 / 0.0844 / 0.117136.30%
274.00 / 74.02 / 74.340.8631 / 0.8636 / 0.81550.1018 / 0.1016 / 0.160640.09%
476.30 / 76.28 / 76.850.8610 / 0.8617 / 0.80230.1290 / 0.1288 / 0.214648.51%
879.08 / 79.09 / 79.870.8601 / 0.8603 / 0.79130.1456 / 0.1449 / 0.267260.38%
1681.44 / 81.43 / 82.090.8547 / 0.8550 / 0.78610.1627 / 0.1621 / 0.293367.11%

Despite its rank of only 4 or 8, the principal text subspace captures 36.30% of the unconstrained residual energy at one shot and 67.11% at 16 shots. The rank-matched random projection stays close to Free across all five settings. Projecting away from the principal text directions improves mean accuracy at every shot count, by 0.25–0.79 percentage points over Free. It also consistently reduces the induced gap and increases orthogonal separation. Together, these measurements show that the complementary directions learned by TCOC cover more of the visual subspace and align more selectively with the corresponding visual classes.

We thank the reviewer for asking us to clarify the geometric role of BPB_P. We will include the full table and this discussion in the revised manuscript to make the connection between orthogonal compensation, visual–classifier mismatch, and class discrimination explicit.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Authors’ Responses to the Reviewers

We thank all three reviewers for their careful assessment of our work. Reviewers 19YE and BY8A recognized the value of interpreting CLIP few-shot adaptation through the geometry of visual–classifier mismatch. H7BU and 19YE highlighted the practical value of transferring what is learned from a few labeled examples to new classes while keeping the encoders frozen. The reviewers also noted the breadth of our generalization experiments, and BY8A expressed particular interest in how TCOC complements CLIP-SVD.

The four main issues raised across the three reviews were what TCOC offers beyond existing category-flexible methods, the respective roles of its two text-derived bases and core components, whether TCOC remains effective when each target class has only one simple description, and our experimental protocol. In our point-by-point responses, we present the following additional experiments and analyses.

1. How TCOC complements existing category-flexible adaptation methods. Prompt-learning methods can already reuse learned context when encoding new classes. TCOC addresses a practical question that arises when only the encoders’ output features are accessible. How can a few labeled images improve a classifier in a way that also transfers to a new vocabulary? TCOC learns a shared function that maps each class’s text representation to a correction for its classifier weight. When the vocabulary changes, the same function generates corrections for the new classes. We extended our original experiments with CLIP-SVD to three prompt-learning methods. After freezing each trained upstream model, adding TCOC improved mean accuracy across 11 datasets for TCP, MaPLe, and CoCoOp at all five shot counts. In base-to-novel generalization, TCOC also improved performance for MMRL and ProLIP. These results show that classifier-side correction can complement several forms of upstream adaptation.

2. What text coordinates and orthogonal projection each contribute. For the input basis BGB_G, we agree that a full-rank change of coordinates preserves the generator’s expressive capacity. The text-derived basis expresses class embeddings along the principal text directions of the current task, giving the generator reference coordinates in which to learn a mapping from class texts to classifier corrections with few labeled examples. Under matched settings, we compared raw text features, a random orthogonal rotation, and text SVD coordinates. The SVD coordinates yielded the highest mean accuracy across 11 datasets at all five shot counts.

For the projection basis BPB_P, the dominant textual direction can underrepresent visual information needed to distinguish those classes. Projecting the residual onto the orthogonal complement confines the correction to directions outside the principal text subspace. We compared an unconstrained generator with two variants that project the residual away from either a random subspace or the principal text subspace of the same rank. Projecting away from the principal text subspace improved mean accuracy by 0.25–0.79 percentage points over the unconstrained generator across the five shot counts and improved the alignment of residuals with their corresponding visual class prototypes.

3. Source-trained compensation remains effective with one simple description per target class. We kept the model trained on ImageNet-1K with 16 shots per class fixed and varied the target prompts. With a single template per target class, TCOC achieved 66.41% mean accuracy across the ten target datasets, 1.26 percentage points above zero-shot CLIP using the same template. For base-to-novel generalization, we also retrained and evaluated TCOC using the same single template. The harmonic mean reached 79.74%. These comparisons quantify the benefit of prompt ensembling and show that single-template TCOC still leads the baselines on the aggregate metrics reported.

4. Clarification of the experimental protocol. We clarified the hyperparameter selection protocol for our TCOC experiments. Each KK-shot run uses KK labeled training images per class. We select the learning rate using at most min⁡(K,4)\min(K,4) additional labeled images per class from a separate validation split. The validation budget therefore never exceeds the size of the training support set. In response to the suggestion to report variability, we also provide means and standard deviations across three random seeds for the new few-shot and domain generalization component ablations.

We will incorporate these comparisons into the revised manuscript and clarify the respective roles of the input coordinates, the shared generator, and the output projection. We thank the reviewers for the specific suggestions that helped us articulate TCOC’s contribution and the conditions under which it applies. Our detailed responses are available in the individual review threads. We would be happy to address any further questions or remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.