- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DEFTS, a training-free few-shot classification method for CLIP. Given text embedding for each class, DEFTS computes their mean, forms a scatter matrix from the centered text embeddings, and eigendecomposes this matrix. It retains the smallest number of leading eigenvectors that explain at least 95% of the text-embedding variance. These eigenvectors define the subspace in which DEFTS projects both the support-image embeddings and the class text embeddings. Within this subspace, it estimates an image prototype for each class using a Gaussian prior centered at the corresponding projected text embedding and a shared covariance estimated from the support set. The posterior-mean prototypes can be used directly for classification or ensembled with existing training-free methods. The approach is evaluated on 12 classification datasets and three CLIP backbones.
Critical strengths
1. The subspace observation is simple, well controlled, and useful. The finding that NCM prototypes estimated inside the text-spanned subspace are far more sample-efficient than full-space prototypes (Figure 2), and that centering the text embeddings helps (Table 7), is clearly shown. Observation 1's comparison against same-rank random and PCA subspaces is the right control. This part of the paper has value independent of the estimator built on top of it.
2. Practical and training-free. DEFTS combines projected visual evidence with a text-derived prior through a closed-form estimator and requires no gradient-based optimization. This makes the approach computationally attractive and potentially useful when only one or two labeled examples per class are available.
Critical weaknesses
1. The writing needs substantial revision. Several parts of the method cannot be followed as written.
- Related work (main text and Appendix A). The paper lists prior methods without giving enough background on them or stating how DEFTS differs from each. CoOp, Zhao et al. (2024) and Zhu et al. (2024) are cited and never discussed, and the Appendix A paragraphs on the modality gap and on James–Stein shrinkage are directly relevant to the prior in Section 2.3 but are not connected to it. The related work also omits Linear Feature Alignment (LFA, Ouali et al., "Black Box Few-Shot Adaptation for Vision-Language Models", ICCV 2023), which addresses the same image–text alignment problem that motivates the paper.
- Section 2.2 (Text Embedding Subspace). The projection matrix is used but never defined. The step from "(I - P_S)F_v(I) is orthogonal to any text embedding" to the displayed equation is not derived, as written, $P_S F_t(T_i) = F_t(T_i)$ trivially, so the equation does not use the orthogonality it invokes. The sentence "few studies have investigated whether the linear space defined by is actually a suitable subspace" gives no citations, and the paper's own related work cites two studies (TAMP, Zhang et al. 2026) that do exactly this. The number of images used to fit the NCM classifiers in Observations 1–3 should be stated where the observations are presented, not only in Appendix E.3. The motivation for Observation 3 is unclear: it argues that reducing the subspace dimension hurts, before any dimensionality reduction has been introduced, and the method then adopts a 95% PCA truncation that Table 9 shows helps at every .
- Section 2.3 (DEFTS). The motivation for the Bayesian estimator is not clear to me. The paper states that the subspace is low-dimensional and that the text embedding is an informative prior, but it does not explain why a Gaussian posterior mean with an isotropic prior and a pooled covariance is the right estimator here, how this choice relates to the projection step (the two can be applied independently, as Figures 4 and 5 show), or what the resulting dimension is on any dataset.
2. The main experiments are confounded, so they do not test the claimed contribution.
The paper's evidence rests on Table 1 (), Table 2 and the ensembling columns, and each of these is confounded in a way that leaves the stated contribution untested.
- At , the Bayesian estimator is not what is being evaluated. With the paper's fixed choices ($\hat\sigma^2 = 0.1 D$, , , ), Eq. (5) reduces to a fixed scalar interpolation with weight on the projected image and on the projected text, identical for every direction and dataset. This is TAMP's mechanism with in a centered PCA subspace, the "direction-dependent, uncertainty-aware fusion" of Appendix D plays no role in the only setting where DEFTS alone is the best method. The comparison is also unequal: baselines run with untuned fixed weights (Tip-Adapter , TAMP , LDA weight chosen "empirically"), while DEFTS's own constants ( from Fig. 9, the 95% variance threshold from Table 9, the 0.1/0.9 split from Table 8) were selected by inspecting test accuracy on the same 12 datasets.
- The ensembling results have no matched control. Every "X + DEFTS" entry is compared to X alone, never to "X + TAMP" or "X + LDA" with the same search budget, and at the "X + DEFTS" rows search a strictly larger leave-one-out grid than X, so the ensemble can hardly lose. The +0.4 to +0.7 gains on the strongest baselines are within what added search freedom alone can produce on a few support images.
This is consequential because these experiments carry the paper. Observations 1–3 are interesting but are already covered by TAMP and Zhang et al. (2026), which the paper cites, from on LDA+ZS is the stronger method, and once the result is seen to be fixed-weight interpolation under an unequal protocol and the ensembling gains are seen to lack a control, there is no remaining experiment that demonstrates the specific contribution the paper claims.
Questions for authors
-
At , as specified above, Eq. (5) appears to reduce to for every dataset and direction. Is this correct as implemented? If so, what in Table 1 is attributable to the Bayesian estimator rather than to fixed-weight interpolation in the centered PCA subspace?
-
Were , the 95% variance threshold and the 0.1/0.9 split chosen using test accuracy on the 12 evaluation datasets (Figs. 9, Tables 8–9)? If yes, why were the baselines at not given an equivalent opportunity (e.g. constants from their original papers, or selection on a disjoint set of datasets)?
-
For the ensembling claim: what do "X + TAMP" and "X + LDA" give under the same leave-one-out grid size as "X + DEFTS", for X ∈ {Tip-Adapter, APE, LDA+ZS}? Without this I cannot tell whether the gains are specific to DEFTS.
-
What is the subspace dimension after 95% truncation, per dataset? Several arguments (data efficiency of estimating , the ImageNet exception) depend on it.
-
Why is Zhang et al. (2026), described in the related work as splitting image embeddings into text-span and orthogonal components, not included as a baseline alongside TAMP?
Suggestions
- Consider recentring the paper on the subspace projection, which I found to be its more interesting part. Section 2.2, Table 7 and Figure 2 stand on their own and could be developed further (why centering helps, behaviour as a function of and , whether the subspace helps other estimators). The Bayesian step contributes little at and could be presented as one optional estimator within that subspace rather than as the core method.
- Define and derive the equation following "orthogonal to any text embeddings" explicitly.
- Fix Figure 3 so each panel has its own y-axis ticks, the shared labels are wrong for all but the leftmost panel.
- Discuss LFA (Ouali et al., ICCV 2023) in the related work.
3 comments
Question 1. Source of the one-shot gain. We agree with your reduction and have added the derivation to Appendix B.3. At , the gain comes from the centered PCA subspace projection and empirical variance estimation.
Question 2. Hyperparameter sensitivity and transfer. We selected the default values of and the threshold using all training data of the 12 datasets. To better understand how these hyperparameters affects the performances of DEFTS. We now report additional results in Table 3, Figure 8, and Tables 13–15.
When , Table 3 shows that DEFTS performs worse than equally weighting the image and text embeddings within the same subspace. Figure 8 shows that one-shot accuracy is relatively insensitive to the variance split. Together, these results suggest that the one-shot advantage of DEFTS over TAMP comes primarily from the centered PCA subspace projection.
In Tables 13–15, we analyze the sensitivity and transferability of and and the 95% variance threshold by performing two additional experiments. First, we performed oracle tuning, optimizing and the threshold directly on each dataset's own test accuracy. Second, we used a leave-one-dataset-out approach: for any given dataset, we tuned the hyperparameters on the other 11 datasets before applying them to the held-out one. The results show that oracle tuning yields an average improvement of less than 1 point for. Moreover, tuning on the other datasets alters the accuracy by at most 0.14 points on average across varying values of, demonstrating that our fixed hyperparameter settings transfer robustly across different datasets.
Question 3. Comparison with X + TAMP and X + LDA. Table 22 now directly compares X + DEFTS with X + TAMP. We chose TAMP for this comparison because both TAMP and DEFTS estimate their main components within the text-spanned subspace. On the other hand, LDA operates in the full feature space and can use information outside that subspace. In fact, Table 2 shows that LDA itself can benefit by ensembling further with DEFTS.
Question 4. Subspace dimension. Table 8 now reports the subspace dimension for every dataset and backbone.
Question 5. Comparison with Zhang et al. (2026). Zhang et al. combine components in the text span and its orthogonal complement, so their estimate uses the full feature space, while the main estimation components of TAMP and DEFTS operate within the text-spanned subspace.
Suggestion 1. Further analysis of the subspace projection. Thank you for the suggestion. We now include Table 8 reporting the subspace ranks, and Table 9 showing how the performance advantage of the subspace varies with the ratio . We leave a more extensive analysis of the subspace as potential future work.
Suggestion 2. Definition and derivation of . Section 2.2 now defines and includes the steps leading to the displayed equation.
Suggestion 3. y-axis of Figure 3. Thank you for spotting this. We have now corrected Figure 3.
Suggestion 4. adding LFA. Thank you for providing the references. We have now added a discussion of LFA to the related work in Appendix A.
Thank you for the detailed response and the careful revision. The paper is clearly better, as the method is now clear, the related work is complete, and the new experiments answer several of my questions. I also appreciate that DEFTS uses one fixed setting for all datasets and shot counts, and Table 15 shows this setting transfers well for k≥2. However, the paper does not yet show that the Bayesian estimator itself is responsible for the gains. In addition, the standalone gains of DEFTS are concentrated at k=2, as the margin at k=1 is small and for k≥4 DEFTS is below LDA+ZS. For the ensembles, many of the reported gains come from DEFTS replacing a weaker method rather than adding to it, so I suggest measuring each gain against max(X, DEFTS alone). By that measure, the clearest improvement is with LDA for k≥4, but it is relatively small (+0.5 to +0.7) and weaker on ViT-H/14. For these reasons I keep my score. I encourage the authors to simplify the method by keeping only the components that clearly contribute to performance. For example, Table 10 suggests that centering and the 95% threshold could be dropped with almost no loss (within 0.35 points at every k), which would make the method simpler and its contribution easier to see.
We thank the reviewer for their careful reading and for finding our results regarding subspace insightful. We address your questions and concerns point by point below.
Weakness 1: We thank the reviewer for pointing out the additional references and highlighting areas for improvement in the writing.
1. Related work: We have now cited and discussed the suggested references including LFA (Ouali et al., ICCV 2023) in Appendix A as suggested.
2. James–Stein shrinkage: We have now explicitly connected this concept in Section 2.3 by adding the following clarification: "In other words, the posterior mean is a weighted average of the sample mean..."
3. Modality gap: We have now incorporated this into Section 2.2 to provide better connection: "A known reason why the raw text embeddings are poor prototypes is the modality gap (Liang et al., 2022)..."
4. Missing citations and the definition of PS: We now formally define PS in Section 2.2, and we explicitly derive the steps leading to the displayed equation. Regarding the sentence on prior manuscript: to the best of our knowledge, while TAMP and Zhang et al. utilize projections, they do not investigate whether the subspace itself constitutes the performance bottleneck, which is our core focus in Section 2.2. We now emphasize this distinction in the main text more clearly.
5. Number of samples: We have now included clarification to the captions of both Figure 1 and Figure 2 for the number of samples.
Motivation for Observation 3. We now revised Observation 3 to explain why we test dimensionality reduction within the text-spanned subspace. The subspace is tight itself, so in practice only discarding small variance improves accuracy, while discarding large variance, e.g. 50% substantially degrades it.
Motivation for the Bayesian estimator. Section 2.3 now starts with motivations for Bayesian formulation and pooled covariance estimation.
Weakness 2: We thank the reviewers for rasing the questions.
Contribution at k=1. We agree that at k=1, the observed gain comes from the subspace projection and empirical variance estimation. We now state this explicitly in the end of Section 2.3. The Bayesian update provides additional gains from k=2 onward. We now also add Table 3, where DEFTS improves on a fixed weight of 0.5 in the same subspace by 0.68, 2.10, 3.49, and 4.27 percentage points at k=2,4,8,16 respectively. The new Figure 8 in Appendix F.4 also shows that one-shot DEFTS is robust to variation in the noise fraction.
Missing baseline for ensembling. We now added the comparison of X + DEFTS and X + TAMP in Table 22 at Appendix F.12. Across the 28 comparisons at k≥2, X + DEFTS is significantly ahead in 21, tied in 6, and behind in 1 (TIP-X at k=8).