acceptodds
Under review as a conference paper at ICLR 2027

DEFTS: Data-Efficient, Training-Free Few-Shot Learning for CLIP in the Text Subspace

Abstract

Contrastive Language-Image Pre-training (CLIP) achieves impressive zero-shot image classification performance by comparing image embeddings against class-specific text embeddings. While recent few-shot methods leverage labeled training samples to improve upon this zero-shot baseline, they often predominantly rely on image embeddings and underutilize the semantic structure of the text embeddings. In this paper, we introduce DEFTS, a data-efficient, training-free few-shot classification method strictly in the text subspace. We first demonstrate empirically that while raw text embeddings can be suboptimal as direct classifiers, the linear subspace they span remains highly discriminative. DEFTS therefore proposes to reduce the image-text modality gap by projecting estimated image prototypes directly into this subspace. Importantly, by expressing these prototypes as linear combinations of text embeddings, DEFTS reduces a high-dimensional estimation problem to the derivation of just a few coefficients. These coefficients are then computed via a Bayesian posterior mean, using a prior naturally centered on the original text embedding coordinates. Extensive experiments across 12 image classification benchmarks demonstrate that DEFTS significantly improves upon existing training-free methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Few-Shot Learning, Vision-Language Models, Training-Free Methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HXAbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DEFTS, a training-free few-shot classification method for CLIP. Given text embedding for each class, DEFTS computes their mean, forms a scatter matrix from the centered text embeddings, and eigendecomposes this matrix. It retains the smallest number of leading eigenvectors that explain at least 95% of the text-embedding variance. These eigenvectors define the subspace in which DEFTS projects both the support-image embeddings and the class text embeddings. Within this subspace, it estimates an image prototype for each class using a Gaussian prior centered at the corresponding projected text embedding and a shared covariance estimated from the support set. The posterior-mean prototypes can be used directly for classification or ensembled with existing training-free methods. The approach is evaluated on 12 classification datasets and three CLIP backbones.

Critical strengths

1. The subspace observation is simple, well controlled, and useful. The finding that NCM prototypes estimated inside the text-spanned subspace are far more sample-efficient than full-space prototypes (Figure 2), and that centering the text embeddings helps (Table 7), is clearly shown. Observation 1's comparison against same-rank random and PCA subspaces is the right control. This part of the paper has value independent of the estimator built on top of it.

2. Practical and training-free. DEFTS combines projected visual evidence with a text-derived prior through a closed-form estimator and requires no gradient-based optimization. This makes the approach computationally attractive and potentially useful when only one or two labeled examples per class are available.

Critical weaknesses

1. The writing needs substantial revision. Several parts of the method cannot be followed as written.

  • Related work (main text and Appendix A). The paper lists prior methods without giving enough background on them or stating how DEFTS differs from each. CoOp, Zhao et al. (2024) and Zhu et al. (2024) are cited and never discussed, and the Appendix A paragraphs on the modality gap and on James–Stein shrinkage are directly relevant to the prior in Section 2.3 but are not connected to it. The related work also omits Linear Feature Alignment (LFA, Ouali et al., "Black Box Few-Shot Adaptation for Vision-Language Models", ICCV 2023), which addresses the same image–text alignment problem that motivates the paper.
  • Section 2.2 (Text Embedding Subspace). The projection matrix PSP_S is used but never defined. The step from "(I - P_S)F_v(I) is orthogonal to any text embedding" to the displayed equation is not derived, as written, $P_S F_t(T_i) = F_t(T_i)$ trivially, so the equation does not use the orthogonality it invokes. The sentence "few studies have investigated whether the linear space SS defined by PSP_S is actually a suitable subspace" gives no citations, and the paper's own related work cites two studies (TAMP, Zhang et al. 2026) that do exactly this. The number of images used to fit the NCM classifiers in Observations 1–3 should be stated where the observations are presented, not only in Appendix E.3. The motivation for Observation 3 is unclear: it argues that reducing the subspace dimension hurts, before any dimensionality reduction has been introduced, and the method then adopts a 95% PCA truncation that Table 9 shows helps at every kk.
  • Section 2.3 (DEFTS). The motivation for the Bayesian estimator is not clear to me. The paper states that the subspace is low-dimensional and that the text embedding is an informative prior, but it does not explain why a Gaussian posterior mean with an isotropic prior and a pooled covariance is the right estimator here, how this choice relates to the projection step (the two can be applied independently, as Figures 4 and 5 show), or what the resulting dimension rr is on any dataset.

2. The main experiments are confounded, so they do not test the claimed contribution.

The paper's evidence rests on Table 1 (k=1k=1), Table 2 and the ensembling columns, and each of these is confounded in a way that leaves the stated contribution untested.

  • At k=1k=1, the Bayesian estimator is not what is being evaluated. With the paper's fixed choices ($\hat\sigma^2 = 0.1 D$, τ^2=0.9D\hat\tau^2 = 0.9 D, Σ−1=βσ^−2I\Sigma^{-1} = \beta\hat\sigma^{-2} I, β=0.1\beta = 0.1), Eq. (5) reduces to a fixed scalar interpolation with weight 0.9/1.9≈0.470.9/1.9 \approx 0.47 on the projected image and 0.530.53 on the projected text, identical for every direction and dataset. This is TAMP's mechanism with λ≈0.47\lambda \approx 0.47 in a centered PCA subspace, the "direction-dependent, uncertainty-aware fusion" of Appendix D plays no role in the only setting where DEFTS alone is the best method. The comparison is also unequal: baselines run with untuned fixed weights (Tip-Adapter (β,α)=(1,1)(\beta,\alpha) = (1,1), TAMP λ=0.5\lambda = 0.5, LDA weight 0.0010.001 chosen "empirically"), while DEFTS's own constants (β=0.1\beta = 0.1 from Fig. 9, the 95% variance threshold from Table 9, the 0.1/0.9 split from Table 8) were selected by inspecting test accuracy on the same 12 datasets.
  • The ensembling results have no matched control. Every "X + DEFTS" entry is compared to X alone, never to "X + TAMP" or "X + LDA" with the same search budget, and at k≥2k \ge 2 the "X + DEFTS" rows search a strictly larger leave-one-out grid than X, so the ensemble can hardly lose. The +0.4 to +0.7 gains on the strongest baselines are within what added search freedom alone can produce on a few support images.

This is consequential because these experiments carry the paper. Observations 1–3 are interesting but are already covered by TAMP and Zhang et al. (2026), which the paper cites, from k=4k=4 on LDA+ZS is the stronger method, and once the k=1k=1 result is seen to be fixed-weight interpolation under an unequal protocol and the ensembling gains are seen to lack a control, there is no remaining experiment that demonstrates the specific contribution the paper claims.

Questions for authors
  1. At k=1k=1, as specified above, Eq. (5) appears to reduce to θ^c=0.526 ec′+0.474 bˉc\hat\theta_c = 0.526\,e'_c + 0.474\,\bar b_c for every dataset and direction. Is this correct as implemented? If so, what in Table 1 is attributable to the Bayesian estimator rather than to fixed-weight interpolation in the centered PCA subspace?

  2. Were β=0.1\beta = 0.1, the 95% variance threshold and the 0.1/0.9 split chosen using test accuracy on the 12 evaluation datasets (Figs. 9, Tables 8–9)? If yes, why were the baselines at k=1k=1 not given an equivalent opportunity (e.g. constants from their original papers, or selection on a disjoint set of datasets)?

  3. For the ensembling claim: what do "X + TAMP" and "X + LDA" give under the same leave-one-out grid size as "X + DEFTS", for X ∈ {Tip-Adapter, APE, LDA+ZS}? Without this I cannot tell whether the gains are specific to DEFTS.

  4. What is the subspace dimension rr after 95% truncation, per dataset? Several arguments (data efficiency of estimating Σ\Sigma, the ImageNet exception) depend on it.

  5. Why is Zhang et al. (2026), described in the related work as splitting image embeddings into text-span and orthogonal components, not included as a baseline alongside TAMP?

Suggestions

  • Consider recentring the paper on the subspace projection, which I found to be its more interesting part. Section 2.2, Table 7 and Figure 2 stand on their own and could be developed further (why centering helps, behaviour as a function of rr and KK, whether the subspace helps other estimators). The Bayesian step contributes little at k=1k=1 and could be presented as one optional estimator within that subspace rather than as the core method.
  • Define PSP_S and derive the equation following "orthogonal to any text embeddings" explicitly.
  • Fix Figure 3 so each panel has its own y-axis ticks, the shared labels are wrong for all but the leftmost panel.
  • Discuss LFA (Ouali et al., ICCV 2023) in the related work.
3 comments

We thank the reviewer for their careful reading and for finding our results regarding subspace insightful. We address your questions and concerns point by point below.

Weakness 1: We thank the reviewer for pointing out the additional references and highlighting areas for improvement in the writing.

1. Related work: We have now cited and discussed the suggested references including LFA (Ouali et al., ICCV 2023) in Appendix A as suggested.

2. James–Stein shrinkage: We have now explicitly connected this concept in Section 2.3 by adding the following clarification: "In other words, the posterior mean is a weighted average of the sample mean..."

3. Modality gap: We have now incorporated this into Section 2.2 to provide better connection: "A known reason why the raw text embeddings are poor prototypes is the modality gap (Liang et al., 2022)..."

4. Missing citations and the definition of PSP_{\mathcal{S}}: We now formally define PSP_{\mathcal{S}} in Section 2.2, and we explicitly derive the steps leading to the displayed equation. Regarding the sentence on prior manuscript: to the best of our knowledge, while TAMP and Zhang et al. utilize projections, they do not investigate whether the subspace itself constitutes the performance bottleneck, which is our core focus in Section 2.2. We now emphasize this distinction in the main text more clearly.

5. Number of samples: We have now included clarification to the captions of both Figure 1 and Figure 2 for the number of samples.

  1. Motivation for Observation 3. We now revised Observation 3 to explain why we test dimensionality reduction within the text-spanned subspace. The subspace is tight itself, so in practice only discarding small variance improves accuracy, while discarding large variance, e.g. 50% substantially degrades it.

  2. Motivation for the Bayesian estimator. Section 2.3 now starts with motivations for Bayesian formulation and pooled covariance estimation.

Weakness 2: We thank the reviewers for rasing the questions.

  1. Contribution at k=1k=1. We agree that at k=1k=1, the observed gain comes from the subspace projection and empirical variance estimation. We now state this explicitly in the end of Section 2.3. The Bayesian update provides additional gains from k=2k=2 onward. We now also add Table 3, where DEFTS improves on a fixed weight of 0.5 in the same subspace by 0.68, 2.10, 3.49, and 4.27 percentage points at k=2,4,8,16k=2,4,8,16 respectively. The new Figure 8 in Appendix F.4 also shows that one-shot DEFTS is robust to variation in the noise fraction.

  2. Missing baseline for ensembling. We now added the comparison of X + DEFTS and X + TAMP in Table 22 at Appendix F.12. Across the 28 comparisons at k≥2k\geq2, X + DEFTS is significantly ahead in 21, tied in 6, and behind in 1 (TIP-X at k=8k=8).

Question 1. Source of the one-shot gain. We agree with your reduction and have added the derivation to Appendix B.3. At k=1k=1, the gain comes from the centered PCA subspace projection and empirical variance estimation.

Question 2. Hyperparameter sensitivity and transfer. We selected the default values of β\beta and the threshold using all training data of the 12 datasets. To better understand how these hyperparameters affects the performances of DEFTS. We now report additional results in Table 3, Figure 8, and Tables 13–15.

When k=1k=1, Table 3 shows that DEFTS performs worse than equally weighting the image and text embeddings within the same subspace. Figure 8 shows that one-shot accuracy is relatively insensitive to the variance split. Together, these results suggest that the one-shot advantage of DEFTS over TAMP comes primarily from the centered PCA subspace projection.

In Tables 13–15, we analyze the sensitivity and transferability of β\beta and and the 95% variance threshold by performing two additional experiments. First, we performed oracle tuning, optimizing and the threshold directly on each dataset's own test accuracy. Second, we used a leave-one-dataset-out approach: for any given dataset, we tuned the hyperparameters on the other 11 datasets before applying them to the held-out one. The results show that oracle tuning yields an average improvement of less than 1 point for. Moreover, tuning on the other datasets alters the accuracy by at most 0.14 points on average across varying values of, demonstrating that our fixed hyperparameter settings transfer robustly across different datasets.

Question 3. Comparison with X + TAMP and X + LDA. Table 22 now directly compares X + DEFTS with X + TAMP. We chose TAMP for this comparison because both TAMP and DEFTS estimate their main components within the text-spanned subspace. On the other hand, LDA operates in the full feature space and can use information outside that subspace. In fact, Table 2 shows that LDA itself can benefit by ensembling further with DEFTS.

Question 4. Subspace dimension. Table 8 now reports the subspace dimension rr for every dataset and backbone.

Question 5. Comparison with Zhang et al. (2026). Zhang et al. combine components in the text span and its orthogonal complement, so their estimate uses the full feature space, while the main estimation components of TAMP and DEFTS operate within the text-spanned subspace.

Suggestion 1. Further analysis of the subspace projection. Thank you for the suggestion. We now include Table 8 reporting the subspace ranks, and Table 9 showing how the performance advantage of the subspace varies with the ratio K/dK/d. We leave a more extensive analysis of the subspace as potential future work.

Suggestion 2. Definition and derivation of PSP_S. Section 2.2 now defines PSP_S and includes the steps leading to the displayed equation.

Suggestion 3. y-axis of Figure 3. Thank you for spotting this. We have now corrected Figure 3.

Suggestion 4. adding LFA. Thank you for providing the references. We have now added a discussion of LFA to the related work in Appendix A.

Thank you for the detailed response and the careful revision. The paper is clearly better, as the method is now clear, the related work is complete, and the new experiments answer several of my questions. I also appreciate that DEFTS uses one fixed setting for all datasets and shot counts, and Table 15 shows this setting transfers well for k≥2. However, the paper does not yet show that the Bayesian estimator itself is responsible for the gains. In addition, the standalone gains of DEFTS are concentrated at k=2, as the margin at k=1 is small and for k≥4 DEFTS is below LDA+ZS. For the ensembles, many of the reported gains come from DEFTS replacing a weaker method rather than adding to it, so I suggest measuring each gain against max(X, DEFTS alone). By that measure, the clearest improvement is with LDA for k≥4, but it is relatively small (+0.5 to +0.7) and weaker on ViT-H/14. For these reasons I keep my score. I encourage the authors to simplify the method by keeping only the components that clearly contribute to performance. For example, Table 10 suggests that centering and the 95% threshold could be dropped with almost no loss (within 0.35 points at every k), which would make the method simpler and its contribution easier to see.

Reviewer_TuMUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates training-free few-shot image classification using CLIP. It builds upon the observation that zero-shot CLIP predictions depend on an image embedding solely via its projection onto the span of class text embeddings. The authors empirically demonstrate (Figs. 1–3) that this text-spanned subspace exhibits strong discriminative power, and that estimating class prototypes within this subspace yields better sample efficiency than estimation over the full embedding space. They propose DEFTS: text embeddings are centered and PCA-reduced (retaining 95% variance). Each class prototype is estimated within the resulting r-dimensional space as the posterior mean of a Gaussian model. The Gaussian prior is centered at the projected text embedding, and the likelihood is derived from the k support images. The prior variance, noise variance, and a pooled, shrunk covariance matrix are calibrated via empirical Bayes, with a fixed 0.9/0.1 split for k = 1. Experiments across 12 benchmarks and 3 backbones show that DEFTS achieves the strongest standalone performance at k = 1 and 2, and can complement existing training-free methods when used in ensembles.

Critical strengths

(1) Controlled empirical insight into CLIP geometry. The equal-rank comparison (text span vs. PCA-of-images vs. random subspace, Fig. 1) and the k-shot estimation study (Fig. 2) disentangle the subspace effect with appropriate controls. The resulting observation — that the text span is discriminative and sample-efficient for prototype estimation, with the exceptions of ImageNet and EuroSAT honestly reported — is valuable beyond the proposed method itself.

(2) Strong tuning-free performance under low-shot settings. DEFTS admits a closed-form solution, calibrates its hyperparameters via empirical Bayes, and delivers the highest average accuracy among competing methods at k = 1 and k = 2 (77.12 / 79.50, Table 2). Its behavior remains consistent across three backbones (Table 14).

Critical weaknesses

(1) The Bayesian machinery is not isolated in the regime where the method matters most. At k = 1 the likelihood precision is isotropic (Σ⁻¹ = β·σ̂⁻²·I_r, Sec. 2.3), so by the authors' own analysis (Appendix D) the posterior reduces to a nearly even scalar blend (about 52:48) of the projected text embedding and the single support image. Meanwhile, switching only the projection from centered to uncentered PCA costs 3.43 points at k = 1 (76.97 vs. 73.54, Table 7) - more than the entire reported DEFTS - TAMP gap at k = 1 (+2.66, Table 3) - and the uncentered DEFTS variant (73.54) even falls below TAMP alone (74.47). No ablation (Tables 3-4) includes the decisive control: centered PCA with a fixed scalar weighting. As it stands, the low-shot advantage may be attributable to the centered projection plus data-adaptive weighting rather than to the Bayesian update.

(2) Two claims are overstated relative to the supporting evidence. DEFTS alone falls below zero-shot CLIP by 3.43 points on ImageNet and 1.10 on Food101 at k = 1 (Table 1); and several ensemble deltas are negative - e.g., Linear Probe + DEFTS on EuroSAT: -0.57 (k = 4), -1.21 (k = 8), -3.98 (k = 16) (Tables 11-13) - which is difficult to reconcile with the stated "significant and consistent performance gains" (Contribution 3).

Questions for authors

(1) Please add the control ablation - centered PCA (95%) with a fixed scalar weighting of the projected image mean and the text prior - at k = 1 and 2. If it matches DEFTS, the low-shot gains stem from the projection and adaptive weighting rather than the Bayesian update, which would materially change my assessment of the core claim.

(2) On ImageNet, where the text subspace approximately spans the full feature space, DEFTS alone is 3.43 points below zero-shot at k = 1. Can you explain this failure mode (e.g., error analysis against zero-shot predictions)?

(3) In Table 3 (k = 1), TAMP is reported without zero-shot ensembling (74.47), whereas Table 2 uses TAMP+ZS (76.21). Please also report DEFTS alone against TAMP+ZS.

(4) The constants β = 0.1, the 95% threshold, and the 0.9/0.1 split were fixed from experiments on the same 12 benchmarks (F.3, F.5, F.6). How sensitive is k = 1 performance to the 0.9/0.1 split (which sets the mixing ratio), and is τ̂² clamped when the estimate in Eq. (3) is negative?

2 comments

We thank the reviewer for their precise analysis and for finding our results regarding CLIP geometry insightful. We address your questions and concerns point by point below.

Weakness 1: We agree that when k=1k = 1, DEFTS acts as a scalar blend of the projected image embeddings and the text embeddings. Thus, its one-shot gain stems from the projection and empirical weighting rather than the Bayesian update (which becomes impactful for k>1k > 1). We now state this explicitly at the end of Section 2.3 and derive the result step-by-step in the new Appendix B.3. Furthermore, we have added Table 3 in Section 3.2.2 to compare DEFTS against a baseline using a fixed weight of 0.5 within the same centered 95% subspace:

Method1 shot2 shots4 shots8 shots16 shots
Fixed weight 0.577.1678.8279.5680.0280.32
DEFTS77.1279.5081.6683.5184.59

We believe this new table provides strong additional evidence demonstrating the effectiveness of the Bayesian updates used in DEFTS when k>1k > 1 .

Weakness 2: We agree that we previously overstated the claim that the gains are "consistent." We have replaced this phrasing with specific empirical counts in both Contribution 3 and the Limitations paragraph in Section 4: out of 480 dataset-level experiments, 366 yielded significantly positive improvements, while 19 were negative (only 5 of which were statistically significant).

Question 1: Thanks for the suggestion! We have added Table 3 (shown above) as a control ablation. At k=1k = 1, the fixed-weight control matches DEFTS (77.16 vs. 77.12), exactly as expected. However, at k=2k = 2, DEFTS pulls ahead by 0.68 points, and this performance gap widens to 4.27 points at k=16k = 16.

Question 2: Thank you for raising this concern. The performance drop occurs because we fixed the covariance inflation parameter at β=0.1\beta = 0.1 and the PCA variance threshold at 95%, which severely penalizes ImageNet at k=1k = 1 (where ImageNet retains r=289r = 289 directions, see Table 8). If we specifically tune these values for ImageNet, accuracy improves by 5.33 points, as detailed in Table 14 (Appendix F.8).

Question 3: For k=1k = 1, the performance of standalone DEFTS compared to TAMP + ZS is now provided in Table 16: DEFTS achieves 77.12, whereas TAMP + ZS achieves 76.21. For k>1k > 1, TAMP + ZS yields 76.65, 78.80, 80.00, and 80.89 for k=2,4,8,16k = 2, 4, 8, 16 (now shown in Table 22), compared to 79.50, 81.66, 83.51, and 84.59 for DEFTS alone.

Question 4: Thank you for the suggestion. We have now analyzed the sensitivity of the hyperparameters and the variance split in Figure 8 (Appendix F.4) and Tables 13–15 (Appendix F.8).

Regarding the variance split when k=1k = 1, the one-shot accuracy is highly stable (between 76.84 and 77.18) for noise fractions ranging from 0.06 to 0.15, demonstrating that the one-shot performance of DEFTS is robust to this choice.

To analyze the sensitivity of β\beta and the 95% threshold, we added a new section in Appendix F.8 including two additional experiments. First, we performed oracle tuning, optimizing β\beta and the threshold directly on each dataset's own test accuracy. Second, we used a leave-one-dataset-out approach: for any given dataset, we tuned the hyperparameters on the other 11 datasets before applying them to the held-out one. The results show that oracle tuning yields an average improvement of less than 1 point for k>1k > 1. Moreover, tuning on the other datasets alters the accuracy by at most 0.14 points on average across varying values of kk, demonstrating that our fixed hyperparameter settings transfer robustly across different datasets.

Finally, regarding the clamping of τ^2\hat{\tau}^2: at k=1k = 1, τ^2=(1−f)∥δ∥2/r\hat{\tau}^2 = (1 - f)\Vert{}\delta\Vert{}^2/r, which is strictly positive by construction (now derived in Appendix B.3). For k≥2k \ge 2, our implementation floors the estimate in Eq. (3) at a small positive constant (10−610^{-6}). In our reported experiments, this floor was never active; the estimate remained positive across all 480 fits (12 datasets ×\times 4 values of kk ×\times 10 seeds).

Thank you for the clarifications and additional experiments. However, my core concern remains: the contribution of the Bayesian estimator cannot be disentangled in the critical operating regime. Therefore, I maintain my original score.

Reviewer_AhwrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DEFTS, a training-free, data-efficient few-shot CLIP adaptation method. The core observation is that CLIP's zero-shot classification depends only on the projection of the image embedding onto the subspace spanned by the K class text embeddings, and that this subspace is itself highly discriminative and not the bottleneck; the real bottleneck is that the raw text-embedding coordinates are suboptimal. Building on this, DEFTS first applies centered PCA to the text embeddings to obtain a low-dimensional subspace, then reduces image prototype estimation to a Bayesian estimation problem over a small set of coefficients within that subspace: the projected text embeddings serve as the Gaussian prior mean, the pooled covariance as the likelihood, Empirical Bayes is used to estimate the prior variance and the shrunk precision matrix, and the posterior mean is taken as the class prototype for cosine classification or for ensembling with other training-free methods. Across 12 datasets and 3 backbones, DEFTS alone achieves the best average accuracy in the 1/2-shot settings, and when ensembled it yields consistent gains over several methods.

Critical strengths
  1. The entry angle of the problem describes well.
  2. The Bayesian formalization and Empirical Bayes hyperparameter estimation are appropriate and self-consistent in the few-shot setting.
  3. The experimental matrix is solid and the statistical reporting of ensemble gains is disciplined.
Critical weaknesses
  1. This Observations 1/2 do not constitute independent evidence at the most representative scale, yet the authors still state them as a universality argument, lacking a quantitative characterization of how the subspace advantage decays with the K/d ratio.
  2. The empirical fixing of the 95% variance threshold and β=0.1 lacks a transferable selection criterion.
  3. The fixed split σ²=0.1·deviation, τ²=0.9·deviation at k=1 is the most fragile assumption in the paper, but it is supported by only one appendix table.
  4. If LDA is also projected into the same text subspace and its class means shrunk toward text coordinates, would most of DEFTS's gain be reproduced? Without this ablation-style baseline, the boundary of DEFTS's independent contribution is unclear.
  5. The magnitude of ensemble gains is strongly negatively correlated with baseline strength, and the paper provides no normalized analysis.
Questions for authors
  1. The anomaly on EuroSAT is attributed to out-of-distribution, but the same explanation cannot account for why DEFTS's gain over SimpleShot on EuroSAT is instead large.
  2. For Figure 6, pooled covariance is overtaken by per-class covariance at k=16, but the paper still fixes pooling globally and gives no switching criterion.
1 comment

We thank the reviewer for the careful reading and for finding the Bayesian formulation appropriate and self-consistent. The concerns are mainly about the robustness of the fixed hyper-parameters and the attribution of the gain of DEFTS. We have now made these points clearer and run additional experiments, which we summarize below.

Weakness 1: We agree that Observations 1/2 are not universal but only hold for most of the datasets considered (10 of 12; the exceptions are ImageNet and EuroSAT), and we now state this more clearly in the paper. Additionally, we now provide an analysis in Table 9 (Appendix F.2) that quantifies the subspace advantage against the K/dK/d ratio. The Spearman correlation between the advantage and K/dK/d is −0.43,−0.29,−0.15,−0.10-0.43, -0.29, -0.15, -0.10 and 0.000.00 for k=1,2,4,8,16k = 1, 2, 4, 8, 16 (p-values 0.170.17 to 0.990.99), so there is no statistically significant relationship.

Weakness 2: We agree that the 95% PCA variance threshold and the covariance inflation parameter β=0.1\beta = 0.1 were chosen based on empirical results. However, the fixed values work well for all 12 datasets considered, as shown in Tables 1 and 2. Additionally, we now include a sensitivity analysis for the two hyper-parameters in Appendix F.8 (Tables 13–15), suggesting that the fixed values transfer robustly for different datasets. Specifically, tuning both on each dataset's own test accuracy (oracle) gains on average +1.82,+0.84,+0.61,+0.33,+0.31+1.82, +0.84, +0.61, +0.33, +0.31 points for k=1,2,4,8,16k = 1, 2, 4, 8, 16, and tuning them on the other 11 datasets changes the accuracy by −0.60,−0.14,+0.01,+0.02,+0.06-0.60, -0.14, +0.01, +0.02, +0.06.

Weakness 3: Thanks for raising your concern about this assumption. To support our choice of the split of the variance, we now add a new Figure 8 (Appendix F.4) showing that the choice is robust: for noise fractions from 0.060.06 to 0.150.15 the one-shot accuracy stays between 76.8476.84 and 77.1877.18, above zero-shot CLIP (74.6574.65).

Weakness 4: Thank you for making the suggestion. We ran the experiment as you suggested, and it shows that the components of DEFTS are useful. Average accuracy over the 12 datasets:

Method1 shot2 shots4 shots8 shots16 shots
LDA, full space50.8869.2478.7283.4986.14
LDA, shrunk towards the text coordinates as in DEFTS75.9978.8981.5683.5984.95
DEFTS77.1279.5081.6683.5184.59

Weakness 5: We thank the reviewer for pointing this out. We now add the relative error reduction of DEFTS in Table 21 (Appendix F.11) for better evaluating the performances of DEFTS.

Question 1: EuroSAT is a well-documented outlier for CLIP, and as shown in Figure 2 (and Table 9), estimating in the text subspace is worse than estimating in the full space (−0.2,−4.4,−6.6,−6.5-0.2, -4.4, -6.6, -6.5 points for k=2,4,8,16k = 2, 4, 8, 16). However, the accuracy still increases with the number of shots kk, which suggests that the text space still has discriminative power for this dataset although it is an outlier. By adding the semantic meaning through ensembling with DEFTS, SimpleShot, which operates in the full space, can still benefit. We would also note that the gain of SimpleShot + DEFTS on EuroSAT at k=2k = 2 is +5.18±2.74+5.18 \pm 2.74 (Table 1), smaller than on CIFAR-10 (+15.84+15.84), CIFAR-100 (+28.50+28.50), DTD (+14.29+14.29) or Food101 (+17.06+17.06).

Question 2: Thank you for pointing this out. The error was in our text, not in the figure. We have now removed the wrong text in Section 3.2.3, which says that pooling is better at k=2k = 2 and 44 and that per-class covariance only gets close at k=8k = 8 and 1616. Per-class covariance never overtakes the pooling method: the difference is −0.12±0.05-0.12 \pm 0.05 at sixteen shots, and per-class is not significantly better on any dataset. Switching rules chosen without test labels gained nothing, so no switching criterion is needed. Accuracy (%), average over 12 datasets:

Method2 shots4 shots8 shots16 shots
Pooled covariance (DEFTS)79.5081.6683.5184.59
Per-class covariance76.6280.8383.2084.47

Thanks again for your careful review and constructive suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.