acceptodds
Withdrawn from ICLR 2027

Project and Mix: Task-Semantic Prototypes for Few-Shot Image Classification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models like CLIP are trained with the objective of aligning text and image pairs. Beyond text prompts alone, recent works show that exploiting few-shot image embeddings from a training set is effective for CLIP-based classification. In this work, we analyze mixing image and text prototypes from a bias-variance perspective and show that mixing prototypes acts like a variance shrinkage estimator. Naively mixing text and image prototypes combines two partially aligned spaces since the two modalities are not perfectly aligned. To address this, we project image prototypes onto the principal directions of the semantic text embedding space to obtain a task-semantic image subspace. Mixing the image prototypes with text embeddings in the task-semantic subspace improves few-shot classification. However, when the task-semantic subspace captures insufficient discriminative visual information, relying on this subspace alone can be suboptimal. On extensive experiments over several few-shot classification benchmarks, we show that combining a task-semantic mixed prototype classifier and an anisotropic image-specific classifier systematically outperforms existing methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision-language Models, Cross-Modal Alignment, Prototypes

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yzKkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates training-free few-shot image classification using vision-language models. The authors analyze the mixing of visual and textual class prototypes from a bias–variance perspective, arguing that incorporating textual prototypes can reduce the estimation variance of visual prototypes in few-shot settings. Based on this analysis, the authors propose constructing a task-semantic subspace using the principal components obtained from SVD of textual prototypes. Visual prototypes are projected into this subspace before being mixed with textual prototypes. To compensate for potentially discriminative visual information lost during projection, the authors further introduce a Mahalanobis-based image classifier using within-class covariance estimation. The predictions from the two classifiers are then combined through weighted score fusion. Experiments are conducted on 15 datasets and multiple vision-language models, demonstrating competitive average classification accuracy across different few-shot settings.

Critical strengths
  1. The research problem is practically relevant. Effectively leveraging a small number of labeled images and textual semantics for classification without updating pretrained vision-language models is a meaningful research problem. The proposed method requires no additional gradient-based training and is relatively simple to implement, making it potentially useful in practical applications.

  2. The experiments cover a broad range of settings and demonstrate competitive performance. The authors evaluate their method on 15 datasets, across different numbers of shots and multiple vision-language models, with comparisons against representative methods such as GDA, APE, and ProKeR. The results show improvements in average classification accuracy, particularly in the 1–4 shot settings.

  3. The paper is generally well organized. The method follows a clear progression from bias–variance analysis of prototype estimation to semantic subspace projection and dual-classifier fusion. The overall approach is intuitive and relatively easy to understand and implement.

Critical weaknesses
  1. Limited methodological novelty

The proposed method primarily combines established techniques, including SVD-based feature projection, visual-textual prototype mixing, Mahalanobis classification, and classifier fusion. In particular, GDA (ICLR 2024) has already explored a closely related framework combining a text-based classifier with an LDA-based visual classifier.

The main distinction of this work lies in projecting visual features into a text-derived subspace before prototype mixing. However, the paper does not sufficiently demonstrate what fundamentally new methodological insights this combination provides beyond existing approaches.

  1. Insufficient justification of the motivation and methodology

The paper identifies cross-modal misalignment as an important motivation. However, the proposed text-guided projection does not explicitly model or optimize cross-modal alignment, and the paper provides limited evidence that it actually improves the correspondence between visual and textual representations.

More importantly, the motivation for task-semantic projection is to exclude task-orthogonal components that may interfere with prototype mixing. However, the final method reintroduces the excluded visual information through a Mahalanobis classifier operating on the full feature space.

Although this may be justified by classifier complementarity, it weakens the connection between the original motivation and the final method. The improved performance of the complete system does not necessarily validate the benefit of excluding task-orthogonal components, as the final predictions still rely on the information initially removed. The paper lacks sufficient analysis to distinguish these effects.

  1. Insufficient experimental analysis and unclear sources of performance gains

The Mahalanobis classifier is already a strong established baseline, and the existing ablations mainly compare different NCM variants. A direct comparison between Mix + Mahalanobis and Proj + Mix + Mahalanobis is needed to isolate the contribution of the proposed semantic projection within the final framework.

  1. The reliability of the cross-modal alignment metric is insufficiently validated.

The proposed principal-angle-based metric is sensitive to subspace dimensionality, as higher-dimensional subspaces may exhibit larger alignment scores even without stronger semantic correspondence. Since the effective subspace dimensions may vary across datasets with different numbers of classes, the metric may conflate geometric overlap with genuine semantic alignment. More importantly, this metric is also used to determine classifier fusion weights, yet the lack of dimension-controlled experiments or random-subspace baselines leaves the proposed adaptive fusion strategy insufficiently justified.

Questions for authors
  1. Methodological novelty and contributions beyond existing approaches

The proposed method combines established techniques, including SVD-based feature projection, visual-textual prototype mixing, Mahalanobis classification, and classifier fusion. Moreover, GDA (ICLR 2024) has already explored a closely related framework combining text-based and covariance-based visual classifiers.

Could the authors clarify what constitutes the key methodological novelty of this work beyond the combination of existing techniques? What fundamentally new insights or technical contributions does the proposed method offer compared with prior approaches such as GDA?

  1. Relationship between semantic projection and the final classifier

The motivation for task-semantic projection is to exclude task-orthogonal components that may interfere with cross-modal prototype mixing. However, the final method reintroduces the excluded visual information through a Mahalanobis classifier operating on the full feature space.

Could the authors clarify how this design is consistent with the original motivation? In particular, how do the performance improvements of the complete system validate the benefit of excluding task-orthogonal components, rather than simply reflecting the gains from combining two classifiers?

  1. Cross-modal misalignment and the effectiveness of semantic projection

The paper identifies cross-modal misalignment as an important motivation, yet the proposed method primarily projects visual features into a subspace defined by textual prototypes.

Could the authors explain more precisely why this operation is expected to alleviate cross-modal misalignment? Is there direct evidence that the projection improves the semantic correspondence between visual and textual representations?

  1. Cross-modal alignment metric and adaptive fusion

Given that principal-angle cosine values depend on subspace dimensionality, how do the authors ensure that alignment scores are comparable across datasets with different numbers of classes?

After controlling for subspace dimensionality, does the proposed metric still reliably reflect cross-modal alignment? Does the resulting adaptive fusion strategy consistently outperform simpler fixed-weight fusion approaches?

If the authors can satisfactorily address these concerns with clear explanations and convincing evidence, I would be willing to raise my score accordingly.

Reviewer_7QMZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a classifier for frozen vision-language features. It projects support-image prototypes onto a subspace defined by class-text prototypes, mixes them with the text prototypes, and combines the resulting classifier with a Mahalanobis image classifier. It reports both validation-tuned and validation-free variants across several backbones and datasets.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The experimental results demonstrate the effectiveness of the proposed method.
  3. Projecting visual prototypes into a text-defined subspace before mixing and classification is a reasonable way to reduce irrelevant visual information.
  4. The theoretical analysis helps explain the bias–variance trade-off in mixing visual and textual prototypes.
Critical weaknesses
  1. Prototype mixing and classifier ensembling are well established in few-shot learning, so the additional contribution needs to be clearer.

  2. Although the paper includes component ablations, it lacks a matched comparison with unprojected Mix combined with the same Mahalanobis classifier. This would help establish how much of the final gain comes from the proposed projection.

  3. The choice of truncated SVD needs clearer justification. The paper motivates using a text-defined space, but does not compare alternative constructions, such as PCA or VAE-based representations.

  4. The sensitivity to the 99% energy threshold should be analysed to show how much the results depend on the chosen subspace dimension.

Questions for authors

Please see weaknesses.

Reviewer_1AgsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses training-free adaptation of CLIP-style vision-language models for few-shot classification, proposing a classifier named CMP. Specifically, task-semantic subspace is spanned by class text prototypes, and decompose image prototypes into a task-semantic component and a task-orthogonal component. Besides, theoretical analysis shows that in the few-shot regime, mixing image prototypes with text prototypes acts as a shrinkage estimator, trading a small amount of bias for variance reduction and thus lowering mean squared error. Moreover, the authors perform mixing only within the task-semantic subspace to obtain the Proj+Mix prototype classifier; in parallel, a Mahalanobis image classifier based on the global covariance captures discriminative information outside the text subspace. The final experiments validate the performance of the proposed method.

Critical strengths
  1. The task-semantic subspace + orthogonal subspace decomposition has a clear geometric interpretation and forms a closed loop with the bias-variance analysis.
  2. This paper offers an honest and quantifiable diagnosis of when the method fails, rather than reporting only average gains.
  3. The validation-free hyperparameter selection scheme has practical deployment value.
Critical weaknesses
  1. For Eq. (4), the authors provide a dataset-independent fixed table, and Appendix D.5 admits only that the trend of the tuned optimal λ increasing with shots is qualitatively consistent with it, without giving any approximate derivation or fit from Eq. (4) to the table.
  2. For Eq. (5), the rank k of the task-semantic subspace is barely specified in the main text, but it is the key hyperparameter of the entire projection operator.
  3. For Eq. (7), this paper uses the n/(n+1) factor to compensate, but at n=1 this factor only drops from 1 to 0.5; whether this compensation matches the actual noise level of 1-shot prototypes is not validated at all.
  4. For Eq. (9), this paper reports strong CMP gains at 1 shot in Tables 1-2, without explaining how Σ is computed at 1 shot, whether it degenerates to identity, or whether other regularization is used.
  5. For Eq. (10), the logits of the Proj+Mix and Image branches are simply added, but the two may not be on comparable scales. Besides, this paper performs no logit standardization, temperature scaling, or calibration analysis.
Questions for authors
  1. For Eq. (11), Why squared rather than first power? Why is the reliability term n/(n+1) rather than n/(n+c) or 1−1/n? This paper does not compare with alternative forms (e.g., γ = 1−μ(S_M), γ = (1−μ(S_M))·n/(n+1)) in the appendix.
    
  2. The method's boundary (failure on OOD) is honestly pointed out, but the controlled-variable analysis of the failure cause is insufficient, and the confounding between prompt template and domain shift is not ruled out.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.