acceptodds
Under review as a conference paper at ICLR 2027

Adapt Part of the Way: Class Text as a Prior for Few-Shot Self-Supervised Vision Models

Abstract

Few-shot classification with vision-language models is dominated by jointly trained encoder pairs such as CLIP, whose shared image-text space allows an image to be compared directly with text. Self-supervised vision encoders such as DINOv3 now give stronger visual representations than the CLIP image tower, but they have no text tower: they are adapted from the support images alone and ignore the class name, a prior about the class that does not depend on which, or how many, support images are available. We introduce TAP (Text-Anchored Partial transport), which supplies this prior to a self-supervised encoder using only few-shot supervision. TAP aligns the class-name embeddings of an independently pre-trained text encoder to the visual space with a closed-form orthogonal alignment estimated from the support prototypes, trains a lightweight residual map from support images onto their aligned class names, and applies the map only part of the way at inference: the query moves a fraction of the way toward the map's output while each class name moves the complementary fraction toward its support mean. Adapting part of the way is what makes the learned map useful. We evaluate TAP at 1 to 16 shots against linear probes and few-shot adapters developed for CLIP, on several self-supervised image encoders and several text encoders. On ImageNet, its distribution-shifted variants and the standard 11-dataset few-shot benchmark, TAP outperforms a linear probe at every shot count, by 8.4 points at 1 shot on ImageNet, with the largest gains in the low-shot regime and under distribution shift, and it does so with frozen encoders, a single short training run and no validation labels. The gain grows with the quality of the text encoder, so a frozen self-supervised vision model keeps improving as text encoders do. TAP thus gives self-supervised vision models access to the class names that their adaptation has so far ignored, at negligible inference cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
few-shot classification, vision-language alignment, flow matching, self-supervised vision encoders, frozen encoders, orthogonal Procrustes, distribution shift, DINOv3

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HHX9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAP, a few-shot adaptation method for incorporating class-name semantics from an independently pretrained text encoder into self-supervised vision encoders such as DINOv3. TAP first aligns text prototypes to the visual space using Orthogonal Procrustes, then learns a residual map from support images toward the aligned class names. Instead of applying this map fully, TAP uses partial transport at inference, which improves robustness, especially in low-shot and distribution-shift settings. Experiments across 11 datasets and ImageNet shifts show consistent gains over strong frozen-feature baselines, with extensive ablations supporting the partial-transport design.

Critical strengths
  1. The paper addresses a meaningful problem: leveraging class-name semantics for strong self-supervised vision encoders that otherwise cannot directly use text information.
  2. The empirical gains are convincing, particularly in the low-shot and distribution-shift settings, where the proposed text prior is most useful.
  3. The experimental analysis is thorough, with targeted ablations on partial vs. full transport, different objectives, incorrect class names, random alignment, and multiple vision/text encoders.
Critical weaknesses
  1. The benefit depends strongly on the choice of text encoder: CLIP/SigLIP2 provide large gains, while text-only embedding models provide much smaller or even negative gains, which makes the generality of the “class-name prior” claim less clear.
  2. The gains diminish substantially as the number of shots increases and become negative on some datasets, suggesting that the method is mainly effective in very-low-shot and OOD regimes.
Questions for authors
  1. To what extent does TAP rely on the text encoder having been trained with visual supervision, rather than on class-name semantics alone?
  2. How should we interpret the scope of TAP given that its gains become very small, and sometimes negative at higher shot counts?
2 comments

We thank the reviewer for pressing on generality and scope. Both points changed how the abstract and the conclusion state the result.

Q1 / W1. How much does TAP rely on a text encoder trained with visual supervision?

TAP relies on the class-name semantics as a prior, and a text encoder trained with images strengthens it when few labels are available. Shuffling the class names removes the gain (Table 9), so the semantics are necessary. Semantics from text alone, without visual supervision, already give a useful prior. With Qwen3-Embedding-4B and -8B, which we added to test this (Sec. 3.4, Tables 14 and 16, Fig. 5, all on ImageNet), TAP outperforms the validated probe in distribution and under shift at 1 and 16 shots, and the prior grows with model size. At 1 shot, text encoders trained with images still give two to three times the gain of the largest text-only model. This extra gain is not explained by model size or embedding width (see our reply to Reviewer CQPW, Q2), and only partly by the higher neighborhood agreement of these encoders (0.41 to 0.47, against 0.38 to 0.40 for Qwen3-Embedding-4B and -8B). Its most likely source is the training against images, though our experiments cannot isolate which part of it matters most.

Q2 / W2. How should the scope of TAP be read, given that its gains become small and sometimes negative at higher shot counts?

TAP is most useful with few labels and under distribution shift. Frozen self-supervised features such as DINOv3's are well suited to linear classification, so with 16 labeled images per class a linear probe already finds a strong in-distribution boundary. That boundary is fitted to the source distribution, while the class names carry a prior about each class that is independent of it and separates the classes better under shift (Sec. 3.2). At 16 shots, TAP matches or improves on the probe on every dataset except DTD and FGVCAircraft (Table 23), and averaging TAP with the validated cross-modal probe (Sec. 2.3) recovers most of the gap on these two. Accordingly, we removed "at every shot count" from the abstract's claim over the three benchmarks, and the conclusion and Limitations now state this scope and name both exceptions.

We are happy to answer further questions.

Thank you for your response. The additional experiments addressed my concerns about the dependence on the text encoder, but the gains at higher shot counts are still limited, so I think I will maintain my rating.

Reviewer_CQPWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to introduce class-name priors into self-supervised vision encoders without paired text towers. TAP uses support-set prototypes to orthogonally align independently pretrained text representations, then learns a residual map from image features toward the aligned class names. At inference, queries move partway toward the mapped endpoints, while class names move partway toward their support means. The default method requires no additional validation labels. The results support the value of class names in few-shot learning and under distribution shift, and the benefits of partial mapping for balancing adaptation and generalization.

Critical strengths
  1. The design coherently addresses the information gap in few-shot adaptation. Class names provide inter-class relationships beyond the support images, prototype alignment connects independently pretrained image and text spaces, and partial transport combines task-specific learning with the original representations. No additional validation labels are required, and the overall results at low shot counts and under distribution shift are consistent with the motivation.
  2. The diagnostic experiments provide useful empirical insights. Shuffled-name and random-alignment controls support the importance of correct semantic correspondence and embedding-space alignment, respectively. The two-sided decomposition reveals the distinct roles of queries and class names. Comparisons across training objectives and against the learned endpoint support the importance of how the learned map is applied.
Critical weaknesses

The mechanism behind the loss of generalization when the map is applied fully remains insufficiently understood. The ablations support two-sided partial transport, but the mapping objectives compared in Section 3.3 all use aligned class names as targets or classification anchors. They do not clarify the relative contributions of support-set fitting bias and the effects of text targets on visual discriminative information. This limits our understanding of why partial application helps and when it is appropriate.

Questions for authors
  1. What roles do support-set fitting bias and text-target constraints play when the map is applied fully? Can the existing evidence further distinguish their effects and explain which biases are mitigated by retaining the original query features and moving class names toward their support means?
  2. What inter-class relationships make the class-name prior useful? In 3.4, what does the low-shot advantage of text towers pretrained with images over text-only towers tell us about the respective roles of general linguistic semantics and visually relevant class structure? Which factors cannot yet be disentangled?
2 comments

We thank the reviewer for two questions that go to the core of the method.

Weakness and Q1. Support-set fitting bias and text-target constraints under full transport. At 16 shots, full transport is 5.9 points below the probe under shift. Two new analyses examine this loss.

  • In the features. To see what the map does to the features themselves, we trained the linear probe on the image features after full transport instead of the original ones. It keeps its in-distribution accuracy (75.7 against 75.9) but is 2.2 points lower under shift (App. D.4, Table 11), so the change to the features alone accounts for part of the loss. The other 3.7 points come from scoring the transported features against class names that have not moved.
  • Alignment. To test whether aligning the targets to the images causes the loss, we replaced Procrustes by a fixed random alignment, which keeps the geometry of the class names but places them without regard to the images. With this alignment, full transport reaches 54.9 under shift, against 54.0 with Procrustes (App. D.6, Table 13), so the loss is tied to the targets themselves, not to their placement relative to the images. The alignment matters under partial transport instead, where Procrustes is ahead of the random alignment by 6.9 / 7.1 points at 1 shot (Table 13, validated rows).

What partial transport mitigates. Partial transport changes both parts of this loss: the visual query preserves part of its original features, and each class name moves toward its visual prototype, the mean of its support features, which adapts the text prior to the observed images. At 16 shots each side alone already recovers most of the loss under shift (Table 4). Stopping the query part of the way gives 76.5 / 61.2, against 75.3 / 55.1 with full transport at the same α\alpha. Moving only the aligned class names improves OOD accuracy but lags in distribution (75.0 against 75.9 for the probe). Combining both, TAP is best in distribution and above the probe on both axes (77.0 / 62.4). At 1 shot, where each visual prototype is a single image, nearly all of the gain comes from moving the query, so the schedule moves the class names only 0.24 of the way.

Q2. Which inter-class relationships make the prior useful, and what does the advantage of image-trained text towers tell us?

Inter-class relationships. We count how many of a class's ten nearest classes are the same in the text and visual spaces (mutual 10-NN agreement, Huh et al., 2024, App. D.5). Across six text encoders, the agreement orders the 1-shot ImageNet gains (Fig. 5), also when computed from the 1-shot support alone. The correlation between neighborhood agreement and the benefit from the prior thus suggests that the useful inter-class relationships lie in the local structure of the class space, of which the nearest-neighbor agreement is one measure.

Linguistic versus visual structure. Language alone already captures part of the visual class structure: for the text-only encoders the agreement is far above chance (0.29 to 0.40 against 0.01) and rises from Qwen3-Embedding-0.6B to 8B, together with the 1-shot gain. The text encoders trained with images, which we call visually grounded, capture more (0.41 and 0.47) and gain more than the text-only encoders would at the same agreement (Fig. 5), so training with images seems to add something beyond which classes are neighbors. The advantage is a low-shot one. At 16 shots, where the support supplies most of the visual structure, the in-distribution gains of the larger text-only models and of CLIP text are similar (about +1.0).

What cannot be disentangled. Neither size nor width explains the advantage: mini-GTE is as large as the CLIP-B/16 text tower (66M against 63M parameters) and, like it, no wider than the image features, yet it gains −1.1 points at 1 shot against +8.4. The most likely source is therefore the training against images. The visually grounded text encoders differ from the text-only ones in three ways at once (training text, objective and joint training with an image encoder), so our experiments cannot isolate which of these matters most.

We are happy to answer further questions.

I thank the authors for their detailed supplementary analyses, but the design of the partial transport schedule and movement magnitude remains largely heuristic. I therefore maintain my score.

Reviewer_7rJuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how class-text information can improve few-shot classification with self-supervised visual models. It proposes TAP, which aligns textual and visual prototypes, learns a residual mapping and applies this mapping partially at inference. A support-based rule determines the mapping strength and prior weight.

Critical strengths
  1. The paper is generally well written and easy to follow.

  2. The low-shot improvements are convincing, including under distribution shift. The comparisons use matched visual features and support sets, with results reported over three seeds.

  3. The reviewer appreciates the relevant controls and parameter analyses, including shuffled class names, random alignment, transport strength, training length and model capacity. Adaptation and inference costs are also reported.

Critical weaknesses
  1. The reliability of the support-based rule needs further clarification. After shuffling class names, the alignment cosine decreases only from 0.40 to 0.35, while classification performance drops considerably. This suggests that the score may remain relatively high even when the text prior is unhelpful. The choice of the 0.6 cap in the rule is also not clearly justified or directly tested.

  2. There appears to be an inconsistency between Section 3.4 and Table 12. The text reports gaps from validation-based selection of at most 0.35 points in distribution and 0.8 under shift, whereas the CAPI-L two-shot results differ by 2.42 and 2.02 points. This should be clarified.

  3. The discussion of independent text encoders could be more precise. The strongest gains use CLIP and SigLIP2 text encoders, which were pretrained with images. Purely text-trained encoders provide smaller gains, and mini-GTE falls below the linear probe at one shot. The conclusions should better reflect this difference.

Questions for authors

Please see the weaknesses above.

1 comment

We thank the reviewer for the careful reading. W2 points to an inconsistency in our text, which the revision corrects.

W1a. The alignment cosine stays high with shuffled class names. Correct: ρ\rho falls only from 0.40 to 0.35 (App. D.2). The Procrustes fit finds the best rotation for any pairing of names and classes, so ρ\rho assumes correct names and cannot detect wrong ones, a limitation that Sec. 2.3 states explicitly. Unlike ρ\rho, the mutual 10-NN agreement (Sec. 3.4, App. D.2) needs no alignment, and, computed from the 1-shot support, it falls from 0.21 to 0.01, the chance level, when the class names are shuffled. Because it costs no labels, the agreement can be checked before running TAP.

W1b. The cap of 0.6. The cap was fixed a priori. It must be below 1, since at α=1\alpha = 1 TAP reduces to the Procrustes-only classifier, and we keep it above 1/2 so that α\alpha decreases with KK (Sec. 2.3). Any value in between affects only K=1K = 1, where we varied it (App. D.3, Table 10):

cap cc0.40.50.60.70.80.9
ImageNet54.5455.3555.8856.0255.5954.46
OOD avg47.4748.0848.5448.6648.1447.13

The default of 0.6 is within 0.15 points of the best value, in distribution and under shift. Validated selection, which searches α\alpha from 0 to 1, picks 0.52 on average at 1 shot across the 11 datasets, close to the default (App. D.3).

W2. Section 3.4 and the CAPI-L rows. Thank you for catching this. The bound of 0.35 / 0.8 points holds for DINOv3-L and DINOv2-L only. For CAPI-L, validation adds up to 2.4 / 2.0 points (Table 15, Table 12 in the submission), as the revised Sec. 3.4 reports. The likely cause is that its features share one dominant direction (mean cosine between two images 0.62, against 0.01 for DINOv3-B), which inflates ρ\rho and pushes τ\tau toward full transport. Under the schedule, TAP on CAPI-L still improves on the probe baseline at every shot count, in distribution and under shift.

W3. Visually grounded and text-only encoders. With the two text-only encoders of the submission, the reviewer's reading is correct, and our conclusions did not state the difference. To test whether the smaller gains are a property of text-only encoders or of these two small models, we added Qwen3-Embedding-4B and -8B (Sec. 3.4, Tables 14 and 16, Fig. 5 and the table in the general comment). With one labeled image per class, the gain over the probe grows with model size, +1.0, +2.8 and +3.5 points for 0.6B, 4B and 8B, while visually grounded text encoders still provide a stronger prior (+8.4 CLIP, +10.2 SigLIP2). At 16 shots the two larger models come within 0.1 points of CLIP text in distribution. TAP is therefore beneficial with text-only encoders as well, outperforming the probe in distribution and under shift at both 1 and 16 shots with these two models. Only mini-GTE falls below the probe, at 1 shot under the schedule. The abstract, Sec. 3.4 and the conclusion now state this difference.

We are happy to answer further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of the revision

We thank the reviewers for their careful and constructive feedback. They found the low-shot and distribution-shift gains convincing (Reviewers 7rJu, HHX9) and consistent with the motivation (Reviewer CQPW), and the controls relevant, useful and thorough (Reviewers 7rJu, CQPW, HHX9). TAP itself, including its schedule, is unchanged. The revision adds two larger text-only encoders (Qwen3-Embedding-4B and -8B, on ImageNet), a class-neighborhood analysis, a sweep of the prior weight α\alpha at 1 shot, and two analyses of full transport: a linear probe on the transported image features, and a random alignment in place of Procrustes. The revised PDF marks all changes in blue.

Text encoders (Reviewers 7rJu, CQPW, HHX9). All three reviewers were interested in how much the prior depends on a text encoder trained with images. We added Qwen3-Embedding-4B and -8B to the 0.6B model (Sec. 3.4, Tables 14 and 16) and measured how many of a class's ten nearest classes coincide in the text and visual spaces (mutual 10-NN agreement, Huh et al., 2024, Fig. 5, App. D.5). On ImageNet, we find that the prior from text-only encoders grows with model size, while text encoders trained with images give the largest gains at 1 shot. With the 4B and 8B models, TAP outperforms the validated probe in distribution and under shift at both 1 and 16 shots. Across all six encoders, the 1-shot gain follows their neighborhood agreement (table below).

Gain of TAP under the closed-form schedule (no validation data) over the validated linear probe, ImageNet / OOD average, 3 seeds. The probe reaches 47.45 / 39.98 at 1 shot and 75.91 / 59.90 at 16 shots. † marks text encoders trained with images.

Text encoder10-NN agreement1 shot16 shots
SigLIP2-B/16 †0.465+10.2 / +10.9+1.0 / +3.3
CLIP-B/16 †0.411+8.4 / +8.5+1.0 / +2.5
Qwen3-Embedding-8B (new)0.395+3.5 / +4.0+1.0 / +2.0
Qwen3-Embedding-4B (new)0.381+2.8 / +3.3+1.0 / +2.1
Qwen3-Embedding-0.6B0.324+1.0 / +0.6+0.6 / +1.7
mini-GTE0.294−1.1 / −1.2+0.6 / +1.7

Why the map is applied only part of the way (Reviewer CQPW). Applying the learned map in full (full transport) costs accuracy under shift, and two new analyses help locate where this loss comes from. At 16 shots, full transport falls 5.9 points below the linear probe on the OOD average. The same probe, trained on the resulting image features, keeps its in-distribution accuracy (75.7 against 75.9) but is 2.2 points lower under shift (Sec. 3.3, App. D.4), so part of this loss is already in the features. With a random alignment in place of Procrustes, full transport is 5.0 points below the probe, against 5.9 with the Procrustes alignment (Table 13), so the loss is tied to the targets, not to their alignment. Partial transport avoids this loss by keeping part of the original features and moving the class names toward their visual prototypes. There the alignment does matter, and at 1 shot Procrustes is 6.9 / 7.1 points ahead of the random alignment (Table 13, validated rows).

Clarifications. The abstract and conclusion now state the scope more precisely (low shot and distribution shift), and the Limitations name the two datasets where the probe is stronger (Reviewer HHX9). Sec. 2.3 makes explicit what the alignment cosine assumes (Reviewer 7rJu), and Sec. 3.4 reports the gap between the schedule and validated selection separately for each image encoder (Reviewer 7rJu).

Tables 1-9 keep their numbers, Tables 10-12 and Fig. 5 are new, and submitted Tables 10-20 and Figs. 5-7 are now Tables 13-23 and Figs. 6-8. Table, figure and section numbers in the replies refer to the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.