- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TAP, a few-shot adaptation method for incorporating class-name semantics from an independently pretrained text encoder into self-supervised vision encoders such as DINOv3. TAP first aligns text prototypes to the visual space using Orthogonal Procrustes, then learns a residual map from support images toward the aligned class names. Instead of applying this map fully, TAP uses partial transport at inference, which improves robustness, especially in low-shot and distribution-shift settings. Experiments across 11 datasets and ImageNet shifts show consistent gains over strong frozen-feature baselines, with extensive ablations supporting the partial-transport design.
Critical strengths
- The paper addresses a meaningful problem: leveraging class-name semantics for strong self-supervised vision encoders that otherwise cannot directly use text information.
- The empirical gains are convincing, particularly in the low-shot and distribution-shift settings, where the proposed text prior is most useful.
- The experimental analysis is thorough, with targeted ablations on partial vs. full transport, different objectives, incorrect class names, random alignment, and multiple vision/text encoders.
Critical weaknesses
- The benefit depends strongly on the choice of text encoder: CLIP/SigLIP2 provide large gains, while text-only embedding models provide much smaller or even negative gains, which makes the generality of the “class-name prior” claim less clear.
- The gains diminish substantially as the number of shots increases and become negative on some datasets, suggesting that the method is mainly effective in very-low-shot and OOD regimes.
Questions for authors
- To what extent does TAP rely on the text encoder having been trained with visual supervision, rather than on class-name semantics alone?
- How should we interpret the scope of TAP given that its gains become very small, and sometimes negative at higher shot counts?
2 comments
Thank you for your response. The additional experiments addressed my concerns about the dependence on the text encoder, but the gains at higher shot counts are still limited, so I think I will maintain my rating.
We thank the reviewer for pressing on generality and scope. Both points changed how the abstract and the conclusion state the result.
Q1 / W1. How much does TAP rely on a text encoder trained with visual supervision?
TAP relies on the class-name semantics as a prior, and a text encoder trained with images strengthens it when few labels are available. Shuffling the class names removes the gain (Table 9), so the semantics are necessary. Semantics from text alone, without visual supervision, already give a useful prior. With Qwen3-Embedding-4B and -8B, which we added to test this (Sec. 3.4, Tables 14 and 16, Fig. 5, all on ImageNet), TAP outperforms the validated probe in distribution and under shift at 1 and 16 shots, and the prior grows with model size. At 1 shot, text encoders trained with images still give two to three times the gain of the largest text-only model. This extra gain is not explained by model size or embedding width (see our reply to Reviewer CQPW, Q2), and only partly by the higher neighborhood agreement of these encoders (0.41 to 0.47, against 0.38 to 0.40 for Qwen3-Embedding-4B and -8B). Its most likely source is the training against images, though our experiments cannot isolate which part of it matters most.
Q2 / W2. How should the scope of TAP be read, given that its gains become small and sometimes negative at higher shot counts?
TAP is most useful with few labels and under distribution shift. Frozen self-supervised features such as DINOv3's are well suited to linear classification, so with 16 labeled images per class a linear probe already finds a strong in-distribution boundary. That boundary is fitted to the source distribution, while the class names carry a prior about each class that is independent of it and separates the classes better under shift (Sec. 3.2). At 16 shots, TAP matches or improves on the probe on every dataset except DTD and FGVCAircraft (Table 23), and averaging TAP with the validated cross-modal probe (Sec. 2.3) recovers most of the gap on these two. Accordingly, we removed "at every shot count" from the abstract's claim over the three benchmarks, and the conclusion and Limitations now state this scope and name both exceptions.
We are happy to answer further questions.