- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for transferring knowledge of inter-image similarity from large vision-language models (LVLMs) to CLIP. Since LVLMs produce textual outputs rather than embeddings, their outputs cannot be directly compared with CLIP embeddings. To address this issue, the proposed method, Alignment from Solicited Kernels (ASK), converts the image similarity matrix obtained from an LVLM into a positive semidefinite (PSD) kernel and updates CLIP so that the PSD kernel computed from its embeddings approaches the LVLM's kernel. Furthermore, since correcting only inter-image similarities may disrupt image-text alignment, the method introduces an additional regularization term to preserve image-text alignment. The experiments demonstrate that the proposed method improves retrieval and zero-shot classification performance on tasks involving fine-grained image datasets.
Critical strengths
- S1: The idea of incorporating VLM preferences into CLIP is interesting.
Critical weaknesses
- W1: The motivation for transferring knowledge of inter-image similarity from VLMs to CLIP is unclear. It is not clear which limitations of CLIP this knowledge transfer is intended to address, or whether these limitations can only be overcome through knowledge transfer from VLMs. Although the results indeed show improvements in CLIP's performance, what specific limitations of CLIP are addressed to achieve these improvements?
- W2: The positioning of this work relative to existing studies is unclear. The authors should discuss what limitations of prior methods, such as KUEA, Park et al. (2019), and Tung & Mori (2019), are addressed by the proposed method. They should also explain why leveraging VLM preferences is preferable to using a stronger image encoder, such as DINO, which provides better image representations than CLIP.
- W3: The paper lacks sufficient ablation studies and sensitivity analyses of the key components of the proposed method.
- Why was the polynomial kernel chosen? How do the polynomial degree , scale , and offset affect performance? What happens when the polynomial kernel is replaced with an RBF kernel? Providing guidelines for kernel design would improve the practical usefulness of the method.
- It would also be beneficial to analyze the effects of the anchor loss weight and batch size .
- It would be useful to compare the effectiveness of using VLM log-probabilities with that of using textual outputs.
- W4: The experiments are limited to fine-grained datasets. It would be interesting to evaluate the method's performance on more general tasks, such as those involving COCO and ImageNet.
Questions for authors
- Q1: Could the eigenvalue clipping operation in Eq. (3) distort the preferences of the teacher VLM?
6 comments
Response to Reviewer 7Whq (2/3)
W3: Why a polynomial kernel? What about degree p, scale γ, offset c, and RBF? What settings should a user choose?
We thank the reviewer for this question. The polynomial kernel provides a nonlinear PSD training similarity, while deployment still uses cosine retrieval. Our choice of cubic is empirical. Extending Table 6(f) and Appendix E.6, we compared cosine, quadratic, cubic, and RBF similarities while keeping teacher judgments, LoRA, KL anchoring, and update budgets fixed:
| Training similarity | Cosine fidelity ρ | Unseen-breed mAP |
|---|---|---|
| Cosine | 0.756 | 59.16 |
| Normalized quadratic (p = 2) | 0.788 | 60.76 |
| Normalized cubic (p = 3) | 0.794 | 61.18 |
| RBF | 0.770 | 60.82 |
Retrieval uses 1,000 unseen-breed images. Cubic gives the highest fidelity and unseen-breed mAP, although quadratic and RBF are close. Thus, cubic is a useful default.
For γ and c, we fit γ > 0 and c ≥ 0 against teacher targets using frozen training features, then hold them fixed during adaptation and evaluation (Appendix E.6). Diagonal normalization cancels a common positive scaling of γ and c, so the effective parameter is their ratio c/γ. In a new sensitivity experiment, halving or doubling this fitted ratio changes unseen-breed and non-dog mAP by at most 0.24 points, supporting local stability around the calibrated setting.
Our practical recommendation is therefore to start with p = 3, calibrate γ and c on training data, and keep them fixed. Kernel alternatives can be selected on a separate validation set according to the intended retrieval task.
Changes in the revision. Section 2.1 now explains that normalization leaves only the ratio c/γ and that p = 3 is an empirical default fitted on frozen training features. Appendix E.6 adds the kernel-family comparison (Table 23), the c/γ sensitivity result, and this practical recommendation; Section 3.3 summarizes the comparison.
W3: How does the anchor weight λ affect results?
We thank the reviewer for this question. The anchor weight λ controls the balance between learning the teacher’s similarity structure and retaining CLIP’s image–text predictions. Table 20 (Appendix E.4) reports the corresponding sweep.
Increasing λ from 0.01 to our default 0.2 improves mean zero-shot accuracy by 1.81 points, while reducing retrieval by only 0.66 mAP points. The default achieves 60.16 mAP and 62.28% zero-shot accuracy. Increasing λ further to 1.0 gains only another 0.15 accuracy points, while reducing retrieval by 8.28 mAP points. Thus, stronger anchoring increasingly restricts similarity adaptation, and λ = 0.2 provides a practical balance in this setting.
W3: How does the scoring/training group size b affect performance?
We thank the reviewer for this question. We evaluated group sizes b = 16, 32, and 64 on the same 1,000-image training pool. To keep total image exposure fixed, we adjusted the number of updates so that each setting processes 96,000 image presentations (group size × updates). Evaluation uses the same 1,000-image unseen-breed test set.
| Group size b | Distinct teacher judgments | Training updates | Unseen-breed mAP |
|---|---|---|---|
| 16 | 21,710 | 6,000 | 58.71 |
| 32 (default) | 44,450 | 3,000 | 59.52 |
| 64 | 85,077 | 1,500 | 59.53 |
The b = 32 entry is the mean over three training seeds. Reducing b from 32 to 16 roughly halves the required teacher judgments, with a 0.81-point mAP decrease. Increasing b to 64 nearly doubles the judgments without a meaningful retrieval gain. These results support b = 32 as a practical default, while b = 16 offers a lower-query alternative. This comparison holds total image exposure fixed; query budgets, update counts, and group composition vary with group size. Appendix D.10 (Table 17) also explores larger groups under sparse pair supervision.
Changes in the revision. The group-size comparison is reported in Table 6(i), summarized in Section 3.3, and detailed in Appendix E.7.
Response to Reviewer 7Whq (3/3)
W3: Do log-probabilities help compared with the VLM’s textual output?
We thank the reviewer for this question. Appendix E.3 (Figure 7) directly compares probability-weighted digit expectations with argmax-digit verdicts using the same cached Qwen responses and training settings. Expected-digit scores improve fidelity from 0.760 to 0.794. Fidelity uses common held-out expected-digit judgments. Thus, retaining the probability distribution across rating levels can improve fidelity to the teacher’s graded judgments. Please note that log-probabilities remain optional: ASK also supports textual verdicts when they are unavailable.
W4 Can ASK generalize beyond fine-grained datasets, including COCO and ImageNet?
We thank the reviewer for this question. The paper evaluates transfer to both COCO-Stuff and ImageNet. Figure 5(b) (Appendix D.5) reports background retrieval on 3,701 non-dog COCO-Stuff images, where the dog-trained background adapter achieves 36.05 nDCG@10, compared with 29.30 for CLIP, 28.63 for DINOv2+KL, and 35.00 for VLM2Vec, without COCO adaptation. Table 8 reports non-dog ImageNet retrieval, and Table 12 includes ImageNet zero-shot classification.
Changes in the revision. Table 5(a) adds non-dog retrieval on 5,000 held-out ImageNet images, where ASK reaches 74.26 mAP versus 73.13 for DINOv2+KL (Section 3.2, Appendix D.2).
Q1 Could eigenvalue clipping distort the teacher’s preferences?
We thank the reviewer for this question. Eigenvalue clipping can change individual preferences. It produces the nearest PSD matrix in Frobenius norm before diagonal normalization, but does not guarantee preservation of rankings. Its purpose is to make potentially indefinite teacher judgments compatible with the student’s PSD kernel.
We quantified these changes after clipping and normalization on 31 scored groups:
| Teacher | Raw–corrected Spearman ρ | Ranking reversals | Reversals for score gaps ≥ 1 |
|---|---|---|---|
| Qwen | 0.999 | 0.97% | 0.000% |
| GPT | 0.972 | 3.75% | 0.015% |
| Gemma | 0.959 | 8.36% | 0.060% |
Reversals compare candidate rankings for a fixed query; score gaps refer to the original 1–9 scale. The changes predominantly affect close preferences, with very few reversals among clearly separated scores. Table 6(a) and Appendix E.2 (Table 19) also show that correction can improve learning. The benefit varies with the teacher and criterion.
Changes in the revision. Section 2.3 now notes that clipping can reverse individual pairwise preferences, mostly between nearly tied scores. The analysis above is reported in Table 6(g), summarized in Section 3.3, and detailed in Appendix E.2.
Response to Reviewer 7Whq (1/3)
We are genuinely grateful for your detailed assessment and valuable insights. Your constructive feedback has significantly contributed to the refinement and advancement of our study. Please note that all table, figure, and section numbers refer to the revised paper (changes in blue). The original Table 5 (ablations) is now Table 6, appendix Tables 6–21 are now Tables 7–22, and Table 22 is now Table 24.
We thank the reviewer for this question. We address two limitations of CLIP’s default similarity: it can underrepresent fine-grained relationships, such as those among breeds or car models, and its fixed ranking does not directly accommodate different comparison criteria, such as colour, viewpoint, pose, or background. ASK transfers both a VLM’s fine-grained knowledge and its criterion-dependent judgments into CLIP. Table 1 shows unseen-breed retrieval improving from 22.1 to 60.2 mAP, while Table 4(b) and the human-annotated evaluations in Appendix D.5 demonstrate that changing the requested criterion changes the learned similarity appropriately.
VLM transfer is not the only solution: label supervision, feature distillation, and conditional-similarity training are valid alternatives. ASK provides a way to obtain both richer similarity supervision and language-based criterion control without collecting new human similarity annotations. Because it requires only pairwise judgments, hosted teachers such as GPT can be used without accessing their internal embeddings.
Changes in the revision. The first paragraph of the Introduction now states both limitations explicitly, using fine-grained identity (dog breed, car model) and chosen properties (colour, viewpoint, background) as examples.
We thank the reviewer for asking us to clarify these distinctions. KUEA matches teacher feature kernels, Park et al. transfer distances and angles between teacher representations, and Tung & Mori preserve similarities between teacher activations. ASK instead transfers fine-grained and criterion-dependent judgments elicited from a VLM, without accessing teacher embeddings. We obtain pairwise scores using probability-weighted digit expectations when available. Unlike feature-derived kernels, these independently elicited scores need not be PSD, so ASK applies symmetrization, rescaling, spectral PSD projection, and diagonal normalization within each group (Secs. 2.2–2.3).
A second substantive difference is joint visual and text adaptation. KUEA freezes the text encoder and penalizes visual-feature drift through an L2 anchor. ASK regularizes image–text prediction distributions with KL while adapting both encoders. This allows both representations to adjust to the requested visual geometry while remaining anchored to frozen CLIP’s predictions. Under the same KL objective, adapting the text encoder increases retrieval from 49.71 to 60.16 mAP, a 10.45-point gain (Table 6(e)); Appendix E.4 provides further regularizer comparisons.
Strong DINO features are a valuable supervision source, but their fixed feature kernel does not directly express which relationship the user wants to emphasize. VLM judgments provide that control through language, while also improving category-based retrieval. Our DINOv2+KL control retains the same LoRA configuration and KL objective: ASK achieves 75.0 versus 71.7 mean retrieval mAP across five domains (Table 1). Table 4(b) and Appendix D.5 additionally demonstrate criterion-specific control on fixed images and against human annotations.
Changes in the revision. The Introduction now contrasts ASK with KUEA (citing Table 6(e)), and notes that a feature teacher such as DINOv2 provides one fixed set of relationships, whereas the prompt selects which relationship to learn. The KUEA paragraph of the Related Work (Appendix B) adds both empirical controls.