acceptodds
Under review as a conference paper at ICLR 2027

Kernel-Based Steering of CLIP with Vision-Language Model Preferences

Abstract

Large vision-language models (VLMs) can judge visual similarity, but their judgments are not directly available as compact image embeddings for efficient comparison. We study how to transfer these preferences into CLIP while retaining its image–text capabilities. We introduce ASK, a kernel-based steering method that learns from elicited pairwise judgments without accessing teacher embeddings or collecting new human similarity annotations. ASK constructs positive semidefinite target kernels within small image groups and combines visual kernel matching with an image–text distributional anchor. Low-rank adapters jointly update the visual and text encoders while regularizing predictions toward frozen CLIP. After adaptation, retrieval uses CLIP image embeddings and cosine similarity, with no VLM calls. Experiments across five image domains, four CLIP backbones, and six judges evaluate teacher agreement, retrieval, and recognition retention. For ViT-B/16, mean retrieval mAP on classes excluded from adaptation increases from 53.8 to 75.0, compared with 71.7 for DINOv2 targets with KL anchoring. Mean zero-shot accuracy with jointly adapted encoders increases from 61.8% to 62.4%, averaged over 12 benchmarks and the five adaptation domains. Prompting provides an additional capability: selecting which visual distinctions the student learns. Human-annotated evaluations across four datasets support this criterion-specific control.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-language models, CLIP adaptation, kernel alignment, image retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_7WhqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for transferring knowledge of inter-image similarity from large vision-language models (LVLMs) to CLIP. Since LVLMs produce textual outputs rather than embeddings, their outputs cannot be directly compared with CLIP embeddings. To address this issue, the proposed method, Alignment from Solicited Kernels (ASK), converts the image similarity matrix obtained from an LVLM into a positive semidefinite (PSD) kernel and updates CLIP so that the PSD kernel computed from its embeddings approaches the LVLM's kernel. Furthermore, since correcting only inter-image similarities may disrupt image-text alignment, the method introduces an additional regularization term to preserve image-text alignment. The experiments demonstrate that the proposed method improves retrieval and zero-shot classification performance on tasks involving fine-grained image datasets.

Critical strengths
  • S1: The idea of incorporating VLM preferences into CLIP is interesting.
Critical weaknesses
  • W1: The motivation for transferring knowledge of inter-image similarity from VLMs to CLIP is unclear. It is not clear which limitations of CLIP this knowledge transfer is intended to address, or whether these limitations can only be overcome through knowledge transfer from VLMs. Although the results indeed show improvements in CLIP's performance, what specific limitations of CLIP are addressed to achieve these improvements?
  • W2: The positioning of this work relative to existing studies is unclear. The authors should discuss what limitations of prior methods, such as KUEA, Park et al. (2019), and Tung & Mori (2019), are addressed by the proposed method. They should also explain why leveraging VLM preferences is preferable to using a stronger image encoder, such as DINO, which provides better image representations than CLIP.
  • W3: The paper lacks sufficient ablation studies and sensitivity analyses of the key components of the proposed method.
    • Why was the polynomial kernel chosen? How do the polynomial degree pp, scale γ\gamma, and offset cc affect performance? What happens when the polynomial kernel is replaced with an RBF kernel? Providing guidelines for kernel design would improve the practical usefulness of the method.
    • It would also be beneficial to analyze the effects of the anchor loss weight λ\lambda and batch size bb.
    • It would be useful to compare the effectiveness of using VLM log-probabilities with that of using textual outputs.
  • W4: The experiments are limited to fine-grained datasets. It would be interesting to evaluate the method's performance on more general tasks, such as those involving COCO and ImageNet.
Questions for authors
  • Q1: Could the eigenvalue clipping operation in Eq. (3) distort the preferences of the teacher VLM?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 7Whq (1/3)

We are genuinely grateful for your detailed assessment and valuable insights. Your constructive feedback has significantly contributed to the refinement and advancement of our study. Please note that all table, figure, and section numbers refer to the revised paper (changes in blue). The original Table 5 (ablations) is now Table 6, appendix Tables 6–21 are now Tables 7–22, and Table 22 is now Table 24.

W1 What specific limitation of CLIP is addressed? Can only VLM knowledge transfer overcome it?

We thank the reviewer for this question. We address two limitations of CLIP’s default similarity: it can underrepresent fine-grained relationships, such as those among breeds or car models, and its fixed ranking does not directly accommodate different comparison criteria, such as colour, viewpoint, pose, or background. ASK transfers both a VLM’s fine-grained knowledge and its criterion-dependent judgments into CLIP. Table 1 shows unseen-breed retrieval improving from 22.1 to 60.2 mAP, while Table 4(b) and the human-annotated evaluations in Appendix D.5 demonstrate that changing the requested criterion changes the learned similarity appropriately.

VLM transfer is not the only solution: label supervision, feature distillation, and conditional-similarity training are valid alternatives. ASK provides a way to obtain both richer similarity supervision and language-based criterion control without collecting new human similarity annotations. Because it requires only pairwise judgments, hosted teachers such as GPT can be used without accessing their internal embeddings.

Changes in the revision. The first paragraph of the Introduction now states both limitations explicitly, using fine-grained identity (dog breed, car model) and chosen properties (colour, viewpoint, background) as examples.


W2 How does ASK differ from KUEA, Park et al., and Tung & Mori? Why use VLM preferences instead of stronger DINO features?

We thank the reviewer for asking us to clarify these distinctions. KUEA matches teacher feature kernels, Park et al. transfer distances and angles between teacher representations, and Tung & Mori preserve similarities between teacher activations. ASK instead transfers fine-grained and criterion-dependent judgments elicited from a VLM, without accessing teacher embeddings. We obtain pairwise scores using probability-weighted digit expectations when available. Unlike feature-derived kernels, these independently elicited scores need not be PSD, so ASK applies symmetrization, rescaling, spectral PSD projection, and diagonal normalization within each group (Secs. 2.2–2.3).

A second substantive difference is joint visual and text adaptation. KUEA freezes the text encoder and penalizes visual-feature drift through an L2 anchor. ASK regularizes image–text prediction distributions with KL while adapting both encoders. This allows both representations to adjust to the requested visual geometry while remaining anchored to frozen CLIP’s predictions. Under the same KL objective, adapting the text encoder increases retrieval from 49.71 to 60.16 mAP, a 10.45-point gain (Table 6(e)); Appendix E.4 provides further regularizer comparisons.

Strong DINO features are a valuable supervision source, but their fixed feature kernel does not directly express which relationship the user wants to emphasize. VLM judgments provide that control through language, while also improving category-based retrieval. Our DINOv2+KL control retains the same LoRA configuration and KL objective: ASK achieves 75.0 versus 71.7 mean retrieval mAP across five domains (Table 1). Table 4(b) and Appendix D.5 additionally demonstrate criterion-specific control on fixed images and against human annotations.

Changes in the revision. The Introduction now contrasts ASK with KUEA (citing Table 6(e)), and notes that a feature teacher such as DINOv2 provides one fixed set of relationships, whereas the prompt selects which relationship to learn. The KUEA paragraph of the Related Work (Appendix B) adds both empirical controls.

AuthorsOpenReview ↗

Response to Reviewer 7Whq (2/3)

W3: Why a polynomial kernel? What about degree p, scale γ, offset c, and RBF? What settings should a user choose?

We thank the reviewer for this question. The polynomial kernel provides a nonlinear PSD training similarity, while deployment still uses cosine retrieval. Our choice of cubic is empirical. Extending Table 6(f) and Appendix E.6, we compared cosine, quadratic, cubic, and RBF similarities while keeping teacher judgments, LoRA, KL anchoring, and update budgets fixed:

Training similarityCosine fidelity ρUnseen-breed mAP
Cosine0.75659.16
Normalized quadratic (p = 2)0.78860.76
Normalized cubic (p = 3)0.79461.18
RBF0.77060.82

Retrieval uses 1,000 unseen-breed images. Cubic gives the highest fidelity and unseen-breed mAP, although quadratic and RBF are close. Thus, cubic is a useful default.

For γ and c, we fit γ > 0 and c ≥ 0 against teacher targets using frozen training features, then hold them fixed during adaptation and evaluation (Appendix E.6). Diagonal normalization cancels a common positive scaling of γ and c, so the effective parameter is their ratio c/γ. In a new sensitivity experiment, halving or doubling this fitted ratio changes unseen-breed and non-dog mAP by at most 0.24 points, supporting local stability around the calibrated setting.

Our practical recommendation is therefore to start with p = 3, calibrate γ and c on training data, and keep them fixed. Kernel alternatives can be selected on a separate validation set according to the intended retrieval task.

Changes in the revision. Section 2.1 now explains that normalization leaves only the ratio c/γ and that p = 3 is an empirical default fitted on frozen training features. Appendix E.6 adds the kernel-family comparison (Table 23), the c/γ sensitivity result, and this practical recommendation; Section 3.3 summarizes the comparison.


W3: How does the anchor weight λ affect results?

We thank the reviewer for this question. The anchor weight λ controls the balance between learning the teacher’s similarity structure and retaining CLIP’s image–text predictions. Table 20 (Appendix E.4) reports the corresponding sweep.

Increasing λ from 0.01 to our default 0.2 improves mean zero-shot accuracy by 1.81 points, while reducing retrieval by only 0.66 mAP points. The default achieves 60.16 mAP and 62.28% zero-shot accuracy. Increasing λ further to 1.0 gains only another 0.15 accuracy points, while reducing retrieval by 8.28 mAP points. Thus, stronger anchoring increasingly restricts similarity adaptation, and λ = 0.2 provides a practical balance in this setting.


W3: How does the scoring/training group size b affect performance?

We thank the reviewer for this question. We evaluated group sizes b = 16, 32, and 64 on the same 1,000-image training pool. To keep total image exposure fixed, we adjusted the number of updates so that each setting processes 96,000 image presentations (group size × updates). Evaluation uses the same 1,000-image unseen-breed test set.

Group size bDistinct teacher judgmentsTraining updatesUnseen-breed mAP
1621,7106,00058.71
32 (default)44,4503,00059.52
6485,0771,50059.53

The b = 32 entry is the mean over three training seeds. Reducing b from 32 to 16 roughly halves the required teacher judgments, with a 0.81-point mAP decrease. Increasing b to 64 nearly doubles the judgments without a meaningful retrieval gain. These results support b = 32 as a practical default, while b = 16 offers a lower-query alternative. This comparison holds total image exposure fixed; query budgets, update counts, and group composition vary with group size. Appendix D.10 (Table 17) also explores larger groups under sparse pair supervision.

Changes in the revision. The group-size comparison is reported in Table 6(i), summarized in Section 3.3, and detailed in Appendix E.7.

AuthorsOpenReview ↗

Response to Reviewer 7Whq (3/3)

W3: Do log-probabilities help compared with the VLM’s textual output?

We thank the reviewer for this question. Appendix E.3 (Figure 7) directly compares probability-weighted digit expectations with argmax-digit verdicts using the same cached Qwen responses and training settings. Expected-digit scores improve fidelity from 0.760 to 0.794. Fidelity uses common held-out expected-digit judgments. Thus, retaining the probability distribution across rating levels can improve fidelity to the teacher’s graded judgments. Please note that log-probabilities remain optional: ASK also supports textual verdicts when they are unavailable.


W4 Can ASK generalize beyond fine-grained datasets, including COCO and ImageNet?

We thank the reviewer for this question. The paper evaluates transfer to both COCO-Stuff and ImageNet. Figure 5(b) (Appendix D.5) reports background retrieval on 3,701 non-dog COCO-Stuff images, where the dog-trained background adapter achieves 36.05 nDCG@10, compared with 29.30 for CLIP, 28.63 for DINOv2+KL, and 35.00 for VLM2Vec, without COCO adaptation. Table 8 reports non-dog ImageNet retrieval, and Table 12 includes ImageNet zero-shot classification.

Changes in the revision. Table 5(a) adds non-dog retrieval on 5,000 held-out ImageNet images, where ASK reaches 74.26 mAP versus 73.13 for DINOv2+KL (Section 3.2, Appendix D.2).


Q1 Could eigenvalue clipping distort the teacher’s preferences?

We thank the reviewer for this question. Eigenvalue clipping can change individual preferences. It produces the nearest PSD matrix in Frobenius norm before diagonal normalization, but does not guarantee preservation of rankings. Its purpose is to make potentially indefinite teacher judgments compatible with the student’s PSD kernel.

We quantified these changes after clipping and normalization on 31 scored groups:

TeacherRaw–corrected Spearman ρRanking reversalsReversals for score gaps ≥ 1
Qwen0.9990.97%0.000%
GPT0.9723.75%0.015%
Gemma0.9598.36%0.060%

Reversals compare candidate rankings for a fixed query; score gaps refer to the original 1–9 scale. The changes predominantly affect close preferences, with very few reversals among clearly separated scores. Table 6(a) and Appendix E.2 (Table 19) also show that correction can improve learning. The benefit varies with the teacher and criterion.

Changes in the revision. Section 2.3 now notes that clipping can reverse individual pairwise preferences, mostly between nearly tied scores. The analysis above is reported in Table 6(g), summarized in Section 3.3, and detailed in Appendix E.2.

Reviewer_X26nOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ASK, a framework for transferring VLM-generated pairwise similarity judgments into CLIP through kernel alignment. The method constructs positive semidefinite (PSD) target kernels from VLM scores and jointly adapts CLIP's image and text encoders using LoRA, with an image-text KL regularizer to preserve zero-shot recognition capabilities.

Critical strengths

The paper is generally well organized and provides extensive evaluations across multiple domains, CLIP backbones, and VLM judges. The improvements in retrieval performance are encouraging.

Critical weaknesses
  1. Limited significance of the proposed problem setting.

Conditional image similarity has been extensively studied in prior work, including Conditional Similarity Networks, GeneCIS, and MagicLens, which offer more flexible approaches to handling different similarity criteria. In contrast, ASK requires separate VLM supervision and offline adaptation for each criterion. The paper does not sufficiently justify why this more restrictive formulation represents a significant research problem or offers compelling advantages over existing approaches.

  1. Insufficient methodological novelty.

The proposed framework largely combines existing techniques, including knowledge distillation, kernel alignment, and regularization. In particular, the overall approach closely follows KUEA, with the main differences lying in the source of supervision and the strategy for preserving CLIP's pretrained capabilities. Using VLM-generated similarity judgments as supervision is a straightforward application of knowledge distillation, while the remaining components are largely adopted from established methods. The paper does not sufficiently demonstrate how their combination constitutes a substantive methodological advance beyond applying existing techniques to a different supervision source.

  1. Limited substantive insight and practical value of the proposed solution.

The paper aims to enhance CLIP's ability to handle diverse visual similarity criteria by leveraging the general visual understanding capabilities of VLMs. However, the proposed solution simply distills each criterion into a separately trained LoRA adapter. This straightforward task-specific distillation strategy offers limited new insight beyond existing knowledge distillation techniques. Furthermore, the paper does not convincingly demonstrate any distinctive practical advantage that would justify the repeated VLM supervision and adaptation required for each new criterion.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer X26n (1/2)

We deeply appreciate your thorough review of our manuscript and are grateful for the insightful feedback you provided. Your constructive comments have been instrumental in enhancing the quality of our work. Please note that all table, figure, and section numbers refer to the revised paper (changes in blue). The original Table 5 (ablations) is now Table 6, appendix Tables 6–21 are now Tables 7–22, and Table 22 is now Table 24.

W1 CSN, GeneCIS, and MagicLens already study conditional similarity and are more flexible. Why is per-criterion adaptation a significant setting?

We thank the reviewer for this comment. We agree that instruction-conditioned models are more flexible when the criterion changes from query to query. ASK targets a complementary setting: a fixed criterion chosen by the application, such as car model for deduplication or viewpoint for dataset curation, reused across an entire image collection. In this setting, per-criterion adaptation offers three advantages.

Higher accuracy. We added GeneCIS (official CC3M-trained ViT-B/16) to our human-annotated evaluations. It receives the same validation-based instruction and gallery-format selection as MagicLens (Figure 3(c), Appendix D.5). Scores are nDCG@10 for CUB-200-2011 attributes, and for car viewpoint (PASCAL3D+), background (COCO-Stuff), and animal pose (AP-10K) without adaptation on these datasets:

MethodWing colourBill shapeBreast patternViewpointBackgroundPose
CLIP34.5549.9852.3134.7729.3052.29
GeneCIS33.9550.0151.8935.7829.9953.30
MagicLens33.2346.7051.1636.5730.8851.82
ASK38.6452.0155.6647.7336.0555.83

GeneCIS and MagicLens remain within −3.3 to +1.8 points of CLIP, whereas ASK improves by 2.0 to 13.0 points and leads on all six criteria. Results on 100 CUB species excluded from adaptation follow the same pattern (Figure 4).

Criterion-specific gallery geometry. GeneCIS and MagicLens are designed to condition the query, while each gallery image keeps a single embedding. Under this native query-only use, accuracy drops sharply: MagicLens reaches 17.58 wing-colour nDCG@10 on CUB (Table 11(b)), and GeneCIS falls from 30.87 to 20.59 on held-out species (Appendix D.5). ASK builds the criterion into the image encoder, so every image–image relationship follows it. This also supports clustering, deduplication, and k-NN over cached embeddings, without an instruction at query time.

Lightweight, prompt-specified supervision. CSN learns from about 200k labeled triplets per notion, GeneCIS from 1.6M caption-mined triplets, and MagicLens from 36.7M web triplets. ASK requires only a comparison prompt and about 44k VLM judgments per criterion (0.61 H100-hours with Qwen3.8-27B). This allows it to target fine-grained criteria, such as bill shape or body pose, that generic conditional training data may cover only sparsely.

Finally, as detailed in our response to W3, separate adapters also outperform a CSN-style shared model with learned criterion masks trained on the same judgments (Table 6(h)). We therefore view per-criterion adaptation as complementary to instruction-conditioned models: it trades query-time flexibility for higher accuracy on a fixed criterion.

Changes in the revision. The Introduction now contrasts selecting the criterion through the teacher prompt with conditioning similarity at query time, and the Related Work (Appendix B) discusses this trade-off. GeneCIS is added as a baseline in Figure 3(c), Figures 4–5, and Table 11(a). Its selection protocol and native query-only results are described in Appendix D.5.

AuthorsOpenReview ↗

Response to Reviewer X26n (2/2)

W2 Is ASK merely KUEA with VLM supervision and a different regularizer?

We thank the reviewer for this question. ASK shares kernel matching with KUEA, but changes both the supervision source and how cross-modal compatibility is maintained.

KUEA transfers relationships encoded in a vision teacher’s features. ASK instead elicits prompt-selectable similarity scores without teacher-embedding access, allowing the same teacher and images to express different relationships, such as breed, colour, or viewpoint. Unlike feature Gram matrices, these judgments can be indefinite, motivating groupwise PSD projection and diagonal normalization. Table 4(b) and Appendix D.5 demonstrate that the resulting encoders learn the requested distinctions.

The regularizer change also enables joint visual–text adaptation: KUEA freezes the text encoder and anchors visual features with L2, whereas ASK anchors image–text prediction distributions while adapting both encoders. Under the same KL objective, adapting the text encoder raises retrieval from 49.71 to 60.16 mAP (Table 6(e)).

Finally, DINOv2+KL controls for the LoRA configuration and KL objective, yet ASK achieves 75.0 versus 71.7 mean retrieval mAP across five domains (Table 1). These controls support the value of both judgment supervision and joint adaptation; we do not claim kernel matching or PSD projection themselves as novel.

Changes in the revision. The Introduction now contrasts ASK with KUEA directly, citing the joint-adaptation gain (Table 6(e)). The KUEA paragraph of the Related Work (Appendix B) adds the prompt-selectable target and both empirical controls.


W3 What practical advantage or insight justifies scoring and training a separate adapter for each criterion?

We thank the reviewer for this comment. Separate adapters target applications where a fixed similarity criterion is reused across many retrieval queries. Teacher judgments are collected once, and the adapted encoder then supports ordinary cosine retrieval without further VLM calls. Changing criteria requires selecting another adapter.

This one-time cost per criterion is modest. At the default budget, a new criterion requires about 44k judgments, which take 0.611 H100-hours to generate locally with Qwen3.8-27B or an estimated $3.4–7.7 through an API, followed by 3,000 adapter updates (about 1.3 GPU-hours on one RTX A6000) (Table 5(b), Appendix D.9, Table 16). Under a matched end-to-end budget that includes this judgment generation, ASK still exceeds DINOv2+KL by about 8 unseen-breed mAP points (Table 5(b)). As shown in our response to W1, the resulting adapters also outperform instruction-conditioned models on fixed criteria.

To test whether separate adaptation offers an accuracy benefit, we compared three ASK adapters with one shared CLIP model using three learned criterion masks. Both use the same cached criterion-specific judgments, objectives, and 9,000 aggregate training updates (3,000 per criterion). On 100 held-out CUB species:

CriterionShared modelSeparate ASK
Wing colour33.9035.71
Bill shape48.4651.37
Breast pattern52.7552.80

Scores are nDCG@10, evaluated using human attributes. Separate adapters improve wing-colour and bill-shape retrieval by 1.81 and 2.91 points, while breast-pattern performance is comparable.

The practical insight is that dedicating adaptation capacity to each criterion can improve retrieval accuracy. This comes with additional adapter storage; the shared model uses fewer adapter parameters. Both approaches use the same teacher judgments in this comparison, so separate adapters do not incur additional scoring costs relative to this shared control. We therefore view separate adaptation as a useful accuracy–storage trade-off, rather than a requirement for conditional similarity.

Changes in the revision. The comparison is reported in Table 6(h), summarized in Section 3.3, and detailed in Appendix D.5. The Introduction and Related Work (Appendix B) now describe the fixed-criterion setting and the accuracy–storage trade-off.

I appreciate the authors' efforts to address my concerns.

However, I still find the proposed method incremental, as it primarily combines existing techniques without offering substantial methodological novelty or new insights in the CLIP adaptation domain.

Therefore, my main concerns remain unresolved, and I will maintain my original rating.

Reviewer_WF3xOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ASK (Alignment from Solicited Kernels), a framework for transferring visual similarity preferences from large Vision-Language Models (VLMs) into CLIP's embedding space without requiring access to teacher embeddings or additional human annotations. The method queries a VLM to obtain pairwise image similarity scores, converts these judgments into positive semidefinite (PSD) kernels, and trains CLIP to reproduce the resulting similarity structure. The authors use LoRA to adapt both image and text encoders while introducing a KL-divergence regularizer to preserve CLIP's original image-text alignment. Experiments across five image domains, four CLIP backbones, and six VLM judges demonstrate improvements in image retrieval and agreement with teacher preferences. Specifically, ASK improves average unseen-class retrieval mAP by around 13 points, compared with 9 points for DINOv2+KL, while approximately preserving zero-shot classification accuracy.

Critical strengths

The paper has the following strengths -

  1. Experiments were considerably thorough.
  2. The problem the authors are solving is relevant to CLIP community.
Critical weaknesses

The paper has the following 2 weaknesses -

  1. Insufficient analysis of computational efficiency and scalability - I find that the paper does not sufficiently establish the computational practicality of the proposed framework. In Section 2.2 and Algorithm 1, ASK requires querying a large VLM to obtain pairwise similarity judgments, followed by kernel construction and LoRA fine-tuning. Although the authors emphasize efficient inference without VLM calls, I do not see a comprehensive comparison of the total offline computational cost against existing methods. In particular, generating pairwise judgments using large VLMs may introduce substantial latency and inference costs as the dataset grows.

I would like to see comparisons of total GPU-hours, VLM query costs, and performance under matched computational budgets, particularly against DINOv2+KL and other feature-based distillation baselines.

  1. Limited methodological novelty and unclear contribution of kernel-based alignment - I find the methodological novelty somewhat limited, as the proposed framework primarily combines established techniques, including relational distillation, PSD projection, kernel alignment, LoRA, and KL-based regularization. In Sections 2.3–2.5, these components closely resemble existing approaches, particularly KUEA's kernel alignment [ref 1] and conventional knowledge distillation [ref 2, 3]. In my opinion, the idea is very similar to [ref 1]. Furthermore, Table 5(f) in Section 3.3 shows that replacing the normalized cubic kernel with cosine similarity reduces retrieval mAP by only 1.81 points (60.16 versus 58.35), suggesting that the specific kernel formulation contributes relatively modest improvements.

I believe the authors should more clearly establish what fundamentally distinguishes ASK from existing relational distillation methods and whether comparable performance can be achieved using simpler pairwise similarity regression objectives.

References -

  1. Gong et. al., Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models. ICML 2025
  2. Park et. al., Relational Knowledge Distillation. CVPR 2019
  3. Tung et. al., Similarity-Preserving Knowledge Distillation, ICCV 2019
Questions for authors

See major weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer WF3x (1/2)

We sincerely thank you for your careful reading of our paper and appreciate the valuable feedback in your comments. The insightful and constructive suggestions have enabled us to effectively improve our work. Please note that all table, figure, and section numbers refer to the revised paper (changes in blue). The original Table 5 (ablations) is now Table 6, appendix Tables 6–21 are now Tables 7–22, and Table 22 is now Table 24.

W1. Compare total GPU-hours, VLM query costs, and performance under matched computational budgets, particularly against DINOv2+KL and other feature-based distillation baselines.

We thank the reviewer for requesting a clearer comparison of cost and performance. Appendix D.9 reports student training times and teacher API-cost estimates (Table 16). For ViT-B/16, 3,000 adaptation steps take a median of approximately 1.3 GPU-hours on one RTX A6000, excluding teacher acquisition. At the default budget of 44,450 distinct training-pair judgments, estimated API costs are $5.4–12.3 for Qwen3.5-397B and $3.4–7.7 for Qwen3.8-27B under the stated pricing assumptions.

To compare performance while accounting for supervision acquisition, we additionally evaluated ASK and DINOv2+KL under 90- and 120-minute end-to-end time budgets. We account for teacher acquisition and target preparation first, then allocate the remaining time to student training. Both methods are evaluated on the same 1,000-image unseen-breed test set.

Budget (min)MethodVLM callsAcquisition (min, estimated)Preparation (min, measured)Student GPU-hoursmAP
90DINOv2 + KL000.581.4952.93
90ASK (Qwen 27B)44,45056.300.250.5659.97
120DINOv2 + KL000.581.9953.13
120ASK (Qwen 27B)44,45056.300.251.0660.04

Preparation includes frozen-feature extraction and target/link construction. After accounting for acquisition time, ASK exceeds DINOv2+KL by 7.04 and 6.91 mAP points, respectively. These results demonstrate higher retrieval performance under the stated estimated time budgets.

The judgment budget does not grow with dataset size: judgments are collected within groups of 32, and in Table 5(c) the pool grows from 400 to 2,000 images under the same ~44,450-judgment budget, with the largest pool performing best (60.90 mAP). We focus on DINOv2+KL because it is the strongest feature-based baseline in Table 1 (71.7 mean mAP, versus 54.1 for KUEA and 59.5 for Qwen2.5-VL vision-tower features).

Changes in the revision. The paper reports the fully measured version of this comparison, described in the next response (Table 5(b), Section 3.2, Appendix D.9).


W1 (continued): matched, measured compute.

We appreciate the reviewer for this comment. We conducted a new comparison under matched, measured budgets of 1.5 and 2.0 H100-hours. Both methods use a single H100, the same CLIP ViT-B/16 student, rank-32 vision/text LoRA, KL weight, training images, and random seed. ASK’s budget includes the full 0.611 H100-hours required to generate its Qwen3.8-27B-FP8 judgments; DINOv2+KL’s budget includes fresh feature extraction. Both budgets also include loading, preparation, calibration, and adaptation.

Budget (H100-hours)MethodTraining updatesUnseen-breed mAP (1,000 images)Non-dog mAP (5,000 images)
1.5ASK5,92458.9673.33
1.5DINOv2+KL9,95750.9872.08
2.0ASK9,28057.6572.53
2.0DINOv2+KL13,29750.0271.85

Including teacher-generation costs, ASK improves unseen-breed retrieval by 7.98 and 7.63 mAP points at the two budgets, respectively, and also improves non-dog retrieval.

Changes in the revision. This comparison is reported in Table 5(b), discussed in Section 3.2, and detailed in Appendix D.9. The Conclusion notes that the gains hold under matched end-to-end compute that includes teacher scoring.

AuthorsOpenReview ↗

Response to Reviewer WF3x (2/2)

W2 The method combines established techniques and resembles KUEA and relational distillation. What is the substantive difference?

We thank the reviewer for asking us to clarify the distinction. ASK builds on kernel and relational distillation to enable criterion-selectable visual similarity from VLM judgments. KUEA transfers relationships encoded in a vision teacher’s features. Similarly, Park et al. transfer distances and angles between teacher representations, and Tung & Mori preserve similarities between teacher activations; all three require teacher features. ASK lets the same teacher and images define different target geometries—emphasizing breed, colour, viewpoint, or background—through the comparison prompt, without teacher-embedding access. We extract pairwise similarity scores using probability-weighted digit expectations when available. Because these independently elicited scores need not form a PSD kernel, ASK applies symmetrization, score rescaling, spectral PSD projection, and diagonal normalization within each group (Secs. 2.2–2.3). Table 4(b) and Appendix D.5 demonstrate that the resulting encoders learn the requested distinctions.

A second substantive difference is joint adaptation of the visual and text encoders. KUEA freezes the text encoder and constrains visual-feature drift through an L2 anchor. ASK instead regularizes image–text prediction distributions with KL, allowing both encoders to accommodate the requested visual geometry while remaining anchored to frozen CLIP’s predictions. This joint adaptation yields a substantial benefit: under the same KL objective, adapting the text encoder increases retrieval from 49.71 to 60.16 mAP, a 10.45-point gain (Table 6(e)). Appendix E.4 provides further regularizer and weight comparisons.

The benefit of judgment supervision also persists after controlling for this anchoring change. DINOv2+KL uses the same LoRA configuration and KL objective, yet ASK achieves 75.0 versus 71.7 mean retrieval mAP across five domains (Table 1). These results support both the value of prompt-selected similarity targets and the joint adaptation mechanism; kernel matching and PSD projection remain established building blocks.

Changes in the revision. The Introduction now contrasts ASK with KUEA directly, citing the joint-adaptation gain (Table 6(e)) and the fact that a feature teacher such as DINOv2 provides one fixed set of relationships. The KUEA paragraph of the Related Work (Appendix B) adds both empirical controls.


W2 The cubic gain over cosine is only 1.81 points. Would simpler pairwise score regression suffice?

We thank the reviewer for this comment. We do not claim the cubic kernel over cosine as a substantive contribution. Our contribution concerns transferring criterion-dependent VLM judgments into CLIP with joint image–text anchoring. The cubic kernel is an empirical design choice supported by its stronger retrieval results.

Table 6(f) and Appendix E.6 already test simpler pairwise regression: cosine-MSE uses the same PSD-processed targets, LoRA configuration, and KL anchor, achieving 58.35 mAP versus 60.16 for normalized-cubic MSE. Both models use cosine similarity at retrieval time. Cubic training also improves Spearman agreement under the common cosine readout from 0.756 to 0.794 (Table 22). These gains support our default choice, while the strong cosine-MSE results confirm that the approach also works with simpler pairwise regression.

Changes in the revision. Section 2.1 now states that p = 3 is an empirical default. Appendix E.6 adds a comparison of cosine, quadratic, cubic, and RBF training similarities (Table 23), which Section 3.3 summarizes.

Reviewer_WF3xOpenReview ↗

Final justification

Authors have answered my queries and as a result, I would like to raise my score to 3.

Some relevant works are not cited, for instance,

  1. Sarkar A. et. al., Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction.
  2. Tong S. et. al., Eyes wide shut? exploring the visual shortcomings of multimodal llms.

Thanks.

Thank you for the feedback. We’re glad our response addressed your questions. We also appreciate the references and will cite both papers in the final version and briefly discuss how they relate to our work.

Reviewer_4CDxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method to adapt CLIP based models for comparing images based on custom criteria defined using text prompts. To do this, the paper takes a dataset of images, and uses a VLM to score pairs of images (not densely, groups are created with dense scoring in groups) to build target similarity matrices which are then used to supervise training of CLIP adapters.

The proposed method involves sampling groups from the dataset using the class labels, and using a VLM to densely compare pairs of images in each group. These are used to create a group similarity matrix which is then repair to be Positive Semi Definite so that it represents a valid normalized kernel matrix. This is used for training supervision. In order to make sure the model doesnt diverge from its frozen behavior on image text embeddings, a loss is incorporated for penalizing divergence between the frozen and adapted model.

Experiments are done over multiple datasets like Imagent dog breeds, stanford cars etc. The proposed method is compared against several baselines and performs competitively across the domains.

Critical strengths
  1. The paper compares against several baselines like frozen CLIP, dinov2 feature targets, etc. to help demonstrate the effectiveness of VLM guided adaptation.
  2. The ablations are interesting and helpful to understand the various components. For example, in table 5e the results show the value of having the KL anchoring for maintaining zero shot performance.
Critical weaknesses
  1. The overall motivation and problem setup of the paper leave room for improvement. The paper tackles this problem of adapting CLIP to custom similarity criteria by using VLMs to build kernel targets. I think this part of the problem is important. However, the method also assumes the availability of class labels in places like when creating groups. This leads to the question that if class labels are needed for this method, then why not just annotate class labels for the target criteria and use prompt learning / adapter methods for directly adapting CLIP to that criteria. It would have been great if the paper could decouple the necessity / benefit of incorporating labels for grouping.
  2. The experiment setup of the paper is quite small scale and requires explanation on some design choices. The paper takes 25 classes for each domain, which are significantly narrowed down. The evaluation set only contains 10 images per class for the teacher fidelity experiment, making it hard to determine whether the results will generalize. Given that the model requires custom comparison prompts, the scalability of the method is unclear. It requires domain level knowledge of different factors across the classes in a domain to be able to provide some criteria to the VLM. It would be great to understand the scaling of this method with the number of adaptation classes, training images etc.

Overall I think this paper formulates an interesting problem of utilizing VLMs to build target kernel matrices. I think it would be greatly improved by polishing the motivation and improving the evaluation setup.

Questions for authors
  1. It would be great to understand evaluation on a larger test set to understand the performance reliably. It would be great if evaluation could be done for at least one domain.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4CDx (1/2)

We sincerely thank you for your comprehensive examination of our paper and value the thoughtful feedback you have offered. Your helpful suggestions have played a crucial role in improving the overall quality of our research. Please note that all table, figure, and section numbers refer to the revised paper (changes in blue). The original Table 5 (ablations) is now Table 6, appendix Tables 6–21 are now Tables 7–22, and Table 22 is now Table 24.

Following the suggestion to polish the motivation, the revised Introduction states the problem more directly: a VLM can compare images by fine-grained identity or by a chosen property such as colour, viewpoint, or background, whereas CLIP provides one fixed similarity (paragraph 1). ASK transfers these prompt-selected judgments into CLIP once, so the resulting encoder supports fast cosine retrieval for a fixed criterion without VLM calls (paragraph 6). We address the evaluation-setup concerns in W2 / Q1 below.

W1. ASK uses class labels for grouping. Why not annotate classes for the target criterion and use prompt learning or adapters? Can you separate the necessity and benefit of labels?

We thank the reviewer for this comment. Labels are not necessary, but they help. Labels enter training only through stratified grouping (Appendix C.2); neither loss uses per-image class targets (Eqs. 5–8). To separate their role, we compared class-stratified groups with random groups that ignore labels. Both use Qwen3.8-27B judgments, with the same image pool, number of groups, judgments (44,450), and per-image exposure:

GroupingUnseen-breed mAP
Random (no labels)57.87 ± 0.47
Class-stratified (default)59.52 ± 0.46

Scores are mean ± SD over three training seeds. Class-stratified grouping adds 1.65 points, while random grouping retains most of the gain. Both conditions draw from the same 1,000-image pool (25 breeds × 40 images). Random groups ignore breed, but the pool itself is breed-balanced, so this experiment isolates the role of labels in grouping rather than in data collection.

Annotating each criterion and training prompts or adapters is a valid alternative, but it needs new human labels for every criterion. Class labels also cannot capture graded similarity, or similarity in colour, viewpoint, or background. ASK learns these from a prompt alone: on the same images, changing the prompt gives a student that performs best on its own criterion (Table 4(b)), as human annotations confirm (Appendix D.5). With the same training recipe, image-level judgments also beat binary class labels (60.16 vs. 43.11 mAP) and VLM class-pair averages (57.17 mAP) (Table 6(b), Appendix E.1).

Changes in the revision. Section 3.1 and Appendix C.2 now state that labels are used only for grouping. The label-blind comparison appears in Table 6(i), Section 3.3, and Appendix E.1.


W2 / Q1. The 25-class adaptation setup and 10-image-per-class fidelity test are small. Do the results hold on a larger test set?

We appreciate the reviewer for this comment. We conducted a larger evaluation on 7,000 additional held-out images: 1,000 images for unseen-breed retrieval, 5,000 non-dog ImageNet images, and 1,000 fidelity images (40 per adaptation breed), with 50,000 uniformly sampled image-pair judgments. All images are disjoint from adaptation, validation, and previous evaluation sets. We evaluated the existing checkpoints without retraining.

ModelUnseen-breed mAPNon-dog mAPFidelity (Spearman ρ)
CLIP23.5167.900.199
SupCon + KL48.4270.080.425
DINOv2 + KL53.2273.130.497
ASK61.1874.260.749

The gains persist on these larger evaluation sets. ASK exceeds DINOv2+KL by 7.96 unseen-breed mAP points and achieves the highest non-dog retrieval and fidelity scores among the evaluated methods.

Changes in the revision. This evaluation is reported in Table 5(a), discussed in Section 3.2, and detailed in Appendix D.2.

AuthorsOpenReview ↗

Response to Reviewer 4CDx (2/2)

W2. How does performance scale with the number of adaptation classes and training images?

We thank the reviewer for this comment. We conducted a scaling study varying both the number of adaptation classes and the number of images per class. All settings use 3,000 training updates and approximately 44,450 teacher judgments for ASK, with judgment budgets matched within 0.51%. Evaluation uses the same 1,000-image unseen-breed test set, disjoint from every adaptation pool. Scores below are retrieval mAP.

Adaptation classesImages per classTotal training imagesDINOv2 + KLASK
104040045.2754.25
252050050.7858.36
25401,00053.2259.67
25802,00052.6259.24
50402,00052.9460.90

ASK outperforms DINOv2+KL in all five settings. With 40 images per class, increasing adaptation classes from 10 to 50 improves ASK from 54.25 to 60.90 mAP. At the same total of 2,000 images, covering 50 classes also performs better than covering 25 classes (60.90 versus 59.24). Increasing images within the same 25 classes does not consistently improve performance under the fixed judgment and update budgets. These results support the benefit of broader class coverage while showing that gains from additional images depend on the available supervision and training budget.

Changes in the revision. The scaling study is reported in Table 5(c), discussed in Section 3.2, and detailed in Appendix D.2.


W2. Custom criteria seem to require domain knowledge. Does prompt design scale as classes increase?

We thank the reviewer for this comment. Within each domain, ASK uses one shared prompt per criterion across all classes and image pairs, rather than class-specific prompts (Appendix G). For example, the car viewpoint prompt asks the teacher to compare pose, viewing angle, and framing while ignoring model identity, colour, and setting. It does not enumerate car models or provide model-specific rules. Adding classes therefore does not require additional prompts.

Choosing a meaningful criterion does require knowledge of the task. However, the user specifies which properties matter and what to ignore; the VLM supplies the image-specific judgments. Our prompts describe visible properties such as colour, texture, viewpoint, and background, without supplying class-specific expert descriptions. Thus, ASK reduces the need for class-by-class prompt design, while still depending on a clearly specified criterion and a teacher capable of interpreting it.

The existing transfer experiment directly demonstrates prompt reuse: Appendix D.3 (Table 9) applies the original colour/lighting, pose/view, and background prompts unchanged to 50 non-dog classes. The matching criterion-trained student leads each evaluation column. This supports reuse across additional classes without redesigning the prompts.

Changes in the revision. Section 2.2 now states that each criterion uses one shared prompt for all classes in a domain, so adding classes requires no new prompts. The introduction to Appendix G explains prompt reuse and the role of task knowledge, and points to the transfer results in Appendix D.3.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 4CDx,

Thank you again for your time and constructive feedback. As the discussion period is ending soon, we would like to kindly ask whether our response has addressed your concerns. If so, we would be grateful if you would consider updating your rating. We are happy to clarify any remaining questions.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Reviewers, Area Chairs, and Senior Area Chairs,

As the discussion period comes to an end, we sincerely thank you for your time and thoughtful feedback, which have strengthened both the clarity and the completeness of our manuscript. We have addressed all reviewer comments and incorporated the resulting experiments, analyses, and clarifications into the revised manuscript, where changes are marked in blue. The main updates are summarized below.

  1. Contribution and distinction from KUEA and relational distillation. ASK transfers prompt-selected similarity judgments from a VLM into CLIP, without accessing teacher embeddings or collecting new human annotations, so the same teacher and images can define different similarity criteria. ASK also adapts CLIP differently: KUEA freezes the text encoder and anchors visual features with an L2 penalty, whereas ASK adapts both the image and text encoders under a KL anchor on image–text predictions. This difference matters substantially. With the same DINOv2 targets, KUEA reaches 54.1 mean mAP (CLIP: 53.8), while our adaptation recipe reaches 71.7; replacing DINOv2 with VLM judgments raises this to 75.0 (Table 1). In a controlled ablation, adapting the text encoder alone raises retrieval from 49.71 to 60.16 mAP, a 10.45-point gain (Table 6(e)). We present kernel matching and PSD projection as established tools that ASK builds on (Reviewer WF3x: W2; Reviewer X26n: W2; Reviewer 7Whq: W1–W2).

  2. Practical setting and conditional-similarity comparisons. ASK targets settings where one application-defined criterion is reused across a collection, so embeddings can be cached and searched by cosine similarity without further VLM calls. We added GeneCIS alongside MagicLens in the human-annotated comparisons, and ASK leads on all six evaluated criteria (Figure 3(c), Table 11). A shared model with learned criterion masks, trained on the same judgments and aggregate update budget, shows that separate adapters improve wing-colour and bill-shape retrieval by 1.81 and 2.91 nDCG@10 points, with comparable breast-pattern performance (Table 6(h)). We discuss the resulting storage and flexibility trade-off (Reviewer X26n: W1, W3).

  3. Label dependence, prompt reuse, and adaptation scale. On the same image pool, random groups reach 57.87 ± 0.47 mAP versus 59.52 ± 0.46 for class-stratified groups: labels help form groups but are not required by the losses (Table 6(i)). One prompt is shared across all classes for each criterion. A new scaling study covers 10–50 classes and 400–2,000 training images under fixed judgment and update budgets; ASK exceeds DINOv2+KL in all five settings, and broader class coverage improves retrieval (Table 5(c)) (Reviewer 4CDx: W1–W2).

  4. Larger evaluation and broader transfer. On 7,000 additional held-out images (1,000 unseen-breed, 5,000 non-dog ImageNet, and 1,000 fidelity images with 50,000 scored pairs), ASK exceeds DINOv2+KL by 7.96 unseen-breed and 1.13 non-dog mAP points (Table 5(a)). We also highlight transfer to COCO-Stuff background retrieval without COCO adaptation (Figure 5) (Reviewer 4CDx: W2, Q1; Reviewer 7Whq: W4).

  5. Measured computational cost. Under matched, measured budgets of 1.5 and 2.0 H100-hours that include teacher scoring, feature extraction, preparation, calibration, and adaptation, ASK, charged the full 0.611 H100-hours for its teacher judgments, exceeds DINOv2+KL by 7.98 and 7.63 unseen-breed mAP points and also improves non-dog retrieval (Table 5(b)). Appendix D.9 reports student runtimes, API-cost estimates, and judgment reuse (Reviewer WF3x: W1; Reviewer X26n: W3).

  6. Design sensitivity and PSD correction. We added quadratic and RBF kernels, offset-ratio sensitivity, and group-size experiments, with practical guidance (Table 23, Appendix E.7). Cubic remains the best default, while direct cosine regression also performs well. PSD correction largely preserves teacher preferences: raw–corrected Spearman correlations range from 0.959 to 0.999, with at most 0.060% reversals among pairs differing by at least one rating point (Table 6(g)). We note that projection can change rankings and that its benefit varies across teachers and criteria (Reviewer WF3x: W2; Reviewer 7Whq: W3, Q1).

We hope these additions clarify ASK's contribution, setting, evidence, and limitations. Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.