- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes new evaluation metrics and a training-free selection method for Language-Only VLM Selection (LOVM), which aims to select an appropriate VLM for a target task using only a textual description of that task. Motivated by the observation that the prior evaluation protocol does not directly measure the performance of the selected model, the authors introduce metrics such as SR and ORNR. Building on these metrics, they propose TC-ESR, which accounts for the similarity between the target task and the source tasks. Under the leave-one-out evaluation protocol, TC-ESR achieves the lowest SR and ORNR among existing LOVM methods.
Critical strengths
- The paper clearly identifies the limitations of the existing evaluation metric and proposes new metrics that directly address them.
- TC-ESR, which builds on the proposed metrics and combines several techniques, outperforms existing LOVM methods under the revised evaluation protocol. Its effectiveness is further supported by extensive ablation studies and robustness analyses.
Critical weaknesses
Limited task scope and limited broader methodological insight.
The LOVM benchmark used in this paper is restricted to a relatively narrow set of visual recognition tasks. It does not evaluate broader VLM capabilities such as VQA or image-text retrieval. Moreover, the proposed TC-ESR method is closely tied to this benchmark setting, as its task similarity relies on closed-set, finite label spaces through label semantics and label granularity. It is therefore unclear how the proposed approach could be extended to open-ended tasks without a predefined label set. As a result, while the paper provides useful observations within the LOVM setting, I found the broader methodological insight and generalizable takeaway beyond this setting to be limited.
Questions for authors
Why can the LOVM setting be considered practically relevant?
The paper argues that this setting is useful when acquiring labeled data for a target task is difficult or costly. However, it also assumes full access to the complete set of target class labels. It is not immediately clear to me in which real-world scenarios this assumption is realistic or useful. I therefore encourage the authors to describe realistic and convincing practical use cases that motivate the LOVM setting and clarify when access to the full target label set can reasonably be assumed despite the lack of labeled examples.
4 comments
Response to Reviewer ntVV (2/2)
On the generalizable takeaway. We will state it in the conclusion together with the scope limitation. Three observations do not depend on classification and are relevant wherever a selection method is scored by its predicted ranking: (a) ranking metrics and the regret of the selected model can disagree substantially ( and reverse the SR conclusion in 25.7% and 43.1% of per-target method comparisons; Table 5); (b) prior LOVM methods, including the two that train a ranker model (ModelGPT and SWAB), select nearly the same model for every target, which averaged metrics do not reveal (Table 22; on VQAv2, ModelGPT and SWAB select the same model as INB), so we report the best-fixed-model reference; and (c) target performance can leak through feature selection and input features even when the target is nominally held out (Table 6).
Q1. Why the class list is a realistic input In zero-shot classification the class list is the minimum input, not an extra assumption: a CLIP-style VLM is deployed by building its classifier from the class names, so no candidate can be used without the list, while labeled images are not needed. “Class list available, labeled examples not yet” is therefore the starting state of anyone deploying a zero-shot classifier. A concrete case is a product-classification service: its category catalogue and image domain are fixed before photographs from the intended collection process exist, so the first model has to be chosen from this specification alone. Other cases are images that cannot leave the site where they are collected, so that the party choosing the model cannot run the candidates on them, and shortlisting among dozens of checkpoints before any of them is downloaded and run.
Without class names. We separate two results. On the classification benchmark, removing the two label axes from TC-ESR is an ablation: the domain metadata were still assigned with the class names available, so it is not a test of a pipeline without class names. In this ablation the mean SR is 6.79 %p, against 5.83 %p for ESR, which uses no information about the target and is itself better than every prior method (7.11-7.39 %p). Together with Table 2, where removing the domain axis gives 5.91 %p, this shows that TC-ESR improves on ESR only when the label axes and the domain axis are used together. On the 13 targets that actually have no class names (W1), domain-only TC-ESR has the lowest mean ORNR (10.7%, against 12.8% for INB and 13.4% for ESR), but the difference is within noise. We therefore claim no advantage of TC-ESR without class names, and we will state this as a limitation.
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns, but I remain unconvinced about the broader generalizability and methodological insights of this work.
While the additional experiments broaden the evaluation scope, I do not find them particularly convincing as evidence of generalization. I find it less convincing as evidence of generalization beyond the original closed-set formulation. The experiment essentially replaces class names with a finite set of candidate answers, without demonstrating how the proposed task similarity could extend to genuinely open-ended tasks. Moreover, the reported improvements are statistically inconclusive. I think these results may suggest that the method remains closely tied to the specific structure of the original LOVM benchmark.
Regarding the claimed generalizable takeaways, the distinction between ranking-based evaluation and the quality of the selected model has already been explicitly discussed in prior transferability estimation research (e.g., [1], Section 3.3). While demonstrating this discrepancy in LOVM is useful, the underlying insight is not new. Similarly, the observation that prior methods tend to select the same model across targets is an interesting benchmark-specific finding. Identifying evaluation leakage is also valuable, but I am not convinced that these observations establish sufficiently generalizable methodological insights beyond the specific setting considered.
I remain unconvinced that the paper offers sufficiently substantial insights for the broader community. I will therefore respectfully maintain my original score. Please let me know if there are any points in your response that I may have overlooked or misunderstood.
[1] Agostinelli, Andrea, et al. "How stable are transferability metrics evaluations?." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.
Response to Reviewer ntVV
Thank you for the careful reply and for pointing us to Agostinelli et al. (2022). We appreciate the chance to clarify a few points.
1. The additional experiments. We agree: the improvements are statistically inconclusive, and these experiments do not show that the gain of TC-ESR extends beyond targets with class names. The revised paper states this as a limitation (Appendices I and J). One clarification, in case our response was unclear: VQAv2 is the only new target that uses a finite answer set. The other 13 targets in Table R1 (image-text retrieval, SugarCrepe, caption evaluation) have no label set or answer vocabulary, so on these targets TC-ESR conditions on the domain alone. What carries over to them is the evaluation by SR and ORNR, which does not use class names. The two leakage issues were identified in the original LOVM benchmark; keeping the target out of feature selection and validation is a general requirement, but we did not test its effect on these additional targets.
2. Novelty of the takeaways. The distinction between ranking quality and the quality of the selected model has indeed been discussed for transferability metrics, and the revised paper now cites Agostinelli et al. (2022) in the Introduction and relates their Rel@1 to SR and ORNR in Appendix A. Our intended contribution is narrower: what this distinction implies for LOVM, where the benchmark and the existing methods are evaluated by and , and ModelGPT and SWAB are also tuned by them. When evaluated by the regret of the selected model, the methods that train a ranker are on par with ranking by ImageNet accuracy and worse than the best fixed model (7.11-7.39 vs. 7.23 and 5.52 %p; Table 1), and retuning their feature selection by SR or ORNR leaves ModelGPT essentially unchanged and makes SWAB worse (Table 21). In addition, the ordering of the methods by is inverted relative to SR (Kendall correlation of -0.83; Table 5), and two sources of target leakage inflate the reported ranking scores (Table 6). The paper also introduces ORNR and a training-free method with the lowest regret among the compared methods.
3. Scope. We understand the concern about breadth. The paper is about LOVM, and we have revised the conclusion to state this scope explicitly rather than suggest a result on model selection in general. We believe the findings are useful for work in this setting, which has an established benchmark and published methods, and we see extending them beyond classification as future work.
Thank you again for your time and for the constructive discussion.
Response to Reviewer ntVV (1/2)
We appreciate your time and constructive comments. Below we address the scope of the experiments and the broader takeaway (W1) and the practical relevance of the setting (Q1), with new experiments.
W1. Scope and what generalizes We agree that the experiments in the submission are limited to zero-shot classification; we used the prior methods’ own benchmark so that the protocol problems could be shown under their conditions. In response to your comment we added two settings outside this benchmark (Table R1), which we will include in the revision together with a limitations paragraph on what still depends on class names.
In TC-ESR, only the label-semantic and label-granularity similarities are computed from class names. The domain similarity compares categorical metadata that an LLM assigns from the task record (the domain and task descriptions and, when they exist, the class names), and the evaluation metrics (SR, ORNR) and the leakage corrections use no class names. We therefore added two kinds of targets. On VQAv2, the answer vocabulary is used in place of the class names and TC-ESR runs unchanged. On 13 targets without class names, such as image-text retrieval, the task is specified by a domain and task description and no class-name set exists; TC-ESR then computes the similarity from the domain alone, in the same way that Eq. 8 already omits label granularity for targets where it does not apply. Both settings use the 35 OpenCLIP candidates of the zoo, the 23 LOVM tasks as sources, and the protocol of the main experiments. We restrict the candidates to these 35 because the new targets are scored with standard OpenCLIP-based protocols (CLIP_benchmark for retrieval, TAP-C for VQAv2), which do not support the other eight candidates, BLIP and BEiT-3 retrieval checkpoints fine-tuned on MSCOCO or Flickr30k.
We also ran ModelGPT and SWAB on VQAv2, with the answer vocabulary as their class set (the same input that TC-ESR uses for its label-semantic axis). Both select the same model as INB, so their SR and ORNR in Table R1 equal those of INB. On the targets without class names the two methods cannot be run, because their features require a fixed target class set.
Table R1. New settings beyond zero-shot classification (lower is better).
The targets without class names are four image-text retrieval targets (Flickr30k and MSCOCO, both directions), the seven SugarCrepe subsets (Hsieh et al., 2023), and two caption-evaluation targets scored by the Kendall correlation between CLIPScore (Hessel et al., 2021) and human judgments; since their metrics differ, we aggregate with ORNR. Without class names only the domain axis remains, and it is identical for all 13 targets, so each method selects a single model for all of them (INB and AvgRank the same one) and the differences are within noise: TC-ESR has a lower ORNR than INB on 8 of the 13 targets (95% paired-bootstrap CI over targets of the mean difference, INB minus TC-ESR: [-2.6, 7.1]); its lower average comes from SugarCrepe, and on the four retrieval targets its mean ORNR is higher than that of INB (9.3 vs. 8.2%). We will state this as a limitation.
VQAv2 is evaluated with TAP-C (Song et al., 2022). It is a single target on which the candidates differ little (robust range of about 4.5 %p), so we present the lower regret of TC-ESR (1.73 vs. 2.09 %p for the four baselines that select the same model, on 3,000 questions) only as a check of applicability; excluding the two MSCOCO-fine-tuned CoCa checkpoints leaves the order of the methods unchanged.
With the counting rule of Table 5, R5 and τ5 disagree with SR in 38.6% and 22.0% of the comparable method pairs on the targets without class names (descriptive only, since each method selects one model there); on VQAv2, TC-ESR has both the lowest R5 (0.00) and the lowest SR.