acceptodds
Under review as a conference paper at ICLR 2027

Good Rankings, Poor Choices? Rethinking Language-Only Vision–Language Model Selection

Abstract

Language-Only Vision–Language Model Selection (LOVM) aims to identify the Vision–Language Model (VLM) with the highest zero-shot accuracy on a target task using only textual information. We revisit the existing LOVM evaluation protocol and argue that ranking metrics, such as top-5 recall and Kendall's rank correlation, do not directly assess the performance of the selected model. We therefore adopt simple regret (SR), defined as the performance gap between the best and selected models, and introduce Outlier-Robust Normalized Regret (ORNR) to account for the varying performance ranges across tasks while reducing sensitivity to outliers. Building on this revised evaluation protocol, we propose Target-Conditioned Expected Source Regret (TC-ESR), a training-free method that selects the model with the lowest similarity-weighted mean regret across source tasks. Across 22 target tasks and 43 VLMs, TC-ESR achieves the lowest mean SR and ORNR among all compared methods. Separately, we identify two sources of target performance leakage in the prior ranking-based evaluation protocol and provide corrections that remove both. The code will be released.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
language-only vlm selection, vision-language models, model selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ntVVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes new evaluation metrics and a training-free selection method for Language-Only VLM Selection (LOVM), which aims to select an appropriate VLM for a target task using only a textual description of that task. Motivated by the observation that the prior evaluation protocol does not directly measure the performance of the selected model, the authors introduce metrics such as SR and ORNR. Building on these metrics, they propose TC-ESR, which accounts for the similarity between the target task and the source tasks. Under the leave-one-out evaluation protocol, TC-ESR achieves the lowest SR and ORNR among existing LOVM methods.

Critical strengths
  1. The paper clearly identifies the limitations of the existing evaluation metric and proposes new metrics that directly address them.
  2. TC-ESR, which builds on the proposed metrics and combines several techniques, outperforms existing LOVM methods under the revised evaluation protocol. Its effectiveness is further supported by extensive ablation studies and robustness analyses.
Critical weaknesses

Limited task scope and limited broader methodological insight.

The LOVM benchmark used in this paper is restricted to a relatively narrow set of visual recognition tasks. It does not evaluate broader VLM capabilities such as VQA or image-text retrieval. Moreover, the proposed TC-ESR method is closely tied to this benchmark setting, as its task similarity relies on closed-set, finite label spaces through label semantics and label granularity. It is therefore unclear how the proposed approach could be extended to open-ended tasks without a predefined label set. As a result, while the paper provides useful observations within the LOVM setting, I found the broader methodological insight and generalizable takeaway beyond this setting to be limited.

Questions for authors

Why can the LOVM setting be considered practically relevant?

The paper argues that this setting is useful when acquiring labeled data for a target task is difficult or costly. However, it also assumes full access to the complete set of target class labels. It is not immediately clear to me in which real-world scenarios this assumption is realistic or useful. I therefore encourage the authors to describe realistic and convincing practical use cases that motivate the LOVM setting and clarify when access to the full target label set can reasonably be assumed despite the lack of labeled examples.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ntVV (1/2)

We appreciate your time and constructive comments. Below we address the scope of the experiments and the broader takeaway (W1) and the practical relevance of the setting (Q1), with new experiments.

W1. Scope and what generalizes We agree that the experiments in the submission are limited to zero-shot classification; we used the prior methods’ own benchmark so that the protocol problems could be shown under their conditions. In response to your comment we added two settings outside this benchmark (Table R1), which we will include in the revision together with a limitations paragraph on what still depends on class names.

In TC-ESR, only the label-semantic and label-granularity similarities are computed from class names. The domain similarity compares categorical metadata that an LLM assigns from the task record (the domain and task descriptions and, when they exist, the class names), and the evaluation metrics (SR, ORNR) and the leakage corrections use no class names. We therefore added two kinds of targets. On VQAv2, the answer vocabulary is used in place of the class names and TC-ESR runs unchanged. On 13 targets without class names, such as image-text retrieval, the task is specified by a domain and task description and no class-name set exists; TC-ESR then computes the similarity from the domain alone, in the same way that Eq. 8 already omits label granularity for targets where it does not apply. Both settings use the 35 OpenCLIP candidates of the zoo, the 23 LOVM tasks as sources, and the protocol of the main experiments. We restrict the candidates to these 35 because the new targets are scored with standard OpenCLIP-based protocols (CLIP_benchmark for retrieval, TAP-C for VQAv2), which do not support the other eight candidates, BLIP and BEiT-3 retrieval checkpoints fine-tuned on MSCOCO or Flickr30k.

We also ran ModelGPT and SWAB on VQAv2, with the answer vocabulary as their class set (the same input that TC-ESR uses for its label-semantic axis). Both select the same model as INB, so their SR and ORNR in Table R1 equal those of INB. On the targets without class names the two methods cannot be run, because their features require a fixed target class set.

Table R1. New settings beyond zero-shot classification (lower is better).

SettingMetricINBAvgRankModelGPTSWABESRTC-ESR
13 targets without class names (retrieval ×4, SugarCrepe ×7, CLIPScore–human correlation ×2)mean ORNR (%)12.812.8n/an/a13.410.7
VQAv2 (TAP-C zero-shot scoring, 3,000 val questions)SR (%p) / ORNR (%)2.09 / 46.52.09 / 46.52.09 / 46.52.09 / 46.51.83 / 40.81.73 / 38.6

The targets without class names are four image-text retrieval targets (Flickr30k and MSCOCO, both directions), the seven SugarCrepe subsets (Hsieh et al., 2023), and two caption-evaluation targets scored by the Kendall correlation between CLIPScore (Hessel et al., 2021) and human judgments; since their metrics differ, we aggregate with ORNR. Without class names only the domain axis remains, and it is identical for all 13 targets, so each method selects a single model for all of them (INB and AvgRank the same one) and the differences are within noise: TC-ESR has a lower ORNR than INB on 8 of the 13 targets (95% paired-bootstrap CI over targets of the mean difference, INB minus TC-ESR: [-2.6, 7.1]); its lower average comes from SugarCrepe, and on the four retrieval targets its mean ORNR is higher than that of INB (9.3 vs. 8.2%). We will state this as a limitation.

VQAv2 is evaluated with TAP-C (Song et al., 2022). It is a single target on which the candidates differ little (robust range of about 4.5 %p), so we present the lower regret of TC-ESR (1.73 vs. 2.09 %p for the four baselines that select the same model, on 3,000 questions) only as a check of applicability; excluding the two MSCOCO-fine-tuned CoCa checkpoints leaves the order of the methods unchanged.

With the counting rule of Table 5, R5R_5 and τ5\tau_5 disagree with SR in 38.6% and 22.0% of the comparable method pairs on the targets without class names (descriptive only, since each method selects one model there); on VQAv2, TC-ESR has both the lowest R5R_5 (0.00) and the lowest SR.

AuthorsOpenReview ↗

Response to Reviewer ntVV (2/2)

On the generalizable takeaway. We will state it in the conclusion together with the scope limitation. Three observations do not depend on classification and are relevant wherever a selection method is scored by its predicted ranking: (a) ranking metrics and the regret of the selected model can disagree substantially (R5R_5 and τ5\tau_5 reverse the SR conclusion in 25.7% and 43.1% of per-target method comparisons; Table 5); (b) prior LOVM methods, including the two that train a ranker model (ModelGPT and SWAB), select nearly the same model for every target, which averaged metrics do not reveal (Table 22; on VQAv2, ModelGPT and SWAB select the same model as INB), so we report the best-fixed-model reference; and (c) target performance can leak through feature selection and input features even when the target is nominally held out (Table 6).

Q1. Why the class list is a realistic input In zero-shot classification the class list is the minimum input, not an extra assumption: a CLIP-style VLM is deployed by building its classifier from the class names, so no candidate can be used without the list, while labeled images are not needed. “Class list available, labeled examples not yet” is therefore the starting state of anyone deploying a zero-shot classifier. A concrete case is a product-classification service: its category catalogue and image domain are fixed before photographs from the intended collection process exist, so the first model has to be chosen from this specification alone. Other cases are images that cannot leave the site where they are collected, so that the party choosing the model cannot run the candidates on them, and shortlisting among dozens of checkpoints before any of them is downloaded and run.

Without class names. We separate two results. On the classification benchmark, removing the two label axes from TC-ESR is an ablation: the domain metadata were still assigned with the class names available, so it is not a test of a pipeline without class names. In this ablation the mean SR is 6.79 %p, against 5.83 %p for ESR, which uses no information about the target and is itself better than every prior method (7.11-7.39 %p). Together with Table 2, where removing the domain axis gives 5.91 %p, this shows that TC-ESR improves on ESR only when the label axes and the domain axis are used together. On the 13 targets that actually have no class names (W1), domain-only TC-ESR has the lowest mean ORNR (10.7%, against 12.8% for INB and 13.4% for ESR), but the difference is within noise. We therefore claim no advantage of TC-ESR without class names, and we will state this as a limitation.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns, but I remain unconvinced about the broader generalizability and methodological insights of this work.

While the additional experiments broaden the evaluation scope, I do not find them particularly convincing as evidence of generalization. I find it less convincing as evidence of generalization beyond the original closed-set formulation. The experiment essentially replaces class names with a finite set of candidate answers, without demonstrating how the proposed task similarity could extend to genuinely open-ended tasks. Moreover, the reported improvements are statistically inconclusive. I think these results may suggest that the method remains closely tied to the specific structure of the original LOVM benchmark.

Regarding the claimed generalizable takeaways, the distinction between ranking-based evaluation and the quality of the selected model has already been explicitly discussed in prior transferability estimation research (e.g., [1], Section 3.3). While demonstrating this discrepancy in LOVM is useful, the underlying insight is not new. Similarly, the observation that prior methods tend to select the same model across targets is an interesting benchmark-specific finding. Identifying evaluation leakage is also valuable, but I am not convinced that these observations establish sufficiently generalizable methodological insights beyond the specific setting considered.

I remain unconvinced that the paper offers sufficiently substantial insights for the broader community. I will therefore respectfully maintain my original score. Please let me know if there are any points in your response that I may have overlooked or misunderstood.

[1] Agostinelli, Andrea, et al. "How stable are transferability metrics evaluations?." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

AuthorsOpenReview ↗

Response to Reviewer ntVV

Thank you for the careful reply and for pointing us to Agostinelli et al. (2022). We appreciate the chance to clarify a few points.

1. The additional experiments. We agree: the improvements are statistically inconclusive, and these experiments do not show that the gain of TC-ESR extends beyond targets with class names. The revised paper states this as a limitation (Appendices I and J). One clarification, in case our response was unclear: VQAv2 is the only new target that uses a finite answer set. The other 13 targets in Table R1 (image-text retrieval, SugarCrepe, caption evaluation) have no label set or answer vocabulary, so on these targets TC-ESR conditions on the domain alone. What carries over to them is the evaluation by SR and ORNR, which does not use class names. The two leakage issues were identified in the original LOVM benchmark; keeping the target out of feature selection and validation is a general requirement, but we did not test its effect on these additional targets.

2. Novelty of the takeaways. The distinction between ranking quality and the quality of the selected model has indeed been discussed for transferability metrics, and the revised paper now cites Agostinelli et al. (2022) in the Introduction and relates their Rel@1 to SR and ORNR in Appendix A. Our intended contribution is narrower: what this distinction implies for LOVM, where the benchmark and the existing methods are evaluated by R5R_5 and τ5\tau_5, and ModelGPT and SWAB are also tuned by them. When evaluated by the regret of the selected model, the methods that train a ranker are on par with ranking by ImageNet accuracy and worse than the best fixed model (7.11-7.39 vs. 7.23 and 5.52 %p; Table 1), and retuning their feature selection by SR or ORNR leaves ModelGPT essentially unchanged and makes SWAB worse (Table 21). In addition, the ordering of the methods by R5R_5 is inverted relative to SR (Kendall correlation of -0.83; Table 5), and two sources of target leakage inflate the reported ranking scores (Table 6). The paper also introduces ORNR and a training-free method with the lowest regret among the compared methods.

3. Scope. We understand the concern about breadth. The paper is about LOVM, and we have revised the conclusion to state this scope explicitly rather than suggest a result on model selection in general. We believe the findings are useful for work in this setting, which has an established benchmark and published methods, and we see extending them beyond classification as future work.

Thank you again for your time and for the constructive discussion.

Reviewer_BycSOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper works on Language-Only Vision–Language Model Selection (LOVM), aiming to identify the with the highest zero-shot accuracy on a target task using only textual information. The authors introduce three-fold contributions:

  1. Evaluation metrics. The authors argue that existing ranking metrics, such as top-5 recall and Kendall’s rank correlation, do not directly assess the performance of the selected model. Thus, they adopt SR to quantify the performance gap between the best and selected models, and introduce ORNR to account for the varying performance ranges across tasks while reducing sensitivity to outliers.
  2. Evaluation protocol corrections. They identify and correct two sources of target performance leakage in the prior protocol.
  3. Selection method. Furthermore, they propose TC-ESR, which uses only textual task information to measure the similarity between source and target tasks along three axes: visual domain, label semantics, and label granularity; it then weights source tasks by these similarities when aggregating the simple regret of each candidate model.
Critical strengths
  • The authors explain their thoughts and motivations clearly, so I can understand why they did some changes and proposed new methods when reading the paper.
  • The proposed metrics and selection methods are interesting and provide new insights.
Critical weaknesses
  • I'm not convinced by the authors' argument on existing evaluation metrics. I think using top-5 recall and Kendall’s rank correlation or SR and ORNR depends on "how we utilize the selection results in LOVM". If we only use the top1 selected model, yes, SR and ORNR are more reasonable. But if we can pick multiple models, such as top5 models, and do some further evaluation on these models, maybe top-5 recall and Kendall’s rank correlation are more important, because they show some overall performance of the selected models. I'm not sure about which is more suitable for LOVM. Maybe the authors can position SR and ORNR as an addition to the existing metrics.

  • Unlike prior LOVM methods, ESR and TC-ESR do not require training a ranker model but needs to call LLMs. I'm wondering whether ESR and TC-ESR is cheaper in computation than previous methods.

Questions for authors
  • I believe the paper is interesting and would have rated a clear acceptance for this paper, but I'm sorry that I am not familiar with LOVM and I'm afraid that I missed some important issues. I need to read other reviewers' comments and then make a decision.
  • I'm wondering what might be the applications of LOVM. In what cases we need to select VLMs only based on the textual information? Considering the current SR and ORNR results, are these existing methods useful in practice? Or do they still need to be further improved?
4 comments
AuthorsOpenReview ↗

Response to Reviewer BycS (1/2)

We appreciate your positive assessment and thoughtful questions. Below we address the role of ranking metrics (W1), the computational cost (W2), and the practical use of LOVM (Questions). Since you mention waiting for the other reviews, the concerns raised there are summarized with the new results in our common response.

W1. Evaluation should reflect how the selection result is used We agree that the appropriate metric depends on how the selection result is used. The submission assigns the roles as follows: SR and ORNR are the primary metrics, because they assess the model selected at rank one, which is the model deployed in the LOVM setting, and R5R_5 and τ5\tau_5 are retained as supplementary diagnostics of the predicted ranking (Sections 3.1 and 5.4, Appendix B.3, Table 14). We keep this position. We will make it explicit in Sections 3.1 and 5.4 and in the conclusion, where the current wording can read as if the ranking metrics were discarded.

When more than one model is selected, the matching measure depends on what is done with the selected models. If the top-kk models are evaluated further and the best one is deployed, it is the regret of the best model within the predicted top-kk (SR@kk; Renggli et al., 2022), of which SR is the case k=1k=1. This use requires labeled target data, which the LOVM setting assumes is not available.

Table R2. R5R_5 (Table 14) and SR@kk, averaged over the 22 targets (SR@kk: lower is better).

MethodR5R_5 (%)SR@1 (%p)SR@3 (%p)SR@5 (%p)
INB41.87.233.342.80
AvgRank41.87.353.962.80
ModelGPT41.17.113.552.67
SWAB47.47.393.152.38
ESR31.85.834.263.52
TC-ESR40.04.663.451.96

TC-ESR has the lowest regret at k=1k=1 and k=5k=5, while SWAB and INB are lower at k=3k=3. R5R_5 is informative for this use, since ESR has both the lowest R5R_5 and the highest SR@5, but it does not determine the outcome: SWAB has the highest R5R_5, while TC-ESR has the lowest SR@5.

If the top-5 models are instead used together as an ensemble, which needs no labeled target data but requires running five models, the result is different. Averaging class probabilities gives a mean accuracy of 62.4-62.7% for INB, AvgRank, ModelGPT, and SWAB, against 60.8% for ESR and TC-ESR, so in this setting the prior methods are better. This supports your point that the conclusion depends on how the selection result is used. With one model deployed, TC-ESR is the most accurate (60.96%, against 58.2-58.5% for the four prior methods). The direct measure for an ensemble is the accuracy of the ensemble itself rather than R5R_5: ESR and TC-ESR differ in R5R_5 (31.8 and 40.0%) but have the same ensemble accuracy, and INB reaches 62.4% with the same five models for every target. We therefore limit the claims of the paper to the deployment of a single model, and we will state this in the revision.

We will add both analyses to the appendix.

AuthorsOpenReview ↗

Response to Reviewer BycS (2/2)

W2. Computational cost Yes. Prior LOVM methods also call an LLM, and far more heavily. ModelGPT and SWAB generate a caption dataset for every target (5,000 captions for a 100-class task) and pass it through the text encoder of every candidate, whereas TC-ESR needs 5-15 LLM calls that each return two categorical labels and one pass of a single text encoder. ESR calls no LLM at all.

We profiled both pipelines on CIFAR100, a 100-class target of the benchmark. Both normally call an LLM through an API; for the timing we replace it in both with the same local LLM (Qwen3-4B-Instruct-2507) on one NVIDIA RTX 4090, because the response time of an API depends on network latency, server load, and rate limits and does not measure the computation of either method. The text-encoding stages of ModelGPT/SWAB are measured for the 35 OpenCLIP candidates only, so they are a lower bound for the full zoo of 43.

Table R3. Computational requirements and reported stage timings.

StageTC-ESRModelGPT / SWAB
LLM generation5-15 calls per target for task metadata (5 in practice); 2.25 s for 5 calls5,000 captions (50 per class) and synonyms; 480 s
Text encodingOne MPNet pass over class names; 0.01 s (GPU) / 0.52 s (CPU)35 candidate text encoders; 119 s
Candidate VLM checkpoint loadingNot requiredAbout 470 s (depends on storage and caching)
Total before selectionAbout 2.3 sAbout 1,070 s
Ranker fitting and final selectionAbout 0.03 s per targetUnder 0.5 s

The LLM calls of TC-ESR are made per target (five votes in practice, and up to fifteen until every metadata field has a strict majority). The 2.25 s is the measured time of five calls (0.45 s each) with this LLM, with which TC-ESR attains an SR of 4.83 %p (Table 16). All timings exclude the loading of the LLM (in both pipelines) and of MPNet. Checkpoint loading is listed separately because it depends on storage and caching and can be amortized when the checkpoints are reused.

Before selection, the ModelGPT/SWAB pipeline thus takes about 1,070 s (about 600 s without checkpoint loading), against about 2.3 s for TC-ESR; the saving comes from avoiding caption generation and feature extraction through every candidate VLM. We will add this breakdown to the paper.

Questions: Applications and practical usefulness LOVM is relevant when the task specification is available before target images can be used for model selection. For example, the category catalogue and image domain of a product-classification service can be specified before photographs from the intended collection process exist, and an organization that maintains a model repository with evaluations on source tasks can then choose an initial model from this specification alone. Other such cases are listed in item 2 of our common response.

Judged by the model they select, existing methods add little over a simple baseline: ModelGPT and SWAB (7.11 and 7.39 %p mean SR) are on par with ranking by ImageNet accuracy (7.23 %p), return their most frequent model in 95.0% and 80.9% of selections (Table 22), and are worse than the single best fixed model (5.52 %p; a reference that requires target accuracies). TC-ESR is the only compared method below that reference, selecting models that are 2.45 %p more accurate on average than those of ModelGPT (Table 1). Regret remains, so further improvement is needed: we see LOVM as a way to improve the initial model choice until target examples are available. We will add this application context and the limitations of the setting.

I thank the authors for their detailed rebuttal. For me, most of my concerns have been addressed. It sounds reasonable to me that "SR and ORNR are the primary metrics, and R5 and tau5 are retained as supplementary diagnostics of the predicted ranking".

Reviewer GJo9 mentioned the limited novelty in SR as a evaluation metric. I agree with the authors that, although SR is an established metric, introducing it to this specific task is interesting if previous works use other non-suitable metrics.

Reviewer ntVV and I both noted the potential limitation of the paper's setting. But previous works have been working on LOVM for years and established benchmarks. Although LOVM seems not very popular (The LOVM paper: 35 citations), I don't think we should blame this paper by the limitations in LOVM. The paper is still valuable, introducing new evaluation metrics and new methods, and the authors are encouraged to extend LOVM in the future.

So I will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer BycS

Thank you for reading our response and the other reviews, and for your comments on the points raised by the other reviewers. We have uploaded a revised version that includes the analyses from our response, with all changes marked in blue, and we are happy to answer any further questions during the discussion period.

Reviewer_GJo9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper revisits language-only vision–language model selection (LOVM) and argues that conventional ranking-based evaluation metrics are insufficient for assessing the quality of the selected model. The authors adopt simple regret (SR), introduce outlier-robust normalized regret (ORNR), and identify two sources of target-performance leakage in prior evaluation protocols. The paper further proposes Target-Conditioned Expected Source Regret (TC-ESR), a training-free selection method that aggregates source-task regrets using text-derived task similarities across visual domain, label semantics, and label granularity. Experiments involving 43 VLMs and 22 target tasks show that TC-ESR achieves the lowest mean SR and ORNR among the compared methods.

Critical strengths

S1.Important problem formulation.

This work convincingly demonstrates that ranking quality does not necessarily translate into effective model selection. Evaluating the performance of the top-ranked model is better aligned with the practical objective of LOVM.

S2.Evaluation protocol improvements

The identification and correction of two potential sources of target-performance leakage represent valuable contributions beyond the proposed algorithm.

Critical weaknesses

W1. Limited novelty of adopting simple regret as an evaluation metric

The paper argues that ranking-based metrics are misaligned with the practical objective of selecting the best-performing VLM and advocates simple regret (SR) as a more appropriate evaluation metric. While this motivation is convincing, SR is an established metric in model selection and related decision-making problems, rather than a novel evaluation measure introduced by this work. Therefore, the contribution primarily lies in applying an existing metric to the LOVM setting. Although this is a useful perspective, its methodological novelty is somewhat limited.

W2. Insufficient justification for the necessity of ORNR

The proposed Outlier-Robust Normalized Regret (ORNR) addresses the sensitivity of conventional range normalization to unusually weak candidate models, and the robustness experiments provide empirical support for this design. However, the paper does not sufficiently justify why normalized regret should be preferred over absolute regret when assessing practical model selection quality.

In particular, ORNR assigns different penalties to identical absolute accuracy gaps depending on the performance distribution of the candidate models. It is unclear why the same loss in target accuracy should be considered more or less severe solely because the candidate models exhibit different performance ranges. While the experiments demonstrate that ORNR is less sensitive to outliers than min–max normalization, this does not necessarily establish that ORNR better reflects the practical consequences of selecting a suboptimal model. A clearer motivation for the decision-making relevance of this normalization would strengthen the contribution.

Questions for authors
  • Q1. Since simple regret is an established metric in model selection, could the authors clarify what the adoption of SR contributes beyond applying an existing metric to the LOVM setting? In particular, is the main contribution the empirical demonstration that ranking-based metrics can be misaligned with selection quality, or does the proposed evaluation perspective provide additional insights that were not captured by prior model selection studies?
  • Q2. Could the authors further clarify the necessity and practical relevance of ORNR as an evaluation metric?
    • (a) Motivation for normalization. Figure 2(a) illustrates that identical or similar absolute regret values can represent different fractions of the candidate models' performance ranges. However, why should the same absolute accuracy loss be penalized differently depending on the performance distribution of the candidate models? Could the authors clarify what aspect of practical model selection quality ORNR captures that SR alone does not?
    • (b) Practical interpretation. Since the objective of LOVM is to select the model with the highest target accuracy, SR directly measures the accuracy lost due to a suboptimal selection. In contrast, ORNR additionally depends on the performance distribution of other candidate models. Could the authors provide a concrete model selection scenario in which ORNR offers a more meaningful assessment than SR?
    • (c) Evidence from robustness experiments. Table 20 demonstrates that ORNR is less sensitive than min–max normalization to perturbations of weak candidate models. However, this establishes robustness relative to min–max normalization rather than the practical necessity of normalization itself. Could the authors clarify how these robustness results support the use of ORNR alongside SR, rather than relying on SR alone?
    • (d) Interpretation of alternative normalization results. Table 19 shows that TC-ESR achieves the lowest normalized regret across 13 alternative denominator choices. While this supports the robustness of the method comparison to the normalization scheme, it does not necessarily establish that normalized regret is a more appropriate measure of selection quality. Could the authors clarify what additional evidence supports the adoption of ORNR as a primary evaluation metric?
4 comments
AuthorsOpenReview ↗

Response to Reviewer GJo9 (1/2)

We appreciate your valuable comments. Below we address what adopting SR contributes (W1/Q1) and why ORNR is reported together with SR (W2/Q2).

W1 / Q1. Contribution of adopting SR

We agree that SR is an established metric and do not claim novelty in its definition. That regret suits the evaluation of model search better than correlation-type scores was already argued for transfer learning by Renggli et al. (2022), whom we cite. What was missing is the consequence for LOVM, where both existing methods and the benchmark were still evaluated, and tuned, by R5R_5 and τ5\tau_5. To answer Q1 directly, the contribution is the empirical demonstration for LOVM together with two observations it exposed: the disagreement with ranking metrics is systematic, and prior methods select nearly the same model for every target.

In Table 5, R5R_5 and τ5\tau_5 disagree with SR in 25.7% and 43.1% of the per-target method comparisons, with mean SR differences of 3.32 and 2.46 %p. At the method level the relationship is inverted: ranking the six methods by mean R5R_5 and by mean SR gives a Kendall correlation of -0.83 (-0.47 for τ5\tau_5).

Ranking metrics also hide what the methods select. Prior methods choose nearly the same model for every target: INB and AvgRank select one model for all 22 targets, and ModelGPT and SWAB select their most frequent model in 95.0% and 80.9% of selections (Table 22), yet none of them attains a lower mean SR than the best fixed model (7.11-7.39 vs. 5.52 %p, Table 1). We will make this contribution clearer and distinguish it from the established definition of SR.

AuthorsOpenReview ↗

Response to Reviewer GJo9 (2/2)

W2 / Q2. Necessity and practical relevance of ORNR We do not propose ORNR as a replacement for SR. When the paper calls SR and ORNR its primary metrics, this separates them from the ranking metrics; between the two, SR is the measure of the absolute accuracy loss and ORNR is a relative complement that is always reported next to it. The ordering of the six methods in Table 1 is identical under either metric, so the advantage of the models selected by the submitted procedure is also seen in absolute SR.

Q2(a). Motivation for normalization You are right that the same accuracy loss is the same downstream cost, whatever the other candidates do, and ORNR does not say otherwise. What differs across targets is the scale: the robust performance range Δtrob\Delta^{\mathrm{rob}}_t varies from about 5 (STL10) to 71 (Diabetic Retinopathy) percentage points across the 22 targets, so an SR of 5 percentage points spans almost the entire robust range on the narrowest target but only about 7% of it on the widest.

SR and ORNR answer different questions. SR measures the consequence for the deployed model, the accuracy points lost. ORNR measures the selection method, the fraction of the spread among the non-outlier candidates that it gave up; a larger ORNR does not mean greater downstream harm. The distinction matters when results are aggregated: in a macro-average of SR, the tasks with a wide range can contribute most of the total, whereas ORNR averages the relative regret of each task with equal weight, which gives narrow-range tasks more influence. Neither weighting is neutral, so we report both and draw conclusions where they agree. We will revise our sentence about a "common scale" to state this explicitly.

Q2(b). A concrete use of normalization

Two concrete situations call for a normalized measure.

  • (i) Benchmarks whose targets use different metrics. The 13 targets without class names added for Reviewer ntVV (Table R1) are scored by recall@1, accuracy, or rank correlation; SR cannot be averaged across them, while ORNR can.
  • (ii) Reading a single outcome. INB loses a similar amount of accuracy on GTSRB (SR 8.73 %p) and on DMLab (9.06 %p), but this is 33.8% of the robust range on GTSRB and 98.5% on DMLab, where the selected model is about as weak as the weakest non-outlier candidate (Table 18).

Q2(c). Evidence from the robustness experiment

The motivation for normalization and the choice of a robust denominator are separate questions, and Table 20 addresses the latter: when only the weakest candidate is degraded, the mean change in normalized regret is 74.0%-82.0% smaller under ORNR than under min-max normalization. This supports using a robust range; as the reviewer notes, it does not establish the need for normalization itself, and we will clarify this.

Q2(d). Interpretation of the alternative denominators

We agree that Table 19 does not establish that normalized regret is the more appropriate measure, and we do not claim that it is; it shows only that the comparison between methods does not depend on the denominator. SR remains our measure of the absolute accuracy loss, and ORNR is reported next to it for the aggregation reason given under Q2(a). Normalizing regret by a per-task range before averaging across datasets is established practice in hyperparameter-optimization benchmarks (e.g., HPO-B; Pineda Arango et al., 2021); ORNR replaces the minimum with Tukey's lower whisker, so that a single degenerate candidate cannot set the scale.

We will make the distinct roles of Tables 19 and 20 explicit in the revision.

Thank you for the detailed responses and additional clarifications. My main concerns have been substantially addressed, particularly through the clearer positioning of SR and ORNR. I appreciate the paper's meaningful problem formulation and improvements to the evaluation protocol. However, I still consider the methodological novelty relatively limited. Therefore, I maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer GJo9

Thank you for the follow-up and for your comments on the positioning of SR and ORNR. The revised paper reflects them in Section 3.3 and Appendix E.4, with changes marked in blue.

On methodological novelty, we agree that SR itself is a standard measure that we adopt rather than propose; ORNR, the two leakage corrections, and TC-ESR are our contributions. Our earlier response covered only the metrics, so we briefly add where we see the new part of the method. ModelGPT and SWAB select their most frequent model in 95.0% and 80.9% of their selections (Table 22), so they behave almost like a fixed choice, and no fixed choice can go below the SR of the best fixed model, the single candidate with the highest mean accuracy across the targets (5.52 %p; Table 1). TC-ESR instead estimates, from text alone, how similar each source task is to the target along three axes (visual domain, label semantics, and label granularity) and weights each candidate’s regret on the source tasks by this similarity, without training a ranker. The gain comes from this similarity: permuting the same weights across the source tasks raises SR to 9.08 %p, worse than uniform weighting at 5.83 %p (Table 3), and class-name similarity alone gives 6.52 %p (Table 4), against 4.66 %p for the three axes. As a result, TC-ESR varies its selection with the target (8 distinct models, the most frequent in 45.5% of its selections; Table 22) and is the only compared method below the best fixed model.

Thank you again for your time and for the constructive discussion.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response to All Reviewers

Dear Reviewers,

We thank all reviewers for their time and constructive comments. We are encouraged that the reviewers found that the paper clearly identifies the limitations of the existing evaluation (ntVV, GJo9), that the metrics and method provide new insights (BycS), and that the corrections of target performance leakage are a valuable contribution (GJo9).

Following the comments, we ran additional experiments and analyses. We summarize them here; details are in the individual responses.

1. Beyond zero-shot classification (ntVV W1; Table R1). We added 13 targets without class names and VQAv2. With an answer vocabulary in the role of the class names (VQAv2), TC-ESR applies unchanged. Without any class names it conditions on the domain only; on these 13 targets its mean ORNR is the lowest, but the difference is within noise, so we claim no advantage without class names and will state this as a limitation.

2. Practical relevance of LOVM and the class-list assumption (ntVV Q1, BycS Q). For zero-shot classification, the class names are the minimum input needed to deploy any candidate VLM, whereas labeled target images are not needed. LOVM addresses this starting state, for example when the taxonomy is fixed before images are collected (e.g., a product catalogue), when images cannot leave the site, or when candidates are shortlisted before their checkpoints are downloaded.

3. Evaluation should reflect how the selection result is used (BycS W1; Table R2). As in the submission, SR and ORNR remain the primary metrics and R5R_5 and τ5\tau_5 supplementary diagnostics of the predicted ranking. For the case where the top-kk models are evaluated further, we report SR@kk; for the case where they are used together, the top-5 ensembles of the prior methods are more accurate than that of TC-ESR (62.4-62.7% vs. 60.8%), so we limit our claims to the deployment of a single model.

4. Computational cost (BycS W2; Table R3). On CIFAR100, with the same local LLM and GPU for both, the preprocessing for a new target takes about 2.3 s for TC-ESR against about 1,070 s for ModelGPT and SWAB (about 600 s without candidate checkpoint loading).

5. What adopting SR contributes, and why ORNR is reported with it (GJo9). We clarify the empirical contribution and the distinct roles of SR and ORNR.

Please let us know if anything remains unclear. We would be glad to provide further clarification during the discussion period.

Best regards, The Authors

Dear Reviewers,

We are currently in the discussion phase. The authors have responded to the comments, and I would greatly appreciate it if you could take a moment to check the rebuttal and leave any follow-up comments or questions. If the response has addressed your concerns, please also consider updating your review accordingly.

Your feedback is very valuable to both the authors and the decision process. Thank you again for your help.

Best regards, AC

AuthorsOpenReview ↗

Revised paper uploaded

Dear Reviewers,

We have uploaded a revised version of the paper. All changes are marked in blue, and the main text remains within the 10-page limit. The changes are listed below with the comments they respond to. The table numbers of the original submission (Tables 1-25) are unchanged, and the new tables are Tables 26-30.

Main text

  • Introduction: when the LOVM setting arises, with use cases (ntVV Q1, BycS Questions); SR is an established measure and our contribution is the result for LOVM, with a citation of Agostinelli et al. (2022) (GJo9 W1/Q1, ntVV).
  • Sections 3.1 and 5.4: SR and ORNR are the primary metrics, and R5R_5 and τ5\tau_5 are supplementary diagnostics of the predicted ranking (BycS W1).
  • Section 3.3: SR and ORNR answer different questions, and neither way of aggregating is neutral; this replaces the sentence about a "common scale" (GJo9 Q2(a)).
  • Conclusion: the claims are stated for the deployment of a single model and for LOVM on zero-shot classification (BycS W1, ntVV W1).

Appendix

  • Appendix A: relation of SR and ORNR to Rel@1 of Agostinelli et al. (2022) (ntVV).
  • Appendix E: the distinct roles of the two analyses of the robustness of ORNR (GJo9 Q2(c), Q2(d)).
  • Appendix G: computational cost with stage timings (BycS W2; Table R3 of our response).
  • Appendix H: SR@kk and the top-5 ensemble (BycS W1; Table R2 of our response).
  • Appendix I: targets without class names and VQAv2, with per-target results (ntVV W1; Table R1 of our response).
  • Appendix J: limitations (scope, dependence on class names, single-model deployment).

We thank the reviewers again for their comments.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.