acceptodds
Under review as a conference paper at ICLR 2027

CARE: Label-Free Cost-Aware Online Routing for Zero-Shot VLM Classification

Abstract

Contrastive vision–language models (VLMs) exhibit complementary strengths and widely different computational costs in zero-shot classification, making a single fixed model suboptimal across diverse inputs. We introduce CARE, a label-free online framework for per-sample routing among heterogeneous CLIP-style VLMs under a global computational budget. Given a stream of unlabeled inputs, CARE dynamically selects one expert for each sample without requiring labeled calibration data or a pre-deployment reference set. To enable cross-model comparison, CARE standardizes logits on a per-sample basis and uses the resulting confidence as a label-free proxy reward. Routing is formulated as a contextual bandit with a dynamic dual update that balances estimated utility and computational cost while enforcing the global budget. Experiments across eight zero-shot classification benchmarks show that CARE achieves strong accuracy–cost trade-offs compared with individual models, full-expert methods, and routing baselines that use additional calibration or reference data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Zero-Shot Classification, Online Model Routing, Cost-Aware Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QdoQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online selection among heterogeneous vision-language models under a global budget, without target labels, calibration data, or a pre-deployment reference set. CARE standardizes the class logits of queried models on a per-sample basis and uses the resulting confidence as a label-free proxy reward. It learns an expert-selection policy using image features from a base model, a contextual bandit, and dynamic budget control. Experiments demonstrate competitive accuracy–cost trade-offs under the paper’s definition of additional routing cost and support the usefulness of standardization for selecting among heterogeneous models. The theoretical analysis concerns budget-constrained optimization of confidence-based proxy rewards.

Critical strengths
  1. The problem is practically relevant, and the design aligns with the deployment constraints. CARE jointly addresses the absence of label feedback, differences in expert output scales, and limited inference budgets by combining confidence processing, online utility learning, and budget control. It progressively learns an expert-selection policy without pre-deployment reference data, and the overall results demonstrate the practical potential of this approach.
  2. The role of standardization receives fairly direct experimental support. Standardization improves performance overall in both confidence-based selection with access to all experts and online CARE. Comparisons across confidence metrics further support the finding that cross-model scale mismatch affects selection quality. Experiments with mixed domains and different input orderings provide additional insight into the framework’s behavior.
Critical weaknesses

The incremental benefit of image-level context has not been sufficiently isolated. The paper motivates per-sample routing through experts’ complementary strengths across inputs. Expert-utilization patterns and mixed-domain experiments support adaptation across domains, but do not sufficiently distinguish gains from identifying the strongest experts for each domain from gains that additionally exploit expert complementarity across images within the same domain. Without a control that removes image features while retaining the same proxy reward, online learning framework, and budget mechanism, the added value of fine-grained contextual decisions remains unclear.

Questions for authors
  1. How do standardized rewards lead to useful expert preferences? Existing ablations already support the usefulness of standardization for expert selection. For binary classification, the authors explain that the nonzero regularization term preserves dependence on the original logit margin. Could existing reward and routing records further clarify how these differences guide expert selection during online learning? This question seeks to understand the source of the reported effects.
  2. To what extent do CARE’s gains depend on image context? Particularly within single-domain streams, does CARE retain a clear advantage over a non-contextual policy that learns only overall expert preferences, when the proxy reward, exploration rules, and budget control are held fixed? Can existing results or routing records further distinguish the contributions of cross-domain specialization and within-domain, per-sample complementarity?
1 comment

I thank the authors for their responses and the additional control experiment. However, given the marginal and inconsistent gains from image context (as discussed above), I do not think the work is yet at the level required for ICLR. I therefore maintain my original score.

Reviewer_fd1HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CARE, a label-free, cost-aware online routing framework for zero-shot VLM classification. It uses a frozen base CLIP encoder to extract image features, learns linear predictors of confidence-based rewards for candidate experts, and employs a contextual bandit with budget constraints to select an expert for each input. The method is evaluated on eight classification benchmarks, with additional ablation studies and a theoretical regret analysis.

Critical strengths

The paper considers a potentially relevant problem: selecting among heterogeneous VLMs under computational constraints without labeled calibration data. The evaluation spans eight datasets and includes comparisons with several routing alternatives. However, I do not find the demonstrated benefits sufficient to overcome the concerns below.

Critical weaknesses
  1. Violation of submission formatting requirements: The manuscript appears to use substantially reduced page margins. This represents a serious deviation from the official ICLR formatting requirements and warrants consideration for desk rejection.

  2. Limited practical significance and insufficient empirical gains: The practical necessity of the proposed framework is unconvincing across different deployment scenarios. For a fixed, known domain, directly using an appropriate specialized VLM can be simpler and even outperform CARE, as demonstrated by the results on PCam. For mixed-domain settings where task-specific class vocabularies are already available (as in Appendix H), straightforward task-aware expert selection or lightweight domain-level routing could be sufficient, yet these alternatives are not adequately evaluated. For genuinely unknown or shifting domains, the method relies on potentially unreliable confidence feedback, while its ability to adapt to unseen distributions remains largely untested. Moreover, the paper does not establish whether CARE learns meaningful sample-level expert complementarity beyond coarse domain-level selection. Overall, the additional complexity is not justified by sufficiently compelling empirical improvements or new insights.

  3. Unreliable confidence-based routing objective: Higher confidence does not necessarily imply higher accuracy, especially across heterogeneous VLMs. The router may consistently favor overconfident but inaccurate experts, and the paper provides insufficient evidence addressing this fundamental issue.

  4. Limited relevance of the theoretical analysis: The extensive mathematical derivations largely revisit established techniques and provide regret bounds for surrogate confidence rewards rather than actual classification accuracy. Much of the analysis feels unnecessarily elaborate, giving the impression of mathematical formalism added for presentation rather than substantive theoretical insight. Consequently, it does little to justify the proposed method or address its fundamental limitations.

Questions for authors

Please refer to the Critical Weaknesses.

Reviewer_CccrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CARE, a label-free and cost-aware online routing framework for zero-shot classification with heterogeneous vision-language models. The proposed method dynamically selects one VLM for each input under a global computational budget. The authors report experiments on eight zero-shot classification benchmarks and claim favorable accuracy-cost trade-offs compared with individual models and alternative routing approaches.

Critical strengths

(1) The paper addresses an important challenge in VLMs: balancing classification performance and computational cost. The label-free setting is particularly important because obtaining ground-truth labels may be difficult in real-world applications.

(2) The authors test CARE on eight datasets, including natural images, medical images, and remote sensing images. They also use both general-purpose and specialized VLMs, which makes the evaluation more convincing.

Critical weaknesses

(1) Most parts of CARE, such as logit standardization, contextual bandits, and dynamic budget updates, have already been studied. The main contribution seems to be putting these existing techniques together.

(2) In Table 2, CARE-60 achieves slightly lower accuracy than Neural-LinUCB, but at a lower average cost. However, since Neural-LinUCB does not have the same global budget constraint, it is difficult to make a fair comparison between the two methods.

(3) The paper uses log-normalized FLOPs and does not include the base model's computation in the reported routing cost. Since efficiency is one of the main goals of this paper, using actual FLOPs or end-to-end inference time is important.

(4) Figures 1 and 2 contain many small plots, and some labels and legends are quite small. This makes it difficult to compare the results across different datasets.

Questions for authors

(1) In Table 2, CARE-60 achieves slightly lower accuracy than Neural-LinUCB, but at a lower average cost. Could the authors compare these methods under the same computational budget to make the comparison fairer?

(2) Since the paper uses log-normalized FLOPs and does not count the base model's computation in the reported cost, could the authors provide results using actual FLOPs or end-to-end inference time?

(3) The paper shows that confidence is related to prediction accuracy, but how well does it work when different models disagree on the same image? Could the authors provide more analysis on whether standardized confidence can reliably select the better model in these cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.