acceptodds
Under review as a conference paper at ICLR 2027

SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks

Abstract

Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost. Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios, and employs training objectives that dilute multi-positive signals via softmax normalization without incorporating cost. We address these limitations with three contributions: (1) VLM-ExecRouterBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains with 11 candidate models spanning nearly two orders of magnitude in pricing; (2) SCOPE-Router, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration (random/diagnostic/diversity sampling), enabling new models to join routing without retraining; (3) CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance targets through per-pair independent scoring, eliminating multi-positive dilution while regularizing queries with similar routing preferences to be closer in the routing space. Empirically, SCOPE-Router achieves the best Rank Score on all three benchmarks, surpassing the runner-up by 1.84 points under OOD settings and by 6.75 points under doubly OOD open-set evaluation. When applied to four diverse routers, CRM+RCCR improves Rank Score by 1.25–6.21 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Open-set, VLM, Router, Cost-aware

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_47rROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces VLM-ExecRouterBench, which records correctness and token-based inference costs for 11 models across Code, Agentic, and Search tasks. SCOPE-Router matches queries to model profiles built from calibration queries and their outcomes, allowing new models to be added without retraining the router. Its training objective combines Cost-aware Relevance Matching (CRM) and Routing-Consistency Contrastive Regularization (RCCR) and can also be used with other router architectures. SCOPE-Router achieves the highest reported mean Rank Score on three benchmarks. Further experiments evaluate generalization to held-out tasks and models, and the transfer of CRM+RCCR to four other routers.

Critical strengths
  1. Open-set capability. I appreciate that new VLMs can be added through calibration without adding classifier outputs or retraining the router. This makes the design useful when the candidate pool changes. The profiles are similar to specifications in the learnware paradigm: they characterize model capabilities to support matching and reuse. The authors may find the related work Constructive Specification for Plug-and-Play Learnware Agents (KDD 2026) useful, as it also uses capability profiling and matching for agent selection.

  2. Broad experimental evaluation. The evaluation spans three benchmarks and multiple baselines, with ablations on the training objective, model profiles, calibration strategy and set size, and encoders. Applying CRM+RCCR to four other router architectures also provides useful evidence that the objective can benefit methods beyond SCOPE-Router.

Critical weaknesses
  1. Weak connections among the three main contributions. The title's emphasis on "VLM Routing for Execution-Oriented Tasks" led me to expect a clearer account of how these tasks shape the method. The profile construction, query–profile matching, and CRM+RCCR appear broadly applicable to model routing; I could imagine presenting the method independently of VLM-ExecRouterBench. I would like the authors to explain how the benchmark's task properties or empirical findings inform these design choices. Evaluating SCOPE-Router and CRM+RCCR with text-only candidate models would also help establish their broader applicability and clarify which benefits, if any, are specific to VLM routing.

  2. Limited evidence for open-set generalization and scalability. Open-set capability is central to the paper, but Table 7 reports only one split of six seen and five unseen models. I would like to see unseen models added progressively to larger candidate pools, with the router kept fixed. With only six profiles seen during training, I am unsure whether ProfileMLP learns a transferable capability representation or mainly fits associations with the seen models' overall strength and price. Multiple held-out model splits, entire model families held out, and newcomers with local strengths absent from the training pool would provide stronger evidence.

    I also suggest comparing SCOPE-Router with closed-set routers, allowing the latter to be retrained after each model addition. I would compare accuracy and end-to-end latency, including routing, on the same expanded candidate pool and test queries. The comparison should report the time and compute needed for new-model evaluation, profile construction and encoding, and router retraining. This would show whether SCOPE-Router can reduce the cost of adding models while retaining accuracy close to that of the retrained baselines. If additional experiments resolve these concerns, I would consider raising my rating to 4.

Minor presentation suggestions:

  • Explain the intuition behind RCCR more clearly in Sec. 2.5, particularly why overlap between relevance vectors provides a useful training signal for query embeddings. A small example showing how the contrastive weights are constructed and what they encourage would help.
  • Define eie_i at first use in Sec. 2.3.2, before its construction in Sec. 2.4.
  • Clarify the correspondence between ym,c~m\mathbf{y}_m,\tilde{\mathbf{c}}_m in Eq. (3) and Yi,m,c~i,mY_{i,m},\tilde c_{i,m} in Eq. (5), including the calibration indices. State which samples and models determine the bounds for cost min-max normalization.
  • Replace the unusual indicator glyph in Eq. (9) with a clearly rendered 1Yi,m=1\mathbf 1\\{Y_{i,m}=1\\}.
  • Define α\alpha in Eq. (9) and report its value.
Questions for authors
  1. What do the authors consider the main reason SCOPE-Router works: the model profiles, the learned query–profile embeddings, the training objective, or their interaction? Which results best support that explanation?

  2. Does Table 7 use only the five held-out models or all eleven candidates? Please clarify how this relates to the strict unseen-pool setting in Appendix C, and report sensitivity to random seeds and model splits. See Weakness 2 for my other questions and suggested experiments on open-set evaluation.

1 comment
Reviewer_47rROpenReview ↗

Follow-up questions on the open-set experiments

Thanks for the detailed reply and the new experiments. The gradual pool expansion, three held-out model splits, and family holdouts address much of my concern about how well SCOPE handles new models without retraining. I’m leaning toward raising my score, but I still have three questions:

  1. W2-1: Choice of retraining baseline. My understanding is that UniRoute-KM can add new models without retraining. Why did you use a retrained version in W2-1? What exactly did you refit after each model addition? I had in mind methods that need a training step for each new model, such as retraining the closed-set MLP in W2-4 or fitting a new model embedding in EmbedLLM.

  2. W2-2: Profile ablation on unseen models. The global8 ablation is helpful. Since the mixed pool includes seen models, I’m still not sure how well ProfileMLP generalizes to unseen models. Could you run a small follow-up using the same two checkpoints on the split_1 test queries, with only the five unseen models in the candidate pool?

  3. W2-4: Reusing existing profiles. The timing table includes converting and encoding all profiles after each expansion. What does this step do? Do the profiles or embeddings of existing models change when a new model joins?

Sorry for following up so late. I’d appreciate any clarification you have time to share before the discussion closes.

Reviewer_Lp7NOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Authors present SCOPE router, which is a cost aware routing approach that matches query representations to model representations for performing query routing. Authors use correctness and cost measurements on calibration queries to build model behavior and semantic vectors. Training combines independent sigmoid BCE with cost adjusted targets, and also use a contrastive loss regularization technique.

Evaluation demonstrates improved performance.

See weaknesses for a detailed explanation.

Critical strengths

Problem identification of extending routers to new models is important, but is already studied in literature and the paper fails to discuss it (see attached references in weaknesses). Benchmark contribution can be useful. Empirical evaluation seems reproducible.

Critical weaknesses
  1. There is no discussion of broad and well recognized existing literature on Model Embeddings (including ICLR 25 Spotlight Embedllm, ICLR 26 Oral LLMDNA), which is very similar to the present ideas on model behavior and semantic vectors.

  2. The presented idea of model behavior vector through correctness and cost etc is not new, KMeans Router performs something very similar at the cluster level (vector of average performances over say K clusters can be called a model behavior vector -> generalize this to clusters with single queries over the calibration data).

  3. RouterDC's contrastive loss can be easily extended to accuracy and cost aware scenarios, and the submission overstates their contribution in this aspect.

  4. The queries required in the calibration data need to be same across all models, and the presented approach fails when all models are not evaluated on same queries (these queries and their informativeness about model performance can also become stale with time).

  5. The evaluation is too limited to establish robustness and generalization to a variety of tasks and evolving query distributions.

Relevant papers not discussed in presented submission:

LLM DNA: Tracing Model Evolution via Functional Representations Zhaomin Wu, Haodong Zhao, Ziyang Wang, Jizhou Guo, Qian Wang, Bingsheng He

EmbedLLM: Learning Compact Representations of Large Language Models Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, Kannan Ramchandran

Learning Compact Representations of LLM Abilities via Item Response Theory Jianhao Chen, Chenxu Wang, Gengrui Zhang, Peng Ye, Lei Bai, Wei Hu, Yuzhong Qu, Shuyue Hu

LOCUS: Low-Dimensional Model Embeddings for Efficient Model Exploration, Comparison, and Selection Shivam Patel, William Cocke, Gauri Joshi

Questions for authors

Mentioned in weaknesses.

Please position the work properly and acknowledge and compare existing literature. Please do literature survey to better position contributions.

Reviewer_qLA2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains. In addition they introduce a new dual tower routing architecture, as well as an architecture agnostic cost aware objective that pulls together suitable queries in the routing space. These all fill relevant gaps in the literature for VLM routing, particular the new benchmark. They validate the performance of their new routing architecture + objective on the new benchmark as well as two other prior benchmarks in the VLM routing space.

Critical strengths
  1. I really appreciate the gap filled in the literature by this new execution driven VLM benchmark as the strongest contribution in my opinion.

  2. In an era of AI written papers, this one is clear, flows smoothly, and has little to no jargon that all make it a pleasure to read.

  3. The performance of ScopeRouter on the 3 benchmarks is strong, and I appreciate the use of 11 models to make this a reasonable scale study.

While I have raised weaknesses - that said, I think this is a good starting point benchmark for future work to improve upon.

Critical weaknesses
  1. For a benchmark paper, I would have expected human validation of the dataset and this was missing. The addition of even a small scale human annotation during rebuttal period would increase my confidence in the results.

  2. There is a little too much jammed into the paper as equal contributions. To be honest, there is nothing that novel about a dual tower architecture or the new cost objective proposed. They don't harm the paper per se, but could have been de-emphasized a little to foreground the VLM execution benchmark itself as the strongest contribution.

  3. The majority of its agentic examples are existing single-question VQA tasks from prior benchmarks augmented with optional image manipulation and OCR tools, which draws some doubt on the realism of these scenarios - especially without human validation.

  4. only 1.8% of the benchmark is search (see Appendix B.3, lines 868–872 which says the dataset is about 34k examples and Appendix A, lines 706–708 which says search contributes 626 examples). This casts some question on the selling of the search component so prominently.

Questions for authors

If you can respond to the weaknesses I raised, particularly on the realism of the agentic examples, as well as a small scale human annotation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.