- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces VLM-ExecRouterBench, which records correctness and token-based inference costs for 11 models across Code, Agentic, and Search tasks. SCOPE-Router matches queries to model profiles built from calibration queries and their outcomes, allowing new models to be added without retraining the router. Its training objective combines Cost-aware Relevance Matching (CRM) and Routing-Consistency Contrastive Regularization (RCCR) and can also be used with other router architectures. SCOPE-Router achieves the highest reported mean Rank Score on three benchmarks. Further experiments evaluate generalization to held-out tasks and models, and the transfer of CRM+RCCR to four other routers.
Critical strengths
-
Open-set capability. I appreciate that new VLMs can be added through calibration without adding classifier outputs or retraining the router. This makes the design useful when the candidate pool changes. The profiles are similar to specifications in the learnware paradigm: they characterize model capabilities to support matching and reuse. The authors may find the related work Constructive Specification for Plug-and-Play Learnware Agents (KDD 2026) useful, as it also uses capability profiling and matching for agent selection.
-
Broad experimental evaluation. The evaluation spans three benchmarks and multiple baselines, with ablations on the training objective, model profiles, calibration strategy and set size, and encoders. Applying CRM+RCCR to four other router architectures also provides useful evidence that the objective can benefit methods beyond SCOPE-Router.
Critical weaknesses
-
Weak connections among the three main contributions. The title's emphasis on "VLM Routing for Execution-Oriented Tasks" led me to expect a clearer account of how these tasks shape the method. The profile construction, query–profile matching, and CRM+RCCR appear broadly applicable to model routing; I could imagine presenting the method independently of VLM-ExecRouterBench. I would like the authors to explain how the benchmark's task properties or empirical findings inform these design choices. Evaluating SCOPE-Router and CRM+RCCR with text-only candidate models would also help establish their broader applicability and clarify which benefits, if any, are specific to VLM routing.
-
Limited evidence for open-set generalization and scalability. Open-set capability is central to the paper, but Table 7 reports only one split of six seen and five unseen models. I would like to see unseen models added progressively to larger candidate pools, with the router kept fixed. With only six profiles seen during training, I am unsure whether ProfileMLP learns a transferable capability representation or mainly fits associations with the seen models' overall strength and price. Multiple held-out model splits, entire model families held out, and newcomers with local strengths absent from the training pool would provide stronger evidence.
I also suggest comparing SCOPE-Router with closed-set routers, allowing the latter to be retrained after each model addition. I would compare accuracy and end-to-end latency, including routing, on the same expanded candidate pool and test queries. The comparison should report the time and compute needed for new-model evaluation, profile construction and encoding, and router retraining. This would show whether SCOPE-Router can reduce the cost of adding models while retaining accuracy close to that of the retrained baselines. If additional experiments resolve these concerns, I would consider raising my rating to 4.
Minor presentation suggestions:
- Explain the intuition behind RCCR more clearly in Sec. 2.5, particularly why overlap between relevance vectors provides a useful training signal for query embeddings. A small example showing how the contrastive weights are constructed and what they encourage would help.
- Define at first use in Sec. 2.3.2, before its construction in Sec. 2.4.
- Clarify the correspondence between in Eq. (3) and in Eq. (5), including the calibration indices. State which samples and models determine the bounds for cost min-max normalization.
- Replace the unusual indicator glyph in Eq. (9) with a clearly rendered .
- Define in Eq. (9) and report its value.
Questions for authors
-
What do the authors consider the main reason SCOPE-Router works: the model profiles, the learned query–profile embeddings, the training objective, or their interaction? Which results best support that explanation?
-
Does Table 7 use only the five held-out models or all eleven candidates? Please clarify how this relates to the strict unseen-pool setting in Appendix C, and report sensitivity to random seeds and model splits. See Weakness 2 for my other questions and suggested experiments on open-set evaluation.
Follow-up questions on the open-set experiments
Thanks for the detailed reply and the new experiments. The gradual pool expansion, three held-out model splits, and family holdouts address much of my concern about how well SCOPE handles new models without retraining. I’m leaning toward raising my score, but I still have three questions:
W2-1: Choice of retraining baseline. My understanding is that UniRoute-KM can add new models without retraining. Why did you use a retrained version in W2-1? What exactly did you refit after each model addition? I had in mind methods that need a training step for each new model, such as retraining the closed-set MLP in W2-4 or fitting a new model embedding in EmbedLLM.
W2-2: Profile ablation on unseen models. The
global8ablation is helpful. Since the mixed pool includes seen models, I’m still not sure how well ProfileMLP generalizes to unseen models. Could you run a small follow-up using the same two checkpoints on thesplit_1test queries, with only the five unseen models in the candidate pool?W2-4: Reusing existing profiles. The timing table includes converting and encoding all profiles after each expansion. What does this step do? Do the profiles or embeddings of existing models change when a new model joins?
Sorry for following up so late. I’d appreciate any clarification you have time to share before the discussion closes.