acceptodds
Withdrawn from ICLR 2027

MMR-Bench V2: From Visual Information to Routing Decisions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models vary in capability and inference cost, motivating query-dependent routing. When do model diversity and visual information translate into better model choices? MMR-Bench V2 expands MMR-Bench to 44 models and 18 benchmarks, recording outcomes, execution status, and reference costs for every query–model pair. We examine model-pool composition, query information, and target-task adaptation. At the pool level, expansion increases answer coverage faster than routing quality. Compact global pools retain the quality of unrestricted task-conditioned policies within a predefined tolerance at medium and high validation budgets on the evaluated suite. At the information level, matched frozen-feature comparisons show that images can improve outcome prediction while worsening selection. Most prediction gains concern the mean success rate across candidates for a given query. Visual effects depend on task composition, and preferences for models that generally perform well on each task provide strong controls when those tasks are represented in training. For known-task adaptation with matched prior weights, concentrating target labels on candidates screened using other tasks can outperform tested broader uniform and sequential allocations at the same label budget. These findings favor learning task-level preferences and evaluating larger pools and richer inputs by their additional selection gains.

Primary area:
datasets and benchmarks
Keywords:
multimodal large language models, model routing, benchmark, cost-aware inference, model selection, cross-dataset generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4CubOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces MMR-Bench V2, a multimodal model-routing benchmark covering 44 models, 18 benchmarks, and corresponding model-query-answer samples. Based on MMR-Bench V2, this paper compares different routing methods and investigates how the model pool composition, visual information availability, and different task setups affect routing performance. In summary, this paper finds that there is still a substantial room for further improvement and exploration.

Critical strengths
  1. The presented benchmark, MMR-Bench V2, has broad model and task coverage, containing full paired records.
  2. The authors configure a careful evaluation setup, including introducing known-task fixed baselines, presenting top-k coverage analysis, and so on.
  3. The authors conducted insightful analysis to explore the effect of model-pool composition, visual information, and task setups.
Critical weaknesses
  1. Poor presentation: the presentation of this paper is unsatisfactory, making it difficult to read and understand.
  • The task definition and descriptions in Section 3.1 are not sufficiently clear.
  • The paper is not well structured, and the paragraph organization is sometimes unreasonable. The authors are encouraged to add more \paragraph{} headings to clarify the structure and purpose of different parts.
  • Some statements in the paper are not written fluently, which further affects readability.
  1. The benchmark setting is overly fixed and does not sufficiently consider practical scenarios
  • It is unclear how the benchmark and the conclusions would change under non-binary outcomes.
  • The treatment of execution failures is questionable. Execution failures should not be included in the primary routing evaluation, because they may result from randomness or inappropriate configurations. Moreover, it is difficult for a router to predict execution failures, and handling such failures should arguably not be part of the router’s responsibility.
  • Cost is currently treated mainly as an auxiliary metric, while most of the analysis still focuses on quality. This setting is not fully satisfactory. The problem could instead be formulated as a constrained optimization problem, such as maximizing quality under a fixed budget, or evaluated using a metric that jointly considers quality and cost.
  • Although the benchmark has broad coverage, it is unclear whether including 44 models is meaningful when the main evaluation focuses on quality. According to the results in Section 6, as well as common practical usage, users are likely to consider only a small subset of these models.
  1. The research contribution is limited
  • The benchmark does not appear to have a sufficiently distinctive design. It seems mainly to be a scale-up of prior work, and it is unclear what is specifically novel about MMR-Bench V2 in the multimodal setting. The results in Section 6 even show that text-only methods can outperform methods using multimodal inputs.

  • The potential impact of this benchmark may be limited. Models are developing rapidly, and within a few months, newly released models may substantially outperform the older models included in the benchmark. A static benchmark may therefore be unsuitable for evaluating model routers. As the paper’s own analysis suggests, simply selecting a newer and stronger model may often be sufficient.

  • Although the paper presents several analyses, it does not clearly indicate how future work should proceed based on these findings.

  1. The analysis of visual information may not be sufficiently robust

The paper’s conclusion regarding the effect of visual information may not be sufficiently well supported, as the aggregate degradation is heavily influenced by MathVerse.

Questions for authors

see weakness

Reviewer_jCE5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents MMR-Bench V2, extending MMR-Bench, a new benchmark that evaluates multimodal model routing, i.e., choosing, given a multimodal query (image + text), which multimodal LLM should answer it among a pool of models. Compared to V1, it extends it from 7 task-specific benchmarks to 18, and from 10 models to 44. Thus, the main contribution is a large table featuring 44×25,50444\times25,504 entries, where each ijij entry records whether model ii correctly answers query jj. This allows fast and cheap evaluation of routing algorithms without recomputing model answers every time. Results on MMR-Bench V2 reveal that a task-level baseline (i.e., predicting the task label and using the best model for that task) is a stronger baseline than using a single fixed model. Additionally, the paper shows that large model pools improve question coverage, while resulting routing performance improves only marginally.

Critical strengths

The proposed task-level baseline is quite compelling. The authors show that predicting the benchmark to be tested and using the best model for that benchmark beats all routing baselines. Thus, while it is not the central argument of the paper, it proposes a baseline that benefits from a model pool rather than a single fixed model.

Critical weaknesses

To me, the paper can have great potential in the field, as it presents a new strong baseline and a large amount of interesting findings. However, the writing quality is very low. Reading the paper was a tedious process, as it is extremely dense with information, while the sentences are convoluted and use heavy jargon, often without introducing it. The paper reads as an inline list of accumulated rebuttal material without clear connection between sentences and paragraphs. Additionally, a large portion of the reported numbers are in the appendix. Finally, while I would not normally flag writing quality as the main critical weakness, I feel this submission, at the current state, requires major manuscript revision before reaching conference quality.

Questions for authors
  1. When comparing routing with and without images, it turns out that most of the loss in routing quality from including images is explained by MathVerse alone. Could the authors provide an intuition for why this happens?
  2. The paper evaluates cost only at three fixed budgets. It never shows per-model cost vs. quality, the Pareto-efficient set, or router trade-off curves across a budget sweep. Wouldn't it make sense to propose a few model pools with different cost tiers such that models within a pool are Pareto efficient in terms of performance vs. cost under a specific budget?
Reviewer_KWEuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MMR-Bench V2, covering 44 models and 18 multimodal benchmarks. It studies model-pool size, visual inputs, and adaptation with limited labels. The experiments separate answer availability, prediction accuracy, and final model selection.

Critical strengths

The evaluation includes strong task-based controls and several robustness checks. It examines execution failures, scoring choices, price changes, and label allocation. These controls make the findings more informative than a simple comparison with one fixed model.

Critical weaknesses
  1. Limited new insight. Several main findings revisit known issues in model routing. The paper does not clearly explain what important new insight it adds.

  2. The benchmark includes more tasks and models. However, Appendix D.3 does not show a stable improvement in method selection. The practical value of this expansion remains unclear.

  3. The findings depend strongly on the evaluated tasks. For example, removing MathVerse changes LatentRouter’s mean visual effect from negative to slightly positive. This limits the generality of the findings.

Questions for authors
  1. What causes the recorded execution failures? Are they concentrated in particular models, tasks, or collection periods?

  2. How much router cost can be added before routing loses its cost advantage over a fixed-model policy?

  3. How often does source-based screening exclude the best model for the target task? How much accuracy is lost for this reason?

  4. Does the task-profile advantage remain if dataset-specific wording and formatting are removed from the router inputs?

  5. For outcomes scored by an LLM judge, how well do the labels agree with human assessment on a small sample?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.