- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces MMR-Bench V2, a multimodal model-routing benchmark covering 44 models, 18 benchmarks, and corresponding model-query-answer samples. Based on MMR-Bench V2, this paper compares different routing methods and investigates how the model pool composition, visual information availability, and different task setups affect routing performance. In summary, this paper finds that there is still a substantial room for further improvement and exploration.
Critical strengths
- The presented benchmark, MMR-Bench V2, has broad model and task coverage, containing full paired records.
- The authors configure a careful evaluation setup, including introducing known-task fixed baselines, presenting top-k coverage analysis, and so on.
- The authors conducted insightful analysis to explore the effect of model-pool composition, visual information, and task setups.
Critical weaknesses
- Poor presentation: the presentation of this paper is unsatisfactory, making it difficult to read and understand.
- The task definition and descriptions in Section 3.1 are not sufficiently clear.
- The paper is not well structured, and the paragraph organization is sometimes unreasonable. The authors are encouraged to add more
\paragraph{}headings to clarify the structure and purpose of different parts. - Some statements in the paper are not written fluently, which further affects readability.
- The benchmark setting is overly fixed and does not sufficiently consider practical scenarios
- It is unclear how the benchmark and the conclusions would change under non-binary outcomes.
- The treatment of execution failures is questionable. Execution failures should not be included in the primary routing evaluation, because they may result from randomness or inappropriate configurations. Moreover, it is difficult for a router to predict execution failures, and handling such failures should arguably not be part of the router’s responsibility.
- Cost is currently treated mainly as an auxiliary metric, while most of the analysis still focuses on quality. This setting is not fully satisfactory. The problem could instead be formulated as a constrained optimization problem, such as maximizing quality under a fixed budget, or evaluated using a metric that jointly considers quality and cost.
- Although the benchmark has broad coverage, it is unclear whether including 44 models is meaningful when the main evaluation focuses on quality. According to the results in Section 6, as well as common practical usage, users are likely to consider only a small subset of these models.
- The research contribution is limited
-
The benchmark does not appear to have a sufficiently distinctive design. It seems mainly to be a scale-up of prior work, and it is unclear what is specifically novel about MMR-Bench V2 in the multimodal setting. The results in Section 6 even show that text-only methods can outperform methods using multimodal inputs.
-
The potential impact of this benchmark may be limited. Models are developing rapidly, and within a few months, newly released models may substantially outperform the older models included in the benchmark. A static benchmark may therefore be unsuitable for evaluating model routers. As the paper’s own analysis suggests, simply selecting a newer and stronger model may often be sufficient.
-
Although the paper presents several analyses, it does not clearly indicate how future work should proceed based on these findings.
- The analysis of visual information may not be sufficiently robust
The paper’s conclusion regarding the effect of visual information may not be sufficiently well supported, as the aggregate degradation is heavily influenced by MathVerse.
Questions for authors
see weakness