- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the generation overhead associated with explicit reasoning in large-scale multimodal models, this paper introduces MARE-Bench, a benchmark for evaluating output-side adaptive reasoning. The benchmark encompasses seven datasets, five capability dimensions, and seven adaptive reasoning methods, categorizing the latter into three types: reasoning mode selection, reasoning effort allocation, and reasoning expression optimization. Using a unified evaluation framework, the authors compare the task accuracy and average output token count of these methods, further analyzing reasoning effort adaptability, sensitivity to output budgets, and error types. Experimental results reveal that the performance gains and generation costs of adaptive reasoning are jointly influenced by task type, the base model, and the reasoning strategy.
Critical strengths
-
The evaluation addresses a problem of clear research significance. While existing research on adaptive reasoning primarily focuses on validating individual methods, this paper integrates various control strategies into a unified evaluation framework. By covering tasks such as mathematical reasoning, logical reasoning, general knowledge, medical QA, and OCR, the study facilitates a better understanding of the applicability of different methods.
-
Section 4.3 goes beyond comparing final performance to examine factors such as task difficulty, output budget, and generation repetition. For instance, ARES-RL-7B shows sensitivity to output budget constraints on MathVision, whereas FAST-7B and VL-Cogito-7B exhibit relatively stable performance. These results demonstrate that increasing reasoning length does not necessarily lead to improved accuracy, offering valuable practical insights.
Critical weaknesses
-
Sections 3.1 and 4.1 use the officially released models of each method, but different methods differ in their base models, training configurations, and parameter sizes. In particular, TON selects different dedicated models based on the evaluation task. Consequently, the performance differences shown in Table 2 cannot be directly attributed solely to adaptive inference strategies. It is recommended to further distinguish between relative gains achieved using the same base model and absolute performance comparisons across different models, thereby clarifying the scope of conclusions drawn from cross-method comparisons.
-
Section 3 primarily consolidates seven existing datasets and compares performance based on accuracy and average output token count. Compared to established benchmarks like AdaptMMBench and EffiReason-Bench, the unique value of the proposed evaluation protocol or analytical approach has not yet been sufficiently demonstrated. While the behavioral analysis in Section 4.3 offers some observations, it fails to yield sufficiently profound new insights. It is recommended to clearly articulate the key evaluation questions that existing benchmarks cannot answer but MARE-Bench can address, and to strengthen the corresponding experimental evidence.
Questions for authors
-
The authors are encouraged to further elaborate on how training configurations and the selection of task-specific models across different methods influence the comparative results. Specifically, when comparing multiple TON-specific models against other fixed models, how do the authors ensure that the conclusions primarily reflect differences in adaptive inference strategies?
-
What are the key differences between MARE-Bench and existing benchmarks such as AdaptMMBench and EffiReason-Bench? It is hoped that the authors will further clarify what capabilities they offer in terms of evaluation protocols, methodological coverage, or behavioral analysis that are not yet available in existing benchmarks.
-
It is recommended to clarify whether token statistics across different models are comparable and to discuss the impact of truncated samples on average accuracy and average generation length. Are there existing statistics to substantiate the current conclusions regarding efficiency gains?
-
Regarding the manual error attribution in Section 4.3.3, how is annotation reliability ensured? It is recommended to explain the sampling rules for the 50 samples, whether only incorrect answers were included, and whether multiple annotations or consistency checks were performed.