acceptodds
Withdrawn from ICLR 2027

CAN LARGE MULTIMODAL MODELS THINK JUST ENOUGH? EVALUATING OUTPUT-SIDE ADAPTIVE REASONING IN LARGE MULTIMODAL MODELS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Explicit reasoning can improve the performance of large multimodal models (LMMs) on complex visual understanding and reasoning tasks, but lengthy and unnecessary reasoning also incurs additional generation cost. Although recent output-side adaptive reasoning methods have shown potential for reducing unnecessary generation cost, their evaluation remains fragmented and inconsistent. In this work, we introduce MARE-Bench, a benchmark for systematically evaluating output-side adaptive reasoning methods for LMMs. MARE-Bench provides unified evaluation protocols across seven representative datasets spanning five evaluation dimensions: General Knowledge, Mathematical Reasoning, Logical Reasoning, Medical QA, and OCR. It covers seven representative adaptive reasoning methods and three LMM families. We analyze adaptive reasoning methods in terms of task performance, average output tokens, and reasoning behaviors, characterizing the performance–generation cost trade-off. Our experiments reveal several findings: (1) some methods outperform their base models, while others perform poorly; (2) FAST-GRPO achieves consistently high overall performance across the evaluated settings; (3) the effects of adaptive reasoning vary across evaluation dimensions, with larger performance gains on Mathematical Reasoning and Logical Reasoning tasks, smaller gains in General Knowledge and Medical QA, and performance degradation on OCR; and (4) the performance–generation cost trade-off of adaptive reasoning methods can vary across base models. We believe that MARE-Bench will serve as a reliable foundation for future research on efficient adaptive reasoning in LMMs.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Adaptive Reasoning, Reasoning Efficiency Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nhCTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the generation overhead associated with explicit reasoning in large-scale multimodal models, this paper introduces MARE-Bench, a benchmark for evaluating output-side adaptive reasoning. The benchmark encompasses seven datasets, five capability dimensions, and seven adaptive reasoning methods, categorizing the latter into three types: reasoning mode selection, reasoning effort allocation, and reasoning expression optimization. Using a unified evaluation framework, the authors compare the task accuracy and average output token count of these methods, further analyzing reasoning effort adaptability, sensitivity to output budgets, and error types. Experimental results reveal that the performance gains and generation costs of adaptive reasoning are jointly influenced by task type, the base model, and the reasoning strategy.

Critical strengths
  1. The evaluation addresses a problem of clear research significance. While existing research on adaptive reasoning primarily focuses on validating individual methods, this paper integrates various control strategies into a unified evaluation framework. By covering tasks such as mathematical reasoning, logical reasoning, general knowledge, medical QA, and OCR, the study facilitates a better understanding of the applicability of different methods.

  2. Section 4.3 goes beyond comparing final performance to examine factors such as task difficulty, output budget, and generation repetition. For instance, ARES-RL-7B shows sensitivity to output budget constraints on MathVision, whereas FAST-7B and VL-Cogito-7B exhibit relatively stable performance. These results demonstrate that increasing reasoning length does not necessarily lead to improved accuracy, offering valuable practical insights.

Critical weaknesses
  1. Sections 3.1 and 4.1 use the officially released models of each method, but different methods differ in their base models, training configurations, and parameter sizes. In particular, TON selects different dedicated models based on the evaluation task. Consequently, the performance differences shown in Table 2 cannot be directly attributed solely to adaptive inference strategies. It is recommended to further distinguish between relative gains achieved using the same base model and absolute performance comparisons across different models, thereby clarifying the scope of conclusions drawn from cross-method comparisons.

  2. Section 3 primarily consolidates seven existing datasets and compares performance based on accuracy and average output token count. Compared to established benchmarks like AdaptMMBench and EffiReason-Bench, the unique value of the proposed evaluation protocol or analytical approach has not yet been sufficiently demonstrated. While the behavioral analysis in Section 4.3 offers some observations, it fails to yield sufficiently profound new insights. It is recommended to clearly articulate the key evaluation questions that existing benchmarks cannot answer but MARE-Bench can address, and to strengthen the corresponding experimental evidence.

Questions for authors
  1. The authors are encouraged to further elaborate on how training configurations and the selection of task-specific models across different methods influence the comparative results. Specifically, when comparing multiple TON-specific models against other fixed models, how do the authors ensure that the conclusions primarily reflect differences in adaptive inference strategies?

  2. What are the key differences between MARE-Bench and existing benchmarks such as AdaptMMBench and EffiReason-Bench? It is hoped that the authors will further clarify what capabilities they offer in terms of evaluation protocols, methodological coverage, or behavioral analysis that are not yet available in existing benchmarks.

  3. It is recommended to clarify whether token statistics across different models are comparable and to discuss the impact of truncated samples on average accuracy and average generation length. Are there existing statistics to substantiate the current conclusions regarding efficiency gains?

  4. Regarding the manual error attribution in Section 4.3.3, how is annotation reliability ensured? It is recommended to explain the sampling rules for the 50 samples, whether only incorrect answers were included, and whether multiple annotations or consistency checks were performed.

Reviewer_UTKHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MARE-Bench, a unified benchmark for evaluating output-side adaptive reasoning methods in large multimodal models. It integrates seven existing public datasets spanning five evaluation dimensions: general knowledge, mathematical reasoning, logical reasoning, medical question answering, and OCR. The evaluation compares seven existing methods across three categories, together with their corresponding base models. Accuracy and average output token count are used to measure performance and generation cost, supplemented by analyses of problem difficulty, output budgets, generation repetition, and error types. The experiments indicate that the benefits of adaptive reasoning depend on the task and base model. Some methods improve the trade-off between accuracy and output length, whereas others exhibit performance degradation or increased output cost.

Critical strengths

The paper evaluates multiple output-side adaptive reasoning methods through a unified evaluation pipeline covering five task categories. This helps address the difficulty of comparing methods evaluated under inconsistent settings and provides a useful reference for understanding how post-training methods affect model performance and output cost. Beyond accuracy and average output length, the analysis examines problem difficulty, output budgets, truncation, and generation repetition. In particular, the budget sensitivity experiments show that some methods are substantially constrained by the output limit, while longer outputs do not necessarily yield higher accuracy. These observations provide useful guidance for selecting reasoning configurations in practice.

Critical weaknesses
  1. MARE-Bench primarily integrates existing datasets, publicly available methods, and established evaluation tools, with accuracy and average output token count as its core metrics. Although the paper provides analyses of difficulty, budgets, and errors, it does not sufficiently establish what distinctive diagnostic capabilities these analyses offer beyond existing reasoning evaluation frameworks. In particular, changes in output length with difficulty and improvements in average performance do not directly demonstrate that a model selects an appropriate reasoning mode or budget for individual examples. The paper lacks controlled comparisons with non-adaptive strategies, such as always using CoT reasoning or always answering directly, leaving the validation of adaptivity itself relatively limited.

  2. The fairness of comparisons across methods and the scope of the conclusions remain limited. Although the paper standardizes the inference pipeline and includes the corresponding base models as references for most methods, it evaluates existing trained models whose training data, training budgets, and supervision procedures are not controlled across methods. Consequently, improvements over the base models cannot be sufficiently attributed to the adaptive mechanisms themselves.

Questions for authors
  1. The authors position MARE-Bench as an evaluation framework for adaptive reasoning, but its primary metrics remain accuracy and average output length. Please clarify what distinctive adaptive behaviors or failure modes this framework reveals beyond existing reasoning evaluation frameworks. Could non-adaptive strategies, such as always reasoning or always answering directly, be evaluated using the same model under comparable budgets to determine whether the gains indeed arise from reasoning mode or budget selection at the individual-example level?

  2. The paper evaluates publicly available trained models against their corresponding base models. Please explain how the effects of the adaptive mechanisms can be distinguished from gains attributable to additional training data, training budgets, and supervision procedures.

Questions for further investigation:

  1. Do different training approaches exhibit similar patterns of performance improvement across different base model families?

  2. The manual error attribution analysis samples 50 cases for each of four models on MMMU. Please clarify whether sampling is restricted to incorrect outputs, whether the same questions are used across models, and the number of annotators, category assignment criteria, and inter-annotator agreement. Could a larger-scale LLM-assisted error analysis yield stable error-type distributions and reveal systematic patterns across different task domains?

Reviewer_3eqHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MARE-Bench for systematically evaluating adaptive reasoning methods in large multimodal models. MARE-Bench collects data from 7 existing benchmarks and divides the data into 5 capability dimensions. The experiments evaluate 7 representative adaptive reasoning methods across 4 MLLM families, presenting their accuracy and average output token. The paper further examines reasoning effort adaptivity, output budget sensitivity, and error attribution.

Critical strengths
  1. All experiments use VLMEvalKit and vLLM under a unified evaluation pipeline, ensuring strong comparability and reproducibility.

  2. Beyond accuracy and average output tokens, this paper further analyzes reasoning effort adaptivity, output budget sensitivity, and error attribution, enabling a deeper characterization of reasoning behaviors and failure modes.

Critical weaknesses
  1. The contribution of MARE-Bench is overstated. It combines existing public benchmarks without collecting new data. In my assessment, evaluating on MARE-Bench therefore offers no fundamental difference from evaluating on its constituent benchmarks separately.

  2. The experimental analysis largely reiterates findings already widely accepted in the community and provides limited new, meaningful insights. It also lacks sufficient depth: for example, the explanation that OCR degradation results from “unnecessary reinterpretation or revision” remains speculative rather than being supported through ablation studies.

  3. The error attribution analysis is limited to 200 examples from a single dataset (MMMU) and a single base model (Qwen2.5-VL-7B). This small sample and narrow coverage limit the generalizability of the findings.

  4. The output-budget sensitivity analysis examines only three methods on Qwen2.5-VL-7B, without covering other foundation models.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.