acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Reasoning under Multimodal Conflict: Separating Score Gains from Evidence Selection

Abstract

Reasoning is used to help multimodal large language models answer questions when visual and textual evidence conflict. Yet higher benchmark scores do not establish more reliable evidence selection, because scores also depend on how answers are elicited and judged. We develop a controlled evaluation protocol that examines correct-source direction, candidate access, and answer scoring through complementary paired comparisons. Its core design crosses reasoning on/off, repeated versus disagreeing incorrect statements, and candidate visibility while holding the required response and output limit fixed within each model. The core comparison covers three open-weight model families, with separate evidence-direction and visual-recognition controls on two. We find that reasoning can improve answers when images are correct while harming them when text is correct. On image-associated knowledge questions, candidates enlarge the reasoning-by-disagreement interaction mainly through a weaker non-reasoning baseline in Qwen, but more through reasoning-enabled accuracy changes in Gemma. The large positive candidate-induced increases in this interaction observed in these two models are not reproduced in the third or under the chart-reading protocol. In the recognition control, the original strict-JSON tests fail, while supplementary wrapper-tolerant scoring reveals label-list benefits and no additional accuracy gain from constrained decoding. These results distinguish format compliance from label matching without attributing vocabulary gains to naming alone. Our findings support evaluating reasoning through source-resolved outcomes, absolute paired accuracies, and explicit answer-information and scoring controls, rather than aggregate gains alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, multimodal conflict, reasoning evaluation, model reliability, evaluation methodology, benchmark validity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EcD7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether higher reasoning scores reflect better evidence selection in multimodal models. It examines which source is correct, whether answer candidates are visible, and how outputs are scored. Experiments cover three model families, using CMC-Bench, MMKC year questions, ChartQA, and Oxford-IIIT Pet, although not every model is tested in every study. The main finding is that a larger reasoning gain can partly result from a weaker non-reasoning baseline.

Critical strengths
  1. The candidate-visibility experiment keeps the required answer format and output limit fixed within each model. This provides a useful example of how a larger reasoning gain can arise partly because the non-reasoning baseline becomes worse.
  2. The paper reports absolute accuracies, paired confidence intervals, invalid outputs, and unsuccessful replications. It also clearly separates the original scoring rules from later analyses, making the results easier to assess.
Critical weaknesses
  1. The experiments provide limited evidence about multimodal evidence selection. For questions concerning "years," a model might be able to derive the answer based solely on the provided text and its own prior knowledge. In cases where a brand name is explicitly mentioned in the text, the model may arrive at the answer by synthesizing that information with its existing knowledge base. There is no image-removal control to show whether the images matter. The chart experiment explicitly tells the model to trust the chart, so it does not test whether the model can identify the reliable source itself. These limitations leave the connection between the main findings and multimodal evidence selection unclear.
  2. Compared to existing research, the new findings are not particularly significant. It is already well-established in the literature that answer options and scoring methods can influence results. The primary new findings in this paper appear only in the "years" task for two of the models and could not be replicated on GLM or ChartQA. The experiments involving "pets" primarily demonstrate that output formats affect scoring, without addressing issues related to conflict or reasoning. While the paper offers some useful observations, its overall research contribution remains limited.
Questions for authors
  1. Would the evaluation results for the "year-task" change if the images were removed or replaced with irrelevant ones? Since the brand name is already mentioned in the text of the year-related questions, the model might be able to provide the correct answer based solely on its existing knowledge; however, the paper does not verify this by removing the images.
  2. Does the ChartQA result change if the prompt does not tell the model to prioritize the chart?
  3. In CMC, how many newly incorrect answers are content errors, and how many are formatting or truncation failures? Can you manually check a sample?
Reviewer_7JBmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the measurement of reasoning gains in MLLMs on multimodal QA with conflicting evidence. It argues that aggregate performance gains are hard to interpret because of unclear baselines and unintended effects of many factors in evaluation settings. In the experiments, various controlled comparisons show that some gains can stem from weaker baselines, and that the effects vary with the modality of the incorrect evidence and with how model outputs are parsed.

Critical strengths
  • The paper pointed out critical limitations in evaluation pipeline for MLLM on QA tasks.
  • The paper empirically shows that aggregated scores make interpreting performance gains hard due to varying baselines, different effects by types of incorrect sources or visibility of candidates, etc.
Critical weaknesses
  • Main issue is presentation. I found that this paper is hard to follow.
  • Many terms are introduced without clear definition or explanation. Identical concepts are represented in different wording.
  • The paper have many comparisons by controlling factors. However, it is hard to understand the aim of each experiment and how each experiment relate a research question.
  • As stated in the paper, it does not aim to identify the mechanism by which various evaluation settings affect performance scores. It provides comparison results for several selected factors and proposes a set of reporting questions for performance gains. However, since many other factors may affect the observed scores, the usefulness of this set of questions is unclear.
Questions for authors
  • I understand the individual analyses, but I had difficulty identifying the main goal that connects them. Could you state it more explicitly?
  • Several of the controls (direction, candidate visibility, scoring rules) seem applicable to any evaluation, not only to reasoning. Could you clarify what is specific to reasoning gain in your design?
Reviewer_HEiPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how evaluation choices affect the apparent benefits of reasoning in multimodal models facing conflicting evidence. It examines which source is correct, whether candidate answers are provided, and how responses are scored. Its central claim is that larger reasoning gains may arise because reasoning produces better answers, because the non-reasoning baseline performs worse, or both. These possibilities must be examined before attributing larger gains to improved evidence selection.

In the image–text conflict experiments, the effects of reasoning vary across models and depend on which source contains the correct answer, illustrating how aggregate scores can conceal important differences. In separate temporal question-answering experiments, the authors find that providing candidate answers can substantially increase the apparent reasoning advantage, partly by reducing non-reasoning accuracy. A follow-up experiment suggests that overlap between candidate answers and incorrect textual claims contributes to this effect for one of the tested models. However, the pattern does not consistently reproduce in a third model or in chart-based questions, limiting its generality.

Overall, the paper argues for a more careful interpretation of reasoning gains and recommends reporting absolute accuracies alongside performance differences.

Critical strengths
  • The paper reports an interesting finding: reasoning can improve answers when the image is correct while reducing accuracy when the text is correct.
  • The paper addresses a useful evaluation problem: a larger reasoning advantage can reflect a weaker non-reasoning baseline rather than better reasoning-enabled answers. Reporting absolute accuracies helps readers distinguish between these possibilities.
  • The main controlled experiments include paired comparisons, confidence intervals, and analyses of how invalid responses could affect the conclusions. I find these checks useful because they help readers assess the uncertainty and robustness of the reported results.
  • The paper is transparent about its limitations. It distinguishes exploratory analyses from the main experiments and reports cases where the findings do not generalize.
Critical weaknesses
  • The connection to evidence selection remains indirect. The experiments primarily measure answer correctness rather than which source the model uses. In the temporal tasks, textual statements may already identify the relevant entity, raising the possibility that correct answers come from prior knowledge without requiring the image.

  • The main positive result has limited demonstrated generality. A clear positive temporal interaction appears in two model families but is not reproduced in the third or under the chart protocol.

  • Response validity still limits some conclusions. For example, Gemma produces 108 invalid chart outputs, and the resulting logical bounds on the content-level interaction range from −13.67 to 15.23 percentage points. The small strict-scoring estimate therefore cannot rule out a substantial content-level effect. The CMC results would also benefit from a comparable breakdown separating substantive errors from formatting failures and truncation.

Questions for authors
  • How much do the temporal results depend on the image?
  • Can image-removal/replacement ablations/experiments distinguish visual evidence use from answering through textual cues and prior knowledge?
  • Can you audit the CMC regressions and Gemma’s invalid chart responses to distinguish substantive answer errors from formatting and truncation failures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.