- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether higher reasoning scores reflect better evidence selection in multimodal models. It examines which source is correct, whether answer candidates are visible, and how outputs are scored. Experiments cover three model families, using CMC-Bench, MMKC year questions, ChartQA, and Oxford-IIIT Pet, although not every model is tested in every study. The main finding is that a larger reasoning gain can partly result from a weaker non-reasoning baseline.
Critical strengths
- The candidate-visibility experiment keeps the required answer format and output limit fixed within each model. This provides a useful example of how a larger reasoning gain can arise partly because the non-reasoning baseline becomes worse.
- The paper reports absolute accuracies, paired confidence intervals, invalid outputs, and unsuccessful replications. It also clearly separates the original scoring rules from later analyses, making the results easier to assess.
Critical weaknesses
- The experiments provide limited evidence about multimodal evidence selection. For questions concerning "years," a model might be able to derive the answer based solely on the provided text and its own prior knowledge. In cases where a brand name is explicitly mentioned in the text, the model may arrive at the answer by synthesizing that information with its existing knowledge base. There is no image-removal control to show whether the images matter. The chart experiment explicitly tells the model to trust the chart, so it does not test whether the model can identify the reliable source itself. These limitations leave the connection between the main findings and multimodal evidence selection unclear.
- Compared to existing research, the new findings are not particularly significant. It is already well-established in the literature that answer options and scoring methods can influence results. The primary new findings in this paper appear only in the "years" task for two of the models and could not be replicated on GLM or ChartQA. The experiments involving "pets" primarily demonstrate that output formats affect scoring, without addressing issues related to conflict or reasoning. While the paper offers some useful observations, its overall research contribution remains limited.
Questions for authors
- Would the evaluation results for the "year-task" change if the images were removed or replaced with irrelevant ones? Since the brand name is already mentioned in the text of the year-related questions, the model might be able to provide the correct answer based solely on its existing knowledge; however, the paper does not verify this by removing the images.
- Does the ChartQA result change if the prompt does not tell the model to prioritize the chart?
- In CMC, how many newly incorrect answers are content errors, and how many are formatting or truncation failures? Can you manually check a sample?