- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies why multimodal large reasoning models sometimes give a final answer that disagrees with their own reasoning. For each question, the authors compare the answer given with empty thinking (pre), the conclusion extracted from the reasoning (mid), and the final answer (post). They call the case where pre post mid a "Reasoning-Bypass Shortcut" (RBS), meaning the model reasons to one answer but returns to the answer it would have given without reasoning, and evaluate across 7 models and 4 benchmarks to show its prevalence. To find the cause, they train a learn probe based on attention head value change before and after reasoning, then intervene on the selected heads to suppress the shortcut. Their experiments show that these interventions reduces RBS and generalize well.
Critical strengths
- The paper is well motivated and clear presented. The pre/mid/post formulation is straightforward, and the output -> internal -> output structures makes it easy to follow.
- The experiments are extensive. It covers 7 MLRMs and 4 benchmarks, and includes held-out and cross-dataset evaluations.
Critical weaknesses
- Effect on consistent samples is unknown. The causal activation intervention experiments exclude samples where pre = mid = post, so it is unclear how the intervention affects them. These samples likely make up the majority of each benchmark, so the intervention could noticeably change overall accuracy. Reporting the flip rate, answer confidence, and accuracy on these samples would strengthen the paper.
- Reasoning Differential Probe is trained by labeling RBS as pre = post and mid post and normal as pre post and mid post. Since pre mid applies to both classes, the label is just whether pre = post or not. A probe on the change between hidden states of pre and post separates these two classes but ignores the cases where pre = mid = post, which ties this issue to Weakness 1
Questions for authors
- Is the RI% denominator in table 1 all samples or only samples excluding the samples where pre = mid = post. The caption mentions the prevalence of reasoning inconsistency. If those samples are excluded, the reported rates may overstate how common reasoning inconsistency actually is.
- Do the findings hold for MLRMs that are not built on Qwen? All seven models appear to be fine-tuned from Qwen 2.5 VL 7B. Have the authors tried models from other model families?