acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Models, Non-Reasoning Answers: Uncovering Reasoning-Bypass Shortcut in MLRMs

11 views

Abstract

Multimodal large reasoning models (MLRMs) aim to move vision language models beyond System-1-like direct prediction toward explicit System-2-like reasoning for complex problems. However, recent studies reveal substantial inconsistency between the reasoning process and the final answer. Existing work often characterizes such behavior using concepts borrowed from human cognition, including hallucination, unfaithfulness, and even deception, but provides limited insight into its internal mechanism. In this work, we identify a prevalent yet underexplored pattern of reasoning inconsistency, termed Reasoning-Bypass Shortcut, in which the final answer specifically reverts to the prediction the model would have produced without explicit reasoning. Across seven MLRMs and four benchmarks, Reasoning-Bypass Shortcut accounts for of all reasoning inconsistency cases, revealing a systematic tendency rather than random disagreement. To investigate its mechanism, we construct more than instances spanning diverse visual tasks that reliably capture this behavior. We then introduce a Reasoning Differential Probe to localize attention head representations associated with Reasoning-Bypass Shortcut, and further apply representation interventions to distinguish causal features from merely correlated ones. Our analysis reveals a sparse set of internal features that can redirect the model from its reasoning based decision (System-2) toward its direct prediction (System-1). Finally, we transfer our analysis to unseen scenarios and show that the identified internal signatures persist beyond the settings where they are first observed. This verifies the generality of our findings and shows that such behaviors can be captured from internal model representations, complementing output-level observations with a mechanistic perspective.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large multimodal reasoning models, failure mode

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_pa53OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies why multimodal large reasoning models sometimes give a final answer that disagrees with their own reasoning. For each question, the authors compare the answer given with empty thinking (pre), the conclusion extracted from the reasoning (mid), and the final answer (post). They call the case where pre eqeq post ≠\neq mid a "Reasoning-Bypass Shortcut" (RBS), meaning the model reasons to one answer but returns to the answer it would have given without reasoning, and evaluate across 7 models and 4 benchmarks to show its prevalence. To find the cause, they train a learn probe based on attention head value change before and after reasoning, then intervene on the selected heads to suppress the shortcut. Their experiments show that these interventions reduces RBS and generalize well.

Critical strengths
  1. The paper is well motivated and clear presented. The pre/mid/post formulation is straightforward, and the output -> internal -> output structures makes it easy to follow.
  2. The experiments are extensive. It covers 7 MLRMs and 4 benchmarks, and includes held-out and cross-dataset evaluations.
Critical weaknesses
  1. Effect on consistent samples is unknown. The causal activation intervention experiments exclude samples where pre = mid = post, so it is unclear how the intervention affects them. These samples likely make up the majority of each benchmark, so the intervention could noticeably change overall accuracy. Reporting the flip rate, answer confidence, and accuracy on these samples would strengthen the paper.
  2. Reasoning Differential Probe is trained by labeling RBS as pre = post and mid ≠\neq post and normal as pre ≠\neq post and mid eqeq post. Since pre ≠\neq mid applies to both classes, the label is just whether pre = post or not. A probe on the change between hidden states of pre and post separates these two classes but ignores the cases where pre = mid = post, which ties this issue to Weakness 1
Questions for authors
  1. Is the RI% denominator in table 1 all samples or only samples excluding the samples where pre = mid = post. The caption mentions the prevalence of reasoning inconsistency. If those samples are excluded, the reported rates may overstate how common reasoning inconsistency actually is.
  2. Do the findings hold for MLRMs that are not built on Qwen? All seven models appear to be fine-tuned from Qwen 2.5 VL 7B. Have the authors tried models from other model families?
Reviewer_mQunOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Reasoning-Bypass Shortcut (RBS), a pattern in which a model’s final answer matches its answer without explicit reasoning but disagrees with the conclusion expressed in its reasoning trace (pre = post ≠ mid). The authors characterize this behavior across seven multimodal reasoning models and four benchmarks, then use differential probing and activation interventions to identify attention heads that influence its occurrence.

Critical strengths
  1. Comparing the final answer with both the reasoning conclusion and the direct-answer prediction provides a useful refinement of reasoning inconsistency. It makes the observed disagreement more concrete and suggests a testable explanation.

  2. The analysis goes beyond probing accuracy. Activation interventions, controls using randomly selected heads from the same layer, and evaluations on unseen samples provide evidence that the selected heads influence the observed behavior.

  3. Appendix C.1 presents an interesting finding: the final answer is often more accurate than the reasoning conclusion on RBS cases. This complicates the interpretation of reasoning inconsistency and raises a worthwhile question about when explicit reasoning should be trusted.

Critical weaknesses

W1. My main concern is the gap between the behavioral definition and the proposed mechanism. Observing pre = post ≠ mid does not establish that the model first adopted the reasoning conclusion and subsequently reverted to its direct prediction. Moreover, mid is extracted from text by an external model, whereas pre and post are elicited at the answer position. The difference could therefore reflect a persistent disagreement between the written explanation and the model’s answer preference. Figure 5 partially addresses this issue, but does not report how often RBS cases actually exhibit a pre → mid → pre trajectory.

W2. The interventions establish an influence on answer selection, but their specificity to reasoning bypass remains unclear. Heads involved in copying the stated conclusion, selecting answer letters, or resolving conflicting predictions could produce similar effects. Gaussian Noise and Random-K do not distinguish these explanations. The composition of the “normal” group also needs to be specified.

W3. Figure 3 is difficult to reconcile with Equation (14). Its axes are labeled as intervention-induced changes, yet the before intervention points have nonzero values. In addition, Equation (14) defines the confidence change on normal samples and the RBS margin change on RBS samples, while the figure presents them as a joint distribution of individual samples. The authors should explain what each point represents and whether the plotted quantities are raw values or differences.

W4. The accuracy analysis deserves a closer connection to the intervention results. Table 6 shows that replacing post with mid can substantially reduce accuracy; for GThinker, the reported accuracies are 65.86% and 22.05%, respectively. The authors appropriately acknowledge this trade-off, but do not show which cases their intervention changes. Separating cases where mid is correct, pre/post is correct, or both are wrong would clarify whether the intervention restores useful reasoning dependence or mainly overrides correct final predictions.

Questions for authors
  • What fraction of RBS cases actually transition from pre to mid and then back to pre under the conditional answer readout?

  • Do the identified heads remain specific to RBS under the behavioral controls described above?

  • Which intervention tuples are used in Table 3, and how should Figure 3 be interpreted under Equation (14)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.