acceptodds
Withdrawn from ICLR 2027

When Visual Truth Loses Control: Diagnosing and Preserving Grounded Predictions in Multimodal Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable multimodal generation requires visual evidence to remain influential throughout decoding, yet existing studies mainly attribute hallucination to insufficient visual perception or excessive language priors, leaving its internal evolution underexplored. We uncover a fundamental failure mode, termed Grounded Control Reversal (GCR), where a visually grounded preference emerges in intermediate representations but is subsequently weakened or overturned before the final answer is produced. Through layer-wise preference tracing and same-image internal interventions on Qwen2.5-VL-7B across HallusionBench and POPE-Adversarial, we show that reversals arise from the combined effects of attenuated visual pathways, competing textual pathways, and feed-forward transformations. Based on this insight, we introduce ReCAP, a training-free decoding framework that identifies a persistent visually grounded intermediate state and restores its lost predictive influence through a minimum-norm projection in hidden space. The intervention selectively recovers the grounded preference while preserving unrelated residual components. Our findings provide a causal perspective on multimodal hallucination, showing that failures can arise not from the absence of visual understanding, but from the subsequent loss of control over already acquired visual evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Visual Hallucination, Representation Steering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2tTNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Grounded Control Reversal (GCR): a correct, visually supported answer preference emerges in intermediate layers of a multimodal model but is subsequently overturned before the final prediction. The authors study this phenomenon through layerwise prediction tracing, same-image visual-pathway interventions, and Shapley attribution across visual attention, textual attention, and feed-forward updates. Building on this diagnosis, the paper proposes ReCAP, a training-free decoding method that selects a persistent, visually supported intermediate candidate and applies a minimum-norm projection to the final hidden state to restore its advantage over the original prediction. Evaluation covers two 7B backbones and multiple hallucination and general-capability benchmarks. Additional experiments examine anchor reliability, alternative correction rules, preservation of useful content, and inference overhead.

Critical strengths

The mechanism analysis includes meaningful intervention controls. The paper does not infer visual grounding solely from a correct intermediate prediction. It includes visual-value suppression, sham replay, norm-matched random perturbations, and norm-matched nonvisual perturbations. Importantly, Table A2 evaluates these controls before applying the positive visual-support filter, reducing the risk that the filtering criterion itself produces an apparently positive effect. Table A4 additionally examines correspondence between the online support score and the intervention-based measure. These experiments make the diagnosis more persuasive than layerwise correlations or attention visualizations alone.

The correction is precisely formulated and supported by useful empirical analyses. ReCAP has a closed-form correction, and the matched-anchor comparisons in Table 5 and Appendix E help separate the effects of anchor selection from correction geometry. The reported improvements are practically relevant: Qwen’s POPE accuracy increases from 84.82% to 89.38%. Table A5 further shows that ReCAP corrects 562 original errors while introducing 152 new errors, making the net benefit explicit. Content-retention and general-capability evaluations provide additional support for the method’s practical

Critical weaknesses

The contribution of reversal-aware triggering is not clearly isolated. Section 4.1 defines b=arg⁡max⁡wzL(w)b=\arg\max_w z_L(w) and requires an eligible anchor to satisfy a⋆≠ba^\star\neq b. Except when final logits are tied, an eligible anchor therefore already satisfies zL(a⋆)<zL(b)z_L(a^\star)<z_L(b), making the subsequent reversal check redundant. However, Section 5.4 describes an always-on variant that additionally intervenes when a supported preference weakens without reversing. It is unclear how such cases arise under the stated definitions. If the variant also changes the candidate selection, competitor, or correction rule, its performance difference cannot be attributed solely to the reversal trigger. A precise description of this variant and the conditions under which it differs from ReCAP would help resolve this concern.

The mechanistic novelty relative to DeCo needs sharper positioning. DeCo[1] already reports that correct visual predictions can emerge in intermediate layers and subsequently be suppressed or overtaken by hallucinated predictions in later layers. Its decoding method also uses candidate filtering, dynamic intermediate-layer selection, and final-output correction. Thus, the broad phenomenon underlying GCR substantially overlaps with an existing observation. The present paper adds meaningful elements, including persistence requirements, same-image visual-pathway interventions, route attribution, and pairwise residual projection. However, these additions support a more precise diagnosis and a different correction strategy more directly than the discovery of an entirely new failure mechanism. The authors should clarify which mechanistic conclusions go beyond DeCo and connect those claims to the intervention evidence already provided. This distinction is consequential because mechanism discovery is presented as a central contribution.

[1]Wang, Chenxi, et al. "Mllm can see? dynamic correction decoding for hallucination mitigation." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors
  1. Clarification of the always-on ablation. Since b=arg⁡max⁡wzL(w)b=\arg\max_w z_L(w) and an eligible anchor must satisfy a⋆≠ba^\star\neq b, the reversal condition zL(a⋆)<zL(b)z_L(a^\star)<z_L(b) appears to follow automatically when the final maximum is unique. How does the always-on variant encounter cases where a supported preference weakens without reversing? Please clarify whether this variant changes anything besides the trigger, such as candidate selection, the competitor, or the correction rule.
  2. Clarification of targeted validation. Please clarify whether the baselines received comparable tuning opportunities. Paired confidence intervals for the main comparisons would help assess the smaller improvements. If the reported test sets informed parameter selection, a limited evaluation on held-out data with fixed parameters would help resolve this concern.
  3. Clarification of the mechanistic contribution relative to DeCo. DeCo already describes correct visual predictions emerging in intermediate layers and subsequently being suppressed or overtaken in later layers. Which specific mechanistic conclusions of GCR go beyond this observation? Please identify how the existing persistence analysis and visual-pathway interventions establish the additional contribution, and clarify whether GCR is intended as a distinct failure mechanism or a more precise operational characterization of the previously observed phenomenon.
Reviewer_bAGzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies grounded control reversal (i.e., an intermediate prediction favors the correct answer), where the preference is supported by visual information, and the final prediction nevertheless becomes incorrect. It combines persistent intermediate-margin measurements with same-image visual-value suppression and coalition-based replay of visual-attention, remaining-attention, and feed-forward routes. The proposed ReCAP method uses an online grounding proxy to select an intermediate alternative and applies a closed-form, minimum-norm correction to the final residual representation under a pairwise control constraint. Experiments on Qwen2.5-VL and LLaVA cover hallucination and visual-reasoning benchmarks, intervention controls, alternative correction strategies, and runtime overhead.

Critical strengths

This paper investigates whether the intermediate preference actually depends on visual computation. Table A2 evaluates visual suppression against matched controls before filtering on a positive grounding effect. This makes the evidence more informative than a comparison conditioned on the diagnostic criterion itself. The route-replay analysis also provides a concrete, architecture-dependent account of how subsequent computation changes the relevant preference.

The minimum-norm intervention is derived for an explicitly limited pairwise constraint, and the experiments include comparisons that match important aspects of the anchor and correction target. Table A5 reports the effect on the full 9,000-example POPE set, which establishes a useful aggregate benefit while making the cost of intervention visible.

Critical weaknesses
  1. In the algorithm, bb is the final argmax, while an eligible alternative must satisfy a≠ba \neq b. Consequently, zL(a)≤zL(b)z_L(a)\leq z_L(b) already follows from these definitions, with strict inequality except for ties. The final-layer reversal check therefore adds essentially no independent selectivity once eligibility is satisfied.

  2. Table 5 describes an always-on variant that also acts when an earlier preference weakens without reversing. It is unclear how such cases are constructed.

Questions for authors
  1. What candidate, competitor, eligibility, and correction definitions are used for the always-on variant?

  2. Which data were used to select KK, ρ\rho, and the persistence requirement? Were these choices frozen before the reported test evaluations?

Reviewer_EpwzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a failure mode termed Grounded Control Reversal (GCR), where a visually supported prediction emerges at intermediate layers of an MLLM. The authors propose ReCAP, a training-free decoding method that identifies a persistent visually supported intermediate candidate and restores its pairwise preference. The method is evaluated on Qwen2.5-VL-7B and LLaVA-1.5-7B across several hallucination and general multiomodal benchmarks.

Critical strengths

The mechanism analysis is thoughtful and controlled. The authors test visual dependence through intervention and nonvisual perturbation controls.

ReCAP is simple, training-free, and empirically effective across multiple tasks. It improves most hallucination metrics while largely preserving general multimodal capability and adding modest inference overhead.

Critical weaknesses

The main claim is established only on Qwen2.5-BL-7B. GCR are analyzed usign Qwen2.5-VL-7B errors from POPE-Adversaria and HallusionBench. ReCAP is evaluated on LLaVA-1.5-7B.

Model generalization is relatively narrow, and hyperparameter selection protocol is unclear. Both evaluated models are 7B-scale. This is particularly relevant for a method that depends on layer deoth, visual tokens, etc. The paper should clarify whether a seprate validation set was used for these choices.

The models prevalance on LLaVA-1.5-7BB, if it is reported, it can be established that GCR is model independant phenomen. Also what'd be the take on other model scales withing the same family (e.g., smaller or larger qwen), I understand there'll always be another model to test, but the authors present it as adaptable sort of work.

Questions for authors
  • GCR relies on direct application of the final output head to intermediate residual states. how is it different from simple intermeidate decodability?
  • can the GCR and grounded-GCR on LLaVA-1.5-7B be reported?
  • for visual attention, textual attention, and FFN routes. are visual attention tokens not influenced by textual one's at all? i mean how are these emantically distint routes
  • minor: hyperparameter selection, were the K and \rho based on performances on the benchmarks, or otherwise?

my main hesitation is the intrerprtation of 'control reversal'. the paper shows that the correct answer can receive a postive margin when the final head is applied to intermediate residual states. The concern is that whether the correct answer was subsequently overwritten, also that it is established only on single model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.