acceptodds
Withdrawn from ICLR 2027

Path Conflict: Understanding and Mitigating Hallucinations in Multimodal Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) are often assumed to hallucinate because they under-utilize visual evidence and over-rely on linguistic priors. We show that this explanation is incomplete: hallucinated generations can exhibit stronger, rather than weaker, visual-pathway influence. To understand this counter-intuitive behavior, we introduce Path Conflict, which characterizes directional inconsistency between Image-to-Text (I2T) and Text-to-Text (T2T) pathways during token prediction. We further define Conflict Ratio and Conflict Strength to quantify the frequency and magnitude of such conflicts at the attention-head level. Our analysis reveals that hallucinated samples exhibit substantially stronger Path Conflict, driven primarily by amplified I2T contributions while T2T contributions remain comparatively stable. This suggests that hallucination can arise from misaligned visual influence, rather than simply insufficient visual grounding. Based on this finding, we propose a lightweight, training-free Conflict-aware Intervention that selectively regulates I2T and T2T flows within conflict-prone heads. Experiments on POPE, MCQ-POPE, and CHAIR consistently outperform strong inference-time baselines, including AllPath, while pathway-specific ablations reveal the asymmetric and complementary roles of I2T and T2T. Our results highlight inter-path directional conflict as an important mechanism-level perspective for understanding and mitigating MLLM hallucination.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Hallucination, Path Conflict, Mechanistic Analysis, Attention Intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7khjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies hallucinations in multimodal large language models from the perspective of interactions between visual and linguistic information pathways. The authors challenge the conventional explanation that hallucination primarily arises from insufficient visual grounding or excessive reliance on linguistic priors. Instead, they observe that hallucinated generations can exhibit stronger visual-pathway influence. To explain this behavior, the paper introduces path conflict, defined as a directional inconsistency between the Image-to-Text (I2T) and Text-to-Text (T2T) pathways during token prediction. The authors further propose conflict ratio and conflict strength to quantify the frequency and magnitude of such conflicts at the attention-head level. Based on their analysis, the authors report that hallucinated samples exhibit stronger path conflict, with the difference primarily attributed to increased I2T contributions while T2T contributions remain comparatively stable.

Critical strengths
  1. The central observation that hallucinated predictions can involve stronger visual influence is counter-intuitive and challenges the commonly adopted insufficient visual grounding explanation. The distinction between pathway strength and pathway directional agreement is conceptually meaningful: two pathways may both contribute strongly while pushing the prediction in opposite directions.
  2. Rather than introducing an unrelated mitigation technique, the authors use conflict strength to identify conflict-prone heads and independently regulate I2T and T2T flows.
  3. The definition of path conflict based on the directional inconsistency between I2T and T2T contributions is intuitive and mechanism-oriented.
Critical weaknesses
  1. The experiments demonstrate a strong association between path conflict and hallucination, but this does not establish that path conflict is a causal source of hallucination. In particular, the observed improvement from intervention could potentially result from simply attenuating strong I2T contributions rather than resolving directional conflict itself. Stronger controlled experiments are needed to distinguish these explanations.
  2. Both conflict ratio and conflict strength depend on the estimated I2T and T2T pathway contributions. It is therefore important to determine whether the reported conflict phenomenon is intrinsic to the model or partly induced by the specific decomposition/attribution procedure used by the authors. The paper does not provide sufficient evidence using alternative attribution formulations.
  3. The method requires selecting the top-K conflict heads and choosing separate I2T/T2T scaling coefficients. The reported optimal values may depend on the particular model and benchmarks. The paper does not sufficiently establish whether these parameters transfer across datasets and models or whether they require tuning for each setting.
Questions for authors
  1. What evidence supports the claim that path conflict is a causal mechanism rather than a consequence or correlate of hallucinated predictions?
  2. Have the authors compared conflict-strength-based head selection with selecting heads solely according to their I2T contribution magnitude?
  3. Do the authors have evidence that the observed path conflict phenomenon and the effectiveness of the intervention extend beyond LLaVA-v1.5-7B?
Reviewer_bLBrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes object hallucination in MLLMs by checking whether the image-to-text (I2T) and text-to-text (T2T) attention pathways push a token prediction in opposite directions, which the authors call Path Conflict. Using AllPath's attribution on LLaVA-v1.5-7B, they find that conflict concentrates in a few heads and is much stronger in hallucinated samples, mainly because of larger I2T contributions. They then propose a training-free method that picks the top-K heads by Conflict Strength and down-scales their I2T and T2T attention separately. It gives small, consistent gains over AllPath and other baselines on POPE, MCQ-POPE and CHAIR, with MME roughly unchanged, and ablations on pathway choice, scaling factors and head selection.

Critical strengths

The paper points out an interesting and previously overlooked aspect of MLLM hallucination: it is not only caused by insufficient use of visual information, but can also come from directional conflict between the visual (I2T) and textual (T2T) pathways. In hallucinated samples, the I2T contribution is actually larger, yet it points in the opposite direction to T2T. Building on this observation, the authors separately rescale the I2T and T2T attention in a few high-conflict heads at inference time, which reduces object hallucination. I find this observation insightful for understanding why hallucination happens, and I consider it the main contribution of the paper.

Critical weaknesses

1. Key quantities are not defined, which hurts clarity and reproducibility.

The whole analysis and method rely on Cl,h,tI2TC^{I2T}_{l,h,t} and Cl,h,tT2TC^{T2T}_{l,h,t}, but the paper only says it follows AllPath and gives no formula for them, nor what their sign is defined relative to. The value of K, the data used to compute Conflict Strength, and whether α was tuned on the test benchmarks are also not reported. This makes the paper hard to follow for readers unfamiliar with AllPath and makes it unclear whether the hyperparameters were selected on test data.

2. The link between the finding and the method is weak, and the method is largely heuristic.

Conflict information is only used offline to select heads; at inference, all tokens in the selected heads are scaled by fixed α values, regardless of whether a conflict actually occurs. This does not make full use of the paper's main observation. The paper also does not show why down-scaling I2T/T2T resolves the conflict (e.g., whether Conflict Strength decreases after intervention). Moreover, renormalization after scaling shifts attention to other tokens such as the system prompt, which may partly explain the gains. Overall, the method reads more like an effective trick than a mechanism-grounded approach.

3. The empirical evidence is limited, and some results are presented inaccurately.

Gains over AllPath are small (0.4–1.1 on POPE). In Table 4, the Position score of 119.4 is bolded although it is neither the best nor the second best, and the Total is lower than AllPath, which does not match the claim in the text. The (1.0, 1.0) row in Table 5 is partly inconsistent with AllPath in Table 3. All experiments use only LLaVA-v1.5-7B, so the generality of the conclusions is unclear.

Questions for authors
  1. Could the authors give the explicit formulas for Cl,h,tI2TC^{I2T}_{l,h,t} and Cl,h,tT2TC^{T2T}_{l,h,t}, and clarify which token their sign is defined with respect to? On which data is Conflict Strength computed, what value of K is used, and were α and K tuned on the test benchmarks?

  2. The paper argues that conflict causes hallucination, but the method does not detect conflict at inference. After intervention, does Conflict Strength in the selected heads actually decrease? After renormalization, where does the removed attention mass go? A control experiment (e.g., scaling the remaining tokens instead) would help rule out that the gains come from attention redistribution. Also, PAI amplifies attention to image tokens while this method attenuates I2T, and both help; how do the authors reconcile this? (Suggestion: could α be set adaptively based on the per-token conflict, or learned?)

  3. Why is the Position score of 119.4 bolded in Table 4? What setting does the (1.0, 1.0) row in Table 5 correspond to, and why does it differ from AllPath in Table 3 on MCQ-POPE? Specifically, What setting does the (1.0, 1.0) row in Table 5 correspond to? The text calls it the no-intervention baseline, but its results are far better than Vanilla in Table 3 (e.g., CHAIRs 26.9 vs. 52.2) and nearly identical to AllPath on POPE, while differing from AllPath on MCQ-POPE (e.g., 78.1 vs. 80.5 on Random). Is the proposed intervention applied on top of AllPath? If so, this should be stated clearly, since it affects how the gains over AllPath should be interpreted.

Reviewer_t4M5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper identifies Path Conflict as a failure mode of MLLM hallucination: at the attention-head level, the Image-to-Text (I2T) and Text-to-Text (T2T) pathways push token predictions in opposite directions. The authors quantify this via Conflict Ratio and Conflict Strength, report that hallucinated generations show stronger conflict driven by amplified I2T contributions (with T2T roughly unchanged), and propose a training-free Conflict-aware Intervention: select Top-K heads by Conflict Strength and scale I2T/T2T flows by αI2T=0.3\alpha_{I2T}=0.3, αT2T=0.2\alpha_{T2T}=0.2 with renormalization. Experiments on LLaVA-v1.5-7B show improvements over inference-time baselines, including AllPath (Qian et al., 2026), on POPE, MCQ-POPE, and CHAIR.

Critical strengths
  • A clean and novel mechanistic observation. The paper reframes hallucination analysis from pathway strength to pathway directional agreement: hallucinated generations exhibit stronger I2T/T2T conflicts, driven by amplified I2T contributions while T2T remains stable. This offers a fresh angle beyond the prevailing "insufficient visual grounding" narrative and provides a concrete, testable signal (Conflict Strength) that the community can build on.

  • Lightweight, training-free, and clearly presented. The intervention requires no training or parameter updates, making it cheap to deploy and easy to reproduce. The paper is well written, the pathway-level ablation (Table 5) is the right kind of analysis in spirit, and the limitations section is honest about the single-model scope.

Critical weaknesses
  • Incremental over AllPath, with no mechanistic evidence that the method is more fundamental, and an unclear relationship to AllPath. The problem setup, attribution framework, and contribution definitions are all inherited from the predecessor AllPath. Beyond modest performance gains, the paper provides no convincing mechanistic analysis showing that the proposed method is more fundamental than AllPath — e.g., whether directional conflict carries information beyond the pathway magnitudes AllPath already exploits, or how the selected heads differ from AllPath's. Nor does the paper clarify whether it is an all-around improvement over AllPath or a parallel

  • Single, outdated model: practical value is questionable. Hallucination behavior can differ entirely across model generations. LLaVA-v1.5-7B is a classic, but papers from over a year ago already include analysis on Qwen2.5-VL-class models. All experiments here use only LLaVA-v1.5 — a coverage regression even relative to AllPath itself, which validates across alignment formats. Since the paper's core finding (amplified I2T, stable T2T) may be specific to LLaVA, the real-world applicability of the method is doubtful. As the method is training-free, evaluating on at least one recent MLLM is a minimal ask.

Questions for authors
  1. Relationship to AllPath. Does directional conflict provide information beyond the pathway magnitudes AllPath already uses? Please provide: (i) CR/CS measured after AllPath's intervention, and (ii) a combined "AllPath + Ours" row, to clarify whether the method is an all-around improvement or a complementary one.
  2. Interpretation of Eq. (7) and baseline consistency. If I2T and T2T cover all valid keys for a text query, renormalization makes the intervention depend only on the ratio of the two scaling factors; identical factors cancel out entirely. In particular, (0.3, 0.2) is equivalent to(1.5, 1), increasing the relative attention weight of I2T over T2T. How does this support the descriptions of “attenuating both pathways” and “attenuating I2T”? Also, (1,1) should be an identity transformation under Eqs. (6)–(7). Why does this row in Table 5 differ substantially from Vanilla in Table 3?
  3. Generality across model generations. Does the conflict pattern (amplified I2T, stable T2T) replicate on a recent MLLM such as Qwen2.5-VL, and does the intervention remain effective there? Since the method is training-free, this should require minimal effort. (If the authors can satisfactorily address my first two concerns, I would also like to see the additional experiments requested in Question 3 to assess the generality of the findings. )

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.