acceptodds
Under review as a conference paper at ICLR 2027

MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads

20 views

Abstract

Multimodal Large Language Models (MLLMs) often struggle with hallucinations, thus hindering their reliable practical applications. Existing attention-based mitigation methods mainly rely on indirect signals (e.g., attention weights) that fail to accurately reflect the actual information shift underlying hallucination generation. In this paper, we propose HEAL, Head-lEvel information disentAnglement and caLibration for identifying and mitigating hallucinations. HEAL first employs causal noise intervention on multi-head outputs to filter out causally redundant heads. Subsequently, it disentangles information distribution within the remaining heads via the counterfactual Difference-in-Differences, categorizing heads into four types. Through analysis, we observe: hallucinations happen when information distribution drifts away from a healthy equilibrium in synergy heads, not strongly correlated with the quantity or strength of modality-specific heads. Motivated by this insight, HEAL injects dynamic information calibration factors into the value vectors of synergy heads and actively regulates visual-language dependencies, steering the output distribution towards factual evidence. Extensive experiments demonstrate that HEAL effectively reduces hallucinations across multiple MLLMs, offering a simple and interpretable pathway to enhance model trustworthiness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, hallucination, head taxonomy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_av3MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HEAL, which is an inference-time method for reducing multimodal hallucinations.HEAL uses counterfactual interventions to classify attention heads and rescales related value vectors in the heads. The improvement ares upported by several models and benchmarks. The paper also argues that an imbalance within synergy heads helps explain hallucination generation.

Critical strengths

The improvements looks fairly broad to me, and I don't think its just because the answers got shorter. On LLaVA-1.5, HEAL bring sentence-level CHAIR down from 51.0 to 36.7, but response length stay roughly same and object recall even goes up a little (Table 1). Plus the extra models and robustness check in appendix, I'd say it's worth to try as a practical fix.

Critical weaknesses

1.Theorem 1 relies on a information measure that is additive and positively homogeneous. But the scores actually used in Equation 8–10 are difference of cosine similarities, and in general those doesn't have either property. The paper do mention a local approximation, but it never show when that approximation holds. So I'm not convinced that rescaling the value vectors really push the measured information ratio toward to the target. 2.The bidirectional interventions do show that shifting the balance between visual and language information change how much the model hallucinate. What they don't show is whether picking these specific head matters, beyond the general gain you get from reweighting visual information at all. A simple way to check this would be apply the same intervention, at similar strength, to same number of randomly chosen heads or non-synergy heads. That would test the author's explanation directly.

Questions for authors

1.After calibration, how close does the recomputed cosine-based information ratio actually gets to the target α? And does the approximation still holds across different head and generation steps? 2.If you calibrate the selected synergy heads, does that actually works better than apply the same intervention to a matched set of random head or non-synergy heads?

5 comments
AuthorsOpenReview ↗

Response 1 to Reviewer av3M

We thank Reviewer av3M for the thoughtful comments. We address each concern below.

W1: 1.Theorem 1 relies on a information measure that is additive and positively homogeneous. But the scores actually used in Equation 8–10 are difference of cosine similarities, and in general those doesn't have either property. The paper do mention a local approximation, but it never show when that approximation holds. So I'm not convinced that rescaling the value vectors really push the measured information ratio toward to the target.

Q1: After calibration, how close does the recomputed cosine-based information ratio actually gets to the target α? And does the approximation still holds across different head and generation steps?

We want to explain this question from two perspectives.

From a theoretical perspective, we derive an approximation error bound for the implemented cosine-based scores. The idea is to use a second-order Taylor expansion around the original, uncalibrated point, and explicitly account for the nonlinear deviations from the desired scaling. The deviation between the recomputed α\alpha and the target α\alpha for every head or step is bounded by

\leq \frac{ (1-\alpha)\varepsilon_{v} +\alpha\varepsilon_{l} }{ \left| I_{v}^{ori} +I_{l}^{ori} -\varepsilon_{v} -\varepsilon_{l} \right| }.$$ where \$ \varepsilon_{v} \:= |$\left.\frac{\partial I_{v}^{cal}}{\partial\beta}\right|_{(1,1)} - I_v^{ori}$ (\beta-1) \$ \$ \+ \left.\frac{\partial I_{v}^{cal}}{\partial\gamma}\right|_{(1,1)}(\gamma-1)| \$ \$ \+\frac{L_{\mathrm{v}}}{2}\left[(\beta-1)^2+(\gamma-1)^2\right], \$ \$ \varepsilon_{l} \:=|\left.\frac{\partial I_{l}^{cal}}{\partial \beta }\right|_{(1,1)}(\beta - 1) \$ \$ \+ $\left.\frac{\partial I_{l}^{cal}}{\partial\gamma }\right|_{(1,1)} - I_l^{ori}$(\gamma -1)| \$ $+\frac{L_{\mathit{l}}}{2}\left[{\left(\beta -1\right)}^{2}+{\left(\gamma -1\right)}^{2}\right].$ This is a per-head, per-decoding-step bound and its numerical tightness depends on the local derivatives and curvature of the cosine-based scores. From an experimental perspective, we directly compute MAE between the recomputed $\alpha$ and the target $\alpha$ for different heads and generation steps. Specifically, we sample decoding steps at intervals of 10. In each sampling, we divide synergy heads into groups of 50. In each group, we report MAE between the average recomputed $\alpha$ and the target $\alpha$. Table R1: MAE between the recomputed $\alpha$ and the target $\alpha$ (0.4) on Qwen2.5-VL-7B. ||1|2|3|4|5|6|7| |-|-|-|-|-|-|-|-| |10|0.032|0.061|0.093|0.015|0.021|0.057|0.108| |20|0.015|0.056|0.052|0.102|0.023|0.007|0.043| |30|0.096|0.004|0.081|0.012|0.021|0.045|0.001| |40|0.052|0.009|0.026|0.073|0.057|0.054|0.034| |50|0.081|0.038|0.088|0.017|0.097|0.075|0.032| Table R2: MAE between the recomputed $\alpha$ and the target $\alpha$ (0.5) on LLaVA-1.5-7B. ||1|2|3|4|5|6|7| |-|-|-|-|-|-|-|-| |10|0.002|0.013|0.106|0.068|0.029|0.042|0.051| |20|0.103|0.075|0.021|0.072|0.058|0.037|0.023| |30|0.046|0.035|0.088|0.021|0.017|0.107|0.032| |40|0.072|0.059|0.036|0.104|0.028|0.074|0.014| |50|0.001|0.028|0.046|0.005|0.053|0.033|0.098| These results provide a direct empirical assessment of whether value-space calibration moves the information ratio toward the target. Most errors are within the range of 0.005-0.095, which demonstrates that the calibration generally moves the information ratio toward the target, though the vector rescaling isn’t very exact.
AuthorsOpenReview ↗

Response 2 to Reviewer av3M

W2: The bidirectional interventions do show that shifting the balance between visual and language information change how much the model hallucinate. What they don't show is whether picking these specific head matters, beyond the general gain you get from reweighting visual information at all. A simple way to check this would be apply the same intervention, at similar strength, to same number of randomly chosen heads or non-synergy heads. That would test the author's explanation directly.

Q2: If you calibrate the selected synergy heads, does that actually works better than apply the same intervention to a matched set of random head or non-synergy heads?

We agree that it is necessary to distinguish the benefit of calibrating synergy heads from the general benefit of visual reweighting in arbitrary attention heads. We conduct a controlled comparison among three intervention strategies: (1) randomly selected heads, (2) non-synergy heads, and (3) the synergy heads identified by HEAL. All three strategies use exactly the same number of heads as well as comparable intervention strength (the target α\alpha is set to 0.5). The decoding configuration is kept identical across all settings.

Table R3: Performance comparison on Qwen2.5-VL-7B when intervening different groups of heads.

POPE F1↑LLaVA-Bench↑
Baseline87.476.8
Random Heads87.877.1
Non-synergy Heads87.575.9
Synergy Heads88.478.5

We observe that random-head calibration provides some improvements over the baseline, suggesting that visual reweighting can have a general mitigating effect. However, calibrating the synergy heads identified by HEAL achieves consistently greater improvements. Additionally, calibrating non-synergy heads has a tiny increase in POPE F1, but a decrease in LLaVA-Bench score. Non-synergy heads include visual, language and redundant heads. For visual heads, they don’t need visual reweighting, because they are already visual-sensitive enough. Calibrating redundant heads has little influence on the output. For language heads, intervening on them may damnify the general linguistic expressiveness, which may have a negative effect on some overall metrics, like the LLaVA-Bench score.

These results provide further evidence for the importance of the proposed synergy-head identification and support our central claim that hallucination is associated with information-distribution drift within these heads.

AuthorsOpenReview ↗

Discussion Window Coming to Close

Dear Reviewer av3M,

Thank you again for your valuable comments!

As the discussion phase is coming to a close, please let us know whether we have satisfactorily addressed your concerns and questions in the revised version. If you have any further questions, please feel free to raise them. We are expecting you to re-consider our work.

Many thanks,

Reviewer_DExsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hallucination in multimodal large language models (MLLMs) and proposes HEAL (Head-lEvel information disentAnglement and caLibration), a training-free, inference-time method. HEAL first replaces each attention head's output with distribution-matched Gaussian noise to filter out causally redundant heads. Inspired by Partial Information Decomposition (PID), it then constructs four counterfactual inputs by masking visual and/or language tokens, and uses a Difference-in-Differences scheme to estimate the visual information, language information, and interaction term of each head. Based on these scores, heads are categorized into redundant, visual, language, and synergy heads. Using this taxonomy, the authors observe that when hallucinated tokens are generated, the number and strength of visual/language heads differ little from those for correct tokens; the main difference is that the visual-language information ratio within synergy heads drifts toward language. They also observe that head types change dynamically with the nature of the generated token. Motivated by this, HEAL introduces an equilibrium factor α and rescales the value vectors of visual and language tokens in synergy heads toward a target ratio, pulling the information distribution back to equilibrium. The paper provides theoretical analysis showing that value-space scaling is equivalent to calibrating the information ratio, and that the alignment between the head output and the visual component increases monotonically with α. For efficiency, head types are updated periodically, and the noise interventions and counterfactual computations are parallelized and batched. Experiments on 7 models from the LLaVA, Qwen-VL, and InternVL families and on POPE, CHAIR, MMHal-Bench, LLaVA-Bench, MME, and BLINK-Twice show consistent improvements on both hallucination and general benchmarks.

Critical strengths
  1. The analytical perspective has some novelty. Unlike prior work that relies directly on attention weights or simply "strengthens visual heads", this paper attempts to measure visual and language contributions at the granularity of individual attention heads via counterfactual interventions, and attributes hallucination to an imbalance in the modality information ratio rather than to the weakness of a single modality. This "rebalance rather than unilaterally amplify" view offers an interpretable angle on the vision-language see-saw problem, and is consistent with HEAL's relatively high recall and generation length on CHAIR.
  2. The method is training-free, plug-and-play, and engineering-friendly. Calibration is applied after the KV-cache update and before the attention kernel, leaving the internals of FlashAttention / PagedAttention untouched; the periodic type update, parallelized intervention, and batched DiD reflect practical deployment considerations.
Critical weaknesses

1. The notion of "synergy head" is a misnomer, and the paper provides no intuitive or concrete analysis of these heads.

The central claim of the paper is that hallucination arises from information-distribution drift within synergy heads, yet the identification of synergy heads has nothing to do with synergistic information itself. In Sec. 3.2, right after defining I_syn = I_total − I_vis − I_lang, the authors state that "I_syn is not the true synergy measure … our subsequent categorization primarily rely on the visual and language information content". The actual rule is that a synergy head is any "remaining" head with I_vis > 0 and I_lang > 0 whose visual ratio does not exceed the median ± λ·MAD threshold. The calibration factors β and γ (Eq. 16) likewise depend only on α_vis and α_lang. In other words, I_syn is not used in categorization, calibration, or any experiment; it only appears as one axis of Fig. 2(c) and in the theoretical discussion of Appendix C.2, where the authors further note that I_syn ≈ S − R, i.e., not synergy in the PID sense. The so-called synergy heads are therefore essentially "mixed heads to which both modalities contribute with a moderate ratio", and naming them after PID synergy is inaccurate.

More importantly, the paper offers no intuitive or concrete analysis of synergy heads: it does not show which tokens a typical synergy head attends to or on what kind of generated content it is active; it does not report the distribution of I_syn, nor compare I_syn between hallucinated and correct tokens; and it does not distinguish the roles in hallucination of heads with I_syn > 0 (both modalities are indispensable) versus I_syn < 0 (the modalities are substitutable). Since this is the core concept on which the paper's title and argument rest, readers are left unable to understand what function "synergy heads" actually serve, or what new insight they provide beyond existing analyses of visual heads and language priors (e.g., VHR).

2. The method is relatively simple, its practical gains are limited (especially on newer models), and it depends on a manually set α.

The intervention essentially rescales the value vectors of visual/language tokens in synergy heads with a global factor, which is close in spirit to existing attention-head reweighting methods (e.g., VHR, PAI). Given this, the empirical gains are not remarkable and shrink noticeably as the base model gets stronger:

  • The largest gain is on LLaVA-1.5-7B (CHAIR_S 51.0 → 36.9), whereas on Qwen2.5-VL-7B, CHAIR_I only goes from 9.0 to 8.5 and POPE-F1 improves by only +1.0 (Table 2);
  • On the newer Qwen3-VL-8B / InternVL3.5-8B, the MMHal hallucination rate drops by only 0.9 / 1.1 points, and the score improves by only +0.09 / +0.18 (Table 15).

The paper does not analyze why the gains are smaller on newer models. Notably, Fig. 1, which supports the core observation, is obtained on Qwen3-VL-8B; if information drift were indeed the main cause of hallucination and HEAL corrected it, it would be hard to explain why the improvement on this model is so limited.

In addition, performance depends on the equilibrium factor α. LLaVA/InternVL and Qwen families require different values of α, and the same model needs different values for "simple" and "complex" tasks (Sec. 4.1), a split that requires knowing the task type in advance in practice. An overly large α leads to ungrammatical or degenerate outputs (Fig. 5(b)). In Appendix D.4, the authors acknowledge that they cannot provide a principled rule for choosing α, offering only an empirical range of 0.4–0.6 to be "slightly adjusted according to the visual capability of the target model". This weakens the method's practicality as a plug-and-play, training-free solution.

Questions for authors
  1. Could you provide a concrete and intuitive analysis of synergy heads? For example, on which types of tokens are typical synergy heads active, and which visual/language tokens do they mainly attend to? Do heads with I_syn > 0 and I_syn < 0 behave differently on hallucinated versus correct tokens? If I_syn plays no role in categorization or calibration, what justifies calling these heads "synergy heads"?
  2. Why are the gains of HEAL on newer models (Qwen2.5-VL, Qwen3-VL, InternVL3.5) clearly smaller than on LLaVA-1.5? Could you report, for different models, the gap in the within-synergy-head information ratio between hallucinated and correct tokens, as well as the fraction of hallucinated tokens that are actually accompanied by a clear information drift?
  3. α is defined as the desired visual-language equilibrium when generating correct tokens. Could α then be estimated directly from data, e.g., by using the actual visual information ratio of correct tokens within synergy heads (about 0.43:0.54 in Table 6) as the target, instead of setting it manually per model family and task type? Moreover, a single fixed α is currently applied to all synergy heads at all decoding steps, whereas the paper itself observes that head roles differ markedly between generating language-centric tokens and visually grounded tokens. Would adapting α to the token type alleviate the ungrammatical and degenerate outputs observed when α is too large?
7 comments
AuthorsOpenReview ↗

Response 1 to Reviewer DExs

We thank Reviewer DExs for the thoughtful comments. We address each concern below.

W1: The notion of "synergy head" is a misnomer.

Q1: If I_syn plays no role in categorization or calibration, what justifies calling these heads "synergy heads?

We are sorry for arousing some misunderstandings, and we are using I_res (residual) to replace it. But we believe that "synergy head" can describe these heads better. These heads contain not only pure language and pure visual information but also synergistic interactions, which the typical synergy head is. "mixed heads" merely shows the coexistence of two modalities, not reflecting synergistic interactions.

According to the classification rule in Lines 249- 274, the remaining heads are those heads to which both modalities contribute with a moderate ratio, and most of them have significant synergistic interactions with I_res = I_total − I_vis − I_lang = S – R = true synergy effect - evidence supported prior information > 0. We note that, in our decomposition (Figure 2(d)), we divide the unimodality-induced prior information into evidence-supported (redundant) and evidence-unsupported parts.

From an analytical perspective, I_res in the remaining heads is greater than 0 in most cases, because I_res <= 0 indicates modality substitutability or evidence-supported redundancy. However, visual and linguistic inputs generally provide complementary information: visual evidence contains details that cannot generally be inferred from the linguistic context alone, while language provides semantic constraints that are not available from visual input alone. Consequently, the information contributed by their joint interaction is expected to exceed the redundant information in most cases.

Consistent with this analysis, in our robustness experiments, only approximately 0.8% of the 9,000 token-level observations have I_res <= 0. More importantly, these rare cases are associated with relatively strong evidence-supported prior information but little or no synergistic information. Such prior information can already benefit the model's predictions, meaning that these heads do not require additional calibration. Meanwhile, because they contain little or no synergistic information, calibrating them is also expected to have little impact on the overall performance. This is consistent with our pre-experiment, which shows nearly identical POPE-F1 scores with and without retaining these cases (87.71 vs. 87.72).

However, we do not perform an additional step to filter I_res <= 0 in the practical implementation. This is a deliberate simplification, and filtering these rare cases introduces additional computational and decision-making overhead without providing a meaningful performance benefit. We thus directly treat the remaining heads after the modality-based classification as synergy heads in the actual implementation. We will explain this process in detail in the revised manuscript. About their name, most of the remaining heads have I_res > 0 (S > 0) and “synergy” may describe them better.

About “I_syn is not used in calibration, or any experiment”, as discussed in Lines 928-932, it is very difficult to compute R, so we can’t get or directly intervene on the synergy effect. This does not mean that we are not calibrating synergistic heads; rather, we regulate the two modality-specific components that jointly form their multimodal interaction representations.

AuthorsOpenReview ↗

Response 2 to Reviewer DExs

W1: the paper offers no intuitive or concrete analysis of synergy heads

Q1: on which types of tokens are typical synergy heads active, and which visual/language tokens do they mainly attend to? Do heads with I_syn > 0 and I_syn < 0 behave differently on hallucinated versus correct tokens?

First, about “the paper offers no intuitive or concrete analysis of synergy heads”, we would like to clarify that studying synergy heads thoroughly may exceed our paper’s topic that is more foucsing on the relationship between hallucinations and modality contributions and their synergy. As illustrated in Figures 2(a) and 2(b) and 7, synergy heads are observed across different generation positions, including both language-centric tokens (e.g., "and") and visually grounded tokens (e.g., "15", "right", and "RE"). This suggests that their role is not restricted to a single semantic token category. For multimodal LLMs, their output should also be the synergistic outcome of all modalities.

At the same time, as is discussed in Lines 81-88 and 272-273, for language/visual-preferred synergy heads, they have some preference patterns and may be more active in some specific content/tokens. When producing language-centric tokens, some language-preferred synergy heads tend to revert toward language heads, and language heads increase; when generating visually grounded tokens, some language heads shift back toward synergy heads, and a small subset of visual-preferred synergy heads may further move toward visual heads. This dynamic transition behavior highlights the adaptive nature of Transformer attention.

Second, about “it does not report the distribution of I_syn, nor compare I_syn between hallucinated and correct tokens”, in Figure 1(c) we will further complement the mean synergy information content (I_syn) of synergy heads when generating hallucinated/correct tokens.

According to our previous statistical results, correct tokens have a greater mean I_syn than hallucinated tokens (0.441:0.283 in LLaVA-NeXT, 0.496:0.307 in Qwen3-VL). High synergy effects tend to require good multimodal cooperation (i.e., an equilibrium visual-language distribution), and it helps the model ground its generated content in visual evidence. This is also consistent with our claim.

Third, about “it does not distinguish the roles in hallucination of heads with I_syn > 0 (both modalities are indispensable) versus I_syn < 0 (the modalities are substitutable)”, as explained above, from an analytical perspective, I_res in the remaining heads is greater than 0 in most cases, because I_res <= 0 indicates modality substitutability or evidence-supported redundancy. However, visual and linguistic inputs generally provide complementary information: visual evidence contains details that cannot generally be inferred from the linguistic context alone, while language provides semantic constraints that are not available from visual input alone. Consequently, the information contributed by their joint interaction is expected to exceed the redundant information in most cases.

Consistent with this analysis, only approximately 0.8% of the 9,000 token-level observations in our robustness experiments have I_res <= 0. More importantly, these rare cases are associated with relatively strong evidence-supported prior information but little or no synergistic information. Such prior information can already benefit the model's predictions, meaning that these heads do not require additional calibration. Meanwhile, because they contain little or no synergistic information, calibrating them is also expected to have little impact on the overall performance. This is consistent with our pre-experiment, which shows nearly identical POPE-F1 scores with and without retaining these cases (87.71 vs. 87.72).

Therefore, hallucinations are associated with synergy heads with I_syn > 0. But considering the practical implementation and cost, we don’t further identify synergy heads with I_syn <= 0, and they have little impact on our results.

Q2: Could you report ... by a clear information drift?

Table R2: Information ratios and fractions of hallucinated tokens for different models.

Modelcorrecthallucinatedfraction
Qwen2.5-VL-7B0.408:0.5920.278:0.72278.5
Qwen3-VL-8B0.453:0.5470.354:0.64669.5
InternVL3.5-8B0.439:0.5610.314:0.68675.0

These results show that hallucinated tokens are associated with the information distribution drift within synergy heads. We also note that not every hallucinated token that is accompanied by an information drift can be solved by our methods. Some of these hallucinated tokens are simultaneously caused by multiple factors (Lines 133-141 and 482-485), one of which is the visual-language information drift. For example, hallucinations jointly caused by visual-language disequilibrium and early visual encoding failures cannot be resolved by attention head calibration alone.

AuthorsOpenReview ↗

Response 3 to Reviewer DExs

W2: The method is relatively simple, its practical gains are limited (especially on newer models), and it depends on a manually set α.

About “The intervention essentially rescales ... which is close in spirit to existing attention-head reweighting methods (e.g., VHR, PAI)”, we agree that the final calibration is, to some extent, vector reweighting. However, we respectfully emphasize that the paper goes beyond vector reweighting in many aspects that are our core contributions.

First, as discussed in Lines 44- 49, most attention-based mitigation methods (like MemVR, VHR) only boost visual evidence or inject visual information, which often leads to a “see-saw” dilemma. This motivates our central question: what heads need to be calibrated /re-weighted and how to calibrate them to improve visual evidence while keeping models’ expressiveness.

To this end, we use DiD to decompose information and identify these need-to-be-calibrated heads that are associated with hallucinations. In these synergy heads, some have more visual information, and others have more language information. We re-weight to balance the visual and language, not just to enhance the uni-modality.

The key is how we identify these heads and decompose information in multi-modal heads, how we get the calibration factor values and how we reduce hallucinations while maintaining expressiveness, though the final step is reweighting.

Then, we provide the theoretical analysis of why vector reweighting works and how vector reweighting changes the output, which the prior reweighting works just lack. In addition, we consider not only visual evidence and language priors, but also their cross-modal interaction and the distinction between modality-induced evidence-supported and unsupported prior information.

About “The paper does not analyze why the gains are smaller ... why the improvement on this model is so limited”, we would like to point out that our paper does not claim that the information drift is the only, or necessarily the dominant, cause of hallucination. Our paper says “MLLMs hallucinate when information distribution drifts away from a healthy equilibrium in synergy heads” and “We identify visual-language information disequilibrium within synergy heads as an intervenable factor that can influence hallucination behavior”. We admit the information drift is one of the causes that lead to hallucinations. In fact, as is discussed in Lines 482-485, hallucinations, like caused by early visual encoding failures or missing visual evidence cannot be resolved by attention head calibration alone.

About the smaller gains, we are sorry for not providing the comparison results, and we will supplement the comparison of different methods on newer models, as another reviewer suggested. We also want to explain the question from the following perspectives:

First, the connection between the information drift extent in Fig. 1 and improvement values is not necessary. For example, supposing older models have 300 hallucinated tokens and newer models have 100 hallucinated tokens, we can observe the information drift across these 300 tokens or these 100 tokens. For older models, we will have more hallucinated tokens to solve. Some metrics reflect the number of solved hallucinated tokens/sentences, like CHAIR_S, CHAIR_I. Naturally, improvements on older models are generally larger than those on newer models. Especially, LLaVA-1.5 has severe inherent hallucinations. The gain on LLaVA-1.5-7B is much larger than on Qwen2.5-VL-7B.

Second, metric computation and design are different. CHAIR_S measures token-level hallucinations and CHAIR_I measures sentence-level hallucinations. Generally, CHAIR_I is much smaller than CHAIR_S, because one sentence consists of many tokens. MMHal-Bench score is a comprehensive metric that not merely measures the hallucination reduction. So, even a small gain on this metric is difficult.

Third, compared to older models (LLaVA-1.5), newer models (Qwen2.5-VL, Qwen3-VL, InternVL3.5) are more likely to learn on these datasets/images in the training stage (please see their technical reports). In this situation, a small gain on newer models is also very difficult. A similar gap can also be observed in recently released papers, like LocoRE (ICLR 2026) and LEAD (CVPR 2026).

Table R1: Performance comparison on Qwen2.5-VL.

MethodLLaVA-Bench↑CHAIR$_S$ ↓CHAIR$_I$ ↓POPE-R↑POPE-F1↑POPE-A↑Expressiveness
Qwen2.5-VL-7B76.827.29.080.487.488.49.25
+MemVR77.424.28.381.487.988.79.03
+VHR77.925.38.180.687.788.28.08
+FarSight77.324.79.476.987.187.68.74
+OPERA78.226.48.881.888.288.78.32
+HEAL78.523.38.581.388.489.19.14

Note. To assess not only hallucination mitigation but also the quality of generated language, we additionally evaluate expressiveness following the setting used in FarSight.

Reviewer_xb1VOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that MLLM hallucinations occur when the visual-language information balance drifts in multimodal "synergy heads". The authors propose HEAL, an inference-time intervention:

  1. It uses causal noise replacement to weed out redundant attention heads.
  2. It applies a 4-way counterfactual Difference-in-Differences (DiD) masking scheme to classify heads into visual, language, and synergy types.
  3. During decoding, it rescales the value vectors (VvisV_{vis} and VlangV_{lang}) of synergy heads using a manual equilibrium factor α\alpha to steer outputs back to visual evidence.
  4. Evaluations are run on LLaVA, Qwen, and InternVL backbones across POPE, CHAIR, MME, and BLINK-Twice.
Critical strengths
  1. Tracking how attention heads transition between linguistic and visual roles during autoregressive decoding is intuitive and well-visualized.

  2. The authors did a solid job testing different masking types, noise distributions, and threshold variations in the appendix.

Critical weaknesses
  1. Rescaling visual vs. text value vectors during decoding is an old idea dressed up in fancy econometric terms like "Difference-in-Differences" and PID. Stripping away the heavy notation, this is just another heuristic value-vector re-weighting trick.

  2. The primary comparison against other methods in Table 1 relies almost entirely on LLaVA-1.5-7B. LLaVA-1.5 is a 2023 model with severe inherent hallucinations; any visual-boosting trick looks great on it. Showing competitive SOTA tables only on such an obsolete backbone is unacceptable for an ICLR 2027 submission.

  3. In Table 2 and Table 15, newer models like Qwen2.5-VL and InternVL3.5 are only compared against their own vanilla versions (Base vs Base + HEAL). Competitive inference-time methods like MemVR, VHR, FarSight, and OPERA are completely missing on these newer architectures. We have no idea if HEAL actually beats existing methods on modern MLLMs.

  4. Running 4 counterfactual passes (H11,H01,H10,H00H_{11}, H_{01}, H_{10}, H_{00}) across all heads and all layers creates massive computational spikes. The paper claims batched tensor operations solve this, but it noticeably omits real-world wall-clock latency (ms/token) and peak VRAM measurements.

  5. The factor α\alpha is entirely manual and sensitive. As shown in Figure 4(b) and Table 3, deviating slightly from the narrow sweet spot hurts performance, and α=0.7\alpha=0.7 even causes degenerative word loops like "six zebra zebras". A static, manually hand-tuned parameter is brittle for real-world deployment.

Questions for authors
  1. What is the exact generation speed (tokens/sec) and peak memory overhead of HEAL compared to standard greedy decoding and MemVR on an NVIDIA GPU?

  2. In Figure 5(b), why does slightly higher visual scaling cause the model to repeat words ("six zebra zebras")? Does value vector manipulation degrade language modeling fluency?

  3. Since α\alpha varies between simple and challenging tasks (0.4 vs 0.6), how can a practitioner deploy HEAL when prompts mix fine-grained OCR, counting, and creative writing in the same session?

  4. In Equation 11, the synergy score IsynI_{syn} can be negative. How often does this occur, and does a negative score distort the classification of synergy heads?

5 comments
AuthorsOpenReview ↗

Response 1 to Reviewer xb1V

We thank Reviewer xb1V for the thoughtful comments. We address each concern below.

W1: Rescaling visual vs. text value vectors ... another heuristic value-vector re-weighting trick.

We agree that the final calibration is the vector re-weighting to some extent. However, we respectfully emphasize that the paper goes beyond vector re-weighting in many aspects that are our core contributions.

First, as discussed in Lines 44- 49, most attention-based mitigation methods (like MemVR, VHR) only boost visual evidence or inject visual information, which often leads to a “see-saw” dilemma. This motivates our central question: what heads need to be calibrated /re-weighted and how to calibrate them to improve visual evidence while keeping models’ expressiveness.

To this end, we use DiD to decompose information and identify these need-to-be-calibrated heads that are associated with hallucinations. In these synergy heads, some have more visual information, and others have more language information. We re-weight to balance the visual and language, not just to enhance the uni-modality.

The key is how we identify these heads, how we decompose information in multimodal heads, how we get the calibration factor values, and how we reduce hallucinations while maintaining expressiveness, though the final step is reweighting.

Second, going beyond the value-vector reweighting trick, we provide the theoretical analysis of why vector reweighting works and how vector reweighting changes the output, which the prior reweighting works just lack.

Third, we apply the econometric DiD based on what our information decomposition really needs. PID is the classical applied mathematical work to decompose multimodal information, and naturally we need to discuss the relationship between it and our decomposition. PID is difficult to instantiate in neural networks, and our method provides a practical decomposition instantiation for multimodal transformers.

Finally, regarding α\alpha, we provide an empirical range and guideline, as well as a theoretical interpretation rather than treating it as an arbitrary heuristic constant. In practice, there are some ways for deployment:

(1) Practitioners can select a factor in the range of 0.4-0.6 using a representative validation set containing a mixture of task types, which may be acceptable for both simple and difficult tasks

(2) When computational resources permit, service providers can provide multiple responses using different α\alpha values in this range, allowing users to choose one. This is similar in spirit to the functionality in the ChatGPT website version.

(3) Supported by Theorem 2, the factor has a geometric interpretation. Companies can expose it as an optional control, allowing users to adjust the extent to which they want the response to rely on visual evidence versus linguistic flexibility and creativity.

In summary, although the final operation involves reweighting, evaluating the novelty solely from the final operation would overlook our core contributions.

W2: The primary comparison ... is unacceptable for an ICLR 2027 submission.

We agree that evaluating methods on newer MLLMs is important. We initially used LLaVA-1.5 as the primary backbone because a substantial number of existing methods (not only the old but also the newer) all report results on this model, making it possible to conduct a more consistent comparison across methods. We complement the comparison results in newer models below.

Q4: In Equation 11, the synergy score can be negative. How often does this occur, and does a negative score distort the classification of synergy heads?

In Equation 11 and 65, I_syn = I_total − I_vis − I_lang = S – R = true synergy effect - evidence supported prior information. In most cases, I_syn > 0, because I_syn <= 0 indicates substantial modality substitutability or evidence-supported redundancy. However, visual and linguistic inputs generally provide complementary information: visual evidence contains details that cannot generally be inferred from the linguistic context alone, while language provides semantic constraints that are not available from visual input alone. Consequently, the information contributed by their joint interaction is expected to exceed the evidence-supported redundancy in most cases.

Empirically, negative values are also uncommon: in our robustness experiments, approximately 0.8% of the 9,000 analyzed token-level observations have I_syn <= 0. Importantly, the head categorization procedure does not directly threshold I_syn. Based on the visual and linguistic information contributions, we first exclude redundant and modality-specific heads; the remaining heads belong to one category. And almost every remaining head contains visual, language information, and positive I_syn, so we name them synergy heads. Therefore, the negative score has no impact on the classification.

AuthorsOpenReview ↗

Response 2 to Reviewer xb1V

W3: In Table 2 and Table 15, newer models like Qwen2.5-VL and InternVL3.5 are only compared against their own vanilla versions (Base vs Base + HEAL). Competitive inference-time methods like MemVR, VHR, FarSight, and OPERA are completely missing on these newer architectures. We have no idea if HEAL actually beats existing methods on modern MLLMs.

We complement the comparison results in newer models as follows:

Table R1: Performance comparison on Qwen2.5-VL.

MethodLLaVA-Bench↑CHAIR$_S$ ↓CHAIR$_I$ ↓POPE-R↑POPE-F1↑POPE-A↑Expressiveness
Qwen2.5-VL-7B76.827.29.080.487.488.49.25
+MemVR77.424.28.381.487.988.79.03
+VHR77.925.38.180.687.788.28.08
+FarSight77.324.79.476.987.187.68.74
+OPERA78.226.48.881.888.288.78.32
+HEAL78.523.38.581.388.489.19.14

Table R2: Performance comparison on the recent benchmark and MLLM.

MethodMMHal-Bench
Halluc. Rate↓Score↑Expressiveness
InternVL3.5-8B19.44.539.12
+MemVR19.04.638.96
+VHR18.34.658.01
+FarSight18.54.678.49
+OPERA18.14.598.15
+HEAL18.34.719.03

Note. To assess not only hallucination mitigation but also the quality of generated language, we additionally evaluate expressiveness following the same setting used in FarSight. This assessment considers grammar, fluency, naturalness, and richness, and reports their average score.

These additional results provide a broader comparison in recent MLLMs. HEAL achieves effective hallucination reduction while retaining relatively strong expressiveness.

W4: Running 4 counterfactual passes across all heads and all layers creates massive computational spikes. The paper claims batched tensor operations solve this, but it noticeably omits real-world wall-clock latency (ms/token) and peak VRAM measurements.

Q1:What is the exact generation speed (tokens/sec) and peak memory overhead of HEAL compared to standard greedy decoding and MemVR on an NVIDIA GPU?

We have conducted an additional efficiency evaluation in Table R3. Specifically, we randomly sampled 50 image-caption pairs from the CHAIR benchmark and measured the average inference efficiency under the same setting for all methods (batch size = 1, 1 × NVIDIA RTX Pro 5000).

Table R3: Performance and computational overhead on Qwen2.5-VL-7B.

MethodLLaVA-Bench↑Expressiveness↑tokens/sPeak GPU Memory (GB)wall-clock latency (s)
Qwen2.5-VL-7B(base)76.89.2517.2515.5311.36
MemVR77.49.0314.3716.6813.75
VCD77.68.937.1118.1728.41
OPERA78.28.324.5323.3238.58
HEAL(ours)78.59.149.1223.8820.53

These results suggest that HEAL offers a favorable trade-off between hallucination mitigation, generation quality, and inference efficiency, although it does incur additional computational costs compared with standard greedy decoding.

W5: The factor α\alpha is ... A static, manually hand-tuned parameter is brittle for real-world deployment.

Q3: Since varies between simple and challenging tasks (0.4 vs 0.6), how can a practitioner deploy HEAL when prompts mix fine-grained OCR, counting, and creative writing in the same session?

In Lines 1274-1320, we offer a guideline on choosing the equilibrium factor, showing that the favorable range is 0.4-0.6. In practice, there are some ways for deployment:

(1) Practitioners can select a factor in the range of 0.4-0.6 using a representative validation set containing a mixture of task types, which may be acceptable for both simple and difficult tasks

(2) When computational resources permit, service providers can provide multiple responses using different α\alpha values in this range, allowing users to choose one. This is similar in spirit to the functionality in the ChatGPT website version.

(3) Supported by Theorem 2, the factor has a geometric interpretation. Companies can expose it as an optional control, allowing users to adjust the extent to which they want the response to rely on visual evidence versus linguistic flexibility and creativity.

Q2: In Figure 5(b), why ... degrade language modeling fluency?

The example in Figure 5(b), where the model generates a repetitive phrase such as “six zebra zebras,” illustrates the potential downside of excessively increasing the visual contribution. As discussed in Lines 43-48, overly strengthening one modality can lead to a “see-saw” dilemma: excessive reliance on visual information may interfere with linguistic generation, whereas excessive reliance on language can increase visual hallucinations.

This observation motivates the central design of HEAL. Rather than continuously increasing visual information, HEAL aims to improve visual grounding while preserving linguistic coherence and expressiveness. The degenerate output is an example caused by excessive visual scaling, rather than evidence that our vector manipulation necessarily degrades fluency.

AuthorsOpenReview ↗

Discussion Window Coming to Close

Dear Reviewer xb1V,

Thank you again for your valuable comments!

As the discussion phase is coming to a close, please let us know whether we have satisfactorily addressed your concerns and questions in the revised version. If you have any further questions, please feel free to raise them. We are expecting you to re-consider our work.

Many thanks,

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair and all reviewers for their thoughtful comments and constructive suggestions. We summarize our main clarifications and additions below.

1. The methodological contributions of HEAL

We clarify that HEAL is not merely a value-vector reweighting trick. Its main contribution lies in an analytical perspective: how we decompose information in multimodal heads and how we reduce hallucinations while maintaining language expressiveness. Unlike approaches that simply enhance visual information, HEAL aims to regulate the balance between visual evidence and linguistic information, mitigating hallucinations while preserving language expressiveness. We further clarify the theoretical motivation and representation-level effects of this calibration.

2. The definition and role of synergy heads

We further clarify why we retain the term synergy heads rather than mixed heads. These heads are characterized by substantial contributions from both visual and linguistic modalities, rather than merely the coexistence of the two. We also explain the distinction between the modality-based operational classification and the residual interaction measure (I_{\mathrm{res}}). A theoretically extended identification rule can additionally retain heads with (I_{\mathrm{res}}>0), indicating that synergistic interaction exceeds redundant information. However, since non-positive cases are extremely rare (approximately 0.8% of the 9,000 token-level observations in our robustness experiments), explicitly computing and filtering them introduces additional overhead with negligible practical benefit. We therefore omit this additional step in the actual implementation. We also provide further analysis of head behaviors across generated tokens and compare the residual interaction scores between correct and hallucinated tokens.

3. Strengthening the theoretical and empirical validation of Theorem 1

To address Reviewer av3M’s concern about applying the theoretical analysis to cosine-based information scores, we derive an error bound for the deviation between the recomputed information ratio and the target equilibrium factor. We further evaluate this deviation across multiple generation steps and groups of synergy heads on both Qwen2.5-VL-7B and LLaVA-1.5-7B. The results provide empirical evidence that the calibration moves the measured information ratio toward the target. In addition, we conduct controlled interventions on randomly selected heads, non-synergy heads, and HEAL-identified synergy heads. The results show that calibrating the identified synergy heads yields greater improvements than the alternative head-selection strategies, supporting the importance of head identification rather than visual reweighting alone.

4. Expanding evaluations in newer models and discussing practical considerations

We extend comparisons with existing hallucination mitigation methods to newer MLLMs, including Qwen2.5-VL-7B and InternVL3.5-8B, and report both hallucination-related metrics and language expressiveness. We also provide additional analyses of visual–linguistic information ratios in correct and hallucinated tokens across different model families. These results support the relevance of information-distribution drift while clarifying that it is one intervenable factor among multiple potential causes of hallucination, rather than the sole explanation.

Furthermore, we report additional inference-efficiency measurements, including generation throughput, peak GPU memory, and wall-clock latency. We acknowledge the computational overhead introduced by HEAL and discuss its trade-offs relative to existing methods. Finally, we provide more detailed guidance on selecting the equilibrium factor (\alpha), discuss data-driven selection on representative mixed-task validation sets, and clarify the limitations of token-type-dependent adaptation under autoregressive decoding.

We hope these clarifications, theoretical developments, and additional experiments have strengthened the paper. More importantly, we reinforce the central message of HEAL: understanding and mitigating MLLM hallucinations requires not only enhancing visual evidence, but also identifying and regulating the internal visual–linguistic information distribution in the attention heads that jointly process both modalities.

We sincerely appreciate the reviewers' time and feedback, and hope that our revisions can address their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.