acceptodds
Withdrawn from ICLR 2027

From Detection to Control: Hallucination Heads as Text-Side Actuators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models remain prone to object hallucination, generating object mentions unsupported by visual evidence. Recent work shows that intervening on particular attention heads can mitigate hallucination, but why these heads constitute effective control points remains unclear. We show that raw text-side reliance in hallucination-related heads is only weakly diagnostic at the token level, even though interventions on selected heads affect hallucinated predictions much more strongly than grounded ones. Hallucinated object predictions are more text-reliant on average, but their distributions overlap substantially with those of grounded predictions. In contrast, attenuating selected text-side pathways disproportionately reduces hallucinated-object probabilities while minimally affecting grounded predictions. We therefore reinterpret these heads as text-side actuators: pathways whose text-side contribution exerts selectively stronger influence on unsupported continuations. Building on this finding, we introduce DEACT, a training-free inference-time method with model-specific calibration that identifies actuator heads using complementary pathway-strength and hallucination-contrastive statistics and regulates their text-side contribution according to the current decoding state. Across multiple LVLM architectures, DEACT consistently reduces object hallucination while maintaining grounded-generation quality. More broadly, our results show that weak diagnostic separability can coexist with strong causal controllability, suggesting that effective hallucination mitigation need not depend on reliable online failure detection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Vision-Language Models, Multimodal Hallucination, Object Hallucination, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Un4yOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose DEACT, a training-free method that identifies hallucination-related attention heads and dynamically regulates their text-side contributions during decoding. Experiments across multiple LVLMs and benchmarks demonstrate its effectiveness in reducing object hallucinations while largely preserving generation quality.

Critical strengths

The experiments cover multiple models and demonstrate clear advantages over baseline methods, with relatively comprehensive ablation studies.

Critical weaknesses
  1. Limited novelty and substantial overlap with existing attention-head intervention methods. The proposed method substantially overlaps with prior work on hallucination mitigation through attention-head interventions, such as AD-HH. Identifying hallucination-related attention heads and subsequently intervening on them has become a common paradigm. Although this paper introduces modifications to head selection and dynamic suppression strategies, these changes primarily concern implementation details rather than fundamental differences in the underlying mechanism.
  2. Insufficient justification for the dynamic control strategy. Section 3.1 shows that text-side reliance poorly distinguishes hallucinated from grounded predictions. However, DEACT still uses this metric to determine suppression strength, applying stronger interventions as text-side reliance increases. While diagnostic capability does not necessarily imply control effectiveness, the paper lacks sufficient justification and empirical evidence for why higher text-side reliance should warrant stronger suppression.
Questions for authors

no other questions

Reviewer_k8kuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why and how attention head interventions mitigate LVLM object hallucinations. Challenging the assumption that these heads act as online detectors, the authors show their diagnostic separability is poor. Instead, they position these heads as asymmetric causal actuators. The proposed DEACT framework uses an offline calibration set (500 images) to select actuator heads via a rank-fused metric of attention quality and contrastive bias. During inference, it applies a position-dependent threshold and an exponential gating function to directly attenuate text-side attention weights without re-normalization. Evaluations on LLaVA and Qwen2.5-VL models report lower CHAIR scores while maintaining general generation metrics.

Critical strengths
  1. Insightful Conceptual Decoupling: The distinction between diagnostic separability and causal controllability is well-motivated and addresses a common conceptual confusion in recent interpretability literature.

  2. Thorough Metric Evaluation: The evaluation goes beyond CHAIR by including object-level precision, recall, and F1, verifying that the intervention does not merely suppress object generation blindly.

  3. Transparent Implementation Details: The authors provide detailed ablation statistics regarding head selection criteria and unnormalized attenuation behavior, ensuring reproducibility.

Critical weaknesses
  1. Opaque and Heuristic Hyperparameter Selection Rules: The identification of actuator heads requires tuning two architecture-specific hyperparameters: the layer range Lact\mathcal{L}_{\text{act}} and the pool size KK. As shown in Appendix A (Table 6), these values vary wildly across models (e.g., K=200K=200 for LLaVA-1.5-13B versus K=33K=33 for Qwen2.5-VL-7B) without a principled, automated selection criterion.
  2. Absence of Computational Overhead and Inference Latency Analysis: Although DEACT is categorized as "training-free," the online phase requires computing real-time online text-reliance scores, evaluating position-dependent reference thresholds τs\tau_s, and calculating exponential gating factors for every selected head at every decoding step. The paper completely omits an empirical analysis of the added inference latency, wall-clock time overhead, and memory footprint during autoregressive generation compared to the baseline models.
  3. Lack of Investigation into Degradation on Multi-Turn Conversations and General Visual QA: The intervention permanently applies unnormalized text-side attenuation to selected middle layers during all generation steps. The authors evaluate this solely on single-turn image descriptions and brief generative benchmarks. There is no investigation into whether this aggressive modification harms multi-turn conversational consistency, complex instruction-following capabilities, or general multimodal reasoning benchmarks (such as MMBench or SEED-Bench), where intermediate layer representations play a critical role in logical synthesis.
Questions for authors

See the weaknesses. If the authors can successfully address these concerns during the rebuttal phase—particularly by providing the requested out-of-distribution (OOD) and general visual question answering evaluations, along with the necessary computational overhead and inference latency analysis—I would be very willing to raise my rating to a Weak Acceptance.

Reviewer_DtTNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the distinction between hallucination detection and intervention in large vision language models. The authors find that text reliance in selected attention heads provides weak discrimination between hallucinated and grounded objects, while suppressing their text contributions affects hallucinated predictions more strongly. Based on this observation, the authors propose DEACT, a training free method that selects attention heads using text attention strength and hallucination contrastive statistics, then dynamically adjusts their text contributions during inference. Extensive experiments across four models and two benchmarks verify the reductions in object hallucination.

Critical strengths

1.Interesting distinction between detection and intervention. The paper shows that weak diagnostic performance does not necessarily imply limited intervention effectiveness. The comparison between observational statistics and intervention responses provides useful empirical evidence for this distinction. 2.Comprehensive experimental evaluation. The method is evaluated across four models and two benchmarks. The authors also provide comparisons with existing methods, head selection controls, and parameter sensitivity analyses. These experiments support the effectiveness of the proposed approach.

Critical weaknesses

1.Lack of novelty compared with prior work. AD-HH has already identified hallucination related attention heads and demonstrated that suppressing their text contributions reduces hallucination. SPIN and CausalLens have further explored dynamic head suppression and sensitivity guided intervention. The key observations in this paper overlaps with those works.

2.Insufficient evidence for the proposed mechanism. The central distinction between detection and control is primarily supported by the low AUC of a single text reliance statistic and the differential response to intervention. However, poor discrimination by this statistic does not establish that the selected heads lack other diagnostic information. Moreover, the analysis does not explain why suppressing these pathways selectively affects hallucinated predictions.

3.Incomplete validation of dynamic regulation. The paper does not sufficiently isolate the contribution of its dynamic controller from that of head selection and static suppression. In particular, comparisons with fixed strength suppression and position dependent suppression on the same selected heads are missing. Consequently, it remains unclear whether the proposed exponential regulation is necessary for the reported improvements.

Questions for authors

1.Could you please further clarify the conceptual difference between this paper and AD-HH, SPIN, and CausalLens beyond its head selection and suppression functions in details. What can be the main novel insight in mechanism observation? 2.Could you please compare the proposed dynamic controller with fixed strength and position dependent suppression using exactly the same selected heads, and report how much improvement comes from online state dependent regulation? 3.I wonder that does the distinction between weak diagnostic performance and strong intervention selectivity persist when using more informative diagnostic features, such as combined attention head representations or hidden states?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.