acceptodds
Under review as a conference paper at ICLR 2027

Looking Is Not Grounding: Rethinking Attention-Based Hallucination Detection and Mitigation in LVLMs

Abstract

Large vision-language models frequently hallucinate objects that are not supported by the visual input, limiting their reliability in visually grounded generation. Existing attention-based approaches often treat visual attention distributions as evidence of object grounding, yet our analyses show that attention is systematically influenced by decoding context and semantic associations, while direct attention manipulation can induce response bias rather than reliably improve object verification. Motivated by this mismatch, we turn to alternative model-internal signals and introduce RASE, which combines Relational Attention structures with native Semantic Evidence to reliably quantify visual grounding. During generation, high-risk object mentions are corrected through rollback and reselection from the model's original prediction distribution. Experiments across four LVLM backbones and multiple benchmarks show that RASE consistently improves object-level hallucination detection and mitigation, and generalizes across model architectures and datasets. Our code is available at https://anonymous.4open.science/r/HalluAttn-1.

Primary area:
interpretability and explainable AI
Keywords:
VLM, Object Hallucination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MmJ7OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work analyzes the limitations in existing attention-based dehallucination methods, highlighting potential confounding effects from generation position, semantic associations, and attention-induced response preferences. It proposes RASE, which employs MLPs to extract and integrate cross-head attention relations with native semantic evidence to detect object hallucinations and guide rollback and reselection.

Critical strengths
  1. I think the analysis of potentially misleading attention signals relatively novel and informative. Examining generation-position bias, object association priors, and directional response changes provides useful perspectives for future works on attention-based hallucination detection and mitigation.
  2. RASE demonstrates a substantial advantage over the evaluated baselines in object hallucination detection.
Critical weaknesses
  1. Several essential implementation details are insufficiently specified: how to identify object words during generation; how multi-word object and multi-token word are handled; labeling of training data; actual thresholds; how the required attention distributions are obtained when using FlashAttention, whether a different attention backend is required, or the resulting computational overhead. These omissions make the method difficult to reproduce and its practical applicability difficult to assess.
  2. I thought the analyses in Section 3 have limited support for generalization. The Object Association Prior analysis primarily relies on qualitative examples, without statistical analysis. For Directional Effects of Attention Intervention, the sample size and aggregation procedure underlying Figure 3b are insufficiently described, while the transition analysis is only reported for ClearSight on the POPE adversarial split.
Questions for authors
  1. The paper suggests that object hallucinations exhibit identifiable internal feature patterns. Can the trained MLP detectors generalize across backbones without retraining? Since the current experiments train a separate detector for each backbone, they do not directly establish detector transferability.
  2. Can rollback and reselection reduce text fluency or introduce hallucinations involving attributes, relations, or other non-object content? The reported object-level hallucination and coverage metrics do not directly assess these effects.

The authors’ responses to these questions and clarification of the missing implementation details could change my assessment.

4 comments
AuthorsOpenReview ↗

Response to Reviewer MmJ7 1/4

W1 Implementation details

We thank the reviewer for pointing out the insufficiently implementation details and providing an opportunity for further clarification. We have incorporated these clarifications into the revised Appendix A.

Multi-Token Object Handling. For multi-token objects, detection is triggered once the complete object mention is generated and identified. We extract diagnostic features from the last subtoken, whose attention patterns and hidden states incorporate the context of the entire object mention through causal attention. If the mention is classified as hallucinated, RASE rolls back to the beginning of the object span and performs reselection.

FlashAttention Compatibility. Replacing FlashAttention with eager attention throughout decoding would introduce considerable latency, but this is unnecessary for RASE. A straightforward solution is to selectively use eager attention for object-token detection while retaining FlashAttention for all other tokens. A more efficient alternative is to keep FlashAttention enabled throughout decoding and independently compute the required attention distributions using the current object-token queries and visual keys stored in the KV cache. Since object tokens constitute only a small fraction of generated tokens, this selective computation limits the additional overhead.

Object Mention Identification and Labeling. We follow the standard CHAIR-based protocol adopted in prior work to identify object mentions using the COCO object vocabulary, including synonym mappings and compound-word matching. During training, we generate captions for 20,000 COCO train2017 images using each backbone and label the identified object mentions as grounded or hallucinated based on the corresponding COCO ground-truth annotations. During inference, we use the same strategy to identify completed object mentions in generated text and trigger RASE detection, without requiring any ground-truth annotations.

Threshold Calibration. For each backbone, we select the detection threshold using five-fold image-level out-of-fold predictions on the training set, maximizing hallucination F1 subject. The resulting thresholds are 0.530, 0.517, 0.571, and 0.589 for LLaVA-1.5, LLaVA-1.6, Qwen2.5-VL, and Qwen3-VL, respectively.

W2-A Quantitative Validation of Object Association Prior

We thank the reviewer for pointing out the limited quantitative evidence supporting our analysis of object association priors. To address this concern, we provide additional statistical validation across multiple LVLM backbones.

To complement the qualitative examples in Figure 3(a), we conduct a quantitative analysis across three LVLM backbones. For each image, we compare the attention allocated to the same visual region under semantically related and unrelated absent-object queries. We consider two types of semantic associations, contextual co-occurrence (car → traffic light) and category similarity (orange → apple), using teddy bear as the unrelated control. Across all three backbones, semantically related queries consistently attract more attention to the same visual regions than unrelated queries. These results provide quantitative evidence that visual attention is influenced by semantic associations even when the target object is absent, reinforcing our finding that attention allocation alone does not establish object-specific grounding.

BackboneAssociationNRelated Attn. (%)Unrelated Attn. (%)Δ Attn. (pp) [95% CI]
LLaVA-1.5Co-occurrence407.865.95+1.91 [1.29, 2.54]
Similarity4020.7315.22+5.51 [4.63, 6.39]
LLaVA-1.6Co-occurrence408.966.87+2.08 [1.18, 3.01]
Similarity4023.6617.52+6.14 [5.05, 7.21]
Qwen3-VLCo-occurrence349.337.29+2.04 [0.41, 4.44]
Similarity4016.708.83+7.87 [6.17, 9.76]
AuthorsOpenReview ↗

Response to Reviewer MmJ7 2/4

W2-B Details and Generalization of Attention Intervention

We thank the reviewer for pointing out the insufficient experimental details and limited generalization evidence. We provide further clarification and additional evaluations below.

Experimental Clarification. Figure 3(b) was intended as a qualitative illustration of the directional effects of attention scaling, rather than an aggregated statistical evaluation. To strengthen the empirical support, we extend this analysis across multiple LVLM backbones and generation settings.

Cross-Backbone Validation. We extend the analysis to three LVLM backbones, covering different versions of LLaVA and the Qwen3-VL architecture. The results consistently show that HSVR scaling increases affirmative response probabilities even when the queried object is absent, while LSVR scaling produces the opposite effect.

BackboneInterventionProbability1.01.251.51.752.02.25
LLaVA-1.5HSVRP(Yes)0.7140.7220.7250.7310.7360.747
P(No)0.0270.0270.0270.0280.0280.028
LSVRP(Yes)0.7140.6880.6660.6470.6330.605
P(No)0.0270.0290.0300.0320.0330.035
LLaVA-1.6HSVRP(Yes)0.1700.1710.1730.1730.1740.177
P(No)0.6300.6070.5940.5810.5710.564
LSVRP(Yes)0.1700.1690.1670.1660.1660.165
P(No)0.6300.6370.6390.6450.6470.624
Qwen3-VLHSVRP(Yes)0.0860.1210.1840.1890.1950.231
P(No)0.9130.7220.6900.6890.6320.590
LSVRP(Yes)0.0860.0350.0330.0310.0310.007
P(No)0.9130.9580.9670.9670.9680.982

Object-Token Probability Analysis. We further examine the effects of attention scaling in generation contexts. The results show that HSVR enhancement increases the probabilities of both visually present objects and semantically related but absent objects, while unrelated object probabilities remain largely unchanged. This suggests that attention enhancement can amplify semantic associations alongside genuine visual evidence.

BackboneObject Token1.001.251.501.752.002.25relative change
LLaVA-1.5Present (motorcycle)0.03810.03930.04000.04050.04070.0415+9.0%
Related absent (traffic light)0.002840.002880.002920.002940.003020.00309+8.8%
Unrelated absent (teddy bear)0.000140.000140.000140.000140.000140.00014−0.6%
LLaVA-1.6Present (motorcycle)0.20830.21110.21250.21560.21660.2179+4.6%
Related absent (traffic light)0.009430.009630.009850.009990.010120.01017+7.8%
Unrelated absent (teddy bear)0.000090.000090.000090.000090.000090.00009−4.1%

Quantitative Transition Analysis. We selected ClearSight for the original transition analysis because its pre-softmax attention scaling sharpens visual attention distributions, emphasizing highly attended visual tokens while suppressing weakly attended ones. To further validate the directional effects illustrated in Figure 3(b), extend the transition analysis to the remaining two POPE splits (Random and Popular) on LLaVA-1.5 and Qwen3-VL.

The results show that ClearSight consistently increases both TPR and FPR across all settings, indicating a shift toward affirmative predictions. Notably, the increase in FPR exceeds that in TPR on the adversarial splits, resulting in decreased accuracy. These findings further demonstrate that attention enhancement can induce directional response bias without consistently improving object-specific verification.

BackboneSplitΔ Yes RateΔ TPRΔ FPRΔ Acc
LLaVA-1.5Random+2.97+3.54+2.40+0.57
Popular+3.44+3.54+3.34+0.10
Qwen3-VLRandom+2.40+3.60+1.20+1.20
Popular+3.10+3.60+2.60+0.50
AuthorsOpenReview ↗

Response to Reviewer MmJ7 3/4

Q1 Cross-Backbone Generalization

We would like to clarify that RASE demonstrates framework-level generalization across different LVLM architectures, rather than the transferability of a single trained detector.

Different LVLM backbones vary in their architectures, including the number of decoder layers and attention heads, resulting in different feature dimensions and internal representations. Moreover, differences in training data and training procedures can lead to distinct internal representations, even when the model architectures are similar. Therefore, RASE trains a separate lightweight detector for each backbone. Nevertheless, our experiments demonstrate that the proposed feature extraction and detection framework consistently achieves strong performance across diverse LVLM architectures.

Q2 Fluency and Non-Object Hallucination Analysis

We thank the reviewer for raising this important concern regarding the potential effects of rollback on text fluency and non-object hallucinations. We provide further clarification and additional evaluation below.

Clarification of Rollback Mechanism. RASE performs hallucination detection only on object mentions, leaving non-object tokens unaffected. Moreover, reselection operates on the backbone's original prediction distribution, preserving its native language modeling behavior without modifying internal representations.

Additional Evaluation. To assess whether rollback affects text fluency or introduces non-object hallucinations, we conduct additional evaluations across three LVLM backbones. Specifically, we use GRUEN [2] to measure linguistic quality and FaithScore [1] to evaluate fine-grained visual faithfulness, including attribute and relation consistency. We compare RASE with vanilla decoding on the same COCO images under identical generation settings. The results show that RASE maintains largely comparable fluency and fine-grained visual faithfulness across all three backbones.

ModelMetricVanillaRASE
LLaVA-1.5GRUEN ↑0.81090.8142
FaithScore (Attribute) ↑0.90650.8970
FaithScore (Relation) ↑0.86550.8817
FaithScore (Overall) ↑0.88100.8882
LLaVA-1.6GRUEN ↑0.76200.7656
FaithScore (Attribute) ↑0.90360.9054
FaithScore (Relation) ↑0.88810.8889
FaithScore (Overall) ↑0.90030.8981
Qwen3-VLGRUEN ↑0.64140.6384
FaithScore (Attribute) ↑0.87810.8699
FaithScore (Relation) ↑0.88880.8820
FaithScore (Overall) ↑0.88450.8874

[1] FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models, Findings of EMNLP 2024

[2] GRUEN for Evaluating Linguistic Quality of Generated Text, Findings of EMNLP 2020

AuthorsOpenReview ↗

Response to Reviewer MmJ7 4/4

We sincerely thank the reviewer for the constructive feedback and the opportunity to clarify our work. We hope our responses have addressed the concerns and would be happy to provide further clarification if any questions remain.

Reviewer_ujRMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the use of visual attention as a grounding signal for detecting and mitigating object hallucinations in LVLMs, showing that it is confounded by generation position and semantic associations, while attention manipulation can introduce response bias. To address this issue, the authors propose RASE, which combines relational patterns across attention heads with semantic evidence from hidden states to estimate object-level hallucination risk. RASE mitigates hallucinations through risk-guided rollback and token reselection, achieving consistent improvements across multiple LVLMs and image/video benchmarks.

Critical strengths
  • RASE provides a technically coherent framework that combines attention head-level structural and semantic signals for object-level hallucination detection, and its rollback-and-reselection mechanism avoids directly modifying the backbone’s attention distributions.

  • The method achieves strong and consistent results across four mainstream LVLMs and multiple image and video benchmarks, demonstrating robust hallucination reduction while largely preserving correctly grounded object content.

Critical weaknesses
  • Limited novelty and insufficient evidence for the “Looking is not Grounding” claim.

    • Although this observation is presented as a central contribution, several of its key components have been reported in prior work. The generation-position bias closely resembles the finding of M3ID [R1] that the influence of image conditioning decreases as more tokens are generated, while Attention Lens [R2] has shown that hallucinated tokens can still interact with semantically related visual objects.

    • Moreover, the evidence for the directional effects of attention intervention (Figure 3b) is limited to a single model under a binary yes/no QA setting, making it unclear whether this phenomenon generalizes across architectures and open-ended generation scenarios.

  • Insufficient validation and efficiency analysis of rollback and reselection, raising concerns about its practical utility.

    • The proposed risk-guided rollback mechanism is motivated by the "response bias" observed in the QA analysis, whereas the Experiments section focus on captioning tasks and do not directly demonstrate that rollback and reselection address the identified bias.

    • Rollback may require repeated forward passes whenever a candidate token is rejected, potentially introducing substantial inference overhead, especially when multiple reselection iterations occur. The paper would benefit from reporting the rollback-trigger frequency, the number of reselection iterations, and end-to-end latency or throughput relative to standard decoding and competing mitigation methods.

[R1] Multi-modal hallucination control by visual information grounding. Favero A, Zancato L, Trager M, et al. CVPR 2024.

[R2] Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. Jiang Z, Chen J, Zhu B, et al. CVPR 2025.

Questions for authors

Please refer to the Critical Weaknesses section.

6 comments
AuthorsOpenReview ↗

Response to Reviewer ujRM 1/3

W1-A Distinction from Prior Work

We thank the reviewer for raising this important concern and providing an opportunity to clarify our contributions in relation to prior work. We agree that our work builds on important insights from prior studies, including M3ID [1], Attention Lens [2], and other related works. However, our work provides a unified perspective on these findings, systematically examining the fundamental limitations of attention-based hallucination detection and mitigation.

Relation to Prior Work. M3ID [1] observes that visual conditioning weakens as generation progresses, while Attention Lens [2] reveals that hallucinated tokens may attend to semantically related visual objects. These studies provide valuable insights into specific aspects of attention behavior. However, these observations have largely been studied in isolation, and their broader implications for the reliability of attention-based hallucination detection and mitigation remain insufficiently explored. Our work addresses this gap by systematically examining how these limitations affect existing approaches. In particular, our position-controlled evaluation reveals that five existing attention-based detectors experience substantial AUROC degradation after controlling for generation position. We further examine how attention interventions can shift response preferences without consistently improving object verification.

Unified Perspective. By revisiting attention's fundamental role in context-dependent information routing, we reveal why visual attention distributions do not necessarily provide reliable evidence of object-specific grounding. We bring together insights from prior work and our new empirical evidence to provide a unified understanding of this limitation. This perspective further motivates us to move beyond conventional reliance on visual attention distributions and explore alternative grounding signals, leading to RASE, which combines cross-head relational structure with native semantic evidence.

We believe this perspective can encourage broader research beyond attention distribution-based approaches to hallucination detection and mitigation.

[1] Multi-modal hallucination control by visual information grounding. Favero A, Zancato L, Trager M, et al. CVPR 2024.

[2] Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. Jiang Z, Chen J, Zhu B, et al. CVPR 2025.

AuthorsOpenReview ↗

Response to Reviewer ujRM 2/3

W1-B Generalizability of Attention-Induced Response Bias

We agree that the original analysis of attention-induced response bias is limited to a single model and binary QA setting. To address this concern, we extend our experiments across multiple LVLM backbones and open-ended image captioning.

Cross-Backbone Validation. We extend the attention intervention analysis in Figure 3(b) to three LVLM backbones, covering different versions of LLaVA and the Qwen3-VL architecture. We examine the question "Is there a traffic light in the image?" on images containing cars and trucks but no traffic lights. The results show consistent directional effects across all three backbones. Increasing attention to HSVRs raises the probability of answering "Yes," even though the queried object is absent, while scaling LSVRs consistently reduces this probability. These findings provide further evidence that attention enhancement can amplify incorrect response preferences rather than reliably improve object verification.

BackboneInterventionProbability1.01.251.51.752.02.25
LLaVA-1.5HSVRP(Yes)0.7140.7220.7250.7310.7360.747
P(No)0.0270.0270.0270.0280.0280.028
LSVRP(Yes)0.7140.6880.6660.6470.6330.605
P(No)0.0270.0290.0300.0320.0330.035
LLaVA-1.6HSVRP(Yes)0.1700.1710.1730.1730.1740.177
P(No)0.6300.6070.5940.5810.5710.564
LSVRP(Yes)0.1700.1690.1670.1660.1660.165
P(No)0.6300.6370.6390.6450.6470.624
Qwen3-VLHSVRP(Yes)0.0860.1210.1840.1890.1950.231
P(No)0.9130.7220.6900.6890.6320.590
LSVRP(Yes)0.0860.0350.0330.0310.0310.007
P(No)0.9130.9580.9670.9670.9680.982

Open-Ended Generation Analysis. We further investigate whether enhancing highly attended visual regions selectively strengthens grounded object predictions. Specifically, we compare the probability changes of visually present objects, semantically related but absent objects, and unrelated absent objects under the same attention intervention. The results show that HSVR enhancement increases the probabilities of both visually present objects and semantically related but absent objects, while unrelated objects remain largely unaffected. This suggests that attention intervention can amplify semantic associations alongside genuine visual evidence, without necessarily improving object-specific grounding.

BackboneObject Token1.001.251.501.752.002.25relative change
LLaVA-1.5Present (motorcycle)0.03810.03930.04000.04050.04070.0415+9.0%
Related absent (traffic light)0.002840.002880.002920.002940.003020.00309+8.8%
Unrelated absent (teddy bear)0.000140.000140.000140.000140.000140.00014−0.6%
LLaVA-1.6Present (motorcycle)0.20830.21110.21250.21560.21660.2179+4.6%
Related absent (traffic light)0.009430.009630.009850.009990.010120.01017+7.8%
Unrelated absent (teddy bear)0.000090.000090.000090.000090.000090.00009−4.1%
AuthorsOpenReview ↗

Response to Reviewer ujRM 3/3

W2-A Rationale and Rationality of Rollback

We would like to clarify the rationale behind our rollback mechanism and its distinction from attention-based intervention to address the reviewer's concern.

The response bias identified in Section 3 arises from directly manipulating attention distributions to influence model predictions. In contrast, RASE extracts hallucination signals from cross-head relational structures and native semantic evidence, and performs correction through risk-guided rollback and reselection from the backbone's original prediction distribution. This avoids modifying internal attention distributions or representations, thereby circumventing the specific source of response bias identified in our analysis.

W2-B Computational Efficiency of Rollback

We thank the reviewer for raising this concern. As computational overhead was also raised by other reviewers, we have added a detailed discussion and additional efficiency evaluations in the revised Appendix B.

(1) Implementation Clarification. Several potential sources of computational overhead can be avoided through careful implementation.

RASE performs hallucination detection only on identified object tokens, which account for a small fraction of generated tokens (e.g., 2.86% for Qwen3-VL-8B on COCO). For non-object tokens, the original decoding process remains unchanged, without additional detection overhead. For object tokens, RASE extracts diagnostic features from the forward pass that processes each candidate, avoiding redundant LVLM forward passes for accepted candidates.

Another potential source of computational overhead is the structural branch's need for attention distributions. Rather than replacing FlashAttention with eager attention throughout decoding, we can retain FlashAttention and independently reconstruct the required visual attention distributions using the current object-token queries and cached visual keys.

(2) Efficiency Evaluation. We report end-to-end decoding latency, peak GPU memory, and average retries per object mention compared with vanilla decoding on COCO.

BackboneVanilla (s/img)RASE (s/img)SlowdownPeak Memory (GiB) Vanilla → RASERetries/mention
LLaVA-1.5-7B4.184.351.04×13.55 → 13.750.20
LLaVA-1.6-Mistral-7B6.127.041.15×14.83 → 15.870.32
Qwen2.5-VL-7B6.337.281.15×15.64 → 15.960.23
Qwen3-VL-8B15.4218.661.21×16.54 → 17.000.33

Comparison with Existing Methods. We further compare RASE with baseline methods on LLaVA-1.5-7B. VCD incurs additional overhead from contrastive decoding, while SADT introduces extra computation for hallucination detection and mitigation. Methods involving direct attention modification, such as ClearSight and PAI, may incur additional latency due to limited compatibility with standard FlashAttention implementations. In contrast, RASE retains FlashAttention throughout decoding, thereby minimizing additional inference overhead.

MethodLatency (s/img)SlowdownPeak Mem (GiB)
Vanilla4.181.00×13.55
PTI4.221.01×13.55
ClearSight6.941.66×13.56
SADT8.782.10×13.60
VCD9.282.22×13.93
PAI31.737.59×13.99
RASE (Ours)4.351.04×13.75
Reviewer_bMeyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

RASE combines cross-head attention similarities with semantic features to detect hallucinated object mentions and uses the score to guide rollback during generation. It is evaluated on four LVLMs, image-captioning benchmarks and a separately trained video extension. The paper also studies confounds in existing attention-based hallucination scores.

Critical strengths

• The detection results are consistently strong, with AUROC above 93 on all four backbones. The captioning evaluations also report correct-object coverage, so the hallucination reductions are not assessed in isolation.

• The analysis of generation-position bias is useful beyond this particular detector. Combining relational attention structure with semantic evidence also gives a practical way to use model-internal signals for online correction.

Critical weaknesses

• The main diagnostic result is not applied to RASE itself. Figure 2 shows a substantial AUROC drop for existing detectors under position control, but the proposed method is evaluated only under the overall detection protocol. Normalizing attention over visual tokens does not establish that its head-pair relations or semantic features are independent of generation position. RASE and its two branches need to be evaluated under the same position-controlled setting. This is the most important missing check because it directly tests the paper's explanation for why its grounding signal is more reliable.

• The training recipe and the cost of mitigation are not sufficiently documented. Section 5 specifies 20,000 training images and training-only threshold calibration, but the mention-labeling procedure, MLP architecture, optimization settings and calibration objective need enough detail to reproduce the detector.

• The internal findings currently support a detector, while correction operates through rollback and reselection. A better strategy would be to use the identified head relations and semantic evidence to guide internal adjustments that prevent unsupported candidates.

Questions for authors

• Does RASE retain its advantage under exactly the position-controlled protocol in Figure 2? Please include the structural-only and semantic-only variants.

• Please provide a reproducible detector-training recipe and report end-to-end decoding time, peak memory and retries per object mention against vanilla decoding.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bMey 1/2

W1 & Q1 Position-controlled robustness

We thank the reviewer for raising this important concern. RASE was designed to mitigate generation-position bias by extracting features that are less sensitive to generation position, but we acknowledge the lack of direct experimental validation. We address this concern through the following clarification and additional experiments.

Clarification of Design Motivation. In Figure 2(b), we control for the influence of text-context length by evaluating object tokens within the same generation-position bins. Motivated by this observation, RASE aims to mitigate position-related bias by normalizing visual attention to remove variations in total attention mass and capturing cross-head relations in the structural branch, while extracting object-specific visual evidence from hidden states and vocabulary predictions in the semantic branch.

Position-Controlled Evaluation. We further evaluate RASE and its structural-only (S) and semantic-only (G) variants under the same position-controlled protocol as Figure 2(b). The results show that RASE and both individual branches maintain strong detection performance under position control, with smaller drops than existing detectors. This confirms that RASE remains an effective hallucination detector without relying primarily on generation-position bias.

BackboneMethodOverall AUROCWithin-bin AUROCDrop
LLaVA-v1.6S only94.8094.370.43
G only93.4792.820.65
S + G95.8695.410.45
Qwen3-VLS only94.2193.690.52
G only94.5493.750.79
S + G95.3094.930.37

We have included these additional experimental results and detailed comparisons in Appendix C of the revised manuscript.

W2 & Q2 - A Training Details

We thank the reviewer for pointing out the missing implementation details in the original submission, and provide additional details below.

Object Mention Labeling. Following the standard CHAIR-based object hallucination evaluation protocol adopted in prior work[1,2], we automatically extract object mentions from backbone-generated captions and label them as grounded or hallucinated. Specifically, for each COCO training image, we first generate a caption using the corresponding backbone, then identify object mentions using CHAIR's object vocabulary and synonym mappings, and label each mention as grounded or hallucinated based on the image's ground-truth object categories.

Detector Architecture and Optimization. RASE employs lightweight structural and semantic MLPs with architectures K → 128 → 32 → 1 and 3L → 32 → 1, respectively. Their risk scores are combined through a linear fusion layer followed by a sigmoid activation to produce the final hallucination risk. All MLPs use ReLU activations and are trained with binary cross-entropy loss using AdamW (learning rate 1e-3, batch size 512).

Threshold Calibration. For each backbone, we select the detection threshold using five-fold out-of-fold predictions on the training set, maximizing hallucination F1.

We have included these additional details in Appendix A of the revised manuscript.

[1] Object Hallucination in Image Captioning

[2] Devils in Middle Layers of Large Vision-Language Models

AuthorsOpenReview ↗

Response to Reviewer bMey 2/2

W2 & Q2 - B End-to-End Decoding Overhead

We thank the reviewer for raising this concern and providing an opportunity for further clarification. As computational overhead was also raised by other reviewers, we have added a detailed discussion and additional efficiency evaluations in Appendix B of the revised manuscript.

Efficient Design. In fact, several potential sources of computational overhead in RASE can be avoided through careful implementation, as clarified below.

(1) Selective Detection. RASE performs hallucination detection only on identified object tokens, which account for a small fraction of generated tokens (e.g., 2.86% for Qwen3-VL-8B on COCO). For non-object tokens, the original decoding process remains unchanged, without additional detection overhead. For object tokens, RASE extracts diagnostic features from the forward pass that processes each candidate, avoiding redundant LVLM forward passes for accepted candidates. Furthermore, the detector's high precision helps minimize unnecessary rollbacks.

(2) FlashAttention Compatibility. Another potential source of substantial computational overhead is the structural branch's need for attention distributions. Replacing FlashAttention with eager attention throughout decoding would introduce considerable latency, but this is unnecessary for RASE. A straightforward solution is to selectively use eager attention for object-token detection while retaining FlashAttention for all other tokens. A more efficient alternative is to keep FlashAttention enabled throughout decoding and independently compute the required visual attention distributions using the current object-token queries and cached keys. Since object tokens constitute only a small fraction of generated tokens, this selective computation limits the additional overhead.

Efficiency Evaluation. We evaluate both implementations against vanilla decoding under identical settings, reporting end-to-end decoding latency, peak GPU memory, and average retries per object mention. The results show that the KV-cache implementation limits decoding overhead to 1.04–1.21× with modest additional GPU memory.

BackboneVanillaSelective EagerKV-cachePeak Memory (GiB)Retries/mention
LLaVA-1.5-7B4.18s4.43s (1.06×)4.35s (1.04×)13.55 → 13.750.20
LLaVA-1.6-Mistral-7B6.12s7.41s (1.21×)7.04s (1.15×)14.83 → 15.870.32
Qwen2.5-VL-7B6.33s7.41s (1.17×)7.28s (1.15×)15.64 → 15.960.23
Qwen3-VL-8B15.42s19.89s (1.29×)18.66s (1.21×)16.54 → 17.000.33

Comparison with Existing Methods. We further compare RASE with baseline methods on LLaVA-1.5-7B. VCD incurs additional overhead from contrastive decoding, while SADT introduces extra computation for hallucination detection and mitigation. Methods involving direct attention modification, such as ClearSight and PAI, may incur additional latency due to limited compatibility with standard FlashAttention implementations. In contrast, RASE retains FlashAttention throughout decoding, thereby minimizing additional inference overhead.

MethodLatency (s/img)SlowdownPeak Mem (GiB)
Vanilla4.181.00×13.55
PTI4.221.01×13.55
ClearSight6.941.66×13.56
SADT8.782.10×13.60
VCD9.282.22×13.93
PAI31.737.59×13.99
RASE (Ours)4.351.04×13.75

W3 Rationale for Risk-Guided Rollback

We agree with the reviewer that leveraging the identified head relations and semantic evidence to guide internal adjustments could be a promising direction for preventing hallucinations.

While the identified signals are effective for hallucination detection, they serve as diagnostic indicators rather than necessarily corresponding to causally controllable directions within the model, making their direct use for internal intervention nontrivial. A natural approach would be to use these signals to guide attention redistribution. However, as shown in Section 3, direct attention manipulation can introduce directional response biases rather than reliably improve visual grounding. Moreover, such interventions require modifying attention computations, complicating compatibility with standard FlashAttention implementations and introducing much more additional computational overhead. Therefore, RASE adopts risk-guided rollback and reselection, correcting high-risk candidates based on the backbone's original prediction distribution without modifying its internal representations.

Thanks for the additional experiments and clarifications. My main concern is whether RASE is also susceptible to generation-position bias. Under the same evaluation protocol as Figure 2(b), the full model’s AUROC dropped by only 0.45 and 0.37 points on the two backbones, while the individual branches also maintained strong performance. The added training details and the latency measurements for the KV-cache implementation also address my concerns about reproducibility and efficiency.

I agree with the point raised by reviewer ujRM regarding the overlap with M3ID and Attention Lens. What I find most convincing is the empirical analysis of generation-position bias and the reduction in object hallucinations while largely preserving coverage of correctly identified objects.

AuthorsOpenReview ↗

Response to Reviewer bMey

We sincerely thank the reviewer for carefully considering our additional experiments and clarifications. As the reviewer noted, our work builds on valuable insights from several influential prior studies. We further develop these insights through a unified perspective, providing additional controlled evaluations and developing a more effective hallucination mitigation framework. We sincerely appreciate the reviewer's recognition of the novelty and value of these contributions.

We will incorporate these additional experiments and insights into the final manuscript. We sincerely thank the reviewer again for the valuable suggestions that have helped us improve our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Further Response to Reviewer MmJ7

We thank the reviewer for the continued discussion. We would like to further clarify the concerns raised and explain our perspective on the evaluation Scope and the key contributions of our work.

Evaluation Scope

We acknowledge that COCO annotations are not exhaustive and that an object absent from the annotations is not necessarily absent from the image. However, we would like to clarify that our experimental evaluation is not limited to COCO.

Following established research on object hallucination[1, 2, 3], we use COCO and the CHAIR-based protocol as our primary evaluation setting. Importantly, however, our evaluation is not limited to COCO. As reported in Figure 5, we additionally evaluate RASE on two independent open-ended generation benchmarks:

  • AMBER-G[4], which provides independently curated annotations covering 337 object categories across 1,004 images.
  • NoCaps[5], which is built upon Open Images V4, providing object annotations for 600 categories, including nearly 400 categories that are novel or rarely mentioned in COCO captions.

For both benchmarks, we use their corresponding annotation vocabularies for object-token identification and hallucination evaluation, rather than restricting the evaluation to COCO's 80 object categories.

Crucially, our detectors are trained using COCO object categories and are directly applied to AMBER-G and NoCaps without retraining or benchmark-specific calibration. Despite the differences in object vocabularies, RASE consistently reduces object hallucinations while largely preserving correct-object coverage. These results provide evidence that the detector captures hallucination-related signals that generalize beyond its COCO training vocabulary.

[1] Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination, CVPR 2026.

[2] Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models, CVPR 2026.

[3] Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation, ICML 2026.

[4] An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation, arXiv 2023

[5] nocaps: novel object captioning at scale, ICCV 2019

Key Contributions

We acknowledge that RASE requires backbone-specific detector training, and we do not claim zero-shot transferability across different LVLMs. Nevertheless, we would like to emphasize that our contributions extend beyond the detector itself.

(1) Our Key Contributions. First, we systematically analyze the limitations of visual attention as an indicator of object grounding, revealing the effects of generation position and semantic associations on hallucination detection and mitigation. Second, building on these findings, we propose RASE, which combines cross-head relational and semantic features for detection with risk-guided rollback and reselection for mitigation, without directly manipulating attention. Third, we demonstrate consistent reductions in object hallucinations across three LVLM backbones and multiple benchmarks while largely preserving correct-object coverage.

(2) Advantages of Our Lightweight Detector. As shown in Figure 5, our detectors trained on COCO object categories generalize to AMBER-G and NoCaps without retraining, reducing dataset-specific training costs. Compared with training-based alternatives, RASE achieves stronger hallucination mitigation with lower training requirements and better generalization. Compared with training-free approaches, it delivers better detection performance while preserving object coverage, with lower inference latency than several training-free baselines, as shown in our computational efficiency analysis with other reviewers.

We sincerely appreciate the reviewer's continued engagement and valuable feedback throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.