acceptodds
Withdrawn from ICLR 2027

SensiEdit: Sensitivity-Guided Spatial Attention Editing for Hallucination Mitigation in LVLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal reasoning and complex scene understanding. However, they still suffer from hallucination, generating descriptions that contradict visual facts. Attention heads differ in their sensitivity to image content, motivating selective modulation of visual attention. To exploit this variation, we propose SensiEdit, a lightweight framework for sensitivity-guided spatial attention editing. Specifically, we introduce Attention-based Visual Sensitivity (AVS) to characterize each head’s response to changes in visual input and train a lightweight predictor to estimate this signal. The predicted sensitivity guides stronger visual attention modulation for less sensitive heads, while learnable spatial biases determine the editing strength at different visual positions. The LVLM backbone remains frozen throughout training, and the spatial biases are learned from only 3K images. Experiments on LLaVA-1.5-7B and Qwen2.5-VL-7B, together with supplementary results on three additional backbones, demonstrate the effectiveness of SensiEdit in mitigating object hallucination. On LLaVA-1.5-7B, SensiEdit achieves relative reductions of 56% and 68% in CHAIR_S and CHAIR_I, respectively, compared with the unedited baseline, while maintaining performance on the evaluated general-capability benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large vision-language models, hallucination, attention editing, visual sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fiPXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SensiEdit predicts head-wise sensitivity to image replacement and uses it to modulate learned spatial biases on visual attention logits. A gate controls when editing occurs, while the LVLM stays frozen. The paper reports object-hallucination reductions across several backbones and examines head weighting, gating, runtime and caption quality.

Critical strengths

The LLaVA result is substantial: CHAIRs/CHAIRi falls from 51.30/16.80 to 22.60/5.32. The full model also retains more recall than the reported equal-head control, 74.07 versus 63.71. The POPE and AMBER gains provide evidence beyond the main captioning experiment.

Critical weaknesses

• The method’s motivation is unclear. Head-wise heterogeneity does not establish the proposed intervention rule. Figure 1 shows that heads respond differently to image replacement, but does not fully explain why low-sensitivity heads require stronger visual attention intervention.

• The relationship between AVS and hallucination is insufficiently established. AVS measures a change in attention weights under image replacement, not whether an object mention is supported by the image.The authors acknowledge that AVS is not a validated causal measure of visual reliance. What is missing is direct evidence that AVS, and its predicted counterpart, track hallucination risk and predict which heads or decoding steps benefit from intervention.

Questions for authors

• On held-out examples, does low AVS predict hallucination and greater benefit from stronger intervention? Can the allocation rule be compared under matched training and bias budgets?

• When trained on the COCO dataset, can it be generalized to other datasets with different distributions, or is retraining required on the dataset with the new distribution?

Reviewer_uHSSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SensiEdit, a sensitivity-guided spatial attention editing framework for mitigating hallucinations in Large Vision-Language Models (LVLMs). The key observation is that different attention heads exhibit varying degrees of sensitivity to visual inputs. To exploit this property, the authors introduce Attention-based Visual Sensitivity (AVS) to measure head-wise responses to image replacement and train a lightweight predictor to estimate these sensitivity scores. Based on the predicted sensitivity, a learnable Spatial Attention Bias Editor selectively adjusts visual attention logits, applying stronger modulation to less sensitive heads.

Critical strengths
  1. The proposed framework is computationally lightweight and practically applicable.

  2. The experimental results demonstrate substantial improvements in hallucination mitigation. The authors evaluate SensiEdit on multiple benchmarks and LVLM architectures, showing notable improvements in CHAIR, POPE, and AMBER. The paper also includes comparisons with recent methods and several ablation studies to investigate the contributions of different components.

Critical weaknesses
  1. The underlying mechanism of visual sensitivity is not sufficiently justified. The paper does not adequately explain why smaller attention changes indicate lower visual sensitivity, as larger changes may still reflect attention to irrelevant or incorrect regions. Analyzing whether different heads consistently focus on semantically important objects could provide stronger evidence for the proposed metric.

  2. The proposed framework lacks sufficient theoretical justification. The paper lacks a principled explanation of why sensitivity-guided attention editing effectively suppresses hallucinations. In particular, it remains unclear why a predictor trained on limited data can generalize across different data distributions and whether the learned sensitivity patterns transfer across MLLM architectures.

  3. The methodological novelty and incremental contribution are relatively limited. The proposed framework largely builds upon existing attention-logit manipulation techniques, with additional sensitivity prediction and learnable spatial biases. Notably, Table 7 shows that a simple fixed-uniform attention bias already reduces CHAIR-S from 51.3 to 28.2, compared with 22.6 achieved by the full method. This suggests that much of the improvement comes from general visual attention enhancement, while the distinctive contribution of sensitivity-guided editing is less substantial.

  4. The improvements in hallucination mitigation come with noticeable trade-offs in generation quality. Although SensiEdit substantially reduces hallucination rates, Table 7 shows that object Recall decreases from 76.97 to 74.07 on LLaVA-1.5-7B. Furthermore, Appendix C.2 reports repeated phrases in 5.4% of generated captions, compared with none in the baseline. These results raise concerns about whether the method consistently improves generation quality beyond reducing hallucination-specific metrics.

Questions for authors
  1. Could the authors provide additional evidence regarding the statistical reliability and generalization of the reported improvements? The main CHAIR evaluation uses only 500 COCO images, while the auxiliary components are also trained on COCO data. Could the authors report results across multiple independent training runs and random seeds, together with evaluations on more diverse image distributions, to establish the robustness of the reported improvements?

  2. Could the authors further clarify the practical training and inference costs of SensiEdit compared with simpler alternatives? Although the backbone remains frozen, the framework requires training both a sensitivity predictor and spatial bias parameters for each backbone, while increasing per-token decoding latency by 28.9%. Could the authors provide an end-to-end cost comparison, including auxiliary training overhead, against the fixed-uniform attention bias baseline to better justify the additional complexity?

Reviewer_2gGwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sensitivity-Guided Spatial Attention Editing (SensiEdit) to mitigate object hallucination in Large Vision-Language Models (LVLMs). The authors focus on the phenomenon that attention is concentrated on non-visual attention sinks rather than visual tokens in some decoder layers, and on the fact that attention heads differ in their sensitivity to changes in images. Based on these observations, they define Attention-based Visual Sensitivity (AVS), which measures changes in attention caused by image replacement, and predict head-wise sensitivity and an editing gate using a lightweight MLP. They then use learnable spatial biases to apply stronger attention biases to heads with low visual sensitivity, thereby encouraging the use of visual information. The proposed method does not modify the LVLM backbone and reports results showing reduced hallucination while largely maintaining general visual understanding performance across multiple LVLMs and hallucination benchmarks.

Critical strengths
  1. This paper proposes a method for selectively adjusting attention based on observations of imbalanced visual attention and differences in visual sensitivity across attention heads. In particular, directly applying biases to pre-softmax attention logits is relatively intuitive and has the advantage of being applicable without modifying the parameters of the LVLM backbone. Since the method uses a trained AVS predictor, it does not require an additional image-replacement forward pass during inference, allowing it to reduce additional backbone computation compared to contrastive decoding methods.

  2. The paper evaluates several LVLMs, including LLaVA-1.5-7B and Qwen2.5-VL-7B, on various hallucination benchmarks such as CHAIR, POPE, and AMBER, and reports generally superior performance compared to existing methods. It also examines whether general visual understanding capabilities are maintained through MME, SEED-Bench, and MMMU.

Critical weaknesses
  1. The main contribution of this paper is not merely strengthening visual attention, but using AVS to identify less visually sensitive heads and selectively modulate their attention biases. However, Appendix C.1 provides limited evidence that sensitivity guidance offers consistent additional benefits across models and metrics. On Qwen2.5-VL-7B, shuffle-head performs better than the full method on CHAIR_S, while equal-head performs better on CHAIR_I. Both controls also substantially reduce hallucination compared to the original model. Although LLaVA shows benefits from sensitivity guidance, these ablations modify the sensitivity signal within an already-trained spatial editor while matching the overall bias strength. Without separately optimizing sensitivity-free alternatives under the same conditions, it remains unclear how much of the improvement comes specifically from AVS-based modulation rather than general visual attention boosting and learned spatial biases. Moreover, no quantitative evaluation of the AVS predictor's prediction accuracy is reported.

  2. The method applies attention biases to eight selected decoder layers, but the paper does not explain how these layers were selected or why eight layers are used. It also omits the exact edited layer indices for LLaVA-1.5-7B and Qwen2.5-VL-7B, leaving unclear whether a consistent layer-selection principle was applied across the two models. Although some Qwen-specific attention and decoding implementation details are provided, the missing layer configurations make accurate reproduction difficult. Furthermore, Figure 3 shows substantial performance differences across edited layer groups. However, the ablation changes both the number and positions of edited layers simultaneously, making it difficult to isolate the effect of layer placement or justify the eight-layer configuration. Consequently, the lack of justification and specification for this performance-sensitive design choice weakens the reproducibility of the proposed method and makes it difficult to verify consistency across different backbones.

Questions for authors

According to the paper, attention editing is applied only during the autoregressive decoding stage, not during the prefill stage. However, in binary question-answering tasks such as POPE, the first generated token generally determines the final answer, and this token is predicted from the last hidden state of the prefill stage. In this case, could you explain specifically how the proposed method affects the prediction of the first answer token and through what process the reported performance improvements arise? In addition, in Stage 1, the AVS Predictor is trained using hidden states from the unmodified frozen LVLM, but during inference, attention editing applied to preceding decoder layers may change the hidden-state distribution of subsequent layers. I would like to know whether you have analyzed how these changes affect the accuracy of the predicted sensitivity and gate, particularly whether the reliability of the predictions is maintained after sequentially passing through multiple edited layers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.