acceptodds
Withdrawn from ICLR 2027

VES: Visual Grounding-Aware Steering For Large Vision Language Models Hallucination Mitigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) achieve strong performance but is susceptible to hallucinating contextually plausible objects. Current LVLM steering methods mitigate hallucination by steering models away from these hallucination directions. However, these directions can encode semantic context shared by both the hallucinated objects and true image content. Consequently, we discover that the current methods suppress the true-content tokens after steering, a failure mode we term Post-Steer Content-Suppression. This problem arises because existing methods suppress hallucinatory directions without grounding their interventions in visual input. Building on this insight, we introduce a novel framework called VES (Visual-Evidenced Steering), where we actively steer the LVLMs towards directions supported by the image. First, VES identifies semantic regions within the image to construct a steering direction for each region, identifying visually grounded directions. It then weights these region directions according to their relevance to the current generated response and steer the model towards the aggregated direction. Across multiple LVLMs and four hallucination benchmarks, VES achieves state-of-the-art performance, showing that effective hallucination mitigation requires not only suppressing unsupported content but also reinforcing image-grounded evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hallucination Mitigation, Vision Language Models, Steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t1ynOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies hallucination mitigation in large vision-language models. It identifies Post-Steer Content-Suppression: existing activation-steering methods may reduce the likelihood of genuine objects alongside hallucinated ones. The proposed training-free method, VES, constructs steering directions from image regions, weights them according to the evolving generation context, and combines them with a precomputed visual steering direction. The authors evaluate the method through token-level logit analysis, linear probes, ablations, and experiments on three models across CHAIR, POPE, MME, and AMBER. VES generally reduces hallucination while maintaining competitive object coverage.

Critical strengths
  1. Clear motivation supported by substantial preliminary analysis. The paper goes beyond presenting a steering heuristic by first investigating a limitation of existing approaches: suppressing hallucinated content may also suppress signals associated with genuine visual content.
  2. Broad empirical validation against strong baselines, with a favorable balance between hallucination reduction and content preservation. The experimental study covers three VLM families and four benchmarks overall, including CHAIR, POPE, MME, and AMBER, and compares VES with a substantial set of competitive methods, such as VCD, VISTA, DeCo, VTI, and DMAS. The generally strong results across different evaluation settings provide convincing evidence of the method’s practical effectiveness.
Critical weaknesses
  1. PSD is calculated from raw logit changes averaged across genuine and hallucinated token positions. However, a decrease in a token’s raw logit does not necessarily imply a decrease in its generation probability. A position-dependent additive shift applied to all vocabulary logits leaves the output distribution unchanged while changing the suppression score. Consequently, PSD alone does not establish that steering selectively suppresses genuine content or increases its omission rate.
  2. The ablation studies only conducted on a 200 images subset of CHAIR.
  3. LLaVA 1.5 and Qwen2-VL are somewhat outdated. Evaluating the method with more recent models would make the results more convincing.
  4. There is a typo on line 795 of the paper: “Table ??” appears to be an unresolved table reference.
Questions for authors
  1. See Weakness 1.
  2. Could the authors conduct more extensive ablation studies to better assess the contribution of each component?
  3. Could the authors evaluate VES on more recent models, such as LLaVA 1.6 and Qwen3-VL/Qwen3.5, to further validate its effectiveness?
  4. Is the Qwen2-VL MME total 721.0 or 717.7? There is a discrepancy in Line 732 and Appendix B1.
Reviewer_6tmROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies Post-Steer Content-Suppression, where activation steering suppresses genuine object tokens alongside hallucinated ones. To address this issue, the authors propose VES, a training-free framework that constructs image-specific region directions through unsupervised segmentation. During decoding, these directions are dynamically weighted by their similarity to the evolving textual context and combined with a global steering direction. Experiments across three LVLMs and four benchmarks show reduced hallucination while largely preserving object coverage.

Critical strengths
  1. The paper addresses a meaningful trade-off in hallucination mitigation: suppressing unsupported objects can also remove genuine content. Reinforcing image-specific evidence provides a sensible response to this problem, and the region bank with contextual weighting forms a coherent, training-free implementation.

  2. The experiments demonstrate a favorable balance between hallucination reduction and object coverage. On LLaVA-1.5, VES reduces CHAIR sentence-level hallucination from 52.0 to 38.8 while retaining similar recall (77.1 versus 76.7). On Molmo 2, it improves both hallucination metrics and recall. Results on POPE, MME, and AMBER further support its effectiveness across architectures and evaluation settings.

Critical weaknesses
  1. The PSD analysis does not fully establish the proposed content-suppression mechanism. A decrease in raw logits does not necessarily reduce a token’s probability or rank, and PSD averages changes across different token positions. Therefore, the reported separation alone cannot establish that genuine content becomes less likely to be generated. Probability changes, relative logit margins, and their relationship to actual object omissions would provide stronger evidence.

  2. The independent contribution of semantic region selection remains incompletely isolated. The ablation shows that adding global visual features already provides a substantial improvement, while region weighting and EMA offer smaller additional gains on a 200-image subset. Comparisons with uniform region weights, random partitions, and region-only steering are missing. Moreover, the probing study uses ground-truth bounding boxes, so it does not directly validate the grounding quality of the unsupervised regions used by VES.

Questions for authors
  1. Does the content-suppression finding remain when measured using token probabilities or relative logit margins under identical prefixes? Can the authors connect these changes to genuine objects omitted during free generation?

  2. How does VES compare with uniform region weighting, random spatial partitions, and region steering without the inherited VTI direction? Please report both hallucination and recall, with uncertainty estimates, to clarify the contribution of semantic region selection.

  3. Please clarify the exact features and layers used for segmentation and region matching across the three architectures, the fallback when all region similarities are nonpositive, and whether calibration samples and hyperparameter-selection data are separate from evaluation.

Reviewer_LMdDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies post-steer content suppression, where activation steering can suppress genuine image content alongside hallucinated content. It proposes VES, a training-free method that dynamically combines image-region representations with a global steering direction to guide generation toward visual evidence.

Critical strengths
  1. Within the evaluated hallucination-mitigation setting, the experiments provide reasonably thorough evidence of the method’s effectiveness. Results show reduced hallucination while maintaining object coverage, supported by component ablations.
Critical weaknesses
  1. I am not fully convinced that VES addresses the specific failure mode motivating the paper (Post-Steer Content-Supression). Although VES achieves higher PSD, Figure 1(b) still shows positive average suppression scores for genuine tokens. Higher PSD establishes stronger suppression of hallucinated tokens relative to genuine ones, but does not demonstrate that genuine content is preserved or reinforced. It leaves a gap between the proposed explanation and the evidence.

  2. The reliability and added value of the region-specific steering mechanism remain insufficiently established. Figure 7 shows that most gains come from global visual features, while region pooling and EMA provide smaller improvements. Moreover, Figure 3 uses ground-truth object regions rather than CLIPTrase predictions, so it does not directly validate the regions actually used by VES.

Questions for authors

Does VES also benefit general visual tasks beyond hallucination mitigation, such as visual question answering or visual reasoning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.