acceptodds
Withdrawn from ICLR 2027

Beyond Visual Attention: Competition-Informed Visual Evidence Refinement for Hallucination Mitigation in LVLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hallucination remains a critical challenge for Large Vision-Language Models (LVLMs), where generated responses may be fluent yet inconsistent with the actual visual content. Existing studies commonly attribute hallucination to insufficient visual contribution and mitigate it by strengthening visual signals during decoding. However, hallucination may still occur even when the model attends strongly to the image. We find that hallucinated predictions exhibit greater overlap between the visual-evidence distributions associated with competing token candidates, suggesting that the attended visual information may not sufficiently distinguish these candidates. We term this phenomenon candidate-level visual-evidence ambiguity and find that it is associated with a higher rate of hallucination. Based on these findings, we propose Competition-Informed Visual Evidence Refinement (CIVER), a training-free decoding framework that disambiguates competing token predictions using four complementary components: a sample-specific visual prior, supporting evidence for the provisional leading candidate, competing evidence favoring its strongest alternative, and cross-step evidence memory. CIVER combines these components into a signed evidence bias and selectively applies it to subsequent visual attention according to the ambiguity of the candidate pair. Extensive experiments on POPE, CHAIR, MME, and AMBER using InstructBLIP, Shikra, LLaVA-1.5, and Qwen-VL demonstrate that CIVER consistently reduces hallucinations and improves visual faithfulness across diverse LVLM architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Hallucination Mitigation, Visual-Evidence Ambiguity, Training-Free Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_m6FwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CIVER, a training-free method for mitigating hallucinations in large vision-language models. CIVER integrates a sample-specific visual prior, candidate-conditioned supporting and competing evidence, and cross-step evidence memory to selectively adjust visual attention during decoding. Experiments across four LVLMs on POPE, CHAIR, MME, and AMBER demonstrate improvements in hallucination mitigation over existing baselines.

Critical strengths

The paper presents a valuable observation that hallucinations are more likely to occur when the visual evidence distributions associated with competing token candidates are difficult to distinguish. This finding offers a new perspective on understanding and detecting hallucinations in LVLMs.

Critical weaknesses
  1. The connection between the research motivation and the core intervention mechanism is insufficiently justified. The paper argues that hallucinations are more likely to occur when the visual evidence distributions of competing candidates exhibit substantial overlap, making them difficult to distinguish. Following this motivation, the intervention should help the model more reliably distinguish competing candidates based on visual evidence. However, CIVER enhances the visual evidence associated with the top-ranked candidate while suppressing evidence that uniquely favors the second-ranked candidate. This design appears to implicitly assume that the leading candidate is more reliable, which may not hold in hallucination-prone scenarios, where the leading candidate itself could be incorrect.
  2. Insufficient justification for the candidate-conditioned visual evidence estimation. The paper estimates candidate-specific visual evidence distributions by computing similarities between candidate-conditioned queries and visual token representations. However, this design relies heavily on heuristic choices and lacks sufficient justification. For example, the candidate-conditioned query qtkq_t^k is constructed by combining the anchor-layer hidden state with the corresponding candidate's LM Head output weight vector, but it remains unclear whether the dot product between the resulting query and visual token representations can reliably capture the visual evidence supporting that candidate. Moreover, since both qt+q_t^+ and qt−q_t^- share the same contextual hidden state hth_t, the overlap between their estimated visual evidence distributions may partly arise from the query construction itself, rather than genuinely reflecting the model's ability to discriminate between candidates based on visual information.
  3. The evaluated models are relatively outdated, leaving the applicability to recent LVLMs unclear.
Questions for authors

no other questions

Reviewer_KwYsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Competition-Informed Visual Evidence Refinement (CIVER), a training-free decoding method for mitigating hallucinations in LVLMs. It identifies candidate-level visual-evidence ambiguity as a potential source of hallucination, showing that hallucinated predictions tend to exhibit greater overlap in the visual-evidence distributions of competing token candidates, even under strong visual attention. To address this, CIVER combines a sample-specific visual prior, candidate-conditioned supporting and competing evidence, and cross-step evidence memory to selectively refine visual attention during decoding. Experiments on four LVLMs across POPE, CHAIR, MME, and AMBER demonstrate consistent improvements over existing training-free baselines.

Critical strengths
  1. Interesting motivation and perspective on hallucination mitigation. The paper introduces candidate-level visual-evidence ambiguity as an alternative perspective to the common assumption that hallucinations primarily result from insufficient visual attention. The preliminary analysis provides empirical evidence that greater overlap between competing candidates' visual evidence is associated with higher hallucination rates, offering useful insights into hallucination mitigation.
  2. Consistent improvements across multiple models and benchmarks. CIVER achieves strong results across four LVLMs and four hallucination benchmarks, with particularly notable improvements on CHAIR. The ablation studies further support the contributions of its main components, while the efficiency analysis demonstrates relatively modest inference overhead.
Critical weaknesses
  1. Insufficient validation of candidate-level visual-evidence ambiguity and its role in hallucination mitigation. The candidate-conditioned evidence distributions are estimated using similarity between hidden representations, but the paper does not verify whether they accurately reflect visually grounded evidence. And CIVER reinforces evidence associated with the leading candidate, which may itself be hallucinated.
  2. Insufficient mechanism-isolation experiments and implementation details. Although the ablation studies demonstrate the contributions of individual components, they do not establish whether the improvements arise from candidate-specific evidence discrimination rather than generic attention redistribution. Additionally, several important hyperparameters and the warm-up procedure are insufficiently specified, limiting reproducibility.
Questions for authors
  1. How to validate that the estimated candidate-conditioned evidence corresponds to actual visual grounding? Why should reinforcing the leading candidate help when that candidate is itself hallucinated?
  2. Can you evaluate CIVER on more recent and advanced LVLMs, such as Qwen2.5-VL or Qwen3-VL, to demonstrate its effectiveness and generalizability beyond the relatively outdated models used in the current experiments? Additionally, can you clarify the missing hyperparameters and warm-up implementation details?
Reviewer_CjHJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CIVER, a training-free decoding framework for mitigating hallucinations in large vision-language models. Its core idea is to move beyond focusing only on the overall amount of visual contribution and instead examine whether the visual evidence associated with competing token candidates is sufficiently discriminative. CIVER estimates candidate-conditioned visual evidence at an intermediate anchor layer, compares the provisional leading candidate with its strongest alternative, and combines supporting evidence, competing evidence, a sample-specific visual prior, and cross-step memory into a signed bias that is selectively applied to evidence-aligned attention heads in subsequent decoder layers. The method can reduce hallucination-related metrics to a certain extent while preserving general multimodal performance, and its inference overhead is moderate.

Critical strengths
  1. This paper provides a useful perspective on an important failure pattern associated with hallucinations in LVLMs, namely, the visual-evidence distributions of competing candidates. It finds that hallucinations can persist even when the model assigns relatively strong attention to image tokens, and that greater overlap between the visual-evidence distributions of competing candidates is associated with higher hallucination rates.

  2. This paper also proposes a training-free hallucination mitigation method and validates its effectiveness to a certain extent across four representative LVLM backbones. Runtime analysis further shows that its inference overhead is moderate.

Critical weaknesses
  1. In Eq. (9), h\mathbf{h} and e\mathbf{e} are combined through a weighted sum. Although they have the same dimensionality, they belong to different semantic spaces. The authors then apply L2 normalization to the weighted sum, but they do not explain why this operation is valid or what the weighted sum is intended to represent. In addition, it remains unclear why the inner product of q\mathbf{q} and v\mathbf{v} in Eq. (10) can be interpreted as candidate-conditioned visual evidence. If the method is intended as a heuristic, the authors should explain its rationale and justification, or provide comparative experiments with alternative designs to validate its reasonableness.

  2. The method contains many threshold parameters and selection rules, but most of them are not adequately justified. These include η\eta in Eq. (9); αs,αc,αm,bmax⁡\alpha_s,\alpha_c,\alpha_m,b_{\max} in Eq. (14); and λm,λo,δg,τg\lambda_m,\lambda_o,\delta_g,\tau_g in Eq. (12). Moreover, sensitivity analyses should be provided for the relevant important parameters. This affects the reproducibility and credibility of the framework.

Questions for authors

1.In the second paragraph below Eq. (1), the authors construct a candidate pair using “the token selected by the final layer” and “the highest-ranked anchor-layer candidate other than ytouty_t^{\mathrm{out}}.” However, in the CIVER method description, they state that the candidate pair is constructed from the anchor-layer Top-2 tokens. Since CIVER uses the anchor-layer Top-2 tokens as the candidate pair, why does the retrospective diagnosis use the final-layer output? Please clarify the distinction between these two candidate pairs.

2.In the paragraph below Eq. (2), the authors define τA\tau_A but do not give the rule for its use, which creates a logical break in the writing. The authors also state that τA\tau_A and τD\tau_D are the medians of visual attention and evidence distance obtained from a calibration split, but what is this calibration split? It is not defined earlier in the paper. In addition, what does “This yields 1,704 events on the held-out analysis split” mean? Is 1,704 the number of all eligible single-token object events, or the number of events with Dt<τDD_t<\tau_D? This is unclear.

3.Why is the competing evidence Ct\mathbf{C}_t not used when selecting the reference map, i.e., the original formula is Ft=N(P+St+Mt)\mathbf{F}_t=\mathcal{N}(\mathbf{P}+\mathbf{S}_t+\mathbf{M}_t), but Ct\mathbf{C}_t is included when adding the bias, i.e., Eq. (14) contains −αcZ(Ct)-\alpha_c\mathcal{Z}(\mathbf{C}_t)?

4.In Eq. (9), h\mathbf{h} and e\mathbf{e} are combined by a weighted sum. However, although h\mathbf{h} and e\mathbf{e} have the same dimension, they are not in the same semantic space. The authors then apply L2 normalization after the weighted sum. The authors do not explain why this operation is valid or what the weighted sum represents. Also, why can the inner product of q\mathbf{q} and v\mathbf{v} in Eq. (10) represent candidate-conditioned visual evidence?

5.The method includes a large number of threshold parameters and selection rules, but most of them are not given reasonable explanations, including η\eta in Eq. (9); αs,αc,αm,bmax⁡\alpha_s,\alpha_c,\alpha_m,b_{\max} in Eq. (14); and λm,λo,δg,τg\lambda_m,\lambda_o,\delta_g,\tau_g in Eq. (12). In addition, sensitivity analyses should be conducted for the relevant important parameters.

6.The paper contains many symbols. It is suggested that the authors provide a complete notation table. There are also notation issues in the paper, where multiple symbols represent the same meaning. For example, is L2Norm(⋅)\mathrm{L2Norm}(\cdot) in Eq. (9) the same as xˉ\bar{x} defined earlier?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.