acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Visual Evidence-Guided Decoding for Hallucination Reduction in LVLMs

Abstract

Large Vision-Language Models (LVLMs) are prone to hallucinations, producing outputs that are linguistically plausible but not grounded in visual evidence. Existing mitigation approaches typically rely on multi-pass interventions, attention reallocation, or look-twice visual re-injection mechanisms. Although effective, these methods either incur significant computational overhead or provide only partial mitigation, limiting their reliability in practical deployment. In this work, we introduce Adaptive Visual Evidence-Guided Decoding (AVED), a unified single-forward-pass framework that explicitly incorporates visual evidence into the decoding process. Instead of treating visual grounding as an external correction step, AVED integrates visual evidence estimation and decoding within a single forward pass, eliminating the need for multiple inference stages or external visual re-injection. Specifically, our method consists of two complementary components: Visual Attention Calibration (VAC), which refines cross-modal attention to capture vision-language correspondence, and Layer-wise Logits Reweighting (LLR), which adaptively aggregates intermediate decoding predictions according to their estimated visual evidence reliability. The two components are tightly connected through cross-attention signals, where calibrated attention provides visual evidence estimates that guide evidence-aware logits reweighting. Extensive experiments on hallucination benchmarks demonstrate that our approach consistently suppresses hallucinations while improving generalization under distribution shifts, establishing a simple yet effective paradigm for reliable LVLM inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hallucination Reduction, LVLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_6HMBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes AVED for hallucination mitigation in LVLMs, which include Visual Attention Calibration (VAC) and Layer-wise Logits Reweighting (LLR). VAC redistributes existing visual attention using text–visual similarities, while LLR combines layer predictions using an attention-conditioned similarity concentration score. The method requires only one full model forward pass per decoding step and reports improvements across hallucination and general VQA benchmarks.

Critical strengths
  1. Practically relevant direction. Combining visual attention calibration with multilayer prediction offers a useful approach to hallucination mitigation without an additional full model forward pass.
  2. Broad experimental coverage. The evaluation covers multiple backbones, hallucination and general VQA tasks, Gaussian image corruption, and comparisons of components and layer-confidence measures.
Critical weaknesses
  1. Serious inconsistencies in quantitative reporting. Tables 2 and 4 report inconsistent LLaVA-1.5 results, while incorrect averages in Tables 7, 10, and 12 understate baseline performance or overstate AVED's. These issues undermine confidence in the reported gains.
  2. Limited scope of VAC. A potential source of hallucination is that LVLMs allocate too little attention to visual inputs relative to text [1,2]. VAC only redistributes visual attention while preserving its total mass. Its effectiveness when visual attention is already very weak remains unclear.
  3. Incomplete efficiency evaluation. Table 23 omits results for the complete AVED (VAC + LLR) configuration, leaving the claimed efficiency advantage unverified.
  4. Poor writing and presentation. The writing lacks a clear logical progression, particularly in the fifth and sixth paragraphs of the introduction and in the method section.
Questions for authors

See Weaknesses.

References

[1] Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs. ECCV 2024.

[2] Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models. CVPR 2026.

Reviewer_xkX9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces AVED, a method for reducing statements by vision-language models that are not supported by the input image. It adjusts the model’s attention to image regions and combines prediction scores from different internal layers, weighting them using a score based on similarities between text and image representations. These changes require no retraining or second full model inference at each generation step. AVED is evaluated across several VLMs (LLaVA, Qwen2.5-VL, Qwen3.5, GLM-4V) using the Caption Hallucination Assessment with Image Relevance (CHAIR) metric for image descriptions and visual question-answering benchmarks (POPE, MME, MM-Vet, MMBench), with reported results showing reduced hallucination rates alongside generally improved task performance.

Critical strengths
  1. The proposed methodology reduces mentions of objects that are not present in an image without retraining the VLM or requiring another inference at each generation step. This makes it easier to apply to existing VLMs, making the contribution practical.
  2. In the image-description evaluation, descriptions mentioning absent objects decreased by 8 percentage points, while the proportion of real objects correctly mentioned increased by 2 percentage points; i.e. the model makes fewer false claims while retaining useful information.
Critical weaknesses
  1. The paper does not establish a substantial methodological advance over closely related prior work that it does not cite. For example:
    [a] The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering (ICML 2025)
    Averages the logits of the layers just before the final layer and combines them with final-layer logits to reduce hallucination, which is similar to LLR with uniform layer weights.
    [b] Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding (NAACL 2025)
    Uses inter-layer differences in image-token attention to select layers for contrastive decoding, establishing a connection between visual attention and layer-dependent decoding.
    [c] SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision (AAAI 2026)
    Selects an earlier layer using a visual grounding score and adds its scaled logits to the final-layer logits, which resembles LLR.
    [d] CAAC: Confidence-Aware Attention Calibration to Reduce Hallucinations in Large Vision-Language Models (WACV 2026)
    Performs visual attention calibration, similar to VAC, along with confidence-guided attention rescaling.
    [e] Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs (NAACL 2025)
    Uses cosine similarity between contextual answer embeddings and image-patch embeddings for hallucination detection and grounding, similar to VEC (Eqs. 4–5).
    None of the decoding methods [a-d] are compared to in experiments, or even appear in the references as related work.
  2. The paper states that some hallucination mitigation methods “require multiple forward passes, limiting their efficiency in practical deployment” (the claim appears 3x across the abstract, introduction, and related work), and Table 23 in the appendix provides some evidence of this for the ICD and VCD baselines. However, experiments in the main paper don't evaluate the efficiency of the proposed method, while the appendix reports runtime measurements for VAC and LLR only separately. The paper even mentions 2x “reflected in the runtime measurement”, but it does not show whether AVED offers a better balance between speed and hallucination reduction than the baselines, which is a major weakness.

Also important to consider:

  • There is no "AI use statement"
  • The ICLR template is altered by using numerical citations that artificially extend the page limit while hindering readability.
  • No code is attached to verify the paper's findings, hindering reproducibility.

Other comments:

  • L47-49: Sentences "Attention reallocation methods dynamically adjust [...] However, they often" require some references to the literature.
  • L83-88: This paragraph is full of jargon that is hard to understand at this point. Also, L90-95 give even more details on the method, and even reference an ablation in Section 4.3, which would be clearer to keep in Section 3.
  • Contribution 2: Again, it is completely unclear what it means, as "confidence" is undefined at this point but used 4 times.

"AVED establishes visual-evidence coupling between attention calibration and layer-wise decoding, together with a layer-confidence criterion grounded in visual evidence rather than linguistic confidence alone. Controlled comparisons with five alternative layer confidence measures substantiate the proposed layer-wise confidence criterion." I would recommend rewriting it in plain, human-comprehensible language.

  • "Visual Attention Calibration (VAC)" and "Layer-wise Logits Reweighting (LLR)" appear 10 and 8 times, respectively
  • Figure 1: In the left column, what are the "1, 2, 3 Output", and what is "input" - only text or both image and text? The middle column does not reflect what the proposed "VAC" approach is doing.
  • Related work could discuss more relevant literature on LLMs; see e.g. Attribution-Guided Decoding (ICLR 2026) and references given there.
  • L196: "Because pp is a softmax of normalized similarities, it is nonnegative and sums to one." Softmax is always nonnegative and sums to one.
Questions for authors
  1. Why does AVED have different CHAIR scores in Tables 1 and 20, and what experimental settings were used for Table 6?

  2. What evidence shows that visual evidence-guided confidence (VEC) gives higher scores to layers that make more visually accurate predictions, beyond the comparisons in Table 6?

  3. How fast is the complete VAC+LLR method compared with the baseline and closely related methods? If the main claim is that the method is efficient, it necessitates plotting the performance-efficiency Pareto frontier.

Reviewer_xhSAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AVED, a single-forward-pass framework for mitigating hallucinations in large vision-language models (LVLMs). AVED has two components: (1) Visual Attention Calibration (VAC), which redistributes the text-query-to-visual-key attention mass within selected heads using a softmax over row-normalized text–visual similarity, preserving the original visual attention mass; and (2) Layer-wise Logits Reweighting (LLR), which computes a per-layer "visual evidence-guided confidence" (VEC) — the max softmax over cosine similarities between a task-aware layer representation and the top-K attention-selected visual tokens — and uses it to weight an intermediate-layer logit ensemble added to the final-layer logits. The authors claim a single autoregressive evaluation per token, no visual re-injection, and improvements on CHAIR, POPE (clean and Gaussian-perturbed), MME, MM-Vet, and MMBench across LLaVA-1.5-7B, Qwen2.5-VL-7B, Qwen3.5-4B, and GLM-4V-9B, plus ablations and runtime measurements.

Critical strengths
  1. The work is well motivated and design choices are interesting. The attention-conditioned top-K evidence selection is directly ablated, and the task-aware VQA representation (question + generated answer mean) is validated against last-token and all-text means. The failure modes of the alternatives are argued, not just asserted.

  2. Experimental evaluation is broad and comprehensive. Four backbones, five benchmarks, a controlled perturbation sweep (0.08–0.38), a head-to-head against both multi-pass (VCD, ICD) and look-twice (MemVR, AIR) families, and component-level runtime/memory measurements. The perturbation robustness results are the most persuasive part of the empirical case, and the AD-HH comparison is a useful, non-trivial baseline.

Critical weaknesses
  1. LLR is structurally DoLA, and the paper does not say so. LLR is DoLA (ref. [38]) with the layer weight swapped from JS-divergence to VEC — and DoLA is in the baseline table. The only comparison is a single CHAIR row (54.40 vs. 44.40). A controlled LLR-vs-DoLA-weighting comparison (and ideally DoLA+VAC) is needed to isolate what the visual-evidence signal buys; Table 6 compares other measures but not DoLA's actual rule.

  2. Key comparison tables are confounded/misaligned. Table 2 uses inverted arrows ("↓" for improvements), mixes backbones without clear separation, and Table 4 has rows that do not align with their headers. These must be rebuilt before any claims can be verified — a formatting issue that is an evidentiary one in practice.

  3. Sparse layer-decoding baseline. DoLA is the only one; it is unclear whether it was re-run under identical settings.

Questions for authors
  1. DoLA control: Report LLR with DoLA's JS-divergence weights vs. VEC under identical settings, and DoLA+VAC if feasible. What survives when the weighting rule is held fixed?

  2. Table integrity: Provide a corrected, backbone-separated Table 2 and a row-aligned Table 4 with arrow conventions stated.

  3. Variance: Report seeds/confidence intervals for the main POPE and CHAIR results, and per-split error bars for Table 3.

Reviewer_755ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Visual Evidence-Guided Decoding (AVED), a single-forward-pass framework for reducing hallucinations in Large Vision-Language Models (LVLMs). AVED consists of two components: Visual Attention Calibration (VAC), which redistributes visual attention based on vision-language similarity, and Layer-wise Logits Reweighting (LLR), which adaptively aggregates intermediate-layer predictions using visual evidence-guided confidence (VEC). Experiments across multiple LVLM backbones and benchmarks demonstrate improvements in hallucination mitigation, general VQA performance, and robustness to visual perturbations.

Critical strengths

The overall framework is well-designed. AVED integrates attention calibration and layer-wise decoding within a single forward pass, providing an adaptive way to incorporate intermediate-layer predictions based on attention-derived visual information.

Critical weaknesses
  • Limited analysis of VAC under irrelevant visual inputs.

    While the mass-preserving property of VAC is a reasonable design choice, its effectiveness relies on the assumption that redistributing attention within the visual tokens is beneficial. However, this assumption may not always hold. In particular, when the original attention mass assigned to visual tokens is already very small, it is unclear whether these visual tokens contain meaningful evidence for the current prediction. More importantly, when the visual input is irrelevant or inconsistent with the textual query, VAC may introduce misleading visual information into decoding. The current experiments primarily evaluate settings in which visual information is expected to be relevant. I think the authors should investigate the behavior of VAC under weak or irrelevant visual evidence, such as mismatched image–question pairs, text-only questions accompanied by unrelated images, or cases with naturally low text-to-visual attention mass, to assess the robustness of proposed method.

  • Insufficient justification of Visual Evidence-Guided Confidence (VEC).

    The proposed VEC score (Eq. 5) is defined as the maximum softmax-normalized similarity between a layer representation and its attention-selected visual tokens. Thus, VEC measures the concentration of visual similarity rather than the factual correctness of layer predictions. A high VEC score does not necessarily indicate more reliable visual evidence. Although Table 6 shows improvements over alternative confidence measures, it does not establish whether high-VEC layers actually produce fewer hallucinations. The authors should analyze the correlation between VEC and layer-level factual correctness, compare high- and low-VEC layers, and examine VEC distributions for correct versus hallucinated predictions.

  • Missing end-to-end inference efficiency results.

    Although single-forward-pass inference is a key motivation, Table 23 only reports the efficiency of VAC and LLR separately, without evaluating the complete AVED framework. Since VAC alone reduces throughput from 11.6 to 7.8 tokens/s, the computational overhead is non-negligible. The authors should report the end-to-end latency and throughput of full AVED under the same hardware settings as the baselines.

Questions for authors

The CHAIR results differ substantially across Tables 1, 6, and 19 (e.g., CHAIR-S of 44.4, 26.0, and 46.8). Could the authors clarify the differences in experimental settings that lead to these discrepancies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.