acceptodds
Withdrawn from ICLR 2027

AMEND: Adaptive Modulation via Enhanced Navigation Decoding for Mitigating Hallucinations in LVLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated extraordinary capabilities in cross-modal tasks. However, they frequently suffer from hallucinations, generating descriptions that are inconsistent with the visual inputs. Existing training-free attention intervention methods lack dynamic adjustment, resulting in poor visual grounding or instruction neglect, while indiscriminately enhancing all visual tokens amplifies background noise. To address this, we propose AMEND, a novel training-free approach that mitigates hallucinations by dynamically recalibrating visual attention. Specifically, this adjustment is performed from two aspects: (1) Uncertainty-Aware Adaptive Scaling, which leverages the entropy of output token logits to estimate generation uncertainty and adaptively adjusts the visual attention enhancement coefficient; and (2) Expert-Layer Attention-Driven Token Extraction, which analyzes attention weights across vision-dominant expert layers to selectively identify the most critical and question-relevant visual tokens, rather than indiscriminately amplifying all image tokens. Finally, using the dynamically enhancement coefficient, we amplify attention to the selected key visual tokens while suppressing attention to irrelevant visual tokens and system prompts. Extensive experiments on standard benchmarks demonstrate that AMEND significantly alleviates hallucinations and outperforms existing baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Hallucination, Training-free Decoding Strategy, Adaptive Visual Attention Mechanism

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Za28OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMEND, a training-free decoding strategy that mitigates hallucinations in LVLMs by dynamically recalibrating visual attention. It consists of two components. Uncertainty-Aware Adaptive Scaling adjusts the visual attention enhancement coefficient based on the entropy of the output logits. Expert-Layer Attention-Driven Token Extraction selects question-relevant visual tokens from vision-sensitive expert layers. Attention to the selected tokens is amplified, while attention to the other visual tokens and the system prompt is suppressed. Experiments on POPE and MME-Hallucination with LLaVA-1.5 and Qwen2.5-VL show that AMEND outperforms existing decoding-based methods in most settings.

Critical strengths
  1. Modulating visual attention according to model uncertainty is a reasonable idea. AMEND decides both when to intervene (via entropy) and where to intervene (via critical tokens), which avoids the over- or under-enhancement of fixed-coefficient methods.
  2. The method is training-free and applicable to different LVLMs (LLaVA-1.5 and Qwen2.5-VL). It is relatively efficient compared with contrastive decoding baselines and achieves strong results across multiple benchmarks.
Critical weaknesses
  1. The novelty is limited. Uncertainty-triggered visual intervention and selective manipulation of visual attention have already been explored, e.g., in MemVR (ICML 2025), AdaptVis (ICML 2025), VAR (ICLR 2025), and Self-Aug (ICLR 2026). The differences from these works should be clarified.
  2. The method has many hyperparameters (entropy center and scale, coefficient bounds, β, γ, λ, and the layer range), and the paper states that layer selection and the token retention ratio are tuned per task type. This raises concerns about generalization to new models and tasks.
  3. The assumption that high entropy indicates hallucination is supported only by an empirical analysis of a single model on a single benchmark (Fig. 2), without theoretical justification. High entropy can also stem from linguistic ambiguity or inherently uncertain predictions, and hallucinations can occur with high confidence.
  4. It is unclear whether high-attention tokens in the expert layers actually correspond to question-relevant objects. Prior work shows that LVLMs often assign high attention to irrelevant background tokens (e.g., Visual Attention Sink, ICLR 2025; DAMRO, EMNLP 2024), so amplifying such tokens may reinforce hallucinations rather than mitigate them.
Questions for authors
  1. Could you provide a quantitative analysis of whether the selected tokens focus on meaningful regions, for example their overlap with ground-truth object regions, or a comparison with randomly selected tokens?
  2. Standard attention weights sum to one after softmax, but multiplying them by coefficients as in Eq. (11) may break this property. Is the modified attention renormalized? If not, how does this affect the model?
  3. Could you evaluate AMEND on open-ended generation benchmarks such as CHAIR, where token-level adaptivity should matter more?
Reviewer_VQR3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out that existing training-free attention intervention methods rely on fixed scaling or uniformly enhance visual tokens, lacking dynamic adjustment. To address these limitations, the authors propose AMEND, a training-free hallucination mitigation method that combines Uncertainty-aware Adaptive Scaling, Expert-Layer Attention-Driven Token Extraction and Contrastive Attention Calibration. Experiments on POPE and MME demonstrate improvements across three models.

Critical strengths
  1. The paper provides a clear motivation. Based on this motivation, the proposed method dynamically adjusts the intensity of visual attention intervention and selectively enhances critical visual information.
  2. The proposed method is training-free and does not significantly impact inference latency.
Critical weaknesses
  1. The experiment only includes the POPE and MME discriminative benchmarks, lacking a comprehensive evaluation of the proposed method. Could the authors add evaluations on more benchmarks, including hallucination benchmarks for long-form generation (e.g., CHAIR, AMBER) and general vision benchmarks (e.g., MMBench, LLaVA-Bench)?
  2. The proposed method involves several hyperparameters, including αmin\alpha_{min}, αmax\alpha_{max}, β\beta, γ\gamma, λ\lambda, kk (number of expert layers) and the selected enhancement layers. Does the effectiveness of the proposed method depend on manually tuning of key parameters? The authors should discuss the resulting tuning cost for new models. Besides, the paper lacks ablation analysis for some parameters such as αmin\alpha_{min}, αmax\alpha_{max}, β\beta and kk.
Questions for authors
  1. Could the critical visual tokens selected by the expert layer provide correct visual evidence? How does the expert layer selection avoid the impact of attention sinks? A quantitative visual grounding analysis can further prove its effectiveness in identifying relevant visual evidence..
  2. Writing: The notations should be consistent. In Eq. (3), attention weight is denoted by αt,i\alpha_{t,i}, and in Eq. (6), the dynamic enhancement coefficient is denoted by αt\alpha_{t}. In Eq. (7-8), kk represents the number of expert layers, but in Eq. (11), kk represents the k-th context token.
  3. Typo: In Table 2, the four MME subset scores for ASCD on LLaVA-1.5-13B sum to 610.00, but the reported total score is 618.34.
Reviewer_7EaWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed AMEND: a training-free decoding method for LVLM hallucination built on by dynamically steering visual attention. This paper assumes the hallucination appears in VLMs because it lost attention to the key target visual patches. The proposed AMEND therefore boost the attention on the visual tokens that already receive above the average attention and suppress the others.

Critical strengths

The proposed AMEND algorithm is a clear, simple method, ready for plug-and-play. It achieves good performance as the best average POPE F1 in 8 of 9 model setting cells, the best MME total on two of three models, with moderate efficiency cost.

Critical weaknesses
  1. This paper has a mismatch between the proposed solution described in this paper and the actual experiment. Sections 3 and 4 are working on one specific VLM paradigm: a cross-attention module between text and image tokens to fetch visual information. As described in Section 3.1, every text token works as a query and cross-attends to the keys and values of the visual tokens, so that visual information is mixed into the updated text token. The updated text tokens themselves then go through ordinary self-attention to generate next answer. However, the 3 models used in this experiment (LLaVA-1.5-7B/13B, Qwen2.5-VL-7B) follow a totally different paradigm that don't have the cross attention module between text and visual token at all. In their working paradigm, visual tokens pass through an MLP projector and are inserted into the text token sequence. All the text, visual tokens together go through ordinary self attention. More than that, the released code diverges from Section 4 significantly. Scaling is applied pre-softmax, mismatch with equation 11 and introduced extra negative sign problem. The scoring layers and edited layers are distinct fixed/dynamic sets, and with \mu = 3.0 the entropy-driven \alpha never leaves [0.03, 0.05] on the reported entropy range. This makes the adaptive component effectively static.”
  2. There exists a clear mismatch between the motivation and the proposed AMEND. In Section 3.2, the authors claim that VLM's hallucination comes from lack of attention weight on the oracle target object region. In the cross attention paradigm, the total attention weight of text token to the visual tokens is 1. So if the mass is not on the oracle position, it is on other visual patches. However, AMEND does not look for the oracle position. I_critical (Eq. 10, λ = 0.1) is the set of patches that already receive above-average attention from the model itself, so the method basically boosts what is already attended and suppresses the rest. If a hallucination arises because the model is not looking at the object, those object patches land in the suppressed set by construction. The method therefore cannot help in the very case its motivation describes, and no evidence is offered that I_critical overlaps the object region.
Questions for authors

1.The premise that hallucination arises when attention mass on the object patches goes to zero (Eq. 3) is only one assumption. Attention is a softmax over the full context, so low attention on ROI patches does not necessary comes with information missing, the visual context can simply become a weighted sum of the remaining background patches. Previous papers [1, 2] show that low-information background patches can act as registers that store global image semantics, so attention to them is not necessarily noise and can not infer that LLM generate the answer base on only text (P_LLM(y_t | y_<t) in equation 3). A simple ablation study can solve this question by showing that attention mass on ROI patches is lower for hallucinated answers than correct ones, and AMEND can raise it. [1] Darcet et al., Vision Transformers Need Registers (ICLR 2024) [2] Kang et al., See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025)

  1. Sec. 4.2 selects expert layers dynamically via cumulative-attention threshold γ (Eq. 7), but Implementation Details / App. A.1 modulate fixed layer ranges (8–13 for LLaVA, 7–19 for Qwen, chosen from 50 images). It’s unclear whether Eq. 7 is actually used, or whether the “expert layers” for scoring differ from the layers where attention is edited.

  2. Why does hyperparameter \lambda → 0 (keeping half of all image tokens) perform best given the paper’s argument against broad enhancement? What is the test set size of experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.