acceptodds
Withdrawn from ICLR 2027

ROAM: Mitigating Vision-Language Hallucinations with Calibrated Grounding Monitors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models can produce fluent responses that drift from visual evidence during generation. We introduce Read-Only Attention Monitoring (ROAM), a training-free decoding method that decouples visual-grounding monitoring from intervention. Offline, ROAM selects reusable monitoring heads using text-to-vision attention statistics and their stability across controlled image and text perturbations. The resulting monitor is calibrated once per model and reused across inputs. Online, the selected heads remain read-only: their image-attention entropy provides a grounding signal that adaptively controls the sampling temperature, sharpening the next-token distribution when the signal indicates weaker grounding. The intervention acts solely on the output logits, requiring a single forward pass per token without auxiliary decoding branches or direct modifications to internal representations. Analyses on LLaVA show that lower grounding scores are associated with higher hallucination rates before intervention, while ablations support the value of cross-environment calibration. Experiments across three VLM families and four benchmarks show the largest gains in long-form generation. On CHAIR, ROAM reduces sentence-level hallucination by 2.4–6.4 percentage points relative to vanilla decoding while increasing object recall on all three models, with near-vanilla end-to-end latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multimodal hallucination, visual grounding, decode-time inference, attention heads, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7rVmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ROAM, a training-free decoding method for mitigating hallucinations in vision-language models. ROAM first calibrates a set of reusable attention heads using entropy-based statistics across perturbed image-text inputs. During inference, the selected heads serve as read-only monitors to estimate visual grounding, which is then used to adaptively adjust the sampling temperature. Experiments across multiple models and benchmarks demonstrate improvements in hallucination mitigation with minimal inference overhead.

Critical strengths

I appreciate the authors' efforts in conducting experiments on more recent vision-language models, such as Qwen3-VL and InternVL3.5, rather than relying primarily on earlier models. The evaluation also covers both generative and discriminative hallucination benchmarks, demonstrating a relatively comprehensive experimental effort.

Critical weaknesses
  1. Limited methodological novelty. The main components of ROAM, including entropy-based attention-head selection, visual-attention entropy monitoring, and adaptive temperature scaling, are relatively straightforward and closely related to existing works. Although the authors emphasize the separation of monitoring and intervention, this distinction appears primarily architectural rather than a substantial methodological advance. The paper does not sufficiently establish the unique advantages of this design beyond computational efficiency.

  2. Insufficient justification for the core methodological assumptions. The method assumes that concentrated visual attention indicates stronger grounding, while diffuse attention suggests a higher hallucination risk. However, attention concentration does not necessarily imply that the model is attending to the correct visual evidence.

  3. Missing critical baselines. Since ROAM primarily mitigates hallucinations by reducing the sampling temperature, comparisons with simple fixed-temperature decoding strategies are essential.

Questions for authors

no other questions

Reviewer_36gGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Read-Only Attention Monitoring (ROAM), a training-free method for reducing hallucinations in long-form generation by large vision-language models. The main idea is to monitor changes in visual attention during decoding and adjust the sampling temperature when the model becomes less grounded in the image. Specifically, the authors introduce an offline Environment-Informed Calibration (EIC) procedure using 8,000 COCO samples under seven perturbation settings to select attention heads and determine their aggregation weights. During inference, ROAM measures the normalized Shannon entropy of the selected attention heads to estimate visual grounding and adjusts the decoding temperature accordingly. The method requires only a single decoding pass and does not modify the model's hidden states. Experiments on several LVLM families show reduced hallucination rates and improved object recall, with little additional inference overhead.

Critical strengths
  1. Decoupled and Lightweight Design: Separates read-only attention monitoring from decoding intervention, without the extra computational cost of dual-branch methods (VCD/M3ID) or modifying internal hidden states.

  2. Reliable Offline Calibration: Evaluates the proposed signal across seven perturbation settings, showing consistent results across data splits and a clear correlation with hallucination risk.

  3. Improved Accuracy and Coverage: Reduces CHAIRS\text{CHAIR}_S by 2.4–6.4% while improving object recall, suggesting that the gains are not simply due to generating shorter captions.

Critical weaknesses
  1. Limited Model Scale and OOD Evaluation: All evaluated models are below 10B parameters. Given the available A100 GPUs, the authors are encouraged to include experiments on larger models (e.g., 13B) to assess the scalability of ROAM. In addition, the current evaluation mainly focuses on COCO-based hallucination benchmarks. Results on general VQA and OCR benchmarks, particularly under OOD settings, would help demonstrate whether the method generalizes to other vision-language tasks.

  2. Missing Fixed-Temperature Baselines: Since ROAM mainly adjusts the sampling temperature during decoding, it is unclear how much of the improvement comes from the proposed grounding signal rather than temperature reduction itself. The authors should compare ROAM with fixed-temperature baselines (τ∈{0.3,0.5,0.7}\tau \in \{0.3, 0.5, 0.7\}). A shuffled-temperature baseline would also help determine whether the grounding-guided adjustment is more effective than simply varying the temperature.

  3. Bias Toward Shallow Attention Layers: The EIC procedure appears to select attention heads mainly from the first two layers (0/1). These early layers may capture low-level visual information but might not fully reflect semantic-level visual grounding. It would be useful to investigate whether including attention heads from middle or deeper layers improves the results. In particular, an ablation combining shallow-layer and middle-layer heads could help clarify whether the current selection strategy misses useful grounding information.

Questions for authors

See the weaknesses. I will improve my score after the author addresses my concerns. I hope the author can supplement the corresponding experimental results in the days before the end of the rebuttal and mark them in different colors in the updated article for my review and score improvement.

Reviewer_jPrgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ROAM, a training-free method for mitigating hallucinations in vision-language models. The method first performs offline calibration through image and text perturbations to select monitoring heads with stable visual attention characteristics. During inference, it constructs a dynamic visual grounding signal using the visual attention entropy of the selected attention heads, and adaptively adjusts the sampling temperature accordingly, without modifying the model's internal representations or adding extra decoding branches. Experiments cover three types of vision-language models and four hallucination evaluation benchmarks. Results show that ROAM can effectively reduce object hallucinations in long-form generation while maintaining or improving object recall, and achieves inference efficiency close to that of the original decoding.

Critical strengths

The experimental coverage is relatively solid. Three models (LLaVA, Qwen3-VL, and InternVL3.5) and four evaluations (CHAIR, AMBER, POPE, and MME), supplemented by ablation, calibration stability, and transfer experiments, are far more convincing than validation on only a single model.

Critical weaknesses

\textbf{Weakness 1: The absence of a key static temperature control baseline makes it difficult to verify the independent contribution of the core mechanism.}

ROAM dynamically lowers the sampling temperature via visual attention entropy to mitigate hallucinations, but the paper does not adequately compare against fixed low-temperature sampling, matched average temperature, or randomly shuffled grounding signals. The existing ablation experiments mainly validate the effectiveness of monitoring head selection and cannot rule out that the performance gains stem primarily from low-temperature sampling itself rather than from the proposed visual grounding guidance mechanism.

\textbf{Weakness 2: The reliability of the visual grounding signal and its mechanistic link to hallucination mitigation remain insufficiently established.}

The paper adopts visual attention concentration as a proxy for grounding, but attention concentration does not necessarily imply that the generated content is supported by correct visual evidence. Although the experiments find a statistically significant correlation between the grounding score and hallucination rate, the correlation strength is weak (rpb=−0.089r_{\mathrm{pb}}=-0.089), and failure cases show that correct and incorrect objects may have similar grounding scores. Therefore, the existing evidence is insufficient to fully demonstrate that this signal can reliably identify hallucination risk and guide effective intervention, weakening the paper's mechanistic explanation and scientific contribution.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.