- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses two related problems: identifying visual regions associated with hallucinations and mitigating hallucinations in vision-language models (VLMs). The proposed method, VIRAH, selectively retains small groups of visual tokens and probes the model to assess the information contained in each region. It then identifies regions associated with high predictive entropy and removes their corresponding visual tokens before generating the final caption, aiming to reduce hallucinations.
Critical strengths
Intuitive and easy-to-implement methodology: The proposed approach to identifying potentially hallucination-prone visual regions is intuitive and relatively easy to implement. The paper also acknowledges some of the limitations of its method.
Integration of existing techniques: Although the individual techniques offer relatively incremental novelty, their integration into a unified framework for visual attribution and hallucination mitigation provides an interesting direction for further research.
Critical weaknesses
Lack of theoretical justification for hallucination mitigation: Although the proposed method may demonstrate empirical improvements, removing high-entropy visual tokens does not necessarily reduce hallucinations. In fact, removing visual information could make the model rely more heavily on contextual or linguistic priors, potentially reinforcing the very hallucinations the method aims to mitigate. The paper does not establish a rigorous connection between regional predictive entropy, hallucination risk, and the effectiveness of token removal.
Limited practical applicability and computational efficiency: The proposed approach requires repeatedly probing different image regions for each new input, making it potentially computationally expensive. Moreover, the method does not determine whether hallucinations would occur before applying the intervention, instead removing high-entropy regions regardless of whether they would actually cause hallucinations. These limitations raise concerns about its practicality for system-level hallucination detection and mitigation. The paper also does not provide end-to-end latency measurements for the complete mitigation pipeline under realistic deployment conditions.
Insufficient justification for the necessity of regional probing: Since the method does not directly identify hallucinated claims but instead removes regions associated with high predictive entropy, it remains unclear whether the expensive regional probing procedure is necessary for the observed improvements. The authors should investigate whether simpler interventions could achieve comparable results at a lower computational cost.
Missing ablation studies validating the central hypothesis: The experimental evaluation lacks sufficient ablation studies to establish the relationship between regional predictive entropy and hallucination risk. In particular, the authors should compare removing high-entropy versus low-entropy regions, evaluate entropy-based selection against alternative uncertainty measures, and investigate whether regional entropy actually correlates with the likelihood of hallucination. Such experiments are essential to validate the proposed uncertainty score and determine whether the observed improvements arise from the claimed mechanism rather than from token removal more generally.
Questions for authors
See above