- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates confident visual hallucinations (CVH) in vision-language models (VLMs). It constructs ConfHall-VL, an MS COCO-derived diagnostic dataset that identifies and labels high-confidence hallucinated object claims using VLM-generated captions and image annotations. The paper further proposes EViC-VL, which combines evidence from two open-vocabulary object detectors, OWLv2 and Grounding DINO, to detect such hallucinations and guide caption editing.
Critical strengths
- Curating a diagnostic dataset for confident visual hallucinations is a meaningful effort, given the lack of comprehensively annotated data targeting this specific failure mode.
- Combining evidence from multiple open-vocabulary object detectors (OWLv2 and Grounding DINO) can improve the reliability of hallucination detection compared with relying on a single detector.
Critical weaknesses
-
The paper motivates CVH primarily through overconfident token probabilities and limited self-uncertainty signals. However, substantial prior research has developed more reliable and better-calibrated uncertainty quantification methods that can, to some extent, mitigate such overconfidence. These methods could potentially provide a stronger basis for identifying confident hallucinations than the generation confidence and self-uncertainty measures adopted here. The paper should better justify its choices in light of these alternatives.
-
The paper often reads more like a technical report than an academic research paper, particularly in its early sections. While research papers need not follow a rigid structure, a clear narrative is essential. E.g., the Research Question of the Introduction mixes research questions, contributions, and experimental findings. Sections 3 and 4 focus largely on procedural details without adequately explaining the rationale behind the methodological choices. In particular, the selection of evaluation metrics and thresholds appears somewhat arbitrary and insufficiently justified.
-
Although the appendices contain substantial technical details, some sections may be overly extensive, particularly Appendix F (Structural Properties) and Appendix G (Full Proofs), contribute relatively little to substantiating the paper's central contributions.
Questions for authors
None beyond the concerns raised above.
Thank you for the detailed responses and revisions. I appreciate the additional analyses, particularly the detector disagreement, fusion comparisons, and matched operating-point results. The responses have clarified several of my concerns, but they also confirmed that some of the original claims were not fully supported. While the revised manuscript now presents a more accurate and appropriately scoped contribution, several issues have been addressed mainly by narrowing the claims rather than providing additional validation. In particular, the methodological novelty remains limited, the practical benefit beyond detection AUROC is still unclear, and independent human-label validation and cross-model replication remain unresolved. I therefore do not think the revisions substantially change my overall assessment of the paper's contribution. I will maintain my original rating of Weak Rejection (2/4). However, if there is any specific evidence or result in the revised manuscript that I may have overlooked and that directly addresses these remaining concerns, I would be happy to reconsider.