acceptodds
Under review as a conference paper at ICLR 2027

EViC-VL: Evidence-Conditioned Verification for Confident Object Hallucinations in Vision-Language Models

Abstract

Vision–language models (VLMs) can produce object claims that are both linguistically confident and unsupported by the image. Such claims can evade verification policies that intervene only when a model exposes high self-uncertainty. We study this failure mode as confident visual hallucination (CVH). We operationalize CVH at the object-claim level and construct ConfHall-VL, a COCO-derived diagnostic that labels each candidate purely from the caption's own confidence and uncertainty signals together with the image annotations, without any detector evidence. We then propose EViC-VL, an evidence-conditioned procedure that queries two frozen open-vocabulary object detectors, OWLv2 and Grounding DINO, with canonical object names, averages their evidence scores, and applies a validation-frozen policy to the original caption. As a secondary robustness check, EViC-VL matches the frozen base generator on standard hallucination benchmarks overall, with its largest gain on object-presence polling accuracy (POPE F1). Our central question is whether combining these independent detectors improves object-level CVH discrimination over a single detector. On a sealed holdout, the equal-weight ensemble reliably raises the area under the ROC curve (AUROC) over the single-detector baseline by about four points. Overall, EViC-VL treats detector output as fallible evidence rather than ground truth, and this design measurably improves confident-hallucination detection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, object hallucination, hallucination detection, hallucination benchmark, verification, open-vocabulary object detection, uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9TqMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates confident visual hallucinations (CVH) in vision-language models (VLMs). It constructs ConfHall-VL, an MS COCO-derived diagnostic dataset that identifies and labels high-confidence hallucinated object claims using VLM-generated captions and image annotations. The paper further proposes EViC-VL, which combines evidence from two open-vocabulary object detectors, OWLv2 and Grounding DINO, to detect such hallucinations and guide caption editing.

Critical strengths
  1. Curating a diagnostic dataset for confident visual hallucinations is a meaningful effort, given the lack of comprehensively annotated data targeting this specific failure mode.
  2. Combining evidence from multiple open-vocabulary object detectors (OWLv2 and Grounding DINO) can improve the reliability of hallucination detection compared with relying on a single detector.
Critical weaknesses
  1. The paper motivates CVH primarily through overconfident token probabilities and limited self-uncertainty signals. However, substantial prior research has developed more reliable and better-calibrated uncertainty quantification methods that can, to some extent, mitigate such overconfidence. These methods could potentially provide a stronger basis for identifying confident hallucinations than the generation confidence and self-uncertainty measures adopted here. The paper should better justify its choices in light of these alternatives.

  2. The paper often reads more like a technical report than an academic research paper, particularly in its early sections. While research papers need not follow a rigid structure, a clear narrative is essential. E.g., the Research Question of the Introduction mixes research questions, contributions, and experimental findings. Sections 3 and 4 focus largely on procedural details without adequately explaining the rationale behind the methodological choices. In particular, the selection of evaluation metrics and thresholds appears somewhat arbitrary and insufficiently justified.

  3. Although the appendices contain substantial technical details, some sections may be overly extensive, particularly Appendix F (Structural Properties) and Appendix G (Full Proofs), contribute relatively little to substantiating the paper's central contributions.

Questions for authors

None beyond the concerns raised above.

Reviewer_DUnPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EViC-VL, a training-free method for detecting confident visual hallucinations (CVHs) in vision-language models. The authors construct ConfHall-VL, a COCO-derived diagnostic based on generation confidence, self-uncertainty, and object absence. EViC-VL averages the evidence scores of OWLv2 and Grounding DINO to verify object claims. The ensemble improves AUROC from 0.887 to 0.927 over OWLv2 alone. The paper also evaluates caption editing, standard hallucination benchmarks, and label quality.

Critical strengths

•The paper addresses an important problem: VLMs may confidently generate objects that are not present in the image. I appreciate that the authors separate hallucination detection from actual caption correction, rather than assuming that better detection necessarily leads to better generation.

•The primary evaluation is reasonably well designed, using a separate holdout and paired image-cluster bootstrap. The AUROC improvement over OWLv2 is clear. The editing study also suggests that the ensemble may help avoid unnecessary changes to supported claims.

Critical weaknesses

•I am not convinced that the proposed method provides sufficient novelty or new insight beyond existing hallucination verification approaches. Section 4 essentially averages the confidence scores of two existing object detectors. Using external visual evidence to verify object claims is already well established, and combining multiple visual signals is not new. For example, MARINE (Zhao et al., ICML 2025) already investigates combining multiple vision models for object-grounded generation. GLSim (Park and Li, NeurIPS 2025) also combines complementary visual signals for object-level hallucination detection. Although these methods differ from EViC-VL, the paper should explain more clearly what new knowledge it contributes. Simply showing that averaging two detectors improves AUROC seems insufficient without analyzing why the combination works, which errors are complementary, or whether the gain extends beyond this particular detector pair.

•The practical benefit beyond improved detection AUROC remains unclear. Table 18 does not establish a clear editing advantage over detector revision. Although EViC-VL preserves more supported claims, its CVH recall is lower. In Table 22, CHAIR-S is slightly worse than the base model, while AMBER hallucination and coverage are unchanged at the reported precision. It is therefore difficult to see whether adding another detector actually improves the final VLM output enough to justify the additional cost. Table 5 also compares different detector scores at the same numerical threshold, which may not represent comparable operating points. Comparisons at matched recall or specificity would be more informative. The paper would also benefit from qualitative examples showing cases where the ensemble succeeds but a single detector fails.

•There are several concerns about the CVH definition and evaluation validity. In Section 3.2, CVH requires high confidence, low uncertainty, and object absence. However, a non-CVH claim is not necessarily supported; an absent-object claim with low confidence is still a hallucination. Equation (10) nevertheless uses all (z=0) claims to calculate "supported specificity." This needs clarification. Similarly, the zero recall of the U-only baseline in Table 5 may largely follow from defining CVH positives with (U\leq0.15). The label audit in Appendix Y also raises concerns: the reported precision of 0.918 applies only to detector-triage-retained candidates, whereas the confirmed-absent proportion across both audited candidate strata is 0.831. Furthermore, the post-audit results in Table 25 depend on detector-based triage, and some cross-model labels in Appendix O are derived from ensemble risk. These analyses do not seem sufficient to establish independent label validity or cross-model generalization.

Questions for authors
  1. What is the main contribution beyond combining two existing detectors? How does EViC-VL differ from MARINE and GLSim? Could the authors analyze detector disagreements and compare simple alternative fusion methods to show why averaging is effective?

  2. What practical improvement does EViC-VL provide beyond AUROC? Does the ensemble outperform a single detector at matched recall or specificity? Could the authors show actual image-caption examples where the ensemble makes a better verification or editing decision?

  3. How is the negative class defined in Equation (10)? Does (z=0) include hallucinations that do not satisfy the confidence or uncertainty thresholds? Also, is the U-only baseline's zero recall partly determined by the CVH definition itself?

  4. Could the primary AUROC comparison be repeated using independently human-adjudicated labels? In Appendix Y, why is the triage-retained precision emphasized rather than the result across all audited candidates? Could the authors also replicate the main detector comparison on another VLM without using detector-derived labels?

  5. Could the authors clarify the editing implementation and evaluation? Algorithm 1 does not specify how keep, drop, hedge, replace, and abstain are selected. Table 18 and Appendix U also appear to use inconsistent conventions for abstention. Which rule was used to calculate Edit F1?

  6. Could the authors clarify the CEBC experiment in Section 7.3? Equation (17) measures risk among hallucinations remaining after editing, which may differ from the original CEBC calibration target. Given the large difference in revision rate from the original study, is it appropriate to conclude that the guarantee is broken?

1 comment

Thank you for the detailed responses and revisions. I appreciate the additional analyses, particularly the detector disagreement, fusion comparisons, and matched operating-point results. The responses have clarified several of my concerns, but they also confirmed that some of the original claims were not fully supported. While the revised manuscript now presents a more accurate and appropriately scoped contribution, several issues have been addressed mainly by narrowing the claims rather than providing additional validation. In particular, the methodological novelty remains limited, the practical benefit beyond detection AUROC is still unclear, and independent human-label validation and cross-model replication remain unresolved. I therefore do not think the revisions substantially change my overall assessment of the paper's contribution. I will maintain my original rating of Weak Rejection (2/4). However, if there is any specific evidence or result in the revised manuscript that I may have overlooked and that directly addresses these remaining concerns, I would be happy to reconsider.

Reviewer_KmiBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses confident visual hallucination (CVH), where VLMs assert absent objects with high confidence and low uncertainty. The authors build ConfHall-VL, a COCO-derived diagnostic set, and propose EViC-VL, which flags unsupported object claims by averaging the evidence scores of two frozen open-vocabulary detectors (OWLv2 and Grounding DINO). On a prospective holdout, the ensemble improves AUROC over the OWLv2-only baseline from 0.887 to 0.927.

Critical strengths
  • The motivation is clear and important: hallucinations accompanied by high confidence can bypass uncertainty-based verification.
  • The limitations of prior work are clearly explained.
Critical weaknesses

Poor readability and unclear experimental setup make the paper difficult to understand.

  • Experimental settings are scattered across the main text and multiple appendices, making it hard to grasp the overall setup.
  • Many terms are used without definition or are defined only in the appendix (“junction-fixed build”, “detector-cache column”, “column-native labels”, “case A/B/C”, “mezzanine”, “macro”), which further hinders understanding.

Insufficient experimental validation.

  • There is no performance comparison with state-of-the-art hallucination detection baselines.
  • At a minimum, the method should be compared with token entropy, CLIP score, VLM self-verification, and internal-state probes.

Limited novelty.

  • That averaging different detectors outperforms a single detector is a general ensemble effect.
  • Detector-based verification has already been used in prior work such as Woodpecker and HALC.
Questions for authors

A clearer and more accessible explanation of the problem the paper aims to solve would be helpful.

Comments on the paper

1 comment

I appreciate the authors' efforts to improve the manuscript. However, as my main concerns regarding the paper's organization and experimental clarity remain largely unresolved, I will maintain my original score.

The revised manuscript adds a glossary of internal pipeline terminology at the end of Section 2 (Related Work), but I believe these implementation-specific definitions would be more appropriate elsewhere. Similarly, the "Setup at a glance" summary appears in Section 7 (Results), after the experimental setup and evaluation metrics have already been introduced.

More importantly, understanding the experiments still requires readers to navigate multiple sections and appendices to piece together the model backbones, label construction procedures, evaluation subsets, and relationships between experimental tracks. I believe this reflects a broader organizational issue that cannot be fully resolved by adding definitions or summaries.

Additionally, a performance comparison with state-of-the-art hallucination detection baselines is still missing, making it difficult to assess the proposed method's relative performance and the extent of its improvements over existing approaches.

I have carefully reviewed the authors' responses and revisions, and I appreciate the work they have put into addressing my concerns. If I have missed or misunderstood any of the changes, I would be happy to reconsider those points with further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.