acceptodds
Under review as a conference paper at ICLR 2027

EViCA: Evidence-Guided Visual Claim Adjudication for Mitigating Hallucinations in Large Vision-Language Models

Abstract

Tool-assisted correction of large vision-language models requires evidence that is relevant to the target claim and sufficient to justify a change. A missing detection alone does not establish that an object is absent, and each revision must remain within the scope justified by its evidence. We introduce EViCA, a training-free framework for evidence-guided visual claim adjudication and selective correction. EViCA associates target claims with object-level observations and their provenance, then applies programmatic checks to assess claim-specific conditions for support and refutation. Unmet conditions guide additional observations within a fixed budget before final adjudication. Task-specific intervention rules govern answer corrections and constrain caption edits to eligible object mentions whose existence claims are refuted. For unresolved or conflicting claims, the controller retains the original answer or object mention. We evaluate EViCA on visual question answering and image captioning, examining correction quality, harmful modifications, and content retention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Hallucination Mitigation, Visual Grounding, Claim Verification, Selective Correction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mc3vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a training-free framework named EViCA for adjudicating visual claims and selectively correcting LVLM outputs. It converts answers or object mentions into claims, gathers SAM 3 observations with provenance and instance associations, and judges support and refutation separately. A claim may therefore be supported, refuted, unresolved, or conflicting, and only eligible decisions are allowed to change a binary answer or remove a caption span. The paper motivates this design through three concrete failure modes, namely target mismatch, treating an empty detection as proof of absence, and deleting valid content while fixing a false mention. Evaluation is conducted on POPE, MME-Hallucination, and COCO captioning with two LVLM backbones.

Critical strengths
  1. The paper identifies specific decision errors rather than treating hallucination correction as a single black-box problem. In Table 1, target correspondence reduces harmful changes from 20 to 4, and combining it with preservation under empty evidence raises net correction from 14 to 31 while eliminating harmful changes in that diagnostic. This is direct evidence for the central claim that a missing detection should not automatically be interpreted as absence.
  2. The ablations separate the contribution of acquisition from that of adjudication. On the same acquired observations, Full avoids 204 harmful changes relative to Score only and improves accuracy by about 0.50 points; relative to Initial only, it gains about 0.69 points. The image-clustered bootstrap intervals make these comparisons more informative than a bare benchmark score.
Critical weaknesses
  1. The reported MME result is affected by test-set model selection. Section 5.1 and Appendix C.2 state that all 240 MME questions are used both to select the count qualification threshold and to report final performance. Table 9 shows that this choice moves the total from 730 to 740 for Qwen and from 678.33 to 695 for LLaVA. Consequently, the headline MME scores in Tables 2 and 3 are not results on an untouched test set. This matters here because thresholded, programmatic decisions are a central part of the method rather than a peripheral implementation detail.
  2. The main comparison does not control for method class, compute, or retained content. EViCA uses SAM 3 and several rounds of post-processing, but is compared mainly with generation-time methods such as VCD, M3ID, SID, and DiVE. There is no controlled comparison with Woodpecker, Pelican, or a budgeted evidence-acquisition method using the same backbone and observation budget. EViCA also costs 21,806 MB and 77.7 ms/token versus 17,568 MB and 33.4 ms/token for the base model (Table 21). On captioning, lower CHAIR comes with recall drops from 71.70 to 65.87 for Qwen and from 80.50 to 75.82 for LLaVA. Without a broader quality assessment, some of the gain may reflect more aggressive deletion rather than a uniformly better caption. Finally, the reproducibility statement says that the core EViCA implementation will remain closed, even though route checks, priorities, query normalization, and deletion eligibility are crucial to the result.
Questions for authors
  1. Can the authors select every POPE and MME threshold on a separate validation set and report MME results, with confidence intervals, on data that played no role in model selection?
  2. How does EViCA compare with Woodpecker, Pelican, and at least one budgeted acquisition method when the backbone, SAM 3 observations, number of tool calls, and latency budget are matched?
  3. Will the authors release the core implementation? If not, can they release all prompts, route conditions, action priorities, calibration splits, random seeds, and per-example adjudication records needed to reproduce the system?
  4. Does the captioning advantage remain under a paired human evaluation that jointly considers factuality, completeness, and fluency rather than CHAIR alone?
  5. On p. 9, the Figure 3 caption says that “stars mark the selected thresholds,” but no star markers are visible in panels (b) or (c). Please add the markers or remove this sentence.
  6. In the reference title on p. 11, “adaptive” in “Global context or local detail? adaptive visual grounding...” should be capitalized.
  7. The reference title on p. 13 ends with “vision-language model”; the published title uses the plural “vision-language models.”
  8. Figures 1 and 5 use “SAM3,” whereas the main text and Figure 2 use “SAM 3.” Please use one form consistently.
Reviewer_tVLcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free, post-hoc correction framework for LVLM hallucination. It turns answers and caption mentions into claims, gathers SAM 3 evidence, and judges support and refutation separately. This gives four states: Supported, Refuted, Unknown and Conflict. The original output is changed only when the evidence is relevant and sufficient. On POPE, MME and CHAIR it beats decoding-based baselines on two 7B backbones.

Critical strengths
  • The three failure modes (target mismatch, premature negation, over-editing) are clear and well motivated. Table 1 shows how each one maps to a design choice.
  • Treating “no detection” as Unknown rather than “absent” is a principled idea, and the “preserve when unsure” default is sensible.
  • The ablations are careful, with paired bootstrap intervals. Score-only vs. Full on identical evidence is a good control.
  • The paper reports harmful changes (H) alongside gains, which is rare and useful and cost is disclosed.
Critical weaknesses
  • Recent baselines are missing. The comparison covers only decoding-based methods, and recent baselines are absent, for example CRoPS. The post-hoc correctors that are the closest competitors (Woodpecker, LURE, Pelican) are discussed in Section 2 but never run in the tables. ONLY is listed as a baseline but is missing from Table 2 and Figure 4.

  • Outdated and narrow model set. Only Qwen2.5-VL-7B and LLaVA-1.5-7B are evaluated. Please add current open models (e.g., Qwen3-VL, InternVL3, Gemma 3, LLaVA-OneVision) and at least one larger or closed model. Stronger models hallucinate less, so the gains may shrink.

  • The gains may come mostly from SAM 3, not the adjudication logic. "Initial only" (one SAM 3 query plus the judge) already reaches 89.58 accuracy against about 84 for Base. Adjudication and acquisition add only about +0.5 and +0.7 points. Baselines use no external detector. Please add a plain "SAM 3 only" baseline and give Woodpecker/Pelican the same detector.

  • Premature negation is reduced, not solved. Table 15 shows 515 beneficial and 355 harmful Yes→No changes, so about 41% of that direction's changes are harmful. Table 16 reports 12,663 Refuted claims on a balanced held-out set with about 12,159 negatives, so at least 504 present objects were declared absent. This weakens the claim that the validity checks make absence decisions reliable.

  • Evaluation scope and calibration.

    • The MME count threshold is selected on the same 240 questions used for evaluation.
    • POPE is existence-only, which matches the method's design closely. Please add AMBER, MMHal-Bench or Object HalBench.
    • Captioning uses only 500 images, and object recall drops by about 6 points on Qwen, so part of the CHAIR gain comes from saying less. Please add a content-aware metric (e.g., FaithScore).
    • It is unclear whether baseline CHAIR numbers use the same three repeats, given that EViCA's repeats differ by about 2.4 points in CHAIR_S.
  • Cost reporting is incomplete. Latency is measured per token on captioning only, so per-question cost on POPE is unknown. EViCA's generation stage (30.0 ms/token) is also lower than the Base latency (33.4). The cost is about 2.3× Base latency plus extra GPU memory.

Questions for authors
  • How does EViCA behave when SAM 3 fails on small or occluded objects?
  • Why does Fixed schedule match Full?
  • What is the per-question POPE latency, and how does it compare with Woodpecker and Pelican?
Reviewer_qKivOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EViCA, a training-free framework for correcting hallucinations in large vision-language models. It uses SAM 3 to collect visual evidence, checks whether that evidence supports or refutes a specific claim, and selectively corrects the answer when sufficient evidence is available. The method also supports additional evidence acquisition and constrained caption editing to avoid removing unrelated information.

Critical strengths
  1. I like the distinction between missing evidence and evidence that actually refutes a claim. This is an important issue in tool-assisted correction, since a failed detection should not automatically lead to changing a correct answer.

  2. The experimental evaluation is fairly comprehensive. The authors test two VLM backbones across several benchmarks and report not only accuracy improvements but also beneficial and harmful corrections. The component ablations are particularly useful for understanding when additional evidence helps and when it introduces new errors.

Critical weaknesses
  1. I think the main weakness is that it is still hard to tell how much of the improvement actually comes from EViCA's proposed adjudication framework. Most of the baselines are decoding-based methods, while EViCA uses an additional SAM 3 model to verify visual claims. The paper discusses more closely related tool-assisted methods, such as Woodpecker and Pelican, but does not compare against them. Also, in Table 4, the full method only improves accuracy by 0.69 points over Initial only and 0.50 points over Score only. The fixed-schedule baseline even performs slightly better (90.30 vs. 90.27), although it uses more observations. I understand that reducing the number of tool calls is useful, but these results make me question how much the more complicated evidence acquisition and adjudication process contributes. A stronger comparison with simpler SAM 3-based correction methods would help clarify this.

  2. Another concern is the trade-off between reducing hallucinations and preserving correct information. The paper emphasizes that EViCA avoids unnecessary edits, but the captioning results show a noticeable drop in object recall: from 71.70% to 65.87% on Qwen and from 80.50% to 75.82% on LLaVA. The edited captions are also shorter. I understand that removing hallucinated objects can naturally reduce caption length, but it is not clear how much of the recall drop comes from removing correct object mentions. Since preserving useful information is one of the main motivations of the paper, I think this deserves more analysis. The controlled editing experiment is helpful, but I would like to see a breakdown of correct and incorrect deletions on naturally generated captions.

  3. EViCA is considerably more expensive than the baseline, with latency increasing from 33.4 to 77.7 ms/token and higher GPU memory usage. This overhead may be reasonable for some applications, but the accuracy improvements from the more sophisticated acquisition and adjudication steps are relatively small.

Questions for authors
  1. Appendix C.2 mentions that the MME count threshold is selected on the same 240 questions used for evaluation. Could the authors clarify this protocol and whether the results hold when the threshold is selected on separate calibration data?

  2. How well does the instance-consistent adjudication work on compositional claims involving multiple similar objects? Is there a dedicated experiment that isolates this component?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.