EViCA: Evidence-Guided Visual Claim Adjudication for Mitigating Hallucinations in Large Vision-Language Models

Tool-assisted correction of large vision-language models requires evidence that is relevant to the target claim and sufficient to justify a change. A missing detection alone does not establish that an object is absent, and each revision must remain within the scope justified by its evidence. We introduce EViCA, a training-free framework for evidence-guided visual claim adjudication and selective correction. EViCA associates target claims with object-level observations and their provenance, then applies programmatic checks to assess claim-specific conditions for support and refutation. Unmet conditions guide additional observations within a fixed budget before final adjudication. Task-specific intervention rules govern answer corrections and constrain caption edits to eligible object mentions whose existence claims are refuted. For unresolved or conflicting claims, the controller retains the original answer or object mention. We evaluate EViCA on visual question answering and image captioning, examining correction quality, harmful modifications, and content retention.