MODEL-SUBSTITUTION RISK CONTROL: VERIFIED REPAIR OF VISUAL CLAIMS

Vision–language models can generate captions and answers that contain details not supported by the image. Removing such details reduces hallucination, but it can also discard information that could be recovered through a correct replacement. We introduce Model-Substitution Risk Control (MSRC), a training-free method that verifies visual claims and attempts to repair unsupported object claims before deleting them. MSRC retains supported claims and uses the frozen model to propose replacements for failed claims. Each replacement must pass fixed admissibility checks and independent visual verification before being accepted; otherwise, the unsupported claim is removed. The method follows a common claimlevel repair process across image captioning and visual question answering, with task-specific handling of short answers. To balance hallucination reduction and information retention, MSRC selects its verification thresholds using held-out data and measures remaining errors relative to the original claim opportunities. Openvocabulary replacements are assessed using benchmark annotations, lexical relations, and human-audited error estimates when automatic labels are unavailable. We evaluate MSRC across captioning, visual question answering, and hallucination benchmarks, reporting both claim-level risk and final-response quality.