Don't Grade Your Own Exam: Iterative External-Detector Verification and Rewind Decoding for LVLM Hallucination

Large vision-language models (LVLMs) often generate fluent responses that are not fully grounded in visual evidence. Most existing hallucination-mitigation methods rely on the generator's own internal representations to revise its predictions, leaving the correction process exposed to the same biases that caused the original error. External vision models offer a complementary source of evidence that can verify generated claims independently of the generator. Building on this idea, we introduce SIEVE, a training-free decoding framework that performs selective, iterative verification during generation. Whenever the model produces a verifiable claim about the image, an external verifier evaluates that claim against the visual evidence. If the claim is rejected, SIEVE rolls generation back to the rejected word, suppresses it at the same decoding position, and resumes from the model's next-best continuation. The regenerated claim is verified again, allowing correction to continue until the output is supported by the evidence rather than simply deleting unsupported content. We instantiate SIEVE for object hallucination, where established benchmarks and reliable external verifiers are readily available, using an object detector as the verifier. Across LLaVA-1.5, InstructBLIP, and Qwen2.5-VL, SIEVE reduces object hallucinations by 42-80% while preserving caption length and fluency.