Better Scores Are Not Enough: Evidence-Dependence Evaluation for Video Hallucination

Video hallucination is commonly associated with weak visual and temporal grounding, over-dependence on language priors, and response bias. Yet benchmark scores alone do not establish whether correct responses depend on video evidence. When a mitigation method improves a video hallucination benchmark score, does that gain actually depend on video evidence? We propose EVIDENCE-DEPENDENCE EVALUATION (EDE), an evaluation framework that uses controlled interventions on visual and temporal evidence to measure how benchmark performance and mitigation gains depend on that evidence, while preserving each benchmark’s original evaluation protocol. EDE combines controlled evidence interventions with complementary diagnostics that assess other evaluation factors relevant to interpreting benchmark scores and mitigation gains, requiring neither additional training nor new benchmark-specific annotations. EDE reveals markedly different evidence-dependence profiles: some benchmarks exhibit substantial performance declines when targeted video evidence is disrupted, whereas others remain stable or even increase after such evidence is removed or disrupted. We evaluate four video LLMs on six video hallucination benchmarks and analyze gains from six mitigation methods on selected benchmarks. Complementary diagnostics show that mitigation gains can coincide with error trade-offs, be partly reproduced by response shifts, and vary substantially across evaluation formats, while predictions can change under meaning-preserving question paraphrases. Together, EDE and these diagnostics reveal dependence on video evidence and sensitivity to other factors that benchmark scores and mitigation gains alone do not capture. We therefore argue that video hallucination evaluation should report benchmark scores and mitigation gains alongside their evidence-dependence profiles.