From Diagnosis to Mitigation: Benchmarking and Routed Mitigation for Visual and Knowledge Hallucinations in MLLMs

Multimodal large language models (MLLMs) perform tasks that differ in the primary source of information required for correct answers. Some tasks are driven by directly verifiable visual evidence, while others rely more on knowledge and reasoning. We denote them as visual-dominant and knowledge-dominant tasks. Hallucination arises when an MLLM fails to properly balance these two sources of information. To diagnose and evaluate visual and knowledge hallucinations, we construct a new hallucination benchmark, VKHalluciBench. Compared with existing benchmarks, VKHalluciBench adopts a novel taxonomy, provides broad coverage across visual- and knowledge-dominant tasks, and incorporates carefully designed open-ended questions and counter-intuition samples that are prone to induce hallucinations. After a comprehensive evaluation, we observe that most mitigation strategies face a common dilemma, which manifests as a trade-off between mitigating visual hallucinations and knowledge hallucinations. To address this limitation, we further propose a plug-and-play routing-based mitigation framework to enable adaptive hallucination mitigation. Specifically, we design a Diagnostic Signal Probe to provide diagnostic results, and an Adaptive Hallucination Mitigation Router then selects suitable mitigation strategies. Experimental results demonstrate that our method achieves the best performance on both visual and knowledge tasks and introduces only 0.33M trainable parameters, resulting in an overall improvement of 12.1% over the baseline model.