- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper revisits object hallucination evaluation in modern LVLMs. It replaces the rule-based object extraction in CHAIR with an LLM-based judge, re-evaluates several recent LVLMs and three hallucination mitigation methods, and further uses BEAF with a persistence metric to study hallucination under counterfactual object removal.
Critical strengths
-
The paper provides a controlled re-evaluation of CHAIR by changing only the object-extraction component, which allows direct comparison with prior CHAIR-based results.
-
It evaluates multiple recent LVLM families rather than focusing mainly on LLaVA-1.5, and tests several representative mitigation methods under the same setup.
-
The paper extends evaluation beyond standard COCO images by using counterfactual object removal in BEAF and introducing persistence to measure whether models continue to mention removed objects.
Critical weaknesses
Concerns
- Limited novelty in revisiting CHAIR and MS-COCO-based evaluation. I am personally very interested in the problem studied in this paper, as I have also been concerned for some time about whether the standard object-hallucination evaluation protocol still reflects the behavior of modern LVLMs. I therefore find the paper’s attempt to revisit existing evaluation practices valuable. However, the concerns raised about CHAIR and MS COCO have already been recognized by several recent works. For example, ALOHa explicitly points out that CHAIR is restricted to a fixed set of MS COCO categories and synonyms, and replaces this closed-vocabulary matching process with an LLM-based, open-vocabulary object extraction and matching framework [1]. THRONE similarly studies hallucination in free-form LVLM generations and uses language models to evaluate object hallucinations beyond conventional CHAIR-style matching [2]. The current paper acknowledges these works, but the distinction between judge-CHAIR and these newer evaluation frameworks is relatively limited: judge-CHAIR mainly replaces rule-based object extraction while intentionally retaining the rest of the CHAIR pipeline. I therefore think the paper needs a much clearer comparison with these recent evaluation frameworks, both conceptually and empirically, to establish what new understanding judge-CHAIR provides beyond what has already been identified.
- The selected hallucination-mitigation baselines are not sufficiently representative of recent methods. The paper evaluates VCD, PAI, and LocoRE, and uses their limited generalization to argue that gains obtained on LLaVA-1.5 do not reliably transfer to modern LVLMs. While this is useful evidence for these particular methods, the choice of baselines feels somewhat dated relative to the current literature. A number of more recent approaches explicitly target stronger or more diverse LVLM families and claim cross-model or cross-architecture generalization, including Nullu [3], ICT [4], AGLA [5], TruthPrInt [6], AIR [7], and RUDDER [8]. For instance, RUDDER reports results across LLaVA-1.5, Idefics2, InstructBLIP, and Qwen2.5-VL, rather than being evaluated only on older LLaVA variants [8]. It would therefore be more informative to test whether such recent methods also fail under the paper’s revised evaluation protocol. Otherwise, the current results mainly establish that several earlier mitigation methods do not transfer well, rather than showing that the broader modern mitigation literature systematically overclaims generalization.
- The proposed evaluation setup does not fully address the more fundamental limitations of hallucination evaluation. More fundamentally, I am not convinced that replacing rule-based extraction with an LLM judge, or repurposing BEAF with the persistence metric, fully resolves the limitations that motivate this paper. Judge-CHAIR still inherits the 80 MS COCO object categories and incomplete COCO annotations, and objects outside these categories remain unscored. BEAF provides a useful counterfactual setup by removing objects and measuring whether models continue to claim their presence, but this general idea of evaluating hallucination through before/after object removal has itself already appeared in prior work [9,10]. More importantly, LVLM outputs are changing substantially as models improve: newer models generate longer and richer descriptions, use more specific visual concepts, and may refer to objects at a different semantic granularity from the fixed COCO taxonomy. A model that correctly says “paraglider,” for example, should not need to collapse its prediction into a coarse legacy category simply because the benchmark ontology does so. Similarly, longer outputs naturally create more opportunities for hallucinated statements, making sentence-level hallucination rates difficult to compare directly across models with very different response styles. In my view, the central challenge is therefore not only better object extraction, but designing an evaluation framework that remains meaningful as model vocabularies, decoding behavior, response length, and descriptive granularity continue to evolve. The current work identifies part of this problem, but its proposed evaluation framework still appears closely tied to the assumptions of CHAIR and MS COCO.
References
[1] Petryk S, Chan DM, Kachinthaya A, et al. ALOHa: A New Measure for Hallucination in Captioning Models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2024:342-357. doi:10.18653/v1/2024.naacl-short.30.
[2] Kaul P, Li Z, Yang H, et al. THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024:27228-27238.
[3] Yang L, Zheng Z, Chen B, et al. Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025:14635-14645.
[4] Chen J, Zhang T, Huang S, et al. ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025:4209-4221.
[5] An W, Tian F, Leng S, et al. Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025:29915-29926.
[6] Duan J, Kong F, Cheng H, et al. TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025:7372-7382.
[7] Sun H, Li Q, Wang P, Zhang M. Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Findings. 2026:8930-8940.
[8] Zou Z, Gao Y, Guan J, Li B, Marttinen P. Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision-Language Models. In: Proceedings of the 43rd International Conference on Machine Learning. Vol 306. PMLR; 2026:168675-168694.
[9] Moon YB, Nam HW, Choi W, Oh TH. BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models. In: European Conference on Computer Vision. 2024.
[10] He Y, Sun H, Ren P, et al. Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects? In: Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics. 2025.
Questions for authors
See weakness above.