acceptodds
Withdrawn from ICLR 2027

Is Object Hallucination in LVLMs Overestimated? Revisiting Evaluation, Models, and Mitigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) have become remarkably capable of describing images at an impressive level of detail. However, object hallucinations have been a long-standing problem and a wide variety of mitigation strategies have been proposed and evaluated. Interestingly, most methods including very recent work rely on one particular metric—the CHAIR metric on COCO—and often report results primarily for LLaVA1.5-7B model. In this work, we take a step back by carefully designing an evaluation to understand if the CHAIR metric on COCO is actually measuring object hallucinations properly and how significant they are for modern models. In particular, we evaluate 10 models (7B to 12B) from 5 different families, and find that measured hallucination rates significantly fall (16 to 38%) from the original CHAIR value for every model except LLaVA-1.5-7B. Since major mitigation methods, however, are still developed and tested on LLaVA-1.5 with this metric, we revisit them on current models and find that none of them reduce hallucination consistently across models on COCO: a large gain on LLaVA-1.5 does not carry over or results in modest gains on newer models. We also perform finer-grained analyses that go beyond CHAIR. Finally, to evaluate models on more challenging images, we repurpose the existing BEAF dataset and introduce the persistence metric to assess models and hallucination mitigation methods under counterfactual object removal. Taken together, these metrics help better understand the prevalence of hallucinations in modern LVLMs and the effectiveness of mitigation methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
COCO, vision-language models, multimodal large language models, object hallucination, hallucination evaluation, CHAIR, hallucination mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UXn4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits object hallucination evaluation in modern LVLMs. It replaces the rule-based object extraction in CHAIR with an LLM-based judge, re-evaluates several recent LVLMs and three hallucination mitigation methods, and further uses BEAF with a persistence metric to study hallucination under counterfactual object removal.

Critical strengths
  1. The paper provides a controlled re-evaluation of CHAIR by changing only the object-extraction component, which allows direct comparison with prior CHAIR-based results.

  2. It evaluates multiple recent LVLM families rather than focusing mainly on LLaVA-1.5, and tests several representative mitigation methods under the same setup.

  3. The paper extends evaluation beyond standard COCO images by using counterfactual object removal in BEAF and introducing persistence to measure whether models continue to mention removed objects.

Critical weaknesses

Concerns

  1. Limited novelty in revisiting CHAIR and MS-COCO-based evaluation. I am personally very interested in the problem studied in this paper, as I have also been concerned for some time about whether the standard object-hallucination evaluation protocol still reflects the behavior of modern LVLMs. I therefore find the paper’s attempt to revisit existing evaluation practices valuable. However, the concerns raised about CHAIR and MS COCO have already been recognized by several recent works. For example, ALOHa explicitly points out that CHAIR is restricted to a fixed set of MS COCO categories and synonyms, and replaces this closed-vocabulary matching process with an LLM-based, open-vocabulary object extraction and matching framework [1]. THRONE similarly studies hallucination in free-form LVLM generations and uses language models to evaluate object hallucinations beyond conventional CHAIR-style matching [2]. The current paper acknowledges these works, but the distinction between judge-CHAIR and these newer evaluation frameworks is relatively limited: judge-CHAIR mainly replaces rule-based object extraction while intentionally retaining the rest of the CHAIR pipeline. I therefore think the paper needs a much clearer comparison with these recent evaluation frameworks, both conceptually and empirically, to establish what new understanding judge-CHAIR provides beyond what has already been identified.
  2. The selected hallucination-mitigation baselines are not sufficiently representative of recent methods. The paper evaluates VCD, PAI, and LocoRE, and uses their limited generalization to argue that gains obtained on LLaVA-1.5 do not reliably transfer to modern LVLMs. While this is useful evidence for these particular methods, the choice of baselines feels somewhat dated relative to the current literature. A number of more recent approaches explicitly target stronger or more diverse LVLM families and claim cross-model or cross-architecture generalization, including Nullu [3], ICT [4], AGLA [5], TruthPrInt [6], AIR [7], and RUDDER [8]. For instance, RUDDER reports results across LLaVA-1.5, Idefics2, InstructBLIP, and Qwen2.5-VL, rather than being evaluated only on older LLaVA variants [8]. It would therefore be more informative to test whether such recent methods also fail under the paper’s revised evaluation protocol. Otherwise, the current results mainly establish that several earlier mitigation methods do not transfer well, rather than showing that the broader modern mitigation literature systematically overclaims generalization.
  3. The proposed evaluation setup does not fully address the more fundamental limitations of hallucination evaluation. More fundamentally, I am not convinced that replacing rule-based extraction with an LLM judge, or repurposing BEAF with the persistence metric, fully resolves the limitations that motivate this paper. Judge-CHAIR still inherits the 80 MS COCO object categories and incomplete COCO annotations, and objects outside these categories remain unscored. BEAF provides a useful counterfactual setup by removing objects and measuring whether models continue to claim their presence, but this general idea of evaluating hallucination through before/after object removal has itself already appeared in prior work [9,10]. More importantly, LVLM outputs are changing substantially as models improve: newer models generate longer and richer descriptions, use more specific visual concepts, and may refer to objects at a different semantic granularity from the fixed COCO taxonomy. A model that correctly says “paraglider,” for example, should not need to collapse its prediction into a coarse legacy category simply because the benchmark ontology does so. Similarly, longer outputs naturally create more opportunities for hallucinated statements, making sentence-level hallucination rates difficult to compare directly across models with very different response styles. In my view, the central challenge is therefore not only better object extraction, but designing an evaluation framework that remains meaningful as model vocabularies, decoding behavior, response length, and descriptive granularity continue to evolve. The current work identifies part of this problem, but its proposed evaluation framework still appears closely tied to the assumptions of CHAIR and MS COCO.

References

[1] Petryk S, Chan DM, Kachinthaya A, et al. ALOHa: A New Measure for Hallucination in Captioning Models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2024:342-357. doi:10.18653/v1/2024.naacl-short.30.

[2] Kaul P, Li Z, Yang H, et al. THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024:27228-27238.

[3] Yang L, Zheng Z, Chen B, et al. Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025:14635-14645.

[4] Chen J, Zhang T, Huang S, et al. ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025:4209-4221.

[5] An W, Tian F, Leng S, et al. Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025:29915-29926.

[6] Duan J, Kong F, Cheng H, et al. TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025:7372-7382.

[7] Sun H, Li Q, Wang P, Zhang M. Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Findings. 2026:8930-8940.

[8] Zou Z, Gao Y, Guan J, Li B, Marttinen P. Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision-Language Models. In: Proceedings of the 43rd International Conference on Machine Learning. Vol 306. PMLR; 2026:168675-168694.

[9] Moon YB, Nam HW, Choi W, Oh TH. BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models. In: European Conference on Computer Vision. 2024.

[10] He Y, Sun H, Ren P, et al. Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects? In: Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics. 2025.

Questions for authors

See weakness above.

Reviewer_3DvNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that rule-based CHAIR overestimates object hallucination in modern LVLMs. It proposes judge-CHAIR, which replaces CHAIR's word matching with an LLM judge, and shows that hallucination rates drop substantially for nine recent models but barely change for LLaVA-1.5-7B. Using this metric and a repurposed BEAF benchmark with a new persistence metric, the authors re-evaluate VCD, PAI, and LocoRE and find that none reduces hallucination consistently across models.

Critical strengths
  1. Timely and practically relevant re-examination of a standard evaluation setup. Many hallucination mitigation methods are still developed and reported mainly on LLaVA-1.5 with rule-based CHAIR. The paper shows that this setup can misjudge both how much modern models hallucinate and how much mitigation methods help. If the findings hold, they matter for how future methods are evaluated, which is the main reason I see value in the paper.

  2. Controlled design with careful statistical analysis. Judge-CHAIR changes only the object extraction step, so its numbers remain directly comparable with prior CHAIR results. The re-evaluation also uses held-out hyperparameter tuning, exact McNemar tests, and a fix/break decomposition. Together these make the negative results on VCD, PAI, and LocoRE more convincing than a simple comparison of aggregate scores.

Critical weaknesses
  1. Limited and partly circular human validation. Judge-CHAIR is validated against a single annotator, who is also an author, on only 100 captions from two models. The human annotation guidelines closely mirror the judge prompt, so high agreement may reflect adherence to the same predefined rules rather than independent validation of their correctness. Moreover, agreement is reported only at the binary CHAIRs level, potentially masking object-level extraction errors, and no inter-annotator reliability can be assessed. Since the paper's central COCO-based conclusions depend on judge-CHAIR, this limited validation substantially weakens confidence in the reported hallucination reductions.

  2. Ground-truth leakage into object extraction. The judge receives each image's COCO ground-truth classes, ostensibly to resolve ambiguous references such as "the animal." However, this makes object extraction dependent on the same labels subsequently used to determine hallucination. Ambiguous mentions may be preferentially mapped to ground-truth classes, potentially biasing hallucination rates downward. Without an ablation withholding the ground-truth list, it is unclear whether the reported reductions reflect more accurate linguistic extraction or ground-truth-conditioned disambiguation.

Questions for authors
  1. Human validation of judge-CHAIR. Table 1 reports only caption-level CHAIRs agreement with a single annotator. Can the authors provide object-level extraction precision, recall, and F1 against human annotations, particularly for cases where rule-CHAIR and judge-CHAIR disagree? Could the authors also clarify how the human annotation guidelines were developed relative to the judge prompt? Independent annotation results would materially strengthen confidence that the reported reductions reflect genuine corrections rather than shared annotation conventions.

  2. Effect of ground-truth conditioning. Can the authors report judge-CHAIR results when the COCO ground-truth class list is withheld from the judge, keeping everything else unchanged? In particular, how much do the CHAIRs and CHAIRi scores in Table 2 change, and what fraction of the differences arise from ambiguous class mappings? This would help establish whether the reported reductions are robust to removing ground-truth information from the extraction process.

Reviewer_pDbUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the standard practice of evaluating object hallucination with rule-based CHAIR on COCO, often using LLaVA-1.5-7B as the sole or primary model. The authors introduce judge-CHAIR, which replaces CHAIR's synonym matching with an LLM that extracts object mentions from a caption without seeing the image. Across ten LVLMs, judge-CHAIR lowers CHAIRs by 16.1–38.3 points for recent models, while changing the LLaVA-1.5 result much less. The paper then reevaluates VCD, PAI, and LocoRE, and repurposes BEAF for counterfactual object removal using a persistence metric that asks whether a removed object continues to be claimed.

Critical strengths
  1. The paper challenges an evaluation convention with a large and interpretable effect. In Table 2, Qwen3.5 drops from 55.6% rule-CHAIRs to 17.3% judge-CHAIRs, and Qwen3-VL drops from 55.2% to 18.2%. Figure 5 traces many disagreements to concrete lexical failures such as “orange” used as a color and broad mappings to “chair.” This makes the critique testable rather than anecdotal.
  2. The mitigation analysis looks beyond aggregate improvement. The paper separates captions that a method fixes from previously correct captions that it breaks, which helps explain why an intervention can help an older, high-hallucination model but fail on newer models with a lower base rate. This is a useful perspective for future evaluations.
Critical weaknesses
  1. The validation of judge-CHAIR is too limited for the paper's central claim. Section 3.2 and Appendix A.1 state that the judge does not see the image, but it does see the image's COCO ground-truth classes and may use them to resolve references such as “the animal.” This is not a purely text-only replacement for the rule extractor: the labels can steer ambiguous mentions toward an annotated class and may make the judge more conservative about classes absent from the list. No label-blind ablation is reported. Moreover, Table 1 validates the metric on only 100 captions—50 each from two models—annotated by a single author. It reports sentence-level verdict agreement but not mention- or class-level precision and recall, uncertainty, inter-annotator agreement, or validation on the other eight models and the BEAF judge. Since the headline conclusion depends on this measurement procedure, internal consistency at this scale is not sufficient.
  2. Some statistical and BEAF comparisons are less stable than the prose suggests. The appendices contain many method-by-model tests, but the paper uses uncorrected p<0.05 significance statements; several marginal results may not survive a Holm or FDR correction. The large rule-versus-judge differences in Table 2 are repeatedly called significant without corresponding intervals or tests. For BEAF, persistence is computed only on units where each method still mentions the object in the original image. As a result, the denominator varies across methods—n ranges from 176 to 239 in Table 4—and a method can change its evaluation set by omitting the object before the edit. The authors correctly avoid treating the hard subset as a prevalence estimate, but the changing comparison set still complicates method rankings.
Questions for authors
  1. What happens to Tables 2 and 3 when the judge is not shown the COCO ground-truth labels? Please report paired label-visible and label-blind results rather than only aggregate scores.
  2. Can the metric be validated by at least two independent, non-author annotators on a larger sample spanning more model families, caption lengths, and BEAF outputs, with mention-, class-, and sentence-level metrics and confidence intervals?
  3. After defining test families and applying Holm or FDR correction, which method-by-model improvements remain statistically significant? Can the main effects also be reported with confidence intervals?
  4. Do the BEAF rankings persist when eligibility is fixed using the vanilla model, or when all 250 units are scored with a joint measure that penalizes both failure to mention the object in the original and persistence after removal? Minor Comments
  5. On p. 2, “if CHAIR measure object hallucination correctly” should read “if CHAIR measures object hallucination correctly.”
  6. On p. 2, “their gains on LLaVA-1.5 does not always carry over” should use “do,” not “does.”
  7. On p. 2, “some of the widely believed hypothesis” should use the plural “hypotheses.”
  8. On p. 3, “VCD ... assume hallucination happens” should read “VCD ... assumes hallucination happens.”
  9. On p. 3, “‘orange’ the the fruit” contains a duplicated “the”; “‘orange’ as the fruit” would be clearer.
  10. On p. 4, “Section A.1 and Table 5 shows...” should read “Section A.1 and Table 5 show....”
  11. The Table 2 caption on p. 5 contains the awkward repetition “judge-CHAIR on vanilla on the 500-image...”; please revise it.
  12. On p. 5, “a recent approahc” should be “a recent approach.”
  13. On p. 6, “the exact McNemar teston the captions” is missing a space between “test” and “on.”
  14. On p. 6, “CHAIR scores only shows the net change” should read “CHAIR scores only show the net change.”
  15. In the AI Use Statement on p. 10, “AI coding agents ... was checked” has an agreement and antecedent problem. One possible revision is: “AI coding agents were used to help implement the code ..., and the code was checked for plausibility.”
  16. Also on p. 10, “AI tools were used creating and refining figures” should read “AI tools were used to create and refine figures.”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.