acceptodds
Withdrawn from ICLR 2027

Same Score, Different Blind Spots: What Object-Hallucination Benchmarks Can and Cannot Certify

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Object-hallucination benchmarks aggregate heterogeneous queries into pooled scores that do not reveal how false-positive risk varies across objects. We present an object-conditioned audit that assigns violation, unresolved, or certified verdicts and assesses whether benchmark allocation supports simultaneous object-wise certification. We evaluate eight open VLM checkpoints on POPE and DASH-B at a nominal 10% pooled false-positive budget. Kimi-VL and Molmo 2-O differ by 0.004 in pooled AUROC yet make 20 versus 49 false assertions on the same 147 POPE-popular cup negatives. The pair-level difference survives a selection-aware permutation test and persists under the models' native decisions. Using split-half held-out calibration and simultaneous exact bounds, we identify violations in 20 of 29 model benchmark cells in each direction, with 15 of 29 violating in both. A conservative calibration sensitivity analysis retains violations in a majority of cells in both directions. In the descriptive full-data census, no cell earns a simultaneous certificate covering all eligible objects. Under this convention, sparse allocation can preclude certification even with zero observed errors. An Open Images positive control on a checkpoint subset yields class-level certificates under frozen labels. A pre-specified readout gate voids Gemma 4's POPE cells, showing that statistical resolution alone does not establish valid model attribution. We recommend object-wise counts and uncertainty bounds, explicit unresolved outcomes, and validity checks alongside pooled scores.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Model evaluation, Object Hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FjzrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper examines object-hallucination benchmarks and shows evidence that aggregated evaluation metric such as FPR and ROC-AUC can hide detailed discrepancies at object class level. The detailed audit process reveals multiple such cases; open-image based augmentation shows that certification is possible with improved benchmark. The conclusion encourages the benchmark generators to provide detailed information about object group, risk budget, and confidence level for certification.

Critical strengths
  • The paper presents clear evidence on how object-hallucination benchmarks for VLMs can assign models similar aggregated scores when they show discrepant performance over specific object classes. The guides for certification is a practical and useful one.
Critical weaknesses
  • In the end, I think there is nothing surprising about the detailed results. Benchmarks are by definition bound to hide details when interpreted through aggregated evaluation metrics such as ROC-AUC. I think object class level variances and discrepancies are simply expected. I appreciate the actual experimentation and evidence, but the principle itself is not new.
Questions for authors
  • What is the core take-home message here? The core finding itself is hardly surprising, unless we suspect a systematic, shared root cause (I don't think there is one, this is just models with varying performances). That leaves pretty standard guideline for benchmark construction and/or certification practice.
Reviewer_ZouxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what object-hallucination benchmarks can establish beyond pooled performance metrics. It audits eight VLM checkpoints on two benchmarks using object-conditioned FPR. It shows that near-tied pooled AUROC can still conceal substantial object-specific differences. Authors introduce BCC, a count-only diagnostic showing whether benchmark allocation permits certification even with zero observed errors. An Open Images positive control demonstrates class-level certification when allocation is sufficient. It also illustrates why statistically detected discrepancies cannot automatically be attributed to model hallucination by using reference-label and readout validity checks.

Critical strengths
  1. The paper addresses a clear gap in aggregate evaluation. The comparison between models with nearly identical AUROC but substantially different error rates on the same object makes the problem concrete.
  2. BCC provides a practical way to assess benchmark design. It identifies whether the available samples would permit certification even if a model made no errors. This helps separate limitations of the benchmark from limitations of the model and gives benchmark designers a concrete sample-allocation criterion.
  3. The paper checks whether measured errors can be attributed to model hallucination. The Gemma 4 analysis is a useful example: rather than taking the initial violation at face value, the authors examine whether the first-token readout reflects the model’s response behavior and exclude the affected results when the pre-specified checks fail.
Critical weaknesses
  1. The methodological contribution is limited and depends on the certification procedure. BCC is a useful diagnostic, but it characterizes sample sufficiency under the chosen procedure rather than establishing a procedure-independent limit. Although the paper acknowledges this distinction, its contribution beyond existing subgroup auditing could be stated more clearly.
  2. The statistical guarantees rely on within-object sampling assumptions that are difficult to verify. Distinct images eliminate duplicate observations but do not establish independence. Shared scenes or semantic patterns may introduce dependence, particularly in a mined stress-test dataset such as DASH-B. Further sensitivity analysis would help assess how such dependence affects the coverage of the reported intervals and the resulting verdicts.
  3. The validity checks identify measurement problems without fully resolving them. The Open Images label inspection relies on a single inspector and therefore cannot establish corrected ground truth. Similarly, excluding Gemma 4’s affected POPE cells avoids unsupported attribution but leaves those cases without an evaluation directly comparable to the main audit. These checks are valuable, but they also limit the completeness of the empirical conclusions.
Questions for authors
  1. How should readers separate violations of the fixed 10% budget from object-specific concentration when the held-out pooled FPR itself exceeds 10%? Could you summarize how the headline conclusions change under the tightened-calibration analysis and explain which concentration results address this distinction?
  2. How sensitive are the main findings to plausible dependence among images within an object group? The duplicate-pair checks are helpful, but they do not establish the within-group independence required for exact-binomial coverage. What empirical evidence or sensitivity analysis most directly supports this assumption?
  3. Beyond the reported one-sided-bound and leave-one-group-out diagnostics, which substantive benchmark-design conclusions would remain under a more statistically efficient simultaneous inference procedure? In particular, is the main practical limitation insufficient allocation itself, or insufficient allocation under the chosen certification convention?
Reviewer_tNPbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper is about evaluating VLMs for object-hallucinations on two well-known benchmarks (POPE and DASH-B). The issue the authors identify is that models can have similar hallucination scores but make different mistakes each. However, the existing benchmarks lack enough instances that help to determine which model is well suited for which category of an object.

Critical strengths
  • The issue the authors are tackling, namely really knowing where a model hallucinates objects and for which category is relevant. I agree with the authors that too many benchmarks lack enough examples to distinguish patterns such as object hallucinations. Also, which motivates a general perspective on how the community should design datasets for answering different research questions.
  • The authors fix some values for the experimental design in section 3, however, they report additional results in the appendix that list further tests. Also the evaluation with 8 different VLMs allows to check the behavior on a wide range of models.
Critical weaknesses

Clarity and organization:

  • The setup (Section 3) is sometimes very difficult to follow, particularly Section 3.2. The section would benefit from concrete examples rather than relying almost entirely on formal definitions. For instance, RgR_g is introduced earlier, but the notation is reused for the Clopper–Pearson bounds without clearly distinguishing the upper and lower bounds. Additionally, the discussion of primary inference and the full-data census is introduced rather abruptly. Providing some motivation for why these two approaches are used would help the reader understand their purpose. Similarly, in Section 3.4, basic clarifications of terms such as the “audit score ss” and “answer mass aa” would improve readability.
  • After the setup, a brief overview of the planned evaluations would be helpful. Splitting the evaluation across four separate sections without first explaining what is being evaluated, why these analyses are necessary, and how they relate to each other makes the paper difficult to follow. Consequently, I found it challenging to identify the central contribution and understand how the individual experiments support it. I believe the paper could be restructured to present a more coherent narrative, making the contributions less fragmented.
  • Lastly, I think an overall conclusion is missing. Section 8 primarily discusses the results and their implications, but its structure could also be clearer. The paper would benefit from a concise conclusion that revisits the main research questions and summarizes the key findings.
Questions for authors
  • What motivates the choice of a 10% budget? While the paper provides sensitivity analyses for alternative budgets, it would be helpful to clarify whether this threshold reflects a practically meaningful reliability requirement or primarily serves as an illustrative operating point.
  • Given that the proposed certificates are restricted to specific benchmark distributions, object categories, and decision thresholds, how should they be interpreted in practical VLM applications? Could you clarify what additional guarantees these certificates provide compared with conventional object-wise error rates and confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.