- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper examines object-hallucination benchmarks and shows evidence that aggregated evaluation metric such as FPR and ROC-AUC can hide detailed discrepancies at object class level. The detailed audit process reveals multiple such cases; open-image based augmentation shows that certification is possible with improved benchmark. The conclusion encourages the benchmark generators to provide detailed information about object group, risk budget, and confidence level for certification.
Critical strengths
- The paper presents clear evidence on how object-hallucination benchmarks for VLMs can assign models similar aggregated scores when they show discrepant performance over specific object classes. The guides for certification is a practical and useful one.
Critical weaknesses
- In the end, I think there is nothing surprising about the detailed results. Benchmarks are by definition bound to hide details when interpreted through aggregated evaluation metrics such as ROC-AUC. I think object class level variances and discrepancies are simply expected. I appreciate the actual experimentation and evidence, but the principle itself is not new.
Questions for authors
- What is the core take-home message here? The core finding itself is hardly surprising, unless we suspect a systematic, shared root cause (I don't think there is one, this is just models with varying performances). That leaves pretty standard guideline for benchmark construction and/or certification practice.