- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the method to construct contrastive branches in contrastive decoding for MLLM hallucination mitigation. The authors argue that exsiting strategies construct negative branches with predefined operations. To address this limitation, the authors propose Visual Evidence guided Role Assignment (VERA), an inference-time hallucination mitigation method which verifies intermediate visual hypothese before constructing contrastive branches. Experiments on CHAIR, POPE, and five general visual question answering benchmarks show the effectiveness of the proposed method.
Critical strengths
- On the whole the writing is clear, and the core of the idea can be grasped.
- Experiments on several benchmarks show the improvement by the proposed method.
Critical weaknesses
- Although the authors report the results of the proposed method on several benchmarks, the evaluation is neither enough nor fair. In table 1, the authors report results on CHAIR and POPE. However, 500 random samples from COCO 2014 validation set is far from enough: The variance is very high; the performance gain, such as an improvement of 0.87% on accuracy for LLaVA V1.5 on POPE adversarial set (equivalent to 4 more correct answers within 500 testcases), do not really imply substantial improvement. To make things worse, as reported by HalluEdit, the performance of HalluEdit for LLaVA V1.5 on POPE adversarial set is 82.5, rather than the 80.13 reported by the authors. There are strong possibilities that the authors suppress the performance of the baselines.
- The authors did not clearly define the meaning of some symbols, such as and .
Questions for authors
- Please provide the experiment results under wider and fairer evaluation protocol.
- What are the meanings of {h_1, h_2, \cdots, h_U\}? How are they computed?
- What are the meanings of and ?