- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an evaluation method (VHA) for generative image super-resolution hallucinations which leverages visual-language models (VLMs) to extract semantic and visual information to provide evidence for judgment. It also trains a lighter model, VHA+, that does not require VLM inference, and constructs VHA-bench to measure the consistency between hallucination evaluation metrics and human judgments. Comparative experiments are provided to validate the model’s effectiveness.
Critical strengths
- The paper proposes a hallucination evaluation method (VHA) and a lightweight model, VHA+, and provides experiments to validate the effectiveness of both methods’ evaluations as well as the lightweight advantage of the VHA+.
- The VHA evaluation method combines semantic and visual information for judgment; ablation experiments demonstrate the complementary nature of these two types of information in hallucination evaluation under this classification framework.
- The paper constructs the hallucination evaluation benchmark VHA-bench, supplementing the manually annotated data required for evaluating hallucinations in super-resolution image reconstruction and helping to measure consistency between evaluation metrics and human judgments.
Critical weaknesses
It remains unclear how reliable the reference answers generated by the Qwen3-VL-8B-Instruct model are, particularly for low-level visual attributes such as texture categories. It is unclear whether the authors conducted manual sampling verification. Regarding BIDIRECTIONAL HALLUCINATION SCORING, the authors obtain the final score by summing the forward and reverse scores. Observing the calculation method, it appears that the current approach allows for offsetting due to opposite signs between the forward and reverse scores. Please explain the rationale behind this design and its practical impact.
Questions for authors
See weaknesses.
3 comments
Thank you for the response and additional analysis. The explanations of reference-answer filtering and forward/reverse score cancellation address my questions to some extent. The additional scoring comparison is also helpful. A quantitative check of the generated reference answers, particularly for low-level visual attributes, would further strengthen the evaluation. Overall, I maintain my original rating.
Thank you for your follow-up and for considering our responses and additional analysis.
Additional quantitative check of generated Q/A pairs
To provide a quantitative check of the reliability of generated Q/A pairs, we used GPT-6 Astra to classify each Q/A pair in VHA-bench as accurate or inaccurate based on its corresponding image.
| Accurate | Inaccurate | |
|---|---|---|
| High-level Q/A | 94.88% | 5.12% |
| Low-level Q/A | 97.32% | 2.68% |
| ALL | 96.48% | 3.52% |
Most Q/A pairs were judged accurate. Notably, the low-level Q/A pairs mentioned in the review were judged accurate at a higher rate than the high-level pairs (97.32% vs. 94.88%).
Manual verification of all generated Q/A pairs would require substantial effort. Although these VLM-based judgments may contain errors, they provide an additional quantitative check covering the full set of generated Q/A pairs.
Please let us know if any concerns remain or if further clarification would be helpful.
We thank for the constructive feedback and for recognizing the complementary roles of semantic and visual information in VHA. We also appreciate the positive assessment of VHA+'s efficiency and VHA-bench's role in evaluating agreement with human judgments. Below, we clarify the reliability of the generated reference answers and the rationale and practical impact of cancellation between the forward and reverse scores.
W1. Reliability of generated reference answers
We use automatic thresholding to limit the influence of the potential unreliable answers generated by a VLM. For each Q/A pair, we measure the difference between the answer's log-probabilities with and without the image used to generate the pair (Eq. 2). If this difference is at or below the corresponding threshold, the pair contributes zero (Eqs. 3 and 4). Table 2(c) shows that using both forward and reverse thresholding improves agreement with human judgments over either alone:
Additionally, we designed detailed prompts for low-level attributes to encourage reliable Q/A generation, requiring questions to identify the relevant image region and answers to describe clearly visible properties. We qualitatively inspected a subset of generated Q/A pairs and found them generally consistent with the visible image content. Figure 4 of the paper provides examples of generated Q/A pairs covering both high-level semantics and low-level visual attributes.
W2. Rationale and practical impact of forward/reverse cancellation
The reverse score R cannot be negative, but the forward score F can be negative and offset R in Eq. 5. A negative F means that SR provides stronger overall VLM support for GT-derived answers than GT does. We retain this offset to reflect the balance between support for original content and unsupported additions when assessing the image as a whole. On VHA-bench, we replaced F with max(F,0) in Eq. 5 and obtained the following correlations with human MOS:
The slight decrease suggests that allowing a negative F to offset R can benefit image-level hallucination assessment.