acceptodds
Under review as a conference paper at ICLR 2027

VHA: VQA-based Hallucination Assessment for Generative Image Super-Resolution

Abstract

Generative image super-resolution produces visually realistic high-resolution images, but may also introduce hallucinations by adding unsupported content or altering existing details. Because existing super-resolution metrics fail to measure whether generated details remain grounded in the reference image, we present a hallucination assessment technique based on bidirectional visual question answering (VQA), which we term VQA-based Hallucination Assessment (VHA). The proposed approach uses a vision-language model (VLM) to determine whether the semantic content and low-level visual evidence in a super-resolved image are supported by its ground-truth counterpart. To this end, it constructs question–answer pairs from each image and evaluates them on both images. Discrepancies between the resulting scores in both directions reveal missing or altered ground-truth information and unsupported additions. However, obtaining these scores requires computationally expensive VLM inference. We therefore develop VHA+, a learned, lightweight model that directly predicts VHA scores from ground-truth and super-resolved image pairs. We construct a hallucination benchmark, VHA-bench, through a human study and show that both VHA and VHA+ correlate highly with human judgments of hallucination. We will publicly release our code, benchmark, and model weights to support reproducibility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Low-level vision.+super-resolution.+hallucination

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bTRBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an evaluation method (VHA) for generative image super-resolution hallucinations which leverages visual-language models (VLMs) to extract semantic and visual information to provide evidence for judgment. It also trains a lighter model, VHA+, that does not require VLM inference, and constructs VHA-bench to measure the consistency between hallucination evaluation metrics and human judgments. Comparative experiments are provided to validate the model’s effectiveness.

Critical strengths
  1. The paper proposes a hallucination evaluation method (VHA) and a lightweight model, VHA+, and provides experiments to validate the effectiveness of both methods’ evaluations as well as the lightweight advantage of the VHA+.
  2. The VHA evaluation method combines semantic and visual information for judgment; ablation experiments demonstrate the complementary nature of these two types of information in hallucination evaluation under this classification framework.
  3. The paper constructs the hallucination evaluation benchmark VHA-bench, supplementing the manually annotated data required for evaluating hallucinations in super-resolution image reconstruction and helping to measure consistency between evaluation metrics and human judgments.
Critical weaknesses

It remains unclear how reliable the reference answers generated by the Qwen3-VL-8B-Instruct model are, particularly for low-level visual attributes such as texture categories. It is unclear whether the authors conducted manual sampling verification. Regarding BIDIRECTIONAL HALLUCINATION SCORING, the authors obtain the final score by summing the forward and reverse scores. Observing the calculation method, it appears that the current approach allows for offsetting due to opposite signs between the forward and reverse scores. Please explain the rationale behind this design and its practical impact.

Questions for authors

See weaknesses.

3 comments

We thank for the constructive feedback and for recognizing the complementary roles of semantic and visual information in VHA. We also appreciate the positive assessment of VHA+'s efficiency and VHA-bench's role in evaluating agreement with human judgments. Below, we clarify the reliability of the generated reference answers and the rationale and practical impact of cancellation between the forward and reverse scores.


W1. Reliability of generated reference answers

We use automatic thresholding to limit the influence of the potential unreliable answers generated by a VLM. For each Q/A pair, we measure the difference between the answer's log-probabilities with and without the image used to generate the pair (Eq. 2). If this difference is at or below the corresponding threshold, the pair contributes zero (Eqs. 3 and 4). Table 2(c) shows that using both forward and reverse thresholding improves agreement with human judgments over either alone:

ThresholdingSpearmanPearson
Forward score thresholding0.5700.614
Reverse score thresholding0.6070.635
Forward + Reverse score thresholding0.6180.645

Additionally, we designed detailed prompts for low-level attributes to encourage reliable Q/A generation, requiring questions to identify the relevant image region and answers to describe clearly visible properties. We qualitatively inspected a subset of generated Q/A pairs and found them generally consistent with the visible image content. Figure 4 of the paper provides examples of generated Q/A pairs covering both high-level semantics and low-level visual attributes.


W2. Rationale and practical impact of forward/reverse cancellation

The reverse score RR cannot be negative, but the forward score FF can be negative and offset RR in Eq. 5. A negative FF means that SR provides stronger overall VLM support for GT-derived answers than GT does. We retain this offset to reflect the balance between support for original content and unsupported additions when assessing the image as a whole. On VHA-bench, we replaced FF with max⁡(F,0)\max(F, 0) in Eq. 5 and obtained the following correlations with human MOS:

Score calculationSpearmanPearson
F+RF + R (original)0.6180.645
max⁡(F,0)+R\max(F, 0) + R0.6130.637

The slight decrease suggests that allowing a negative FF to offset RR can benefit image-level hallucination assessment.

Thank you for the response and additional analysis. The explanations of reference-answer filtering and forward/reverse score cancellation address my questions to some extent. The additional scoring comparison is also helpful. A quantitative check of the generated reference answers, particularly for low-level visual attributes, would further strengthen the evaluation. Overall, I maintain my original rating.

Thank you for your follow-up and for considering our responses and additional analysis.


Additional quantitative check of generated Q/A pairs

To provide a quantitative check of the reliability of generated Q/A pairs, we used GPT-6 Astra to classify each Q/A pair in VHA-bench as accurate or inaccurate based on its corresponding image.

AccurateInaccurate
High-level Q/A94.88%5.12%
Low-level Q/A97.32%2.68%
ALL96.48%3.52%

Most Q/A pairs were judged accurate. Notably, the low-level Q/A pairs mentioned in the review were judged accurate at a higher rate than the high-level pairs (97.32% vs. 94.88%).

Manual verification of all generated Q/A pairs would require substantial effort. Although these VLM-based judgments may contain errors, they provide an additional quantitative check covering the full set of generated Q/A pairs.


Please let us know if any concerns remain or if further clarification would be helpful.

Reviewer_9S4MOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1(−2)
Read review
Paper summary

Generative image super-resolution models often suffer from hallucinations. This paper aims to assess hallucinations in super-resolved images using vision-language models (VLMs). The authors propose a bidirectional visual question answering (VQA) approach to hallucination assessment. The method assesses hallucinations bidirectionally by examining whether the original content is preserved in the SR image and whether newly generated content is supported by the GT image.

To address the high computational cost of VLM inference, the authors further propose VHA+, which fine-tunes a DINOv3 model to directly predict VHA scores from pairs of GT and SR images. The authors then use VHA+ as a reward model, together with MUSIQ, to fine-tune VOSR. Experimental results show that the fine-tuned model achieves lower VHA and VHA+ scores while maintaining generation quality.

Critical strengths
  1. The research objective is clear, and detecting and mitigating hallucinations in generative image super-resolution is of practical importance.
  2. The bidirectional VQA design is effective. Rather than simply asking a VLM to assign an overall score, the proposed method assesses both the loss of original information and the introduction of unsupported content.
  3. VHA+ significantly accelerates hallucination assessment, alleviating the high computational cost of VQA-based methods. It can also be used to optimize SR models.
Critical weaknesses
  1. The methodological novelty is limited. Compared with the baseline Hallucination Score, the paper mainly replaces direct VLM-based scoring with bidirectional VQA and replaces direct DINO-based scoring with scoring using a fine-tuned DINO model. These modifications appear to be relatively incremental improvements.
  2. The paper only uses Qwen3-VL-8B-Instruct for VQA-based assessment. It does not investigate how larger VLMs or proprietary models might affect hallucination assessment performance.
  3. The paper uses Qwen3-VL-8B-Instruct and the corresponding VHA+ model to optimize VOSR, while evaluating the fine-tuned model using the same VHA and VHA+ metrics. This raises concerns about potential evaluation bias and metric overfitting. Furthermore, the paper does not provide human evaluations of the VOSR* results.
Questions for authors
  1. Does VOSR* suffer from potential overfitting to the VHA/VHA+ metrics?
  2. Why does VHA+ achieve a higher correlation with human judgments than VHA, despite being trained to predict VHA scores?
  3. VHA+ is trained solely on image pairs and scalar VHA scores. However, VQA-based scoring and DINOv3-based scoring differ substantially in their evaluation mechanisms. Can VHA+ preserve the fine-grained hallucination detection capabilities of VHA, or does it merely learn correlated statistical patterns?
4 comments

We thank for the thoughtful feedback and for recognizing the practical importance of hallucination assessment and mitigation in generative SR. We appreciate the positive assessment of our bidirectional VQA design and the efficiency and utility of VHA+. Below, we clarify our contributions and address the questions about VLM backbones, metric overfitting, and VHA+'s fine-grained hallucination detection capacity.


W1. Limited methodological novelty

Our contribution is the design of a structured VQA-based hallucination assessment framework. Our taxonomy and evidence-based Q/A generation enable assessment across diverse semantic content and visual attributes, while bidirectional scoring captures both lost or altered GT content and unsupported additions in SR outputs. Compared with the baseline Hallucination Score, which provides a discrete rating and textual explanation, our continuous VHA score quantifies each Q/A pair's contribution, showing which queried content or attributes raise or lower the score.

Tables 1 and 2(a) of the revised manuscript show the effect of this design using the same Qwen3-VL-8B-Instruct backbone and the VHA-bench human ratings.

Assessment methodSpearmanPearson
Direct VLM rating0.4970.482
Hallucination Score prompting0.5430.563
VHA, forward only0.5160.562
VHA, reverse only0.4650.560
VHA, both directions0.6180.645

VHA+ builds on the DINO feature-distance approach of Hallucination Score. Our contribution here is a training pipeline that uses our VHA scores to learn a lightweight, differentiable predictor. This makes the assessment signal available as a differentiable SR training loss and reduces evaluation time from 101s to 17ms per image pair on the same GPU.


W2. Dependence on a single VLM backbone

We evaluated VHA with several VLM backbones on the same 540 GT–SR pairs in VHA-bench, with the results reported below.

VLM backboneSpearmanPearson
Qwen3-VL-4B-Instruct0.5760.623
Qwen3-VL-8B-Instruct (ours)0.6180.645
Qwen3-VL-32B-Instruct0.6430.631

Within Qwen3-VL, the 8B model outperforms the 4B model on both correlation measures. Scaling further to 32B improves Spearman correlation but reduces Pearson correlation relative to 8B. These results suggest that increasing model size does not necessarily yield consistent improvements in VHA performance.

VHA is backbone-agnostic and can be applied to VLMs that provide access to the answer-token probabilities required for scoring. However, applying VHA to some proprietary models is challenging because their APIs do not expose the necessary answer-token logits or log-probabilities.


W3. Q1. Regarding metric overfitting and human evaluation

We added a human comparison of VOSR and VOSR* to Section 5.4 of the revised manuscript. On the 100-image RealSR test set, each of 21 participants viewed 30 randomly sampled output pairs alongside the corresponding ground-truth image. Participants selected the output with fewer hallucinations or indicated a tie. They selected VOSR* more than twice as often as VOSR (Table 5 of the revised manuscript).

VOSRTieVOSR*
20.48%27.62%51.90%

We also repeated the VHA evaluation with two VLM backbones that were used in neither VHA+ training nor VOSR fine-tuning. Both assigned lower hallucination scores to VOSR*, consistent with the human preference on RealSR (Table 6 of the revised manuscript).

VHA's VLM backboneVOSRVOSR*
Ministral-3-8B-Instruct0.44400.3938
Gemma-3-12B-IT2.44642.1038

Together with the consistent improvements in all evaluation metrics (Table 4 of the main paper), these results suggest that the gains are not solely driven by metric overfitting.

Q2. Higher human correlation of VHA+ than VHA

Although VHA provides the targets, VHA+ learns from image pairs spanning different degrees of content preservation and alteration, using both MSE and correlation losses. These training choices may improve agreement with human judgments; Table 13(a) of the revised manuscript supports the contribution of the correlation loss. Furthermore, VHA+'s higher correlations may result from combining VHA supervision with pretrained DINOv3 representations through partial fine-tuning. The trainable block-count ablation in Table 13(b) of the revised manuscript gives the following correlations with human judgments:

Trainable blocksSpearmanPearson
Last 20.59620.6212
Last 40.65940.6753
Last 100.63890.6631

Too few trainable blocks may limit adaptation to the VHA targets, while updating too many may alter useful pretrained representations. Together, this training strategy and the pretrained DINOv3 representations may explain why VHA+ achieves higher correlations with human judgments than VHA.


Q3. Fine-grained detection versus correlated statistical patterns

We added a VHA+ feature analysis in Section 5.3 of the revised manuscript, comparing GT–SR patch-feature differences from pretrained DINOv3 and VHA+. The results in the Figure 6 of the revised manuscript explicitly show that VHA+ can capture fine-grained hallucination-related differences. In the top example, VHA+ exhibits strong feature differences around the altered alphabets in “eduroam” text. In the bottom example, VHA+ shows pronounced feature differences around bottle labels containing altered patterns, textures, and text.

Reviewer_9S4MOpenReview ↗

Update

The author's response addressed most of my concerns, so I have changed my score to weak accept.

Thank you for taking the time to review our responses and reconsider your assessment. We appreciate your constructive feedback and thoughtful discussion.

Reviewer_7f3GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hallucination assessment in generative image SR. It defines hallucination as losing or changing visual evidence in the GT, or adding content that the GT does not support. The authors propose VHA, which uses a VLM to generate Q/A from both GT and SR images. The method scores the same answer under both images to measure whether original content is preserved and new content is supported by GT. Because VHA requires repeated VLM inference, the authors also propose VHA+, which predicts VHA scores from DINOv3 feature distances without running the VLM at evaluation time.

Critical strengths
  1. The method addresses both missing and unsupported content.

  2. The paper offers a more efficient learned evaluator and an initial human comparison.

Critical weaknesses
  1. VHA+ is used directly as a training loss, and the paper then uses VHA and VHA+ scores to show that hallucination decreases after fine-tuning. This does not independently establish that human-perceived hallucination decreases. The provided text does not report an independent human evaluation of VOSR before and after fine-tuning.

  2. The final values of hyper-parameters (such as thresholds \tau_g, \tau_s, \tau_m) are selected using human ratings from VHA-bench. Authors should show results on a separate test set that was not used for these choices.

Questions for authors
  1. Equation (3) uses signed differences, so contributions from different Q/A pairs can cancel and the forward score can be negative. How should negative scores and this cancellation be interpreted? Could a severe content error be offset by other Q/A pairs, reducing the final hallucination score?

  2. Did the authors conduct an independent human evaluation of VOSR and VOSR*, or compare their hallucinations using an evaluator not involved in VHA/VHA+ training or optimization?

4 comments

We thank for the constructive feedback and for recognizing the value of assessing both missing and unsupported content, our efficient learned evaluator and initial human comparison. Below, we address the concerns about independent evaluation, hyperparameter selection, and signed scoring.


W1. Q2. Independent evaluation of VOSR fine-tuning

We added a human comparison of VOSR and VOSR* to Section 5.4 of the revised manuscript. On the 100-image RealSR test set, each of 21 participants viewed 30 randomly sampled output pairs alongside the corresponding ground-truth image. Participants selected the output with fewer hallucinations or indicated a tie. They selected VOSR* more than twice as often as VOSR (Table 5 of the revised manuscript).

VOSRTieVOSR*
20.48%27.62%51.90%

We also repeated the VHA evaluation with two VLM backbones that were used in neither VHA+ training nor VOSR fine-tuning. Both assigned lower hallucination scores to VOSR*, consistent with the human preference on RealSR (Table 6 of the revised manuscript).

VHA's VLM backboneVOSRVOSR*
Ministral-3-8B-Instruct0.44400.3938
Gemma-3-12B-IT2.44642.1038

Together with the consistent improvements in all evaluation metrics (Table 4 of the main paper), these results suggest that the gains are not solely driven by metric overfitting.


W2. Hyperparameter selection and a separate test set

Table 12 in the appendix of the paper shows that VHA is relatively insensitive to the tested scoring thresholds. We varied each threshold and obtained the following ranges of correlations with human mean opinion scores (MOS).

HyperparameterSpearman rangePearson range
GT answer threshold, τg0.5919–0.61900.6295–0.6449
SR answer threshold, τs0.5941–0.61790.6295–0.6449
Reverse margin, τm0.5909–0.61790.6366–0.6449

The reported setting (τg = −2, τs = 0, τm = 0.5) gives Spearman/Pearson correlations of 0.6179/0.6449. Nearby settings yield similar results.

We are collecting human ratings to construct a separate test set and will add the results when the analysis is complete.


Q1. Negative forward scores and cancellation

Yes. A severe error on one question can be partially offset by another question. A negative forward term means that the VLM assigns stronger support to the same GT-derived answer under the SR image than under the GT image. For example, in Fig. 4 of the manuscript, the altered sign text gives a forward difference of +10.62 for the answer “eduroam,” while the answer “no smoking sign” on the same image gives −1.62. These terms partially offset in Eq. (3). Since an image contains multiple attributes, we retain both increases and decreases in answer support to capture their combined effect on the overall score.

To assess the effect of negative forward contributions, we evaluated two alternatives on VHA-bench: clamping negative contributions to zero and excluding the corresponding Q/A pairs. The table reports correlations between the VHA score and human MOS.

Negative forward Q/A contributionsSpearmanPearson
Signed contributions (Ours)0.6180.645
Set negative contributions to zero0.6070.632
Remove Q/A pairs of negative contribution0.5800.614

These results show that retaining signed forward contributions improves agreement with image-level human judgments.

W2. Hyperparameter selection and a separate test set

To complement the threshold sensitivity analysis, we have now completed data collection for an additional evaluation set, DRealSR-bench. Following the same procedure used to construct VHA-bench, we built DRealSR-bench using 50 DRealSR GT images and outputs from the same six generative SR methods, yielding 300 GT–SR pairs. Each of 34 participants evaluated 54 pairs, giving approximately six ratings per pair on average. We averaged the ratings for each pair to obtain its mean opinion score (MOS).

We evaluated three hyperparameters used to compute VHA scores—the forward support threshold (τg), reverse support threshold (τs), and reverse margin (τm)—on DRealSR-bench. Both tables below report Spearman (ρ) and Pearson (r) correlations between VHA scores and human MOS.

τgρr│τsρr│τmρr
−50.51200.5046│−10.51310.5069│−20.49690.5134
−40.51310.5073│00.51260.5112│−10.51040.5173
−30.51540.5149│10.51540.5149│00.51540.5149
−20.51670.5128│20.50330.5163│10.50000.5028
−10.50580.5089│30.48580.5096│20.47360.4840

We then evaluated the DRealSR-bench-tuned setting, (τg, τs, τm) = (−3, 1, 0), on VHA-bench by recomputing scores.

Hyperparameter setting (τg, τs, τm)VHA-benchDRealSR-bench
Spearman / PearsonSpearman / Pearson
VHA-bench-tuned (−2, 0, 0.5)0.6179 / 0.64490.5126 / 0.5062
DRealSR-bench-tuned (−3, 1, 0)0.6111 / 0.64100.5154 / 0.5149

Compared with the original setting, Spearman and Pearson correlations on VHA-bench decreased by 0.0068 and 0.0039, respectively. Together with the threshold sensitivity analysis reported above, these results suggest that VHA is relatively insensitive to the tested hyperparameter choices and that tuning on DRealSR-bench yields similar agreement with human judgments on VHA-bench.

I thank the authors for their response. VHA offers an interesting and useful perspective on assessing hallucinations in generative image super-resolution using visual question answering. By evaluating the faithfulness of both semantic content and visual attributes to the reference image, it complements existing evaluation metrics. However, it does not provide a comprehensive assessment of overall perceptual quality, which limits its use as a standalone evaluation metric. It should therefore be used alongside established image quality metrics. Overall, I maintain my recommendation of weak accept.

Thank you for your thoughtful review and for recognizing the usefulness of our work. We agree that VHA focuses on hallucination assessment rather than serving as a standalone measure of overall perceptual quality. When used alongside existing image quality metrics, VHA can contribute to a more comprehensive assessment of generative super-resolution results. We appreciate your constructive feedback and careful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.