- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses semantic hallucinations in generative image super-resolution. The authors argue that full-image preference optimization cannot accurately localize failure regions, and that object-level semantic regions may not align with hallucination boundaries. They propose HGM, which uses an indexed grid and a VLM for candidate localization, followed by GT/SR local crops for verification, and constructs AHS from region severity and area. Building on HGM, HSPO exchanges local content across multiple random candidates, with GT fallback when necessary, to construct chosen/rejected preference pairs and restricts the preference loss to the relevant regions. The authors claim that this reduces hallucinations while maintaining competitive perceptual quality, supported by two initializations (C-FLUX and DP²O), two VLM evaluators, component ablations, and human evaluation. The core contribution is organizing preference supervision around actual failure locations, rather than being the first to propose SR hallucination or DPO.
Critical strengths
- Clear method and strong motivation. The paper is well organized and easy to follow. The argument for localizing hallucinations rather than optimizing globally is convincing, and the preliminary taxonomy of SR hallucinations offers a useful reference for future work.
- Region-level preference construction is sound. Building local chosen/rejected pairs at the same spatial location avoids local strengths and weaknesses canceling out in image-level comparisons. Treating failure regions as supervision units for post-training has strong potential for transfer to other generative restoration tasks.
Critical weaknesses
- Validity and reproducibility of the severity metric remain insufficient. Figure 3 mainly validates area correlation and localization F1, but does not directly verify the consistency between the continuous severity score and human judgment. The threshold used to convert Stage 2 scores into the retained binary mask is unclear. In addition, the matching and counting rule when multiple predicted cells correspond to the same human-annotated region needs clarification.
- The meaning of "fewer hallucinations" should be further qualified. A lower average AHS does not always coincide with a lower number of images containing hallucinations, and LPIPS degrades in some settings. The current evidence is therefore better suited to supporting improvements in average area/severity than synchronized fidelity gains across all dimensions.
- Training cost is not disclosed. Without GPU resources, training steps, batch size, learning rate, and total time, the practical advantage for online training remains unclear.
Questions for authors
- Can the authors provide a curve with the score threshold (0 to 1) on the x-axis and F1@score on the y-axis, to show how finely the current VLM localizes hallucinations beyond a single operating point?
- In online training, what is the additional time cost of introducing an extra VLM for Set-of-Marks localization compared with directly performing flow matching? How many training GPU hours are typically needed for convergence, and can the corresponding training cost be disclosed?
- Since training is performed only on a small masked region, does this affect the generation result of the full image? How well are non-masked regions preserved before and after optimization? Is there any risk of degrading the original generation quality outside the masked regions?
- What is the additional contribution of local preference optimization relative to direct GT supervision? The authors are encouraged to prioritize controlled experiments under the same mask and the same training budget: (i) a local GT reconstruction baseline, and (ii) an HSPO variant with GT fallback removed. These comparisons would address the attribution question more directly than adding several external SR baselines.
3 comments
Response to LaRE (Part2/2) : Effects of Masked Optimization and Contribution Beyond GT SFT
Q3. Impact of Masked Optimization on Unmasked Regions
HSPO computes the preference loss within the detected hallucination masks and introduces a global GT-MSE loss with a weight of 0.1 to prevent degradation of overall reconstruction capability. Local preferences are designed to avoid imposing unnecessary preference constraints on regions that are already correct, thereby reducing the risk of disturbing normal regions.
As shown in Table 2 of the paper, HSPO achieves lower hallucination levels without degrading perceptual metrics. In the ablation study, Ours achieves a lower hallucination level than Global, which computes the loss globally without a mask. Overall, because model parameters are shared spatially, local optimization affects unmasked regions. However, local masks prevent local strengths and weaknesses from potentially canceling each other out and avoid incorrectly computing preferences for normal regions. Therefore, they preserve the original generation quality while reducing hallucinations.
Q4. HSPO versus Masked GT SFT
We have added two ablations: (1) removing GT fallback and the GT-MSE loss, with the mask defined as the union of the masks for A and B minus their intersection; and (2) SFT training, computing the GT loss only within the masked regions to measure the difference between HSPO and SFT while keeping the training budget the same. The results are shown below:
Table 3. Ablation study.
| Method | PSNR ↑ | MUSIQ ↑ | CLIPIQA ↑ | AHS-Q ↓ | Hal.Img-Q ↓ |
|---|---|---|---|---|---|
| Ours | 22.09 | 65.91 | 0.6950 | 0.0286 | 81/300 |
| w/o GT | 22.23 | 64.89 | 0.6820 | 0.0302 | 82/300 |
| Random Mask | 22.01 | 64.83 | 0.6360 | 0.0372 | 85/300 |
| GT SFT | 21.26 | 69.16 | 0.6850 | 0.0381 | 89/300 |
| Global | 21.89 | 65.27 | 0.6340 | 0.0389 | 89/300 |
| Global w/o exchange | 21.65 | 66.07 | 0.6400 | 0.0455 | 92/300 |
The experiments show that GT fallback has some impact on performance, but preference learning makes the main contribution. GT SFT also reduces hallucinations, potentially benefiting from both adaptation to the training degradation distribution and hallucination-aware supervision within the detected regions, while HSPO achieves further improvements by leveraging region-level preference learning.
Reviewer Comments
Thank you for the additional experiments and clarifications. My concerns have been largely addressed.
Response to LaRE (Part1/2) : Evaluation Protocol and Computational Costs
We thank the reviewer for their careful review. We provide clarifications below.
Q1. Severity Threshold and F1 Matching Protocol
The second stage outputs a region score in [0,1]; by default, we retain regions with score>0, while score=0 classifies a candidate as not containing a semantic hallucination. For localization evaluation, we process annotated regions in annotation order and select the unmatched prediction with the highest coverage, defined as intersection area divided by annotated-region area. A match is counted as TP when coverage is at least 0.3. Each annotated region contributes at most one TP; the remaining predictions are FP and undetected annotated regions are FN.
The reviewer asks us to clarify how the second-stage threshold affects the results. Our prompt specifies that a score greater than 0 indicates a hallucination. In practice, the VLM's scores for local hallucinations are not uniformly distributed. The distribution computed from the test-set scoring results is shown below.
Table 1. Distribution of second-stage hallucination scores.
Thus, the VLM effectively produces a discretized assessment: score=0 indicates no hallucination, while score>0 measures hallucination severity.
Q2. Offline Data Construction and Training Costs
Our method uses offline preference optimization. Before optimization begins, we generate SR candidates in advance and use HGM for Set-of-Marks localization and verification. The dataset is saved for subsequent optimization.
During dataset construction, the VLM performs HGM evaluation on the candidates, taking approximately 2 seconds per image on two A800 GPUs. For 38K images, this requires 53 GPU-hours.
During training, the cost of preference optimization is equivalent to that of DPO training. With a batch size of 128 and 500 steps, HSPO requires 25 GPU-hours, while direct flow-matching SFT requires 10 GPU-hours for the same number of iterations. Although HSPO has a higher training cost, SFT alone cannot provide semantic preference information and mitigate SR hallucinations (see Q4).