- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
A large-scale human-annotated benchmark is introduced for evaluating IQA metrics on diffusion-based SR images using an NR-MOS protocol. The authors evaluated 51 standard FR and NR IQA configurations and 11 adapted variants, on both diffusion and non-diffusion SR outputs. They find that agreement with MOS decreases substantially on diffusion SR, and the performance varies by SR generator, input degradation, and upscaling factor. They also studied the effect of fine-tuning. However, some important IQA models, especially SR-dedicated and rank-learning based models are missing. Overall, the benchmark helps future development of quality predictors for diffusion-based and other generative SR, but the comparison and the subsequent analysis can be largely improved.
Critical strengths
- A large-scale subject-rated image dataset and benchmark directly address the lack of strong IQA models that could guide future construction of diffusion-based or other generative image SR methods.
- Extensive evaluations on known generic IQA models, including 51 standard FR and NR IQA configurations and 11 adapted variants, spanning traditional, CNN-based, and VLM-based metrics.
- Experiments on varies existing IQA methods provide some interesting observations, for example, fine-tuning could harm general IQA performance.
Critical weaknesses
- Main weakness: Although the evaluation on generic IQA models is extensive, the paper omits many important IQA models dedicated to evaluating SR images (only few are included), for example, W. Zhou et al, “Image super-resolution quality assessment: structural fidelity versus statistical naturalness,” QoMEX 2021. W. Zhou et al, “Quality assessment of image super-resolution: balancing deterministic and statistical fidelity,” ACM Multimedia 2022. L. Chen et al, “Graph-perceptron with semantic fidelity for no-reference super-resolution image quality assessment,” ACM Multimedia 2025. X. Li et al, “Hybrid vision transformer and convolutional neural network for super-resolution image quality assessment,” ICCVW 2025. L. Shu et al, “A dual-stream network with attention-guided feature fusion for blind super-resolution image quality assessment,” Displays 2026. These methods are highly relevant because they target SR-specific artifacts, and directly address the fidelity vs naturalness trade-off and the deterministic vs statistical fidelity trade-off, which are central to the perceptual quality of diffusion or generative SR.
- Also missing are rank-learning and rank-regularized regression based IQA models, which explicitly optimize for ordering and could potentially perform better on diffusion SR. Given that these methods are explicitly trained to optimize ordering, they might be more robust to the distribution shift of diffusion SR.
- The work reveals the performance drop but lacks deeply analysis on why diffusion outputs are harder to assess. Is it the synthesized texture that looks natural but differs from the reference? The missed evaluation metrics mentioned above may help with the explanation, but deeper investigation is missing.
- The fine-tuning experiments and the student model results are both interesting, but the results are mixed and inconclusive, and the improvement is modest.
Questions for authors
- Some important models especially SR-dedicated models are missing. In addition to those mentioned in “Critical Weaknesses”, the authors should perform a thorough search on other SR-dedicated, rank-learning based, and rank-regularization based IQA models. This is essential not only for completeness, but also because some of them explicitly address the main criticisms the current benchmark had on generic FR and NR IQA models. Therefore, the test results on these other IQA models may either change or strengthen the conclusions of the current paper. In addition to including these models in the comparison, it’d also be highly desirable to offer deeper analysis of them using the proposed benchmark. Furthermore, the authors may strongly demonstrate the value of the proposed benchmark by calibrating some of those models, for example, the balance between structural fidelity and statistical naturalness, or between deterministic and statistical fidelities.
- The paper notes that performance varies substantially by diffusion generator. Can you provide more insight into why some generators are harder to assess? Is it related to the types of artifacts they produce, or the degree of hallucination?
- The student model is the best among the evaluated configurations but is still far from human level performance. Again, it’d be helpful to provide some insight on where the main bottleneck is. The same applies to the fine-tuning experiments on SR adaptation harming general IQA performance. Why and what do we learn from it?
We thank the reviewer for the specific suggestions on SR-oriented and ranking-based IQA and for requesting deeper analysis. We have expanded the evaluation accordingly. W1–W2 / Q1 — Additional SR-specific and ranking-based methods.
We added seven models: SFSN, SRIF, VQ-R1, HiRQA, QCN-SPAQ, RankIQA+FT (TID2013), and UNIQUE. SFSN and SRIF directly test structural fidelity versus statistical naturalness and deterministic versus statistical fidelity. The other additions broaden coverage of ranking, comparison, and uncertainty-aware learning. RankIQA+FT denotes its TID2013 checkpoint, not adaptation on DISRQAD.
All seven obtain lower SRCC on diffusion than classic SR. Explicit naturalness modeling and ranking-oriented supervision therefore do not, in the tested pretrained configurations, eliminate the generalization gap. This does not preclude stronger results after diffusion-specific adaptation.
We also investigated Graph-Perceptron with Semantic Fidelity (ACM MM 2025), Hybrid ViT–CNN (ICCVW 2025), and DSAG-Net (Displays 2026). The source code and weights were not available, so we could not evaluate them on our benchmark.
W3 / Q2 — Diffusion artifacts and differences between generators.
We distinguish two plausible mechanisms. First, synthesized details can appear convincing without matching the HR reference. Humans viewing only the output may prefer them, while FR metrics penalize the reference mismatch. Lower FR–MOS agreement can therefore reflect a difference between fidelity and the benchmark's perceived-quality target.
Second, diffusion outputs can contain plastic-looking textures, inconsistent structures, or sharp details incompatible with their surroundings. Generic NR models may reward familiar sharpness or naturalness cues while missing these artifacts because of a training-domain mismatch. We present this as a hypothesis consistent with the results, not as an experimentally isolated cause.
We will expand the qualitative discussion to distinguish visually coherent departures from GT from visibly inconsistent synthesis. Under NR assessment, the perceptual coherence of generated detail matters alongside its amount.
Generator dependence also appears in the added models: VQ-R1 reaches SRCC 0.632 on PassionSR but 0.159 on DiT4SR. DiT4SR nevertheless has split-half human Spearman agreement of 0.814. Thus, low metric correlation coexists with reproducible human rankings. Different artifact mixtures plausibly contribute, although attributing this variation to particular generator components requires further controlled study.
W4 / Q3 — Adaptation, the remaining bottleneck, and general IQA.
DISRQAD's main contribution is a benchmark for diagnosing existing IQA methods; the student is an adaptation case study. In the updated evaluation, Q-ReAlign-mini improves from diffusion SRCC 0.345 to 0.494 after direct fine-tuning, and the distilled student reaches 0.531. Other methods exhibit mixed outcomes, including improved diffusion alignment accompanied by reduced classic-SR performance.
A plausible explanation is that limited SR-specific supervision shifts the learned quality cues, reducing transfer to other distortion domains. This is consistent with the non-SR losses in Section 5.2, but does not show that specialization inevitably harms general IQA. Replay and anchoring address this trade-off in the student; the current comparisons do not isolate every component's contribution.
The remaining gap is not readily explained solely by annotation noise: diffusion split-half Spearman agreement is 0.862. This is not an exact human ceiling for metric–MOS correlation, but it indicates substantial scope for better prediction. Reliable annotations, generator-specific diagnostics, and the mixed adaptation outcomes make DISRQAD useful for investigating that bottleneck without claiming that the student solves it.