acceptodds
Under review as a conference paper at ICLR 2027

DISRQAD: Diffusion Image Super-Resolution Quality Assessment Dataset and Benchmark

Abstract

Diffusion-based image super-resolution (SR) can create visually plausible detail that is not supported by the low-resolution input. We introduce DISRQAD, a subjective-quality dataset and diagnostic benchmark for this setting. It contains mean opinion scores (MOS) for 14,000 SR outputs from ten diffusion and four non-diffusion methods, spanning four low-resolution degradation conditions and / upscaling. We evaluate 51 standard full-reference and no-reference metric configurations and 11 adapted variants. Agreement with MOS is substantially weaker on diffusion outputs: the strongest standard no-reference baseline reaches 0.431 SRCC on diffusion SR versus 0.813 on non-diffusion SR. As a case study in benchmark use, a pruned and distilled Q-ReAlign-mini student reaches 0.496 SRCC on diffusion SR. DISRQAD measures perceived output quality, not faithfulness to the input; it enables analysis of metric behavior across generator families and input conditions. Our findings reveal a substantial gap in the assessment of diffusion-based SR and provide a basis for developing quality models sensitive to diffusion-specific artifacts.

Primary area:
datasets and benchmarks
Keywords:
super-resolution, dataset, image quality assessment, diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HvjjOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

A large-scale human-annotated benchmark is introduced for evaluating IQA metrics on diffusion-based SR images using an NR-MOS protocol. The authors evaluated 51 standard FR and NR IQA configurations and 11 adapted variants, on both diffusion and non-diffusion SR outputs. They find that agreement with MOS decreases substantially on diffusion SR, and the performance varies by SR generator, input degradation, and upscaling factor. They also studied the effect of fine-tuning. However, some important IQA models, especially SR-dedicated and rank-learning based models are missing. Overall, the benchmark helps future development of quality predictors for diffusion-based and other generative SR, but the comparison and the subsequent analysis can be largely improved.

Critical strengths
  1. A large-scale subject-rated image dataset and benchmark directly address the lack of strong IQA models that could guide future construction of diffusion-based or other generative image SR methods.
  2. Extensive evaluations on known generic IQA models, including 51 standard FR and NR IQA configurations and 11 adapted variants, spanning traditional, CNN-based, and VLM-based metrics.
  3. Experiments on varies existing IQA methods provide some interesting observations, for example, fine-tuning could harm general IQA performance.
Critical weaknesses
  1. Main weakness: Although the evaluation on generic IQA models is extensive, the paper omits many important IQA models dedicated to evaluating SR images (only few are included), for example, W. Zhou et al, “Image super-resolution quality assessment: structural fidelity versus statistical naturalness,” QoMEX 2021. W. Zhou et al, “Quality assessment of image super-resolution: balancing deterministic and statistical fidelity,” ACM Multimedia 2022. L. Chen et al, “Graph-perceptron with semantic fidelity for no-reference super-resolution image quality assessment,” ACM Multimedia 2025. X. Li et al, “Hybrid vision transformer and convolutional neural network for super-resolution image quality assessment,” ICCVW 2025. L. Shu et al, “A dual-stream network with attention-guided feature fusion for blind super-resolution image quality assessment,” Displays 2026. These methods are highly relevant because they target SR-specific artifacts, and directly address the fidelity vs naturalness trade-off and the deterministic vs statistical fidelity trade-off, which are central to the perceptual quality of diffusion or generative SR.
  2. Also missing are rank-learning and rank-regularized regression based IQA models, which explicitly optimize for ordering and could potentially perform better on diffusion SR. Given that these methods are explicitly trained to optimize ordering, they might be more robust to the distribution shift of diffusion SR.
  3. The work reveals the performance drop but lacks deeply analysis on why diffusion outputs are harder to assess. Is it the synthesized texture that looks natural but differs from the reference? The missed evaluation metrics mentioned above may help with the explanation, but deeper investigation is missing.
  4. The fine-tuning experiments and the student model results are both interesting, but the results are mixed and inconclusive, and the improvement is modest.
Questions for authors
  1. Some important models especially SR-dedicated models are missing. In addition to those mentioned in “Critical Weaknesses”, the authors should perform a thorough search on other SR-dedicated, rank-learning based, and rank-regularization based IQA models. This is essential not only for completeness, but also because some of them explicitly address the main criticisms the current benchmark had on generic FR and NR IQA models. Therefore, the test results on these other IQA models may either change or strengthen the conclusions of the current paper. In addition to including these models in the comparison, it’d also be highly desirable to offer deeper analysis of them using the proposed benchmark. Furthermore, the authors may strongly demonstrate the value of the proposed benchmark by calibrating some of those models, for example, the balance between structural fidelity and statistical naturalness, or between deterministic and statistical fidelities.
  2. The paper notes that performance varies substantially by diffusion generator. Can you provide more insight into why some generators are harder to assess? Is it related to the types of artifacts they produce, or the degree of hallucination?
  3. The student model is the best among the evaluated configurations but is still far from human level performance. Again, it’d be helpful to provide some insight on where the main bottleneck is. The same applies to the fine-tuning experiments on SR adaptation harming general IQA performance. Why and what do we learn from it?
1 comment

We thank the reviewer for the specific suggestions on SR-oriented and ranking-based IQA and for requesting deeper analysis. We have expanded the evaluation accordingly. W1–W2 / Q1 — Additional SR-specific and ranking-based methods.

We added seven models: SFSN, SRIF, VQ-R1, HiRQA, QCN-SPAQ, RankIQA+FT (TID2013), and UNIQUE. SFSN and SRIF directly test structural fidelity versus statistical naturalness and deterministic versus statistical fidelity. The other additions broaden coverage of ranking, comparison, and uncertainty-aware learning. RankIQA+FT denotes its TID2013 checkpoint, not adaptation on DISRQAD.

MetricDiffusion SRClassic SR
VQ-R10.3960.773
SFSN0.3280.695
SRIF0.2760.456
HiRQA0.0560.494
QCN-SPAQ0.3540.508
RankIQA+FT (TID2013)0.0740.540
UNIQUE0.1620.630

All seven obtain lower SRCC on diffusion than classic SR. Explicit naturalness modeling and ranking-oriented supervision therefore do not, in the tested pretrained configurations, eliminate the generalization gap. This does not preclude stronger results after diffusion-specific adaptation.

We also investigated Graph-Perceptron with Semantic Fidelity (ACM MM 2025), Hybrid ViT–CNN (ICCVW 2025), and DSAG-Net (Displays 2026). The source code and weights were not available, so we could not evaluate them on our benchmark.

W3 / Q2 — Diffusion artifacts and differences between generators.

We distinguish two plausible mechanisms. First, synthesized details can appear convincing without matching the HR reference. Humans viewing only the output may prefer them, while FR metrics penalize the reference mismatch. Lower FR–MOS agreement can therefore reflect a difference between fidelity and the benchmark's perceived-quality target.

Second, diffusion outputs can contain plastic-looking textures, inconsistent structures, or sharp details incompatible with their surroundings. Generic NR models may reward familiar sharpness or naturalness cues while missing these artifacts because of a training-domain mismatch. We present this as a hypothesis consistent with the results, not as an experimentally isolated cause.

We will expand the qualitative discussion to distinguish visually coherent departures from GT from visibly inconsistent synthesis. Under NR assessment, the perceptual coherence of generated detail matters alongside its amount.

Generator dependence also appears in the added models: VQ-R1 reaches SRCC 0.632 on PassionSR but 0.159 on DiT4SR. DiT4SR nevertheless has split-half human Spearman agreement of 0.814. Thus, low metric correlation coexists with reproducible human rankings. Different artifact mixtures plausibly contribute, although attributing this variation to particular generator components requires further controlled study.

W4 / Q3 — Adaptation, the remaining bottleneck, and general IQA.

DISRQAD's main contribution is a benchmark for diagnosing existing IQA methods; the student is an adaptation case study. In the updated evaluation, Q-ReAlign-mini improves from diffusion SRCC 0.345 to 0.494 after direct fine-tuning, and the distilled student reaches 0.531. Other methods exhibit mixed outcomes, including improved diffusion alignment accompanied by reduced classic-SR performance.

A plausible explanation is that limited SR-specific supervision shifts the learned quality cues, reducing transfer to other distortion domains. This is consistent with the non-SR losses in Section 5.2, but does not show that specialization inevitably harms general IQA. Replay and anchoring address this trade-off in the student; the current comparisons do not isolate every component's contribution.

The remaining gap is not readily explained solely by annotation noise: diffusion split-half Spearman agreement is 0.862. This is not an exact human ceiling for metric–MOS correlation, but it indicates substantial scope for better prediction. Reliable annotations, generator-specific diagnostics, and the mixed adaptation outcomes make DISRQAD useful for investigating that bottleneck without claiming that the student solves it.

Reviewer_yvGsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DISRQAD, a subjective quality assessment benchmark containing 14,000 outputs from ten diffusion-based and four non-diffusion super-resolution methods. The dataset covers four input degradation conditions and two upscaling factors, with 205,578 human quality judgments. Its annotation target is perceived output quality, assessed without showing the low-resolution input or high-resolution reference. Evaluation of 51 standard metric configurations and 11 adapted variants reveals substantially weaker agreement with human scores on diffusion outputs, while a pruned and distilled quality predictor provides an additional adaptation case study.

Critical strengths
  1. The dataset represents a substantial annotation effort across a broad set of SR generators.
  2. The benchmark evaluates many metric families and provides useful generator- and condition-specific analyses.
Critical weaknesses
  1. Metric-dependent sampling may influence the benchmark conclusions. Source selection uses Q-Align thresholds, while output selection incorporates Q-Align and TOPIQ-FR scores before these metrics are evaluated. This does not establish that the resulting rankings are biased in a particular direction, but it creates a dependency between benchmark construction and evaluation. A randomly sampled or metric-independent control subset would help assess the robustness of the conclusions.

  2. The diffusion versus non-diffusion comparison is not sufficiently controlled. The two output families are clustered and sampled separately, so their rated subsets may differ in source content, degradation composition, and MOS distribution. Differences in quality range can themselves affect correlation coefficients. Table 4’s within-generator analyses are useful, but matched-source comparisons, quality-range controls, and source-level bootstrap intervals are needed to strengthen the interpretation of the aggregate correlation gap.

  3. Annotation reliability is supported mainly by a limited pilot. The annotation-budget analysis uses only 50 outputs, and the relationship between reduced-rating MOS and the 20-rating baseline is not an independent reliability estimate if ratings overlap. The protocol-selection analysis additionally fits three parameters to five outputs per reference image. The final dataset would benefit from disjoint-rater agreement, per-image uncertainty, and observer-consistency statistics beyond the pilot results.

  4. Source independence across training, validation, and testing is insufficiently documented. Appendix C states that SRIQA-Bench is split at the individual-image level, potentially placing related HR and SR images in both training and validation. For further adaptation on DISRQAD, source-disjointness between the open and private portions is not specified. Because the source pool also includes KonIQ-10K, which is used during student training, an explicit content-overlap audit is important for interpreting generalization.

Questions for authors
  1. How many unique HR sources are represented in the 14,000 rated outputs, and are the relevant training, validation, and private-test partitions source-disjoint?

  2. Does the diffusion versus non-diffusion correlation gap persist under matched source content, degradation conditions, and MOS distributions, with source-level confidence intervals?

  3. Can you report independent rater-group agreement and per-image MOS uncertainty for the final dataset?

1 comment

We thank the reviewer for identifying the controls and documentation needed to strengthen DISRQADl. We address related concerns jointly below and provide additional clarifications and experimental evidence.

W1 — Dependence on TOPIQ/Q-Align sampling.

We annotated an additional 400 classic and 1,000 diffusion SR outputs selected without TOPIQ or Q-Align in output sampling. TOPIQ-FR remains the strongest evaluated FR metric by SRCC on both controls, reaching 0.764 on classic SR and 0.470 on diffusion SR. Q-Align reaches 0.713 and 0.501, respectively. Thus, their strong relative performance persists after removing their scores from output selection.

MetricDiffusion SRDiffusion (w/o TOPIQ && Q-Align)Classic SRClassic (w/o TOPIQ && Q-Align) SR
Q-ReAlign-pro0.4520.5010.7600.684
Q-Align0.4470.5010.8130.713
TOPIQ-FR0.4450.4700.8110.764
Q-ReAlign-lite0.4270.4650.7720.698
FSIM0.4230.4260.7580.723
WaDIQaM-FR0.4090.4190.7150.672
VSI0.3910.3930.7650.720
VIF0.3840.4090.6870.677
NIMA0.3630.3760.7020.617
GMSD0.3610.3750.7650.727
DMM0.3580.3820.7390.682
MANIQA0.3570.3840.6290.579
CKDN0.3560.4090.0930.123
AFINE (FR)0.3540.3900.7870.734
Q-ReAlign-mini0.3450.3550.6830.576
MS-SSIM0.3440.3670.7250.710
LPIPS-VGG0.3380.4000.7630.738
NLPD0.3280.3470.6780.677
AHIQ0.3270.3680.6400.602
LPIPS-AlexNet0.3140.3710.7230.689
MAD0.2940.2990.6850.629
SSIM0.2910.3370.5970.617
TOPIQ-NR0.2880.3380.6680.582
ERQA0.2830.3210.4930.506
PSNR0.2800.2850.6300.625
MUSIQ0.2770.3420.6050.529
DISTS0.2630.3330.7510.682
DBCNN0.2490.2770.6420.555
NeuralSBS0.2490.227-0.184-0.079
PieAPP0.2380.2800.5670.524
HyperIQA0.2380.2950.6880.563
CW-SSIM0.2250.2460.5040.506
CNNIQA0.1840.1600.5660.468
TReS0.1730.2270.5670.447
PaQ-2-PiQ0.1580.2260.4010.267
WaDIQaM (NR)0.1510.1670.5580.446
ARNIQA0.1100.1710.5330.413
AFINE (NR)0.0190.0740.6230.521
BRISQUE0.014-0.0580.3700.210
PIQE-0.068-0.0960.3800.179
NRQM-0.076-0.0890.4030.290
NIQE-0.085-0.1070.3600.119
PI-0.106-0.1330.4190.191
ILNIQE-0.237-0.2840.3960.198

More broadly, diffusion SRCC is lower for 42 of the 44 configurations evaluated on both sets. This supports the robustness of the headline finding to the output-sampling criteria.

W2 / Q2 — Content, degradation, MOS range, and correlation uncertainty.

The correlation decrease is reproduced in both pairs of annotated datasets: diffusion SRCC is lower for 67 of 69 configurations with TOPIQ-FR/Q-Align-based output sampling and for 42 of 44 without it. Thus, the finding persists after removing these scores from output selection.

The pattern also extends to stratified analyses. In the original tracks, diffusion SRCC is lower for 67–68 of 69 configurations within each degradation condition and for 64 of 69 at each upscaling factor.

W3 / Q3 — Reliability beyond the pilot and MOS uncertainty.

We evaluated the final annotations on both main tracks and both sampling controls. For classic and diffusion SR, aggregate ICC(1,k effective) is 0.945 and 0.926, and split-half Spearman is 0.870 and 0.862. The diffusion ICC interval is [0.924, 0.928]. Aggregate ICC remains above 0.90 on both controls.

Reliability is also substantial within diffusion subdomains: aggregate ICC ranges from 0.879 to 0.937 across generators and remains high across degradations and scales. Thus, reproducible MOS rankings are not merely a consequence of pooling different generators.

W4 / Q1 — Unique HR sources and partition indepedence.

The 14,000 rated outputs represent 7,326 distinct HR sources: 6,341 in the diffusion track and 3,221 in the classic track.

DISRQAD is partitioned by HR source, with no source shared between its open and private portions. This source-disjoint partitioning is already documented in the repository linked in the paper.

For SRIQA-Bench adaptation, the training/validation split is performed at the individual-image level, so HR images and SR outputs derived from the same source may occur in both partitions. Test evaluation, however, uses an independent set of HR sources. The main adapted baselines are evaluated on DISRQAD without training on its MOS; further adaptation on open DISRQAD and evaluation on its source-disjoint private portion are reported separately.

Reviewer_wLjuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the DISRQAD dataset, which comprises 14,000 super-resolution (SR) results paired with subjective human quality ratings. Systematic evaluation reveals that existing image quality assessment (IQA) metrics exhibit a significant decline in alignment with human judgment when applied to diffusion-based super-resolution outputs. To address this, the authors developed a lightweight, no-reference IQA model based on Q-ReAlign-mini; this model achieves an improvement in SRCC with human ratings—from 0.341 to 0.496.

Critical strengths

The paper is clearly motivated and addresses an important problem, highlighting a major out-of-distribution generalization challenge for current IQA models in diffusion-based super-resolution.

Critical weaknesses

The ranking reliability of the subjective annotations requires further validation. Despite the small-scale pilot study, it remains unclear whether absolute ratings adequately capture relative quality differences. Larger-scale pairwise comparisons, similar to those used in AFINE, could provide stronger evidence of ranking consistency. AFINE: Toward Generalized Image Quality Assessment

The paper presents an important observation, but does not sufficiently establish a solution that performs reliably on both conventional and diffusion-based SR. A controlled comparison with models such as Q-Align under the same data and training conditions is needed to determine whether the gains stem from the proposed design or domain adaptation itself.

Questions for authors

Do the final labeled subsets used for diffusion-based SR and traditional SR correspond to the same LR inputs? The appendix mentions that the two sets of outputs were selected via clustering, but it does not explicitly state whether the chosen samples are identical. If the input samples differ, the observed disparity in correlation could be influenced by variations in image content, the SR method, and the degradation conditions.

1 comment

We thank the reviewer for highlighting annotation reliability, controlled adaptation, and comparability between SR tracks. Our additional experiments strengthen the benchmark's central finding while clarifying its scope. W and Q follow the order of the review.

W1 — Reliability of absolute ratings and pairwise validation.

We have evaluated reliability on the final annotated data rather than relying only on the protocol-selection pilot. For classic and diffusion SR, aggregate ICC(1,k effective) is 0.945 and 0.926, respectively; split-half Spearman correlation is 0.870 and 0.862. The diffusion ICC has a 95% interval of [0.924, 0.928].

The additional classic and diffusion subsets sampled without TOPIQ/Q-Align also show reliable MOS: aggregate ICC is 0.936 and 0.902, and split-half Spearman is 0.858 and 0.813. Reliability extends to diffusion subdomains: aggregate ICC ranges from 0.879 to 0.937 across generators and exceeds 0.90 for every degradation condition and both upscaling factors. Thus, reproducibility is present in both sampling settings and within subgroups, rather than arising only from pooling generators.

Individual disagreement remains: diffusion ordinal Krippendorff alpha is 0.469. This is compatible with reliable aggregate MOS because averaging reduces individual rating noise. We report individual agreement and MOS reliability separately; full results will be included in the supplementary material.

Appendix B already compares NR ratings, FR ratings, and pairwise preferences on 50 outputs. This pilot motivates the protocol but does not establish equivalence. A larger pairwise study would require substantial additional annotation: within-source comparisons yield local rankings requiring links across sources, whereas cross-content comparisons can introduce content preferences. The full-dataset reliability analysis provides direct evidence supporting the reproducibility of the chosen MOS target.

W2 — Contribution and controlled adaptation.

DISRQAD's primary contribution is a dataset and diagnostic benchmark for existing IQA methods. The student is an efficiency and adaptation case study. In the updated evaluation, unadapted Q-ReAlign-mini, direct fine-tuning, and the distilled student achieve diffusion SRCCs of 0.345, 0.494, and 0.531. Direct adaptation thus accounts for much of the gain; the student further improves alignment while reducing the model from 0.8B to 0.34B parameters.

These results support effective adaptation of a compact predictor, but do not isolate every training component or constitute a matched-training comparison with Q-Align. We will emphasize this distinction and avoid claiming a universally superior architecture.

Q1 — Whether the rated tracks use identical LR inputs.

The generation pool uses common sources, degradation recipes, and scales, but the final rated tracks were sampled separately and are not identical LR sets. They represent 6,341 diffusion and 3,221 classicl HR sources, with 7,326 distinct sources in their union. We will distinguish these counts from the larger generation pool reported in the manuscript.

We additionally annotated 1,000 diffusion and 400 classic outputs sampled without TOPIQ or Q-Align. Lower diffusion SRCC is reproduced in both pairs of datasets: for 67 of 69 configurations in the original tracks and 42 of 44 in the controls. The broad finding therefore persists after removing these metrics from output selection.

Stratified analyses reinforce this result. In the original tracks, diffusion SRCC is lower for 67–68 of 69 configurations within each degradation condition and 64 of 69 at each scale. In the controls, the decrease persists for 39–42 of 44 configurations within noise, JPEG, resized inputs, and both scales; the blur control is an exception. Generator-specific analyses also identify difficult diffusion outputs in both settings: TOPIQ-FR's within-DiT4SR SRCC is 0.205 and 0.127, respectively.

Together, these results support a broad assessment gap across both sampling settings and within input-condition, scale, and SR-method subsets. The tracks are not fully matched in source content or MOS distribution, so we interpret the result as a robust empirical pattern rather than an isolated causal effect of diffusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.