- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces FAIM, a super-resolution fidelity metric that does not need an HR reference. It judges whether the SR output contradicts its LR input. First, the LR image is upsampled with bicubic interpolation. Then, a frozen DINOv3 extracts multi-layer features from both the upsampled LR and the SR images, and small learned modules score the differences between them through two branches (a local branch and a region-level branch). The final score is the negative sum of both branches, and their maps can be used for artifact localization. The authors also propose a benchmark, FAIM-Bench, with 465 LR images from five real-world SR datasets, each super-resolved with eight diffusion-based models (3,720 LR–SR pairs). Twelve annotators rated LR quality, SR quality, and LR–SR fidelity, with at least eight judgments per pair.
Critical strengths
- Well-posed problem.
- The benchmark, although small, could be useful for the community.
Critical weaknesses
- Formulation of the problem. The paper is motivated by the lack of GT data in real-world SR. However, FAIM is trained on the labels of its own benchmark and all fidelity results are reported on it, where FAIM is the only method trained in-domain. Holding out SR methods helps, but the annotators and the protocol remain the same. Evaluating FAIM on independently collected human judgments, and on non-diffusion SR methods, would show whether it generalizes beyond this benchmark.
- Missing related work and baselines. The authors miss two important works that, like FAIM, do not assume that the reference is perfect: A-FINE [1] and RQI [2]. I think both should be evaluated on FAIM-Bench. In addition, the Hallucination Score [3], although a preprint, also targets hallucinations in generative SR and should be discussed in the related work.
- Benchmark size. The benchmark is a central part of the paper, as FAIM is both trained and evaluated on it. However, it is small, 465 LR images (only 149 used for training), 12 annotators, and at least eight judgments per pair, below the 15 observers usually recommended for subjective studies (ITU-R BT.500). Together with the low agreement between annotators (Krippendorff's alpha of 0.307), I am not sure that solid conclusions can be drawn from it.
- Effectiveness. FAIM reaches an average SRCC of 0.458. In Table 2, several components barely help (e.g., RCCB alone obtains 0.452 versus 0.458 for the full model). A noise ceiling for the human scores would help to judge whether 0.458 is a good result.
[1] D. Chen et al. Toward Generalized Image Quality Assessment: Relaxing the Perfect Reference Quality Assumption. CVPR 2025.
[2] S. Su. et al. Bridging the Perception Gap in Image Super-Resolution Evaluation. CVPR 2026.
[3] W. Ren et al. Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution. Arxiv 2025.
Questions for authors
- Why do 6 different ablation variants nearly identical numbers? Given Table 2, do most of the modules actually contribute to the final performance?
- How is the localization of FAIM evaluated? Fig. 5 is only qualitative, FAIM-Bench has no artifact annotations, and on SR-Prominence FAIM is below the baselines in pixel-level metrics.
8 comments
Response to the Review (2/4)
W2: Missing Related Work and Baseline Comparisons
R: We have expanded Section 2.2 (Image Quality Assessment) to discuss A-FINE, RQI, and Hallucination Score, and additionally included RefReward-SR in our related work and baseline comparisons.
Specifically, A-FINE, RQI, and RefReward-SR have been incorporated into the FAIM-Bench evaluation under the same protocols as other metrics. Hallucination Score is evaluated on the separate 20-image RealLQ250 evaluation set in Appendix G.1 using gpt-6.1-sol, rather than on the full FAIM-Bench, due to the substantial computational cost of MLLM-based evaluation.
The corresponding revisions are located at:
- Related Work: Section 2.2 (Image Quality Assessment).
- Main Experiments: Sections 5.1–5.2 and Table 1.
- Additional Analyses: Appendix G.1, Tables 18–19 (external evaluation); Appendix G.2, Table 21 (known-method evaluation); Appendix G.3, Table 22 (ground-truth HR assessment); and Appendix G.8, Table 37 (computational costs).
W3: Benchmark Size and Annotation Reliability
R: We acknowledge that FAIM-Bench has a relatively limited number of LR images and annotators. While ITU-R BT.500 generally recommends at least 15 observers, our study involves 12 annotators, with at least eight valid judgments per LR–SR pair. We recognize this as a limitation. To further assess the reliability of the collected annotations, we conducted additional agreement and reproducibility analyses.
Importantly, the originally reported Krippendorff's α = 0.307 reflects agreement on individual absolute ratings rather than the final aggregated fidelity labels. As described in Sec. 3.2 and Appendix B.3, our two-stage annotation protocol combines five-level absolute fidelity ratings (Stage I) with relative rankings of eight SR outputs sharing the same LR input (Stage II). Following Ye and Doermann [5], both sources are jointly aggregated into the final latent fidelity scores. We therefore assess annotation reliability at three levels:
-
For absolute ratings, we identified and corrected a calculation error that excluded the lowest valid rating category. The corrected interval Krippendorff's is 0.359, compared with the previously reported 0.307.
-
For relative rankings, Stage II achieves a Kendall's τ-b of 0.312, measuring rank-order consistency between annotators, and a pairwise ordering agreement of 65.6%, indicating that annotators agree on which SR output is better in 65.6% of pairwise comparisons.
-
For the final aggregated fidelity labels, we conducted a split-half analysis across all 465 LR images. We independently reconstructed the hybrid fidelity scores from two non-overlapping groups of six annotators across 184 valid splits. The median macro-averaged SRCC between the two groups' scores is 0.571. Across all 13,020 within-LR output pairs, the median ordering agreement reaches 77.1%, exceeding the 63.0% obtained using absolute ratings alone under identical splits.
These results demonstrate that, despite variability in individual judgments, the final aggregated fidelity labels retain reproducible ranking information across annotator groups. We have corrected Table 4 and expanded Appendix B.4 to report the detailed agreement and reproducibility analyses.
Response to the Review (3/4)
W4&Q1: Architectural Contributions and Human-Score Noise Ceiling
R: We provide additional statistical analyses to clarify the contributions of individual components and examine the human-score noise ceiling to help interpret FAIM's performance.
(1) Architectural Contributions.
FAIM integrates multi-level features, spatial discrepancies, and region-level contextual reasoning. The ablations show that different components contribute to different extents.
-
Component effectiveness. We ran paired bootstrap analyses with 9,999 resamples using the predictions underlying main-text Table 2 and applied Holm correction to the resulting p-values. Cross-Attention significantly improves SRCC (Δ = +0.0687, p = 0.0014), while multi-layer DINOv3 features significantly improve PLCC-log (Δ = +0.0625, p = 0.0221). Additional Appendix G.6 ablations under the known-method protocol support Spatial Discrepancy Gate, Geometry, and Contextual Contradiction (ΔSRCC = +0.0605, +0.0308, +0.0168; adjusted p = 0.014 for each).
-
Overall architectural benefit. Across three test datasets (RealDeg-test, RealSR, and RealPhoto60) and two evaluation protocols (leave-one-SR-method-out and known-method LR-held-out), Full outperforms the stronger single-branch variant by up to 0.035 SRCC, while never underperforming it by more than 0.010 SRCC. In contrast, RCCB-only and SCB-only exhibit larger performance gaps of up to 0.051 and 0.113 SRCC, respectively. These results highlight the complementary strengths of SCB and RCCB, with their combination providing more consistent performance across datasets and evaluation settings.
-
Computational efficiency. Table 4 compares the performance and computational cost of the architectural variants under the known-method LR-held-out protocol. Full FAIM adds at most 0.023 MB of model storage over the ablated variants, with latency increases of 0.025–0.328 s per image pair.
Table 4. Accuracy and computational cost of architectural ablations. The parameter and storage columns exclude the shared frozen DINOv3 backbone (Appendix G.6, Table 34).
| Variant | Avg. SRCC / PLCC-log ↑ | FAIM parameters (M) ↓ | FAIM model size (MB) ↓ | GMACs ↓ | Latency (s) ↓ | Peak GPU memory (GB) ↓ |
|---|---|---|---|---|---|---|
| Full FAIM | 0.719 / 0.766 | 0.046 | 0.175 | 1410.282 | 0.943 | 1.520 |
| SCB-only | 0.699 / 0.746 | 0.046 | 0.175 | 1399.403 | 0.615 | 1.520 |
| RCCB-only | 0.684 / 0.732 | 0.046 | 0.175 | 1410.263 | 0.918 | 1.520 |
| DINOv3 layer 24 only | 0.564 / 0.616 | 0.040 | 0.152 | 1409.811 | 0.856 | 1.464 |
| w/o Cross-Attention | 0.697 / 0.754 | 0.046 | 0.175 | 1410.282 | 0.852 | 1.520 |
| w/o Spatial Discrepancy Gate | 0.658 / 0.713 | 0.046 | 0.175 | 1410.282 | 0.825 | 1.520 |
| w/o Geometry | 0.688 / 0.730 | 0.046 | 0.175 | 1410.282 | 0.886 | 1.520 |
| w/o Contextual Contradiction | 0.702 / 0.744 | 0.046 | 0.175 | 1399.408 | 0.616 | 1.520 |
Together, these results support the full architecture as a balanced design that combines effective components with consistent performance across evaluation settings.
(2) Human-Score Noise Ceiling.
A noise ceiling refers to the highest agreement with human scores that a model can be expected to achieve given variability in human judgments. To examine this variability, we assessed the reproducibility of independently aggregated human scores through a split-half analysis (Appendix B.4).
We divided the 12 annotators into two non-overlapping groups of six and retained 184 valid splits, with at least four absolute ratings per LR–SR pair in each group. For each split, we independently reconstructed each group's hybrid fidelity scores from its absolute ratings and relative rankings over all 465 LR images. On the fixed test set of 279 LR images, we computed human–human SRCC separately for each dataset and SR method, then averaged equally over methods and datasets, following the aggregation protocol in main-text Table 2.
The median human–human SRCC across these splits was 0.575, providing an empirical reference for FAIM's average SRCC of 0.458. This result shows that the aggregated human scores retain reproducible fidelity information while remaining affected by annotation variability. However, 0.575 is not a strict upper bound on FAIM's performance: the split-half analysis compares two smaller annotator groups, whereas FAIM is evaluated against the full consensus.
Response to the Review (1/4)
We thank the reviewer for the constructive feedback. In response, we have added an evaluation using newly collected human judgments on images outside FAIM-Bench, including non-diffusion SR methods (Appendix G.1); expanded the discussion of A-FINE, RQI, and Hallucination Score and added the requested A-FINE and RQI comparisons (Section 2.2, Section 5.2, and Appendix G.2); expanded the annotation-agreement and split-half reproducibility analyses (Appendix B.4); and expanded the quantitative artifact-localization evaluation and the analysis of architectural contributions (Appendices G.5–G.6). All revisions to the manuscript are highlighted in blue. Below, we provide point-by-point responses to the weaknesses and questions raised in your review.
R: To assess generalization beyond diffusion-based SR and ×4 upscaling, we evaluated FAIM on 20 RealLQ250 LR images disjoint from FAIM-Bench. This evaluation covered six methods from three SR families: regression-based (RCAN), GAN-based (MambaIR_real, BSRGAN, and FeMaSR), and diffusion-based (DreamClear and SeeSR). We tested ×4 and ×16, with seven annotators evaluating 120 LR–SR pairs per scale (840 fidelity ratings). Human fidelity scores combine absolute ratings (Stage I) and relative rankings (Stage II). Full details are provided in Appendix G.1.
We compare FAIM with 11 IQA metrics (four FR-IQA and seven NR-IQA) and four additional methods:
(1) Overall performance across scales.
Table 1. SRCC/PLCC-log comparison of ILR–SR proxy-reference metrics (Appendix G.1, Tables 18–19).
Table 2. SRCC/PLCC-log comparison of NR-IQA metrics (Appendix G.1, Tables 18–19).
As shown in Tables 1 and 2, at ×4, FAIM achieves the highest SRCC of 0.638, exceeding the strongest baseline, Hallucination Score (0.484), by 0.154. Its PLCC-log of 0.664 closely matches the best result of 0.668 from RefReward-SR. At ×16, FAIM ranks first in both SRCC (0.757) and PLCC-log (0.763), outperforming the strongest baselines by 0.099 and 0.077, respectively. These results demonstrate strong overall agreement with human fidelity judgments across both upscaling factors.
(2) Generalization across SR families.
We further examine FAIM's performance across different SR families and training exposure. Correlations are computed across 20 images per method and averaged within each group. Brackets show 95% CIs from 5,000 bootstrap resamples of LR images.
Table 3. FAIM correlations across SR families (Appendix G.1, Table 20).
FAIM achieves positive correlations on unseen GAN-based methods at both scales (95% CIs above zero). Across all five unseen methods, average correlations are positive in both metrics at ×4 and in PLCC-log at ×16. Together, these findings support FAIM’s applicability to unseen SR methods beyond diffusion-based SR.
In summary, FAIM leads in overall SRCC at both scales and PLCC-log at ×16, with positive correlations on unseen GAN-based methods supporting generalization beyond diffusion-based SR. Details are provided in Appendix G.1, Tables 18–20.