- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a score-recovery-based defense method against adversarial attacks on NR-IQA models. It first estimates the distributions of quality score changes induced by different input transformations on clean images. These distributions are then modeled using Gaussian kernels to construct a score agreement function. Finally, the recovered quality score is obtained by maximizing this agreement function.
Critical strengths
-
The idea of score recovery is interesting and has potential practical value. Despite the limited experimental results, the proposed method appears to be effective.
-
The authors provide a theoretical analysis to explain the underlying principles and effectiveness of the proposed score recovery method, though it is hard to follow for me.
Critical weaknesses
-
The theoretical analysis is difficult to follow due to confusing notation and insufficient explanations of key mathematical concepts. For instance, MMD which plays a central role in the analysis, is not explicitly defined in the main manuscript, making the theoretical results harder to understand.
-
The paper only reports performance averaged across different NR-IQA models and attack methods, making it difficult to identify how individual models respond to different attacks and defenses. Such detailed results are important for understanding the effectiveness and limitations of the proposed method under different settings. Moreover, the lack of model-specific and attack-specific results makes it difficult to reproduce and verify the reported findings.
-
The proposed score recovery method relies on score-shift distributions calibrated on clean images. Although the theoretical analysis accounts for image-specific mismatch and residual attack effects, the recovery guarantee depends on the conditions in Eq. (10) and Eq. (11). The experiments do not directly assess whether this condition holds in practice. This is particularly relevant because calibration uses PaQ-2-PiQ, whereas evaluation uses KonIQ-10k and KADID-10k.
Questions for authors
-
The average clean SROCC of the undefended NR-IQA models is only 0.624, which is considerably lower than the performance typically reported for these models on KonIQ-10k and KADID-10k. For instance, even LinearityIQA, one of the earliest models considered in this paper, achieves an SROCC of over 0.93 on KonIQ-10k, as reported in its original paper. Could the authors provide per-model, per-dataset clean SROCC results and clarify the evaluation protocol, including dataset splits, model checkpoints, and aggregation procedure?
-
The recovered score is obtained through a global grid search. What is the step size of the search grid, and how sensitive are the final results to this choice?
I would like to reconsider the rating based on the authors’ rebuttal.
2 comments
After Rebuttal
Thank the authors for the detailed responses and additional results. Most of my concerns have been adequately addressed. However, I believe the paper still has some limitations. First, as the authors acknowledge, the theoretical conditions may not necessarily hold in practice, which limits the practical applicability of the theoretical guarantees. Second, although the authors have improved the presentation of the theoretical analysis, I have not been able to thoroughly verify the correctness of the theoretical results due to time constraints.
Overall, I consider this paper to be borderline, leaning slightly toward acceptance. Since the review system does not provide a borderline option, I have decided to increase my score to weak accept.
Rebuttal for Reviewer ygpP
Thank you for your helpful comments on the evaluation protocol and theoretical analysis. We address several shared concerns in our general comment and address your specific questions below.
Q1: Clean SROCC and model checkpoints. We made the training protocol more clear (Section 5.1, Appendix D.1). All nine models were retrained on TID2013 dataset for the fair evaluation, since the original checkpoints of several metrics (e.g., KonCept, Linearity) are trained on KonIQ-10k. Our results follow the same pattern as published TID2013-trained models, which reach 0.40–0.77 on KADID-10k, which shares TID2013's synthetic distortions, and 0.18–0.52 on KonIQ-10k, which has authentic distortions [1-3].
Table 9 now reports clean SROCC for each model on the evaluation subsets:
Each dataset contains 1,000 evaluation images. SROCC is computed separately for each model and dataset and then averaged.
Q2: Grid spacing. We provide more details in Tables 5 and 6. The default spacing is g=0.2h, with per-model values in Table 5. Table 6 varies g/h from 0.05 to 0.8 on the same attacked images. Clean recovery error changes only from 7.91 to 8.01, and white-box error from 11.12 to 11.26; both are unchanged to two decimal places for g/h≤0.2.
W1: Readability of the theory. Following your suggestion, we now explicitly define MMD in Section 4.1 and explain its role in the analysis. Intuitively, we ask whether the calibrated distributions allow us to distinguish the original score from an incorrect candidate. The margin measures this separation: a larger margin makes recovery more reliable, while calibration error, limited sampling, image-specific mismatch, residual attack effects, and numerical approximation make it harder.
W2: Detailed results. We include Tables 11–14 in Appendix in the revised version of the paper, which provide results for each model, attack and dataset separately. The results show that the TransDiff has the lowest Dscore in 81/90 white-box and 47/54 black-box cells. Table 10 additionally reports paired differences against Gaussian shifts with 95% bootstrap confidence intervals.
W3: Calibration transfer and theoretical conditions. We use PaQ-2-PiQ for calibration and separate datasets, KonIQ-10k and KADID-10k, for evaluation. The results show that the method can work despite this dataset mismatch, although they do not directly establish that Eq. (11) holds. The theorem helps explain when recovery is possible and what can cause it to fail. Its bound is too conservative to guarantee recovery with four views, but Figure 3 shows that four views already provide most of the observed benefit. We therefore choose this setting based on its empirical accuracy and computational cost.
We hope these answers and the revision address the reviewer's concerns, and we are happy to clarify further.
[1] W. Zhang, K. Ma, G. Zhai, X. Yang. Learning to Blindly Assess Image Quality in the Laboratory and Wild. IEEE ICIP, pp. 111–115, 2020. doi:10.1109/ICIP40778.2020.9191278 (Table 3)
[2] W. Zhang, K. Ma, G. Zhai, X. Yang. Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild. IEEE Transactions on Image Processing, 30:3474–3486, 2021. doi:10.1109/TIP.2021.3061932 (Table 4)
[3] S. Liu, Q. Mao, C. Li, J. Chen, F. Meng, Y. Tian, Y. Liang. Content-Distortion High-Order Interaction for Blind Image Quality Assessment. arXiv:2504.05076, 2025 (Table 8)