acceptodds
Under review as a conference paper at ICLR 2027

TransDiff: Plug-and-Play Adversarial Defense for No-Reference Image Quality Assessment via Calibrated Score Consensus

Abstract

Adversarial perturbations can substantially alter the predictions of no-reference image quality assessment (NR-IQA) models while remaining nearly imperceptible. Because these models guide image processing and quality-based selection, manipulated scores can compromise downstream decisions without corresponding changes in perceived quality. We introduce TransDiff, a method for estimating a model’s pre-attack score directly from an attacked image. The method evaluates multiple transformed versions of the image to obtain complementary observations related to its original score. Crucially, transformations themselves affect quality predictions. We account for these effects using score-shift distributions calibrated on images without adversarial perturbations, then combine the calibrated observations through distributional agreement. Our theoretical analysis establishes conditions under which this procedure accurately recovers the pre-attack score despite image mismatch, finite calibration and transformation sampling, residual attack effects, and optimization error. TransDiff retains the shape of the calibrated score-shift distributions rather than reducing them to low-order statistics. Experiments on KADID-10k and KonIQ-10k across nine NR-IQA models and white-box and black-box attacks show that TransDiff improves score recovery over test-time image defenses and matched score-recovery baselines. Relative to a matched Gaussian-shift baseline, it reduces aggregate recovery error by 11.5% under white-box attacks and 7.5% under black-box attacks on the nested stress subset. These results demonstrate the value of retaining calibrated score-shift distribution shape for recovering a model’s original prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image quality assessment, adversarial robustness, adversarial defense

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ygpPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a score-recovery-based defense method against adversarial attacks on NR-IQA models. It first estimates the distributions of quality score changes induced by different input transformations on clean images. These distributions are then modeled using Gaussian kernels to construct a score agreement function. Finally, the recovered quality score is obtained by maximizing this agreement function.

Critical strengths
  1. The idea of score recovery is interesting and has potential practical value. Despite the limited experimental results, the proposed method appears to be effective.

  2. The authors provide a theoretical analysis to explain the underlying principles and effectiveness of the proposed score recovery method, though it is hard to follow for me.

Critical weaknesses
  1. The theoretical analysis is difficult to follow due to confusing notation and insufficient explanations of key mathematical concepts. For instance, MMD which plays a central role in the analysis, is not explicitly defined in the main manuscript, making the theoretical results harder to understand.

  2. The paper only reports performance averaged across different NR-IQA models and attack methods, making it difficult to identify how individual models respond to different attacks and defenses. Such detailed results are important for understanding the effectiveness and limitations of the proposed method under different settings. Moreover, the lack of model-specific and attack-specific results makes it difficult to reproduce and verify the reported findings.

  3. The proposed score recovery method relies on score-shift distributions calibrated on clean images. Although the theoretical analysis accounts for image-specific mismatch and residual attack effects, the recovery guarantee depends on the conditions in Eq. (10) and Eq. (11). The experiments do not directly assess whether this condition holds in practice. This is particularly relevant because calibration uses PaQ-2-PiQ, whereas evaluation uses KonIQ-10k and KADID-10k.

Questions for authors
  1. The average clean SROCC of the undefended NR-IQA models is only 0.624, which is considerably lower than the performance typically reported for these models on KonIQ-10k and KADID-10k. For instance, even LinearityIQA, one of the earliest models considered in this paper, achieves an SROCC of over 0.93 on KonIQ-10k, as reported in its original paper. Could the authors provide per-model, per-dataset clean SROCC results and clarify the evaluation protocol, including dataset splits, model checkpoints, and aggregation procedure?

  2. The recovered score is obtained through a global grid search. What is the step size of the search grid, and how sensitive are the final results to this choice?

I would like to reconsider the rating based on the authors’ rebuttal.

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer ygpP

Thank you for your helpful comments on the evaluation protocol and theoretical analysis. We address several shared concerns in our general comment and address your specific questions below.

Q1: Clean SROCC and model checkpoints. We made the training protocol more clear (Section 5.1, Appendix D.1). All nine models were retrained on TID2013 dataset for the fair evaluation, since the original checkpoints of several metrics (e.g., KonCept, Linearity) are trained on KonIQ-10k. Our results follow the same pattern as published TID2013-trained models, which reach 0.40–0.77 on KADID-10k, which shares TID2013's synthetic distortions, and 0.18–0.52 on KonIQ-10k, which has authentic distortions [1-3].

Table 9 now reports clean SROCC for each model on the evaluation subsets:

ModelKonIQ-10kKADID-10k
Meta-IQA0.4520.468
MANIQA0.6030.802
CLIP-IQA+0.7120.768
TOPIQ0.5960.794
KonCept0.5580.686
SPAQ0.5710.641
PAQ2PIQ0.5210.606
Linearity0.5490.708
FPR0.4330.764
Mean0.5550.693

Each dataset contains 1,000 evaluation images. SROCC is computed separately for each model and dataset and then averaged.

Q2: Grid spacing. We provide more details in Tables 5 and 6. The default spacing is g=0.2h, with per-model values in Table 5. Table 6 varies g/h from 0.05 to 0.8 on the same attacked images. Clean recovery error changes only from 7.91 to 8.01, and white-box error from 11.12 to 11.26; both are unchanged to two decimal places for g/h≤0.2g/h \le 0.2.

W1: Readability of the theory. Following your suggestion, we now explicitly define MMD in Section 4.1 and explain its role in the analysis. Intuitively, we ask whether the calibrated distributions allow us to distinguish the original score from an incorrect candidate. The margin measures this separation: a larger margin makes recovery more reliable, while calibration error, limited sampling, image-specific mismatch, residual attack effects, and numerical approximation make it harder.

W2: Detailed results. We include Tables 11–14 in Appendix in the revised version of the paper, which provide results for each model, attack and dataset separately. The results show that the TransDiff has the lowest DscoreD_{score} in 81/90 white-box and 47/54 black-box cells. Table 10 additionally reports paired differences against Gaussian shifts with 95% bootstrap confidence intervals.

W3: Calibration transfer and theoretical conditions. We use PaQ-2-PiQ for calibration and separate datasets, KonIQ-10k and KADID-10k, for evaluation. The results show that the method can work despite this dataset mismatch, although they do not directly establish that Eq. (11) holds. The theorem helps explain when recovery is possible and what can cause it to fail. Its bound is too conservative to guarantee recovery with four views, but Figure 3 shows that four views already provide most of the observed benefit. We therefore choose this setting based on its empirical accuracy and computational cost.

We hope these answers and the revision address the reviewer's concerns, and we are happy to clarify further.

[1] W. Zhang, K. Ma, G. Zhai, X. Yang. Learning to Blindly Assess Image Quality in the Laboratory and Wild. IEEE ICIP, pp. 111–115, 2020. doi:10.1109/ICIP40778.2020.9191278 (Table 3)

[2] W. Zhang, K. Ma, G. Zhai, X. Yang. Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild. IEEE Transactions on Image Processing, 30:3474–3486, 2021. doi:10.1109/TIP.2021.3061932 (Table 4)

[3] S. Liu, Q. Mao, C. Li, J. Chen, F. Meng, Y. Tian, Y. Liang. Content-Distortion High-Order Interaction for Blind Image Quality Assessment. arXiv:2504.05076, 2025 (Table 8)

Reviewer_ygpPOpenReview ↗

After Rebuttal

Thank the authors for the detailed responses and additional results. Most of my concerns have been adequately addressed. However, I believe the paper still has some limitations. First, as the authors acknowledge, the theoretical conditions may not necessarily hold in practice, which limits the practical applicability of the theoretical guarantees. Second, although the authors have improved the presentation of the theoretical analysis, I have not been able to thoroughly verify the correctness of the theoretical results due to time constraints.

Overall, I consider this paper to be borderline, leaning slightly toward acceptance. Since the review system does not provide a borderline option, I have decided to increase my score to weak accept.

Reviewer_pa1eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work targets the adversarial robustness of NR-IQA through pre-attack score recovery rather than reconstructing the original pixel values, using transformations as calibrated measurement channels and aggregating their score-shift distributions via KDE to estimate the original score. Theoretical analysis has been provided for why and when the method succeeds. The experiments are comprehensive, covering a wide range of evaluation settings across nine NR-IQA models, two datasets and white-box and black-box attacks.

Critical strengths
  1. It formulates the defense objective as pre-attack score recovery, instead of recovering the rax pixel values. This formulation offers a fresh perspective on adversarial defense for NR-IQA.

  2. It provides a theoretical analysis of score recoverability and a finite-sample recovery guarantee. The theoretical analysis is well developed, providing insights into the conditions for successful score recovery and explicitly accounting for different sources of estimation error.

Critical weaknesses
  1. Main results in table 1 only report aggregated results across different IQA metrics and attacks. TransDiff may perform significantly better in some settings but show only marginal improvements or even underperform in others. Without detailed results, it is difficult to assess the consistency of performances of TransDiff and baselines. I suggest providing per-model and per-attack results, at least in the appendix.

  2. Though the task formulation is novel, the proposed approach seems like combining exiting tools to NR-IQA task. Moreover, the performance of the Gaussian-shift baseline is already quite close to TransDiff. These relatively modest gains may not be sufficient to demonstrate the significance of modeling full empirical score-shift distributions using KDE. If possible, can authors visualize the actual distributions to emphasize properties such as asymmetry and multimodality and support the motivation.

Questions for authors

My main concerns are listed in the Weaknesses section.

1 comment
AuthorsOpenReview ↗

Rebuttal for reviewer pa1e

We thank the reviewer for thoughtful review. We summarize the main revisions in our general comment and address your specific concerns below.

W1: Consistency across models and attacks. We report results for each model, attack, and dataset in Tables 11–14. Table 10 additionally reports paired differences Dscore(TransDiff)−Dscore(Gaussian)D_{\mathrm{score}}(\mathrm{TransDiff})-D_{\mathrm{score}}(\mathrm{Gaussian}) with 95% bootstrap confidence intervals (for KADID-10k, resampled by reference image):

DatasetWhite-boxBlack-box
KADID-10k−1.33 [−1.51, −1.14]−0.51 [−0.60, −0.42]
KonIQ-10k−1.55 [−1.66, −1.45]−0.59 [−0.68, −0.52]

All model-wise aggregate intervals favor TransDiff. Attack-wise intervals also favor it except for Opt.-UAP on KADID-10k. There are some exceptions: Gaussian shifts is better in 16 out of 144 cases, and the largest gains occur under I-FGSM attack.

W2: Contributions. Our approach addresses one of a key difficulties in NR-IQA: transformations can weaken an attack, but they also change the quality score themselves. We see the contribution of this work in new formulation for score recovery in NR IQA task: accounting for the changes that image transformations introduce, combining their scores to recover clean quality prediction and theoretically analyzing when this recovery succeeds.

Gaussian shifts can be seen as the simplified version of our approach: it uses the same score-recovery method with Gaussian shift distributions. Its strong performance supports the score-recovery idea itself. With all other components held fixed, TransDiff performs better on average for all nine models, although individual settings include exceptions. Retaining the full distribution shape thus provides an additional benefit, while the paper’s broader contribution is calibrated score recovery.

Reviewer_M9kHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies recovery of a frozen no-reference image-quality model's pre-attack score. It calibrates the distribution of score changes induced by each image transformation, then estimates the original score by maximizing agreement between transformed predictions and these distributions. The estimator retains empirical distribution shape through Gaussian kernel smoothing and uses four transformations sampled from a bank of 48 by default. The analysis relates population identification to MMD and separates finite-sample error, image-specific mismatch, residual attack effects, and optimization error. Experiments cover two datasets, nine IQA models, and white-box and black-box attacks, including matched score-recovery baselines.

Critical strengths
  1. Recovering the original IQA score is a well-motivated objective because image purification can change genuine distortions that the metric is intended to measure. The scalar recovery procedure does not require retraining the underlying IQA model.

  2. Table 2 provides a meaningful controlled comparison. Relative to the matched Gaussian baseline, TransDiff reduces Dscore from 12.56 to 11.12 under white-box attacks and from 6.71 to 6.21 under black-box attacks. Matching transformations, calibration data, bandwidth, and aggregation helps isolate the contribution of empirical distribution shape.

  3. The analysis explicitly identifies image-specific and attack-induced shifts shared across transformations as failure modes that additional views cannot remove. The attack-strength and view-budget studies provide further evidence about the method's behavior. The discussion correctly distinguishes score recovery from ranking against human ratings and acknowledges clean-score drift.

Critical weaknesses
  1. The adaptive-attack evaluation needs validation against the implemented estimator. Appendix C.2 uses 10 I-FGSM iterations and eight four-view EOT samples per iteration, with a local implicit derivative of an isolated continuous stationary mode. Appendix B.2 describes grid-based global selection with optional refinement, and the selected mode can change. Please specify how this derivative corresponds to the actual solver and verify on the existing stress subset whether the results remain stable under longer optimization, restarts, and increased EOT sampling.

  2. Essential calibration and optimization settings are absent. Neither Section 5.1 nor Appendix B specifies the calibration sample count, actual per-metric bandwidths and their selection criterion, or the grid spacing and optimization tolerance used in the reported experiments. Appendix B.2 supplies only a symbolic tolerance bound. These choices directly affect distribution shape, mode selection, attack gradients, and cost. The reported 92.16 ms inference time also lacks hardware and batching specifications. Reporting the settings actually used is necessary to assess the central comparisons.

  3. The finite-sample sufficient condition cannot hold at ordinary confidence levels in the default four-view setting. For m=4m = 4 and β=0.025\beta = 0.025, Eq. (9) gives ϵm≈1.86\epsilon_m \approx 1.86. Appendix A.2 implies H(d)≤wh(d)H(d) \leq w_h(d), because the relevant Gaussian-kernel mean has norm at most one. Thus Bn,m≥ϵm>1B_{n,m} \geq \epsilon_m > 1 already prevents the positive-margin condition in Eq. (11), even before the other nonnegative error terms are included. This does not invalidate the theorem or the empirical results, but it limits the claimed finite-sample explanation of default inference. Please state this limitation explicitly and clarify which inference regimes the bound usefully describes.

Questions for authors
  1. How does the attack gradient correspond to the implemented recovery solver? Does inference return a grid point or a refined stationary maximum, and how is the implicit derivative in Appendix C.2 applied to that implementation?

  2. What numerical calibration and solver settings were used in the reported experiments? Please specify the development and calibration split sizes, the per-metric bandwidths and their selection criterion, and the grid spacing or stopping tolerance. A compact table of the actual settings would suffice.

  3. How consistent is the improvement over Gaussian shifts in Table 2? Could you provide per-model and per-attack differences in Dscore, together with paired confidence intervals from the existing evaluations that respect the reference-image grouping in KADID? This would clarify whether the distribution-shape benefit is broadly shared or concentrated in particular settings.

1 comment
AuthorsOpenReview ↗

rebuttal for Reviewer M9kH

We thank the reviewer for their time and feedback on our paper which we find very valuable. We summarize the changes in our general comment, including expanded experimental results and reproducibility details. Below, we also address the reviewers’ specific concerns.

W1 / Q1: Adaptive attacks. With you feedback in mind, we make Appendix C.2 now distinguish the grid-search forward pass from the implicit-gradient surrogate, which is used for backward propagation.

To evaluate TransDiff under stronger attacks we increased the attack budget on Meta-IQA and TOPIQ using 100 images per dataset (Tables 3 and 7 in the revised paper):

Iterations / restarts / EOTKADID: Gaussian / TransDiffKonIQ: Gaussian / TransDiff
10 / 1 / 818.60 / 12.1022.90 / 16.00
100 / 1 / 819.60 / 12.9023.90 / 16.90
100 / 5 / 3220.40 / 13.6024.80 / 17.70

Values in the table are DscoreD_{\mathrm{score}}. Stronger attacks increase both methods’ errors, but TransDiff remains its advantage on the two tested models.

W2 / Q2: Reproducibility. We have added the missing implementation details in Section 5.1 and Table 5. Calibration uses 2,000 images, and bandwidth selection uses a separate set of 500 development images. We also clarified that runtime was measured on an A100 80 GB GPU with batch size 1 and includes both the transformed model evaluations and score recovery.

W3: Four-view guarantee. You are right: with m=4m=4 and β=0.025\beta=0.025, the bound is too loose to satisfy Eq. (11). We see the theorem as supporting analysis rather than a certificate for that configuration. Our experiments show that the four-view configuration works well, but the theorem does not certify this setting. Using the full transformation bank removes the sampling error term, but the other conditions of the theorem still need to hold.

Q3: Consistency. Tables 11–14 provide per-setting results. Table 10 reports paired differences Dscore(TransDiff)−Dscore(Gaussian)D_{\mathrm{score}}(\text{TransDiff})-D_{\mathrm{score}}(\text{Gaussian}) with 95% bootstrap CIs, respecting reference-image groups in KADID:

DatasetWhite-boxBlack-box
KADID−1.33 [−1.51, −1.14]−0.51 [−0.60, −0.42]
KonIQ−1.55 [−1.66, −1.45]−0.59 [−0.68, −0.52]

In this setting, TransDiff is outperforming Gaussian shifts in all four aggregate comparisons. Attack-wise intervals also favor it except for Optimised-UAP on KADID. See the Table 10 for detailed results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

We thank the reviewers and address important shared concerns

We thank all three reviewers for their time and valuable feedback. We revised the paper and added detailed experimental results, stronger adaptive-attack evaluations, and implementation details. We summarize the most important changes:

  • Detailed results and statistical comparisons. Tables 11–14 now report results separately for each NR-IQA model, attack, and dataset. Table 10 provides paired comparisons with Gaussian shifts and 95% bootstrap confidence intervals, with reference-image grouping for KADID-10k. These results show where TransDiff improves recovery and where a Gaussian approximation performs comparably or better.
  • Stronger adaptive attacks. On Meta-IQA and TOPIQ, we increased the attack budget from 10 iterations / 1 restart / 8 EOT samples to 100 / 5 / 32, using 100 images per dataset (Tables 3 and 7). TransDiff retains its advantage under the tested budget increases. Appendix C.2 now explicitly distinguishes the grid-search forward pass from the implicit-gradient surrogate used in the backward pass.
  • Reproducibility and evaluation protocol. Section 5.1 and Table 5 specify calibration and development set sizes, per-model bandwidths, grid settings, and runtime measurement conditions. Table 6 reports sensitivity to grid spacing. We also clarify that all nine models were retrained on TID2013, and Table 9 reports their clean cross-dataset SROCC values.
  • Theoretical interpretation. Section 4.1 now explicitly defines MMD. In our responses, we clarify the distinction between the theorem’s sufficient recovery conditions and the empirical evidence supporting the four-view configuration and calibration transfer across datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.