- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a training-free test-time defense framework for improving the adversarial robustness of CLIP, with two main components: Anchor Quality Score (AQS) and Anchor-Guided Rectification using Compression (ARC). AQS is introduced to systematically evaluate candidate anchors. It characterizes anchor quality using four complementary properties: attack removal, collateral avoidance, clean fidelity, and separability. These four terms are combined through a weighted harmonic mean. The authors evaluate ten transformation families and select JPEG compression with quality 50 as the strongest anchor. Based on this selection, ARC uses the representation of the JPEG-compressed input as the anchor and performs test-time correction by attracting the perturbed representation toward the anchor while repelling it from the original adversarial representation. A soft anchor-consistency score further scales the correction budget adaptively.
Critical strengths
- The paper is clearly written and presents a coherent and easy-to-follow story. It identifies anchor selection as an important factor in test-time adversarial rectification, then naturally develops the AQS criterion for anchor evaluation and the ARC framework for defense, resulting in a complete methodology.
- Reasonable defense design. The proposed approach is technically reasonable. AQS provides a systematic strategy to evaluate and select an effective anchor based on multiple complementary properties, while ARC uses the selected anchor to guide test-time correction.
Critical weaknesses
- The paper lacks a sufficiently thorough comparison with DBD [1], which appears closely related to ARC in mechanism. Both methods leverage transformed views as references for recovering adversarial representations and use transformation-induced consistency or similarity signals to determine subsequent correction. Given the existence of DBD, the incremental contribution of ARC is therefore not yet sufficiently clear. A direct comparison with DBD under matched datasets, backbones, attack settings, and especially adaptive attacks would be important. The paper should also more explicitly articulate what new capability or insight ARC provides beyond DBD.
- The analysis of AQS is still limited relative to its central role in the paper. From both the title and the overall narrative, AQS is one of the main contributions, yet its robustness and generality are not fully established. It remains unclear whether AQS produces consistent anchor rankings across different attack types, attack strengths, backbones, and datasets, and whether JPEG with remains the optimal choice under these variations. It would also be valuable to investigate whether combinations of transformations can yield higher AQS values than any single transformation, and whether changing the relative weights of the four AQS components leads to different optimal anchors. Such analyses would substantially strengthen the claim that AQS provides a general and reliable criterion for anchor selection.
[1] Liangsheng Liu, Si Chen, Jiamin Wu, Weiwei Feng, Zhixin Cheng, Xiaotian Yin, Wenfei Yang, and Tianzhu Zhang. Adversarial attacks already tell the answer: Directional biasguided test-time defense for vision-language models. In The Fourteenth International Conference on Learning Representations, 2026.
Questions for authors
- Could the authors clarify how the hyperparameters involved in Eq. (11), including the consistency threshold, temperature, and gating threshold, are selected? Since these parameters directly control whether and to what extent rectification is applied, please provide a sensitivity analysis to demonstrate the robustness of ARC with respect to these choices.
- Could the authors evaluate a simpler variant that removes the optimization in Eq. (12)? Specifically, after using Eq. (11) to determine whether an input should be rectified, the JPEG50-transformed image could be directly used as the corrected input instead of optimizing the additional correction perturbation. This comparison would help clarify how much of ARC’s performance gain comes from the selected JPEG50 anchor itself and how much comes from the optimization-based rectification in Eq. (12).
- Could the authors provide a more comprehensive robustness evaluation for the stronger and adaptive attacks reported in Table 3? Currently, Table 3 only evaluates three datasets, whereas Table 1 covers ten datasets. It would be helpful to report the corresponding results across all ten datasets under the three attack settings for a more complete comparison. In addition, please consider including representative black-box attacks, such as Square Attack, as well as low-frequency attack [1], to further evaluate whether ARC remains effective against attacks with substantially different perturbation characteristics.
- Could the authors clarify how the feature-space analysis in Theorem 1 and Corollary 1 relates to the actual rectification procedure used in ARC? The theoretical results analyze a linear interpolation between the adversarial feature and the anchor feature, whereas ARC performs a constrained optimization in the input space through the nonlinear CLIP encoder, as formulated in Eq. (12). It would be helpful to explain under what assumptions the feature-space analysis can justify the actual pixel-space rectification, and, if possible, provide empirical evidence showing that anchors favored by the theoretical criteria indeed induce more reliable correction trajectories toward the clean representation.
[1] Guo, C., Frank, J.S., Weinberger, K.Q., 2018. Low frequency adversarial perturbation. arXiv preprint arXiv:1809.08758 .
8 comments
Response 2/5 to Reviewer skmU
Response to W2.
We have added several of these analyses. Below we address each factor, including where the answer is qualified or where we do not yet have results.
Table R2: Rank by AQS vs. measured rank of the full defense, for the ten family representatives. Held-out: AQS computed on Stanford Cars, FGVC-Aircraft and UCF101; defense measured on Food101 and ImageNet (none used for anchor selection). AA→AA: AQS and defense both under AutoAttack. PGD→AA: AQS from PGD only, defense measured under AutoAttack (AQS not recomputed).
| Anchor | AQS rank (AA) | AQS rank (PGD) | Defense rank under AA | Measured AA mean rank |
|---|---|---|---|---|
| JPEG50 | 1 | 1 | 1 | 2.9 |
| Crop | 2 | 2 | 2 | 3.3 |
| Block quantization | 3 | 4 | 3 | 4.0 |
| Low-pass | 4 | 3 | 6 | 5.4 |
| Gaussian noise | 5 | 6 | 5 | 5.1 |
| Down-up | 6 | 5 | 4 | 5.1 |
| TV minimization | 7 | 7 | 7 | 6.9 |
| Median | 8 | 8 | 8 | 7.1 |
| Bit-depth | 9 | 9 | 9 | 8.1 |
| Color normalization | 10 | 10 | 10 | 10.0 |
The corresponding Figure 12 in Appendix A.12 is available in the revised version.
Attack types. AQS computed under AutoAttack agrees with performance measured under AutoAttack at ρ = 0.95. A ranking computed only from PGD and compared with AutoAttack-measured performance reaches ρ = 0.93 (Table R2, PGD→AA). JPEG50 ranks first in both.
Datasets. On the five benchmarks not used for selection, the AQS ranking and the measured ranking agree with ρ = 0.96, with the weights fixed (Table R5, held-out). JPEG50 ranks first and Crop second under both.
Attack strength. The cell-wise AQS in Figure 5 shows that the mean rank depends on the budget. At ε = 1/255 Crop leads JPEG50 (mean rank 1.9 vs. 2.3); at ε = 4/255 JPEG50 leads Crop (2.0 vs. 3.0). Within the JPEG family, q = 75 scores highest in four of the five ε = 1/255 cells, while q = 50 is best or tied-best in the other six (Appendix A.7). So q = 50 is not uniformly optimal. It is the most consistent choice across the grid, with the highest worst-cell score (0.68, shared with q = 30), and the weaker the attack, the milder the optimal compression. We will state this explicitly.
Backbones. AQS was computed on ViT-B/16 only. As indirect evidence, JPEG50, chosen on ViT-B/16, gives ARC the best average robust accuracy on ViT-B/32 and widens its margin over CSR from 6.8 to 14.3 points (Tables 1 and 2). This supports transfer of the choice, but it is not a test of AQS on a second backbone.
Combinations of transformations. We have not evaluated these. AQS is defined only on the anchor map , so it applies to composed anchors without modification. We list composed anchors as future work in Section 5.
Relative weights. Changing the weights can change the top anchor. Sweeping from 0.5 to 8 (Table R3), the top pick is JPEG50 for and Crop for . Spearman correlation with the measured ranking stays at least 0.93 throughout. With equal weights, AQS places Crop first and JPEG50 second, at ρ = 0.96 (Table R4). Leaving out clean fidelity, separability or collateral keeps JPEG50 first. Leaving out removal does not, and gives the weakest agreement (ρ = 0.86).
Table R3: Sensitivity to the removal weight , with the other three weights held at 1. Rank correlation between AQS and measured defense rank, and the anchor AQS ranks first. Bold = paper setting.
| Removal weight | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 4 | 5 | 6 | 8 |
|---|---|---|---|---|---|---|---|---|---|---|
| Spearman ρ | 0.94 | 0.96 | 0.97 | 0.98 | 0.99 | 1.00 | 0.99 | 0.96 | 0.96 | 0.93 |
| Kendall τ | 0.81 | 0.87 | 0.90 | 0.93 | 0.96 | 1.00 | 0.96 | 0.91 | 0.91 | 0.87 |
| Top pick | Crop | Crop | Crop | Crop | JPEG50 | JPEG50 | JPEG50 | JPEG50 | JPEG50 | JPEG50 |
The corresponding Figure 5 is available in the revised version.
Table R4: Rank of each anchor under AQS variants (full score, equal weights, and leave-one-component-out).
| Anchor | Full | Equal | Omit rem. | Omit collat. | Omit fidelity | Omit sep. | Measured mean |
|---|---|---|---|---|---|---|---|
| JPEG50 | 1 | 2 | 2 | 1 | 1 | 1 | 2.5 |
| Crop | 2 | 1 | 1 | 2 | 4 | 2 | 3.5 |
| Low-pass | 3 | 4 | 4 | 3 | 3 | 4 | 4.3 |
| Block quantization | 4 | 3 | 3 | 5 | 6 | 3 | 5.5 |
| Down-up | 5 | 6 | 7 | 4 | 2 | 5 | 5.6 |
| Gaussian noise | 6 | 5 | 5 | 6 | 5 | 6 | 5.7 |
| TV minimization | 7 | 7 | 8 | 7 | 7 | 7 | 5.8 |
| Median | 8 | 9 | 9 | 8 | 8 | 9 | 6.7 |
| Bit-depth | 9 | 8 | 6 | 9 | 9 | 8 | 8.4 |
| Color normalization | 10 | 10 | 10 | 10 | 10 | 10 | 9.9 |
The corresponding Figure 13 in Appendix A.13 is available in the revised version.
Response 3/5 to Reviewer skmU
Response to W2 (continued).
Table R5: Held-out anchor rank agreement (Spearman ).
| Anchor | AQS rank | Measured defense rank | Measured mean rank |
|---|---|---|---|
| JPEG50 | 1 | 1 | 2.4 |
| Crop | 2 | 2 | 2.6 |
| Low-pass* | 3 | 5 | 5.8 |
| Block quantization* | 4 | 3 | 3.8 |
| Gaussian noise* | 5 | 4 | 4.5 |
| Down-up | 6 | 6 | 6.0 |
| TV minimization | 7 | 7 | 6.4 |
| Bit-depth | 8 | 8 | 7.8 |
| Median | 9 | 9 | 8.8 |
| Color normalization | 10 | 10 | 10.0 |
* denotes change in rank. The corresponding Figure 11 in Appendix A.12 is available in the revised version.
We will add these analyses to the revision and qualify the generality claim to match them.
Response to Q1.
In Eq. (11), the consistency threshold is fixed to the value used in CSR [2], along with all its rectification settings (, , , ), while the temperature controls the sharpness of the soft consistency scaling and determines when rectification is activated. Our reported configuration is and . To assess its sensitivity, we vary and jointly while holding the remaining settings fixed, and report mean clean/robust top-1 accuracy (%) over OxfordPets, Flowers102, and StanfordCars in Table R6.
Table R6: Sensitivity analysis for and .
| / | 0.05 | 0.10 | 0.25 | 0.50 | 0.75 |
|---|---|---|---|---|---|
| 0.005 | 74.24 / 71.38 | 74.25 / 71.37 | 74.30 / 70.95 | 74.39 / 70.09 | 74.46 / 68.68 |
| 0.010 | 74.08 / 72.43 | 74.10 / 72.42 | 74.25 / 71.93 | 74.41 / 70.12 | 74.51 / 67.33 |
| 0.020 | 73.09 / 74.44 | 73.27 / 74.40 | 73.92 / 73.47 (default) | 74.41 / 70.04 | 74.59 / 63.87 |
| 0.040 | 69.42 / 76.16 | 69.49 / 76.17 | 72.28 / 75.61 | 74.43 / 69.87 | 74.66 / 56.42 |
| 0.080 | 65.09 / 76.59 | 65.09 / 76.59 | 65.97 / 76.58 | 74.44 / 69.62 | 74.66 / 39.32 |
The results reveal a systematic trade-off: more permissive rectification (smaller , and in much of the grid higher ) tends to increase robust accuracy but sacrifices clean accuracy; conservative gating better preserves clean accuracy while reducing protection against attacks. The default offers a balanced operating point (73.92% clean / 73.47% robust) without optimizing solely for adversarial accuracy. Several nearby settings offer alternative trade-offs; therefore, we do not claim that the default is uniquely optimal. This sweep demonstrates how performance changes across a broad range of gate settings. We will add this analysis to the revised version.
Response to Q2.
We evaluated the requested variant, keeping the gate from Eq. (11) while using the JPEG50 image directly rather than optimizing a correction. It appears as “Gated JPEG50 as direct input” in Table R7.
Table R7: Separating compression from rectification. Clean and robust accuracy (%), averaged over ten datasets, and inference time (ms per image, clean / attacked).
| Pipeline | Clean | ε = 1/255 | ε = 4/255 | ms/img (clean / attacked) |
|---|---|---|---|---|
| No defense | 65.7 | 2.0 | 1.3 | 0.48 / 0.47 |
| JPEG50 as input | 62.6 | 50.3 | 29.5 | 2.31 / 2.12 |
| Gated JPEG50 as direct input | 64.9 | 49.3 | 29.5 | 2.73 / 2.52 |
| JPEG50 + hard gate, fixed budget | 61.9 | 62.8 | 42.5 | 3.92 / 6.62 |
| ARC | 63.0 | 64.0 | 45.3 | 4.27 / 6.69 |
Table R8: Contribution of each stage to ARC's robust accuracy (percentage points), derived from Table R7.
| Stage | Gain at ε = 1/255 (pp) | Gain at ε = 4/255 (pp) |
|---|---|---|
| No defense gated JPEG50 directly | +47.3 | +28.2 |
| Add Eq. (12) rectification, fixed budget | +13.5 | +13.0 |
| Add soft budget scaling, full ARC | +1.2 | +2.8 |
| Total versus no defense | +62.0 | +44.0 |
Thus, optimization contributes 13.5 and 13.0 points over the gated JPEG50-input variant and gives a 14.7/15.8-point margin for full ARC over the no-optimization variant, respectively. The gated-JPEG alternative has higher clean accuracy and lower inference cost.
Response 1/5 to Reviewer skmU
We thank the reviewer for the constructive review, and for finding the paper clear and the methodology coherent. The reviewer's concerns are the relation to DBD, the generality of AQS, the sensitivity of the gate hyperparameters, how much of ARC's gain comes from the JPEG50 anchor versus the optimization, broader attack coverage, and how the feature-space theory relates to pixel-space rectification. We address each below.
Response to W1.
Comparison with DBD. Both approaches use transformation-induced changes in CLIP representations, but their questions and mechanisms differ. DBD [1] aggregates multiple transformed views to estimate a defense direction, uses a directional-consistency DB-score to select a reconstruction stream, and directly reconstructs a feature representation. Our main contribution is instead AQS: an offline criterion for selecting an individual rectification anchor, without implementing the full rectifier for every candidate. AQS separately quantifies attack-aligned removal, orthogonal collateral distortion, clean fidelity, and clean/adversarial separability. ARC then instantiates the selected JPEG50 anchor in an input-space, contrastive rectification procedure with an adaptively scaled correction budget. Thus, our contribution is not the general idea of exploiting transformed views, but making anchor quality measurable and comparable before defense construction and validating this selection in a concrete rectifier.
In DBD's evaluation, robust accuracy is often far above clean accuracy; on DTD, for example, it is 79.7% vs. 45.1%. DBD's adversarial examples are crafted on vanilla CLIP, using ground-truth labels, with no knowledge of the defense. Such a perturbation pushes the feature away from the true class, and input transformations partly undo it. The resulting shift therefore points back toward the true class, and DBD's Defense Direction recovers that signal. Transferred attacks thus favor defenses that read this signal; ARC's repulsion term is also partly affected. A defense-aware attacker controls the perturbation and can remove the signal, so we agree with the reviewer that comparing the two methods under adaptive attacks is more informative.
Table R1: Matched adaptive comparison. ViT-B/16, all ten datasets, ε = 1/255. The attack runs the full pipeline with EOT + BPDA, the same protocol as Table 3, with BPDA applied to DBD's entropy filtering and branch selection.
Under a matched defense-aware attack, ARC is more robust than DBD on all ten datasets (34.87% vs. 13.27% on average).
Additional contributions over DBD.