acceptodds
Under review as a conference paper at ICLR 2027

What makes a good anchor? Offline Anchor Selection for Test-Time Adversarial Robustness of CLIP

10 views

Abstract

Test-time defense techniques improve the zero-shot adversarial robustness of CLIP without retraining. Among them, rectification-based methods provide a direct and low-overhead strategy by seeking a correction that counteracts the attack by moving the adversarial representation toward an anchor, a transformation of the input intended to approximate the unavailable clean representation. Because this anchor serves as the reference for estimating the correction, its selection is fundamental to successful rectification. Yet existing methods typically prescribe an anchor a priori, with no criterion for comparing it against alternatives, and justify it only through the performance of the resulting defense. In this paper, we formulate anchor selection as an offline measurement problem and ask: what makes a good anchor? We identify four complementary properties required for effective rectification and quantify them in CLIP feature space to construct an Anchor Quality Score (AQS). We rank representative anchors from 10 transformation families based on their AQS and validate this ordering against the ranking obtained by building the full defense around each anchor. This experiment selects JPEG compression at quality 50 as the strongest anchor. Building on this selection, we introduce Anchor-Guided Rectification using Compression (ARC), a lightweight test-time defense that combines the selected anchor with a soft anchor-consistency scaling of the rectification budget. Across ten zero-shot classification benchmarks, ARC delivers substantial improvements in adversarial robust accuracy over SOTA test-time defenses while maintaining comparable clean accuracy and inference cost, and remains effective under adaptive attacks that explicitly target the complete defense.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial robustness, test time defense, CLIP, zero-shot classification, anchor selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_skmUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free test-time defense framework for improving the adversarial robustness of CLIP, with two main components: Anchor Quality Score (AQS) and Anchor-Guided Rectification using Compression (ARC). AQS is introduced to systematically evaluate candidate anchors. It characterizes anchor quality using four complementary properties: attack removal, collateral avoidance, clean fidelity, and separability. These four terms are combined through a weighted harmonic mean. The authors evaluate ten transformation families and select JPEG compression with quality 50 as the strongest anchor. Based on this selection, ARC uses the representation of the JPEG-compressed input as the anchor and performs test-time correction by attracting the perturbed representation toward the anchor while repelling it from the original adversarial representation. A soft anchor-consistency score further scales the correction budget adaptively.

Critical strengths
  1. The paper is clearly written and presents a coherent and easy-to-follow story. It identifies anchor selection as an important factor in test-time adversarial rectification, then naturally develops the AQS criterion for anchor evaluation and the ARC framework for defense, resulting in a complete methodology.
  2. Reasonable defense design. The proposed approach is technically reasonable. AQS provides a systematic strategy to evaluate and select an effective anchor based on multiple complementary properties, while ARC uses the selected anchor to guide test-time correction.
Critical weaknesses
  1. The paper lacks a sufficiently thorough comparison with DBD [1], which appears closely related to ARC in mechanism. Both methods leverage transformed views as references for recovering adversarial representations and use transformation-induced consistency or similarity signals to determine subsequent correction. Given the existence of DBD, the incremental contribution of ARC is therefore not yet sufficiently clear. A direct comparison with DBD under matched datasets, backbones, attack settings, and especially adaptive attacks would be important. The paper should also more explicitly articulate what new capability or insight ARC provides beyond DBD.
  2. The analysis of AQS is still limited relative to its central role in the paper. From both the title and the overall narrative, AQS is one of the main contributions, yet its robustness and generality are not fully established. It remains unclear whether AQS produces consistent anchor rankings across different attack types, attack strengths, backbones, and datasets, and whether JPEG with q=50q=50 remains the optimal choice under these variations. It would also be valuable to investigate whether combinations of transformations can yield higher AQS values than any single transformation, and whether changing the relative weights of the four AQS components leads to different optimal anchors. Such analyses would substantially strengthen the claim that AQS provides a general and reliable criterion for anchor selection.

[1] Liangsheng Liu, Si Chen, Jiamin Wu, Weiwei Feng, Zhixin Cheng, Xiaotian Yin, Wenfei Yang, and Tianzhu Zhang. Adversarial attacks already tell the answer: Directional biasguided test-time defense for vision-language models. In The Fourteenth International Conference on Learning Representations, 2026.

Questions for authors
  1. Could the authors clarify how the hyperparameters involved in Eq. (11), including the consistency threshold, temperature, and gating threshold, are selected? Since these parameters directly control whether and to what extent rectification is applied, please provide a sensitivity analysis to demonstrate the robustness of ARC with respect to these choices.
  2. Could the authors evaluate a simpler variant that removes the optimization in Eq. (12)? Specifically, after using Eq. (11) to determine whether an input should be rectified, the JPEG50-transformed image could be directly used as the corrected input instead of optimizing the additional correction perturbation. This comparison would help clarify how much of ARC’s performance gain comes from the selected JPEG50 anchor itself and how much comes from the optimization-based rectification in Eq. (12).
  3. Could the authors provide a more comprehensive robustness evaluation for the stronger and adaptive attacks reported in Table 3? Currently, Table 3 only evaluates three datasets, whereas Table 1 covers ten datasets. It would be helpful to report the corresponding results across all ten datasets under the three attack settings for a more complete comparison. In addition, please consider including representative black-box attacks, such as Square Attack, as well as low-frequency attack [1], to further evaluate whether ARC remains effective against attacks with substantially different perturbation characteristics.
  4. Could the authors clarify how the feature-space analysis in Theorem 1 and Corollary 1 relates to the actual rectification procedure used in ARC? The theoretical results analyze a linear interpolation between the adversarial feature and the anchor feature, whereas ARC performs a constrained optimization in the input space through the nonlinear CLIP encoder, as formulated in Eq. (12). It would be helpful to explain under what assumptions the feature-space analysis can justify the actual pixel-space rectification, and, if possible, provide empirical evidence showing that anchors favored by the theoretical criteria indeed induce more reliable correction trajectories toward the clean representation.

[1] Guo, C., Frank, J.S., Weinberger, K.Q., 2018. Low frequency adversarial perturbation. arXiv preprint arXiv:1809.08758 .

8 comments
AuthorsOpenReview ↗

Response 1/5 to Reviewer skmU

We thank the reviewer for the constructive review, and for finding the paper clear and the methodology coherent. The reviewer's concerns are the relation to DBD, the generality of AQS, the sensitivity of the gate hyperparameters, how much of ARC's gain comes from the JPEG50 anchor versus the optimization, broader attack coverage, and how the feature-space theory relates to pixel-space rectification. We address each below.


Response to W1.

Comparison with DBD. Both approaches use transformation-induced changes in CLIP representations, but their questions and mechanisms differ. DBD [1] aggregates multiple transformed views to estimate a defense direction, uses a directional-consistency DB-score to select a reconstruction stream, and directly reconstructs a feature representation. Our main contribution is instead AQS: an offline criterion for selecting an individual rectification anchor, without implementing the full rectifier for every candidate. AQS separately quantifies attack-aligned removal, orthogonal collateral distortion, clean fidelity, and clean/adversarial separability. ARC then instantiates the selected JPEG50 anchor in an input-space, contrastive rectification procedure with an adaptively scaled correction budget. Thus, our contribution is not the general idea of exploiting transformed views, but making anchor quality measurable and comparable before defense construction and validating this selection in a concrete rectifier.

In DBD's evaluation, robust accuracy is often far above clean accuracy; on DTD, for example, it is 79.7% vs. 45.1%. DBD's adversarial examples are crafted on vanilla CLIP, using ground-truth labels, with no knowledge of the defense. Such a perturbation pushes the feature away from the true class, and input transformations partly undo it. The resulting shift therefore points back toward the true class, and DBD's Defense Direction recovers that signal. Transferred attacks thus favor defenses that read this signal; ARC's repulsion term is also partly affected. A defense-aware attacker controls the perturbation and can remove the signal, so we agree with the reviewer that comparing the two methods under adaptive attacks is more informative.

Table R1: Matched adaptive comparison. ViT-B/16, all ten datasets, ε = 1/255. The attack runs the full pipeline with EOT + BPDA, the same protocol as Table 3, with BPDA applied to DBD's entropy filtering and branch selection.

DatasetCaltech101OxfordPetsStanfordCarsFlowers102FGVC-AircraftDTDEuroSATUCF101Food101ImageNetAverage
DBD34.2025.6717.457.354.269.933.075.475.3519.9113.27
ARC73.4646.1739.9733.836.5721.498.9743.0647.7927.4134.87

Under a matched defense-aware attack, ARC is more robust than DBD on all ten datasets (34.87% vs. 13.27% on average).

Additional contributions over DBD.

  1. A way to choose the anchor. DBD fixes its set of transformations in advance. AQS measures anchor quality offline using four properties, and its rankings agree with realized defense performance (Fig. 3, App. A.7, and on held-out data in Table R4 of our response to Reviewer XLym). AQS could also be used to choose DBD's transformations.
  2. Theory that applies to DBD. DBD's correction is a linear shift in feature space, (1−β)fa+βaˉ(1-\beta)f_a + \beta\bar a, which is the update analyzed in Theorem 1. The theorem shows that the benefit of such a shift is governed by the anchor's removal rr and collateral cc. Corollary 1 shows that no choice of step size makes up for a poor anchor.
  3. A single view, corrected in input space. ARC encodes one JPEG50 view and optimizes a bounded input perturbation, at 4.4 ms per image. DBD encodes many transformed views and corrects in feature space, at 25 ms per image.
  4. Continuous control instead of a switch. DBD switches between a feature shift and averaging over views depending on its DB-score. ARC scales its correction budget continuously and leaves high-consistency inputs unchanged.
AuthorsOpenReview ↗

Response 2/5 to Reviewer skmU

Response to W2.

We have added several of these analyses. Below we address each factor, including where the answer is qualified or where we do not yet have results.

Table R2: Rank by AQS vs. measured rank of the full defense, for the ten family representatives. Held-out: AQS computed on Stanford Cars, FGVC-Aircraft and UCF101; defense measured on Food101 and ImageNet (none used for anchor selection). AA→AA: AQS and defense both under AutoAttack. PGD→AA: AQS from PGD only, defense measured under AutoAttack (AQS not recomputed).

AnchorAQS rank (AA)AQS rank (PGD)Defense rank under AAMeasured AA mean rank
JPEG501112.9
Crop2223.3
Block quantization3434.0
Low-pass4365.4
Gaussian noise5655.1
Down-up6545.1
TV minimization7776.9
Median8887.1
Bit-depth9998.1
Color normalization10101010.0

The corresponding Figure 12 in Appendix A.12 is available in the revised version.

Attack types. AQS computed under AutoAttack agrees with performance measured under AutoAttack at ρ = 0.95. A ranking computed only from PGD and compared with AutoAttack-measured performance reaches ρ = 0.93 (Table R2, PGD→AA). JPEG50 ranks first in both.

Datasets. On the five benchmarks not used for selection, the AQS ranking and the measured ranking agree with ρ = 0.96, with the weights fixed (Table R5, held-out). JPEG50 ranks first and Crop second under both.

Attack strength. The cell-wise AQS in Figure 5 shows that the mean rank depends on the budget. At ε = 1/255 Crop leads JPEG50 (mean rank 1.9 vs. 2.3); at ε = 4/255 JPEG50 leads Crop (2.0 vs. 3.0). Within the JPEG family, q = 75 scores highest in four of the five ε = 1/255 cells, while q = 50 is best or tied-best in the other six (Appendix A.7). So q = 50 is not uniformly optimal. It is the most consistent choice across the grid, with the highest worst-cell score (0.68, shared with q = 30), and the weaker the attack, the milder the optimal compression. We will state this explicitly.

Backbones. AQS was computed on ViT-B/16 only. As indirect evidence, JPEG50, chosen on ViT-B/16, gives ARC the best average robust accuracy on ViT-B/32 and widens its margin over CSR from 6.8 to 14.3 points (Tables 1 and 2). This supports transfer of the choice, but it is not a test of AQS on a second backbone.

Combinations of transformations. We have not evaluated these. AQS is defined only on the anchor map A(⋅)A(\cdot), so it applies to composed anchors without modification. We list composed anchors as future work in Section 5.

Relative weights. Changing the weights can change the top anchor. Sweeping wremw_{rem} from 0.5 to 8 (Table R3), the top pick is JPEG50 for wrem∈[2.5,8]w_{rem} \in [2.5, 8] and Crop for wrem≤2w_{rem} \le 2. Spearman correlation with the measured ranking stays at least 0.93 throughout. With equal weights, AQS places Crop first and JPEG50 second, at ρ = 0.96 (Table R4). Leaving out clean fidelity, separability or collateral keeps JPEG50 first. Leaving out removal does not, and gives the weakest agreement (ρ = 0.86).

Table R3: Sensitivity to the removal weight wremw_{rem}, with the other three weights held at 1. Rank correlation between AQS and measured defense rank, and the anchor AQS ranks first. Bold = paper setting.

Removal weight wremw_\mathrm{rem}0.511.522.534568
Spearman ρ0.940.960.970.980.991.000.990.960.960.93
Kendall τ0.810.870.900.930.961.000.960.910.910.87
Top pickCropCropCropCropJPEG50JPEG50JPEG50JPEG50JPEG50JPEG50

The corresponding Figure 5 is available in the revised version.

Table R4: Rank of each anchor under AQS variants (full score, equal weights, and leave-one-component-out).

AnchorFullEqualOmit rem.Omit collat.Omit fidelityOmit sep.Measured mean
JPEG501221112.5
Crop2112423.5
Low-pass3443344.3
Block quantization4335635.5
Down-up5674255.6
Gaussian noise6556565.7
TV minimization7787775.8
Median8998896.7
Bit-depth9869988.4
Color normalization1010101010109.9

The corresponding Figure 13 in Appendix A.13 is available in the revised version.

AuthorsOpenReview ↗

Response 3/5 to Reviewer skmU

Response to W2 (continued).

Table R5: Held-out anchor rank agreement (Spearman ρ=0.96\rho=0.96).

AnchorAQS rankMeasured defense rankMeasured mean rank
JPEG50112.4
Crop222.6
Low-pass*355.8
Block quantization*433.8
Gaussian noise*544.5
Down-up666.0
TV minimization776.4
Bit-depth887.8
Median998.8
Color normalization101010.0

* denotes change in rank. The corresponding Figure 11 in Appendix A.12 is available in the revised version.

We will add these analyses to the revision and qualify the generality claim to match them.


Response to Q1.

In Eq. (11), the consistency threshold τ=0.85\tau=0.85 is fixed to the value used in CSR [2], along with all its rectification settings (ϵη\epsilon_\eta, αη\alpha_\eta, λ\lambda, NN), while the temperature TT controls the sharpness of the soft consistency scaling and gτg_\tau determines when rectification is activated. Our reported configuration is T=0.02T=0.02 and gτ=0.25g_\tau=0.25. To assess its sensitivity, we vary TT and gτg_\tau jointly while holding the remaining settings fixed, and report mean clean/robust top-1 accuracy (%) over OxfordPets, Flowers102, and StanfordCars in Table R6.

Table R6: Sensitivity analysis for TT and gτg_\tau.

TT / gτg_\tau0.050.100.250.500.75
0.00574.24 / 71.3874.25 / 71.3774.30 / 70.9574.39 / 70.0974.46 / 68.68
0.01074.08 / 72.4374.10 / 72.4274.25 / 71.9374.41 / 70.1274.51 / 67.33
0.02073.09 / 74.4473.27 / 74.4073.92 / 73.47 (default)74.41 / 70.0474.59 / 63.87
0.04069.42 / 76.1669.49 / 76.1772.28 / 75.6174.43 / 69.8774.66 / 56.42
0.08065.09 / 76.5965.09 / 76.5965.97 / 76.5874.44 / 69.6274.66 / 39.32

The results reveal a systematic trade-off: more permissive rectification (smaller gτg_\tau, and in much of the grid higher TT) tends to increase robust accuracy but sacrifices clean accuracy; conservative gating better preserves clean accuracy while reducing protection against attacks. The default (T,gτ)=(0.02,0.25)(T,g_\tau)=(0.02,0.25) offers a balanced operating point (73.92% clean / 73.47% robust) without optimizing solely for adversarial accuracy. Several nearby settings offer alternative trade-offs; therefore, we do not claim that the default is uniquely optimal. This sweep demonstrates how performance changes across a broad range of gate settings. We will add this analysis to the revised version.


Response to Q2.

We evaluated the requested variant, keeping the gate from Eq. (11) while using the JPEG50 image directly rather than optimizing a correction. It appears as “Gated JPEG50 as direct input” in Table R7.

Table R7: Separating compression from rectification. Clean and robust accuracy (%), averaged over ten datasets, and inference time (ms per image, clean / attacked).

PipelineCleanε = 1/255ε = 4/255ms/img (clean / attacked)
No defense65.72.01.30.48 / 0.47
JPEG50 as input62.650.329.52.31 / 2.12
Gated JPEG50 as direct input64.949.329.52.73 / 2.52
JPEG50 + hard gate, fixed budget61.962.842.53.92 / 6.62
ARC63.064.045.34.27 / 6.69

Table R8: Contribution of each stage to ARC's robust accuracy (percentage points), derived from Table R7.

StageGain at ε = 1/255 (pp)Gain at ε = 4/255 (pp)
No defense →\rightarrow gated JPEG50 directly+47.3+28.2
Add Eq. (12) rectification, fixed budget+13.5+13.0
Add soft budget scaling, full ARC+1.2+2.8
Total versus no defense+62.0+44.0

Thus, optimization contributes 13.5 and 13.0 points over the gated JPEG50-input variant and gives a 14.7/15.8-point margin for full ARC over the no-optimization variant, respectively. The gated-JPEG alternative has higher clean accuracy and lower inference cost.

Reviewer_7NtvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies anchor selection for test-time adversarial defense of CLIP. It introduces an Anchor Quality Score AQSAQS that evaluates candidate anchors in terms of attack-aligned residual displacement, orthogonal collateral distortion, clean fidelity, and gate separability. The authors compare ten transformation families and their strength parameters, select JPEG50, and use it in Anchor-Guided Rectification using Compression ARCARC. ARC adopts a contrastive rectification objective and adjusts the rectification budget according to input–anchor consistency. The evaluation includes agreement between anchor rankings, experiments on ten classification datasets, adaptive attacks, and ablation studies.

Critical strengths

I consider the quantification of anchor quality to be the paper’s main contribution. The distinction between attack-aligned residual displacement and orthogonal collateral distortion, together with clean fidelity and gate separability, provides a structured account of the properties relevant to anchor selection. Theorem 1 and Corollary 1 give a clear geometric interpretation under the stated linear feature interpolation setting. The experiments across ten transformation families and their strength parameters also provide useful insight into the role of the anchor.

Critical weaknesses
  • Limited cross-attack generalizability evaluation of AQS. AQS-based anchor selection is evaluated only under PGD attacks with two perturbation budgets. Although ARC is further tested against AutoAttack and adaptive attacks, these experiments do not establish whether AQS can reliably predict anchor quality across different attack methods. Since its removal and collateral metrics depend on attack-induced feature displacements, the resulting anchor rankings may be attack-dependent. If anchor selection must be repeated for each attack method, the practical utility of AQS as a generalizable selection criterion would be limited.
  • The adaptive attack evaluation is limited. The evaluation uses a perturbation budget of 1/2551/255 and 10 PGD steps, whereas SS-TPT and MAC specify 4/2554/255 and 100 steps for CLIP-ViT. On Flowers102, ARC’s robust accuracy drops from 70.4% against transferred AutoAttack examples to 33.9% under the adaptive attack, both at 1/2551/255. Although this comparison does not isolate the effect of making the attack defense-aware, the gap makes the adequacy of attack optimization a relevant concern. Without a convergence check over attack steps or adaptive evaluation at a larger budget, the evidence for robustness against adaptive attacks remains limited.
  • The benefit of rectification over JPEG compression alone is unclear. The paper does not include a baseline that directly classifies JPEG50-compressed inputs. Both the anchor comparison and the JPEG quality study retain the rectification stage, so they do not distinguish the effect of compression from the additional benefit of rectification. Since rectification requires additional optimization steps, this comparison is needed to establish its contribution and computational cost.
Questions for authors
  • Could the authors increase the number of adaptive attack steps at 1/2551/255 to check whether robust accuracy has converged, and also report adaptive attack results at 4/2554/255? Does ARC retain its performance and relative advantage under these settings?

  • Could the authors compare direct classification of JPEG50-compressed inputs, JPEG50-guided rectification with a fixed budget, and full ARC on the same evaluation samples, using matched perturbation budgets and a defense-aware attack protocol tailored to each pipeline? Reporting clean accuracy, robust accuracy, and inference time would help establish the contribution of rectification beyond compression alone.

  • Appendix A.6.2 describes 300 development pairs per dataset–attack condition, while Section 4.1 and Appendix A.8 state that the final evaluation uses the test splits. Could the authors clarify which splits the development pairs come from and whether they are disjoint from the final evaluation samples? Is the agreement between AQS rankings and defense-performance rankings also evaluated on samples not used for anchor selection?

4 comments
AuthorsOpenReview ↗

Response 1/2 to Reviewer 7Ntv

We thank the reviewer for the thorough review, and for identifying the quantification of anchor quality as the paper's main contribution. The reviewer's concerns are whether AQS rankings depend on the attack, whether the adaptive attack is strong enough to support the robustness claims, how much rectification adds beyond JPEG compression alone, and whether the development samples overlap with the evaluation samples. We address each below.


Response to W1.

We agree that removal and collateral depend on attack-induced displacement, so an attack-transfer test is essential. We performed two comparisons using AutoAttack, summarized in Table R1:

Same attack (AA → AA). We computed AQS under AutoAttack and measured defense performance under AutoAttack. The two rankings agree with Spearman ρ = 0.95. JPEG50 ranks first and Crop second under both, and the top-3 shortlist (JPEG50, Crop, Block quantization) are in agreement.

Cross-attack (PGD → AA). This experiment tests whether selection must be repeated for each attack. We used the AQS ranking computed only from PGD and compared it with defense performance measured under AutoAttack. The correlation remains high at ρ=0.93\rho=0.93, and JPEG50 and Crop keep the top two places in both columns. Ranks 7--10 are identical. The main error is that the PGD-derived score over-rates Low-pass (3rd by AQS, 6th measured), so two of the top three shortlisted anchors carry over.

Table R1: Anchor ranking across attack methods.

AnchorAQS rank (AA)AQS rank (PGD)Defense rank under AAMeasured AA mean rank
JPEG501112.9
Crop2223.3
Block quantization3434.0
Low-pass4365.4
Gaussian noise5655.1
Down-up6545.1
TV minimization7776.9
Median8887.1
Bit-depth9998.1
Color normalization10101010.0

The corresponding Figure 12 in Appendix A.13 is available in the revised version.

The second experiment directly tests the reviewer's concern that selection must be repeated for every attack. The original PGD-derived top anchor remains best under AutoAttack; the main discrepancy concerns mid-ranked Low-pass (third by AQS, sixth in measured AutoAttack performance). The absolute AQS values do vary with the attack, as they do with the budget (Appendix A.5), and we limit this generalization claim to the tested ℓ∞\ell_\infty attacks rather than arbitrary attack families.


Response to W2, Q1.

Repeating from our response to Reviewer XLym, we extend the defense-aware adaptive attack from 10 to 50 and 100 PGD steps, with three restarts at both ϵ=1/255\epsilon=1/255 and 4/2554/255, on the same three datasets used in Table 3 (Pets, Flowers102, and Cars). We report per-image worst-case accuracy (WC) across restarts.

Table R2: Adaptive evaluation at ϵ=1/255\epsilon=1/255 (per-image worst-case robust accuracy across three restarts, %)

MethodPets (50 / 100)Flowers102 (50 / 100)Cars (50 / 100)Average (50 / 100)
ARC21.42 / 19.6516.18 / 15.1222.29 / 20.2519.96 / 18.34
CSR6.16 / 4.583.29 / 2.411.70 / 1.493.72 / 2.83
MAC23.30 / 23.2018.70 / 16.9015.00 / 15.5019.00 / 18.50

At 50 steps, ARC leads on mean WC (19.96% versus MAC's 19.00% and CSR's 3.72%), although MAC outperforms ARC on Pets and Flowers102. At 100 steps, ARC remains ahead of CSR by 15.51 points (18.34% versus 2.83%) while performing similar to MAC staying within half a point. The attack converges by 50 steps. Going from 50 to 100 steps lowers ARC's average robust accuracy by only 1.6 points, CSR's by 0.9, and MAC's by 0.5.

At ϵ=4/255\epsilon=4/255, all methods, including ARC, converge to 0% WC robust accuracy at both 50 and 100 steps. We therefore make no claim of robustness at this larger perturbation budget. An equivalent multi-restart evaluation of SS-TPT was computationally prohibitive within the rebuttal period because its multi-augmentation pipeline must be evaluated repeatedly inside the adaptive attack.

AuthorsOpenReview ↗

Response 2/2 to Reviewer 7Ntv

Response to W3, Q2.

We agree this is the right baseline and have added it in Table R3. It uses the same samples and matched budgets on all ten datasets, under PGD-10 crafted on base CLIP and defense-aware protocol as in A.9.

Table R3: Separating compression from rectification. Clean and robust accuracy (%), averaged over ten datasets, and inference time (ms per image, clean / attacked).

PipelineCleanε = 1/255ε = 4/255Adaptivems/img (clean / attacked)
No defense65.72.01.32.00.48 / 0.47
JPEG50 as input62.650.329.524.02.31 / 2.12
JPEG50 + hard gate, fixed budget61.962.842.526.73.92 / 6.62
ARC63.064.045.328.84.27 / 6.69

Compression alone is already a substantial defense under this attack, reaching 50.3% at ϵ=1/255\epsilon=1/255 and 29.5% at ϵ=4/255\epsilon=4/255. Rectifying toward the JPEG50 anchor with a fixed budget raises this to 62.8% and 42.5%, a gain of 12.5 and 13.0 points. Full ARC reaches 64.0% and 45.3%, which is 13.7 and 15.8 points above direct JPEG50 classification. Relative to always compressing the input, ARC gives this gain with no loss of clean accuracy (63.0% vs. 62.6%). Gating the compression recovers more clean accuracy (64.9%), at a cost of 14.7 and 15.8 robust points relative to ARC. Under adaptive attack at ϵ=1/255\epsilon=1/255 (per-image worst case over each pipeline's own attack and a plain-CLIP attack), ARC reaches 28.8%, 4.8 points above direct JPEG50 classification (24.0%) and 2.1 above fixed-budget rectification (26.7%).

Cost. Rectification adds about 2.0 ms per clean image and 4.6 ms per attacked image over direct JPEG50 input (4.27 / 6.69 ms vs. 2.31 / 2.12 ms). This is small next to the multi-view prompt-tuning defenses in Table 4, which take 133.6 ms or more.


Response to Q3.

In the submitted experiments, we used the term "development pairs" to mean disjoint samples from the same split: the test split for Pets, Flowers102, and DTD, and the full dataset for EuroSAT and Caltech101, which have no official split. They make up at most 8.2%, 4.9%, 16.0%, 1.1% and 3.5% of those sets, respectively. To remove any overlap, we ran the evaluation excluding these samples. We will clarify the writing in the revised version.

Agreement on samples not used for selection. Yes. We re-ran the ranking experiment using the remaining five benchmarks: the AQS ranking was obtained on Stanford Cars, FGVC-Aircraft, and UCF101, while realized defense performance was measured on held-out Food101 and ImageNet (Table R4). The AQS ranking and the measured ranking agree with Spearman ρ=0.96\rho=0.96, and JPEG50 ranks first in both.

Table R4: Held-out rank agreement (ρ=0.96\rho=0.96).

AnchorAQS rankMeasured defense rankMeasured mean rank
JPEG50112.4
Crop222.6
Low-pass*355.8
Block quantization*433.8
Gaussian noise*544.5
Down-up666.0
TV minimization776.4
Bit-depth887.8
Median998.8
Color normalization101010.0

* denotes change in rank. The corresponding Figure 11 in Appendix A.12 is available in the revised version.

Thanks the authors for the detailed rebuttal and additional experiments.

The new results adequately address my concerns regarding the cross-attack generalizability of AQS (W1), the JPEG-only baseline (W3), and the evaluation data split (Q3).

Regarding W2, while the stronger adaptive evaluation addresses my concern about attack optimization, ARC's average robust accuracy drops substantially from approximately 40% to 18.34% at ε = 1/255, becoming comparable to MAC. This weakens the adaptive robustness claims in the original submission.

Overall, I appreciate the authors' efforts and acknowledge that most of my concerns have been addressed. I will raise my Soundness score from 1 to 2 to reflect the improved experimental validation. However, given the limited robustness under stronger adaptive attacks and the relatively incremental nature of the proposed defense, I will maintain my overall rating.

We thank the reviewer for their thoughtful reassessment and for acknowledging that most concerns have been addressed. We would like to offer two additional clarifications.

Regarding adaptive robustness, we acknowledge the reduction in absolute robust accuracy under stronger attacks. However, the approximately 40% reported in the original submission was obtained using a single 10-step adaptive attack, whereas the new 18.34% reflects per-image worst-case accuracy across three restarts of a 100-step attack. This substantially stricter evaluation is applied consistently to all methods. While ARC remains competitive with MAC on ViT-B/16 (18.34% vs. 18.50%), we additionally evaluated the same 100-step adaptive attack on ViT-B/32, obtaining:

MethodPetsFlowers102CarsAverage
ARC27.7518.4610.9819.06
MAC22.6012.6012.4015.87

ARC outperforms MAC by 3.19 percentage points on ViT-B/32 and exhibits more consistent average robustness across the two backbones (18.34% and 19.06%, compared with MAC's 18.50% and 15.87%). This further supports ARC's relative effectiveness under stronger adaptive attacks, while acknowledging the limitations in absolute robustness.

Regarding the incremental nature of the defense, we would like to emphasize that our primary contribution is not a new rectification objective, but AQS: a principled, offline criterion for evaluating and selecting anchors without constructing the full defense for each candidate. ARC serves as an instantiation of this criterion using the selected JPEG50 anchor. The additional held-out validation and cross-attack evaluation directly support the generality and predictive utility of AQS, which we believe constitutes the central methodological contribution beyond existing rectification approaches.

We appreciate the reviewer's constructive feedback and hope these additional clarifications help contextualize the strengthened evaluation and the paper's principal contribution.

Reviewer_XLymOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an Anchor Quality Score (AQS) for selecting anchors in CLIP test-time rectification. It combines attack removal, collateral avoidance, clean fidelity, and clean/adversarial separability. AQS selects JPEG50 from ten transformation families. ARC uses this anchor with CSR's contrastive objective and a consistency-scaled correction budget. Experiments cover ten datasets and two CLIP backbones. On ViT-B/16, average accuracy under transferred PGD-10 attacks rises from 57.2% for CSR to 64.0% for ARC.

Critical strengths
  1. The main contribution is a criterion for comparing anchors, not JPEG itself. Theorem 1 links attack removal and collateral error to correction quality. Figures 3 and 8 connect AQS rankings to measured defense performance. This is useful beyond reporting that one transform works better.
  2. The ablation supports the role of anchor selection: JPEG50 with CSR's hard gate and fixed budget reaches 62.8%, versus CSR's 57.2%; soft scaling adds 1.2 points (Table 6). ARC also leads the reported adaptive tests on all three datasets (Table 3).
Critical weaknesses
  1. AQS ranking generalization remains unclear. Appendix A.7 validates JPEG-quality rankings on the same ten dataset-attack settings used for selection. The main defense is tested more broadly, but the ranking criterion itself needs held-out validation. The weights (3, 1, 1, 1) also lack a sensitivity test.
  2. Adaptive evaluation uses 10 steps, no restarts, and epsilon = 1/255 on three datasets (Table 3; Appendix A.9). PGD and AutoAttack instead transfer examples from base CLIP. ARC's lead is encouraging, but attack convergence and stronger adaptive robustness remain untested.
Questions for authors
  1. Can AQS predict anchor rankings on a held-out dataset with all settings fixed? Please compare rank correlations for the full score, equal weights, and leave-one-component-out variants across the ten real anchor families.
  2. Does ARC retain its lead under 50/100-step adaptive attacks with multiple restarts at epsilon = 1/255 and 4/255? Please report per-image worst-case accuracy for each budget.
  3. In Appendix A.11.2, is s = 1 enforced? Otherwise, c = λ/s appears to omit a normalization term in the orthogonal residual. Please check the derivation and confirm whether Table 5 uses the full residual norm.
5 comments
AuthorsOpenReview ↗

Response 1/3 to Reviewer XLym

We thank the reviewer for the careful review, and for recognizing that the main contribution is a criterion for comparing anchors rather than JPEG itself. The reviewer's concerns are whether AQS rankings generalize to held-out data and different weights, whether ARC holds up against stronger multi-step adaptive attacks, and the normalization in Appendix A.11.2. We address each below.


Response to W1, Q1.

We agree that the criterion itself requires validation beyond the settings used for selection. We re-ran the ranking experiment using the remaining five benchmarks: the AQS ranking was obtained on Stanford Cars, FGVC-Aircraft, and UCF101, while realized defense performance was measured on held-out Food101 and ImageNet. We retained the same ten transformation-family representatives, ViT-B/16, PGD-10, ϵ∈1,4/255\epsilon\in1,4/255, and original weights (3,1,1,1)(3,1,1,1). Table R1 shows Spearman correlation ρ=0.96\rho=0.96, with JPEG50 ranked first and Crop second by both AQS and measured defense performance. The anchor AQS selects is therefore also the best measured anchor on data it was not chosen from. Most rank changes occur in the middle with Low-pass shifting from third (AQS) to fifth (measured), and Block quantization and Gaussian noise moving up one place.

Table R1: Held-out anchor rank agreement (Spearman ρ=0.96\rho=0.96).

AnchorAQS rankMeasured defense rankMeasured mean rank
JPEG50112.4
Crop222.6
Low-pass*355.8
Block quantization*433.8
Gaussian noise*544.5
Down-up666.0
TV minimization776.4
Bit-depth887.8
Median998.8
Color normalization101010.0

* denotes change in rank. The corresponding Figure 11 in Appendix A.12 is available in the revised version.

Weight sensitivity. In Table R2, sweeping the removal weight wremw_{\mathrm{rem}} from 0.5 to 8 while fixing the other three weights at 1 yields Spearman ρ≥0.93\rho\ge 0.93 throughout and Kendall τ≥0.81\tau\ge 0.81. The top pick is JPEG50 for wrem∈[2.5,8]w_{\mathrm{rem}}\in[2.5,8], while Crop is first for wrem≤2w_{\mathrm{rem}} \le 2. Our default weight of 3 therefore sits inside a wide range that yields the same selection, rather than at an isolated value. The sweep also shows that the selection is not independent of the weighting where below wrem=2.5w_\mathrm{rem}=2.5 AQS prefers Crop, while the measured best remains JPEG50. Removal has to be weighted above the other terms to recover the measured best anchor, which is consistent with our reason for weighting it, namely that it most directly reflects adversarial robustness.

Table R2: Removal-weight sensitivity.

Removal weight wremw_\mathrm{rem}0.511.522.534568
Spearman ρ0.940.960.970.980.991.000.990.960.960.93
Kendall τ0.810.870.900.930.961.000.960.910.910.87
Top pickCropCropCropCropJPEG50JPEG50JPEG50JPEG50JPEG50JPEG50

The corresponding Figure 5 is available in the revised version.

AuthorsOpenReview ↗

Response 2/3 to Reviewer XLym

Response to W1, Q1 (continued).

Equal-weight and leave-one-component-out variants. Table R3 and Table R4 compare the variants across the ten real families. These comparisons use the same selection cells as the paper so that the reason for rank disagreements can be isolated.

Table R3: Rank correlation with measured defense rank for each AQS variant. Weights are listed as removal / collateral / clean fidelity / separability.

VariantWeightsSpearman ρKendall τAQS top pick
Full score (default)3/1/1/11.001.00JPEG50
Equal weights1/1/1/10.960.87Crop (JPEG50 second)
Omit removal0/1/1/10.860.72Crop (JPEG50 second)
Omit collateral3/0/1/10.990.96JPEG50
Omit clean fidelity3/1/0/10.880.76JPEG50
Omit separability3/1/1/00.970.90JPEG50

Table R4: Rank of each anchor under AQS variants (full score, equal weights, and leave-one-component-out).

AnchorFullEqualOmit rem.Omit collat.Omit fidelityOmit sep.Measured mean
JPEG501221112.5
Crop2112423.5
Low-pass3443344.3
Block quantization4335635.5
Down-up5674255.6
Gaussian noise6556565.7
TV minimization7787775.8
Median8998896.7
Bit-depth9869988.4
Color normalization1010101010109.9

The corresponding Figure 13 in Appendix A.13 is available in the revised version.

Removing removal or clean fidelity produces the greatest reduction in agreement (ρ=0.86\rho=0.86 and 0.880.88). Removing collateral barely changes ranks within this particular set of real transforms, but this does not imply that collateral is dispensable: the targeted synthetic DcolD_{\mathrm{col}} construction in original Table 5 isolates attack-orthogonal error while controlling the other metrics. The real-family and controlled-degenerate analyses test complementary aspects of component necessity. We will incorporate these findings into the revision.

AuthorsOpenReview ↗

Response 3/3 to Reviewer XLym

Response to W2, Q2.

We extend the defense-aware adaptive attack from 10 to 50 and 100 PGD steps, with three restarts at both ϵ=1/255\epsilon=1/255 and 4/2554/255, on the same three datasets used in Table 3 (Pets, Flowers102, and Cars). We report per-image worst-case accuracy (WC) across restarts.

Table R5: Adaptive attack results at ϵ=1/255\epsilon=1/255 (per-image worst-case robust accuracy across three restarts, %)

MethodPets (50 / 100)Flowers102 (50 / 100)Cars (50 / 100)Average (50 / 100)
ARC21.42 / 19.6516.18 / 15.1222.29 / 20.2519.96 / 18.34
CSR6.16 / 4.583.29 / 2.411.70 / 1.493.72 / 2.83
MAC23.30 / 23.2018.70 / 16.9015.00 / 15.5019.00 / 18.50

At 50 steps, ARC leads on mean WC (19.96% versus MAC's 19.00% and CSR's 3.72%), although MAC outperforms ARC on Pets and Flowers102. At 100 steps, ARC remains ahead of CSR by 15.51 points (18.34% versus 2.83%) while performing similar to MAC staying within half a point.

At ϵ=4/255\epsilon=4/255, all methods, including ARC, converge to 0% WC robust accuracy at both 50 and 100 steps. We therefore make no claim of robustness at this larger perturbation budget. An equivalent multi-restart evaluation of SS-TPT was computationally prohibitive within the rebuttal period because its multi-augmentation pipeline must be evaluated repeatedly inside the adaptive attack.


Response to Q3.

Thank you for checking this carefully. The reviewer is correct on both counts: s=1s = 1 is not enforced, and the closed form c=λ/sc = \lambda/s omits a term. The expression for rr is correct. Table 5 is unaffected because it uses the full residual norm from Eq. (2).

Derivation. With z=fˉc+αu+λ∥u∥2nz = \bar f_c + \alpha u + \lambda\|u\|_2 n, s=∥z∥2s = \|z\|_2 and v=z/s−fˉcv = z/s - \bar f_c,

v=(1s−1)fˉc+αsu+λs∥u∥2 n.v = \left(\tfrac{1}{s} - 1\right)\bar f_c + \tfrac{\alpha}{s}u + \tfrac{\lambda}{s}\|u\|_2\, n.

Using ⟨fˉc,u⟩=−∥u∥22/2\langle \bar f_c, u\rangle = -\|u\|_2^2/2 recovers r=(α−12)/s+12r = (\alpha - \tfrac12)/s + \tfrac12, as stated in the paper. Subtracting ruru gives

v−ru=(1s−1)(fˉc+u2)+λs∥u∥2 n.v - ru = \left(\tfrac{1}{s} - 1\right)\left(\bar f_c + \tfrac{u}{2}\right) + \tfrac{\lambda}{s}\|u\|_2\, n.

The vector fˉc+u/2\bar f_c + u/2 is the midpoint of fˉc\bar f_c and fˉa\bar f_a. It is orthogonal to both uu and nn, and its squared norm is 1−∥u∥22/41 - \|u\|_2^2/4. The two terms are therefore orthogonal, so

c2=∥v−ru∥22∥u∥22=λ2s2+(1−s)2s2(1∥u∥22−14).c^2 = \frac{\|v - ru\|_2^2}{\|u\|_2^2} = \frac{\lambda^2}{s^2} + \frac{(1-s)^2}{s^2}\left(\frac{1}{\|u\|_2^2} - \frac14\right).

Hence c=λ/sc = \lambda/s holds only when s=1s = 1, that is, when λ2=α(1−α)\lambda^2 = \alpha(1-\alpha). The omitted term is the part of the normalization shift that lies along the midpoint direction. Because it is non-negative, the true collateral is at least λ/s\lambda/s.

Table 5 Implementation. We used the closed form only to set the construction parameters of DcolD_{col}. The collateral score in Table 5 is computed by applying Eq. (2) directly to the constructed features, c=∥v−ru∥2/∥u∥2c = \|v - ru\|_2/\|u\|_2, as for every other anchor, so it includes the full residual. Each degenerate's measured components are then calibrated against the reference, which absorbs any gap between the closed form and the measured value. The rankings in Table 5 are therefore unchanged. We will correct the expression in A.11.2, state that the closed form is used only for calibration, and fix the notation for the working space.

Comments on the paper

3 comments
Area_Chair_bdGkOpenReview ↗

Reviewer–Author Discussion Period Ending Soon

Dear Reviewers,

With the discussion period coming to a close, I encourage you to revisit the authors’ responses and let us know if you have any further comments.

Thank you for your careful reviews and engagement in the discussion!

AuthorsOpenReview ↗

Summary of changes 1/2

We sincerely thank all reviewers for their careful reading of our submission and for their constructive feedback and suggestions. We have revised the manuscript to strengthen the empirical validation of the Anchor Quality Score (AQS) and Anchor-Guided Rectification using Compression (ARC), clarify the experimental protocol, and make our robustness claims more precise. The major changes are summarized below.

Major changes in the revised paper and appendix:

  1. Expanded adaptive-attack evaluation and refined robustness claims (Section 4.2; Appendix A.14, Table 9). We extend the evaluation under end-to-end, defense-aware EOT+BPDA attacks from three fine-grained datasets to all ten benchmarks, comparing ARC with CSR and MAC. ARC achieves the highest mean adaptive robust accuracy (34.9%, versus 32.6% for MAC and 19.9% for CSR) and leads on six of the ten datasets; MAC performs better on the other four. We also report results under transferred AutoAttack and stronger PGD (ε = 4/255), explicitly acknowledging that MAC leads on average under the latter (48.1% versus ARC's 45.3%). Accordingly, we revise the introduction and contribution statements to emphasize the gains attributable to anchor choice without implying that ARC dominates under every adaptive or stronger-attack setting. Additionally, during the rebuttal, we conducted more computationally intensive adaptive evaluations using 50/100 PGD steps, three restarts, and per-image worst-case accuracy. These results, reported in our reviewer responses, reveal substantial reductions in absolute robustness under stronger attack optimization highlighting an important limitation of the current defense. Nevertheless, ARC remains competitive with MAC on ViT-B/16 (18.34% vs. 18.50%) and outperforms it on ViT-B/32 (19.06% vs. 15.87%) at ϵ=1/255\epsilon=1/255 under the 100-step evaluation.

  2. New held-out validation of anchor selection (Section 2.3; Appendix A.12, Figure 11). We add a cross-dataset experiment that computes AQS rankings on Stanford Cars, FGVC-Aircraft, and UCF101, then compares them with realized defense-performance rankings on held-out Food101 and ImageNet. The rankings remain strongly correlated (Spearman ρ = 0.96), with JPEG50 ranked first by both criteria. This provides additional evidence that AQS can identify a useful anchor beyond the datasets used for the original selection.

  3. New cross-attack generalization analysis (Appendix A.12, Figure 12). We test whether anchor rankings transfer between attack types. AQS and defense rankings agree when both are evaluated under AutoAttack (ρ = 0.95), while AQS rankings obtained under PGD remain predictive of defense performance under AutoAttack (ρ = 0.93). JPEG50 remains the top-ranked anchor in both settings, although some lower-ranked transformations change positions.

  4. Added AQS weight-sensitivity and component-ablation experiments (Section 4.3, Figure 5; Appendix A.13, Figure 13). We sweep the removal weight from 0.5 to 8 while keeping the other weights fixed. Rank agreement remains high (Spearman ρ ≥ 0.93; Kendall τ ≥ 0.81), and JPEG50 remains the selected anchor for removal weights from 2.5 to 8. We further add equal-weight and leave-one-component-out comparisons over the ten real transformation families, complementing the existing targeted-degenerate tests and clarifying the contributions of individual AQS components.

AuthorsOpenReview ↗

Summary of changes 2/2

  1. Expanded decomposition of ARC's gains (Section 4.3, Table 6). We replace the earlier budget-scaling-only comparison with an analysis of undefended CLIP, direct JPEG50 classification, gated JPEG50 classification, fixed-budget rectification, and full ARC, evaluated at ε = 1/255 and 4/255. The results distinguish the benefit of using JPEG50 as a rectification anchor from merely classifying its transformed output. Relative to fixed-budget rectification, anchor-consistency budget scaling adds 1.2 and 2.8 percentage points of robust accuracy at the two budgets, respectively.

  2. New gate-hyperparameter sensitivity study (Section 4.3, Figure 6). We evaluate ARC across temperature and gate-threshold settings on OxfordPets, Flowers102, and StanfordCars, reporting the clean–robust trade-off through individual sweeps and a full parameter grid. This makes the behavior of the soft consistency gate and the rationale for the reported default setting more transparent.

  3. Clarified the meaning and use of “development pairs” (Section 4.1; Appendix A.6.2). We explicitly describe these as clean/adversarial examples used only for offline anchor scoring and selection, not for training the defense. We now specify that 300 development examples per dataset–attack condition were drawn from the test splits of OxfordPets, Flowers102, and DTD, and from the full datasets of EuroSAT and Caltech101. Importantly, these development examples are excluded from the corresponding final evaluations, avoiding sample overlap between anchor selection and reported defense accuracy. We have also added an explicit cross-reference to this protocol in the main text.

  4. Corrected the theoretical derivation for the collateral degenerate (Appendix A.11.2). In response to a reviewer’s comment, we correct the post-normalization expression for collateral distortion in the synthetic DcolD_{\mathrm{col}} construction. The earlier simplification c=λ/sc = \lambda/s is valid only in the special case s=1s = 1; in general, ℓ2\ell_2 normalization introduces an additional attack-orthogonal residual. We now provide the full expression for c2c^2 and clarify that the collateral scores reported in Table 5 are calculated using the complete residual norm in Equation (2), rather than the simplified construction expression.

We hope these additional analyses and clarifications make the scope, strengths, and limitations of the proposed approach easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.