acceptodds
Under review as a conference paper at ICLR 2027

AFOC: ADAPTIVE FREQUENCY-ORTHOGONAL COUNTERACTION FOR ROBUST CLIP INFERENCE

Abstract

Zero-shot CLIP inference is vulnerable to adversarial perturbations, yet many defenses require training, labels, or prompt optimization. We study the more constrained setting of frozen encoders, unchanged prompts, and no training data. We propose Adaptive Frequency-Orthogonal Counteraction (AFOC), a training-free defense that estimates input risk from local embedding instability and sensitivity to frequency attenuation. A unified risk score calibrates orthogonal exploration, trajectory momentum, Fourier-domain update shaping, and a soft final gate, allowing conservative corrections for stable inputs and stronger corrections for risky inputs. Our local analysis formalizes clean-drift control and characterizes the directional and spectral effects of the update; this analysis is mechanistic rather than a robustness certificate. An embedding-geometry diagnostic further shows that PGD can disrupt zero-shot alignment while preserving local visual neighborhoods, whereas AFOC recovers decisions by rearranging embeddings relative to fixed text anchors. Across CIFAR-10, CIFAR-100, and ImageNet, we evaluate both post-hoc defense against attacks on the base classifier and defense-aware EOT-PGD attacks. On CIFAR-100 with CLIP ViT-B/32, AFOC attains 30.18% accuracy under adaptive EOT-PGD and a descriptive average of 39.22% across the four reported attacks, while retaining 56.55% clean accuracy. On ImageNet, it attains 29.53% adaptive accuracy with a 0.11-percentage-point drop in clean accuracy. These results demonstrate a competitive fixed-prompt operating point and motivate broader end-to-end adaptive evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adversarial Robustness, CLIP, Zero-Shot Learning, Training-Free Defense, Inference-Time Defense, Risk Estimation, Frequency-Domain Shaping, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_T1poOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the adversarial vulnerability of zero-shot CLIP without modifying pretrained models or prompts. It proposes AFOC, a training-free defense combining geometric and spectral risk estimation with adaptive counteraction. Experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate substantial robustness improvements over existing test-time defenses while largely preserving clean accuracy.

Critical strengths

None

Critical weaknesses

The manuscript contains numerous instances of unnecessary defensive writing. Additionally, several figures suffer from poor visual quality and insufficient clarity. Figures 4, 7, and 9 appear to contain AI-generated watermarks, yet the AI usage declaration does not disclose the use of AI tools for figure generation. Furthermore, the blue curve in Figure 10(c) appears to have missing data segments.

Questions for authors
  1. The manuscript contains numerous instances of unnecessary defensive writing, which undermine the clarity and conciseness of the presentation. The authors are encouraged to carefully review the entire manuscript, eliminate redundant explanations and excessive qualifications, and adopt a more direct, precise, and objective academic writing style.
  2. The authors should carefully review all figures in the manuscript to ensure that they are visually clear. In particular, figures presenting experimental results should be generated directly from the underlying data using appropriate visualization tools, rather than relying on AI-generated images.
Reviewer_jAorOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a training-free test-time defense method for CLIP. They construct a risk score by combining local embedding instability and sensitivity to frequency attenuation, and use this score to adaptively control orthogonal exploration, trajectory momentum, Fourier-domain update shaping, and the final correction strength.

Critical strengths

1.The research problem is clear. The authors focus on the problem that fixed defense strategies cannot adapt to different levels of input risk, and use a unified risk score to control multiple optimization processes.

2.The experimental results reported in the paper are competitive. On CIFAR-10, CIFAR-100, and ImageNet, AFOC achieves significant improvements in adversarial performance compared with TTC and DOC, while largely preserving clean accuracy.

Critical weaknesses
  1. The experiments in Appendix D.7 show that the spectral risk cue alone already has strong ability to distinguish adversarial attacks, whereas the geometric risk cue is anti-aligned with some attacks. How, then, can the necessity and actual contribution of combining the two risk cues be explained?
  2. In Appendix D.6, the Square Attack experiment on CIFAR-100 uses a hybrid counteraction objective and gating hyperparameters that differ from those in the main experiments. This is unfair. How can this demonstrate the robustness of the default AFOC configuration under this gradient-free attack?
Questions for authors
  1. Could the authors compare the end-to-end defense performance when using only the geometric risk cue, only the spectral risk cue, and the fusion of both cues?
  2. Why does the CIFAR-100 Square Attack experiment use a different counteraction objective and gating parameters? Could the authors report the results of the default AFOC configuration under the same Square Attack setting?
Reviewer_u8xSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AFOC is an inference-time defense for frozen CLIP with unchanged prompts. It combines geometric and spectral sensitivity into a risk score controlling exploration, momentum, frequency shaping, and correction strength. Experiments evaluate post-hoc recovery and defense-aware robustness on CIFAR-10, CIFAR-100, and ImageNet.

Critical strengths
  1. Frozen encoders and unchanged prompts make the approach applicable when adaptation is unavailable. Reported improvements over TTC and DOC are substantial under the main protocols.
  2. The paper distinguishes attack protocols and reports gradient-free attacks, EOT sensitivity, and computational costs, enabling closer assessment of its limitations.
Critical weaknesses
  1. The dual-cue risk estimator is inadequately justified. Geometric risk has AUROC 0.00 for several attacks, while spectral risk alone reaches at least 0.999. Fusion reduces EOT-PGD AUROC to 0.755. Matched single-cue defense ablations are needed to establish whether the geometric term helps.
  2. The data-free claim conflicts with unclear calibration. The paper references held-out clean calibration, but its stated rule does not reproduce the deployed backbone offsets. Since bias changes substantially affect clean accuracy, parameter provenance and separation from evaluation data must be clarified.
Questions for authors
  1. Why does the CIFAR-100 Square Attack evaluation change the defense objective, bias, gate, and attack budget? What results are obtained against the unchanged main configuration?
  2. Does geometric risk improve defense accuracy over spectral-only risk under matched calibration and adaptive attacks, despite its reversed attack ranking?
  3. What data and selection procedure produced the deployed backbone offsets? How does this calibration reconcile with the data-free claim and the stated held-out calibration rule?

Comments on the paper

1 comment

The rebuttal addresses some concerns, but key issues remain unresolved. I therefore maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.