acceptodds
Withdrawn from ICLR 2027

Backdoor as Probe: Test-Time Adversarial Defense for CLIP

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time adversarial defense improves the robustness of vision-language foundation models such as CLIP without retraining. However, adversarial activation shifts are typically treated as distortions to suppress, rather than signals to exploit. We turn these shifts into defense signals by repurposing the trigger-to-target mechanism of backdoors. The key is to implant a defender-controlled backdoor as a probe that is weakly activated by clean inputs but strongly activated by adversarial shifts. Based on this insight, we propose Backdoor as Probe (BaP), a test-time adversarial defense for CLIP. BaP constructs the probe through a closed-form model edit to a selected MLP layer. It projects the average adversarial activation shift and a defender-specified semantic direction onto the layer's low-energy input and output activation subspaces to obtain the trigger and target directions, respectively. At inference time, adversarial inputs produce measurable responses along the target direction for detection. BaP then selectively rectifies detected inputs by optimizing a small perturbation that steers their representations away from adversarial shifts and toward the clean subspace. Experiments across 16 benchmarks show that BaP improves average robust accuracy from 1.0% to 52.3% while retaining clean accuracy, achieving performance comparable to state-of-the-art methods with up to a 5.7 X inference speedup. BaP further shows the generalization to adversarial attacks on large vision-language models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial defense, backdoor attack, vision-language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EzBKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BaP introduces a defender-controlled probe into CLIP through a closed-form rank-one MLP edit. The probe detects adversarial activation shifts and selectively triggers a three-step input rectification procedure. Across 16 datasets, BaP improves average robust accuracy from 1.0% to 52.3%, while reducing clean accuracy from 63.6% to 61.1%.

Critical strengths

1、The class-agnostic probe offers an interesting extension of backdoor-inspired defenses to open-vocabulary models, with a simple closed-form construction. 2、 Under the reported evaluation, BaP achieves competitive robustness with substantially lower inference latency than R-TPT. Experiments cover multiple datasets and CLIP backbones.

Critical weaknesses

• Incomplete adaptive evaluation. Equation (16) attacks the edited classifier and suppresses detection, but does not explicitly optimize against the final prediction after rectification. Adaptive evaluation is limited to 10-step PGD at 1/255, and the reported “AutoAttack” uses only targeted APGD. Moreover, the existing adaptive attack already reduces ImageNet robust accuracy from 39.6% to 27.2%, making stronger evaluation consequential. • Insufficient evidence for the model edit’s independent contribution. Equations (8)–(9) show that the detection score can be computed exactly as a linear readout of the original hidden activations. Detection separation alone therefore does not establish the benefit of editing the encoder. Table 7 also shows that adding the escape stage increases robustness from 22.3% to 52.3%, but lacks escape-only and unedited-model controls.

Questions for authors

• Are the main attacks generated against the complete defense? How does BaP perform under stronger adaptive attacks targeting the final rectified prediction, with additional iterations and restarts? • How does full BaP compare with an unedited encoder using the equivalent external detector and identical rectification, and with escape-only rectification? • How are the threshold, clean-subspace dimension, and initialization ξ_0 selected? How are zero or very small singular values handled in Equation (7)? • Can the authors verify that all decoded adversarial examples satisfy the stated perturbation bounds?

Reviewer_3qTuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a test-time defense, Backdoor as Probe (BaP), that uses a defender-controlled backdoor to detect and rectify adversarial examples. The key idea is that adversarial examples produce consistent shifts in the model's activations relative to clean samples. Therefore, the authors construct a trigger direction by projecting the mean adversarial activation shift into a low-energy input subspace of a chosen MLP layer. They also define a target direction using a semantic direction defined from the defender's side. They apply a closed-form model edit to connect the trigger to the target direction, while using a probe that remains inactive on clean data but not on adversarial shifts. During inference, adversarial samples can be detected if they produce responses in the target direction. When the response exceeds a pre-defined threshold, the method optimizes for an input perturbation that moves the representation away from its current adversarial feature and then pulls it toward the clean subspace. The authors show that BaP shows consistent improvement across multiple CLIP architectures and 16 zero-shot classification benchmarks, with additional testing on large vision-language models.

Critical strengths
  1. The paper is well written and motivated, and the method is clearly explained.

  2. One of the important strengths is the extension of backdoor-based defense to open-vocabulary models like CLIP

  3. The method is training-free and not too expensive computationally.

  4. The empirical evaluations use multiple CLIP backbones, attack budgets, and attack objectives

Critical weaknesses

Inadequate adaptive attacks:

  1. The trigger direction is defined using the PGD adversarial examples under the L-inf threat model. While the authors do test the defense against several gradient-based attacks, all of them remain L-infinity gradient-based attacks. It is not clear whether the proposed defense can withstand attacks that are significantly different from such attacks. The examples would be black box attacks or query-based attacks, attacks with different norms (L1, L2) , or even optimization-based minimal-distortion attacks such as Carlini-Wagner.

  2. While the paper does include an adaptive attack, it is mostly targeted at the detection mechanism. This seems reasonable to test, but a standard adaptive attack would be against the complete BaP pipeline.

Questions for authors
  1. Would the activation shifts resulting from the gradient-based attacks (like PGD) be similar to other types of attacks? (Check weakness point 1).

  2. Could the authors do an adaptive attack that specially targets the rectifying step of the BaP? In this, the detection mechanism should not be used, and especially test whether it is possible to generate an adversary by maximizing the final classification loss after unrolling steps and the subsequent clean-subspace repair step?

Reviewer_9DbSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BaP which is BACKDOOR AS PROBE: TEST-TIME ADVERSARIAL DEFENSE FOR CLIP, a test-time defense that helps CLIP handle adversarial images. The main idea is to deliberately insert a backdoor and use its response to detect attacks.

The authors modify a single MLP layer, using layer 6 by default. They first use clean images to identify directions in the layer’s activation space that have very little activity. They then measure the average activation shift caused by PGD attacks on 1,000 ImageNet samples at ε = 1/255 and project this shift onto those quiet directions to form a trigger. A text prompt, such as “a white teapot,” provides a target direction, which is also projected onto quiet directions in the output space. A rank-one weight edit connects the trigger to the target without any additional training.

At test time, BaP measures the response along the target direction and flags inputs whose response exceeds a fixed threshold. For flagged images, it applies a small counter-perturbation in three steps: two steps move the feature away from its current position, and the final step brings it closer to the clean subspace. Images that are not flagged skip this correction, keeping the extra computation low.

Across 16 zero-shot datasets with ViT-B/16 at ε = 1/255, the reported robust accuracy increases from 1.0% to 52.3%, while clean accuracy drops by about 2.5 percentage points. The authors also evaluate other backbones, stronger attacks, different attack objectives, LLaVA captioning, and a simple adaptive attack. They report roughly 5.7 times lower latency than R-TPT.

Critical strengths
  1. The low cost of an attack is the main strength of this paper for me. Adding the probe takes about 2.7 ms through a closed-form weight edit and requires no training. Only flagged images go through rectification, so clean images take almost the same time as plain CLIP: 3.75 ms versus 3.37 ms. Many other test-time defenses add substantial computation to every input. This makes BaP appealing from a practical standpoint.

  2. The evaluation covers a good range of settings, including 16 datasets, three CLIP backbones, several attack objectives, and a LLaVA captioning task. One useful finding is that the probe is built using only ImageNet PGD samples at ε = 1/255, yet it still helps in the other settings shown in Table 4. This suggests that the direction it captures can transfer beyond the dataset and attack used to build it, at least for the non-adaptive attacks tested.

Critical weaknesses
  1. My main concern is that the adaptive attack is too weak to show whether the detector can be bypassed. Detection comes down to a dot product between the hidden activation and a fixed direction t, followed by a threshold check. If an attack stays below that threshold, rectification never runs, so bypassing the detector also bypasses the defense. An attacker with both the original and edited weights could recover t from W′ − W. They might also estimate a similar direction using public CLIP and PGD samples. They could then try to cause misclassification while keeping the hidden activation shift orthogonal to t.

The attack in Eq. 16 does not fully address this concern. It uses only 10 steps, the ReLU penalty stops providing a gradient once p(x) is below τ, and it does not account for rectification. Even with this limited attack, robust accuracy drops from 39.6% to 27.2%. Accuracy rises again at β = 1, which makes me question whether the attack was tuned well enough. For a defense that relies on detection, this remains a major unanswered question.

  1. I am also unsure what the backdoor edit adds. The extra response added by the edit, ŷᵀΔW h(x), equals γ tᵀh(x)/‖t‖², which is just a linear projection onto t. This suggests that the same detector could be computed without changing the model weights. Table 7 shows that adding the escape loss raises robust accuracy from 21.3% to 52.3%, so much of the improvement appears to come from this step. The escape loss also seems very similar to the TTC counterattack idea.

It would be helpful to include comparisons with the same linear detector without the weight edit, TTC applied only to images flagged by that detector, and CSR (Nie et al., 2026a) and Liu et al. (2026). The paper describes CSR as the closest prior work, and CSR evaluates on the same 16 datasets, yet neither method appears in the results. Without these comparisons, I cannot tell what the backdoor edit adds beyond a linear detector combined with an existing correction method.

  1. There are also two citation issues:

(a) Line 44 cites He et al. (2016) as prior work on adversarial defense, but this is the ResNet paper. It does not support the claim about adversarial defenses or backdoors. The authors should replace it with the relevant reference.

(b) Line 107 cites Madry et al. (2018) to support the claim that CLIP is vulnerable to adversarial attacks. That paper predates CLIP and studies models on MNIST and CIFAR, so it cannot support a claim specifically about CLIP. A reference that directly evaluates attacks on CLIP is needed.

Questions for authors
  1. How does BaP perform against an attacker who knows t? For example, the attacker could use a penalty such as λ(tᵀ(h(x+δ) − h(x)))² to keep the hidden activation shift orthogonal to t. Please test this with at least 100 attack steps and pass gradients through rectification, using BPDA or EOT where needed. Strong results under this setting would make the defense more convincing.

  2. Can the authors compare BaP with the same detector, tᵀh(x), without the weight edit, and with TTC applied only to images flagged by that detector? These comparisons would help show what the backdoor edit adds. Comparisons with CSR and Liu et al. (2026) would also be useful, especially since the paper identifies CSR as the closest related work.

  3. How did you choose τ = 1.93? Was any test data used to set it? What is the process or what was the intuition of choosing that specific number?Please report the true positive and false positive rates for each dataset at this threshold. It would also help to report false positives on images with common corruptions, such as JPEG compression, noise, and blur.

  4. Clean accuracy drops by up to 6.4 percentage points on StanfordCars and 5.5 points on OxfordPets. How much of this drop comes from the weight edit, and how much does it comes from clean images being incorrectly flagged and rectified?

  5. The rectification budget is 4/255, while the attack budget is 1/255. How does BaP perform when both budgets are set to 1/255?

  6. Are the 1,000 ImageNet images used to build the probe separate from the ImageNet images used for evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.