acceptodds
Under review as a conference paper at ICLR 2027

ImmuneCLIP: Adaptation-Stable Defense Against Backdoor Rebound in Purified CLIP Models

Abstract

Multimodal contrastive learning models such as CLIP align images and text in a shared representation and are widely adapted to downstream tasks. Yet existing backdoor purification defenses are typically evaluated only immediately after purification, leaving their security during later adaptation largely unexplored. We identify backdoor rebound, a failure mode in which a purified model initially achieves a low attack success rate (ASR) but regains strong backdoor behavior after benign downstream fine-tuning. By analyzing fine-tuning update directions, we find that clean optimization can move the model along parameter-space directions that reactivate residual backdoor behavior. To address this threat, we introduce ImmuneCLIP, an adaptation-stable defense that protects purified models beyond the initial purification checkpoint. ImmuneCLIP first uses trigger inversion to construct a differentiable proxy for residual backdoor risk. It then collects parameter-update directions from representative clean adaptation procedures to build an adaptation probe bank. Based on these probes, we define reactivation susceptibility as the largest first-order increase in proxy risk along clean adaptation directions. ImmuneCLIP jointly minimizes residual backdoor risk and reactivation susceptibility at the purified checkpoint and at nearby states reached through simulated clean updates. Contrastive regularization and knowledge distillation are further used to preserve clean performance. We evaluate four backdoor attacks, four purification defenses, and five adaptation pipelines. ImmuneCLIP's worst-case post-adaptation ASR anchors range from 2.0±0.7% to 9.5±1.0%, with final clean-accuracy anchors between 50.1±0.4% and 54.9±0.4%. The evaluation measures both delivery-time suppression and resistance to backdoor rebound during subsequent adaptation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Backdoor Defense, Vision-Language Models, Backdoor Purification, Backdoor Rebound, Downstream Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GVWrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates backdoor rebound in purified CLIP models, where backdoor behavior suppressed by purification can re-emerge during subsequent benign fine-tuning. To address this issue, the authors propose ImmuneCLIP, a post-purification defense that estimates residual backdoor risk through trigger inversion and constructs an update probe bank to characterize potential downstream adaptation directions. Experiments across multiple backdoor attacks, purification defenses, and adaptation settings demonstrate that ImmuneCLIP can substantially reduce post-adaptation attack success rates.

Critical strengths
  1. The research perspective is practical. Pretrained CLIP models are often further fine-tuned by downstream users for different tasks. A low ASR immediately after purification does not necessarily guarantee that the model remains secure during subsequent fine-tuning. Investigating this potential security gap is meaningful.
  2. The authors investigate the relationship of backdoor rebound with parameter-update directions. This provides useful evidence that risk-increasing components of benign updates contribute to backdoor rebound.
Critical weaknesses
  1. The novelty is not sufficiently clarified. Lin et al. [1] have already observed that clean fine-tuning can unintentionally reactivate backdoor behavior after an unlearning-based defense. However, the manuscript does not discuss this relevant work or sufficiently distinguish its contributions from previous findings.
  2. The proposed definition of backdoor rebound is not fully validated by the experiments. The paper explicitly defines successful purification using both an ASR threshold and a clean-utility requirement. However, the authors repeatedly acknowledge that the current experimental records lack the delivery-time utility (U_0) and pre-defense utility (U_pre), preventing verification of the complete purification-success criterion.
  3. ImmuneCLIP depends on an estimated backdoor-risk function and a predefined bank of parameter-update directions. However, the defender has access to neither the true backdoor trigger nor the downstream user’s private training data and fine-tuning procedure. The practical generalizability of the defense insufficiently established.
Questions for authors
  1. How does the observed backdoor rebound differ from the backdoor reactivation during clean fine-tuning reported by Lin et al. [1]?
  2. What exactly does the statement "The current preview verifies the ASR threshold but does not claim full utility-gate passage" in Contribution 1 (page 2) mean?
  3. What do the uncertainty values (±) in Table 1 represent? How many independent experimental runs were conducted, and are these values standard deviations, standard errors, or confidence intervals?
  4. How well does ImmuneCLIP generalize to more recent CLIP architectures and different model scales? The current evaluation mainly uses RN50 and ViT-B/32, both of which are relatively early CLIP architectures.

[1] Lin et al. Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness. NeurIPS 2024.

1 comment
AuthorsOpenReview ↗

Response to Reviewer GVWr

We thank the Reviewer for the comments. We first summarize our paper in plain words, and then give our point-to-point responses.

Our paper in short. A provider removes a backdoor from a CLIP model and delivers it to a user with a low ASR. The user fine-tunes it on clean data, and the backdoor returns (e.g., 7.5% → 55.9%). We find that the cause is one direction inside each ordinary update: removing it stops the return, while removing a random component of the same size does not. Our ImmuneCLIP trains the model before delivery so that ordinary updates no longer contain this direction, which keeps the worst ASR at 2.0–9.5% on four attacks.

W1 / Q1. Relation to Lin et al. (NeurIPS 2024). Our Reply: We thank the Reviewer for the reference. We have cited and discussed it in Sections 1 and 8 of the revised paper. Our contribution is not the observation that fine-tuning can bring a backdoor back. The differences are as follows:

Lin et al. (2024)Ours
Who runs the reactivating fine-tuningThe defender, inside its own defenseA third-party user after delivery
Can the defender change it?Yes; the remedy is a modified fine-tuning ruleNo; neither the data nor the optimizer
ExplanationBackdoor neurons are more active (larger gradient norm)Benign updates have a positive component along the risk-increasing direction
Where the defense actsInside the defender's fine-tuningOn the delivered weights
ModelsSupervised image classifiersMultimodal contrastive CLIP
  1. Setting. In Lin et al., the reactivating fine-tuning belongs to the defender, so it can be replaced. In our work, it belongs to the user, so our defense has to act on the delivered weights.
  2. Explanation. In Figure 2b, removing a random component of the same norm still lets the ASR return to about 56%, while removing the component along the risk direction keeps it at 0.04%. Therefore, the direction matters, not the magnitude or activeness of the updates.
  3. Scope. Our Table 1 shows that the latest CLIP purifiers, PAR and InverTune, pass delivery with an ASR as low as 0.0% and then return to 31.6–85.5% under clean third-party fine-tuning. This was not known for these methods.

W2 / Q2. The gate is not validated; what does the sentence in Contribution 1 mean? Our Reply: This sentence meant that our logs recorded the accuracy after downstream fine-tuning and not at delivery, so we labeled the utility part of the gate as unverified. This label was more cautious than necessary. As shown in General Response 1, the accuracy changes by only about one point during the workload (Figure 2b), and the ratio to the undefended accuracy is 0.88–0.98 for ImmuneCLIP and 0.89–1.00 for PAR and InverTune, all above 0.85. Therefore, every checkpoint behind our claims meets the definition of rebound. We have deleted this sentence and added the ratio column to Table 1.

W3. The defense depends on an estimated risk and a predefined bank, while the defender knows neither the trigger nor the user's data and procedure. Our Reply: Our evaluation tests each of the three unknowns:

Unknown to the defenderWhat ImmuneCLIP usesEvidence in the paper
Trigger and targetInverted trigger, inferred targetAll four attacks in Table 1 are defended without the planted trigger. Gradient distance between inverted and planted risk is 0.11. An attacker who optimizes against the inversion reaches 24.6% (82.8% without the defense).
User's data10K clean pairs disjoint from the user'sDownstream data in Table 1 are disjoint from the defender's. Cross-domain MSCOCO fine-tuning: 7.8%.
User's procedureA published family of proceduresOptimizers, a task, an architecture and a parameterization absent from the bank: 7.8–17.6% (Table 5).

In every row, we evaluate our defense on something that it has not seen, and the worst result is less than half of the weakest unprotected rebound.

Q3. More recent architectures and other scales. Our Reply: We choose RN50 and ViT-B/32 because BadCLIP, CleanCLIP, PAR and InverTune are all evaluated on them, so our numbers can be compared with the published ones. They cover a convolutional encoder and a transformer, and both full fine-tuning and LoRA. Besides, our method does not depend on the architecture: the risk score is defined on the model outputs, the probes are optimizer displacements of the trainable parameters, and Proposition 1 only assumes local smoothness. The cost (12.6 GPU-hours on RN50) grows only with the cost of a training step. Therefore, our method can be applied to ViT-B/16 and ViT-L/14 without any change.

Reviewer_CaQEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ImmuneCLIP is a post-purification stage for backdoored CLIP. It inverts a trigger and a target to build a proxy risk, collects real clean optimizer steps, and treats reactivation susceptibility as the largest increase of that proxy along those steps. Training penalizes this increase at the delivered checkpoint and after one simulated clean step, caps the proxy risk, and keeps utility with a contrastive loss plus distillation from the purified model. The main bank has eight directions from four clean batches. On the reported ResNet-50 runs the worst post-adaptation attack success stays in the single digits, while the edit that drives attack success to zero in the diagnostic uses the planted-trigger direction.

Critical strengths

The method leaves a clean update unchanged, replaces its risk-increasing part by a random direction of the same length, or deletes only that part. The first two restore a high attack success rate, and the third keeps it near zero at similar clean accuracy. Rebound is therefore carried by one direction of the clean update, rather than by fine-tuning in general.

Critical weaknesses
  1. ImmuneCLIP does not delete that direction. It inverts a trigger, freezes eight clean update directions, and penalizes growth of the proxy along those directions at the delivered weights and after one simulated step. The paper’s bound reaches the planted trigger only when later updates stay close to those eight directions, and the main table uses updates from that same family, so the trained objective is not the edit that was shown to stop rebound.

  2. A checkpoint enters the rebound summary only when attack success at delivery is at most 10% and clean accuracy at delivery is at least 85% of the accuracy before purification. The released records omit the two delivery-time accuracy numbers this check uses, so the single-digit attack success and the final accuracy in the abstract are not the endpoint the paper defined.

Questions for authors

For a clean update that does not stay close to the eight probes, what is the worst post-delivery attack success of the planted trigger? This asks whether the trained objective controls the risk outside the probe bank.

1 comment
AuthorsOpenReview ↗

Response to Reviewer CaQE

We thank the Reviewer for the precise summary and for pointing out the central question. Our point-to-point responses are given below.

W1. The trained objective is not the edit that was shown to stop rebound.

Our Reply: The two are the same operation, and they are only applied in different places.

  1. Our diagnostic edit replaces each update Δ_t by Δ_t − [⟨Δ_t, ŝ_t⟩]_+ ŝ_t, where ŝ_t is the unit gradient of the risk. It removes the part of the update that has a positive inner product with the risk gradient.
  2. Our ImmuneCLIP trains the delivered weights so that [⟨∇S, u⟩]_+ = 0 for the allowed update directions u, i.e., ρ = 0. At such a checkpoint, the bracket in the edit is zero for every allowed update, and the edit becomes the identity: there is nothing left to remove.

The edit cannot be deployed, because it needs the planted trigger and the control of the user's optimizer. Our objective is the deployable form of the same condition: we move it from the user's optimizer into the delivered weights.

There are two approximations between the two. Proposition 1 names both of them, and we measure both in the paper: we use the inverted trigger instead of the planted one (gradient distance ε_q = 0.11), and we use a finite bank instead of all updates (ε_u = 0.36, a median angle of 21°). Besides, our reach term extends the condition from the delivered checkpoint to the states after a simulated step, and our ablation shows that this is important (14.2% → 6.4%, F vs. H). The final results agree with this equivalence: the susceptibility decreases from 0.44 to 0.05 (Table 4), and the worst ASR decreases from 82.8% to 6.4%.

Q1. For a clean update that does not stay close to the eight probes, what is the worst post-delivery ASR of the planted trigger?

Our Reply: We have measured this in our paper. The eight probes are raw-gradient and sign-preconditioned SGD updates on RN50. The following updates are outside this set, and the ASR is measured on the planted trigger:

Update outside the bankWorst post-delivery ASR
MSCOCO retrieval fine-tuning (different data and task)7.8%
SGD-momentum10.3%
ViT-B/32 LoRA (low-rank update subspace)12.8%
AdamW at 5× the probe learning rate17.6%
Backdoor optimized against the bank's coverage (adaptive attack)21.9%

The worst value for a benign update is 17.6%, and it is 21.9% when an attacker targets the coverage on purpose. The same purified checkpoints without ImmuneCLIP reach 46.9–82.8%. Therefore, our objective controls most of the risk outside the bank.

There are 2 reasons for this result:

  1. Our eight probes reach a median angle of 21° to the real updates in a space of millions of parameters, where unrelated directions are close to 90°. Clean contrastive updates concentrate in a low-dimensional subspace, and a small bank covers most of it. An update that is far from all probes is also far from this subspace, and clean training on image–text pairs does not move the model there.
  2. Our safety anchor lowers the risk level itself, and our distillation term keeps the model close to the purified checkpoint. Neither of them depends on the bank.

W2. The delivery-time accuracy numbers are missing.

Our Reply: Please see General Response 1 for details. Evaluated from Table 1, our four ImmuneCLIP checkpoints keep 0.88–0.98 of the undefended accuracy, and PAR and InverTune keep 0.89–1.00. Therefore, the single-digit ASR and the final accuracy in our abstract refer to checkpoints that meet both conditions of the endpoint defined in Section 2.

Reviewer_nL36OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that purification-based defenses against backdoor attacks in multimodal contrastive learning models are evaluated only when checkpoints are released, and that this protection expires throughout the model’s lifecycle following benign user adaptation (such as lightweight fine-tuning). The authors refer to this phenomenon as “backdoor rebound”. An analysis of this failure mode shows that removing the risk-increasing component of clean optimizer updates effectively suppresses backdoor rebound and motivates the proposed defense, ImmuneCLIP. The final design uses trigger inversion as a proxy for the backdoor (which is unknown to the end user in the paper’s threat model) and minimizes proxy risk over a set of sampled optimizer updates both at the current checkpoint and at reachable future update states. The evaluation covers four attacks and four additional defenses on ResNet-50 CLIP and ViT-B/32 under different degrees of fine-tuning, and shows a significant reduction in post-adaptation attack success rates using their method.

Critical strengths

The paper exposes a substantial gap in the way defenses against CLIP backdoor attacks are evaluated and the impact of the work is clear. Table 1 in particular shows even the strongest recent purifiers lowering attack success rates at delivery and then rebounding following benign fine-tuning (PAR going from 7.5% to 55.9% on BadCLIP-Align and 2.3% to 80.9% on BadEncoder, InverTune going from 0.0% to 82.8% on BadCLIP-Align and 0.0% to 64.5% on BadCLIP). The delivery-time rates are clearly low enough to certify these checkpoints. CLIP models are routinely fine-tuned after release, and used within embodied and robotic perception stacks, so this result alone should impact how future works report their results.

Critical weaknesses

Several parts of the evaluation raise concern. In Section 6 and Appendix C the paper states that the ablation and extended-setting results are single runs. In Section 6.1 it is stated that the authors cannot confirm that checkpoints retain at least 85% of their original clean accuracy at delivery as the required records are missing, and ImmuneCLIP’s final clean accuracy is noticeably lower than some baselines (50.1% compared to 56.8% for InverTune on BadCLIP in Table 1), so the utility cost of the defense cannot be fully assessed. The no-backdoor control results for the main benchmark are also missing from Section 6.2. Section 3.3 reports that clean and purified-clean controls remain below 1% ASR in the diagnostic run but it remains unclear whether the same holds for the Table 1 experiments. It remains plausible that the workload itself contributes to the ASR increase and including these controls would help separate the effect of the backdoor from the effect of the workload.

Although the threat model states that the defender does not have access to the downstream user’s data, it does not state whether the defender knows the user’s training procedure, apart from Section 7’s recommendation that model providers “declare the defended adaptation family and path-length budget.” Such knowledge appears necessary for effective protection. ImmuneCLIP exceeds the 10% threshold under ViT-B/32 LoRA (12.8%) and higher-rate AdamW (17.6%). Section 4.2 states that the probe bank is designed to capture “adaptive scaling, and trainable parameter scope,” and LoRA is listed among the considered adaptations in Section 2. However, the paper does not report an experiment that adds AdamW or LoRA probes to the bank and re-measures ASR. It would also help to discuss how a user would know whether their fine-tuning falls within the protected family, for example by reporting reactivation susceptibility after adaptation.

Questions for authors

Which purifier is ImmuneCLIP applied on top of in Table 1? Table 2’s reference row seems to match InverTune for BadCLIP-Align but PAR for BadCLIP. What do the +/- values represent? Please make this explicit. Multi-seed results for the cells near the 10% threshold would be helpful. Does rebound persist at more typical fine-tuning settings (such as a learning rate of 1e-5 and longer training)? How is the matched random direction in Section 3.3 chosen? A random direction is nearly orthogonal to the risk gradient in a high dimensional space. As a result some kind of structured non-backdoor direction might be a better control and should be added. How does backdoor rebounding relate to the finding of Qi et al. (2024) cited in Appendix D that benign fine-tuning reverts safety-aligned behavior in language models? A short discussion in the main text would help position the paper’s contribution.

1 comment
AuthorsOpenReview ↗

Response to Reviewer nL36

We thank the Reviewer for the constructive comments. Our responses are given below.

W1a. The 85% check; ImmuneCLIP's accuracy is lower than some baselines. Our Reply: The check can be evaluated from Table 1: our four checkpoints keep 0.88–0.98 of the undefended accuracy (see General Response 1). For the cost, we think accuracy should be compared together with security. On BadCLIP, InverTune keeps 56.8% accuracy but its post-adaptation ASR is 64.5%, while ours are 50.1% and 5.2%. On average, our accuracy is 52.4%, between PAR (50.6%) and InverTune (55.2%).

W1b. No-backdoor controls; the workload itself might raise ASR. Our Reply: Three results in our paper rule this out. 1) Under the same workload (Section 3), clean and purified-clean models stay below 1% ASR. 2) Our workload removes target-class captions and near-duplicate images, so the data cannot teach the trigger–target association. 3) Our ImmuneCLIP checkpoints go through the same workload with the same triggers, and their ASR stays at 2.0–9.5%. Therefore, the workload itself does not raise the ASR.

W2a. Does the defender know the user's training procedure? Our Reply: No. The defender knows neither the user's data nor the exact procedure. The defender chooses a family of procedures, builds the probe bank from it, and publishes the family with the checkpoint. Like the perturbation set in adversarial robustness, we claim protection inside the family and measure it outside.

W2b. No experiment adds AdamW or LoRA probes to the bank. Our Reply: Our framework supports this: adding a pipeline only changes the configuration (Section 4.2), not our method. Two results show its effect. 1) Widening the bank gives the largest gain in our ablation: the ASR decreases from 41.9% to 18.6% on BadCLIP-Align and from 17.5% to 9.8% on BadCLIP (D vs. E). 2) Our bank already contains sign-preconditioned updates, the limiting form of Adam. This is why AdamW at 5× the learning rate stays at 17.6%. Besides, building a bank costs only 1.1 GPU-hours (Table 7). For LoRA, we obtain 12.8% without any LoRA probe, so most of our protection transfers.

W2c. How would a user know whether their fine-tuning is protected? Our Reply: The provider releases the probe pipelines with the checkpoint, and the user computes the angle between their own updates and the nearest probe (median 21° for our main workload). We also agree to report ρ̂_proxy after adaptation: it is cheap and predicts the post-adaptation ASR well (Spearman 0.98).

Q1. Which purifier is ImmuneCLIP applied on top of? Our Reply: Our ImmuneCLIP does not depend on a specific purifier. For Table 2, we use one fixed starting point for each attack: InverTune for BadCLIP-Align and PAR for BadCLIP. This is why Variant A matches different rows in Table 1. Within each attack, all variants share the same purified checkpoint, so the improvements come from our components and not from the purifier.

Q2. Does rebound persist at lr 1e-5 and longer training? Our Reply: At lr 1e-6, the PAR checkpoint already regains about 50 points of ASR within 50 steps (Figure 2a). Our normalized exposure is a ratio of displacements, so to first order it does not change with the step size. Therefore, we expect the rebound at 1e-5 to be at least as fast. For longer schedules, the provider can declare a larger path-length budget.

Q3. How is the matched-random direction chosen? Our Reply: At each step, our matched-random arm removes a random component with the same norm as the risk-projected arm, and matches the final update norm. This rules out the update magnitude as the explanation. Our paper also has a structured control: the exposure of the same benign updates is below 0.02 on clean models and 0.19–0.24 on purified backdoored models. Therefore, the alignment is specific to the backdoor history.

Q4. Relation to Qi et al. (2024). Our Reply: Both works show that benign fine-tuning can undo a safety property. However, alignment is a broad behavioral constraint, while a backdoor is a single trigger–target association. This allows us to define a scalar risk, find the risk-increasing component of each update, and remove it (Section 3.3). Our contribution is this directional explanation and a pre-delivery defense built on it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.