- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates backdoor rebound in purified CLIP models, where backdoor behavior suppressed by purification can re-emerge during subsequent benign fine-tuning. To address this issue, the authors propose ImmuneCLIP, a post-purification defense that estimates residual backdoor risk through trigger inversion and constructs an update probe bank to characterize potential downstream adaptation directions. Experiments across multiple backdoor attacks, purification defenses, and adaptation settings demonstrate that ImmuneCLIP can substantially reduce post-adaptation attack success rates.
Critical strengths
- The research perspective is practical. Pretrained CLIP models are often further fine-tuned by downstream users for different tasks. A low ASR immediately after purification does not necessarily guarantee that the model remains secure during subsequent fine-tuning. Investigating this potential security gap is meaningful.
- The authors investigate the relationship of backdoor rebound with parameter-update directions. This provides useful evidence that risk-increasing components of benign updates contribute to backdoor rebound.
Critical weaknesses
- The novelty is not sufficiently clarified. Lin et al. [1] have already observed that clean fine-tuning can unintentionally reactivate backdoor behavior after an unlearning-based defense. However, the manuscript does not discuss this relevant work or sufficiently distinguish its contributions from previous findings.
- The proposed definition of backdoor rebound is not fully validated by the experiments. The paper explicitly defines successful purification using both an ASR threshold and a clean-utility requirement. However, the authors repeatedly acknowledge that the current experimental records lack the delivery-time utility (U_0) and pre-defense utility (U_pre), preventing verification of the complete purification-success criterion.
- ImmuneCLIP depends on an estimated backdoor-risk function and a predefined bank of parameter-update directions. However, the defender has access to neither the true backdoor trigger nor the downstream user’s private training data and fine-tuning procedure. The practical generalizability of the defense insufficiently established.
Questions for authors
- How does the observed backdoor rebound differ from the backdoor reactivation during clean fine-tuning reported by Lin et al. [1]?
- What exactly does the statement "The current preview verifies the ASR threshold but does not claim full utility-gate passage" in Contribution 1 (page 2) mean?
- What do the uncertainty values (±) in Table 1 represent? How many independent experimental runs were conducted, and are these values standard deviations, standard errors, or confidence intervals?
- How well does ImmuneCLIP generalize to more recent CLIP architectures and different model scales? The current evaluation mainly uses RN50 and ViT-B/32, both of which are relatively early CLIP architectures.
[1] Lin et al. Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness. NeurIPS 2024.
Response to Reviewer GVWr
We thank the Reviewer for the comments. We first summarize our paper in plain words, and then give our point-to-point responses.
Our paper in short. A provider removes a backdoor from a CLIP model and delivers it to a user with a low ASR. The user fine-tunes it on clean data, and the backdoor returns (e.g., 7.5% → 55.9%). We find that the cause is one direction inside each ordinary update: removing it stops the return, while removing a random component of the same size does not. Our ImmuneCLIP trains the model before delivery so that ordinary updates no longer contain this direction, which keeps the worst ASR at 2.0–9.5% on four attacks.
W1 / Q1. Relation to Lin et al. (NeurIPS 2024). Our Reply: We thank the Reviewer for the reference. We have cited and discussed it in Sections 1 and 8 of the revised paper. Our contribution is not the observation that fine-tuning can bring a backdoor back. The differences are as follows:
W2 / Q2. The gate is not validated; what does the sentence in Contribution 1 mean? Our Reply: This sentence meant that our logs recorded the accuracy after downstream fine-tuning and not at delivery, so we labeled the utility part of the gate as unverified. This label was more cautious than necessary. As shown in General Response 1, the accuracy changes by only about one point during the workload (Figure 2b), and the ratio to the undefended accuracy is 0.88–0.98 for ImmuneCLIP and 0.89–1.00 for PAR and InverTune, all above 0.85. Therefore, every checkpoint behind our claims meets the definition of rebound. We have deleted this sentence and added the ratio column to Table 1.
W3. The defense depends on an estimated risk and a predefined bank, while the defender knows neither the trigger nor the user's data and procedure. Our Reply: Our evaluation tests each of the three unknowns:
In every row, we evaluate our defense on something that it has not seen, and the worst result is less than half of the weakest unprotected rebound.
Q3. More recent architectures and other scales. Our Reply: We choose RN50 and ViT-B/32 because BadCLIP, CleanCLIP, PAR and InverTune are all evaluated on them, so our numbers can be compared with the published ones. They cover a convolutional encoder and a transformer, and both full fine-tuning and LoRA. Besides, our method does not depend on the architecture: the risk score is defined on the model outputs, the probes are optimizer displacements of the trainable parameters, and Proposition 1 only assumes local smoothness. The cost (12.6 GPU-hours on RN50) grows only with the cost of a training step. Therefore, our method can be applied to ViT-B/16 and ViT-L/14 without any change.