- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies vision-languge models backdoor defense. The main idea is to identify projection fragility. Based on the clean-loss increases under projection perurbtation, the proposed PSR methods find backdoor-associated projection output channels and restores their parameters. Experiments on different backdoor attacks show that PSR can reduce attack success rates.
Critical strengths
- The paper solves backdoor defense from a different perspective. Different from fine-tuning or inference phases operation, this papers proposes a lightweight post-training purification. The method only needs small amount of data. The three phases are also naturally applied to this task. Hence, overall, the work is interesting.
- The method is evaluated on 6 attack methods over different datasets. The ASR reduction is clear, showing strong empirical performances.
Critical weaknesses
My biggest concern is about the attack and defense setting. It assumes only the vision-to-language projection layer is tunable while the visual encoder and language model are frozen. This is quite strong and in many practical applications the assumption may not hold. For example, LoRA updates on language models or some prompt learning of input tokens.
Even more, the defense method requires the pretained clean base for restoration, which is typically infeasible. If clean weights are available, why not simply replace all weights without perturb and select?
Questions for authors
-
The authors are encouraged to discuss about real scenarios of the attack/defense settings, and how the proposed method can be extend to more general cases other than projection layer-only. Why do we need to select particular rwos to copy from clean base instead of copying all?
-
Methodwise, figure 1 shows that clean model also seems to have a non-ignorable loss increase. How to ensure the top K smallest channels will not mix in good ones?
4 comments
Dear Reviewer sBxG,
In our earlier response, we clarified the projector-adaptation setting and trusted-base requirement, and explained why selective restoration preserves task-specific adaptation that full replacement would discard. We have since added evaluations on three additional VLMs and a dual-trigger setting (Table 5), and compared PSR with RACER, a recent post-training backdoor defense for VLMs (Tables 1–2). These additions extend the evaluation across model families and scales and provide a comparison with another post-training defense.
We hope these results help clarify the scope and performance of PSR. We would value your feedback and are glad to address any remaining questions.
Best regards, The Authors
Thanks for these clarifications. "PSR requires the original pretrained checkpoint" still does not convince me why it is practical. I keep this concerns.
Thank you for the follow-up. If the pretrained checkpoint is unavailable, PSR can instead zero the selected channels. Our ablation (Table 3) shows that this variant reduces ASR to 0.00%, although it incurs a slightly larger CU drop than pretrained restoration. Its backdoor suppression remains competitive with RACER, a recently proposed post-training defense (Table 1).
Response to Reviewer sBxG
Thank you for your comments. We clarify the deployment setting and the purpose of selective restoration below.
W1. The “projection layer-only” assumption.
We target developers and deployers adapting open-weight VLMs through their projectors, where freezing the backbones enables efficient adaptation. PSR repairs the resulting interface after adaptation on untrusted data and before deployment. LoRA and prompt learning modify different components and require corresponding extensions of the defense; our current evaluation covers projector adaptation.
W2. The “pretrained clean base” and why not “replace all weights.”
PSR requires the original pretrained checkpoint, which developers can retain or obtain from a known public release, and which may also be maintained internally by a model provider and made available to authorized personnel. It does not require a clean version of the task-adapted model.
During poisoned fine-tuning, the model learns the downstream task while also acquiring the backdoor. Reverting all adapted parameters to the base therefore removes both the malicious changes and the useful task-specific knowledge acquired during fine-tuning. This explains the lower clean utility of full restoration in Table 7: the base model has not undergone this task-specific adaptation. For Qwen3-VL-8B on VQAv2, full restoration leaves accuracy near zero, whereas undefended settings reach around 70-80% (Table 7). Selective restoration aims to suppress the backdoor while retaining these adaptation gains (Appendix B.3, Table 7; Figure 4).
Answers to the questions
Q1. Real scenarios, broader adaptation, and selective copying?
The intended use is to repair a projector after adaptation on external data or before deployment of an externally adapted projector, provided the corresponding trusted base checkpoint is available either through a known public release or through authorized access to a checkpoint retained by the model provider. Extension to LoRA or learned prompts would require scoring and repairing the corresponding adapted components, followed by separate evaluation. Copying only selected rows preserves task-specific changes that full replacement would discard.
Q2. Can the “top K smallest channels” include good channels?
Selected channels may also contribute to normal behavior, which is why PSR does not rank channels by perturbation sensitivity alone. The learned θ jointly accounts for projection fragility and clean-task preservation: the unperturbed clean-loss term discourages suppressing channels needed for the task. The new backdoor-output probability experiment connects the perturbation signal to attack-related behavior, and the perturbation ablation shows that this signal contributes to effective purification (Figure 2; Tables 3 and 6).
PSR further limits the impact on useful channels by restoring their pretrained parameters rather than deleting them. For the same selected channels, restoration preserves substantially more clean utility than zeroing while still suppressing the backdoor (Table 3). Thus, PSR addresses the overlap between useful and backdoor-related functionality through both its scoring objective and its repair operation, without requiring a perfect separation of the two.