acceptodds
Under review as a conference paper at ICLR 2027

Selective Channel Restoration for Backdoored Vision-Language Models

Abstract

Vision-language models (VLMs) exhibit strong multimodal capabilities but remain vulnerable to backdoors implanted through poisoned fine-tuning data. Existing defenses often require extensive parameter updates during fine-tuning or incur per-query overhead during inference. To address these limitations, we propose Perturb-Select-Restore (PSR), a post-training defense that performs sparse updates to the projection interface and introduces no additional computation during inference. We reveal that backdoored VLM projectors are substantially more sensitive to bounded perturbations than clean VLM projectors, a phenomenon we term projection fragility. Building on this finding, PSR identifies the output channels most sensitive to perturbations in each projection layer of a backdoored VLM and restores their parameters to the corresponding pretrained values. Experiments across multiple tasks show that PSR reduces attack success rates to near zero while preserving clean-task performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Backdoor Defense, Post-training Defense, Trustworthy AI, Fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_sBxGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies vision-languge models backdoor defense. The main idea is to identify projection fragility. Based on the clean-loss increases under projection perurbtation, the proposed PSR methods find backdoor-associated projection output channels and restores their parameters. Experiments on different backdoor attacks show that PSR can reduce attack success rates.

Critical strengths
  • The paper solves backdoor defense from a different perspective. Different from fine-tuning or inference phases operation, this papers proposes a lightweight post-training purification. The method only needs small amount of data. The three phases are also naturally applied to this task. Hence, overall, the work is interesting.
  • The method is evaluated on 6 attack methods over different datasets. The ASR reduction is clear, showing strong empirical performances.
Critical weaknesses

My biggest concern is about the attack and defense setting. It assumes only the vision-to-language projection layer is tunable while the visual encoder and language model are frozen. This is quite strong and in many practical applications the assumption may not hold. For example, LoRA updates on language models or some prompt learning of input tokens.

Even more, the defense method requires the pretained clean base for restoration, which is typically infeasible. If clean weights are available, why not simply replace all weights without perturb and select?

Questions for authors
  1. The authors are encouraged to discuss about real scenarios of the attack/defense settings, and how the proposed method can be extend to more general cases other than projection layer-only. Why do we need to select particular rwos to copy from clean base instead of copying all?

  2. Methodwise, figure 1 shows that clean model also seems to have a non-ignorable loss increase. How to ensure the top K smallest channels will not mix in good ones?

4 comments
AuthorsOpenReview ↗

Response to Reviewer sBxG

Thank you for your comments. We clarify the deployment setting and the purpose of selective restoration below.

W1. The “projection layer-only” assumption.

We target developers and deployers adapting open-weight VLMs through their projectors, where freezing the backbones enables efficient adaptation. PSR repairs the resulting interface after adaptation on untrusted data and before deployment. LoRA and prompt learning modify different components and require corresponding extensions of the defense; our current evaluation covers projector adaptation.

W2. The “pretrained clean base” and why not “replace all weights.”

PSR requires the original pretrained checkpoint, which developers can retain or obtain from a known public release, and which may also be maintained internally by a model provider and made available to authorized personnel. It does not require a clean version of the task-adapted model.

During poisoned fine-tuning, the model learns the downstream task while also acquiring the backdoor. Reverting all adapted parameters to the base therefore removes both the malicious changes and the useful task-specific knowledge acquired during fine-tuning. This explains the lower clean utility of full restoration in Table 7: the base model has not undergone this task-specific adaptation. For Qwen3-VL-8B on VQAv2, full restoration leaves accuracy near zero, whereas undefended settings reach around 70-80% (Table 7). Selective restoration aims to suppress the backdoor while retaining these adaptation gains (Appendix B.3, Table 7; Figure 4).

Answers to the questions

Q1. Real scenarios, broader adaptation, and selective copying?

The intended use is to repair a projector after adaptation on external data or before deployment of an externally adapted projector, provided the corresponding trusted base checkpoint is available either through a known public release or through authorized access to a checkpoint retained by the model provider. Extension to LoRA or learned prompts would require scoring and repairing the corresponding adapted components, followed by separate evaluation. Copying only selected rows preserves task-specific changes that full replacement would discard.

Q2. Can the “top K smallest channels” include good channels?

Selected channels may also contribute to normal behavior, which is why PSR does not rank channels by perturbation sensitivity alone. The learned θ jointly accounts for projection fragility and clean-task preservation: the unperturbed clean-loss term discourages suppressing channels needed for the task. The new backdoor-output probability experiment connects the perturbation signal to attack-related behavior, and the perturbation ablation shows that this signal contributes to effective purification (Figure 2; Tables 3 and 6).

PSR further limits the impact on useful channels by restoring their pretrained parameters rather than deleting them. For the same selected channels, restoration preserves substantially more clean utility than zeroing while still suppressing the backdoor (Table 3). Thus, PSR addresses the overlap between useful and backdoor-related functionality through both its scoring objective and its repair operation, without requiring a perfect separation of the two.

Dear Reviewer sBxG,

In our earlier response, we clarified the projector-adaptation setting and trusted-base requirement, and explained why selective restoration preserves task-specific adaptation that full replacement would discard. We have since added evaluations on three additional VLMs and a dual-trigger setting (Table 5), and compared PSR with RACER, a recent post-training backdoor defense for VLMs (Tables 1–2). These additions extend the evaluation across model families and scales and provide a comparison with another post-training defense.

We hope these results help clarify the scope and performance of PSR. We would value your feedback and are glad to address any remaining questions.

Best regards, The Authors

Thanks for these clarifications. "PSR requires the original pretrained checkpoint" still does not convince me why it is practical. I keep this concerns.

Thank you for the follow-up. If the pretrained checkpoint is unavailable, PSR can instead zero the selected channels. Our ablation (Table 3) shows that this variant reduces ASR to 0.00%, although it incurs a slightly larger CU drop than pretrained restoration. Its backdoor suppression remains competitive with RACER, a recently proposed post-training defense (Table 1).

Reviewer_JJ8NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a heuristic method to clear the backdoor in the projection model, which contains a Perturb-Select-Restore pipeline.

Critical strengths

1、This paper studies an important question and provides an effective method to defend against backdoor attacks. 2. This paper has a comprehensive evaluation of different conditions. 3、The writing is good and easy to understand for readers.

Critical weaknesses
  1. Although there is "projection fragility" in existing, the author did not explain what's the reason is, which makes the foundation of the proposed method questionable.

2、 The causal link between the fragility probe and the channel-selection mechanism is missing; there is no evidence that the channels ranked lowest by the robustness score are the ones that encode the backdoor.

3、 Why sensitivity to adversarial clean-task perturbations correlate with the presence of a backdoor? since a clean model can also be driven worse by deliberately chosen worst-case perturbation directions.

4、 The probe uses a single model for only three attacks and just 64 clean examples on each of two tasks, and it omits VLOOD, which is used as the primary comparison in the main evaluation; moreover, Figure 1 is the sole evidence, with no accompanying numeric table or analysis of effect size or statistical significance.

Questions for authors
  1. Why do you select linear layers in the projection model? 2、Since the restoration target is selected purely by robustness sensitivity, could you clarify the empirical basis for equating the least-robust channels with the backdoor-carrying ones? Is there any validation that the channels ranked lowest by the robustness score actually encode the backdoor?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JJ8N

Thank you for the constructive comments. We have strengthened the evidence connecting projection fragility to PSR.

W1. “Projection fragility” and the foundation of the method.

We added a backdoor-output probability experiment showing that perturbations derived from clean-task loss can increase the conditional probability of backdoor-related outputs on untriggered inputs. For example, on COCO, the conditional probability of completing the attack target for BadNet rose from 0.12% at ϵ=0\epsilon=0 to 3.12% at ϵ=0.05\epsilon=0.05 (26.1×; Table 6). This connects projection fragility to backdoor behavior and strengthens the empirical foundation of PSR (Figure 2; Appendix B.1, Table 6).

W2. The “causal link” and whether low-scoring channels “encode the backdoor.”

We clarified that θ balances perturbation vulnerability with clean-task preservation and ranks candidates for restoration. The new ablation shows that removing Perturb weakens backdoor suppression even when score learning, selection, and restoration are retained. For example, on Qwen3-VL-8B COCO, removing perturbations raises Blended ASR from 1.70% to 41.74% while retaining score learning, channel selection, and restoration (Table 3). This demonstrates the need for Perturb in reliable purification and supports projection fragility as the empirical basis of PSR (Section 4; Section 5.3, Table 3).

W3. A “clean model can also be driven worse.”

We agree that loss sensitivity alone is insufficient. The new experiment shows that perturbations also shift poisoned models toward backdoor-related outputs, while clean controls show no sustained corresponding shift. This behavioral difference supplies the connection missing from the original loss-only probe (Figure 2; Table 6).

W4. Limited probe coverage and numerical evidence.

We have expanded the probe to include VLOOD on both COCO and VQAv2 (Figures 1–2; Appendix B.1, Table 6). We use 64 matched clean examples per task to examine the phenomenon under limited clean-data access, consistent with our defender setting in which only a small clean dataset is available. Table 6 reports absolute conditional target-completion probabilities alongside the relative changes in Figure 2, while the perturbation ablation tests the contribution of perturbations to purification (Table 3).

Answers to the questions

Q1. Why linear layers?

We target these layers because they are central to the adaptation process through which the backdoor is introduced. In the projector-tuning setting studied here, the visual encoder and language model remain frozen, while the projector’s linear layers learn the task-specific mapping from visual features to the language space. Poisoned fine-tuning modifies this mapping, making its trainable parameters a direct target for repair. Their channel structure then enables selective restoration of weight rows and biases while preserving other useful adaptation. The layer-coverage ablation further supports repairing these transformations jointly (Sections 3.1–3.2; Appendix A.1; Table 3).

Q2. Do the lowest-scoring channels actually encode the backdoor?

θ is not based on fragility alone: it jointly accounts for perturbation vulnerability and clean-task preservation. We therefore interpret low-scoring channels as restoration candidates, not as a direct identification of backdoor carriers. The new backdoor-output probability experiment establishes an empirical connection between clean-task perturbations and attack-related behavior. The ablation then shows that removing Perturb weakens purification while retaining score learning, selection, and restoration. Together, these findings support the selection mechanism: projection fragility provides a signal relevant to backdoor behavior, and using this signal improves the effectiveness of selective repair (Section 4; Figure 2; Tables 3 and 6).

Dear Reviewer JJ8N,

In our earlier response, we added a backdoor-output probability experiment examining the relationship between projector perturbations and backdoor behavior, along with an ablation showing the contribution of the Perturb step to the defense (Figure 2; Tables 3 and 6). We have since expanded our evaluation to three additional VLMs and dual-trigger attacks (Table 5), and added RACER, a recent post-training backdoor defense for VLMs, as a baseline (Tables 1–2). The results extend the evaluation across models and trigger settings, while the RACER comparison shows more consistent backdoor suppression with generally better clean-utility retention.

We would appreciate your feedback on these additions and remain available to answer any remaining questions.

Best regards, The Authors

Reviewer_JJ8NOpenReview ↗

Thanks for your response

Thanks for your response, after review, I decide to maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.