acceptodds
Under review as a conference paper at ICLR 2027

Purifying Backdoored Large Vision-Language Models by Removing Hijacked Directions

Abstract

Large vision-language models (LVLMs) are increasingly deployed in safety-critical applications, yet they remain vulnerable to backdoor attacks. Defending against such attacks remains costly, as existing methods require either extensive retraining on clean data or per-query intervention at inference time. To address this limitation, we propose OrthoPurify, a more efficient method to purify backdoored model weights via one-step orthogonal projection. Specifically, through structural analysis of backdoor weight updates, we find that the backdoor is encoded by diverting a small number of weight update directions from task adaptation to backdoor shortcut encoding, a phenomenon we term *direction hijacking*. However, identifying these hijacked directions requires a benign reference model, which is typically inaccessible to the defender. We show that a pseudo-benign model, obtained by fine-tuning the pretrained weights on only a small set of clean samples, provides a sufficient approximation, as the dominant update directions stabilize within the first few gradient steps. OrthoPurify uses this pseudo-benign reference to isolate the hijacked directions and removes them through a single projection on the weight update. Extensive experiments show that OrthoPurify reduces the attack success rate to near zero while preserving the original performance across diverse benchmarks, without retraining the backdoored model or introducing inference-time overhead. Our code is publicly available at [this repository](https://anonymous.4open.science/r/OrthoPurify-paper-code-225B/).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Backdoor Defense, Large Vision-Language Models, Backdoor Purification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ncTXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OrthoPurify is an offline backdoor defense that compares suspicious weight updates with a pseudo-benign reference constructed from limited clean data, then removes directions with large principal angles. The paper reports strong attack suppression across multiple models, tasks, and attacks without additional inference overhead.

Critical strengths

The method is simple and computationally efficient. The experimental evaluation is substantial, including adaptive attacks, clean-model evaluations, and direction-selection controls. Figure 4(b) and Appendix H provide useful empirical support for the method’s design.

Critical weaknesses
  1. Adaptive robustness depends critically on threshold selection. Table 9 reports residual ASR of 91.4–96.1% for several alignment-based adaptive attacks at the default threshold of 50°. Thus, the default configuration does not withstand these attacks. Lowering the threshold to 20° achieves 0% ASR across the tested alignment-regularization settings, providing a meaningful countermeasure. However, how would a defender select 20°, or another effective threshold, without access to triggered validation data or attack success rates? Appendix M recommends monitoring held-out clean utility, but the precise selection rule and acceptable utility loss remain unclear. The results establish the effectiveness of a particular threshold, while robustness under the proposed clean-only calibration procedure remains insufficiently demonstrated.

  2. The clean-model safety claim needs qualification. Table 8 reports that purification reduces an unpoisoned LLaVA model’s VQAv2 score from 71.03 to 66.93. The other three clean-model settings improve, so the evidence does not indicate uniformly harmful purification. Nevertheless, the 4.10-point decrease requires explanation and is difficult to reconcile with Appendix J’s general claim of negligible degradation. Its reproducibility and whether the recommended calibration procedure prevents it remain unresolved.

Questions for authors
  1. Please specify the clean-only parameter-selection procedure and report ASR and clean utility for the configuration it selects, with parameters fixed before final evaluation. If 20° is intended as a revised fixed default, how does it perform across the evaluated settings?
  2. What explains the clean-model VQAv2 decrease? Please provide variability estimates and clarify whether calibration avoids this loss while retaining attack suppression.
6 comments
AuthorsOpenReview ↗

Response to Reviewer ncTX (W1)

We sincerely thank you for the rigorous and constructive evaluation. We are grateful for your recognition of OrthoPurify's simplicity and computational efficiency, and for your positive assessment of our experimental evaluation including adaptive attacks, clean-model evaluations, and direction-selection controls. Below we address each concern with additional experiments and analysis.


W1 & Q1: How can the threshold be reliably selected using only clean data?

Thank you for this thorough analysis of our adaptive robustness evaluation.

The calibration procedure achieves 0% ASR across all 7 standard models and all 40 adaptive attack combinations, with all parameters fixed before evaluation. It is entirely clean-only and produces the precise selection rule with an explicit acceptable utility loss threshold that you requested.

The key insight is that the defender can trade off purification aggressiveness against clean utility using only a held-out clean set. The procedure first ensures the pseudo-benign reference is well-trained by increasing T until the clean utility drop is within δ=2%, then scans θ downward from 50° to find the most aggressive purification the clean utility permits.

Calibration procedure. Given a model of unknown backdoor status and a small held-out clean set (128 images), the defender executes the following steps.

  1. Start with T=8 pseudo-benign training steps and θ=50° (the paper's default). Apply OrthoPurify and evaluate CU on the held-out set.
  2. If the held-out CU drops by more than δ=2% relative to the undefended model, increase T (→32→128) and repeat, until the drop is within δ or T reaches 128.
  3. With the selected T, search θ from 50° down to 20° in 5° steps and select the smallest θ whose held-out CU drop remains within δ, retaining θ=50° if none qualifies.

We set δ=2% based on the empirical variance across different pseudo-benign seeds, which is approximately ±1-2% in CU. All parameters are fixed by this procedure before final evaluation on the test set. The procedure is entirely clean-only, as the defender never observes ASR or any attack-related signal.

Results on COCO (LLaVA-1.5-7B). We apply this procedure to all 7 models in our COCO evaluation (6 standard attacks and 1 benign model). All models select T=8. Other selected parameters and test-set results are shown below.

ModelSelected θTest CIDErNo-defense CIDErCU ChangeASR (%)
BadNet35°119.15121.07−1.920
Blended50°131.72129.20+2.520
WaNet25°126.44121.51+4.930
ISSBA50°127.43126.75+0.680
TrojVLM20°121.13106.67+14.460
VLOOD20°125.89118.60+7.290
Benign25°132.68122.72+9.960

The calibration procedure achieves 0% ASR across all 7 models with parameters fixed before evaluation (Table 23 in the updated manuscript). The same procedure also handles adaptive attacks: across all 40 attack×strategy×λ combinations (Appendix L, Tables 14–17), the calibration-selected parameters achieve ASR ≤ 3.13% (36 of 40 at 0%). This demonstrates that the proposed clean-only procedure is sufficient for reliable threshold selection.

The calibration procedure adaptively selects θ per model. On COCO, it selects θ as low as 20° where the held-out CU permits (e.g., TrojVLM, VLOOD), while selecting higher θ when aggressive purification would reduce CU (e.g., BadNet at 35°). On VQAv2, the procedure selects higher θ to preserve performance. Please see our response to W2 for the VQAv2 calibration results. The complete calibration procedure and results are included in Appendix N (Tables 23–24) of the updated manuscript.

AuthorsOpenReview ↗

Response to Reviewer ncTX (W2)

W2 & Q2: What explains the clean-model VQAv2 decrease?

Thank you for your careful examination of this result.

The decrease is caused by insufficient pseudo-benign training at the default T=8, and is fully eliminated by increasing T to 128 (mean CU change −0.15 ± 0.42 across 5 seeds, with every individual seed within ±1 point of the undefended score). The calibration procedure from W1 automatically detects and resolves this case. With limited training, the pseudo-benign model does not fully capture all benign fine-tuning directions, and the projection introduces minor interference with some of these directions. Simply increasing T, without additional data, allows the pseudo-benign reference to better approximate the benign subspace.

Variability estimates. We evaluate the LLaVA-1.5-7B benign model on VQAv2 across 5 seeds at T=8, T=32, and T=128 (θ=50°).

SettingMean ± Std CU Change
T=8−4.38 ± 0.64
T=32−1.67 ± 0.52
T=128−0.15 ± 0.42

The drop decreases monotonically with T, confirming that it originates from insufficient pseudo-benign training. At T=128, the drop is eliminated across all seeds (mean −0.15 ± 0.42), with every individual seed within ±1 point of the undefended score.

Calibration avoids this loss while retaining attack suppression. The calibration procedure described in W1 automatically detects this case. When T=8 produces a held-out CU drop exceeding δ=2%, the procedure increases T until the drop is within tolerance. We apply the full procedure to all 7 VQAv2 models.

ModelTSelected θTest VQANo-defense VQACU ChangeASR (%)
Benign12845°71.3571.09+0.260
BadNet820°75.9676.95−0.990
Blended12850°69.0170.25−1.240
WaNet3220°78.4576.95+1.500
ISSBA12850°67.9766.15+1.820
TrojVLM820°76.5676.76−0.200
VLOOD12850°69.6070.05−0.450

The calibration selects T=128 for the benign model and achieves CU of +0.26 (no decrease). All 7 models achieve 0% ASR with parameters fixed before evaluation (Table 24 in the updated manuscript).

We have revised Appendix K in the updated manuscript to reflect that certain model-task combinations require larger T to eliminate degradation.

Thank you for the additional experiments. The multi-seed results substantially alleviate the clean-model degradation concern, and the expanded calibration evaluation provides useful evidence. Two issues remain:

  1. Could calibration select a configuration that preserves clean utility but retains the backdoor? The procedure selects the smallest threshold satisfying the clean-utility tolerance, but this criterion alone does not establish attack suppression. This concern remains relevant to the original BadNet/LLaVA/COCO alignment attacks in Table 12, for which I could not find calibration-selected results. At λ=0.27\lambda=0.27, 50∘50^\circ leaves 91.4% ASR with only 0.4% relative utility loss, whereas 20∘20^\circ achieves zero ASR with approximately 5% utility loss, exceeding the 2% tolerance. These test results do not establish calibration failure, but motivate directly evaluating the complete procedure on the original cases. Please report the selected T,θT,\theta, validation utility, and final test utility/ASR, ideally with reproducible calibration scripts.
  2. What is the complete calibration cost? The reported 46-second runtime and 64-sample budget do not establish the cost of the revised procedure. Please report end-to-end runtime, including reference training, purification, and all validation evaluations, particularly when T=128T=128. Please also clarify whether the 128 calibration images are additional to and disjoint from the 64 reference-training samples. These costs should be included in the efficiency assessment of the calibrated defense.
Reviewer_mbPdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OrthoPurify, a post-training defense method for removing backdoors from large vision-language models (LVLMs). The authors identify a phenomenon called direction hijacking, where backdoor fine-tuning diverts a small number of weight update directions away from benign task adaptation. Based on this observation, OrthoPurify constructs a pseudo-benign reference using a small set of clean samples, identifies hijacked directions through principal angle analysis, and removes them via orthogonal projection. Experiments across different LVLM architectures, backdoor attacks, and fine-tuning settings demonstrate that OrthoPurify can substantially reduce attack success rates while largely preserving clean-task performance, without retraining the backdoored model or introducing inference-time overhead.

Critical strengths
  1. The paper identifies direction hijacking as a potential mechanism of backdoor learning in LVLMs.
  2. OrthoPurify removes backdoors with few clean samples and no additional inference-time overhead.
  3. The method is evaluated across multiple LVLM architectures, backdoor attacks, and fine-tuning settings, showing strong defense performance.
Critical weaknesses

Major Weaknesses:

  1. The paper provides empirical evidence for direction hijacking through principal angle analysis and directional removal experiments. However, it remains unclear whether the observed large principal angles are specific to backdoor learning or can also arise from normal fine-tuning variations, such as different training data, random seeds, or optimization settings. The authors should include comparisons between independently fine-tuned benign models and further investigate whether direction hijacking consistently occurs across different backdoor objectives and training configurations.
  2. The main experiments compare OrthoPurify with only four post-training baselines, most of which are relatively old and originally designed for image classification. The authors should clarify whether more recent post-training defenses for MLLMs/LVLMs have been proposed in the past two years and, if so, explain why they are not included in the comparison. Although the paper includes two recent test-time defenses, PurMM and CleanSight, it does not consider other relevant methods, such as Trap on Text (AAAI 2026) [1].
  3. The authors claim that OrthoPurify can be safely applied to benign models without knowing whether a backdoor is present. However, Table 8 shows that applying OrthoPurify to a benign LLaVA model on VQAv2 reduces clean accuracy from 71.03% to 66.93%. This degradation raises concerns about the reliability of applying the defense to models with unknown backdoor status. The authors should explain this performance drop and provide more comprehensive evaluations on benign models.
  4. The two adaptive attacks in Appendix K are evaluated only on BadNet with LLaVA-1.5-7B (COCO), which is insufficient to support the paper's broad claims of robustness across different models, tasks, and attack types.

Minor Weakness:

  1. The Introduction categorizes existing LVLM backdoor defenses into only post-training and test-time approaches, overlooking training-time defenses that aim to prevent backdoor implantation during fine-tuning. Although the threat model focuses on post-training purification, the authors should clarify the scope of this categorization and discuss how their approach differs from representative training-time defenses. If feasible, including comparisons with such methods under comparable experimental settings would further strengthen the empirical evaluation and provide a more comprehensive assessment of the proposed defense's effectiveness.

Reference:
[1] Rong X, Huang W, Jiang W, et al. Probing semantic insensitivity for inference-time backdoor defense in multimodal large language model[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(42): 35775-35783.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer mbPd (W1)

We sincerely thank you for the detailed review and valuable suggestions. We appreciate you recognizing direction hijacking as a meaningful mechanism of backdoor learning, as well as OrthoPurify's efficiency in removing backdoors with few clean samples and no inference-time overhead. Below we address each concern with additional experiments and analysis.


W1: Are large principal angles specific to backdoor learning?

Thank you for this thought-provoking question.

Large principal angles are indeed specific to backdoor learning. We compute principal angles against a benign base model for both benign fine-tuning variations and backdoor attacks. All benign variations produce maximum principal angles of at most 46°, while all six backdoor attacks produce angles of at least 76°. A consistent 30° gap separates the two groups across both adapter weight matrices.

Your question points to a natural concern. If normal fine-tuning variations such as different training data, random seeds, or optimization settings could also produce large principal angles, then principal angles alone would not reliably indicate backdoor presence. To test this, we designed benign variations that maximize the chance of shifting the learned subspace away from the base model. A different random seed, a 5× learning rate change (1e-4 vs 2e-5), and a completely non-overlapping training set each perturb a different aspect of training, together covering the sources of variation you identified. If even these aggressive variations fail to produce large angles, the specificity claim is well-supported.

Starting from a single base model (LLaVA-1.5-7B fine-tuned on COCO), we independently vary one factor at a time and compute principal angles against the base model. L1 and L2 denote the two weight matrices of the adapter. We compare these three benign variations and six backdoor attacks, all computed against the same base model.

VariationL1 Max AngleL2 Max Angle
Different random seed37°26°
Different learning rate (1e-4 vs 2e-5)29°30°
Non-overlapping training data46°36°
BadNet90°84°
Blended83°76°
WaNet87°80°
ISSBA87°77°
TrojVLM85°78°
VLOOD85°79°

The largest benign angle is 46° (non-overlapping training data, L1), while the smallest backdoor angle is 76° (Blended, L2). Even replacing the entire training set only shifts the subspace by 46°, whereas every backdoor attack pushes it past 76°. This 30° gap holds consistently across both layers and all nine conditions.

Consistency across different backdoor objectives and training configurations. The six attacks span fundamentally different mechanisms, including patch overlay (BadNet), global pattern blending (Blended), spatial warping (WaNet), sample-specific invisible perturbation (ISSBA), semantic-preserving training (TrojVLM), and clean-knowledge-preserving training (VLOOD). Despite these diverse strategies, all six consistently produce large principal angles in both layers. This consistency suggests that direction hijacking is a structural consequence of backdoor learning itself, not an artifact of any particular trigger design. The full results are included in Appendix H (Table 5) of the updated manuscript.

AuthorsOpenReview ↗

Response to Reviewer mbPd (W2)

W2 & Minor: Are there more recent defenses that should be compared? What about training-time defenses?

Thank you for this suggestion.

By the time of our submission, to our knowledge no post-training purification method existed for LVLMs, which is why our four baselines are adapted from established methods for image classifiers. Two concurrent works have appeared around the same time, RACER [1] (arXiv August 2026) and PSR [2] (arXiv September 2026), and we reproduce both on all 24 backdoor models from Tables 1–2. We additionally compare with Trap on Text (AAAI 2026) as you suggested. Below we also describe how each existing baseline was adapted and clarify the scope distinction with training-time defenses.

Trap on Text comparison. Trap on Text is a test-time defense that detects backdoor inputs by comparing model outputs under the original and text-scrambled prompts. We evaluate it on LLaVA-1.5-7B under both VQAv2 and COCO.

TaskAttackNo-defense CUToT CUCU ChangeNo-defense ASRToT ASRFalse Reject (%)
VQAv2BadNet76.9564.65−12.3091.99%0%17.77
VQAv2TrojVLM76.7664.19−12.5799.02%0%17.38
COCOBadNet121.070−121.0799.41%0%100.00
COCOTrojVLM107.348.94−98.40100%10.16%91.99

On VQAv2, Trap on Text effectively reduces ASR to 0%. The CU cost is approximately 12 points, with a false rejection rate of about 17%, mainly because some benign inputs coincidentally produce overlapping answers before and after text scrambling, triggering the detection criterion. On COCO captioning, the false rejection rate reaches 91-100%, resulting in near-zero CU. This is because the captioning model generates descriptions primarily based on the image content, so scrambling the text prompt does not significantly change the output, causing nearly all inputs to be flagged as backdoor samples. This comparison is included in Appendix J (Table 10) of the updated manuscript.

Concurrent post-training defenses. RACER divides fused representations into visual-token and text-token regions, computes a layer-wise consistency objective in deep layers, and adversarially fine-tunes the model using 100 clean samples. PSR attaches learnable scores and bounded perturbations to each projection channel, identifies fragile channels via min-max optimization on clean data, and restores them to pre-trained values. We reproduce both following their recommended settings. For RACER, several hyperparameters not specified in the paper (perturbation budget, regularization weight, learning rate) are selected on the test set, which is favorable to this baseline (Appendix C.4).

Due to the character limit, the full RACER and PSR results across all 24 settings are provided in our supplementary tables reply below. Across all 24 settings, RACER achieves ASR ≤ 5% on 3 and PSR on 3 (2 with CU collapsed below 6). OrthoPurify achieves 0% ASR on all 24 with clean utility preserved. The complete reproduction details are in Appendix C.4 of the updated manuscript.

Existing baselines and adaptation. Our four baselines are adapted from established post-training methods originally designed for image classifiers, all adapted to the LVLM setting and applied to the adapter module consistent with our threat model.

MethodOriginal SettingAdaptation to LVLM
Clean Fine-tuningGeneralTrain adapter on 1,000 clean samples for 2 epochs
Fine-Pruning (RAID 2018)DNN classifiersRank adapter neurons by average clean-data activation, auto-select pruning ratio (CU within 2.5%), fine-tune 2 epochs
ANP (NeurIPS 2021)Image classification CNNsLearnable perturbation masks on adapter weights, adversarial optimization on 500 clean samples, prune mask < 0.5
CLP (ECCV 2022)CNN convolutional channelsPrune adapter channels with weight-row L2 norm > μ + σ, no data required

The full adaptation details are in Appendix C.4.

Training-time defenses. We appreciate you pointing out this distinction. Training-time defenses such as BYE (NeurIPS 2025), BYORn (ICML 2026), and TCAP (ICML 2026) assume that the defender controls the fine-tuning process and can apply protective measures during training. Our threat model addresses a different scenario where the defender receives an already-trained model and cannot intervene in the training process. Post-training purification and training-time defenses are complementary rather than competing. We have clarified this scope distinction in the related work (Appendix A) of the updated manuscript.

References

[1] Wei, J. et al. Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs. arXiv:2608.24354, 2026.

[2] Liu, S. et al. Selective Channel Restoration for Backdoored Vision-Language Models. arXiv:2609.37759, 2026.

AuthorsOpenReview ↗

Response to Reviewer mbPd (W2 Supplementary Tables)

Below are the full RACER and PSR results for all four model-task combinations. Each cell reports ASR (%) / clean utility.

LLaVA-1.5-7B, COCO (CU = CIDEr).

MethodBadNetBlendedWaNetISSBATrojVLMVLOOD
No defense99.41 / 121.4897.85 / 129.9898.63 / 121.4898.44 / 126.74100 / 107.6681.64 / 117.36
RACER2.34 / 119.5535.35 / 119.5121.68 / 121.0912.70 / 127.4357.03 / 118.210.20 / 117.52
PSR96.68 / 0.0179.88 / 36.500 / 1.420.39 / 5.7595.51 / 59.0565.62 / 16.87
OrthoPurify0 / 125.850 / 131.790 / 126.200 / 125.120 / 130.080 / 121.84

LLaVA-1.5-7B, VQAv2 (CU = VQA score).

MethodBadNetBlendedWaNetISSBATrojVLMVLOOD
No defense92.29 / 76.9298.83 / 67.8495.70 / 76.1499.22 / 66.4199.80 / 75.4299.41 / 70.18
RACER99.61 / 66.0896.88 / 64.4595.12 / 67.3892.38 / 64.0699.80 / 64.2699.41 / 66.02
PSR84.77 / 55.6091.80 / 62.5783.98 / 61.8529.88 / 61.4698.24 / 59.7088.48 / 62.83
OrthoPurify0 / 77.470 / 65.490 / 79.690 / 66.800 / 76.690 / 61.39

Qwen3-VL-8B, COCO (CU = CIDEr).

MethodBadNetBlendedWaNetISSBATrojVLMVLOOD
No defense99.80 / 70.9295.70 / 67.8699.80 / 68.0499.41 / 66.01100 / 79.2099.80 / 24.97
RACER100 / 45.9137.50 / 43.8211.33 / 42.000 / 34.8648.24 / 38.9436.91 / 49.82
PSR82.23 / 13.8137.11 / 10.4125.00 / 13.6630.66 / 16.1737.50 / 13.8559.18 / 17.06
OrthoPurify0 / 71.590 / 69.950 / 69.790 / 64.940 / 74.060 / 25.03

Qwen3-VL-8B, VQAv2 (CU = VQA score).

MethodBadNetBlendedWaNetISSBATrojVLMVLOOD
No defense100 / 84.2497.46 / 83.5696.78 / 84.8699.80 / 83.07100 / 83.72100 / 85.29
RACER99.41 / 79.3089.65 / 82.4286.91 / 80.1441.60 / 81.3233.01 / 81.51100 / 81.32
PSR94.14 / 81.5148.05 / 78.5827.15 / 75.652.54 / 80.6696.29 / 84.7796.68 / 75.07
OrthoPurify0 / 79.950 / 86.000 / 84.570 / 84.770 / 81.250 / 84.57
Reviewer_oEWLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OrthoPurify, a framework designed for backdoor purification in Large vision-language models (LVLMs). The authors first identify a phenomenon termed direction hijacking, where backdoor fine-tuning redirects a small number of weight update directions away from benign task adaptation. Based on this, OrthoPurify constructs a pseudo-benign reference via a few tuning steps on a small clean sets, identifies potentially hijacked directions through SVD and principal-angle analysis, and removes them via a single orthogonal projection.

Critical strengths
  • OrthoPurify requires only a small amount of clean data and a few fine-tuning steps to construct a pseudo-benign reference. The subsequent purification involves a single orthogonal projection, avoiding additional training of the backdoored model and inference-time overhead.
  • OrthoPurify achieves near-zero ASR across multiple backdoor attacks, LVLM architectures, and downstream tasks while largely preserving clean utility.
Critical weaknesses
  • The main experiments primarily compare against conventional fine-tuning and pruning-based defenses originally designed for image classifiers. The lack of comparisons with more recent post-training backdoor purification methods limits the comprehensiveness of the evaluation.
  • As shown in Figure 2(b), several baseline defenses retain or even amplify the hijacked energy, yet still achieve substantial reductions in ASR. This raises questions about how reliably the proposed hijacked energy retention ratio reflects actual backdoor behavior. A more systematic analysis of the relationship between hijacked energy and ASR would strengthen the evidence supporting the direction hijacking mechanism.
  • The adaptive attack results in Appendix K reveal that OrthoPurify can be largely bypassed under its default principal-angle threshold, with residual ASR exceeding 90% in several settings. Although lowering the threshold restores defense effectiveness, this adjustment raises concerns about practical deployment, where defenders typically lack knowledge of the attack strategy or access to triggered validation samples. The paper does not sufficiently establish how an appropriate threshold can be reliably selected under such uncertainty.
Questions for authors
  • Could the authors include comparisons with more recent post-training backdoor purification methods to provide a stronger evaluation of OrthoPurify?
  • Why do some baseline defenses substantially reduce ASR despite retaining or amplifying the hijacked energy? Could the authors further analyze the relationship between hijacked energy and ASR?
  • How can the principal-angle threshold be reliably selected against adaptive attacks when defenders have no knowledge of the attack strategy or access to triggered validation samples?
6 comments
AuthorsOpenReview ↗

Response to Reviewer oEWL (W1)

We sincerely thank you for the thorough and constructive feedback. We appreciate you recognizing the efficiency of OrthoPurify in requiring only a small amount of clean data and a single projection step, as well as its strong defense performance across multiple backdoor attacks, LVLM architectures, and downstream tasks. Below we address each concern with additional experiments and analysis.


W1 & Q1: Are there more recent post-training backdoor purification defenses for LVLMs?

Thank you for raising this point.

By the time of our submission, to our knowledge OrthoPurify is the first post-training purification method for LVLMs. Two concurrent works have appeared around the same time, RACER [1] (arXiv August 2026) and PSR [2] (arXiv September 2026). We reproduce both on all 24 backdoor models from Tables 1–2.

RACER divides fused representations into visual-token and text-token regions, computes a layer-wise consistency objective in deep layers, and adversarially fine-tunes the model using 100 clean samples. PSR attaches learnable scores and bounded perturbations to each projection channel, identifies fragile channels via min-max optimization on clean data, and restores them to pre-trained values. We reproduce both following their recommended settings. For RACER, several hyperparameters not specified in the paper (perturbation budget, regularization weight, learning rate) are selected on the test set, which is favorable to this baseline (Appendix C.4).

Results for all four model-task combinations are shown below. Each cell reports ASR (%) / clean utility.

LLaVA-1.5-7B, COCO (CU = CIDEr).

MethodBadNetBlendedWaNetISSBATrojVLMVLOOD
No defense99.41 / 121.4897.85 / 129.9898.63 / 121.4898.44 / 126.74100 / 107.6681.64 / 117.36
RACER2.34 / 119.5535.35 / 119.5121.68 / 121.0912.70 / 127.4357.03 / 118.210.20 / 117.52
PSR96.68 / 0.0179.88 / 36.500 / 1.420.39 / 5.7595.51 / 59.0565.62 / 16.87
OrthoPurify0 / 125.850 / 131.790 / 126.200 / 125.120 / 130.080 / 121.84

LLaVA-1.5-7B, VQAv2 (CU = VQA score).

MethodBadNetBlendedWaNetISSBATrojVLMVLOOD
No defense92.29 / 76.9298.83 / 67.8495.70 / 76.1499.22 / 66.4199.80 / 75.4299.41 / 70.18
RACER99.61 / 66.0896.88 / 64.4595.12 / 67.3892.38 / 64.0699.80 / 64.2699.41 / 66.02
PSR84.77 / 55.6091.80 / 62.5783.98 / 61.8529.88 / 61.4698.24 / 59.7088.48 / 62.83
OrthoPurify0 / 77.470 / 65.490 / 79.690 / 66.800 / 76.690 / 61.39

Qwen3-VL-8B, COCO (CU = CIDEr).

MethodBadNetBlendedWaNetISSBATrojVLMVLOOD
No defense99.80 / 70.9295.70 / 67.8699.80 / 68.0499.41 / 66.01100 / 79.2099.80 / 24.97
RACER100 / 45.9137.50 / 43.8211.33 / 42.000 / 34.8648.24 / 38.9436.91 / 49.82
PSR82.23 / 13.8137.11 / 10.4125.00 / 13.6630.66 / 16.1737.50 / 13.8559.18 / 17.06
OrthoPurify0 / 71.590 / 69.950 / 69.790 / 64.940 / 74.060 / 25.03

Qwen3-VL-8B, VQAv2 (CU = VQA score).

MethodBadNetBlendedWaNetISSBATrojVLMVLOOD
No defense100 / 84.2497.46 / 83.5696.78 / 84.8699.80 / 83.07100 / 83.72100 / 85.29
RACER99.41 / 79.3089.65 / 82.4286.91 / 80.1441.60 / 81.3233.01 / 81.51100 / 81.32
PSR94.14 / 81.5148.05 / 78.5827.15 / 75.652.54 / 80.6696.29 / 84.7796.68 / 75.07
OrthoPurify0 / 79.950 / 86.000 / 84.570 / 84.770 / 81.250 / 84.57

Across all 24 settings, RACER achieves ASR ≤ 5% on 3 and PSR on 3 (2 with CU collapsed below 6). OrthoPurify achieves 0% ASR on all 24 with clean utility preserved. The complete reproduction details are in Appendix C.4 of the updated manuscript.

Existing baselines and adaptation. Our four baselines are adapted from established post-training methods originally designed for image classifiers, all adapted to the LVLM setting and applied to the adapter module consistent with our threat model.

MethodOriginal SettingAdaptation to LVLM
Clean Fine-tuningGeneralTrain adapter on 1,000 clean samples for 2 epochs
Fine-Pruning (RAID 2018)DNN classifiersRank adapter neurons by average clean-data activation, auto-select pruning ratio (CU within 2.5%), fine-tune 2 epochs
ANP (NeurIPS 2021)Image classification CNNsLearnable perturbation masks on adapter weights, adversarial optimization on 500 clean samples, prune mask < 0.5
CLP (ECCV 2022)CNN convolutional channelsPrune adapter channels with weight-row L2 norm > μ + σ, no data required

The full adaptation details are in Appendix C.4.


References

[1] Wei, J. et al. Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs. arXiv:2608.24354, 2026.

[2] Liu, S. et al. Selective Channel Restoration for Backdoored Vision-Language Models. arXiv:2609.37759, 2026.

AuthorsOpenReview ↗

Response to Reviewer oEWL (W2)

W2 & Q2: Why do some baseline defenses reduce ASR despite retaining hijacked energy?

Thank you for this important question.

The disconnect arises because reducing ASR does not require geometrically modifying the hijacked subspace. The hijacked energy retention ratio in Figure 2(b) captures whether a defense modifies the geometric structure of this subspace. OrthoPurify removes the backdoor-carrying directions from the hijacked subspace, which simultaneously reduces hijacked energy and eliminates ASR. Baseline defenses reduce ASR through other means without targeting the subspace geometry, so their hijacked energy remains high. These are two fundamentally different pathways to ASR reduction, and the energy metric reflects only the geometric one.

To verify this mechanistic distinction, we go beyond the aggregate energy and examine the hijacked subspace at the level of individual directions. If the backdoor signal is concentrated in specific directions, and OrthoPurify removes exactly those directions, then the energy drop directly reflects backdoor removal. We isolate each direction removed by OrthoPurify (θ=50°) on LLaVA-1.5-7B COCO and measure its causal contribution to ASR independently. For each direction, "Remove only" starts from the undefended model and removes that single direction while keeping all others. "Restore only" starts from the fully purified model (ASR=0) and adds back that single direction. "Energy share" is the fraction of total removed energy in that layer contributed by this direction. L1 and L2 denote the two weight matrices of the adapter.

BadNet (5 directions removed in L1, 3 in L2, undefended ASR=99.41%).

DirectionAngleEnergy ShareRemove Only (ASR)Restore Only (ASR)
L1_089°28.8%99.41%0%
L1_188°35.0%82.42%0%
L1_274°11.1%99.41%0%
L1_368°11.9%99.41%0%
L1_454°13.3%99.41%0%
L2_090°41.1%2.73%0%
L2_185°41.8%74.22%0%
L2_263°17.2%99.41%0%

L2_0 is the primary backdoor-carrying direction (removing it alone reduces ASR to 2.73%), with L1_1 and L2_1 having partial contributions. The remaining directions do not significantly affect ASR.

Blended (4 directions removed in L1, 4 in L2, undefended ASR=97.85%).

DirectionAngleEnergy ShareRemove Only (ASR)Restore Only (ASR)
L1_088°47.2%74.61%0%
L1_188°22.3%97.46%0%
L1_284°15.3%97.46%0%
L1_370°15.2%97.66%0%
L2_089°33.6%95.12%0%
L2_183°35.7%1.56%15.62%
L2_281°15.4%97.07%0%
L2_354°15.2%97.85%0%

L2_1 is the primary backdoor-carrying direction (removing it alone reduces ASR to 1.56%). Restoring L2_1 alone raises ASR from 0% back to 15.62%, confirming a direct causal link between this direction and the backdoor behavior.

Relationship between hijacked energy and ASR. The hijacked energy retention ratio in Figure 2(b) captures whether a defense geometrically modifies the hijacked subspace. OrthoPurify directly removes directions from this subspace, including the primary backdoor-carrying ones identified above, resulting in low energy retention and 0% ASR. Baseline defenses such as fine-pruning reduce ASR through a different pathway by disrupting the trigger-to-response mapping, without geometrically targeting the hijacked directions. This is why the hijacked energy remains while ASR decreases. The direction-level analysis confirms this mechanistic distinction at a finer granularity, showing that the directions mainly responsible for ASR are removed by OrthoPurify's projection. The complete direction-level results are reported in Appendix I (Tables 7–8) of the updated manuscript.

AuthorsOpenReview ↗

Response to Reviewer oEWL (W3)

W3 & Q3: How can the threshold be reliably selected without knowledge of the attack?

Thank you for this important question about practical deployment.

A clean-only calibration procedure achieves 0% ASR across all 7 standard models and all 40 adaptive attack combinations, with all parameters fixed before evaluation. The procedure requires no knowledge of the attack strategy or access to triggered samples.

The key insight is that the defender can trade off purification aggressiveness against clean utility using only a held-out clean set. The procedure first ensures the pseudo-benign reference is well-trained by increasing T until the clean utility drop is within δ=2%, then scans θ downward from 50° to find the most aggressive purification the clean utility permits.

Calibration procedure. Given a model of unknown backdoor status and a small held-out clean set (128 images), the defender executes the following steps.

  1. Start with T=8 pseudo-benign training steps and θ=50° (the paper's default). Apply OrthoPurify and evaluate CU on the held-out set.
  2. If the held-out CU drops by more than δ=2% relative to the undefended model, increase T (→32→128) and repeat, until the drop is within δ or T reaches 128.
  3. With the selected T, search θ from 50° down to 20° in 5° steps and select the smallest θ whose held-out CU drop remains within δ, retaining θ=50° if none qualifies.

The entire procedure is based solely on clean utility, as the defender never observes ASR or any attack-related signal. We set δ=2% based on the empirical variance across different pseudo-benign seeds, which is approximately ±1-2% in CU.

Results. We apply this procedure to all 7 models in our COCO evaluation (6 standard attacks and 1 benign model). All models select T=8. Other selected parameters and test-set results are shown below.

ModelSelected θTest CIDErNo-defense CIDErCU ChangeASR (%)
BadNet35°119.15121.07−1.920
Blended50°131.72129.20+2.520
WaNet25°126.44121.51+4.930
ISSBA50°127.43126.75+0.680
TrojVLM20°121.13106.67+14.460
VLOOD20°125.89118.60+7.290
Benign25°132.68122.72+9.960

The calibration procedure achieves 0% ASR across all 7 models with parameters fixed before evaluation (Table 23 in the updated manuscript). The same procedure also handles adaptive attacks: across all 40 attack×strategy×λ combinations (Appendix L, Tables 14–17), the calibration-selected parameters achieve ASR ≤ 3.13% (36 of 40 at 0%).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal

(Reviewer oEWL---R1, Reviewer mbPd---R2, Reviewer ncTX---R3).

We sincerely thank all reviewers for their great efforts in providing many treasurable and constructive feedback. We are encouraged that they consider direction hijacking a meaningful mechanism of backdoor learning (R2), recognize OrthoPurify's efficiency in requiring only a small amount of clean data and a single projection step with no inference-time overhead (R1, R2), and positively assess the comprehensive experimental evaluation including adaptive attacks, clean-model evaluations, and direction-selection controls (R3). In response, we have conducted extensive additional experiments and revised the manuscript accordingly. Below are some key changes to our manuscript and three main concerns raised by reviewers.


Key Manuscript Updates

  1. We have reproduced two concurrent post-training baselines, RACER (Aug 2026) and PSR (Sep 2026), on all 24 backdoor settings across 4 model-task combinations and 6 attacks (suggested by R1 and R2). OrthoPurify achieves 0% ASR on all 24 with clean utility preserved (updated Tables 1-2, Appendix C.4).

  2. We have proposed a clean-only calibration procedure that selects T and θ using only 128 clean images with no attack knowledge (suggested by R1 and R3). It achieves 0% ASR on all 7 standard models and ASR ≤ 3.13% on all 40 adaptive combinations (new Appendix N).

  3. We have provided additional evaluations on benign models showing that the VQA performance drop is specific to one setting and is fully resolved by increasing T to 128 (suggested by R2 and R3). The calibration procedure detects and handles this automatically (revised Appendix K).

  4. We have computed principal angles against a benign base model for both benign training variations and backdoor attacks to verify specificity (suggested by R2). All benign variations produce maximum angles ≤ 46° while all backdoor attacks produce angles ≥ 76°, confirming a consistent 30° gap (new Appendix H).

  5. We have added direction-level experiments to explain the relationship between hijacked energy and ASR (suggested by R1). Remove-only and restore-only results confirm that OrthoPurify removes exactly the backdoor-carrying directions, while baselines reduce ASR through other pathways without modifying the hijacked subspace (new Appendix I).

  6. We have extended the adaptive attack evaluation to 40 combinations across different attack types, model architectures, and tasks (suggested by R2). All combinations are successfully defended by the calibration procedure without attack knowledge (updated Appendix L).

  7. We have added a comparison with Trap on Text as a test-time baseline to provide a more comprehensive evaluation (suggested by R2). See new Appendix J.

  8. We have clarified in the related work that training-time and post-training defenses address complementary threat scenarios (suggested by R2). See updated Appendix A.


Common Concerns

  1. Threshold selection without attack knowledge (R1, R3). The calibration procedure in Appendix N provides a complete solution with all parameters fixed before evaluation, achieving 0% ASR across all standard and adaptive settings.

  2. Benign model performance drop (R2, R3). This is specific to one model-task combination at default settings and is resolved by increasing T. The calibration procedure handles this automatically.

  3. Recent baselines (R1, R2). We reproduce RACER and PSR across all 24 settings. OrthoPurify is the only method achieving 0% ASR universally while preserving clean utility.


We have carefully revised the manuscript to incorporate all feedback and welcome any further questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.