- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OrthoPurify is an offline backdoor defense that compares suspicious weight updates with a pseudo-benign reference constructed from limited clean data, then removes directions with large principal angles. The paper reports strong attack suppression across multiple models, tasks, and attacks without additional inference overhead.
Critical strengths
The method is simple and computationally efficient. The experimental evaluation is substantial, including adaptive attacks, clean-model evaluations, and direction-selection controls. Figure 4(b) and Appendix H provide useful empirical support for the method’s design.
Critical weaknesses
-
Adaptive robustness depends critically on threshold selection. Table 9 reports residual ASR of 91.4–96.1% for several alignment-based adaptive attacks at the default threshold of 50°. Thus, the default configuration does not withstand these attacks. Lowering the threshold to 20° achieves 0% ASR across the tested alignment-regularization settings, providing a meaningful countermeasure. However, how would a defender select 20°, or another effective threshold, without access to triggered validation data or attack success rates? Appendix M recommends monitoring held-out clean utility, but the precise selection rule and acceptable utility loss remain unclear. The results establish the effectiveness of a particular threshold, while robustness under the proposed clean-only calibration procedure remains insufficiently demonstrated.
-
The clean-model safety claim needs qualification. Table 8 reports that purification reduces an unpoisoned LLaVA model’s VQAv2 score from 71.03 to 66.93. The other three clean-model settings improve, so the evidence does not indicate uniformly harmful purification. Nevertheless, the 4.10-point decrease requires explanation and is difficult to reconcile with Appendix J’s general claim of negligible degradation. Its reproducibility and whether the recommended calibration procedure prevents it remain unresolved.
Questions for authors
- Please specify the clean-only parameter-selection procedure and report ASR and clean utility for the configuration it selects, with parameters fixed before final evaluation. If 20° is intended as a revised fixed default, how does it perform across the evaluated settings?
- What explains the clean-model VQAv2 decrease? Please provide variability estimates and clarify whether calibration avoids this loss while retaining attack suppression.
6 comments
Response to Reviewer ncTX (W2)
W2 & Q2: What explains the clean-model VQAv2 decrease?
Thank you for your careful examination of this result.
The decrease is caused by insufficient pseudo-benign training at the default T=8, and is fully eliminated by increasing T to 128 (mean CU change −0.15 ± 0.42 across 5 seeds, with every individual seed within ±1 point of the undefended score). The calibration procedure from W1 automatically detects and resolves this case. With limited training, the pseudo-benign model does not fully capture all benign fine-tuning directions, and the projection introduces minor interference with some of these directions. Simply increasing T, without additional data, allows the pseudo-benign reference to better approximate the benign subspace.
Variability estimates. We evaluate the LLaVA-1.5-7B benign model on VQAv2 across 5 seeds at T=8, T=32, and T=128 (θ=50°).
| Setting | Mean ± Std CU Change |
|---|---|
| T=8 | −4.38 ± 0.64 |
| T=32 | −1.67 ± 0.52 |
| T=128 | −0.15 ± 0.42 |
The drop decreases monotonically with T, confirming that it originates from insufficient pseudo-benign training. At T=128, the drop is eliminated across all seeds (mean −0.15 ± 0.42), with every individual seed within ±1 point of the undefended score.
Calibration avoids this loss while retaining attack suppression. The calibration procedure described in W1 automatically detects this case. When T=8 produces a held-out CU drop exceeding δ=2%, the procedure increases T until the drop is within tolerance. We apply the full procedure to all 7 VQAv2 models.
| Model | T | Selected θ | Test VQA | No-defense VQA | CU Change | ASR (%) |
|---|---|---|---|---|---|---|
| Benign | 128 | 45° | 71.35 | 71.09 | +0.26 | 0 |
| BadNet | 8 | 20° | 75.96 | 76.95 | −0.99 | 0 |
| Blended | 128 | 50° | 69.01 | 70.25 | −1.24 | 0 |
| WaNet | 32 | 20° | 78.45 | 76.95 | +1.50 | 0 |
| ISSBA | 128 | 50° | 67.97 | 66.15 | +1.82 | 0 |
| TrojVLM | 8 | 20° | 76.56 | 76.76 | −0.20 | 0 |
| VLOOD | 128 | 50° | 69.60 | 70.05 | −0.45 | 0 |
The calibration selects T=128 for the benign model and achieves CU of +0.26 (no decrease). All 7 models achieve 0% ASR with parameters fixed before evaluation (Table 24 in the updated manuscript).
We have revised Appendix K in the updated manuscript to reflect that certain model-task combinations require larger T to eliminate degradation.
Thank you for the additional experiments. The multi-seed results substantially alleviate the clean-model degradation concern, and the expanded calibration evaluation provides useful evidence. Two issues remain:
- Could calibration select a configuration that preserves clean utility but retains the backdoor? The procedure selects the smallest threshold satisfying the clean-utility tolerance, but this criterion alone does not establish attack suppression. This concern remains relevant to the original BadNet/LLaVA/COCO alignment attacks in Table 12, for which I could not find calibration-selected results. At , leaves 91.4% ASR with only 0.4% relative utility loss, whereas achieves zero ASR with approximately 5% utility loss, exceeding the 2% tolerance. These test results do not establish calibration failure, but motivate directly evaluating the complete procedure on the original cases. Please report the selected , validation utility, and final test utility/ASR, ideally with reproducible calibration scripts.
- What is the complete calibration cost? The reported 46-second runtime and 64-sample budget do not establish the cost of the revised procedure. Please report end-to-end runtime, including reference training, purification, and all validation evaluations, particularly when . Please also clarify whether the 128 calibration images are additional to and disjoint from the 64 reference-training samples. These costs should be included in the efficiency assessment of the calibrated defense.
Response to Reviewer ncTX (W1)
We sincerely thank you for the rigorous and constructive evaluation. We are grateful for your recognition of OrthoPurify's simplicity and computational efficiency, and for your positive assessment of our experimental evaluation including adaptive attacks, clean-model evaluations, and direction-selection controls. Below we address each concern with additional experiments and analysis.
W1 & Q1: How can the threshold be reliably selected using only clean data?
Thank you for this thorough analysis of our adaptive robustness evaluation.
The calibration procedure achieves 0% ASR across all 7 standard models and all 40 adaptive attack combinations, with all parameters fixed before evaluation. It is entirely clean-only and produces the precise selection rule with an explicit acceptable utility loss threshold that you requested.
The key insight is that the defender can trade off purification aggressiveness against clean utility using only a held-out clean set. The procedure first ensures the pseudo-benign reference is well-trained by increasing T until the clean utility drop is within δ=2%, then scans θ downward from 50° to find the most aggressive purification the clean utility permits.
Calibration procedure. Given a model of unknown backdoor status and a small held-out clean set (128 images), the defender executes the following steps.
We set δ=2% based on the empirical variance across different pseudo-benign seeds, which is approximately ±1-2% in CU. All parameters are fixed by this procedure before final evaluation on the test set. The procedure is entirely clean-only, as the defender never observes ASR or any attack-related signal.
Results on COCO (LLaVA-1.5-7B). We apply this procedure to all 7 models in our COCO evaluation (6 standard attacks and 1 benign model). All models select T=8. Other selected parameters and test-set results are shown below.
The calibration procedure achieves 0% ASR across all 7 models with parameters fixed before evaluation (Table 23 in the updated manuscript). The same procedure also handles adaptive attacks: across all 40 attack×strategy×λ combinations (Appendix L, Tables 14–17), the calibration-selected parameters achieve ASR ≤ 3.13% (36 of 40 at 0%). This demonstrates that the proposed clean-only procedure is sufficient for reliable threshold selection.
The calibration procedure adaptively selects θ per model. On COCO, it selects θ as low as 20° where the held-out CU permits (e.g., TrojVLM, VLOOD), while selecting higher θ when aggressive purification would reduce CU (e.g., BadNet at 35°). On VQAv2, the procedure selects higher θ to preserve performance. Please see our response to W2 for the VQAv2 calibration results. The complete calibration procedure and results are included in Appendix N (Tables 23–24) of the updated manuscript.