- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SPEAR, a data-purification framework for backdoor defense in vision-language instruction tuning. A diagnostic surrogate is first LoRA-tuned on a subset of the potentially poisoned data. The surrogate then extracts layer-wise visual attention-energy profiles from a disjoint screening subset. The proposed Dual-Path Perception module combines a global GMM over the full energy profile with a local GMM over high-variance layers, removing samples flagged by both branches. The final model is trained on the retained data. Experiments report substantial reductions in attack success rates across multiple VLMs and datasets, together with preserved or improved clean-task performance.
Critical strengths
-
Informative visual presentation. Figure 3 provides a useful overview of the surrogate-training, energy-extraction, and dual-path filtering stages, making the pipeline easier to follow. The energy-landscape visualization in Figure 1 also helps readers understand the layer-wise patterns motivating the method.
-
A coherent design for data purification. Using a lightweight surrogate to examine potentially poisoned data is a practical idea that does not require a verified clean reference dataset. The disjoint proxy and screening subsets avoid evaluating detection solely on the surrogate’s training samples. Combining local and global evidence through intersection is also a reasonable design choice for limiting unnecessary removal of benign data.
Critical weaknesses
-
HED is not sufficiently distinguished from benign distribution differences, and the poison-detection criterion is underspecified. Energy differences may reflect visual content, supervision, or task heterogeneity rather than backdoor associations. Section 3.4 fits two Gaussian components and labels the higher-energy local component as suspicious, but provides no explicit threshold or criterion for declaring that no poisoning is present. It therefore remains unclear how DPP avoids removing legitimate high-energy subgroups in entirely clean data.
-
The added value of HED over existing internal-statistic approaches is insufficiently demonstrated. Prior work has used cross-modal attention anomalies for LVLM backdoor detection [1] and energy-based scores for backdoor defense [2]. Although their formulations and settings differ, these precedents warrant a clearer assessment of SPEAR’s specific contribution. Comparisons with existing attention statistics and sample loss on the same adapted surrogate are needed to separate the benefit of HED from ordinary LoRA adaptation.
-
The comparative defense advantage is established mainly under a single controlled attack protocol. Table 1 uses a black-square trigger and fixed <BD> response. Table 10 evaluates TrojVLM and VLOOD but omits competing defenses, leaving unclear whether SPEAR’s relative advantage extends to these established VLM attacks. Since TrojVLM already achieves 98.84% pre-defense ASR, attack ineffectiveness does not explain the missing comparison.
[1] Zhifang Zhang, Bojun Yang, Shuo He, Weitong Chen, Wei Emma Zhang, Olaf Maennel, Lei Feng, Miao Xu: Test-Time Attention Purification for Backdoored Large Vision Language Models. CVPR 2026: 22826-22835
[2] Yudong Gao, Honglong Chen, Peng Sun, Zhe Li, Junjian Li, Huajie Shao: Energy-based Backdoor Defense without Task-Specific Samples and Model Retraining. ICML 2024: 14611-14637
Questions for authors
-
Does DPP include a threshold or criterion for declaring that no poisoning is present? Please clarify how it avoids treating a benign high-energy subgroup as poisoned, and report its false-positive rate on entirely clean data.
-
Could you compare HED with existing attention-based statistics and sample loss on the same adapted surrogate under matched filtering conditions? This would separate the benefit of the energy representation from surrogate adaptation.
-
Could you report competing defenses under the TrojVLM and VLOOD settings in Table 10? This would establish whether SPEAR’s comparative advantage extends beyond the black-square/fixed-response protocol in Table 1.
7 comments
Response to Reviewer CQzB (Part 2/2)
W2, Q2: Comparison with existing attention statistics and sample loss.
(1) Clarifying the technical differences between SPEAR and two related works
CleanSight [1] uses the ratio of visual to textual attention at selected layers, calibrates its detector using clean reference statistics, and prunes suspicious visual tokens during inference. EBBA/EBBA+ [2] constructs energy from classifier output scores to detect backdoored models or poisoned inputs.
These methods differ from ours in their detection signals and operational stages. SPEAR extracts layer-wise visual attention energy trajectories from an adapted proxy model and filters training samples through local and global grouping mechanisms.
(2) Comparison with CleanSight and sample loss
To address the reviewer's concern, we conducted comparative evaluations on LLaVA-1.5-7B using the same LoRA-adapted surrogate checkpoint and the same screening samples for all detection signals. This controls for the influence of surrogate adaptation. HED and conventional attention statistics are evaluated with the same DPP procedure, allowing us to compare their feature representations directly. Sample loss produces a single scalar per sample and is therefore evaluated using a two-component GMM; CleanSight uses its visual-to-textual attention ratio. We report detection Precision and Recall for each method.
| Method | IconQA Precision ↑ | IconQA Recall ↑ | Flickr30k Precision ↑ | Flickr30k Recall ↑ |
|---|---|---|---|---|
| Sample loss + GMM | 19.19% | 100.00% | 95.93% | 97.43% |
| CleanSight (attention ratio) | 28.95% | 4.65% | 88.70% | 78.68% |
| Attention Statistic + DPP | 98.33% | 99.30% | 98.63% | 98.99% |
| HED + DPP | 100.00% | 99.30% | 99.36% | 99.54% |
The comparison uses the same adapted checkpoint and evaluation samples, while retaining the detection procedure appropriate to each feature representation. With DPP held fixed, HED reduces false positives from 12 to 0 at the same recall on IconQA, and from 15 to 7 while detecting six additional poisoned samples on Flickr30k. These results suggest that the HED representation can offer a modest additional benefit over conventional attention statistics under the shared-surrogate setting. Figures 5 and 1 additionally provide evidence of the role of surrogate adaptation through the energy profiles before and after adaptation. We will include these controlled comparisons in the revised manuscript.
W3, Q3: Competing defense experiments under TrojVLM and VLOOD.
We thank the reviewer for the suggestion. Table 1 of the original paper compares multiple defense methods under the black-square trigger and fixed <BD> response protocol. The original Table 10 reported SPEAR's results under TrojVLM and VLOOD; the updated matched defense comparisons are now presented in Table 3 of the revised main text.
We further conducted new comparative experiments with CleanSight, ZIP, W2SDefense, and SEER under a unified experimental protocol. For each attack, the undefended and defended settings use the same reconstruction configuration and the same set of test samples.
| Method | TrojVLM ASR ↓ | Clean CIDEr ↑ | VLOOD ASR ↓ | Clean Accuracy ↑ |
|---|---|---|---|---|
| No Defense | 99.80% | 73.51 | 17.75% | 69.73% |
| CleanSight | 72.80% | 72.79 | 16.91% | 69.85% |
| ZIP | 91.40% | 72.22 | 12.32% | 69.22% |
| W2SDefense | 20.30% | 69.26 | 0.00% | 65.44% |
| SEER (VLM-adapted) | 99.80% | 70.38 | 73.21% | 58.34% |
| SPEAR | 0.00% | 74.19 | 0.00% | 70.42% |
Under the TrojVLM attack, SPEAR reduces ASR from 99.80% to 0% while achieving the highest clean CIDEr score of 74.19. In comparison, although W2SDefense significantly suppresses the attack, its clean performance decreases to 69.26. CleanSight, ZIP, and SEER retain relatively high residual ASRs.
These results suggest that SPEAR offers a favorable balance between security and clean-task performance under the evaluated VLM backdoor attacks. These supplementary comparisons are now integrated into Table 3 of the revised main text to make the evaluation across different attack settings more complete and clear.
Thank you for the detailed responses and additional experiments. I appreciate the new comparisons, and I consider W2 largely addressed. However, I still have two concerns:
-
W1: The clean-only experiments show a low false-positive rate, but it remains unclear whether HED captures backdoor-specific patterns or general distribution anomalies. Could the authors further evaluate whether benign but distributionally different subgroups are incorrectly identified as poisoned? This would help clarify the specificity of HED and its limitations.
-
W3: The additional defense comparisons are helpful, but the undefended ASR of VLOOD changes from 61.20% in the original Table 10 to 17.75% in the rebuttal. Could the authors explain the differences in experimental settings? It would also be helpful to provide comparisons under the same attack configuration as the original Table 10.
These clarifications would help me better assess whether the remaining concerns have been addressed.
Response to Reviewer CQzB (Part 1/2)
Dear Reviewer CQzB
We thank the reviewer for the careful evaluation of our work and for recognizing our method. Below, we respond to each concern in turn.
(1) Distinguishing HED from normal data heterogeneity
Our central argument is that there exists a trade-off between the stealthiness and effectiveness of backdoor attacks. A successful backdoor attack requires the model to learn a stable trigger–target response association, which alters the model's internal information flow and attention energy distribution.
We agree that visual content, supervision information, and task heterogeneity may all cause energy fluctuations. However, the energy changes associated with backdoor correlations appear more pronounced after proxy adaptation than those arising from normal variations in visual content, supervision information, and task heterogeneity, which may help distinguish poisoned samples from heterogeneous clean data.
The results in our paper also support this observation. Figure 5 in Appendix B.2 shows that the energy curves of clean and poisoned groups nearly overlap before adaptation. The energy landscape in Figure 1 and the subsequent filtering results suggest that the separation after adaptation remains observable despite the original variations in the data.
(2) Detection thresholds and determining the absence of poisoning
One advantage of our method is that it requires neither a predefined poisoning ratio nor a manually specified fixed energy detection threshold. The current implementation does not include an additional explicit rule for determining whether a dataset contains no poisoned samples.
To address the reviewer's concern regarding the erroneous removal of clean samples, we have conducted dedicated experiments under completely clean conditions, as described below.
(3) Clean-only experimental results
Figure 4 in the main text includes an experimental setting with a poisoning ratio of 0%.
To address this question more thoroughly, we have conducted dedicated clean-only experiments. We use completely clean datasets containing no poisoned samples and apply the same energy extraction and clustering pipeline as in the standard SPEAR procedure. We observe an interesting phenomenon: approximately 99.5% of the clean samples are assigned to the main cluster, while approximately 0.5% are assigned to another cluster with anomalous energy characteristics. After removing this small subset, the model fine-tuned on the remaining majority of the data achieves even better clean-task performance than the model fine-tuned on the complete clean dataset.
Even in completely clean datasets, a small fraction of samples may exhibit attention-energy profiles that deviate from the dominant distribution. These energy outliers may reflect heterogeneous or less compatible internal representations, which are not necessarily identifiable through visual inspection. Our clean-only experiments suggest that removing such samples can benefit downstream fine-tuning.
This phenomenon is also consistent with the fundamental principles of energy-based models. LeCun et al. pointed out in A Tutorial on Energy-Based Learning (2006) that an energy function can characterize the compatibility between variable configurations, with higher energy corresponding to lower compatibility.
Our experiments further suggest that anomaly detection based on internal attention energy may have value for data quality filtering beyond detecting backdoor-poisoned samples. In completely clean datasets, a small number of samples deviating from the main energy distribution may adversely affect downstream fine-tuning, and SPEAR's filtering mechanism can mitigate this effect to some extent.We will include these clean-only experiments in the revised manuscript to further clarify the data purification effect of SPEAR.