acceptodds
Under review as a conference paper at ICLR 2027

SPEAR: Piercing Stealthy Backdoors in Vision-Language Models with Energy Signatures

Abstract

VLMs have transformed multimodal understanding, yet they remain vulnerable to stealthy backdoor attacks that traditional input-centric or distillation-based defenses often fail to detect. We observe that although the triggers may be visually subtle, successful trigger-target shortcuts consistently induce measurable shifts in the model's internal attention-energy profiles, which we formalize as Heterogeneous Energy Divergence (HED). Based on this discovery, we propose a data purification framework called SPEAR (Surrogate-based Poisoned-sample Energy Analysis and Removal). SPEAR utilizes Energy-Driven Surrogate Learning (EDSL) to transform latent triggers into measurable energy anomalies and a Dual-Path Perception (DPP) mechanism to excise poisoned samples by intersecting global distribution shifts with local layer-wise spikes. Extensive evaluations across multiple benchmarks and VLM architectures show that SPEAR reduces ASR to near-zero while in most cases improving clean reasoning fidelity under equal-volume tuning. Additional stress tests under ultra-low poisoning rates, diverse VLM-effective triggers, existing VLM backdoor attacks, and an adaptive energy-regularized attacker suggest that SPEAR captures a recurring internal energy signature in evaluated backdoor shortcuts rather than a fixed trigger artifact. The code is available at https://anonymous.4open.science/r/backdoorvlm-071D/link.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vlm backdoor defense

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_CQzBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPEAR, a data-purification framework for backdoor defense in vision-language instruction tuning. A diagnostic surrogate is first LoRA-tuned on a subset of the potentially poisoned data. The surrogate then extracts layer-wise visual attention-energy profiles from a disjoint screening subset. The proposed Dual-Path Perception module combines a global GMM over the full energy profile with a local GMM over high-variance layers, removing samples flagged by both branches. The final model is trained on the retained data. Experiments report substantial reductions in attack success rates across multiple VLMs and datasets, together with preserved or improved clean-task performance.

Critical strengths
  1. Informative visual presentation. Figure 3 provides a useful overview of the surrogate-training, energy-extraction, and dual-path filtering stages, making the pipeline easier to follow. The energy-landscape visualization in Figure 1 also helps readers understand the layer-wise patterns motivating the method.

  2. A coherent design for data purification. Using a lightweight surrogate to examine potentially poisoned data is a practical idea that does not require a verified clean reference dataset. The disjoint proxy and screening subsets avoid evaluating detection solely on the surrogate’s training samples. Combining local and global evidence through intersection is also a reasonable design choice for limiting unnecessary removal of benign data.

Critical weaknesses
  1. HED is not sufficiently distinguished from benign distribution differences, and the poison-detection criterion is underspecified. Energy differences may reflect visual content, supervision, or task heterogeneity rather than backdoor associations. Section 3.4 fits two Gaussian components and labels the higher-energy local component as suspicious, but provides no explicit threshold or criterion for declaring that no poisoning is present. It therefore remains unclear how DPP avoids removing legitimate high-energy subgroups in entirely clean data.

  2. The added value of HED over existing internal-statistic approaches is insufficiently demonstrated. Prior work has used cross-modal attention anomalies for LVLM backdoor detection [1] and energy-based scores for backdoor defense [2]. Although their formulations and settings differ, these precedents warrant a clearer assessment of SPEAR’s specific contribution. Comparisons with existing attention statistics and sample loss on the same adapted surrogate are needed to separate the benefit of HED from ordinary LoRA adaptation.

  3. The comparative defense advantage is established mainly under a single controlled attack protocol. Table 1 uses a black-square trigger and fixed <BD> response. Table 10 evaluates TrojVLM and VLOOD but omits competing defenses, leaving unclear whether SPEAR’s relative advantage extends to these established VLM attacks. Since TrojVLM already achieves 98.84% pre-defense ASR, attack ineffectiveness does not explain the missing comparison.

[1] Zhifang Zhang, Bojun Yang, Shuo He, Weitong Chen, Wei Emma Zhang, Olaf Maennel, Lei Feng, Miao Xu: Test-Time Attention Purification for Backdoored Large Vision Language Models. CVPR 2026: 22826-22835

[2] Yudong Gao, Honglong Chen, Peng Sun, Zhe Li, Junjian Li, Huajie Shao: Energy-based Backdoor Defense without Task-Specific Samples and Model Retraining. ICML 2024: 14611-14637

Questions for authors
  1. Does DPP include a threshold or criterion for declaring that no poisoning is present? Please clarify how it avoids treating a benign high-energy subgroup as poisoned, and report its false-positive rate on entirely clean data.

  2. Could you compare HED with existing attention-based statistics and sample loss on the same adapted surrogate under matched filtering conditions? This would separate the benefit of the energy representation from surrogate adaptation.

  3. Could you report competing defenses under the TrojVLM and VLOOD settings in Table 10? This would establish whether SPEAR’s comparative advantage extends beyond the black-square/fixed-response protocol in Table 1.

7 comments
AuthorsOpenReview ↗

Response to Reviewer CQzB (Part 1/2)

Dear Reviewer CQzB

We thank the reviewer for the careful evaluation of our work and for recognizing our method. Below, we respond to each concern in turn.

W1, Q1: Sources of energy differences and false positives on completely clean data.

(1) Distinguishing HED from normal data heterogeneity

Our central argument is that there exists a trade-off between the stealthiness and effectiveness of backdoor attacks. A successful backdoor attack requires the model to learn a stable trigger–target response association, which alters the model's internal information flow and attention energy distribution.

We agree that visual content, supervision information, and task heterogeneity may all cause energy fluctuations. However, the energy changes associated with backdoor correlations appear more pronounced after proxy adaptation than those arising from normal variations in visual content, supervision information, and task heterogeneity, which may help distinguish poisoned samples from heterogeneous clean data.

The results in our paper also support this observation. Figure 5 in Appendix B.2 shows that the energy curves of clean and poisoned groups nearly overlap before adaptation. The energy landscape in Figure 1 and the subsequent filtering results suggest that the separation after adaptation remains observable despite the original variations in the data.

(2) Detection thresholds and determining the absence of poisoning

One advantage of our method is that it requires neither a predefined poisoning ratio nor a manually specified fixed energy detection threshold. The current implementation does not include an additional explicit rule for determining whether a dataset contains no poisoned samples.

To address the reviewer's concern regarding the erroneous removal of clean samples, we have conducted dedicated experiments under completely clean conditions, as described below.

(3) Clean-only experimental results

Figure 4 in the main text includes an experimental setting with a poisoning ratio of 0%.

To address this question more thoroughly, we have conducted dedicated clean-only experiments. We use completely clean datasets containing no poisoned samples and apply the same energy extraction and clustering pipeline as in the standard SPEAR procedure. We observe an interesting phenomenon: approximately 99.5% of the clean samples are assigned to the main cluster, while approximately 0.5% are assigned to another cluster with anomalous energy characteristics. After removing this small subset, the model fine-tuned on the remaining majority of the data achieves even better clean-task performance than the model fine-tuned on the complete clean dataset.

ModelDatasetFull Clean FT (CP↑)Random Filtering (CP↑)SPEAR (CP↑)Cluster DistributionFiltering Ratio / FPR ↓ΔCP ↑
LLaVA-1.5-7BVSR73.8373.7574.5299.4% / 0.6%0.60%+0.69
LLaVA-1.5-7BIconQA58.1258.0559.1099.3% / 0.7%0.70%+0.98
LLaVA-1.5-7BFlickr30k62.7462.5563.2899.5% / 0.5%0.50%+0.54
LLaVA-1.5-7BScienceQA76.7876.6077.4199.6% / 0.4%0.40%+0.63

Even in completely clean datasets, a small fraction of samples may exhibit attention-energy profiles that deviate from the dominant distribution. These energy outliers may reflect heterogeneous or less compatible internal representations, which are not necessarily identifiable through visual inspection. Our clean-only experiments suggest that removing such samples can benefit downstream fine-tuning.

This phenomenon is also consistent with the fundamental principles of energy-based models. LeCun et al. pointed out in A Tutorial on Energy-Based Learning (2006) that an energy function can characterize the compatibility between variable configurations, with higher energy corresponding to lower compatibility.

Our experiments further suggest that anomaly detection based on internal attention energy may have value for data quality filtering beyond detecting backdoor-poisoned samples. In completely clean datasets, a small number of samples deviating from the main energy distribution may adversely affect downstream fine-tuning, and SPEAR's filtering mechanism can mitigate this effect to some extent.We will include these clean-only experiments in the revised manuscript to further clarify the data purification effect of SPEAR.

AuthorsOpenReview ↗

Response to Reviewer CQzB (Part 2/2)

W2, Q2: Comparison with existing attention statistics and sample loss.

(1) Clarifying the technical differences between SPEAR and two related works

CleanSight [1] uses the ratio of visual to textual attention at selected layers, calibrates its detector using clean reference statistics, and prunes suspicious visual tokens during inference. EBBA/EBBA+ [2] constructs energy from classifier output scores to detect backdoored models or poisoned inputs.

These methods differ from ours in their detection signals and operational stages. SPEAR extracts layer-wise visual attention energy trajectories from an adapted proxy model and filters training samples through local and global grouping mechanisms.

(2) Comparison with CleanSight and sample loss

To address the reviewer's concern, we conducted comparative evaluations on LLaVA-1.5-7B using the same LoRA-adapted surrogate checkpoint and the same screening samples for all detection signals. This controls for the influence of surrogate adaptation. HED and conventional attention statistics are evaluated with the same DPP procedure, allowing us to compare their feature representations directly. Sample loss produces a single scalar per sample and is therefore evaluated using a two-component GMM; CleanSight uses its visual-to-textual attention ratio. We report detection Precision and Recall for each method.

MethodIconQA Precision ↑IconQA Recall ↑Flickr30k Precision ↑Flickr30k Recall ↑
Sample loss + GMM19.19%100.00%95.93%97.43%
CleanSight (attention ratio)28.95%4.65%88.70%78.68%
Attention Statistic + DPP98.33%99.30%98.63%98.99%
HED + DPP100.00%99.30%99.36%99.54%

The comparison uses the same adapted checkpoint and evaluation samples, while retaining the detection procedure appropriate to each feature representation. With DPP held fixed, HED reduces false positives from 12 to 0 at the same recall on IconQA, and from 15 to 7 while detecting six additional poisoned samples on Flickr30k. These results suggest that the HED representation can offer a modest additional benefit over conventional attention statistics under the shared-surrogate setting. Figures 5 and 1 additionally provide evidence of the role of surrogate adaptation through the energy profiles before and after adaptation. We will include these controlled comparisons in the revised manuscript.

W3, Q3: Competing defense experiments under TrojVLM and VLOOD.

We thank the reviewer for the suggestion. Table 1 of the original paper compares multiple defense methods under the black-square trigger and fixed <BD> response protocol. The original Table 10 reported SPEAR's results under TrojVLM and VLOOD; the updated matched defense comparisons are now presented in Table 3 of the revised main text.

We further conducted new comparative experiments with CleanSight, ZIP, W2SDefense, and SEER under a unified experimental protocol. For each attack, the undefended and defended settings use the same reconstruction configuration and the same set of test samples.

MethodTrojVLM ASR ↓Clean CIDEr ↑VLOOD ASR ↓Clean Accuracy ↑
No Defense99.80%73.5117.75%69.73%
CleanSight72.80%72.7916.91%69.85%
ZIP91.40%72.2212.32%69.22%
W2SDefense20.30%69.260.00%65.44%
SEER (VLM-adapted)99.80%70.3873.21%58.34%
SPEAR0.00%74.190.00%70.42%

Under the TrojVLM attack, SPEAR reduces ASR from 99.80% to 0% while achieving the highest clean CIDEr score of 74.19. In comparison, although W2SDefense significantly suppresses the attack, its clean performance decreases to 69.26. CleanSight, ZIP, and SEER retain relatively high residual ASRs.

These results suggest that SPEAR offers a favorable balance between security and clean-task performance under the evaluated VLM backdoor attacks. These supplementary comparisons are now integrated into Table 3 of the revised main text to make the evaluation across different attack settings more complete and clear.

Thank you for the detailed responses and additional experiments. I appreciate the new comparisons, and I consider W2 largely addressed. However, I still have two concerns:

  1. W1: The clean-only experiments show a low false-positive rate, but it remains unclear whether HED captures backdoor-specific patterns or general distribution anomalies. Could the authors further evaluate whether benign but distributionally different subgroups are incorrectly identified as poisoned? This would help clarify the specificity of HED and its limitations.

  2. W3: The additional defense comparisons are helpful, but the undefended ASR of VLOOD changes from 61.20% in the original Table 10 to 17.75% in the rebuttal. Could the authors explain the differences in experimental settings? It would also be helpful to provide comparisons under the same attack configuration as the original Table 10.

These clarifications would help me better assess whether the remaining concerns have been addressed.

Reviewer_sUJGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the backdoor defense and proposes a data purification framework. The motivation is: poisoned samples exhibit heterogeneous attention-energy divergence (HED), which becomes more distinguishable after lightweight surrogate fine-tuning. Based on this, the paper designs methods to remove flagged samples from suspect training data, and then fine-tune the model with cleaned data. The evaluation is across multiple VLM architectures, datasets, poisoning rates, and attack settings. The reported results demonstrate near-perfect poisoned-sample detection and ASR drop while keeping clean-task performance comparable.

Critical strengths

1.The attention-energy is an attention-based method: At each transformer layer, take the last instruction token as the query. Examine its raw attention scores toward visual tokens, average those scores across heads. The attention score can be powerful to distinguish the obvious poisoned samples vs. clean samples.

2.The appendix is very comprehensive and contains a lot of work. appreciate it.

Critical weaknesses

1.The paper requires to identify the suspect training data first. In real practice, it can be hard to identify the suspect training data, and the real poisoned data can be hard to detect/identified by attention score. This paper seems use obvious trigger, what about the trigger is invisible, what about the trigger is text instead of image trigger? if you cannot successfully identify the real poisoned data, the proposed defense method will not work.

2.Robustness analysis of the real poisoned data in suspect training data. In paper's setting, What is the proportion of real poisoned data suspect training data? ANy analysis, for example, across 10%-90% of suspect training data is poisoned data, what is the model performance? Especially when there is only 10% of suspect training data is real poisoned data, the defense may not work.

3.The paper does not provide sufficient evidence on defense performance against different attack baselines. I saw some attack baseline analysis in appendix, but it is not well-centered and it would be better to put together in main paper.

Questions for authors

1.How does SPEAR behave when the training data contains no poisoned samples.

5 comments
AuthorsOpenReview ↗

Response to Reviewer sUJG (Part 1/2)

Dear Reviewer sUJG

We thank the reviewer for the careful evaluation of our work and for recognizing the completeness of our appendix experiments. Below, we respond to each concern in turn.

W1: Identification of the suspect dataset and detection of stealthy triggers.

(1) SPEAR does not require prior identification of suspicious samples

We appreciate the opportunity to clarify our method's setting. The entire SPEAR pipeline does not require prior knowledge of which samples are suspicious. The term "suspect dataset" in our paper refers to the entire training dataset that may have been contaminated by poisoning.

Specifically, we first randomly partition a subset DproxyD_{\mathrm{proxy}} from the full training dataset and use it to obtain a proxy model through LoRA fine-tuning. We then use the proxy model to extract attention energy features from another independent subset and identify and remove poisoned samples through unsupervised clustering. The entire process requires no prior knowledge of poisoning labels, trigger locations, or attack target information.

(2) Stealthy triggers and the generalization capability of HED

Our central argument is that there exists a trade-off between the stealthiness and effectiveness of backdoor attacks. A successful backdoor attack requires the model to learn a stable trigger–target response association, which alters the model's internal information flow and attention energy distribution. HED is designed to capture such internal changes.

As reported in the main text and appendix, we evaluated different triggers, including randomized trigger locations, scale variations, texture triggers, low-visibility blended triggers, and text-semantics-conditioned Semantic Hijacking. Table 9 shows that SPEAR reduces ASR to 0% across all these settings. Furthermore, the adaptive attack experiment in Table 12 provides additional support for this mechanism: when the attacker suppresses the HED signal through energy regularization, the attack success rate decreases from 99.50% to 8.96%. This suggests a trade-off between concealing internal energy anomalies and maintaining an effective backdoor under the evaluated setting.

For traditional image classification backdoor attacks such as WaNet, BppAttack, and ISSBA, we have also conducted dedicated experiments in Appendix Table 10. The results show that directly transferring these triggers to the VLM instruction fine-tuning setting fails to establish effective backdoors, with all pre-defense ASRs being 0%. This observation may reflect differences between VLM backdoor attacks and traditional classification-model backdoor attacks.

W2: Robustness under different poisoning ratios.

The default poisoning ratio is explicitly specified in Section 4.1 (Attack Configuration and Trigger Validity) of the main text: our primary experiments use a poisoning ratio of 15%.

The main text and appendix report ablations over different poisoning ratios. Figure 4 in Section 4.3 examines poisoning ratios ranging from 0% to 45%, covering settings from no poisoning and low-ratio poisoning to high-ratio poisoning. Furthermore, Table 7 in Appendix D.1 reports five new independent runs at extremely low poisoning ratios of 0.1%, 0.5%, and 1.0%. At 0.1%, SPEAR achieves 100.00% Precision, 100.00% Recall, and 0.00% post-defense ASR, with pre-defense ASR of 28.0 ± 6.0%. At 0.5% and 1.0%, Precision remains 100.00%, Recall reaches 99.17% and 99.57%, respectively, and post-defense ASR is 0.00% in both settings. The new low-ratio results are based on five independently sampled partitions and are separate from the original ablation results.

In addition, we have conducted supplementary experiments with poisoning ratios ranging from 45% to 90% on LLaVA-1.5-7B × IconQA, as shown in the table below. These results suggest that SPEAR can maintain high detection precision and recall even under extremely high poisoning ratios, although downstream performance gradually degrades. They also suggest that the proxy learning process can extract informative energy anomaly features across poisoning ratios ranging from 0% to 90%.

Poisoning RatePrecision ↑Recall ↑CP ↑ASR ↓
15%100.00%99.50%61.86%0.00%
45%99.80%99.70%61.20%0.00%
60%99.70%99.60%60.40%0.30%
75%99.40%99.20%59.00%1.50%
90%98.50%98.80%56.50%6.80%
AuthorsOpenReview ↗

Response to Reviewer sUJG (Part 2/2)

W3: Comparison with different attack methods and defense baselines.

The main text and appendix report comparisons with several defense methods. Table 1 in the main text includes representative defense methods such as ZIP, W2SDefense, and SEER. The appendix further includes methods such as Spectral Signatures, Activation Clustering, and SCAn, while Table 3 in the revised main text reports the matched defense comparisons under two existing VLM backdoor attacks, TrojVLM and VLOOD.

We thank the reviewer for the suggestion regarding the organization of our experiments. We agree that presenting these results together in the main text would help readers more directly understand SPEAR's performance under different attack settings. The revised manuscript now integrates the key attack generalization and defense comparisons into the main text (Table 3).

Q1: How does SPEAR perform when the training data contain no poisoned samples?

We thank the reviewer for raising this question. Figure 4 in the main text includes a setting with a poisoning ratio of 0%.

To address this question more thoroughly, we have conducted dedicated clean-only experiments. We use completely clean datasets containing no poisoned samples and apply the same energy extraction and clustering pipeline as in the standard SPEAR procedure. We observe an interesting phenomenon: approximately 99.5% of the clean samples are assigned to the main cluster, while approximately 0.5% are assigned to another cluster with anomalous energy characteristics. After removing this small subset, the model fine-tuned on the remaining majority of the data achieves even better clean-task performance than the model fine-tuned on the complete clean dataset.

ModelDatasetFull Clean FT (CP↑)Random Filtering (CP↑)SPEAR (CP↑)Cluster DistributionFiltering Ratio / FPR ↓ΔCP ↑
LLaVA-1.5-7BVSR73.8373.7574.5299.4% / 0.6%0.60%+0.69
LLaVA-1.5-7BIconQA58.1258.0559.1099.3% / 0.7%0.70%+0.98
LLaVA-1.5-7BFlickr30k62.7462.5563.2899.5% / 0.5%0.50%+0.54
LLaVA-1.5-7BScienceQA76.7876.6077.4199.6% / 0.4%0.40%+0.63

Even in completely clean datasets, a small fraction of samples may exhibit attention-energy profiles that deviate from the dominant distribution. These energy outliers may reflect heterogeneous or less compatible internal representations, which are not necessarily identifiable through visual inspection. Our clean-only experiments suggest that removing such samples can benefit downstream fine-tuning, indicating a potential additional role of HED in data quality filtering.

This phenomenon is also consistent with the fundamental principles of energy-based models. LeCun et al. pointed out in A Tutorial on Energy-Based Learning (2006) that, even when all samples are normal, an energy function can characterize the compatibility between variable configurations, with higher energy corresponding to lower compatibility. We will include these clean-only experiments in the revised manuscript to further clarify the data purification effect of SPEAR.

Dear Reviewer sUJG,

As the discussion deadline is approaching, we would like to kindly follow up on our rebuttal and revised manuscript.

We believe that our detailed responses and additional experiments have adequately addressed your main concerns, particularly regarding the method's assumptions, robustness across poisoning ratios, and generalization to different attacks. We would appreciate it if you could revisit our responses and reconsider your assessment in light of the additional evidence. If you have any remaining questions or concerns, we would be happy to provide further clarification.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.