acceptodds
Under review as a conference paper at ICLR 2027

Trigger Appearance Is Not Enough: Evidence Routing for Diverse Backdoor Activation in Large Vision-Language Models

Abstract

Backdoor attacks against large vision-language models (LVLMs) take increasingly diverse forms. Backdoor triggers can be stealthy, adaptive, or distributed across modalities. Existing analyses built on trigger inversion or attention behavior rely on a specific assumption about what a trigger looks like and break down when the true trigger form does not match it, leaving little common ground for studying these otherwise disparate phenomena. We propose a unified perspective on backdoor activation: rather than asking what a trigger looks like, we examine whether an input causes the model to reach its output through an abnormal evidence route during autoregressive generation. This view treats diverse trigger mechanisms as instances of the same underlying phenomenon, an abnormal change in how evidence is routed toward the output. Our framework localizes a decision moment at which the committed token strongly depends on multimodal evidence, constructs an evidence support graph over visual patches, textual tokens, and their causally verified cross-modal interactions through counterfactual masking, and compares the resulting route against clean evidence output behavior estimated from a small calibration set, requiring no trigger inversion, no poisoned samples, and no prior knowledge of the target response. Across multiple LVLMs, datasets, and trigger types, this evidence route deviation remains a consistent signature of backdoor activation regardless of trigger form, and its use for detection consistently achieves AUROC and AUPR greater than 0.99.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large vision-language models, backdoor triggers, evidence routing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WnrVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an evidence-routing perspective for detecting diverse backdoor activations in LVLMs. Instead of relying on the appearance or location of backdoor triggers, the method localizes a decision moment during autoregressive generation, constructs an evidence support graph using attention and counterfactual masking, and measures deviations from clean evidence-routing behavior estimated from a calibration set. Experiments across multiple LVLMs, datasets, and backdoor configurations show consistently strong detection performance.

Critical strengths
  • The paper provides an interesting unified perspective on diverse LVLM backdoors. Modeling backdoor activation through abnormal evidence routing, rather than relying on specific trigger appearances or locations, is well motivated and potentially applicable to heterogeneous trigger mechanisms.

  • The empirical evaluation is relatively comprehensive and shows consistently strong performance. The method is evaluated across multiple LVLM backbones, datasets, and substantially different backdoor configurations, including multimodal, instance-adaptive, and stealthy visual triggers.

Critical weaknesses
  • The overall methodological presentation could be improved. The paper lacks a framework overview illustrating the complete pipeline. In particular, Figure 1 directly presents the visualization of decision-moment localization rather than an overview of the proposed method. Given that the method consists of several sequential components, including decision-moment localization, evidence support graph construction, counterfactual verification, and calibration-based detection, an overview figure would substantially improve readability.

  • The extremely high detection performance raises questions about generalization to genuinely unseen backdoor mechanisms. The reported results are nearly perfect across the evaluated settings, while the considered backdoor configurations are known and instantiated during evaluation. It would be important to clarify whether the proposed evidence-routing signature remains equally effective for unseen trigger mechanisms or attack families that are not represented in the current experimental design.

  • The discussion of related internal-signal-based detection approaches appears incomplete. The paper primarily positions the proposed approach against trigger-inversion and attention-based backdoor detection, with limited discussion of methods that detect abnormal LVLM behavior using hidden representations, embeddings, or uncertainty signals. Relevant examples include evidential uncertainty-based LVLM misbehavior detection [1], TruthPrInt [2], and UniProbe [3]. Discussing these approaches would better clarify how evidence routing differs from existing internal-representation-based detection paradigms.

[1] Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification.
[2] TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination via Latent Truthful-Guided Pre-Intervention.
[3] UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations.

Questions for authors

See the weakness.

1 comment

1. Overview figure

We thank the reviewer for this valuable suggestion and agree that an overview would improve readability. We will add an overview figure that illustrates the complete pipeline of our framework.

2. Generalization to unseen backdoor mechanisms

Thank you for this question. Our framework does not learn what any particular backdoor looks like. It characterizes how evidence is routed to the output, using only clean calibration data. Every backdoor in our evaluation is unseen by our framework.

We respectfully note that evaluating on an attack is different from training on it. Any study must run attacks to test them, and our framework never sees them beforehand. The near-perfect results reflect the large gap between backdoored and clean evidence routes, not familiarity with the attacks.

This is what makes the signature general. The four configurations differ in modality, trigger form, and implantation stage: multimodal instruction tuning (VL-Trojan), cross-modal invisible triggers (BadCM), instance-adaptive triggers (IAG), and encoder-level stealthy triggers (BadVision). They all show the same abnormal evidence routing.

3.Related internal-signal-based methods

Thank you for pointing out these works. We agree that discussing them makes our positioning more complete. They address hallucination and general misbehavior rather than backdoors, and they examine whether an output is abnormal. We will add these to the related work. Our work instead examines how an output is reached. We causally verify, via counterfactual interventions, which evidence supports the output, and compare the route with clean inputs that produce the same output, using clean data only.

Reviewer_YFWjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the detection of backdoor activations in large vision-language models (LVLMs). The proposed method first identifies a generation step that exhibits strong input dependency, based on changes in generation probabilities following input masking. It then extracts marginal contributions and interaction features by masking individual image tokens, text tokens, and pairs of image-text tokens. Finally, by referencing the correspondence between "evidence" features (derived from clean samples) and output tokens, the method employs a classifier—trained on samples with randomized mismatches—to perform detection. The paper reports high detection performance across various models, datasets, and attack configurations.

Critical strengths
  1. The method accounts for image inputs, text inputs, and their interactions, enabling the analysis of input dependencies across different modalities within a unified framework.
  2. The detector does not require actual backdoor samples or prior knowledge of the attack target. While this setup is valuable for research, its effectiveness warrants more robust experimental validation.
Critical weaknesses
  1. There are discrepancies in the results for identical configurations between Table 2 and Table 4. For instance, the AUROC for Idefics3 under the RefCOCO/BadVision setting is reported as 0.9984 in Table 2, but as 1.0000 in Table 4.
  2. The current ablation studies focus primarily on the selection of the generation step and the size of the calibration set. They fail to sufficiently validate the necessity of individual components, such as interaction features, output-conditioned nearest-neighbor matching, and the classifier trained on randomized mismatches. Further analysis is needed to compare results when using only the maximum input-masking probability difference, only marginal contributions, or only interaction intensity, as well as when applying simple anomaly detection techniques to the same feature set.
  3. Each test input requires the initial generation, re-computation under masking, and multiple individual or paired interventions. The lightweight nature of the final MLP does not reflect the computational cost of the entire detection pipeline. The paper lacks reporting on the number of additional forward passes, per-sample latency, throughput, and VRAM consumption, nor does it analyze how these costs scale with input length or the number of image tokens.
  4. The method relies on a small-scale clean calibration set, yet its false-positive rate under natural distribution shifts remains unverified. Furthermore, there is no assessment of the computational cost per input. Consequently, the current experimental evidence is insufficient to support the method's effectiveness or deployability in real-world, open-world scenarios.
Questions for authors

see Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer YFWj (Part 1)

Thank you for the constructive review and for recognizing the value of our work. We respond to your questions below.

1. Discrepancy in tables (W1)

We thank the reviewer for carefully checking our results. The values in Table 4 are correct. We have fixed Table 2 in the revised manuscript and re-checked all others for consistency.

Backdoor ConfigurationMetricLLaVA-1.5-7bQwen2-VL-7B-InstructQwen3-VL-8B-InstructIdefics3-8B-Llama3InternVL3.5-8B
VL-TrojanPrecision0.99980.99960.99960.99920.9993
Recall1.00000.99980.99970.99910.9990
F1-Score0.99990.99970.99950.99920.9994
AUROC1.00000.99500.99880.99800.9987
AUPR1.00000.99830.99860.99920.9990
BadCMPrecision0.99890.99980.99940.99890.9992
Recall0.99850.99960.99890.99970.9990
F1-Score0.99870.99970.99860.99930.9993
AUROC0.99920.99900.99900.99950.9979
AUPR0.99850.99920.99840.99930.9982
IAGPrecision0.99910.99900.99950.99910.9993
Recall0.99930.99910.99900.99920.9987
F1-Score0.99920.99910.99920.99920.9991
AUROC0.99830.99900.99850.99830.9990
AUPR0.99800.99810.99880.99790.9989
BadVisionPrecision0.99970.99940.99970.99990.9995
Recall0.99960.99970.99961.00000.9994
F1-Score0.99970.99960.99981.00000.9990
AUROC0.99960.99900.99841.00000.9989
AUPR0.99900.99870.99861.00000.9985

2. Ablation studies (W2)

We would like to clarify that our framework is a sequential pipeline in which each stage depends on the previous one, rather than a set of independent modules.

We would like to raise:

(1) Why we ablate the decision moment. All components of our framework are computed at the decision moment t∗t^*. Choosing t∗t^* is the foundation of the whole pipeline, and it directly tests our central hypothesis: backdoor activation is concentrated at a few key generation steps rather than spread over the whole response. We show this results in Table 6.

(2) How our existing results support the other components. The components the reviewer mentions are not independent modules that can be evaluated in isolation. They describe an evidence route: how strongly evidence from different modalities is coupled, how strongly single-modality evidence supports the output, and whether the route agrees with clean inputs that produce the same output. Table 1 and Figure 4 show that every backdoor deviates clearly on these angles.

Interaction (Ψint\Psi_{\mathrm{int}})Single-modality (Ψm\Psi_{\mathrm{m}})Consistency (ScS_c)Combined (FF)
Clean1.9–2.3%1.5–1.8%1.2–1.9%2.5–2.9%
VL-Trojan257.6–295.3%68.8–79.1%122.9–138.5%493.1–594.2%
BadCM285.4–310.1%71.5–88.9%133.7–144.3%532.7–603.2%
IAG70.8–87.5%98.6–153.8%135.8–161.0%426.9–511.7%
BadVision39.4–50.6%195.5–256.7%129.4–139.5%489.6–529.1%

Even the smallest deviation of any attack on any component (39.4%, Ψint\Psi_{\mathrm{int}} under BadVision) is about 17×17\times the largest clean deviation (2.3%). Attacks differ only in which angle deviates most: multimodal triggers mainly in interaction, and stealthy visual triggers mainly in single-modality reliance. Consistency deviates by over 120% for every attack. When the three angles are combined, all attacks reach a similar level (427-603%), about 150150-240×240\times the clean level. This shows that the combination describes one shared signature rather than a patchwork of trigger-specific signals, and Figure3 confirms that clean and backdoored inputs separate best in this joint space.

AuthorsOpenReview ↗

Response to Reviewer YFWj (Part 2)

3. Computational overhead (W3)

We measured the end-to-end cost of every stage on a single NVIDIA A100 (RefCOCO, batch size 1):

ModelExtra FPGeneration (ms)Detection (ms)OverheadThroughput (samples/s)Peak VRAM (GB)
LLaVA-1.5-7B3352416393.130.4617.8
Qwen2-VL-7B-Instruct3362515012.400.4719.4
Qwen3-VL-8B-Instruct3362816172.570.4520.9
Idefics3-8B-Llama33375130234.030.2623.7
InternVL3.5-8B3379336364.590.2324.4

Extra FP: forward passes beyond normal generation. Overhead: detection time relative to normal generation. Throughput: end-to-end samples per second (generation + detection).

Forward passes. Each sample requires a fixed 33 extra forward passes, regardless of the backbone or input length. Each pass is a single prefill truncated at the decision moment, with no token-by-token decoding. Decision-moment localization needs only one pass, because Δt for all generation steps is computed jointly.

Latency and throughput. Detection takes 1.5–3.6 s per sample, i.e., 2.4–4.6× the time of normal generation, and the full pipeline processes 0.23–0.47 samples/s.

Memory. Peak VRAM stays between 17.8 and 24.4 GB, well within a single GPU.

Scaling. Because the number of passes is fixed, the cost grows only through the cost of each pass, which depends mainly on the number of image tokens. This explains the higher latency of Idefics3 and InternVL3.5, which encode images into more visual tokens.

In summary, the full pipeline requires a fixed 33 forward passes, 1.5–3.6 s per sample, and at most 24.4 GB of memory on a single GPU, and its cost grows only through the per-pass cost as inputs get longer, making it practical for real-world deployment.

Thank you for the response. My main concern regarding W2 remains: the component-wise statistics show that the signals differ between clean and backdoored inputs, but they do not establish how much each component contributes to detection performance. After considering the rebuttal and the other reviewers’ comments, I am maintaining my original rating.

Thank you for the follow-up. We would like to clarify that the quantities in our framework work as a whole rather than as independent parts. They are measured on the same object. The interaction and marginal terms are computed on the same evidence support graph at the same decision moment. The consistency score ScS_c builds on an evidence descriptor that already encodes both terms (Appendix C.3), and FF couples them multiplicatively (Eq. 10). Removing one term does not isolate its contribution. It changes the meaning of the others. Strong reliance on evidence is normal, since clean inputs also rely strongly on evidence. An input is suspicious only when its evidence differs from that of clean inputs producing the same output, and capturing this requires all terms together. Our contribution is the mechanism: whatever the trigger form, backdoor activation appears as an abnormal change in evidence routing. For completeness, we will add a leave-one-out analysis in the revised version.

Reviewer_BbHnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an evidence-routing-based framework for detecting backdoor activation in large vision-language models (LVLMs). It identifies an input-dependent decision moment, constructs an evidence support graph using attention-guided candidate selection with counterfactual masking, and measures evidence-route consistency against a small clean calibration set. A lightweight classifier is then trained using self-supervised evidence-output consistency learning to detect potential backdoor activation.

Critical strengths
  • The proposed framework does not require trigger inversion, poisoned samples, or knowledge of the target response. It relies on a small clean calibration set and internal model signals, making it potentially applicable to different backdoor configurations.
  • The experiments cover five LVLM architectures, multiple downstream datasets, and four representative backdoor configurations, including multimodal, instance-adaptive, and stealthy visual triggers.
Critical weaknesses
  • The main experiments primarily report the performance of the proposed method without direct quantitative comparisons against existing detectors, such as CleanSight[1], TCAP[2], and PurMM[3]. Such comparisons are important for establishing whether evidence routing provides a meaningful improvement over established detection signals.
  • The proposed framework requires multiple counterfactual evaluations to identify decision moments and verify evidence interactions. These operations may introduce substantial inference overhead for large vision-language models. However, the paper does not provide sufficient measurements of detection latency, additional forward passes, or computational costs to establish its practicality.
  • Although the experiments cover several trigger types, they do not explicitly consider attackers aware of the evidence-routing-based detector. An adaptive attacker might attempt to preserve normal evidence dependencies while inducing malicious outputs. The robustness of the proposed detection mechanism against such defense-aware attacks remains unclear.
  • The paper lacks a clear overview figure illustrating the complete pipeline of the proposed framework.

[1] Zhang Z, et al. Test-time attention purification for backdoored large vision language models. In CVPR 2026.

[2] Liu M, et al. TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuning. In ICML 2026.

[3] Jiang W, et al. PurMM: Attention-Guided Test-Time Backdoor Purification in Multimodal Large Language Models. In AAAI 2026.

Questions for authors
  • Could the authors provide direct quantitative comparisons with existing LVLM backdoor detection methods, such as CleanSight, TCAP, and PurMM?
  • What is the computational overhead of the proposed framework in terms of detection latency and additional forward passes? How does it compare with existing detection methods?
  • How robust is the proposed framework against adaptive attackers who explicitly attempt to minimize evidence-routing deviations while maintaining high attack success rates?
3 comments

Thank you for the constructive review and for recognizing the value of our work. We respond to your questions below.

1. About existing detection methods (W1,Q1)

Our main contribution is not another detector but a new way to understand backdoor activation in LVLMs. Existing work asks where the trigger is or what it looks like. We ask how the trigger drives the output. We show that very different backdoors (stealthy, instance-adaptive, and cross-modal) share one mechanism: an abnormal change in how evidence is routed to the committed output. Detection serves as a quantitative check that this mechanism holds consistently across trigger types and backbones.

The works you mention (CleanSight,TCAP, and PurMM) all rely on abnormal attention to visual tokens. They implicitly assume an image-side trigger that draws concentrated attention. Our method:

(1) Verifies causal support instead of measuring attention. Counterfactual interventions test whether a piece of evidence actually supports the output, not just whether it is attended to.

(2) Assumes no trigger form or location. Each input’s evidence route is compared with clean inputs that produce the same output, so no prior on how or where the trigger appears is needed.

2. Computational overhead (W2,Q2)

We measured the end-to-end cost of every stage on a single NVIDIA A100 (RefCOCO, batch size 1):

ModelExtra FPGeneration (ms)Detection (ms)OverheadThroughput (samples/s)Peak VRAM (GB)
LLaVA-1.5-7B3352416393.130.4617.8
Qwen2-VL-7B-Instruct3362515012.400.4719.4
Qwen3-VL-8B-Instruct3362816172.570.4520.9
Idefics3-8B-Llama33375130234.030.2623.7
InternVL3.5-8B3379336364.590.2324.4

Extra FP: forward passes beyond normal generation. Overhead: detection time relative to normal generation. Throughput: end-to-end samples per second (generation + detection).

Forward passes. Each sample requires a fixed 33 extra forward passes, regardless of the backbone or input length. Each pass is a single prefill truncated at the decision moment, with no token-by-token decoding. Decision-moment localization needs only one pass, because Δt for all generation steps is computed jointly.

Latency and throughput. Detection takes 1.5–3.6 s per sample, i.e., 2.4–4.6× the time of normal generation, and the full pipeline processes 0.23–0.47 samples/s.

Memory. Peak VRAM stays between 17.8 and 24.4 GB, well within a single GPU.

Scaling. Because the number of passes is fixed, the cost grows only through the cost of each pass, which depends mainly on the number of image tokens. This explains the higher latency of Idefics3 and InternVL3.5, which encode images into more visual tokens.

In summary, our framework adds a moderate cost of 2.4–4.6× normal generation with a fixed number of forward passes and no iterative optimization, which supports its practicality for large vision-language models.

3. Robustness against adaptive attacks (W3,Q3)

We thank the reviewer for raising this point. We would like to clarify why an attacker who tries to “preserve normal evidence dependencies” cannot easily evade our framework. A backdoor must change the output from the benign y0y_0 to the target y∗y* because of the trigger, so the trigger evidence must causally support the committed token. Our method measures exactly this causal support through counterfactual interventions, not through appearance or attention patterns that an attacker can reshape. If an attacker suppressed this causal footprint to clean levels, removing the trigger would no longer change the output, which would defeat the backdoor itself. Our evaluation already covers attacks designed for stealth and variability: stealthy encoder-level triggers (BadVision), instance-adaptive triggers (IAG), and cross-modal distributed triggers (VL-Trojan, BadCM). All of them are detected consistently without tuning to any specific attack. The contribution of our work is that any effective backdoor must still route its trigger causally into the output.

4.Overview figure (W4)

Thank you for this helpful suggestion. We will add an overview figure that illustrates the complete pipeline of our framework.

Thank you for the detailed response. The additional computational cost analysis largely addresses my concern regarding efficiency, and I appreciate the clarification of the proposed method's motivation. However, two major concerns remain unresolved. First, the lack of quantitative comparisons with existing detection methods makes it difficult to assess the empirical advantages of evidence routing. Second, while the authors argue that effective backdoors must leave causal footprints, this does not necessarily imply that such footprints must exhibit abnormal deviations from clean evidence-routing patterns. The robustness against defense-aware adaptive attacks therefore remains unclear. Overall, I will maintain my original rating.

Thank you for the follow-up. We are glad that your concern regarding efficiency has been addressed. We would like to clarify the two remaining points.

1. Quantitative comparison

Rather than focusing only on detection performance, our work explains why detection is possible in the first place. It offers a new perspective that explains how triggers drive the output, and this view unifies backdoor mechanisms that have so far been studied separately. A direct numerical comparison with methods built on narrower trigger assumptions would not speak to this contribution.

2. Adaptive attacks

We respectfully note that this concern does not point to any specific adaptive attack, and such an attack would also be difficult to construct in principle. Evading our framework would require the triggered output to be supported by the input in the same way as a clean output, and an output supported in this way would no longer be a backdoor behavior.

Comments on the paper

1 comment
Area_Chair_jaGhOpenReview ↗

Please engage with authors

Dear Reviewers,

The author-reviewer discussion will be closed on November 18 (AoE). If you haven’t yet replied to the authors’ latest response, please do so on OpenReview as soon as possible so they have time to follow up. A brief reply indicating which of your concerns have been addressed and what questions remain would be helpful. Thank you for your time and for engaging with the authors.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.