- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the backdoor attack detection in multimodal LLMs under strict blackbox access. It proposes B³M², a training free detector motivated by the observation that backdoor responses can persist when the trigger modality is preserved. The method separately replaces images and prompts with clean references, measures semantic similarity to the original response, and flags abnormal persistence in either direction. Across nine targeted response attacks on LLaVA-1.5-7B and Qwen2.5-VL-7B using VQA-E and COCO, the paper reports at least 99.5% average TPR in each model/dataset setting at the benchmark’s 1% FPR operating point. Additional experiments examine reference selection, reference-aware adaptive attacks, and extensions for jointly activated triggers and partially controlled responses.
Critical strengths
-
A useful detection principle under realistic access constraints. Preserving one modality while replacing the other provides an interpretable way to probe suspicious response persistence without directly disturbing a potential trigger in the retained modality. The detector requires neither victim model internals nor prior knowledge of the trigger or target response. Its default audit uses five reference pairs and eleven victim queries per input, excluding threshold calibration. These properties make the approach relevant to auditing hosted MLLMs with restricted access.
-
Consistent empirical performance across the evaluated trigger modalities. The main experiments show strong detection across textual, visual, and bimodal triggers on both backbones and datasets. For example, on LLaVA/VQA-E, B³M² reports 99.8% average TPR versus 88.8% for the strongest baseline, BEAT, at the reported 1% FPR operating point. This consistency supports the value of probing both replacement directions when the trigger modality is unknown.
Critical weaknesses
-
Response persistence is not specific to backdoor activation. The modality shortcut observation is interesting and potentially useful, but the contrast with benign responses needs qualification. Legitimate prompts whose answers do not depend on the image, including instructions to ignore it, can also produce persistent responses under image replacement. Since B³M² flags abnormal persistence in either direction, these cases may overlap with its backdoor signal. Although the method calibrates thresholds on clean data, the paper does not establish whether calibration can accommodate such benign behavior without reducing detection sensitivity. A targeted evaluation with benign prompts of varying visual dependence, reporting category specific False Positive rates and attack TPR before and after recalibration, would help establish the method’s applicability beyond VQA and captioning.
-
The Dual-Key extension lacks comparative evaluation at low false positive rates. The paper acknowledges that its standard cross-modal replacements can suppress strictly joint-trigger activation and introduces a separate probing procedure in §5.4.1. The reported AUROCs of 0.983 on VQA-E and 0.952 on COCO indicate useful separation, but do not establish performance at the low False Positive rates emphasized in the main evaluation. Moreover, the absence of baseline comparisons under the same Dual-Key setting leaves the extension’s relative effectiveness unclear. Reporting TPR at 1% and 5% FPR, together with suitable baseline comparisons under comparable query budgets, would provide stronger support for this extension.
Questions for authors
-
How to distinguish backdoor persistence from legitimate modality independence. How does B³M² distinguish backdoor induced persistence from benign responses that legitimately do not depend on the image? Could you evaluate benign prompts with weak or no visual dependence, including instructions to ignore the image, and report their False Positive rates alongside backdoor TPR, both before and after recalibration on a representative benign mixture?
-
How does a single detection procedure work when the attack type is unknown. The paper introduces the standard two dimension detector, a separate Dual-Key score, and factor level persistence for alternative objectives. What single configuration should a defender use when neither the activation mechanism nor the attack objective is known? If these variants are combined, how are their decisions calibrated jointly, and what overall FPR, detection performance, and query/computation costs result?
-
Have you ever considered a unified detection procedure across settings. Given the unresolved challenge of benign modality independent persistence and the need for a separate extension for strictly joint trigger activation, have the authors considered a unified detection procedure that handles these cases without prior knowledge of the attack type? The current presentation leaves unclear how a defender should select or combine the different variants in practice. Could the authors formulate and evaluate a single detection pipeline, reporting its overall False Positive rate and detection performance across these settings?
4 comments
Q2 and Q3. A single procedure when the attack type is unknown.
The variants form one conditional pipeline (Section 5.6), in which each stage examines only the inputs left unflagged by the previous one. Whole-response persistence runs first and covers fixed-target attacks with textual, visual or bimodal triggers; factor-level persistence then handles partially attacker-controlled responses, reusing the same responses with no extra victim query; defenders who expect joint triggers can add last. We evaluate the first two stages end to end under one joint FPR budget (new, Appendix M, Table C).
Table C. TPR at 1% / 5% FPR of the end-to-end pipeline (realised FPR ≤ 0.8% / 4.7%; LLaVA-1.5-7B; Appendix M, Table 21).
| Objective | Trigger | VQA-E | COCO |
|---|---|---|---|
| Fixed target | Textual | 1.000 / 1.000 | 1.000 / 1.000 |
| Visual | 0.992 / 0.992 | 1.000 / 1.000 | |
| Malicious injection | Textual | 0.996 / 1.000 | 1.000 / 1.000 |
| Visual | 0.848 / 0.953 | 0.805 / 0.957 | |
| Perceptual hijacking | Textual | 1.000 / 1.000 | 1.000 / 1.000 |
| Visual | 0.793 / 0.969 | 0.973 / 1.000 |
In Table C, the pipeline reaches an average TPR of 0.951 at 1% FPR and 0.989 at 5% FPR. We keep the fixed target for the main evaluation because it is the most common objective in this line of work, and we deliberately adopt the least assumptions: strict input-output access with no knowledge of the trigger, the target or the model internals. Within this setting, B³M² handles unimodal and bimodal triggers without knowing the attack type, and joint triggers call for the optional stage; a single-pass detector covering every trigger type would need stronger access assumptions, which we see as a promising direction for future work.
We hope these additions address the reviewer's concerns and would be grateful if the reviewer would consider raising the score.
References
[1] Z. Zhang et al. Test-time attention purification for backdoored large vision language models. CVPR, 2026.
[2] X. Rong et al. Probing semantic insensitivity for inference-time backdoor defense in multimodal large language model. AAAI, 2026.
[3] M. Liu, H. Fang, R. Cong. TCAP: Tri-component attention profiling for unsupervised backdoor detection in MLLM fine-tuning. ICML, 2026.
[4] I. Sabolić et al. BYORn: Bootstrap your own responses to defend large vision-language models against backdoor attacks. ICML, 2026.
[5] M. Walmer et al. Dual-key multimodal backdoors for visual question answering. CVPR, 2022.
Introducing a trusted reference MLLMs raise a new concern: why B³M² is necessary if we can access a trusted reference MLLMs?
Thanks for the explanation and supplemental experiments. However, I still have concerns about adding a trusted reference MLLM. 'dependence check' works because of the trusted reference MLLM's ability and contribution, not the proposed method(mainly based on Response Persistence) in this paper.
More importantly, why don't we use it to detect backdoor activation directly, if we can introduce a trusted reference MLLM? One way we could do this is that letting the trusted reference MLLM respond to the same prompt and compare the outputs with the victor model? This method, as a baseline, would be intuitively straightforward and requires less computations compared to B³M², right?
Then the following up questions would be:
- What's the necessity of the proposed method when having the reference MLLM baseline?
- How does the proposed method perform comparing to the reference MLLM baseline?
Thank you for the follow-up. We address both questions below.
Q1. Necessity of B³M² given a reference MLLM. A direct comparison between the victim and a reference MLLM requires a trusted model whose clean responses match the victim's. This holds only when the deployed model behaves like a public one. A model fine-tuned for a specific domain, style, or output format departs from public models by design, and the comparison then cannot separate a backdoor from the victim's legitimate behaviour. B³M² does not rely on such a model. Its detection signal compares the victim only with itself under source-preserving interventions, and all main results (Table 1) use no reference MLLM. The dependence check asks the reference model a weaker question, namely whether the answer to an input depends on the image. This is a property of the input, not of the victim's answer, so the reference model does not need to reproduce the victim's behaviour, and it is queried only for inputs flagged through image replacement.
Q2. Comparison with the reference-MLLM baseline. B³M² does not detect a backdoor from how different one response looks. It detects it from what stays the same: the attacker-specified content persists across the victim's responses to several counterfactual queries, while the content that depends on the replaced source changes. The suggested baseline has only one victim response and one reference response, so it sees a single difference, and a single difference cannot tell attacker-specified content apart from the ordinary variation between two models' answers to the same input. Its threshold must tolerate this variation to keep false positives low, and any backdoor output that differs from the reference no more than a normal answer does is then missed.
We implemented the baseline: the stock Qwen3-VL-8B answers the same input, and an input is flagged when the MPNet similarity between the victim's and the reference model's responses falls below the threshold at the target FPR. A fixed target such as "You have been hacked lol." is far from any normal answer, so the baseline separates it well (0.997 average TPR at 1% FPR on VQA-E). Once the backdoor output looks like an ordinary answer, the difference falls within normal output variation. Under malicious injection, where the victim answers correctly and adds the attacker's content, the baseline detects 1.2%–4.3% of triggered inputs at 1% FPR, against 90.2%–100% for B³M². Under perceptual hijacking, where the victim describes an attacker-chosen concept in a normal style, it reaches 28.1%–69.1%, against 94.1%–100% for B³M². The conditional pipeline, which runs both persistence stages under one joint FPR budget, stays at 80.5%–100% and 79.3%–100% for the two objectives (Table R).
Table R. TPR at 1% / 5% FPR on LLaVA-1.5-7B victims, 256 triggered and 256 clean inputs per setting: the reference-MLLM baseline, B³M² with factor-level persistence (Table 3), and the B³M² conditional pipeline (whole-response then factor-level persistence under one joint FPR budget, Appendix Table 21). The best value in each row is in bold.
| Objective | Trigger | Dataset | Reference-MLLM baseline | B³M² (factor-level) | B³M² (conditional pipeline) |
|---|---|---|---|---|---|
| Malicious injection | Textual | VQA-E | 0.012 / 0.098 | 0.996 / 0.996 | 0.996 / 1.000 |
| Malicious injection | Textual | COCO | 0.023 / 0.203 | 1.000 / 1.000 | 1.000 / 1.000 |
| Malicious injection | Visual | VQA-E | 0.027 / 0.133 | 0.902 / 0.969 | 0.848 / 0.953 |
| Malicious injection | Visual | COCO | 0.043 / 0.215 | 0.914 / 0.965 | 0.805 / 0.957 |
| Perceptual hijacking | Textual | VQA-E | 0.281 / 0.820 | 1.000 / 1.000 | 1.000 / 1.000 |
| Perceptual hijacking | Textual | COCO | 0.590 / 0.957 | 1.000 / 1.000 | 1.000 / 1.000 |
| Perceptual hijacking | Visual | VQA-E | 0.316 / 0.836 | 0.941 / 0.988 | 0.793 / 0.969 |
| Perceptual hijacking | Visual | COCO | 0.691 / 0.910 | 0.984 / 1.000 | 0.973 / 1.000 |
| Average | 0.248 / 0.521 | 0.967 / 0.990 | 0.927 / 0.985 |
We hope this response addresses your concern about the reference MLLM, and we would be grateful if you would consider raising your score.
Response to Reviewer PDfP
We thank the reviewer for the detailed review and for recognising the value of our detection principle under realistic access constraints.
W1 and Q1. Specificity of persistence.
Our main experiments follow prior MLLM backdoor defences (CleanSight [1], ToT [2], TCAP [3], BYORn [4]), which evaluate on VQA and captioning where the image and the text jointly determine the response. We agree that a deployed system will also receive prompts with weak or no visual dependence, so we added a targeted study (Section 5.4) on 500 image-independent questions each from ScienceQA and ARC-Easy. Calibration on clean VQA data alone flags most of them (Table A, first row), so we add a dependence check: when an input is flagged through image replacement, a trusted reference MLLM answers the same input with the same reference images, and the flag stands only if its answer depends on the image, with the threshold calibrated on clean data; otherwise B³M² reports the input as unsupported rather than clean. The reference MLLM is a public, off-the-shelf model, like the sentence encoder B³M² already uses; it never accesses the victim, and the threat model (Section 2) now states it.
Table A. Share of inputs flagged on 500 image-independent questions per dataset, before and after the check (LLaVA-1.5-7B, 1% FPR budget; mean over three attacks per trigger type; Section 5.4, Table 2).
As Table A shows, the check removes most false positives (second row), while visual triggers are still caught through the direction that keeps the image (third row). Textual triggers on these questions are reported as unsupported, like clean ones, and requiring the victim's answer to also agree with the reference answer recovers them (last row of Table A). On the regular VQA-E and COCO test inputs, attack TPR before and after the check differs by at most 1.4 points across all nine attacks, with clean FPR at 0.4%–1.4% (Appendix G, Table 14).
W2. Dual-Key at low FPR with baseline comparisons.
As suggested, we now report the Dual-Key [5] stage dJ at fixed FPR budgets against baselines under the same setting (Section 5.6.2, Table B); the AUROC values remain in Appendix L.
Table B. TPR on the Dual-Key victim (LLaVA-1.5-7B; Section 5.6.2, Table 4).
dJ outperforms every baseline at every budget in Table B. Single-source baselines struggle here because any perturbation that disrupts one key deactivates the AND-gated backdoor, so the response changes as it would for a benign input.