acceptodds
Under review as a conference paper at ICLR 2027

Black-Box Backdoor Detection in Multimodal Large Language Models via Counterfactual Response Persistence

Abstract

Multimodal large language models (MLLMs) are vulnerable to backdoor attacks, where hidden triggers in images, text, or both can induce attacker-specified behaviours. Detecting such backdoors is particularly challenging in the black-box setting, where defenders can access only model inputs and generated responses. Existing defences are often designed for unimodal models or rely on model internals, while directly perturbing a modality in MLLMs can simultaneously alter task-relevant semantics, making response changes difficult to interpret. We uncover a previously overlooked phenomenon, termed the **modality shortcut**: backdoor-induced responses exhibit abnormally high persistence under cross-modal interventions, unlike benign responses. Building on this observation, we propose , a training-free black-box detector for backdoor activation in MLLMs. independently replaces each input modality with clean references and measures how strongly the original response persists after each intervention. Inputs exhibiting abnormally high persistence in either modality are identified as backdoored. requires no access to model parameters, logits, or internal activations, and assumes no knowledge of the trigger, attack mechanism, or target response. Under this strictly black-box setting, achieves at least 99.5% average TPR across all model–dataset settings at a 1% FPR operating point across nine attacks on LLaVA-1.5-7B and Qwen2.5-VL-7B over VQA-E and COCO, while remaining effective against adaptive attacks. Notably, it uses only five clean reference pairs and eleven queries per input, highlighting its effectiveness and practicality for black-box backdoor detection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal large language models, vision-language models, backdoor detection, adversarial machine learning, black-box detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PDfPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the backdoor attack detection in multimodal LLMs under strict blackbox access. It proposes B³M², a training free detector motivated by the observation that backdoor responses can persist when the trigger modality is preserved. The method separately replaces images and prompts with clean references, measures semantic similarity to the original response, and flags abnormal persistence in either direction. Across nine targeted response attacks on LLaVA-1.5-7B and Qwen2.5-VL-7B using VQA-E and COCO, the paper reports at least 99.5% average TPR in each model/dataset setting at the benchmark’s 1% FPR operating point. Additional experiments examine reference selection, reference-aware adaptive attacks, and extensions for jointly activated triggers and partially controlled responses.

Critical strengths
  1. A useful detection principle under realistic access constraints. Preserving one modality while replacing the other provides an interpretable way to probe suspicious response persistence without directly disturbing a potential trigger in the retained modality. The detector requires neither victim model internals nor prior knowledge of the trigger or target response. Its default audit uses five reference pairs and eleven victim queries per input, excluding threshold calibration. These properties make the approach relevant to auditing hosted MLLMs with restricted access.

  2. Consistent empirical performance across the evaluated trigger modalities. The main experiments show strong detection across textual, visual, and bimodal triggers on both backbones and datasets. For example, on LLaVA/VQA-E, B³M² reports 99.8% average TPR versus 88.8% for the strongest baseline, BEAT, at the reported 1% FPR operating point. This consistency supports the value of probing both replacement directions when the trigger modality is unknown.

Critical weaknesses
  1. Response persistence is not specific to backdoor activation. The modality shortcut observation is interesting and potentially useful, but the contrast with benign responses needs qualification. Legitimate prompts whose answers do not depend on the image, including instructions to ignore it, can also produce persistent responses under image replacement. Since B³M² flags abnormal persistence in either direction, these cases may overlap with its backdoor signal. Although the method calibrates thresholds on clean data, the paper does not establish whether calibration can accommodate such benign behavior without reducing detection sensitivity. A targeted evaluation with benign prompts of varying visual dependence, reporting category specific False Positive rates and attack TPR before and after recalibration, would help establish the method’s applicability beyond VQA and captioning.

  2. The Dual-Key extension lacks comparative evaluation at low false positive rates. The paper acknowledges that its standard cross-modal replacements can suppress strictly joint-trigger activation and introduces a separate probing procedure in §5.4.1. The reported AUROCs of 0.983 on VQA-E and 0.952 on COCO indicate useful separation, but do not establish performance at the low False Positive rates emphasized in the main evaluation. Moreover, the absence of baseline comparisons under the same Dual-Key setting leaves the extension’s relative effectiveness unclear. Reporting TPR at 1% and 5% FPR, together with suitable baseline comparisons under comparable query budgets, would provide stronger support for this extension.

Questions for authors
  1. How to distinguish backdoor persistence from legitimate modality independence. How does B³M² distinguish backdoor induced persistence from benign responses that legitimately do not depend on the image? Could you evaluate benign prompts with weak or no visual dependence, including instructions to ignore the image, and report their False Positive rates alongside backdoor TPR, both before and after recalibration on a representative benign mixture?

  2. How does a single detection procedure work when the attack type is unknown. The paper introduces the standard two dimension detector, a separate Dual-Key score, and factor level persistence for alternative objectives. What single configuration should a defender use when neither the activation mechanism nor the attack objective is known? If these variants are combined, how are their decisions calibrated jointly, and what overall FPR, detection performance, and query/computation costs result?

  3. Have you ever considered a unified detection procedure across settings. Given the unresolved challenge of benign modality independent persistence and the need for a separate extension for strictly joint trigger activation, have the authors considered a unified detection procedure that handles these cases without prior knowledge of the attack type? The current presentation leaves unclear how a defender should select or combine the different variants in practice. Could the authors formulate and evaluate a single detection pipeline, reporting its overall False Positive rate and detection performance across these settings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PDfP

We thank the reviewer for the detailed review and for recognising the value of our detection principle under realistic access constraints.

W1 and Q1. Specificity of persistence.

Our main experiments follow prior MLLM backdoor defences (CleanSight [1], ToT [2], TCAP [3], BYORn [4]), which evaluate on VQA and captioning where the image and the text jointly determine the response. We agree that a deployed system will also receive prompts with weak or no visual dependence, so we added a targeted study (Section 5.4) on 500 image-independent questions each from ScienceQA and ARC-Easy. Calibration on clean VQA data alone flags most of them (Table A, first row), so we add a dependence check: when an input is flagged through image replacement, a trusted reference MLLM answers the same input with the same reference images, and the flag stands only if its answer depends on the image, with the threshold calibrated on clean data; otherwise B³M² reports the input as unsupported rather than clean. The reference MLLM is a public, off-the-shelf model, like the sentence encoder B³M² already uses; it never accesses the victim, and the threat model (Section 2) now states it.

Table A. Share of inputs flagged on 500 image-independent questions per dataset, before and after the check (LLaVA-1.5-7B, 1% FPR budget; mean over three attacks per trigger type; Section 5.4, Table 2).

ScienceQAARC-Easy
Clean, no check (FPR)0.9370.955
Clean, with check (FPR)0.0350.006
Visual trigger, with check (TPR)0.9830.977
Textual trigger, check + response agreement (TPR)0.8530.809

As Table A shows, the check removes most false positives (second row), while visual triggers are still caught through the direction that keeps the image (third row). Textual triggers on these questions are reported as unsupported, like clean ones, and requiring the victim's answer to also agree with the reference answer recovers them (last row of Table A). On the regular VQA-E and COCO test inputs, attack TPR before and after the check differs by at most 1.4 points across all nine attacks, with clean FPR at 0.4%–1.4% (Appendix G, Table 14).

W2. Dual-Key at low FPR with baseline comparisons.

As suggested, we now report the Dual-Key [5] stage dJd_J at fixed FPR budgets against baselines under the same setting (Section 5.6.2, Table B); the AUROC values remain in Appendix L.

Table B. TPR on the Dual-Key victim (LLaVA-1.5-7B; Section 5.6.2, Table 4).

MethodVQA-E 1%5%7%COCO 1%5%7%
CleanSight0.0310.1050.1680.3830.5900.609
ToT0.3550.4100.4340.3550.4220.426
STRIP0.1250.2460.2460.3050.4530.547
BEAT0.0780.1720.2270.1520.3010.387
B³M² (dJd_J)0.8140.8560.9020.9060.9690.969

dJd_J outperforms every baseline at every budget in Table B. Single-source baselines struggle here because any perturbation that disrupts one key deactivates the AND-gated backdoor, so the response changes as it would for a benign input.

Q2 and Q3. A single procedure when the attack type is unknown.

The variants form one conditional pipeline (Section 5.6), in which each stage examines only the inputs left unflagged by the previous one. Whole-response persistence runs first and covers fixed-target attacks with textual, visual or bimodal triggers; factor-level persistence then handles partially attacker-controlled responses, reusing the same responses with no extra victim query; defenders who expect joint triggers can add dJd_J last. We evaluate the first two stages end to end under one joint FPR budget (new, Appendix M, Table C).

Table C. TPR at 1% / 5% FPR of the end-to-end pipeline (realised FPR ≤ 0.8% / 4.7%; LLaVA-1.5-7B; Appendix M, Table 21).

ObjectiveTriggerVQA-ECOCO
Fixed targetTextual1.000 / 1.0001.000 / 1.000
Visual0.992 / 0.9921.000 / 1.000
Malicious injectionTextual0.996 / 1.0001.000 / 1.000
Visual0.848 / 0.9530.805 / 0.957
Perceptual hijackingTextual1.000 / 1.0001.000 / 1.000
Visual0.793 / 0.9690.973 / 1.000

In Table C, the pipeline reaches an average TPR of 0.951 at 1% FPR and 0.989 at 5% FPR. We keep the fixed target for the main evaluation because it is the most common objective in this line of work, and we deliberately adopt the least assumptions: strict input-output access with no knowledge of the trigger, the target or the model internals. Within this setting, B³M² handles unimodal and bimodal triggers without knowing the attack type, and joint triggers call for the optional dJd_J stage; a single-pass detector covering every trigger type would need stronger access assumptions, which we see as a promising direction for future work.

We hope these additions address the reviewer's concerns and would be grateful if the reviewer would consider raising the score.

References

[1] Z. Zhang et al. Test-time attention purification for backdoored large vision language models. CVPR, 2026.

[2] X. Rong et al. Probing semantic insensitivity for inference-time backdoor defense in multimodal large language model. AAAI, 2026.

[3] M. Liu, H. Fang, R. Cong. TCAP: Tri-component attention profiling for unsupervised backdoor detection in MLLM fine-tuning. ICML, 2026.

[4] I. Sabolić et al. BYORn: Bootstrap your own responses to defend large vision-language models against backdoor attacks. ICML, 2026.

[5] M. Walmer et al. Dual-key multimodal backdoors for visual question answering. CVPR, 2022.

Reviewer_PDfPOpenReview ↗

Introducing a trusted reference MLLMs raise a new concern: why B³M² is necessary if we can access a trusted reference MLLMs?

Thanks for the explanation and supplemental experiments. However, I still have concerns about adding a trusted reference MLLM. 'dependence check' works because of the trusted reference MLLM's ability and contribution, not the proposed method(mainly based on Response Persistence) in this paper.

More importantly, why don't we use it to detect backdoor activation directly, if we can introduce a trusted reference MLLM? One way we could do this is that letting the trusted reference MLLM respond to the same prompt and compare the outputs with the victor model? This method, as a baseline, would be intuitively straightforward and requires less computations compared to B³M², right?

Then the following up questions would be:

  1. What's the necessity of the proposed method when having the reference MLLM baseline?
  2. How does the proposed method perform comparing to the reference MLLM baseline?

Thank you for the follow-up. We address both questions below.

Q1. Necessity of B³M² given a reference MLLM. A direct comparison between the victim and a reference MLLM requires a trusted model whose clean responses match the victim's. This holds only when the deployed model behaves like a public one. A model fine-tuned for a specific domain, style, or output format departs from public models by design, and the comparison then cannot separate a backdoor from the victim's legitimate behaviour. B³M² does not rely on such a model. Its detection signal compares the victim only with itself under source-preserving interventions, and all main results (Table 1) use no reference MLLM. The dependence check asks the reference model a weaker question, namely whether the answer to an input depends on the image. This is a property of the input, not of the victim's answer, so the reference model does not need to reproduce the victim's behaviour, and it is queried only for inputs flagged through image replacement.

Q2. Comparison with the reference-MLLM baseline. B³M² does not detect a backdoor from how different one response looks. It detects it from what stays the same: the attacker-specified content persists across the victim's responses to several counterfactual queries, while the content that depends on the replaced source changes. The suggested baseline has only one victim response and one reference response, so it sees a single difference, and a single difference cannot tell attacker-specified content apart from the ordinary variation between two models' answers to the same input. Its threshold must tolerate this variation to keep false positives low, and any backdoor output that differs from the reference no more than a normal answer does is then missed.

We implemented the baseline: the stock Qwen3-VL-8B answers the same input, and an input is flagged when the MPNet similarity between the victim's and the reference model's responses falls below the threshold at the target FPR. A fixed target such as "You have been hacked lol." is far from any normal answer, so the baseline separates it well (0.997 average TPR at 1% FPR on VQA-E). Once the backdoor output looks like an ordinary answer, the difference falls within normal output variation. Under malicious injection, where the victim answers correctly and adds the attacker's content, the baseline detects 1.2%–4.3% of triggered inputs at 1% FPR, against 90.2%–100% for B³M². Under perceptual hijacking, where the victim describes an attacker-chosen concept in a normal style, it reaches 28.1%–69.1%, against 94.1%–100% for B³M². The conditional pipeline, which runs both persistence stages under one joint FPR budget, stays at 80.5%–100% and 79.3%–100% for the two objectives (Table R).

Table R. TPR at 1% / 5% FPR on LLaVA-1.5-7B victims, 256 triggered and 256 clean inputs per setting: the reference-MLLM baseline, B³M² with factor-level persistence (Table 3), and the B³M² conditional pipeline (whole-response then factor-level persistence under one joint FPR budget, Appendix Table 21). The best value in each row is in bold.

ObjectiveTriggerDatasetReference-MLLM baselineB³M² (factor-level)B³M² (conditional pipeline)
Malicious injectionTextualVQA-E0.012 / 0.0980.996 / 0.9960.996 / 1.000
Malicious injectionTextualCOCO0.023 / 0.2031.000 / 1.0001.000 / 1.000
Malicious injectionVisualVQA-E0.027 / 0.1330.902 / 0.9690.848 / 0.953
Malicious injectionVisualCOCO0.043 / 0.2150.914 / 0.9650.805 / 0.957
Perceptual hijackingTextualVQA-E0.281 / 0.8201.000 / 1.0001.000 / 1.000
Perceptual hijackingTextualCOCO0.590 / 0.9571.000 / 1.0001.000 / 1.000
Perceptual hijackingVisualVQA-E0.316 / 0.8360.941 / 0.9880.793 / 0.969
Perceptual hijackingVisualCOCO0.691 / 0.9100.984 / 1.0000.973 / 1.000
Average0.248 / 0.5210.967 / 0.9900.927 / 0.985

We hope this response addresses your concern about the reference MLLM, and we would be grateful if you would consider raising your score.

Reviewer_zywgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes B³M², a simple yet effective black-box backdoor detection method for multimodal large language models. The key insight is that backdoor-induced responses tend to exhibit abnormal persistence when the trigger-bearing modality is preserved while the complementary modality is replaced. Based on this observation, B³M² performs bidirectional cross-modal interventions and detects suspicious inputs through response similarity, without requiring model internals, trigger knowledge, or additional training.

Experiments across two MLLMs, two datasets, and nine backdoor attacks demonstrate strong detection performance. Additional studies examine reference selection, adaptive attacks, joint triggers, and alternative backdoor objectives.

Critical strengths

1. Simple yet effective detection framework.

The proposed source-preserving cross-modal intervention is intuitive and practically meaningful. Unlike methods that perturb a potentially trigger-bearing modality, B³M² preserves each modality in turn, enabling unified detection across different trigger modalities. Its training-free and output-only design is particularly attractive for realistic black-box deployments.

2. Strong and relatively comprehensive experimental validation.

B³M² achieves consistently high detection performance across different attacks, datasets, and victim models. The additional analyses of reference selection, adaptive attacks, joint-trigger activation, and alternative backdoor objectives further strengthen the empirical contributions.

Critical weaknesses

1. Further clarification of the specificity of response persistence would be valuable.

The proposed method assumes that abnormally persistent responses indicate backdoor activation. However, certain benign inputs may also exhibit strong modality-insensitive behavior, such as image-independent questions, naturally stable answers, or consistent refusals.

The authors have already partially addressed this concern through the reference-count ablation and clean-data calibration experiments. Nevertheless, a small-scale evaluation on benign inputs with inherently stable responses would further strengthen the evidence that the persistence signal is sufficiently specific to backdoor activation.

2. Generalization across backdoor objectives could be further strengthened.

The nine main benchmark attacks share a fixed target response. While the additional experiments on malicious injection and perceptual hijacking provide encouraging evidence beyond this setting, greater diversity in target responses would make the generalization claim more convincing.

Questions for authors
  1. Have the authors examined benign inputs that naturally produce highly persistent responses under cross-modal interventions? How does B³M² perform on such inputs after clean-data threshold calibration?
  2. Could the authors further clarify how the persistence-based detection mechanism generalizes to more diverse attacker objectives, beyond the fixed-target setting in the main experiments? The supplementary experiments are encouraging, and a brief discussion of their implications would be helpful.
1 comment
AuthorsOpenReview ↗

Response to Reviewer zywg

We thank the reviewer for the positive assessment and the constructive questions.

W1 and Q1. Benign inputs with naturally persistent responses.

Our main experiments follow prior MLLM backdoor defences (CleanSight [1], ToT [2], TCAP [3], BYORn [4]), which evaluate on VQA and captioning where both modalities shape the response. We agree that image-independent prompts will appear in deployment, so we ran the small-scale study the reviewer suggested (Section 5.4) on 500 image-independent questions each from ScienceQA and ARC-Easy. Clean-data calibration alone flags most of them (Table A, first row), so we add a dependence check: when an input is flagged through image replacement, a trusted reference MLLM answers the same input with the same reference images, and the flag stands only if its answer depends on the image; otherwise B³M² reports the input as unsupported rather than clean. The reference MLLM is a public, off-the-shelf model, like the sentence encoder B³M² already uses; it never accesses the victim, and the threat model (Section 2) now states it.

Table A. Share of inputs flagged on 500 image-independent questions per dataset (LLaVA-1.5-7B, 1% FPR budget; mean over three attacks per trigger type; Section 5.4, Table 2).

ScienceQAARC-Easy
Clean, no check (FPR)0.9370.955
Clean, with check (FPR)0.0350.006
Visual trigger, with check (TPR)0.9830.977
Textual trigger, check + response agreement (TPR)0.8530.809

As Table A shows, the check removes most false positives (second row), while visual triggers are still caught through the direction that keeps the image (third row). Textual triggers on these questions are reported as unsupported, like clean ones, and requiring the victim's answer to also agree with the reference answer recovers them (last row of Table A). On the regular VQA-E and COCO test inputs the check costs at most 1.4 TPR points across all nine attacks, with clean FPR at 0.4%–1.4% (Appendix G, Table 14).

W2 and Q2. Generalisation beyond the fixed target.

A fixed target is the most common objective in this line of work, but the persistence principle does not depend on it: it only requires the attacker-controlled content to survive the replacement of the complementary modality. When only part of the response is attacker-controlled, as in malicious injection and perceptual hijacking, we measure persistence over semantic factors (Section 5.6.1), and the defender runs both stages as one conditional pipeline (Section 5.6). We now evaluate this pipeline under one joint FPR budget (new, Appendix M, Table B).

Table B. TPR at 1% / 5% FPR of the end-to-end pipeline (whole-response then factor-level persistence; realised FPR ≤ 0.8% / 4.7%; LLaVA-1.5-7B; Appendix M, Table 21).

ObjectiveTriggerVQA-ECOCO
Fixed targetTextual1.000 / 1.0001.000 / 1.000
Visual0.992 / 0.9921.000 / 1.000
Malicious injectionTextual0.996 / 1.0001.000 / 1.000
Visual0.848 / 0.9530.805 / 0.957
Perceptual hijackingTextual1.000 / 1.0001.000 / 1.000
Visual0.793 / 0.9690.973 / 1.000

In Table B, the pipeline reaches an average TPR of 0.951 at 1% FPR and 0.989 at 5% FPR, and the factor-level stage reuses the same responses with no extra victim query. We also add Qwen3-VL-8B victims with three triggers, each tied to its own target (new, Appendix P), where B³M² reaches 0.996–1.000 TPR at 1% FPR without any change.

We hope these additions answer the reviewer's questions and would be grateful if the reviewer would consider raising the score.

References

[1] Z. Zhang et al. Test-time attention purification for backdoored large vision language models. CVPR, 2026.

[2] X. Rong et al. Probing semantic insensitivity for inference-time backdoor defense in multimodal large language model. AAAI, 2026.

[3] M. Liu, H. Fang, R. Cong. TCAP: Tri-component attention profiling for unsupervised backdoor detection in MLLM fine-tuning. ICML, 2026.

[4] I. Sabolić et al. BYORn: Bootstrap your own responses to defend large vision-language models against backdoor attacks. ICML, 2026.

Reviewer_JrwZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes B3M2, a training-free black-box method for detecting backdoor activation in MLLMs through response persistence under cross-modal interventions. The research question is interesting, and the proposed method is intuitive and clearly presented. The authors also provide extensive experimental analyses, with promising results across different attacks. However, I still have several concerns about the core assumptions and experimental design. In particular, the limited diversity of attack settings, small evaluation size, and lack of validation on more recent models raise questions about the generalizability of the reported results. Overall, I think this work is interesting and well written, but I believe stronger experimental evidence is needed to fully support its claims.

Critical strengths
  1. Clear motivation and intuitive method. The paper identifies an interesting phenomenon in multimodal backdoor detection. The proposed method is easy to understand and well motivated. It also requires no access to model internals or knowledge of the trigger, making it practical for black-box settings.

  2. Good experimental coverage. The authors evaluate nine attacks across textual, visual, and bimodal triggers. The paper also includes comparisons with multiple baselines, ablation studies, and additional experiments on adaptive attacks and other backdoor objectives. These experiments provide useful support for the proposed method.

Critical weaknesses
  1. Questionable assumption behind modality shortcut. High response persistence does not necessarily indicate backdoor activation. A benign prompt may naturally produce the same response regardless of the image. How robust is the proposed method when benign inputs naturally exhibit high response persistence? Can the authors provide a more targeted analysis of such cases?

  2. Lack of detection rule for bimodal triggers. The original method may fail when both visual and textual triggers are required simultaneously. The authors introduce an additional score for Dual-Key attacks, but it is unclear how this extension fits into the overall detection framework. How should the defender decide which detection strategy to use when the trigger type is unknown?

  3. Missing comparison with combined baselines. Most existing baselines mainly examine one input modality, while B³M² considers both image and text. Could the authors include a simple combination of image-based and text-based detectors, along with single-direction ablations? This would help clarify whether the improvement mainly comes from checking both modalities or from the proposed detection design.

  4. Limited diversity of backdoor settings. The main experiments adapt nine attacks to the same 8K training mixture, 10% poisoning ratio, and a single fixed target response ("You have been hacked lol."). I would like to see more diverse attack settings. In particular, what happens when multiple attack objectives coexist in the same victim model? Since the proposed detector relies on response persistence, I am concerned that the near-perfect TPR may be partly influenced by the use of fixed-response attacks. More experiments would help demonstrate its general backdoor detection capability.

  5. Limited evaluation size. Each attack setting uses only 256 clean and 256 triggered inputs, which I find somewhat limited. This is especially concerning for TPR@1% FPR. With only 256 clean samples, two false positives give an FPR of 0.78%, while three already give 1.17%. A single sample can therefore affect the threshold selection. I would like to see evaluations on larger test sets, especially considering the near-perfect TPRs reported in the paper. How stable are these results with more evaluation samples?

  6. Limited and outdated model selection. The experiments only use LLaVA-1.5-7B and Qwen2.5-VL-7B. Both are relatively old models by 2026, and many newer and stronger MLLMs are now available. I would encourage the authors to include more recent models from different families and with larger model sizes. Evaluating only two 7B models makes it difficult to fully assess the generalizability of the proposed method.

Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer JrwZ

We thank the reviewer for the careful reading and constructive suggestions. We ran nearly all of the requested experiments; the results are summarised below and included in the revised PDF.

W1. Benign inputs with naturally high persistence.

Our main experiments follow prior MLLM backdoor defences (CleanSight [1], ToT [2], TCAP [3], BYORn [4]), which evaluate on VQA and captioning benchmarks where the image and the question jointly shape the response. We agree that a deployed system will also see prompts whose answer does not need the image, so we added a dedicated study (Section 5.4) on 500 image-independent questions each from ScienceQA and ARC-Easy. Without any change, B³M² flags most of them (Table A, first row), which confirms the concern. We therefore add a dependence check: when an input is flagged through image replacement, a trusted reference MLLM answers the same input with the same reference images, and the flag stands only if its answer depends on the image; otherwise B³M² reports the input as unsupported rather than clean. The reference MLLM is a public, off-the-shelf model, like the sentence encoder B³M² already uses; it never accesses the victim, and the threat model (Section 2) now states it.

Table A. Share of inputs flagged on 500 image-independent questions per dataset (LLaVA-1.5-7B, 1% FPR budget; mean over three attacks per trigger type; Section 5.4, Table 2).

ScienceQAARC-Easy
Clean, no check (FPR)0.9370.955
Clean, with check (FPR)0.0350.006
Visual trigger, with check (TPR)0.9830.977
Textual trigger, check + response agreement (TPR)0.8530.809

As Table A shows, the check removes most false positives (second row), while visual triggers are still caught through the direction that keeps the image (third row). Textual triggers on these questions are reported as unsupported, like clean ones, and requiring the victim's answer to also agree with the reference answer recovers them (last row of Table A). On the regular VQA-E and COCO test inputs the check costs at most 1.4 TPR points across all nine attacks, with clean FPR at 0.4%–1.4% (Appendix G, Table 14).

W2. Detection rule for bimodal triggers and the role of Dual-Key.

B³M² replaces one modality at a time, so it is designed to preserve any trigger carried by the other modality. The bimodal attacks we evaluate (BadNets-MM [5], VL-Trojan [6], MABA [7]) do not require joint activation, in line with BackdoorVLM's [8] observation that the textual trigger usually dominates, and B³M² detects them like the unimodal ones. Dual-Key [9] is stricter: both keys must be present at once, which requires an attacker who controls both modalities at training and inference time, and a one-modality replacement cannot preserve it by design. For defenders who expect this case, we provide an optional final stage dJd_J (Section 5.6.2) that keeps both sources and perturbs only the surrounding text. As also requested by Reviewer PDfP, we now compare it with baselines at fixed FPR budgets (Table B), where dJd_J outperforms every baseline at every budget.

Table B. TPR on the Dual-Key victim (LLaVA-1.5-7B; Section 5.6.2, Table 4).

MethodVQA-E 1%5%7%COCO 1%5%7%
CleanSight0.0310.1050.1680.3830.5900.609
ToT0.3550.4100.4340.3550.4220.426
STRIP0.1250.2460.2460.3050.4530.547
BEAT0.0780.1720.2270.1520.3010.387
B³M² (dJd_J)0.8140.8560.9020.9060.9690.969

Section 5.6 describes how the stages run as one conditional pipeline, so the defender does not need to know the trigger type in advance.

W3. Combined baselines and single-direction ablations.

Prior black-box detectors face a dilemma: a weak perturbation can leave benign responses unchanged (false positives), while a strong one can destroy the trigger itself (false negatives; Appendices I and J show this for BEAT [10] and STRIP [11]). B³M² avoids it by never perturbing the source it audits, which to our knowledge makes it the first strict input-output black-box detector of triggered inputs in MLLMs built on this factorised view. Following the reviewer's suggestion, we added both experiments (Appendix F) and summarise them in Table C.

Table C. Average TPR over the nine attacks at 1% / 5% FPR (LLaVA-1.5-7B; Appendix F, Tables 11 and 12).

DetectorVQA-ECOCO
dId_I only (keep image)0.551 / 0.5510.549 / 0.564
dTd_T only (keep text)0.555 / 0.5550.556 / 0.556
STRIP + BEAT0.888 / 0.8880.777 / 0.999
SCALE-UP + BEAT0.222 / 0.8880.667 / 1.000
B³M²0.998 / 0.9981.000 / 1.000

In Table C, each direction alone catches only the attacks whose trigger it keeps (first two rows), so both are needed. Combining image- and text-side detectors (third and fourth rows) does not close the gap at the 1% budget, because each component still perturbs the source it inspects: STRIP's blending misses most BadNets-I [5] and all ImgTrojan [12] inputs on VQA-E, and BEAT's probe misses BadNets-MT [13] entirely. The gain therefore comes from the source-preserving design, not only from looking at both modalities.

W4. Diversity of backdoor settings.

A fixed target is the standard objective in MLLM backdoor defence (CleanSight, ToT, TCAP and BYORn all use predefined targets), and we keep it for comparability. Beyond it, we add victims with multiple triggers and multiple targets on Qwen3-VL-8B (new, Appendix P, Table D), and we evaluate malicious injection and perceptual hijacking under one joint FPR budget (new, Appendix M, Table E).

Table D. Pooled TPR at 1% FPR on Qwen3-VL-8B victims with three triggers, each tied to its own target (Appendix P, Table 23).

VictimVQA-ECOCO
Textual triggers0.9991.000
Visual triggers0.9990.996

Table E. TPR at 1% / 5% FPR of the end-to-end pipeline (whole-response then factor-level persistence; realised FPR ≤ 0.8% / 4.7%; LLaVA-1.5-7B; Appendix M, Table 21).

ObjectiveTriggerVQA-ECOCO
Fixed targetTextual1.000 / 1.0001.000 / 1.000
Visual0.992 / 0.9921.000 / 1.000
Malicious injectionTextual0.996 / 1.0001.000 / 1.000
Visual0.848 / 0.9530.805 / 0.957
Perceptual hijackingTextual1.000 / 1.0001.000 / 1.000
Visual0.793 / 0.9690.973 / 1.000

Table D shows that B³M² needs no change when each trigger has its own target, and Table E shows that the joint pipeline reaches an average TPR of 0.951 at 1% FPR and 0.989 at 5% FPR across the three objectives.

W5. Evaluation size.

We agree that with 256 clean inputs one false positive moves the FPR by 0.39 points. We therefore recalibrate the thresholds on 512 clean inputs per dataset and test on a disjoint set of 500 clean and 500 triggered inputs (Appendix H), where one sample moves the FPR by only 0.2 points. Table F reports the results.

Table F. Average TPR and held-out FPR at a 1% budget with thresholds calibrated on 512 separate clean inputs (Appendix H, Tables 16 and 17).

VictimDatasetTPR (%)Held-out FPR (%)
LLaVA-1.5-7BVQA-E99.841.02
COCO99.960.80
Qwen2.5-VL-7BVQA-E99.531.20
COCO99.380.24

In Table F, TPR stays within 0.12 points of the main results and is flat from 1% to 5% budgets; the small FPR excess on held-out data reflects finite-sample transfer, since the calibration-set FPR never exceeds the budget. Tables 12–14 also use these 500-input sets, while Table 11 uses the 256-input split of Table 1 so that B³M² and every baseline are scored on the same inputs.

W6. Model selection.

We use the backbones of the benchmark we reproduce: BackdoorVLM evaluates exactly LLaVA-1.5-7B and Qwen2.5-VL-7B, and CleanSight, ToT and TCAP use models at the same 7B–8B scale. To address newer models, we additionally train and audit the multi-trigger victims on Qwen3-VL-8B-Instruct, where B³M² reaches 0.996–1.000 TPR at 1% FPR without any change (Table D).

We hope these results address the reviewer's concerns and would be grateful if the reviewer would consider raising the score.

References

[1] Z. Zhang et al. Test-time attention purification for backdoored large vision language models. CVPR, 2026.

[2] X. Rong et al. Probing semantic insensitivity for inference-time backdoor defense in multimodal large language model. AAAI, 2026.

[3] M. Liu, H. Fang, R. Cong. TCAP: Tri-component attention profiling for unsupervised backdoor detection in MLLM fine-tuning. ICML, 2026.

[4] I. Sabolić et al. BYORn: Bootstrap your own responses to defend large vision-language models against backdoor attacks. ICML, 2026.

[5] T. Gu et al. BadNets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 2019.

[6] J. Liang et al. VL-Trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. IJCV, 2025.

[7] S. Liang et al. Revisiting backdoor attacks against large vision-language models from domain shift. CVPR, 2025.

[8] J. Li et al. BackdoorVLM: A benchmark for backdoor attacks on vision-language models. arXiv:2511.18921, 2025.

[9] M. Walmer et al. Dual-key multimodal backdoors for visual question answering. CVPR, 2022.

[10] B. Yi et al. Probe before you talk: Towards black-box defense against backdoor unalignment for large language models. ICLR, 2025.

[11] Y. Gao et al. STRIP: A defence against trojan attacks on deep neural networks. ACSAC, 2019.

[12] X. Tao et al. ImgTrojan: Jailbreaking vision-language models with one image. NAACL, 2025.

[13] G. Cui et al. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. NeurIPS, 2022.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

We thank the AC and all reviewers for their time and constructive feedback. All three reviewers found the source-preserving idea intuitive and practical for strict black-box auditing, and asked for further evidence on specificity, generality and evaluation scale. We added new experiments for each concern; all results are in the revised PDF (changed paragraphs are marked in blue), and Table S summarises them.

Table S. Main concerns, the evidence added in the revision, and the key results (TPR/FPR at a 1% FPR budget unless stated).

Concern (reviewers)What we addedKey result
Benign inputs that naturally persist (JrwZ, zywg, PDfP)Sec. 5.4, Table 2, App. G: 500 image-independent questions each from ScienceQA and ARC-Easy; dependence check with a trusted reference MLLMClean FPR 0.937 / 0.955 → 0.035 / 0.006; visual-trigger TPR 0.983 / 0.977; VQA-E / COCO TPR with the check 0.998 / 0.996
Dual-Key at low FPR with baselines (PDfP, JrwZ)Sec. 5.6.2, Table 4B³M² (dJd_J) TPR: 0.814 (VQA-E), 0.906 (COCO); best baseline 0.355 / 0.383
Combined baselines and single-direction ablation (JrwZ)App. F, Tables 11–13TPR: B³M² 0.998 / 1.000; best combination 0.888 / 0.777; each direction alone ≈ 0.55
One procedure when the attack type is unknown (PDfP, JrwZ)Sec. 5.6, App. M, Table 21: conditional pipeline under one joint FPR budgetAverage TPR 0.951 at 1% FPR and 0.989 at 5%; realised FPR ≤ 0.8% / 4.7%
Objectives beyond a fixed target (zywg, JrwZ)Sec. 5.6.1, Table 3 and App. M, Table 21 (malicious injection, perceptual hijacking); App. P, Table 23 (3 triggers × 3 targets)Injection / hijacking TPR 0.902–1.000 (factor-level), joint-pipeline average 0.951; multi-target TPR 0.996–1.000
Evaluation size (JrwZ)App. H: 512 calibration inputs, disjoint 500 + 500 test setTPR 99.38–99.96%, held-out FPR 0.24–1.20%
Newer models (JrwZ)App. P: Qwen3-VL-8B-Instruct victimsTPR 0.996–1.000 without any change
[New] Comparison with a reference-MLLM baseline (PDfP)Follow-up response, Table R: victim vs. stock Qwen3-VL-8B response similarity, injection and hijacking victimsInjection / hijacking average TPR 0.967 (B³M²) vs. 0.248 (baseline)

These additions confirm that B³M²'s source-preserving persistence remains effective beyond the original setting, and clarify how the defender handles image-independent prompts and jointly activated triggers. We hope this summary helps the AC's assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.