acceptodds
Withdrawn from ICLR 2027

It Takes Two to Tango: Backdoor Attack for Multimodal Diffusion Models via Cross-modal Collaborative Triggers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

With the increasing popularity of decoupled cross-attention-based Text-to-Image (T2I) Diffusion Models (DMs) in image editing tasks, maintaining their integrity and reliability is becoming an important issue. However, when dealing with multimodal feature fusion, such DMs rely heavily on two decoupled cross-attentions, which inevitably leak feature-projection information, making them vulnerable to exploitation by malicious adversaries. In light of this vulnerability, this paper introduces a novel Cross-Modal Backdoor Attack (XMBA) targeting these decoupling-based (i.e., decoupled cross-attention-based) T2I DMs. Unlike backdoor attacks on T2I DMs, which rely solely on text triggers, XMBA focuses on scenarios that involve the joint use of multimodal triggers. To enhance the effectiveness of the attack, XMBA adopts a target inversion method, which inverts the target image into a target text vector through the reverse process of DMs, improving the alignment between target texts and images. Meanwhile, based on our projection alignment approach, XMBA modifies the parameters of the text and image projection matrices to align trigger features with those of their corresponding targets in the latent space, enabling backdoor injection without additional training data. Comprehensive experiments demonstrate that XMBA can effectively attack decoupling-based T2I DMs in just 1.6 seconds, achieving a success rate of up to 99.5% without compromising the model's generative ability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Backdoor Attack, Cross-modal Triggers, Diffusion Model, Multimodal, Text-to-Image

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_soZWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the backdoor vulnerability of decoupling-based T2I DMs. Specifically, the authors introduce a two-stage injection method, including target inversion and projection alignment, to achieve multimodal trigger injection. Experiments show that the proposed method achieves a high ASR while maintaining benign performance.

Critical strengths

The proposed backdoor injection method is lightweight. The experiments consider existing backdoor attacks and achieve good performance on three models with multiple metrics.

Critical weaknesses
  • Since the title claims to target “multimodal diffusion models”, more recent and widely used architectures, such as the SD3.5 and FLUX families, should be considered in the evaluation. Otherwise, the scope should be more clearly limited to decoupling-based models.

  • The definition of stealthiness is inaccurate and the evaluation is insufficient. The paper lacks formal definitions of ASR-T and ASR-I. Stealthiness should also reflect the ability to bypass defense methods. However, the paper only evaluates UFID as a defense, which is insufficient given that several effective backdoor defense methods have already been proposed [1][2][3][4]. A more comprehensive evaluation against existing defenses is necessary to substantiate the claimed stealthiness of XMBA.

[1] Wang et al. T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models. ECCV’24.

[2] Xu et al. Fine-grained Prompt Screening: Defending Against Backdoor Attack on Text-to-Image Diffusion Models. IJCAI’25.

[3] Zhai et al. Efficient Input-level Backdoor Defense on Text-to-Image Synthesis via Neuron Activation Variation. ICCV’25.

[4] Li et al. BlackMirror: Black-Box Backdoor Detection for Text-to-Image Models via Instruction-Response Deviation. CVPR’26.

Questions for authors

What detection threshold is used for UFID? Please provide the exact threshold value and explain how it is selected. Without this information, it is difficult to assess the validity of the claim that “UFID has a false acceptance rate of 0% and a false rejection rate of 100% on XMBA.” Such a result could be caused by an improperly selected threshold. I therefore suggest reporting threshold-independent or more comprehensive metrics, such as ROC-AUC, together with metrics such as F1 score, to provide a more reliable evaluation of the defense performance.

Reviewer_CgWXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a white-box, parameter-level backdoor attack on text-image dual-conditioned diffusion models with decoupled cross-attention. The authors propose a method called XMBA. In Stage 1, they use Target Inversion to optimize continuous text embeddings with a frozen diffusion U-Net. This makes the target text representation better match the chosen target image. In Stage 2, they map the text and image trigger features to their target features. They then edit the two cross-attention projection matrices using a closed-form solution based on regularized least squares, without using an additional poisoned training dataset. The attack aims to generate outputs from the target class when both text and image triggers are present, while keeping normal generation behavior largely unchanged. Experiments report ASRs ranging from 90.30% to 99.50% across different models and datasets. Compared with the authors' simple poisoning-based fine-tuning baseline, XMBA shows clear improvements in both ASR and FID on clean inputs.

Critical strengths
  1. The paper focuses on white-box backdoor threats in models with decoupled text and image conditioning, rather than traditional text-only triggers. To address the lack of matching text conditions for target images, the authors design Target Inversion and propose a closed-form projection editing method with weight-shift regularization. This problem setting has clear research value, especially in resource-limited scenarios where no additional poisoned training data is available.

  2. The method shows strong attack performance across several victim models. In all six settings in Table 1, XMBA achieves an ASR above 90%. Compared with the Naive Baseline used in the paper, XMBA achieves much higher ASR and lower FID.

Critical weaknesses
  1. The central claim of "Cross-Modal Collaborative Trigger" and its advantages lack fair and well-controlled validation. Equations (2)–(4) edit the text and image projection branches separately, without any explicit constraint that requires both triggers to activate together. Therefore, the optimization does not guarantee that the two conditions depend on each other.

  2. The key metrics, runtime costs, and comparison protocols are not reported in enough detail to support the main conclusions or allow reliable reproduction. Table 2 only compares the Naive Baseline (1312.5 s) with XMBA (1.6 s), but does not include EvilEdit, which also focuses on training-free parameter editing. More importantly, the paper does not clarify whether the reported 1.6 s for XMBA includes the full iterative gradient optimization of continuous vectors in Stage 1. Therefore, the claim of an approximately 800× speedup is not well supported. Table 3 does not provide the actual prompts used for GPT-4o, the number and backgrounds of human annotators, the annotation guidelines, or the actual target descriptions. The comparison also mixes two key differences: continuous optimizable vectors versus discrete natural language, and optimization using victim model feedback versus external descriptions. In addition, the reported cross-modal cosine similarity scores exceed 1, which raises concerns about how this metric is calculated and reported.

Questions for authors
  1. Please add four controlled settings using exactly the same victim model, target image, and Stage 1 representation: Text-only XMBA (editing only W_T), Image-only XMBA (editing only W_I), Full XMBA (editing both), and a Clean baseline. For each setting, report the target class hit rate, ASR, ASR-T, ASR-I, and Fidelity under all four trigger combinations. Does the full method significantly outperform either single-branch method?

  2. Please add a joint-trigger fine-tuning baseline with a reasonable hyperparameter search. At a minimum, report the ASR–FID trade-off of the Naive Baseline under different poisoning rates (n), training epochs, and regularization settings. Please also explain why the current training budget was chosen. Is there a stronger joint-trigger baseline that achieves performance close to XMBA?

  3. Does the 1.6 s reported in Table 2 include the total runtime of both Stage 1 and Stage 2? Please also report the runtime of the other comparison methods, not just the baseline.

  4. What was the full prompt used for GPT-4o in Table 3, and how were the candidate descriptions selected? How many human annotators were involved, what were their backgrounds, what instructions were they given, and what were the exact descriptions they wrote? How were the five zebra images divided between description creation and evaluation? Please also report robustness results using multiple strong GPT-4o prompts and human-written descriptions under the same budget.

  5. Why are the Cosine Similarity scores in Table 3 reported as 31.33, 29.94, and 22.21, rather than raw cosine similarity values within the range [-1, 1]?

  6. Please keep the backdoored model fixed and change only the trigger positions at test time. For text triggers, place the same "beautiful cat" phrase at the beginning, middle, and end of the prompt. For image triggers, place the same visual subject on the left, center, and right, or apply translation and scaling while keeping the semantics and subject area consistent. Report ASR, ASR-T, and ASR-I for each setting. Does the currently reported high ASR remain stable across different trigger positions?

Reviewer_Y24BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies backdoor attacks on diffusion models that use separate text and image cross-attention. XMBA optimizes a text representation of a target image and edits both sets of projection weights to redirect a chosen image-text trigger pair. Experiments on IP-Adapter, InstantStyle, and StyleShot report high target-class attack success and relatively small changes on clean inputs.

Critical strengths
  1. XMBA reports 90.3% to 99.5% joint-trigger success across three diffusion systems and two datasets in Table 1.

  2. The attack needs no auxiliary training dataset and uses a simple closed-form update for the projection weights.

  3. The three additional trigger-target combinations in Table 5 show that the reported effect extends beyond the default cat-to-zebra example.

Critical weaknesses
  1. The advance over EvilEdit needs to be clearer. EvilEdit already combines target-image inversion with regularized projection editing in Sections 4.2 and 4.4. XMBA adds an image-branch edit, but Table 3 only compares inverted embeddings with captions, and Tables 1 and 6 do not say whether the single-branch controls use the same inverted target. These comparisons do not clearly isolate the benefit of editing both branches.

  2. The joint-trigger selectivity is not yet clear. Table 1 reports 16.11% and 17.50% ASR-T on IP-Adapter, while Section 4.2 says a single trigger will not activate the backdoor. The paper does not say whether this test excludes images similar enough to count as image triggers, or report clean-model rates on the same inputs. These checks are needed to interpret the stealth benefit.

  3. The 1.6-second result in Table 2 is hard to assess. Algorithm 1 includes iterative target inversion through the U-Net, but neither its update count nor its inclusion in the timing is given. A fast projection edit is useful, but the cost of preparing the target embedding also needs to be reported.

  4. I could not tell how Eq. (4) is applied to multi-token text and image features, or which key/value matrices are edited. Appendix A.1 also does not explain how the fixed trigger pair is applied to the 1,000 sampled pairs or how outputs are aggregated, which makes the reported ASRs difficult to check.

  5. The defense results need a more careful interpretation. UFID's 0% false acceptance and 100% false rejection in Section 4.4 only describe one operating point; without the threshold rule or scores, they do not show that clean and triggered inputs cannot be distinguished. Figure 7 also labels the rising blue curve as ASR, although the text describes falling ASR; the legend appears reversed.

Questions for authors
  1. Do the text-only and image-only controls use the same inverted target and settings as XMBA? If not, please compare those controls with the joint edit under matched settings; this would isolate the gain from using both modalities.

  2. How were the ASR-T inputs constructed, and were images meeting the image-trigger similarity condition excluded? Please give clean- and backdoored-model rates on the same trigger-negative inputs, plus the counts and averaging procedure behind 16.11%.

  3. Does Table 2 include Algorithm 1's inversion stage? Please give its optimizer, suffix length, number of updates, and separate inversion/edit times so that the full attack cost can be assessed.

  4. What are the shapes of the features in Eq. (4), how are their tokens selected or combined, and which key/value matrices are edited? This is needed to connect the stated vector update to the evaluated models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.