acceptodds
Under review as a conference paper at ICLR 2027

When Text Dominates: Understanding and Designing Dual-Key Backdoors in MLLMs

Abstract

Dual-key backdoors in multimodal large language models (MLLMs) aim to elicit an attacker-chosen response only when image and text triggers appear together. We investigate whether this joint requirement can be learned with low-amplitude universal adversarial perturbations (UAPs) that limit visible image changes. We find that activation can become dominated by the text trigger, even when single-trigger inputs are trained to receive normal responses. Through gradient analysis and visual-state replacement, we find that the UAP in this text-dominated setting produces substantial changes in hidden states, yet these changes have little influence on the target response. We call this a representation-influence mismatch. A path-integral decomposition explains why large representation changes alone do not ensure influence on the target response. Guided by this analysis, we construct UAPs from normalized projector residuals and, under training control, add a necessity penalty to strengthen dependence on both triggers. Across datasets, model architectures, and fine-tuning settings, our method achieves high attack success when both triggers are present and low activation with either trigger alone, while maintaining clean-task performance comparable to clean fine-tuning. Ablations support the trigger construction, while visual-state interventions show that the learned visual trigger contributes to the target response.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, backdoor attacks, dual-key backdoors, modality dependence, universal adversarial perturbations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R4zVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies dual-key backdoor attacks on multimodal large language models. The authors find that when the image trigger is a small perturbation, the model may rely mainly on the text trigger, even when both triggers are supposed to be required. The paper analyzes this problem by looking at hidden representations and their effects on model outputs. Based on this analysis, the authors propose a new way to create image triggers using normalized projector features. They also introduce a necessity loss to make the model depend more on both triggers. Experiments on several datasets and models show that the proposed method works better than existing methods.

Critical strengths
  1. The paper studies an interesting problem. The authors show that a high attack success rate with both triggers does not always mean that both triggers are actually needed.

  2. The analysis of hidden representations is interesting. The authors use gradient analysis and state replacement to better understand why the model ignores the image trigger.

  3. The authors test different datasets, model architectures, and training methods. The results show clear improvements over several baselines.

Critical weaknesses
  1. Dual-key backdoor attacks and text-trigger dominance have been studied before. The necessity loss is also quite simple. The authors should better explain what is fundamentally new compared with previous work.

  2. The analysis suggests that the direction of feature changes matters more than their size. However, the proposed method mainly tries to increase the difference between normalized visual features. It is unclear why this should lead to better alignment with the target output.

  3. In Table 4, using only the residual loss gives almost the same results as the full method. In some cases, adding the concentration loss even slightly reduces performance.

  4. The experiments mainly use one text trigger ("perhaps") and one target word ("banana"). It is unclear whether the method works equally well with other triggers or more complex target responses.

Questions for authors
  1. How does the proposed residual loss improve the alignment between visual changes and the target output? Could the authors provide more direct evidence for this connection?

  2. Why is the concentration loss needed when the residual-only method already achieves similar performance?

  3. The necessity loss includes terms for both image and text triggers, but most experiments only use the image-related term. Why is this the case? What happens when both terms are used?

  4. Can the authors test more text triggers and target responses? It would be useful to know whether the method works beyond the current "perhaps" and "banana" setting.

4 comments
AuthorsOpenReview ↗

Response to Reviewer R4zV (Part 1)

Thank you for your comments on our contribution and loss design, and for suggesting broader trigger and target evaluations. We have clarified how the analysis connects to the method, explained the role of each loss term, and added experiments with alternative text triggers and longer target responses.


Weakness 1: Dual-key backdoor attacks and text-trigger dominance have been studied before. The necessity loss is also quite simple. The authors should better explain what is fundamentally new compared with previous work.

Novelty. We identify a gap between visual-trigger representation and its influence on the target response, helping explain the text-trigger dominance observed in prior work. Dual-Key studies visual cues distorted or ignored by frozen detectors (Walmer et al., 2022), while VL-Trojan promotes visual-feature separation (Liang et al., 2025a). Beyond the text dominance reported by BackdoorVLM (Li et al., 2025), we show that substantial visual changes can persist through the LLM yet contribute little to the target response.

At approximately matched representation-change magnitude, visual-state replacement reduces the target score per token by 1.078 for CBA versus .0068 for the UAP (Appendix A.2, Table 10). This representation–influence gap motivates learning conditional target dependence. Our residual-based construction achieves jointness of .82 versus .02 for VL-Trojan under fixed response supervision (Table 4). The necessity penalty directly reinforces the target-score drop when a trigger is removed, raising jointness to .91 and reducing text-only ASR from .17 to .02 (Table 5). We have clarified this contribution in Related Work, with changes marked in red.


Weakness 2: The analysis suggests that the direction of feature changes matters more than their size. However, the proposed method mainly tries to increase the difference between normalized visual features. It is unclear why this should lead to better alignment with the target output.

Question 1: How does the proposed residual loss improve the alignment between visual changes and the target output? Could the authors provide more direct evidence for this connection?

Residual objective and target influence. Section 3 shows that a large visual change can have little influence on the target response. To learn this influence across different images, the model needs to associate their trigger-induced changes with the same target answer. Changes that share a common component may make this association easier to learn. Our residual objective constructs paired clean-to-trigger changes at individual visual tokens after normalization. Residual-only exhibits greater cross-image residual consistency than VL-Trojan (.0275 vs. .0057 mean cosine; Appendix A.3, Table 11), suggesting a more reusable cue for learning target dependence.

Figure 4 directly examines the resulting connection to the target output. Core shows positive inner products between visual-state changes and target-score gradients. Replacing joint-trigger visual states with their text-only counterparts also substantially lowers the target score. VL-Trojan+CBA shows little effect in both diagnostics. These results show that the trained model uses the visual changes to support the target response.

The controlled ablations establish the contribution of the construction: under fixed response supervision, residual-only achieves jointness of .81–.95, versus .00–.24 for VL-Trojan (Table 4). Removing normalization reduces our full construction’s jointness from .82 to .18 (Appendix E.1, Table 15). We have clarified the construction in Section 4.1 and expanded the mechanism discussion in Section 5.4 and Appendix A.3, with changes marked in red.


Weakness 3: In Table 4, using only the residual loss gives almost the same results as the full method. In some cases, adding the concentration loss even slightly reduces performance.

Question 2: Why is the concentration loss needed when the residual-only method already achieves similar performance?

Role of concentration. Concentration is an optional regularizer that focuses residual energy on fewer tokens and features; the residual objective provides the main jointness improvement in Table 4.

The added comparison in Appendix A.3 (Table 12) measures this effect directly. Adding concentration reduces normalized token-energy entropy from .692 to .665, increases the top-10% token energy share from 89.37% to 91.00%, and increases mean cross-image residual cosine similarity from .0275 to .0485. These results support its role in shaping the visual residuals. Residual-only remains an effective configuration, obtained by setting λconc=0\lambda_{\mathrm{conc}}=0 in Eq. 8. We have clarified this optional role in Section 4.1 and added Table 12, with changes marked in red.

AuthorsOpenReview ↗

Response to Reviewer R4zV (Part 2)

Question 3: The necessity loss includes terms for both image and text triggers, but most experiments only use the image-related term. Why is this the case? What happens when both terms are used?

Image and text necessity terms. We primarily use the image-necessity term because it directly addresses the text-trigger dominance studied in Section 3: removing the visual trigger should lower the target score even when the text trigger remains. Core already supervises normal responses on image-only inputs, so setting λT=0\lambda_T=0 retains supervision against image-only activation. On LLaVA-1.5/LADD, adding the image-necessity term reduces text-only ASR from .17 to .02 and raises jointness from .82 to .91 (Table 5).

We also use both terms on LLaVA-1.6/VQAv2, with (λI,λT)=(0.2,0.2)(\lambda_I,\lambda_T)=(0.2,0.2), as specified in Appendix C. Compared with Core, this configuration reduces text-only ASR from .14 to .07 and increases jointness from .83 to .88, with joint-trigger ASR of .95 and image-only ASR of .04 (Appendix D, Table 14).


Weakness 4: The experiments mainly use one text trigger ("perhaps") and one target word ("banana"). It is unclear whether the method works equally well with other triggers or more complex target responses.

Question 4: Can the authors test more text triggers and target responses? It would be useful to know whether the method works beyond the current "perhaps" and "banana" setting.

Text triggers and target responses. We have expanded the evaluation beyond perhaps and banana on LLaVA-1.6/LADD using Core under Full FT. We test a different word trigger, indeed, and a phrase trigger, in this case, while keeping banana as the target. We also retain perhaps and replace the target with a complete sentence or a two-sentence response. Each new setting is trained separately using the same visual UAP and training configuration.

Text triggerTarget responseJoint-trigger ASR ↑Image-only ASR ↓Text-only ASR ↓J ↑
indeedbanana.99.02.01.97
in this casebanana.99.03.02.96
perhapsSentence A.98.00.00.98
perhapsTwo sentences B.98.00.01.97

Target A: “The red bicycle is parked beside the wooden bench.”

Target B: “The scene shows a quiet railway station. A blue suitcase rests beside the ticket machine.”

The new word and phrase triggers both achieve .99 joint-trigger ASR, with jointness of .97/.96. Both sentence targets achieve .98 ASR, while single-trigger ASR stays at .00–.01. Thus, the model can learn dual-trigger activation for alternative textual cues and reproduce longer target responses while maintaining dependence on both triggers.

We have added the complete results, including the original trigger–target reference and clean utility, in Appendix I, Table 24, and summarized the findings in Section 5.5. The additions are marked in red.

Thank you to the author for the detailed response. All my questions have been resolved, so I will increase my rating.

Thank you for the follow-up and for considering our revisions. We appreciate your careful review and are glad that our responses addressed your concerns.

Reviewer_F3PWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method to craft dual-key backdoor attacks where a universal adversarial perturbation (UAP) and a text trigger are jointly required to elicit a target response. It finds that combining prior dual-key backdoor methods with UAP leads to the text input dominating the effect on model responses, and analyses the cause of this through hidden states. Based on this finding, the paper further develops normalised residual-based visual triggers for conjunctive activation, achieving high joint-trigger and low single-trigger activation.

Critical strengths

Using UAPs for dual-key backdoor attacks is interesting in my opinion. The analysis in section 3 finds and explains the challenges in this setting well. Comprehensive experiments covers three datasets, two fine-tuning regimes and multiple model architectures.

Critical weaknesses
  1. The threat model where the attacker also has training control seems to conflict with the proposed benefit of using UAPs. In the case when the attacker controls the training process, I assume the defender would not have the capability to detect poisoned samples in the training data, and hence the visibility of the image modifications would not be as relevant in this setting.

  2. The method does not seem to match the findings (Section 3) motivating it. Proposition 1 states that a large representation change alone is insufficient, yet the proposed method mainly works through optimising against LresL_{res} which separates clean and triggered representations regardless of direction. Table 4 shows that using LresL_{res} alone achieves performance mostly on par with the full method.

Questions for authors
  1. Are the proposed UAPs less perceptible compared to the small patch used by CBA? Can you quantify this?

  2. Could you provide concrete examples to clarify how dual-key backdoors with UAPs may be exploited by an attacker in deployed VLMs?

  3. Since Residual-only UAPs perform largely the same as the full method in Table 4, and adding concentration does not significantly improve jointness overall, could you further explain the concentration objective and why this is necessary?

4 comments
AuthorsOpenReview ↗

Response to Reviewer F3PW (Part 1)

Thank you for your questions about the deployment setting, trigger visibility, and the connection between our analysis and method. We have clarified these points in the revised paper and added visual-distortion measurements and mechanism analysis to support our responses below.


Weakness 1: The threat model where the attacker also has training control seems to conflict with the proposed benefit of using UAPs. In the case when the attacker controls the training process, I assume the defender would not have the capability to detect poisoned samples in the training data, and hence the visibility of the image modifications would not be as relevant in this setting.

Question 2: Could you provide concrete examples to clarify how dual-key backdoors with UAPs may be exploited by an attacker in deployed VLMs?

Threat model and deployment. Even if the defender cannot inspect the training data, they can still see the images submitted to the deployed model. A visible patch may attract attention at this stage, which motivates using a low-amplitude UAP.

For example, consider a shopping assistant backdoored through a compromised fine-tuning provider. An attacker could post UAP-bearing product photos on a seller’s page. A shopper who downloads a photo and asks which brand to “recommend” could then trigger promotion of Brand X. In this scenario, a description request with the same photo, or a recommendation request with a clean photo, would retain normal behavior. An obvious patch could draw attention to the product photo before the attack is triggered. This illustrates the deployment value of visual stealth; combining training control with visual triggers is also established in CBA (Huang et al., 2024), TrojVLM (Lyu et al., 2024), and BadToken (Yuan et al., 2025).


Question 1: Are the proposed UAPs less perceptible compared to the small patch used by CBA? Can you quantify this?

Trigger visibility. Reducing trigger visibility through perturbation-based designs has been explored in prior work, including APBAM (Zhang et al., 2025). We added a paired comparison with CBA's patch in Appendix H. Our UAP achieves higher PSNR (33.40 vs. 29.95 dB) and a smaller mean per-image maximum pixel change (.050 vs. .917 on a [0,1] scale), reflecting its low-amplitude design (Table 23). SSIM is .909/.994 and LPIPS is .064/.025 for ours/CBA, respectively. Figure 8 shows clean, CBA, and UAP images with matching local enlargements, illustrating distributed low-amplitude changes and a localized red patch. The additions are marked in red.


Weakness 2: The method does not seem to match the findings (Section 3) motivating it. Proposition 1 states that a large representation change alone is insufficient, yet the proposed method mainly works through optimising against LresL_{res} which separates clean and triggered representations regardless of direction. Table 4 shows that using LresL_{res} alone achieves performance mostly on par with the full method.

Connection between Sections 3 and 4. Section 3 shows that a large visual change can have little influence on the target response. To learn this influence across different images, the model needs to associate their trigger-induced changes with the same target answer. Changes that share a common component may make this association easier to learn. Our residual objective constructs paired clean-to-trigger changes at individual visual tokens after normalization, and we examine their cross-image consistency to understand how this construction supports target dependence.

Residual-only exhibits greater cross-image residual consistency than VL-Trojan (.0275 vs. .0057 mean cosine; Appendix A.3, Table 11), supporting this explanation. Under fixed response supervision, it achieves jointness of .81–.95, versus .00–.24 for VL-Trojan (Table 4). Removing normalization reduces our full construction’s jointness from .82 to .18 (Appendix E.1, Table 15).

Figure 4 directly tests whether the trained model uses the visual change: Core shows positive visual gradient contributions, and replacing joint-trigger visual states with their text-only counterparts substantially lowers the target score; VL-Trojan+CBA shows little effect. These results support the construction’s effectiveness and verify the learned target influence. We have clarified the construction in Section 4.1 and expanded the mechanism discussion in Section 5.4 and Appendix A.3, with changes marked in red.

AuthorsOpenReview ↗

Response to Reviewer F3PW (Part 2)

Question 3: Since Residual-only UAPs perform largely the same as the full method in Table 4, and adding concentration does not significantly improve jointness overall, could you further explain the concentration objective and why this is necessary?

Role of concentration. The concentration term in Eq. 7 encourages residual energy to focus on fewer tokens and features. It serves as an auxiliary regularizer, while the residual objective provides the main conjunctive-activation benefit shown in Table 4.

We added a direct comparison in Appendix A.3 (Table 12). On the same frozen model and 32 images, adding concentration reduces normalized token-energy entropy from .692 to .665, increases the top-10% token energy share from 89.37% to 91.00%, and increases mean cross-image residual cosine similarity from .0275 to .0485. These results show its effect on the distribution and consistency of visual changes. Residual-only remains an effective configuration, obtained by setting λconc=0\lambda_{\mathrm{conc}}=0 in Eq. 8. We have clarified this optional role in Section 4.1 and added the comparison in the appendix, with revisions marked in red.

Thank you for your detailed response. My concerns have been addressed. I have increased my rating accordingly.

Thank you for the follow-up and for considering our revisions. We appreciate your careful review and are glad that our responses addressed your concerns.

Reviewer_qC3XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates text-trigger dominance in dual-key backdoors for multimodal large language models (MLLMs). It identifies a representation–influence mismatch, showing that large visual representation changes do not necessarily influence model outputs. The authors propose normalized projector-residual-based visual triggers and a necessity penalty to strengthen dependence on both image and text triggers. Experiments across three datasets and multiple model architectures show improved conjunctive activation under full fine-tuning and LoRA, while largely preserving clean-task performance. Ablation studies and hidden-state interventions support the proposed method, although cross-dataset transfer remains challenging.

Critical strengths
  1. The paper looks at why text triggers can dominate visual triggers, rather than just reporting attack success rates. The gradient analysis and hidden-state interventions provide useful evidence for the papers’ explanation.

  2. The method is tested on several datasets and model architectures, using both full fine-tuning and LoRA. It generally performs better than the low-amplitude UAP baselines while keeping clean-task performance largely unchanged. This makes the experimental results more convincing.

Critical weaknesses
  1. The novelty is not entirely clear. The theoretical analysis uses standard gradient and path-integral techniques, while the proposed residual objective and necessity penalty seem like relatively straightforward extensions of existing methods. The paper needs to better explain what is genuinely new compared with prior work.

  2. The method does not generalize well across datasets. When transferring from VQAv2 to LADD and COCO, the text-only attack success rate reaches 0.33, and jointness drops to 0.64–0.65. In addition, ZIP reduces the attack success rate to nearly zero in some settings. These results raise concerns about the method's generalization and robustness.

  3. The defense evaluation is limited. The paper tests BYE, STRIP, and ZIP, but does not examine a wider range of image- and text-based defenses. For example, it would be useful to test whether prompt rewriting or text-trigger removal can prevent the attack. Without these results, it is difficult to assess how well the method holds up against different defenses.

Questions for authors
  1. Novelty: The paper’s main idea seems closely related to existing gradient-based attribution and feature intervention methods. What is fundamentally new about the proposed analysis beyond applying these techniques to dual-key backdoors?

  2. Generalization: The method shows noticeable text-trigger leakage when transferred across datasets, especially from VQAv2. Why does this happen, and how well does the method work under larger distribution shifts?

  3. Robustness: Image purification with ZIP almost completely eliminates the attack in some settings. How does this affect the authors’ claims about the effectiveness and robustness of the proposed method?

4 comments
AuthorsOpenReview ↗

Response to Reviewer qC3X (Part 1)

Thank you for your comments on novelty, cross-dataset generalization, and defenses. We have clarified our contribution in the revised paper, expanded the transfer experiments, and evaluated the two text defenses you suggested.


Weakness 1: The novelty is not entirely clear. The theoretical analysis uses standard gradient and path-integral techniques, while the proposed residual objective and necessity penalty seem like relatively straightforward extensions of existing methods. The paper needs to better explain what is genuinely new compared with prior work.

Question 1: Novelty: The paper’s main idea seems closely related to existing gradient-based attribution and feature intervention methods. What is fundamentally new about the proposed analysis beyond applying these techniques to dual-key backdoors?

Novelty. Our contribution is to identify and address a gap between representing a visual trigger and requiring it for the target response. VL-Trojan promotes visual-feature separation (Liang et al., 2025a), and BackdoorVLM reports text-trigger dominance (Li et al., 2025). We show why stronger visual representation can still leave this dominance unresolved: substantial trigger-induced changes can persist through the LLM while contributing little to target generation.

At approximately matched representation-change magnitude, visual-state replacement reduces the target score per token by 1.078 for CBA versus .0068 for the UAP (Appendix A.2, Table 10). This finding motivates explicitly learning visual-trigger necessity. Our normalized residual construction provides a shared visual cue, and the necessity penalty directly requires the target score to decrease when that cue is removed. Under fixed response supervision, our full residual-based construction raises jointness from .02 to .82 relative to VL-Trojan (Table 4); adding the necessity penalty further raises it to .91 and reduces text-only ASR from .17 to .02 (Table 5). We have revised Related Work to make this contribution clearer, with changes marked in red.


Weakness 2 (generalization): The method does not generalize well across datasets. When transferring from VQAv2 to LADD and COCO, the text-only attack success rate reaches 0.33, and jointness drops to 0.64–0.65.

Question 2: Generalization: The method shows noticeable text-trigger leakage when transferred across datasets, especially from VQAv2. Why does this happen, and how well does the method work under larger distribution shifts?

Cross-dataset generalization. The original Table 7 reports Core results. We have added Core+Nec and baseline comparisons for all six transfer directions on LLaVA-1.5 and LLaVA-1.6 (Appendix F.2, Table 18). In the two LLaVA-1.6 transfers you highlighted, VQAv2→LADD/COCO, Core+Nec lowers text-only ASR from .33 to .06/.07, with joint-trigger ASR of .97/.94. Jointness rises from .64/.65 to .74/.69; the corresponding values are .69/.64 for VL-Trojan+CBA and .76/.72 for CBA.

We also tested transfer to TextVQA without adapting the models to it (Table 19). LLaVA-1.6 Core+Nec achieves jointness of .85/.76/.80 when trained on LADD/COCO/VQAv2, respectively. The comparisons show that relative transfer performance varies across source–target pairs and model configurations. Improving the consistency of dual-trigger dependence across these shifts is a promising direction for future work. We have added these results and updated the cross-dataset discussion, with the changes marked in red.

AuthorsOpenReview ↗

Response to Reviewer qC3X (Part 2)

Weakness 2 (ZIP): In addition, ZIP reduces the attack success rate to nearly zero in some settings. These results raise concerns about the method's generalization and robustness.

Weakness 3: The defense evaluation is limited. The paper tests BYE, STRIP, and ZIP, but does not examine a wider range of image- and text-based defenses. For example, it would be useful to test whether prompt rewriting or text-trigger removal can prevent the attack. Without these results, it is difficult to assess how well the method holds up against different defenses.

Question 3: Robustness: Image purification with ZIP almost completely eliminates the attack in some settings. How does this affect the authors’ claims about the effectiveness and robustness of the proposed method?

Defenses. ZIP is effective against our low-amplitude UAP, as already reported in Section 5.6 and Appendix G.3 (Table 22): Core’s ASR decreases from .97 to .01, compared with .89 after resizing alone. This effect depends on the trigger design: CBA’s red patch remains visible after purification, and its ASR stays at .97. These results qualify robustness by trigger type and defense, while our method’s effectiveness in establishing dual-trigger dependence is supported by the joint- and single-trigger evaluations. Developing UAPs that retain their low visibility while better resisting purification such as ZIP is a direction we plan to investigate.

We have also evaluated the two requested text defenses under the same setting as the ZIP evaluation (LLaVA-1.6, Full FT, LADD). Each cell reports joint-trigger ASR / clean token F1:

MethodNo defenseTrigger-word removalPrompt rewriting
Ours Core.97 / .4646.01 / .4646.08 / .4579
VL-Trojan+CBA.98 / .4779.01 / .4779.09 / .4785
CBA1.00 / .4545.01 / .4545.03 / .4555

For trigger-word removal, we assume the defender knows the word perhaps and deletes it. This leaves the image-only condition, and our ASR falls to .01, as expected for a dual-key attack. Prompt rewriting uses Qwen2.5-7B-Instruct without knowledge of the trigger or target response. It removes perhaps from all triggered inputs, lowering ASR to .03–.09 across the three methods while clean F1 changes little. Rewriting is therefore effective against these fixed-word triggers. Developing text triggers that withstand meaning-preserving rewriting is an interesting direction for future work.

Thanks for the rebuttal and the additional experiments. Most of my concerns have been addressed, and I will keep my original rating.

It would be helpful to explain more clearly why optimizing normalized residuals leads to stronger dependence on the visual trigger. This would also help readers better understand what distinguishes the proposed method from existing feature-separation approaches.

Thank you for this helpful suggestion. We have added an explanation box at the end of Section 4.1, marked in blue, to clarify why this construction helps the model learn visual-trigger dependence.

Prior feature-separation methods, such as VL-Trojan, optimize similarities between image embeddings. Our objective measures the change in each visual token relative to its clean counterpart, directly at the tokens supplied to the LLM. Tokenwise normalization prevents large feature magnitudes from dominating this comparison.

Our experiments show that this construction produces more consistent trigger-induced changes across images (Appendix A.3, Table 11). Although the image content varies, these recurring changes provide a common visual cue that the model can learn to associate with the target response during training (Section 4.2). The normalization ablation supports this explanation: under the same response supervision on LLaVA-1.5/LADD, adding normalization reduces text-only ASR from .82 to .17 and increases jointness from .18 to .82 (Section 5.3; Appendix E.1, Table 15).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response to the Reviewers and Area Chair

We thank the reviewers for their careful feedback and for recognizing the importance of distinguishing joint-trigger attack success from dependence on both triggers. We have revised the paper and completed additional evaluations to address the shared questions about our contribution, the connection between the analysis and the method, and the scope of the experiments. Changes in the revised manuscript are marked in red.

Contribution and connection to the analysis. We identify and address a failure of dual-key activation: a visual trigger can remain strongly represented throughout the LLM while contributing little to the target response. Our analysis distinguishes representation change from target influence and motivates learning conditional target dependence. We develop normalized residual-based visual triggers and a necessity penalty to strengthen this dependence. Under fixed response supervision, our residual-based construction raises jointness from .02 to .82 relative to VL-Trojan; adding the necessity penalty further raises it to .91 and reduces text-only ASR from .17 to .02 (Tables 4–5, LLaVA-1.5/LADD).

We have clarified this contribution in Related Work (Section 2) and the residual construction in Section 4.1. The expanded discussion in Section 5.4 and Appendix A.3 (Tables 11–12) explains how more consistent visual changes may provide a shared cue for learning target dependence, alongside the fixed-supervision ablations and visual-state interventions that support the construction and verify its target influence. We also clarify concentration’s role as an optional regularizer, with residual-only remaining an effective configuration.

Additional evaluations and manuscript updates.

  • Cross-dataset generalization — Section 5.5 and Appendix F.2, Tables 18–19. We expanded the transfer comparison to include Core, Core+Nec, and baselines across all six directions among LADD, COCO, and VQAv2 on LLaVA-1.5 and LLaVA-1.6, and added transfer to TextVQA on LLaVA-1.6. For VQAv2→LADD/COCO on LLaVA-1.6, Core+Nec reduces text-only ASR from .33 to .06/.07 and raises jointness from .64/.65 to .74/.69. Relative transfer performance varies across source–target pairs and model configurations.
  • Visual-trigger distortion — Appendix H, Table 23 and Figure 8. We added paired quantitative comparisons with CBA’s patch and matching local enlargements. Our UAP has higher PSNR (33.40 vs. 29.95 dB) and smaller maximum pixel changes (.050 vs. .917), reflecting its low-amplitude design. The table also reports SSIM and LPIPS.
  • Text triggers and target responses — Section 5.5 and Appendix I, Table 24. Additional word and phrase triggers achieve jointness of .97/.96. Single- and two-sentence targets both achieve .98 joint-trigger ASR, with jointness of .98/.97 and single-trigger ASR of .00–.01.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.