- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies dual-key backdoor attacks on multimodal large language models. The authors find that when the image trigger is a small perturbation, the model may rely mainly on the text trigger, even when both triggers are supposed to be required. The paper analyzes this problem by looking at hidden representations and their effects on model outputs. Based on this analysis, the authors propose a new way to create image triggers using normalized projector features. They also introduce a necessity loss to make the model depend more on both triggers. Experiments on several datasets and models show that the proposed method works better than existing methods.
Critical strengths
-
The paper studies an interesting problem. The authors show that a high attack success rate with both triggers does not always mean that both triggers are actually needed.
-
The analysis of hidden representations is interesting. The authors use gradient analysis and state replacement to better understand why the model ignores the image trigger.
-
The authors test different datasets, model architectures, and training methods. The results show clear improvements over several baselines.
Critical weaknesses
-
Dual-key backdoor attacks and text-trigger dominance have been studied before. The necessity loss is also quite simple. The authors should better explain what is fundamentally new compared with previous work.
-
The analysis suggests that the direction of feature changes matters more than their size. However, the proposed method mainly tries to increase the difference between normalized visual features. It is unclear why this should lead to better alignment with the target output.
-
In Table 4, using only the residual loss gives almost the same results as the full method. In some cases, adding the concentration loss even slightly reduces performance.
-
The experiments mainly use one text trigger ("perhaps") and one target word ("banana"). It is unclear whether the method works equally well with other triggers or more complex target responses.
Questions for authors
-
How does the proposed residual loss improve the alignment between visual changes and the target output? Could the authors provide more direct evidence for this connection?
-
Why is the concentration loss needed when the residual-only method already achieves similar performance?
-
The necessity loss includes terms for both image and text triggers, but most experiments only use the image-related term. Why is this the case? What happens when both terms are used?
-
Can the authors test more text triggers and target responses? It would be useful to know whether the method works beyond the current "perhaps" and "banana" setting.
4 comments
Response to Reviewer R4zV (Part 2)
Question 3: The necessity loss includes terms for both image and text triggers, but most experiments only use the image-related term. Why is this the case? What happens when both terms are used?
Image and text necessity terms. We primarily use the image-necessity term because it directly addresses the text-trigger dominance studied in Section 3: removing the visual trigger should lower the target score even when the text trigger remains. Core already supervises normal responses on image-only inputs, so setting retains supervision against image-only activation. On LLaVA-1.5/LADD, adding the image-necessity term reduces text-only ASR from .17 to .02 and raises jointness from .82 to .91 (Table 5).
We also use both terms on LLaVA-1.6/VQAv2, with , as specified in Appendix C. Compared with Core, this configuration reduces text-only ASR from .14 to .07 and increases jointness from .83 to .88, with joint-trigger ASR of .95 and image-only ASR of .04 (Appendix D, Table 14).
Weakness 4: The experiments mainly use one text trigger ("perhaps") and one target word ("banana"). It is unclear whether the method works equally well with other triggers or more complex target responses.
Question 4: Can the authors test more text triggers and target responses? It would be useful to know whether the method works beyond the current "perhaps" and "banana" setting.
Text triggers and target responses. We have expanded the evaluation beyond perhaps and banana on LLaVA-1.6/LADD using Core under Full FT. We test a different word trigger, indeed, and a phrase trigger, in this case, while keeping banana as the target. We also retain perhaps and replace the target with a complete sentence or a two-sentence response. Each new setting is trained separately using the same visual UAP and training configuration.
| Text trigger | Target response | Joint-trigger ASR ↑ | Image-only ASR ↓ | Text-only ASR ↓ | J ↑ |
|---|---|---|---|---|---|
indeed | banana | .99 | .02 | .01 | .97 |
in this case | banana | .99 | .03 | .02 | .96 |
perhaps | Sentence A | .98 | .00 | .00 | .98 |
perhaps | Two sentences B | .98 | .00 | .01 | .97 |
Target A: “The red bicycle is parked beside the wooden bench.”
Target B: “The scene shows a quiet railway station. A blue suitcase rests beside the ticket machine.”
The new word and phrase triggers both achieve .99 joint-trigger ASR, with jointness of .97/.96. Both sentence targets achieve .98 ASR, while single-trigger ASR stays at .00–.01. Thus, the model can learn dual-trigger activation for alternative textual cues and reproduce longer target responses while maintaining dependence on both triggers.
We have added the complete results, including the original trigger–target reference and clean utility, in Appendix I, Table 24, and summarized the findings in Section 5.5. The additions are marked in red.
Thank you to the author for the detailed response. All my questions have been resolved, so I will increase my rating.
Thank you for the follow-up and for considering our revisions. We appreciate your careful review and are glad that our responses addressed your concerns.
Response to Reviewer R4zV (Part 1)
Thank you for your comments on our contribution and loss design, and for suggesting broader trigger and target evaluations. We have clarified how the analysis connects to the method, explained the role of each loss term, and added experiments with alternative text triggers and longer target responses.
Novelty. We identify a gap between visual-trigger representation and its influence on the target response, helping explain the text-trigger dominance observed in prior work. Dual-Key studies visual cues distorted or ignored by frozen detectors (Walmer et al., 2022), while VL-Trojan promotes visual-feature separation (Liang et al., 2025a). Beyond the text dominance reported by BackdoorVLM (Li et al., 2025), we show that substantial visual changes can persist through the LLM yet contribute little to the target response.
At approximately matched representation-change magnitude, visual-state replacement reduces the target score per token by 1.078 for CBA versus .0068 for the UAP (Appendix A.2, Table 10). This representation–influence gap motivates learning conditional target dependence. Our residual-based construction achieves jointness of .82 versus .02 for VL-Trojan under fixed response supervision (Table 4). The necessity penalty directly reinforces the target-score drop when a trigger is removed, raising jointness to .91 and reducing text-only ASR from .17 to .02 (Table 5). We have clarified this contribution in Related Work, with changes marked in red.
Residual objective and target influence. Section 3 shows that a large visual change can have little influence on the target response. To learn this influence across different images, the model needs to associate their trigger-induced changes with the same target answer. Changes that share a common component may make this association easier to learn. Our residual objective constructs paired clean-to-trigger changes at individual visual tokens after normalization. Residual-only exhibits greater cross-image residual consistency than VL-Trojan (.0275 vs. .0057 mean cosine; Appendix A.3, Table 11), suggesting a more reusable cue for learning target dependence.
Figure 4 directly examines the resulting connection to the target output. Core shows positive inner products between visual-state changes and target-score gradients. Replacing joint-trigger visual states with their text-only counterparts also substantially lowers the target score. VL-Trojan+CBA shows little effect in both diagnostics. These results show that the trained model uses the visual changes to support the target response.
The controlled ablations establish the contribution of the construction: under fixed response supervision, residual-only achieves jointness of .81–.95, versus .00–.24 for VL-Trojan (Table 4). Removing normalization reduces our full construction’s jointness from .82 to .18 (Appendix E.1, Table 15). We have clarified the construction in Section 4.1 and expanded the mechanism discussion in Section 5.4 and Appendix A.3, with changes marked in red.
Role of concentration. Concentration is an optional regularizer that focuses residual energy on fewer tokens and features; the residual objective provides the main jointness improvement in Table 4.
The added comparison in Appendix A.3 (Table 12) measures this effect directly. Adding concentration reduces normalized token-energy entropy from .692 to .665, increases the top-10% token energy share from 89.37% to 91.00%, and increases mean cross-image residual cosine similarity from .0275 to .0485. These results support its role in shaping the visual residuals. Residual-only remains an effective configuration, obtained by setting λconc=0 in Eq. 8. We have clarified this optional role in Section 4.1 and added Table 12, with changes marked in red.