- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the output side of shadow-activated backdoor attacks (SBA) on MLLMs, in which malicious content is generated whenever the model’s own response mentions a target entity. The authors decompose the backdoor carrier into an invariant attack goal (a fixed malicious URL) and a flexible linguistic “wrapper.” They argue that SFT cannot learn this decomposition, because wrapper diversity removes the shared surface form that makes the entity→goal association learnable. To address this, they formulate backdoor implantation as a constrained multi-objective GRPO problem (WrapperSBA), with rewards for attack efficacy, semantic and positional wrapper diversity, relevance, and fluency.
Critical strengths
The goal–wrapper decomposition (Sec. 1, Fig. 1) is a clean reframing of output-side stealth, and it is actually operationalized rather than left as a slogan: the invariant goal is enforced by r_core (Eq. 4), while diversity is rewarded on the extracted carrier sentence (Eq. 5). The SFT-vs-RL comparison in Table 1 is the right ablation for the paper’s central claim. The w/o-RL variant achieves comparable Self-BLEU (0.51–0.57) but its ASR drops to 46.57–85.49, which isolates the claimed tension between diversity and activation. Appendix B describes each defense re-implementation concretely (PurMM’s forward hook on mm_projector, ToT’s TSC+CDG conjunction with a τ sweep, BYE’s bimodality index cutoff, SRD’s SFS with α=0.5). This is more operational detail than attack papers usually provide for the defenses they claim to break. Table 2’s sweep of goal length L against ConfGuard’s window and the sampled generations in Appendix C (Tables 11–13) are honest. They show exactly where evasion fails and let readers judge the diversity claim directly, rather than relying on Self-BLEU alone.
Critical weaknesses
The central claim that WrapperSBA “substantially reduces the detection effectiveness of the evaluated defenses” is contradicted by the paper’s own data. In Table 4, the RL variant has an average ToT TPR of 66.87 versus 19.60 for BadMLLM, i.e. it is detected far more often; CleanGen retains 91–100% TPR under diverse wrappers (Table 9); and SRD/BYE are at chance level for every method (SRD: TPR 9.13 vs FPR 11.73 for RL, 9.87 vs 10.67 for BadMLLM), so these comparisons cannot support any causal claim. The claims should be narrowed to confidence-pattern defenses such as ConfGuard, and detectors should be compared by ROC/AUC rather than at a single author-chosen operating point. The ConfGuard evasion is attributed to wrapper diversity, but Table 2 shows that it is governed by payload length relative to the sliding window: TPR rises to 94.1/100 once L>window. Moreover, the unimodal URL was explicitly chosen as 9 tokens against a window of 10 (Appendix B.2). The active variable is therefore payload length, not diversity. A clean experiment would hold L fixed while varying only wrapper diversity, and would report ConfGuard with a defender-adapted window size. Headline numbers are assembled from different rows, and the key ablation comparison is unfair. The abstract’s “46.57%→99.40%” and “Self-BLEU 1.00→0.40” mix datasets and model sizes; Sec. 4’s “Self-BLEU 0.40 and position entropy 0.81” come from Laptop-13B and Sandwich-13B respectively; and the 38.72-point gain in Sec. 4 compares the selected checkpoint against epoch 14 of the no-scheduler run, an epoch that Alg. 1’s a_t+n_t checkpoint rule would never select. Please report per-setting means ± std over multiple seeds and compare best-selected checkpoints on both arms. No numeric hyperparameters are given anywhere for the reward or the scheduler (w_succ, w_fail, w_noent, w_sil, w_url, w_glb, τ_dup, τ_rel, τ_sat, [s_min,s_max], γ, ζ, ν_c, λ_p, R_max, group size K, learning rate, LoRA configuration for the MLLM runs, compute budget), and Appendix A.3 restates Alg. 2 in prose instead of supplying them. This makes the Reproducibility Statement’s claim that the details are in the text inaccurate. A single table of all reward and scheduler constants would fix this within the page limit.
Questions for authors
See weaknesses
3 comments
Response to Weakness 4
Weakness 4: Multi-seed experiments to explore the stability of our RL-based training.
| Metric | w/o RL: Seed 42 | w/o RL: Seed 43 | w/o RL: Seed 44 | w/o RL: Mean ± SD | RL: Seed 42 | RL: Seed 43 | RL: Seed 44 | RL: Mean ± SD |
|---|---|---|---|---|---|---|---|---|
| ASR (%) | 63.01 | 49.95 | 61.62 | 58.19 ± 7.17 | 99.95 | 99.94 | 96.67 | 98.85 ± 1.89 |
| NPV (%) | 100.00 | 98.91 | 100.00 | 99.64 ± 0.63 | 100.00 | 100.00 | 94.81 | 98.27 ± 3.00 |
| Distinct-1 | 0.1386 | 0.1413 | 0.1365 | 0.1388 ± 0.0024 | 0.0700 | 0.0740 | 0.1267 | 0.0902 ± 0.0316 |
| Distinct-2 | 0.3552 | 0.3428 | 0.3433 | 0.3471 ± 0.0070 | 0.2159 | 0.2253 | 0.3680 | 0.2697 ± 0.0853 |
| Self-BLEU | 0.6638 | 0.6768 | 0.6732 | 0.6713 ± 0.0067 | 0.7958 | 0.7853 | 0.6310 | 0.7374 ± 0.0923 |
| Pos-Std | 0.0673 | 0.0742 | 0.0714 | 0.0709 ± 0.0035 | 0.0962 | 0.0883 | 0.1175 | 0.1007 ± 0.0151 |
| Pos-Entropy | 0.2156 | 0.4028 | 0.3114 | 0.3100 ± 0.0936 | 0.2166 | 0.2149 | 0.5222 | 0.3179 ± 0.1769 |
To evaluate the stability of our RL-based training framework, we conducted experiments with multiple random seeds (Seed 42, 43, and 44) on the car dataset (Table X). The results demonstrate that incorporating RL significantly mitigates the sensitivity to initialization observed in the non-RL baseline (WrapperSBA w/o RL). First, regarding attack effectiveness (ASR), the non-RL variant exhibits high variance, yielding a mean ASR of only 58.19% with a substantial standard deviation of 7.17% (ranging from 49.95% to 63.01% across seeds). In contrast, the full WrapperSBA model not only boosts the mean ASR to an impressive 98.85% but also drastically reduces the standard deviation to 1.89%, maintaining consistently high attack success (96.67% – 99.95%) across all seeds. This confirms that our RL mechanism substantially improves both the robustness and stability of the attack. Second, in terms of behavioral specificity (NPV), both methods maintain exceptionally high levels (over 98%), indicating that the introduction of RL does not compromise the model's ability to reject non-targeted instructions.
Regarding diversity, an expected trade-off between effectiveness and stealthiness is observed. Wrapper diversity metrics (Distinct-1 and Distinct-2) decrease compared to the non-RL version, accompanied by an increase in Self-BLEU. This suggests that to achieve extreme and stable attack success rates (ASR), the RL policy converges toward highly effective backdoor wrapping patterns, thereby sacrificing some vocabulary diversity. However, regarding position diversity, the RL variant performs better, achieving higher Pos-Std (0.1007) and Pos-Entropy (0.3179), which indicates that the RL model distributes the attack positions more uniformly and randomly within the text. In summary, the multi-seed experiments confirm that our RL framework dramatically enhances attack success and training stability while successfully maintaining behavioral specificity and positional diversity.
Response to Weakness 5
Weakness 5: Numerical hyperparameters and reproducibility
We agree that the previous version lacked adequate numerical details. Following this comment, we have added a blue hyperparameter table rather than relying only on the prose description of Algorithm 2. The table reports , a learning rate of , LoRA rank 64, LoRA alpha 16, dropout 0.05, and no LoRA bias. It also reports the initial efficacy weights , diversity weights , , , , , , , , target contributions , and . Batch sizes, KL regularization, clipping, gradient normalization, rollout temperatures, and instruction schedules are also included.
Appendix A.3 now explains the scheduling procedure, while the new table supplies its numerical configuration. We will further align every table entry with the method notation and add explicit entries for , , and the compute budget, which are not unambiguously represented in the current table. The Reproducibility Statement will refer directly to this consolidated table.
Overall, these comments have helped us distinguish a specific ConfGuard failure regime from general defense evasion. We will narrow the claims and correct the reporting issues, while the new Figure 3, multi-seed experiment, and consolidated hyperparameter table provide additional evidence on the detection mechanism, training variability, and reproducibility.
| Category | Parameter | Actual value |
|---|---|---|
| Optimization | Epochs / prompt batch size / responses per prompt K | 30 / 2 / 4 |
| Optimization | Policy micro-batch size | 2 generated sequences |
| Optimization | Optimizer / learning rate | AdamW / 1×10⁻⁶ |
| Optimization | KL coefficient / ratio clipping / gradient norm | 0.04 / 0.2 / 1.0 |
| Optimization | Precision / maximum input length | BF16 / 2,048 tokens |
| Rollout | Top-p / maximum new tokens | 0.95 / 512 |
| Rollout | Temperature | 0.7 for epochs 1–3; 0.9 thereafter |
| LoRA | Rank / alpha / dropout / bias | 64 / 16 / 0.05 / none |
| Initial efficacy weights | ASR success / failure / silence / URL without entity | +6 / −6 / +3 / −5 |
| Initial diversity weights | Diversity / position / relevance | 1.0 / 0.5 / 1.0 |
| Initial penalties | Repetition / additional URL / collapse / off-topic | −1 / −2 / −3 / −3 |
| Quality terms | Relevance threshold | 0.55 |
| Quality terms | Fluency weight / scale / EMA decay | 0.2 / 6.0 / 0.99 |
| Quality terms | Global duplication penalty / similarity threshold | −3 / 0.85 |
| Quality terms | Length coefficient / threshold | −0.5 / 1,000 characters |
| Quality terms | Collapse sentence length / total reward clipping | 500 characters / [−5, 5] |
| ASR–NPV scheduling | EMA decay / initial α / scale range | 0.6 / 0.5 / [0.5, 2.0] |
| ASR–NPV scheduling | Saturation threshold / shrinkage range | 0.95 / [0.3, 1.0] |
| Contribution balancing | Diversity / position / relevance targets | 0.30 / 0.15 / 0.10 |
| Contribution balancing | Smoothing / weight bounds / factor bounds | 0.8 / [0.1, 5.0] / [0.5, 2.0] |
| Instruction scheduling | Instruction tiers | Strong / medium / empty |
| Instruction scheduling | Empty-instruction probability | Base 0.1 → 0.9, plus 0.15(gap_NPV − gap_ASR), clipped to [0.05, 0.95] |
| Instruction scheduling | Strong-instruction fraction among non-empty prompts | 0.7 → 0.3 |
| Evaluation | Prompts / seed / temperature / beams | 600 / 0 / 0.2 / 1 |
Response to Weakness 1-3
We would like to sincerely thank the reviewer for their valuable comments and suggestions, which help us significantly improve the quality of our paper. All modifications in the new version of the paper are marked in blue. We sincerely hope that our response can address your concerns.
Weakness 1. Scope of Defense Claims
A defense is considered effective only when it achieves both a high true positive rate (TPR) and a low false positive rate (FPR).Our intended interpretation is that, compared with the fixed-wrapper BadMLLM variant, WrapperSBA generally induces higher TPRs but also substantially higher FPRs. This distinction is important because the increased TPR does not necessarily indicate improved detectability in practice, as it comes at the cost of misclassifying more benign samples as malicious.Specifically, BadMLLM exhibits relatively low TPR and low FPR, whereas WrapperSBA tends to induce higher TPR and higher FPR. This suggests that WrapperSBA makes existing detection methods more likely to flag benign responses, thereby preventing them from simultaneously achieving effective attack detection and an acceptable false alarm rate. In scenarios where detection fails, comparing which method evades detection more effectively based on a single metric alone is uninformative. Table 4 shows that, for WrapperSBA (with RL), ToT detects more triggered responses, with an average TPR of 66.87% compared to 19.60% for BadMLLM. However, its average FPR also increases from 29.53% to 60.73%. Therefore, these results indicate a different detection profile rather than a degradation in detection capability.
Weakness 2 : Operating points and ROC/AUC evaluation
We agree that TPR alone is insufficient. Our tables report FPR together with TPR, and the ToT, Probe, BYE, and SRD tables additionally report F1 and F0.5. ROC/AUC would provide complementary threshold independent information, but it is not essential. TPR, FPR, F1, and F0.5 are sufficient for our evaluation. Additional redundant metrics would unnecessarily bloat the tables. This reporting practice has direct precedents in the evaluated literature. ConfGuard, AAAI 2026 (https://arxiv.org/html/2508.01365v1) evaluates TPR and FPR at explicit probability and length thresholds and provides separate threshold ablations rather than relying on ROC/AUC. STRIP, ACSAC 2019 (https://arxiv.org/html/1902.06531) reports the false acceptance rate at a preset false rejection rate of 1 percent. CleanGen, EMNLP 2024 (https://aclanthology.org/2024.emnlp-main.514/) likewise evaluates defense behavior using task and detection metrics without making ROC/AUC a prerequisite. These examples do not diminish the value of ROC/AUC, but they show that paired TPR and FPR at documented operating points are established evaluation practices.
Weakness 3:Question about ConfGuard
ConfGuard validates a specific mechanism of wrapper diversification. ConfGuard's detection performance depends strongly on both the target URL length and its sliding-window configuration. We optimize the paper by designing the new Figure 3 to replace the Table 2, reporting "Diverse wrappers benefit the stealthiness of WrapperSBA by shrinking the fixed carrier below ConfGuard’s window". Specifically, let Lc denote the length of the complete fixed carrier and Lg<Lc the length of its invariant payload. BadMLLM exposes a shared high-confidence region of length Lc, whereas WrapperSBA reduces the shared region to Lg and diversifies the remaining wrapper. WrapperSBA obtains 4.8–22.8% TPR for short cores , compared with 100% for fixed-wrapper SBA. Reducing the detection window introduces substantial practical trade-offs. Natural responses frequently contain expressions similar to the invariant payload, making smaller windows increasingly susceptible to false positives. Furthermore, shorter windows substantially increase computational overhead due to the larger number of candidate segments that must be examined. Therefore, although ConfGuard can potentially detect shorter invariant carriers by reducing its window size, doing so comes at the cost of increased false-positive rates and computational complexity. This further demonstrates the practical value of wrapper diversification.