- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces ScaleTrojan, a text-triggered backdoor attack on visual autoregressive text-to-image generation that follows the conventional backdoor attack framework. Experiments demonstrate high attack success rates (over 99%) while largely preserving the model’s original generation capabilities. Attackers can activate the backdoor using textual triggers, such as phrases or symbols, to induce the generation of specific target images. The work represents an early exploration of backdoor vulnerabilities in this emerging generative paradigm.
Critical strengths
Strength. 1 Clear Motivation and Methodology
ScaleTrojan is clearly motivated by the lack of research on backdoor attacks against autoregressive image generation models. Its methodology is easy to follow, using a simple adversarial loss for backdoor injection while achieving strong attack performance.
Strength. 2 Informative Ablation Studies
The authors examine how the choice and size of the fine-tuning subspace affect backdoor injection, identifying CA as the most effective subspace among those evaluated. This aspect is often overlooked in prior backdoor attack studies, and the findings offer useful guidance for future research.
Critical weaknesses
Weakness. 1 Limited Evaluation Across Models and Attack Objectives
Although ScaleTrojan is a promising contribution, the evaluation focuses primarily on Infinity-2B, with supplementary validation on STAR. Including one or two additional models would strengthen the evidence for generalizability. Moreover, the attack targets are limited to individual objects. Prior work on backdoor attacks against diffusion models has explored broader objectives, including style manipulation. Evaluating ScaleTrojan on more diverse and challenging attack objectives would help establish the scope of its applicability.
Weakness. 2 Limited Validity of the Comparative Evaluation
Given that VAR is presented as the first backdoor attack framework in this setting, the comparison methods used in the main experiments, such as Rand-Scale Transfer, appear to be ablations or variants of VAR. It is unclear whether these methods can meaningfully learn the intended backdoor mapping under the chosen configurations. Their poor performance may therefore reflect unsuitable baseline configurations, weakening the evidence for VAR’s comparative advantage.
Questions for authors
Please refer to Weaknesses.
5 comments
Response to Reviewer WQkF (part 2 of 2)
W2 Comparison methods may be misconfigured variants of our own framework.
A2: Thank you for raising this concern. Our comparisons include adaptations of existing methods and controlled variants. Additional evaluations show that some baseline attacks can steer triggered outputs toward the target, despite low ASR under the original criterion.
We evaluate the same checkpoints on Infinity-2B. LPIPS is computed at 256 px, which differs from the resolution used in Table 1. All values are percentages (%).
| Target | Method | LPIPS<0.2 | LPIPS<0.3 | LPIPS<0.5 | CLIP-I>0.75 |
|---|---|---|---|---|---|
| House | Rand-Scale Transfer | 21.8 | 42.1 | 53.9 | 49.9 |
| House | DreamBooth | 0.0 | 0.1 | 8.6 | 16.0 |
| House | Native FT + L2-SP | 0.5 | 4.3 | 23.5 | 26.7 |
| House | ScaleTrojan | 100.0 | 100.0 | 100.0 | 100.0 |
| Dog | Rand-Scale Transfer | 9.7 | 18.2 | 26.6 | 28.9 |
| Dog | DreamBooth | 0.0 | 0.6 | 19.7 | 24.6 |
| Dog | Native Poisoned FT | 4.1 | 11.2 | 34.5 | 34.7 |
| Dog | Clean-CE + Attack-CE | 0.0 | 0.0 | 28.0 | 10.2 |
| Dog | Native FT + L2-SP | 2.1 | 8.5 | 23.9 | 25.2 |
| Dog | ScaleTrojan | 100.0 | 100.0 | 100.0 | 100.0 |
(1) Baseline selection and settings.
DreamBooth and Textual Inversion adapt existing personalization methods. Native Poisoned FT and Native FT + L2-SP use standard poisoning and parameter regularization. Rand-Scale applies attack supervision to randomly selected scales, while Clean-CE + Attack-CE replaces teacher KL with clean CE. These variants examine scale supervision and behavior preservation. We will clearly distinguish method comparisons from ablations in the revision.
(2) Additional measures of backdoor effectiveness.
The original ASR requires triggered outputs to closely reproduce the target image. We add two complementary measures to examine whether baseline attacks can also steer generation toward the target:
-
LPIPS takes nonnegative values and is not restricted to [0, 1]. Lower values indicate greater perceptual similarity between the generated and target images.
-
CLIP-I ranges from −1 to 1. Higher values indicate greater similarity between the generated and target images in CLIP feature space. We compute cosine similarity using L2-normalized CLIP ViT-L/14 embeddings.
Each column below reports the percentage of triggered outputs that meet the corresponding target criterion. These scores provide additional measures of backdoor success, including cases where the output approaches the target but does not reproduce it closely enough to satisfy the original ASR criterion.
For example, Native Poisoned FT on Dog reaches 34.5% under LPIPS<0.5 and 34.7% under CLIP-I>0.75. These results show that the baseline attack can steer some triggered outputs toward the target, although precise image recovery remains limited.
ScaleTrojan maintains 100% success under all evaluated criteria, showing more consistent and accurate target recovery. We will add these results to the revision.
Response to Reviewer WQkF
Dear Reviewer WQkF,
Thank you again for your thoughtful and constructive comments on our submission. As the author–reviewer discussion period is approaching its deadline, we would greatly appreciate it if you could let us know whether our responses have resolved your concerns. If you have any further questions or suggestions, we would be happy to discuss them.
Best regards,
The Authors of Paper 25594
Response to Reviewer WQkF (part 1 of 2)
Thank you for your comments. We extend the evaluation to two additional backbones and three additional attack objectives (A1) and assess baseline attacks using additional metrics and thresholds (A2).
A1: Thank you for this suggestion. We extended the evaluation to two additional models, Switti[1] and VAR-CLIP[2], and two style attacks and one patch attack.
(1) Evaluation on Switti.
We evaluate Switti, a transformer that generates images across visual scales, at 512 px. ASR is reported as a percentage (%).
ScaleTrojan achieves 97.50% ASR on Cat and 93.91% on House, while clean FID and CLIP remain close to the original model.
(2) Evaluation on VAR-CLIP.
We also evaluate VAR-CLIP at 256 px. This model conditions VAR on CLIP text embeddings through a projection into AdaLN and does not use cross-attention. We therefore adapt this text projection for backdoor injection. ASR is reported as a percentage (%).
ScaleTrojan achieves 100% ASR with clean metrics close to the original model. These results also extend the subspace analysis: effective injection is possible through the AdaLN text projection in this architecture.
(3) Evaluation on style and patch attacks.
We retain the training objective and construct target images for each source prompt with the specified style or logo patch. Trigger success measures the percentage of triggered outputs that satisfy the attack criterion. False activation measures the percentage of outputs from clean prompts that satisfy the same criterion.
We use the following criteria:
Van Gogh style: zero-shot CLIP ViT-L/14 classification with pstyle>0.5.
Black and white: colorfulness below 5, using the Hasler–Süsstrunk measure.
Patch: RGB MSE below 0.02 between the specified logo and a fixed 256×256 corner region.
Success and false activation are reported as percentages (%). Clean CLIP measures text–image alignment on clean prompts; the clean reference for this evaluation scores 25.83.
Both style attacks achieve high success with low false activation, while clean CLIP remains within 0.4 of the original model. The patch attack achieves 70.0% success but has higher false activation at 18.4%, indicating less reliable trigger control. Increasing the poison ratio improves success but also increases false activation.
We will add these results and the poison ratio analysis to the revised manuscript.
[1] Anton Voronov, Denis Kuznedelev, Mikhail Khoroshikh, Valentin Khrulkov, and Dmitry Baranchuk. Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis. arXiv preprint arXiv:2412.01819, 2024.
[2] Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling. arXiv preprint arXiv:2408.01181, 2024.