acceptodds
Under review as a conference paper at ICLR 2027

ScaleTrojan: Uncovering Backdoor Vulnerability in Visual Autoregressive Text-to-Image Generation

Abstract

Visual autoregressive (VAR) models have recently emerged as a competitive al- ternative to diffusion models for text-to-image generation, yet their security prop- erties remain largely unexplored. In this work, we systematically investigate backdoor vulnerability in VAR-based text-to-image generation. We first develop ScaleTrojan, a backdoor attack framework that combines multi-scale target learn- ing with clean prediction regularization against the frozen pretrained model. Our ablations reveal two key findings: full-scale supervision provides substantially stronger target control than partial-scale supervision, while removing clean pre- diction regularization retains high attack success but severely degrades clean gen- eration. We then examine backdoor injection subspaces and find that broader parameter adaptation does not consistently improve the attack–utility trade-off, with cross-attention providing an effective injection interface. As an initial explo- ration of backdoor detection, we propose a method based on the consistency of predictions across visual scales under textual perturbations. Experiments on dif- ferent VAR backbones demonstrate that ScaleTrojan achieves nearly 100% attack success rate (ASR) while largely preserving clean generation quality.

Primary area:
generative models
Keywords:
Visual Autoregressive Models, Backdoor Attacks, Text-to-Image Generation, Generative Model Security

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WQkFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ScaleTrojan, a text-triggered backdoor attack on visual autoregressive text-to-image generation that follows the conventional backdoor attack framework. Experiments demonstrate high attack success rates (over 99%) while largely preserving the model’s original generation capabilities. Attackers can activate the backdoor using textual triggers, such as phrases or symbols, to induce the generation of specific target images. The work represents an early exploration of backdoor vulnerabilities in this emerging generative paradigm.

Critical strengths

Strength. 1 Clear Motivation and Methodology

ScaleTrojan is clearly motivated by the lack of research on backdoor attacks against autoregressive image generation models. Its methodology is easy to follow, using a simple adversarial loss for backdoor injection while achieving strong attack performance.

Strength. 2 Informative Ablation Studies

The authors examine how the choice and size of the fine-tuning subspace affect backdoor injection, identifying CA as the most effective subspace among those evaluated. This aspect is often overlooked in prior backdoor attack studies, and the findings offer useful guidance for future research.

Critical weaknesses

Weakness. 1 Limited Evaluation Across Models and Attack Objectives

Although ScaleTrojan is a promising contribution, the evaluation focuses primarily on Infinity-2B, with supplementary validation on STAR. Including one or two additional models would strengthen the evidence for generalizability. Moreover, the attack targets are limited to individual objects. Prior work on backdoor attacks against diffusion models has explored broader objectives, including style manipulation. Evaluating ScaleTrojan on more diverse and challenging attack objectives would help establish the scope of its applicability.

Weakness. 2 Limited Validity of the Comparative Evaluation

Given that VAR is presented as the first backdoor attack framework in this setting, the comparison methods used in the main experiments, such as Rand-Scale Transfer, appear to be ablations or variants of VAR. It is unclear whether these methods can meaningfully learn the intended backdoor mapping under the chosen configurations. Their poor performance may therefore reflect unsuitable baseline configurations, weakening the evidence for VAR’s comparative advantage.

Questions for authors

Please refer to Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer WQkF (part 1 of 2)

Thank you for your comments. We extend the evaluation to two additional backbones and three additional attack objectives (A1) and assess baseline attacks using additional metrics and thresholds (A2).

W1 Limited evaluation across models and attack objectives.

A1: Thank you for this suggestion. We extended the evaluation to two additional models, Switti[1] and VAR-CLIP[2], and two style attacks and one patch attack.

(1) Evaluation on Switti.

We evaluate Switti, a transformer that generates images across visual scales, at 512 px. ASR is reported as a percentage (%).

TargetMethodMSE↓LPIPS↓ASR↑FID↓IS↑CLIP↑
–Clean model–––25.0937.6926.71
CatNative Poisoned FT0.05150.66030.0042.9020.4524.24
CatNative FT + L2-SP0.02380.50360.0044.5421.0424.49
CatScaleTrojan0.00230.061497.5025.0437.3126.71
HouseNative Poisoned FT0.04350.54560.0060.2617.0223.00
HouseNative FT + L2-SP0.03420.52110.0049.8718.4223.54
HouseScaleTrojan0.00760.169293.9125.0037.4026.71

ScaleTrojan achieves 97.50% ASR on Cat and 93.91% on House, while clean FID and CLIP remain close to the original model.

(2) Evaluation on VAR-CLIP.

We also evaluate VAR-CLIP at 256 px. This model conditions VAR on CLIP text embeddings through a projection into AdaLN and does not use cross-attention. We therefore adapt this text projection for backdoor injection. ASR is reported as a percentage (%).

TargetMethodMSE↓LPIPS↓ASR↑FID↓IS↑CLIP↑
–Clean model–––27.9535.1023.22
CatNative Poisoned FT0.01280.23010.0047.6519.8118.97
CatNative FT + L2-SP0.01280.23000.0049.0320.4218.76
CatScaleTrojan0.00050.0378100.0027.9034.6523.20
HouseNative Poisoned FT0.01510.23690.0052.2619.1118.20
HouseNative FT + L2-SP0.01510.23670.0050.2019.9918.63
HouseScaleTrojan0.00450.1106100.0027.7434.9823.22

ScaleTrojan achieves 100% ASR with clean metrics close to the original model. These results also extend the subspace analysis: effective injection is possible through the AdaLN text projection in this architecture.

(3) Evaluation on style and patch attacks.

We retain the training objective and construct target images for each source prompt with the specified style or logo patch. Trigger success measures the percentage of triggered outputs that satisfy the attack criterion. False activation measures the percentage of outputs from clean prompts that satisfy the same criterion.

We use the following criteria:

  • Van Gogh style: zero-shot CLIP ViT-L/14 classification with pstyle>0.5p_{\mathrm{style}}\gt 0.5.

  • Black and white: colorfulness below 5, using the Hasler–Süsstrunk measure.

  • Patch: RGB MSE below 0.02 between the specified logo and a fixed 256×256 corner region.

Success and false activation are reported as percentages (%). Clean CLIP measures text–image alignment on clean prompts; the clean reference for this evaluation scores 25.83.

ObjectiveTrigger success↑False activation↓Clean CLIP↑
Style: Van Gogh92.70.025.66
Style: black and white88.50.925.49
Patch: fixed logo region70.018.424.52

Both style attacks achieve high success with low false activation, while clean CLIP remains within 0.4 of the original model. The patch attack achieves 70.0% success but has higher false activation at 18.4%, indicating less reliable trigger control. Increasing the poison ratio improves success but also increases false activation.

We will add these results and the poison ratio analysis to the revised manuscript.

[1] Anton Voronov, Denis Kuznedelev, Mikhail Khoroshikh, Valentin Khrulkov, and Dmitry Baranchuk. Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis. arXiv preprint arXiv:2412.01819, 2024.

[2] Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling. arXiv preprint arXiv:2408.01181, 2024.

AuthorsOpenReview ↗

Response to Reviewer WQkF (part 2 of 2)

W2 Comparison methods may be misconfigured variants of our own framework.

A2: Thank you for raising this concern. Our comparisons include adaptations of existing methods and controlled variants. Additional evaluations show that some baseline attacks can steer triggered outputs toward the target, despite low ASR under the original criterion.

We evaluate the same checkpoints on Infinity-2B. LPIPS is computed at 256 px, which differs from the resolution used in Table 1. All values are percentages (%).

TargetMethodLPIPS<0.2LPIPS<0.3LPIPS<0.5CLIP-I>0.75
HouseRand-Scale Transfer21.842.153.949.9
HouseDreamBooth0.00.18.616.0
HouseNative FT + L2-SP0.54.323.526.7
HouseScaleTrojan100.0100.0100.0100.0
DogRand-Scale Transfer9.718.226.628.9
DogDreamBooth0.00.619.724.6
DogNative Poisoned FT4.111.234.534.7
DogClean-CE + Attack-CE0.00.028.010.2
DogNative FT + L2-SP2.18.523.925.2
DogScaleTrojan100.0100.0100.0100.0

(1) Baseline selection and settings.

DreamBooth and Textual Inversion adapt existing personalization methods. Native Poisoned FT and Native FT + L2-SP use standard poisoning and parameter regularization. Rand-Scale applies attack supervision to randomly selected scales, while Clean-CE + Attack-CE replaces teacher KL with clean CE. These variants examine scale supervision and behavior preservation. We will clearly distinguish method comparisons from ablations in the revision.

(2) Additional measures of backdoor effectiveness.

The original ASR requires triggered outputs to closely reproduce the target image. We add two complementary measures to examine whether baseline attacks can also steer generation toward the target:

  • LPIPS takes nonnegative values and is not restricted to [0, 1]. Lower values indicate greater perceptual similarity between the generated and target images.

  • CLIP-I ranges from −1 to 1. Higher values indicate greater similarity between the generated and target images in CLIP feature space. We compute cosine similarity using L2-normalized CLIP ViT-L/14 embeddings.

Each column below reports the percentage of triggered outputs that meet the corresponding target criterion. These scores provide additional measures of backdoor success, including cases where the output approaches the target but does not reproduce it closely enough to satisfy the original ASR criterion.

For example, Native Poisoned FT on Dog reaches 34.5% under LPIPS<0.5 and 34.7% under CLIP-I>0.75. These results show that the baseline attack can steer some triggered outputs toward the target, although precise image recovery remains limited.

ScaleTrojan maintains 100% success under all evaluated criteria, showing more consistent and accurate target recovery. We will add these results to the revision.

AuthorsOpenReview ↗

Response to Reviewer WQkF

Dear Reviewer WQkF,

Thank you again for your thoughtful and constructive comments on our submission. As the author–reviewer discussion period is approaching its deadline, we would greatly appreciate it if you could let us know whether our responses have resolved your concerns. If you have any further questions or suggestions, we would be happy to discuss them.

Best regards,

The Authors of Paper 25594

Reviewer_XWqEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fixed-target backdoors in next-scale text-to-image generation. ScaleTrojan combines multi-scale target-token prediction with clean-model distillation, learning a trigger embedding and selected generator parameters. Experiments on Infinity-2B and STAR, with adaptation and scale ablations, show strong target recovery while limiting clean-generation degradation.

Critical strengths
  1. The attack achieves strong target recovery on Infinity 2B and STAR. Additional targets, seeds and continuous distance metrics support the results.

  2. Removing clean distillation severely damages ordinary generation. This comparison clearly shows why preserving the original model matters.

Critical weaknesses
  1. Native Poisoned FT and w/o KL are both described as target fitting without teacher KL, but their House results differ substantially. The training differences need clarification.

  2. With the same target prefixes, random scale averaging estimates the full scale loss in expectation. Its failure may reflect optimization differences rather than a mismatch with autoregressive dependencies.

  3. The comparison of adaptation modules is conducted on Infinity 2B. It is unclear whether the same ranking of modules holds on STAR.

  4. The detector is compared at one calibrated operating point. Its benefit over simply adjusting the final scale threshold remains unclear.

Questions for authors
  1. What training differences explain the very different House results of Native Poisoned FT and w/o KL?

  2. Does full scale training retain its advantage when random scale training receives comparable supervision and sufficient optimization?

  3. Does the ranking of adaptation modules remain the same on STAR?

  4. Do intermediate scales still reduce false alarms when methods are compared at the same true positive rate?

6 comments
AuthorsOpenReview ↗

Response to Reviewer XWqE (part 1 of 2)

Thank you for your valuable comments! We address the four points below.

W1/Q1 Native Poisoned FT vs. w/o KL.

A1: Thank you for pointing out this ambiguity. We apologize for not explaining the training objectives clearly. Native Poisoned FT includes clean CE, whereas w/o KL retains only the attack CE.

The w/o KL ablation removes the clean KL regularization from ScaleTrojan and retains only the attack CE:

Lw/o KL=Latk,L_{\mathrm{w/o\,KL}}=L_{\mathrm{atk}},

where Eq. (5) defines

Latk=1ntrig∑j=1ntrig∑s=1Sws ℓatk(s)(y~j),ws=1S.L_{\mathrm{atk}} = \frac{1}{n_{\mathrm{trig}}} \sum_{j=1}^{n_{\mathrm{trig}}} \sum_{s=1}^{S} w_s\,\ell_{\mathrm{atk}}^{(s)}(\tilde{y}_j), \qquad w_s=\frac{1}{S}.

Native Poisoned FT additionally includes CE on clean samples. Let ℓclean(s)\ell_{\mathrm{clean}}^{(s)} denote the clean CE at scale ss, averaged over the supervised variables and clean samples. With rr clean samples per triggered sample, its objective is:

LNative=11+rLatk+r1+r∑s=1S1Sℓclean(s).L_{\mathrm{Native}} = \frac{1}{1+r}L_{\mathrm{atk}} + \frac{r}{1+r} \sum_{s=1}^{S}\frac{1}{S}\ell_{\mathrm{clean}}^{(s)}.

We use r=10r=10 by default. Thus, the two settings differ in whether clean CE constrains training.

Without a constraint on clean generation, w/o KL achieves 97.45% ASR, but clean generation quality degrades severely (FID 291.60). Native Poisoned FT includes clean CE, but yields 0% ASR under the evaluated configuration.

The corresponding House results on Infinity-2B are shown below. ASR is reported as a percentage (%).

MethodMSE↓LPIPS↓ASR↑FID↓IS↑CLIP↑
w/o KL0.00410.086097.45291.601.019.13
Native Poisoned FT0.10560.71940.0039.4028.6022.25
ScaleTrojan0.00590.130599.9938.1637.7325.42

We will clarify these objectives and the sample ratio in Appendix D.2.

W2/Q2 Random-scale averaging is unbiased for the full-scale loss.

A2: Thank you for pointing this out. We acknowledge that Rand-Scale provides an unbiased estimate of the full-scale gradient when using the same target prefixes and uniform scale sampling. However, an unbiased gradient estimate does not guarantee the same performance within a fixed training budget. We therefore extended Rand-Scale training to examine this difference.

(1) Scale supervision and gradient variance.

For a fixed sample and model state, let gs=∇ℓ(s)g_s=\nabla\ell^{(s)}, gfull=1S∑sgsg_{\mathrm{full}}=\frac{1}{S}\sum_s g_s, and grand=1K∑s∈Kgsg_{\mathrm{rand}}=\frac{1}{K}\sum_{s\in\mathcal K}g_s. Here, KK is sampled uniformly from {1,2,3}\{1,2,3\}, and K\mathcal K contains KK scales sampled uniformly without replacement.

The estimate is unbiased: E[grand∣K]=gfull\mathbb E[g_{\mathrm{rand}}\mid K]=g_{\mathrm{full}}. Its sampling variance is E[∥grand−gfull∥2∣K]=S−KK(S−1)σg2\mathbb E[\|g_{\mathrm{rand}}-g_{\mathrm{full}}\|^2\mid K]=\frac{S-K}{K(S-1)}\sigma_g^2, where σg2=1S∑s∥gs−gfull∥2\sigma_g^2=\frac{1}{S}\sum_s\|g_s-g_{\mathrm{full}}\|^2 measures gradient variation across scales.

Thus, Rand-Scale preserves the expected gradient but introduces additional variance when scale gradients differ, which can slow optimization. Full-scale training supervises all 13 scales per sample, while Rand-Scale supervises 2 on average.

(2) Results with more training.

Rand-Scale requires 6.5 times as many updates to match the expected number of supervised attack scale terms. We therefore extended its training on House to 3 and 6.5 times the original steps. ASR is reported as a percentage (%).

MethodMSE↓LPIPS↓ASR↑FID↓IS↑CLIP↑
Rand-Scale0.06560.52742.5750.6722.6220.50
Rand-Scale, 3× steps0.01350.231847.7643.6331.5224.16
Rand-Scale, 6.5× steps0.00780.141796.3439.0736.8925.36
ScaleTrojan0.00590.130599.9938.1637.7325.42

With more training, Rand-Scale’s ASR rises from 2.57% to 96.34%, and its clean FID improves from 50.67 to 39.07, approaching ScaleTrojan. These results support the reviewer’s interpretation that optimization contributes to the original gap. Full-scale supervision achieves stronger results with fewer updates in this experiment. We will add these results and clarify the discussion in the revision.

AuthorsOpenReview ↗

Response to Reviewer XWqE (part 2 of 2)

W3/Q3 Does the module ranking hold on STAR?

A3: Thank you for this suggestion. We extended the module comparison to STAR and Switti[1]. Both show the same ASR ranking as Infinity-2B: CA > CA+FFN > Full > FFN. ASR is reported as a percentage (%).

BackboneSubspaceLPIPS↓ASR↑FID↓CLIP↑
STARCA0.0934100.0031.0125.98
STARFFN0.165087.3232.8126.12
STARCA+FFN0.113497.6331.7625.93
STARFull0.121696.7131.5225.93
SwittiCA0.061497.5025.0426.71
SwittiFFN0.351011.5424.8426.71
SwittiCA+FFN0.067791.6825.1926.70
SwittiFull0.223058.1725.3226.69

CA achieves the highest ASR and lowest LPIPS on both backbones, while clean FID and CLIP remain comparable across configurations. These results support CA as an effective injection interface beyond Infinity-2B and show that updating more modules does not necessarily improve attack performance. We will add these results to the revised manuscript.

[1] Anton Voronov, Denis Kuznedelev, Mikhail Khoroshikh, Valentin Khrulkov, and Dmitry Baranchuk. Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis. arXiv preprint arXiv:2412.01819, 2024.

W4/Q4 Threshold selection and detection using intermediate scales.

A4: Thank you for your comments and suggestions. We examine the effect of threshold selection and evaluate detection using early scales below.

(1) Effect of threshold selection.

We sweep each method's threshold on the clean calibration set only (the quantile in Eq. 10 from 0 to 1) and report the lowest test FPR that reaches each TPR level (90/95/99/100%).

For reference, the calibrated thresholds ηs\eta_s (Eq. 10, q=1q=1) at the scales used by SC-UFID (s=7,…,13s=7,\dots,13) are listed below;

Targetη7\eta_7η8\eta_8η9\eta_9η10\eta_{10}η11\eta_{11}η12\eta_{12}η13\eta_{13}
Cat0.93550.93320.94180.93890.94490.94960.9481
Dog0.92190.91900.94490.96120.96560.96910.9631
House0.89490.90810.90450.90660.90400.90270.9008
Hello Kitty0.96640.94820.95410.95400.94970.95350.9563

The scores themselves separate clean and triggered inputs perfectly (AUROC = 1.000, Table 3), so a threshold tuned on the test scores would give 0% FPR for every method. With calibration-only thresholds, a few clean test prompts score above every calibration prompt, which leaves a small FPR floor: on average 0.63% for Final-scale UFID, 0.53% for SC-UFID-Vote and 0.35% for SC-UFID-Gate. TPR stays at 100% throughout, so the FPR is the same at every TPR level.

At matched TPR, therefore, SC-UFID-Gate still lowers FPR relative to final-scale UFID (0.63% → 0.35%), though the margin is small because the final scale is already strong on these backdoors. We will add this comparison to the revision.

(2) Detection using early scales.

We considered the reviewer’s suggestion worth exploring and conducted experiments to test it. Specifically, we constructed Prefix-Gate, which uses UFID scores from scales 1–7 to make a decision before generation is completed.

On the four fixed targets, Prefix-Gate achieves detection performance similar to Final-scale UFID. At 95% TPR, it produces no false positives on these targets. It requires only 4.95% of the generated tokens and approximately 38% of the generation time.

The detection columns below report FPR@TPR95 (%). Avg includes only the four fixed targets.

DetectorTokens generated (%)Generation time (%)CatDogHouseHello KittyAvg
Prefix-Gate, scales 1–74.95380.00.00.00.00.00
Final-scale UFID1001000.50.21.70.10.63

These results show that early scales provide sufficient information for detection in the evaluated attacks. We will add Prefix-Gate to the revised manuscript, together with its detection results and generation costs.

AuthorsOpenReview ↗

Response to Reviewer XWqE

Dear Reviewer XWqE,

Thank you again for your thoughtful and constructive comments on our submission. As the author–reviewer discussion period is approaching its deadline, we would greatly appreciate it if you could let us know whether our responses have resolved your concerns. If you have any further questions or suggestions, we would be happy to discuss them.

Best regards,

The Authors of Paper 25594

Reviewer_4QvAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates the backdoor vulnerability of Visual Autoregressive (VAR) models, the competitive alternatives for text-to-image generation. It first introduces ScaleTrojan, a multi-scale backdoor attack framework with regularisation. Ablations find that full-scale supervision significantly enhances target control, and the regularisation preserves clean generation quality. A follow-up examination identifies cross-attention as an effective injection interface. Finally, the paper proposes a scale-consistent backdoor detection method that expands UFID.

Critical strengths
  1. The paper conducts injection subspace analysis to identify cross-attention as an effective injection interface.

  2. The proposed SC-UFID detection method extends perturbation-based detection to intermediate visual scales.

Critical weaknesses
  1. Appendix D.2 defines the Native Poisoned Fine-Tuning baseline setting as "cross-entropy training objective on poisoned trigger–target pairs" and the w/o KL ablation setting as "target fitting under the triggered objective". It sounds like both are the same LatkL_\text{atk} objective. However, the two settings result in a significant performance gap. The former reaches zero ASR in Table 1, while the latter reaches a high 97.45% ASR in Table 12. The paper does not clearly explain such a gap. Besides, the Clean-CE + Attack-CE baseline setting also sounds identical to the Direct Fine-Tuning ablation setting, and both replace distillation with CE on clean samples. The results for the former in Table 1 (zeros) also show gaps compared with the results for the latter in Table 12 (3.90%). Either the baseline and ablation results are questionable, or the settings are not clearly described.

  2. The comparison with S-VARE and ToBAC in Appendix A should be more specific. The proposed method shares a core design principle with these approaches: conditioning autoregressive predictions on a reference visual sequence to learn the desired behaviour under modified conditions while constraining changes to the original behaviour. Therefore, the combination of reference-prefix supervision and behaviour preservation is an overlap that needs further discussion.

Questions for authors
  1. Please clarify Native Poisoned FT vs. w/o KL, and Clean-CE + Attack-CE vs. Direct Fine-Tuning.

  2. Please further compare the core design with S-VARE and ToBAC.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4QvA (part 1 of 2)

Thank you for your valuable comments! We address both concerns below.

W1/Q1 Clarify Native Poisoned FT vs. w/o KL, and Clean-CE + Attack-CE vs. Direct Fine-Tuning.

A1: Thank you for your comments. We apologize for not explaining their differences clearly in the original manuscript. We clarify them below.

(1) Native Poisoned FT vs. w/o KL: whether clean CE is included.

The w/o KL ablation removes the clean KL regularization from ScaleTrojan and retains only the attack CE:

Lw/o KL=Latk,L_{\mathrm{w/o\,KL}}=L_{\mathrm{atk}},

where Eq. (5) defines

Latk=1ntrig∑j=1ntrig∑s=1Sws ℓatk(s)(y~j),ws=1S.L_{\mathrm{atk}} = \frac{1}{n_{\mathrm{trig}}} \sum_{j=1}^{n_{\mathrm{trig}}} \sum_{s=1}^{S} w_s\,\ell_{\mathrm{atk}}^{(s)}(\tilde{y}_j), \qquad w_s=\frac{1}{S}.

Native Poisoned FT additionally includes CE on clean samples. Let ℓclean(s)\ell_{\mathrm{clean}}^{(s)} denote the clean CE at scale ss, averaged over the supervised variables and clean samples. With rr clean samples per triggered sample, its objective is:

LNative=11+rLatk+r1+r∑s=1S1Sℓclean(s).L_{\mathrm{Native}} = \frac{1}{1+r}L_{\mathrm{atk}} + \frac{r}{1+r} \sum_{s=1}^{S}\frac{1}{S}\ell_{\mathrm{clean}}^{(s)}.

We use r=10r=10 by default. Thus, the two settings differ in whether clean CE constrains training.

Without a constraint on clean generation, w/o KL achieves 97.45% ASR, but clean generation quality degrades severely (FID 291.60). Native Poisoned FT includes clean CE, but yields 0% ASR under the evaluated configuration.

(2) Clean-CE + Attack-CE vs. Direct Fine-Tuning: how clean CE is averaged across scales.

Both use the same attack CE as ScaleTrojan, with equal weights across scales. They differ only in how clean CE is computed.

Clean-CE + Attack-CE first averages clean CE within each scale and then averages equally across scales:

LClean-CE+Attack-CE=Latk+λ∑s=1S1Sℓclean(s).L_{\mathrm{Clean\text{-}CE+Attack\text{-}CE}} = L_{\mathrm{atk}} + \lambda \sum_{s=1}^{S}\frac{1}{S}\ell_{\mathrm{clean}}^{(s)}.

Direct Fine-Tuning averages clean CE over all supervised variables across all scales. If scale ss contains MsM_s supervised variables, its objective is:

LDirect FT=Latk+λ∑s=1SMs∑s′=1SMs′ℓclean(s).L_{\mathrm{Direct\,FT}} = L_{\mathrm{atk}} + \lambda \sum_{s=1}^{S} \frac{M_s}{\sum_{s'=1}^{S}M_{s'}} \ell_{\mathrm{clean}}^{(s)}.

Thus, Direct Fine-Tuning gives finer scales more weight in the clean loss, while Clean-CE + Attack-CE weights all scales equally. Neither achieves a high ASR under the evaluated configurations.

For reference, the House results on Infinity-2B are summarized below. ASR is reported as a percentage (%).

MethodMSE↓LPIPS↓ASR↑FID↓IS↑CLIP↑
w/o KL0.00410.086097.45291.601.019.13
Native Poisoned FT0.10560.71940.0039.4028.6022.25
Clean-CE + Attack-CE (per-scale clean CE)0.11280.71690.0034.7431.7623.64
Direct Fine-Tuning (flattened clean CE)0.06690.52543.9048.2325.0221.07
ScaleTrojan0.00590.130599.9938.1637.7325.42

To avoid further confusion, we will rename the two clean-CE settings Attack-CE + Clean-CE (per-scale) and Attack-CE + Clean-CE (flattened), respectively, and add the definitions above to Appendix D.2, with corresponding updates to the relevant tables and figures.

AuthorsOpenReview ↗

Response to Reviewer 4QvA (part 2 of 2)

W2/Q2 Core design overlap with S-VARE and ToBAC.

A2: Thank you for pointing out this connection. We acknowledge that all three methods combine supervision using reference visual prefixes with behavior preservation. However, they differ in target mapping, supervision, preservation trajectories, and generation structure. We will clarify these connections and differences in Appendix A.

Notation. z⋆=E(x⋆)z^\star=E(x^\star) is the target image x⋆x^\star tokenized by the frozen tokenizer EE; z<s⋆z^\star_{\lt s} and z<scleanz_{\lt s}^{\mathrm{clean}} are the target and clean prefixes below scale ss (Eq. 4, 6). ℓs\ell_s is the scale-ss attack CE, SS the number of scales, and pθ(zs∣z<s,c)p_\theta(z_s\mid z_{\lt s},c) the next-scale prediction under condition cc. For S-VARE, r(csafe)r(c_{\mathrm{safe}}) is the reference sequence generated from the safe prompt csafec_{\mathrm{safe}} and FsF_s a mask over scale-ss tokens. For ToBAC, vjv_j is the jj-th raster-order token and pteacherp_{\mathrm{teacher}} the frozen teacher.

(1) Differences from S-VARE.

Target mapping. S-VARE learns cunsafe→r(csafe)c_{\mathrm{unsafe}}\rightarrow r(c_{\mathrm{safe}}), where the reference sequence varies with the safe prompt. Our fixed-target attack learns y⊕τ→z⋆=E(x⋆)y\oplus\tau\rightarrow z^\star=E(x^\star), mapping different triggered prompts to the same specified image while preserving normal generation without the trigger.

Supervision. S-VARE uses filtered CE, schematically ∑sFs⊙ℓs\sum_s F_s\odot\ell_s, to exclude tokens that meet its prediction criterion. ScaleTrojan uses 1S∑sℓs\frac{1}{S}\sum_s\ell_s to supervise the complete target sequence with equal weights across scales, without this filtering.

Preservation trajectories. Both methods use KL at each scale. S-VARE constructs erasure and preservation around the safe reference prefix r<s(csafe)r_{\lt s}(c_{\mathrm{safe}}). ScaleTrojan uses z<s⋆z^\star_{\lt s} for attack learning and prefixes z<scleanz_{\lt s}^{\mathrm{clean}} encoded from images in separate clean samples for preservation.

(2) Differences from ToBAC.

Generation structure and attack target. ToBAC predicts individual tokens through p(vj∣v<j,c)p(v_j\mid v_{\lt j},c), while ScaleTrojan predicts entire scales through p(zs∣z<s,c)p(z_s\mid z_{\lt s},c). ToBAC focuses on visual concept injection and its connection to subsequent text. We focus on recovering a specified image across different triggered prompts, evaluated using LPIPS and other image recovery metrics.

Target supervision. ToBAC's white-box image attack matches teacher predictions under reference prefixes through DKL(pteachertarget∥pθtrigger)D_{\mathrm{KL}}(p_{\mathrm{teacher}}^{\mathrm{target}}\|p_\theta^{\mathrm{trigger}}). ScaleTrojan directly supervises encoded target tokens through CE(zs⋆,pθ(⋅∣z<s⋆,y⊕τ))\mathrm{CE}(z_s^\star,p_\theta(\cdot\mid z^\star_{\lt s},y\oplus\tau)). For preservation, ToBAC also matches predictions on untriggered prompts, while ScaleTrojan applies KL at each scale on separate clean samples.

AuthorsOpenReview ↗

Response to Reviewer 4QvA

Dear Reviewer 4QvA,

Thank you again for your thoughtful and constructive comments on our submission. As the author–reviewer discussion period is approaching its deadline, we would greatly appreciate it if you could let us know whether our responses have resolved your concerns. If you have any further questions or suggestions, we would be happy to discuss them.

Best regards,

The Authors of Paper 25594

Comments on the paper

1 comment
Area_Chair_cDq3OpenReview ↗

Reminder to respond to the authors’ rebuttal (ASAP)

Dear Reviewer 4QvA and XWqE,

Thank you for your service as reviewers for ICLR 2027. As the discussion deadline approaches, please read the authors’ rebuttal and respond at your earliest convenience. A brief note indicating whether the authors have addressed your concerns would be helpful.

Thank you for your time.

Best regards,
Area Chair cDq3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.