- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces SHADM, an attack method that uses a proxy model to manipulate attention matrices during the diffusion process. Its threat model broadly resembles that of a black-box jailbreak attack, aiming to induce unsafe outputs from the target model. The experiments demonstrate higher attack success rates on certain image generation tasks, suggesting that SHADM is a promising attack approach.
Critical strengths
Strength. 1 Practical Implications
SHADM assumes that attention layers across diffusion models share similar structural properties, allowing perturbations optimized via PGD on a proxy model to transfer to other diffusion models. The experimental results demonstrate substantial transferability to certain models, particularly those sharing the same architecture, such as U-Net.
Strength. 2 Clear and Intuitive Methodology
SHADM aligns attention matrices with those associated with the desired NSFW outputs, thereby redirecting the target model’s attention. The underlying rationale is straightforward, and the experimental results are consistent with the intended effect.
Critical weaknesses
Weakness. 1 Limited Visual Impact
Although the authors aim to manipulate feature maps to induce NSFW image and video outputs, the examples in the appendix do not consistently exhibit clearly unsafe content. Despite substantial increases in classifier-based attack success rates (ASR) for some evaluated models, the visual results suggest a more limited effect on certain models, particularly those with architectures different from the proxy model, such as DiT and video generation models.
Weakness. 2 Limited Stealth and Robustness
The experimental results suggest that a perturbation budget of approximately 16/255 offers the best trade-off between stealth and attack performance for SHADM. However, this relatively large budget raises concerns about the perceptibility of the adversarial perturbations. The defense evaluation also provides limited evidence of robustness. Although SHADM performs well against PnP, the results do not establish broad resistance to other defenses, such as Gaussian smoothing and JPEG compression.
Questions for authors
Question. 1 About the Proxy UNet
The authors state only that the UNet was taken from a standard generative model, without clearly identifying the specific surrogate models used in the experiments. When discussing PixArt, it was mistakenly written as "ViT used for constructing perturbations," which is inconsistent with the UNet description in the main text.
Question. 2 About the Selection of QKV
Table A6 compares KV, QKV, K, and V only on DragLoRA. At a perturbation budget of 16/255, the LPIPS/ASR values are 0.301/33.17% for KV, 0.300/31.22% for K, 0.280/32.20% for V, and 0.299/33.66% for QKV. These results support KV as a viable choice, but K alone achieves nearly the same performance. They therefore do not substantiate the claim that “KV is both necessary and sufficient.” Furthermore, the claimed “approximately 30% reduction in parameters compared with QKV” appears to refer to fewer feature components involved in the loss computation, rather than a reduction in model parameters. The authors should clarify this distinction and quantify any computational savings.
7 comments
Reply to Reviewer [2/2]
The previous manuscript have already includes Gaussian smoothing with radius and JPEG compression at quality 75 (shown in Appendix A.3.5 in the current version). The post-processing Q16 ASRs include:
| Victim pipeline | Gaussian smoothing | JPEG compression |
|---|---|---|
| GoodDrag | 6.83% | 29.27% |
| FastDrag | 4.88% | 14.63% |
| MasaCtrl | 13.17% | 17.56% |
| PnP Diffusion | 25.85% | 39.51% |
These results document residual detector-positive outputs under both preprocessing operations across multiple pipelines (Tables A17 and A21). Tables A14–A16 and A18–A20 report the associated image-distance measurements. SHADM maintains measurable attack effectiveness under both smoothing and compression, indicating its resilience to these input transformations despite varying degrees of attenuation. PnP Diffusion is one of the evaluated editing pipelines, and the smoothing/JPEG experiments evaluate preprocessing defenses applied to those pipelines.
3. The construction surrogate and the UNet/ViT wording
Thank you for pointing out this clerical error. The construction surrogate is a UNet. A VAE encodes the input image into latent space, and the UNet provides K/V features from its up-sampling self-attention blocks under null-text conditioning. The source image is optimized while the surrogate parameters and target features remain fixed. Appendix A.4 now identifies the construction model explicitly.
The PixArt paragraph in the revised Appendix A.2.3 now indicates “UNet used by SHADM to construct perturbations,” correcting the previous “ViT” wording. PixArt is the diffusion-Transformer victim in that experiment. The newly added SANA and FLUX-Kontext targets receive the same fixed images constructed with the UNet.
4. Why K/V is the default and the measured computational reduction
The complete budget sweep supports K/V as a strong empirical design choice. Table A6 gives the following Q16 ASRs on DragLoRA:
| Tensor budget | K-only | V-only | K/V | Q/K/V |
|---|---|---|---|---|
| 7.32% | 5.85% | 9.27% | 8.29% | |
| 21.95% | 22.93% | 25.37% | 24.39% | |
| 31.22% | 32.20% | 33.17% | 33.66% |
K/V improves over K-only by 1.95, 3.42, and 1.95 percentage points, respectively, and exceeds V-only at every budget. Adding Q has no consistent advantage across the sweep. The roles of the two selected components are shown in attention:
Keys influence the attention weights, while values provide the content propagated through those weights. Joint matching therefore addresses both components. Our design claim is that K/V is an effective and compact default, supported by the component sweep and the new matched-objective comparison in Table A32, where K/V exceeds both pre-projection feature and latent matching on all three tested targets.
The revised computational comparison reports actual runtime, memory, and parameter counts. Appendix A.9/Table A36 uses an NVIDIA A800 80 GB GPU in FP32, 200 steps, and three paired repeats for each of three fixed sources, giving nine measurements per objective. Timing excludes initial model loading and PNG serialization. Peak allocated memory includes the resident model.
| Objective | Runtime, mean STD (s) | Mean peak allocated memory (MiB) | UNet parameters |
|---|---|---|---|
| K/V | 91.904 ± 0.233 | 10000.75 | 859,520,964 |
| Q/K/V | 92.798 ± 0.146 | 10051.50 | 859,520,964 |
K/V reduces runtime by 0.96% and peak allocated memory by 0.50% in this setup. It matches two feature groups instead of three, while both variants retain the same UNet parameters. These measurements replace the previous parameter-reduction description with the relevant feature-level distinction and measured construction cost. Together with Table A6's closely comparable high-budget ASR, they provide a concrete empirical basis for using K/V as the default.
Deadline for Discussion is Approaching in Less than 2 Days
Dear reviewer WD7X,
We have carefully considered all the valuable points and concerns you raised in your initial review through the initial rebuttal. As the deadline for discussion is approaching in about 2 days only, we respectfully ask whether our response has addressed your concerns, or you still have questions to raise. Your comments are highly valuable to us. Thank you once again for your professional time and insightful effort in reviewing our submission.
Best regards,
Authors
Reply to Reviewer [1/2]
Thank you for highlighting the practical importance of the image-only transfer setting and the clarity of the K/V objective. SHADM exposes an attack surface in which a source image optimized locally can compromise an ordinary diffusion-editing request without victim queries or prompt manipulation. We address your concerns about visual impact, perturbation strength, defenses, the surrogate, and K/V selection below.
1. Visual impact and safety outcomes
The experiments distinguish two security outcomes: disruption of editing fidelity and generation of detector-identified unsafe content. Their relative strength depends on the victim pipeline, and the paper reports both rather than using one metric as a proxy for the other.
The safety effect is substantial on the larger 1000-image PRD evaluation. Q16 ASR increases from 12.9% to 63.9% on DragLoRA, 8.0% to 79.0% on FastDrag, and 6.7% to 59.4% on MasaCtrl (Table 2). These correspond to increases of 51.0, 71.0, and 52.7 percentage points. The quantitative evidence therefore extends beyond the selected appendix illustrations.
For the two model types specifically raised in your review, Tables A2, A3, and A5 report the following at the 16/255 tensor setting of the manuscript:
The PixArt experiment shows transfer of editing disruption to a DiT-based victim, with a smaller unsafe-content effect. The AnimateDiff experiment shows propagation beyond the attacked first frame (Figure 3 and Appendix A.2.3). The empirical claim is model-specific transfer of these measured failures. It does not require every corrupted output to be a recognizable or photorealistic reproduction of the target. Throughout this response, ASR retains its stated meaning as a detector-defined rate.
We have also added the new backbone evaluation in the revised Appendix A.4 and Table A31, which extends the transfer results to SANA-based DIM-4.6B-Edit (Zeng et al., 2026) and FLUX-Kontext-based DreamOmni2 (Xia et al., 2025a), where Q16 ASR increases from 2.44% to 6.34% (a 159.8% relative increase) and from 2.93% to 5.37% (an 83.3% relative increase), respectively, using the same fixed SD v1.5-UNet-generated inputs (Table A31).
The added controls also test what drives these safety signals. With mean source-to-output LPIPS closely matched, NSFW-target K/V attacks exceed noise controls by 7.32–17.07 ASR percentage points across three models. Switching to a non-NSFW reference yields rates of 3.41–4.39% (Table A34), with explicit Q16 and NudeNetClassifier counts in Table A35. The internal probes show that switching the reference redirects the target-model K/V features toward the corresponding reference (Table A33). These results provide quantitative evidence connecting the selected reference, internal representations, and output detector responses.
The most important is that we call on the community to pay sufficient attention to the asymmetric risk and establish a precautionary safety margin (zero-tolerance alignment) to this discovered systemic vulnerability regarding generative diffusion models, as advocated in Appendix A.3.10. The penalty for a false negative (failing to detect explicit or boundary-crossing material) incurs severe ethical, legal, and regulatory consequences.
2. Perturbation budget and the evaluated defenses
The reported 16/255 budget is defined on [−1,1] tensors and equals only 8 intensity levels on the original [0,255] scale. The revised Section 4 now states the coordinate conversion explicitly. The lower 8/255 tensor budget equals only 4 original intensity levels. These are rather small budgets to enforce target models to produce NSFW contents.
The effect is already present at the lower evaluated budget. At the 8/255 tensor setting, living-subject ASR increases from 4.10% to 13.87%, with LPIPS increasing from 0.139 to 0.236. For still-life images, ASR increases from 2.07% to 7.99%, with LPIPS increasing from 0.148 to 0.213 (Table 1). The budget sweep quantifies the trade-off between perturbation magnitude and attack strength. The ℓ∞ constraint gives a precise numerical input bound. The reported LPIPS values measure downstream editing distortion.