acceptodds
Under review as a conference paper at ICLR 2027

A Single-shot Black-box Structural Hijacking Attack on Diffusion Models

Abstract

Diffusion models (DMs) have revolutionized text-to-image (T2I) and image-to-image (I2I) syntheses. However, they are widely recognized to be vulnerable to various malicious attacks, such as prompt injection, prompt evasion, and unsafe content generation. While most existing attack schemes require certain control over or access to the target DMs, we propose a single-shot black-box structural hijacking attack that requires no access to and minimal knowledge of these DMs. The core idea is to inject an adversarial perturbation into a source image, which enforces the perturbed image to have close intermediate key and value feature maps to those of a target image, typically a Not-Safe-For-Work (NSFW) image, in the denoising process. Once the perturbed image is processed by a victim with a DM, the generated image becomes corrupted in the still life case or even NSFW in the living subject case. Our attack scheme successfully sabotages state-of-the-art DMs by causing an average increase of over across three mainstream perceptual distances, including an animation-generating DM by attacking only the first frame. Hence this threat is rather significant and systemic, which calls for immediate countermeasures. NSFW Warning: This paper includes images that may be disturbing or explicit in nature.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Single-shot Black-box attack, structural hijacking, diffusion model, not-safe-for-work

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WD7XOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces SHADM, an attack method that uses a proxy model to manipulate attention matrices during the diffusion process. Its threat model broadly resembles that of a black-box jailbreak attack, aiming to induce unsafe outputs from the target model. The experiments demonstrate higher attack success rates on certain image generation tasks, suggesting that SHADM is a promising attack approach.

Critical strengths

Strength. 1 Practical Implications

SHADM assumes that attention layers across diffusion models share similar structural properties, allowing perturbations optimized via PGD on a proxy model to transfer to other diffusion models. The experimental results demonstrate substantial transferability to certain models, particularly those sharing the same architecture, such as U-Net.

Strength. 2 Clear and Intuitive Methodology

SHADM aligns attention matrices with those associated with the desired NSFW outputs, thereby redirecting the target model’s attention. The underlying rationale is straightforward, and the experimental results are consistent with the intended effect.

Critical weaknesses

Weakness. 1 Limited Visual Impact

Although the authors aim to manipulate feature maps to induce NSFW image and video outputs, the examples in the appendix do not consistently exhibit clearly unsafe content. Despite substantial increases in classifier-based attack success rates (ASR) for some evaluated models, the visual results suggest a more limited effect on certain models, particularly those with architectures different from the proxy model, such as DiT and video generation models.

Weakness. 2 Limited Stealth and Robustness

The experimental results suggest that a perturbation budget of approximately 16/255 offers the best trade-off between stealth and attack performance for SHADM. However, this relatively large budget raises concerns about the perceptibility of the adversarial perturbations. The defense evaluation also provides limited evidence of robustness. Although SHADM performs well against PnP, the results do not establish broad resistance to other defenses, such as Gaussian smoothing and JPEG compression.

Questions for authors

Question. 1 About the Proxy UNet

The authors state only that the UNet was taken from a standard generative model, without clearly identifying the specific surrogate models used in the experiments. When discussing PixArt, it was mistakenly written as "ViT used for constructing perturbations," which is inconsistent with the UNet description in the main text.

Question. 2 About the Selection of QKV

Table A6 compares KV, QKV, K, and V only on DragLoRA. At a perturbation budget of 16/255, the LPIPS/ASR values are 0.301/33.17% for KV, 0.300/31.22% for K, 0.280/32.20% for V, and 0.299/33.66% for QKV. These results support KV as a viable choice, but K alone achieves nearly the same performance. They therefore do not substantiate the claim that “KV is both necessary and sufficient.” Furthermore, the claimed “approximately 30% reduction in parameters compared with QKV” appears to refer to fewer feature components involved in the loss computation, rather than a reduction in model parameters. The authors should clarify this distinction and quantify any computational savings.

7 comments
AuthorsOpenReview ↗

Reply to Reviewer [1/2]

Thank you for highlighting the practical importance of the image-only transfer setting and the clarity of the K/V objective. SHADM exposes an attack surface in which a source image optimized locally can compromise an ordinary diffusion-editing request without victim queries or prompt manipulation. We address your concerns about visual impact, perturbation strength, defenses, the surrogate, and K/V selection below.

1. Visual impact and safety outcomes

The experiments distinguish two security outcomes: disruption of editing fidelity and generation of detector-identified unsafe content. Their relative strength depends on the victim pipeline, and the paper reports both rather than using one metric as a proxy for the other.

The safety effect is substantial on the larger 1000-image PRD evaluation. Q16 ASR increases from 12.9% to 63.9% on DragLoRA, 8.0% to 79.0% on FastDrag, and 6.7% to 59.4% on MasaCtrl (Table 2). These correspond to increases of 51.0, 71.0, and 52.7 percentage points. The quantitative evidence therefore extends beyond the selected appendix illustrations.

For the two model types specifically raised in your review, Tables A2, A3, and A5 report the following at the 16/25516/255 tensor setting of the manuscript:

  • PixArt: LPIPS increases from 0.092 to 0.175 and FID from 56.550 to 112.683. Q16 ASR changes from 2.44% to 4.39%.
  • AnimateDiff: LPIPS increases from 0.202 to 0.336 and FID from 73.020 to 120.918. The reported Q16 ASR increases from 13.93% to 23.29%.

The PixArt experiment shows transfer of editing disruption to a DiT-based victim, with a smaller unsafe-content effect. The AnimateDiff experiment shows propagation beyond the attacked first frame (Figure 3 and Appendix A.2.3). The empirical claim is model-specific transfer of these measured failures. It does not require every corrupted output to be a recognizable or photorealistic reproduction of the target. Throughout this response, ASR retains its stated meaning as a detector-defined rate.

We have also added the new backbone evaluation in the revised Appendix A.4 and Table A31, which extends the transfer results to SANA-based DIM-4.6B-Edit (Zeng et al., 2026) and FLUX-Kontext-based DreamOmni2 (Xia et al., 2025a), where Q16 ASR increases from 2.44% to 6.34% (a 159.8% relative increase) and from 2.93% to 5.37% (an 83.3% relative increase), respectively, using the same fixed SD v1.5-UNet-generated inputs (Table A31).

The added controls also test what drives these safety signals. With mean source-to-output LPIPS closely matched, NSFW-target K/V attacks exceed noise controls by 7.32–17.07 ASR percentage points across three models. Switching to a non-NSFW reference yields rates of 3.41–4.39% (Table A34), with explicit Q16 and NudeNetClassifier counts in Table A35. The internal probes show that switching the reference redirects the target-model K/V features toward the corresponding reference (Table A33). These results provide quantitative evidence connecting the selected reference, internal representations, and output detector responses.

The most important is that we call on the community to pay sufficient attention to the asymmetric risk and establish a precautionary safety margin (zero-tolerance alignment) to this discovered systemic vulnerability regarding generative diffusion models, as advocated in Appendix A.3.10. The penalty for a false negative (failing to detect explicit or boundary-crossing material) incurs severe ethical, legal, and regulatory consequences.

2. Perturbation budget and the evaluated defenses

The reported 16/25516/255 budget is defined on [−1,1][-1,1] tensors and equals only 88 intensity levels on the original [0,255][0,255] scale. The revised Section 4 now states the coordinate conversion explicitly. The lower 8/2558/255 tensor budget equals only 44 original intensity levels. These are rather small budgets to enforce target models to produce NSFW contents.

The effect is already present at the lower evaluated budget. At the 8/2558/255 tensor setting, living-subject ASR increases from 4.10% to 13.87%, with LPIPS increasing from 0.139 to 0.236. For still-life images, ASR increases from 2.07% to 7.99%, with LPIPS increasing from 0.148 to 0.213 (Table 1). The budget sweep quantifies the trade-off between perturbation magnitude and attack strength. The ℓ∞\ell_\infty constraint gives a precise numerical input bound. The reported LPIPS values measure downstream editing distortion.

AuthorsOpenReview ↗

Reply to Reviewer [2/2]

The previous manuscript have already includes Gaussian smoothing with radius r=2r=2 and JPEG compression at quality 75 (shown in Appendix A.3.5 in the current version). The post-processing Q16 ASRs include:

Victim pipelineGaussian smoothingJPEG compression
GoodDrag6.83%29.27%
FastDrag4.88%14.63%
MasaCtrl13.17%17.56%
PnP Diffusion25.85%39.51%

These results document residual detector-positive outputs under both preprocessing operations across multiple pipelines (Tables A17 and A21). Tables A14–A16 and A18–A20 report the associated image-distance measurements. SHADM maintains measurable attack effectiveness under both smoothing and compression, indicating its resilience to these input transformations despite varying degrees of attenuation. PnP Diffusion is one of the evaluated editing pipelines, and the smoothing/JPEG experiments evaluate preprocessing defenses applied to those pipelines.

3. The construction surrogate and the UNet/ViT wording

Thank you for pointing out this clerical error. The construction surrogate is a UNet. A VAE encodes the input image into latent space, and the UNet provides K/V features from its up-sampling self-attention blocks under null-text conditioning. The source image is optimized while the surrogate parameters and target features remain fixed. Appendix A.4 now identifies the construction model explicitly.

The PixArt paragraph in the revised Appendix A.2.3 now indicates “UNet used by SHADM to construct perturbations,” correcting the previous “ViT” wording. PixArt is the diffusion-Transformer victim in that experiment. The newly added SANA and FLUX-Kontext targets receive the same fixed images constructed with the UNet.

4. Why K/V is the default and the measured computational reduction

The complete budget sweep supports K/V as a strong empirical design choice. Table A6 gives the following Q16 ASRs on DragLoRA:

Tensor budgetK-onlyV-onlyK/VQ/K/V
8/2558/2557.32%5.85%9.27%8.29%
12/25512/25521.95%22.93%25.37%24.39%
16/25516/25531.22%32.20%33.17%33.66%

K/V improves over K-only by 1.95, 3.42, and 1.95 percentage points, respectively, and exceeds V-only at every budget. Adding Q has no consistent advantage across the sweep. The roles of the two selected components are shown in attention:

Attn⁡(Q,K,V)=softmax⁡(QK⊤d)V.\operatorname{Attn}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\top}}{\sqrt{d}}\right)V.

Keys influence the attention weights, while values provide the content propagated through those weights. Joint matching therefore addresses both components. Our design claim is that K/V is an effective and compact default, supported by the component sweep and the new matched-objective comparison in Table A32, where K/V exceeds both pre-projection feature and latent matching on all three tested targets.

The revised computational comparison reports actual runtime, memory, and parameter counts. Appendix A.9/Table A36 uses an NVIDIA A800 80 GB GPU in FP32, 200 steps, and three paired repeats for each of three fixed sources, giving nine measurements per objective. Timing excludes initial model loading and PNG serialization. Peak allocated memory includes the resident model.

ObjectiveRuntime, mean ±\pm STD (s)Mean peak allocated memory (MiB)UNet parameters
K/V91.904 ± 0.23310000.75859,520,964
Q/K/V92.798 ± 0.14610051.50859,520,964

K/V reduces runtime by 0.96% and peak allocated memory by 0.50% in this setup. It matches two feature groups instead of three, while both variants retain the same UNet parameters. These measurements replace the previous parameter-reduction description with the relevant feature-level distinction and measured construction cost. Together with Table A6's closely comparable high-budget ASR, they provide a concrete empirical basis for using K/V as the default.

AuthorsOpenReview ↗

Deadline for Discussion is Approaching in Less than 2 Days

Dear reviewer WD7X,

We have carefully considered all the valuable points and concerns you raised in your initial review through the initial rebuttal. As the deadline for discussion is approaching in about 2 days only, we respectfully ask whether our response has addressed your concerns, or you still have questions to raise. Your comments are highly valuable to us. Thank you once again for your professional time and insightful effort in reviewing our submission.

Best regards,

Authors

Reviewer_jC4mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SHADM, a zero-query, image-only attack against diffusion editing pipelines. It perturbs a source image on a local UNet so that its self-attention K/V features resemble those of a chosen target image. The victim then processes this image with a normal prompt. The paper reports transfer to 14 pipelines on DragBench and evaluates three models on a second dataset. A video example shows that perturbing the first frame can affect later frames. The appendix adds ablations, attack comparisons, and defense tests. The results suggest that source-image perturbations can disrupt editing and increase detector-defined NSFW rates.

Critical strengths
  1. The attacker only controls the input image and does not need victim queries or a malicious prompt. A user could therefore encounter the attack during normal editing. The K/V objective offers a concrete way to study this risk, and I see value in developing defenses for this setting.

  2. The experiments cover 14 pipelines, several budgets, and a second dataset, with comparisons and controls in the appendix. I found the first-frame video example particularly useful: it shows how an image perturbation can propagate through a generation sequence. This breadth supports weak acceptance, although some pipelines may share pretrained components.

Critical weaknesses
  1. SHADM constructs perturbations using a local surrogate UNet without querying the victim model, which is a reasonable zero-query, transfer-based setting. However, the paper should clarify whether any prior knowledge of the victim's architecture or editing pipeline is assumed.

  2. The evidence for structural hijacking is not yet conclusive. Although the outputs appear to shift toward the target category, the severe degradation makes their semantic content largely unrecognizable. It is therefore unclear whether SHADM achieves meaningful target-specific structural transfer or merely induces generic image corruption. Moreover, Table A6 shows similar performance for K-only and KV, leaving the benefit of jointly matching K and V insufficiently justified.

  3. The optimization tools are standard, so the contribution should be framed around the K/V objective and the zero-query setting. Section 3.3 does not establish a fundamental difference from PGD. The causal claims also seem too strong: K-only is close to KV in Table A6 (LPIPS 0.300 vs. 0.301; ASR 31.22% vs. 33.17%). This does not show that both K and V are necessary.

  4. Several evaluation details need clarification. The stated pixel budget does not match the [-1,1] normalization, and Table A31 reports an overall ASR below both subgroup rates. Detector agreement is also not a substitute for independently labeled ground truth. These issues affect how reliably the reported success rates and defense results can be interpreted.

Questions for authors
  1. Does the attacker require any prior knowledge of the victim’s architecture, pretrained weights, or editing pipeline beyond access to a local surrogate model?

  2. What is the key methodological novelty of SHADM beyond applying K/V feature matching to offline transfer attacks on diffusion models?

  3. What evidence demonstrates that K/V feature matching causes the intended structural or semantic hijacking, rather than generic output degradation? Could the authors clarify this distinction through controlled comparisons?

  4. How are the budget and step size converted between [0,255] and [-1,1]? Please specify the clipping convention, surrogate checkpoint, and whether noise is fixed across the 200 optimization steps.

  5. Please reconcile Table A31 and provide the positive counts and denominators. With the stated 155/50 split, FastDrag’s subgroup rates imply about 4.61% overall, not 2.68%. Do Tables A26/A29 use the default or alternative target, and would correcting or relabeling these results change the main comparisons?

  6. Please specify the detector checkpoints, thresholds, and reference labels for Table A27, including how images are processed for QwenGuard. Which IMPRESS code and settings were used? The current bibliography entry appears to refer to a KV-storage system.

7 comments
AuthorsOpenReview ↗

Reply to Reviewer [1/3]

Thank you for recognizing the relevance of the zero-query, image-only threat model and the breadth of the evaluation. The central contribution is a target-conditioned K/V intervention that enters through the source image and propagates through ordinary editing operations. We address the attack assumptions, mechanism, and evaluation protocol point by point.

1. Attacker knowledge and the meaning of “single-shot”

SHADM requires no prior knowledge of the victim's architecture, pretrained weights, or editing algorithm. The attacker has differentiable access only to a local surrogate and controls the source image. The construction uses the pretrained Stable Diffusion v1.5 UNet and the VAE for latent encoding. The victim later processes the resulting image using its ordinary prompt and editing pipeline. Perturbation construction uses no victim outputs, gradients, or intermediate activations (Sections 1 and 3, and Appendix A.4).

“Single-shot” describes the one-time delivery of the adversarial image to the victim. The 200 optimization steps take place entirely on the local surrogate. Appendix A.2.1 specifies the construction hyperparameters. The target-model probes added in Appendix A.6 are performed after the attack inputs are fixed and serve only to evaluate transfer.

2. Methodological novelty and the role of optimization

The contribution of SHADM is the representation it targets and the image-only route by which that intervention reaches downstream editing. Sections 3.1–3.2 match the chosen target's self-attention keys and values in the surrogate's up-sampling blocks under null-text conditioning. Denoting these layers by U\mathcal{U}, the objective is

LSHL(x)=1∣U∣∑ℓ∈U(∣Kℓ(zS(x))−Kℓtgt∣22+∣Vℓ(zS(x))−Vℓtgt∣22).\mathcal{L}_{\mathrm{SHL}}(x)=\frac{1}{|\mathcal{U}|}\sum_{\ell\in\mathcal{U}}\left(\\|K_\ell(z_S(x))-K_\ell^{\mathrm{tgt}}\\|_2^2+\\|V_\ell(z_S(x))-V_\ell^{\mathrm{tgt}}\\|_2^2\right).

The target features and surrogate parameters remain fixed while the source pixels are optimized in the perturbation constraint. Projected gradient updates solve this optimization, pathwise differentiation propagates gradients through latent construction, and stop-gradient fixes the target branch. The scientific contribution lies in the K/V objective, the attack surface, and the demonstrated transfer without victim feedback.

The new matched-objective comparison in Table A32 isolates the choice of representation under shared construction settings. K/V achieves Q16 ASRs of 33.17%, 37.07%, and 16.59% on DragLoRA, FastDrag, and MasaCtrl, compared with 27.32%, 30.73%, and 12.68% for pre-projection feature matching, and 21.46%, 24.39%, and 8.78% for latent matching. The component ablation in Table A6 additionally shows K/V exceeding K-only by 1.95, 3.42, and 1.95 percentage points at the three evaluated budgets. These results support K/V as an effective joint objective. Tables A36 and A6 indicate that Q/K/V requires more computational resources than K/V but similar performance to K/V, thus the latter is preferable.

3. Structural targeting and generic image corruption

The revised evidence links the chosen reference to both internal target-model features and output safety signals. Section 3.4 traces the propagation path through latent encoding, LoRA adaptation, motion supervision, and point tracking in DragLoRA. Appendix A.2.3 identifies source-feature reuse in control-based methods such as MasaCtrl. The new experiments test the reference dependence of that intervention directly.

First, the target-model probes in Table A33 show directional feature transfer. NSFW-target K/V matching reduces NSFW-reference MSE by 15.80%, 18.00%, and 11.60% across DragLoRA, FastDrag, and MasaCtrl, while changing to a non-NSFW reference shifts the larger reductions toward that reference: 17.60%, 20.30%, and 13.80%. Random perturbations produce reductions of at most 1.54%.

Second, Table A34 compares safety responses at closely matched mean source-to-output LPIPS. NSFW-target K/V matching exceeds its corresponding noise control by 16.10, 17.07, and 7.32 ASR percentage points. Switching to a non-NSFW target yields Q16 rates of 3.41%, 4.39%, and 3.90%, versus 19.02%, 19.02%, and 8.78% for the corresponding controls. The largest mean LPIPS difference in these six comparisons is 0.002.

Third, Table A32 shows higher Q16 ASR with lower LPIPS, (1−CLIP)(1-\mathrm{CLIP}), and FID than intermediate-feature matching on every tested target. These comparisons show reference-dependent feature changes and detector responses at comparable mean output distortion. Our use of “structural hijacking” refers to this reference-directed intervention in representations that support editing consistency.

AuthorsOpenReview ↗

Reply to Reviewer [2/3]

4. Budget coordinates, clipping, surrogate, and noise

The budget and step size are defined on the normalized image tensor x∈[−1,1]x\in[-1,1]. With u=(x+1)/2∈[0,1]u=(x+1)/2\in[0,1], the conversion is

ϵu=ϵx/2,ηu=ηx/2.\epsilon_u=\epsilon_x/2,\qquad \eta_u=\eta_x/2.

Thus, the reported tensor budgets 8/2558/255, 12/25512/255, and 16/25516/255 correspond to 4, 6, and 8 intensity levels on the original [0,255][0,255] scale, respectively. The tensor step size ηx=1/255\eta_x=1/255 corresponds to 0.5/2550.5/255 in [0,1][0,1], or 0.5 original intensity level. The revised Section 4 explicitly states this coordinate convention.

Each update is projected onto the intersection of the source-centered perturbation box and the valid image domain, as in Eq. (14). Coordinate ii is clipped to

[max⁡(−1,xisrc−ϵx),;min⁡(1,xisrc+ϵx)].[\max(-1,x_i^{\mathrm{src}}-\epsilon_x),\\;\min(1,x_i^{\mathrm{src}}+\epsilon_x)].

The surrogate is the pretrained Stable Diffusion v1.5 UNet. The construction uses S=261S=261 and 200 updates. Appendix A.5 specifies VAE posterior means, target diffusion noise sampled once, and cached target K/V features. Source diffusion noise is resampled at each optimization step. In each update, the sampled noise is held constant during differentiation. These settings are shared by the K/V, feature, and latent objectives in the controlled comparison.

5. Reconciliation of the original Table A31 and explicit detector counts

The non-NSFW-target evaluation is now reported with explicit positive counts and denominators in the revised Table A35, replacing the original Table A31. The existing generated images and image-distance measurements are retained, and the outputs are evaluated with the documented Q16 and NudeNetClassifier rules in Appendix A.8.

Target modelQ16: still-lifeQ16: living subjectsQ16: all outputsNudeNetClassifier: all outputs
FastDrag7/155 (4.52%)2/50 (4.00%)9/205 (4.39%)0/205 (0.00%)
MasaCtrl6/155 (3.87%)2/50 (4.00%)8/205 (3.90%)0/205 (0.00%)
DragLoRA5/155 (3.23%)2/50 (4.00%)7/205 (3.41%)1/205 (0.49%)

For each model, the overall numerator is exactly the sum of the two subgroup numerators and the denominator is 155+50=205155+50=205. In particular, FastDrag's overall rate is 9/205 = 4.39%, replacing the original 2.68%. The single NudeNetClassifier-positive DragLoRA output belongs to the still-life subset, where the count is 1/155 (0.65%).

The revised condition-specific comparisons pair these non-NSFW results with the default NSFW-reference K/V results in Tables A32 and A34: 33.17%, 37.07%, and 16.59% for DragLoRA, FastDrag, and MasaCtrl. The target-switching conclusion is therefore supported by explicit conditions and reconciled counts, together with the matched-objective and noise-control experiments.

AuthorsOpenReview ↗

Reply to Reviewer [3/3]

6. Detector configuration, reference labels, and defense evaluation

The revised manuscript specifies the detector and defender configurations used for the count-based evaluation.

  • Q16: OpenAI CLIP ViT-L/14 with its learned prompt pair and normalized image/prompt features. An output is positive when the inappropriate-class score exceeds the non-inappropriate-class score. The same rule is applied to clean, attacked, and control outputs.
  • NudeNetClassifier: NudeClassifier() in ONNX Runtime with the default v2 classifier_model.onnx. Images are converted to RGB, resized to 256×256256\times256, and divided by 255 to obtain [0,1][0,1] inputs. The batch size is 4. A positive decision requires unsafe score greater than safe score, equivalent to unsafe probability above 0.5 for the normalized binary outputs.
  • QwenGuard-v1.2-7B: it is based on Qwen2.5-VL-7B-Instruct. RGB images are passed directly to AutoProcessor with the fixed O3 sexual-content/O4 nudity policy, without an intermediate caption. We use min_pixels=3136=3136, max_pixels=262144=262144, BF16, deterministic decoding, and at most 192192 new tokens. A valid JSON response is positive only for rating Unsafe with category O3 or O4. Safe is negative. Invalid responses remain missing, not safe. No separate confidence threshold is applied.
  • IMPRESS: we uses the image-purification implementation from the repository of the authors (https://github.com/AAAAAAsuka/Impress), with the core routine impress.py. Our run_impress_dragbench.py adapts this routine for batched image processing, retaining the VAE reconstruction-consistency MSE and VGG-LPIPS hinge objective, Adam, and cosine learning-rate scheduling. The purification configuration uses the SD2.1-base VAE, FP16, 30003000 purification iterations, learning rate 0.010.01, LPIPS threshold 0.10.1, penalty coefficient 0.10.1, initialization-noise standard deviation 0.10.1, seed 00, and batch size 88, with image values clipped to [−1,1][-1,1]. The LPIPS threshold constrains perceptual deviation during purification.

The reference labels for Table A27's pairwise error columns are the corresponding companion detector's decisions. For example, the DragLoRA comparison against NudeNet gives FPR = 3/(205 − 39) = 1.81% and FNR = 1/39 = 2.56%. These statistics quantify agreement and disagreement between Q16 and the companion detector. Throughout our response, ASR denotes the detector-positive fraction. The new controlled evidence additionally tests reference direction inside target-model features (Table A33), responses under closely matched mean distortion (Table A34), and counts under non-NSFW targeting (Table A35).

For preprocessing defenses, Appendix A.3.5 specifies Gaussian smoothing with radius r=2r=2 and JPEG compression at quality 75. Their Q16 results are reported in Tables A17 and A21. For example, FastDrag retains rates of 4.88% after smoothing and 14.63% after JPEG, and MasaCtrl retains 13.17% and 17.56%, respectively. These experiments measure residual detector-positive outputs under the stated preprocessing settings.

Reviewer_x8mXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SHADM, a single-shot black-box adversarial attack against diffusion models. The method constructs adversarial images by minimizing the discrepancy between the self-attention Key and Value features of benign source images and malicious target images using a surrogate UNet. The resulting perturbations are transferred to victim diffusion models without requiring access to their parameters or outputs. Experiments across 14 diffusion models demonstrate substantial image degradation and increased NSFW generation rates.

Critical strengths

Practical and Relevant Threat Model. The paper investigates a meaningful security scenario where attackers can manipulate source images but have no access to the victim models. The zero-query, single-shot setting is relevant to real-world image editing services and highlights potential vulnerabilities in diffusion-based applications.

Broad Experimental Coverage. The evaluation covers 14 diffusion models with different editing mechanisms and architectures. The authors also provide comparisons with existing attacks, multiple evaluation metrics, and additional experiments on larger datasets, offering a relatively comprehensive empirical assessment.

Critical weaknesses

Insufficient Evidence for Cross-Architecture Transferability. Although the paper evaluates 14 diffusion models, many of them share similar underlying diffusion backbones, potentially overstating the architectural diversity of the evaluation. More importantly, the attack achieves only marginal NSFW success on PixArt, suggesting that its effectiveness does not consistently transfer to substantially different architectures. Evaluations on independently trained diffusion backbones are needed to substantiate the claimed model-agnostic transferability.

Limited Methodological Novelty. The proposed method essentially combines feature-matching objectives with standard gradient-based adversarial perturbation optimization. Aligning intermediate representations through MSE and applying projected gradient updates are well-established techniques, while the reparameterization trick is also standard practice in differentiable generative models. The paper does not sufficiently clarify what constitutes a fundamentally new methodological contribution beyond applying these existing techniques to attention K/V features.

Questions for authors
  1. How does SHADM perform against diffusion models with independently trained backbones and substantially different architectures?

  2. What improvements does K/V feature matching provide over conventional intermediate-feature or latent-representation matching under identical perturbation and optimization budgets?

  3. Can the authors provide direct evidence that the victim models' intermediate structural representations are hijacked, rather than merely showing degraded outputs?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer [1/2]

Thank you for recognizing the practical relevance of the threat model and the breadth of the evaluation. SHADM targets the feature consistency mechanisms used by diffusion editing: an image-only intervention aligns surrogate K/V features with a chosen target and transfers the resulting disruption without victim queries or a malicious prompt. The revision adds experiments across pretrained backbone families, comparisons between matching objectives under identical construction budgets, and direct probes of target-model representations.

1. Transfer across pretrained backbones and architectures

The same fixed attacks constructed with the Stable Diffusion v1.5 (SD v1.5) pretrained UNet transfer to SANA-based DIM-4.6B-Edit (Zeng et al., 2026) and FLUX-Kontext-based DreamOmni2 (Xia et al., 2025a), alongside PixArt. The revised Appendix A.4 and Table A31 evaluate these three targets on 205 DragBench sources. All targets receive the same surrogate-generated inputs, with ϵ=16/255\epsilon=16/255 on [−1,1][-1,1] image tensors, which is equivalent to only ϵ=8/255\epsilon=8/255 on the [0,1][0,1] coordinates.

Target modelPretrained backboneLPIPS: clean → SHADMQ16 ASR: clean → SHADM
PixArtPixArt diffusion Transformer0.092 → 0.1752.44% → 4.39%
DIM-4.6B-EditSANA0.180 → 0.2172.44% → 6.34%
DreamOmni2FLUX-Kontext0.160 → 0.1912.93% → 5.37%

Relative to the clean baselines, the Q16 ASR of SHADM increases by 79.9%, 159.8%, and 83.3% for PixArt, DIM-4.6B-Edit, and DreamOmni2, respectively. All three also show increases in (1−CLIP)(1-\mathrm{CLIP}) and FID. This directly extends the transfer evaluation to additional pretrained backbone families while holding attack construction and attack inputs fixed. The original fourteen-model benchmark covers editing pipelines. The added SANA and FLUX-Kontext experiments provide further tests of transfer beyond that pipeline coverage. The results report the strength of both editing disruption and detector-defined safety effects for each tested target.

2. Contribution and comparison with intermediate-feature and latent matching

The methodological contribution is a target-conditioned objective at the representations used for feature propagation, together with its source-image injection mechanism. SHADM extracts self-attention K/V features from the surrogate's up-sampling blocks at an intermediate diffusion timestep, using null-text conditioning, and matches them to a selected target's features (Sections 3.1–3.2). The source image is the optimized variable. The surrogate parameters and target-reference features remain fixed. Projected gradient updates and pathwise differentiation implement this objective. Its value is established by the controlled representation comparisons.

The revised Table A32 directly answers the requested comparison under identical perturbation and optimization budgets. The feature baseline matches the features FF entering the same self-attention layers immediately before K/V projection. The latent baseline matches the noisy source and target VAE latents. All three objectives share the source and reference sets, VAE, perturbation budget, 200 steps, S=261S=261, noise-sampling protocol, step-size convention, and projection. The following table presents the corresponding Q16 ASRs.

Target modelK/V matchingFeature FF matchingLatent matching
DragLoRA33.17%27.32%21.46%
FastDrag37.07%30.73%24.39%
MasaCtrl16.59%12.68%8.78%

K/V improves over feature matching by 5.85, 6.34, and 3.91 percentage points, and over latent matching by 11.71, 12.68, and 7.81 points. K/V also produces lower LPIPS, (1−CLIP)(1-\mathrm{CLIP}), and FID than feature matching on every target. Thus the higher detector-positive rate is achieved with less measured output distortion than the feature baseline. The choice of projected attention representations produces a consistent downstream advantage under the matched construction conditions.

AuthorsOpenReview ↗

Reply to Reviewer [2/2]

3. Direct evidence of hijacked intermediate structural representations inside target models

The revised Appendix A.6 now measures reference-directed changes in the target models' own K/V features. These probes are performed only after the attack inputs have been fixed, so they provide evaluation evidence without entering the zero-query construction process.

For DragLoRA, FastDrag, and MasaCtrl, respectively, Table A33 reports:

  • With the NSFW attack reference, MSE to the NSFW-reference K/V features decreases by 15.80%, 18.00%, and 11.60%. The reductions toward the non-NSFW reference are only 1.10%, 0.39%, and 1.00%.
  • Switching to a non-NSFW attack reference redirects the larger reductions toward that reference: 17.60%, 20.30%, and 13.80%, compared with 3.80%, 4.40%, and 3.00% toward the NSFW reference.
  • Random perturbations yield reductions of at most 1.54% across the tested models and references.

The revised Appendix A.7 further compares attacks with noise controls whose mean source-to-output LPIPS differs by at most 0.002 across all six attack/control pairs. For the NSFW-target attacks, Q16 ASRs are 33.17% versus 17.07% on DragLoRA, 37.07% versus 20.00% on FastDrag, and 16.59% versus 9.27% on MasaCtrl (Table A34). With non-NSFW targeting, the rates become 3.41%, 4.39%, and 3.90%, while the corresponding noise controls give 19.02%, 19.02%, and 8.78%.

In summary, these experiments connect reference selection to the direction of internal feature changes and to the downstream detector response at comparable mean perceptual distortion. Feature matching also induces reference-directed K/V changes in Table A33. Table A32 establishes the additional downstream advantage of directly matching K/V under the same construction conditions. This provides both an internal measurement and controlled output evidence for the structural interpretation.

AuthorsOpenReview ↗

Deadline for Discussion is Approaching in Less than 2 Days

Dear reviewer x8mX,

We have carefully considered all the valuable points and concerns you raised in your initial review through the initial rebuttal. As the deadline for discussion is approaching in about 2 days only, we respectfully ask whether our response has addressed your concerns, or you still have questions to raise. Your comments are highly valuable to us. Thank you once again for your professional time and insightful effort in reviewing our submission.

Best regards,

Authors

Comments on the paper

2 comments
Area_Chair_v5SUOpenReview ↗

[Urgent] Please check the authors' rebuttal soon

Dear Reviewers,

The authors have posted their responses to the issues you raised. As the discussion period will end soon, please take a look and have a discussion with the authors at your earliest convenience. Thank you very much!

Best, AC

AuthorsOpenReview ↗

Summary of Discussion

Dear Area Chair,

Thank you for overseeing the discussion. SHADM shows that an attacker can influence what a diffusion editor generates for another user without ever querying that editor. The attacker provides an ordinary-looking source image and takes no further part. The attacker needs no access to the editor's parameters or outputs and no knowledge of the user's prompt. When the user later requests a routine edit in their chosen editor, the editor can distort the subject's appearance and introduce disturbing content the user never asked for.

On DragBench, we limit the change in each color channel to eight intensity levels on a 0–255 scale. Q16 flags a larger share of outputs as inappropriate in all 14 evaluated pipelines. The average flag rate across pipelines rises from 2.56% with clean inputs to 28.7% with altered inputs (Table A3). On the separate 1000-image PRD benchmark, the increase ranges from 51 to 71 percentage points across the three tested editors (Table 2). On DragBench, SHADM also yields a higher Q16 flag rate than noise controls at comparable mean output distortion in each of the three editors tested (Table A34).

Reviewer WD7X asked whether the changes could produce recognizable harmful content. At a higher perturbation budget, both DragLoRA and FastDrag give the same sculpture a disturbing creature-like face (Figure A15). After two rounds of discussion, WD7X indicated that ``My concerns have been almost completely resolved, and I have changed the rating to positive.''

SHADM reveals a vulnerability in how diffusion editors use features from the source image to preserve a subject's appearance. In all three editors we probed, changing the attacker's reference image shifted the internal attention features toward those of the new reference (Table A33). We believe this vulnerability deserves the ML community's attention and that routine edits of third-party images should be a standard safety test for diffusion editors.

Thank you for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.