acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Image Immunization for Transferable and Purification-Robust Protection

Abstract

Image immunization has been actively studied to protect images from unauthorized diffusion-based manipulation. For a practical deployment, however, image protection requires two key properties: black-box transferability to unknown generative models and robustness against perturbation purification, which have largely been studied along separate directions. In this paper, we present a unified immunization framework that jointly targets both requirements. Our controlled analysis identifies early VAE encoder representations as a promising common attack surface: they exhibit strong cross-model correspondence, and perturbations optimized at these stages better retain their induced latent shifts after purification. Based on these observations, we directly disrupt early VAE encoder representations, rather than relying on architecture-specific denoisers or explicit purification-aware designs. We further introduce a relation-disruption objective that disrupts the internal cross-channel structure of early representations. Extensive experiments demonstrate strong black-box transferability across heterogeneous diffusion-based generative models within multiple downstream manipulation tasks, and robustness against diverse purification techniques, with low computational overhead.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Image Immunization, Diffusion model, AI safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_wGVUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an image immunization framework targeting black-box transferability and purification robustness. The core design targets early VAE features and combines feature differences with interference from cross-channel relationships to optimize defensive perturbations. Experiments across diverse generative models, manipulation tasks, and purification methods demonstrate effective protection with low computational overhead.

Critical strengths
  1. The research question has clear practical value, where black-box transferability and purification robustness are more closely aligned with real-world applications.
  2. The experiments are relatively well-designed.
Critical weaknesses
  1. Early feature perturbation (DiffusionGuard, ICLR'25) and VAE representation perturbation (PCA, TIFS'24) have already been explored in existing research. The advantages of combining these two methods have not yet been fully analyzed, so their integration primarily depends on engineering practice and empirical exploration, lacking substantial technical innovation.
  2. The paper argues that early VAE representations provide a unified attack surface for both black-box transferability and purification robustness. However, different generative models employ VAEs with distinct feature distributions, latent spaces, and encoding mappings. So, why perturbations optimized on one VAE can consistently disrupt other heterogeneous VAEs while remaining effective after purification?
Questions for authors
  1. The relationship perturbation loss is constructed using cross-channel cosine similarity, but the paper does not fully explain why disrupting the internal channel relationships in the surrogate model can enhance transferability to unknown models.
  2. The paper uses PWCCA to explain the cross-model transfer advantages of early VAE features, but it remains unclear whether higher representational correlation implies stronger adversarial transfer capabilities.
  3. Lack of comparison with the most relevant work: 10.1109/TIFS.2025.3639957
3 comments

[W1, Q3] Distinction from DiffusionGuard and PCA

Thank you for the valuable comment. We first clarify our distinction from DiffusionGuard and PCA.

DiffusionGuard targets the early stage of the diffusion denoising process, specifically the first denoising step, by maximizing the norm of the noise predicted by the diffusion denoiser. In contrast, early in our method refers to network depth within the VAE encoder, specifically B0. Our optimization targets this earliest intermediate representation without involving the denoising network or selecting diffusion timesteps.

PCA [A1] targets the VAE encoder’s final output by manipulating the posterior mean and variance through a KL-divergence-based objective to induce collapse. In contrast, our method targets early intermediate encoder features, without optimizing the final posterior parameters.

To further examine the transferability benefit of targeting early VAE encoder representations, we additionally compare our method with PCA and DiffusionGuard.

Table A. Comparison with PCA and DiffusionGuard

SettingSurrogate → TargetPCA+PCA−DiffusionGuardOurs
White-boxSD1.4 → SD1.40.580‾\underline{0.580}0.5920.4060.553
SD3 → SD30.4730.6090.5230.563‾\underline{0.563}
FLUX.1 → FLUX.10.5210.5830.4890.558‾\underline{0.558}
Black-boxSD1.4 → SD30.4160.553‾\underline{0.553}0.4190.593
SD1.4 → FLUX.10.3870.468‾\underline{0.468}0.3820.567
SD3 → SD1.40.391$ \underline{0.479}$0.3460.540
SD3 → FLUX.10.3730.5290.4180.523‾\underline{0.523}
FLUX.1 → SD1.40.456‾\underline{0.456}0.4500.4310.546
FLUX.1 → SD30.4890.493‾\underline{0.493}0.4840.585

As shown in Table A, our method achieves strong black-box protection. DiffusionGuard’s weaker transferability is consistent with our analysis that relying on denoiser representations can limit cross-model transfer. Compared with PCA’s final encoder-output objectives, our method targets only early encoder representations and achieves superior overall black-box performance, further supporting our choice of attack surface.

We also acknowledge that encoder-based protection is well established. PhotoGuard’s encoder attack, PCA, DCT-Shield, Editshield, BlurGuard’s encoder-based configuration, and PID have already demonstrated the value of encoder-based objectives for reducing computational cost or dependence on the denoising network. We therefore do not claim encoder-only optimization itself as our contribution.

Our contribution is to identify which encoder representations provide a favorable attack surface for both black-box transferability and purification robustness, and to investigate why. To this end, we conduct controlled depth-wise comparisons under a common objective, analyze cross-encoder correspondence using clean-PWCCA and shift-PWCCA, and measure the survival of perturbation-induced latent shifts after purification. These analyses motivate targeting early encoder representations. We have included the corresponding result in the revision, and will revise the discussion to clarify these distinctions and the scope of our contribution.

[W2, Q2] Cross-VAE Transferability and Purification Robustness

Thank you for these constructive comments. We agree that higher representational correspondence alone does not establish adversarial transferability (L237-241). Our intention was not to establish a direct causal relationship between PWCCA and attack transferability, but to provide complementary evidence for the observed black-box protection behavior. The clean-PWCCA, shift-PWCCA, and latent shift survival analyses in Fig. 3 complement this evaluation by examining cross-encoder correspondence and the persistence of perturbation-induced effects after purification.

Importantly, our analysis goes beyond clean representation correspondence. Despite differences in VAE architectures and latent spaces, perturbation-induced changes also exhibit stronger cross-encoder correspondence at earlier stages (shift-PWCCA, Fig. 3b). This is consistent with the stronger black-box transferability observed in Fig. 2a, suggesting that shared responses to perturbations may help explain the transferability of early-stage attacks.

To further investigate shared responses across encoders, we additionally conduct an input-gradient analysis in response to your concern. At each stage, we compute the gradient of the ℓ2\ell_2 norm of the channel vector at the central spatial location with respect to the RGB input. We then compare the full input gradients across encoders for the same clean image using cosine similarity. These diagnostic probes characterize input sensitivity without assuming one-to-one channel correspondence; they are not attack-loss gradients.

Table B. Cross-encoder input-gradient cosine similarity

DepthSD1.4 ↔ SD3SD1.4 ↔ FLUX.1SD3 ↔ FLUX.1
B00.527 ± 0.2070.515 ± 0.2900.525 ± 0.384
B10.417 ± 0.2250.315 ± 0.3430.389 ± 0.217
B20.419 ± 0.1250.134 ± 0.1920.197 ± 0.184
B3−0.097 ± 0.197−0.020 ± 0.1110.130 ± 0.075
B4−0.070 ± 0.171−0.003 ± 0.0370.083 ± 0.083

As shown in Table B, B0 exhibits the highest mean gradient alignment for all three encoder pairs. This indicates that activation magnitudes at B0 respond to more similar input directions across encoders. Together with shift-PWCCA and the controlled depth-wise evaluation, these observations support the interpretation that shared early-stage input sensitivities and corresponding perturbation-induced responses may contribute to the observed transferability.

We also quantify the association between shift-PWCCA and actual black-box protection performance. For each surrogate–target pair, we compute Spearman and Pearson correlations across B0–B4 using results obtained under our method.

Table C. Correlation between shift-PWCCA and black-box protection performance (LPIPS)

Surrogate → TargetSpearman ρPearson r
SD1.4 → SD30.9000.915
SD1.4 → FLUX.10.9000.982
SD3 → SD1.40.9000.977
SD3 → FLUX.10.7000.720
FLUX.1 → SD1.41.0000.973
FLUX.1 → SD30.9000.948

As shown in Table C, both correlation measures are positive across all six pairs, with Spearman ranging from 0.700 to 1.000 and Pearson r from 0.720 to 0.982. Thus, attack depths with stronger perturbation-induced cross-encoder correspondence also tend to yield stronger transferred protection. This quantitatively supports the relationship between the shift-PWCCA analysis and the depth-wise protection results.

For purification robustness, latent shift survival (Fig. 3c) measures how much of the perturbation-induced change in the final VAE latent remains after purification, accounting for both directional alignment and retained magnitude. Perturbations targeting early representations show greater latent shift survival, consistent with the protection-retention results in Fig. 2b.

Together, these analyses support early VAE representations as a favorable common attack surface: they exhibit stronger cross-encoder correspondence and input-gradient alignment, while perturbations optimized at these stages better preserve their effects on the final latent after purification. We have incorporated the additional gradient analysis and its detailed protocol into the revised manuscript. We sincerely thank you for helping us strengthen the analysis and clarify this rationale.

[Q1] Clarification on Relation Disruption Loss

Thank you for raising this concern. We distinguish the complementary roles of attack-surface selection and relation disruption. Our rationale for transferability primarily comes from targeting early VAE representations, particularly B0, supported by the controlled depth-wise evaluation and cross-encoder correspondence analyses in Figs. 2a and 3. We introduce LRelL_{Rel} to complement LDisL_{Dis} at this favorable attack surface.

While LDisL_{Dis} directly maximizes feature discrepancy, it does not explicitly require changes in inter-channel relationships. Since subsequent layers combine spatially weighted information across channels, disrupting these relationships can alter how early-stage perturbations are transformed downstream, potentially amplifying distortion in the final VAE latent supplied to the diffusion denoiser. This motivates applying LRelL_{Rel} to the normalized relationships between channel activation maps within B0’s residual branches.

To examine this downstream effect, we compare representation changes under the two objectives while keeping the attack location fixed at B0. Because feature scales vary across VAEs and encoder depths, we normalize the representation MSE for each image by the mean squared activation of the corresponding clean representation before averaging.

Table D. Downstream representation distortion measured by scaled MSE of B0-targeted attacks

RepresentationWB LDisL_{Dis}WB LRel+LDisL_{Rel}+L_{Dis}BB LDisL_{Dis}BB LRel+LDisL_{Rel}+L_{Dis}
B10.95600.91260.85940.8644
B20.73390.77300.71060.7450
B30.30150.34600.30160.3201
B40.26230.28930.26010.2704
Latent0.37830.39740.38900.4037

Table E. Reconstruction distortion relative to clean reconstructions under B0-targeted attacks

SettingSurrogate → TargetMetricLDisL_{Dis}LDisL_{Dis} + LRelL_{Rel}
White-boxSD1.4 → SD1.4PSNR ↓24.24923.744
LPIPS ↑0.48600.5255
SD3 → SD3PSNR ↓25.66025.287
LPIPS ↑0.45380.4798
FLUX.1 → FLUX.1PSNR ↓25.58325.280
LPIPS ↑0.48860.5172
Black-boxSD1.4 → SD3PSNR ↓25.59624.432
LPIPS ↑0.48270.5220
SD1.4 → FLUX.1PSNR ↓25.66025.188
LPIPS ↑0.48370.5245
SD3 → SD1.4PSNR ↓24.36824.105
LPIPS ↑0.45470.4795
SD3 → FLUX.1PSNR ↓26.01125.596
LPIPS ↑0.45120.4775
FLUX.1 → SD1.4PSNR ↓24.22724.005
LPIPS ↑0.49220.5144
FLUX.1 → SD3PSNR ↓25.17124.991
LPIPS ↑0.48910.5106

As shown in Table D, adding LRelL_{Rel} generally increases distortion in deeper encoder representations and, most importantly, in the final VAE latent under both white-box and black-box settings. Table E further confirms this effect at the reconstruction level: LDis+LRelL_{Dis}+L_{Rel} consistently produces lower PSNR and higher LPIPS than LDisL_{Dis} alone across all white-box and black-box surrogate-target pairs. Together with the improved black-box editing protection reported in Table 6, these results support our motivation for designing LRelL_{Rel} to complement direct feature displacement by propagating stronger distortion to the latent representation supplied to the diffusion denoiser.

As described in L314–316, this rationale does not assume shared relation matrices or corresponding channel indices across encoders. B0 provides favorable conditions for transfer, while LRelL_{Rel} strengthens disruption at that surface and its downstream effects. We sincerely thank you for helping us clarify this rationale. We have added this motivation and the accompanying analysis to the revised manuscript, with the additions highlighted in blue.


[A1] A Gray-Box Attack Against Latent Diffusion Model-Based Image Editing by Posterior Collapse. IEEE TIFS, 2025.

Reviewer_zNRhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a simple approach to make an image protection model more transferable and purification-robust. In their analysis, early layers of VAE encoder is better in terms of both aspects. To do so, they introduce dis and rel losses on the B0 block of VAE encoder. The experimental results show that the proposed protection is cheap, transferable, and purification-robust. Then also yields I2V and VQ encoder results, which is essential in this field.

Critical strengths

Overall analysis and the resulting observation are insightful e.g. Figure 1 illustrates the systemically analysis of transferability. In their analysis and comparison, broad range of baselines and setups are used so that the results are more reliable and insightful.

The proposed method is very straightforward and cheap since they only use a small portion of the encoder in the protection optimization hence cheaper than PhotoGuard, which is believe the one of the most simple and effective protection scheme.

The performance on the purification-robust and transferability looks solid.

Critical weaknesses

While the proposed method is simple yet effective, I have some concerns.

  • Recent works [1, 2, 3] have discussed VAE encoders in related contexts (although I admit that they do not exactly match the setting of this work), and I think it would be good to discuss them together. Also, LrelL_{rel} seems to have a similar form to the Gram loss widely used in style transfer, so I think it would be useful to discuss the relationship between them.
  • While the experiments and analyses are generally well conducted, there are still some missing points. For example, in the black-box transfer experiment and analysis, SD1.4 as the surrogate and SD1.5-Inpainting and InstructPix2Pix seem to use the same VAE encoder. If so, I wonder whether this can really be considered a black-box setting. Also, I think results on general purification methods such as Noisy Upscaling [4] or img2img-based approaches [5] should be reported.
  • Although the authors analyze the B0 block using PWCCA, I feel that this is somewhat of a post-hoc correlation observation. I think the prior intuition and the depth-wise analysis are useful analysis and motivation, but more in-depth analysis would be needed. For example, in my opinion, one possible reason why B0 transfers well is simply that it is located at an early stage of the network. If it learns relatively local filters, as in the conventional interpretation of early network layers, such features may transfer relatively well. It would be useful to include some dissection-style analysis related to this point. In fact, otherwise, I find it difficult to understand why it transfers well even to FLUX. I recommend including more analysis on this aspect.
  • Also, in the comparison within the same backbone model (e.g., SD1.5 -> SD1.5), why does the proposed method perform better than PhotoGuard? It is unclear whether this comes from using only the B0 layer or from the additional loss. I think this should be clarified.

[1] StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations, NeurIPS2025

[2] Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models, CVPR25

[3] Anti-I2V: Safeguarding your photos from malicious image-to-video generation, CVPR26

[4] Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI, ICLR25

[5] Off-The-Shelf Image-to-Image Models Are All You Need To Defeat Image Protection Schemes, SaTML26

Questions for authors

In the weaknesses I raised some of the questions:

  • in the comparison within the same backbone model (e.g., SD1.5 -> SD1.5), why does the proposed method perform better than PhotoGuard? It is unclear whether this comes from using only the B0 layer or from the additional loss. I think this should be clarified.
  • Why B0 is good?
7 comments

[W1-1] Distinction from VAE Encoder-based Methods

Thank you for pointing this out. StyleGuard [A1] uses an instance-normalization (IN)-inspired style objective on the final VAE latent features, encouraging their means and variances to approach those of a different-style target image while departing from the original image. FastProtect and Anti-I2V both use layer-wise intermediate VAE encoder features. FastProtect additionally incorporates a final-latent objective alongside its intermediate-feature objectives, while Anti-I2V uses intermediate features from all VAE encoder stages, together with denoising features.

Our method instead targets only the early VAE encoder block B0, without incorporating final-latent, later-stage encoder, or denoiser features into its objectives. We compare our method with StyleGuard, FastProtect, and an encoder-only variant of Anti-I2V (denoted as Anti-I2V-E) in Table A. Since FastProtect’s pretrained perturbation weights are not publicly available, we implement an image-specific optimization variant using its combined final-latent and intermediate-feature objective, denoted as FastProtect-IS.

Table A. Comparison with StyleGuard, FastProtect-IS, and Anti-I2V-E

MethodBlack-box mean LPIPS ↑
StyleGuard0.453 ± 0.032
FastProtect-IS0.505 ± 0.033
Anti-I2V-E0.464 ± 0.037
Ours0.559 ± 0.027

As shown in Table A, our method achieves the highest mean black-box LPIPS and the lowest standard deviation among the compared methods. These results support targeting early VAE representations for strong and consistent protection across unknown models, aligning with our primary goal of black-box transferability.

[W1-2] Clarification for Relation Disruption Loss

Thank you for raising this concern. Gram matrices are widely used in style transfer [A2] to summarize relationships between feature channels. Conventional Gram-based style transfer encourages a generated image to reproduce a reference style by minimizing the discrepancy between their Gram matrices:

\left|G(x_{\mathrm{gen}})-G(x_{\mathrm{style}})\right|_F^2.$$ On the other hand, our goal is to disrupt the selected early representation, including its internal channel relationships. Subsequent convolutional layers combine spatially weighted information across channels, so changing these relationships can alter how early-stage perturbations are transformed downstream, potentially amplifying distortion in the final VAE latent supplied to the diffusion denoiser. Motivated by this, we introduce $L_{Rel}$ to complement direct feature displacement. We construct a Gram matrix from channel-wise normalized residual-branch activation maps within B0 and maximize its discrepancy from the clean counterpart by minimizing: $$L_{Rel}= -\frac{1}{|\mathcal I_{\mathrm{B0}}|} \sum_{i\in\mathcal I_{\mathrm{B0}}} \frac{ \left|C^{(i)}(x_{\mathrm{adv}})-C^{(i)}(x)\right|_F }{ \left|C^{(i)}(x)\right|_F },$$ where C(i) is the normalized Gram matrix in B0. Thus, while style transfer matches a reference style, our objective explicitly changes the clean representation’s channel relationships without requiring a target image. Our additional analyses support this motivation. Specifically, we measure the MSE between clean and protected representations at downstream encoder stages and the final VAE latent, scaling it by the corresponding clean representation’s mean squared activation for each image before averaging to account for feature-scale differences across VAEs and encoder depths. **Table B. Downstream representation distortion measured by scaled MSE of B0-targeted attacks** | Representation | WB $L_{Dis}$ | WB $L_{Rel}+L_{Dis}$ | BB $L_{Dis}$ | BB $L_{Rel}+L_{Dis}$ | |---|---:|---:|---:|---:| | B1 | **0.9560** | 0.9126 | 0.8594 | **0.8644** | | B2 | 0.7339 | **0.7730** | 0.7106 | **0.7450** | | B3 | 0.3015 | **0.3460** | 0.3016 | **0.3201** | | B4 | 0.2623 | **0.2893** | 0.2601 | **0.2704** | | Latent | 0.3783 | **0.3974** | 0.3890 | **0.4037** | **Table C. Reconstruction distortion relative to clean reconstructions under B0-targeted attacks** | Setting | Surrogate → Target | Metric | $L_{Dis}$ | $L_{Dis}$ + $L_{Rel}$ | |---|---|---|---:|---:| | **White-box** | SD1.4 → SD1.4 | PSNR ↓ | 24.249 | **23.744** | | | | LPIPS ↑ | 0.4860 | **0.5255** | | | SD3 → SD3 | PSNR ↓ | 25.660 | **25.287** | | | | LPIPS ↑ | 0.4538 | **0.4798** | | | FLUX.1 → FLUX.1 | PSNR ↓ | 25.583 | **25.280** | | | | LPIPS ↑ | 0.4886 | **0.5172** | | **Black-box** | SD1.4 → SD3 | PSNR ↓ | 25.596 | **24.432** | | | | LPIPS ↑ | 0.4827 | **0.5220** | | | SD1.4 → FLUX.1 | PSNR ↓ | 25.660 | **25.188** | | | | LPIPS ↑ | 0.4837 | **0.5245** | | | SD3 → SD1.4 | PSNR ↓ | 24.368 | **24.105** | | | | LPIPS ↑ | 0.4547 | **0.4795** | | | SD3 → FLUX.1 | PSNR ↓ | 26.011 | **25.596** | | | | LPIPS ↑ | 0.4512 | **0.4775** | | | FLUX.1 → SD1.4 | PSNR ↓ | 24.227 | **24.005** | | | | LPIPS ↑ | 0.4922 | **0.5144** | | | FLUX.1 → SD3 | PSNR ↓ | 25.171 | **24.991** | | | | LPIPS ↑ | 0.4891 | **0.5106** | As shown in Table B, combining $L_{Dis}$ and $L_{Rel}$ produces greater average scaled MSE from B2 through the final latent in both white-box and black-box settings. Table C further shows that adding $L_{Rel}$ consistently yields lower PSNR and higher LPIPS relative to clean reconstructions across all evaluated surrogate–target pairs. Together with the protection gains reported in Table 6 of the revised manuscript, these results support the complementary role of $L_{Rel}$ in strengthening the downstream impact of B0-targeted perturbations. We sincerely thank you for helping us clarify this rationale. We have added an explanation of the motivation behind the relation-disruption loss and an analysis of its effects to the revised manuscript, with the additions highlighted in blue.

[W2] Definition of Black-box and Evaluation on Additional Purification

Thanks for the valuable comment. We agree that pairs sharing VAE encoder weights should be considered white-box at the attacked component. To avoid ambiguity, we have explicitly marked the shared-encoder white-box settings in gray in Tables 1, 2, and 3. Following your valuable suggestion, we further conduct purification-robustness evaluations using Noisy Upscaling [A4], Conditional DiffPure [A4], Gaussian blur, and off-the-shelf img2img-based purification [A5].

Table D. Purification robustness evaluation against Noisy Upscaling

TargetSurrogateAPUniDefBGDSPGESPIDOurs
SD1.4SD1.40.3200.3850.3730.3320.3480.3480.3340.423
SD1.4SD30.3270.3120.3470.3650.3560.3370.3290.438
SD1.4FLUX.10.3200.3330.3380.3360.3320.3310.3190.453
SD3SD1.40.3170.3830.3420.3310.3280.3300.3240.438
SD3SD30.3170.3200.3330.3690.3580.3500.3450.432
SD3FLUX.10.3160.3310.3160.3430.3220.3130.3210.471
FLUX.1SD1.40.2650.3480.3080.2770.2750.2770.2780.389
FLUX.1SD30.2670.2540.2900.3020.2930.2930.2880.379
FLUX.1FLUX.10.2680.2850.2680.2840.2670.2610.2700.432

Table E. Purification robustness evaluation against off-the-shelf img2img denoiser

TargetSurrogateAPUniDefBGDSPGESPIDOurs
SD1.4SD1.40.4290.4260.3450.4970.4000.4950.4870.558
SD1.4SD30.4280.3850.2950.5050.4010.4800.4250.540
SD1.4FLUX.10.4220.4310.2640.4720.3530.4080.4500.539
SD3SD1.40.4360.4040.3070.5270.4030.4750.4640.583
SD3SD30.4710.4110.3050.5450.4190.5310.4740.551
SD3FLUX.10.4510.5200.2490.4740.4000.3910.4520.579
FLUX.1SD1.40.3840.3830.2730.4510.3140.4430.4290.556
FLUX.1SD30.4060.3420.2710.4960.3590.4600.4310.514
FLUX.1FLUX.10.4130.4530.2100.4900.3900.3340.4070.551

Table F. Purification robustness evaluation against Conditional DiffPure

TargetSurrogateAPUniDefBGDSPGESPIDOurs
SD1.4SD1.40.4290.3900.4050.4490.4400.4950.4610.504
SD1.4SD30.4330.4020.3690.4700.4230.4710.4160.508
SD1.4FLUX.10.4230.4020.3330.4660.3870.4100.4390.501
SD3SD1.40.4460.3980.3490.4580.4210.4810.4630.540
SD3SD30.4660.4230.3420.5150.4590.5180.4580.526
SD3FLUX.10.4390.4270.3200.4840.4000.4070.4460.541
FLUX.1SD1.40.4130.4090.3460.4090.3840.4810.4330.505
FLUX.1SD30.4290.3980.3320.4640.4010.4780.4340.478
FLUX.1FLUX.10.4040.3660.3010.4300.3880.3650.3980.513

Table G. Purification robustness evaluation against Gaussian Blur

TargetSurrogateAPUniDefBGDSPGESPIDOurs
SD1.4SD1.40.4090.5440.3530.5060.5200.5460.4880.511
SD1.4SD30.3840.3370.2840.4360.3650.4380.3800.511
SD1.4FLUX.10.3660.3550.2200.4220.3230.3320.3900.521
SD3SD1.40.4060.3940.2880.5110.4300.4430.4330.526
SD3SD30.4690.4180.3060.5110.4240.6050.5190.504
SD3FLUX.10.4070.4450.2180.4390.3730.3480.4380.534
FLUX.1SD1.40.3460.3810.2630.3960.2710.3990.3820.487
FLUX.1SD30.3680.3020.2550.4160.3180.4210.4080.463
FLUX.1FLUX.10.3770.4370.1930.4400.3800.2830.3790.509

As shown in Tables D, E, F and G, our method maintains strong protection across these additional purification methods, further supporting its robustness against diverse perturbation-removal strategies. We thank the reviewer for suggesting these evaluations, which have helped strengthen our robustness assessment. We have incorporated these results into the revised manuscript and cited the corresponding purification methods.

Reviewer_jcPaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how VAE encoder depth affects transferability and purification robustness in image immunization. It finds that early features show stronger cross-model correspondence and retain larger latent shifts after purification, and proposes early-stage feature disruption with a relation-disruption objective.

Critical strengths
  1. The paper moves clearly from the depth-wise study to the representation analysis and then to the method. The choice of early VAE features is well supported by these results.

  2. The paper connects transferability and purification robustness through a shared observation about early encoder features. This is a useful perspective, as the two goals have usually been studied separately.

  3. The evaluation covers several manipulation tasks, U-Net and DiT models, KL and VQ encoders, and four purification methods. The low optimization cost of 6 s and 3.2 GB also supports the method’s practical value.

Critical weaknesses
  1. Some targets, including SD-Inpaint/IP2P and FLUX-Fill/FLUX-Kontext, appear to share VAE encoder weights with the surrogate. Since the method attacks only the VAE encoder, these pairs may be considered white-box at the attacked component. Identifying these pairs and reporting black-box averages excluding them would help clarify the transferability results.

  2. A closer comparison with FastProtect and Anti-I2V would help clarify the contribution of the B0 objective. Both methods use intermediate encoder features, so the statement that they mainly rely on VAE outputs or latents could benefit from further explanation. A B0-only versus multi-layer ablation and a comparison with FastProtect, or a brief explanation of its omission, would also be helpful.

Minor Comments

  1. A method overview figure could make Sec. 4 easier to follow.

  2. Some details in Fig. 5 are difficult to see. Including a larger, more detailed example from Appendix D could help.

Questions for authors
  1. Which targets share VAE encoder weights with the surrogate, and what are the black-box averages after excluding these pairs?

  2. How does the B0 objective differ from FastProtect’s multi-layer loss, and how do they compare experimentally?

  3. Does the early-stage advantage hold for structurally different encoders, beyond the three KL encoders that share the same backbone?

5 comments

[W1, Q1] Clarification for Shared VAE Encoders and Definition of Black-box

Thank you for highlighting this distinction. Our original intention in these experiments was to examine whether perturbations optimized using a surrogate VAE encoder remain effective across different downstream manipulation tasks and pipelines, such as image inpainting, instruction-guided image editing, and I2V. As the reviewer pointed out, SD1.4 shares its VAE encoder with SD-Inpaint, IP2P, and DynamiCrafter, while FLUX.1 shares its VAE encoder with FLUX-Fill and FLUX-Kontext. We agree that these settings may be considered white-box at the encoder level, even though their downstream pipelines differ.

To separately evaluate cross-encoder transferability across different downstream tasks, we report task-wise averages after excluding all shared-encoder and same-model pairs. The remaining pairs evaluate protection on target encoders that were not used during perturbation optimization, with the same exclusion criterion applied to every compared method. As summarized in Table A, our method retains the strongest average protection across all four tasks under this stricter black-box evaluation.

Table A. Average black-box protection performance. Higher LPIPS and lower Anti-Prompt scores indicate stronger protection

TaskMetricStrongest baselineOurs
I2I editingLPIPS ↑DS: 0.4950.559
InpaintingLPIPS ↑DS: 0.4750.531
Instruction-guided editingLPIPS ↑ES: 0.3990.440
I2V generationAnti-Prompt score ↓ES: 3.0042.930

We have marked the shared-encoder white-box settings in gray in the revised Tables 1, 2, and 3 and clarified this designation in their captions. These annotations make the encoder-access assumptions explicit and help distinguish protection across downstream pipelines with a shared encoder from black-box transfer to unseen encoders.

[W2 & Q2] Distinction from FastProtect and Anti-I2V

Thank you for the constructive feedback. We agree that more detail is needed to clarify the distinction between our method and prior approaches. We have revised and expanded Related Work to explain that both FastProtect and Anti-I2V use layer-wise intermediate VAE encoder features: FastProtect combines these intermediate features with a final-latent objective, while Anti-I2V uses features from all VAE encoder stages together with denoising features.

In contrast to their use of features across multiple encoder stages, our method restricts optimization to the single early-block B0, motivated by our finding that early representations provide more favorable cross-model transferability. For a comprehensive comparison, we evaluate black-box transferability against FastProtect and Anti-I2V in Table B. Since FastProtect’s pretrained perturbation weights are not publicly available, we implement an image-specific optimization variant (denoted as FastProtect-IS) using its combined final-latent and intermediate-feature objective. For Anti-I2V, we use only its VAE encoder loss, denoting this variant as Anti-I2V-E.

Table B. Black-box I2I transferability. Mean LPIPS is reported

MethodBlack-box mean LPIPS ↑
FastProtect-IS0.505 ± 0.033
Anti-I2V-E0.464 ± 0.037
Ours0.559 ± 0.027

As shown in Table B, our B0-only method consistently outperforms all additional baselines with the lowest standard deviation on black-box settings. These results support the benefit of targeting early representations alone for cross-model transferability, rather than jointly optimizing final-latent and intermediate features in this setting. We have revised the related-work discussion to clarify these distinctions, with the changes highlighted in red.

Furthermore, following the valuable suggestion, we conduct the multi-layer ablation comparing optimization over B0 alone, B0–B1 jointly, B0–B2, B0–B3, B0–B4, and B0-latent jointly, as reported in Table C.

Table C. Effect of jointly optimizing additional encoder blocks on black-box transferability

Attacked stagesBlack-box mean LPIPS ↑
B0-only (ours)0.559 ± 0.027
B0–B1 jointly0.558 ± 0.022
B0–B2 jointly0.534 ± 0.020
B0–B3 jointly0.497 ± 0.025
B0–B4 jointly0.484 ± 0.024
B0–Latent jointly0.479 ± 0.026

As shown in Table C, incorporating later-stage features tends to reduce transferability compared with B0-only optimization. This trend is consistent with Fig. 3a and Fig. 3b, where both clean-PWCCA and shift-PWCCA generally decrease with encoder depth, supporting the interpretation that incorporating deeper features can produce more surrogate-specific perturbations.

[Q3] Analysis of Early-stage Trends on Different Encoders

Thank you for the valuable feedback. To provide a more comprehensive assessment of whether the early-stage advantage extends to structurally different encoders, we further evaluate perturbations optimized at B0–B4 using our overall objective and report their transferability to Kandinsky 2.2 and Kandinsky 3 in Table D.

Table D. Depth-wise attack transferability to Kandinsky models

TargetSurrogateB0B1B2B3B4
Kandinsky 2.2SD1.40.3820.3390.3230.2760.288
Kandinsky 2.2SD30.3670.3910.3650.2890.311
Kandinsky 2.2FLUX.10.3760.3520.3090.2520.278
Kandinsky 3SD1.40.4980.4790.4570.3610.369
Kandinsky 3SD30.4630.5130.4890.4090.421
Kandinsky 3FLUX.10.4820.4670.4240.3440.351

As shown in Table D, B0 or B1 achieves the strongest transferred protection for every surrogate–target pair, while later stages yield weaker protection. These results support the early-stage advantage beyond the three KL encoders, although the best-performing depth can vary across encoder pairs. We have included this additional depth-wise comparison, and will discuss these findings in the final revision.

[MW1] Inquiry of Method overview

Thank you for the helpful suggestion. We agree that an overview figure would make Sec. 4 easier to follow. We will add a figure illustrating the B0 attack target and the complementary roles of the two loss components to the final manuscript if the paper is accepted.

[MW2] Examples of Immunized Images

We apologize for the difficulty in viewing the examples. We kindly ask you to refer to Fig. 19 in the Appendix of the revision, where we have added larger, clearer examples of immunized images.

Reviewer_psb4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates image immunization techniques for diffusion models, aiming to address two critical challenges in practical deployment: black-box transferability to unseen generative models and robustness against perturbation purification. Through a systematic analysis of different stages of the diffusion pipeline, including intermediate VAE encoder layers and denoiser features, the authors find that targeting early VAE features (e.g., the B0B_0 block) yields stronger cross-model feature correspondence and better preservation of protection effectiveness after purification. Building on these findings, the authors propose a unified image immunization framework that combines a feature discrepancy maximization loss (LDisL_{Dis}) with an inter-channel relationship disruption loss (LRelL_{Rel}). By restricting optimization to the early stages of a surrogate VAE encoder, the proposed method achieves effective protection across heterogeneous architectures (U-Net and DiT) and diverse downstream tasks, including image editing, inpainting, and image-to-video generation, with relatively low computational overhead.

Critical strengths
  1. Clear motivation and systematic analysis. The paper is well-written and logically organized. Through controlled experiments, the authors systematically investigate how the choice of attack depth affects transferability and robustness against purification. Metrics such as PWCCA further quantify cross-model feature correspondence, providing empirical support for targeting early VAE features.

  2. Novel and computationally efficient design. The proposed inter-channel relationship disruption loss (LRel\mathcal{L}_{Rel}) effectively disrupts internal channel dependencies and complements the feature discrepancy loss. By restricting optimization to the early stages of a surrogate VAE encoder, the method avoids backpropagation through the entire denoiser, substantially reducing computational overhead.

  3. Comprehensive experimental evaluation. The experiments cover a broad range of generative architectures (U-Net and DiT), downstream applications (image editing, inpainting, instruction-based editing, and image-to-video generation), and existing purification methods, providing a relatively thorough evaluation of the proposed framework.

Critical weaknesses
  1. Unclear generalizability beyond VAE-based architectures. The core motivation and design of the proposed method rely heavily on hierarchical features extracted from VAE encoders. However, it remains unclear whether the method can generalize to pixel-space diffusion models that do not employ conventional VAE architectures, such as DeepFloyd IF. Evaluating or discussing its applicability to such models would help clarify the scope and limitations of the proposed approach.

  2. Incomplete evaluation against purification-based defenses. Although the authors evaluate several common purification techniques, including JPEG compression, Adverse Cleaner, and GridPure, the evaluation could be strengthened by incorporating more recent and representative purification or defense methods. In particular, Noisy Upscaling [1] would be a relevant baseline for assessing the robustness of the proposed method against stronger purification strategies.

  3. Lack of evaluation against adaptive attacks. Evaluating robustness against informed adversaries is essential for image immunization methods. If an attacker has full knowledge of the defense mechanism, including the specific formulations of Eqs. (1) and (5), they may be able to design adaptive optimization objectives to undermine the protection. For example, an attacker could construct a purification objective by reversing the signs of the corresponding loss terms. It is suggested that the authors evaluate the proposed method under adaptive attack scenarios.

[1] Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI. ICLR 2025.

Questions for authors

See Weaknesses. If the authors can clarify the concerns raised above, I would be willing to reconsider and increase my score.

4 comments

[W1] Transferability to Heterogeneous VAE and Pixel-space Diffusion Models

Thank you for the valuable comment. We recognize the importance of transferability to diffusion models with different VAE architectures and evaluated it in Appendix C.7. Specifically, we evaluated Kandinsky 2.2 and Kandinsky 3, whose structurally different VQ-based encoders extend our evaluation beyond KL-based VAEs, and our method showed superior protection performance compared to the baselines.

We would also like to clarify that our method was designed specifically for latent diffusion models. We agree that evaluating transferability beyond this scope, including to pixel-space diffusion models, is essential. Following your suggestion, we further evaluate black-box transferability to DeepFloyd IF, a pixel-space diffusion model that does not use a VAE.

Table A. Black-box comparison on DeepFloyd IF

SurrogateAPUniDefBGDSPGESPIDOurs
SD1.40.1250.1710.1920.1020.1560.1300.1270.176‾\underline{0.176}
SD30.1490.0920.2190.1120.1550.1280.1370.164‾\underline{0.164}
FLUX.10.1150.1160.2250.0980.1310.1060.0920.136‾\underline{0.136}

As shown in Table A, BlurGuard performs best, potentially owing to its low-frequency perturbation design and the target model’s direct processing of perturbed images in pixel space without VAE encoding. Nevertheless, our method achieves the second-highest LPIPS across all three surrogates, outperforming all remaining baselines.

To examine whether the early-stage advantage also holds when transferring to a VAE-free model, we additionally compare perturbations optimized at different surrogate encoder depths.

Table B. Depth-wise evaluation on DeepFloyd IF

SurrogateB0B1B2B3B4Latent
SD1.40.1760.1650.1530.1450.1290.130
SD30.1640.1280.1480.1450.1230.125
FLUX.10.1360.1120.1180.1050.0940.104

As shown in Table B, B0 achieves the highest LPIPS among the evaluated optimization targets for all three surrogates, outperforming final-latent optimization. Your insightful comment prompted us to evaluate beyond LDMs, revealing that the early-stage advantage also holds when transferring to the pixel-space diffusion model. We sincerely thank you for this suggestion and will incorporate these results into the revision.

[W2] Robustness Evaluation on Additional Purifications

Thanks for the constructive feedback. We further evaluate our method’s robustness against Noisy Upscaling [A1] and Conditional DiffPure [A1].

Table C. Purification robustness evaluation against Noisy Upscaling

TargetSurrogateAPUniDefBGDSPGESPIDOurs
SD1.4SD1.40.3200.3850.3730.3320.3480.3480.3340.423
SD1.4SD30.3270.3120.3470.3650.3560.3370.3290.438
SD1.4FLUX.10.3200.3330.3380.3360.3320.3310.3190.453
SD3SD1.40.3170.3830.3420.3310.3280.3300.3240.438
SD3SD30.3170.3200.3330.3690.3580.3500.3450.432
SD3FLUX.10.3160.3310.3160.3430.3220.3130.3210.471
FLUX.1SD1.40.2650.3480.3080.2770.2750.2770.2780.389
FLUX.1SD30.2670.2540.2900.3020.2930.2930.2880.379
FLUX.1FLUX.10.2680.2850.2680.2840.2670.2610.2700.432

Table D. Purification robustness evaluation against Conditional DiffPure

TargetSurrogateAPUniDefBGDSPGESPIDOurs
SD1.4SD1.40.4290.3900.4050.4490.4400.4950.4610.504
SD1.4SD30.4330.4020.3690.4700.4230.4710.4160.508
SD1.4FLUX.10.4230.4020.3330.4660.3870.4100.4390.501
SD3SD1.40.4460.3980.3490.4580.4210.4810.4630.540
SD3SD30.4660.4230.3420.5150.4590.5180.4580.526
SD3FLUX.10.4390.4270.3200.4840.4000.4070.4460.541
FLUX.1SD1.40.4130.4090.3460.4090.3840.4810.4330.505
FLUX.1SD30.4290.3980.3320.4640.4010.4780.4340.478
FLUX.1FLUX.10.4040.3660.3010.4300.3880.3650.3980.513

As shown in Tables C and D, our method achieves the strongest protection across all evaluated settings under both purification methods, with one tie under Conditional DiffPure. These results further support our robustness against diverse purification techniques. We thank the reviewer for suggesting these evaluations, which strengthen our robustness assessment.

[W3] Concerns on Robustness against Adaptive Attacks

Thanks for pointing this out. We recognize the importance of robustness against adaptive attacks. However, an adaptive strategy based on directly reversing the protection objective is less straightforward in our setting than for objectives with fixed references. PhotoGuard’s encoder attack minimizes the distance to a prescribed target latent, while DCT-Shield minimizes the latent norm, effectively using the zero vector as its reference. When these target images or latents are known, opposite-direction objectives can be formulated without access to the original clean image.

In contrast, our objectives maximize discrepancies from image-specific clean features and channel relationships. Exactly reversing these objectives would require the corresponding clean reference representations, which are unavailable to the attacker under our threat model. Thus, knowing our method alone does not provide the information needed for exact loss reversal. This makes such a direct strategy challenging, although it does not rule out other adaptive approaches.

To broaden our robustness evaluation, we further evaluate Gaussian blur and off-the-shelf img2img purification [A2], with results reported in Tables E and F. Our method achieves the strongest protection in all black-box transfer settings under both purification methods, demonstrating robustness against spatial smoothing and image-to-image regeneration.

Together with the Noisy Upscaling and Conditional DiffPure evaluations presented in our response to W2, these experiments cover a broader range of strategies for removing protective perturbations. Although we do not directly evaluate adaptive attacks, we believe these additional results strengthen our robustness assessment. We have incorporated these results into the revised manuscript and cited the corresponding purification methods. We will discuss their implications and limitations regarding robustness against adaptive attacks.

Table E. Purification robustness evaluation against Gaussian Blur

TargetSurrogateAPUniDefBGDSPGESPIDOurs
SD1.4SD1.40.4090.5440.3530.5060.5200.5460.4880.511
SD1.4SD30.3840.3370.2840.4360.3650.4380.3800.511
SD1.4FLUX.10.3660.3550.2200.4220.3230.3320.3900.521
SD3SD1.40.4060.3940.2880.5110.4300.4430.4330.526
SD3SD30.4690.4180.3060.5110.4240.6050.5190.504
SD3FLUX.10.4070.4450.2180.4390.3730.3480.4380.534
FLUX.1SD1.40.3460.3810.2630.3960.2710.3990.3820.487
FLUX.1SD30.3680.3020.2550.4160.3180.4210.4080.463
FLUX.1FLUX.10.3770.4370.1930.4400.3800.2830.3790.509

Table F. Purification robustness evaluation against off-the-shelf img2imgdenoiser

TargetSurrogateAPUniDefBGDSPGESPIDOurs
SD1.4SD1.40.4290.4260.3450.4970.4000.4950.4870.558
SD1.4SD30.4280.3850.2950.5050.4010.4800.4250.540
SD1.4FLUX.10.4220.4310.2640.4720.3530.4080.4500.539
SD3SD1.40.4360.4040.3070.5270.4030.4750.4640.583
SD3SD30.4710.4110.3050.5450.4190.5310.4740.551
SD3FLUX.10.4510.5200.2490.4740.4000.3910.4520.579
FLUX.1SD1.40.3840.3830.2730.4510.3140.4430.4290.556
FLUX.1SD30.4060.3420.2710.4960.3590.4600.4310.514
FLUX.1FLUX.10.4130.4530.2100.4900.3900.3340.4070.551

[A1] Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI. ICLR, 2025.

[A2] Off-The-Shelf Image-to-Image Models Are All You Need To Defeat Image Protection Schemes. IEEE SaTML, 2026.

Thank you for the detailed response and additional experiments. The new results on DeepFloyd IF and the additional purification methods have substantially addressed my concerns in W1 and W2.

Regarding W3, I find the authors' explanation reasonable. While a systematic evaluation against adaptive attacks would further strengthen the work, I acknowledge that the proposed direct loss-reversal strategy may not be straightforward in this setting. I would still encourage the authors to briefly discuss this limitation and clarify the scope of their robustness claims in the revised manuscript.

Overall, the rebuttal has adequately addressed my major concerns and significantly strengthened the experimental evaluation. I am willing to increase my rating from 2 to 3.

Dear Reviewer psb4,

We are glad our responses addressed your major concerns and sincerely appreciate your thoughtful follow-up.

Regarding W3, we agree that systematic robustness evaluation against adaptive attacks remains important. Based on your valuable feedback, we will carefully discuss this limitation in the revised manuscript and clarify the scope of our robustness claims, distinguishing the evaluated purification methods from adaptive attacks.

We will also discuss our method’s applicability across diffusion architectures and its purification robustness, drawing on the additional results addressing W1 and W2.

We sincerely appreciate your constructive feedback and careful consideration.

1864 Authors

Comments on the paper

1 comment

We sincerely thank all reviewers for their time, effort, and thoughtful feedback on our paper, as well as for recognizing that our work:

  • Is supported by comprehensive experimental evaluations (psb4, jcPa, zNRh, wGVU).
  • Provides systematic and insightful analyses of how attack depth affects transferability and purification robustness (psb4, jcPa, zNRh).
  • Demonstrates solid transferability and purification robustness (psb4, zNRh, wGVU).
  • Offers a computationally efficient design (psb4, jcPa, zNRh, wGVU).
  • Is clearly motivated, logically organized, and easy to follow (psb4, jcPa).
  • Has practical value for real-world image immunization (jcPa, wGVU).

We sincerely appreciate these positive assessments and the reviewers’ constructive suggestions, which have helped us clarify our contributions and strengthen the manuscript.


We have conducted additional experiments and provided clarifications in response to the reviewers’ feedback. The key updates are as follows:

  • Comparisons with additional baselines (jcPa, zNRh, wGVU)
  • Clarification of white-box and black-box evaluation settings (jcPa, zNRh)
  • Evaluations against additional purification methods (psb4, zNRh)
  • Further analyses of early-stage transferability (zNRh, wGVU)
  • Clarification and analysis of the complementary loss components (zNRh, wGVU)
  • Additional evaluations and discussion of heterogeneous and VAE-free models (psb4, jcPa)
  • Ablation of single-stage versus multi-stage optimization (jcPa)
  • Discussion of adaptive attacks and evaluation limitations (psb4)
  • Improved visual examples of immunized images (jcPa)

We believe these additions strengthen the manuscript and clarify the contributions and scope of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.