acceptodds
Under review as a conference paper at ICLR 2027

DiTCloak: Protecting Facial Privacy Against Personalized Text-to-Image Diffusion Transformers via Unlearnable Examples

Abstract

Diffusion Transformers (DiTs) increasingly serve as denoising backbones for modern text-to-image (T2I) models. Their adoption in subject personalization, however, enables publicly shared portraits to be learned and reproduced without permission. Unlearnable examples (UEs) seek to mitigate this by applying concealed perturbations before image release. Yet prior approaches largely inherit objectives developed for U-Net architectures, leaving Transformer block designs in DiTs largely unexplored. Accordingly, we present DiTCloak, which optimizes perturbations at both the latent diffusion and DiT structural levels. At the latent diffusion level, we study how guidance interacts across VAE encoding and diffusion prediction, revealing that guiding both toward a shared target pattern is more effective, thereby motivating our target-guided dual alignment objective. At the DiT structural level, token relation anchoring reshapes inter-token dependencies according to the target pattern, limiting the learning of subject-specific local structure. Meanwhile, we disrupt multi-head attention within Transformer blocks to impair visual information aggregation in DiTs, forming the multi-head attention reorientation objective. Together, these objectives are optimized by alternating surrogate adaptation with projected gradient descent (PGD). Experiments on CelebA-HQ and VGGFace2 across diverse personalization schemes and DiT architectures demonstrate that DiTCloak outperforms nine state-of-the-art defenses in protection performance. Our code is available at [https://anonymous.4open.science/r/DiTCloak-7E67](https://anonymous.4open.science/r/DiTCloak-7E67).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Privacy protection, diffusion transformers, text-to-image

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_53WzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a novel approach to avoid unauthorized personalization in DiT-based T2I generation. Their approach, called DiTCloak, combines target-guided dual alignment, token relation anchoring, and multi-head attention reorientation objectives to optimize unlearnable examples (UEs) in order to prevent malicious reproduction of personal content. DiTCloak addresses two limitations of DiT-based T2I generation: 1) insufficient token-relation decorrelation; and 2) limited token-attention distortion. Extensive experiments on three DiT-based T2I backbones, two face datasets, two diffusion architectures, and both training-based and training-free personalization settings show the superiority of DiTCloak with respect to the state of the art. Furthermore, DiTCloak demonstrates strong robustness against purification methods (JPEG compression, resizing, Gaussian blur/noise, etc.). Additionally, UEs generated via DiTCloak transfer effectively across different diffusion architectures, including both DiT and U-Net models.

Critical strengths

These are the paper's strengths:

  • it is one of the first papers that addresses the problem of unauthorized personalization in DiT-based T2I generation
  • the approach overcomes two limitations of DiT-based T2I generation: 1) insufficient token-relation decorrelation; and 2) limited token-attention distortion
  • experimental validation is extensive and demonstrates the superiority of the proposed approach with respect to the state of the art
  • DiTCloak shows robustness against several purification methods
  • UEs generated via DiTCloak transfer effectively across different diffusion architectures
Critical weaknesses

These are the paper's weaknesses:

  • the evaluation focuses mainly on facial privacy and identity reproduction
  • the performance relies on several loss-balancing and optimization hyper-parameters (λdp\lambda_{dp}, λtr\lambda_{tr}, λma\lambda_{ma}, etc.), which require careful manual fine-tuning
  • optimizing perturbations requires a surrogate model and specific personalization setups (e.g. DreamBooth LoRA), which might limit optimization efficiency if the adversary uses a different pipeline.
  • the optimization process (alternating rounds of surrogate adaptation and M-step PGD with multiple structural loss evaluations per update) could add a significant computational overhead
  • the paper lacks an user study in order to confirm the findings obtained through automated metrics
Questions for authors

These are my concerns:

  • how well does your approach cope with the protection of non-face subjects, such as: personalized objects, pets, artistic styles? It would be appreciated if you can show through expeiments in some of these cases
  • since the UEs are optimized using specific surrogate models and fine-tuning pipelines (such as DreamBooth LoRA), how does protection efficiency is affected if an adversary adopts an entirely different personlization scheme?
  • what is the computational overhead introduced by your optimization process (alternating rounds of surrogate adaptation and M-step PGD with multiple structural loss evaluations per update)?
  • how sensitive is the protection performance to the hyper-parameters ((λdp\lambda_{dp}, λtr\lambda_{tr}, λma\lambda_{ma}, etc.) used in the loss functions across different DiT backbones? Is there an automated way to tune them?
  • the structural objectives target evenly spaced blocks from the middle to the final DiT blocks. How do a different subset of blocks contribute to the preservation of identity? Could targeting a different subset of blocks optimize the trade-off between protection and perturbation budget?
  • I recommend to perform an user study in order to confirm the results obtained by automated metrics
9 comments
AuthorsOpenReview ↗

Response to Reviewer 53Wz [1/3]

We sincerely thank the reviewer for the thoughtful, professional, and detailed feedback, which helps us further improve our work. Our point-by-point responses are provided below:

W1 & Q1: Limited non-face evaluation.

Re: We sincerely thank you for drawing attention to this aspect of our work. Based on your suggestion, we extend our evaluation to personalized objects and pets (backpack and dog), and artistic styles (Edward Hopper).

  • Objects and pets. We use the DreamBooth dataset and report DINO, CLIP-I, and CLIP-T. Text prompts for the dog subject are a photo of sks dog and a photo of sks dog wearing a red hat, for the backpack subject are a photo of sks backpack and a photo of sks backpack on a wooden floor.
  • Artistic styles. We use Edward Hopper paintings from the WikiArt dataset and report CSD style similarity. Text prompts are a painting of mountain by nulevoy and a painting of piano by nulevoy.
SchemeBackpack DINO (↓)Backpack CLIP-I (↓)Backpack CLIP-T (↓)Dog DINO (↓)Dog CLIP-I (↓)Dog CLIP-T (↓)Edward Hopper CSD (↓)
Clean baseline0.4790.8430.3090.8360.8690.3260.352
Glaze0.4460.7890.3170.7810.8330.3260.311
Anti-DB0.4560.7830.3170.6930.8060.3210.190
ACE0.4130.7610.3140.5850.7960.3220.152
BlurGuard0.5620.8670.3010.8110.8590.3230.349
TAFAP0.4420.7650.3090.7660.8170.3260.277
DiTCloak (Ours)0.3300.7470.3000.4550.7390.3140.128

It reveals that DiTCloak achieves the lowest scores across reported metrics, indicating better protection against personalization of objects, pets, and artistic styles in the tested cases. In the revised manuscript, we added the experimental results in 🔗L486-494 and the discussion in 🔗L464-466.


W2 & Q4: (a) Parameter sensitivity across DiT backbones; (b) automated tuning option.

Re: We deeply appreciate for this helpful suggestion. (a) Guided by your suggestions, we fixed λvl\lambda_{vl} and jointly scaled λdp\lambda_{dp}, λtr\lambda_{tr}, and λma\lambda_{ma} by 0.5×, 1.0×, or 2.0× relative to that backbone’s default weights on different backbones. We reoptimized the perturbations for each setting and held the personalization and evaluation protocols fixed:

BackboneWeight multiplierFSS (↓)SER-FQA (↓)IMS-CLIP (↓)IMS-VGG (↓)AVG (↓)
PixArt-α\alpha-XL/20.5×0.1350.3610.5890.2960.345
PixArt-α\alpha-XL/21.0×-0.9330.0240.5870.226-0.024
PixArt-α\alpha-XL/22.0×0.4230.6810.6340.4510.547
Stable Diffusion 3.5 Medium0.5×0.5170.7230.7870.5070.633
Stable Diffusion 3.5 Medium1.0×0.3250.6940.6880.3180.506
Stable Diffusion 3.5 Medium2.0×0.4980.7590.7640.4510.618

The default weights gave the lowest scores on all four metrics for both backbones. Halving or doubling the weights increased AVG by 0.369 or 0.571 on PixArt-α\alpha-XL/2 and by 0.127 or 0.112 on Stable Diffusion 3.5 Medium, indicating greater sensitivity to joint scaling on PixArt-α\alpha-XL/2 in this sweep; (b) To automate selection, we initialized the loss weights for Stable Diffusion 3.5 Medium from (λvl\lambda_{vl}, λdp\lambda_{dp}, λtr\lambda_{tr}, λma\lambda_{ma})=(1, 1, 0.1, 0.1) and automatically selected a shared multiplier for λdp\lambda_{dp}, λtr\lambda_{tr}, and λma\lambda_{ma} from {0.5, 1.0, 2.0} while keeping λvl\lambda_{vl} fixed. We performed a grid search over three loss-weight multipliers by minimizing validation Identity Verification Success Rate (IVSR) (percentage of generated images with a detected face whose target-identity similarity is above threshold). On Stable Diffusion 3.5 Medium, at a verification threshold of 0.6, the rates were 6.7%, 0%, and 3.3% for 0.5×, 1.0×, and 2.0×, respectively, so this rule selects 1.0×, i.e., λvl=1\lambda_{vl} = 1, λdp=1\lambda_{dp} = 1, λtr=0.1\lambda_{tr} = 0.1, and λma=0.1\lambda_{ma} = 0.1. We added the above results and discussion to the revised manuscript 🔗L1120-1133.

AuthorsOpenReview ↗

Response to Reviewer 53Wz [2/3]

W3 & Q2: DreamBooth–optimized UEs' transferability across a different personalization scheme?

Re: We truly appreciate it for asking us to clarify this point. We note that, in the Table 1 of the manuscript (UEs are optimized using DreamBooth LoRA), the evaluation covers attacker's personalization pipelines DreamBooth LoRA and FreeCus. The results reveal that our scheme DiTCloak transfers effectively against FreeCus, outperforming baselines. We have made this more clearly in revision 🔗L326.


W4 & Q3: Computational overhead of DiTCloak optimization.

Re: We greatly appreciate it for raising this point. Based on your suggestion, we measured DiTCloak’s perturbation optimization cost on a system with an NVIDIA RTX PRO 5000 GPU with a memory size of 71.69 GiB and an AMD EPYC 9115 16-core CPU. DiTCloak’s perturbation optimization with FLUX.2-klein-base-4B took 2,482.8 s on CelebA-HQ and 2,486.4 s on VGGFace2, requiring 109,631.76 TFLOPs per run. With PixArt-α\alpha-XL/2, it took 839.4 s on CelebA-HQ and 878.4 s on VGGFace2, requiring 17,740.35 TFLOPs per run. We have added discussion about this point In revision 🔗L792-798.


W5 & Q6: User-study on automated results.

Re: We sincerely thank you for raising this important point. Based on your suggestion, we recruited 10 participants: 3 PhD students, 2 master students, and 5 undergraduates. Participants were aged 21–26 years, with a mean age of 23.3 years. Each participant compared 30 generated images individually with one reference image of the target identity and answered, “Do these two images show the same person?” ('yes' or 'no') Each “yes” counted as one match. We averaged the match counts across the 10 participants and report the mean number of matches out of 30 (↓).

SchemeCelebA-HQ (PixArt-α-XL/2 + DreamBooth LoRA)VGGFace2 (FLUX.2-klein-base-4B + FreeCus)
Glaze21.524.6
ACE29.417.0
BlurGuard25.827.8
TAFAP26.925.2
DiTCloak (Ours)4.70.2

DiTCloak provided the best facial identity protection in this user study setting, with participants identifying fewer generated images as the target person than baselines. We added these results to the revised manuscript 🔗L1134-1144.

AuthorsOpenReview ↗

Response to Reviewer 53Wz [3/3]

Q5: (a) Block-subset effects; (b) selections for better protection-budget trade-offs.

Re: We truly appreciate this constructive suggestion. (a) Guided by suggestions, we further evaluate performance with three different subset of blocks on CelebA-HQ using PixArt-XL-2-512x512 with DreamBooth LoRA:

Selected blocksFSS (↓)SER-FQA (↓)IMS-CLIP (↓)IMS-VGG (↓)
Clean baseline0.7070.8090.8670.618
Full0.1480.3670.5910.246
First half0.1140.4750.6380.230
Middle third0.0670.4180.6140.286
Middle to the final (Ours)-0.9330.0240.5870.226

It reveals that our strategy yields better performance, likely because latter DiT blocks progressively refine high-frequency visual details [1]. Disrupting this may hinder faithful reproduction of the target identity; (b) We further evaluated the same block subsets at a lower perturbation budget of ρ=8/255\rho=8/255, with results as below:

Selected blocksFSS, 8/255SER-FQA, 8/255IMS-CLIP, 8/255IMS-VGG, 8/255FSS, 16/255SER-FQA, 16/255IMS-CLIP, 16/255IMS-VGG, 16/255
Clean baseline0.7070.8090.8670.6180.7070.8090.8670.618
Full0.5850.7390.7700.5570.1480.3670.5910.246
First half0.5680.7700.7690.5670.1140.4750.6380.230
Middle third0.6190.7510.7540.5470.0980.5230.6140.286
Middle to the final (Ours)0.5410.7370.7530.495−0.9330.0240.5870.226

It can be seen that targeting blocks from the midpoint to the end gave the strongest protection at both 8/2558/255 and 16/25516/255, achieving the lowest scores on all four metrics. Thus, among the subsets tested, our block selection offers the best protection at either budget. In the revised manuscript, we added the above results in 🔗L495-503 and the discussion in 🔗L471-474.

[1] W-EDIT: A wavelet-based frequency-aware framework for text-driven image editing. In Proc. of ICLR'26

Reviewer_mjtDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates the objectives for optimizing an unlearnable example for DiT-based text-to-image models. The paper first establishes the design choices of disrupting the outputs of both the VAE and the diffusion model, and aligning both their output with the feature of a target pattern (TT) instead of with clean guidance. Further, the paper proposes two additional objectives that exploit the inductive bias of DiTs and disrupts the features of token relations and multi-head attention. Experiments demonstrate significant improvements over existing methods across model architectures, datasets, and personalization methods.

Critical strengths
  • The paper is useful and in time as image generation pivots to Diffusion Transformer-based architectures.
  • Ablation studies confirm the efficacy of each design choice, including clean vs. target, VAE vs. DiT, and all four objectives.
  • Experiments demonstrate significant improvements over existing methods across model architectures, datasets, and personalization methods.
Critical weaknesses
  • The motivation for the two new DiT-specific objectives are mostly intuitive. Could there be quantitative evidence supporting the motivation, e.g. how the token anchoring objective improves "the learning of subject-specific relations across local regions"?
  • More generally speaking, it is unclear whether the above intuitive structural reasons directly account for the significant improvements. In particular, I wonder if similar effects can be achieved by maximizing the displacement of other intermediate features, such as hidden states after each block.
  • The radial pattern is claimed to be a new contribution, though the motivation and experimental evidence supporting it is relatively scarce.
  • As previous methods are primarily implemented for non-DiT architectures, it would be helpful to include more information on the re-implementation of them with the new pipeline.
  • Section 3.2 could be improved for clarity, particularly the acronyms CC, CT, TC, and TT need more formal introduction in the main text.
Questions for authors
  • Have other intermediate features been attempted during the development of the method?
  • What are the motivations of using the radial target pattern for this task? Have other candidates been attempted?
  • How are previous methods adapted to the new models?
5 comments
AuthorsOpenReview ↗

Response to Reviewer mjtD [1/2]

We sincerely appreciate the reviewer’s careful assessment and constructive feedback. We address each point below:

W1: Quantitative evidence of token anchoring’s effect on learning subject relations across local regions.

Re: We sincerely thank the reviewer for this helpful suggestion. Guided by your suggestion, we compare clean images with UEs (using only Jtr\mathcal{J}_{tr} to optimize). We extract intermediate image tokens from a frozen DiT PixArt-α\alpha-XL/2 with conditions fixed. We pool tokens within four facial local regions segmented by BiSeNet (left eye, right eye, nose, and mouth) and use the six pairwise cosine similarities as the inputs to a binary identity classifier. We use two CelebA-HQ identities (44⇔A and 47⇔B), with 10 training images and 4 test images per identity. At test time, we replace only identity A’s clean images with their paired UEs and use the same four clean images of identity B in both conditions.

Test setOverall accuracyA recognition rateMean predicted probability of AB recognition rateBalanced accuracy
Clean A + clean B0.8750.7500.5311.000.875
UE A + clean B0.7500.5000.4771.000.750

The classifier’s accuracy on identity A dropped from 75% to 50%, and the mean predicted probability for A decreased from 0.531 to 0.477. These results suggest that token relation anchoring makes identity A harder to recognize from cross-region token relations in the DiT. In the revised manuscript, we added the above results and discussion in 🔗L1111-1119.


W2 & Q1: Effects of maximizing the displacement of other intermediate features.

Re: We greatly appreciate this insightful suggestion. Following this thoughtful comment, we retained Jvl\mathcal{J}_{vl} and Jdp\mathcal{J}_{dp} and replaced DiTCloak’s structural losses with a hidden-state displacement objective:

max⁡δ1∣B∣∑ℓ∈BMSE(Sℓ(x+δ),Sℓ(x))\max_{\delta}\frac{1}{|\mathcal{B}|}\sum_{\ell\in\mathcal{B}}\mathrm{MSE}\left(S^\ell(x+\delta), S^\ell(x)\right)

where xx is the clean image, δ\delta is the perturbation, and SℓS^\ell denotes the image-token features output by block ℓ\ell in the set of blocks B\mathcal{B}. We tested weights λhd∈{0.1,1.0}\lambda_{hd}\in\{0.1,1.0\} on CelebA-HQ using PixArt-α\alpha-XL/2 w/ DreamBooth LoRA:

SchemeFSSSER-FQAIMS-CLIPIMS-VGG
Clean0.7070.8090.8670.618
Jvl+Jdp+0.1Jhd\mathcal{J}_{vl}+\mathcal{J}_{dp}+0.1\mathcal{J}_{hd}-0.5490.2120.5870.209
Jvl+Jdp+1.0Jhd\mathcal{J}_{vl}+\mathcal{J}_{dp}+1.0\mathcal{J}_{hd}-0.1500.3020.5890.248
DiTCloak-0.9330.0240.5870.226

It can be seen that hidden-state displacement improved protection over the clean baseline at both weights. DiTCloak substantially outperformed both hidden-state displacement variants on FSS and SER-FQA, lowering FSS by 0.384–0.783 and SER-FQA by 89–92%. IMS-CLIP and IMS-VGG are comparable. We added this experiment to the revised manuscript 🔗L467-470.


W3 & Q2: Insufficient motivation and evidence supporting radial pattern design (other candidates need be tested).

Re: We deeply appreciate this thoughtful feedback. Our motivation is radial pattern spokes connect distant image regions in a regular, identity-independent way, giving the loss a structured target. Building on your suggestion, we follow the setting in Fig. 3(b), and test three alternatives to our radial anchor on CelebA-HQ using PixArt-XL-2-512x512 w/ DreamBooth LoRA:

Target anchorFSS (↓)SER-FQA (↓)IMS-CLIP (↓)IMS-VGG (↓)
Checkerboard0.3640.6400.6960.506
Concentric ring0.5180.7130.7280.504
Spiral stripe0.4490.6320.7190.435
Radial pattern (Ours)0.0270.3600.6410.331

It can be seen that our design outperforms the other choices. We added the experiment and discussion to the revised manuscript 🔗L1080-1093, 🔗L1104-1109.

AuthorsOpenReview ↗

Response to Reviewer mjtD [2/2]

W4 & Q3: Baseline adaptation details for the new pipeline.

Re: We sincerely thank you for raising this important point. Our baseline adaptation details are as follows: In the main experiments, all baselines use the same ℓ∞\ell_\infty perturbation budget of 16/25516/255 as DiTCloak. For the remaining hyper-parameters, we follow the official implementations where available and the original papers otherwise.

PixArt-α-XL/2

  • AdvDM. We encode images with PixArt’s VAE and use projected gradient ascent to maximize the T5-conditioned DiT’s noise prediction error over sampled diffusion timesteps.
  • PhotoGuard. We use the encoder-only variant and apply projected gradient descent to bring the protected image’s PixArt VAE latent closer to a target latent.
  • Glaze. We optimize perturbations to bring the protected image’s PixArt VAE features closer to those of a target image.
  • Anti-DreamBooth. In each ASPL round, we fine-tune a copy of the PixArt DiT’s LoRA adapter on clean images, use it to maximize the denoising loss via PGD, and update the running adapter on the perturbed images.
  • MetaCloak. We adapt sampled PixArt DiT LoRA checkpoints and use PGD with randomly sampled image transformations to optimize perturbations across fine-tuning stages.
  • MYOPIA. We alternate PixArt DiT LoRA updates with perturbation updates that minimize the denoising loss. We use Gaussian timestep sampling biased toward higher noise levels and a JND mask to constrain the perturbations.
  • ACE. We alternate PixArt DiT LoRA updates with perturbation updates that drive the DiT’s noise prediction toward a fixed target tensor. Our fused variant also includes a correlation term between predicted and ground-truth noise and a latent alignment loss.
  • BlurGuard. We use a PixArt VAE encoder loss to optimize perturbations through region-wise Gaussian filters. We calibrate the filter scales using the power spectrum discrepancy between the original and protected images.
  • TAFAP. We differentiate through PixArt DiT LoRA fine-tuning steps to optimize perturbations whose induced parameter updates match segments of precomputed target trajectories.

For Stable Diffusion 3.5 Medium and FLUX.2-klein-base-4B, we use flow-matching velocity prediction for baseline objectives that depend on the denoiser. Each model retains its own VAE latent preprocessing, text conditioning, and Transformer input format.

Stable Diffusion 3.5 Medium and FLUX.2-klein-base-4B

  • AdvDM. We encode images with each model’s VAE and apply its latent preprocessing. We then use projected gradient ascent to maximize the text-conditioned Transformer’s velocity prediction error.
  • PhotoGuard. We use the encoder-only variant with each model’s VAE and apply projected gradient descent to bring the protected image’s latent representation closer to a fixed target latent.
  • Glaze. We use each model’s preprocessed VAE latents as features and iteratively optimize pixel perturbations to reduce the feature distance to a target image.
  • Anti-DreamBooth. In each ASPL round, we fine-tune a copy of the Transformer’s LoRA adapter on clean images, use it to maximize the flow-matching velocity prediction error via PGD, and update the running adapter on the perturbed images.
  • MetaCloak. We adapt LoRA surrogates sampled from different fine-tuning checkpoints and use PGD with random image transformations to maximize the flow-matching loss.
  • MYOPIA. We alternate Transformer LoRA updates with perturbation updates that minimize the velocity prediction error. We bias flow-matching timestep sampling toward higher noise levels and use a JND mask to constrain the perturbations.
  • ACE. We alternate Transformer LoRA updates with perturbation updates that drive the predicted velocity toward a fixed target tensor derived from the target image’s latent representation. Our fused variant also includes a correlation term between predicted and ground-truth velocities and a latent alignment loss.
  • BlurGuard. We use each model’s VAE to define the latent suppression objective and optimize perturbations through region-wise Gaussian filters. We learn the filter scales by minimizing the power spectrum discrepancy between the original and protected images.
  • TAFAP. We differentiate through flow-matching fine-tuning steps in each Transformer’s LoRA parameter space and optimize perturbations so that the resulting parameters match the endpoint of a sampled segment of a precomputed reference trajectory.

We added the above details to the revised manuscript 🔗L990-1056.

W5: Definitions of CC, CT, TC, and TT can be improved more clearly.

Re: We sincerely thank you for this helpful comment. We have revised the manuscript 🔗L230-232 to make these definitions more clearly.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer mjtD,

We understand this discussion period can be a busy time and we are grateful for your helpful reviews. Guided by your suggestions, we added experiments on region token relations, hidden-state displacement, and alternative target anchors, and clarified details in the revised paper. As the discussion period draws to a close (Nov 18, AOE), we would appreciate hearing whether any concerns remain. If your concerns have been adequately addressed, we would be truly grateful if you could consider a more favorable assessment. Thank you very much!

Best regards,

The authors

Reviewer_KSzAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies image perturbations that prevent unauthorized facial personalization of diffusion transformers. DiTCloak aligns protected images with a target pattern at the VAE and diffusion-prediction levels, and adds losses on token relations and attention maps. The perturbations are optimized by alternating surrogate DreamBooth-LoRA adaptation with projected gradient updates. Experiments cover CelebA-HQ and VGGFace2, three DiT backbones, and both training-based and training-free personalization. The paper also reports cross-model transfer, objective ablations, and eight image-processing or purification countermeasures. The main result is a substantial reduction in the reported identity and face-quality scores relative to nine defenses.

Critical strengths
  1. The empirical improvement is large in several relevant settings. For example, on CelebA-HQ with PixArt and DreamBooth LoRA, Table 1 reports FSS of -0.93 for DiTCloak, compared with 0.71 for clean references and 0.29 for Anti-DB. The evaluation also includes flow-matching backbones and FreeCus, so the contribution is supported by more than one generator or personalization procedure. This breadth is the main reason I view the paper positively.
  2. The proposed structural losses are explicit and straightforward to implement, and Table 2 provides useful evidence that each term contributes to the full method. The guidance comparison is also helpful: it tests alternatives rather than simply assuming that a shared target is preferable. Together with the countermeasure experiments, these results make the method a credible starting point for protecting images against DiT personalization.
Critical weaknesses
  1. FSS assigns -1 to undetected faces, mixing face-generation failure with identity mismatch. Reporting detection rates and similarity on detected faces separately would clarify how much identity leakage remains; AVG also combines metrics with different ranges.
  2. Protection weakens across several model pairs and after preprocessing: Gaussian blur raises FSS from -0.9327 to 0.3758 in Table 6. Fixed countermeasure settings do not establish robustness to an attacker who tunes preprocessing or the personalization budget.
  3. The claim of imperceptible perturbations lacks quantitative evaluation of the protected portraits. A shared perturbation budget does not ensure comparable visual quality across methods; matched-distortion comparisons and optimization costs would help assess practical use.
  4. Key experimental details are missing, including identity and sample counts, surrogate adaptation steps, and PGD step size. Ablation uncertainty and a comparison giving targeted baselines the same radial anchor would also help isolate the contribution of the structural losses.
Questions for authors
  1. Can you separate face-detection failures from identity similarity on detected faces, and report an identity-verification success rate for the main comparisons?
  2. What explains the different clean values in Tables 1 and 6? How much protection remains when the attacker varies blur or JPEG strength and the personalization training budget?
  3. What are the evaluated identity and sample counts, surrogate adaptation steps, PGD step size, and optimization cost per subject?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KSzA [1/2]

Thanks for your detailed and insightful suggestions on our work! You can find the responses to your concerns as follows:

W1 & Q1: (a) Report face-detection failures detection rate and verification success rate; (b) AVG’s mismatched metric ranges.

Re: We sincerely thank for this insightful comment. (a) Guided by your suggestion, we report three metrics on CelebA-HQ, PixArt-α\alpha-XL/2 + DreamBooth LoRA over 30 generated samples: Detection Rate (DR): Percentage of images with detected faces, Similarity on Detected Faces (SDF): Identity similarity over detected faces, Identity Verification Success Rate (IVSR): Percentage of generated images with a detected face whose target-identity similarity is above threshold τ\tau. The detailed results are below:

SchemeDR (↓)SDF (↓)IVSR (τ=0.4\tau=0.4, ↓)IVSR (τ=0.5\tau=0.5, ↓)IVSR (τ=0.6\tau=0.6, ↓)IVSR (τ=0.7\tau=0.7, ↓)
PhotoGuard1.0000.5400.730.600.430.03
Glaze1.0000.5560.930.670.330.10
MYOPIA1.0000.6781.000.970.830.63
DiTCloak (Ours)0.0670.0100.000.000.000.00

The results reveal that our protection does not rely solely on face-detection failures, as detected faces also exhibit low identity similarity and fail identity verification, highlighting the superiority of DiTCloak. Specifically, at a verification threshold of 0.4, identity leakage was 73% (22/30) for PhotoGuard, 93% (28/30) for Glaze, and 100% (30/30) for MYOPIA. With DiTCloak, no identity leakage was observed (0/30) at this threshold; (b) We agree that "AVG" alone does not fully capture overall performance. Since all four metrics are lower-is-better, "AVG" serves only as a directionally consistent descriptive summary, while our conclusions are drawn jointly from the individual metrics. In the revision, we have added discussion about face detection rate 🔗L459-463 and clarified how AVG should be used 🔗L882-887.


W2 & Q2: (a) Protection weakens across model pairs and after preprocessing; (b) protection under varying blur/JPEG strengths/personalization budgets; (c) inconsistent clean baselines in Tabs 1 and 6.

Re: We deeply appreciate this constructive feedback. (a) We agree that transferability varies across surrogate-target pairs and we have explicitly discussed this issue in the main text accompanying Fig. 4 🔗L411-413. We acknowledge that countermeasures weaken protection, as blurring and JPEG compression can suppress high-frequency components of the protective perturbations; (b) Following your suggestion, we add results as below:

Gaussian Blur

Kernel sizeFSSSER-FQAIMS-CLIPIMS-VGG
Clean0.7070.8090.8670.618
30.3760.6920.6940.408
50.4770.6760.7080.451
70.5010.7130.7200.466
90.5240.7230.7150.454

JPEG Compression

Quality factorFSSSER-FQAIMS-CLIPIMS-VGG
Clean0.7070.8090.8670.618
650.2250.5330.6720.367
750.1670.4480.6870.359
850.1100.2080.6580.388
95-0.7930.1010.6140.265

Personalization Training Budget

Training stepsFSSSER-FQAIMS-CLIPIMS-VGG
Clean0.7070.8090.8670.618
600-0.5540.2740.5280.180
800-0.5730.2690.5690.266
1000-0.9330.0240.5870.226
1200-0.7660.1110.5960.257

These results verify that DiTCloak retains its protective effectiveness across varying countermeasure strengths. In the revised manuscript, we have added these results 🔗L476-478; (c) The discrepancy arose from mistakenly copying the Stable Diffusion 3.5 Medium clean baseline into Tab. 6. We have corrected this in the revision 🔗L922.

AuthorsOpenReview ↗

Response to Reviewer KSzA [2/2]

W3: (a) Lacks quantitative visual evaluation; (b) matched-distortion comparison; (c) optimization cost.

Re: We greatly appreciate this feedback. (a) We report visual metrics PSNR, LPIPS, and SSIM on CelebA-HQ w/ PixArt-α\alpha-XL/2 + DreamBooth LoRA:

SchemePSNRLPIPSSSIM
PhotoGuard28.040.410.72
Glaze27.341.510.63
MetaCloak27.680.500.69
DiTCloak (Ours)27.170.460.65

DiTCloak achieves visual quality comparable to Glaze and MetaCloak, while PhotoGuard slightly outperforms them. Fig. 1 of the manuscript also shows that DiTCloak’s UEs are visually subtle under this norm constraint; (b) We increased PhotoGuard’s perturbation budget to match DiTCloak’s PSNR. The PSNR-distortion comparison shows that DiTCloak achieves stronger protection compared with PhotoGuard.

SchemeFSSSER-FQAIMS-CLIPIMS-VGGPSNRLPIPSSSIMρ\rho
PhotoGuard0.5220.7210.7320.53527.170.450.6918 / 255
DiTCloak-0.9330.0240.5870.22627.170.460.6516 / 255

(c) We measured DiTCloak’s perturbation optimization cost on a system with an NVIDIA RTX PRO 5000 GPU with a memory size of 71.69 GiB and an AMD EPYC 9115 16-core CPU. DiTCloak’s perturbation optimization for each subject with FLUX.2-klein-base-4B took 2,482.8 s on CelebA-HQ and 2,486.4 s on VGGFace2, requiring 109,631.76 TFLOPs per run. With PixArt-α\alpha-XL/2, it took 839.4 s on CelebA-HQ and 878.4 s on VGGFace2, requiring 17,740.35 TFLOPs per run. In revision, we address (a) and (b) in 🔗L1060-1066 and (c) in 🔗L792-798.


W4 & Q3: (a) Missing details; (b) ablation uncertainty; (c) targeted baselines comparison w/ the same radial anchor

Re: We sincerely thank you for careful reviews. (a) Identity and sample counts are 4 and 200. Surrogate adaptation runs for 20 steps per round over 5 rounds, totaling 100 steps. PGD uses a step size of 2/255, with 50 updates per round and 250 updates in total; (b) Following your comments, we report mean ± standard deviation over three random seeds ([42], [1024], [2027]) to quantify ablation uncertainty:

Jvl\mathcal{J}_{vl}Jdp\mathcal{J}_{dp}Jtr\mathcal{J}_{tr}Jma\mathcal{J}_{ma}FSS (↓)SER-FQA (↓)IMS-CLIP (↓)IMS-VGG (↓)AVG (↓)
✓✓✓0.474 ± 0.0060.767 ± 0.0190.714 ± 0.0050.473 ± 0.0140.607 ± 0.002
✓✓✓0.161 ± 0.0260.362 ± 0.0410.646 ± 0.0060.326 ± 0.0080.374 ± 0.017
✓✓✓-0.551 ± 0.0250.208 ± 0.0210.589 ± 0.0020.227 ± 0.0090.118 ± 0.009
✓✓✓0.116 ± 0.0780.309 ± 0.1020.642 ± 0.0180.334 ± 0.0320.350 ± 0.032
✓✓✓✓-0.933 ± 0.0270.029 ± 0.0120.586 ± 0.0030.226 ± 0.004-0.023 ± 0.011

The variability across the three seeds is small relative to the effect of removing any loss term, indicating that the ablation conclusions are stable across the tested seeds; (c) We compare DiTCloak with three baselines involving target anchors (PhotoGuard, Anti-DreamBooth, and ACE), using the same radial anchor to optimize perturbations for these methods, on CelebA-HQ w/ PixArt-α\alpha-XL/2 + DreamBooth LoRA:

The same radial anchorFSSSER-FQAIMS-CLIPIMS-VGG
Clean0.7070.8090.8670.618
PhotoGuard0.2660.4770.6510.395
Anti-DB0.2320.4310.6590.422
ACE0.4170.7340.6990.445
DiTCloak-0.9330.0240.5870.226

The results show that DiTCloak still outperforms baselines. In the revised manuscript, we address (a) in 🔗L770-773, (b) in 🔗L1067-1072, and (c) in 🔗L1074-1079

Reviewer_KSzAOpenReview ↗

Comments on Rebuttal

I thank the authors for reporting additional results regarding my concerns, so I maintain my positive assessment and my rating.

AuthorsOpenReview ↗

Thank You for Your Follow-Up

Thank you for taking the time to review our additional results. Your feedback helped us strengthen the evaluation, and we appreciate your positive assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.