- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose a novel approach to avoid unauthorized personalization in DiT-based T2I generation. Their approach, called DiTCloak, combines target-guided dual alignment, token relation anchoring, and multi-head attention reorientation objectives to optimize unlearnable examples (UEs) in order to prevent malicious reproduction of personal content. DiTCloak addresses two limitations of DiT-based T2I generation: 1) insufficient token-relation decorrelation; and 2) limited token-attention distortion. Extensive experiments on three DiT-based T2I backbones, two face datasets, two diffusion architectures, and both training-based and training-free personalization settings show the superiority of DiTCloak with respect to the state of the art. Furthermore, DiTCloak demonstrates strong robustness against purification methods (JPEG compression, resizing, Gaussian blur/noise, etc.). Additionally, UEs generated via DiTCloak transfer effectively across different diffusion architectures, including both DiT and U-Net models.
Critical strengths
These are the paper's strengths:
- it is one of the first papers that addresses the problem of unauthorized personalization in DiT-based T2I generation
- the approach overcomes two limitations of DiT-based T2I generation: 1) insufficient token-relation decorrelation; and 2) limited token-attention distortion
- experimental validation is extensive and demonstrates the superiority of the proposed approach with respect to the state of the art
- DiTCloak shows robustness against several purification methods
- UEs generated via DiTCloak transfer effectively across different diffusion architectures
Critical weaknesses
These are the paper's weaknesses:
- the evaluation focuses mainly on facial privacy and identity reproduction
- the performance relies on several loss-balancing and optimization hyper-parameters (, , , etc.), which require careful manual fine-tuning
- optimizing perturbations requires a surrogate model and specific personalization setups (e.g. DreamBooth LoRA), which might limit optimization efficiency if the adversary uses a different pipeline.
- the optimization process (alternating rounds of surrogate adaptation and M-step PGD with multiple structural loss evaluations per update) could add a significant computational overhead
- the paper lacks an user study in order to confirm the findings obtained through automated metrics
Questions for authors
These are my concerns:
- how well does your approach cope with the protection of non-face subjects, such as: personalized objects, pets, artistic styles? It would be appreciated if you can show through expeiments in some of these cases
- since the UEs are optimized using specific surrogate models and fine-tuning pipelines (such as DreamBooth LoRA), how does protection efficiency is affected if an adversary adopts an entirely different personlization scheme?
- what is the computational overhead introduced by your optimization process (alternating rounds of surrogate adaptation and M-step PGD with multiple structural loss evaluations per update)?
- how sensitive is the protection performance to the hyper-parameters ((, , , etc.) used in the loss functions across different DiT backbones? Is there an automated way to tune them?
- the structural objectives target evenly spaced blocks from the middle to the final DiT blocks. How do a different subset of blocks contribute to the preservation of identity? Could targeting a different subset of blocks optimize the trade-off between protection and perturbation budget?
- I recommend to perform an user study in order to confirm the results obtained by automated metrics
9 comments
Response to Reviewer 53Wz [2/3]
W3 & Q2: DreamBooth–optimized UEs' transferability across a different personalization scheme?
Re: We truly appreciate it for asking us to clarify this point. We note that, in the Table 1 of the manuscript (UEs are optimized using DreamBooth LoRA), the evaluation covers attacker's personalization pipelines DreamBooth LoRA and FreeCus. The results reveal that our scheme DiTCloak transfers effectively against FreeCus, outperforming baselines. We have made this more clearly in revision 🔗L326.
W4 & Q3: Computational overhead of DiTCloak optimization.
Re: We greatly appreciate it for raising this point. Based on your suggestion, we measured DiTCloak’s perturbation optimization cost on a system with an NVIDIA RTX PRO 5000 GPU with a memory size of 71.69 GiB and an AMD EPYC 9115 16-core CPU. DiTCloak’s perturbation optimization with FLUX.2-klein-base-4B took 2,482.8 s on CelebA-HQ and 2,486.4 s on VGGFace2, requiring 109,631.76 TFLOPs per run. With PixArt--XL/2, it took 839.4 s on CelebA-HQ and 878.4 s on VGGFace2, requiring 17,740.35 TFLOPs per run. We have added discussion about this point In revision 🔗L792-798.
W5 & Q6: User-study on automated results.
Re: We sincerely thank you for raising this important point. Based on your suggestion, we recruited 10 participants: 3 PhD students, 2 master students, and 5 undergraduates. Participants were aged 21–26 years, with a mean age of 23.3 years. Each participant compared 30 generated images individually with one reference image of the target identity and answered, “Do these two images show the same person?” ('yes' or 'no') Each “yes” counted as one match. We averaged the match counts across the 10 participants and report the mean number of matches out of 30 (↓).
| Scheme | CelebA-HQ (PixArt-α-XL/2 + DreamBooth LoRA) | VGGFace2 (FLUX.2-klein-base-4B + FreeCus) |
|---|---|---|
| Glaze | 21.5 | 24.6 |
| ACE | 29.4 | 17.0 |
| BlurGuard | 25.8 | 27.8 |
| TAFAP | 26.9 | 25.2 |
| DiTCloak (Ours) | 4.7 | 0.2 |
DiTCloak provided the best facial identity protection in this user study setting, with participants identifying fewer generated images as the target person than baselines. We added these results to the revised manuscript 🔗L1134-1144.
Response to Reviewer 53Wz [3/3]
Q5: (a) Block-subset effects; (b) selections for better protection-budget trade-offs.
Re: We truly appreciate this constructive suggestion. (a) Guided by suggestions, we further evaluate performance with three different subset of blocks on CelebA-HQ using PixArt-XL-2-512x512 with DreamBooth LoRA:
| Selected blocks | FSS (↓) | SER-FQA (↓) | IMS-CLIP (↓) | IMS-VGG (↓) |
|---|---|---|---|---|
| Clean baseline | 0.707 | 0.809 | 0.867 | 0.618 |
| Full | 0.148 | 0.367 | 0.591 | 0.246 |
| First half | 0.114 | 0.475 | 0.638 | 0.230 |
| Middle third | 0.067 | 0.418 | 0.614 | 0.286 |
| Middle to the final (Ours) | -0.933 | 0.024 | 0.587 | 0.226 |
It reveals that our strategy yields better performance, likely because latter DiT blocks progressively refine high-frequency visual details [1]. Disrupting this may hinder faithful reproduction of the target identity; (b) We further evaluated the same block subsets at a lower perturbation budget of , with results as below:
| Selected blocks | FSS, 8/255 | SER-FQA, 8/255 | IMS-CLIP, 8/255 | IMS-VGG, 8/255 | FSS, 16/255 | SER-FQA, 16/255 | IMS-CLIP, 16/255 | IMS-VGG, 16/255 |
|---|---|---|---|---|---|---|---|---|
| Clean baseline | 0.707 | 0.809 | 0.867 | 0.618 | 0.707 | 0.809 | 0.867 | 0.618 |
| Full | 0.585 | 0.739 | 0.770 | 0.557 | 0.148 | 0.367 | 0.591 | 0.246 |
| First half | 0.568 | 0.770 | 0.769 | 0.567 | 0.114 | 0.475 | 0.638 | 0.230 |
| Middle third | 0.619 | 0.751 | 0.754 | 0.547 | 0.098 | 0.523 | 0.614 | 0.286 |
| Middle to the final (Ours) | 0.541 | 0.737 | 0.753 | 0.495 | −0.933 | 0.024 | 0.587 | 0.226 |
It can be seen that targeting blocks from the midpoint to the end gave the strongest protection at both and , achieving the lowest scores on all four metrics. Thus, among the subsets tested, our block selection offers the best protection at either budget. In the revised manuscript, we added the above results in 🔗L495-503 and the discussion in 🔗L471-474.
[1] W-EDIT: A wavelet-based frequency-aware framework for text-driven image editing. In Proc. of ICLR'26
Response to Reviewer 53Wz [1/3]
We sincerely thank the reviewer for the thoughtful, professional, and detailed feedback, which helps us further improve our work. Our point-by-point responses are provided below:
Re: We sincerely thank you for drawing attention to this aspect of our work. Based on your suggestion, we extend our evaluation to personalized objects and pets (backpack and dog), and artistic styles (Edward Hopper).
a photo of sks doganda photo of sks dog wearing a red hat, for the backpack subject area photo of sks backpackanda photo of sks backpack on a wooden floor.a painting of mountain by nulevoyanda painting of piano by nulevoy.It reveals that DiTCloak achieves the lowest scores across reported metrics, indicating better protection against personalization of objects, pets, and artistic styles in the tested cases. In the revised manuscript, we added the experimental results in 🔗L486-494 and the discussion in 🔗L464-466.
Re: We deeply appreciate for this helpful suggestion. (a) Guided by your suggestions, we fixed λvl and jointly scaled λdp, λtr, and λma by 0.5×, 1.0×, or 2.0× relative to that backbone’s default weights on different backbones. We reoptimized the perturbations for each setting and held the personalization and evaluation protocols fixed:
The default weights gave the lowest scores on all four metrics for both backbones. Halving or doubling the weights increased AVG by 0.369 or 0.571 on PixArt-α-XL/2 and by 0.127 or 0.112 on Stable Diffusion 3.5 Medium, indicating greater sensitivity to joint scaling on PixArt-α-XL/2 in this sweep; (b) To automate selection, we initialized the loss weights for Stable Diffusion 3.5 Medium from (λvl, λdp, λtr, λma)=(1, 1, 0.1, 0.1) and automatically selected a shared multiplier for λdp, λtr, and λma from {0.5, 1.0, 2.0} while keeping λvl fixed. We performed a grid search over three loss-weight multipliers by minimizing validation Identity Verification Success Rate (IVSR) (percentage of generated images with a detected face whose target-identity similarity is above threshold). On Stable Diffusion 3.5 Medium, at a verification threshold of 0.6, the rates were 6.7%, 0%, and 3.3% for 0.5×, 1.0×, and 2.0×, respectively, so this rule selects 1.0×, i.e., λvl=1, λdp=1, λtr=0.1, and λma=0.1. We added the above results and discussion to the revised manuscript 🔗L1120-1133.