- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the privacy leakage risks associated with personalized diffusion models in scenarios where an attacker possesses a partially visible version of a target image. The authors propose a method called Partial-Image-Assisted Reconstruction (PIAR), which integrates BrushNet-based image inpainting with "Cross-Model Contrastive Guidance" to reconstruct the occluded portions of a target image.
Critical strengths
- This work considers a scenario where the attacker has already obtained a partial view of a target image.
- The overall approach is straightforward to implement.
- The authors evaluate the effectiveness of the method across different datasets and provide further analysis of factors.
Critical weaknesses
- The practical plausibility of the threat model is not sufficiently justified. The paper assumes that an attacker can simultaneously access a publicly released personalized diffusion model checkpoint and a partially occluded version of an image used in its training set. However, the paper fails to adequately explain how such an attack scenario would realistically occur. Specifically, it is unclear why a user would train a personalized model using private photos and subsequently release the checkpoint publicly while simultaneously posting an occluded version of the same photo on social media or other public platforms. Although the authors mention potential sources like social media or public websites, they provide no concrete real-world examples or evidence to support this assumption. The current threat model appears overly idealized, and its practical significance regarding privacy risks remains unproven.
- The technical innovation of the proposed method is limited. PIAR primarily combines existing BrushNet inpainting with the "Cross-Model Guidance" technique previously introduced in FineXtract. Both core components are derived from prior research, and their integration is relatively straightforward, lacking significant novel technical design. Furthermore, the experimental results indicate that the majority of the performance gains stem from using the personalized model for inpainting, rather than from the "Contrastive Guidance" mechanism emphasized by the authors. For instance, on the CelebA-HQ dataset, "Personalized Inpainting" already achieves an average similarity of 0.886, while PIAR improves this only marginally to 0.909.
- Current evaluation metrics are insufficient to accurately measure the recovery of hidden, private information. The primary objective of this work is to reconstruct the content obscured within partially visible images. However, all evaluation metrics are calculated based on the entire image, including the visible regions already accessible to an attacker. In the main experimental setup, only 32.3% of the image area is obscured on average, meaning that approximately 67.7% of the image content is already available to the attacker as input. Consequently, whole-image similarity scores may be heavily skewed by the visible regions, and a high reconstruction similarity does not necessarily imply that the hidden content has been accurately recovered. The authors should provide evaluation results specifically for the masked regions, such as LPIPS, SSIM, and reconstruction error, to more precisely validate the claimed privacy leakage risks.
- In real-world scenarios, privacy protection or information loss in images often manifests through techniques like blurring, pixelation (mosaicing), cropping, or low-resolution downsampling, rather than just solid black masks. The paper has not verified whether PIAR remains effective under these more realistic image transformations. Furthermore, the experiments do not account for other personalization methods, such as LoRA or Custom Diffusion, nor do they evaluate the approach against updated diffusion model architectures. Additionally, while the study employs BrushNet as the inpainting backbone, it lacks comparisons with other diffusion-based inpainting methods, such as VIPaint or PowerPaint.
- All experiments in this paper rely on personalized model checkpoints trained by the authors under uniform, strictly controlled conditions. In practice, however, publicly released personalized models vary significantly in terms of training data scale, fine-tuning strategies, computational budgets, regularization techniques, and the extent to which they memorize their training data. Given that the paper’s primary motivation concerns the privacy risks posed by personalized models shared on platforms like Hugging Face or Civitai, the authors should validate the effectiveness of PIAR on actual, publicly available models.
Questions for authors
- Could the authors provide a concrete, realistic scenario illustrating how an attacker might simultaneously obtain a checkpoint of a personalized diffusion model and the partially occluded images corresponding to its training data?
- Given that the primary objective of PIAR is to recover occluded private content, why does the paper use whole-image similarity as the main evaluation metric, rather than directly measuring the reconstruction quality within the masked regions? If metrics such as SSIM, LPIPS, or others were calculated specifically for the occluded areas, would PIAR still demonstrate a significant advantage over "Personalized Inpainting" and "Base Inpainting" methods?
- Since "Cross-Model Guidance" has already been introduced in FineXtract, and BrushNet is an existing image inpainting method, what are the core technical innovations of PIAR?
7 comments
Masked-region results supporting W3/Q2
For convenient reference, we reproduce Table 17 from Appendix D.1 of the revised manuscript below. PIAR outperforms both inpainting baselines on all three masked-region metrics across all three datasets.
The evaluation uses reconstructions generated with the same seeds, protocol, and candidate-selection procedure as Table 3’s first run. Metrics are averaged over the masked region. Results are mean ± sample standard deviation across subjects within a single run: 50 for CelebA-HQ and 30 for each object dataset. Values are not scaled by 100.
| Dataset | Method | MSE ↓ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| CelebA-HQ | Base Inpainting | 2037.6 ± 842.6 | 0.4392 ± 0.0662 | 0.4111 ± 0.0607 |
| Personalized Inpainting | 488.6 ± 229.2 | 0.6568 ± 0.0574 | 0.1636 ± 0.0315 | |
| PIAR | 398.2 ± 202.1 | 0.6777 ± 0.0535 | 0.1314 ± 0.0240 | |
| DreamBooth | Base Inpainting | 5801.7 ± 4654.9 | 0.2268 ± 0.0938 | 0.6268 ± 0.0903 |
| Personalized Inpainting | 3400.9 ± 3162.5 | 0.3534 ± 0.1449 | 0.4105 ± 0.1147 | |
| PIAR | 2414.4 ± 1152.5 | 0.4285 ± 0.1792 | 0.3200 ± 0.0886 | |
| CustomConcept101 | Base Inpainting | 3958.8 ± 1905.0 | 0.2737 ± 0.0770 | 0.5515 ± 0.0738 |
| Personalized Inpainting | 2282.2 ± 1408.2 | 0.3777 ± 0.1033 | 0.3725 ± 0.1155 | |
| PIAR | 2164.2 ± 1420.9 | 0.4137 ± 0.0982 | 0.3151 ± 0.1103 |
Thanks for your rebuttal.
W3/Q2: Thanks for your additional experiments. The results are promising.
W2/Q3+W1/Q1: I got that this paper reveals a new privacy risk. But I am still curious why we need to mask the image directly. If we mask all faces to protect our privacy, it is not necessary for us to upload our image. Right? I can not find any suitable practical scenario here, although this is a new risk. It would be better if you could illustrate the scenario more practically.
W4/W5: Thanks for your additional experiments. It would be better if you could add the blurring experiment to the discussion.
Overall, I have increased my score to 2 considering the new experimental results. However, the practical threat should be illustrated to show the significance in the real world. If you can solve this, I would like to further increase my score. Thanks.
Response to Reviewer eXSc
Dear Reviewer eXSc,
Thank you for your review.
W3 / Q2: Masked-region metrics
We considered masked-region evaluation but chose whole-image metrics for comparability across five methods: free-form baselines lack spatial correspondence to the target’s masked region. We agree that masked metrics strengthen evidence for hidden-content recovery.
We have evaluated masked-region MSE, SSIM, and LPIPS on all three datasets using the same seeds, protocol, and candidate selection as Table 3’s first run. PIAR outperforms both inpainting baselines on every metric and dataset, with larger gains than under whole-image evaluation. On DreamBooth, masked-region LPIPS falls from 0.4105 for Personalized Inpainting to 0.3200 for PIAR. The gains reflect hidden-content recovery, not merely visible-region agreement. Results and details are in Appendix D.1 (Table 17).
W2 / Q3: Guidance and novelty
Our central novelty is the identification and systematic validation of an overlooked privacy risk, instead of providing the optimal and novel attack for this risk. The fact that combining existing techniques already yields strong attacks actually further strengthens our contribution, which demonstrates that the overlooked privacy risk that we identified is indeed practical and crucial. As motivated in the Introduction (lines 83–95), concealing sensitive content may be insufficient when a personalized checkpoint of the same subject is accessible, a risk unaddressed by prior free-form extraction. Our contributions include personalization and guidance controls, checkpoint-specificity tests, and held-out and generation-budget evaluations. PIAR is modular and requires no additional attack training; FLUX–ControlNet demonstrates applicability to another base model and inpainting module (Appendix D.2, Table 22). We acknowledge inpainting and cross-model guidance in Section 4 (lines 195–204 and 214–216), without claiming them as novel.
We respectfully disagree that contrastive guidance contributes little: gains are substantial on object datasets and under extensive masking. Guidance raises AS over Personalized Inpainting from 0.681 to 0.764 on DreamBooth and 0.725 to 0.780 on CustomConcept101 (Table 1), and from 0.631 to 0.775 under extra-large CelebA-HQ masks (Table 6). The cited small-mask case, where Personalized Inpainting already achieves AS 0.886, does not capture this broader benefit. Recovery despite concealing much more of the photograph is particularly concerning.
W1 / Q1: Threat-model realism
The target photograph need not belong to the fine-tuning set. Section 3 (lines 174–181) describes this setting; Section 5.4 (lines 453–461) shows that personalization aids reconstruction of held-out photographs of the same subject.
An attacker could train a personalized model from public photographs of a person, then reconstruct a different photograph with their face obscured for privacy. The subject need not share a checkpoint. Making a subject available for personalized generation does not imply an intention to disclose the concealed content of every photograph depicting that subject. We have added this example to the Introduction (lines 83–90).
W4: Obfuscations, personalization, and inpainting
Obfuscation types. Solid masking removes the concealed content entirely. Localized blur or pixelation may retain information; even without exploiting it, an attacker can discard the affected region and apply masked reconstruction when its location is known and surrounding content remains available. Exploiting residual information could further aid recovery, a future direction noted in Section 6. We will attempt a blurring experiment, but may not finish within the rebuttal period.
Personalization and architectures. The risk extends beyond the original configuration. We evaluate DreamBooth-LoRA, Custom Diffusion, DisenBooth, and FLUX.1-dev with DreamBooth-LoRA. With LoRA on DreamBooth, PIAR achieves AS 0.702, versus 0.638 for Personalized Inpainting, 0.473 for Base Inpainting, and 0.464 for FineXtract. Results and settings are in Appendix D.2 (Tables 19–22).
Inpainting backbones. BrushNet adds pretrained inpainting without modifying either checkpoint’s weights or requiring additional training; both branches share the module (Section 4). FLUX instead uses ControlNet inpainting: on ten DreamBooth subjects, personalization raises AS from 0.496 to 0.646, and PIAR to 0.678. This shows that the risk persists in this alternative setup (Appendix D.2, Table 22).
W5: Validation on public checkpoints
Controlled checkpoints enable reproducible, scientifically rigorous evaluation: known training photographs allow ground-truth comparison and separation of fine-tuning from held-out images. These checks are not reliably possible for community checkpoints with undisclosed training data. As stated in our submitted Ethics Statement, we deliberately avoided attacking third-party personalized checkpoints to limit potential harm.