acceptodds
Under review as a conference paper at ICLR 2027

Partial Images Raise Further Privacy Concerns in Personalized Diffusion Models

Abstract

Personalized diffusion models can adapt to user-specific subjects from a small set of training images, enabling generation in novel scenes, styles, and compositions. Their checkpoints can be shared for reuse without disclosing the private fine-tuning images, which may appear privacy-preserving. However, information retained in the checkpoints can still enable recovery of those images. While prior extraction methods primarily study this risk through free-form generation, partially obscured or redacted versions of private images may also unintentionally remain accessible through social media or other public sources. The privacy implications of such auxiliary information remain largely unexplored. We therefore investigate how this information can raise further privacy concerns in released personalized checkpoints. To demonstrate it, we introduce Partial-Image-Assisted Reconstruction (PIAR), which combines image inpainting with personalization guidance to reconstruct complete images from partial observations. Our experiments show that PIAR reconstructs private images substantially more accurately than free-form extraction. The risk extends beyond the fine-tuning images, as PIAR also improves reconstruction of unseen photographs of the same personalized subject. Together, these findings highlight how partial visual information can expose sensitive content through released checkpoints and motivate stronger privacy protections for personalized diffusion models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Privacy, Reconstruction, Masked image, Training image, Fine-tuning, Personalized model, Diffusion model, Inpainting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eXScOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the privacy leakage risks associated with personalized diffusion models in scenarios where an attacker possesses a partially visible version of a target image. The authors propose a method called Partial-Image-Assisted Reconstruction (PIAR), which integrates BrushNet-based image inpainting with "Cross-Model Contrastive Guidance" to reconstruct the occluded portions of a target image.

Critical strengths
  1. This work considers a scenario where the attacker has already obtained a partial view of a target image.
  2. The overall approach is straightforward to implement.
  3. The authors evaluate the effectiveness of the method across different datasets and provide further analysis of factors.
Critical weaknesses
  1. The practical plausibility of the threat model is not sufficiently justified. The paper assumes that an attacker can simultaneously access a publicly released personalized diffusion model checkpoint and a partially occluded version of an image used in its training set. However, the paper fails to adequately explain how such an attack scenario would realistically occur. Specifically, it is unclear why a user would train a personalized model using private photos and subsequently release the checkpoint publicly while simultaneously posting an occluded version of the same photo on social media or other public platforms. Although the authors mention potential sources like social media or public websites, they provide no concrete real-world examples or evidence to support this assumption. The current threat model appears overly idealized, and its practical significance regarding privacy risks remains unproven.
  2. The technical innovation of the proposed method is limited. PIAR primarily combines existing BrushNet inpainting with the "Cross-Model Guidance" technique previously introduced in FineXtract. Both core components are derived from prior research, and their integration is relatively straightforward, lacking significant novel technical design. Furthermore, the experimental results indicate that the majority of the performance gains stem from using the personalized model for inpainting, rather than from the "Contrastive Guidance" mechanism emphasized by the authors. For instance, on the CelebA-HQ dataset, "Personalized Inpainting" already achieves an average similarity of 0.886, while PIAR improves this only marginally to 0.909.
  3. Current evaluation metrics are insufficient to accurately measure the recovery of hidden, private information. The primary objective of this work is to reconstruct the content obscured within partially visible images. However, all evaluation metrics are calculated based on the entire image, including the visible regions already accessible to an attacker. In the main experimental setup, only 32.3% of the image area is obscured on average, meaning that approximately 67.7% of the image content is already available to the attacker as input. Consequently, whole-image similarity scores may be heavily skewed by the visible regions, and a high reconstruction similarity does not necessarily imply that the hidden content has been accurately recovered. The authors should provide evaluation results specifically for the masked regions, such as LPIPS, SSIM, and reconstruction error, to more precisely validate the claimed privacy leakage risks.
  4. In real-world scenarios, privacy protection or information loss in images often manifests through techniques like blurring, pixelation (mosaicing), cropping, or low-resolution downsampling, rather than just solid black masks. The paper has not verified whether PIAR remains effective under these more realistic image transformations. Furthermore, the experiments do not account for other personalization methods, such as LoRA or Custom Diffusion, nor do they evaluate the approach against updated diffusion model architectures. Additionally, while the study employs BrushNet as the inpainting backbone, it lacks comparisons with other diffusion-based inpainting methods, such as VIPaint or PowerPaint.
  5. All experiments in this paper rely on personalized model checkpoints trained by the authors under uniform, strictly controlled conditions. In practice, however, publicly released personalized models vary significantly in terms of training data scale, fine-tuning strategies, computational budgets, regularization techniques, and the extent to which they memorize their training data. Given that the paper’s primary motivation concerns the privacy risks posed by personalized models shared on platforms like Hugging Face or Civitai, the authors should validate the effectiveness of PIAR on actual, publicly available models.
Questions for authors
  1. Could the authors provide a concrete, realistic scenario illustrating how an attacker might simultaneously obtain a checkpoint of a personalized diffusion model and the partially occluded images corresponding to its training data?
  2. Given that the primary objective of PIAR is to recover occluded private content, why does the paper use whole-image similarity as the main evaluation metric, rather than directly measuring the reconstruction quality within the masked regions? If metrics such as SSIM, LPIPS, or others were calculated specifically for the occluded areas, would PIAR still demonstrate a significant advantage over "Personalized Inpainting" and "Base Inpainting" methods?
  3. Since "Cross-Model Guidance" has already been introduced in FineXtract, and BrushNet is an existing image inpainting method, what are the core technical innovations of PIAR?
7 comments
AuthorsOpenReview ↗

Response to Reviewer eXSc

Dear Reviewer eXSc,

Thank you for your review.

W3 / Q2: Masked-region metrics

We considered masked-region evaluation but chose whole-image metrics for comparability across five methods: free-form baselines lack spatial correspondence to the target’s masked region. We agree that masked metrics strengthen evidence for hidden-content recovery.

We have evaluated masked-region MSE, SSIM, and LPIPS on all three datasets using the same seeds, protocol, and candidate selection as Table 3’s first run. PIAR outperforms both inpainting baselines on every metric and dataset, with larger gains than under whole-image evaluation. On DreamBooth, masked-region LPIPS falls from 0.4105 for Personalized Inpainting to 0.3200 for PIAR. The gains reflect hidden-content recovery, not merely visible-region agreement. Results and details are in Appendix D.1 (Table 17).

W2 / Q3: Guidance and novelty

Our central novelty is the identification and systematic validation of an overlooked privacy risk, instead of providing the optimal and novel attack for this risk. The fact that combining existing techniques already yields strong attacks actually further strengthens our contribution, which demonstrates that the overlooked privacy risk that we identified is indeed practical and crucial. As motivated in the Introduction (lines 83–95), concealing sensitive content may be insufficient when a personalized checkpoint of the same subject is accessible, a risk unaddressed by prior free-form extraction. Our contributions include personalization and guidance controls, checkpoint-specificity tests, and held-out and generation-budget evaluations. PIAR is modular and requires no additional attack training; FLUX–ControlNet demonstrates applicability to another base model and inpainting module (Appendix D.2, Table 22). We acknowledge inpainting and cross-model guidance in Section 4 (lines 195–204 and 214–216), without claiming them as novel.

We respectfully disagree that contrastive guidance contributes little: gains are substantial on object datasets and under extensive masking. Guidance raises AS over Personalized Inpainting from 0.681 to 0.764 on DreamBooth and 0.725 to 0.780 on CustomConcept101 (Table 1), and from 0.631 to 0.775 under extra-large CelebA-HQ masks (Table 6). The cited small-mask case, where Personalized Inpainting already achieves AS 0.886, does not capture this broader benefit. Recovery despite concealing much more of the photograph is particularly concerning.

W1 / Q1: Threat-model realism

The target photograph need not belong to the fine-tuning set. Section 3 (lines 174–181) describes this setting; Section 5.4 (lines 453–461) shows that personalization aids reconstruction of held-out photographs of the same subject.

An attacker could train a personalized model from public photographs of a person, then reconstruct a different photograph with their face obscured for privacy. The subject need not share a checkpoint. Making a subject available for personalized generation does not imply an intention to disclose the concealed content of every photograph depicting that subject. We have added this example to the Introduction (lines 83–90).

W4: Obfuscations, personalization, and inpainting

Obfuscation types. Solid masking removes the concealed content entirely. Localized blur or pixelation may retain information; even without exploiting it, an attacker can discard the affected region and apply masked reconstruction when its location is known and surrounding content remains available. Exploiting residual information could further aid recovery, a future direction noted in Section 6. We will attempt a blurring experiment, but may not finish within the rebuttal period.

Personalization and architectures. The risk extends beyond the original configuration. We evaluate DreamBooth-LoRA, Custom Diffusion, DisenBooth, and FLUX.1-dev with DreamBooth-LoRA. With LoRA on DreamBooth, PIAR achieves AS 0.702, versus 0.638 for Personalized Inpainting, 0.473 for Base Inpainting, and 0.464 for FineXtract. Results and settings are in Appendix D.2 (Tables 19–22).

Inpainting backbones. BrushNet adds pretrained inpainting without modifying either checkpoint’s weights or requiring additional training; both branches share the module (Section 4). FLUX instead uses ControlNet inpainting: on ten DreamBooth subjects, personalization raises AS from 0.496 to 0.646, and PIAR to 0.678. This shows that the risk persists in this alternative setup (Appendix D.2, Table 22).

W5: Validation on public checkpoints

Controlled checkpoints enable reproducible, scientifically rigorous evaluation: known training photographs allow ground-truth comparison and separation of fine-tuning from held-out images. These checks are not reliably possible for community checkpoints with undisclosed training data. As stated in our submitted Ethics Statement, we deliberately avoided attacking third-party personalized checkpoints to limit potential harm.

AuthorsOpenReview ↗

Masked-region results supporting W3/Q2

For convenient reference, we reproduce Table 17 from Appendix D.1 of the revised manuscript below. PIAR outperforms both inpainting baselines on all three masked-region metrics across all three datasets.

The evaluation uses reconstructions generated with the same seeds, protocol, and candidate-selection procedure as Table 3’s first run. Metrics are averaged over the masked region. Results are mean ± sample standard deviation across subjects within a single run: 50 for CelebA-HQ and 30 for each object dataset. Values are not scaled by 100.

DatasetMethodMSE ↓SSIM ↑LPIPS ↓
CelebA-HQBase Inpainting2037.6 ± 842.60.4392 ± 0.06620.4111 ± 0.0607
Personalized Inpainting488.6 ± 229.20.6568 ± 0.05740.1636 ± 0.0315
PIAR398.2 ± 202.10.6777 ± 0.05350.1314 ± 0.0240
DreamBoothBase Inpainting5801.7 ± 4654.90.2268 ± 0.09380.6268 ± 0.0903
Personalized Inpainting3400.9 ± 3162.50.3534 ± 0.14490.4105 ± 0.1147
PIAR2414.4 ± 1152.50.4285 ± 0.17920.3200 ± 0.0886
CustomConcept101Base Inpainting3958.8 ± 1905.00.2737 ± 0.07700.5515 ± 0.0738
Personalized Inpainting2282.2 ± 1408.20.3777 ± 0.10330.3725 ± 0.1155
PIAR2164.2 ± 1420.90.4137 ± 0.09820.3151 ± 0.1103

Thanks for your rebuttal.

W3/Q2: Thanks for your additional experiments. The results are promising.

W2/Q3+W1/Q1: I got that this paper reveals a new privacy risk. But I am still curious why we need to mask the image directly. If we mask all faces to protect our privacy, it is not necessary for us to upload our image. Right? I can not find any suitable practical scenario here, although this is a new risk. It would be better if you could illustrate the scenario more practically.

W4/W5: Thanks for your additional experiments. It would be better if you could add the blurring experiment to the discussion.

Overall, I have increased my score to 2 considering the new experimental results. However, the practical threat should be illustrated to show the significance in the real world. If you can solve this, I would like to further increase my score. Thanks.

Reviewer_GCMWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the privacy risk of reconstructing a target photograph from a personalized diffusion checkpoint and a partially observed version of the image. The proposed method, PIAR, combines image inpainting with contrastive guidance based on the difference between the personalized and pretrained models noise predictions. Experiments on CelebA-HQ, DreamBooth, and CustomConcept101 show improved reconstruction over free-form extraction and inpainting baselines. The paper also examines checkpoint specificity, mask coverage, generation budgets, and held-out photographs of the same subject.

Critical strengths

The main strength is the study of personalized checkpoints together with partial observations of a target photograph. This is a simple and plausible setting that goes beyond checkpoint-only extraction. The paper also includes relevant controls: Base Inpainting isolates the contribution of personalization, while Personalized Inpainting isolates the additional benefit of contrastive guidance. The checkpoint-specificity experiment further shows that an unrelated personalized checkpoint does not provide the same reconstruction advantage. These comparisons make the central finding reasonably convincing.

Critical weaknesses

1. The interpretation of the reconstruction success rate needs further validation.

  • The matched inpainting baselines and multiple evaluation metrics support an overall reconstruction improvement. However, the reported success rate is defined by an SSCD similarity threshold, without directly validating whether successful outputs accurately reconstruct the missing regions. Since much of the target image is visible in the main setting, agreement with observed content may also contribute to the score.
  • The larger-mask results provide additional support, but a complementary masked-region evaluation would clarify how reliably the success criterion reflects missing-content recovery. This concern affects the interpretation of the reported success rates rather than the existence of the reconstruction improvement.

2. The evidence is limited to one victim-training configuration.

  • The three datasets cover different subjects, but all experiments use DreamBooth with Stable Diffusion v1.4 and disable prior preservation. The results establish a vulnerability in this configuration, but leave its sensitivity to personalization regularization unclear. A comparison with prior preservation enabled, reporting both reconstruction performance and personalization quality, would help clarify the scope of the findings.
Questions for authors
  1. Do the improvements over Base Inpainting and Personalized Inpainting remain when evaluation focuses on the masked regions? How well does the SSCD threshold of 0.8 distinguish accurate from inaccurate reconstruction of those regions?

  2. How do reconstruction performance and personalization quality change when prior preservation is enabled? This would help clarify whether the observed risk depends strongly on the chosen victim-training configuration.

  3. The mask ablation varies coverage and shape, but does not separately examine which facial regions remain visible. How does reconstruction change when masking the left/right half or the lower face, compared with the central masks used in the paper? Comparing masks with similar coverage would help clarify whether PIAR’s advantage depends on the amount of visible content or on the particular facial cues that remain available.

  4. On held-out CelebA-HQ photographs, PIAR performs slightly worse than Personalized Inpainting under partial and small masks (Table 9), despite improving performance under larger masks. What explains this difference? Does stronger contrastive guidance sometimes favor the learned subject appearance over the details of the target photograph?

7 comments
AuthorsOpenReview ↗

Response to Reviewer GCMW

Dear Reviewer GCMW,

Thank you for your thoughtful review and for recognizing the privacy setting and the role of our controlled comparisons. We address your concerns below.

W1 / Q1: Masked-region recovery and the SSCD success threshold We considered masked-region evaluation when designing our experiments, but chose whole-image metrics to provide a common evaluation basis across all five methods: the free-form baselines generate images without spatial correspondence to the target’s masked region. We agree that adding masked-region evaluation strengthens the evidence for concealed-content recovery. Following your suggestion, we have evaluated masked-region MSE, SSIM, and LPIPS across all three datasets using representative reconstructions generated with the same seeds and protocol as the first run in Table 3, without changing the candidate-selection procedure. PIAR outperforms both Base Inpainting and Personalized Inpainting on all three masked-region metrics across all three datasets, with larger gains than under whole-image evaluation. On DreamBooth, for example, PIAR reduces masked-region LPIPS from 0.4105 to 0.3200 and increases masked-region SSIM from 0.3534 to 0.4285 compared with Personalized Inpainting. These results confirm that the reconstruction improvements extend to the concealed content. Full results and evaluation details are provided in Appendix D.1 (Table 17) of the revised manuscript.

Regarding the SSCD threshold, our A-ESR follows prior extraction work, including FineXtract. We report FineXtract’s strict threshold of 0.7 and an additional, more stringent cutoff of 0.8, alongside continuous AS; PIAR’s advantage holds across these measures. A-ESR evaluates whole-image similarity, while the masked-region metrics above provide a complementary assessment of concealed-content recovery.

W2 / Q2: Prior preservation and personalization quality

Our original experiments follow FineXtract’s victim-training configuration, providing a common basis for comparing free-form extraction and masked-input reconstruction. We agree that prior preservation is a relevant component of the training recipe when assessing how broadly the findings apply. We have therefore additionally evaluated a reference recipe based on the original DreamBooth paper, including prior preservation, using the same attack and evaluation protocol as Table 3.

The privacy risk persists under this recipe. On DreamBooth, PIAR achieves AS 0.779, compared with 0.767 under the original configuration. Within the new configuration, personalization raises AS from 0.473 for Base Inpainting to 0.703 for Personalized Inpainting, and contrastive guidance further improves it to 0.779. The same AS ordering holds across all three datasets, with PIAR’s mean AS comparable to or higher than under the original recipe. Full results and training settings are provided in Appendix D.2 (Table 18) of the revised manuscript, alongside other personalization methods and architectures in Tables 19–22.

Q3: Which facial regions remain visible

Our experiments examine sensitivity to the available visual information by varying mask sizes and shapes across different target photographs. The observed gains therefore span a range of visible contexts. While we did not isolate specific facial regions at matched coverage, our evaluation addresses the related question of how reconstruction depends on the amount and pattern of visible content. We are now running an additional comparison of mask locations at matched coverage to further distinguish the role of particular facial cues, and will report the results once the evaluation is complete.

Q4: Guidance on held-out photographs

Across the held-out results for all three datasets and four mask tiers, PIAR improves AS over Personalized Inpainting in 9 of 12 settings (Tables 9–11). In the two CelebA-HQ cases highlighted by the reviewer, the decreases are only 0.004 and 0.010, indicating comparable performance. Under large and extra-large masks on CelebA-HQ, PIAR improves AS by 0.021 and 0.044, respectively. Taken together, these results support improved or comparable reconstruction, with the clearest benefit when less visual information remains. We therefore interpret the small decreases as limited exceptions to the overall benefit of guidance. Whether they reflect a preference for learned subject appearance over target-specific details cannot be determined from these scores alone.

AuthorsOpenReview ↗

Masked-region results supporting W1/Q1

For convenient reference, we reproduce Table 17 from Appendix D.1 of the revised manuscript below. PIAR outperforms both inpainting baselines on all three masked-region metrics across all three datasets.

The evaluation uses reconstructions generated with the same seeds, protocol, and candidate-selection procedure as Table 3’s first run. Metrics are averaged over the masked region. Results are mean ± sample standard deviation across subjects within a single run: 50 for CelebA-HQ and 30 for each object dataset. Values are not scaled by 100.

DatasetMethodMSE ↓SSIM ↑LPIPS ↓
CelebA-HQBase Inpainting2037.6 ± 842.60.4392 ± 0.06620.4111 ± 0.0607
Personalized Inpainting488.6 ± 229.20.6568 ± 0.05740.1636 ± 0.0315
PIAR398.2 ± 202.10.6777 ± 0.05350.1314 ± 0.0240
DreamBoothBase Inpainting5801.7 ± 4654.90.2268 ± 0.09380.6268 ± 0.0903
Personalized Inpainting3400.9 ± 3162.50.3534 ± 0.14490.4105 ± 0.1147
PIAR2414.4 ± 1152.50.4285 ± 0.17920.3200 ± 0.0886
CustomConcept101Base Inpainting3958.8 ± 1905.00.2737 ± 0.07700.5515 ± 0.0738
Personalized Inpainting2282.2 ± 1408.20.3777 ± 0.10330.3725 ± 0.1155
PIAR2164.2 ± 1420.90.4137 ± 0.09820.3151 ± 0.1103

Thank you for the detailed response and additional experiments.

The new results substantially address my main concerns, particularly regarding masked-region reconstruction and the robustness of the privacy risk under alternative training configurations.

Regarding W2/Q2, a comparison of personalization quality with and without prior preservation would further strengthen the findings by demonstrating whether substantial privacy leakage persists while maintaining comparable personalization capabilities.

For Q3, I look forward to the additional results comparing mask locations at matched coverage.

Reviewer_SUuEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates privacy leakage in personalized diffusion models and proposes PIAR, which combines image inpainting with contrastive guidance to recover hidden image regions using personalized model checkpoints. Experiments demonstrate good improvements over existing extraction and inpainting baselines. Additional evaluations across masking ratios, generation budgets, and held-out images further validate the method and highlight privacy risks beyond the original training images.

Critical strengths
  1. The paper identifies a meaningful and underexplored privacy risk arising from combining personalized diffusion model checkpoints with partially observed images. This extends existing research on training-data extraction and highlights practical vulnerabilities.

  2. The proposed PIAR method consistently outperforms existing extraction and inpainting baselines across datasets. Extensive experiments on masking ratios, checkpoint specificity, and held-out images further support the effectiveness and generality of the findings.

Critical weaknesses
  1. PIAR primarily combines existing inpainting techniques with contrastive guidance adapted from prior work. The technical contribution is relatively incremental, which weakens its novelty for ICLR.

  2. Experiments focus exclusively on DreamBooth personalization with Stable Diffusion v1.4. The lack of evaluation on more recent diffusion architectures and alternative personalization methods limits confidence in the broader applicability of the findings.

Questions for authors

Please see the above weakness and respond to them. Thank you!

1 comment
AuthorsOpenReview ↗

Response to Reviewer SUuE

Dear Reviewer SUuE,

Thank you for your thoughtful review and for recognizing the privacy risk identified by our work and the supporting experimental evidence. We address your two concerns below.

W1: Technical novelty

Our central novelty is the identification and systematic validation of an overlooked privacy risk. As motivated in the Introduction (lines 83–95), concealing sensitive content in a photograph may provide insufficient protection when a personalized checkpoint of the same subject is accessible. This risk is not addressed by prior work on free-form extraction. Our contributions include controlled experiments isolating personalization and guidance, checkpoint-specificity tests, and evaluations of held-out photographs and generation budgets. We also provide a modular PIAR framework requiring no additional attack training, whose applicability to a different base model and inpainting module is demonstrated by our FLUX–ControlNet experiments (Appendix D.2, Table 22). We explicitly acknowledge the origins of inpainting and cross-model guidance in Section 4 (lines 195–204 and 214–216) and do not claim those individual components as novel.

W2: Other personalization methods and recent architectures

We agree that evaluating other personalization methods and newer architectures strengthens the scope of the findings. We have extended the evaluation to DreamBooth-LoRA, Custom Diffusion, and DisenBooth, as well as FLUX.1-dev personalized with DreamBooth-LoRA. These experiments support our central conclusion beyond the original DreamBooth–SD1.4 configuration, although the magnitude of leakage and the benefit of guidance vary across settings.

On the DreamBooth dataset, LoRA personalization raises AS from 0.473 for Base Inpainting to 0.638 for Personalized Inpainting, and PIAR further improves it to 0.702. In our FLUX.1-dev evaluation on the DreamBooth dataset with ControlNet inpainting, the respective scores are 0.496, 0.646, and 0.678. These results extend the evidence to alternative personalization methods and a newer architecture with a different inpainting module.

Full results, together with the training and inference settings, are provided in Appendix D.2 (Tables 19–22) of the revised manuscript.

Comments on the paper

1 comment
Area_Chair_chEzOpenReview ↗

Dear Reviewers

Since the author-reviewer discussion ends soon, I encourage the reviewer to review the authors' rebuttals and let them know whether their responses are satisfactory and whether any concerns remain, if you haven't done so.

Thank you for your time and effort in reviewing for ICLR 2027.

Best wishes,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.