acceptodds
Withdrawn from ICLR 2027

When Diffusion Models Forget Who You Are: Identity Preservation in Face Inpainting under Large Occlusions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Face inpainting with diffusion models has recently achieved impressive visual quality, yet preserving identity fidelity under significant occlusion and conflicting text guidance remains a major challenge. To address this issue, we present Reference Semantic Inpainting for Face (ReSem-Face), a cascaded diffusion framework that introduces an explicit identity-conditioned semantic prior for multi-reference face inpainting. Our approach distills representative identity features from multiple references to reconstruct missing semantic regions, which then guide the diffusion process through a multi-stream conditioning architecture. This design provides strong semantic constraints when pixels are absent and stabilizes identity reconstruction while remaining compatible with prompt-driven edits. Experiments on CelebAHQ-IDI-5 and VGGFace2 demonstrate that ReSem-Face yields more reliable identity-preserving completion under severe semantic masks and improves text-controlled editing quality compared with representative baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
face inpainting · diffusion models · multi-reference · semantic prior · identity preservation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xkQWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper fills in large missing parts of a face using a few reference photos. It keeps the person’s identity and supports text-based edits. The method combines identity features, predicts the missing areas, and guides the image generation process. Experiments on two face datasets show better identity similarity and image quality than existing methods.

Critical strengths

Existing methods take identity and missing content together during generation. This can lead to identity loss when large areas are masked. This paper predicts the missing content first, then guides the generation process.

The paper shows consistent improvements over different methods with two datasets, and it shows better identity similarity and image quality.

Critical weaknesses

The two key ideas already exist in CAT-Diffusion and PVA. The paper does not compare against a simple combination of the two methods. It feels more like combining existing methods than introducing a new approach.

Table 3 shows a large gain from semantic prompts, even when they are not used during generation. But the main model is frozen during the second training stage. It is unclear how the prompts could improve identity similarity without affecting generation.

Questions for authors

See Critical Weaknesses 1 and 2

Reviewer_yJn6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper works on identity preserving face inpainting. An identity-aware semantic pre-inpainting module was designed to extract semantic prior, which was later integrated into a diffusion-based generative model. This specific module extracts consistent identity features from reference faces, and leverages identity prior to guide the generation of missing facial regions.The authors validate their approach on CelebAHQ-IDI-5 and VGGFace2 datasets. Experimental results demonstrate that the proposed method achieves impressive performance in identity preservation, visual fidelity and text controllability.

Critical strengths
  1. The paper proposed an identity-aware semantic pre-inpainting module that provides a high-level semantic prior for diffusion-based face inpainting.
  2. The proposed framework achieves impressive performance in identity preservation, visual fidelity and text controllability.
Critical weaknesses
  1. The technical innovation of the proposed framework is limited. The main idea of multi-reference identity semantic aggregation has been studied in PVA and identity-based semantic guidance for face inpainting have also been investigated in prior works.
  2. It seems that ReSem-Face does not preserve well the unmasked region. For example, the eye regions in both Fig 2 and Fig 3 changed after inpainting.
Questions for authors

Please explain why ReSem-Face does not preserve well the unmasked region.

Reviewer_DjM4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes ReSem-Face (Reference Semantic Inpainting for Face), a diffusion-based framework for multi-reference face inpainting that aims to reconstruct masked or occluded facial regions while preserving the identity of a target individual. The framework also supports text-guided face inpainting. The proposed method introduces an identity-aware semantic pre-inpainting module that aggregates identity and texture cues from multiple reference images. These cues are subsequently incorporated into the diffusion process through a semantic attention mechanism, alongside text-based guidance, to generate identity-consistent facial content.

Critical strengths
  • The addressed problem is practically relevant and has useful applications in photo editing.
  • The quantitative results demonstrate superior or competitive performance compared to existing methods.
Critical weaknesses
  1. Figure 1 could be more informative. The current illustration is overly abstract and does not provide sufficient architectural details to clearly understand the proposed framework.

  2. The method section is difficult to follow. For example, Section 3.2.2 provides only a high-level description of the proposed components, without adequately explaining their architectural design or implementation. These details are also missing from Figure 1, making it difficult to understand how the different components interact.

  3. Concerns regarding the experimental evaluation. There are several potential issues in the qualitative results, including possible test data leakage, as well as concerns regarding the hyperparameter analysis, as elaborated in the questions below.

  4. What does MSA represent in Equation (6)? The abbreviation should be explicitly defined.

Questions for authors
  1. Potential test data leakage in the qualitative results: How does the proposed method generate facial poses and expressions that closely match the ground truth? For example, in the first row of Figure 2, the generated facial expression is nearly identical to the ground truth. My understanding is that the guidance comes from reference images (which have different poses and expressions) and text prompts. Therefore, such a close resemblance raises concerns about potential test data leakage. Similar observations can be made in several other qualitative examples. Could the authors clarify how this level of similarity is achieved and rule out possible data leakage?

  2. Inconsistency in the hyperparameter analysis: In Figure 4(a), increasing ss appears to improve identity similarity (the circle has a higher value than the square). However, Figure 4(b) shows the opposite trend, where identity similarity decreases as ss increases. Could the authors explain this apparent inconsistency?

  3. Missing notation definitions and methodological clarity: Several mathematical notations are not clearly defined throughout the paper, making the method difficult to follow. Could the authors provide explicit definitions of the notation and more detailed explanations of the proposed components?

Comments on the paper

1 comment
Area_Chair_XHnLOpenReview ↗

Please read the response from authors and discuss

Dear reviewer fellows,

The author rebuttal has been posted. The rebuttal period is a critical part of our review process, allowing authors to clarify misunderstandings and address specific concerns.

Please take a moment to read the author response, engage in the discussion by replying to the authors' responses, and update your review if the rebuttal changes your assessment.

Your active participation is vital to ensuring a fair and thorough evaluation of the submissions. A constructive discussion helps us make the most informed decisions.

Thank you for your time and valuable contributions to the review process.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.