acceptodds
Withdrawn from ICLR 2027

Adaptive Inpainting Masking from Unconditional Concept Erasure Bias for Safe Inpainting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent diffusion models enable high-fidelity image generation, editing, and inpainting, but they also reopen a subtle attack surface: an adversary can specify a narrowly localized inpainting mask so that the target concept is underrepresented in the masked region, thereby bypassing Anti-editing Concept Erasure (ACE). This mismatch stems from a design gap—current erasure methods operate on the denoising pathway but do not verify whether the user-specified inpainting mask actually covers the concept to be removed. We address this gap by introducing Adaptive Inpainting Masking (AIM) from Unconditional Concept Erasure Bias, a training-free pipeline that turns the ACE-imposed unconditional bias into a pre-inpainting mask refinement signal. We first isolate the object into an OOD-like view, then run deterministic unconditional DDIM inversion through the ACE model, and measure reconstruction consistency. Regions whose visual evidence aligns with the ACE-style unconditional bias are interpreted as containing the erased concept and are automatically merged into the inpainting mask. The refined mask thus captures concept-bearing areas that an attacker attempted to exclude, enabling the downstream inpainting step to remain concept-safe without any explicit text labels. Our approach effectively extends diffusion-based concept erasure to the inpainting setting, closing a practical evasion channel while preserving the training-free nature of existing ACE frameworks.

Primary area:
generative models
Keywords:
generative model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qZQcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AIM to address attempts to bypass concept erasure through localized inpainting by automatically adjusting the mask before generation. The method uses the ACE model’s unconditional concept bias to detect prohibited concepts through object isolation, inversion-based reconstruction, and similarity assessment, then expands the mask to remove residual visual cues. The authors evaluate the framework’s protection capability across various types of content.

Critical strengths
  1. The method addresses a real vulnerability: unmasked visual context can bypass concept erasure during localized inpainting. The proposed mask adaptation effectively mitigates this vulnerability under the evaluated settings, providing more comprehensive protection.

  2. Repurposing the ACE model’s unconditional bias as a concept-detection signal through reconstruction behavior is an interesting idea that reduces reliance on explicit concept-name matching. The experiments also cover multiple concept types.

Critical weaknesses
  1. In my view, the main contribution is detecting target objects through reconstruction behavior. It remains unclear how the full framework differs from simple alternatives such as rejecting the request or regenerating the content after detection. The overall design is also an incremental extension of ACE, so its distinct technical contribution and practical advantages need stronger justification.

  2. The paper appears insufficiently polished, with several noticeable errors in the figures, algorithms, text, and references.

Questions for authors
  1. What are AIM’s key advantages over simple alternatives such as rejecting the request or directly regenerating the content after detection? Which experiments demonstrate the necessity of the full framework?
  2. Does the reconstruction difference between target and non-target concepts remain consistent across different image distributions and concept-erased models? How do the authors assess the effects of false positives and threshold selection on protection effectiveness and benign editing?
  3. Where is Figure 5?
  4. Is inversion performed using the original SD model followed by ACE reconstruction, or using ACE itself? The descriptions in the paper appear inconsistent.
  5. When Are Concepts Erased From Diffusion Models? (NeurIPS 2025) is closely related and already identifies the recovery of erased concepts through visual context in inpainting. Please review this work and clarify its relationship to AIM.
Reviewer_5pVjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AIM, an image protection strategy against inpainting-based image manipulation. AIM is designed as a pre-processing layer prior to ACE, a concept erasure method. The paper argues that ACE fails under inpainting tasks due to strong semantic provided by the unmasked portion of the image. To this end, the paper first isolates the main object through target-agnostic object isolation, then determines if that object contains a concept that should be erased with ACE. If so, it reshapes the mask given by the adversary. The experiment results show that

Critical strengths
  1. The paper addresses a practical and important scenario for defense against an adversary who attempts to bypass concept erasure via mask pinpointing.
Critical weaknesses
  1. Heuristic nature of methodology. The proposed method relies on a threshold τ\tau for similarity-based mask adaptation. This requires tuning for different concepts or images. The paper also lacks a rigorous algorithm for tuning this value, simply reporting the results for different values of τ\tau. The optimal value can thus differ for different prompts used for evaluation.
  2. Inconsistent effects of τ\tau. In some cases, using lower τ\tau does not necessarily lead to stronger protection. e.g., In Table 2, AIM90 shows better CLIP_p of 28.63 compared to 29.80 for AIM80. This is counterintuitive, since lower τ\tau would mean deciding to erase the concept more aggressively.
  3. Overclaim for protection against inpainting attacks. It seems like an overclaim to say that existing methods fail under inpainting attacks, as the proposed method underperforms or shows similar performance with those methods under erase concepts in Table 1.
  4. Missing analysis concept detection. The paper is missing analysis on how well the similarity-based mask adaptation indeed captures the concepts captured by ACE.
  5. Missing baselines. A more straightforward solution to this problem would be identifying if the given image contains the target concept. The paper compares with Grounding DINO, but it is known to be less reliable at capturing specific concepts. A stronger option would be using MLLMs such as GPT, Opus, Qwen-VL, or LLaVA (e.g., simply asking these MLLMs whether the concept exists in the image). In my opinion, assumption that we have access to the target concept is not unrealistic, since these concepts are already being used for training ACE.
  6. Missing related works. The paper is missing a significant related work that addresses robustness of concept erasure under mask inpainting [1].
  7. Incomplete manuscript. e.g., Fig. 5 is missing with a placeholder.
  8. Limitations and failure cases. Discussions on limitations and failure cases are missing.

[1] Lu et al., When Are Concepts Erased From Diffusion Models?, NeurIPS 2025

Questions for authors

N/A

Reviewer_5DBdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the bypass problem associated with concept erasure via diffusion models in local image restoration: an attacker can deliberately shrink the restoration mask to exclude most areas of the target concept, thereby rendering existing anti-edit concept erasure (ACE) methods ineffective. This paper proposes AIM, an adaptive masking method that requires neither training nor explicit text labels. At its core, the method utilises the ACE model’s own unconditional generative bias to determine which regions of the input image are likely to contain the concept being erased. Specifically, the method first transforms the object into a view that lies outside the approximate distribution, then performs deterministic unconditional DDIM inversion and reconstruction via the ACE model. Concept-relevant regions are extracted based on reconstruction consistency and merged into the user-provided mask.

Critical strengths
  1. The problem is clearly defined. By focusing on the mismatch between the user-specified mask and the intended concept to be erased, it highlights potential security vulnerabilities in localised restoration, which is of practical significance.

  2. No additional training is required. As a pre-restoration processing step, AIM does not require further updates to model parameters, making it easy to use.

  3. It does not rely on explicit text labels. By identifying suspected concept regions based on the model’s inversion and reconstruction behaviour, it reduces the reliance on users providing accurate conceptual descriptions.

Critical weaknesses

I have already explained all my doubts regarding the weaknessess above.

Questions for authors
  1. Could the authors provide experimental results for more complex scenarios (such as the multi-character scenarios and large-scale scenes with minor characters that I mentioned)?
  2. Could the authors further explain the necessity of calculating the character region masks in step (c) of Figure. 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.