acceptodds
Withdrawn from ICLR 2027

Probing Concept Erasure in Diffusion Models via Initial Latent Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Concept erasure aims to suppress designated content in text-to-image diffusion models while preserving generation of other content. An illusion of forgetting occurs when suppression under ordinary prompts coexists with recoverable concept generation under adversarial inputs. Across the evaluated models, larger distributional discrepancy between erased and vanilla models' noise predictions during denoising is associated with lower target-concept detection rates. Motivated by this observation, we propose Initial Latent Variable Optimization (IVO) to assess concept recoverability through white-box attacks. IVO optimizes initial latents while keeping the erased model's parameters and original prompt fixed. Target-concept reference images provide the default initialization through DDIM inversion. A distribution matching loss and a direction calibration loss guide optimization using surrogate noise predictions. Experiments cover eleven concept erasure methods and three scenarios involving nudity, objects, and artistic styles. Average attack success rates reached 79.6%, 87.7%, and 91.8% for nudity, parachute, and Van Gogh, respectively. Both average attack success rates and CLIP scores exceeded those of the evaluated prompt-based attacks. A blinded human evaluation of prompt alignment also favored IVO over P4D and UDiff across these scenarios. The findings motivate including latent optimization in concept erasure evaluations and assessing recovery together with alignment to the original prompt.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Models, Concept Erasure, Adversarial Attacks, Initial Latent Optimization, Semantic Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BLb3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether initial latent optimization can recover target content from concept-erased diffusion models. In its default configuration, IVO first applies DDIM inversion to a reference image containing the target concept. It then optimizes the initial latent using a distribution matching loss (DML) and a direction calibration loss (DCL), while keeping the erased model's parameters and the original prompt fixed. Successful optimized latents can also be stored and reused. The experiments cover eleven concept-erasure methods and tasks involving nudity, objects, and artistic styles. They report both target recovery and prompt alignment, with additional ablations and human evaluation. The paper reports improvements over the evaluated prompt-based attacks in its main experiments.

Critical strengths
  1. The paper evaluates recovery while keeping the original prompt fixed and considers both target-concept presence and semantic alignment with that prompt. This goes beyond detector-defined success alone and better addresses whether the generated image satisfies the requested content.
  2. The experiments cover multiple erasure methods and concept types, with loss and initialization ablations and human evaluation. The reported results provide empirical evidence that the optimization procedure can recover target content under the tested conditions.
Critical weaknesses

1. The additional contribution and benefits of optimization are not sufficiently established

W01: Missing comparisons with existing latent-inversion recovery procedures. Although the paper cites Memories of Forgotten Concepts and M-ErasureBench, it does not experimentally compare against the latent-inversion recovery procedures used in these works or clearly explain their omission. Consequently, IVO's advantages over existing inversion-based recovery methods are not sufficiently established.

W02: Missing ablation without optimization after inversion-based initialization. Although the paper compares initialization strategies and loss combinations, it does not report direct generation from the inverted initial latent without subsequent optimization, under the same erased model, reference image, and prompt. The ablations therefore do not sufficiently quantify the additional benefit of optimization over inversion-based initialization alone.

W03: Omission of TINA/TINA+ and insufficient clarification of the additional contribution and validity of inversion-based probing. The paper neither cites nor compares against the closely related TINA and TINA+, and does not clearly explain what additional contributions IVO offers over these optimization-based inversion methods. Moreover, the randomly initialized UNet negative control in TINA+ shows that unconstrained inversion can produce recognizable target reconstructions even when the model has not learned the target knowledge. Successful recovery alone is therefore insufficient to establish that the corresponding knowledge remains in the model. Since IVO also uses reference images and latent optimization for recovery, appropriate negative controls or other effective validation are needed to rule out a similar issue and distinguish functional recovery from evidence of residual knowledge.

Together, these concerns limit the assessment of the method's additional contribution and the interpretation of its results.

2. The motivation and key procedural details are insufficiently explained

W04: Figure 1 provides limited support for the method's motivation. Figure 1 mainly illustrates an association between changes in noise predictions after erasure and target-concept detection rates. Since concept erasure is already expected to alter the model's noise-prediction behavior, this observation provides limited additional insight. Furthermore, the paper does not clearly explain how this association supports the design of initial latent optimization, weakening its role in motivating the method.

W05: Parts of the workflow in Figure 2 are unclear. Some components of Figure 2 are not clearly connected to the method description. The “Search” step in panel (a) is not specified, and the illustration of multiple reference images leading to “Init Latent” can suggest that several images are jointly inverted into one latent, whereas the text describes inversion of a single reference image. Panel (c) corresponds to the latent-reuse mechanism described in the paper, but pool construction, sampling rules, and their use in the experiments are insufficiently specified. This affects both understanding and reproducibility.

Reference:

[1] TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Models. CVPR 2026.

[2] TINA+: Probing Residual Visual Knowledge in Unlearned Diffusion Models via Diffusion-Consistent Text-Free Inversion. ArXiv 2026.

Questions for authors
  1. What are IVO's main additional contributions relative to the inversion-based recovery procedures used in Memories of Forgotten Concepts and M-ErasureBench, and to TINA/TINA+? Can recovery, prompt alignment, and computational cost be compared under suitable shared settings, or can the reasons such comparisons are inapplicable be explained?
  2. What are the results when generation uses the inverted initial latent without DML/DCL optimization, with the erased model, reference image, and prompt held fixed? How much improvement does the subsequent optimization provide?
  3. In light of the spurious reconstruction issue identified by TINA+, what negative controls or other evidence support the residual-knowledge interpretation? If a randomly initialized UNet is used, how would the control account for the external information supplied by IVO's default pretrained surrogate?
  4. How is “Search” in Figure 2(a) implemented, and how are reference images selected? How is the latent pool in panel (c) constructed, sampled, and used in the main experiments? Which results use fresh optimization and which use latent reuse?

Reference:

[1] TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Models. CVPR 2026.

[2] TINA+: Probing Residual Visual Knowledge in Unlearned Diffusion Models via Diffusion-Consistent Text-Free Inversion. ArXiv 2026.

Reviewer_1fAPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Initial Latent Variable Optimization (IVO), a method for evaluating the recoverability of erased concepts in diffusion models by optimizing the initial latent while keeping the erased model's parameters and the original text prompt fixed. The method is motivated by an observed association between discrepancies in the noise-prediction distributions of erased and vanilla models and target-concept detection rates. IVO initializes the latent through DDIM inversion of a reference image containing the target concept and optimizes it using a Distribution Matching Loss (DML) and a Direction Calibration Loss (DCL), based on conditional and unconditional noise predictions from a surrogate model, respectively. Experiments cover 11 concept erasure methods, with the main evaluations focusing on nudity, parachute, and Van Gogh style erasure. IVO achieves higher average attack success rates (ASR) and CLIP scores than the evaluated baseline attacks across these three scenarios.

Critical strengths

The paper clearly formulates the problem of concept recoverability in erased diffusion models. In particular, it distinguishes concept suppression under standard prompts from recovery under adversarial inputs. The white-box evaluation setting is also clearly specified, with the erased model's parameters and the original text prompt kept fixed.

The experimental evaluation covers a range of concept erasure methods and provides empirical support for the proposed optimization. The main experiments report higher average ASR and CLIP scores than the evaluated baselines. Table 4 shows that combining DML and DCL achieves higher ASR than using either loss alone, while Figure 4(b) shows improvements in ASR over optimization iterations.

Critical weaknesses
  1. The incremental contribution of IVO over closely related latent-based recovery methods is not sufficiently established. Memories of Forgotten Concepts (CVPR 2025) demonstrates that erased concepts can be recovered through diffusion inversion. M-ErasureBench (WACV 2026) also evaluates inverted-latent recovery, including surrogate-model inversion followed by resampling with the erased model. RECALL (ICLR 2026) performs reference-guided image-prompt latent optimization using noise-prediction matching while preserving the original text prompt. However, the main experimental comparisons focus on Sneaky, MMA, Ring, P4D, and UDiff, making it difficult to assess IVO's added value over closely related latent-based recovery methods. A comparison on more diverse concepts and related methods would help establish IVO's advantages.

  2. The generalizability of IVO's recovery performance across target concepts remains unclear. The main evaluation focuses on nudity, parachute, and Van Gogh, while only two additional object concepts are evaluated in the appendix. Notably, IVO improves the mean ASR over UDiff by 38.7 percentage points on parachute, but only by 1.6 and 3.3 percentage points on garbage truck and tench, respectively. Moreover, despite being evaluated on the same six erasure methods, IVO underperforms UDiff on several methods for garbage truck, including AdvU (32% vs. 100%) and SalUn (44% vs. 74%). These results raise questions about how representative the main evaluation is of IVO's performance across different concepts. A broader evaluation covering systematically selected target concepts, along with clarification of the concept selection criteria, would help establish whether the reported performance gains generalize beyond the main scenarios.

Questions for authors

Q1. Novelty and comparison with closely related recovery methods: Could the authors further clarify the methodological differences between IVO and existing latent-based recovery approaches, particularly Memories of Forgotten Concepts and RECALL? A quantitative comparison under matched evaluation conditions, where applicable, would help establish IVO's advantages over these methods.

Q2. Generalizability across target concepts: What criteria were used to select the target concepts for the main evaluation? Although additional results are reported for garbage truck and tench, IVO shows substantially smaller mean ASR improvements over UDiff on these concepts than on parachute. Could the authors provide results on a broader, systematically selected set of target concepts/styles under a consistent evaluation protocol? This would help assess whether the main results are representative of IVO's performance across different target concepts.

Reviewer_eFNJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the problem of concept erasure in text-to-image diffusion models. Specifically, motivated by the observation of higher difference between the vanilla and erased DMs resulting in lower detection rate, this paper proposes IVO (Initial Latent Variable Optimization). IVO utilizes a white-box models as the attacker, which can access the the erased model’s gradients and controls its initial latent while keeping the model parameters and original prompt fixed. A DML for reducing the distribution difference and a DCL for semantic alignment are the objective of the proposed method. Experiments on extensive datasets show the superiority of IVO.

Critical strengths
  1. The problem studied in this work is critical and well-defined. And the motivation of this work is clear.

  2. The figures presented is well-designed and clear.

  3. Extensive experiments are conduct in this work.

Critical weaknesses
  1. The paper writing needs to be refined. Although this paper's motivation is clear, the presentation of the proposed method is implicit. For example, in line 67, the introduction of IVO looks like a description of each component.

  2. This paper highlights the problems in unlearning. However, it lacks the discussion about existing unlearning work in related work sections.

  3. In line 43, the meaning of the sentence " Initial latents offer another input through which to test the robustness of concept erasure." is unclear. And there is not the citations. The definition of "initial latents" need to be clarified.

  4. Its unclear that the datasets are used in Figure 1.

  5. In line 71, what is the "DDIM inversion"? This term is used at the first time, but the paper doesn't define it.

  6. Citation format in this paper is weird.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.