acceptodds
Withdrawn from ICLR 2027

Concepts Persist as Directions: Visual-State Auditing of Unlearned Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Concept unlearning aims to suppress unsafe, private, or copyrighted concepts in text-to-image diffusion models. Suppression under direct prompting, however, does not establish whether the unlearned model's intermediate visual states can still support concept recovery when the target prompt is excluded from the optimization objective. To examine this question, we introduce Visual-State RECALL (VS-RECALL), an auditing framework that guides latent optimization with a direction defined by a pair of reference states. A positive reference and a matched negative reference define this direction between their U-Net mid-block states under null-text conditioning. VS-RECALL optimizes an adversarial latent by aligning its state displacement from the negative reference with that direction, and requires no access to the original pretrained model during optimization. The optimized latent is then decoded into an image and supplied to the model's image-to-image pathway with the original evaluation prompt. We evaluate VS-RECALL across seven tasks covering unsafe attributes, artistic styles, and object concepts, against ten concept-unlearning methods. VS-RECALL recovers target concepts against all ten unlearning methods and achieves the highest recall success rate, averaged over these methods, on six of the seven tasks, with the largest margins over the strongest respective baselines reaching 10.00% on Golf ball and 8.31% on I2P. These findings show that reference-defined directions in the unlearned model's visual states can guide concept recovery, motivating unlearning evaluations that assess both prompt-level suppression and internal recoverability. Code, prompt sets, and reference pairs are available at https://anonymous.4open.science/r/VS-RECALL.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
concept unlearning, concept erasure, text-to-image diffusion models, red-teaming, model auditing, internal representations, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JREtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper investigates the idea of recovering specific concepts against unlearned models. Based on previous attack papers, the paper investigates adversarial latent alignment and other techniques to improve performance. The paper contains various experiments on various concepts, showing the strength of the proposed method.
Critical strengths
  • The paper investigates an important question in the unlearning domain.
  • The paper utilizes both positive and negative reference pairs to optimize the latent.
  • The paper includes a nice illustration of the proposed framework.
Critical weaknesses

Please refer to the Questions below.

Questions for authors
  • As far as I understand, the method requires additional computation compared to RECALL for three forward passes, i.e., positive reference, negative reference, and adversarial latent. Can you compare the additional computational burden at each time step and the total convergence, including early stopping?
  • The paper mainly deals with latent alignment, but still utilizes various optimization techniques, such as mid-block activation, instance normalization, initialization with i.i.d. Gaussian, and so on. Can the authors perform an ablation study of each framework, in addition to the single-reference ablation in the paper?
  • The paper needs experimental results on recent models, such as SDXL or Flux.
  • The reviewer believes that the proposed framework is not only useful for RECALL. To broaden the impact of the proposed method, can the authors use their method for plug-and-play on other attack baselines?
Reviewer_eQi9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a whitebox attack against unlearned text-to-image (T2I) diffusion models.

The threat model assumes two conditions:

  1. full white box access to the victim T2I diffusion model weight
  2. the T2I diffusion models takes both text (prompt) and an image as the input

The proposed attack VS-RECALL focuses on optimizing an adversarial image, so that when it is inputed together with the prompt, the unlearned model can re-generate images of unlearned concept.

The key idea of optimizing such an adversarial image input is by minimizing its feature cosine distance towards a target image in the h-space of the victim model's UNet.

In experiment, the paper shows the proposed attack can recover unlearned models produced by different unlearning algorithms, on art style, nudity, and object concept categories.

Critical strengths
  • The proposed attack method is effective in finding a visual semantic direction in the h-space.
    • With just 1 pair of images, the optimized adversarial image encode the concept directions (e.g., clothed -> nude, no style -> style), this is interesting.
Critical weaknesses
  • The threat model is unrealistic.

    • Unlike other prompt-based attacks, the proposed attack instead optimizing through image input space, which reviewer thinks is not a realistic setup for text-to-image generative models. In natural and normal use cases, text-to-image generative models take textual prompt as input. When generating an image, they sample a random noise in the latent space then run prompt-conditioned denoising. To take image as input, T2I models need special adaptation that it needs an additional image encoder, which by default is not what T2I models run with.
  • The attack surface is expanded and unbounded, which makes solving the attack problems trivial. The provided comparisons is also not entirely fair.

    • The proposed attack essentially optimizes variables at a larger parameter space, then compares its performance with other prompt-based attacks, this is problematic.

      • Assuming the optimizee image input is in size of 256x256 (the paper itself did not include image size detail), the optimize variable is in the dimension of 256x256x3 (RGB), and the pixel value is not bounded by any small value ϵ\epsilon so it can resemble basically any noisy image -- this setup is nearly an unbounded search problem.
      • Whereas prompt or token-based attacks, UnlearnDiffAtk only search over few discrete tokens, and CCE optimizes only 1 token embedding. For SDv1.4, the prompt embedding dimension is 768, so the search space for CCE as an example, is x∈R768×1\mathbf{x}\in\mathbb{R}^{768\times1}, vs VS-RECALL's x∈R256×256×3\mathbf{x}\in\mathbb{R}^{256\times256\times3}, both with variable value in continuous range $x\in [0,1] $
      • If authors try to use an existing work [1] RECALL (ICLR 2026) as their justification for the image space attack setup, this paper also has the same issue that overlooked the search space dimension, but somehow went through the peer review.
    • As Table 1,2,3 show, some text-based attacks like UnlearnDiffAtk, are already providing high or even better attack success rates (ASRs), so even without an image attack vector (they used significant less parameters), this attack problem is not very hard.

  • The paper utilizes h-space for semantic exploration, this technique is also not novel, and similar application has been shown before. [2] introduced their findings that h-space encodes semantic information, and [3] shows h-space can provide visual editing in text-to-image diffusion model. Both [2] [3] are missing in the paper citation.


References:

[1] Image can bring your memory back: A novel multi-modal guided attack against image generation model unlearning, ICLR 2026

[2] Diffusion Models Already Have A Semantic Latent Space, ICLR 2023

[3] Black Hole-Driven Identity Absorbing in Diffusion Models, CVPR 2025

Questions for authors

See Critical Weaknesses.

Reviewer_q3eQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors try to show target concept recovery from unlearned models when the target prompt is excluded from the optimization objective. The authors propose VS-Recall, an auditing framework that optimizes an input latent by aligning its normalized U-Net mid-block displacement with a direction defined by positive and matched-negative reference images. Optimization uses empty-text conditioning, followed by prompt-conditioned image-to-image recovery. Across 10 unlearning methods it shows high recovery rate on all taks.

Critical strengths
  • The idea is simple and shows high recovery with various unlearning methods and tasks.
Critical weaknesses
  • The motivation is weak. The unlearning literature has already shown that current unlearning methods don't effectively unlearn concepts, and that the unlearned concept can be recovered by fine-tuning, adversarial prompts, null-text conditioning, textual inversion, latent inputs, etc. So why is this setup required or important needs justification.
  • This work is missing comparisons with other concept recovery works like The Illusion of Unlearning, TINA, IVO and M-EraseBench. These are relevant works which needs to be mentioned in the paper as well.
  • The work is missing experiments on important robust unlearning baselines like STEREO, Meta-Unlearning, Score Forgetting Distillation and AEGIS. These works try to address different robustness settings.
  • The paper doesn't report runtime cost comparison, latency and memory utilization, which is important to know whether it is feasible to run.
Questions for authors
  • Address the weakness mentioned.
  • Does VS-RECALL outperform other recovery methods under matched wall-clock and generation-attempt budgets?
  • What are the average and worst-case runtime, memory use? Are trajectories reused across prompts?
  • How does it perform against robust unlearning methods like STEREO, Meta-Unlearning, SFD and AEGIS.
  • Are the gains stable across seeds, reference pairs, and independent evaluators?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.