acceptodds
Withdrawn from ICLR 2027

RASteer: Retain-Aware Activation Steering for Concept Erasure in Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Concept erasure aims to remove a target concept, such as a copyrighted style, a recognizable character, or unsafe content, from a pretrained text-to-image diffusion model while preserving its ability to generate other content. Existing activation steering methods build an erasure direction mainly from the target concept and adjust model activations along it at inference time. However, target and retained concepts often overlap in the model's representation space, so this direction also contains shared components that retained concepts rely on. Steering directly along this direction can therefore suppress retained concepts and harm the generation of non-target content. To address this issue, we propose Retain-aware Activation Steering (RASteer), a training-free method. RASteer first builds a retain subspace from the concepts to preserve. Retain-Orthogonal Steering (ROS) then removes components aligned with this subspace from the erasure direction, making steering more specific to the target. Since fully removing the shared components can weaken erasure, we further introduce Overlap-Adaptive Calibration (OAC). At each layer and denoising step, OAC uses the overlap between the erasure direction and the retain subspace to control how much of each shared component is removed, balancing target erasure and concept preservation. Experiments on unsafe-content, instance, and artistic-style erasure across multiple backbones and benchmarks show that RASteer matches or outperforms the activation steering and weight editing baselines we evaluate, achieving a better balance between erasure and preservation. The source code will be released.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Diffusion Models, Concept Erasure

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uFXsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RASteer, a training-free method for concept erasure in text-to-image diffusion models that aims to preserve non-target content. RASteer combines Retain-Orthogonal Steering (ROS), which partially removes components overlapping with retain concepts, and Overlap-Adaptive Calibration (OAC),which adjusts the degree of removal based on the overlap between the erasure direction and the retain subspace. Experiments across different concept domains and Stable Diffusion backbones show favorable performance compared with existing methods.

Critical strengths
  • RASteer is training-free, making it easier to apply than training-based concept erasure methods.
  • The paper clearly motivates the need for overlap-adaptive calibration and demonstrates its effectiveness.
  • The method achieves favorable performance across different domains, including unsafe content, instances, and styles, as well as multiple Stable Diffusion backbones.
Critical weaknesses
  • Limited evaluation on recent architectures. The experiments are limited to Stable Diffusion models (SD-v1.4, SD-v1.5, SD-v2.1, and SDXL). Evaluation on newer DiT/flow-based architectures such as FLUX would help demonstrate the generalizability of RASteer.
  • Lack of adversarial robustness evaluation. The paper does not evaluate robustness against adversarial attacks. As adversarial robustness is an important aspect of concept erasure, evaluating RASteer using commonly adopted benchmarks, such as UnlearnDiffAtk [1] (white-box), Ring-A-Bell [2], and MMA-Diffusion [3] (black-box), would strengthen the evaluation.
  • Limited analysis of retain set selection. The criteria for selecting retain concepts and the sensitivity to these choices are not sufficiently discussed. Ablations on the size and semantic composition of the retain set would help clarify their impact on erasure and preservation.

[1] Zhang, Yimeng, et al. "To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now." ECCV, 2024.

[2] Tsai, Yu-Lin, et al. "Ring-a-bell! how reliable are concept removal methods for diffusion models?." ICLR, 2024.

[3] Yang, Yijun, et al. "Mma-diffusion: Multimodal attack on diffusion models." CVPR, 2024.

Questions for authors
  • Does preservation using a predefined retain set generalize to unseen non-target concepts that are not part of the set? Additionally, does concept-level preservation generalize to the preservation of broader compositional generation capabilities, such as those assessed by benchmarks like GenEval[4]?
  • Could the authors clarify how sequential erasure works in RASteer? When a new target concept is added, are the existing steering directions recomputed, or are they kept fixed while a new direction is added? How does this differ from one-shot multi-concept erasure?
  • How does the erasure-preservation trade-off change as the number of erased concepts increases? Is there a point where performance starts to degrade significantly?
  • Can RASteer erase concepts from different semantic domains simultaneously? For example, how does it perform when erasing nudity and a character instance together?

[4] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." NeurIPS, 2023.

Reviewer_W45QOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new inference-based method for concept suppression in diffusion models. The proposed method, ROS, builds a retain subspace and removes the components aligned with this subspace from the erasure direction. Then, the authors propose a method to adaptively balance target erasure and concept preservation, referred to as OAC.

Critical strengths
  1. The paper proposes a new inference-based erasure method that improves the preservation of non-target concepts. Although the proposed algorithm's contribution is just incremental, it tackles the important goal of an improved preservation
  2. The adaptive calibration method improves the projection and thus erasure performance
Critical weaknesses
  1. The method relies on the presence of a retain set, differently from all other existing similar methods, and this also negatively impacts its computational complexity
  2. The paper is limited to U-Net-based T2I models. However, these models are now considered outdated and should focus on demonstrating its effectiveness for DiT-based models.
Questions for authors

Main Questions

  • In the paper, in page 3, it is written that "The part of a retained concept’s activation that carries the concept lies, to good approximation, in S;". However, there is no theoretical or empirical proof for this claim. Can you provide theoretical and empirical proofs?
  • In the paper, in page 4, it is written that "Removing the shared term in full is not always desirable: when the target itself has a component along a retain direction, suppressing that direction also weakens the erasure." Although OAC improves the problem of a weaker erasure, there should still be a trade-off. However, to my surprise, the proposed method outperforms all other methods both in terms of erasure and preservation. Can the authors explain how can this happen? As it seems counterintuitive.
  • The related work is insufficient. Can the authors justify why many inference-based unlearning methods are missing from the references? The comparison appears incomplete at the current stage. Few examples: TraSCE [1], SEOT [2], SPELL [3]. In addition, many recent fine-tuning methods are missing. The authors should significantly revise the related work section and comparisons.

Clarifications

  • Can you explain why do you consider your method to perform concept erasure instead of concept suppression? Since you are not modifying the model parameters.
  • CURE is from Neurips 2025, not 2026

[1] Jain, Anubhav, et al. "Trasce: Trajectory steering for concept erasure." arXiv preprint arXiv:2412.07658 (2024).
[2] Li, Senmao, et al. "Get what you want, not what you don't: Image content suppression for text-to-image diffusion models." International Conference on Learning Representations. Vol. 2024. 2024.
[3] Kirchhof, Michael, et al. "Shielded diffusion: Generating novel and diverse images using sparse repellency." ICML 2025

Reviewer_zJYAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RASteer, a training-free method for concept erasure in diffusion models. The main idea is that target and retain concepts may share representations, so removing the target can unintentionally affect other concepts. At the same time, protecting these shared representations too much can weaken erasure. To address this trade-off, the authors introduce ROS to reduce the overlap with retain representations and OAC to adaptively adjust the projection strength. The experiments show promising results across several Stable Diffusion backbones, particularly compared to existing activation steering methods

Critical strengths

Clear motivation and intuitive methodology

I think the paper presents a clear and convincing motivation. The authors point out how removing a target concept can unintentionally affect other concepts due to overlapping representations, while protecting these shared representations too much can weaken the erasure itself (as illustrated in Figure 1). I find this trade-off interesting and think it is an important issue to address. The proposed method seems like a reasonable and intuitive way to handle both problems. I particularly appreciate that the method is simple and directly addresses the issues identified in the paper

Critical weaknesses

Limited experimental evidence of generalizability

My main concern is that the experimental evaluation is too limited to support the generalizability of the proposed method. While I find the motivation and methodology reasonable, I would have liked to see more systematic experiments across diverse concepts and settings

First, the experiments cover only four instances and four artistic styles, with a few predefined retain concepts. Although the authors evaluate multiple Stable Diffusion backbones, the same small set of concepts is used throughout. This makes it difficult to judge whether the improvements generalize to other concepts and retain-set configurations. I also find the analysis of retain-set construction insufficient, especially since the proposed method directly depends on the retain subspace.

Second, I have similar concerns about the component ablation. Table 4 evaluates ROS and OAC using only Snoopy and Van Gogh. The results are also somewhat mixed across metrics. I would like to see whether both components consistently improve performance across more concepts and experimental settings. Similarly, Figure 1 provides a convincing motivation, but it is unclear how representative these examples are

Finally, I believe the evaluation could be strengthened by following existing benchmarks, such as HUB [1], UnlearnCanvas [2], and EraseBench [3]. These benchmarks offer broader concept coverage and more comprehensive evaluations of erasure and its effects on non-target concepts

  • [1] Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearning
  • [2] UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models
  • [3] Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Concepts
Questions for authors

Q1. Selection of target and retain concepts

I noticed that the experiments use a relatively small and predefined set of concepts. Could the authors explain how these target and retain concepts were selected? In particular, were they chosen based on their semantic similarity or representation overlap? I think this would help me better understand how representative the current experimental settings are

Q2. Consistency of the component ablation

In Table 4, removing OAC improves the target classification accuracy and retained FID for Van Gogh, although other metrics favor the full method. How do the authors interpret these mixed results? Given that the ablation covers only two target concepts, I would also like to understand how confident we can be that the contributions of ROS and OAC are consistent across different settings

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.