- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RASteer, a training-free method for concept erasure in text-to-image diffusion models that aims to preserve non-target content. RASteer combines Retain-Orthogonal Steering (ROS), which partially removes components overlapping with retain concepts, and Overlap-Adaptive Calibration (OAC),which adjusts the degree of removal based on the overlap between the erasure direction and the retain subspace. Experiments across different concept domains and Stable Diffusion backbones show favorable performance compared with existing methods.
Critical strengths
- RASteer is training-free, making it easier to apply than training-based concept erasure methods.
- The paper clearly motivates the need for overlap-adaptive calibration and demonstrates its effectiveness.
- The method achieves favorable performance across different domains, including unsafe content, instances, and styles, as well as multiple Stable Diffusion backbones.
Critical weaknesses
- Limited evaluation on recent architectures. The experiments are limited to Stable Diffusion models (SD-v1.4, SD-v1.5, SD-v2.1, and SDXL). Evaluation on newer DiT/flow-based architectures such as FLUX would help demonstrate the generalizability of RASteer.
- Lack of adversarial robustness evaluation. The paper does not evaluate robustness against adversarial attacks. As adversarial robustness is an important aspect of concept erasure, evaluating RASteer using commonly adopted benchmarks, such as UnlearnDiffAtk [1] (white-box), Ring-A-Bell [2], and MMA-Diffusion [3] (black-box), would strengthen the evaluation.
- Limited analysis of retain set selection. The criteria for selecting retain concepts and the sensitivity to these choices are not sufficiently discussed. Ablations on the size and semantic composition of the retain set would help clarify their impact on erasure and preservation.
[1] Zhang, Yimeng, et al. "To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now." ECCV, 2024.
[2] Tsai, Yu-Lin, et al. "Ring-a-bell! how reliable are concept removal methods for diffusion models?." ICLR, 2024.
[3] Yang, Yijun, et al. "Mma-diffusion: Multimodal attack on diffusion models." CVPR, 2024.
Questions for authors
- Does preservation using a predefined retain set generalize to unseen non-target concepts that are not part of the set? Additionally, does concept-level preservation generalize to the preservation of broader compositional generation capabilities, such as those assessed by benchmarks like GenEval[4]?
- Could the authors clarify how sequential erasure works in RASteer? When a new target concept is added, are the existing steering directions recomputed, or are they kept fixed while a new direction is added? How does this differ from one-shot multi-concept erasure?
- How does the erasure-preservation trade-off change as the number of erased concepts increases? Is there a point where performance starts to degrade significantly?
- Can RASteer erase concepts from different semantic domains simultaneously? For example, how does it perform when erasing nudity and a character instance together?
[4] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." NeurIPS, 2023.