acceptodds
Withdrawn from ICLR 2027

A Concept Is More Than the Sum of Its Parts: Concept Erasure by Excising the Surplus

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Concept erasure aims to remove undesired concepts from text-to-image diffusion models without harming their ability to generate other concepts. Recent training-free methods tackle the problem by suppressing a direction that represents the target concept. However, this direction also carries parts the concept shares with similar ones (e.g., the wheels a garbage truck shares with cars), damaging these nearby concepts as well. Prior works mitigate this issue by protecting a retain set of related concepts, yet their performance is sensitive to which neighbors are chosen. Instead of searching for better neighbors outside the concept, we look inside the concept itself and let the parts of the concept decide what to keep. Since the same parts can form different wholes, a concept is more than the sum of its parts. We call this extra component the concept surplus: the portion of its features that its parts cannot explain. Thus, erasing only this surplus prevents the model from assembling the concept. Meanwhile, its parts remain intact, and so does every concept that shares them. Based on this motivation, we propose to **Ex**tract the **C**oncept **I**dentity as its **S**urplus over its parts, and **E**rase it. With the same principle, our method, named **ExCISE**, can erase a single part or a subset thereof, such as nudity, while preserving the rest of the person.

Primary area:
generative models
Keywords:
concept erasure, text-to-image diffusion, machine unlearning, activation steering, training-free, representation geometry, segmentation-guided unelearning orthogonal projection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9ko7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper deals with concept erasure which aims to remove target concepts from diffusion models without harming their ability to generate unrelated concepts. Prior works used external information / other concepts in order to define what should be retained. This paper defines the surplus of the concept which is the portion of the concept’s activation that the span of its parts cannot explain. Contrary to previous works, this surplus is retrieved from the concept itself. Thus, erasing only this surplus prevents the model from assembling the concept while different concepts sharing its parts are still available for generation.

Critical strengths
  1. The positioning of the paper in the context of the related work and its idea is well motivated and clear.
  2. The proposed erasure method is training-free.
Critical weaknesses
  1. The results are not compelling. Table 1 is not entirely clear. What are data req.? gates? what is strong/mid/weak? Robustness? Qualitative comparison values would be appreciated. In Table 2, ExCISE looks only marginally better compared to CASteer. Showcasing part erasure only on a single domain quantitatively is not enough. A qualitative example of another domain (cowboy hat) could be cherry-picked. A further qualitative superiority demonstration could convince effectiveness. Figure 3: on y axis, CASteer is only ~0.016 behind in CLIP score, which is marginal. On Identity Erasure task, again, only a single domain is demonstrated. What about erasure of other concepts such as garbage truck that was mentioned previously? Animals?
Questions for authors
  1. Please see Weakness #1. How would ExCISE perform on other domains compared to the other approaches?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 9ko7 (1/3)

We thank the reviewer 9ko7 for the careful review and for recognizing the training-free nature of our method, the clarity of our motivation and its positioning within the existing literature. Below, we address each point with new quantitative results.

[W1, Q1] Clarification on performance improvement and additional qualitative and quantitative experimental results.

Answer: We thank the reviewer for highlighting the need for additional experimental results and clearer comparisons with existing baseline methods. Accordingly, we extend our quantitative and qualitative evaluations across multiple concept categories and provide further clarification for several aspects of our experimental evaluation. We address each of the concerns raised under W1 in detail below.

[W1-1, Q1] Table 1 Improvements

Answer: We have clarified the table caption to highlight its purpose: summarizing the erasure–preservation trade-off across tasks, data requirements, and inference-time complexity. We removed the unclear erasure-performance column (strong/mid/weak); quantitative comparisons are reported in the experimental section (Section 5).

Data Requirements and Gates: Data requirements refer to manually selected or externally retrieved information that is needed to improve the erasure-retain trade-off through a constructed retain space. CASteer [G2] requires prompt pairs, NLCE [G3] an external retain set, and ExCISE only the target concept to improve utility preservation. Gates indicate the number of inference-time gating mechanisms a method uses for the same purpose of refining erasure-retain trade-off. Unlike CASteer, ExCISE applies cross-attention output steering without spatial clipping, using a uniform per-layer direction and strength across all spatial positions. ExCISE's reduced dependence on external data matters, as manually selected data (e.g., prompt pairs) reduce efficiency while retrieved sets are heavily dependent on retrieval success and concept neighborhoods as seen in the varying performance of NLCE [G3] on SDXL. Under the same method settings, NLCE showcases near-zero concept leakage on celebrity erasure (Table 3, Lines 397-408), while ranging from 5.9% on English springer to 18.9% leakage on garbage truck. Inference-time gates often improve preservation, but add computational overhead.

Robustness: Robustness is quantitatively evaluated in the experimental section (Sections 5.1 and 5.2) for nudity and celebrity erasure, with additional paraphrasing results provided in Appendix E.3. In Table 1, a checkmark is assigned to methods that do not rank among the two worst-performing methods on any metric within the respective task. ExCISE is the only method satisfying this criterion across all three robustness settings: nudity erasure under Ring-A-Bell [G6] and UnlearnDiffAtk [G5], celebrity erasure with pairwise prompts, and garbage truck erasure under paraphrased prompts. Robustness matters because an erased concept should not be recoverable through adversarial or paraphrased prompts. Under 58 paraphrases of garbage truck (580 images; Appendix E.3, Figure E.1), ExCISE leaks 9.1% (unedited: 88.1%), compared with 38.1% for NLCE and 70.5% for UCE; CASteer (1.7%) and ReCARE (7.1%) leak less but degrade the neighboring concepts far more (neighbor FID 134.0 and 142.1 vs. 65.8).

AuthorsOpenReview ↗

Response to Reviewer 9ko7 (2/3)

(continued)

[W1-2, Q1] Differentiation from CASteer

Answer: We agree with the reviewer that, in some standard benchmark settings, the performance differences between ExCISE and CASteer are marginal. In particular, we acknowledge the reviewer's concern regarding the nudity erasure results in Table 2, where a difference of a single detected image does not provide sufficient evidence of a meaningful improvement.

However, we emphasize that ExCISE's contribution extends beyond strong performance on standard erasure benchmarks. Our method specifically addresses nearby-concept preservation by protecting visual components. Our extended evaluations demonstrate that this mechanism leads to meaningful improvements over CASteer in nearby-concept preservation and adversarial robustness, while consistently maintaining strong target erasure across tasks. We provide quantitative evidence of these differences below:

  • Nudity Erasure and Robustness: While both methods achieve the strongest erasure on standard I2P [G14] prompts (Table 2, Lines 324-336), ExCISE shows greater erasure robustness under UnlearnDiffAtk (official default setup, 118 prompts), achieving an attack success rate of 10.2%, compared with 19.5% for CASteer and 39.0% for ReCARE [G4], showing that ExCISE leaks the erased nudity on roughly half as many attack prompts as CASteer (Section 5.1, Lines 352-372).
  • Celebrity Erasure: Under pairwise celebrity prompts (Table 3, Lines 397-408), ExCISE substantially improves non-target identity preservation over CASteer (54.55% vs. 15.05% Acc_s) while maintaining near-zero target recognition (0.50% vs. 0.05%). This results in a higher harmonic mean (70.34 vs. 25.61), showcasing that in contrast to CASteer, ExCISE maintains a strong erasure-preservation trade-off even under more complex prompts. NLCE preserves more on pairwise prompts; combining our direction with NLCE’s gate gives the best pairwise result (Acc_s 67.20, H 80.30).
  • Object Erasure: To further extend the comparison to object erasure, we evaluate retention on nearby concepts (Section 5.3, Lines 505-510). On SDXL [G11], ExCISE improves preservation across 11 neighboring concepts under garbage truck erasure, reducing FID from 134.0 to 65.8 and ΔCLIP (×100) from 5.54 to 1.18 compared with CASteer, while maintaining low target leakage (2.6% vs. 0.0%). The figure and corresponding quantitative table have been added to the main paper and Appendix, respectively (Figure 6; Tables E.1 and E.2).

[W1-3, Q1] Part Erasure: Clarification and extension.

Answer: To clarify, nudity erasure serves as our primary evaluation of part-level concept removal, where the goal is to suppress targeted visual components while preserving the remaining parent concept (the person and its benign characteristics). To further demonstrate the generality of this setting, we have extended our qualitative examples of cowboy hat erasure (Figure 4b, Lines 338-346) to address part erasure tasks beyond the common nudity evaluations. We additionally evaluated cowboy-hat erasure quantitatively with the same protocol as for objects (800 images per prompt set from 80 ImageNet templates × 10 seeds). Since "cowboy hat" is an ImageNet class, it is detected with the same classifier as our object leakage. Top-1 detection on cowboy hat drops from 73.4% to 2.3% on prompts naming the hat and from 66.0% to 1.3% on prompts naming only the parent ("cowboy"), while COCO [G8] preservation is maintained (caption CLIP unchanged at 0.78; FID 11.52). Since the compared methods do not report part erasure, we evaluate it against the unedited model.

[W1-4, Q1] Extended Qualitative Evaluation

Answer: We have extended the qualitative results to include additional generated examples across different concepts, prompts, and random seeds (Figures 4(b,c) and 5; Appendix Figures C.1–C.3).

These additions include per-method comparisons and aim to provide a broader view of the method's behavior across tasks, including variations in target erasure and nearby-concept preservation, while the quantitative evaluations provide a more systematic comparison.

[W1-5, Q1] Figure 3.

Answer: In Figure 3 (now Figure 7(a), Lines 486-495), CASteer was included as a reference point on a limited prompt subset rather than for comprehensive comparison. Measured from the unedited model, CASteer loses three times more bystander CLIP than ExCISE (0.023 vs. 0.007) at similar target suppression. Our extended evaluations provide stronger evidence of ExCISE's advantages in nearby-concept preservation and adversarial robustness.

AuthorsOpenReview ↗

Response to Reviewer 9ko7 (3/3)

(continued)

[W1-6, Q1] Celebrity Erasure: Per-Concept Results

Answer: We clarify that the celebrity results presented in the main paper (Table 3, Lines 397-408) are averaged across four celebrities: Taylor Swift, Andrew Garfield, Elon Musk, and Emma Watson. The preservation accuracy (Acc_s) measures how often the non-target celebrities of the full MACE [G7] retain set (100 celebrities) are still identified.

To improve transparency, we have further added complete per-celebrity comparisons for all baseline methods in Appendix E.1 (Table E.3) and specified the evaluated celebrities in the experimental section (Section 5.2, Lines 422-423) to facilitate comparisons.

[W1-7, Q1] Extended object erasure evaluation

Answer: To address concerns about the limited scope of object erasure experiments and follow the request to provide results on garbage truck and an animal example (English springer), we have extended our SDXL evaluation with additional quantitative and qualitative results on object erasure in Section 5.3 of the revised manuscript (Lines 477–510).

The extended evaluations provide a broader comparison across methods and concept categories with different characteristics across three new objects: garbage truck, golf ball, and English springer.

MethodGT T1↓GT COCO CLIP↑GT COCO FID↓Vehicle FID↓GB T1↓GB COCO CLIP↑GB COCO FID↓Ball FID↓ES T1↓ES COCO CLIP↑ES COCO FID↓Dog FID↓
Orig86.90.78––78.90.78––91.50.78––
UCE59.60.7815.8141.356.00.7526.6279.07.40.7718.3969.7
AdaVD27.00.780.000.071.10.780.000.01.00.780.000.0
NLCE18.90.789.9264.117.20.789.5278.25.90.798.5954.0
ESD6.80.7718.26117.210.50.7813.95156.92.90.7719.10108.7
ReCARE6.50.7527.63130.311.10.7617.64108.90.00.7619.69107.6
CASteer0.00.7711.8079.80.00.787.87129.90.00.788.2971.4
Ours2.60.7910.5355.51.00.799.6179.50.10.799.8477.2

GT: garbage truck, GB: golf ball, ES: English springer; T1: ImageNet top-1 leakage (%); superclass FID on Vehicle, Ball and Dog prompts. Best per column in bold; AdaVD leaves non-target prompts unchanged (FID 0.0). ESD is ESD-x-strict.

On SDXL, across all three object categories, ExCISE demonstrates a consistently strong erasure–preservation trade-off, without ranking among the two worst-performing methods on any metric. It achieves near-zero target leakage (0.1–2.6%) while maintaining the highest COCO CLIP preservation (0.79, tied on English springer) across all three concepts and competitive FID scores. These results highlight the robustness of ExCISE's erasure–preservation performance across visually distinct object categories. The full table is inserted into the manuscript as Table 4 (Lines 432-449). A garbage truck erasure evaluation on FLUX.1 dev and full comparisons on SD v1.4 can be found in the revised appendix (C.2 and E.2).

As discussed in point [W1-2, Q1] above, we further extended the garbage truck evaluation to include additional visual and semantic neighbors. Complete quantitative results and comparisons across object categories are provided in the newly added object erasure subsection (Section 5.3).

Overall, our extended evaluations demonstrate ExCISE's key strength: a consistently strong erasure–preservation trade-off across diverse concept categories without training or manually constructed retain sets. Its advantages are particularly evident in nearby-concept preservation and adversarial robustness, while its applicability extends beyond U-Net architectures to FLUX.1 dev [G13].

 

We hope that these additional results and clarifications adequately address the reviewer's concerns and kindly invite the reviewer to reconsider their assessment in light of the provided evidence. We are happy to provide further details if needed.

Reviewer_JkBKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes EXCISE, a training-free method for erasing concepts from text-to-image diffusion models. It generates reference images, identifies a target and its visual parts using attention saliency and segmentation masks, then extracts a “surplus” representation by projecting the target features away from a low-rank subspace formed by those parts. The resulting direction is used to steer generation. The same approach handles whole-concept erasure and part-level erasure, aiming to remove the target while retaining shared components and nearby concepts.

Experiments on nudity, identity, object, character, and style erasure support the conclusion that EXCISE can suppress targets while preserving related content in several tested settings. Ablations also support the use of a shared component subspace to improve neighboring-concept preservation. The results suggest that deriving the retain structure from the target’s own parts can be an effective alternative to external retain-concept lists.

Critical strengths
  1. EXCISE constructs the preservation subspace from the target’s own spatial components is an appealing alternative to manually selected or retrieved neighboring concepts. This could reduce dependence on vocabulary design and make the method easier to apply consistently across targets.

  2. EXCISE reports excellent suppression on nudity and celebrity erasure, including I2P, Ring-A-Bell, iterative adversarial attacks, SDXL, and FLUX.1-dev. The results suggest that the method is not merely effective for one narrow benchmark or one model architecture. The ablations provide some support for the proposed mechanism. The comparisons among parent-only directions, per-image surplus directions, shared retain spans, and different ranks are useful and sufficient.

Critical weaknesses
  1. The quantitative evaluation is incomplete and sometimes too narrow for the strength of the claims. Object and character erasure are supported mainly by qualitative figures, while quantitative neighboring-concept preservation is concentrated on one Taylor Swift experiment with eight related concepts. These gaps make it difficult to determine whether the reported improvements are robust and general rather than selected examples.

  2. Important reproducibility and experiments details are under-specified. The experimental protocol is underspecified at the per-task level. Although the paper states that EXCISE does not require an external retain set, it does not explicitly list, for each experiment, which representations are used to construct the retain subspace. The paper also omits the exact parent/component masks, component sets, reference prompts, edited layers, and mask construction details. Consequently, the claimed absence of an external retain set and the reproducibility of the reported results cannot be fully audited.

Questions for authors
  1. Can the authors evaluate erasure across several ordinary object categories, such as church, golf ball, garbage truck ,and so on ,as well as celebrity experiments? Please report target-recognition success, preservation of unrelated objects and generation quality.

  2. Can the authors provide two specific experiment tables to show the detailed experimental setup, including component sets, reference prompts, edited layers, and so on?

4 comments
AuthorsOpenReview ↗

Response to Reviewer JkBK (1/2)

We thank reviewer JkBK for the detailed review and for recognizing ExCISE's strength of reducing vocabulary dependence, as well as the broad applicability across tasks and models. The provided comments truly help us to further analyze and show strengths and weaknesses of our method and reported results. Below, we address the reviewer's concerns and include a significant number of new experiments and additional implementation details.

 

[W1, Q1] Additional quantitative evaluations.

[W1-1, Q1-1] Extension of quantitative evaluations on objects.

Answer: As suggested, we have extended our evaluation to three object categories (including an animal) with different characteristics: garbage truck, golf ball, and English springer. We evaluate ImageNet top-1 leakage using a pretrained ResNet-50 classifier [G15], as well as CLIP Score [G9] and Fréchet Inception Distance (FID) [G10] for the target concepts, using 800 images generated from ImageNet template prompts (80 templates × 10 seeds). Preservation is assessed on the first 3000 samples of COCO-30K [G8] and the respective superclass of each object. Our results demonstrate consistent erasure and retention performance across all evaluated concepts. The table below summarizes the main metrics; the complete results are in Table 4 of the revised manuscript (Lines 432-449).

MethodGT T1↓GT COCO CLIP↑GT COCO FID↓Vehicle FID↓GB T1↓GB COCO CLIP↑GB COCO FID↓Ball FID↓ES T1↓ES COCO CLIP↑ES COCO FID↓Dog FID↓
Orig86.90.78––78.90.78––91.50.78––
UCE59.60.7815.8141.356.00.7526.6279.07.40.7718.3969.7
AdaVD27.00.780.000.071.10.780.000.01.00.780.000.0
NLCE18.90.789.9264.117.20.789.5278.25.90.798.5954.0
ESD6.80.7718.26117.210.50.7813.95156.92.90.7719.10108.7
ReCARE6.50.7527.63130.311.10.7617.64108.90.00.7619.69107.6
CASteer0.00.7711.8079.80.00.787.87129.90.00.788.2971.4
Ours2.60.7910.5355.51.00.799.6179.50.10.799.8477.2

GT: garbage truck, GB: golf ball, ES: English springer; T1: ImageNet top-1 leakage (%); superclass FID on Vehicle, Ball and Dog prompts. Best per column in bold; AdaVD leaves non-target prompts unchanged (FID 0.0). ESD is ESD-x-strict.

On SDXL [G11], across all three object categories, ExCISE maintains a consistently strong erasure–preservation trade-off without ranking among the two worst-performing methods on any metric.

  • ExCISE achieves near-zero top-1 leakage of 2.6%, 1.0%, and 0.1% for garbage truck, golf ball, and English springer, respectively, with target CLIP Scores among the lowest (0.59, 0.56, and 0.44).
  • For preservation, ExCISE achieves the highest COCO CLIP Score of 0.79 across all three categories (tied on English springer), while maintaining competitive FID scores (10.53, 9.61, and 9.84).
  • Superclass preservation improves over CASteer [G2] for garbage truck (FID 55.5 vs. 79.8) and golf ball (79.5 vs. 129.9), and is comparable for English springer (77.2 vs. 71.4).

Garbage truck erasure on FLUX.1 dev and full comparisons on SD v1.4 can be found in the revised appendix (C.2 and E.2). For character erasure, Figure 5(b,c) compares Mickey Mouse erasure against all baselines, including the preservation of related characters (Pikachu and Snoopy).

[W1-2, Q1-1] Neighbor-concept preservation.

Answer:To further substantiate our claim of nearby-concept preservation, we have expanded the garbage truck experiments. In addition to the Vehicle superclass, we select four semantically related concepts that do not share visual components (garbage, trash, garbage bin and trash can) and six visually overlapping concepts (car, truck, pickup truck, delivery truck, ambulance and minivan), generating 400 images per concept using ImageNet templates. The results demonstrate that our method combines the strong erasure performance of output-side steering with improved preservation of nearby concepts compared to CASteer, a closely related steering-based approach. Figure 6 (Lines 451-467) shows the corresponding erasure–preservation trade-offs across baselines, with full numbers in Tables E.1 and E.2.

  • Across all 11 neighboring concepts, ExCISE substantially improves preservation over CASteer, reducing FID from 134.0 to 65.8 and ΔCLIP (×100) from 5.54 to 1.18. The gains are particularly strong for semantic neighbors (FID: 224.0 → 84.4; ΔCLIP: 9.39 → 2.36), while maintaining near-zero target leakage (2.6% vs. CASteer's 0.0%).
AuthorsOpenReview ↗

Response to Reviewer JkBK (2/2)

(continued)

[W1-2, Q1-2] Celebrity erasure evaluation.

Answer:The celebrity results in Table 3 (Lines 397-408) do not come from a single celebrity but are averaged across four celebrities. Acc_s measures how often the non-target celebrities of the full MACE [G7] retain set (100 held-out celebrities) are still identified.

MetricModelTaylor SwiftAndrew GarfieldElon MuskEmma WatsonMean
Acc_e ↓Original97.2054.0095.2094.0085.10
Ours0.001.600.000.000.40
Acc_s ↑Original86.3286.3286.3286.3286.32
Ours82.2082.6080.8480.7681.60

Per-celebrity tables for all methods have been added to the appendix (Appendix E.1, Table E.3) and exact celebrities are now specified in the manuscript (Section 5.2), and qualitative results across methods have been added in Figure 5(a) to complement the quantitative evaluation. Please note the retained-identity identification in Table 3 now reads 81.60% instead of 81.55%: the former averages the rounded per-celebrity values (Table E.3), the latter rounds the average of the unrounded ones, and we report the former for consistency with how the baseline averages are computed.

 

[W2, Q2] Implementation details.

[W2-1, Q2-1] Specification of reproducibility and experimental details.

Answer: We have added two implementation tables to Appendix D (Tables D.1 and D.2, Lines 933-940, 972-982), one per task and one per model, listing the fixed reference prompt, the number of reference images, the steering strength, the edited layers, the timestep used for direction extraction, and the representations forming the retain subspace.

[W2-2, Q2-2] Mask construction and component sets.

Answer: We have made the mask construction more explicit in Section 4.1 (Lines 212-233), with implementation details in Appendix D. Figure D.1 in Appendix D (Lines 1424-1457) shows the reference images of garbage truck and Taylor Swift with their respective parent and automatically selected component masks.

These masks are discarded once the pooled activations are obtained, as the retain subspace is constructed from the pooled representations rather than selecting individual visual mask components. The number of components varies across images and no fixed set exists, but we discard those that are near-duplicates of the parent and those that are too small to survive pooling, using fixed thresholds determined by the model's feature-grid resolution. We have added this detail to the method section (Section 4.1).

[W2-3, Q2-3] Reference prompts.

Answer: The reference prompt is fixed to "A high quality image of [concept]", with the number of generated images fixed to n = 20 (30 for nudity). For nudity erasure, we extend the prompt to "A high quality full body image of a nude person, [part-concept] exposed", while for part erasure, we use "A high quality image of [parent-concept] with [part-concept]" to ensure the generation of both the parent concept and the targeted part. Steering strengths are set to 1.5 for all parent erasure cases (objects, celebrities, characters) and 2.5 for part erasure (nudity and cowboy hat) on SD-1.4 and SDXL, following our analysis in Appendices A and B.

[W2-4, Q2-4] Edited Layers.

Answer: As specified in our method section (Section 4.3, Lines 294-298), we apply steering to the outputs of all cross-attention layers of the U-Net with one direction per layer used across timesteps. Additional implementation details for FLUX.1 dev [G13] are provided in Appendix C.1.

 

We hope the additional quantitative results and implementation details address the reviewer's concerns. We thank the reviewer again for the constructive feedback and are happy to provide further details if needed.

Reviewer_JkBKOpenReview ↗

Response to the Rebuttal

Thank you for your detailed responses and the effort devoted to the additional experiments and clarifications. I will also carefully consider the other reviewers’ comments to identify any aspects I may have overlooked and support a balanced assessment. Thank you again for your time and effort.

AuthorsOpenReview ↗

Thank you for your response and positive feedback.

We thank the reviewer for acknowledging our detailed rebuttal and efforts and for maintaining their positive assessment. We are happy to provide further clarification on any questions that may have arisen from other reviewers' comments and our response to them.

Once again, we would like to express our sincere gratitude for going through our paper and rebuttal and providing constructive feedback, despite busy schedules. We deeply appreciate your commitment and contributions.

Reviewer_w929OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes EXCISE, a training-free concept-erasure method that derives reusable steering directions from a target's visual components rather than external neighboring concepts. It pools masked cross-attention outputs, constructs a shared low-rank component subspace in whitened coordinates, and removes the target's projection onto that subspace. The resulting residual steers generation. Parent erasure retains the target's parts, while part erasure retains non-target siblings. Experiments evaluate nudity and identity erasure, with additional object, character, and style examples across SD v1.4, SDXL, and FLUX.

Critical strengths
  1. A coherent framework for parent and part erasure. The same construction supports both tasks without external neighboring-concept lists. Shared component statistics improve the reported preservation trade-off over parent-only and averaged per-image directions (Figure 3).

  2. Strong reported suppression on important benchmarks. EXCISE reduces I2P detector-positive outputs from 323 to 12 and reports UnlearnDiffAtk ASR of 10.2%, versus 19.5% for CASteer (Section 5.1). On SDXL single-celebrity prompts, target identification is 0.40% while retained-identity identification remains 81.55% (Table 3).

Critical weaknesses
  1. Key methodological steps are insufficiently specified. Although Eqs. (2)-(4) define representation pooling and surplus construction, the mask-generation procedure in Section 4.1 remains largely qualitative. Section 4.3 ("Steering Generation") does not explain how the extracted direction vℓ,0v_{\ell,0} is applied during generation. A complete mathematical or algorithmic specification of these steps is needed.

  2. Component specification and its supervision requirements are unclear. The paper does not explain whether a target's components are selected manually or automatically, or how the corresponding segmentation and selection criteria are obtained. Consequently, it is unclear whether avoiding external neighboring-concept lists merely shifts the selection burden to target components. The claimed advantage in reducing dependence on hand-picked retain sets is therefore insufficiently substantiated.

Questions for authors
  1. Could you clarify the method with a complete mathematical or algorithmic specification, particularly mask construction and the application of �0��0� during generation?

  2. How are each target concept's components specified and selected, and which steps require manual supervision? How does this reduce the selection burden compared with neighboring-concept retain sets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer w929 (1/2)

We thank the reviewer w929 for the detailed review, and for praising the coherence of our framework across parent and part erasure and recognizing our strong reported suppression. Below, we respond to the reviewer's concerns, which included clarification of our methodology, its steps and the component selection process.

[W1, Q1] Elaboration of the methodological steps.

[W1-1, Q1-2] Mask Generation and Steering application.

Answer: We provide a more detailed explanation of the mask generation steps, as well as specifications for the steering application. To facilitate a better understanding of the purpose of those steps, we shortly describe the full pipeline and embed the detailed descriptions within their respective position in the pipeline. Steps 2 and 3 are the complete mask construction, and Step 7 adapts CASteer's [G2] cross-attention output steering, with one fixed direction per layer and no gate.

The pipeline is fully automatic and includes no labeling. The only text input is a generic fixed prompt naming the concept (e.g., garbage truck). Neither SAM2 nor any later step receives a label or a text prompt: the parent and its components are found from the model's own attention and from SAM2's box and point prompts. Masks are used only once, to pool activations; generation uses one fixed direction per layer and no masks.

  • Step 1: Reference generation. We generate n=20n=20 reference images {I(i)}i=1n\lbrace I^{(i)}\rbrace_{i=1}^{n} (30 for nudity) from a fixed prompt naming the target and cache the cross-attention outputs and the concept-token attention maps in the same forward pass.

  • Step 2: Parent mask. The cached concept-token attention maps, averaged over heads, layers and steps, give a saliency map a(i)\boldsymbol{a}^{(i)}. For each of several fixed thresholds τ\tau (quantiles and Otsu, Table D.2), the bounding box of the region a(i)≥τ\boldsymbol{a}^{(i)}\ge\tau is passed to SAM2 [G1] as a box prompt; SAM2 receives no text or class label. The parent mask is the largest valid returned mask, M0(i)=arg⁡max⁡M∈M(i)∣M∣,M(i)={SAM2(I(i),box(a(i)≥τ)):τ∈T}∩V,M_0^{(i)}=\arg\max_{M\in\mathcal{M}^{(i)}}|M|,\qquad \mathcal{M}^{(i)}=\bigl\lbrace\mathrm{SAM2}\bigl(I^{(i)},\mathrm{box}(\boldsymbol{a}^{(i)}\ge\tau)\bigr):\tau\in\mathcal{T}\bigr\rbrace\cap\mathcal{V}, where T\mathcal{T} holds the thresholds and V\mathcal{V} the valid masks (a single connected object covering at most 95% of the image and at least half of its prompt region and of the top-2% saliency pixels; Algorithm 1). If SAM2 returns no valid mask, the thresholded attention region itself is used.

  • Step 3: Component masks (no text prompt, no labels). Within the parent's bounding box, SAM2's automatic mask generator, prompted only by a regular grid of points, proposes unlabeled segments, which are clipped to the parent mask. Nested segments form a containment tree, and its leaves are the components {Mj(i)}j=1Ji\lbrace M_j^{(i)}\rbrace_{j=1}^{J_i} (at most 20 per image, the largest are kept). Segments that nearly duplicate the parent (IoU >0.3>0.3) or fall below a fixed size floor are dropped. Images with fewer than four components are skipped. No component is named, chosen or labeled: the number of components JiJ_i varies per image, and Step 5 reduces them automatically to a rank-8 retain basis. This step removes the need for manual component selection.

  • Step 4: Pooling (Eq. 2). Cross-attention outputs are mean-pooled inside each mask relative to the image-wide mean, at the first sampling step (every step on FLUX.1 dev [G13]): the parent mask gives the target, each component mask a retain representation. The masks are then discarded.

  • Step 5: Retain subspace (Eq. 3). After layer-wise ZCA whitening, the mask-derived retain representations of all images are stacked, and their leading k=8k=8 left singular vectors form the retain basis Bℓ,0\boldsymbol{B}_{\ell,0}. This step works as a refinement of the full span built from all automatically collected components.

  • Step 6: Surplus direction (Eq. 4). The whitened mean target representation is projected off Bℓ,0\boldsymbol{B}_{\ell,0} and mapped back to activation space, giving one unit direction v^ℓ,0\hat{\boldsymbol{v}}_{\ell,0} per layer.

AuthorsOpenReview ↗

Response to Reviewer w929 (2/2)

(continued)

  • Step 7: Steering during generation (Eq. 5). For any prompt, at every edited cross-attention layer ℓ\ell, step tt and position ss of the conditional branch of classifier-free guidance, the cross-attention output hℓ,s,t\boldsymbol{h}_{\ell,s,t} is replaced, before it enters the residual stream, by hℓ,s,t′=hℓ,s,t−α ⟨hℓ,s,t,v^ℓ,0⟩ v^ℓ,0,\boldsymbol{h}'_{\ell,s,t}=\boldsymbol{h}_{\ell,s,t}-\alpha\thinspace\langle\boldsymbol{h}_{\ell,s,t},\hat{\boldsymbol{v}}_{\ell,0}\rangle\thinspace\hat{\boldsymbol{v}}_{\ell,0}, with a fixed α\alpha (1.5 for parent and 2.5 for part erasure on SD-1.4 and SDXL; FLUX.1 dev values in Table D.2), so α>1\alpha>1 pushes activations away from the target. Positions with little component along v^ℓ,0\hat{\boldsymbol{v}}_{\ell,0} are barely changed, and because the direction is orthogonal to the whitened retain subspace, the coordinates within the retain span stay unchanged. Classifier-free guidance then combines the steered conditional prediction ϵt′\boldsymbol{\epsilon}'_t with the unsteered unconditional one, ϵ~t=ϵt∅+w(ϵt′−ϵt∅)\tilde{\boldsymbol{\epsilon}}_t=\boldsymbol{\epsilon}^{\varnothing}_t+w(\boldsymbol{\epsilon}'_t-\boldsymbol{\epsilon}^{\varnothing}_t), with the guidance scale ww unchanged.

Part erasure (sibling cut, Eq. 1). Only the target changes: the reference prompt names parent and part, the part mask is obtained from the attention of the part tokens with SAM2, again without any label, and its pooled representation is contrasted against the remaining unlabeled components. Whitening, the retain subspace and steering are unchanged.

For completeness, we have added the previously missing mathematical formulations of the steering operation to the revised manuscript (Section 4.3, Eq. 5, Lines 294-298) and further specified the mask construction in Section 4.1 (Lines 212-243).

[W1-2, Q1-1] Full algorithmic specifications.

Answer: We thank the reviewer for pointing out the missing explicit algorithmic steps. We summarize our pipeline and its steps above, revise the respective details in the method section (Section 4) and add the full algorithm to Appendix D (Algorithm 1, Lines 1026-1064).

 

[W2, Q2] Component specification and manual supervision requirement clarification.

Answer: As suggested, we further elaborate on component selection and manual supervision. Our entire pipeline is automated, requiring no manual specification or selection of components. The part names shown in Figure 1 are for illustration only; the components are unlabeled SAM2 segments, and we have removed the names from the method overview (Figure 2) to avoid confusion.

  • First, the model generates reference images of the target concept (Section 4.1, Lines 207-210).
  • Cross-attention localizes the target region within each generated image, while SAM2 refines the target mask and identifies its visual components (Section 4.1, Lines 212-233). All components above a size floor calculated from the model's pooling resolution are retained, and their corresponding activations are obtained through mask pooling (Section 4.1, Eq. 2, Lines 232-243).
  • These component representations are then stacked across all reference images, and a low-rank basis is extracted, effectively reducing the broad component collection to its dominant, high-energy directions (Section 4.2, Eq. 3, Lines 257-266).
  • This basis is used as the retain span for surplus extraction (Section 4.2, Eq. 4, Lines 268-269). Thus, no manual component selection is required to construct the retain basis. For Taylor Swift, this basis leaves a residual of 0.385–0.805 of the parent, versus 0.996–0.998 for random rank-8 subspaces (Appendix A.3).

Moreover, as we use one fixed prompt for all reference images (“a high quality image of [concept]”), no additional prompt preparation is required. We use only 20 reference image generations (30 for nudity), reducing not only manual selection efforts but also the number of U-Net forward passes required for direction extraction compared to CASteer's default of 100 images per manually constructed prompt pair.

In summary, our framework does not require any manual selection of retain components, fully removing the retain space selection burden. Additionally, our framework is designed to share fixed parameters within concept groups (e.g. Objects) with no need for individual parameter selection. We have made these benefits explicit in the method and experiment sections (Sections 4.1, 4.2 and 5) of the revised manuscript.

The retained-identity identification in Table 3 now reads 81.60% instead of 81.55%: the former averages the rounded per-celebrity values (Table E.3), the latter rounds the average of the unrounded ones, and we report the former for consistency with how the baseline averages are computed.

 

We hope these clarifications address the reviewer's concerns about the specification of our method, and we are happy to clarify any remaining points.

AuthorsOpenReview ↗

Friendly Reminder

Dear Reviewer w929,

Thank you again for your careful review of our method. As the discussion period ends on November 18 (AoE), we would like to hear whether our responses have addressed your concerns. If any issue remains or any new questions have come up, please let us know, and we will respond promptly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Rebuttal

The authors thank all reviewers (w929, JkBK, 9ko7) for taking the time to review the manuscript thoroughly and provide valuable comments that significantly contribute to improving the quality of this paper. This global response contains a summary of strengths highlighted by the reviewers, revisions of the manuscript to address the reviewers' concerns, and the references for the literature cited in our individual responses. The individual responses address each reviewer's weaknesses and questions.

 

Strengths of our work, as highlighted by reviews:

  1. [9ko7, JkBK] Positioning within related works and contrast to other methods (novelty and clarity of construction method).
  2. [9ko7, JkBK] Training-free characteristic of ExCISE is beneficial.
  3. [w929, JkBK] ExCISE can be applied to various unlearning tasks.
  4. [w929, JkBK] ExCISE shows strong performance on important nudity and identity removal benchmarks
  5. [w929, JkBK] Omitting manual neighbor selection or vocabulary-based neighbor retrieval is a strong benefit in practice.
  6. [w929, JkBK] Component ablations show the improved erasure–preservation trade-off of ExCISE.
  7. [w929] ExCISE combines parent and part erasure within one framework.
  8. [JkBK] Cross-model application of ExCISE shows its broad applicability.
  9. [JkBK] Component-constructed retain subspace can reduce vocabulary dependence.

 

Revisions of the manuscript

  1. Page 4, Lines 162–175: We removed the unclear strong/mid/weak column from Table 1 and defined data requirements, gates and the robustness criterion in its caption. [ 9ko7 ]
  2. Pages 4–5, Lines 212–233: We specified the fully automatic construction of parent and component masks. [ w929, JkBK ]
  3. Page 5, Lines 216–229: We added the subspace construction to the overview figure (Figure 2). [ w929 ]
  4. Page 5, Lines 264–266: We clarified how the shared retain subspace removes manual component selection. [ w929 ]
  5. Page 6, Lines 294–298: We added the steering equation (Eq. 5) and how the direction is applied during generation. [ w929 ]
  6. Pages 6–7, Lines 317–322: We specified the implementation details of all experiments. [ JkBK ]
  7. Page 7, Lines 338–346: We extended the qualitative part-erasure examples to the cowboy hat (Figure 4). [ 9ko7 ]
  8. Page 8, Lines 378–395: We added qualitative comparisons across baselines for pairwise celebrity erasure, Mickey Mouse erasure and nearby-concept preservation (Figure 5). [ JkBK, 9ko7 ]
  9. Page 8, Lines 413–418: We clarified the scope of part erasure and added quantitative cowboy-hat results. [ 9ko7 ]
  10. Page 8, Lines 422–423: We named the four celebrities evaluated in Table 3. [ 9ko7, JkBK ]
  11. Pages 9–10, Lines 432–467, 477–484, 498–510: We added an object-erasure section with three ImageNet classes (Table 4) and a nearby-concept preservation analysis (Figure 6). [ JkBK, 9ko7 ]
  12. Page 10, Lines 486–495, 524–527: We added a surplus-injection ablation showing that the direction encodes the target concept (Figure 7b). [ w929 ]

Appendix

  1. Pages 17–18, Lines 879–906, 918–930: We added object-erasure results on FLUX.1 dev and a cross-model comparison (Tables C.2 and C.3). [ JkBK, 9ko7 ]
  2. Pages 17–18, Lines 913–917, 955–961: We added implementation details for the direction construction. [ JkBK, w929 ]
  3. Page 18, Lines 964–970: We specified the construction of the pairwise celebrity prompts. [ 9ko7, JkBK ]
  4. Pages 19, 972–991: We added the requested setup tables per task and per model (Tables D.1 and D.2). [ JkBK ]
  5. Page 20, Lines 1027–1064: We added the complete procedure as Algorithm 1. [ w929, JkBK ]
  6. *Page 19, Lines 1002–1020: We added SDXL breakdowns, including per-celebrity results, and additional results on SD v1.4. [ 9ko7, JkBK ]
  7. Page 19, Lines 1022–1025; Page 28, Lines 1458–1473: We added a paraphrase-robustness evaluation (Figure E.1, Table E.6). [ 9ko7 ]
  8. Pages 20–21, Lines 1067–1079, 1128–1133: We added the details and quantitative results of the concept-injection ablation (Appendix F). [ w929 ]
  9. Page 26, Lines 1370–1403: We added visualizations of the parent and component masks (Figure D.1). [ JkBK, w929 ]
AuthorsOpenReview ↗

Global References

[G1] SAM 2: Segment Anything in Images and Videos, Ravi et al. 2025
[G2] CASteer: Cross-Attention Steering for Controllable Concept Erasure, Gaintseva et al. 2026
[G3] Neighbor-aware localized concept erasure in text-to-image diffusion models, Shi et al. 2026
[G4] Co-occurring Associated REtained concepts in Diffusion Unlearning, Kim et al. 2026
[G5] To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now, Zhang et al. 2024
[G6] Ring-a-bell! How reliable are concept removal methods for diffusion models?, Tsai et al. 2024
[G7] MACE: Mass Concept Erasure in Diffusion Models, Lu et al. 2024
[G8] Microsoft COCO: Common Objects in Context, Lin et al. 2014
[G9] CLIPScore: A reference-free evaluation metric for image captioning, Hessel et al. 2021
[G10] GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, Heusel et al. 2017
[G11] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, Podell et al. 2024
[G12] High-Resolution Image Synthesis with Latent Diffusion Models, Rombach et al. 2022
[G13] FLUX.1 dev, Black Forest Labs 2024
[G14] Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models, Schramowski et al. 2023
[G15] Deep Residual Learning for Image Recognition, He et al. 2016

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.