- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SHARE, a concept erasure method that addresses the limitations of equality-based preservation constraints in existing closed-form editing approaches, which restrict the available editing space. In Stage 1, SHARE learns a rectified selector to preserve erasure-relevant information while suppressing activations on retained concepts. In Stage 2, it computes layer-specific residual readouts in closed form. Extensive experiments demonstrate competitive erasure-preservation trade-offs across various concept erasure settings.
Critical strengths
- The paper analyzes the structural limitations of equality-based preservation constraints and proposes a rectified half-space formulation to alleviate these restrictions.
- The proposed method demonstrates competitive performance across multi-concept, implicit and instance/style erasure settings.
- The paper provides extensive ablation studies on loss components, retained-set coverage, and representation dimension, supporting the effectiveness of the proposed design.
Critical weaknesses
[W1] Unclear separation between training and evaluation data. It is unclear whether the retain set used during training is independent of the final evaluation data. In addition, the construction of the validation set used for hyperparameter selection and its separation from the final evaluation data are not sufficiently specified.
[W2] Limited hyperparameter sensitivity analysis across concept domains. Although the representation dimension and inference scaling factor vary across concept domains, the corresponding sensitivity analyses are primarily conducted in the celebrity erasure setting.
Questions for authors
-
Could the authors clarify whether the retain set used during training is independent of the final evaluation data? Additionally, how is the validation set for hyperparameter selection constructed, and is it separate from the final evaluation data? If there is any overlap, could the authors provide additional results using an independent evaluation set?
-
Does preservation using a predefined retain set generalize to unseen concepts and different generation capabilities? For example, when MS-COCO prompts are used as the general retain set, can compositional generation capabilities measured by GenEval[1] also be preserved, or does the retain set need to be tailored to specific capabilities?
-
Does preservation performance in generated images improve as the size of the retain set increases, and how does this affect erasure performance? Comparing SHARE with SPEED across different retain set sizes would help clarify the practical benefits of the proposed half-space relaxation.
-
While SHARE demonstrates effective multi-concept erasure in the celebrity setting, does it achieve similarly favorable erasure-preservation trade-offs when erasing multiple concepts in other domains, such as characters or artistic styles?
-
Why is a substantially larger inference scaling factor used for nudity erasure compared to other concept domains?
-
How robust is the final SHARE model against adaptive white-box attacks, such as UnlearnDiffAtk[2]? It would be valuable to evaluate whether the erasure effectiveness is maintained under such attacks.
-
Given the importance of generalization to modern DiT/flow-based architectures, could the authors extend the FLUX.1-dev evaluation beyond I2P to include adversarial benchmarks? Such results would provide a more comprehensive assessment of SHARE's effectiveness and robustness on recent architectures.
[1] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." NeurIPS, 2023.
[2] Zhang, Yimeng, et al. "To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now." ECCV, 2024.
7 comments
Response to Reviewer 2VK4 (2/3)
[2VK4-4] (Q3) Increasing retained-set coverage improves SHARE's image-level preservation.
Does preservation performance in generated images improve as the size of the retain set increases?
Comparing SHARE with SPEED across different retain set sizes would help clarify the practical benefits.
[TL;DR] Increasing general-retention coverage improves SHARE's retained accuracy from 92.61% to 95.46% and reduces FID from 11.97 to 3.35, while maintaining strong erasure. Under the evaluated SPEED configuration, the available null-space editing dimension collapses at coverage and above. The image-level comparison is included in Appendix D.5.4.
Coverage experiment.
We vary the fraction of the full Stage 1 MS-COCO retention pool over , using the same nested retained subsets for SHARE and SPEED at each available coverage level.
| Method / COCO fraction | CS | FID | |||
|---|---|---|---|---|---|
| SPEED / 0.00 | 5.40 | 92.64 | 93.61 | 26.28 | 17.92 |
| SPEED / 0.05 | 5.82 | 87.47 | 90.70 | 26.25 | 16.57 |
| SHARE / 0.00 | 0.81 | 92.61 | 95.79 | 26.55 | 11.97 |
| SHARE / 0.05 | 0.20 | 93.65 | 96.62 | 26.52 | 6.28 |
| SHARE / 0.20 | 0.61 | 94.46 | 96.86 | 26.57 | 4.45 |
| SHARE / 0.50 | 0.20 | 94.48 | 97.07 | 26.56 | 3.81 |
| SHARE / 1.00 | 0.81 | 95.46 | 97.28 | 26.56 | 3.35 |
SHARE: As coverage increases, retained accuracy improves by 2.85 points, FID decreases by 8.62, and erased accuracy remains between 0.20% and 0.81%.
SPEED: Coverage slightly improves FID but reduces retained accuracy. At coverage and above, the available editing subspace becomes zero-dimensional under the fixed official spectral threshold, even with IPF enabled. We report these configurations as unavailable because they cannot produce a nontrivial erasure update.
These results show SHARE's ability to exploit larger retained sets while maintaining erased-concept fitting capacity, consistent with the half-space feasible-set analysis.
The image-level sweep uses one controlled run per coverage fraction and the full Stage 1 MS-COCO pool. Figure 4 separately studies representation-level fitting and leakage with at most 2,000 retained prompts.
[2VK4-5] (Q4) SHARE supports joint multi-character erasure and composed character–style edits.
Does SHARE achieve similarly favorable trade-offs when erasing multiple concepts in other domains, such as characters or artistic styles?
We already evaluate simultaneous erasure of three characters (Snoopy, Mickey, and SpongeBob), as well as composition of this instance-erasure branch with separately trained artistic-style edits. Section 5.3, Figure 3, and Appendix E.3 (Table 13) report these evaluations using the SPEED few-concept protocol.
These experiments extend SHARE beyond celebrities to simultaneous character erasure and character–style composition. Jointly erasing multiple artistic styles within a single branch has not yet been evaluated; further multi-style experiments are in progress.
[2VK4-6] (Q5) Larger is needed to suppress implicit nudity concepts effectively.
Why is a substantially larger inference scaling factor used for nudity erasure compared with other concept domains?
The larger provides the stronger residual intervention needed for implicit nudity erasure. The sensitivity analysis in [2VK4-2] shows that leaves detection rates of 0.37 on I2P, 0.39 on MMA, and 0.58 on RAB, whereas reduces them to 0.03, 0.02, and 0.00, respectively.
We interpret this domain dependence through the erasure targets. Celebrity prompts explicitly name the target, and the replacement phrase "a person" is relatively close to the original representation. Nudity can be activated indirectly through visually diverse prompts, while the replacement "a person in modest clothing" involves a larger residual displacement.
The empirical sweep shows that – achieves most of the suppression benefit; further scaling yields limited improvements while increasing FID. We therefore select using validation data.
CPE (Lee+2025) similarly uses domain-dependent scaling, setting its corresponding parameter to 0.5 for celebrity erasure and 3.0 for explicit-content erasure.
Reference
- [Lee+2025] "Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate," ICLR 2025.
Response to Reviewer 2VK4 (3/3)
[2VK4-7] (Q6) Adversarial refinement and inference scaling improve robustness against UnlearnDiffAtk.
How robust is the final SHARE model against adaptive white-box attacks, such as UnlearnDiffAtk?
Adversarially refined SHARE achieves UnlearnDiffAtk ASR 0.29 at , with FID 6.47. CPE achieves a lower ASR of 0.21 but has substantially higher FID (20.08).
As detailed in [G-4], we evaluate UnlearnDiffAtk against the final edited checkpoints. Adversarial refinement reduces SHARE's ASR from 0.82 to 0.46 at the default . Increasing to 24 further reduces ASR to 0.29, with FID increasing from 4.82 to 6.47.
Without adversarial refinement, SHARE at also matches OCE's ASR of 0.73 while achieving much lower FID (5.92 vs. 46.42).
These results demonstrate that SHARE's robustness against adaptive attacks can be strengthened through adversarial refinement and inference scaling while maintaining substantially better general-image retention than the compared methods. We have added the full comparison to Appendix D.3.
[2VK4-8] (Q7) FLUX.1 evaluation to adversarial prompts is ongoing
Could the authors extend the FLUX.1-dev evaluation beyond I2P to include adversarial benchmarks?
The revised manuscript currently reports FLUX.1-dev performance on I2P, but not MMA-Diffusion or Ring-A-Bell. We are extending the evaluation to these adversarial benchmarks using the final FLUX.1-dev checkpoint. Results are not yet available; we will report them if completed during the discussion period.
Response to Reviewer 2VK4 (1/3)
We thank the reviewer for the detailed and practically important questions. We address the shared concerns regarding data separation and adaptive white-box robustness in [G-1] and [G-4] of our General Response. Below, we report additional evaluations of cross-distribution retention, retained-set coverage, and hyperparameter sensitivity. The cross-domain multi-concept and FLUX.1 adversarial evaluations remain in progress.
[2VK4-1] (W1, Q1) Final evaluation prompts are held out from training and validation.
Training, validation, and final evaluation prompts are disjoint. As detailed in [G-1], celebrity identities are shared across stages following the benchmark protocol, while prompt templates are disjoint. General MS-COCO training and evaluation captions are also disjoint.
These experiments evaluate generalization to unseen prompt formulations for the same identities by following the OCE protocol. We have documented the complete splits in the revised appendix and included machine-readable prompt lists in the updated supplementary material.
[2VK4-2] (W2) We have extended the sensitivity analysis regarding r and γ to the implicit concept erasure case.
[TL;DR] In implicit concept erasure, increasing representation dimension r or inference scaling γ improves erasure at the cost of greater image deviation. Performance largely saturates around r=16 and γ=8–12. We have added both sensitivity analyses to Appendix E.6.
Representation dimension (r).
We vary r without adversarial refinement.
Erasure improves from r=4 to r=16 and then largely saturates, apart from a further MMA reduction at r=64. FID increases steadily, while CLIP Score remains nearly unchanged.
Inference scaling factor (γ).
Using one adversarially refined checkpoint, we vary only γ.
Most erasure gains occur by γ=8–12, while further scaling increases FID with limited additional suppression. We select γ=12 using the validation criterion detailed in the revised Appendix E.6.
[2VK4-3] (Q2) SHARE preserves unseen compositional generation capabilities.
SHARE preserves compositional generation performance on GenEval, whose prompts are excluded from training, Stage 2 fitting, and hyperparameter selection. For N=100 celebrity erasure, SHARE scores 42.09%, close to the original model's 42.45%, and retains 96.88% of originally successful cases.
We also evaluate N∈{10,50,100} in Appendix E.2.1. Across these scales, SHARE's GenEval score differs from the original by −0.36 to +0.08 percentage points, with original-success retention of 96.88%–97.77%.
These results demonstrate preservation on unseen object-compositional prompts without tailoring the retained set to GenEval, extending the preservation evidence beyond the MS-COCO distribution.
Reference