acceptodds
Withdrawn from ICLR 2027

Spectral Saliency-Guided Concept Erasure via Ordered Gradient Projection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Concept erasure aims to suppress a target concept in a generative model while preserving unrelated behavior. Existing methods commonly restrict either the coordinates or the directions of an update, although these operations need not commute. We propose Spectral Saliency-Guided Erasure (SSGE), which estimates low-rank forget and retain gradient subspaces once at initialization (the retain subspace is a denoising-gradient proxy), derives a continuous coordinate-wise gate from their loadings, and projects the gated forget gradient onto the orthogonal complement of the sampled retain subspace, without restricting updates to a single forget-derived adapter subspace. For an orthonormal retain basis, this ordering makes the resulting forget component orthogonal to the sampled basis; reversing the operator order or subsequently applying adaptive preconditioning need not preserve this property. On Stable Diffusion 1.5, SSGE reduces Van Gogh target CLIP from 0.3160 to 0.2279 across three training seeds, compared with 0.2678 for a protocol-matched deterministic UCE baseline, but produces larger changes on retain prompts. Across five Imagenette objects, suppression transfers to held-out synonyms and compositional prompts, while all adjacent-class prompt sets also lose alignment. SSGE therefore provides a well-defined local gradient constraint and consistent target suppression, but does not guarantee broad functional preservation.

Primary area:
generative models
Keywords:
machine unlearning, concept erasure, text-to-image diffusion models, spectral saliency, gradient projection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xkwwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Spectral Saliency-Guided Erasure (SSGE), a concept erasure method that combines spectral saliency-based parameter gating with ordered gradient projection. The method estimates forget and retain gradient subspaces to identify concept-relevant parameters and projects the gated forget gradient away from the retain subspace. Experiments on Stable Diffusion 1.5 demonstrate effective suppression of artistic styles and object concepts, but also reveal noticeable degradation in non-target generation.

Critical strengths
  1. The proposed framework is reasonably motivated, and the analysis of operator ordering provides a clear mathematical explanation of the local orthogonality constraint.

  2. The experiments demonstrate consistent target suppression across multiple concepts, including held-out synonym and compositional prompts.

Critical weaknesses
  1. Limited evidence for the effectiveness of the core designs is the major problem. The ablation results do not demonstrate clear or consistent benefits from the proposed spectral saliency gate and ordered gradient projection. In Table 2, removing the gate yields nearly identical erasure performance with better preservation, while reversing the projection order also improves preservation metrics. These results raise concerns about the practical contributions of the proposed components.

  2. The baseline comparisons are insufficient. The experiments compare against only a small set of baselines, with limited coverage of recent concept erasure methods. In particular, comparisons with closely related gradient-based and preservation-aware approaches are missing, making it difficult to establish the competitiveness of SSGE.

  3. The paper is not well-written. The manuscript would benefit from more careful organization and proofreading. For example, the comparison table on page 8 lacks a table number and caption, affecting the clarity and professionalism of the presentation.

Questions for authors
  1. Could the authors further clarify the practical contributions of the spectral saliency gate and ordered gradient projection?

  2. Could the authors provide additional comparisons with recent and closely related concept erasure methods to better establish the competitiveness of SSGE?

Reviewer_vRx6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Spectral Saliency-Guided Erasure (SSGE), a method for concept erasure in T2I diffusion models that separate coordinate selection from directional protection by first estimating low-rank forget and retain gradient subspaces at initialization, then applying a continuous leverage-score gate to the forget gradient before projecting it onto the orthogonal complement of the retain subspace. Evaluated experiments on Stable Diffusion 1.5 across one artistic style and five object concepts with three training seeds, SSGE achieves stronger target suppression than a protocol-matched deterministic UCE baseline and transfers to held-out synonyms and compositional prompts.

Critical strengths
  1. This paper clearly separates two operations that are often conceptually confused, i.e., coordinate selection (via a diagonal leverage-score gate) and directional protection (via retain-subspace projection), and proves that their order matters. The qualitative and quantitative analysis make the insight reasonable and technically sound.
  2. Float64 SVD re-orthogonalization with a Cholesky correction, verification of basis orthonormality, and a computational-cost report improve the reproducibility.
  3. The acknowledged the negative outcomes, e.g., the incidental variations of adjacent concepts and the lack of proof advantages for each component. The distinction between local gradient orthogonality and actual functional preservation has also been clearly stated, avoiding unreasonable theoretical guarantees.
Critical weaknesses
  1. The proposed method combines existing techniques, such as gradient-based weight saliency [1], retain-subspace gradient projection [2], and concept-preserving gradient surgery [3]. Its main distinction lies in the leverage-score-based continuous gate and the gate-then-project ordering. However, the claimed orthogonality follows directly from standard projection properties, while Tab 2 shows no clear improvement over simpler variants. Therefore, the novelty and necessity of the proposed design need to be justified. [1] SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation, ICLR 2024. [2] Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection, WACV 2024. [3] Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient, AAAI 2025.
  2. Tab 8 shows that SSGE causes larger adjacent-concept degradation than ESD-X (0.0667 vs. 0.0477), despite introducing a retain-subspace constraint. The author should explain this limitation and provide a comparison of the matching erasing intensity to demonstrate a better erasure-preservation trade-off.
  3. Tab 5 shows that the randomly retained basis has a better preservation effect than the learned basis, raising concerns about the validity of the proposed spectral geometry. Furthermore, the retain basis is estimated based on the denoising gradient rather than the actual distillation gradient. The author should quantify the extent to which the subspace captured during the optimization process maintains the relevant direction.
  4. All experiments were conducted on Stable Diffusion 1.5, with only one style and five object concepts. There is a lack of comparisons with related methods, e.g., SalUn[1], and multiple baselines lack multiple training seeds. Experiments on other diffusion architectures and stronger baseline comparisons will enhance the empirical evidence.
Questions for authors

Please refer to the weakness

Reviewer_p2GBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Spectral Saliency-Guided Erasure, which estimates separate forget and retain subspaces from normalized gradient samples, converts their coordinate loadings into a continuous gate, and then projects the gated forget gradient away from the retain subspace. The authors evaluate the method on Stable Diffusion v1.5 using one artistic style and five object concepts.

Critical strengths

The manuscript proposed saliency-guided erasure through the estimation of separate forget and retain subspaces is an interesting direction, and applying gating before projection is a nice formulation.

Critical weaknesses

The paper is difficult to follow, and readability is a major issue throughout.

  1. The definition of “coordinates” is missing from both the abstract and the introduction.
  2. There is no proof or experimental validation of the claim made in lines 038–042.
  3. The paper lacks a clear motivation for estimating separate forget and retain subspaces. Why do the authors choose to formulate them as subspaces?
  4. The claim on line 046 is not supported by evidence or citations.
  5. The proposed approach is difficult to understand from the introduction. The authors should improve its explanation.

Experiments:

  1. The experimental evaluation is insufficiently thorough. Only one artistic style is considered for erasure, with no comparison against prior approaches.
  2. Comparisons with baselines are missing across the experiments.
  3. Furthermore, only Stable Diffusion v1.5 is used, which is somewhat outdated. The authors should evaluate the proposed approach on recent flow-based models.
  4. Evaluation on not-safe-for-work (NSFW) prompts and additional artistic styles is also recommended to demonstrate generalizability.

Overall, the paper lacks sufficient novelty, as many existing approaches already use orthonormal projectors for erasure and retention objectives. In addition, the experimental evaluation lacks breadth and baseline comparisons, and generalizability is not established because the method is evaluated on only one model. The paper also requires revision to address unsupported claims and improve the writing.

Questions for authors

please refer weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.