acceptodds
Under review as a conference paper at ICLR 2027

Continual Concept Erasure in Diffusion Models by Suppressing Cross-Edit Interference

Abstract

Concept erasure removes copyright-protected, privacy-sensitive, or otherwise undesirable concepts from pretrained text-to-image diffusion models to support content governance and compliance. As erasure requests arrive over time, models must remove new targets without undoing prior erasures. Existing methods do not constrain interference across edits: residual perturbations outside the retain set interact and accumulate, degrading unrelated generations and sometimes collapsing previously erased targets into noise. We propose CEASE (Continual Erasure via Adaptive Subspace Editing), a training-free method that imposes two subspace constraints on a closed-form solver. CEASE adds the token representation of the shared replacement to the solver's invariance matrix and, when interference is detected, projects the current update onto the orthogonal complement of dominant output directions extracted from cumulative past updates. A closed-form decomposition attributes the accumulated interference to repeated activation of the shared replacement and overlap between successive update directions, showing that the two constraints suppress these respective sources. Across continual erasure of celebrities, artistic styles, and instances, CEASE achieves the most consistent erase–preserve trade-off, while existing methods either degrade general generation or insufficiently erase targets. The source code will be released.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Diffusion Model, Continual Erasure

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3hvkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual concept erasure, where new erasure requests arrive over time and the model must remove new targets without undoing prior erasures or degrading unrelated generation. The authors identify a cross-edit interference accumulation problem in existing closed-form editing methods: residual perturbations outside the retain set interact and accumulate across sequential updates, degrading general generation quality and sometimes collapsing previously erased targets into noise. They propose CEASE, a training-free method that imposes two subspace constraints on a closed-form solver: AIC adds the shared replacement token representation to the invariance matrix, and HOC projects the current update onto the orthogonal complement of dominant historical output directions when interference is detected. A theoretical decomposition attributes interference to repeated activation of the shared replacement and overlap between successive update directions. Experiments on continual erasure of celebrities, artistic styles, and instances show that CEASE achieves a more consistent erase–preserve trade-off.

Critical strengths
  1. The problem is clearly defined, the continual erasure setting is practically relevant, and the authors explicitly point out the limitation of existing methods that "protect only a finite retain set."
  2. The theoretical analysis is solid: based on the rank-one update structure, it derives a noise decomposition into anchor, off-anchor, and cross terms, and designs AIC and HOC accordingly, with tight motivation–design correspondence.
  3. The method is training-free and closed-form, with low computational overhead and easy integration into existing editing frameworks.
  4. Experiments cover three task types and multiple backbones, with comprehensive metrics (erase set, retain set, COCO), sufficient ablation and layer-scope analysis, and convincing qualitative results.
Critical weaknesses
  1. The theoretical decomposition relies on rank-one approximation and ideal constraint conditions (e.g., βᵀc*=0); in practice, numerical precision and regularization terms may break this structure, leaving a gap between theoretical guarantees and actual behavior.
  2. HOC uses the SVD of cumulative historical updates as a proxy for historical directions, but the paper admits the column space may be a strict subset due to cancellation, potentially missing important interference directions; the impact of projection threshold and rank selection is not fully discussed.
  3. Experiments are mainly based on the Stable Diffusion family with sequences of up to 100 edits; generalizability to longer sequences, newer architectures (e.g., FLUX), and training-based erasure methods is limited.
  4. On erase-set metrics, CEASE does not erase most deeply; the authors explain this as "redirecting to coherent anchor substitutes" rather than weak erasure, but there is no quantitative characterization of the trade-off between erasure thoroughness and generation coherence.
  5. In baseline comparisons, some methods (e.g., UCE, RECE) perform extremely poorly in the continual setting, which may exaggerate CEASE's relative advantage; comparison with stronger continual-learning baselines is insufficient.
Questions for authors
  1. After adding the anchor token to the invariance matrix in AIC, does it affect preservation of non-anchor prompts? How is βᵀc*≈0 guaranteed under numerical precision with a zero regularizer?
  2. How are the SVD rank and projection trigger condition in HOC set? Are they sensitive across tasks? Can an adaptive selection strategy be provided?
  3. The column space of cumulative update Δ_hist may lose historical directions; have you considered keeping explicit historical α vectors or using more stable subspace tracking?
  4. CEASE's CLIP score on the erase set is higher than some baselines; how do you distinguish "redirecting to the anchor" from "incomplete erasure"? Can human evaluation or an anchor semantic consistency metric be added?
  5. Is the method applicable to training-based erasure or non-closed-form updates? How do interference and computational overhead change for longer sequences (e.g., 500+)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3hvk (Part 1/2)

Thank you for your encouraging review and for recognizing the practical setting, the solid analysis and the training-free design. We address each point below.

W1 / Q1. The theoretical decomposition relies on rank-one approximation and ideal constraint conditions (e.g., βᵀc*=0); numerical precision and regularization terms may break this structure. After adding the anchor token to the invariance matrix in AIC, does it affect preservation of non-anchor prompts? How is βᵀc*≈0 guaranteed under numerical precision with a zero regularizer?

The rank-one form is exact for a single target token and anchor, not an approximation. AIC adds c\*c_\* to the invariance matrix (Eq. 9), and with a zero regularizer the closed-form correction enforces Δic\*=0\Delta_i c_\* = 0 as an equality, giving βi⊤c\*=0\beta_i^\top c_\* = 0. The condition is thus exact in exact arithmetic, and floating-point error in the solve leaves only a residual at numerical precision, which does not restore the anchor and cross channels in any meaningful amount. We will distinguish the exact constraint from its numerical implementation.

AIC protects non-anchor prompts rather than restricting them. Because every edit redirects to the same anchor, any prompt with a component along c\*c_\* accumulates noise through the anchor and cross channels (Eq. 8). Once βi⊤c\*=0\beta_i^\top c_\* = 0, both channels vanish for every prompt (Eq. 10). Consistently, CEASE improves retain-set LPIPS and all COCO metrics over SPEED on both SD1.4 and SD2.1.

W2 / Q2 / Q3. HOC uses the SVD of cumulative historical updates as a proxy… the column space may be a strict subset due to cancellation. How are the SVD rank and projection trigger set? Are they sensitive across tasks? Have you considered keeping explicit historical α vectors?

In each layer, HOC triggers when the interference ∣Δhistc1(i)∣2\\|\Delta_{\mathrm{hist}} c_1^{(i)}\\|^2, i.e., how much past updates already change the current target embedding, exceeds μ+ησ, where μ and σ are its running mean and standard deviation (EMA, decay 0.9). It then keeps the singular directions whose eigenvalue exceeds 1e-6 of the largest, up to a rank cap given as a fraction of the output dimension. The defaults are η = 1.5 and a 75% rank cap. We varied each one at a time on SD1.4 after 100 edits, with the erasure strength fixed at its default of 1.0:

TaskηRank capTarget Acc ↓Retain Acc ↑COCO FID ↓
Celebrity1.575%3.565.946.20
Celebrity1.075%3.667.046.41
Celebrity2.075%3.066.446.27
Celebrity1.550%2.866.046.44
Celebrity1.5100%3.465.846.42
Instance1.575%11.871.062.10
Instance1.075%11.971.262.02
Instance2.075%11.671.161.33
Instance1.550%11.470.061.17
Instance1.5100%11.770.761.05

Across all settings, Target Acc varies by at most 0.8 points, Retain Acc by at most 1.2 and COCO FID by at most 1.1. CEASE is therefore not sensitive to these hyperparameters. The same defaults work on the instance task without re-tuning, and we use them for all tasks and backbones.

As you suggested, we also compared the cumulative SVD with an explicit bank of per-update directions in a matched 200-edit SD1.4 run, using CLIP top-1 accuracy on 100 target and 100 retain celebrities:

MethodTarget ↓Retain ↑COCO LPIPS ↓
CEASE, cumulative SVD10820.497
CEASE, explicit bank11790.495

The two variants perform comparably, so the explicit bank offers no clear advantage. At the endpoint, the cumulative update also preserves the full span of the stored directions in all examined layers.

AuthorsOpenReview ↗

Response to Reviewer 3hvk (Part 2/2)

W4 / Q4. CEASE does not erase most deeply… there is no quantitative characterization of the trade-off between erasure thoroughness and generation coherence. How do you distinguish "redirecting to the anchor" from "incomplete erasure"? Can human evaluation or an anchor semantic consistency metric be added?

As you suggested, we measured anchor consistency as the CLIP similarity between the SD1.4 erase-set images after 100 edits and the anchor "person":

MethodCLIP similarity to anchor ↑CLIP similarity to original prompt
Original model17.1030.95
SPEED18.1118.38
CEASE19.8821.54

CEASE's erased images are the closest to the anchor. They also stay more aligned with the original prompt than SPEED's, because the redirected images keep the non-identity content of the prompt, such as the portrait style or scene. Meanwhile, the Qwen2-VL-7B identity judge recognizes the erased celebrity in only 3.5% of these images. Following Reviewer UtoL's suggestion, we also computed LPIPS separately on the person region and the background using person masks: on the erase set, CEASE substantially changes the person region while changing the background less than SPEED, so its edits are confined to the identity. Together, this indicates that CEASE redirects targets to the anchor rather than leaving them incompletely erased. The erasure depth is also adjustable: a larger erasure strength scales each update for deeper erasure. On SD2.1 after 100 edits:

Erasure strengthTarget Acc ↓Retain Acc ↑COCO LPIPS ↓COCO FID ↓
1.034.269.60.29038.20
1.512.068.60.31040.78
2.03.466.60.33941.77

Larger strengths progressively deepen erasure. We will add both analyses.

W3 / Q5. Generalizability to longer sequences, newer architectures (e.g., FLUX), and training-based erasure methods is limited. Is the method applicable to training-based erasure or non-closed-form updates? How do interference and computational overhead change for longer sequences (e.g., 500+)?

As you suggested, we extended the SD1.4 celebrity task to 500 sequential edits of distinct celebrities:

MethodTarget Acc ↓Retain Acc ↑COCO LPIPS ↓COCO FID ↓
Reference71.772.00.0000.0
SPEED0.447.20.62897.2
CEASE0.757.20.59683.3

Both methods keep target recognition very low, and CEASE still preserves retained identities and general images better, so its advantage persists over longer sequences. AIC and HOC add only about 2% to SPEED's edit time.

For training-based erasure, both constraints apply to any additive update: AIC keeps the anchor response unchanged, and HOC projects the update's outputs away from the selected historical directions. A training-based editor can therefore use them to restrict the range of its parameter updates.

For newer architectures, we evaluated 100 sequential edits on SDXL and FLUX.1-dev (DiT, flow matching). On FLUX.1-dev, we edit the text-stream value projections and evaluate 200 target and 200 retain images.

ModelMethodTarget Acc ↓Retain Acc ↑COCO LPIPS ↓COCO FID ↓
SDXLReference88.872.60.0000.0
SDXLSPEED5.465.00.34539.1
SDXLCEASE4.465.60.32736.8
FLUX.1-devReference63.068.00.0000.0
FLUX.1-devSPEED18.052.00.58861.0
FLUX.1-devCEASE19.553.00.50954.1

On both backbones, CEASE keeps target and retain accuracy comparable to SPEED and changes general images less.

W5. Some methods (e.g., UCE, RECE) perform extremely poorly in the continual setting, which may exaggerate CEASE's relative advantage; comparison with stronger continual-learning baselines is insufficient.

Table 1 already compares CEASE with two continual unlearning baselines, CUIG-GradProj and Sculpting Memory, run from their public implementations. Both are training-based and constrain how each request is optimized, but not the weight update that is finally committed. CUIG-GradProj projects the gradient, yet the committed Adam update leaves the protected subspace, and this leakage grows by several orders of magnitude over the sequence. Sculpting Memory restricts which parameters each request updates, but does not prevent an update from changing the outputs of the shared anchor or of earlier edits. Interference therefore accumulates in both, whereas AIC and HOC constrain the committed update directly. We will add this analysis.

Thank you again for your valuable comments.

Reviewer_fKkMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of continual concept erasure in text-to-image diffusion models, where a model must erase new concepts from already-edited weights without compromising previous erasure results or the generation quality of unrelated concepts. This paper propose CEASE (Continual Erasure via Adaptive Subspace Editing), a training-free method that imposes two subspace constraints on a closed-form solver: it incorporates token representations—which share replacement anchors—into the solver’s invariant matrix, and restricts updates to the orthogonal complement of the space spanned by historical update directions, thereby minimizing interference.

Critical strengths
  1. The paper provides an analysis and experimental validation of the proposed method at a mechanistic level. Rather than treating the degradation caused by continuous erasure as a generic form of "forgetting," the authors explicitly decompose the accumulated noise into two distinct sources—the repeated activation of shared anchors and the overlap in update directions—and derive two corresponding constraints based on this decomposition.
Critical weaknesses
  1. There are shortcomings in the experimental setup. Regarding the main experiment (Table 1), which mentions the erasure of 100 celebrities: how were these 100 individuals selected? Can all 100 actually be generated by the SD1.4 model? Furthermore, why is the CLIP Score used to evaluate the "erase set"? An "erasure rate" (or success rate of removal) would be a more appropriate metric. Additionally, experimental details—such as the number of images generated per prompt—are missing. A more common use case for erasure methods involves removing NSFW concepts (e.g., pornography or graphic violence), yet the paper does not evaluate performance on such abstract concepts, nor does it address the removal of specific intellectual property (IP) characters. Moreover, while the method performs well in preserving non-targeted concepts, its effectiveness in actually erasing the intended targets falls short of the desired level (Table 8).
  2. The practical significance of the proposed method is questionable. While "continual unlearning" is indeed a valid problem, a simpler and more effective approach would be to implement a blacklist—detecting the presence of a specific celebrity and replacing the output accordingly. A more challenging scenario would involve handling adversarial prompts or indirect descriptions; however, methods relying on closed-form solutions are typically fragile in such contexts, and the proposed approach might even exacerbate this vulnerability. Furthermore, the experiments are conducted exclusively on SD1.4 and SD2.1—models that are now considered outdated. Modern generative models are built upon DiT (Diffusion Transformer) architectures and Flow Matching; since earlier techniques based on attention mechanisms and fine-tuning (such as ESD) perform poorly on these newer architectures, it is highly questionable whether the method proposed in this paper can be successfully transferred to them.
Questions for authors
  1. Address the questions regarding the experiments mentioned in "Weakness 1."
  2. Explain the advantages over blacklist-based keyword replacement, and include additional experiments focused on adversarial prompts.
  3. Demonstrate whether the approach remains effective on modern models.
10 comments
AuthorsOpenReview ↗

Response to Reviewer fKkM (Part 1/2)

Thank you for your detailed review and for recognizing our mechanistic decomposition of the accumulated noise into two sources, each addressed by one constraint. We address each point below.

W1(a). Regarding the main experiment (Table 1), which mentions the erasure of 100 celebrities: how were these 100 individuals selected? Can all 100 actually be generated by the SD1.4 model?

We follow the experimental setting of SPEED (Li et al., 2026) and take the 100 celebrities from its benchmark. we will state this more clearly. On images from the original SD1.4, the Qwen2-VL-7B identity judge recognizes the intended celebrity in 88.0% of the 500 target images and recognizes 99 of the 100 celebrities in at least one of their five images.

W1(b). Why is the CLIP Score used to evaluate the "erase set"? An "erasure rate" would be a more appropriate metric.

We agree, and Table 1 already reports one: its Acc column is the recognition accuracy of Qwen2-VL-7B, an open-ended VLM judge asked to name the person (Sec. 4), so lower target Acc means stronger erasure. We will make this clearer.

W1(c). Experimental details—such as the number of images generated per prompt—are missing.

We follow SPEED's protocol. Each celebrity has five prompt templates with one image each under a fixed seed, giving 500 images for the 100 erased and 500 for the 100 retained celebrities. MS-COCO metrics use the first 1,000 captions, one image each. Sampling uses 20-step DPM-Solver, guidance 7.5, at 512×512. We will add these details.

W1(d). A more common use case involves removing NSFW concepts (e.g., pornography or graphic violence)… nor does it address the removal of specific intellectual property (IP) characters.

For NSFW, as you suggested, we erase "nudity" on I2P following SPEED's protocol and count the exposed body parts NudeNet detects (threshold 0.6):

MethodTotal ↓
SD-v1.4646
ESD-x274
UCE182
SPEED113
CEASE117

CEASE reduces detected nudity by 82% relative to SD-v1.4, comparable to SPEED and well below ESD-x and UCE.

IP characters are covered by our instance task (Table 3), whose 100 erased concepts are mostly copyrighted characters such as Mickey, Pikachu, SpongeBob, Hello Kitty, Mario and Bugs Bunny:

MethodErase Acc ↓Retain Acc ↑COCO CS ↑COCO FID ↓
SD1.480.278.626.520.0
SPEED5.766.222.8078.0
CEASE11.871.025.2662.1

We will add the NSFW results and state that the instance task consists of IP characters.

W1(e). While the method performs well in preserving non-targeted concepts, its effectiveness in actually erasing the intended targets falls short of the desired level (Table 8).

On SD2.1 (Table 8), the methods with zero target Acc also lose nearly all retained identities (retain Acc ≤ 0.4), while CEASE keeps retain Acc near the original model. CEASE's default changes general images the least (lowest COCO LPIPS and FID in Table 8), and a larger erasure strength makes it erase more. On SD2.1 after 100 edits:

Erasure strengthTarget Acc ↓Retain Acc ↑COCO LPIPS ↓COCO FID ↓
1.034.269.60.29038.20
1.512.068.60.31040.78
2.03.466.60.33941.77

A larger strength lowers target recognition and changes general images only modestly. At strength 2.0, CEASE erases more than SPEED in Table 8 with comparable COCO quality. We will add this result.

AuthorsOpenReview ↗

Response to Reviewer fKkM (Part 2/2)

W2(a) / Q2. A simpler and more effective approach would be to implement a blacklist—detecting the presence of a specific celebrity and replacing the output accordingly. Explain the advantages over blacklist-based keyword replacement.

You are right that a blacklist is simple to deploy. Blacklists and output filters complement weight editing, which has two advantages. First, users of a released or self-hosted model can remove a filter, whereas an edited checkpoint keeps the change. Second, weight editing adds no inference cost, whereas a filter needs a detector call on every image and a new generation for every flagged one. A deployed service can combine both.

W2(b). Methods relying on closed-form solutions are typically fragile with adversarial prompts or indirect descriptions; the proposed approach might even exacerbate this vulnerability.

To test your concern, we attacked the SD1.4 celebrity models with UnlearnDiffAtk [1] on 20 erased celebrities across the five templates. An attack succeeds if the VLM judge recognizes the target.

MethodTypeBefore attack ↓After attack ↓
ESDTraining-based20%47%
SPEEDClosed-form0%25%
CEASEClosed-form1%27%

The two closed-form editors recover fewer targets than the training-based ESD. CEASE differs from SPEED by two prompts out of 100, so its constraints do not make the update more vulnerable. Every method still recovers some identities under attack; we will add these results and discuss this limitation.

W2(c) / Q3. The experiments are conducted exclusively on SD1.4 and SD2.1… it is highly questionable whether the method can be successfully transferred to DiT and Flow Matching models.

To address your concern, we ran 100 sequential celebrity edits on SDXL and FLUX.1-dev (DiT, flow matching). On FLUX.1-dev, we edit the text-stream value projections of the joint-attention blocks, which play the role of cross-attention value projections in UNets, and evaluate 200 target and 200 retain images.

ModelMethodTarget Acc ↓Retain Acc ↑COCO LPIPS ↓COCO FID ↓
SDXLReference88.872.60.0000.0
SDXLSPEED5.465.00.34539.1
SDXLCEASE4.465.60.32736.8
FLUX.1-devReference63.068.00.0000.0
FLUX.1-devSPEED18.052.00.58861.0
FLUX.1-devCEASE19.553.00.50954.1

On both backbones, CEASE keeps target and retain accuracy comparable to SPEED and changes general images less. CEASE thus transfers to a flow-matching DiT. We will add these results.

[1] To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy to Generate Unsafe Images ... For Now

Thank you again for your valuable comments.

Dear Reviewer fKkM,

Thank you for your time and valuable feedback. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We would be happy to discuss them.

Best regards, The Authors

Reviewer_UtoLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper CEASE tells that most of the time erasure of multiple concepts is not simultaneous but sequential. The erasure requests arrive over time, and in such cases, the erasure targets interfere with each other and with retain set priors, reducing the generative quality of the models. Figure 1 shows the effects of continual erasure- the unrelated concepts get harmed after 100 edits, the prompt semantics are not followed and in general, FID scores increase and ClipScores decrease. The authors try to quantify the effect of continuous additive updates and hypothesize that edits sharing same general anchor leads to the quality degradation. Then, the accumulated noise through the sequential edits are decomposed into anchor-associated, off-anchor components and cross-components between anchor and off-anchor edits. The authors then try to zero out the anchor-associated and cross- noise in Anchor Invariance Constraint (AIC) by appending the anchor concepts to the null embedding matrix. The off-anchor noise is handled by Historical Orthogonality Constraint (HOC). The SVD space of the previous updates is calculated and the new edit is projected onto its orthogonal space to reduce interference. In summary, CEASE presents a practical remedy to degrading interference between edits effectively with promising qualitative and quantitative results.

Critical strengths
  1. The paper identifies the shared anchor for closed-form KV weight update based erasure as the prime reason why the sequential edits harm retention quality. The authors provide valid, grounded solutions to counter the sequential edit interferences.

  2. Even 100 celebrities, art, instance erasure leads to best retain set results, compared to other SOTA methods.

  3. The qualitative results look good prior preservation with correct target erasures even after 100 edits.

Critical weaknesses
  1. The authors mention that they use SPEED solver in section 3.3 AIC subsection, without any citation and explanation as to what it contains/implies. Specially, the authors introduce invariance matrix K_2 in the equations without explaining in the main section what it is. A direction to better understanding of the method would be to include what SPEED does originally and where they brought the changes exactly.

  2. In HOC, projecting away the historical directions might reduce erasure efficacy, is that why the erasure set measures are suboptimal?

  3. With respect to the earlier weakness, even though the erasure set results are sub-optimal the qualitative results look solid. So, an evaluation of localized erasure with metrics like FG-LPIPS and BG-LPIPS might highlight the efficacy more.

  4. The diffusion architecture coverage for evaluations is limited. It would be helpful if the extensions to SDXL, DiT backbones were shown.

Questions for authors
  1. What happens in case of the weight edits which are not additive (W + <>) but multiplicative (W * <>). [a]
  2. Why the rank-1 assumption? (papers with additive edits but not rank-1)
  3. Does the noise directly relate to the generative quality? Is it linearly related?
  4. Not every edit has the same anchor, how does the work handle that?
  5. The erase set results have a significant gap wrt to the other erase methods. Could the authors also show BG-LPIPS and FG-LPIPS to highlight the effectiveness of their method?
  6. The authors have shown noise variance across the 100 erasure steps, could they also show the forget-retain set value variation with the steps?

References: [a] Sun, Yuhao, et al. "Orthogonal Concept Erasure for Diffusion Models." arXiv preprint arXiv:2605.28902 (2026).

5 comments
AuthorsOpenReview ↗

Response to Reviewer UtoL (Part 1/3)

Thank you for your constructive review and for recognizing that the shared anchor is the main source of cross-edit interference and that our retain-set results are strong. We address each point below.

W1. The authors mention that they use SPEED solver in section 3.3 AIC subsection, without any citation and explanation as to what it contains/implies. Specially, the authors introduce invariance matrix K_2 in the equations without explaining in the main section what it is. A direction to better understanding of the method would be to include what SPEED does originally and where they brought the changes exactly.

Thank you for pointing this out. K2\mathbf{K}_2 is the matrix of SPEED's Invariant Equality Constraints (Sec. 4.3 of SPEED), denoted C2\mathbf{C}_2 there. We will cite SPEED, adopt its notation and rewrite the AIC paragraph as follows:

Anchor Invariance Constraint (AIC). CEASE builds on the closed-form solver of SPEED (Li et al., 2026). SPEED redirects each target toward its anchor (Eq. 1), projects the update onto the near-null space P\mathbf{P} of a refined retain set, and enforces the Invariant Equality Constraints (IEC) (ΔP)C2=0(\boldsymbol{\Delta}\mathbf{P})\mathbf{C}_2=\mathbf{0} in closed form. C2\mathbf{C}_2 stacks the [SOT] and null-text embeddings, which every prompt and the unconditional branch share. CEASE keeps these steps and changes the solver in two places: AIC enlarges C2\mathbf{C}_2, and HOC projects the solved update (Eq. 14). Although every request redirects to the task anchor c\*\boldsymbol{c}_\*, C2\mathbf{C}_2 does not contain it. AIC therefore appends the anchor token embedding:

C2AIC=[C2, c\*],ΔiC2AIC=0.(9)\mathbf{C}_2^{\mathrm{AIC}} = [\mathbf{C}_2,\ \boldsymbol{c}_\*], \qquad \boldsymbol{\Delta}_i\mathbf{C}_2^{\mathrm{AIC}} = \mathbf{0}. \qquad (9)

W2. In HOC, projecting away the historical directions might reduce erasure efficacy, is that why the erasure set measures are suboptimal?

W3. With respect to the earlier weakness, even though the erasure set results are sub-optimal the qualitative results look solid. So, an evaluation of localized erasure with metrics like FG-LPIPS and BG-LPIPS might highlight the efficacy more.

Q5. The erase set results have a significant gap wrt to the other erase methods. Could the authors also show BG-LPIPS and FG-LPIPS to highlight the effectiveness of their method?

HOC is not the main cause of the weaker default erasure. In Table 4, HOC alone improves preservation with only slightly weaker erasure, and adding HOC to AIC leaves target recognition unchanged while improving retained identities and all COCO metrics. HOC therefore mainly preserves the model's general utility, and a larger erasure strength recovers erasure depth. On SD2.1 after 100 edits:

Erasure strengthTarget Acc ↓Retain Acc ↑COCO LPIPS ↓COCO FID ↓
1.034.269.60.29038.20
1.512.068.60.31040.78
2.03.466.60.33941.77

A larger strength lowers target recognition and changes general images only modestly. A moderate increase keeps retain Acc nearly unchanged; the strongest setting costs about three points of retention.

CS, FID and LPIPS favor images that move far from the prompt and the original images, whereas redirection to "person" keeps the non-identity content of the prompt. As you suggested, we therefore also measure where images change with localized LPIPS, using person masks from the original-model images:

MethodErase FG-LPIPS ↑Erase BG-LPIPS ↓Retain FG-LPIPS ↓Retain BG-LPIPS ↓
SPEED0.6330.6080.3940.363
CEASE0.6010.5330.3400.302

On the erase set, CEASE changes the background less than SPEED, so its edits are more confined to the person. On the retain set, it better preserves both regions. We will add these results.

AuthorsOpenReview ↗

Response to Reviewer UtoL (Part 2/3)

W4. The diffusion architecture coverage for evaluations is limited. It would be helpful if the extensions to SDXL, DiT backbones were shown.

As you suggested, we ran 100 sequential celebrity edits on SDXL (UNet) and FLUX.1-dev (DiT, flow matching). On FLUX.1-dev, we edit the text-stream value projections of the joint-attention blocks, which play the role of cross-attention value projections in UNets, and evaluate 200 target and 200 retain images.

ModelMethodTarget Acc ↓Retain Acc ↑COCO LPIPS ↓COCO FID ↓
SDXLReference88.872.60.0000.0
SDXLSPEED5.465.00.34539.1
SDXLCEASE4.465.60.32736.8
FLUX.1-devReference63.068.00.0000.0
FLUX.1-devSPEED18.052.00.58861.0
FLUX.1-devCEASE19.553.00.50954.1

On SDXL, CEASE matches SPEED in target and retain accuracy and changes general images less (lower COCO LPIPS and FID). On FLUX.1-dev, it erases slightly less, keeps retain accuracy comparable and changes general images notably less. The benefit for general images thus carries over to a DiT backbone. We will add these results.

Q1. What happens in case of the weight edits which are not additive (W + <>) but multiplicative (W * <>). [a] Sun et al., "Orthogonal Concept Erasure for Diffusion Models."

A multiplicative edit can be rewritten as an additive one. OCE [a] edits a layer as W∗=PWW^* = PW with an orthogonal PP, which equals W+ΔW + \Delta with Δ=(P−I)W\Delta = (P-I)W. Our noise analysis therefore applies directly, and it does not require Δ\Delta to be rank one (see Q2). We will cite [a] and add this discussion.

Q2. Why the rank-1 assumption? (papers with additive edits but not rank-1)

Rank one is a property of our setting, not an assumption of the analysis. Each request edits the last subject-token embedding, and the closed-form update for one target and one anchor has rank one. For any additive update with SVD Δ=∑kαkβk⊤\Delta=\sum_k \alpha_k\beta_k^\top, Eq. 5 holds with the sum over all components, so a rank-rr update adds rr interference terms of the same form. The constraints also need no rank-one form: AIC requires Δc\*=0\Delta c_\* = 0, and HOC left-projects the update onto the orthogonal complement of the selected historical output directions, which preserves an existing anchor null in exact arithmetic. We will clarify this.

Q3. Does the noise directly relate to the generative quality? Is it linearly related?

The noise is defined on the weights: it measures how much the accumulated updates change a prompt's value-projection output (∑iΔi)c\big(\sum_i \boldsymbol{\Delta}_i\big)\boldsymbol{c} beyond its intended edit (Eq. 4). It is related to, but not the same as, generation quality, because the value output still passes through attention and the denoising network. In our experiments, lower noise consistently accompanies better quality. We will clarify this.

AuthorsOpenReview ↗

Response to Reviewer UtoL (Part 3/3)

Q4. Not every edit has the same anchor, how does the work handle that?

You are right that edits can use different anchors. AIC extends to multiple anchors by adding their embeddings to the invariance matrix; HOC is unchanged. We compared three policies: protecting the current anchor, all anchors seen so far, or all anchors from the first edit (which assumes the anchor set is known in advance). On SD1.4 after 100 edits with changing anchors:

Method / anchor protectionTarget Acc ↓Retain Acc ↑
SPEED0.659.2
CEASE, current anchor only3.069.2
CEASE, all anchors seen so far2.265.6
CEASE, all anchors known in advance2.667.2

Every CEASE policy keeps target recognition low and retains more identities than SPEED. Protecting more anchors keeps more anchor prompts unchanged but does not improve retain accuracy, because the anchors are generic concepts that are not necessarily among the retained identities. We will add these results and clarify the scope of the shared-anchor analysis.

Q6. The authors have shown noise variance across the 100 erasure steps, could they also show the forget-retain set value variation with the steps?

As you requested, we report forget accuracy (on the targets erased so far) and retain accuracy (on the fixed retain set) on SD1.4 every 10 edits.

StepSPEED forget ↓CEASE forget ↓SPEED retain ↑CEASE retain ↑
100.04.069.069.2
200.08.065.068.8
300.76.063.470.2
402.05.060.669.4
500.46.058.468.8
601.04.757.866.0
700.93.754.267.4
800.53.556.067.6
900.74.755.267.8
1000.33.554.465.9

Both methods keep forget accuracy low. SPEED loses retention from the first edits, whereas CEASE stays nearly flat through step 50 and declines only slightly afterwards, so the gap widens over the sequence. We will add this trajectory.

Thank you again for your valuable comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.