acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Unlearning via Exact Continuous Mixture Scores

Abstract

Machine unlearning aims to mitigate exposure to sensitive training data and undesirable semantic concepts without retraining foundation diffusion models from scratch. While there are concept unlearning methods relying only on inference-time steering, data unlearning has predominantly relied on parameter fine-tuning, an approach that is computationally intensive and requires synthetic retain sets to prevent generation performance collapse. Emerging training-free alternatives circumvent retraining by steering reverse sampling trajectories away from target data, but they choose their repulsion force heuristically and avoid explicit computation of posterior terms as intractable. Conversely, we show that the mathematically principled repulsion weight is the instantaneous continuous-time posterior odds of entering forbidden data under the variance-preserving stochastic differential equation. We establish that unconditional data unlearning and conditional concept erasure share a common mathematical formulation, unifying both within a single training-free operator. We show that our proposed method, namely, Exact Mixture Score (EMIS) constitutes the exact spatial score of an explicit normalized mixture density with an a-priori closed-form leakage law and replaces heuristic time-window cutoffs with an uncertainty-aware continuous temperature schedule. Across unconditional pixel models (CelebA-HQ 256) and conditional text-to-image synthesis (Stable Diffusion on UnlearnCanvas for both style- and object-unlearning), EMIS not only outperforms heuristic training-free diffusion guidance methods but also matches the unlearning efficacy of full parameter fine-tuning while preserving generative fidelity on retained concepts with zero weight updates.

Primary area:
generative models
Keywords:
diffusion models, machine unlearning, score-based models, classifier guidance, continuous-time diffusion, concept erasure

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UwDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new inference-guiding concept suppression method that, instead of choosing the repulsion force heuristically, it uses a principled method. Furthermore, the authors show that it is not necessary to avoid the explicit computation of posterior terms, as they are not intractable, and can be computed using the original model's denoising posterior. The proposed method, EMIS, formulates unlearning as posterior-odds re-weighting. To ensure the guidance remains stable, the authors propose to bound the odds and show that this bounded guidance is the exact score of a regularized mixture density. Finally, EMIS works for both concept and data unlearning.

Critical strengths
  • Demonstrating that evaluating the odds does not require the retain dataset is an important contribution
  • The framework works for both concept and data unlearning
Critical weaknesses
  • Multiple facts stated in the paper and information reported are false or contain inaccuracies. For instance, not all parameter fine-tuning methods are computationally intensive and require synthetic retain sets to prevent performance collapse. For instance, all closed-form methods (like UCE and RECE) are computationally cheap and require less VRAM than some inference-time steering methods. Another example: wrong paper title in the reference Kirchhof et al., 2025. The paper should be significantly revised for fact checking and claims restatements.
  • The paper does not demonstrate the usage of the proposed method for Diffusion Transformer (DiT) architectures, which are now the state-of-the-art. This significantly limits the applicability of the framework
Questions for authors
  • Can you show the computational time of the proposed algorithm compared to other inference-guiding techniques?
  • Since the unlearning method relies on a classifier, can you provide an ablation study to what happens when using an under-trained classifier? For instance, what happens when using a classifier that achieves 50-60-70-80-90% accuracy? Because training a classifier in a controlled benchmark like UnlearnCanvas is rather simple, but in real-world scenarios we can expect to not have such a well-trained classifier.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer UwDj (1/2): W1 and W2

We thank the reviewer for the review, for the attention to accuracy, and for the reference pointed out. We are glad the reviewer found that "Demonstrating that evaluating the odds does not require the retain dataset is an important contribution" and that "The framework works for both concept and data unlearning". These comments helped us make the related-work discussion more precise and broaden the evaluation (DiT, timing, classifier quality); we address each point below.

(W1) We thank the reviewer for pointing us to closed-form editors and to the published title of Kirchhof et al.; both are now reflected in the paper. Our statements on compute and synthetic retain sets were written for iterative fine-tuning for instance unlearning (e.g., SISS, EraseDiff). Following the reviewer's comment, this scope is now explicit (abstract, Sec. 1–2), and Sec. 1–2 discuss closed-form editors separately: as the reviewer notes, UCE (Gandikota et al., 2024) and RECE (Gong et al., 2024) are "computationally cheap and require less VRAM than some inference-time steering methods", and they add no cost at inference. We present EMIS as complementary to them: it needs no text conditioning, so it also applies to unconditional instance unlearning, and it serves every request from one unmodified set of weights. For Kirchhof et al. (2025), our entry carried the title of the arXiv version (arXiv:2410.06025); it now carries the ICML title, "Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency." Following the reviewer's recommendation, we also re-verified every reference against its published version, updating 5 entries, and revised our statements about other methods accordingly; the requested timing table is in Table R4 (part 2/2).

(W2) We thank the reviewer for this suggestion: EMIS applies to DiTs without modification; new DiT-XL/2 results. As the reviewer notes, DiT architectures "are now the state-of-the-art". Our theory is agnostic to how the score is obtained, and the estimator queries the denoiser as a black box (Alg. 1). The benchmarks in the paper are tied to specific U-Net checkpoints (CelebA-HQ DDPM in Alberti et al., 2025; the stylized SD v1.4 checkpoint and evaluation judges in UnlearnCanvas, Zhang et al., 2024), and for text-to-image models Carlini et al. (2023) catalog memorized targets for U-Net checkpoints (SD v1.4) with no reference set for DiT checkpoints. We therefore built a compact protocol on class-conditional ImageNet DiT-XL/2 (256×256) covering both settings, re-running training-free baselines under identical conditions; because EMIS only queries the denoiser, the same procedure carries over directly to text-to-image transformers.

Table R3. DiT-XL/2 instance unlearning (10 targets; kernel membership, no classifier; erased = SSCD < 0.5).

MethodTarget SSCD ↓Erased ↑Retain SSCD ↑
Base1.000%1.00
Constant guidance0.16100%0.26
SLD0.6610%1.00
Safe Denoiser0.960%0.91
SPELL0.5940%1.00
EMIS0.4280%0.91

Classes (10 ImageNet classes paired with similar retain classes; same ResNet-50 guide for all; ViT-B/16 judge): EMIS reaches UA 95.5 / RA 94.5 / H 95.0 with the best retain SSCD (0.853, vs. 0.812 and 0.815), ahead of a constant weight (94.7 H) and plug-in (93.4 H).

As in the main benchmarks, the trade-off is what matters: constant guidance erases everything by destroying retained content, while SLD and SPELL preserve content but erase little. EMIS gives the strongest erasure among the methods that preserve retained content (80% of targets, vs. at most 40% for the others).

AuthorsOpenReview ↗

Reply to Reviewer UwDj (2/2): Q1 and Q2

(Q1) Inference time: small overhead without a decoder; on SD v1.4 the cost is decoding for a pixel-space classifier. We thank the reviewer for asking for this comparison; we timed all methods under one protocol. CelebA-HQ (Table 1, V100): base 2.07 s, SLD / SPELL 2.10 s, Safe Denoiser 2.14 s, EMIS 2.46 s. DiT-XL/2: EMIS 1.33 s vs. 1.25 s base and 1.27–1.32 s for others.

Table R4. SD v1.4 inference time, one protocol (50 steps, batch 1, including VAE decoding; fp16 EMIS classifier).

s/imgBaseNeg. promptSEOTSLDTraSCESAFREEEMIS
V1002.923.143.174.104.124.535.93
H1000.910.920.921.000.991.001.77

With the off-the-shelf pixel-space ViT, 73–79% of the SD v1.4 overhead comes from decoding x^0\hat{x}_0 through the VAE at each active step (2.37 s on V100, 0.63 s on H100). Without per-step VAE decoding (the classifier plus the ~7/50 steps where the gate triggers the third U-Net pass), EMIS takes 3.56 s on V100 (+22%) and 1.14 s on H100 (+25%), consistent with the Table 2 footnote and with Table 1 (+19% on CelebA-HQ, +5.9% on DiT-XL/2), and faster on V100 than SLD (4.10 s), TraSCE (4.12 s), and SAFREE (4.53 s), which run a third U-Net pass at all 50 steps. We note that the decoder is a learned mapping that can be distilled or folded into the classifier to remove this overhead; even with pixel-space decoding, the extra cost on an H100 is under 0.9 s/img, with no fine-tuning or stored checkpoints (App. E.4).

(Q2) Instances need no classifier; for concepts, EMIS degrades more gracefully than plug-in guidance at every reduced accuracy. We thank the reviewer for suggesting this ablation, which is one of the most informative additions to the paper. For instance unlearning, m(x0)m(x_0) is an analytic kernel (Eq. 8). For concepts, we fine-tuned ImageNet-21k ViT-L classifiers on UnlearnCanvas styles and stopped early: (i) standard LR, few steps; (ii) 10× lower LR, checkpoints at 50–90% validation accuracy. To isolate the variance-aware temperature τt\tau_t without confounding variables, we compare EMIS (κ=1\kappa = 1) on 10 styles against plug-in odds (κ=0\kappa = 0) using the exact same classifier, clipped-odds formula wλ(πt)w_\lambda(\pi_t), and 1/λ=3.51/\lambda = 3.5 cap.

Table R5. Classifier-accuracy sweep on UnlearnCanvas styles.

RecipeAcc.EMIS HPlug-in H
(i)97.6%66.549.2
(i)80.3%58.838.4
(i)52.0%51.329.6
(ii)89.95%44.136.8
(ii)80.05%42.933.3
(ii)69.80%36.330.3
(ii)60.05%37.633.3
(ii)50.05%35.634.1

Performance drops with accuracy, since the classifier defines what is erased, but EMIS beats plug-in guidance at every reduced accuracy by +1.5 to +21.7 H, mainly via retention (CRA +12 to +29 pp): while the posterior is uncertain, κ>0\kappa > 0 inflates τt\tau_t, damping confident but wrong outputs. We attribute the lower H of recipe (ii) at equal accuracy to the flatter, more under-confident logits that its 10× smaller learning rate produces (at 80% accuracy: mean max-probability 0.49 and ECE 30.7%, vs. 0.64 and 15.9%): with the gate offset set for the converged classifier, the gate fires on nearly all retain images, which lowers retention (IRA 20–27% vs. 32–49%). We added this ablation (App. E.3) and discuss classifier quality as a limitation.

We thank the reviewer again; these comments made the paper more precise and its evaluation broader. We hope these updates and new experiments address the reviewer's concerns, and we are happy to discuss further. If they do, we would be grateful if the reviewer would consider updating the score.

I thank the authors for their responses. The paper now surely improved from my point of view, but I still will wait for the other reviewers to comment on their side, as I am still unsure about the importance and impact of the overall contribution, mainly due to the dependency on the classifier.

AuthorsOpenReview ↗

On the dependence on the classifier

We thank the reviewer for the follow-up and are glad the revision improved the paper in their view. We would like to address the remaining concern, the dependence on the classifier, directly.

The classifier specifies what to unlearn, which is part of the task, not an extra assumption. Every unlearning method must be told what to forget: the forget set, or membership function, is part of the problem definition. Fine-tuning methods receive it as forget images or prompts and optimize the generator once per concept; prompt-based inference-time methods receive it through the text encoder. EMIS takes the same information as an explicit membership function. Without it, the unlearning problem is undefined for any method. Once the target is specified correctly, EMIS erases it almost losslessly: with the evaluation classifier as an oracle, it reaches 98.25 / 97.39 H on UnlearnCanvas styles / objects (Table 2).

Instance unlearning uses no classifier. There, membership is an analytic kernel around the target image (Eq. 8). Without any classifier, EMIS matches or improves on the fine-tuning methods SISS and NegGrad in Target SSCD on CelebA-HQ (0.40 vs. 0.51 / 0.46 at M = 6; 0.38 vs. 0.39 / 0.40 at M = 50) while keeping Clean-FID at or below the base model's (45.0 / 41.3 vs. 45.2), without the 2.7–3.1 h of optimization and 0.45 GB checkpoint they need per target cohort (Sec. 4.1, Table 1). On DiT-XL/2, it erases 80% of targets vs. at most 40% for the other methods that preserve retained content (Table R3, App. Table A10).

For concepts, the classifier is cheap, and EMIS reduces the dependence on its quality.

  • One standard image classifier covers all 50 styles; it is trained once, separately from the generator.
  • In Table 2, the inference-time methods that rely on the prompt alone erase 23.25–49.85% of objects; EMIS with an independently seeded classifier erases 92.80% (H = 93.92), on par with the strongest fine-tuning methods.
  • With deliberately degraded classifiers (50–98% accuracy), EMIS beats plug-in guidance with the same classifier at every accuracy level tested, by +1.5 to +21.7 H (Table R5, App. Table A7). Better estimators of the membership function (what concepts to forget) therefore translate directly into better unlearning, with no retraining of the generator.

The core results do not depend on how membership is specified. Lemma 3.1 (evaluating the odds does not require the retain set) and Theorem 3.2 (exact mixture score with closed-form leakage) hold for any membership function, kernel or classifier.

Together, this lets a provider serve changing deletion requests, from specific training images to entire concepts, with one frozen model, no retain data, and no per-request checkpoints. We hope this resolves the reviewer's remaining concern; if anything is still unclear, we are happy to provide more details.

Reviewer_NPvzOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method for inference-time concept unlearning in diffusion models. The authors derive a mixture-score formulation with a closed-form leakage relationship and approximate the posterior odds using Tweedie's prediction. Experiments cover instance and concept unlearning. The method is evaluated on instance unlearning with CelebA-HQ and concept unlearning with Stable Diffusion on UnlearnCanvas.

Critical strengths
  • Interesting theoretical formulation. Instead of introducing another heuristic guidance schedule, the authors derive the repulsion weight from a probabilistic mixture model. The explicit connection to residual leakage is shown to be very useful.

  • Strong empirical results. The proposed method achieves competitive unlearning and retention performance across instance and concept unlearning, without updating diffusion model weights. The comparison between decoupled and oracle settings is also informative.

Critical weaknesses
  • I am not fully sure about the theory part and wonder what's the gap between theory and implementation. The theoretical results assume exact posterior odds and component scores, while the implementation relies on several approximations. For example, Eq. (A11) does not appear to compute the exact posterior expectation. It is unclear to me whether the claimed leakage guarantee remains valid in practice.

  • Dependence on auxiliary classifiers. Table 2 shows a significant gap between decoupled and oracle settings. I think this suggests that classifier alignment strongly affects the results, making it difficult to isolate the contribution of posterior-odds weighting.

Overall, I think this work provides a new perspective to concept unlearning. My current rate is borderline.

Questions for authors
  • Can the authors verify whether the predicted leakage relationship holds under the approximate posterior used in practice?

  • How sensitive is EMIS to classifier errors? How would simpler guidance methods perform with the same auxiliary classifier?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer NPvz (1/2): W1 and Q1

We thank the reviewer for the thoughtful and encouraging review. We are delighted that the reviewer thinks "this work provides a new perspective to concept unlearning", and found an "Interesting theoretical formulation", whose "explicit connection to residual leakage is shown to be very useful", and "Strong empirical results", noting that "The comparison between decoupled and oracle settings is also informative." We especially appreciate how carefully the reviewer read the paper, down to Eq. (A11) in the appendix. The reviewer's two questions led to new results (Tables R2, R5, and R6) that make the paper stronger.

(W1 / Q1) The guarantee holds for the exact gate; the deployed sampler estimates it, and the revision makes this explicit. The reviewer is right that "The theoretical results assume exact posterior odds and component scores". The paper separates two levels: Sec. 3.1 derives the exact result for the ideal gate πt(xt)=E[m(x0)∣xt]\pi_t(x_t) = \mathbb{E}[m(x_0) \mid x_t], giving one retain-free framework for both instance and concept unlearning; Sec. 3.2 (App. A–B) estimates that gate, since the exact posterior expectation is intractable for a pretrained high-dimensional model. This is standard practice: classifier guidance (Dhariwal & Nichol, 2021) is derived for the exact noisy posterior and implemented with a learned classifier, and DPS (Chung et al., 2023) and Safe Denoiser (Kim et al., 2025) evaluate clean-space quantities at Tweedie's mean x^0\hat{x}_0. Our estimator refines this plug-in with the posterior variance and a closed-form probit marginalization (App. B).

Eq. (A11). We thank the reviewer for checking this closely. Eq. (A11) is a deliberate approximation rather than the exact posterior expectation. In D=3×2562D = 3 \times 256^2 dimensions, the exact Gaussian convolution carries a factor (σ0/τt)D(\sigma_0/\tau_t)^D, which vanishes exponentially as soon as the posterior is wider than the kernel and would switch the gate off at every noise level. Eq. (A11) instead evaluates the convolution along the radial direction uu toward the target (Eq. A10) and normalizes it to peak at 1 when x^0=x∗\hat{x}_0 = x^*, so the gate stays on a usable scale. Like the rest of the estimator, this belongs to Sec. 3.2 rather than to the exact result of Sec. 3.1, and we now state it explicitly in App. B.2.

Q1. Theorem 3.2 predicts that target odds shrink by λ/(1+λ)\lambda/(1+\lambda). Where every quantity is tractable (1D), observed and predicted agree (Fig. 1e: 0.5% vs. 0.5% at λ=10−2\lambda = 10^{-2}). With the Gaussian gate of Eq. (9) (v0∣t=1−αˉtv_{0|t} = 1 - \bar\alpha_t, κ=1.5\kappa = 1.5), the same 1D setting tracks the prediction through λ=0.1\lambda = 0.1 (19.1%, 8.6%, 3.5% target probability vs. 20.0%, 10.3%, 4.3% for λ=1,0.3,0.1\lambda = 1, 0.3, 0.1) and levels off at 2.1%, 1.7% (vs. 1.4%, 0.5%) at λ=0.03,0.01\lambda = 0.03, 0.01. On high-dimensional real data, a faithful comparison needs the true target prior π\pi and an exact membership oracle, neither of which is available. As the closest test, we swept λ\lambda "under the approximate posterior used in practice" (Table R6), measuring target survival (the fraction of target-producing runs that still produce the target under guidance: seeds reproducing the M=50M = 50 target images with SSCD > 0.5 on CelebA-HQ, or prompts for the target style on UnlearnCanvas) and unrelated-seed preservation (SSCD to the base model's output on 100 unrelated seeds). The qualitative prediction holds: target survival decreases monotonically as λ\lambda decreases, with high preservation on unrelated seeds (0.98 to 0.86); on CelebA-HQ it levels off for λ≤0.1\lambda \le 0.1, while on UnlearnCanvas it continues to decrease through λ=0.01\lambda = 0.01. We attribute the CelebA-HQ level to how well membership is estimated rather than to λ\lambda: with the oracle classifier (the evaluation classifier) as the gate, the UnlearnCanvas residual is 0 of 100 at every λ\lambda, consistent with the theorem being a statement about the exact gate. The absolute factor λ/(1+λ)\lambda/(1+\lambda) cannot be read from these runs, because they start from seeds and prompts that reach the target almost surely under the base model, whereas the theorem concerns an interior prior. The revision clarifies that the guarantee holds for the exact gate and that the sampler approximates it.

Table R6. λ\lambda sweep on real data (CelebA-HQ: M=50M = 50, σ0=0.10\sigma_0 = 0.10, κ=1.5\kappa = 1.5; UnlearnCanvas: 10 styles × 10 objects).

λ\lambda310.30.10.030.01
CelebA-HQ, target survival (n=148n = 148)0.840.510.280.220.220.22
CelebA-HQ, unrelated-seed preservation0.980.930.890.870.860.86
UnlearnCanvas (decoupled), target survival (n=100n = 100)0.390.340.340.270.200.18
UnlearnCanvas (oracle), target survival (n=100n = 100)0.000.000.000.000.000.00
AuthorsOpenReview ↗

Reply to Reviewer NPvz (2/2): W2 and Q2

(W2 / Q2) The Decoupled–Oracle gap comes from the classifier, not the weighting. We thank the reviewer for these two questions; the new ablations below isolate the contribution of posterior-odds weighting. Guided by the evaluation classifier (the oracle), EMIS reaches H = 98.25 / 97.39 on styles / objects (H: harmonic mean of unlearning and retain accuracies): once the target is specified correctly, the weighting erases it almost losslessly. With an independently seeded classifier (99.95% accuracy), the remaining gap comes from a few visually adjacent styles where the two classifiers disagree (App. C.2, Table A3). Specifying the target takes one classifier for all 50 styles, whereas fine-tuning methods optimize the generator once per concept (Table 2).

Isolating the weighting rule. EMIS steers each step along the repulsion direction s−sqs - s_q with weight w=wλ(πt)=πt/(1−πt+λ)w = w_\lambda(\pi_t) = \pi_t/(1-\pi_t+\lambda), the clipped posterior odds. To isolate each component without confounding variables, Table R2 keeps the same 99.95% classifier and margin C=2.5C = 2.5 and varies only how ww is formed: a constant step weight ww once πt>0.5\pi_t > 0.5 (where w=3.5w = 3.5 is rescaled to EMIS's own maximum 1/λ1/\lambda); linear probability (w=πtw = \pi_t); plug-in odds, which still uses EMIS's clipped-odds formula wλ(πt)w_\lambda(\pi_t) and 1/λ=3.51/\lambda = 3.5 ceiling but drops the posterior-variance correction (κ=0\kappa = 0, πt=m(x^0)\pi_t = m(\hat{x}_0)); and unclipped odds (λ→0\lambda \to 0).

Table R2. Weighting rules with the same classifier (10-style UnlearnCanvas subset, SD v1.4, scored by the benchmark's evaluation classifiers; settings differ from Table 2, so H is not directly comparable: the 10 styles include several of the hardest in Table A3, where they average UA 79.5 and H 81.7 vs. 92.7 and 91.04 over all 50).

WeightUA / CRA / IRAH
Constant, w=1w = 163.5 / 98.0 / 90.581.1
Constant, w=3.5w = 3.576.0 / 96.5 / 83.584.5
Linear prob. (w=πtw = \pi_t)64.0 / 98.5 / 92.582.0
Plug-in odds (κ=0\kappa = 0)75.0 / 97.5 / 87.085.5
Unclipped odds (λ→0\lambda \to 0)82.0 / 83.0 / 83.082.7
EMIS70.5 / 99.5 / 88.584.4

Posterior-odds weighting improves over linear probability w=πtw = \pi_t (84.4 vs. 82.0 H) and constant w=1w = 1 (81.1 H), and clipping is what protects retention (CRA 99.5 and IRA 88.5 vs. 83.0 / 83.0 for unclipped odds). The only rows that match EMIS (κ=0\kappa = 0 plug-in odds at 85.5 H and constant w=3.5w = 3.5 at 84.5 H) both borrow EMIS's 1/λ=3.51/\lambda = 3.5 ceiling and clipped-odds transformation, lying within 1 binomial SE (UA SE ≈ 3 pp) when the classifier is 99.95% accurate, whereas at the uncalibrated prior offset (C=log⁡KC = \log K, Table A2) κ=1\kappa = 1 already improves over κ=0\kappa = 0 by +5.4 H (62.26 vs. 56.85).

Sensitivity to classifier errors. This is why we stress-tested the classifier: the difference appears once it is imperfect. Across eight deliberately degraded classifiers (50–98% accuracy; Table R5 in our reply to Reviewer UwDj), EMIS outperforms plug-in guidance with the same classifier at every reduced accuracy, by +1.5 to +21.7 H, mostly through retention (CRA +12 to +29 pp): while the posterior is still uncertain, κ>0\kappa > 0 inflates τt\tau_t, damping confident but wrong classifier outputs instead of triggering repulsion.

In short: with a near-perfect classifier, any reasonable weight performs similarly; the posterior-odds weighting pays off once the classifier is imperfect, which is the realistic case, and its clipping is what protects retained content.

We thank the reviewer again for the careful reading and positive assessment. We hope these clarifications and new results answer the reviewer's questions and reinforce the reviewer's positive view of the paper, and we would be glad to discuss any remaining points. If they do, we would be grateful if the reviewer would consider raising the score.

I thank authors for the rebuttal, which largely addressed my concerns. I will take these into account during the subsequent discussion period.

Reviewer_CHNhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EMIS, a training-free way to stop a diffusion model from generating certain images (instance unlearning) or concepts (style/object erasure). It starts from a known identity: the score of the “safe” distribution equals the base score plus a repulsion term weighted by the posterior odds of the forbidden set. The authors show these odds can be written as the expected membership under the base model’s denoising posterior, so no retain data is needed. To keep the guidance stable they clip the odds, and they prove the clipped guidance is the exact score of a mixture of the safe and base distributions, with a closed-form formula for how much forbidden content remains. The posterior expectation is approximated with a Gaussian around the Tweedie estimate, which gives a noise-dependent temperature. Experiments cover CelebA-HQ instance unlearning and UnlearnCanvas style/object erasure on Stable Diffusion v1.4.

Critical strengths
  1. Clear and unified view. Treating training-free guidance methods as different choices of one repulsion weight is a useful way to organize the field, and it covers both instance and concept unlearning.

  2. Simple, computationally inexpensive method. No fine-tuning, no retain set, no stored checkpoints, and small inference overhead.

  3. Results are competitive with fine-tuning baselines, and the per-class tables are appreciated.

Critical weaknesses

I thank the authors for their contributions. A few critical weaknesses remain. Please address the following weaknesses. A satisfactory response is needed to increase the score. A few suggestions are also given; please incorporate those with proper citations.

  1. The “exact” guarantee does not hold for the method actually run. Theorem 3.2 assumes the true posterior gate π_t and the true forbidden score s_q. In practice: π_t comes from a Gaussian approximation with an isotropic variance schedule, and κ absorbs an unknown Lipschitz factor. s_q for instances is a Gaussian mixture around the target images, which is not the q₀ = m·p₀/π implied by the RBF membership used in the gate. With these approximations, the guided field is no longer the score of ρ^(λ), so the leakage formula is not a guarantee. Suggestion: Replace “exact” and “a-priori safety guarantee” with more careful wording, or bound the error introduced by each approximation

  2. The theory is mostly a re-reading of known results. Eq. (1) is from Kim et al. (2025), as the paper notes. Lemma 3.1 (π_t = E[m(x₀)|x_t]) is a direct application of Bayes’ rule, and plugging a classifier into the Tweedie estimate is common in guidance work (e.g., DPS, universal guidance). Theorem 3.2 is a short algebraic step. Suggestion: State the novelty more modestly and focus on what is new.

  3. The claim that “all existing training-free heuristics approximate the posterior odds” is also stronger than what is shown.

  4. The CelebA-HQ metric is unclear. The base model has Target SSCD exactly 1.00 ± 0.00, which suggests SSCD is measured against the base model’s own output for the same seed rather than against the training image. If so, any change to the image lowers the score, and the metric rewards change rather than identity removal.

  5. The paper calls inference-time steering “unlearning.” The weights still contain the forbidden knowledge, so anyone with the weights, or who removes the guidance, gets it back. The limitations section says this, but the main comparison treats EMIS and weight-editing methods as solving the same problem. Suggestion: Frame the method as safe generation or guided sampling, and state the threat model in the introduction. Or introduce the weak unlearning framework, citing the following works

a. Baumhauer et al., Machine Unlearning: Linear Filtration for Logit-based Classifiers (Machine Learning, 2022) b. Thaker et al., Guardrail Baselines for Unlearning in LLMs (arXiv:2403.03329) c. Shumailov et al., UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI (arXiv:2407.00106) d. Pham et al., Circumventing Concept Erasure Methods for Text-to-Image Generative Models (ICLR 2024) e. Panda & Prathosh, Feature-Aware Similarity Thresholding (arXiv:2312.14895) - IEEE TAI

  1. The paper argues that fine-tuning methods need retain sets, but VDU is a fine-tuning method for diffusion models that needs only the forget subset. This weakens the paper’s framing that being “retain-free” is unique to training-free guidance. Few other works also perform unlearning under data restrictions. Please cite the following works appropriately.

a. Chundawat et al., Zero-Shot Machine Unlearning (arXiv:2201.05629, IEEE TIFS) b. Panda et al., Unlearning in Diffusion Models Under Data Constraints (TMLR 2026) c. Cha et al., Learning to Unlearn: Instance-Wise Unlearning for Pre-trained Classifiers (AAAI 2024) d. Chen et al., Boundary Unlearning (CVPR 2023)

Questions for authors
  1. What exactly is Target SSCD compared against: the training image or the base model’s output?
  2. How close is the observed leakage to the predicted λ/(1+λ) factor on any real dataset?
  3. How was C chosen, and how do results change if it is tuned on held-out styles?
  4. What is the inference time per image on Stable Diffusion, including decoding and the classifier?
  5. How to unlearn a concept when the concept is present in all of the training data? Will unlearning also works in that case?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer CHNh (1/3): W1 and W2

We thank the reviewer for the careful review and concrete suggestions. We are glad the reviewer found that "Treating training-free guidance methods as different choices of one repulsion weight is a useful way to organize the field", that the method needs "No fine-tuning, no retain set, no stored checkpoints", and that "Results are competitive with fine-tuning baselines". We adopt every wording and citation suggestion and add the training-image evaluation the reviewer asked about (part 2/3).

(W1) "Exact" refers to the idealized mixture; following the reviewer's suggestion, the revision now says so explicitly. "Exact" names the ideal case that motivates the method: with the true denoising posterior and forbidden score, clipped-odds guidance is the exact score of the mixture, for instances and concepts alike. A pretrained model only approximates the data score, and Sec. 3.2 estimates the gate; the paper already separates the idealized mixture (Sec. 3.1) from the gate estimator (Sec. 3.2), as is standard in guidance theory (Song et al., 2021; Dhariwal & Nichol, 2021; Chung et al., 2023; Ye et al., 2024). Accordingly, "exact" and "a-priori safety guarantee" now refer only to the idealized mixture of Sec. 3.1, the abstract now reads "With the true posterior gate and forbidden score, our method, Exact Mixture Score (EMIS), is the exact score of an explicit normalized mixture density with a closed-form leakage law; in practice, the pretrained model approximates the score, and EMIS estimates the gate in closed form, replacing heuristic time-window cutoffs with an uncertainty-aware continuous temperature schedule", and the main text points to the gap analysis in the appendix:

  • Gate: Gaussian posterior with a schedule variance that omits the Jacobian term of the exact covariance (App. A.1, B.1).
  • sqs_q for instances: the Gaussian-mixture sqs_q matches q0=mp0/πq_0 = m p_0/\pi to first order in σ02/v0∣t\sigma_0^2 / v_{0|t} over the active repulsion window (App. A.2, Eqs. A4–A7).
  • κ\kappa: κ2\kappa^2 absorbs the Lipschitz factor (App. B.2, Eqs. A13–A14); Table A2 reports the sensitivity to it.
  • Leakage law: verified where every quantity is tractable (1D, Fig. 1e), and an idealized specification in high dimension (Sec. 5); on real data, λ\lambda acts as a monotone control (Table R6).

Following the reviewer's suggestion to "bound the error introduced by each approximation", the revision adds a stability result (new App. D.4): if the gate and forbidden-score errors are at most δ\delta and η\eta uniformly on [0,T][0, T], then KL(ρ0(λ)∥ρ^0(λ))≲(1+λ)2λ4δ2sup⁡tE∥s−sq∥22+η2λ2\mathrm{KL}(\rho_0^{(\lambda)} \Vert \hat\rho_0^{(\lambda)}) \lesssim \frac{(1+\lambda)^2}{\lambda^4}\delta^2 \sup_t \mathbb{E}\Vert s - s_q\Vert_2^2 + \frac{\eta^2}{\lambda^2}, where ρ^0(λ)\hat\rho_0^{(\lambda)} is the law of the sampler under the estimated gate and score. Tighter bounds for each approximation are future work (Sec. 5).

(W2) We now state the novelty more modestly, as the reviewer suggests. We agree the individual ingredients are known: the paper credits Eq. (1) to Kim et al. (2025) (Sec. 2, App. D.1) and Tweedie plug-in guidance to DPS (Chung et al., 2023), and now also cites Universal Guidance (Bansal et al., 2024). The contribution is how they combine into one retain-free framework for instances and concepts, which Sec. 1 now states precisely:

  1. the retain denominator, which Kim et al. (2025) replace with a tuned constant, cancels via p(x0∣xt)p(x_0 \mid x_t) (Lemma 3.1), so no retain data is needed;
  2. odds clipping yields the exact score of the mixture ρ0(λ)\rho_0^{(\lambda)} (Theorem 3.2), with closed-form leakage;
  3. probit variance scaling, τt=σ02+κ2v0∣t\tau_t = \sqrt{\sigma_0^2 + \kappa^2 v_{0|t}}, replaces naive plug-in m(x^0)m(\hat{x}_0) (κ=0\kappa = 0) without backpropagation through the denoiser or the classifier, which DPS and Universal Guidance require.

Point 3 matters in practice: plug-in fires on ambiguous high-noise states (it already reads 0.69 on a safe trajectory at t=880t = 880, vs. 0.96 on the target one; Fig. 2), and with the same classifier, probit scaling improves H from 56.85 to 62.26 (Table A2), and by +1.5 to +21.7 H across eight degraded classifiers (Table R5, reply to UwDj).

AuthorsOpenReview ↗

Reply to Reviewer CHNh (2/3): W3 to W5

(W3) We narrowed the unification claim to score-repulsion methods. In Sec. 1–2 we replaced "all existing training-free unlearning heuristics approximate the continuous-time posterior odds" with:

"Score-repulsion methods that steer along s(xt,t)−sq(xt,t)s(x_t,t) - s_q(x_t,t), such as Negative CFG, SLD, SEGA, SPELL, and Safe Denoiser, correspond to different choices of the posterior-odds weight."

Methods that also modify text embeddings or attention (SEOT, SAFREE, TraSCE) fall outside this view; Table 2 still compares against them.

(W4 / Q1) In Table 1, Target SSCD is measured against the base model's output for the same seed. We now also report SSCD against the training images themselves (Alberti et al. protocol). We thank the reviewer for this question, which led to a more complete evaluation. Following Alberti et al. (2025), we unlearn six CelebA-HQ training images, noise each to t=250t = 250, denoise, and measure SSCD against the training image. SSCD (Pizzi et al., 2022) is a copy-detection descriptor trained to match edited copies: small edits keep the score high, and it drops only when the output is no longer a copy of the reference. This fits instance unlearning, where the goal is to stop reproducing one specific image rather than the person's identity, and Alberti et al. (2025) use the same metric. We also report Unrelated Seed SSCD: generations from 100 other seeds, compared with the base model.

Table R1. Training-image unlearning on CelebA-HQ (Alberti et al. protocol, M=6M = 6).

MethodTarget SSCD ↓Unrelated Seed SSCD ↑
Base DDPM0.8611.000
Parameter fine-tuning
NegGrad0.0470.006
EraseDiff0.2360.022
SISS (Alberti et al., 2025)0.4180.656
Inference-time steering
SLD0.7991.000
Safe Denoiser0.7720.373
SPELL0.7141.000
EMIS (σ0=0.10\sigma_0 = 0.10)0.5780.936

Settings: Table 1 / Table A1 Clean-FID-calibrated settings (Clean-FID ≤ 46.2): SLD (θ=0.12,γ=1\theta = 0.12, \gamma = 1), Safe Denoiser (η=0.20,t∈[0,1000]\eta = 0.20, t \in [0, 1000], active below t=250t = 250), SPELL (r=0.20,γ=1r = 0.20, \gamma = 1), and EMIS (σ0=0.10\sigma_0 = 0.10, the Table 1 M=50M = 50 bandwidth, used because trajectories start at t=250t = 250).

As in Table 1, the trade-off is what matters in Table R1: NegGrad and EraseDiff lower Target SSCD by collapsing unrelated generations (0.006 and 0.022), and SISS, which updates the weights, alters them noticeably (0.656). Among inference-time methods, EMIS reaches the lowest Target SSCD (0.578 vs. 0.714–0.799) while keeping unrelated generations close to the base model (0.936).

Why Table 1 uses full trajectories (Q1). Starting at t=250t = 250 (αˉ250=0.73\bar\alpha_{250} = 0.73) is already in the low-noise regime where x^0\hat{x}_0 is sharp and the low-frequency layout is fixed (as in SDEdit; Meng et al., 2022), leaving guidance little room: the unguided model returns to 0.861, whereas EMIS's uncertainty-aware gate τt\tau_t acts during the early high-noise semantic transition (t∈[600,950]t \in [600, 950], Fig. A1), and pushing uncalibrated barriers wider below t=250t = 250 degrades full-trajectory generation quality (Clean-FID 49.8–56.4 at r=0.30r = 0.30 in Table A1b). Table 1 therefore runs full trajectories from fixed seeds and takes the base model's output for each seed as the target image (hence Base = 1.00; Sec. 4.1 now states this). Arbitrary change does not win there either: constant guidance and EraseDiff reach 0.00 only by collapsing quality (Clean-FID 278.1 / collapse), whereas EMIS reaches 0.40 / 0.38 at base-level Clean-FID (45.0 / 41.3 vs. 45.2).

The paper now says "target images" instead of "identities" (Sec. 4.1, App. C.1, C.4): instance unlearning removes a specific image, not the person's identity, which may appear in other training photos and can still be generated.

(W5) We thank the reviewer for this framing; Sec. 1 now states the threat model within the weak unlearning framework the reviewer suggested. Following the distinction drawn by Baumhauer et al. (2022), Thaker et al. (2024), and Shumailov et al. (2024) between removing knowledge from the weights and suppressing it at the output (weak unlearning; Panda & Prathosh, 2025), Sec. 1 frames EMIS as output-level unlearning for provider-controlled samplers. Open-weight releases still require weight-level deletion (Sec. 5); following Pham et al. (2024) and Shumailov et al. (2024), we present the two levels as complementary. Tables 1–2 separate weight-editing and inference-time methods, and their captions now state that they address different threat models.

AuthorsOpenReview ↗

Reply to Reviewer CHNh (3/3): W6 and Q2 to Q5

(W6) We thank the reviewer for these references; we now credit retain-free fine-tuning. We agree that retain-free unlearning is not exclusive to inference-time guidance. Sec. 1–2 now cite VDU (Panda et al., TMLR 2026), which shows that a diffusion model can be unlearned by fine-tuning on the forget subset alone, together with Chundawat et al. (IEEE TIFS 2023), Cha et al. (AAAI 2024), and Chen et al. (CVPR 2023) for classifiers, and the abstract no longer presents retain-free operation as specific to guidance. EMIS is complementary to these methods: it leaves the weights untouched and stores no checkpoint, which suits requests that change from one query to the next, and it covers memorized instances as well as concepts.

(Q2) The leakage law matches exactly in 1D; on real data, λ\lambda acts as a monotone control (new Table R6). Where every quantity is tractable (1D), observed and predicted odds reduction λ/(1+λ)\lambda/(1+\lambda) agree (Fig. 1e: 0.5% vs. 0.5% at λ=10−2\lambda = 10^{-2}). On high-dimensional data, a faithful comparison needs the true target prior π\pi and an exact membership oracle, neither of which is available for real-world datasets. As the closest test, we swept λ\lambda on CelebA-HQ and UnlearnCanvas (Table R6, reply to NPvz): the fraction of target-producing runs that still produce the target under guidance (target survival) decreases monotonically as λ\lambda decreases (CelebA-HQ: 0.84, 0.51, 0.28, 0.22 for λ\lambda = 3, 1, 0.3, 0.1) and then levels off, at a level that we attribute to how well membership is estimated rather than to λ\lambda: on UnlearnCanvas, with the oracle classifier (the evaluation classifier) as the gate, survival is 0 of 100 at every λ\lambda. This theoretical result nonetheless remains useful in practice. It specifies exactly what the ideal sampler would do, so any gap between observed and predicted leakage can be attributed to estimation error rather than to the objective itself. It also gives λ\lambda a meaning that does not depend on the dataset (the relative reduction in target odds, whatever the unknown prior), so it can be set without knowing the prior.

(Q3) Held-out tuning of CC lands within 0.3 pp of the best fixed value. C=2.5C = 2.5 was selected from the sweep over all 50 styles in Table A2 (H = 91.04 / 90.68 / 88.98 for CC = 2.5 / 3.0 / 3.5). Following the reviewer's suggestion, we tuned it on held-out styles: selecting CC on 10 random styles and testing on the other 40 (1,000 splits) gives H = 90.77 ± 0.97, with C=2.5C = 2.5 chosen in 70% of splits. We added this to App. C.2.

(Q4) SD v1.4 inference time, including decoding and the classifier. We thank the reviewer for asking; we timed all methods under one protocol (50 steps, batch size 1; Table R4 in our reply to UwDj). End-to-end with the off-the-shelf pixel-space UnlearnCanvas ViT, which decodes x^0\hat{x}_0 through the VAE at every active step, EMIS takes 5.93 s/img on a V100 and 1.77 s on an H100 (6.53 / 2.02 s in fp32), vs. 2.92 / 0.91 s for base SD v1.4. Per-step VAE decoding accounts for 73–79% of that overhead (2.37 s on V100, 0.63 s on H100), and we note that the decoder is a learned mapping that can be distilled or folded into the classifier to remove this overhead: without the VAE decode (i.e., the classifier plus the ~7/50 steps where the gate triggers the third U-Net pass), EMIS takes 3.56 s on V100 (+22%) and 1.14 s on H100 (+25%), consistent with the Table 2 footnote and Table 1 (+19% on CelebA-HQ, +5.9% on DiT-XL/2), and faster on V100 than SLD (4.10 s), TraSCE (4.12 s), and SAFREE (4.53 s), which query a third U-Net pass at all 50 steps.

(Q5) We thank the reviewer for this interesting question: a concept "present in all of the training data" is the limiting case π=1\pi = 1, which App. D.3 (Proof of Theorem 3.2) now discusses. If π=Ep0[m]=1\pi = \mathbb{E}_{p_0}[m] = 1, then q0=p0q_0 = p_0 (Eq. 7), so sq=ss_q = s, the guidance vanishes, and the sampler returns p0p_0: no method that only reweights the base distribution can create concept-free samples the model never learned; removing such a concept requires external concept-free data or a replacement target. This is why Theorem 3.2 assumes that the target prior lies strictly between 0 and 1. Whenever a concept-free fraction exists (π<1\pi < 1), the theorem applies unchanged.

We thank the reviewer again for the detailed and constructive review, which has made the paper more precise. We hope these changes and new results address the reviewer's concerns, and we are happy to discuss further. If they do, we would be grateful if the reviewer would consider updating the score.

Comments on the paper

1 comment

We would like to thank all three reviewers for their careful and constructive reviews. They engaged closely with both the theory and the experiments, and the paper is clearly better after iterating on it and addressing their concerns.

We were encouraged that the reviewers appreciated the core idea of the manuscript. CHNh wrote that "Treating training-free guidance methods as different choices of one repulsion weight is a useful way to organize the field, and it covers both instance and concept unlearning." NPvz noted that "Instead of introducing another heuristic guidance schedule, the authors derive the repulsion weight from a probabilistic mixture model. The explicit connection to residual leakage is shown to be very useful." UwDj highlighted that "evaluating the odds does not require the retain dataset is an important contribution", and the results were found "competitive with fine-tuning baselines" (CHNh), obtained "without updating diffusion model weights" (NPvz). According to the subscores, all three reviewers also rate the paper as "generally understandable", measurably "meaningful or useful", and altogether sound with "minor or fixable" concerns; in our revision, we now address each concern.

The reviewers also raised several points addressed in the revision, summarized below (Tables R1–R6 appear in the replies).

  • Theory vs. estimator (CHNh, NPvz). "Exact" refers to the ideal case that motivates this work: with the true denoising posterior and forbidden score, a single clipped-odds rule provides the exact score of an explicit mixture and unifies instance and concept unlearning (Sec. 3.1). In practice, the pretrained model only approximates the score, and the gate is computed by our estimator (Sec. 3.2), as is standard in guidance theory. Following the reviewers, the revision states this distinction from the abstract onward, lists the estimator's approximations (e.g., Eq. A11), brings the gap analysis (App. A.2, B.2) into the main text, and adds a λ\lambda sweep on real data (Table R6). The contributions are unchanged: the retain set drops out of the posterior odds, odds clipping yields the exact mixture score with closed-form leakage, and the variance-aware gate makes the estimator robust to classifier errors.
  • Training-image evaluation (CHNh). Under the Alberti et al. (2025) t=250t = 250 protocol, EMIS reaches the lowest Target SSCD among inference-time methods (0.578 vs. 0.714–0.799) while keeping unrelated generations close to the base model (0.936), without the optimization run and per-request checkpoint of fine-tuning methods (Table R1).
  • Dependence on the classifier (NPvz, UwDj). For concepts, EMIS relies on an auxiliary classifier to specify what to erase (instance unlearning uses no classifier). Guided by the oracle classifier (the evaluation classifier), EMIS reaches 98.25 / 97.39 H on UnlearnCanvas styles / objects (H: harmonic mean of unlearning and retain accuracies): once the target is specified correctly, the weighting erases it almost losslessly, and the remaining gap in Table 2 comes from two independently seeded classifiers disagreeing on a few visually similar styles. Across eight deliberately degraded classifiers (50–98% accuracy), EMIS beats plug-in guidance with the same classifier at every reduced accuracy, by up to 21.7 H (Table R5), mostly via retention (cross-domain retain accuracy +12 to +29 pp).
  • Diffusion Transformers (UwDj). Our theory (Sec. 3.1) is agnostic to how the score is obtained: the denoiser enters only in the estimation step (Sec. 3.2), where it is queried as a black box, with no dependence on U-Net layers, text conditioning, or cross-attention. EMIS therefore carries over to DiT-XL/2 without modification, where it erases the most instances among methods that preserve retained content and is on par with or ahead of simpler weights on classes (Table R3).
  • Positioning (CHNh, UwDj). We now state the threat model up front: following the distinction drawn by Baumhauer et al. (2022), Thaker et al. (2024), and Shumailov et al. (2024) between removing knowledge from the weights and suppressing it at the output, EMIS operates at the output level, for provider-controlled samplers. We also narrow our unification claim to score-repulsion methods, credit retain-free fine-tuning and closed-form editors such as UCE and RECE, and report same-protocol inference times for all methods (Table R4). As UwDj recommended, we re-verified every reference against its published version and revised our statements about other methods.

We are grateful for this feedback, which has made the paper's claims more precise and its empirical case considerably stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.