- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a new inference-guiding concept suppression method that, instead of choosing the repulsion force heuristically, it uses a principled method. Furthermore, the authors show that it is not necessary to avoid the explicit computation of posterior terms, as they are not intractable, and can be computed using the original model's denoising posterior. The proposed method, EMIS, formulates unlearning as posterior-odds re-weighting. To ensure the guidance remains stable, the authors propose to bound the odds and show that this bounded guidance is the exact score of a regularized mixture density. Finally, EMIS works for both concept and data unlearning.
Critical strengths
- Demonstrating that evaluating the odds does not require the retain dataset is an important contribution
- The framework works for both concept and data unlearning
Critical weaknesses
- Multiple facts stated in the paper and information reported are false or contain inaccuracies. For instance, not all parameter fine-tuning methods are computationally intensive and require synthetic retain sets to prevent performance collapse. For instance, all closed-form methods (like UCE and RECE) are computationally cheap and require less VRAM than some inference-time steering methods. Another example: wrong paper title in the reference Kirchhof et al., 2025. The paper should be significantly revised for fact checking and claims restatements.
- The paper does not demonstrate the usage of the proposed method for Diffusion Transformer (DiT) architectures, which are now the state-of-the-art. This significantly limits the applicability of the framework
Questions for authors
- Can you show the computational time of the proposed algorithm compared to other inference-guiding techniques?
- Since the unlearning method relies on a classifier, can you provide an ablation study to what happens when using an under-trained classifier? For instance, what happens when using a classifier that achieves 50-60-70-80-90% accuracy? Because training a classifier in a controlled benchmark like UnlearnCanvas is rather simple, but in real-world scenarios we can expect to not have such a well-trained classifier.
4 comments
Reply to Reviewer UwDj (2/2): Q1 and Q2
(Q1) Inference time: small overhead without a decoder; on SD v1.4 the cost is decoding for a pixel-space classifier. We thank the reviewer for asking for this comparison; we timed all methods under one protocol. CelebA-HQ (Table 1, V100): base 2.07 s, SLD / SPELL 2.10 s, Safe Denoiser 2.14 s, EMIS 2.46 s. DiT-XL/2: EMIS 1.33 s vs. 1.25 s base and 1.27–1.32 s for others.
Table R4. SD v1.4 inference time, one protocol (50 steps, batch 1, including VAE decoding; fp16 EMIS classifier).
| s/img | Base | Neg. prompt | SEOT | SLD | TraSCE | SAFREE | EMIS |
|---|---|---|---|---|---|---|---|
| V100 | 2.92 | 3.14 | 3.17 | 4.10 | 4.12 | 4.53 | 5.93 |
| H100 | 0.91 | 0.92 | 0.92 | 1.00 | 0.99 | 1.00 | 1.77 |
With the off-the-shelf pixel-space ViT, 73–79% of the SD v1.4 overhead comes from decoding through the VAE at each active step (2.37 s on V100, 0.63 s on H100). Without per-step VAE decoding (the classifier plus the ~7/50 steps where the gate triggers the third U-Net pass), EMIS takes 3.56 s on V100 (+22%) and 1.14 s on H100 (+25%), consistent with the Table 2 footnote and with Table 1 (+19% on CelebA-HQ, +5.9% on DiT-XL/2), and faster on V100 than SLD (4.10 s), TraSCE (4.12 s), and SAFREE (4.53 s), which run a third U-Net pass at all 50 steps. We note that the decoder is a learned mapping that can be distilled or folded into the classifier to remove this overhead; even with pixel-space decoding, the extra cost on an H100 is under 0.9 s/img, with no fine-tuning or stored checkpoints (App. E.4).
(Q2) Instances need no classifier; for concepts, EMIS degrades more gracefully than plug-in guidance at every reduced accuracy. We thank the reviewer for suggesting this ablation, which is one of the most informative additions to the paper. For instance unlearning, is an analytic kernel (Eq. 8). For concepts, we fine-tuned ImageNet-21k ViT-L classifiers on UnlearnCanvas styles and stopped early: (i) standard LR, few steps; (ii) 10× lower LR, checkpoints at 50–90% validation accuracy. To isolate the variance-aware temperature without confounding variables, we compare EMIS () on 10 styles against plug-in odds () using the exact same classifier, clipped-odds formula , and cap.
Table R5. Classifier-accuracy sweep on UnlearnCanvas styles.
| Recipe | Acc. | EMIS H | Plug-in H |
|---|---|---|---|
| (i) | 97.6% | 66.5 | 49.2 |
| (i) | 80.3% | 58.8 | 38.4 |
| (i) | 52.0% | 51.3 | 29.6 |
| (ii) | 89.95% | 44.1 | 36.8 |
| (ii) | 80.05% | 42.9 | 33.3 |
| (ii) | 69.80% | 36.3 | 30.3 |
| (ii) | 60.05% | 37.6 | 33.3 |
| (ii) | 50.05% | 35.6 | 34.1 |
Performance drops with accuracy, since the classifier defines what is erased, but EMIS beats plug-in guidance at every reduced accuracy by +1.5 to +21.7 H, mainly via retention (CRA +12 to +29 pp): while the posterior is uncertain, inflates , damping confident but wrong outputs. We attribute the lower H of recipe (ii) at equal accuracy to the flatter, more under-confident logits that its 10× smaller learning rate produces (at 80% accuracy: mean max-probability 0.49 and ECE 30.7%, vs. 0.64 and 15.9%): with the gate offset set for the converged classifier, the gate fires on nearly all retain images, which lowers retention (IRA 20–27% vs. 32–49%). We added this ablation (App. E.3) and discuss classifier quality as a limitation.
We thank the reviewer again; these comments made the paper more precise and its evaluation broader. We hope these updates and new experiments address the reviewer's concerns, and we are happy to discuss further. If they do, we would be grateful if the reviewer would consider updating the score.
I thank the authors for their responses. The paper now surely improved from my point of view, but I still will wait for the other reviewers to comment on their side, as I am still unsure about the importance and impact of the overall contribution, mainly due to the dependency on the classifier.
On the dependence on the classifier
We thank the reviewer for the follow-up and are glad the revision improved the paper in their view. We would like to address the remaining concern, the dependence on the classifier, directly.
The classifier specifies what to unlearn, which is part of the task, not an extra assumption. Every unlearning method must be told what to forget: the forget set, or membership function, is part of the problem definition. Fine-tuning methods receive it as forget images or prompts and optimize the generator once per concept; prompt-based inference-time methods receive it through the text encoder. EMIS takes the same information as an explicit membership function. Without it, the unlearning problem is undefined for any method. Once the target is specified correctly, EMIS erases it almost losslessly: with the evaluation classifier as an oracle, it reaches 98.25 / 97.39 H on UnlearnCanvas styles / objects (Table 2).
Instance unlearning uses no classifier. There, membership is an analytic kernel around the target image (Eq. 8). Without any classifier, EMIS matches or improves on the fine-tuning methods SISS and NegGrad in Target SSCD on CelebA-HQ (0.40 vs. 0.51 / 0.46 at M = 6; 0.38 vs. 0.39 / 0.40 at M = 50) while keeping Clean-FID at or below the base model's (45.0 / 41.3 vs. 45.2), without the 2.7–3.1 h of optimization and 0.45 GB checkpoint they need per target cohort (Sec. 4.1, Table 1). On DiT-XL/2, it erases 80% of targets vs. at most 40% for the other methods that preserve retained content (Table R3, App. Table A10).
For concepts, the classifier is cheap, and EMIS reduces the dependence on its quality.
- One standard image classifier covers all 50 styles; it is trained once, separately from the generator.
- In Table 2, the inference-time methods that rely on the prompt alone erase 23.25–49.85% of objects; EMIS with an independently seeded classifier erases 92.80% (H = 93.92), on par with the strongest fine-tuning methods.
- With deliberately degraded classifiers (50–98% accuracy), EMIS beats plug-in guidance with the same classifier at every accuracy level tested, by +1.5 to +21.7 H (Table R5, App. Table A7). Better estimators of the membership function (what concepts to forget) therefore translate directly into better unlearning, with no retraining of the generator.
The core results do not depend on how membership is specified. Lemma 3.1 (evaluating the odds does not require the retain set) and Theorem 3.2 (exact mixture score with closed-form leakage) hold for any membership function, kernel or classifier.
Together, this lets a provider serve changing deletion requests, from specific training images to entire concepts, with one frozen model, no retain data, and no per-request checkpoints. We hope this resolves the reviewer's remaining concern; if anything is still unclear, we are happy to provide more details.
Reply to Reviewer UwDj (1/2): W1 and W2
We thank the reviewer for the review, for the attention to accuracy, and for the reference pointed out. We are glad the reviewer found that "Demonstrating that evaluating the odds does not require the retain dataset is an important contribution" and that "The framework works for both concept and data unlearning". These comments helped us make the related-work discussion more precise and broaden the evaluation (DiT, timing, classifier quality); we address each point below.
(W1) We thank the reviewer for pointing us to closed-form editors and to the published title of Kirchhof et al.; both are now reflected in the paper. Our statements on compute and synthetic retain sets were written for iterative fine-tuning for instance unlearning (e.g., SISS, EraseDiff). Following the reviewer's comment, this scope is now explicit (abstract, Sec. 1–2), and Sec. 1–2 discuss closed-form editors separately: as the reviewer notes, UCE (Gandikota et al., 2024) and RECE (Gong et al., 2024) are "computationally cheap and require less VRAM than some inference-time steering methods", and they add no cost at inference. We present EMIS as complementary to them: it needs no text conditioning, so it also applies to unconditional instance unlearning, and it serves every request from one unmodified set of weights. For Kirchhof et al. (2025), our entry carried the title of the arXiv version (arXiv:2410.06025); it now carries the ICML title, "Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency." Following the reviewer's recommendation, we also re-verified every reference against its published version, updating 5 entries, and revised our statements about other methods accordingly; the requested timing table is in Table R4 (part 2/2).
(W2) We thank the reviewer for this suggestion: EMIS applies to DiTs without modification; new DiT-XL/2 results. As the reviewer notes, DiT architectures "are now the state-of-the-art". Our theory is agnostic to how the score is obtained, and the estimator queries the denoiser as a black box (Alg. 1). The benchmarks in the paper are tied to specific U-Net checkpoints (CelebA-HQ DDPM in Alberti et al., 2025; the stylized SD v1.4 checkpoint and evaluation judges in UnlearnCanvas, Zhang et al., 2024), and for text-to-image models Carlini et al. (2023) catalog memorized targets for U-Net checkpoints (SD v1.4) with no reference set for DiT checkpoints. We therefore built a compact protocol on class-conditional ImageNet DiT-XL/2 (256×256) covering both settings, re-running training-free baselines under identical conditions; because EMIS only queries the denoiser, the same procedure carries over directly to text-to-image transformers.
Table R3. DiT-XL/2 instance unlearning (10 targets; kernel membership, no classifier; erased = SSCD < 0.5).
Classes (10 ImageNet classes paired with similar retain classes; same ResNet-50 guide for all; ViT-B/16 judge): EMIS reaches UA 95.5 / RA 94.5 / H 95.0 with the best retain SSCD (0.853, vs. 0.812 and 0.815), ahead of a constant weight (94.7 H) and plug-in (93.4 H).
As in the main benchmarks, the trade-off is what matters: constant guidance erases everything by destroying retained content, while SLD and SPELL preserve content but erase little. EMIS gives the strongest erasure among the methods that preserve retained content (80% of targets, vs. at most 40% for the others).