acceptodds
Under review as a conference paper at ICLR 2027

PruneForget: Joint Unlearning and Pruning of Vision Models

Abstract

Machine unlearning and model pruning are increasingly *coupled* in the real world. Models must support unlearning requests, e.g., for safety concerns, *while also* meeting requirements in latency and memory budget. Until recently, existing works have studied each aspect as an independent problem, i.e., running unlearning and pruning sequentially. In this work, we show that unlearning and pruning are naturally aligned and should be solved jointly to be aware of each other. Intuitively, parameters that encode information of the unlearned samples are natural pruning targets, as unlearning and pruning both call for the “deletion” of such parameters. We propose PruneForget, a method that uses the unlearn set as a guide for pruning, so that unlearning and pruning mutually benefit each other. Extensive experiments on image classifiers and generative models show that PruneForget removes the influence of the unlearned samples while producing a more compact model with reduced inference cost.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine unlearning, Model pruning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_e2ccOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PruneForget, which uses the unlearn set to guide a three-stage structured pruning pipeline. A per-group sensitivity score is computed on the pretrained model as the ratio of forget-set to retain-set gradient norms. It weights the sparsity regularizer during sparse training, rescales importance scores so that forget-sensitive groups are pruned first, and post-pruning fine-tuning is replaced with unlearning fine-tuning. Experiments on ResNet-18, Swin-T, DDPM, and DiT report the smallest gap to a retrain-then-prune oracle among the baselines, at lower runtime than the baselines.

Critical strengths
  1. The setting of jointly unlearning and structurally pruning a model, producing an actually smaller and faster network, is practically relevant vision models.

  2. The method is simple and reasonable; it is applied to three pruning methods across CNNs, ViTs, and diffusion models.

  3. The experiments cover both classification and generation with ablations of each component, and uniform pruning ensures identical pruned architectures across methods for a fair efficiency comparison.

Critical weaknesses
  1. Novelty is limited and related work is missing. Ranking parameters by forget-vs-retain importance and then masking or pruning them is a well-studied idea in unlearning. Examples include SSD [1], Scissorhands [2], selective pruning [3], ConceptPrune [4], and Forget-It-All [5]. Only [1] is cited, and only in passing. Table 5 also shows that a Fisher score (0.10) or an unlearn-only score (0.07) does about as well as the proposed ratio (0.06), so the specific score design does not seem to matter much.

References

[1] J. Foster, S. Schoepf, A. Brintrup. Fast Machine Unlearning without Retraining through Selective Synaptic Dampening. AAAI 2024.

[2] J. Wu, M. Harandi. Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks. ECCV 2024.

[3] N. Pochinkov, N. Schoots. Dissecting Language Models: Machine Unlearning via Selective Pruning. arXiv:2403.01267, 2024.

[4] R. Chavhan, D. Li, T. Hospedales. ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning. ICLR 2025.

[5] K. Deng et al. Forget-It-All: Multi-Concept Machine Unlearning via Concept-Aware Neuron Masking. ICML 2026.

  1. Most of the gain comes from Stage 3, which is standard unlearning fine-tuning. In Table 4, Stage 2 (the main contribution) only moves Avg Δ from 0.41 to 0.34. Adding Stage 3, which is SFRon-style fine-tuning per Appendix A2, brings it to 0.06. A Stage-3-only ablation, i.e., standard pruning followed by unlearning fine-tuning, is needed to show that Stages 1–2 actually help.

  2. The improvements are mostly within noise. In Table 1 the ΔUA differences (e.g., 0.03 vs. 0.16) are smaller than the reported std (0.3–0.6). In Table 2 and Table 9, Pretrain→nSP performs no unlearning and still essentially matches the oracle (Avg Δ 0.15 vs. 0.14; UA 97.94 vs. 98.39), so these benchmarks cannot separate the methods. PruneForget's D_KL to the oracle is also consistently higher than this no-unlearning baseline (Tables 1, 3, A14), which works against the claim that it better approximates the oracle.

  3. Hyperparameters are tuned against the oracle. f_base is chosen by the gap to the retrained oracle (Tables A3/A4). In practice that oracle is exactly what the method is meant to avoid computing.

Questions for authors
  1. How does the sensitivity score differ from the forget/retain importance ratios used in SSD and selective pruning? Can you compare against SSD, Scissorhands, ConceptPrune, or Forget-It-All?

  2. What do you get with Stage 3 alone, i.e., standard pruning followed by unlearning fine-tuning? Does adding Stages 1–2 give a significant improvement over that?

  3. How would you choose f_base without access to the retrained oracle? How were the baselines' hyperparameters tuned?

3 comments

We thank the reviewer for the detailed feedback and the references.

Q1. Comparison with unlearning-by-pruning methods (SSD, Scissorhands, selective pruning, ConceptPrune, Forget-It-All)?

Thank you for providing these related works. We did not claim that ranking channels by forget-vs-retain importance is novel. We claim that it should guide structured pruning across all stages. The related works didn't discuss these papers because they address a different problem from ours. We aim to produce a model that is both unlearned and structurally compressed to a target ratio. In contrast, these methods prune parameters to unlearn, so they modify only a tiny fraction of weights (e.g., 0.12% in ConceptPrune, <0.3% in Forget-It-All), and their pruning is unstructured, so the inference cost is unchanged. We agree that citing these papers and discussing the differences will make our work more complete. We thank the reviewer for the comment, and would discuss them in related works.

We replace our S with the score of each method and show the comparison in Table R3; our gradient-norm ratio still gives the lowest Avg Δ. We will include this comparison in the paper.

Table R3. Comparison of sensitivity score designs (CIFAR-10, ResNet-18, 50% random unlearning).

Design of SΔUA ↓ΔRA ↓ΔTA ↓Avg Δ ↓
SSD (Fisher)0.15 (7.74)0.00 (100.00)0.15 (92.05)0.10
Scissorhands: |θ·∇L|0.37 (7.23)0.00 (100.00)0.13 (92.33)0.17
Selective pruning: mean |act|0.48 (8.08)0.00 (100.00)0.44 (91.77)0.31
ConceptPrune: |W|·‖X‖0.49 (8.09)0.00 (100.00)0.62 (91.58)0.37
Forget-It-All: |W|·‖X‖·cos0.37 (7.97)0.00 (100.00)0.65 (91.55)0.34
Ours (gradient-norm ratio)0.16 (7.44)0.00 (100.00)0.02 (92.22)0.06

Q2. Stage 3-only ablation. We agree that Tab. 4 is cumulative and does not isolate Stage 3. Table R4 therefore reports each stage on its own. Each stage alone already improves over Pretrain→3SP (Avg Δ 0.78). Combining all three stages gives the smallest TA gap and the lowest Avg Δ (0.06). Stage 3 alone is therefore a strong component but not the only source of the gain.

Table R4. Stage ablation (CIFAR-10, ResNet-18, 50% random unlearning).

Stage 1Stage 2Stage 3ΔUA ↓ΔRA ↓ΔTA ↓Avg Δ ↓
1.54 (6.06)0.00 (100.00)0.81 (93.01)0.78
✓0.93 (6.67)0.00 (100.00)0.29 (92.50)0.41
✓0.13 (7.47)0.00 (100.00)0.16 (92.05)0.10
✓0.19 (7.79)0.00 (100.00)0.11 (92.09)0.10
✓✓✓0.16 (7.44)0.00 (100.00)0.02 (92.22)0.06

Q3 (a). Improvements are within noise. We thank the reviewer for the question. In hindsight, the way we presented the results makes it hard to compare methods. We now summarize them and explain where PruneForget is and is not separated from the baselines.

First, the three gaps (ΔUA, ΔRA, ΔTA) trade off against one another, so the meaningful question is which method is close to the oracle on all metrics at once. This is what Avg Δ in our tables measures. Table R5 collects it for every setting.

Table R5. Avg Δ to the oracle (lower is better). Bold = best, italic = second best.

MethodTab. 1 CIFAR-10 10%Tab. 1 CIFAR-10 50%Tab. 2 CIFAR-10 1-classTab. 2 CIFAR-10 5-classTab. 3 TinyImageNet 10%Tab. 3 TinyImageNet 50%Tab. 9 DDPM*Tab. A9 CIFAR-100 50%
Pretrain→nSP0.430.780.150.070.590.880.381.57
nSP→SFRon6.703.191.702.334.243.510.312.50
SFRon→nSP0.070.160.230.300.621.010.391.20
ℓ1-MU→2SP5.034.483.121.6432.6937.760.486.67
Un-pruning1.141.910.520.554.944.89163.503.45
Bilevel7.255.6936.9211.136.177.922.645.04
LLM-Eraser13.8111.6813.036.813.864.835.2520.76
PruneForget0.010.060.140.070.540.430.200.72

*Mean of ΔUA and ΔFID. This mixes an accuracy and a distance, so it is only a rough summary.

  • PruneForget has the lowest Avg Δ in all 8 settings.
  • The second-best method changes with the setting (SFRon→nSP in 3, Pretrain→nSP in 3, nSP→SFRon on DDPM). PruneForget is the only method that is consistently best.

Second, we agree that several single-metric differences are within noise. We use a conservative criterion: two results are separated only if their means differ by more than the sum of their standard deviations (non-overlapping ±1 std intervals over the 10 trials). Every case in the main tables where a baseline has a lower gap than ours on a single metric is a tie under this criterion:

Table R6. Cases where a baseline has a lower single-metric gap than PruneForget (raw values).

SettingMetricBaselinePruneForgetGap vs. σ₁+σ₂
Tab. 1, CIFAR-10 50%UASFRon→3SP: 7.45 ± 0.217.44 ± 0.320.01 < 0.53 → tie
Tab. 3, TinyImageNet 50%RAPretrain→2SP: 99.32 ± 0.0499.35 ± 0.030.03 < 0.07 → tie
Tab. 3, TinyImageNet 50%RASFRon→2SP: 99.34 ± 0.0599.35 ± 0.030.01 < 0.08 → tie
Tab. 9, DDPMFIDnSP→SFRon: 16.92 ± 0.7016.92 ± 0.760.00 < 1.46 → tie
Tab. 9, DDPMFIDSFRon→nSP: 16.96 ± 0.7116.92 ± 0.760.04 < 1.47 → tie

These results show that PruneForget consistently achieves the lowest Avg Δ, and on each individual gap (ΔUA, ΔRA, ΔTA) it is either the best or tied with prior work.

Q3 (b). Pretrain→nSP matches the oracle. We agree that class-wise unlearning on CIFAR-10 and DDPM does not separate Pretrain→nSP from the oracle. Below are the results in a harder random-unlearning settings, where Pretrain→nSP under-forgets (Table R7).

Table R7. UA of the oracle, Pretrain→nSP and PruneForget. Pretrain→nSP under-forgets (is separated from the oracle) in these settings.

SettingOraclePretrain→nSPPruneForgetSeparated from the oracle?
Tab. 1, CIFAR-10 50%7.60 ± 0.226.06 ± 0.177.44 ± 0.32Pretrain→nSP: yes (gap 1.54 > 0.39). Ours: no
Tab. A9, CIFAR-100 50%32.48 ± 0.3129.27 ± 0.7133.94 ± 1.24Pretrain→nSP: yes (gap 3.22 > 1.02). Ours: no
Tab. 3, TinyImageNet 50%22.61 ± 0.1720.30 ± 0.1321.64 ± 0.12Both yes; ours is closer

Q3 (c). Pretrain→nSP has a lower D_KL. We agree that Pretrain→nSP has a slightly lower D_KL in Tables 1, 3 and A14. However, the same tables show that it does not forget: in every one of these settings, its UA stays below the oracle's, by up to 2.37 points (Table R8). D_KL therefore cannot be used as the sole metric to determine whether a model has unlearned effectively. Moreover, among methods that actually unlearn, ours is best or tied: in Tab. 1, SFRon→3SP has 0.22 / 0.27 and 3SP→SFRon has 0.24 / 0.27, all others are ≥ 0.27, and ours is 0.22 / 0.25.

Table R8. Pretrain→nSP fails to forget (low UA) in Tables 1, 3 and A14.

SettingOracle UAPretrain→nSP ΔUA (UA)PruneForget ΔUA (UA)
Tab. 1, CIFAR-10 10%6.030.89 (5.15)0.03 (6.01)
Tab. 1, CIFAR-10 50%7.601.54 (6.06)0.16 (7.44)
Tab. 3, TinyImageNet 10%19.881.64 (18.24)1.50 (18.38)
Tab. 3, TinyImageNet 50%22.612.32 (20.30)0.98 (21.64)
Tab. A14, TinyImageNet (global)22.952.37 (20.58)1.16 (21.79)

Q4 (a). How to choose f_base without the oracle?

We follow SFRon, which also sets a different f_base per architecture. We agree that the oracle cannot be used to select f_base in practice. Since f_base only controls the strength of unlearning (i.e., a larger f_base means weaker unlearning), its effect is directly observable in the training curves, and a practitioner can monitor the unlearning and retain losses to adjust it.

Here we provide a simple heuristic that tunes f_base dynamically from these losses. It already gives results close to the oracle-selected f_base, within 0.01–0.06 Avg Δ in all four settings (Table R9).

f_base starts at 100 (other start values in Table R10) and is updated once per epoch from training-time losses only, by a factor of 1.25 in either direction:

  • If the retain loss rises by more than 0.1 above its running minimum, f_base is increased (less gradient ascent).
  • Random unlearning: if the forget loss is more than 10% below the held-out loss, f_base is decreased (more gradient ascent); if it is more than 10% above, f_base is increased.
  • Class-wise unlearning: if the forget-set accuracy is above 1%, f_base is decreased; otherwise it is increased.

Table R9. Oracle-free f_base vs. the best fixed f_base selected with the oracle (Avg Δ).

SettingOracle-free f_baseBest fixed f_base (oracle-selected)
CIFAR-10, random 10%0.070.01
CIFAR-10, random 50%0.100.06
CIFAR-10, 1 class0.150.14
CIFAR-10, 5 classes0.120.07

One may ask whether the heuristic simply replaces f_base with a new hyperparameter, its start value. Table R10 shows that it does not. We start the rule from values that span a 5× range (20 to 100). Avg Δ stays between 0.10 and 0.14, and every start value yields a lower Avg Δ than every baseline in Tab. 1, the best of which is SFRon→nSP with 0.16.

Table R10. Oracle-free f_base with different start values (CIFAR-10, ResNet-18, 50% random unlearning).

Initial f_baseΔUA ↓ΔRA ↓ΔTA ↓Avg Δ ↓
200.41 (7.18)0.00 (100.00)0.01 (92.21)0.14
500.25 (7.35)0.00 (100.00)0.06 (92.14)0.10
800.37 (7.23)0.00 (100.00)0.05 (92.26)0.14
1000.28 (7.31)0.00 (100.00)0.00 (92.20)0.10

Q4 (b). How were baselines tuned?

Note that f_base is the only hyperparameter we tune; all others are fixed or derived. For every baseline, we mostly follow the original settings and adapt them slightly to make the runtimes comparable and obtain reasonable results; see Appendix A2, "Details for baselines". Specifically, we pick one hyperparameter per method, tune it over 3–5 values, and select the best for comparison. Tables R11–R13 show the hyperparameter search for LLM-Eraser, Un-pruning, and SFRon. We will add the hyperparameter search process to the paper.

Table R11. LLM-Eraser with different weights α of the retain term (CIFAR-10, ResNet-18, 50% random unlearning).

αΔUA ↓ΔRA ↓ΔTA ↓Avg Δ ↓
0.0110.60 (18.20)14.27 (85.73)10.81 (81.39)11.89
0.110.46 (18.05)14.00 (85.99)10.59 (81.61)11.68
1.010.97 (18.57)14.71 (85.29)11.17 (81.04)12.28

Table R12. Un-pruning with different fine-tuning learning rates (CIFAR-10, ResNet-18, 50% random unlearning).

Learning rateΔUA ↓ΔRA ↓ΔTA ↓Avg Δ ↓
0.0016.79 (14.39)4.49 (95.51)7.09 (85.12)6.12
0.0052.88 (10.48)0.12 (99.88)3.16 (89.04)2.05
0.012.72 (10.31)0.07 (99.93)2.94 (89.26)1.91

Table R13. SFRon→3SP with different forget frequencies (CIFAR-10, ResNet-18, 50% random unlearning).

f_baseΔUA ↓ΔRA ↓ΔTA ↓Avg Δ ↓
13.46 (11.06)0.10 (99.90)3.52 (88.68)2.36
22.34 (9.93)0.05 (99.95)2.55 (89.65)1.65
50.15 (7.45)0.00 (100.00)0.34 (91.87)0.16
100.43 (7.17)0.00 (100.00)0.17 (92.38)0.20
Reviewer_prHJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper deals with solving the two tasks of unlearning and pruning simultaneously.

  • The idea behind this is that the unlearning set can hint what parameter groups to prune.

  • They propose three steps that incorporate the unlearning set within the three typical steps in model pruning: (1) Unlearn-aware sparsification, (2) Sensitivity-guided pruning, and (3) Post-pruning unlearning.

  • The three stages use a sensitivity score to bind the pruning and unlearning.

  • The authors compare to relevant baselines, and used CIFAR-10 and TinyImageNet benchmarks to assess the method on classification tasks and CIFAR-10 and ImageNet for generative tasks.

Critical strengths
  • The approach sounds and the motivation is clear. The usage of senisitivity score is agnostic to the pruning method used.
  • Results show improvements against most of the baselines and is the closest to the oracle in most of the cases.
Critical weaknesses
  • LLM-Eraser already proposed this connection (as the author mentioned in the introduction). However, in the abstract there is a strong claim: “In this work, we show that unlearning and pruning are naturally aligned and should be solved jointly to be aware of each other.”, I would soften this claim in the abstract. Also, in the caption for Figure 1: “Sequential baselines (Unlearn→Prune or Prune→Unlearn) treat unlearning and pruning as independent tasks. Our proposed PruneForget integrates unlearning awareness into the three stages “. I would avoid stating that the conceptual idea of learning pruning and unlearning simultaneously is novel. I understand that the paper deals with vision-models, but still, the general idea is not new, and I would make it clearer in the paper text.
  • The authors claim “extensive unlearning experiments across image classification and generation”, however the approach are tested mainly on CIFAR-10, TinyImageNet and ImageNet (for generative model unlearning). This is a thin experimental setting with lack of benchmark diversity to my opinion.
  • When comparing to baselines, I would expect focusing on explaining where and why the PruneForget beats close-related approaches, like LLMEraser. Specifically, how did the results obtained for the other baselines? And whether better hyper-parameter search could lead to higher results of the LLMEraser? The authors mentioned: “we find that a direct application of LLM-Eraser to vision models does not perform well.” - but why? Which exact component makes the PruneForget better than LLMEraser?
  • Besides the metric results on the final test, I would expect to deeper analysis for understanding the advantages of the proposed approach via intermediate visualization of the components. For example, how the regularization term R(\theta) differs between the sparse training and in the proposed method, or deeper look on the distribution of the scores S_l, and how it changes with different examples (images in our cases). How different the score distribution from uniform in typical cases, for example, when we are required to unlearn complete classes or specific samples across different classes?
  • Another thing that bother me is that I don’t see anything special in terms of the vision modality - I would expect to see what make the method unique for images, and if it is not unique to images, why not to compare to LLMEraser on text tasks as well?
Questions for authors
  • What do you think is the main gap in the method between LLMEraser and the proposed approach that leads to the gap in accuracy?
6 comments

We thank the reviewer for the feedback.

Q1. LLM-Eraser already connects unlearning and pruning. We apologize for the misunderstanding. We do not claim that joint unlearning and pruning is new (cf. LLM-Eraser, Un-pruning, Bilevel). Our contribution is to make all three pruning stages unlearning-aware (prior work covers only fine-tuning, or pruning plus fine-tuning; Tab. A1) for structured pruning of vision models, with a sensitivity score that works with any importance criterion. We will clarify the abstract and introduction accordingly.

Q2. Limited datasets for an "extensive" claim. We follow SFRon and evaluate on the same 4 datasets (CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet), and our use of "extensive" follows SFRon's paper. In any case, we will soften this claim as suggested.

Q3. Which component makes PruneForget better than LLM-Eraser?

We thank the reviewer for this suggestion. To find out which component makes PruneForget better than LLM-Eraser, we swap the corresponding components of the two methods. We consider (i) the score used to select channels and (ii) the fine-tuning loss. We implement two variants:

  • LLM-Eraser score + PruneForget loss: we replace our S in Stages 1 and 2 with LLM-Eraser's importance score and keep our Stage 3.
  • PruneForget score + LLM-Eraser loss: we keep our Stages 1 and 2 and replace the Stage 3 loss with LLM-Eraser's fine-tuning loss.

The results are in Table R1. Both components matter, and the fine-tuning loss matters most. This is consistent with our view that unlearning-aware designs need to be specific to the modality: LLM-Eraser's loss and score were designed for removing a genre of knowledge from an LLM, whereas ours target forgetting a set of images in vision models.

Table R1. Component swap between LLM-Eraser and PruneForget (CIFAR-10, ResNet-18, 50% random unlearning).

PipelineΔUA ↓ΔRA ↓ΔTA ↓Avg Δ ↓
LLM-Eraser10.46 (18.05)14.00 (85.99)10.59 (81.61)11.68
LLM-Eraser score + PruneForget loss1.33 (8.92)0.00 (100.00)1.45 (90.75)0.93
PruneForget score + LLM-Eraser loss3.81 (11.41)0.53 (99.47)4.13 (88.07)2.83
PruneForget0.16 (7.44)0.00 (100.00)0.02 (92.22)0.06

Q4. Intermediate visualizations of the regularization terms and sensitivity score. We thank the reviewer for the suggestion and will add two analyses to the paper: (i) how the regularization term R(θ) differs between sparse training and our method, and (ii) the distribution of the scores S under random and class-wise unlearning.

For (i), Table R2 tracks the norm of forget-dominant (top 20% of S) and retain-dominant (bottom 20% of S) channels during Stage 1, relative to the median channel norm of the same layer. Under standard sparse training, the two groups keep similar norms and are pruned at almost the same rate. Under our method, forget-dominant channels shrink to 0.456× the layer median while retain-dominant channels grow to 1.921×. Stage 2 then removes 93% of the forget-dominant channels but only 12% of the retain-dominant ones.

For (ii), we examine the distribution of S for random and class-wise unlearning. A narrow distribution of S means that all channels contribute similarly to the retain and forget sets, while a broad distribution means that certain channels contribute significantly more to the forget set than others. For random unlearning, S is narrow: its 99th percentile is 4.12. The forget samples come from the same distribution as the retain samples, so few channels are specific to them. For class-wise unlearning, S has a heavy tail (99th percentile 8.30), so the score isolates a small set of class-specific channels.

Table R2. Norm of forget- and retain-dominant channels during Stage 1, and the fraction of each group pruned in Stage 2 (CIFAR-10, ResNet-18, 50% random unlearning).

MethodChannelsEpoch 1Epoch 25Epoch 50Epoch 75Epoch 100Pruned in Stage 2
Sparse trainingforget-dominant0.9650.9700.9831.0061.01852.3%
Sparse trainingretain-dominant0.9740.9720.9851.0121.02551.9%
PruneForgetforget-dominant0.9010.7830.6570.5190.45693.2%
PruneForgetretain-dominant1.0201.1431.3221.6801.92112.0%

Q5. Uniqueness for the vision modality. While many method seems general, it is non-trivial to apply them across different modality. For example LLM-Eraser is designed for language models and when transferred to other modalities the performance drops. Our method targets vision models in several ways: (a) The pruning methods we apply are designed for vision models (e.g., Diff-Pruning for diffusion models), and (b) the unlearning loss is modality-specific (KL-to-uniform for classifiers; noise MSE for diffusion). These design choices are specific to the vision modality and do not apply to language tasks, which have different loss functions and archiectures.

Reviewer_prHJOpenReview ↗

Comment on the Authors Response

I thank the authors for their detailed response.

  1. Experimental setup: I agree with the authors. The paper follows the setting of prior work, so the four datasets (CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet) are a fair choice.
  2. Comparison to LLM-Eraser: I appreciate the additional ablation, which is meaningful and necessary. From the paper and the rebuttal, I understand that the proposed approach achieves better results. However, neither explains why it outperforms LLM-Eraser. Taking this together with the response to Q4 (intermediate visualizations of the regularization terms and the sensitivity score), I consider this concern only partially addressed.
  3. Specificity to the vision modality: I agree that adapting a general method to another modality is not trivial. However, I could not find any analysis or explanation in the paper of why these specific losses are needed for the vision modality. In my view, an academic paper needs more than a proposed method and final comparison results. The authors should explain why their approach handles the vision modality better.
Reviewer_gUjWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PruneForget, a machine unlearning method. The authors argue that machine unlearning and model pruning have similar purposes and therefore combine machine unlearning with ideas from the model pruning literature, introducing an unlearning-aware three-stage procedure consisting of model sparsification, model pruning, and subsequent fine-tuning. The key contribution is making these steps unlearning-aware by leveraging access to the forget and retain sets.

The authors conduct thorough experiments on both class-level and instance-level unlearning, as well as machine unlearning for generative models (DDPM and DiT). Extensive results and ablations show that their method outperforms similar baselines that combine machine unlearning with model pruning. Their method achieves results closest to an oracle model, which is retrained from scratch with subsequent sparsification, pruning, and fine-tuning.

Critical strengths
  1. The paper is exceptionally well written and clearly structured. The method is clearly introduced and motivated, differences from other approaches are explained, and design choices are thoroughly ablated. The paper also provides very thorough details to support reproducibility. The only missing elements are a code release and a reproducibility statement, which would make the paper exemplary in this regard.
  2. The paper provides a thorough experimental evaluation covering both class-level and instance-level unlearning, as well as machine unlearning for generative models. The authors present convincing results and extensive ablations across the specified tasks and various experimental settings. In particular, they ablate the individual pipeline stages, the design and computation of the sensitivity score (which determines which weights are pruned), the number of pruning rounds, and the proportion of pruned weights. Overall, the results are convincing for their specific unlearning + pruning setting.
Critical weaknesses
  1. The proposed method is only applicable to scenarios where machine unlearning and model pruning are performed together, rather than unlearning-only scenarios. However, Table A5 shows that, in terms of absolute performance, unlearning without pruning achieves better results on the test set. Hence, there seems to be a trade-off between test set performance and model pruning. This raises the question of how to select the pruning fraction pp. While the authors provide an ablation in the appendix, there seems to be no practical guideline for selecting pp or understanding its impact on downstream performance. The method also requires access to the forget set and is comparatively slow in terms of training time, still requiring approximately 1/3 of the time needed for retraining from scratch. How does the method scale to larger models or scenarios where unlearning requests arrive sequentially rather than all at once?
  2. The method introduces a large number of hyperparameters, including the loss weights (α\alpha, η\eta, λ\lambda), κ\kappa for the sensitivity coefficient λ\lambda, the dynamic loss weight ζ\zeta (which is specified as a function), the number of training steps TsT_s and TfT_f, and the pruning threshold τ\tau. How is τ\tau selected to achieve the desired pruning fraction pp? More generally, how should these hyperparameters be selected for different datasets, and how sensitive is the method to their values?

Minor comments:

  • In Line 49: Please list the works by Fang et al. in chronological order.
  • Table 5: Is “1 (Uniform)” written like this on purpose?
Questions for authors
  1. How do you balance the number of steps in the different stages, TsT_s and TfT_f? Could this lead to overfitting on the retain set, particularly given that both the oracle and most other methods seem to saturate on this set, especially for CIFAR-10, achieving 100% accuracy?
  2. The sensitivity score design experiment is unclear. Could you elaborate on the different design choices for SS by providing their corresponding formulas, so that the comparison with Eq. 7 (the default choice) becomes clearer?
7 comments

We thank the reviewer for the positive and constructive feedback.

W1 (a). How to select the pruning fraction p? Sorry for the confusion. We do not select p. In practice, p is set by the deployment budget (latency or memory), and PruneForget then returns a model that is pruned to that budget and has also unlearned the forget set. For reference, we report p = 0.3/0.5/0.7 for ResNet-18, giving 2.07×/3.98×/11.17× speedup (Tab. A2).

W1 (b). Runtime is still ~1/3 of retraining; scalability and repeated requests? PruneForget's runtime is comparable to that of Pretrain→nSP (< 9% on ResNet-18), which is the time required to prune the model in any case. PruneForget makes each step of a pruning pipeline unlearning-aware and generalizes to different pruning approaches (we experiment with 3 pruning algorithms in the paper). Hence, whenever a model is to be pruned, our method only adds little overhead.

On repeated requests: as noted in W1(a), the pruning ratio is fixed by the target device, whose capacity is constant, so pruning is a one-time cost rather than a per-request cost. Later unlearning requests on the already-pruned model need no further pruning and reduce to plain unlearning on the smaller model.

W2. Hyperparameter settings. f_base is the only hyperparameter we tune. The grid search over f_base is in Tab. A3 and A4, and the results are fairly flat: at 50% unlearning, Avg Δ is 0.06/0.07 for f_base = 80/100, and at 10% all values give ≤ 0.12.

All other quantities are fixed or derived:

  • α = |U|/|D\U|;
  • η follows the learning rate of the underlying pruning method;
  • λ = 1e-3 in all experiments;
  • κ = 4t/T_s, a time-scaled version of the constant 4 used in the original method;
  • ζ_t is computed from the current unlearning loss;
  • the pruning threshold τ is set automatically as the importance-score cutoff at which the model reaches the target pruning ratio.

Minor. We will list Fang et al. chronologically. "1 (Uniform)" was intentional (S ≡ 1); we will rename it "Uniform (S=1)" to avoid confusion.

Q1. Balance of T_s and T_f; overfitting to the retain set? T_s and T_f mostly follow the default schedule of each pruning method and are not tuned. The only exception is Swin-T, where we set T_f = 50 instead of the original 300 because performance already saturates by then. We reduce it to speed up the experiments; see "Details for pruning methods" in Appendix A2. Note that the same T_s and T_f are applied to the pruning procedure (nSP) of every baseline that does not specify its own. Regarding the retain set, CIFAR-10 is a relatively easy dataset, so reaching 100% retain accuracy is common in unlearning experiments and is also observed in prior work (e.g., SFRon).

Q2. Sensitivity score design choice. Tab. 5 already compares several designs of S: uniform (S ≡ 1), random, unlearn-only, retain-only, and a Fisher-based variant. Our choice, the ratio of unlearn to retain gradient norms, gives the lowest Avg Δ. If the reviewer has another design in mind, we are happy to add it.

Reviewer_gUjWOpenReview ↗

Comment on the Authors Response

I want to the authors for their time and their responses. While I appreciate the clarifications, some of my concerns remain unresolved, as outlined below.

W1b

My runtime concern was relative to Retrain→nSP, rather than Pretrain→nSP. While the additional overhead compared to standard pruning is small, PruneForget still requires approximately one-third of the retraining time, which I would not consider negligible.

More importantly, the method remains applicable only when pruning and unlearning are required simultaneously. The explanation regarding sequential requests further narrows this use case: if pruning is a one-time operation dictated by deployment constraints, why would it necessarily coincide with an unlearning request? Requests arriving before pruning must already be addressed, while those arriving afterward require standard unlearning. This raises questions about the practical relevance of the proposed joint procedure.

W2

Thank you for clarifying the hyperparameter selection. However, I would disagree that the results are generally "fairly flat" when it comes to selecting f_base. In Table A3, Avg Δ for 50% random unlearning ranges from 0.06 to 0.70 depending on f_base, indicating considerable sensitivity. The optimal value also differs substantially between 10% and 50% random unlearning.

As I understand it, the main results were obtained using f_base values selected based on performance relative to the retrained oracle, as pointed out by Reviewer e2cc. However, this oracle would not be available in practice. While the oracle-free heuristic introduced in response to Reviewer e2cc is a useful addition, the main results still rely on oracle-based hyperparameter selection. This raises concerns about the validity of the reported performance as an estimate of what can be achieved in practice, particularly given the sensitivity to f_base. Could the authors clarify this and discuss whether the main experiments would need to be re-evaluated using oracle-free selection?

The proposed oracle-free heuristic has only been evaluated on CIFAR-10 with ResNet-18. Moreover, substantially different values of f_base are used across architectures (20–100 for ResNet-18, 3 for Swin-T, and 5 for DDPM and DiT). How were these values selected, why do they differ so considerably, and how robust are the results to these choices? In particular, can the authors demonstrate that their main findings generalize under oracle-free hyperparameter selection across the evaluated architectures?

Q2

Thank you for the clarification. However, my question was not about adding further sensitivity score designs, but about understanding the differences between the existing variants in Table 5. Beyond their names, it is unclear how these scores are computed and how they differ mathematically. For example, does "Unlearn Only" simply use the numerator of Eq. 7? How are "Retain Only" and "Fisher Score" constructed?

I would appreciate explicit formulas and a brief explanation of each variant to better understand what the experiment is comparing. More detailed descriptions of the experimental setup and the individual variants, potentially in the appendix, would improve the clarity and reproducibility of this ablation.

We thank the reviewer for the follow-up and address the remaining concerns below.

W1 (b). Runtime. We apologize for the confusion. Our task is to produce a model that is both pruned and unlearned. This is the task studied by prior work such as LLM-Eraser and Un-pruning. In this task, every method has to pay for compression (nSP), including the oracle: Retrain→nSP is retraining plus nSP. The cost of unlearning is therefore the time spent beyond compression alone (nSP). Measured this way, PruneForget adds a modest overhead to the cost of compression, whereas retraining adds more (Tab. R14). The reason is that retraining repeats the full training schedule. Note that retraining on TinyImageNet takes less effort, since it starts from a pretrained model instead of training from scratch.

Table R14. Extra time spent on unlearning by PruneForget and Retrain→nSP, relative to compression only (nSP).

SettingCompression only (nSP)PruneForgetRetrain→nSP
Tab. 1, CIFAR-10 10%984 s+14 s (+1.4%)+1231 s (+125%)
Tab. 1, CIFAR-10 50%556 s+38 s (+6.8%)+962 s (+173%)
Tab. 2, CIFAR-10 1 class1111 s+116 s (+10%)+1130 s (+102%)
Tab. 2, CIFAR-10 5 classes610 s+196 s (+32%)+908 s (+149%)
Tab. A9, CIFAR-100 50%567 s+14 s (+2.5%)+1984 s (+350%)
Tab. 3, TinyImageNet (Swin-T) 10%6278 s+748 s (+12%)+2536 s (+40%)
Tab. 3, TinyImageNet (Swin-T) 50%3771 s+1141 s (+30%)+1933 s (+51%)
Tab. 9, CIFAR-10 (DDPM)21304 s+4950 s (+23%)+377318 s (+1771%)

W1 (b). Applicability. Pruning and unlearning coincide whenever a model is released for deployment. Deletion requests can be batched and honored at the next release, and every release for new hardware requires a new compression run. At that point the provider has to do both in either case. The only question is whether to do them sequentially (Unlearn→Prune or Prune→Unlearn) or jointly. Our results show that the joint procedure is better at the same cost: on CIFAR-10 with 50% unlearning, PruneForget reaches Avg Δ 0.06, versus 0.16 for SFRon→3SP and 3.19 for 3SP→SFRon, with run times of 594 s, 627 s and 593 s, respectively. Requests that arrive between releases are a standard unlearning problem on a fixed model, which is the focus of pure unlearning methods such as SFRon, and can be achieved by running only PruneForget's third stage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.