- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes PruneForget, which uses the unlearn set to guide a three-stage structured pruning pipeline. A per-group sensitivity score is computed on the pretrained model as the ratio of forget-set to retain-set gradient norms. It weights the sparsity regularizer during sparse training, rescales importance scores so that forget-sensitive groups are pruned first, and post-pruning fine-tuning is replaced with unlearning fine-tuning. Experiments on ResNet-18, Swin-T, DDPM, and DiT report the smallest gap to a retrain-then-prune oracle among the baselines, at lower runtime than the baselines.
Critical strengths
-
The setting of jointly unlearning and structurally pruning a model, producing an actually smaller and faster network, is practically relevant vision models.
-
The method is simple and reasonable; it is applied to three pruning methods across CNNs, ViTs, and diffusion models.
-
The experiments cover both classification and generation with ablations of each component, and uniform pruning ensures identical pruned architectures across methods for a fair efficiency comparison.
Critical weaknesses
- Novelty is limited and related work is missing. Ranking parameters by forget-vs-retain importance and then masking or pruning them is a well-studied idea in unlearning. Examples include SSD [1], Scissorhands [2], selective pruning [3], ConceptPrune [4], and Forget-It-All [5]. Only [1] is cited, and only in passing. Table 5 also shows that a Fisher score (0.10) or an unlearn-only score (0.07) does about as well as the proposed ratio (0.06), so the specific score design does not seem to matter much.
References
[1] J. Foster, S. Schoepf, A. Brintrup. Fast Machine Unlearning without Retraining through Selective Synaptic Dampening. AAAI 2024.
[2] J. Wu, M. Harandi. Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks. ECCV 2024.
[3] N. Pochinkov, N. Schoots. Dissecting Language Models: Machine Unlearning via Selective Pruning. arXiv:2403.01267, 2024.
[4] R. Chavhan, D. Li, T. Hospedales. ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning. ICLR 2025.
[5] K. Deng et al. Forget-It-All: Multi-Concept Machine Unlearning via Concept-Aware Neuron Masking. ICML 2026.
-
Most of the gain comes from Stage 3, which is standard unlearning fine-tuning. In Table 4, Stage 2 (the main contribution) only moves Avg Δ from 0.41 to 0.34. Adding Stage 3, which is SFRon-style fine-tuning per Appendix A2, brings it to 0.06. A Stage-3-only ablation, i.e., standard pruning followed by unlearning fine-tuning, is needed to show that Stages 1–2 actually help.
-
The improvements are mostly within noise. In Table 1 the ΔUA differences (e.g., 0.03 vs. 0.16) are smaller than the reported std (0.3–0.6). In Table 2 and Table 9, Pretrain→nSP performs no unlearning and still essentially matches the oracle (Avg Δ 0.15 vs. 0.14; UA 97.94 vs. 98.39), so these benchmarks cannot separate the methods. PruneForget's D_KL to the oracle is also consistently higher than this no-unlearning baseline (Tables 1, 3, A14), which works against the claim that it better approximates the oracle.
-
Hyperparameters are tuned against the oracle. f_base is chosen by the gap to the retrained oracle (Tables A3/A4). In practice that oracle is exactly what the method is meant to avoid computing.
Questions for authors
-
How does the sensitivity score differ from the forget/retain importance ratios used in SSD and selective pruning? Can you compare against SSD, Scissorhands, ConceptPrune, or Forget-It-All?
-
What do you get with Stage 3 alone, i.e., standard pruning followed by unlearning fine-tuning? Does adding Stages 1–2 give a significant improvement over that?
-
How would you choose f_base without access to the retrained oracle? How were the baselines' hyperparameters tuned?
3 comments
Q3 (a). Improvements are within noise. We thank the reviewer for the question. In hindsight, the way we presented the results makes it hard to compare methods. We now summarize them and explain where PruneForget is and is not separated from the baselines.
First, the three gaps (ΔUA, ΔRA, ΔTA) trade off against one another, so the meaningful question is which method is close to the oracle on all metrics at once. This is what Avg Δ in our tables measures. Table R5 collects it for every setting.
Table R5. Avg Δ to the oracle (lower is better). Bold = best, italic = second best.
| Method | Tab. 1 CIFAR-10 10% | Tab. 1 CIFAR-10 50% | Tab. 2 CIFAR-10 1-class | Tab. 2 CIFAR-10 5-class | Tab. 3 TinyImageNet 10% | Tab. 3 TinyImageNet 50% | Tab. 9 DDPM* | Tab. A9 CIFAR-100 50% |
|---|---|---|---|---|---|---|---|---|
| Pretrain→nSP | 0.43 | 0.78 | 0.15 | 0.07 | 0.59 | 0.88 | 0.38 | 1.57 |
| nSP→SFRon | 6.70 | 3.19 | 1.70 | 2.33 | 4.24 | 3.51 | 0.31 | 2.50 |
| SFRon→nSP | 0.07 | 0.16 | 0.23 | 0.30 | 0.62 | 1.01 | 0.39 | 1.20 |
| ℓ1-MU→2SP | 5.03 | 4.48 | 3.12 | 1.64 | 32.69 | 37.76 | 0.48 | 6.67 |
| Un-pruning | 1.14 | 1.91 | 0.52 | 0.55 | 4.94 | 4.89 | 163.50 | 3.45 |
| Bilevel | 7.25 | 5.69 | 36.92 | 11.13 | 6.17 | 7.92 | 2.64 | 5.04 |
| LLM-Eraser | 13.81 | 11.68 | 13.03 | 6.81 | 3.86 | 4.83 | 5.25 | 20.76 |
| PruneForget | 0.01 | 0.06 | 0.14 | 0.07 | 0.54 | 0.43 | 0.20 | 0.72 |
*Mean of ΔUA and ΔFID. This mixes an accuracy and a distance, so it is only a rough summary.
- PruneForget has the lowest Avg Δ in all 8 settings.
- The second-best method changes with the setting (SFRon→nSP in 3, Pretrain→nSP in 3, nSP→SFRon on DDPM). PruneForget is the only method that is consistently best.
Second, we agree that several single-metric differences are within noise. We use a conservative criterion: two results are separated only if their means differ by more than the sum of their standard deviations (non-overlapping ±1 std intervals over the 10 trials). Every case in the main tables where a baseline has a lower gap than ours on a single metric is a tie under this criterion:
Table R6. Cases where a baseline has a lower single-metric gap than PruneForget (raw values).
| Setting | Metric | Baseline | PruneForget | Gap vs. σ₁+σ₂ |
|---|---|---|---|---|
| Tab. 1, CIFAR-10 50% | UA | SFRon→3SP: 7.45 ± 0.21 | 7.44 ± 0.32 | 0.01 < 0.53 → tie |
| Tab. 3, TinyImageNet 50% | RA | Pretrain→2SP: 99.32 ± 0.04 | 99.35 ± 0.03 | 0.03 < 0.07 → tie |
| Tab. 3, TinyImageNet 50% | RA | SFRon→2SP: 99.34 ± 0.05 | 99.35 ± 0.03 | 0.01 < 0.08 → tie |
| Tab. 9, DDPM | FID | nSP→SFRon: 16.92 ± 0.70 | 16.92 ± 0.76 | 0.00 < 1.46 → tie |
| Tab. 9, DDPM | FID | SFRon→nSP: 16.96 ± 0.71 | 16.92 ± 0.76 | 0.04 < 1.47 → tie |
These results show that PruneForget consistently achieves the lowest Avg Δ, and on each individual gap (ΔUA, ΔRA, ΔTA) it is either the best or tied with prior work.
Q3 (b). Pretrain→nSP matches the oracle. We agree that class-wise unlearning on CIFAR-10 and DDPM does not separate Pretrain→nSP from the oracle. Below are the results in a harder random-unlearning settings, where Pretrain→nSP under-forgets (Table R7).
Table R7. UA of the oracle, Pretrain→nSP and PruneForget. Pretrain→nSP under-forgets (is separated from the oracle) in these settings.
| Setting | Oracle | Pretrain→nSP | PruneForget | Separated from the oracle? |
|---|---|---|---|---|
| Tab. 1, CIFAR-10 50% | 7.60 ± 0.22 | 6.06 ± 0.17 | 7.44 ± 0.32 | Pretrain→nSP: yes (gap 1.54 > 0.39). Ours: no |
| Tab. A9, CIFAR-100 50% | 32.48 ± 0.31 | 29.27 ± 0.71 | 33.94 ± 1.24 | Pretrain→nSP: yes (gap 3.22 > 1.02). Ours: no |
| Tab. 3, TinyImageNet 50% | 22.61 ± 0.17 | 20.30 ± 0.13 | 21.64 ± 0.12 | Both yes; ours is closer |
Q3 (c). Pretrain→nSP has a lower D_KL. We agree that Pretrain→nSP has a slightly lower D_KL in Tables 1, 3 and A14. However, the same tables show that it does not forget: in every one of these settings, its UA stays below the oracle's, by up to 2.37 points (Table R8). D_KL therefore cannot be used as the sole metric to determine whether a model has unlearned effectively. Moreover, among methods that actually unlearn, ours is best or tied: in Tab. 1, SFRon→3SP has 0.22 / 0.27 and 3SP→SFRon has 0.24 / 0.27, all others are ≥ 0.27, and ours is 0.22 / 0.25.
Table R8. Pretrain→nSP fails to forget (low UA) in Tables 1, 3 and A14.
| Setting | Oracle UA | Pretrain→nSP ΔUA (UA) | PruneForget ΔUA (UA) |
|---|---|---|---|
| Tab. 1, CIFAR-10 10% | 6.03 | 0.89 (5.15) | 0.03 (6.01) |
| Tab. 1, CIFAR-10 50% | 7.60 | 1.54 (6.06) | 0.16 (7.44) |
| Tab. 3, TinyImageNet 10% | 19.88 | 1.64 (18.24) | 1.50 (18.38) |
| Tab. 3, TinyImageNet 50% | 22.61 | 2.32 (20.30) | 0.98 (21.64) |
| Tab. A14, TinyImageNet (global) | 22.95 | 2.37 (20.58) | 1.16 (21.79) |
Q4 (a). How to choose f_base without the oracle?
We follow SFRon, which also sets a different f_base per architecture. We agree that the oracle cannot be used to select f_base in practice. Since f_base only controls the strength of unlearning (i.e., a larger f_base means weaker unlearning), its effect is directly observable in the training curves, and a practitioner can monitor the unlearning and retain losses to adjust it.
Here we provide a simple heuristic that tunes f_base dynamically from these losses. It already gives results close to the oracle-selected f_base, within 0.01–0.06 Avg Δ in all four settings (Table R9).
f_base starts at 100 (other start values in Table R10) and is updated once per epoch from training-time losses only, by a factor of 1.25 in either direction:
- If the retain loss rises by more than 0.1 above its running minimum, f_base is increased (less gradient ascent).
- Random unlearning: if the forget loss is more than 10% below the held-out loss, f_base is decreased (more gradient ascent); if it is more than 10% above, f_base is increased.
- Class-wise unlearning: if the forget-set accuracy is above 1%, f_base is decreased; otherwise it is increased.
Table R9. Oracle-free f_base vs. the best fixed f_base selected with the oracle (Avg Δ).
| Setting | Oracle-free f_base | Best fixed f_base (oracle-selected) |
|---|---|---|
| CIFAR-10, random 10% | 0.07 | 0.01 |
| CIFAR-10, random 50% | 0.10 | 0.06 |
| CIFAR-10, 1 class | 0.15 | 0.14 |
| CIFAR-10, 5 classes | 0.12 | 0.07 |
One may ask whether the heuristic simply replaces f_base with a new hyperparameter, its start value. Table R10 shows that it does not. We start the rule from values that span a 5× range (20 to 100). Avg Δ stays between 0.10 and 0.14, and every start value yields a lower Avg Δ than every baseline in Tab. 1, the best of which is SFRon→nSP with 0.16.
Table R10. Oracle-free f_base with different start values (CIFAR-10, ResNet-18, 50% random unlearning).
| Initial f_base | ΔUA ↓ | ΔRA ↓ | ΔTA ↓ | Avg Δ ↓ |
|---|---|---|---|---|
| 20 | 0.41 (7.18) | 0.00 (100.00) | 0.01 (92.21) | 0.14 |
| 50 | 0.25 (7.35) | 0.00 (100.00) | 0.06 (92.14) | 0.10 |
| 80 | 0.37 (7.23) | 0.00 (100.00) | 0.05 (92.26) | 0.14 |
| 100 | 0.28 (7.31) | 0.00 (100.00) | 0.00 (92.20) | 0.10 |
Q4 (b). How were baselines tuned?
Note that f_base is the only hyperparameter we tune; all others are fixed or derived. For every baseline, we mostly follow the original settings and adapt them slightly to make the runtimes comparable and obtain reasonable results; see Appendix A2, "Details for baselines". Specifically, we pick one hyperparameter per method, tune it over 3–5 values, and select the best for comparison. Tables R11–R13 show the hyperparameter search for LLM-Eraser, Un-pruning, and SFRon. We will add the hyperparameter search process to the paper.
Table R11. LLM-Eraser with different weights α of the retain term (CIFAR-10, ResNet-18, 50% random unlearning).
| α | ΔUA ↓ | ΔRA ↓ | ΔTA ↓ | Avg Δ ↓ |
|---|---|---|---|---|
| 0.01 | 10.60 (18.20) | 14.27 (85.73) | 10.81 (81.39) | 11.89 |
| 0.1 | 10.46 (18.05) | 14.00 (85.99) | 10.59 (81.61) | 11.68 |
| 1.0 | 10.97 (18.57) | 14.71 (85.29) | 11.17 (81.04) | 12.28 |
Table R12. Un-pruning with different fine-tuning learning rates (CIFAR-10, ResNet-18, 50% random unlearning).
| Learning rate | ΔUA ↓ | ΔRA ↓ | ΔTA ↓ | Avg Δ ↓ |
|---|---|---|---|---|
| 0.001 | 6.79 (14.39) | 4.49 (95.51) | 7.09 (85.12) | 6.12 |
| 0.005 | 2.88 (10.48) | 0.12 (99.88) | 3.16 (89.04) | 2.05 |
| 0.01 | 2.72 (10.31) | 0.07 (99.93) | 2.94 (89.26) | 1.91 |
Table R13. SFRon→3SP with different forget frequencies (CIFAR-10, ResNet-18, 50% random unlearning).
| f_base | ΔUA ↓ | ΔRA ↓ | ΔTA ↓ | Avg Δ ↓ |
|---|---|---|---|---|
| 1 | 3.46 (11.06) | 0.10 (99.90) | 3.52 (88.68) | 2.36 |
| 2 | 2.34 (9.93) | 0.05 (99.95) | 2.55 (89.65) | 1.65 |
| 5 | 0.15 (7.45) | 0.00 (100.00) | 0.34 (91.87) | 0.16 |
| 10 | 0.43 (7.17) | 0.00 (100.00) | 0.17 (92.38) | 0.20 |
We thank the reviewer for the detailed feedback and the references.
Q1. Comparison with unlearning-by-pruning methods (SSD, Scissorhands, selective pruning, ConceptPrune, Forget-It-All)?
Thank you for providing these related works. We did not claim that ranking channels by forget-vs-retain importance is novel. We claim that it should guide structured pruning across all stages. The related works didn't discuss these papers because they address a different problem from ours. We aim to produce a model that is both unlearned and structurally compressed to a target ratio. In contrast, these methods prune parameters to unlearn, so they modify only a tiny fraction of weights (e.g., 0.12% in ConceptPrune, <0.3% in Forget-It-All), and their pruning is unstructured, so the inference cost is unchanged. We agree that citing these papers and discussing the differences will make our work more complete. We thank the reviewer for the comment, and would discuss them in related works.
We replace our S with the score of each method and show the comparison in Table R3; our gradient-norm ratio still gives the lowest Avg Δ. We will include this comparison in the paper.
Table R3. Comparison of sensitivity score designs (CIFAR-10, ResNet-18, 50% random unlearning).
Q2. Stage 3-only ablation. We agree that Tab. 4 is cumulative and does not isolate Stage 3. Table R4 therefore reports each stage on its own. Each stage alone already improves over Pretrain→3SP (Avg Δ 0.78). Combining all three stages gives the smallest TA gap and the lowest Avg Δ (0.06). Stage 3 alone is therefore a strong component but not the only source of the gain.
Table R4. Stage ablation (CIFAR-10, ResNet-18, 50% random unlearning).