- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propose the novel approximate unlearning method DIMU (Distillation-Regularized Importance-Masked Unlearning). Pointing out the problems of existing approximate unlearning that risk leaving residual information, this study seeks to address these issues by utilizing self knowledge distillation. DIMU consists of CRM feature-channel masking and self knowledge distillation. For the efficiency, they freeze the feature extractor and mask the forget sensitive channel. With the various dataset and model architecture experiments, DIMU shows that performance and speed effectiveness.
Critical strengths
- The motivation and approach is very intuitive and clear.
- Experiments are conducted extensively.
- In particular, the control experiments used to explain each component of the algorithm are well organized, making the presentation highly effective.
Critical weaknesses
- Although authors emphasize the importance of removing the influence of forgotten data in the introduction seciton, DIMU freezes the feature extractor, leaving the underlying representations unchanged. This raises a fundatmental concern about whether DIMU actually removes the learned information or merely suppress its contribution to the predictions. I think this is very important issue in machine unlearning.
- In sample-wise unlearning setting, authors interpret the discrepancy between DIMU and retrain model as targeted suppression rather than retraining-equivalent forgetting. However, this interpretation seems to deviate significantly from the fundamental objective of machine unleanring. Although such an evaluation perspective may be understandable from a practical prespective, suppression-based forgetting is arugably more acceptable in class/concept-wise unlearning. In sample-wise, preserving generalization while selectively removing the influence of samples is particularly important. That's why we use MIA and relearning to evaluate the unleanring method.
- It is encouraging that the authors consider relearning attacks. Evaluating unlearning from the perspective of relearning, in addition to MIA, is important. However, rather than simply comparing DIMU with retrain, it would be more informative to investigate how resistant DIMU is to relearning compared with other approximate unlearning. In particular, I suspect that attack such as the Prototypical Relearning Attack proposed in [1] could be effective against DIMU, given its frozen feature extractor.
- It remains unclear whether DIMU offers a significant computational advantage over other approximate unlearning methods. I wonder why Boundary Unlearning [2], which achieves efficient forgetting by modifying decision boundary, is not included as a baseline. A comparison with this approach would help better establish the computational efficiency and advantage of DIMU.
[1] Ha, S., Park, S. & Yoon, S.W.. (2026). Unlearning’s Blind Spots: Over-Unlearning and Prototypical Relearning Attack. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:39108-39126 [2] Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, Chen Wang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 7766-7775
Questions for authors
See the weaknesses.
5 comments
Response to Reviewer Tbz3
CW2: Sample-level suppression may cause over-unlearning rather than selective forgetting, raising concerns about generalization, MIA, and relearning robustness.
We thank the reviewer for highlighting the important distinction between sample-level forgetting and over-unlearning. We agree that random sample-level deletion requires a different interpretation from class-level forgetting. When individual training samples are removed, other examples from the same classes remain in the retain set. Consequently, a correctly retrained model may continue to classify removed samples accurately, despite their exclusion from training.
Our manuscript explicitly examines this distinction in Section 5.3 (Sample-Level Unlearning, Table 5). Using identical forget subsets for DIMU and Retrain across 1%, 5%, and 10% random deletion on CIFAR-10, Retrain achieves 91.19–91.87% accuracy on removed samples, whereas DIMU achieves 65.89–68.47%. DIMU also produces larger forgotten-sample confidence reductions (33.13–36.62 percentage points) than Retrain (5.23–6.39 percentage points), while maintaining 96.79–97.28% retained accuracy and 88.26–90.42% overall test accuracy. We emphasize that lower removed-sample accuracy and confidence do not necessarily indicate stronger instance-level information removal. Instead, these differences may reflect targeted predictive suppression or over-unlearning.
To characterize this discrepancy beyond accuracy, Appendix B (Table 23, Comparison with Retrain) reports additional predictive and representation-level analyses. Symmetric KL divergence relative to Retrain is substantially smaller on retained samples (0.189–0.366) than on forgotten samples (1.818–2.177). Representation CKA is likewise higher on retained samples (0.807–0.854) than on forgotten samples (0.740–0.762). Meanwhile, overall test-set confusion-matrix similarity remains 0.998–0.999. These findings demonstrate stronger preservation of retained behavior while revealing substantial differences in forgotten-sample predictions. They do not establish retraining-equivalent instance deletion or exclude localized collateral damage to retained samples.
We also appreciate the reviewer's reference to Ha et al. (2026). Their over-unlearning metric, (Over-Unlearning at ε), is particularly relevant because it evaluates collateral damage near the forget set, which global retained accuracy may not capture. Our existing evaluations do not directly measure , and we therefore do not claim that DIMU eliminates localized over-unlearning.
Accordingly, we clarify that class-level forgetting is DIMU's primary demonstrated strength, while the sample-level experiments provide complementary evidence of targeted suppression, retained utility, and limitations relative to Retrain. We acknowledge that stronger instance-level privacy and localized over-unlearning evaluations are needed before claiming retraining-equivalent sample deletion. This distinction is consistent with the approximate-unlearning scope and limitations discussed in Section 6 (Limitations and Future Directions).
Response to Reviewer Tbz3
CW3: DIMU's resistance to Prototypical Relearning Attacks should be evaluated against other approximate unlearning methods, not just Retrain.
We thank the reviewer for emphasizing the importance of evaluating resistance to stronger relearning attacks, particularly those exploiting feature representations preserved by a frozen backbone. We agree that comparisons with Retrain alone are insufficient and that representative approximate unlearning methods should also be considered.
Our submitted manuscript already evaluates one-epoch forgotten-class relearning (Appendix Table 18), where forgotten-class accuracy increases by 22.40 percentage points (pp) for DIMU, compared with 9.35 pp for Retrain and 41.37 pp for Head-only Retain-FT. However, prototype-based attacks provide a complementary evaluation of forgotten-class recoverability. To investigate this concern, we additionally evaluate the Prototypical Relearning Attack (PRA) proposed by Ha et al. (ICML 2026) on CIFAR-100 using ResNet-18, with class 0 as the unlearning target. We compare Retrain, Head-only Retain-FT, DELETE, DIMU, and DIMU-L4 using 1, 5, and 10 forgotten-class support samples.
The results show that DIMU and DIMU-L4 exhibit lower forgotten-class recovery than Retrain and the evaluated approximate methods across all three support sizes. Under five-shot PRA, DIMU achieves 26.00% forgotten-class accuracy, compared with 30.00% for Retrain, 49.00% for DELETE, and 58.00% for Head-only Retain-FT. DIMU-L4 further reduces recovery to 17.00%. This trend persists under ten-shot PRA, where DIMU and DIMU-L4 obtain 38.00% and 28.00%, respectively, compared with 43.10% for Retrain. Importantly, retained-class test accuracy after five-shot PRA remains 77.94% for DIMU and 77.35% for DIMU-L4. These findings provide additional evidence that DIMU's importance-guided CRM masking and classifier adaptation can limit recovery under the evaluated prototype-based attack. The lower recovery observed for DIMU-L4 further suggests that limited backbone adaptation can strengthen resistance in this setting.
Table 28. Prototypical Relearning Attack (PRA) on CIFAR-100 using ResNet-18. Forgotten-class test accuracy before and after PRA with different support sizes, together with retained-class test accuracy after five-shot PRA. All values are percentages. Lower post-attack forgotten-class accuracy indicates stronger resistance to the evaluated attack.
| Method | Before ↓ | PRA-1 ↓ | PRA-5 ↓ | PRA-10 ↓ | PRA-5 ↑ |
|---|---|---|---|---|---|
| Retrain | 0.00 | 12.50 | 30.00 | 43.10 | 78.00 |
| Head-only Retain-FT | 12.75 | 27.10 | 58.00 | 72.00 | 77.00 |
| DELETE | 0.00 | 19.50 | 49.00 | 64.00 | 76.50 |
| DIMU | 0.00 | 11.00 | 26.00 | 38.00 | 77.94 |
| DIMU-L4 | 0.00 | 8.50 | 17.00 | 28.00 | 77.35 |
PRA and the existing one-epoch fine-tuning experiment evaluate distinct relearning procedures, providing complementary evidence of post-unlearning recoverability. The observed improvements demonstrate resistance under the evaluated PRA settings rather than guaranteeing robustness against all possible relearning attacks. We have incorporated the PRA analysis into Appendix Table 28 and summarized the findings in Section 5.3.
Response to Reviewer Tbz3
CW1: Does DIMU genuinely remove forgotten information or merely suppress its predictive influence while preserving the underlying representations?
We thank the reviewer for highlighting this fundamental distinction between suppressing the predictive influence of forgotten data and removing information from the underlying learned representations. We agree that these are not equivalent. Canonical DIMU does not claim complete information erasure from the frozen backbone. Rather, it is designed as an approximate unlearning approach that attenuates forget-associated contributions to deployed predictions through a fixed, importance-guided CRM mask.
Specifically, DIMU identifies forget-sensitive penultimate channels using gradients computed on the forget set, suppresses their contribution through the fixed CRM mask, and adapts the classifier head using forgetting, retention, and self-knowledge-distillation objectives. Therefore, although the raw backbone representation remains unchanged, the CRM-masked effective representation supplied to the updated classifier and its predictive behavior are modified. This distinction is central to our formulation and is not intended to imply that forgotten information has been physically removed from the underlying feature extractor.
Importantly, we directly evaluate this limitation using adaptive representation-level attacks. On CIFAR-10 (Table 3), canonical DIMU exhibits a raw-feature MIA AUROC* of 0.989, compared with 0.672 for the deployed CRM-masked effective features. We additionally investigate DIMU-L4, which updates the final residual stage and classifier head while retaining the fixed CRM mask. DIMU-L4 further reduces raw-feature and CRM-masked-feature AUROC* to 0.616 and 0.549, respectively, while achieving 0.00% forgotten-class test accuracy and 95.40% retained-class accuracy. Appendix Table 12 confirms similar improvements on CIFAR-100, where DIMU-L4 achieves raw-feature and CRM-masked-feature AUROC* of 0.620 and 0.550.
These results distinguish the predictive suppression achieved by canonical DIMU from the stronger representation adaptation offered by DIMU-L4. We emphasize that reduced empirical distinguishability does not constitute a guarantee of complete information removal. We have further clarified this scope in the Introduction and Section 6, positioning DIMU as an efficient approximate unlearning framework rather than a certified representation-erasure method.