- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Learning to Unlearn (L2U), a framework that improves continual learning by selectively removing outdated, redundant, or poisoned information. The authors introduce two approaches: Heuristic-L2U, which uses predefined rules or limited supervision to select samples for unlearning, and Meta-L2U, which learns the selection strategy through meta-learning. The framework is evaluated under both natural distribution shifts and adversarial poisoning attacks. Experiments on cybersecurity and image datasets show that L2U generally outperforms existing continual learning methods, especially under poisoning attacks and limited model capacity.
Critical strengths
-
The paper addresses an important limitation of traditional continual learning: preserving historical knowledge is not always beneficial, especially when previous data becomes outdated or poisoned. Introducing selective unlearning into continual adaptation is a meaningful direction, particularly for cybersecurity applications where data distributions frequently change.
-
The authors evaluate L2U on multiple datasets, including malware detection, network intrusion detection, and image classification. The experiments cover natural distribution shifts, poisoning attacks, different unlearning strategies, and reduced model capacity. The results are particularly promising under poisoning attacks, where Meta-L2U achieves higher accuracy than existing continual learning baselines.
Critical weaknesses
-
The main contribution of this paper is learning which samples to unlearn through meta-learning. However, as shown in Table 1, Meta-L2U only provides small improvements over Heuristic-L2U on several datasets. For example, the improvement is less than 0.3 percentage points on both EMBER 2018 and EMBER 2024. Meanwhile, Meta-L2U requires additional computational cost to train the selection strategy. It is therefore unclear whether the performance improvements justify the additional complexity of meta-learning.
-
Although L2U achieves promising results under poisoning attacks as shown in Table 2, the evaluation mainly focuses on classification accuracy rather than directly measuring the success rate of the proposed long-range poisoning attack. In addition, the adaptive selection methods use a small number of labeled clean and poisoned samples to identify suspicious data, while the compared continual learning baselines do not necessarily have access to the same information. Therefore, it remains unclear how much of the improvement comes from the proposed unlearning framework itself.
Questions for authors
-
In the poisoning experiments, L2U filters detected poisoned samples from the replay buffer, while MADAR does not. Could the authors provide comparisons where all methods use the same replay-buffer filtering and poisoning-related supervision?
-
How can the authors distinguish the benefits of selective unlearning from those of simply removing suspicious samples and performing additional fine-tuning? Could the authors clarify whether the reported improvements reflect actual removal of the selected samples' influence, rather than the effects of data filtering or additional optimization?
-
Meta-L2U requires substantial computation for meta-training and additional adaptation under adversarial drift. What is the total computational overhead compared with Heuristic-L2U and other baselines? Also, have you evaluated the learned selection strategy on distribution shifts not used during meta-training?
4 comments
Request for feedback
Dear Reviewer,
We have tried our best to address the concerns. Please let us know if you have any further concerns. We will do our best to address those. Your response will be greatly appreciated.
Thanks
I appreciate the authors' efforts and acknowledge that the rebuttal has addressed most of my concerns. I am inclined to increase my score.
Thank you. If there are any additional clarifications that we can provide, please do let us know!
We thank the reviewers for their insights and suggestions:
Q1/W2: Baselines with additional supervision: We provide three supervision-compatible baselines with the same poison supervision as L2U (labeled "Informed" below): the identical poison-aware detector, annotated set (60 / class), flagged set, and replay-buffer filtering. Every method is tested under independent poison, where each run crafts its own poisoned samples and test-time attacks, as in the original paper pipeline.
The detector improves MADAR and DEDUCE, but no informed baseline approaches L2U: the best reaches 74.4%, over 10 points below Heuristic-L2U (84.7%). Since all informed baselines receive the same detector, the remaining gap reflects how L2U uses the flagged samples (unlearning) rather than detection itself. Basic SSF's high Mean Acc reflects stability rather than adaptation: it learns from few labeled samples per task, so it retains older tasks but largely fails on recent ones (Task 10: 37.7%), also reflected in the table below.
W2: Attack Success Rate Evaluation: Thank you for pointing this out. We report the adversarial error rate at the final task: the percentage of attacked test samples from each task still misclassified by the final model. Lower indicates better resistance to poisoning.
Note:Joint conducts a full retrain on all clean data at every task, making it an upper-bound reference rather than a baseline.
Q2: Effect of Selective Forgetting: In the poisoned setting, we ablated unlearning, retaining a framework that only filters the data fed to our CL pipeline, fine-tuning with samples that are not flagged as perturbed/poisoned.
We see a similar trend in the natural drift setting with a Gaussian Naive Bayes Classifier (GNB) on EMBER 2018.
Q3/W1: Benefit of Meta-L2U: The primary advantage of Meta-L2U is generalizability, rather than superior performance. Heuristic-L2U relies on hand-designed rules, assuming that the donut's middle band is beneficial to forget. When that assumption does not hold, Meta-L2U serves as a reliable alternative that automatically learns what needs to be forgotten. Meta-L2U is meta-trained once offline (8 candidate scorers × 5 ES generations = 40 continual-learning runs), and at deployment the learned scorer requires only a forward pass to rank candidate samples. On EMBER 2018, Meta-L2U takes 4.92 GPU hrs on an NVIDIA RTX 6000 GPU, compared with 7.0 min for Heuristic-L2U, 73.9 min for DEDUCE and 1015.5 min (17 hrs) for MalCL. In the poisoned setting, Meta-L2U takes about 20.3 min on CPU, compared with 29.1 min for Heuristic-L2U, which trains all three unlearning lineages (Dropped Rows, Approximate Unlearning and Flip Label) in a single run. Thus, while Meta-L2U incurs additional training overhead, it offers the flexibility to adapt across data distributions without manually redesigning and validating selection heuristics for each setting. We meta-trained the learned selector on EMBER 2018 and tested it on EMBER 2024 (different distribution) without any further meta-training.