- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the joint problem of continual learning and machine unlearning, where a fixed-capacity model must learn new tasks, retain useful knowledge, and remove the influence of specified past training data. The proposed method, LwU, divides the network parameters into four regions based on their importance to the retain and forget sets, and uses parameter protection and resetting, gradient projection in the conflict region, and teacher distillation to support knowledge retention, deletion, and subsequent learning. Experiments evaluate continual learning and unlearning separately on image classification and haptic datasets. The method is also tested for content suppression in diffusion models, but this extension does not include continual learning experiments.
Critical strengths
The problem setting is relevant to the practical needs of long-term model deployment.
Critical weaknesses
My main concerns are the method’s technical novelty and the limited experimental evidence. Estimating parameter importance from gradients and using it to select and partition parameters is already common practice. The paper needs to explain what the four-region design adds to existing importance-based methods and how it improves the joint handling of continual learning and machine unlearning.
Most experiments evaluate the two separately, and the joint experiments are largely limited to a small-scale Task-IL setting on Split CIFAR-10. This is not enough to support the main claims of the framework. I would like to see comparisons with existing joint methods and reasonable combinations of CL and MU methods across more varied sequences of interleaved learning and deletion, to determine whether the unified design offers a practical advantage.
Questions for authors
Please refer to the weakness.
3 comments
Response (2/4)
"Estimating parameter importance from gradients is common practice"
We agree, and we use the simplest estimator, diagonal Fisher, on purpose. The contribution is what is built on it. Existing importance-based methods compute one score to make one binary decision: protect or free in CL (EWC, SI, WSN, NISPA), dampen or mask in MU (SSD, SalUn). LwU contributes:
- A synapse-level formulation of the joint problem. Retained and forgotten knowledge share parameters, so the CL–MU conflict must be resolved per weight, not per task with buffers (UniCLUN) or per module (CLPU).
- Dual scoring with dominance, giving four zones and four update rules: freeze (A), reset (B), retain-orthogonal ascent (C), plastic learning (D). One estimator, two scores, four actions.
- Coupled deletion and learning. Reset Zone B weights rejoin the plastic pool, so deletion frees capacity for the next task, and the next task overwrites what the reset leaves. A hard reset is too destructive for standalone MU and has no role in CL; only the joint setting makes it affordable.
- Retain-orthogonal ascent on contested weights, with a first-order guarantee on the retain loss and no forget–retain trade-off coefficient. The measured cosine to the retain gradient is at most 3e-7 (Tab. 16).
- One masked privileged teacher for both objectives. It preserves old classes during learning and repairs retained behavior after deletion, and because it is conditioned on retained classes it never supplies a target on the forget set.
- Buffer-free throughout. For unlearning this matters: a replay buffer can hold the very data a user later asks to delete.
Every component is ablated
- Partition vs. estimator (Tab. 14–15). With everything else fixed, a single Fisher map leaves forget accuracy at 42.4% (CNN) and 78.9% (ResNet-18). Dual scoring with four zones reaches 0.0% on both; the gap to Retrain falls from 16.3 to 1.0 and from 28.5 to 0.8. Random and magnitude scores are worse still. The gain comes from the partition, not the estimator.
- Against SSD, which also uses diagonal Fisher (Tab. 2): U-LiRA AUC of 50.9 vs. 75.9 (CIFAR-10, ResNet-18) and 51.6 vs. 87.9 (CIFAR-100, ResNet-18).
- Zone B reset (Tab. 13): removing it raises the gap to Retrain from 1.0 to 13.6, 0.8 to 27.8, and 0.5 to 27.2.
- Dominance (Tab. 5, 11–12): removing the margin drops retain accuracy to 31.8 (CNN) and 52.5 (ResNet-18); in the joint stream ACC falls from 95.9 to 87.1 and BWT from +0.4 to −9.5.
- Teacher masking (Tab. 5): with an unmasked teacher the deleted class survives at 95.0% in the joint stream.
- Components across 5 settings (Tab. 6–10, 13), sensitivity (Tab. 11–12), and test-time memory (Tab. 17).
- Negative results (App. D): 14 alternatives we tried and rejected, including binary partitioning, forget-only scoring, permanent dampening, and a single weighted retain–forget objective. Each design choice in LwU replaces a failure we measured.
Soundness
The review rates soundness 1 ("significant concerns that do not appear easy to address") but does not identify an incorrect claim, derivation, or experimental flaw; both weaknesses concern novelty and scope. We would be grateful if the reviewer could point to the soundness issue, or reconsider that score.
A request
A Clear Rejection largely decides the outcome regardless of the other reviews. We therefore respectfully ask that it rest on concerns we can act on: (a) which existing importance-based method the reviewer considers equivalent to the four-zone design, and (b) which joint baselines, CL+MU compositions, or deletion schedules would change the assessment. We will run what is named within the discussion period. Please note that the paper already includes 15 baselines and a unified CL+MU baseline in its comparisons.
Response (3/4): new results on CIFAR-100
As promised, we ran the joint protocol on CIFAR-100. Setup: 5 tasks × 20 classes, ResNet-18, Task-IL, 3 seeds. One class c_f is deleted after task 3, then tasks 4–5 are learned. F_del is c_f accuracy right after deletion, F_end after task 5.
| Method | ACC ↑ | BWT ↑ | F_del ↓ | F_end ↓ |
|---|---|---|---|---|
| Retrain (never sees c_f) | 71.75±0.73 | −2.95±0.11 | – | 0.00±0.00 |
| LwU, no deletion | 72.65±1.20 | −2.78±0.32 | – | 71.92±13.65 |
| CLPU-DER++ | 65.81±1.25 | −15.63±1.82 | 0.00±0.00 | 0.00±0.00 |
| LwU | 73.65±1.13 | −1.45±0.67 | 0.00±0.00 | 0.00±0.00 |
Varied interleaving. Same stream; the default retain subset is 10%.
| Setting | ACC ↑ | BWT ↑ | F_del ↓ | F_end ↓ |
|---|---|---|---|---|
| Delete after task 2 | 73.81±1.09 | −1.59±0.47 | 0.00±0.00 | 0.00±0.00 |
| Delete after task 3 | 73.65±1.13 | −1.45±0.67 | 0.00±0.00 | 0.00±0.00 |
| Delete after task 4 | 74.12±1.32 | −0.82±0.60 | 0.00±0.00 | 0.00±0.00 |
| Two requests (after 2 and 4) | 74.28±1.11 | −1.14±0.88 | 0.00±0.00 | 0.00±0.00 |
| Retain subset 1% | 71.80±1.52 | −3.90±0.98 | 0.00±0.00 | 0.00±0.00 |
| Retain subset 5% | 73.35±1.43 | −1.69±0.72 | 0.00±0.00 | 0.00±0.00 |
| Retain subset 25% | 73.75±1.02 | −1.17±0.87 | 0.00±0.00 | 0.00±0.00 |
what the four-zone design adds, measured in the joint stream
The reviewer asked what the four-zone design adds to existing importance-based methods. A protect-or-free method computes one score and makes one decision per weight. LwU adds a dominance margin between two scores, a reset zone, and a teacher that both preserves old classes and repairs retained behavior after deletion. Removing each of them on Split CIFAR-100 (same protocol as 1/3) shows what it contributes.
Continual learning only (no deletion)
| Variant | ACC ↑ | BWT ↑ |
|---|---|---|
| w/o self-distillation teacher | 45.78±1.52 | −36.72±0.55 |
| w/o cumulative freeze mask | 72.00±1.48 | −5.75±1.43 |
| LwU | 72.65±1.20 | −2.78±0.32 |
Joint stream (delete c_f after task 3)
| Variant | ACC ↑ | BWT ↑ | F_del ↓ | F_end ↓ |
|---|---|---|---|---|
| No dominance margin (ω = 1) | 34.15±0.65 | −45.08±0.59 | 0.00±0.00 | 0.00±0.00 |
| w/o self-distillation teacher | 48.80±2.95 | −32.91±3.45 | 0.00±0.00 | 0.00±0.00 |
| w/o teacher repair | 71.12±1.56 | −4.54±1.27 | 6.05±3.86 | 8.41±8.14 |
| w/o Zone B reset | 73.84±1.05 | −1.31±0.58 | 7.69±10.10 | 2.87±4.31 |
| w/o Zone-C repair pass | 73.01±0.81 | −2.15±0.56 | 0.00±0.00 | 0.00±0.00 |
| w/o cumulative freeze mask | 73.97±0.12 | −3.30±0.13 | 0.00±0.00 | 0.00±0.00 |
| LwU | 73.65±1.13 | −1.45±0.67 | 0.00±0.00 | 0.00±0.00 |
The dominance result matches Split CIFAR-10 (Tab. 5: ACC 95.9 → 87.1), now at 100 classes with 20 per task.
Response (1/4)
We thank the reviewer. We respectfully disagree that the experimental evidence is limited.
LwU is a single unified method, and every experiment below evaluates that same model. We compare it against 15 baselines (EWC, SI, LwF, SpaceNet, WSN, NISPA, PEC, SNV, Amnesiac, UNSIR, SSD, SCRUB, Bad Teacher, SalUn, and the unified CL–MU method UniCLUN) and 5 reference models (joint training, SGD, Original, Fine-tune, Retrain), on CIFAR-10, CIFAR-100, Tiny-ImageNet, our real-world haptic dataset RTI, and Stable Diffusion v1.4. The unified baseline UniCLUN is evaluated on all four classification datasets. LwU is the only method that places on both overall podiums (Tab. 1 and Tab. 2).
The submission is 57 pages, most of it experiments:
"Limited" is not a fair description of this. However, as reviewers asked, we are running the same ablation in the paper on CIFAR-100 and will publish it here before the discussion phase. We will add the following ASAP: the interleaved protocol on Seq-CIFAR-100 and RTI, Task-IL and Class-IL, against UniCLUN and composed pipelines (LwF + SSD, LwF + SalUn, EWC + SCRUB), with 1–3 deletion requests at different stream positions. Class-IL was left out of Tab. 5 because on 2-class Split CIFAR-10 old tasks fall to 0% without replay; on Seq-CIFAR-100 LwU keeps 33.2% Class-IL accuracy (Tab. 1), so deletion is measurable there.