- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work explores machine unlearning using forward-forward (FF) algorithms as an alternative to the predominant back-propagation (BP) unlearning literature. This paper discusses and resolves two FF-specific challenges: (i) layers lack coordinated yet layer-specific unlearning targets, and (ii) the balance between under- and over-forgetting differs across layers. The paper also proposes G-MIA: an MLP membership classifier trained on temperature-scaled layer-wise goodness vectors, to help better evaluate FF unlearning quality.
Critical strengths
-
Clear diagnosis of a concrete failure mode. The layer-wise CKA analysis (Tab. 26) shows that output-only unlearning leaves FF layers 1–12 essentially unchanged, even though the predictor still relies on those layers. It also shows that naive per-layer extensions over-perturb every layer. This is a useful caution for any model trained with layer-local objectives.
-
The audit is well validated. Table 14 reports AUCs for retained vs. unseen and retained vs. forgotten samples. This rules out the trivial explanation that the frozen attacker simply fails across the board after unlearning. Table 12 cross-checks G-MIA against eight other membership inference attacks (MIAs).
Critical weaknesses
-
The case for FF unlearning is not made. FF is still a research-stage training method with accuracy gap to BP. The paper’s own Tab. 27 reports 82.2% vs. 87.0% test accuracy for VGG-13/CIFAR-10. I am not aware of FF-trained models in deployment. The paper motivates the problem with edge devices, yet every experiment runs on A800 GPUs, and Appendix I states that the memory measurements do not reflect edge hardware. There can be scenarios for unlearning where BP isn't available either, but the authors should discuss and test on those settings. FF-Erase also needs access to the full retained set and a held-out validation set, which weakens the on-device argument.
-
The technical novelty over existing unlearning methods is limited. Challenge (i) is the problem that reference-guided and teacher-based unlearning already address. ReGUn, which the paper uses as a baseline, already distills forget samples toward a reference distribution built from disjoint held-out data. Bad Teacher distills forget samples from an incompetent, randomly initialized teacher. FF goodness vectors act as per-layer class-logit heads. FF-Erase is therefore essentially reference-model distillation applied at every head, with per-sample targets from a model trained on held-out data. Challenge (ii) is handled by gradient-norm balancing in the style of GradNorm and MetaBalance. The paper should clarify what is fundamentally FF-specific beyond applying these components layer by layer.
-
The claim that BP methods are “incompatible” with FF is not well tested. The paper considers only two adaptations: a. “direct” adaptation, which according to Table 26 changes only the final layer; b. “layer-wise” adaptation, with inputs detached between layers. End-to-end BP unlearning through the whole FF network, on the aggregated predictor or on the sum of layer losses, is missing. The baselines also appear to be at a disadvantage:All methods share one stopping rule, based on relative improvement of the local objective. This rule does not stop gradient-ascent methods before collapse. Standard practice is to stop when forget-set accuracy approaches held-out accuracy. Layer-wise SCRUB has an explicit retain-set term, yet it collapses to exactly chance level in every setting. Learning-rate sweeps are given only for GA and ReGUn. Standard baselines are absent: fine-tuning on the retained data alone, NegGrad+, SalUn, and the training-free SSD.
-
Several reported statistics raise reproducibility concerns. In Table 6 (VGG-13/CIFAR-100), the relative standard deviations (RSDs) equal those in Table 3 (VGG-13/CIFAR-10) plus exactly 0.1 in essentially every cell. The same +0.1 relation holds between Table 7 (MNIST) and Table 8 (Fashion-MNIST) in 54 of 57 cells. It also holds between Table 6 and Table 9 (ResNet-50) for all rows other than the layer-wise ones. With five seeds, dispersions measured independently on different datasets and architectures would not line up this way? The MNIST shadow model reportedly takes 982.7s to train, while the MNIST target model takes 436.6s (Table 7). Training the Cross-Domain guidance model on about 5k CIFAR-100 images takes 953.7 s, compared with 57.1 s for Val-Train on 5k CIFAR-10 images...Taken together, these make it hard to tell whether the reported variability and costs were measured as described at this point.
Questions for authors
Please refer to Critical Weaknesses section. Also 1. How does FF-Erase compare with end-to-end BP-based unlearning (e.g., SCRUB, ReGUn, SalUn) applied through all layers of the FF network? In what concrete deployment setting is backpropagation unavailable at unlearning time? 2. What is the guidance model’s test accuracy at each validation split (1–10%)? How does FF-Erase compare with layer-wise ReGUn when ReGUn is given the same per-sample guidance targets?