- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a federated unlearning framework that handles three kinds of requests. A request can remove a whole client, a whole class, or the data that specific clients contributed to specific classes. The last case, client-class unlearning, is the main new setting. The method has two parts. The first part is weight negation. For each layer, it takes the mean gradient of the target data as a direction. It then scales the weight component along that direction by 1 − 2λ_l, where λ_l depends on layer sensitivity and selectivity. The second part, which the paper calls gradient residual correction or proxy filtering, replays stored client updates from a checkpoint. Each update is weighted by exp(−γι), where ι is the inner product between the update and the gradient of the target loss. For client unlearning, the target clients' updates are dropped during replay and the model is then calibrated on the remaining clients. For client-class unlearning, the gradients are computed on D_{k,c} and only the target clients' updates are filtered. Appendix C gives four theorems on smoothness, utility, a forgetting margin and membership inference. The experiments use MNIST, FMNIST and CIFAR10 with 5 or 10 clients under IID, non-IID and label-flip settings. Class unlearning is compared with FUSED, NoT, and FedAF using error rates, confusion matrices, F1 scores, run time and MIA metrics. Client and client-class results are shown for the proposed method only.
Critical strengths
-
Client-class unlearning is a useful problem. A realistic case is a client that has wrong labels for one or two classes. Removing the whole client throws away its clean data, and removing the whole class throws away what other clients contributed. The paper handles all three request types in one pipeline, and in the MNIST label-flip experiment in Fig. 5 the flipped class recovers after the request. This is the main reason the paper interests me.
-
On MNIST and FMNIST class unlearning, the method reaches 100% Error_f, and its Error_r after unlearning is 0.25% and 5.44%, lower than all three baselines in Table 1. Its MIA accuracy is close to 50% in Table 3. The paper also gives pseudo-code for all three modes and a full hyperparameter table, and it openly says that the method is not best on every metric.
Critical weaknesses
-
The method is not consistent as written, and the theory does not support it. The two phases are not connected in the pseudo-code. Algorithm 1 computes the negated model W^(0) in step 18, but replay starts from the checkpoint W_τ in step 9 and the algorithm returns the replay output. W^(0) is never used again. Algorithms 1.2 and 1.3 have the same problem. Appendix C writes the method as U = C(R(N(W))), yet the replay operator R in Eq. 29 starts from the checkpoint and not from N(W). So it is unclear what the experiments actually ran.
-
The paper says its unlearning needs no access to local training data. The abstract calls it data-independent and Appendix B calls negation data-free. Yet Eqs. 4, 10, 16 and 20 all need gradients on target samples, and Eq. 10 needs them again at every replay step. Phase 1 is called client-side, but Fig. 6 and line 903 have the server run the weight negation step. The paper never says who computes these gradients or on which data.
-
The theory relies on assumptions that do not fit the setting, and several steps do not follow. Theorem 2 assumes that every per-sample loss is strongly convex in W and that the original model is already close to the retrained one. Neither is shown for the neural networks in the experiments. The proof then treats L_r as Lipschitz because it is smooth, which does not follow. It also says that a larger β brings L_r as close as we like to L_r(W_r), but Eq. 43 keeps the fixed ε_replay + ε_train term. Theorem 3 assumes |1 − 2λ_l| ≥ κ, which does not make the coefficient negative. Setting λ_l = 0 meets this condition and leaves the model unchanged. Theorem 3 is also about the intermediate W^(0) and not about the model that is returned. In Theorem 4, Eq. 61 bounds the distance to the original model. This can at most show that the attack advantage after unlearning is close to the advantage before it, not that it is small. The step from Eq. 61 to the exp(−γῑ) + C2/η bound in Eq. 56 is not derived.
-
The experiments do not show that the unlearning works as claimed. There is no retrained model to compare against, although Eq. 2 defines the goal relative to W_r and Appendix E calls W_r the gold standard. There is also no ablation of negation, replay and calibration from one common starting model. Given W1.1, this ablation is needed to see which part does the forgetting.
-
The FedAF baseline is cited as Yang et al. 2024, but that reference is a time-series forecasting model, not an unlearning method. FedRecon is a paper on missing-modality reconstruction and is cited as unlearning work. In Fig. 2, classes 8 and 9 are already poorly recognized before client unlearning, which conflicts with the text around line 401. The algorithms contain leftover LaTeX color names such as lightyellow and lightblue. Appendix C contains the placeholder phrase "the reference example", and "Eq. equation" appears throughout.
Questions for authors
-
Who computes the gradients in Eqs. 4, 10, 16 and 20, and on which data? How does the server get target gradients at each replay step without holding target data?
-
Which checkpoint τ was used for t* = 25 and t* = 40, and was the replay window empty? For client unlearning, how is the influence of the target clients before τ removed?
-
Can you add results for a model retrained from scratch, covering Error_f, Error_r, Error_t, MIA and time for all three request types? Can you also add an ablation of negation, replay and calibration from the same starting model?
-
Did all methods start from the same model? Why does FUSED barely change on CIFAR10, why is FedAF missing there, and which FedAF implementation did you use, given that the cited paper is about forecasting?
5 comments
Q3: All configurations start from the same converged model and are compared with a model retrained from scratch on the retained data for the same number of rounds. For CIFAR-10 class unlearning (class 3):
| Method | Forget Error (%) | Retained Error (%) | Test Error (%) | MIA AUC | Time |
|---|---|---|---|---|---|
| Before unlearning | 16.82 | 4.86 | 18.47 | 0.648 | — |
| Retrain from scratch | 100.00 | 4.38 | 23.40 | 0.495 | 10,991 |
| Calibration only | 100.00 | 3.88 | 23.04 | 0.538 | 1,721 |
| Negation + calibration | 100.00 | 9.14 | 24.29 | 0.499 | 890 |
| Replay only | 100.00 | 4.44 | 23.41 | 0.492 | 7,889 |
| Replay + calibration | 100.00 | 4.11 | 23.25 | 0.497 | 8,769 |
| Replay + negation + calibration | 100.00 | 9.80 | 23.95 | 0.488 | 8,767 |
The ablation shows which part does the forgetting. Calibration alone removes the class from the predictions, but the attack still ranks members above non-members (MIA AUC 0.538 vs. 0.495 for retraining), even though its advantage at the default threshold is zero. Replay removes this remaining signal, and replay with calibration matches retraining on every metric. Negation with calibration reaches complete forgetting at about one twelfth of retraining time, at a cost of about 5 points in Error_r. Our framework therefore offers two operating points: replay with calibration for the closest match to retraining, and negation with calibration for complete forgetting at low cost. Client unlearning follows the same pattern. Replay from round 0 raises the forgotten client’s Error_f from 6.98% to 19.71% (retraining: 20.26%), keeps Error_r at 6.32% (7.47%), and lowers MIA AUC from 0.591 to 0.501 (0.494), at about 80% of retraining time (8,691 s vs. 10,724 s).
Client-class unlearning. Two of ten clients hold half of all class-3 training images, relabeled as class 5, while the other clients keep class 3 correctly labeled. Before unlearning, the model predicts class 5 on 76.0% of the flipped images, and the corruption spreads to all clients (class-3 test error 89.6%):
| Method | Agreement with Flipped Label (%) | Class-3 Test Error (%) | Retained Error (%) | MIA AUC | Time |
|---|---|---|---|---|---|
| Before unlearning | 75.96 | 89.6 | 11.51 | 0.582 | — |
| Retrain from scratch | 33.92 | 70.9 | 13.07 | 0.526 | 8,464 |
| Calibration only | 43.40 | 65.6 | 9.82 | 0.561 | 1,591 |
| Negation + calibration | 43.52 | 73.8 | 16.79 | 0.543 | 863 |
| Replay only | 33.60 | 74.2 | 13.90 | 0.521 | 7,827 |
| Replay + calibration | 38.16 | 74.3 | 12.98 | 0.522 | 8,531 |
| Replay + negation + calibration | 43.40 | 81.9 | 18.88 | 0.526 | 8,715 |
Replay removes the flipped contribution to the level of retraining: replay alone reaches agreement 33.6% (retraining: 33.9%) and MIA AUC 0.521 (0.526), and replay with calibration reaches 38.2% and 0.522 while keeping retained error at 12.98% (13.07%). Calibration alone removes most of the flip at about one fifth of the cost but leaves more membership signal (AUC 0.561), and adding negation increases retained error, as in class unlearning. Class 3 recovers only partly, even for retraining, because the benign clients hold only 2,500 class-3 images.
Q4: In the submitted experiments, each baseline was trained in its own federation. We now run every method from the same converged model, with the same evaluation code, and compare each with retraining from scratch. On CIFAR-10 class unlearning (class 3):
| Method | Forget Error (%) | Retained Error (%) | Test Error (%) | MIA AUC | Time |
|---|---|---|---|---|---|
| Before unlearning | 16.82 | 4.86 | 18.47 | 0.648 | — |
| Retrain from scratch | 100.00 | 4.38 | 23.40 | 0.496 | 10,991 |
| FUSED | 47.16 | 3.45 | 19.16 | 0.650 | 2,375 |
| FedAF | 100.00 | 86.30 | 87.78 | 0.508 | 126 |
| Class-discriminative pruning | 100.00 | 6.13 | 24.40 | 0.543 | 368 |
| Replay + calibration (ours) | 100.00 | 4.11 | 23.25 | 0.498 | 8,769 |
| Negation + calibration (ours) | 100.00 | 9.14 | 24.29 | 0.500 | 890 |
FUSED. Under this protocol, with its trained adapters applied before every metric, FUSED does change the model: it removes about half of the target class (Error_f 47.16%) while preserving retained accuracy, but leaves membership detectable (MIA AUC 0.650).
FedAF. refers to Li et al., “Class-wise Federated Unlearning: Harnessing Active Forgetting with Teacher-Student Memory Generation” (arXiv:2307.03363); we have corrected the citation. It was missing on CIFAR-10 because our implementation initially covered only the MNIST-scale model. As no official code is available, we implemented it following the paper (teacher-generated memories and the sketched EWC preserver), with its specified schedule and hyperparameters, and it is now included on CIFAR-10. At the learning rate its authors specify for CIFAR-10, it forgets the class but loses most retained accuracy on our converged model.
We also added class-discriminative pruning (Wang et al., 2022), a strong class-level method that forgets fully at low cost; it is designed for class removal only, whereas our framework handles client, class, and client-class requests within one pipeline. Among all methods, replay with calibration is the closest to retraining overall.
Thank you for the careful review and for recognizing client-class unlearning as a useful problem, along with the unified handling of all three request types. Your comments helped us clarify how the stages of our method connect, who computes what during unlearning, and how our method compares with retraining and with baselines started from the same model.
Q1: Our procedure needs no gradients on forget data, and the server never needs target gradients during replay. This is a key privacy property: a client can delete its forget samples as soon as it submits a request. Eqs. (4), (10), (16), and (20) give the general gradient-based form of the two stages; in our experiments, both stages are computed without target gradients.
Weight negation (Eqs. 4, 20): the server flips the sign of the selected layers of the global model, as shown in Fig. 6. This uses no data at all.
Replay weights (Eq. 10, 16): the server weights each client’s stored update by that client’s share of target labels, computed from per-class label histograms reported during training. The weights are fixed before replay begins, so no gradient is required at any replay step.
The only information the server needs beyond standard federated updates is these label counts. We will present this gradient-free form as the main formulation in Section 3 and describe negation as server-side throughout.
Q2: The replay window is not empty. We use the last checkpoint strictly before the request: τ=20 for t^⋆=25 and τ=35 for t^⋆=40, as in Algorithm 1.1 and Appendix E, so replay covers five rounds. We will correct the “≤” in Section 3 to “<”.
Rounds before the checkpoint are already part of it, so the choice of τ trades cost against fidelity. Starting replay from round 0 removes the target clients’ influence from every round: the history is rebuilt from the initial model, which no client has influenced, with the target clients excluded at every round and each remaining client’s stored update recalibrated by a short local training run, following FedEraser (Liu et al., 2021). On a converged CIFAR-10 model, this matches retraining from scratch:
These results use the fully federated procedure, in which the target clients take no part in unlearning; in our earlier client-unlearning experiments, the calibration step was simulated centrally over all clients’ data.