- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces RWFUBench to evaluate federated unlearning under cross-domain non-IID data. It examines requester forgetting and utility changes on retaining clients, compares several adapted federated unlearning methods, and proposes FedCCCU as a proof-of-concept mitigation strategy.
Critical strengths
The paper explores an interesting and relevant topic: federated unlearning under cross-domain heterogeneity, particularly its potential impact on retaining clients.
Critical weaknesses
-
The benchmark does not isolate the claimed unlearning-induced collateral damage. Section 3.1 acknowledges that retaining-client pre/post utility changes include ordinary training variation, yet no matched no-request trajectory is reported. The observed losses therefore cannot be attributed specifically to unlearning. This concern is compounded by the reliance on target-class suppression as an important forgetting measure, which Section 3.1 also acknowledges cannot establish removal of deleted-data influence.
-
The cross-method comparison lacks a common experimental basis. Methods start from different pre-unlearning models, and several are modified to satisfy Process Independence. Retrain-PI retrains only the requester while retaining clients continue from existing weights. Although Appendix F separately reports global retraining, this does not resolve the confounding in the main comparisons. Consequently, the reported rankings combine differences in initial models, method adaptations, and unlearning interventions.
Questions for authors
Can the authors provide a matched no-request training trajectory to distinguish unlearning-induced collateral damage from ordinary training variation? Section 3.1 acknowledges that the reported pre/post utility changes include normal training variation, so the current comparison does not isolate the effect of unlearning.
4 comments
To W2: A common experimental basis for cross-method comparison
All methods are evaluated on the same cross-domain tasks, client data assignments, unlearning requests, and metrics, with the common configurations provided in Section 4.2 and Table 2. To preserve each method’s original logic, we use its official code and make task and Process Independence adaptations within that implementation. We also follow each method’s training procedure. For example, FedAU trains an auxiliary unlearning module during learning, while Amnesiac requires historical updates corresponding to the trained model. Following these method-specific procedures produces different pre-unlearning FL models in some cases.
We assess retaining-client damage by the accuracy drop from each method’s own starting point, rather than by final accuracy alone. Table 3 reports the before-and-after results, and Figure 5 shows client- and class-level changes, . Drops for the requester’s target class reflect changes in target predictions; drops on retaining clients show utility loss. We assess these changes alongside MIA results on deleted samples to examine both unlearning performance and its cost to retaining clients.
The Process Independence adaptations enforce a common condition on client participation and are specified in Section 4.3. Amnesiac retains historical-update removal without requiring compensatory fine-tuning by retaining clients. FedCDP retains scoring, pruning, and fine-tuning, with scoring and fine-tuning restricted to the requester. These adaptations preserve the core mechanisms, and the reported results correspond to these implementations under Process Independence.
Retrain-PI follows this participation condition: the requester retrains from scratch, while retaining clients continue normal FL updates from existing weights. Retrain-PI evaluates retraining under Process Independence, whereas Retrain-All in Appendix F provides the complete-retraining reference. They serve distinct evaluation purposes.
Remaining Concern: Consistency Between Claims, Evaluation, and Implementation
Thank you for the detailed responses. They suggest that FedAU, DEPN, and FedCCCU involve no intervening FL updates, so their pre/post changes can be attributed to the edits.
My remaining concern is whether the paper's claims, evaluation, and implementation form a consistent chain of evidence.
First, the issue is not whether target accuracy and MIA are useful metrics, but how the paper uses them to interpret forgetting effectiveness. Section 3.1 acknowledges that target accuracy alone cannot establish removal of the deleted data's influence. Yet Section 5 lists high forget-set accuracy as shortcoming of existing methods and motivates a method that suppresses target predictions. The authors' Retrain-All results (Table R1, IR-9) retain 85.50% target accuracy, while FedCCCU reduces it to 18.00%, with similar MIA (52.41% vs. 51.37%). Strong target suppression is therefore not inherent to data deletion. What evidence supports interpreting the strength of this suppression, and its collateral effects on retaining clients, as a trade-off between effective unlearning and retaining-client utility, rather than as consequences of particular editing procedures?
Second, to better understand how FedCCCU is implemented, I examine a previously downloaded code snapshot. I then tried to revisit the repository to verify whether this was the latest version, but the link appeared inaccessible.
The snapshot appears inconsistent with the method described in the paper. The Utility Conflict Ratio in man_neruon.py skips clients with client_id < 5 but not using all retaining clients as in Eq. (6). The attribution in neuron_forget.py uses a single cross-entropy gradient–activation product but not the integrated target-class attribution of Eqs. (3)–(4). I also could not locate the MIA implementation in that snapshot. I cannot confirm whether this snapshot reflects the final experiments. Could the authors clarify which implementation produced the reported FedCCCU and MIA results?
Given the paper's role as a benchmark, I therefore remain concerned that the interpretation of target-class accuracy, the comparison of forgetting effectiveness across methods, and the scope of the paper's conclusions need to be reconsidered accordingly.
Thank you for raising this question. We agree that strong target suppression is not inherent to data deletion. Below, we clarify the evidence behind our trade-off framing.
The clearest evidence for this trade-off comes from the within-method comparison in Table 15. On IR-9, we fixed the number of candidate neurons at 30 and changed only the number edited, from 8 to 16. The requester’s target-class accuracy fell from 18.00% to 16.37%, a further drop of just 1.63 percentage points. Yet retaining clients’ average accuracy on the same class fell from 71.04% to 16.74%, a further drop of 54.30 percentage points.
Figure 5 shows this trade-off across methods. On IR-65, FedAU sharply reduced the requester’s target-class accuracy, but retaining clients also became much less accurate on that class. FedCCCU also reduced the requester’s target-class accuracy, with a smaller drop on retaining clients. The figure shows both how strongly each method suppresses target predictions on the requester and how much utility retaining clients lose in the process.
We would also like to clarify our wording in Section 5. When Section 5 describes unlearning as having a limited effect, we are referring to its effect on the model’s target-class predictions for the requester. After the request is processed, the requester’s target-class accuracy remains high and has dropped only slightly from its level before unlearning. Section 3.1 explains why we make this distinction. Other clients still hold data from the same class, so the model’s ability to recognize that class does not mean it still relies on the requester’s deleted data. Section 5 therefore asks whether we can focus the effect of unlearning more on the requester while preserving the model’s ability to recognize the same class on other clients.
Thank you for pointing out the issue with the link. The anonymous repository link expired during the review period, and we apologize for the difficulty this caused when you tried to check the code. The original link in the paper is now working again. Before this submission, we revised the paper and the implementation based on earlier feedback. We did not update the public repository at the time, so the snapshot you downloaded was an earlier version. We have now uploaded the latest code for reproducing the results in the paper. The client_id < 5 condition in the old snapshot was a temporary filter for an early test. We did not use it in the final experiments. In the reported FedCCCU experiments, we followed Eq. (6) by excluding only the requester and taking the maximum score across all retaining clients. For Eqs. (3)–(4), we integrated the gradient of the target-class probability over multiple steps as each neuron’s activation went from zero to its original value. We have also uploaded the MIA evaluation script, evaluate_mia.py. It uses the model after unlearning to compute the cross-entropy loss for each of the requester’s original target-class training samples and for each test sample from the same class. It predicts that a sample is a member if its loss is below the threshold. The reported ASR is the highest balanced accuracy obtained by trying different thresholds on those same samples.
To W1: Unlearning-induced utility loss and normal training variation
1. Attributing utility loss to unlearning rather than normal training variation
For FedAU, DEPN, and FedCCCU, the unedited model is the no-request control, and no normal FL updates occur between the pre- and post-edit evaluations. Their observed retaining-client losses therefore come from the unlearning edits. FedAU applies a linear parameter operation using its trained auxiliary module, while DEPN and FedCCCU edit neurons. FedAU’s public implementation explicitly follows “evaluate the original model, edit parameters, and evaluate the edited model,” with no training in between. Without the request, the model remains unchanged over this interval. The statement in Section 3.1 about “any intervening normal training variation” applies when such updates actually occur; it does not describe every method’s evaluation.
We follow the core mechanisms and procedures of the original papers and open-source implementations, with the task and Process Independence adaptations described in Section 4.3. For methods involving optimization or recovery, these updates are part of the prescribed deletion procedure. Ferrari performs unlearning by optimizing feature sensitivity, while FedCDP fine-tunes after pruning. We report performance after these operations. These steps are triggered by the deletion request, so their overall utility impact belongs to the unlearning procedure being evaluated. Gradient updates alone do not make them ordinary training variation unrelated to the request.
Retraining is the gold-standard reference because it trains a model without the deleted data, not because its accuracy never changes during training. Retrain-PI follows this retraining-based approach under Process Independence: the requester retrains from scratch after excluding the deleted data, while retaining clients continue normal FL updates from existing weights and participate in aggregation. We use this baseline to evaluate retraining under Process Independence, with training and aggregation forming part of the baseline itself. Appendix F also provides Retrain-All as the complete-retraining reference.
2. Assessing unlearning beyond target-class suppression
Our evaluation combines client-level prediction changes with privacy results on the deleted samples. Appendix H, Table 12 shows that FedCCCU reduces requester target accuracy while retaining clients maintain their ability to recognize the same class. For example, on IR-65’s apple class, requester accuracy falls from 62.86% to 44.16%, a decrease of 18.70 percentage points. Retaining-client accuracy falls from 71.11% to 68.61%, only 2.50 percentage points. Across the target classes in Table 12, requester accuracy consistently decreases more than retaining-client accuracy. These results show that the effect is not simply a model-wide loss of target-class recognition.
The MIA results provide separate privacy evidence. FedCCCU’s attack success rates are 51.37% on IR-9 and 52.53% on IR-65, close to Retrain-All’s 52.41% and 51.43% under the same attack. Thus, we observe both a stronger effect on requester predictions than on retaining-client predictions and membership leakage close to the retraining reference. We assess unlearning and utility preservation using these results together, rather than judging unlearning solely by whether requester predictions become incorrect.