acceptodds
Under review as a conference paper at ICLR 2027

RWFUBench: A Cross-Domain Benchmark for Revealing Collateral Damage in Federated Unlearning

Abstract

Machine unlearning is essential for enforcing data deletion rights and is increasingly demanded in Federated Learning (FL). However, existing studies largely overlook a critical requirement, Unlearning Independence, which requires that uninvolved clients incur no extra unlearning computation or utility loss. This issue is further exacerbated in Cross-Domain Non-IID (CD-NonIID) settings, where clients share the same task and label space but draw data from different sources. Current evaluations, however, predominantly rely on simplified Pseudo-NonIID assumptions. To address this gap, we present a systematic benchmark named RWFUBench for the task of class unlearning in FL under CD-NonIID conditions. Our results reveal a tension between requester target suppression and retaining client utility. Retraining based methods retain high target accuracy consistent with cross-domain knowledge replenishment, while several model editing methods also reduce target accuracy on uninvolved clients. These findings show that aggregate evaluations can overlook collateral damage in federated unlearning. We further provide FedCCCU as a Proof-of-Concept mitigation strategy that reduces collateral damage, supporting future research on Unlearning Independence.

Primary area:
datasets and benchmarks
Keywords:
Federated Unlearning, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HH86OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces RWFUBench to evaluate federated unlearning under cross-domain non-IID data. It examines requester forgetting and utility changes on retaining clients, compares several adapted federated unlearning methods, and proposes FedCCCU as a proof-of-concept mitigation strategy.

Critical strengths

The paper explores an interesting and relevant topic: federated unlearning under cross-domain heterogeneity, particularly its potential impact on retaining clients.

Critical weaknesses
  1. The benchmark does not isolate the claimed unlearning-induced collateral damage. Section 3.1 acknowledges that retaining-client pre/post utility changes include ordinary training variation, yet no matched no-request trajectory is reported. The observed losses therefore cannot be attributed specifically to unlearning. This concern is compounded by the reliance on target-class suppression as an important forgetting measure, which Section 3.1 also acknowledges cannot establish removal of deleted-data influence.

  2. The cross-method comparison lacks a common experimental basis. Methods start from different pre-unlearning models, and several are modified to satisfy Process Independence. Retrain-PI retrains only the requester while retaining clients continue from existing weights. Although Appendix F separately reports global retraining, this does not resolve the confounding in the main comparisons. Consequently, the reported rankings combine differences in initial models, method adaptations, and unlearning interventions.

Questions for authors

Can the authors provide a matched no-request training trajectory to distinguish unlearning-induced collateral damage from ordinary training variation? Section 3.1 acknowledges that the reported pre/post utility changes include normal training variation, so the current comparison does not isolate the effect of unlearning.

4 comments

To W1: Unlearning-induced utility loss and normal training variation

1. Attributing utility loss to unlearning rather than normal training variation

For FedAU, DEPN, and FedCCCU, the unedited model is the no-request control, and no normal FL updates occur between the pre- and post-edit evaluations. Their observed retaining-client losses therefore come from the unlearning edits. FedAU applies a linear parameter operation using its trained auxiliary module, while DEPN and FedCCCU edit neurons. FedAU’s public implementation explicitly follows “evaluate the original model, edit parameters, and evaluate the edited model,” with no training in between. Without the request, the model remains unchanged over this interval. The statement in Section 3.1 about “any intervening normal training variation” applies when such updates actually occur; it does not describe every method’s evaluation.

We follow the core mechanisms and procedures of the original papers and open-source implementations, with the task and Process Independence adaptations described in Section 4.3. For methods involving optimization or recovery, these updates are part of the prescribed deletion procedure. Ferrari performs unlearning by optimizing feature sensitivity, while FedCDP fine-tunes after pruning. We report performance after these operations. These steps are triggered by the deletion request, so their overall utility impact belongs to the unlearning procedure being evaluated. Gradient updates alone do not make them ordinary training variation unrelated to the request.

Retraining is the gold-standard reference because it trains a model without the deleted data, not because its accuracy never changes during training. Retrain-PI follows this retraining-based approach under Process Independence: the requester retrains from scratch after excluding the deleted data, while retaining clients continue normal FL updates from existing weights and participate in aggregation. We use this baseline to evaluate retraining under Process Independence, with training and aggregation forming part of the baseline itself. Appendix F also provides Retrain-All as the complete-retraining reference.

2. Assessing unlearning beyond target-class suppression

Our evaluation combines client-level prediction changes with privacy results on the deleted samples. Appendix H, Table 12 shows that FedCCCU reduces requester target accuracy while retaining clients maintain their ability to recognize the same class. For example, on IR-65’s apple class, requester accuracy falls from 62.86% to 44.16%, a decrease of 18.70 percentage points. Retaining-client accuracy falls from 71.11% to 68.61%, only 2.50 percentage points. Across the target classes in Table 12, requester accuracy consistently decreases more than retaining-client accuracy. These results show that the effect is not simply a model-wide loss of target-class recognition.

The MIA results provide separate privacy evidence. FedCCCU’s attack success rates are 51.37% on IR-9 and 52.53% on IR-65, close to Retrain-All’s 52.41% and 51.43% under the same attack. Thus, we observe both a stronger effect on requester predictions than on retaining-client predictions and membership leakage close to the retraining reference. We assess unlearning and utility preservation using these results together, rather than judging unlearning solely by whether requester predictions become incorrect.

To W2: A common experimental basis for cross-method comparison

All methods are evaluated on the same cross-domain tasks, client data assignments, unlearning requests, and metrics, with the common configurations provided in Section 4.2 and Table 2. To preserve each method’s original logic, we use its official code and make task and Process Independence adaptations within that implementation. We also follow each method’s training procedure. For example, FedAU trains an auxiliary unlearning module during learning, while Amnesiac requires historical updates corresponding to the trained model. Following these method-specific procedures produces different pre-unlearning FL models in some cases.

We assess retaining-client damage by the accuracy drop from each method’s own starting point, rather than by final accuracy alone. Table 3 reports the before-and-after results, and Figure 5 shows client- and class-level changes, Δ=Accbefore−Accafter\Delta=\mathrm{Acc}_{\mathrm{before}}-\mathrm{Acc}_{\mathrm{after}}. Drops for the requester’s target class reflect changes in target predictions; drops on retaining clients show utility loss. We assess these changes alongside MIA results on deleted samples to examine both unlearning performance and its cost to retaining clients.

The Process Independence adaptations enforce a common condition on client participation and are specified in Section 4.3. Amnesiac retains historical-update removal without requiring compensatory fine-tuning by retaining clients. FedCDP retains scoring, pruning, and fine-tuning, with scoring and fine-tuning restricted to the requester. These adaptations preserve the core mechanisms, and the reported results correspond to these implementations under Process Independence.

Retrain-PI follows this participation condition: the requester retrains from scratch, while retaining clients continue normal FL updates from existing weights. Retrain-PI evaluates retraining under Process Independence, whereas Retrain-All in Appendix F provides the complete-retraining reference. They serve distinct evaluation purposes.

Reviewer_HH86OpenReview ↗

Remaining Concern: Consistency Between Claims, Evaluation, and Implementation

Thank you for the detailed responses. They suggest that FedAU, DEPN, and FedCCCU involve no intervening FL updates, so their pre/post changes can be attributed to the edits.

My remaining concern is whether the paper's claims, evaluation, and implementation form a consistent chain of evidence.

First, the issue is not whether target accuracy and MIA are useful metrics, but how the paper uses them to interpret forgetting effectiveness. Section 3.1 acknowledges that target accuracy alone cannot establish removal of the deleted data's influence. Yet Section 5 lists high forget-set accuracy as shortcoming of existing methods and motivates a method that suppresses target predictions. The authors' Retrain-All results (Table R1, IR-9) retain 85.50% target accuracy, while FedCCCU reduces it to 18.00%, with similar MIA (52.41% vs. 51.37%). Strong target suppression is therefore not inherent to data deletion. What evidence supports interpreting the strength of this suppression, and its collateral effects on retaining clients, as a trade-off between effective unlearning and retaining-client utility, rather than as consequences of particular editing procedures?

Second, to better understand how FedCCCU is implemented, I examine a previously downloaded code snapshot. I then tried to revisit the repository to verify whether this was the latest version, but the link appeared inaccessible.

The snapshot appears inconsistent with the method described in the paper. The Utility Conflict Ratio in man_neruon.py skips clients with client_id < 5 but not using all retaining clients as in Eq. (6). The attribution in neuron_forget.py uses a single cross-entropy gradient–activation product but not the integrated target-class attribution of Eqs. (3)–(4). I also could not locate the MIA implementation in that snapshot. I cannot confirm whether this snapshot reflects the final experiments. Could the authors clarify which implementation produced the reported FedCCCU and MIA results?

Given the paper's role as a benchmark, I therefore remain concerned that the interpretation of target-class accuracy, the comparison of forgetting effectiveness across methods, and the scope of the paper's conclusions need to be reconsidered accordingly.

Thank you for raising this question. We agree that strong target suppression is not inherent to data deletion. Below, we clarify the evidence behind our trade-off framing.

The clearest evidence for this trade-off comes from the within-method comparison in Table 15. On IR-9, we fixed the number of candidate neurons at 30 and changed only the number edited, from 8 to 16. The requester’s target-class accuracy fell from 18.00% to 16.37%, a further drop of just 1.63 percentage points. Yet retaining clients’ average accuracy on the same class fell from 71.04% to 16.74%, a further drop of 54.30 percentage points.

Figure 5 shows this trade-off across methods. On IR-65, FedAU sharply reduced the requester’s target-class accuracy, but retaining clients also became much less accurate on that class. FedCCCU also reduced the requester’s target-class accuracy, with a smaller drop on retaining clients. The figure shows both how strongly each method suppresses target predictions on the requester and how much utility retaining clients lose in the process.

We would also like to clarify our wording in Section 5. When Section 5 describes unlearning as having a limited effect, we are referring to its effect on the model’s target-class predictions for the requester. After the request is processed, the requester’s target-class accuracy remains high and has dropped only slightly from its level before unlearning. Section 3.1 explains why we make this distinction. Other clients still hold data from the same class, so the model’s ability to recognize that class does not mean it still relies on the requester’s deleted data. Section 5 therefore asks whether we can focus the effect of unlearning more on the requester while preserving the model’s ability to recognize the same class on other clients.

Thank you for pointing out the issue with the link. The anonymous repository link expired during the review period, and we apologize for the difficulty this caused when you tried to check the code. The original link in the paper is now working again. Before this submission, we revised the paper and the implementation based on earlier feedback. We did not update the public repository at the time, so the snapshot you downloaded was an earlier version. We have now uploaded the latest code for reproducing the results in the paper. The client_id < 5 condition in the old snapshot was a temporary filter for an early test. We did not use it in the final experiments. In the reported FedCCCU experiments, we followed Eq. (6) by excluding only the requester and taking the maximum score across all retaining clients. For Eqs. (3)–(4), we integrated the gradient of the target-class probability over multiple steps as each neuron’s activation went from zero to its original value. We have also uploaded the MIA evaluation script, evaluate_mia.py. It uses the model after unlearning to compute the cross-entropy loss for each of the requester’s original target-class training samples and for each test sample from the same class. It predicts that a sample is a member if its loss is below the threshold. The reported ASR is the highest balanced accuracy obtained by trying different thresholds on those same samples.

Reviewer_g7bAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RWFUBench, a benchmark for evaluating class-level federated unlearning under cross-domain non-IID (CD-NonIID) data distributions. The benchmark emphasizes Unlearning Independence, which requires minimizing additional computation and utility degradation for clients not requesting unlearning. By constructing heterogeneous federated scenarios from multiple image datasets, RWFUBench evaluates existing unlearning approaches in terms of forgetting effectiveness, privacy leakage, and retaining-client utility. The experimental analysis identifies potential conflicts between suppressing target-class knowledge and preserving the performance of uninvolved clients.

Critical strengths

The paper investigates an important practical challenge in federated unlearning by emphasizing the impact of unlearning requests on uninvolved clients. The proposed RWFUBench extends conventional evaluations beyond synthetic label-distribution heterogeneity by considering cross-domain feature shifts. Its distinction between Process Independence and Utility Independence provides a useful framework for examining computational overhead and unintended utility degradation.

Critical weaknesses

W1. The motivation for Unlearning Independence is insufficiently convincing. Sections 1–2 argue that uninvolved clients should not participate in additional unlearning computation. However, knowledge learned from the requesting client's data can propagate to other clients through shared global model parameters. Consequently, removing this influence may require coordinated updates involving uninvolved clients. The paper should clarify whether strict Process Independence is theoretically achievable without compromising unlearning effectiveness and better justify this requirement.

W2. The proposed Unlearning Independence dilemma lacks formal theoretical analysis. Section 3.1 introduces Process Independence and Utility Independence primarily through computational cost and accuracy comparisons. However, these definitions do not establish the conditions under which the two requirements conflict or whether their simultaneous satisfaction is feasible. A formal analysis characterizing this trade-off under cross-domain heterogeneity would strengthen the theoretical foundation.

W3. The research problem and proposed methodology require clearer formalization. Sections 3–5 introduce RWFUBench and FedCCCU, but the relationship between the unlearning objective, independence constraints, and optimization procedure remains insufficiently explicit. In particular, the paper does not formulate a unified objective balancing requester forgetting and retaining-client utility. A clearer problem definition and methodological formulation would improve technical clarity.

W4. The experimental coverage and evaluation setup are insufficiently comprehensive. Section 4 evaluates a limited selection of model architectures and federated unlearning methods, primarily using image classification tasks. Additionally, the descriptions of experimental configurations, baseline adaptations, and training protocols are not sufficiently detailed in the main text. Broader evaluations and clearer experimental specifications would strengthen the reliability and generalizability of the findings.

Questions for authors

refer to weakness

3 comments

To W1: Motivation for Unlearning Independence and feasibility of Process Independence

When one client requests unlearning, the system should not take it for granted that other clients can bear the extra cost of local retraining or recovery. This is the practical requirement behind Process Independence. It limits extra computation caused by the request, but allows normal FL training and model updates.

Thank you for raising this concern. We should have explained this practical motivation more clearly in the introduction. A client’s agreement to participate in normal FL training does not mean it can provide extra resources whenever another client requests deletion. Local retraining, recovery rounds, and additional communication all require computation, communication resources, and participation time. We therefore study whether existing FU methods can handle deletion requests and preserve other clients’ utility when this additional participation cannot be assumed.

We agree that the requesting client’s data can affect other clients through the global model. However, the need to update the shared model does not necessarily mean that every retaining client must perform additional local training. Another option is to edit the model at the server using client information that has already been collected.

FedCCCU explores this option. In Section 5.1, each client reports the indices and Sensitivity Scores of its top-NN sensitive neurons for each class. When handling a deletion request, the server uses these existing scores to calculate the Utility Conflict Ratio and select neurons to edit. This follows the condition stated at the beginning of Section 5: the required scores have already been collected. Retaining clients are not asked to recompute the scores or perform additional recovery training after the request. Section 3.1 also explicitly permits normal FL updates.

We evaluate unlearning using requester target accuracy, MIA success rate, global accuracy, and retaining-client target accuracy. Together, these measurements examine the model’s behavior after deletion and its effects on uninvolved clients.

To W2: Conditions under which the two independence requirements conflict

Meeting both requirements depends on whether the target edit can avoid changing parameters that retaining clients rely on. Shared dependence can restrict the available edits, while differences in dependence can offer opportunities to reduce the conflict.

Thank you for suggesting that we clarify these conditions. To answer when the requirements conflict and when they can be met together, we give a simple local explanation using the target probabilities and sensitivity concepts in the paper.

Consider a small edit around the current model. Let ΔPreq\Delta P_{\mathrm{req}} and ΔPj\Delta P_j denote the changes in mean target-class probability for the requester and retaining client jj. If their first-order responses are proportional over the allowed edit directions, then

\qquad a>0.$$ Under this condition, lowering the requester’s target response also lowers the retaining client’s target response. If this causes previously correct predictions to cross a classification boundary, the retaining client loses accuracy. If the method then asks that client to perform additional recovery training, $\Delta\mathrm{Cost}_j$, as defined in Section 3.1, becomes positive. Shared dependence can therefore make it difficult to edit the target, preserve retaining-client utility, and avoid extra client computation at the same time. Conversely, an edit that changes requester responses while leaving retaining-client predictions unchanged can preserve utility without additional client training. If retaining-client responses are less sensitive to the edit, the resulting damage may also be smaller. **Process Independence permits these server-side edits, but does not permit extra retaining-client training to compensate for the damage.** Regarding the reviewer’s request for an explanation under cross-domain heterogeneity, **Appendix A, Figures 6–7**, shows that sensitive neurons overlap across domains, while their contribution scores can differ. Selecting neurons only by their importance to the requester does not account for whether other clients also depend on them. FedCCCU therefore compares sensitivity across clients in Section 5.2. It prioritizes neurons with lower estimated retaining-client dependence relative to requester dependence.

To W3: Formulation of the objective and independence constraints

FedCCCU’s objective is to suppress requester target predictions while limiting damage to retaining clients. We balance these goals through the Utility Conflict Ratio in Equation (6): retaining-client sensitivity is measured relative to requester sensitivity, and the method selects the neurons with the lowest ratios.

Thank you for pointing out that we should state this objective explicitly. Let UreqU_{\text{req}} denote the requester’s existing top-NN candidate neurons and EE the set of PP neurons to edit. These two set symbols are introduced here to express the existing selection rule. Using the original neuron notation ulqu_l^q and the Sensitivity Scores in Equation (6), the selection rule in Section 5.2 is equivalently written as

\sum_{u_l^q\in E} \frac{\operatorname{ReLU}(\mathcal{S}_{\text{max}}(u_l^q))} {\operatorname{ReLU}(\mathcal{S}_{\text{req}}(u_l^q))+\epsilon}$$ **The two goals enter the same objective.** The denominator measures a candidate’s sensitivity to the requester’s target class. The numerator measures the largest sensitivity reported by retaining clients for that candidate. Minimizing the sum of these ratios favors neurons with lower retaining-client dependence relative to their importance to the requester. Restricting the choice to the requester’s top-$N$ candidates keeps the edits focused on the requested target. We use these sensitivity scores as proxies for the effects of editing a neuron. The objective therefore expresses the balance used by the PoC when choosing edits. **Selecting the $P$ lowest-ratio neurons minimizes this objective over the fixed candidate pool**, exactly as described in Section 5.2. Process Independence is imposed through the request-stage computation requirement in Section 3.1. Under the condition stated in Section 5, the neuron indices and Sensitivity Scores have already been collected. The server performs selection and editing **without additional computation by retaining clients in response to the request**. We will state this objective explicitly in **Section 5.2** and clarify how it incorporates requester sensitivity, retaining-client sensitivity, and the request-stage computation constraint.

To W4: Experimental coverage and evaluation setup

Our experiments first examine cross-domain federated unlearning in image classification. They cover multiple architectures and unlearning methods, as well as unlearning tasks at different granularities, like client-level unlearning. The data allocation, baseline adaptations, and evaluation procedures are described in the main text and appendices. We will make these locations easier to find.

Thank you for the suggestions on experimental coverage and clarity. Several recent baselines in our comparison (FedAU, FCU, FUSED...) , also study unlearning in image-based tasks. We chose these image datasets because they provide shared classes and clear differences between source domains, while allowing stable initial training. This lets us examine how unlearning changes utility across clients. Section 4.1 explains the dataset choice, and Section 4.2 and Table 2 specify the client allocation, input resolutions, and model architectures for the three tasks.

Image and text tasks involve different data organization, models, and evaluation procedures. We first compare the methods within the same cross-domain image classification setting, keeping the tasks and evaluation targets consistent. The paper also includes the following additional experiments:

  • Section 5.3, Table 6: VGG11-BN results in addition to the residual networks used in the main experiments.
  • Appendix B, Figures 8–9: FUSED results under the same CD-NonIID protocol, including client-level target accuracy and accuracy-change heatmaps.
  • Appendix G, Table 11: Requests with a broader deletion scope, extending from the requester’s local target-class data to all of its local training data.
  • Appendix H, Table 12: Different target classes on IR-9 and IR-65, checking whether the results are specific to the main target class.
  • Appendix J, Table 13: Complete client-level results for class unlearning, including statistics across three runs.

The original manuscript also describes how the baselines are applied and compared:

ItemLocation in the paperExisting description
Data and model setupSection 4.2, Table 2Client-to-dataset allocation, shared classes, input resolutions, and model architectures
Retrain-PI procedureOpening of Section 4 and Section 4.3The requester retrains from scratch, while retaining clients continue normal training from their existing weights
Amnesiac adaptationSection 4.3Only the requester’s historical parameter updates are revoked; retaining clients do not perform compensatory fine-tuning
FedCDP adaptationSection 4.3Pruning-score calculation and subsequent fine-tuning are restricted to the requester
Pre/post comparisonSection 4.3Pre/post values refer to each method’s own models before and after unlearning; both starting accuracy and accuracy changes are considered
Dataset sizes and hardwareAppendix E, Table 9Dataset statistics, shared-class filtering, and experimental hardware

We will strengthen the cross-references in Sections 4.3 and 5.3 so readers can directly locate the baseline adaptations, comparison procedures, and additional experiments.

We also believe that the effects of shared-model updates on uninvolved clients deserve further study in text-based federated unlearning. In future work, we will examine these effects in text tasks and investigate suitable evaluation and mitigation methods.

Reviewer_LSHzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RWFUBench, a cross-domain benchmark for evaluating federated unlearning under Cross-Domain Non-IID (CD-NonIID) settings. RWFUBench aims to reveal collateral damage caused by unlearning requests while assessing Unlearning Independence in terms of computational overhead and utility preservation for uninvolved clients.

The proposed benchmark evaluates eight representative federated unlearning methods across three cross-domain classification tasks, examining requester target suppression, privacy leakage, and retaining-client utility. The authors further introduce FedCCCU, a proof-of-concept mitigation method that estimates neuron sensitivity across clients and employs a utility conflict ratio to guide selective neuron pruning.

Experiments demonstrate a trade-off between effective target suppression and retaining-client utility. While some retraining-based methods preserve high target-class accuracy, several model-editing methods cause accuracy degradation on uninvolved clients. FedCCCU shows the potential to reduce such collateral damage by concentrating unlearning effects on the requesting client.

Critical strengths

1. Well-motivated problem formulation.

The paper addresses an important limitation in federated unlearning evaluation, where existing studies predominantly rely on Pseudo-NonIID settings and often overlook the impact of unlearning on uninvolved clients. The introduction of Unlearning Independence and Cross-Domain Non-IID settings provides a meaningful perspective for assessing both unlearning effectiveness and collateral utility loss.

2. Systematic experimental evaluation.

The experiments evaluate eight representative federated unlearning methods across three cross-domain classification tasks, covering global accuracy, client-level target accuracy, and privacy leakage. The results reveal collateral damage that aggregate metrics may overlook. The proposed FedCCCU further provides an encouraging proof-of-concept for mitigating such effects through cross-client neuron sensitivity constraints.

Critical weaknesses

1. Insufficient evidence of genuine unlearning effectiveness.

The evaluation relies heavily on target-class accuracy reduction to assess unlearning effectiveness, although lower accuracy does not necessarily indicate that the influence of the requested data has been removed. While the authors additionally report membership inference results, these are limited to a specific attack. More direct comparisons with a properly retrained reference model are needed to establish whether the methods achieve genuine unlearning rather than simply suppressing target-class predictions.

2. Insufficient validation of the proposed mitigation strategy.

The effectiveness of FedCCCU is not fully established through comparisons at equivalent levels of unlearning. For example, on IR-65, FedCCCU retains 45.16% target accuracy on the requesting client, while FedAU reduces it to 0%, making it difficult to determine whether the lower collateral damage results from the proposed cross-client constraints or weaker target suppression. Additional comparisons under matched unlearning effectiveness and targeted ablation studies would strengthen the evaluation.

Questions for authors

1. How can the evaluation distinguish genuine unlearning from target-class prediction suppression?

The paper primarily measures unlearning effectiveness through target-class accuracy and membership inference attacks. Although Appendix F includes a global retraining baseline, it remains unclear how closely the evaluated methods approximate the model trained without the requested data. Could the authors provide more direct evidence that the requested data influence is removed, rather than merely suppressing target-class predictions?

2. Does FedCCCU still reduce collateral damage under matched unlearning effectiveness?

FedCCCU generally preserves higher retaining-client accuracy but sometimes achieves weaker target suppression than existing methods. Could the authors compare FedCCCU with relevant baselines at comparable levels of requester target suppression or privacy leakage? Additionally, an ablation study comparing the Utility Conflict Ratio with simpler neuron selection strategies would help clarify the contribution of the proposed cross-client constraints.

2 comments

To Question 1: How can the evaluation distinguish genuine unlearning from target-class prediction suppression?

We test whether the deleted samples remain identifiable as original training members, separately from measuring target-class prediction suppression. We additionally compare this membership leakage with Retrain-All, the full-retraining reference in Appendix F.

Thank you for raising this distinction. Since FedCCCU edits target-sensitive neurons, lower target accuracy alone cannot establish removal of the deleted samples’ influence. We therefore evaluated Retrain-All using the same deleted C1C_1 samples and MIA protocol as FedCCCU.

Table R1. Target accuracy and membership leakage compared with full retraining. All values are percentages.

TaskMethodRequester target accuracy (post)MIA attack success rate
IR-9Retrain-All85.5052.41
IR-9FedCCCU18.0051.37
IR-65Retrain-All47.9351.43
IR-65FedCCCU45.1652.53

Retrain-All excludes the deleted samples from training, yet retains substantial target accuracy because other clients still hold same-class data. Its MIA success rates are close to random guessing, showing why high target accuracy need not imply retained membership information.

FedCCCU also gives near-chance MIA results, within 1.10 percentage points of Retrain-All on both tasks. Under this attack, the deleted samples are therefore about as difficult to identify as after full retraining. We interpret this as evidence about deleted-sample membership leakage, rather than proof that prediction suppression and data removal are equivalent.

We will clarify this distinction in Section 4.3 and add the retraining-based privacy comparison to Appendices C and F.

To Question 2: Does FedCCCU still reduce collateral damage under matched unlearning effectiveness?

At comparable privacy leakage measured by the same MIA, FedCCCU causes less retaining-client accuracy loss than FedAU and random selection. The random-selection comparison also holds the candidate pool and number of edited neurons fixed.

Thank you for suggesting comparisons at similar requester suppression or privacy leakage. We follow the latter option and compare methods at similar MIA success rates. We also added a random-selection ablation to examine the contribution of the Utility Conflict Ratio.

Comparison with FedAU at similar MIA success rates

FedCCCU’s smaller retaining-client loss remains when privacy leakage is taken into account. Table R2 combines the MIA results from Table 7 with the retaining-client accuracies from Tables 5, 13, and 15.

Table R2. Privacy and retaining-client accuracy for FedCCCU and FedAU. Accuracy and MIA success rate are percentages; accuracy drops are percentage points. Pre/post values refer to each method’s own models before and after unlearning.

TaskMethodMIA attack success rateRetaining-client target accuracy (pre)Retaining-client target accuracy (post)Retaining-client target accuracy drop
IR-9FedAU50.8585.500.3185.19
IR-9FedCCCU51.3793.2371.0422.19
IR-65FedAU52.8770.120.0070.12
IR-65FedCCCU52.5371.1168.332.78

On IR-65, the requester accuracies of 45.16% for FedCCCU and 0% for FedAU tell only part of the story. Their MIA success rates differ by just 0.34 percentage points. Retaining-client target accuracy falls by 2.78 percentage points with FedCCCU, compared with 70.12 with FedAU.

IR-9 shows the same pattern. MIA success rates differ by 0.52 percentage points, while retaining-client accuracy drops by 22.19 percentage points with FedCCCU and 85.19 with FedAU. Thus, FedCCCU preserves more of the target-class knowledge held by uninvolved clients at similar leakage levels under this attack.

Ablation of the neuron selection rule

Using the Utility Conflict Ratio preserves more retaining-client accuracy than random selection, even when both variants edit the same number of neurons.

We ran an additional ablation using the same initial model and the same pool of the requester’s top-NN sensitive neurons. Both variants zero out exactly PP neurons. FedCCCU selects the candidates with the lowest Utility Conflict Ratios, while the random baseline selects PP neurons randomly from the same pool.

Table R3. FedCCCU versus random selection at the same edit count. All values are percentages.

Task and settingSelection ruleRequester target accuracy (post)Retaining-client target accuracy (post)MIA attack success rate
IR-9, N=30, P=8N=30,\ P=8FedCCCU18.0071.0451.37
IR-9, N=30, P=8N=30,\ P=8Random selection0.000.1350.82
IR-65, N=20, P=2N=20,\ P=2FedCCCU45.1668.3352.53
IR-65, N=20, P=2N=20,\ P=2Random selection6.9212.8751.39

Random selection suppresses requester predictions more strongly, but reduces retaining-client target accuracy to 0.13% on IR-9 and 12.87% on IR-65. With Utility Conflict Ratio selection, the corresponding accuracies remain at 71.04% and 68.33%. Both variants select from the same sensitive-neuron pool, but random selection does not account for how much other clients depend on each candidate. The conflict ratio includes this dependence when ranking neurons for editing. The lower retaining-client loss supports the effectiveness of this selection step in limiting damage to other clients.

We will add these comparisons to Section 5.3 and summarize the privacy and retaining-client results together in Appendix C. The random-selection setup and results will be reported in a new Appendix M.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.