acceptodds
Under review as a conference paper at ICLR 2027

f -Fum: Federated Unlearning via min–max and f -divergence

Abstract

Federated learning (FL) enables collaborative training while keeping raw data on clients. Deletion requests and the discovery of poisoned data create a need to remove selected contributions from an already trained model. This is challenging in FL because client data are decentralized and their influence is entangled through repeated aggregation. We present \method, an active federated unlearning method that builds on teacher–student forget/retain optimization. Starting from a pretrained global model, the method keeps that model fixed as a teacher. Clients maximize an -divergence between student and teacher predictive distributions on forget examples, then minimize KL-based teacher-student disagreement and supervised loss on retained examples. The updates are aggregated in separate, sample-weighted synchronization phases without moving raw data to the server or changing the model architecture. We evaluate forget-side divergence choices in client-level and data-level deletion settings involving backdoors, label confusion, and clean-data deletion. Under equal synchronization-phase budgets, divergence choice changes the forgetting-utility trade-off, yielding improvements in some settings and mixed results in others. In the evaluated configurations, \method uses up to fewer synchronization phases than full federated retraining.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Unlearning, Unlearning, f-Divergence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YnseOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the f-fum, which is an active federated unlearning methods on teacher-student optimization. This method treat original model as a fixed teacher while letting clients maximize f-divergence between student and teacher predictive distributions on the forget examples. It then minimize the KL disntance between two models. It is only launched locally. The author evaluate the forget-side divergence choices in client-level and data-level unlearning and stated that they achieve better results.

Critical strengths
  1. Federated unlearning is a trending and practical research direction.
  2. The paper makes good definition for active federated unlearning (server initialzted unlearning) and passive unlearning (reconstruct the model without access to the forgetting client).
  3. From reading this paper (e.g., related work and the definition of unlearning), I can see that the author has a deep understanding of the FU. Thus, the position of the work in interesting.
  4. The unlearning dynamics is helpful for understanding the method.
Critical weaknesses
  1. I think the active federated unlearning is an important setting and agree with the paper. However, this paper reproduces a limited baselines in the active FU scenarios. It would be nice if the paper could add them into the baseline table. This would facilitate proving the methods effectiveness in the field.
  2. The paper could be benefit from the differentiating them from the unbounded unlearning paper (teacher-student forget optimization), especially how it applies their trick into the federated decentralized domain with active unlearning.
  3. There is a lack of the ablation study, specifically in the combination of the four components (forget set divergence maximization, retained-data divergence minimization, supervised retained data optimization and post training recovery).
  4. The paper could benefit for scaling up their experiments using foundation models and scale up the clients. Overall, I think the active federated unlearning is an important field and this paper contributes to it. I am happy to raise the score to be positive if the above can be adequately addressed.
Questions for authors

Questions are generally the weakness above.

  1. Could the paper reproduce more active FU baselines like you reviewed in the appendix?
  2. Could the paper differentiate them from the unbounded unlearning paper?
  3. Could the author provide a comprehensive ablation study?
  4. Could the author scale up the FU with modern models and datasets like FU papers published in this year?
3 comments

We thank the reviewer for the positive assessment of the setting and the clear list of requests.

Q3: Comprehensive ablation of the four components. Table 13 (Appendix I.1; also shown in our response to Dgg5) ablates each component and adds two controls.

  • (i) Forget-set maximization is the only component that removes the forget signal. Without it, the backdoor stays (ASR 74.3% at client level, against 4.2% for f-FUM and 1.9% for retraining).
  • (ii) Retention divergence and (iii) the supervised loss each preserve utility, and their effects add up: 46.4% → 40.9% without (ii), 37.5% without (iii), and 30.4% without both.
  • (iv) Post-training restores utility and completes the removal. Without it, accuracy is 22.5% with ASR 95.6%. The same budget spent on more cycles gives 27.6%, so it is post-training, not extra rounds.

The controls show that neither half works alone. The max step alone collapses the model (10.3%), which is the CIFAR-10 counterpart of the toy collapse in Fig. 2. Plain fine-tuning is competitive at client level, where the departing client just stops contributing (78.3%, ASR 3.1%). But it cannot remove data-level forget samples that stay on active clients (ASR 68.5%). Those require the max step with a learning rate scaled to the forget set; the paper's main data-level results show it does: on CIFAR-100 with 6–10% of each client's data deleted, f-FUM lowers backdoor ASR from 74–81% before unlearning to 0.25–0.69% (Table 1).

Q2: Difference from the unbounded-unlearning paper (SCRUB). The teacher–student max–min objective comes from SCRUB, as stated in the introduction, and the revised conclusion now names SCRUB explicitly. Our contribution is making it work when forget and retained data stay on clients and the server only averages. This involves (1) two separately weighted aggregation phases per cycle, in which a departing client participates only in the max phase (Theorem 1: an inexact step on one contrastive objective); (2) a fixed global teacher; (3) post-training after aggregation, which is necessary (22.5% without it); and (4) a bounded forget-side f-divergence that prevents the collapse caused by averaging max updates (JS up to +24 points over KL where KL collapses). Please see also our answer to MQWK Q1.

Q1: More active FU baselines. The paper compares against NoT, Halimi and MoDE in every setting. These are the active methods we found that fit the aggregator-only protocol: the server only averages full local models and holds no data, generator, auxiliary model or per-client state. The other active methods we review need server-side computation. Goldfish and Jin et al. need server-side data. Wu et al. needs server-side distillation. FFMU uses server-side operator-based aggregation. The recently suggested FedOSD (AAAI 2025) solves an SVD-based projection over individual client gradients on the server. FedCARE (IJCAI 2026) trains a data-free generator on the server and filters head updates against stored heads (details in our response to MQWK). Comparing against them would mix two system models, so we instead make this distinction explicit in Sec. 2 and App. A.

Q4: Scale (modern models, more clients). We agree that scale matters, and we want to be transparent about what we could and could not run. Our experiments use a single H100 (App. D.7), on which each CIFAR-10 retraining reference alone takes ≈2.3 h. Within the rebuttal period we gave priority to the ablation, retraining and learning-rate studies that all three reviewers asked for. A foundation-model or many-client study was not feasible with our compute, and we do not want to report a rushed, under-tuned version of it. On scalability by design: f-FUM adds no server-side state and no per-client computation on the server. The server performs only the sample-weighted averaging of FedAvg, and each client's cost depends only on its own forget and retained sets, not on N. Its cost therefore grows with the number of clients exactly as standard federated training does. Nothing in the method is specific to small models either. It only needs predictive distributions, so it applies unchanged to pretrained backbones and to adapter-based fine-tuning.

Reviewer_YnseOpenReview ↗

Thank the Author for the Response

Thank the Author for the response. I have increased my score to weak accept.

We sincerely thank the reviewer for reading our response and for raising the score to weak accept. Your suggestions, especially the component ablation and the clarification of the relation to SCRUB, have clearly improved the paper. Thank you again for your time and constructive feedback.

Reviewer_MQWKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes ff-Fum, an active federated unlearning algorithm where the unlearning-requested client participates in the federated process to remove the influence of their data. In ff-Fum, clients alternate between ff-divergence maximization on their local forget set and ff-divergence minimization on their local retain set, where the updates after each maximization/minimization phase are sent back to the server for aggregation.

Critical strengths
  1. Federated unlearning is an important problem to address, where the influence of local, private/backdoored/noisy data can influence models resided on server and other clients through contiuous update aggregation.

  2. The paper is easy to follow, and experiments cover wide range of unlearning applications for private/backdoored/noisy data removal.

Critical weaknesses
  1. The proposed method is not novel and the paper contribution is not significant enough. Teacher-model objectives are widely used in centralized machine unlearning (e.g., [1]), and can be trivially adopted for federated settings. Additionally, studying the effects of different ff-divergence functions to federated unlearning performance is rather trivial, where experimental results show marginal and inconsistent differences among the tested ff-divergence functions.

  2. Empirically, ff-Fum doesn't show consistent improvement over the comparison baselines. Furthermore, the improvement is marginal.

  3. The experimental setup should be improved.

  • The comparison baselines are not extensive, comparing only against 3 other baselines, and golden retraining performance isn't reported. Additional baselines such as FedOSD, FedCARE should be included.
  • Privacy experiments rely on MIA scores, but MIA performance is nearly saturated by previous methods. A similar saturation problem also occurs with attack success rate in backdoor unlearning and unlearning accuracy in confused data unlearning. The saturating performance makes it difficult to evaluate the improvement of the proposed method.
  • The experiments should also evaluate unlearning robustness. It is commonly known that existing methods are not robust to additional fine-tuning post-unlearning.
  • The experiments should extend to more realistic settings, such as unlearning for text generation with LLMs, where clients perform fine-tuning on local parameter-efficient adapters.
Questions for authors
  1. Fundamentally, ff-Fum shares the same mechanism of alternating min-max optimization with SCRUB [1] in centralized unlearning. Does ff-Fum contain any non-trivial modifications to SCRUB that makes it more suitable for federated unlearning?

  2. How sensitive is ff-Fum to the choice of local learning rates and number of fine-tuning epochs? My understanding is that, as the objective in the maximization step is not bounded above, it could quickly deteriorate local model performance and aggregated model performance.

4 comments

We thank the reviewer for the precise questions, which led to the new retraining, learning-rate and divergence studies (general response).

Q1 / W1: Non-trivial modifications to SCRUB? f-FUM builds on SCRUB's teacher–student max–min scheme (Kurmanji et al., 2023), as stated in the introduction; the revised conclusion now names SCRUB explicitly. SCRUB assumes pooled data. The federated version differs in four places, each of which matters empirically:

  1. Two aggregation phases per cycle, with separate sample weights. Only forget-holding clients upload in the max phase, so a departing client participates only there. Theorem 1 shows the aggregated round is an inexact gradient step on one contrastive objective.
  2. A fixed global teacher, so every client diverges from the same reference.
  3. Post-training after aggregation. Without it, accuracy ends at 22.5% instead of 78.0%.
  4. A bounded forget-side f-divergence. Averaging clients' max updates can collapse the global model. JS rescues variants that collapse with KL (up to +24 points), consistent with the analysis in Appendix H (Remarks 1–2).

The alternation also matters: without min steps, accuracy falls from 46.4% to 30.4%, and the max step alone ends at chance.

W1 (cont.): Studying f-divergences is trivial; differences are marginal. In the full method, the differences are indeed moderate, and we will say so. They are not marginal where the max step dominates: up to +24 points with JS at the same seed and LR. JS is better by >1 point in 20 of 43 paired runs and worse by >1 point in only one (−1.7). The reason is visible in Figure 5c: KL's forget objective reaches 12.6 nats, while JS stays at 0.19–0.62, below log 2. In the paper's CIFAR-10 backdoor/confusion results, JS beats KL in 13 of 14 settings (up to +21). On clean-data deletion, KL is as good or better. We frame JS as a safer default for the forget side, not a uniform gain.

Q2: Sensitivity to LR and epochs; the max objective is unbounded. The reviewer is right that the KL max objective is unbounded. Run alone, it sends the model to chance. Inside f-FUM, the min steps pull it back after every max step. Table 15 (Adam, max–min cycles):

LevelLRCollapsed (all variants / f-FUM)f-FUM Acc / ASR
Client5e-2100% / 100%10.0 / –
Client5e-325% / 0%78.0 / 4.2
Data5e-30% / 0%77.0 / 71.2 (no forgetting)

Collapsed: final accuracy ≤ 15%. Retraining: 80.3 / 1.9 (client), 81.2 / 2.1 (data).

The LR must match the forget-set size: about 20 optimizer steps per max phase at client level, about 1 at data level. At client level, LR 5e-2 collapses every run, and 5e-3 works. At data level, 5e-3 is too small to forget anything (ASR 71.2%), because the forget set gives about one optimizer step per max phase; the paper's main data-level experiments remove the backdoor (CIFAR-100, Table 1: ASR 0.25–0.69% after unlearning). A too-large LR shows up after the first cycle (accuracy at chance), i.e. after two synchronization phases, so it is detected early and cheaply on the validation set.

W3: Golden retraining not reported. It is now reported in every CIFAR-10 setup (Table 14 in the general response and Appendix I.2). We had deliberately kept it out of the submission so that f-FUM's hyperparameters could not be tuned toward an oracle that is unavailable in practice; checkpoints were selected only on held-out validation accuracy, and the f-FUM numbers are unchanged. On clean-data deletion, f-FUM is within 3.5 points of retraining's accuracy in all setups except client level N=10 (6.4). On backdoor/confusion, it is 7.3–26.0 points below, at ≈15% of the wall-clock. Forgetting metrics at N=5: ASR 4.2 vs 1.9 and Conf. 8.8 vs 2.1 at client level. For data-level forgetting, see the paper's Table 1 (CIFAR-100: ASR 0.25–0.69% after unlearning).

W3: More baselines (FedOSD, FedCARE). We thank the reviewer for the pointers. We read both papers. Neither fits the protocol we study, so a direct comparison would not be like-for-like. f-FUM targets active, aggregator-only FU (Sec. 2, "Positioning"): the server only broadcasts, receives full local models and computes a sample-weighted average, exactly as in FedAvg. It holds no data, generator, auxiliary model or per-client state. Our three baselines (NoT, Halimi, MoDE) were chosen because they fit this protocol.

  • FedOSD (Pan et al., AAAI 2025) puts the core unlearning computation on the server. In each unlearning round, the server stacks the individual gradients of all remaining clients into a matrix G and solves for the direction closest to the target client's descent direction under Gd = 0, via an SVD of GGᵀ (Alg. 1, Eq. 6). In post-training, the server keeps the original model ω⁰ and projects each client gradient that points back toward ω⁰ (Eq. 7). This requires the server to see every client's update separately, which is incompatible with secure aggregation, and it adds per-round linear algebra on the server.
  • FedCARE (Li et al., IJCAI 2026) trains a class-conditional, data-free generator on the server from the global model (Sec. 3.1, Eqs. 1–5) and ships it to the target client to synthesize a reference set. In recovery, the server stores both the original and the unlearned classifier heads and filters the aggregated head update to remove its rollback component (Sec. 3.3, Eqs. 18–19).

Both are therefore in the "beyond accumulator" category of our Related Work (Sec. 2 / App. A), along with Goldfish, FFMU and Wu et al. We will add them there with this characterization. If the reviewer considers a server-side comparison essential, we would be glad to report it in a separate table marked as a different, stronger server model. We would not present it as a like-for-like comparison.

W3: MIA / ASR / confused-label accuracy look saturated. Retraining, which never saw the forget data, sets the targets: MIA ≈50%, ASR and Conf. ≈2%. Reaching them is the goal of unlearning, not saturation. The metrics still separate methods. f-FUM's MIA ranges from 1.4 to 11.5 points above retraining. Models that keep the forget data are clearly separated from retraining: without max steps, ASR is 74.3% and MIA 62.8%, and fine-tuning only gives MIA 61.6%, against 1.9% and 52.1% for retraining.

W2: No consistent improvement. Please see our answer to Dgg5 W4. We now characterise where f-FUM is preferable (data-level deletion) and where it is not (heavily corrupted teacher), instead of claiming uniform gains.

W3: Robustness to fine-tuning after unlearning, and the LLM / PEFT-adapter setting. We answer these two points together, because they come from the same line of work. The relearning threat model, in which an adversary fine-tunes the unlearned model on a small amount of forget-related data to restore the removed behaviour, has been developed mainly in LLM unlearning. There, unlearning targets knowledge or hazardous capabilities in large pretrained models, and the attacker has access to the weights. Federated LLM fine-tuning with local parameter-efficient adapters brings the same questions into FL, together with others such as adapter-specific leakage and paraphrase-based extraction.

Our paper deliberately studies a different and complementary problem: removing the effect of identifiable training contributions (backdoored, mislabeled or deletion-requested data, or a whole client) from a federated classifier, measured against retraining (Eq. 4) with the metrics standard in FU (ASR, forget accuracy, MIA). Robustness to post-hoc adversarial relearning and the LLM/adapter regime are important. They would, however, need a different threat model, different models and different evaluation protocols, which is beyond the scope of this paper. We will state this explicitly as a limitation and a direction for future work. We note that f-FUM itself only needs predictive distributions and standard aggregation, so nothing in the method prevents applying it to adapter-based federated fine-tuning.

AuthorsOpenReview ↗

Follow-up: summary of new results addressing your review

Dear Reviewer MQWK,

We apologise for the late rebuttal. The points you raised required a large number of new experiments, including retraining from scratch in every CIFAR-10 setup and a learning-rate study, and these took most of the discussion period on our compute. We would be grateful if you could take a look before the discussion closes. In short, the response and the revised PDF (new Appendix I, in blue) address your main points:

  • Relation to SCRUB (Q1/W1). f-FUM builds on SCRUB's teacher–student max–min objective, and the revision now credits this explicitly. What is new is making it work when the data stay on the clients and the server only averages. This requires separately weighted max and min aggregation phases, a fixed global teacher, post-training, and a bounded forget-side divergence. The ablation (Table 13) shows that each of these is needed.
  • f-divergence choice (W1). The differences are moderate in the full method, and we now say so. They are large where the max step dominates (up to +24 points with JS), because KL's forget objective is unbounded (12.6 nats) while JS stays below log 2 (Figure 5c).
  • LR / epoch sensitivity and the unbounded max objective (Q2). Table 15 shows that the learning rate must be scaled to the forget-set size, and that a too-large rate is visible after the first cycle.
  • Retraining reference (W3). It is now reported in every CIFAR-10 setup (Table 14). It also shows that the forgetting metrics are not saturated.
  • FedOSD / FedCARE (W3). Both rely on server-side computation (per-client gradient SVD; a server-side generator and head filtering), which our aggregator-only setting excludes. We explain this in detail and will discuss both in Related Work.
  • Relearning robustness and LLM/PEFT (W3). These are now stated as a limitation and as future work.

We hope these address your concerns. If they do, we would appreciate it if you would consider updating your score. We are happy to answer any remaining questions.

Reviewer_Dgg5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies federated unlearning through an alternating optimization procedure that encourages a model to move away from its original predictions on data marked for deletion (under KL) while preserving its behavior on the remaining data (under f-divergence). The authors explore several divergence measures for the forgetting objective and evaluate the approach across different deletion settings. The paper considers both client-level and data-level unlearning, covering backdoor removal, label-confusion removal, and privacy-oriented clean-data deletion. It also investigates different (f)-divergences, including KL, Jensen–Shannon, and (\chi^2), to examine their effects on the forgetting–utility trade-off. Experimental results across several datasets and model architectures demonstrate competitive performance in selected settings, although the results are mixed across deletion scenarios and baselines.

Critical strengths
  1. This paper proposes a method that operates in an active federated setting where forgetting and retained data remain on clients, and the server only performs standard sample-weighted aggregation. This avoids requiring centralized access to raw data or additional server-side auxiliary datasets, making the formulation relevant to practical federated learning constraints.

  2. This paper considers both client-level and data-level deletion and evaluates the method under backdoor attacks, label corruption, and clean-data removal. These scenarios cover both robustness-oriented and privacy-oriented unlearning, providing a relatively broad assessment of the proposed framework.

  3. Rather than relying on a single forgetting objective, the paper compares KL, JS, and (\chi^2) divergences and discusses their effects on the forgetting–utility trade-off. The results reveal meaningful setting-dependent behavior, suggesting that divergence selection is an important design consideration for optimization-based federated unlearning.

Critical weaknesses
  1. The main idea of encouraging disagreement on forgotten data while maintaining agreement on retained data is already closely related to existing teacher–student unlearning methods. The authors acknowledge this connection and discuss FedAF in Section 2, but the distinction between the proposed method and prior approaches is not particularly clear. It is also unclear which limitation of existing forgetting objectives motivates the use of a general (f)-divergence family. The experiments suggest that different divergences produce different trade-offs, but this alone does not establish why the proposed formulation is necessary or what it enables beyond existing methods.

  2. Equation (4) defines approximate unlearning in terms of the discrepancy between the unlearned model and a model retrained on the retained dataset. However, the optimization in Section 4 does not directly target this discrepancy. Instead, it maximizes disagreement with the original model on forgotten samples. These are not necessarily equivalent objectives: moving away from the original predictions does not imply moving toward the predictions of a retrained model. The local curvature analysis mentioned in Appendix H may help explain optimization behavior, but it is unclear whether it provides any guarantee on forgetting effectiveness or proximity to retraining. The connection between the proposed objective and the stated unlearning criterion needs to be better justified.

  3. It is difficult to tell how much each component contributes to the reported performance. Figures 2 and 3 illustrate the importance of alternating optimization on a toy example, but they do not replace ablations on the actual benchmarks. In particular, the paper would benefit from experiments removing the retention divergence term, the supervised retention loss, or the post-training recovery stage. Without these comparisons, it is hard to determine whether the improvements mainly come from the proposed forgetting objective or from the additional retention and recovery steps.

  4. Although several results show that the proposed approach is competitive, but there are also cases where existing baselines perform substantially better. For example, in Table 2, MoDE achieves 64.30% accuracy on Fashion-MNIST client-level backdoor unlearning with five clients, while the best proposed variant reaches 55.93%. In Table 3, Halimi achieves 79.53% on Fashion-MNIST data-level backdoor unlearning with ten clients, compared with 76.70% for the best proposed variant. On CIFAR-100, NoT also obtains lower attack success rates in some client-level settings, although with lower clean accuracy. The paper acknowledges these mixed results, but offers limited explanation for when the proposed approach should be preferred. A clearer analysis of the conditions under which it works well would help establish its practical value.

  5. The paper highlights a reduction of synchronization phases relative to federated retraining. However, synchronization count alone is not sufficient to characterize the actual cost of an unlearning method. The proposed procedure alternates between two optimization stages and includes additional recovery training, potentially introducing substantial local computation.

Questions for authors

see weakness

3 comments

We thank the reviewer for the detailed and constructive review. New results are summarised in the general response. We answer point by point.

W3: Ablate the retention divergence, the supervised retention loss and post-training. Table 13 (N=5, CIFAR-10, accuracy averaged over the three tasks, each forgetting metric from its own task):

VariantClient Acc↑ASR↓Conf↓MIA (50)Data Acc↑ASR↓Conf↓MIA
Retrain from scratch80.31.92.152.181.22.12.050.1
f-FUM78.04.28.859.377.071.298.459.2
f-FUM, γ=178.02.09.059.476.871.298.359.6
↳ w/o post-training22.595.66.249.072.684.797.959.8
↳ 10 cycles, no post-training27.697.57.847.072.785.297.759.3
f-FUM, Alg. 1 post-training46.449.8*6.347.672.985.498.159.8
↳ w/o max steps75.274.325.162.872.484.796.760.7
↳ w/o retention divergence40.92.110.048.576.974.397.859.0
↳ w/o supervised loss37.518.710.050.469.079.696.659.7
↳ w/o min steps30.416.710.050.772.483.696.259.2
Fine-tuning only78.33.113.261.677.068.598.360.2
Max step only10.399.110.050.668.875.890.955.9

All rows: new single runs (seed 17, KL, unlearning LR 5e-3; *JS, since the KL run is pending). The paper's 3-seed means with validation-selected checkpoints are compared with retraining in Table 14. ↳ removes one component from the row heading its block. Conf. 10.0 on a model at chance accuracy carries no information. Data level: at LR 5e-3, no variant forgets the 2% forget set (≈1 optimizer step per max phase); see Table 15.

  • Post-training is necessary. At client level, accuracy drops from 78.0% to 22.5% without it, and the backdoor stays active (ASR 95.6% vs 2.0%). Spending the same 20 rounds on more cycles does not help (27.6%), so the gain comes from post-training itself, not from extra rounds. Accuracy recovers from 26% to 74% within three post-training rounds.
  • The retention divergence and the supervised loss each preserve utility, and their effects add up. Under the same post-training, accuracy drops from 46.4% to 40.9% without the retention divergence, to 37.5% without the supervised loss, and to 30.4% without both.

W3 (cont.): Do the gains come from the forgetting objective or from retention and recovery? Neither works alone. Without max steps, the backdoor survives (ASR 74.3% vs 4.2%). The max step alone collapses the model (10.3%). We also want to be transparent about one control. At client level, where the departing client simply stops contributing, plain fine-tuning on retained data is a strong baseline (78.3%, ASR 3.1%). f-FUM matches it in accuracy and is better on confused labels (8.8 vs 13.2) and MIA (59.3 vs 61.6). The max step matters most at data level, where the forget samples stay on clients that keep training. There, fine-tuning cannot remove them (ASR 68.5%, Conf. 98.3%). The ablation's learning rate (5e-3) is too small to remove a 2% forget set (about one optimizer step per max phase; Table 15), but the paper's main data-level results show it does: on CIFAR-100 with 6–10% of each client's data deleted, f-FUM lowers backdoor ASR from 74–81% before unlearning to 0.25–0.69% (Table 1).

W2: Eq. 4 targets the retrained model, but the objective pushes away from the original model. We agree that the two objectives are not equivalent, and we no longer leave the connection implicit. The max step removes the forget-set signal. Post-training then moves the model toward the retained-data solution. Proposition 2 (App. H.3.3) bounds the distance between the post-trained model and the retained-data ERM solution, and transfers it to Lipschitz output-based MIA scores. It does not claim equality with a fresh retraining run. We therefore now measure Eq. 4 directly against retraining from scratch (Table 14 in the general response and Appendix I.2, all CIFAR-10 setups). At client level, f-FUM's forgetting is close to retraining: ASR 4.2 vs 1.9, and Conf. 8.8 vs 2.1. At data level, the ablation's learning rate is too small to forget (Table 15); the paper's data-level results (CIFAR-100, Table 1) show backdoor ASR reduced to 0.25–0.69%. The remaining gaps are accuracy (paper: 72.0 vs 80.3 with KL, 77.8 with JS; new run: 78.0) and MIA (paper: 63.6 vs 52.1, 49.9 with JS; new run: 59.3).

W1: Distinction from FedAF and teacher–student methods; why a general f-divergence family? FedAF (Li et al., 2023b) overwrites old knowledge by learning new memories (constructed targets) through teacher–student training. f-FUM constructs no targets: it moves the student away from a fixed global teacher on the forget set, and keeps it anchored on the retained set, with the two steps in separate, sample-weighted aggregation phases. We use the f-family because the forget step is a maximization of an averaged objective. With KL this objective is unbounded: in our max-step-only runs it reaches 12.6 nats, while JS stays between 0.19 and 0.62. JS is bounded by log 2 (Remark 1), and its destabilizing curvature is scaled by f''(1) = 1/4 instead of 1 (Remark 2). Empirically, at the same seed and LR, JS is better than KL by >1 point in 20 of 43 paired runs. It is worse by more than 1 point in only one, by 1.7. The gain is largest exactly where nothing counteracts the max step (+24 on average without post-training, up to +65 on a single task). In the paper's CIFAR-10 backdoor/confusion results, JS beats KL in 13 of 14 settings (up to +21). On clean-data deletion, KL is as good or better. We therefore present JS as the safer forget-side default, not a uniform improvement.

W4: Mixed results against baselines. We agree that the paper should say when f-FUM is preferable. (i) It is strongest on data-level deletion, where forget samples stay on clients that keep training. There, recovery-style methods cannot remove them. Examples: MNIST label confusion, 98.4% vs Halimi 73.5%; CIFAR-10 backdoor N=10, 73.4% vs 62.0%. (ii) It is weakest when the pretrained teacher is itself badly corrupted. FMNIST client-level backdoor N=5 starts from 31.3% accuracy, and there MoDE (64.3%) outperforms us (55.9%). This is consistent with the retention term anchoring the student to a poor teacher. (iii) On CIFAR-100, NoT's lower ASR comes with a 5.7-point accuracy drop (0.59% ASR at 44.4%, vs 5.80% at 50.1%). That pattern indicates a degraded model rather than a removed backdoor.

W5: Synchronization count does not capture cost. We agree. Wall-clock (one H100, App. D.7): f-FUM takes ≈20 min, against ≈2.3 h for the schedule retraining repeats. Local computation: 110 local epochs per client, against 200 for retraining. The max epochs touch only the forget set, which is small at data level. Communication: each phase sends one ResNet-18 (≈11.2M params, ≈45 MB fp32) per participating client, and only forget-holding clients upload in max phases. Client level N=5: 65 uploads vs 800 for retraining. Data level N=5: 100 vs 1,000. That is roughly a 10–12× reduction (Appendix I.4).

AuthorsOpenReview ↗

Follow-up: summary of new results addressing your review

Dear Reviewer Dgg5,

We apologise for the late rebuttal. Your requests (and those of the other reviewers) required a large number of new experiments, including retraining from scratch in every CIFAR-10 setup, and these took most of the discussion period on our compute. We would be grateful if you could take a look before the discussion closes. In short, the response and the revised PDF (new Appendix I, in blue) address your main points:

  • Ablation of the components (W3). Table 13 removes each component in turn. Post-training is necessary (22.5% to 78.0% accuracy), the retention divergence and the supervised loss each preserve utility, and neither half works alone: without the max step the backdoor stays (ASR 74.3%), and the max step alone collapses the model (10.3%).
  • Eq. 4 and retraining (W2). We now measure Eq. 4 directly against retraining from scratch in every CIFAR-10 setup (Table 14). We state the remaining gaps explicitly, and Proposition 2 makes the link between post-training and the retained-data solution precise.
  • FedAF and the f-divergence family (W1). f-FUM constructs no targets, unlike FedAF. The f-family matters on the forget side: KL reaches 12.6 in the max step, while JS stays below log 2. We present JS as the safer default, not as a uniform improvement.
  • Mixed results (W4). We now state when f-FUM is preferable (data-level deletion) and when it is not (a badly corrupted pretrained teacher).
  • Cost (W5). We report communication and local computation in addition to synchronization phases (Appendix I.4).

We hope these address your concerns. If they do, we would appreciate it if you would consider updating your score. We are happy to answer any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading. Three requests were shared across reviews, and we address them with new CIFAR-10 / ResNet-18 experiments (N=5, client level and 2% data level, all three tasks, same pretrained model, split and seed for every variant). The results are in the new Appendix I of the revised PDF (Tables 13–15, Figures 4–5).

  1. Component ablation (all reviewers). We ablate each of the four components (forget-set maximization, retention divergence, supervised retention loss, post-training) plus two controls: plain fine-tuning on retained data, and the max step alone. Neither half works alone. Without the max step, the backdoor stays (ASR 74.3% vs 4.2% for f-FUM at client level). The max step alone collapses the model (10.3% accuracy). Post-training is needed to restore utility (22.5% to 78.0%).
  2. Retraining from scratch (Dgg5, MQWK). We now report the Eq. 4 reference in every CIFAR-10 setup (Table 14 below). At client level, f-FUM's forgetting metrics are close to retraining (ASR 4.2 vs 1.9, Conf. 8.8 vs 2.1). The largest gaps are accuracy and MIA (+1.4 to +11.5 points).
  3. Behaviour of the unbounded maximization (all reviewers). We report learning-rate sensitivity and the divergence reached in each max step. KL reaches 12.6 nats, while JS stays between 0.19 and 0.62, below log 2. The learning rate must be scaled to the forget-set size. This is also where the f-divergence choice matters.
AuthorsOpenReview ↗

General Response

Table 14 (Appendix I.2). Retraining from scratch vs f-FUM in every CIFAR-10 setup of the paper (%).

LevelSetupModelBackdoor Acc↑ASR↓Confusion Acc↑Conf.↓Clean-deletion Acc↑MIA
ClientN=5Retrain80.61.979.42.180.852.1
f-FUM69.64.267.48.878.963.6
ClientN=10Retrain80.51.979.62.280.151.0
f-FUM71.8–71.2–73.752.4
DataN=5, 2%Retrain81.22.180.32.082.150.1
f-FUM66.6–73.0–78.758.6
DataN=10, 2%Retrain79.61.478.92.778.749.3
f-FUM68.6–70.7–78.460.8
DataN=10, 6%Retrain78.61.678.82.078.550.7
f-FUM52.6–67.8–78.161.4
DataN=10, 10%Retrain77.61.877.72.378.550.4
f-FUM52.7–64.1–78.859.9

Retrain: the full pretraining schedule (200 rounds × 1 local epoch) on the retained data only. f-FUM (KL): accuracy and MIA as reported in the paper (Tables 2–5, 8, 10; 3-seed means). ASR and Conf. are not reported for CIFAR-10 in the paper; client level N=5 values come from the new runs.

Why retraining was not in the submission. Retraining is the oracle that approximate unlearning approximates (Eq. 4), and it is not available in practice: if it were, there would be nothing to unlearn. We therefore deliberately kept it out of the original tables. We did not want to tune f-FUM's unlearning hyperparameters (learning rate, number of cycles, post-training length) toward the retrained model's numbers, which would be an oracle-informed choice no deployed system could make. All checkpoints were selected only on held-out validation accuracy (Sec. 5), never on proximity to retraining. We now report retraining as an evaluation reference only. All f-FUM numbers in Table 14 are the unchanged submission numbers.

On FedOSD and FedCARE (MQWK, Ynse). We read both carefully and did not add them as baselines, because each needs server-side computation that our aggregator-only setting (Sec. 2, "Positioning") excludes and that our baselines also do not use. FedOSD has the server collect each remaining client's gradient individually, solve an SVD-based problem for an orthogonal unlearning direction, and project post-training gradients against a stored original model. FedCARE has the server train a data-free generator from the global model and filter aggregated head updates against stored original and unlearned heads. Details are in our responses to MQWK and Ynse. We will add both to Related Work under "beyond accumulator" in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.