- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FedCoDS to address federated LLM unlearning, which poses specific challenges, such as when limited local retain data makes it difficult to preserve general retain knowledge or when heterogeneous local forget data produce inconsistent forgetting directions. The authors address these issues with token-wise SimNPO suppression and knowledge distillation.
Critical strengths
- An important contribution of this paper is the formulation of the federated LLM unlearning problem. The authors identify the unique challenges that differ from those of common LLM unlearning, and their algorithm design also targets resolving these challenges.
- The empirical results validate the effectiveness of their proposed algorithm, which outperforms the baseline NPO unlearning algorithms in the federated LLM unlearning setting.
Critical weaknesses
- The token-wise suppression itself is not new relative to SATIMP. The contribution of the proposed algorithm mainly lies in the knowledge distillation from a shared pretrained model, from my perspective. Therefore, the authors should consider comparing FedCoDS with SimNPO or SATIMP plus knowledge distillation and KL regularization for a more direct comparison.
- One common weakness of suppression-based unlearning algorithms is the worry about true unlearning, where it is difficult to validate whether the forget set is truly removed or simply suppressed. This concern may need more attention in the federated LLM unlearning setting. Some evaluations on whether the forget set can be recovered through paraphrased prompting or re-emerge during subsequent federated training can be considered.
- The experiment compares the proposed method only with likelihood suppression based unlearning algorithms, but it lacks representative alternative approaches such as adaptations of Grad Diff (which was also included in the TOFU benchmark) or SCRUB.
- In the main experimental results (Table 1), FedCoDS may produce an ES-U score lower than retraining. This brings up a worry of over-forgetting or whether the unlearned model makes unreasonable responses. Can the authors at least provide some qualitative examples of the model's responses on the forget set after unlearning?
Questions for authors
- The authors report the best selected checkpoints for the MUSE results. May I ask why the authors chose to do so instead of reporting the average? What are the results on other seeds?
- Can the authors explain the lack of representative alternative unlearning baselines?
12 comments
Answer to Reviewer Euj9 [Part2]
Q2/W3: “Can the authors explain the lack of representative alternative unlearning baselines?”
Thank you for suggesting these alternatives. Our original comparisons prioritized popular LLM unlearning methods such as NPO, SimNPO, and SATIMP. In centralized TOFU Forget05, Fan et al. (2025, Table 1) compared SimNPO with GradDiff and reported better forget quality (0.99 vs. ) with higher model utility (0.58 vs. 0.56), and so we had not included GradDiff in our evaluation. Based on the reviewer’s comments, we now provide a direct comparison with GradDiff in our federated setting. However we are not able to compare with SCRUB in this short period of time due to major differences in the setup.
GradDiff. We implemented GradDiff in the same federated LoRA training framework and added the retain-side KL regularizer used by FedCoDS. For each TOFU setting, we evaluated 16 learning-rate/retain-weight configurations and report the mean ± standard deviation over three seeds.
Table. Comparison with GradDiff on TOFU Forget05 (round 10; mean ± standard deviation over three seeds). Arrows indicate preferred directions; bold indicates the best mean in each setting.
(a) Single-client requests
| Setting | Method | ES-U ↓ | ES-R ↑ | MU ↑ | FTR ↑ |
|---|---|---|---|---|---|
| SS | GradDiff + retain-KL | .113 ± .037 | .339 ± .039 | .393 ± .027 | .459 ± .014 |
| FedCoDS | .037 ± .004 | .921 ± .003 | .540 ± .004 | .726 ± .002 | |
| RS | GradDiff + retain-KL | .087 ± .005 | .233 ± .013 | .374 ± .015 | .438 ± .019 |
| FedCoDS | .036 ± .000 | .931 ± .003 | .552 ± .001 | .738 ± .003 |
(b) Multiple-client requests
| Setting | Method | ES-U ↓ | ES-R ↑ | MU ↑ | FTR ↑ |
|---|---|---|---|---|---|
| SM | GradDiff + retain-KL | .071 ± .012 | .524 ± .039 | .480 ± .014 | .500 ± .038 |
| FedCoDS | .045 ± .006 | .971 ± .007 | .521 ± .009 | .674 ± .002 | |
| RM | GradDiff + retain-KL | .104 ± .024 | .687 ± .017 | .483 ± .024 | .447 ± .027 |
| FedCoDS | .045 ± .002 | .973 ± .006 | .543 ± .008 | .652 ± .004 |
FedCoDS achieves lower ES-U and higher ES-R, MU, and FTR in all four settings. For example, in SS, ES-U/ES-R improves from 0.113/0.339 (GradDiff) to 0.037/0.921 (FedCoDS); in RM, it improves from 0.104/0.687 to 0.045/0.973. These results demonstrate that, compared to GradDiff, FedCoDS has stronger forgetting with substantially better retention under the same federated protocol.
SCRUB. SCRUB (Kurmanji et al., 2023) was developed and evaluated for image classification; its error-based stopping rules are defined for class predictions. SCRUB was evaluated only for image-classification networks (ResNet and All-CNN) on CIFAR and Lacuna. It is difficult to faithfully adapt SCRUB to a federated LLM unlearning framework in such a short time, so we are unable to provide this comparison here.
Answer to Reviewer Euj9 [Part3]
W1: “ […] Therefore, the authors should consider comparing FedCoDS with SimNPO or SATIMP plus knowledge distillation and KL regularization for a more direct comparison. ”
Thank you for this suggestion. We conducted additional experiments by adding the same retain-side KL regularization to the fine-tuned model for NPO, SimNPO, and SATIMP on TOFU. The results confirm that retain-KL generally improves retention, increasing ES-R in 10 of the 12 baseline–setting comparisons.
Importantly, FedCoDS maintains its superior forgetting–retention trade-off. It achieves both lower ES-U and higher ES-R in 11 of the 12 comparisons. For example, in RS, FedCoDS achieves ES-U/ES-R of 0.036/0.931, compared with 0.090/0.906 for SATIMP + retain-KL. In SM, FedCoDS further reduces ES-U from 0.076 to 0.045 while improving ES-R from 0.923 to 0.971. In SS, although SATIMP + retain-KL achieves slightly higher ES-R (0.938 vs. 0.921), FedCoDS provides substantially stronger forgetting (0.037 vs. 0.086 ES-U).
Table. Effect of retain KL regularization on TOFU Forget05. (a) Single-client and (b) multiple-client unlearning under author-silo and random partitions. All baselines are augmented with retain-side KL regularization, as in FedCoDS. Results are reported as mean ± standard deviation over three seeds; bold indicates the best mean among unlearning methods. FedCoDS achieves both lower ES-U and higher ES-R in 11 of 12 baseline comparisons, demonstrating that its forgetting–retention gains extend beyond retain-side KL regularization alone.
(a) Single Client
SS: author silos
| Method | ES-U↓ | ES-R↑ | MU↑ | FTR↑ |
|---|---|---|---|---|
| Finetuned | .978 | .979 | .544 | .492 |
| Retrain | .087 | .976 | .543 | .634 |
| NPO + KL | .095±.001 | .751±.012 | .528±.002 | .598±.002 |
| SimNPO + KL | .074±.017 | .712±.055 | .513±.002 | .620±.005 |
| SATIMP + KL | .086±.003 | .938±.005 | .533±.004 | .645±.010 |
| FedCoDS | .037±.004 | .921±.003 | .540±.004 | .726±.002 |
RS: random
| Method | ES-U↓ | ES-R↑ | MU↑ | FTR↑ |
|---|---|---|---|---|
| Finetuned | .990 | .971 | .542 | .495 |
| Retrain | .093 | .980 | .546 | .647 |
| NPO + KL | .055±.006 | .750±.017 | .519±.001 | .587±.003 |
| SimNPO + KL | .096±.015 | .728±.001 | .534±.009 | .623±.009 |
| SATIMP + KL | .090±.010 | .906±.003 | .554±.002 | .603±.015 |
| FedCoDS | .036±.000 | .931±.003 | .552±.001 | .738±.003 |
(b) Multiple Clients
SM: author silos
| Method | ES-U↓ | ES-R↑ | MU↑ | FTR↑ |
|---|---|---|---|---|
| Finetuned | .980 | .982 | .527 | .499 |
| Retrain | .089 | .984 | .544 | .641 |
| NPO + KL | .120±.006 | .887±.009 | .523±.006 | .594±.010 |
| SimNPO + KL | .085±.003 | .886±.006 | .525±.009 | .557±.009 |
| SATIMP + KL | .076±.015 | .923±.013 | .510±.010 | .666±.016 |
| FedCoDS | .045±.006 | .971±.007 | .521±.009 | .674±.002 |
RM: random
| Method | ES-U↓ | ES-R↑ | MU↑ | FTR↑ |
|---|---|---|---|---|
| Finetuned | .979 | .984 | .559 | .486 |
| Retrain | .090 | .984 | .561 | .636 |
| NPO + KL | .092±.003 | .906±.012 | .536±.010 | .587±.004 |
| SimNPO + KL | .090±.007 | .872±.011 | .533±.005 | .579±.012 |
| SATIMP + KL | .050±.013 | .925±.008 | .536±.012 | .683±.033 |
| FedCoDS | .045±.002 | .973±.006 | .543±.008 | .652±.004 |
Answer to Reviewer Euj9 [Part1]
Q1: “The authors report the best selected checkpoints for the MUSE results. May I ask why the authors chose to do so instead of reporting the average? What are the results on other seeds?”
Note that due to time constraints, our original submission reported the best selected checkpoint rather than the three-seed mean and standard deviation. We acknowledge that this did not adequately characterize run-to-run variability. We have now rerun the evaluation with three seeds and report the mean ± standard deviation for all methods.
Table. MUSE privacy leakage and utility preservation over three random seeds. (a) Single-client and (b) multiple-client unlearning. Results are mean ± standard deviation; PrivLeak closer to zero and higher UtilPres are preferred. Bold indicates the best mean among unlearning methods in each metric. FedCoDS achieves the mean PrivLeak closest to zero in both single-client and multiple-client settings while preserving higher UtilPres than SimNPO and SATIMP.
(a) Single-Client
(b) Multiple Client
The additional results confirm that FedCoDS achieves the mean PrivLeak closest to zero in all four MUSE settings: −6.5 and −18.3 on News, and −6.4 and −20.1 on Books, for single- and multiple-client unlearning, respectively. For example, on News with multiple clients, FedCoDS achieves PrivLeak of −18.3, compared with −46.4 for SimNPO and −58.8 for SATIMP, while improving UtilPres to 37.5 versus 28.8 and 33.1, respectively. FedCoDS also achieves higher mean UtilPres than SimNPO and SATIMP across all four settings. Although NPO maintains comparable utility, its PrivLeak remains between −92.8 and −100, indicating substantially weaker forgetting.
These results demonstrate that FedCoDS maintains a favorable privacy–utility trade-off across different seeds, rather than relying on a single best-performing checkpoint.