acceptodds
Under review as a conference paper at ICLR 2027

FedCoDS: Federated LLM Unlearning by Coupling Knowledge Distillation with Likelihood Suppression

Abstract

Federated large language model (LLM) unlearning aims to remove targeted training data without retraining the model from scratch or sharing clients' raw data. However, existing likelihood-based unlearning objectives face two key challenges in federated settings. When the forget set is held by a single client, the requesting client has access to only limited local retain data, making it difficult to preserve knowledge learned from other clients. When the forget set is distributed across multiple clients, heterogeneous local forget data can produce inconsistent forgetting directions that are weakened during federated aggregation. To address these challenges, we propose FedCoDS, a federated LLM unlearning method that couples token-wise likelihood suppression with knowledge distillation from a shared frozen pretrained model. The suppression term directly reduces the likelihood of targeted tokens, while knowledge distillation provides a common distributional reference that constrains model changes and promotes cross-client consistency. FedCoDS combines their complementary strengths to achieve a better balance between effective forgetting and knowledge retention. Experiments on TOFU and MUSE under both single- and multi-client unlearning settings show that FedCoDS consistently improves the forgetting–retention performance over SOTA baselines. Further analysis shows that FedCoDS improves cross-client forget-gradient alignment and produces forget-answer likelihoods closer to those of the retrained model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated LLM Unlearning, Federated Learning, Large Language Model Unlearning, Knowledge Distillation, Negative Preference Optimization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Euj9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedCoDS to address federated LLM unlearning, which poses specific challenges, such as when limited local retain data makes it difficult to preserve general retain knowledge or when heterogeneous local forget data produce inconsistent forgetting directions. The authors address these issues with token-wise SimNPO suppression and knowledge distillation.

Critical strengths
  • An important contribution of this paper is the formulation of the federated LLM unlearning problem. The authors identify the unique challenges that differ from those of common LLM unlearning, and their algorithm design also targets resolving these challenges.
  • The empirical results validate the effectiveness of their proposed algorithm, which outperforms the baseline NPO unlearning algorithms in the federated LLM unlearning setting.
Critical weaknesses
  • The token-wise suppression itself is not new relative to SATIMP. The contribution of the proposed algorithm mainly lies in the knowledge distillation from a shared pretrained model, from my perspective. Therefore, the authors should consider comparing FedCoDS with SimNPO or SATIMP plus knowledge distillation and KL regularization for a more direct comparison.
  • One common weakness of suppression-based unlearning algorithms is the worry about true unlearning, where it is difficult to validate whether the forget set is truly removed or simply suppressed. This concern may need more attention in the federated LLM unlearning setting. Some evaluations on whether the forget set can be recovered through paraphrased prompting or re-emerge during subsequent federated training can be considered.
  • The experiment compares the proposed method only with likelihood suppression based unlearning algorithms, but it lacks representative alternative approaches such as adaptations of Grad Diff (which was also included in the TOFU benchmark) or SCRUB.
  • In the main experimental results (Table 1), FedCoDS may produce an ES-U score lower than retraining. This brings up a worry of over-forgetting or whether the unlearned model makes unreasonable responses. Can the authors at least provide some qualitative examples of the model's responses on the forget set after unlearning?
Questions for authors
  • The authors report the best selected checkpoints for the MUSE results. May I ask why the authors chose to do so instead of reporting the average? What are the results on other seeds?
  • Can the authors explain the lack of representative alternative unlearning baselines?
12 comments
AuthorsOpenReview ↗

Answer to Reviewer Euj9 [Part1]

Q1: “The authors report the best selected checkpoints for the MUSE results. May I ask why the authors chose to do so instead of reporting the average? What are the results on other seeds?”

Note that due to time constraints, our original submission reported the best selected checkpoint rather than the three-seed mean and standard deviation. We acknowledge that this did not adequately characterize run-to-run variability. We have now rerun the evaluation with three seeds and report the mean ± standard deviation for all methods.

Table. MUSE privacy leakage and utility preservation over three random seeds. (a) Single-client and (b) multiple-client unlearning. Results are mean ± standard deviation; PrivLeak closer to zero and higher UtilPres are preferred. Bold indicates the best mean among unlearning methods in each metric. FedCoDS achieves the mean PrivLeak closest to zero in both single-client and multiple-client settings while preserving higher UtilPres than SimNPO and SATIMP.

(a) Single-Client

MethodBooks: PrivLeak → 0Books: UtilPres ↑News: PrivLeak → 0News: UtilPres ↑
Finetuned-10049.1−10044.2
Retrain050.4041.3
NPO−100.0 ± 0.046.1 ± 5.0−92.8 ± 0.938.9 ± 1.7
SimNPO−14.9 ± 22.944.0 ± 0.5−87.3 ± 3.931.4 ± 1.3
SATIMP−21.8 ± 3.813.5 ± 23.5−93.0 ± 6.722.4 ± 9.0
FedCoDS−6.4 ± 13.444.4 ± 3.1−6.5 ± 1.735.2 ± 3.2

(b) Multiple Client

MethodBooks: PrivLeak → 0Books: UtilPres ↑News: PrivLeak → 0News: UtilPres ↑
Finetuned−10051.4−10043.2
Retrain051.5039.9
NPO−100.0 ± 0.051.4 ± 1.1−99.5 ± 0.336.5 ± 2.4
SimNPO−30.9 ± 5.549.9 ± 0.9−46.4 ± 44.328.8 ± 1.6
SATIMP−39.0 ± 5.748.1 ± 0.4−58.8 ± 25.633.1 ± 1.2
FedCoDS−20.1 ± 7.750.9 ± 3.0−18.3 ± 19.237.5 ± 0.9

The additional results confirm that FedCoDS achieves the mean PrivLeak closest to zero in all four MUSE settings: −6.5 and −18.3 on News, and −6.4 and −20.1 on Books, for single- and multiple-client unlearning, respectively. For example, on News with multiple clients, FedCoDS achieves PrivLeak of −18.3, compared with −46.4 for SimNPO and −58.8 for SATIMP, while improving UtilPres to 37.5 versus 28.8 and 33.1, respectively. FedCoDS also achieves higher mean UtilPres than SimNPO and SATIMP across all four settings. Although NPO maintains comparable utility, its PrivLeak remains between −92.8 and −100, indicating substantially weaker forgetting.

These results demonstrate that FedCoDS maintains a favorable privacy–utility trade-off across different seeds, rather than relying on a single best-performing checkpoint.

AuthorsOpenReview ↗

Answer to Reviewer Euj9 [Part2]

Q2/W3: “Can the authors explain the lack of representative alternative unlearning baselines?”

Thank you for suggesting these alternatives. Our original comparisons prioritized popular LLM unlearning methods such as NPO, SimNPO, and SATIMP. In centralized TOFU Forget05, Fan et al. (2025, Table 1) compared SimNPO with GradDiff and reported better forget quality (0.99 vs. 3.6×10−93.6\times10^{-9}) with higher model utility (0.58 vs. 0.56), and so we had not included GradDiff in our evaluation. Based on the reviewer’s comments, we now provide a direct comparison with GradDiff in our federated setting. However we are not able to compare with SCRUB in this short period of time due to major differences in the setup.

GradDiff. We implemented GradDiff in the same federated LoRA training framework and added the retain-side KL regularizer used by FedCoDS. For each TOFU setting, we evaluated 16 learning-rate/retain-weight configurations and report the mean ± standard deviation over three seeds.

Table. Comparison with GradDiff on TOFU Forget05 (round 10; mean ± standard deviation over three seeds). Arrows indicate preferred directions; bold indicates the best mean in each setting.

(a) Single-client requests

SettingMethodES-U ↓ES-R ↑MU ↑FTR ↑
SSGradDiff + retain-KL.113 ± .037.339 ± .039.393 ± .027.459 ± .014
FedCoDS.037 ± .004.921 ± .003.540 ± .004.726 ± .002
RSGradDiff + retain-KL.087 ± .005.233 ± .013.374 ± .015.438 ± .019
FedCoDS.036 ± .000.931 ± .003.552 ± .001.738 ± .003

(b) Multiple-client requests

SettingMethodES-U ↓ES-R ↑MU ↑FTR ↑
SMGradDiff + retain-KL.071 ± .012.524 ± .039.480 ± .014.500 ± .038
FedCoDS.045 ± .006.971 ± .007.521 ± .009.674 ± .002
RMGradDiff + retain-KL.104 ± .024.687 ± .017.483 ± .024.447 ± .027
FedCoDS.045 ± .002.973 ± .006.543 ± .008.652 ± .004

FedCoDS achieves lower ES-U and higher ES-R, MU, and FTR in all four settings. For example, in SS, ES-U/ES-R improves from 0.113/0.339 (GradDiff) to 0.037/0.921 (FedCoDS); in RM, it improves from 0.104/0.687 to 0.045/0.973. These results demonstrate that, compared to GradDiff, FedCoDS has stronger forgetting with substantially better retention under the same federated protocol.

SCRUB. SCRUB (Kurmanji et al., 2023) was developed and evaluated for image classification; its error-based stopping rules are defined for class predictions. SCRUB was evaluated only for image-classification networks (ResNet and All-CNN) on CIFAR and Lacuna. It is difficult to faithfully adapt SCRUB to a federated LLM unlearning framework in such a short time, so we are unable to provide this comparison here.

AuthorsOpenReview ↗

Answer to Reviewer Euj9 [Part3]

W1: “ […] Therefore, the authors should consider comparing FedCoDS with SimNPO or SATIMP plus knowledge distillation and KL regularization for a more direct comparison. ”

Thank you for this suggestion. We conducted additional experiments by adding the same retain-side KL regularization to the fine-tuned model for NPO, SimNPO, and SATIMP on TOFU. The results confirm that retain-KL generally improves retention, increasing ES-R in 10 of the 12 baseline–setting comparisons.

Importantly, FedCoDS maintains its superior forgetting–retention trade-off. It achieves both lower ES-U and higher ES-R in 11 of the 12 comparisons. For example, in RS, FedCoDS achieves ES-U/ES-R of 0.036/0.931, compared with 0.090/0.906 for SATIMP + retain-KL. In SM, FedCoDS further reduces ES-U from 0.076 to 0.045 while improving ES-R from 0.923 to 0.971. In SS, although SATIMP + retain-KL achieves slightly higher ES-R (0.938 vs. 0.921), FedCoDS provides substantially stronger forgetting (0.037 vs. 0.086 ES-U).

Table. Effect of retain KL regularization on TOFU Forget05. (a) Single-client and (b) multiple-client unlearning under author-silo and random partitions. All baselines are augmented with retain-side KL regularization, as in FedCoDS. Results are reported as mean ± standard deviation over three seeds; bold indicates the best mean among unlearning methods. FedCoDS achieves both lower ES-U and higher ES-R in 11 of 12 baseline comparisons, demonstrating that its forgetting–retention gains extend beyond retain-side KL regularization alone.

(a) Single Client

SS: author silos

MethodES-U↓ES-R↑MU↑FTR↑
Finetuned.978.979.544.492
Retrain.087.976.543.634
NPO + KL.095±.001.751±.012.528±.002.598±.002
SimNPO + KL.074±.017.712±.055.513±.002.620±.005
SATIMP + KL.086±.003.938±.005.533±.004.645±.010
FedCoDS.037±.004.921±.003.540±.004.726±.002

RS: random

MethodES-U↓ES-R↑MU↑FTR↑
Finetuned.990.971.542.495
Retrain.093.980.546.647
NPO + KL.055±.006.750±.017.519±.001.587±.003
SimNPO + KL.096±.015.728±.001.534±.009.623±.009
SATIMP + KL.090±.010.906±.003.554±.002.603±.015
FedCoDS.036±.000.931±.003.552±.001.738±.003

(b) Multiple Clients

SM: author silos

MethodES-U↓ES-R↑MU↑FTR↑
Finetuned.980.982.527.499
Retrain.089.984.544.641
NPO + KL.120±.006.887±.009.523±.006.594±.010
SimNPO + KL.085±.003.886±.006.525±.009.557±.009
SATIMP + KL.076±.015.923±.013.510±.010.666±.016
FedCoDS.045±.006.971±.007.521±.009.674±.002

RM: random

MethodES-U↓ES-R↑MU↑FTR↑
Finetuned.979.984.559.486
Retrain.090.984.561.636
NPO + KL.092±.003.906±.012.536±.010.587±.004
SimNPO + KL.090±.007.872±.011.533±.005.579±.012
SATIMP + KL.050±.013.925±.008.536±.012.683±.033
FedCoDS.045±.002.973±.006.543±.008.652±.004
Reviewer_Km22OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FedCoDS, a federated LLM unlearning method for both single-client and multiple-client forget requests. It multiplicatively couples a token-wise SimNPO suppression term with a knowledge distillation term, whose teacher is the frozen pretrained model shared by all clients, and adds retain-side KL and cross-entropy regularization on local retain data. Experiments on TOFU and MUSE under both settings demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper studies single-client and multiple-client forget requests under both author-silo and random partitions, which makes the federated challenges concrete.
  2. Using the frozen pretrained model as a shared distillation teacher is simple and needs no extra communication in federated LoRA, and FedCoDS achieves the highest ES-R on all TOFU settings and the |PrivLeak| closest to 0 on all MUSE settings.
Critical weaknesses
  1. The comparison with the baselines is not fully controlled. FedCoDS adds a retain-side KL term to the fine-tuned model (λKL\lambda_{KL} from 1 to 8), while NPO, SimNPO, and SATIMP use only retain cross-entropy with a retain-KL coefficient of 0 (Lines 309–311 and 717–718). The ablation in Table 3 applies this term to KD and T-SimNPO, but no baseline is run with it, and MUSE-News single-client uses an extra gap regularizer only for FedCoDS. Running the baselines with the same retain-KL term would show how much of the retention gain comes from the coupled forget loss.
  2. The claim of better forgetting is not well supported. Lines 313–315 state that forget-side generation scores are interpreted relative to Retrain, but Table 1 marks the lowest ES-U as best, although FedCoDS (.036–.045) is far below Retrain (.087–.093). Also, in Table 3, T-SimNPO without KD reaches almost the same multiple-client ES-U as FedCoDS (.0449 vs. .0447), so the gain from KD in this setting seems to be retention rather than more consistent forgetting. Reporting a Retrain-relative metric such as the TOFU forget quality would make the claim more convincing.
Questions for authors
  1. Lines 188–189 say the two terms are coupled multiplicatively rather than optimized independently, but Table 3 only tests each term alone. How does FedCoDS compare with the additive form, KD plus T-SimNPO with tuned weights?
  2. How were the hyperparameters and MUSE checkpoints selected? FedCoDS uses different β\beta, (β1,β2)(\beta_1, \beta_2), and λ\lambda in each setting (Tables 6 and 7), and Lines 751–753 report the best round of the best seed without stating the selection metric.
  3. Since the KD teacher is the base model, the KD gradient at θπ\theta_\pi may point toward undoing the LoRA update on any input. What is the cosine between a client's KD gradients on its forget data and on its retain data?
  4. Table 3 shows that KD alone has ES-U close to Retrain (.0926 and .0837). What is the NLL MSE of KD alone in Figure 2?
  5. Section 2 discusses distillation-based unlearning methods such as RKLD and UNDIAL, but the paper does not compare with them. How does FedCoDS perform against them in the multiple-client setting?

Minor:

  1. Metric names are inconsistent: ES-U/ES-R in Table 1, ES-Unl/ES-Retain in Table 3, and ES-Unlearn in Figure 3.
  2. Lines 725–726 and Table 4: The number of retain samples and local steps per unlearning round is not given.
5 comments
AuthorsOpenReview ↗

Answer to Reviewer Km22 [Part 1]

Q1: “[…] How does FedCoDS compare with the additive form, KD plus T-SimNPO with tuned weights?”

Thank you for this great observation. Multiplicative coupling models the interaction between the two objectives. While a weighted sum combines KD and T-SimNPO with fixed coefficients, multiplication allows each term to scale the gradient of the other. The product rule gives ∇(KD×T-SimNPO)=T-SimNPO⋅∇KD+KD⋅∇T-SimNPO\nabla(\text{KD}\times\text{T-SimNPO})=\text{T-SimNPO}\cdot\nabla\text{KD}+\text{KD}\cdot\nabla\text{T-SimNPO}. Thus, likelihood suppression is weighted by the teacher–student discrepancy, reducing suppression pressure as their distributions converge, while distillation is weighted by the forget-token likelihood-dependent loss

We did an ablation study where the multiplicative forget loss is replaced with a weighted additive combination of KD and T-SimNPO, (1−α)⋅KD+α⋅T-SimNPO(1-\alpha)\cdot\text{KD}+\alpha\cdot\text{T-SimNPO}, for α∈0.2,0.4,0.6,0.8\alpha\in\\{0.2,0.4,0.6,0.8\\}. All other settings, including the learning rate, loss weights, retain regularization, and training rounds, were kept the same. Results are averaged over three seeds.

In the single-client setting (RS), FedCoDS reduces ES-U from 0.060 (best additive) to 0.036 while improving ES-R from 0.774 to 0.931. In the multi-client setting (RM), the additive combination with α=0.8\alpha=0.8 achieves comparable ES-U (0.049 vs. 0.045) with FedCoDS, but FedCoDS achieves much higher ES-R (0.973 vs. 0.964).

Table. Multiplicative coupling versus additive combination on TOFU Forget05. (a) Single-client (RS) and (b) multiple-client (RM) unlearning with random partitions. Sum(α) denotes (1−α)KD+αT-SimNPO(1-\alpha)\text{KD}+\alpha\text{T-SimNPO}. Results are mean ± standard deviation across three seeds; bold marks the best mean. FedCoDS achieves lower ES-U and higher ES-R than all additive combinations.

(a) Single-client unlearning (random partition)

Forget lossES-U ↓ES-R ↑MU ↑FTR ↑
Sum (α=0.2\alpha = 0.2).061 ± .001.657 ± .012.532 ± .001.719 ± .000
Sum (α=0.4\alpha = 0.4).061 ± .002.682 ± .012.531 ± .000.719 ± .001
Sum (α=0.6\alpha = 0.6).061 ± .003.714 ± .010.530 ± .001.718 ± .001
Sum (α=0.8\alpha = 0.8).060 ± .003.774 ± .005.530 ± .003.716 ± .002
FedCoDS.036 ± .000.931 ± .003.552 ± .001.738 ± .003

(b) Multiple-client unlearning (random partition)

Forget lossES-U ↓ES-R ↑MU ↑FTR ↑
Sum (α=0.2\alpha = 0.2).075 ± .002.971 ± .003.543 ± .001.669 ± .003
Sum (α=0.4\alpha = 0.4).070 ± .004.971 ± .009.543 ± .002.668 ± .003
Sum (α=0.6\alpha = 0.6).061 ± .002.970 ± .009.542 ± .003.670 ± .004
Sum (α=0.8\alpha = 0.8).049 ± .004.964 ± .005.540 ± .002.679 ± .004
FedCoDS.045 ± .002.973 ± .006.543 ± .008.652 ± .004
AuthorsOpenReview ↗

Answer to Reviewer Km22 [Part 2]

Q2: “How were the hyperparameters and MUSE checkpoints selected?[…]”

Thank you for bringing this up. We will expand Appendix A to report the range of hyperparameters for both TOFU and MUSE.

On TOFU, we tune five parameters: the learning rate, the T-SimNPO parameter β, and the loss weights λf\lambda_{f}, λKL\lambda_{KL}, and λCE\lambda_{CE}, while fixing (β1\beta_{1}, β2\beta_{2}) = (1, 1). We initially explored learning rates from 1×10−51\times 10^{-5} to 1×10−21\times 10^{-2}, and then did a fine-grained search on the reduced range shown in the table. On MUSE, we additionally varied β1\beta_{1} and β2\beta_{2}. The gap-regularization coefficient λg\lambda_{g} is fixed at 1 only for MUSE-News with a single requester and is set at 0 otherwise. Within each setting, the selected hyperparameters are fixed across all three random seeds. We choose the configuration that provides good performance for both forgetting (ES-U) and retaining (ES-R).

TOFU search ranges

SettingLearning rateβλf\lambda_{f}λKL\lambda_{KL}λCE\lambda_{CE}
SS3×10−53\times 10^{-5} – 1×10−41\times 10^{-4}4–61–2{1,2,4,8}{2,3,4,5,6}
RS3×10−53\times 10^{-5} – 1×10−41\times 10^{-4}4–61–2{1,2,4,8}{2,3,4,5,6}
SM5×10−45\times 10^{-4} – 2×10−32\times 10^{-3}1–21–2{1,2,4,8}{2,3,4,5,6}
RM5×10−45\times 10^{-4} – 2×10−32\times 10^{-3}1–21–2{1,2,4,8}{2,3,4,5,6}

MUSE search ranges

SettingLearning rateβ(β1\beta_{1}, β2\beta_{2})(λf,λKL,λCE\lambda_{f},\lambda_{KL},\lambda_{CE})
News · RS1×10−51\times 10^{-5} – 1×10−31\times 10^{-3}1–2(0.1–2, 0.1–2)(1–2, 1–8, {2,3,4,5,6})
News · RM1×10−51\times 10^{-5} – 1×10−31\times 10^{-3}1–2(0.1–2, 0.2–2)(1–2, 1–8, {2,3,4,5,6})
Books · RS1×10−51\times 10^{-5} – 1×10−31\times 10^{-3}1–2(0.1–2, 0.2–2)(1–2, 1–8, {2,3,4,5,6})
Books · RM1×10−51\times 10^{-5} – 1×10−31\times 10^{-3}1–2(0.1–2, 0.2–2)(1–2, 1–8, {2,3,4,5,6})

Effect of different numbers of seeds on MUSE. Note that due to time constraints, our original submission reported the best selected checkpoint rather than the three-seed mean and standard deviation. We acknowledge that this did not adequately characterize run-to-run variability. We have now rerun the evaluation with three seeds and report the mean ± standard deviation for all methods.

Table. MUSE privacy leakage and utility preservation over three random seeds. (a) Single-client and (b) multiple-client unlearning. Results are mean ± standard deviation; PrivLeak closer to zero and higher UtilPres are preferred. Bold indicates the best mean among unlearning methods in each metric. FedCoDS achieves the mean PrivLeak closest to zero in both single-client and multiple-client settings while preserving higher UtilPres than SimNPO and SATIMP.

(a) Single-Client

MethodBooks: PrivLeak → 0Books: UtilPres ↑News: PrivLeak → 0News: UtilPres ↑
Finetuned-10049.1−10044.2
Retrain050.4041.3
NPO−100.0 ± 0.046.1 ± 5.0−92.8 ± 0.938.9 ± 1.7
SimNPO−14.9 ± 22.944.0 ± 0.5−87.3 ± 3.931.4 ± 1.3
SATIMP−21.8 ± 3.813.5 ± 23.5−93.0 ± 6.722.4 ± 9.0
FedCoDS−6.4 ± 13.444.4 ± 3.1−6.5 ± 1.735.2 ± 3.2

(b) Multiple Client

MethodBooks: PrivLeak → 0Books: UtilPres ↑News: PrivLeak → 0News: UtilPres ↑
Finetuned−10051.4−10043.2
Retrain051.5039.9
NPO−100.0 ± 0.051.4 ± 1.1−99.5 ± 0.336.5 ± 2.4
SimNPO−30.9 ± 5.549.9 ± 0.9−46.4 ± 44.328.8 ± 1.6
SATIMP−39.0 ± 5.748.1 ± 0.4−58.8 ± 25.633.1 ± 1.2
FedCoDS−20.1 ± 7.750.9 ± 3.0−18.3 ± 19.237.5 ± 0.9

The additional results confirm that FedCoDS achieves the mean PrivLeak closest to zero in all four MUSE settings: −6.5 and −18.3 on News, and −6.4 and −20.1 on Books, for single- and multiple-client unlearning, respectively. For example, on News with multiple clients, FedCoDS achieves PrivLeak of −18.3, compared with −46.4 for SimNPO and −58.8 for SATIMP, while improving UtilPres to 37.5 versus 28.8 and 33.1, respectively. FedCoDS also achieves higher mean UtilPres than SimNPO and SATIMP across all four settings. Although NPO maintains comparable utility, its PrivLeak remains between −92.8 and −100, indicating substantially weaker forgetting.

These results demonstrate that FedCoDS maintains a favorable privacy–utility trade-off across different seeds, rather than relying on a single best-performing checkpoint. We will update the appendix to include the full search configurations, checkpoint-selection protocol, and three-seed results.

AuthorsOpenReview ↗

Answer to Reviewer Km22 [Part 3]

Q3:” What is the cosine between a client’s KD gradients on its forget data and on its retain data?”

Thank you for raising this point. We agree that distillation toward the pretrained model can reverse fine-tuning changes. While this is desirable on forget data, it may be harmful if applied indiscriminately to retained data. FedCoDS explicitly separates these objectives on retain and forget data: on forget data, pretrained-teacher distillation is coupled with token-wise likelihood suppression; on retain data, we use cross-entropy and KL regularization toward the frozen fine-tuned model. Figure 1 measures cross-client alignment of forget-loss gradients on only forget data. Importantly, our method does not apply KD from the frozen fine-tuned model.

Q4: “[…] What is the NLL MSE of KD alone in Figure 2?”

Thank you for this question. We additionally evaluated standalone KD using the NLL MSE relative to Retrain. KD alone obtains an NLL MSE of 2.76, compared with 1.40 for FedCoDS.

Table. Paired forget-answer NLL MSE relative to Retrain (lower is better).

MethodNLL MSE ↓
NPO6.30
SimNPO15.89
SATIMP2.44
KD alone2.76
FedCoDS1.40

Although standalone KD achieves an aggregate ES-U close to Retrain, its larger paired NLL MSE indicates a weaker match to Retrain at the individual-answer level.

Q5: “[…] distillation-based unlearning methods such as RKLD and UNDIAL […]How does FedCoDS perform against them in the multiple-client setting?”

Thank you for suggesting these comparisons.

RKLD. Fan et al. (2025, Table 1) compared SimNPO and RKLD on centralized TOFU Forget05. SimNPO achieved higher Forget Quality (0.99 versus 0.79) and Model Utility (0.58 versus 0.56), so we used SimNPO as one of our baselines.

UNDIAL. Following your suggestion, we adapted the official UNDIAL implementation to our federated unlearning framework and evaluated it on TOFU with multiple requesting clients. To control retain-side regularization, our implementation of UNDIAL uses the same retain cross-entropy and retain-KL terms as FedCoDS. Shown below are the results for UNDIAL and FedCoDS averaged over 3 seeds.

SM: author silos

MethodES-U↓ES-R↑MU↑FTR↑
UNDIAL.097±.005.950±.010.522±.008.557±.004
FedCoDS.045±.006.971±.007.521±.009.674±.002

RM: random

MethodES-U↓ES-R↑MU↑FTR↑
UNDIAL.114±.004.962±.003.546±.003.536±.003
FedCoDS.045±.002.973±.006.543±.008.652±.004

FedCoDS consistently improves forgetting and retention over UNDIAL in both multiple-client settings. In SM, it lowers ES-U from 0.097 to 0.045 and improves ES-R from 0.950 to 0.971; in RM, ES-U falls from 0.114 to 0.045 while ES-R increases from 0.962 to 0.973. FedCoDS also improves FTR in both settings (0.674 versus 0.557 in SM; 0.652 versus 0.536 in RM), with comparable MU.

“Metric names are inconsistent […] local steps per unlearning round is not given”

Thank you for pointing these out. We apologize for the inconsistencies. We will standardize the metric notation to ES-U and ES-R throughout the manuscript and add the local retain-sample counts and the number of local optimization steps per unlearning round to the experimental setup.

Reviewer_ctFHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies LLM unlearning after federated LoRA fine-tuning, distinguishing a single client holding the whole forget set from a forget set spread across clients. It argues that the first setting makes utility hard to preserve with limited local retain data, while in the second, heterogeneous local updates weaken each other under aggregation. FedCoDS multiplies a token-level distillation term toward the frozen base model, which all clients share, with token-wise SimNPO suppression, and adds retain-side regularization. On TOFU and MUSE, it reports better forgetting–retention trade-offs than NPO, SimNPO, and SATIMP.

Critical strengths
  1. The paper addresses a practical problem. Distinguishing single-client from multiple-client forget allocations is a useful way to organize federated LLM unlearning.

  2. The method is simple and fits federated LoRA. The frozen base model is already on every client, so the shared teacher costs no extra communication.

  3. Retention gains in the single-client setting are clear and consistent across seeds, and the experimental details are reported thoroughly.

Critical weaknesses
  1. The two stated challenges may not be directly addressed. With multiple clients, the gradient alignment seems largely expected, since all clients are pulled toward the same base model. It also does not directly show that each client’s forgetting survives aggregation. SimNPO applied per token also appears to forget about as well without the shared teacher. With a single client, the added KL term uses the same local retain data, so it is unclear how it protects knowledge from other clients.

  2. The coupling would benefit from theoretical support. Equation 6 shows that the gradients vanish when the model matches the teacher. However, the loss also approaches zero when the target token is fully suppressed, so the objective itself may not favor stopping at the teacher. An analysis of convergence, and of why multiplying the two terms works better than adding them, would strengthen the paper.

  3. The novelty seems somewhat limited. FedCoDS keeps standard FedAvg and modifies only the local loss, which combines existing components such as distillation, SimNPO, and retain regularization. Apart from the shared base model, the design does not appear specific to the federated setting, so calling it the first loss designed for federated LLM unlearning may be an overstatement.

  4. The evaluation could be more consistent and fair. The paper notes that forgetting is best judged by closeness to the retrained model, but the comparisons treat lower scores on the forget data as better. In addition, only FedCoDS uses the extra KL term on the retain data, and there is no ablation that adds the two terms instead of multiplying them.

Questions for authors
  1. Could the authors clarify whether the gradient alignment is specific to the forget data? For example, does the same loss on the retain data align just as well?
  2. Under what conditions does the loss stop at the teacher rather than fully suppressing the target token? What motivates multiplying the two terms instead of adding them?
  3. Do the gains still hold when the same KL term on the retain data added to all baselines?
  4. Beyond the shared base model, which parts of FedCoDS are specific to federated learning, and how does it compare with existing methods in centralized unlearning?
8 comments
AuthorsOpenReview ↗

Answer to Reviewer ctFH [Part 1]

Q1: “Could the authors clarify whether the gradient alignment is specific to the forget data? For example, does the same loss on the retain data align just as well?”

Thank you for this question. The gradient alignment reported in Figure 1 is measured specifically for the forget-loss gradients on each client's local forget data and does not include retain-side losses. FedCoDS and standalone KD achieve a mean pairwise cosine similarity of approximately 0.86, compared with 0.56 for SATIMP and 0.24/0.29 for SimNPO/NPO. These results demonstrate that the shared pretrained teacher improves alignment among client forgetting gradient directions. Importantly, FedCoDS implements knowledge distillation on the pretrained model only on forget data. On retain data, it uses cross-entropy and KL regularization, preserving knowledge acquired during federated fine-tuning. We do not apply the forget loss to retain data since this would introduce unwanted suppression of retained answers.

Q2: “What motivates multiplying the two terms instead of adding them?”

Multiplicative coupling: Thank you for this great observation. Multiplicative coupling models the interaction between the two objectives. While a weighted sum combines KD and T-SimNPO with fixed coefficients, multiplication allows each term to scale the gradient of the other. The product rule gives ∇(KD×T-SimNPO)=T-SimNPO⋅∇KD+KD⋅∇T-SimNPO\nabla(\text{KD}\times\text{T-SimNPO})=\text{T-SimNPO}\cdot\nabla\text{KD}+\text{KD}\cdot\nabla\text{T-SimNPO}. Thus, likelihood suppression is weighted by the teacher–student discrepancy, reducing suppression pressure as their distributions converge, while distillation is weighted by the forget-token likelihood-dependent loss

We did an ablation study where the multiplicative forget loss is replaced with a weighted additive combination of KD and T-SimNPO, (1−α)⋅KD+α⋅T-SimNPO(1-\alpha)\cdot\text{KD}+\alpha\cdot\text{T-SimNPO}, for α∈0.2,0.4,0.6,0.8\alpha\in\\{0.2,0.4,0.6,0.8\\}. All other settings, including the learning rate, loss weights, retain regularization, and training rounds, were kept the same. Results are averaged over three seeds.

In the single-client setting (RS), FedCoDS reduces ES-U from 0.060 (best additive) to 0.036 while improving ES-R from 0.774 to 0.931. In the multi-client setting (RM), the additive combination with α=0.8\alpha=0.8 achieves comparable ES-U (0.049 vs. 0.045) with FedCoDS, but FedCoDS achieves much higher ES-R (0.973 vs. 0.964).

Table. Multiplicative coupling versus additive combination on TOFU Forget05. (a) Single-client (RS) and (b) multiple-client (RM) unlearning with random partitions. Sum(α) denotes (1−α)KD+αT-SimNPO(1-\alpha)\text{KD}+\alpha\text{T-SimNPO}. Results are mean ± standard deviation across three seeds; bold marks the best mean. FedCoDS achieves lower ES-U and higher ES-R than all additive combinations.

(a) Single-client unlearning (random partition)

Forget lossES-U ↓ES-R ↑MU ↑FTR ↑
Sum (α=0.2\alpha = 0.2).061 ± .001.657 ± .012.532 ± .001.719 ± .000
Sum (α=0.4\alpha = 0.4).061 ± .002.682 ± .012.531 ± .000.719 ± .001
Sum (α=0.6\alpha = 0.6).061 ± .003.714 ± .010.530 ± .001.718 ± .001
Sum (α=0.8\alpha = 0.8).060 ± .003.774 ± .005.530 ± .003.716 ± .002
FedCoDS.036 ± .000.931 ± .003.552 ± .001.738 ± .003

(b) Multiple-client unlearning (random partition)

Forget lossES-U ↓ES-R ↑MU ↑FTR ↑
Sum (α=0.2\alpha = 0.2).075 ± .002.971 ± .003.543 ± .001.669 ± .003
Sum (α=0.4\alpha = 0.4).070 ± .004.971 ± .009.543 ± .002.668 ± .003
Sum (α=0.6\alpha = 0.6).061 ± .002.970 ± .009.542 ± .003.670 ± .004
Sum (α=0.8\alpha = 0.8).049 ± .004.964 ± .005.540 ± .002.679 ± .004
FedCoDS.045 ± .002.973 ± .006.543 ± .008.652 ± .004
AuthorsOpenReview ↗

Answer to Reviewer ctFH [Part 2]

Q3:” Do the gains still hold when the same KL term on the retain data added to all baselines?”

Thank you for this suggestion. We conducted additional experiments by adding the same retain-side KL regularization to the fine-tuned model for NPO, SimNPO, and SATIMP on TOFU. The results confirm that retain-KL generally improves retention, increasing ES-R in 10 of the 12 baseline–setting comparisons.

Importantly, FedCoDS maintains its superior forgetting–retention trade-off. It achieves both lower ES-U and higher ES-R in 11 of the 12 comparisons. For example, in RS, FedCoDS achieves ES-U/ES-R of 0.036/0.931, compared with 0.090/0.906 for SATIMP + retain-KL. In SM, FedCoDS further reduces ES-U from 0.076 to 0.045 while improving ES-R from 0.923 to 0.971. In SS, although SATIMP + retain-KL achieves slightly higher ES-R (0.938 vs. 0.921), FedCoDS provides substantially stronger forgetting (0.037 vs. 0.086 ES-U).

Table. Effect of retain KL regularization on TOFU Forget05. (a) Single-client and (b) multiple-client unlearning under author-silo and random partitions. All baselines are augmented with retain-side KL regularization, as in FedCoDS. Results are reported as mean ± standard deviation over three seeds; bold indicates the best mean among unlearning methods. FedCoDS achieves both lower ES-U and higher ES-R in 11 of 12 baseline comparisons, demonstrating that its forgetting–retention gains extend beyond retain-side KL regularization alone.

(a) Single Client

SS: author silos

MethodES-U↓ES-R↑MU↑FTR↑
Finetuned.978.979.544.492
Retrain.087.976.543.634
NPO + KL.095±.001.751±.012.528±.002.598±.002
SimNPO + KL.074±.017.712±.055.513±.002.620±.005
SATIMP + KL.086±.003.938±.005.533±.004.645±.010
FedCoDS.037±.004.921±.003.540±.004.726±.002

RS: random

MethodES-U↓ES-R↑MU↑FTR↑
Finetuned.990.971.542.495
Retrain.093.980.546.647
NPO + KL.055±.006.750±.017.519±.001.587±.003
SimNPO + KL.096±.015.728±.001.534±.009.623±.009
SATIMP + KL.090±.010.906±.003.554±.002.603±.015
FedCoDS.036±.000.931±.003.552±.001.738±.003

(b) Multiple Clients

SM: author silos

MethodES-U↓ES-R↑MU↑FTR↑
Finetuned.980.982.527.499
Retrain.089.984.544.641
NPO + KL.120±.006.887±.009.523±.006.594±.010
SimNPO + KL.085±.003.886±.006.525±.009.557±.009
SATIMP + KL.076±.015.923±.013.510±.010.666±.016
FedCoDS.045±.006.971±.007.521±.009.674±.002

RM: random

MethodES-U↓ES-R↑MU↑FTR↑
Finetuned.979.984.559.486
Retrain.090.984.561.636
NPO + KL.092±.003.906±.012.536±.010.587±.004
SimNPO + KL.090±.007.872±.011.533±.005.579±.012
SATIMP + KL.050±.013.925±.008.536±.012.683±.033
FedCoDS.045±.002.973±.006.543±.008.652±.004
AuthorsOpenReview ↗

Answer to Reviewer ctFH [Part 3]

Q4/W3: “Beyond the shared base model, which parts of FedCoDS are specific to federated learning, and how does it compare with existing methods in centralized unlearning?”

Thank you for this comment. We agree that FedCoDS builds upon established components, including knowledge distillation, token-wise SimNPO, and retain regularization. Our main contribution is the proposed multiplicative coupling of distillation and likelihood suppression, specifically motivated by the challenges of federated LLM unlearning, rather than a modification of the underlying FedAvg algorithm.

Unlike centralized unlearning, federated unlearning introduces two challenges: (1) heterogeneous forget data across clients can produce conflicting local forgetting directions, weakening updates during aggregation; and (2) a single requesting client has access to only limited local retain data, making it difficult to preserve globally learned knowledge. FedCoDS addresses these challenges through its coupled objective. Specifically, distillation toward the shared pretrained model provides a common reference across clients, while token-wise suppression targets each client's local forget data. Their multiplicative coupling weights likelihood suppression by the teacher–student discrepancy, reducing suppression pressure as their distributions converge, while weighting distillation by the forget-token likelihood-dependent loss. In addition, retain-side KL regularization toward the globally fine-tuned model helps preserve knowledge acquired during federated training, even when only local retain data are available.

Our experiments demonstrate the effectiveness of this design. In the multi-client setting, FedCoDS achieves a mean pairwise forget-gradient cosine similarity of 0.86, compared with 0.56 for SATIMP and 0.24/0.29 for SimNPO/NPO. On TOFU-SM, FedCoDS reduces ES-U from 0.090 (SATIMP) to 0.045 while improving ES-R from 0.961 to 0.971. Furthermore, our new controlled ablation demonstrates the benefit of multiplicative coupling over simply combining existing objectives: in TOFU-RS, FedCoDS achieves ES-U/ES-R of 0.036/0.931, compared with 0.060/0.774 for the best tested additive combination.

We compare FedCoDS with established centralized unlearning SOTA (NPO, SimNPO, and SATIMP) adapted to the same federated training framework. Although the proposed objective can also be applied in centralized settings, its design is motivated by limited local data access and cross-client forget alignment, which are characteristic challenges of federated unlearning.

We will clarify our contribution statement to emphasize the federated motivation and novel coupling mechanism, rather than claim novelty in the individual components or FedAvg itself.

W4:” […] The paper notes that forgetting is best judged by closeness to the retrained model, but the comparisons treat lower scores on the forget data as better.”

The statement: “Forget-side generation scores are interpreted relative to Retrain; lower scores alone do not establish better unlearning” is incorrect! This sentence was inadvertently retained from an earlier draft. We sincerely apologize for this error and the confusion that it caused.

In our evaluation, on TOFU, lower ES-U indicates better forgetting shown by ES-U ↓, while higher ES-R indicates better retention of learned knowledge shown by ES-R ↑. Higher MU indicates better overall model utility shown by MU ↑, and higher FTR indicates better forget truth ratio shown by FTR ↑. On MUSE, lower VerbMem indicates less verbatim memorization of forget data shown by VerbMem ↓, and lower KnowMem indicates less retained knowledge of the forget data shown by KnowMem ↓. Higher UtilPres indicates better preservation of model utility shown by UtilPres ↑. PrivLeak closer to zero indicates better privacy performance shown by PrivLeak → 0. The Retrain row is listed in the tables only as a reference.

The only exception is PrivLeak on MUSE, which is why it is the only metric marked “→ 0”. PrivLeak is defined relative to Retrain: it is the relative difference between the membership-inference AUC of the unlearned model and that of Retrain. A value of 0 means that the unlearned model separates the forget data from unseen data exactly as Retrain does, so Retrain is 0 by definition and a value closer to 0 is better.

Reviewer_CZxLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FedCoDS for federated LLM unlearning. A single requester uses its local retain set, while multiple requesters combine their updates. FedCoDS uses SimNPO to weight distillation token by token, and KL and cross-entropy losses to preserve performance on the remaining data. In experiments, FedCoDS consistently achieves a better balance between forgetting and retention than state-of-the-art baselines.

Critical strengths

This paper explicitly distinguishes between single-client and multi-client unlearning problems and proposes a interesting method. Experiments conducted on 1B and 7B models demonstrate the effectiveness of the proposed approach.

Critical weaknesses

The paper fails to provide a sufficient justification for the necessity of the proposed multiplicative coupling. The ablation study compares FedCoDS only against independent KD and T-SimNPO, omitting an ablation that considers a simple weighted combination of the two loss functions. Furthermore, FedCoDS introduces several setting-specific hyperparameters, yet the paper does not report how these parameters were selected. Consequently, it remains unclear whether the performance gains stem from the proposed coupling mechanism or from extensive hyperparameter tuning.

Questions for authors

[1] On pages 3–4, in the section "Likelihood-based Unlearning Objectives... Loss Regularization," the text could be interpreted in two ways: (1) that these two challenges are pervasive across all federated unlearning scenarios; or (2) that these challenges are introduced by the likelihood-based objectives themselves, even though they actually stem from federated data isolation and data heterogeneity.

[2] The core contributions of the paper should be summarized, and the inclusion of quantitative results is recommended.

[3] FedCoDS introduces several hyperparameters, such as β\beta, β1\beta_1, β2\beta_2, λf\lambda_f, λKL\lambda_{\mathrm{KL}}, and λCE\lambda_{\mathrm{CE}}, which vary significantly across different datasets and experimental settings. This makes it difficult to determine whether the reported performance gains stem from the proposed coupling method or simply from hyperparameter tuning tailored to specific settings.

[4] The paper does not demonstrate that the proposed multiplicative coupling is necessary. The ablation studies only compare FedCoDS against independent implementations of KD and T-SimNPO. It is important to verify whether the performance gains truly arise from the multiplicative coupling, rather than merely from the combination of the two individual components.

[5] The current federated setting, as presented for multi-client scenarios, is overly idealized. It is recommended to consider more realistic conditions, such as varying numbers of requesting clients, partial participation, and imbalances in local data sizes.

5 comments
AuthorsOpenReview ↗

Answer to Reviewer CZxL [Part 1]

Q1 “ […]The current wording may cause two misunderstandings […]Please revise the motivation to clarify these two points.”

Thank you for pointing out these ambiguities. The two challenges arise separately in the multi-client and single-client unlearning settings and stem from data distribution and heterogeneity, rather than from likelihood-based objectives themselves.

Specifically, in the single-client setting, the requesting client has access only to its local retain data, making it difficult to preserve knowledge learned from other clients. In the multi-client setting, heterogeneous local forget data can produce inconsistent forgetting directions, weakening the effectiveness of federated aggregation. Although likelihood-based objectives effectively suppress targeted forget responses, they do not explicitly address these challenges. Based on this discussion, we plan to revise the motivation in Section 4 as follows:

Federated LLM unlearning presents distinct challenges depending on the distribution of forget data. First, in the multi-client setting, clients hold different local forget data and therefore suppress different tokens’ likelihood, which can lead to inconsistent local forgetting directions during aggregation. Second, in the single-client setting, preserving model utility is challenging because the requesting client has access only to its limited local retained data for retain loss regularization. Existing likelihood-based methods cannot address these challenges because they rely primarily on decreasing the likelihood of targeted forget tokens. The targeted tokens vary across clients under data heterogeneity. Also, aggressive likelihood suppression can disrupt knowledge acquired during fine-tuning when only a small retain set is available for regularization. These limitations motivate us to develop an unlearning objective that promotes more consistent client forgetting updates while preserving fine-tuned knowledge with limited retain data.

Q2 ”The core contributions of the paper should be summarized rather than stated in general terms; it is recommended to include quantitative results.”

Thank you for this suggestion. We will revise the contributions to highlight quantitative improvements. We plan to revise the last bullet of combinations as follows:

We evaluate FedCoDS against SOTA on TOFU and MUSE under both single- and multi-client settings. Compared to STAIMP, which is the highest-performing SOTA, on TOFU, FedCoDS reduces forget extraction strength by approximately 50% over STAIMP in the multi-client setting while maintaining retain extraction strength above 0.97. On MUSE, FedCoDS achieves the privacy-leakage scores closest to zero for the News and Book datasets, for both single- and multi-client settings.

AuthorsOpenReview ↗

Answer to Reviewer CZxL [Part 2]

Q3 ” FedCoDS introduces several hyperparameters[…]”

Thank you for bringing up the missing hyperparameter search. We will expand Appendix A to report the range of hyperparameters for both TOFU and MUSE.

On TOFU, we tuned five parameters: the learning rate, the T-SimNPO parameter β, and the loss weights λf\lambda_{f}, λKL\lambda_{KL}, and λCE\lambda_{CE}, while fixing (β1\beta_{1}, β2\beta_{2}) = (1, 1). We initially explored learning rates from 1×10−51\times 10^{-5} to 1×10−21\times 10^{-2}, and then did a fine-grained search on the reduced range shown in the table. On MUSE, we additionally varied β1\beta_{1} and β2\beta_{2}. The gap-regularization coefficient λg\lambda_{g} is fixed at 1 for MUSE-News with a single requester and at 0 otherwise. Within each setting, the selected hyperparameters are fixed across all three random seeds. We choose the configuration that provides good performance for both forgetting (ES-U/ PrivLeak) and retaining (ES-R/ UtilPres).

TOFU search ranges

SettingLearning rateβλf\lambda_{f}λKL\lambda_{KL}λCE\lambda_{CE}
SS3×10−53\times 10^{-5} – 1×10−41\times 10^{-4}4–61–2{1,2,4,8}{2,3,4,5,6}
RS3×10−53\times 10^{-5} – 1×10−41\times 10^{-4}4–61–2{1,2,4,8}{2,3,4,5,6}
SM5×10−45\times 10^{-4} – 2×10−32\times 10^{-3}1–21–2{1,2,4,8}{2,3,4,5,6}
RM5×10−45\times 10^{-4} – 2×10−32\times 10^{-3}1–21–2{1,2,4,8}{2,3,4,5,6}

MUSE search ranges

SettingLearning rateβ(β1\beta_{1}, β2\beta_{2})(λf,λKL,λCE\lambda_{f},\lambda_{KL},\lambda_{CE})
News · RS1×10−51\times 10^{-5} – 1×10−31\times 10^{-3}1–2(0.1–2, 0.1–2)(1–2, 1–8, {2,3,4,5,6})
News · RM1×10−51\times 10^{-5} – 1×10−31\times 10^{-3}1–2(0.1–2, 0.2–2)(1–2, 1–8, {2,3,4,5,6})
Books · RS1×10−51\times 10^{-5} – 1×10−31\times 10^{-3}1–2(0.1–2, 0.2–2)(1–2, 1–8, {2,3,4,5,6})
Books · RM1×10−51\times 10^{-5} – 1×10−31\times 10^{-3}1–2(0.1–2, 0.2–2)(1–2, 1–8, {2,3,4,5,6})

Effect of different numbers of seeds on MUSE. Note that due to time constraints, our original submission reported the best selected checkpoint rather than the three-seed mean and standard deviation. We acknowledge that this did not adequately characterize run-to-run variability. We have now rerun the evaluation with three seeds and report the mean ± standard deviation for all methods.

Table. MUSE privacy leakage and utility preservation over three random seeds. (a) Single-client and (b) multiple-client unlearning. Results are mean ± standard deviation; PrivLeak closer to zero and higher UtilPres are preferred. Bold indicates the best mean among unlearning methods in each metric. FedCoDS achieves the mean PrivLeak closest to zero in both single-client and multiple-client settings while preserving higher UtilPres than SimNPO and SATIMP.

(a) Single-Client

MethodBooks: PrivLeak → 0Books: UtilPres ↑News: PrivLeak → 0News: UtilPres ↑
Finetuned-10049.1−10044.2
Retrain050.4041.3
NPO−100.0 ± 0.046.1 ± 5.0−92.8 ± 0.938.9 ± 1.7
SimNPO−14.9 ± 22.944.0 ± 0.5−87.3 ± 3.931.4 ± 1.3
SATIMP−21.8 ± 3.813.5 ± 23.5−93.0 ± 6.722.4 ± 9.0
FedCoDS−6.4 ± 13.444.4 ± 3.1−6.5 ± 1.735.2 ± 3.2

(b) Multiple Client

MethodBooks: PrivLeak → 0Books: UtilPres ↑News: PrivLeak → 0News: UtilPres ↑
Finetuned−10051.4−10043.2
Retrain051.5039.9
NPO−100.0 ± 0.051.4 ± 1.1−99.5 ± 0.336.5 ± 2.4
SimNPO−30.9 ± 5.549.9 ± 0.9−46.4 ± 44.328.8 ± 1.6
SATIMP−39.0 ± 5.748.1 ± 0.4−58.8 ± 25.633.1 ± 1.2
FedCoDS−20.1 ± 7.750.9 ± 3.0−18.3 ± 19.237.5 ± 0.9

The additional results confirm that FedCoDS achieves the mean PrivLeak closest to zero in all four MUSE settings: −6.5 and −18.3 on News, and −6.4 and −20.1 on Books, for single- and multiple-client unlearning, respectively. On News with multiple clients, FedCoDS achieves PrivLeak of −18.3, compared with −46.4 for SimNPO and −58.8 for SATIMP, while improving UtilPres to 37.5 versus 28.8 and 33.1, respectively. FedCoDS also achieves higher mean UtilPres than SimNPO and SATIMP across all four settings. Although NPO maintains comparable utility, its PrivLeak remains between −92.8 and −100, indicating substantially weaker forgetting.

These results demonstrate that FedCoDS maintains a favorable privacy–utility trade-off across different seeds, rather than relying on a single best-performing checkpoint. We will update the appendix to include the full search configurations, checkpoint-selection protocol, and three-seed results.

AuthorsOpenReview ↗

Answer to Reviewer CZxL [Part 3]

Q4 ” […] I want to check whether the gains come specifically from multiplicative coupling rather than merely combining the two components.”

Thank you for bringing up this point. Multiplicative coupling models the interaction between the two objectives. While a weighted sum combines KD and T-SimNPO with fixed coefficients, multiplication allows each term to scale the gradient of the other. The product rule gives ∇(KD×T-SimNPO)=T-SimNPO⋅∇KD+KD⋅∇T-SimNPO\nabla(\text{KD}\times\text{T-SimNPO})=\text{T-SimNPO}\cdot\nabla\text{KD}+\text{KD}\cdot\nabla\text{T-SimNPO}. Thus, likelihood suppression is weighted by the teacher–student discrepancy, reducing suppression pressure as their distributions converge, while distillation is weighted by the forget-token likelihood-dependent loss.

We did an ablation study where the multiplicative forget loss is replaced with a weighted additive combination of KD and T-SimNPO, (1−α)⋅KD+α⋅T-SimNPO(1-\alpha)\cdot\text{KD}+\alpha\cdot\text{T-SimNPO}, for α∈0.2,0.4,0.6,0.8\alpha\in\\{0.2,0.4,0.6,0.8\\}. All other settings, including the learning rate, loss weights, retain regularization, and training rounds, were kept the same. Results are averaged over three seeds.

In the single-client setting (RS), FedCoDS reduces ES-U from 0.060 (best additive) to 0.036 while improving ES-R from 0.774 to 0.931. In the multi-client setting (RM), the additive combination with α=0.8\alpha=0.8 achieves comparable ES-U (0.049 vs. 0.045) with FedCoDS, but FedCoDS achieves much higher ES-R (0.973 vs. 0.964).

Table. Multiplicative coupling versus additive combination on TOFU Forget05. (a) Single-client (RS) and (b) multiple-client (RM) unlearning with random partitions. Sum(α) denotes (1−α)KD+αT-SimNPO(1-\alpha)\text{KD}+\alpha\text{T-SimNPO}. Results are mean ± standard deviation across three seeds; bold marks the best mean. FedCoDS achieves lower ES-U and higher ES-R than all additive combinations.

(a) Single-client unlearning (random partition)

Forget lossES-U ↓ES-R ↑MU ↑FTR ↑
Sum (α=0.2\alpha = 0.2).061 ± .001.657 ± .012.532 ± .001.719 ± .000
Sum (α=0.4\alpha = 0.4).061 ± .002.682 ± .012.531 ± .000.719 ± .001
Sum (α=0.6\alpha = 0.6).061 ± .003.714 ± .010.530 ± .001.718 ± .001
Sum (α=0.8\alpha = 0.8).060 ± .003.774 ± .005.530 ± .003.716 ± .002
FedCoDS.036 ± .000.931 ± .003.552 ± .001.738 ± .003

(b) Multiple-client unlearning (random partition)

Forget lossES-U ↓ES-R ↑MU ↑FTR ↑
Sum (α=0.2\alpha = 0.2).075 ± .002.971 ± .003.543 ± .001.669 ± .003
Sum (α=0.4\alpha = 0.4).070 ± .004.971 ± .009.543 ± .002.668 ± .003
Sum (α=0.6\alpha = 0.6).061 ± .002.970 ± .009.542 ± .003.670 ± .004
Sum (α=0.8\alpha = 0.8).049 ± .004.964 ± .005.540 ± .002.679 ± .004
FedCoDS.045 ± .002.973 ± .006.543 ± .008.652 ± .004

Comments on the paper

1 comment

We thank the reviewers for their constructive comments and discussions. Based on their feedback, we have revised the manuscript and incorporated some of the additional experimental results and clarifications to address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.