- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study recipient-specific federated unlearning for large language models. They propose PULSAR, which allows different clients to forget different source clients’ data. The method uses source-specific filters and sends them only to the clients that need to forget the data. It also uses local optimization to reduce the loss in task performance. Experiments on TOFU and MUSE-News cover several LLMs, federated aggregation methods, and unlearning algorithms. The results show that PULSAR can achieve forgetting performance close to retraining while preserving reasonable utility for other clients.
Critical strengths
-
The paper addresses an important and practical problem in federated unlearning. Existing methods usually produce one unlearned global model, so all clients are affected by the same forgetting request. PULSAR introduces a source–target relation and allows only selected clients to forget specific source data. This setting is useful when different clients have different data permissions.
-
The proposed framework use different underlying unlearning algorithms and convert their effects into source-specific filters. The server then sends these filters only to the clients that need them. This design does not require a completely new unlearning algorithm and may be easier to integrate into existing federated learning systems.
-
The method supports reversible and client-specific model updates. The filters are applied during model delivery, while relative updates are used for global aggregation. As a result, the forgetting operation does not need to be permanently written into the global model. This makes it possible to revoke a forgetting relation and restore the normal model for a client.
Critical weaknesses
-
The paper mainly uses FTR, ROUGE, and model utility to measure forgetting. It does not report the original TOFU Forget Quality metric or sufficient privacy tests such as membership inference. In addition, multiple filters are combined by a weighted average, but this is not equivalent to unlearning the union of the corresponding datasets.
-
The paper does not clearly explain how the source, target, and authorized clients are assigned or how many samples each client receives. It also does not provide enough information about random seeds, repeated runs, or statistical significance. Some FedAvgM and FedYogi results in Tables 5 and 6 are nearly identical across many settings, which may indicate a reporting or implementation problem.
Questions for authors
-
The paper evaluates several unlearning algorithms within the PULSAR framework, but it does not provide direct comparisons with existing personalized or reversible federated unlearning methods, such as FUSED, ZeroFU, Mimir, or ConPFU. Why are these methods not included as baselines?
-
Can the authors compare PULSAR with the following baselines under the same data split and computing budget: full retraining, global unlearning for all clients, local unlearning for each target client, and a standard personalized federated learning method without filtering?
-
Tables 5 and 6 show almost identical results for FedAvgM and FedYogi in many settings. Are these values correct? Please provide the results for every seed and explain how the two aggregators were implemented.
-
In PULSAR, each source client must run the underlying unlearning algorithm locally and upload a source-specific filter. What happens if the source client refuses to participate, is offline, or uploads an incorrect or malicious filter? Can the server generate or verify the filter without access to the source client’s data? If not, the selective forgetting guarantee appears to depend on source-client cooperation.
3 comments
Follow-up on Author Response
Dear Reviewer jBVo,
We hope our rebuttal and revisions have addressed your concerns. As the discussion deadline approaches, we would be happy to clarify any remaining questions.
If your concerns have been addressed, we hope that you may very kindly reconsider your rating for our paper, and perhaps champion our paper.
Best regards, Authors of Paper 7666
Summary of Revisions in Response to Reviewer jBVo
Dear Reviewer jBVo,
Thank you again for your careful review and constructive feedback. We have made several revisions and conducted additional experiments to address your concerns. The main changes are summarized below.
1. Baseline Comparisons and Evaluation Fairness.
(1) We further clarified the distinction between PULSAR and existing personalized/reversible FU methods (Response Q1). These methods do not directly implement the source–recipient authorization policy studied in our work and therefore are not directly equivalent baselines under the same task setting (Sec. 5).
(2) We also added comparisons with FFA, target-local unlearning, and filter-free personalized fine-tuning. In Table 2(b), PULSAR achieves comparable target forgetting to FFA (FTR: 0.658 vs. 0.644 on Llama-1B), while maintaining -level performance for source () and authorized () clients, thereby avoiding the collateral forgetting caused by global unlearning. We further compare PULSAR with a target-local unlearning oracle (Table 18). Under , , and , PULSAR achieves comparable or slightly higher FTR without requiring target clients to access the source-side forget set . Appendix F.11 further details the operational budgets and data-access assumptions of these baselines.
2. Unlearning and Privacy Evaluation.
We added the original TOFU Forget Quality (FQ) metric and membership inference attack (MIA) evaluations (Table 11). For example, on Llama-1B, improves the log10 FQ from −11.77 () to −1.50. Its Min-K/loss MIA AUCs are 0.379/0.384, closely matching (0.380/0.383). We also added external-domain evaluations on MUSE-News (Tables 26–27). The target 's MIA Min-K reaches 0.5141, close to (0.5075). Meanwhile, the authorized recipient 's MIA Min-K and PrivLeak remain highly consistent with , with mean absolute deviations of only 0.000081 and 0.016250, respectively, across eight unlearning operators and two aggregation operators .
3. Reproducibility and Experimental Consistency.
(1) We clarified the client roles and data partitioning in Appendix A.5. The source client holds 400 forget QA pairs and 1,200 retain QA pairs, while the target and authorized clients each hold 1,200 retain QA pairs. The partition of the 3,600 retain QA pairs is kept fixed across the three unlearning seeds (42, 43, and 44).
(2) We also added results across three random seeds (Tables 7–8). Table 7 reports the mean and standard deviation under , while Table 8 provides per-seed results for and on both model backbones. In response to the previously observed similarity between the and results, we rechecked the corresponding experiments and corrected the affected results.
4. Filter Composition and Threat Model.
(1) We added a union-data oracle comparison (Table 21, Appendix F.13) to demonstrate that normalized filter composition is not equivalent to unlearning the union of forget sets. For T123, normalized composition achieves mean ΔFTR/ΔFOR/ΔMU of +0.011/−0.015/+0.001, compared with +0.025/−0.050/−0.015 for the union-data oracle. This shows that union unlearning achieves stronger forgetting but incurs greater model utility loss.
(2) We also clarified the source-client cooperation requirements and the limitations of filter verification under the current threat model in Section 3.1. A cached filter can be reused when the source client is temporarily offline (Table 16(b), Appendix F.10). However, initial filter construction still requires source-client cooperation, as the server cannot independently construct or semantically verify a filter without access to the source-side forget data.
We would appreciate your reviewing the revised manuscript and our responses before the discussion closes. If you find the revisions satisfactory, we hope to have your support for our work.
Thank you again for your input.
Sincerely,
The Authors of Paper 7666
Response to Reviewer jBVo: Experimental Validation and Clarifications
Thank you for the valuable review. We below provide the point-to-point responses. Revisions are marked in blue.
Q1. Personalized/reversible FU baselines. FUSED, ZeroFU, Mimir, and ConPFU target reversibility, adaptation, or forgetting intensity. PULSAR instead specifies who-forgets-whom contribution: designated targets forget while authorized recipients retain source knowledge. These methods lack this explicit source–recipient policy. We therefore add policy-aligned comparisons with FFA, target-local unlearning, and filter-free personalized FT (Sec. 5, p.10, Apps. F.10–F.11). These controls test recipient-specific dispatch under the intended policy.
Q2①. Full retraining and budgets. All controls use Split90. Retrain serves as a full-cost reference with 11 retain-only rounds. FFA and the target-local oracle match operator/FT epochs, while filter-free controls share the local-FT budget (App. F.11, p.44). Retrain is an effectiveness reference, not an equal-cost control.
Q2②. Global unlearning. FFA broadcasts the same source-derived filter to all recipients. Target FTR is comparable (PULSAR/FFA: 0.658/0.644 Llama, 0.647/0.651 Qwen). Authorized-client FTR increases from 0.565/0.569 to 0.641/0.649, and FOR falls from 0.509/0.465 to 0.449/0.434. MU changes by only −0.004/−0.002. PULSAR avoids unintended forgetting of knowledge still authorized for non-target recipients (Table 2(b), p.9, Table 16(a), p.43).
Q2③. Target-local unlearning. The target-local oracle receives source forget data and applies the same operator. At κ=1 with matched epochs, PULSAR/oracle MU/FTR is 0.541/0.636 vs. 0.561/0.635 (GradDiff), 0.546/0.618 vs. 0.547/0.612 (NPO), and 0.540/0.631 vs. 0.557/0.628 (SimNPO). PULSAR matches FTR without sharing source forget data, though the oracle has higher MU (Table 18, p.45).
Q2④. Personalized FT without filtering. Finetune and No-F+JDOO reach FTR 0.571 and 0.583, versus 0.658 for PULSAR (Table 2(a), p.9). The source-derived filter contributes forgetting beyond local personalization. Together, these controls substantiate policy-specific forgetting.
Q3①. FedAvgM/FedYogi results. The original near-duplicates arose from a duplicated warm-up checkpoint. We corrected the checkpoint, reran affected warm-ups, and regenerated Tables 3–4. Llama Finetune MU/FTR is now 0.535/0.493 (FedAvgM) versus 0.467/0.687 (FedYogi) (p.33).
Q3②. Per-seed evidence. Table 8 (p.37) lists seeds 42/43/44 for both aggregators and backbones. FedYogi's Llama SimNPO FTR exceeds FedAvgM's in each reported seed (App. F.2, p.32).
Q3③. Implementation. FedAvgM uses server momentum. FedYogi adds a sign-controlled second-moment update, implemented using squared server momentum (App. A.4.2, pp.20–21). The corrected results and per-seed comparisons distinguish FedAvgM from FedYogi, addressing the reported duplication.
W2①–②. Client roles and seeds. Under Split90, client 0 is the source (400 forget + 1,200 retain QA), client 1 the target (1,200 retain), and client 2 authorized (1,200 retain). Table 1 reports mean ± SD over unlearning seeds 42/43/44, averaging ten aggregators per seed. FedAvg per-seed results (Table 7, p.37) improve FTR and FOR against same-seed Finetune for all eight operators on both backbones. Roles, partitions, and seed-level reporting are now explicit.
W2③. Aggregator correctness. See Q3 and corrected Tables 3–4.
W1①. TOFU Forget Quality. Table 11 (p.39) adds KS-based FQ. For GradDiff, the KS test does not reject at p>0.05 in 15/22 Llama and 18/22 Qwen evaluations (App. F.5, p.35). This adds the benchmark's distributional test beyond FTR/ROUGE.
W1②. Membership inference. Table 11 adds Min-K/loss MIA AUC and PrivLeak. For Llama/GradDiff, AUC is 0.379/0.384 versus Retrain's 0.380/0.383. PrivLeak is −0.2%/−8.9% (Llama/Qwen) versus −63.1%/−59.0% for Finetune (App. F.5, p.35). These metrics add privacy evidence beyond FTR/ROUGE.
W1③. Multiple filters. Normalized composition is not equivalent to union unlearning. Tables 20–21 (p.49) report per-relation results and a union-data oracle. On T123, normalized ΔFTR/ΔFOR/ΔMU is +0.011/−0.015/+0.001 versus +0.025/−0.050/−0.015 for the oracle (App. F.13). Composition preserves more MU at weaker forgetting than the union oracle.
Q4①–③. Source availability and trust. Filter construction requires a cooperative source with forget data. Without it, the server cannot construct or semantically verify the filter. A cached filter remains usable while the source is offline (mean ΔFTR +0.085 versus +0.087 with refresh, Table 16(b), p.43). Malicious filters are outside the protocol-compliant-source assumption (Sec. 3.1, p.4).