acceptodds
Under review as a conference paper at ICLR 2027

Prepare in Proportion: Deletion-Hazard-Aware Training for Unlearning-Ready Federated Language Models

Abstract

Federated unlearning (FU) removes a withdrawing client's contribution from a model trained across clients, and preparing at training time lowers the cost of later requests. Preparation faces a distinctive obstacle: it is bought with utility, and one budget must be divided across clients whose withdrawal likelihood differs and is predictable from signals the deployer holds. Current methods isolate every client's contribution equally, and they spend its utility cost on clients who will never leave. To address these, we first formulate preparation as a budgeted allocation over deletion hazards, whose uniform-preparation excess has a closed form in the hazard variance. We then design PREMIUM, a framework that prepares each client in proportion to its hazard of deletion. Within this framework, Hazard Estimation fits a discrete-time survival model on operating signals and returns a deletion probability per client. Budgeted Readiness Allocation then solves the budgeted program by water-filling with one dual price. In addition, Readiness-Proportional Training routes each client's gradient between a shared and a private adapter in the assigned share, so a request detaches the private part and removes the shared part in the priced steps. Across three hazard profiles on Reddit and six published baselines at a common utility budget, PREMIUM needs fewer deletion steps than every baseline and less than half the steps of uniform preparation when hazards are dispersed, with comparable forgetting and membership inference. For example, with fitted hazards, PREMIUM serves a request in 110 steps, against 243 under uniform preparation and 272 for MemSinks, the strongest published baseline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, federated learning, language models, unlearning-ready training, survival analysis, budgeted allocation, data deletion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_87kNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on a novel issue: Different clients have different probabilities to quit training, and further propose PREMIUM, a framework that prepares each client in proportion to its hazard of deletion. Experiments demonstrate the effectiveness of proposed methods.

Critical strengths
  1. The paper identifies that clients have different probabilities of issuing future deletion requests, while uniform preparation may waste the limited utility budget on low-risk clients. Its a novel perspective beyond conventional request-time unlearning.
  2. The experiments are comprehensive and convincing. The ablation studies and scalability experiments further support the proposed design.
Critical weaknesses
  1. The deletion cost metrics and baseline comparisons make me confused. Eq. (4) optimizes the expected total cost over a compliance period, but the reported 110 steps are sometimes described as the cost per request. These are different metrics. Table 6 also reports 652 deletion steps when all deletion hazards are zero, which needs explanation.

  2. The practical feasibility of request-time unlearning is unclear. Section 3.4 requires gradient ascent on the departing client's loss, but the server normally has no access to client data, and the client may already be offline. The authors should clarify how this operation is performed.

  3. The theoretical analysis is established based on strong assumptions. Eq. (3) assumes a linear relationship between deletion cost and shared exposure, but the exposure identity in Eq. (8) does not establish this relationship for actual unlearning steps. Moreover, Assumption 2 in Appendix C.1 already assumes a similar bound, making the theoretical guarantees somewhat dependent on the assumed cost model.

Questions for authors

Please see weaknesses above.

Reviewer_AxRpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a deletion-hazard-aware preparation framework for federated unlearning that efficiently removes a client's contribution when the client withdraws from the system. The framework computes the deletion probability for each client to prepare each client in proportion to its hazard of deletion. It then converts those predicted deletion probabilities into private shares and balances preparation efforts across clients. The clients' contributions (private adapter) are routed away from the federated training (shared adapter) in the assigned share based on the predicted deletion probabilities, so a request detaches the private part and removes the shared part in the priced steps. Evaluations across three hazard profiles on Reddit, Stack Overflow, and GitHub data show that, with a common utility budget, PREMIUM is more efficient, requiring fewer deletion steps and less than half the steps of uniform preparation, with comparable forgetting and membership inference.

Critical strengths
  • The paper addresses a meaningful problem in personalized federated unlearning in practical resource-constrained environments. Experimental results show that the proposed framework improves efficiency compared to conventional solutions.
  • Each of the three modules in the framework (i.e., HE, BRA, and RPT) has its own role in the system, and the judgment of the implementation is clear.
  • Experiments span multiple datasets, showing the robustness of the proposed framework. Evaluations are across multiple perspectives, including model utility, forget quality, MIA performance, and costs.
Critical weaknesses
  • The hazard estimation might not be practical in real-world scenarios since the assumption is that the probability of unlearning requests can be predicted from historical events. However, the request probability does not follow a pattern in practice most of the time.
  • While the time and communication costs for different model sizes, adapter ranks, optimizer types, and training realization are compared in the appendix, the results mainly focus on deletion cost and fail to report the full cost for all stages of the experiments.
  • In practical federated learning systems, the clients' data might be unbalanced and non-IID. There is a lack of analyses on such scenarios.
Questions for authors
  • In equation (3), why would the authors assume that the expected cost of a request is proportional only to the shared exposure? As there are many factors, such as unlearning difficulties (gradient alignment between clients, information retention in the model, etc.), should the authors consider additional factors in the cost function?
  • How would the authors determine the deletion threshold? Why would it be considered that unlearning is sufficient at the threshold?
  • How robust is the proposed hazard prediction to shifts in future deletion behavior?
  • Can the authors provide the unlearning efficacy compared to the retraining baseline (i.e., retrain the model without unlearning the client's data)?
Reviewer_FuBVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PREMIUM, a novel framework for federated unlearning that prepares federated language models for future client deletion by estimating deletion hazards and allocating preparation resources accordingly. Experiments on Reddit, Stack Overflow, and GitHub demonstrate its potential to reduce unlearning costs while maintaining model utility.

Critical strengths
  1. The paper takes a proactive approach to federated unlearning by incorporating potential future client deletion requests into the training process. Considering client-specific deletion hazards provides a meaningful alternative to uniform preparation strategies.
  2. The paper presents a unified pipeline that integrates deletion hazard estimation, budgeted readiness allocation, and readiness-proportional training, with a theoretically motivated allocation strategy.
  3. Experiments on Reddit, Stack Overflow, and GitHub demonstrate the potential effectiveness of the method in reducing unlearning costs while maintaining model utility, with additional evaluations of forgetting quality and robustness under different settings.
Critical weaknesses
  1. The proposed allocation strategy relies on strong assumptions about the deletion-cost model and client deletion hazards. In particular, the assumed linear relationship between shared contributions and deletion costs may not fully capture the complexity of real-world federated unlearning.
  2. The discussion of privacy needs further clarification. DP-SGD is applied only to hazard estimation, leaving unclear how shared updates and audit records are protected.
  3. Forgetting-quality improvements are limited, and the efficiency–quality trade-off needs further examination. In Table 1, PREMIUM achieves an FQ of 0.74, while below Exact-Fun (0.82). PREMIUM’s utility is also lower than that of uniform preparation (0.659) and MemSinks (0.688). The paper should verify whether the efficiency advantage persists at matched forgetting quality and retained utility.
Questions for authors
  1. How accurately does the linear deletion-cost model predict actual steps across individual clients? When the model overestimates or underestimates these costs, does the proposed allocation still outperform uniform preparation?
  2. Given the differences in retained accuracy and FQ in Table 1, does PREMIUM’s efficiency advantage persist when both retained utility and forgetting quality are matched?
  3. Does PREMIUM assume a trusted server? Who can access the shared updates and client-level audit records?
  4. How much additional computation and storage do private adapters, hazard refitting, and exposure auditing require relative to uniform preparation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.