acceptodds
Withdrawn from ICLR 2027

ModULE: Learning Modular Expert Representations for Localized Machine Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Machine unlearning aims to remove the influence of specified training data while preserving performance on retained data. Existing methods typically modify models post hoc and often require broad parameter updates, resulting in substantial computational cost and collateral degradation. We study whether structured deletion requests can instead be made localizable to small expert subsets. We introduce Modular Unlearning via Localized Mixture of Experts (ModULE), a deletion-aware MoE framework that promotes routing-space localizability. During learning, ModULE combines sparse routing, balanced expert utilization, and centered kernel alignment to encourage sparse and differentiated expert pathways. Given a deletion request, it selects experts that provide high forget-route coverage with limited retain exposure and identifies retained examples whose routes intersect the selected experts. Unlearning then updates only the selected experts, while output and route distillation protect exposed retained examples and limit layer-wise routing shifts. Under hard routing and frozen shared parameters, retained inputs whose pathways avoid the updated experts remain unchanged. Experiments on structured class- and domain-level deletion, with random deletion as a stress test, show that ModULE improves the forgetting-utility trade-off while updating fewer parameters than matched dense and conventionally trained MoE baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_RypyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ModULE, a mixture-of-experts (MoE) framework designed to localize the impact of future machine unlearning requests. The framework integrates four learning objectives: routing concentration, gated-mass balancing, expert-representation differentiation, and route separation. The authors evaluate class-level and domain-level unlearning across PACS, OfficeHome, CIFAR-100, and Tiny-ImageNet.

Critical strengths

When deletion requests target known groups, such as classes or domains, encouraging group-dependent expert pathways during training offers a compelling alternative to global updates. The theoretical analysis in Lemma 1 remains appropriately bounded by its stated assumptions, providing a formal explanation for why localized updates can preserve exact predictions on the retain set. Furthermore, the paper compares ModULE against several full-model and MoE-based unlearning baselines, evaluating performance across forget accuracy (FA), retain accuracy (RA), test accuracy (TA), and membership-inference attack (MIA) metrics.

Critical weaknesses

(1) The proposed method combines several distinct objectives and introduces multiple hyperparameters, raising concerns regarding its practicality and generalizability. (2) Given the framework's complexity, thorough ablation studies are essential to isolate the impact of each individual component on performance. However, all ablation studies have been deferred to the Appendix, and the paper lacks an ablation evaluating the individual learning objectives introduced in Section 3.3. (3) It remains unclear whether the method is applicable to random unlearning. Although the authors state that they evaluate ModULE in this setting, no experimental results for random unlearning are provided. (4) It remains unclear what evidence demonstrates that forgotten information cannot be readily recovered from the frozen backbone, prediction head, or non-selected experts. (5) The evaluation results fail to establish clear superiority over strong baselines, and the ablation studies do not offer practical guidance on how to tune critical hyperparameters.

Questions for authors

Please see the weaknesses.

Reviewer_vnxROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of machine unlearning for MoE Models. The authors propose a method ModULE. The method follows the principle of unlearning-by-design which prepares the unlearning from training stage. In the training, 4 regularizers are considered so that data from similar classes mainly shares similar experts. When unlearning requests arrives, ModULE select the experts that cover most of the forget data and rarely used by retain data, and update these experts. Experiments show that ModULE achieves a good unlearn-utility trade-offs.

Critical strengths

Strengths:

1.The paper studies an important problem of unlearning for MoE structure, and follow the spirit of unlearning-by-design. The idea of unlearning during training is interesting.

2.The method is easy to understand and follow.

3.The paper covers many unlearning baselines from different categories and multiple datasets and includes different settings such as sequential unlearning.

Critical weaknesses

Weaknesses:

1.Main concern: The idea of unlearning by design is interesting, but it is not practical because the method needs to know the class or domain to be deleted at training time. In practice, for many requests, such as deleting one user’s data, a copyrighted document, or a piece of harmful knowledge, the corresponding class or domain is unknown. Further, for non-image data such as text, this is even less feasible. This is fundamentally different from other “unlearning by design” methods. For example, traditional data-sharding method like SISA. SISA does not rely on class or domain labels and directly uses sharding to reduce the cost of retraining.

2.One core idea of the paper is that we update only the selected experts is enough to forget, however in ablation results, this core idea is not supported. For example, in Tab. 4, Selected Experts Only FA score is 81.75 while Selected Experts + Last Encoder Block drop to 5.72. This means update only the selected experts is not enough and a good forgetting requires the last encoder block to be updated too, which does not satisfy the assumption of Lemma 1. The paper does not explain the gap.

3.The paper treats lower FA as better (Tab. 1, Figs. 2 and 3), but the quality of unlearning should be measured by the gap to the retrained model, since an FA lower than retraining means over-unlearning. For domain unlearning on PACS, ModULE’s FA is already lower than retraining (22.15% vs. 29.82%). The sequential results in Fig. 3 also claim the lowest FA at every step, without showing the retraining reference. I suggest reporting the gap to retraining.

Questions for authors

See weaknesses.

Reviewer_ZUKsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that post-hoc unlearning is costly and damages retained data. It further argues that standard MoEs, trained only for prediction, do not make deletion requests localizable. ModULE trains top-k expert adapters in two late blocks of a frozen DeiT-S with four regularizers: routing entropy, gated-mass balancing, inter-expert CKA and a group-expert mutual-information term (Eq. (2)-(5)). At deletion, it selects experts by forget coverage and retain exposure (Eq. (8)). It then updates only those experts, using gradient ascent plus retain, output and route distillation on retained samples whose routes intersect them (Eq. (10)). Evaluation covers class and domain deletion on PACS, OfficeHome, CIFAR-100 and Tiny-ImageNet against 15 baselines. The paper concludes that ModULE improves the forgetting-utility trade-off with fewer updated parameters.

Critical strengths
  1. The coverage-exposure criterion (Section 3.4) is a clear, testable notion of localizability, and the limits of Lemma 1 are stated candidly.
  2. Evaluation is broad: four datasets, three seeds, and sequential deletion (Figs. 2-3). It includes controls reported against the authors’ interest (Tables 4, 5, 12).
Critical weaknesses
  1. A plain MoE with ModULE’s unlearning procedure nearly matches ModULE (6.55 vs 5.72 FA, single run, Table 12). The gain therefore lies in the procedure, not the regularizers. Route separation is also absent from the default config (Appendix E) and from Table 13.
  2. Updating only the selected experts barely forgets (FA 81.75, Table 4). Full-model updating is best on all metrics in Table 5. Eq. (8) is never evaluated; Appendix E.1 reports no results.
  3. With token-level routing and δ = 0, the at-risk set likely covers most retained data, which makes Lemma 1 near-vacuous. The at-risk ratio promised in Section 4.1 is not reported.
  4. PACS class results are within one std of GRIP and SEUF (Table 1). Lowest-FA ranking rewards domain FA below retraining (Tables 1, 15), and Avg. Gap is not reported.
  5. The same PACS class setting yields FA 8.76, 5.72, 14.92 and 81.75 across Tables 1, 12, 6 and 4.
  6. Each component has a precedent: route separation follows Mod-Squad, balancing follows Switch Transformer, routing-based selection follows SEUF and TRACE, router stability follows GRIP, and retain distillation follows SCRUB.
  7. Training with known group labels is the setting of group-partitioned exact unlearning (ARCANE, SAFE, S3T), none of which is compared. TRACE is discussed but not compared. SCADA, POUR and Q-MUL are listed but not reported.
  8. “MoE (1)” is undefined.
Questions for authors
  1. What fraction of the retain set is at risk at δ = 0 in the Table 1 setting?
  2. Is route separation used in the main results, and are seed intervals available for Table 12?
  3. Do the baselines start from the ModULE-S checkpoint (Appendix D.4), and which split selected the learning rate?
  4. Which update scope produces Table 1, given FA 81.75 in Table 4? How is Eq. (8) solved?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.