acceptodds
Under review as a conference paper at ICLR 2027

Targeted Routing-Aware Calibration for Machine Unlearning in Mixture-of-Experts Language Models

Abstract

Machine unlearning aims to remove the undesired data from large language models (LLMs) while preserving model utility. As Mixture-of-Experts (MoE) LLMs become prevalent, there is a growing need for effective unlearning methods that account for their distinctive architecture. In this work, we observe that a small subset of experts has substantially higher activation frequencies on forget data than the other experts, while generic retain data activates these experts much less frequently. These observations motivate considering both which experts to update and how to direct retain regularization toward them. We propose , argeted outing-ware alibration for xperts, a two-stage framework for MoE unlearning. TRACE first selects experts with high activation frequencies on a small calibration subset of forget data. During unlearning, it reweights the retain loss to prioritize retain samples whose activation frequencies on the selected experts more closely match those of the forget data, providing targeted retain regularization. Experiments across multiple MoE LLMs show that TRACE preserves substantially higher utility at comparable forgetting levels, including a 4.55-point MMLU improvement over the strongest external baseline on Qwen1.5-MoE-A2.7B-Chat and a 3.96-point improvement on DeepSeek-V2-Lite-Chat.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r9wNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies gradient-based machine unlearning for sparse MoE LLMs. It identifies a forget–retain routing mismatch: a small subset of experts is activated disproportionately often by forget data, while generic retain data activates the same experts much less frequently, so under a GradDiff-style objective the retain gradient barely reaches the experts being unlearned. The proposed method TRACE has two stages. It first selects high forget frequency experts by applying an IQR outlier rule to the forget-side activation frequencies. During unlearning it updates only these experts and reweights retain samples by a softmax over their routing distance to the current forget batch, so that retain samples engaging the selected experts in forget-like patterns receive larger weight. Experiments on WMDP and MUSE-BOOKS report higher utility than baselines at comparable forgetting levels.

Critical strengths
  1. The routing mismatch is established as a mechanism rather than a mere observation, supported both empirically and by a per-expert gradient decomposition. Both components of TRACE follow directly from this analysis, so the paper reads as a coherent mechanism-to-method pipeline instead of a heuristic stack.
  2. The empirical gains are consistent across three MoE models and two benchmarks, and TRACE nearly matches a baseline using task-curated retain data while relying only on generic retain data. This targets the practically important setting, since avoiding curated retain data is the main operational cost the method claims to remove.
Critical weaknesses
  1. The baseline tuning appears weak enough to inflate the comparative claims. In Table 2, GA, GradDiff, and NPO collapse to 22.95 to 33.07% MMLU, far below what these methods typically retain on WMDP-style setups, and in Table 1 GA collapses to 0.00 retain KnowMem. Appendix C.1 shows a narrow search grid (learning rate in [5×10−5,10−4][5\times10^{-5}, 10^{-4}], λ∈[1,2.5]\lambda \in [1, 2.5]), and the strongest external baseline is ESFT, an expert-selection fine-tuning method repurposed for unlearning rather than a dedicated unlearning method. Since the headline contribution is utility preservation relative to baselines, the margins depend directly on how hard the baselines were tuned, and the current evidence does not establish that they were.
  2. The ablations study hyperparameters (λf\lambda_f, τ\tau, calibration set size) but not the method design itself, so the utility gains cannot be cleanly attributed to the proposed components. Three conditions are missing: reweighting without selection (for example GradDiff over all experts, or ESFT-selected experts, combined with routing-aware reweighting), a control for the reweighting direction (inverted or random weights of the same variance), and alternatives for the distance dmd_m (cosine, or unrestricted all-expert vectors, given that Figure 2 itself uses cosine similarity for analysis). This is consequential because the paper's central claim is that routing-aware retain reweighting causes the utility improvement; without these controls, the trivial explanation that any non-uniform retain weighting helps remains open.
Questions for authors
  1. What motivates the IQR outlier rule in Equation 3 for thresholding forget activation frequencies? The paper does not explain why distribution outliers should coincide with the experts worth updating, nor why quartiles are computed over all layers pooled together despite their different base activation rates. How does this compare to simpler rules such as per-layer top-K or a fixed quantile?
  2. TRACE-Select, the key ablation in Tables 2 and 3 and Figures 3 and 4, is never formally defined beyond table captions. Please state its objective, hyperparameters, and training budget. Note also that Table 3 compares TRACE and TRACE-Select at different steps (200 versus 250 on DeepSeek).
  3. On MUSE-BOOKS, GA, GradDiff, and NPO use LoRA with r=16r=16 (Appendix C.1), but the parameterization of TRACE, SEUF, and ESFT is not stated. Does TRACE update full parameters of the selected experts? If so, the comparison confounds the method with the trainable parameter budget and could explain the baseline collapses in Table 1.
  4. TRACE freezes the router while SEUF updates the router of its selected expert. Does freezing the router contribute to TRACE's utility advantage or limit its forgetting depth?
  5. Figure 1, 3, 4 are PNG/JPG images. Can you replace them with SVG?
Reviewer_AyJ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on the machine forgetting task of the MoE large language model and analyzes the problem of mismatch between the expert routing of Forget and Retain data. It proposes a two-stage method called TRACE. This method first selects high-frequency experts based on the frequency of expert activation, and then dynamically adjusts the Retain Loss weight through routing similarity to alleviate the degradation of model capability during the forgetting process. The research problem of this paper is novel to some extent, and the method design is simple. Moreover, it has achieved good forgetting and capability retention effects on multiple MoE models and forgetting tasks. However, the paper's verification of the universality of the Forget–Retain Routing Mismatch is still insufficient, and there is a lack of in-depth analysis of the interaction between the gradients of Forget and Retain. At the same time, the expert selection threshold and the stability of calibration data still need to be further verified. The necessity of using routing similarity instead of directly increasing the expert activation coverage for sample weighting in Stage 2 also lacks sufficient theoretical explanation and experimental support.

Critical strengths
  1. This paper focuses on the machine forgetting task of the MoE model, analyzes the differences in expert routing between the Forget and Retain data, and points out that such differences may lead to the lack of sufficient retention regularization for frequently activated experts, providing a new research perspective for the problem of knowledge forgetting and ability retention in the MoE model.
  2. TRACE combines HFF Expert Selection based on the frequency of expert activation and Routing-Aware Retain Reweighting to achieve selective parameter updates and retention loss adjustment without modifying the model architecture. The overall framework is relatively simple and easy to integrate into the existing MoE training process.
Critical weaknesses
  1. The paper takes "Forget-Retain Routing Mismatch" as the main research motivation, but the related analysis is mainly based on specific data combinations and has not fully verified the universality of this phenomenon in different fields, data distributions, and forgetting tasks. This makes the applicable scope of the TRACE method and its dependence on data distribution still unclear.
  2. The paper mainly adjusts the Retain Loss based on the activation frequency of experts, but does not deeply analyze the interaction between Forget and Retain gradients. Therefore, the mechanism explanation for the effectiveness of the method is still insufficient.
  3. TRACE relies on the selection of thresholds by experts and a small number of calibration samples to determine trainable experts. However, the current experiments have not fully verified the stability under different models, tasks, and random calibration subsets. At the same time, Stage 2 uses Routing Similarity for sample weighting, but has not fully demonstrated that this strategy has independent advantages compared to other more direct expert coverage weighting methods.
Questions for authors
  1. The paper mainly observed Routing Mismatch based on specific Forget/Retain data combinations. Is this phenomenon still prevalent in different fields, data distributions, and forgetting tasks?
  2. The paper analyzed the impact of different IQR thresholds on Forget Quality and Utility, but the two indicators did not show a consistent trend of change. In practical applications, how to determine appropriate thresholds for different models and datasets?
  3. The author introduced two stages. Have they ever tried a single stage, such as using some contrastive learning methods or studying the gradient update issues of the two? Because the forget loss and retain loss are two different gradient update directions?
  4. The paper analyzed the impact of different Calibration Set Sizes on the final performance, but has not verified the stability of different random calibration subsets under the same sample quantity. Will different Calibration Subsets significantly change the selected HFF Experts and further affect Forget Quality and Utility?
  5. In Stage 2, the Retain Loss weight is adjusted based on the routing similarity between the Retain Sample and the Forget Batch. Compared to directly increasing the activation coverage of HFF Experts by the Retain Sample, what is the advantage of using Routing Similarity? Can a control experiment comparing the two weighting strategies be supplemented?
Reviewer_UhJ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TRACE selects experts with unusually high activation frequency on a forget calibration set, using a global IQR outlier rule. During unlearning, it upweights retain samples whose selected-expert routing-frequency vectors are close to the current forget batch. Only selected expert parameters are updated.

Critical strengths

MoE-specific framing is valuable. The paper studies both update localization and retain-gradient delivery. The method is simple and operational. It adds no extra forward pass and updates a small parameter subset.

Critical weaknesses

The paper does not establish genuine unlearning. Chance-level WMDP accuracy and low ROUGE-L memorization scores are useful indicators, but do not establish that the targeted knowledge is inaccessible under alternative prompts or absent from the model. Each “original model” is first fine-tuned on the benchmark’s forget and retain data, so the study primarily tests removal of recently injected knowledge. Without retain-only, never-trained, or retrained-from-scratch references, it does not establish deletion of naturally acquired knowledge.

Questions for authors

How robust is forgetting to paraphrases, extraction attempts, and short relearning runs? Which models are fine-tuned on forget and retain data before unlearning, and can you provide retain-only or never-exposed references?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.