- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper addresses an important question in Machine Unlearning (MUL) community, how to apply MUL in mixture-of-experts model. The main motivation behind the work is accounting for routing during MUL, since some of the experts (e.g. activations) may still have information about the data to be forgotten. To account for this, the authors propose a new iterative unlearning algorithm which recursively finds highest-recovery route in MOE (with respect to the forget set), and updating the experts in this family. They provide strong empirical evidence for the iterative unlearning procedure proposed: against the SEUF baseline, RExU (the proposed algorithm by the authors) provides better forget-utility Pareto front. This is further strengthened by reasonable ablation studies. The closest prior work is SEUF, which also proposes to select experts for unlearning using an affinity-based localization. In addition, RExU investigates alternative routes in MOE and updates the weights for these that may still carry information. Theoretical novelty is modest and primarily explanatory, so most of the novelty relies on empirical gains.
Critical strengths
Strengths
- The empirical gains over prior work (SEUF) are promising, where RExU provides a better utility-forget trade-off. This is with respect to the range of MUL algorithms authors test, including gradient ascent and negative preference optimization. The promising empirical result is further displayed in paired analysis, where masking recovers 190 of 230 pretrained predictions that became wrong under SEUF’s route.
- The evaluation is bnroad with respect to unlearning objectives and models. The authors cover multiple MOE models, and use both multiple-choice and generation tasks, including low-rank adaptation to show the trade-off between MUL algorithms (in terms of forgetting efficiency and broader utility).
Critical weaknesses
Weaknesses:
- On the empirical side, while the experimental setup is broad (multiple MoE and tasks), the machine unlearning techniques compared against do not have strong baselines from classification literature, and NLP. For example, algorithms like SCRUB/NegGrad from Kurmanji et al., for cases where the retain set is available. This would broaden the scope of the paper, and show that, for the SOTA algorithms, the improvements still stand even if replicated in a reduced setting for MoE.
- The theoretical argument is mostly explanatory and does not have sufficient depth. For example, theorem 6 does not prove a convergence guarantee for the implemented algorithm. Theorem 3 has strong assumptions (Lipschitz) which may not hold true in practice, therefore highlighting a disconnect.
Questions for authors
- Do the improvement translate to other machine unlearning algorithms that show SOTA improvements beyond MoE? Gradient Ascent and preference optimisation are too narrow to qualify the claims, and it is well known that gradient ascent will induce overforgetting.
- Can the theoretical claims be adjusted to account for the strong assumptions, and to show convergence of the recursive operator proposed?
4 comments
Official Author Responses to Reviewer aHfX [2/3]
W1: On the empirical side, while the experimental setup is broad (multiple MoE and tasks), the machine unlearning techniques compared against do not have strong baselines from classification literature and NLP. For example, algorithms like SCRUB/NegGrad from Kurmanji et al., for cases where the retain set is available. This would broaden the scope of the paper, and show that, for the SOTA algorithms, the improvements still stand even if replicated in a reduced setting for MoE.
Q1: Do the improvement translate to other machine unlearning algorithms that show SOTA improvements beyond MoE? Gradient Ascent and preference optimisation are too narrow to qualify the claims, and it is well known that gradient ascent will induce overforgetting.
Thank you for suggesting SCRUB and NegGrad. We agree that GA alone can overforget and that comparisons with stronger retain-aware methods are important. We respectfully disagree only that our original evaluation was limited to GA and preference optimization: as already reported in §6.1 (Lines 420–426), it also covers retain-aware GDIFF and representation-level RMU alongside GA and NPO, with MoE-specific GRIP[R1] and TRACE[R2] comparisons in Table 10. NegGrad and NegGrad+ share objective-level principles with GA and GDIFF, respectively, but we do not treat this as an evaluation of their exact implementations.
Following your suggestion, we added the retain-aware teacher–student SCRUB baseline on DeepSeek-V2-Lite/WMDP-Cyber (Appendix F.4, Table 14, Lines 3080–3091):
| Method | Natural FE ↓ | WRFE ↓ | MMLU ↑ |
|---|---|---|---|
| SCRUB | 28.18% | 30.35% | 50.68% |
| SCRUB+SEUF | 28.23% | 35.88% | 48.60% |
| SCRUB+RExU | 28.49% | 30.20% | 54.10% |
Compared with SCRUB+SEUF, SCRUB+RExU reduces WRFE by 5.69 percentage points and improves MMLU by 5.50 points. Relative to standalone SCRUB, it attains comparable WRFE (30.20% vs. 30.35%) with +3.42 MMLU points. This supports RExU's benefit for an additional retain-aware objective in the tested setting, without claiming universal superiority across unlearning methods.
References
[R1] GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints. arXiv:2601.16905 (2026)
[R2] Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models. arXiv:2606.10338
Official Author Responses to Reviewer aHfX [3/3]
W2: The theoretical argument is mostly explanatory and does not have sufficient depth. For example, theorem 6 does not prove a convergence guarantee for the implemented algorithm. Theorem 3 has strong assumptions (Lipschitz) which may not hold true in practice, therefore highlighting a disconnect.
Q2: Can the theoretical claims be adjusted to account for the strong assumptions, and to show convergence of the recursive operator proposed?
Thank you for raising these concerns. We agree that Theorem 3 relies on a restrictive assumption and that Theorem 6 does not prove convergence. We would, however, distinguish the conditional results from our existing structural guarantees: as established in Appendices C.6–C.7, Lemma 5 shows exact invariance when a routed computation avoids all edited experts under fixed router/shared parameters, and Theorems 4–5 give optimization-independent lower bounds on residual recovery and necessary edit budgets. Neither requires Lipschitz continuity or convergence of RExU.
As already stated in Appendix C.5, Theorem 3 uses local Lipschitz continuity only as a sufficient condition for approximate redundancy; the existing exact-copy special case (Lines 1512–1524) preserves the target margin without a quantitative Lipschitz bound. We do not claim the assumption holds universally in practical MoEs.
Theorem 6 gives a conditional one-round descent bound, not an unconditional convergence guarantee. Progress requires the targeted recovery reduction to exceed cross-route shifts and finite-audit error. As already illustrated in Appendix I (Figure 12), WRFE rises from 27.78% at round 5 to 36.03% at round 7 as recovery shifts between routes, showing why monotonic progress cannot be assumed. Following your feedback, we clarified in Appendix C.8 (Lines 1935–1951) that Theorem 6 does not establish parameter convergence or attainment of the forgetting threshold, and that the differentiable-route argument does not automatically cover hard top-k boundary crossings. Establishing convergence for the implemented recursive operator remains open.
Dear Reviewer aHfX,
As the discussion period nears its end, we sincerely thank you for your thoughtful review and valuable suggestions. Your feedback motivated us to extend our evaluation with SCRUB, a stronger retain-aware unlearning baseline, and clarify the assumptions and guarantees of our theoretical analysis.
We have incorporated these additions into the revised manuscript. If you have a moment, we would greatly appreciate hearing whether our new results and theoretical clarifications address your concerns.
Regardless, we sincerely appreciate your time and constructive feedback, which have helped improve both the empirical and theoretical presentation of our work.
Best regards,
Official Author Responses to Reviewer aHfX [1/3]
Thank you for your constructive feedback! We agree that stronger retain-aware unlearning baselines and careful qualification of the theory would strengthen the paper. Following your suggestions, we added a SCRUB experiment (Appendix F.4, Table 14) and clarified what Theorem 6 does-and does not-guarantee (Appendix C.8). We address both concerns below.