- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies gradient-based machine unlearning for sparse MoE LLMs. It identifies a forget–retain routing mismatch: a small subset of experts is activated disproportionately often by forget data, while generic retain data activates the same experts much less frequently, so under a GradDiff-style objective the retain gradient barely reaches the experts being unlearned. The proposed method TRACE has two stages. It first selects high forget frequency experts by applying an IQR outlier rule to the forget-side activation frequencies. During unlearning it updates only these experts and reweights retain samples by a softmax over their routing distance to the current forget batch, so that retain samples engaging the selected experts in forget-like patterns receive larger weight. Experiments on WMDP and MUSE-BOOKS report higher utility than baselines at comparable forgetting levels.
Critical strengths
- The routing mismatch is established as a mechanism rather than a mere observation, supported both empirically and by a per-expert gradient decomposition. Both components of TRACE follow directly from this analysis, so the paper reads as a coherent mechanism-to-method pipeline instead of a heuristic stack.
- The empirical gains are consistent across three MoE models and two benchmarks, and TRACE nearly matches a baseline using task-curated retain data while relying only on generic retain data. This targets the practically important setting, since avoiding curated retain data is the main operational cost the method claims to remove.
Critical weaknesses
- The baseline tuning appears weak enough to inflate the comparative claims. In Table 2, GA, GradDiff, and NPO collapse to 22.95 to 33.07% MMLU, far below what these methods typically retain on WMDP-style setups, and in Table 1 GA collapses to 0.00 retain KnowMem. Appendix C.1 shows a narrow search grid (learning rate in , ), and the strongest external baseline is ESFT, an expert-selection fine-tuning method repurposed for unlearning rather than a dedicated unlearning method. Since the headline contribution is utility preservation relative to baselines, the margins depend directly on how hard the baselines were tuned, and the current evidence does not establish that they were.
- The ablations study hyperparameters (, , calibration set size) but not the method design itself, so the utility gains cannot be cleanly attributed to the proposed components. Three conditions are missing: reweighting without selection (for example GradDiff over all experts, or ESFT-selected experts, combined with routing-aware reweighting), a control for the reweighting direction (inverted or random weights of the same variance), and alternatives for the distance (cosine, or unrestricted all-expert vectors, given that Figure 2 itself uses cosine similarity for analysis). This is consequential because the paper's central claim is that routing-aware retain reweighting causes the utility improvement; without these controls, the trivial explanation that any non-uniform retain weighting helps remains open.
Questions for authors
- What motivates the IQR outlier rule in Equation 3 for thresholding forget activation frequencies? The paper does not explain why distribution outliers should coincide with the experts worth updating, nor why quartiles are computed over all layers pooled together despite their different base activation rates. How does this compare to simpler rules such as per-layer top-K or a fixed quantile?
- TRACE-Select, the key ablation in Tables 2 and 3 and Figures 3 and 4, is never formally defined beyond table captions. Please state its objective, hyperparameters, and training budget. Note also that Table 3 compares TRACE and TRACE-Select at different steps (200 versus 250 on DeepSeek).
- On MUSE-BOOKS, GA, GradDiff, and NPO use LoRA with (Appendix C.1), but the parameterization of TRACE, SEUF, and ESFT is not stated. Does TRACE update full parameters of the selected experts? If so, the comparison confounds the method with the trainable parameter budget and could explain the baseline collapses in Table 1.
- TRACE freezes the router while SEUF updates the router of its selected expert. Does freezing the router contribute to TRACE's utility advantage or limit its forgetting depth?
- Figure 1, 3, 4 are PNG/JPG images. Can you replace them with SVG?