- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that natural-routing evaluation of MoE unlearning conflates parameter-level removal with routing changes. It proposes a forced-routing probe that reads the forget domain through experts outside the natural top-k. A positive control on the unmodified model supports this. It also proposes LEAK, a recoverability-based expert selection method.
Critical strengths
- The work shows that lower natural-routing accuracy does not imply knowledge removal, demonstrated cleanly by a positive control on the unmodified model.
- Routing affinity and recoverability select almost disjoint experts. This provides a useful perspectivee on expert selection.
Critical weaknesses
-
Limited novelty. Routing shortcuts (SEUF’s “fake unlearning”) and recovery via expert forcing (Zhu et al., 2026) are known, and the forcing operator is adopted directly. The router is frozen for all arms, so “SEUF” here is only its selection rule. The paper also does not relate forced FA to established recovery attacks such as relearning.
-
LEAK is acknowledged as not practical, so its value is evidential. Yet the worst-case margin is 0.0037 (2 of 537 items), the ordering fails under NPO, and the positive control does not reproduce on Qwen1.5-MoE, with only one backbone and one forget domain tested
Questions for authors
-
Router weights are frozen, but expert updates at earlier layers change router inputs downstream. How much does natural top-k selection actually shift after unlearning?
-
Does correlate with the actual effect of unlearning expert j alone?
-
Are the results robust to other forcing strategies, e.g., random non-selected experts or adaptive layer selection?
5 comments
Response to Reviewer 8yeh (2/2): Q1 to Q3
Q1: Router weights are frozen, but expert updates at earlier layers change router inputs downstream. How much does natural top-k selection actually shift after unlearning?
A: A lot on the forget domain, very little on general text. On all 28 depth-cell checkpoints of the new-seed runs (seeds 5 to 9), the Jaccard overlap of each token's top-6 set with the unmodified model's is:
| Layer 6 | Layer 7 | Layers 8–26 | |
|---|---|---|---|
| Ours | 0.44 | 0.33 | 0.35 |
| SEUF | 0.60 | 0.51 | 0.38 |
Layer 5 is unchanged, as its router reads its input before any updated expert. On held-out WikiText the overlap is 0.98–0.99 (Appendix E.8, Table 29). The shift matters. Under our arm and under SEUF, replaying the unmodified model's expert selections recovers 21.4 and 11.2 items (5 of 5 runs each): representation-induced rerouting hides part of what natural accuracy counts as removed, the phenomenon of Section 4.1 arising through training rather than through a mask. We have withdrawn the sentence of Section 5.1 that implied otherwise.
Q2: Does Δ_j correlate with the actual effect of unlearning expert j alone?
A: We have not measured this. It would take one unlearning run per candidate expert, 192 for our three layers. Δ itself is also noisy: its split-half reliability is 0.46, and the selected sets from two halves of the calibration set overlap by 0.25 (Section 4.3, Appendix E.6). Our evidence is at the level of selected sets compared at matched depth, and it is mixed (see W2). We therefore do not claim that Δ_j measures an expert's causal contribution. The paper uses Δ as evidence that recoverability and routing preference pick different experts (Figure 5), not as a recipe.
Q3: Are the results robust to other forcing strategies, e.g., random non-selected experts or adaptive layer selection?
A: We tested the following.
- Random non-selected experts. At layers 5–7 of every depth-cell checkpoint, we forced eight experts drawn at random from outside the natural top-6. Under random non-selected forcing the ordering of Table 4(a) holds in 5 of 5 seed pairs (mean difference 0.099, 53 items; Appendix E.8).
- Pre-unlearning routes (GRIP's Algorithm 2). Under route replay our arm reads 16.4 items higher than SEUF at matched natural accuracy, which works against the ordering of Table 4(a) (Appendix E.8).
- A larger policy family. We searched the backup policy at every budget, single experts, fixed random sets and a greedy search on one half of the questions. On the masked model the selected policy answers 364 of 900 evaluation items correctly, against 370 under the backup policy, inside a margin of 18 items fixed in advance (Appendix E.3). This search was run on the unmodified and masked models, not on the trained arms.
- Other layers. On four other sets of three layers of Moonlight, the routing-only control does not reproduce (Appendix E.7). We did not run an adaptive search over layers; Section 7 lists it as future work.
We hope this answers your questions, and we are happy to discuss further.
Dear Reviewer 8yeh,
Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.
The discussion phase closes on November 18 at 11:59 pm AoE.
Thank you for your attention and contribution.
Best regards,
Area Chair
Response to Reviewer 8yeh (1/2): W1 and W2
Thank you for the clear summary and for questions that pointed to measurements we were missing. We ran them and report the results in a new Appendix E of the revised PDF; changed text in the main body and in Appendices A and D is shown in blue. Each criterion was fixed before the run, and we report the results whether or not they support our claims. Due to the character limit, we split our response into two comments: this one answers W1 and W2, and the next one answers Q1 to Q3.
W1-1: Limited novelty. Routing shortcuts (SEUF's "fake unlearning") and recovery via expert forcing (Zhu et al., 2026) are known, and the forcing operator is adopted directly.
A: We agree that both are known. The paper says so in Section 2 and in its first contribution, and it uses GRIP's operator as is (Section 4.2, after Eq. 5). Our contribution is the evaluation around that operator:
The new route replay adds that, on Moonlight, rerouting caused by training itself hides part of what natural accuracy counts as removed (see Q1).
W1-2: The router is frozen for all arms, so "SEUF" here is only its selection rule.
A: Correct. Every arm trains with one recipe and frozen router weights, so "SEUF" means its selection rule; we do not run its router anchor loss. This keeps the selection rule as the only difference between arms (Section 5.1).
W1-3: Forced FA is not related to established recovery attacks such as relearning.
A: Relearning and fine-tuning attacks are established, and they work through the representations, using data and optimisation (Section 2). Our probe tests a different channel, the router's choice at inference, which needs neither. The paper makes no robustness claim that a relearning attack would test, and Section 7 lists adaptation after unlearning as future work. We did not run relearning attacks.
W2: The worst-case margin is 0.0037, the ordering fails under NPO, and the control does not reproduce on Qwen1.5-MoE, with one backbone and one forget domain.
A: We agree, and the new experiments sharpen both sides.
The revised PDF states the selection result as a case study on Moonlight under RMU and multiple-choice scoring, and the control as a result on Moonlight at the edited layers.