acceptodds
Under review as a conference paper at ICLR 2027

Routing Can Hide What Unlearning Did Not Remove: A Forced-Routing Probe for Mixture-of-Experts

Abstract

In Mixture-of-Experts (MoE) language models, unlearning can lower forget-domain accuracy in two ways: by removing knowledge stored in expert parameters, or by changing which experts the router activates. Evaluation under the model's own routing conflates the two, scoring a removed capability and one the router merely hides alike. We introduce a forced-routing evaluation framework that separates them by also reading the forget domain through the experts that natural routing bypasses. With it, we show that natural-routing metrics can substantially overestimate unlearning, as capabilities that appear forgotten can remain recoverable through alternative expert paths. We further propose a recoverability-based expert selection criterion, which chooses experts by how much target capability remains recoverable through them rather than by how often or how strongly the router selects them. The two criteria select largely disjoint experts. Under representation-based unlearning, recoverability-based selection leaves less capability recoverable than SEUF's affinity-based selection at matched forgetting depth, with comparable retain utility. Unlearning in sparse models should therefore be evaluated by what the parameters retain, not only by what the default computation path exposes. Anonymous code is available at https://anonymous.4open.science/r/ForcedRoutingProbe.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, mixture of experts, expert routing, unlearning evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8yehOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that natural-routing evaluation of MoE unlearning conflates parameter-level removal with routing changes. It proposes a forced-routing probe that reads the forget domain through experts outside the natural top-k. A positive control on the unmodified model supports this. It also proposes LEAK, a recoverability-based expert selection method.

Critical strengths
  1. The work shows that lower natural-routing accuracy does not imply knowledge removal, demonstrated cleanly by a positive control on the unmodified model.
  2. Routing affinity and recoverability select almost disjoint experts. This provides a useful perspectivee on expert selection.
Critical weaknesses
  1. Limited novelty. Routing shortcuts (SEUF’s “fake unlearning”) and recovery via expert forcing (Zhu et al., 2026) are known, and the forcing operator is adopted directly. The router is frozen for all arms, so “SEUF” here is only its selection rule. The paper also does not relate forced FA to established recovery attacks such as relearning.

  2. LEAK is acknowledged as not practical, so its value is evidential. Yet the worst-case margin is 0.0037 (2 of 537 items), the ordering fails under NPO, and the positive control does not reproduce on Qwen1.5-MoE, with only one backbone and one forget domain tested

Questions for authors
  1. Router weights are frozen, but expert updates at earlier layers change router inputs downstream. How much does natural top-k selection actually shift after unlearning?

  2. Does Δj\Delta_j correlate with the actual effect of unlearning expert j alone?

  3. Are the results robust to other forcing strategies, e.g., random non-selected experts or adaptive layer selection?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8yeh (1/2): W1 and W2

Thank you for the clear summary and for questions that pointed to measurements we were missing. We ran them and report the results in a new Appendix E of the revised PDF; changed text in the main body and in Appendices A and D is shown in blue. Each criterion was fixed before the run, and we report the results whether or not they support our claims. Due to the character limit, we split our response into two comments: this one answers W1 and W2, and the next one answers Q1 to Q3.

W1-1: Limited novelty. Routing shortcuts (SEUF's "fake unlearning") and recovery via expert forcing (Zhu et al., 2026) are known, and the forcing operator is adopted directly.

A: We agree that both are known. The paper says so in Section 2 and in its first contribution, and it uses GRIP's operator as is (Section 4.2, after Eq. 5). Our contribution is the evaluation around that operator:

  1. a routing-only control on the unmodified model, which lowers natural accuracy with no parameter changed (Section 4.1);
  2. forcing applied to every item rather than only to rerouted ones, compared with the unmodified model under the same forcing, with the budget swept and its worst case reported (Section 4.2);
  3. comparisons only at matched depth.

The new route replay adds that, on Moonlight, rerouting caused by training itself hides part of what natural accuracy counts as removed (see Q1).

W1-2: The router is frozen for all arms, so "SEUF" here is only its selection rule.

A: Correct. Every arm trains with one recipe and frozen router weights, so "SEUF" means its selection rule; we do not run its router anchor loss. This keeps the selection rule as the only difference between arms (Section 5.1).

W1-3: Forced FA is not related to established recovery attacks such as relearning.

A: Relearning and fine-tuning attacks are established, and they work through the representations, using data and optimisation (Section 2). Our probe tests a different channel, the router's choice at inference, which needs neither. The paper makes no robustness claim that a relearning attack would test, and Section 7 lists adaptation after unlearning as future work. We did not run relearning attacks.

W2: The worst-case margin is 0.0037, the ordering fails under NPO, and the control does not reproduce on Qwen1.5-MoE, with one backbone and one forget domain.

A: We agree, and the new experiments sharpen both sides.

  • Worst-case margin. The 0.0037 is read at the step-449 endpoints, where depths are not matched. Against SEUF as well, reading every depth-cell checkpoint under all nine probe budgets and taking each checkpoint's worst case, our arm is lower in all 5 seed pairs (mean margin 0.020; p = 0.031, the smallest value 5 pairs allow; item-bootstrap 95% interval −0.008 to 0.047). That interval includes zero (Appendix E.4).
  • Second backbone. We trained both arms on Ling-mini-2.0, whose router is in the same family as Moonlight's. On Ling-mini-2.0 the two selections do not separate at matched depth (mean margin −0.010); the ordering of Table 4(a) is specific to our backbone (Appendix E.9). As a description only, the gap between the two readings does appear there: at the final step, natural accuracy is near chance under both arms (0.264, 0.289), while forced accuracy stays at 0.411 and 0.410, against 0.425 for the unmodified model.
  • Other readings. The ordering does not carry to NPO (Section 5.3), reverses under a free-form scorer (Appendix E.5), and works against us under route replay (Q3).
  • The control. Besides Qwen1.5-MoE, the routing-only control does not reproduce on DeepSeek-V2-Lite, OLMoE-1B-7B or Qwen3-30B-A3B, so it appears on one of the five backbones we have measured. On WMDP-Bio it is exploratory (Appendices E.1, E.2).

The revised PDF states the selection result as a case study on Moonlight under RMU and multiple-choice scoring, and the control as a result on Moonlight at the edited layers.

AuthorsOpenReview ↗

Response to Reviewer 8yeh (2/2): Q1 to Q3

Q1: Router weights are frozen, but expert updates at earlier layers change router inputs downstream. How much does natural top-k selection actually shift after unlearning?

A: A lot on the forget domain, very little on general text. On all 28 depth-cell checkpoints of the new-seed runs (seeds 5 to 9), the Jaccard overlap of each token's top-6 set with the unmodified model's is:

Layer 6Layer 7Layers 8–26
Ours0.440.330.35
SEUF0.600.510.38

Layer 5 is unchanged, as its router reads its input before any updated expert. On held-out WikiText the overlap is 0.98–0.99 (Appendix E.8, Table 29). The shift matters. Under our arm and under SEUF, replaying the unmodified model's expert selections recovers 21.4 and 11.2 items (5 of 5 runs each): representation-induced rerouting hides part of what natural accuracy counts as removed, the phenomenon of Section 4.1 arising through training rather than through a mask. We have withdrawn the sentence of Section 5.1 that implied otherwise.

Q2: Does Δ_j correlate with the actual effect of unlearning expert j alone?

A: We have not measured this. It would take one unlearning run per candidate expert, 192 for our three layers. Δ itself is also noisy: its split-half reliability is 0.46, and the selected sets from two halves of the calibration set overlap by 0.25 (Section 4.3, Appendix E.6). Our evidence is at the level of selected sets compared at matched depth, and it is mixed (see W2). We therefore do not claim that Δ_j measures an expert's causal contribution. The paper uses Δ as evidence that recoverability and routing preference pick different experts (Figure 5), not as a recipe.

Q3: Are the results robust to other forcing strategies, e.g., random non-selected experts or adaptive layer selection?

A: We tested the following.

  • Random non-selected experts. At layers 5–7 of every depth-cell checkpoint, we forced eight experts drawn at random from outside the natural top-6. Under random non-selected forcing the ordering of Table 4(a) holds in 5 of 5 seed pairs (mean difference 0.099, 53 items; Appendix E.8).
  • Pre-unlearning routes (GRIP's Algorithm 2). Under route replay our arm reads 16.4 items higher than SEUF at matched natural accuracy, which works against the ordering of Table 4(a) (Appendix E.8).
  • A larger policy family. We searched the backup policy at every budget, single experts, fixed random sets and a greedy search on one half of the questions. On the masked model the selected policy answers 364 of 900 evaluation items correctly, against 370 under the backup policy, inside a margin of 18 items fixed in advance (Appendix E.3). This search was run on the unmodified and masked models, not on the trained arms.
  • Other layers. On four other sets of three layers of Moonlight, the routing-only control does not reproduce (Appendix E.7). We did not run an adaptive search over layers; Section 7 lists it as future work.

We hope this answers your questions, and we are happy to discuss further.

Dear Reviewer 8yeh,

Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.

The discussion phase closes on November 18 at 11:59 pm AoE.

Thank you for your attention and contribution.

Best regards,

Area Chair

Reviewer_YEEYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies machine unlearning for MoE-based LLMs by introducing: (1) a forced-routing evaluation framework that separates them by also reading the forget domain through the experts that natural routing bypasses; (2) a recoverability-based expert selection criterion, which chooses experts by how much target capability remains recoverable through them rather than by how often or how strongly the router selects them.

Critical strengths

The paper writing is very poor with its introduction section, proposed methods, and experiments presented unclearly, making it difficult for the reviewer to identify its strengths.

Critical weaknesses
  1. The paper mentions the key contributions are “forced-routing evaluation framework” and “recoverability-based expert selection criterion” but never explain them in the main paper.
  2. The paper uses terms “reading” and “forced reading” a lot but never defines them clearly.
  3. Section 1 - Introduction: paper mentions “building on the expert forcing of Zhu et al. (2026)” but never explains “expert forcing” before. The baseline also skips GRIP (Zhu et al. (2026)). Figure 1 without legends, settings or explanations is very hard to understand.
  4. Section 2 - Related Work: “SEUF (Zhuang et al., 2025) scores an expert by its mean gate score averaged over all tokens” is not correct; ESFT (Wang et al., 2024) is not the original for Unlearning; RACE (Xie et al., 2026) is a withdraw Arxiv paper.
  5. Section 3 - Preliminaries: Since that knowledge is in previous research, there is no need to spend a whole section repeating it.
  6. Section 4 : it is unclear if the section is the method section or analysis section. Three insights are also difficult to understand or have evidence to prove for those insights.
  7. Section 5 - Experiments: it is very limited with one model (Moonlight-16B-A3B-Instruct) and one benchmark (WMDP). Since the settings and metrics are different compared to the baselines, it is really hard to judge the result.
  8. The befitnit of the proposed method is unclear: Table 3 (main results), the method does not beat the baselines on three metrics.
Questions for authors

Please refer to the above weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YEEY (1/2): points 1 to 4

Thank you for the detailed reading. Several of your points show where the paper was unclear, and the point on TRACE found an error in our references. We have uploaded a revised PDF; changed text in the main body and in Appendices A and D is shown in blue, and a new Appendix E reports the experiments we ran after submission. We answer your eight points in order; due to the character limit, points 1 to 4 are in this comment and points 5 to 8 in the next one.

1. The two key contributions are not explained in the main paper.

A: Both are defined in Section 4. The forced-routing evaluation is Section 4.2: the threat model, the three routing policies (Eq. 4), the two readings (Eq. 5) and the rules for comparing them. The recoverability-based criterion is Section 4.3, Eq. 6: Δ_j is the forget accuracy on a calibration split when the model is forced through expert j alone, and we select the B experts with the largest Δ_j in each layer.

2. "Reading" and "forced reading" are never clearly defined.

A: Thanks for pointing this out. The revised introduction defines both at first use (Lines 70–73). The natural reading (natFA) is forget-domain accuracy under the model's own routing. The forced reading (forced FA) is the accuracy when, at the edited layers, the experts the router selected are bypassed and the next-highest-scoring experts serve the token instead.

3. Introduction: "expert forcing" is not explained, the baselines skip GRIP, and Figure 1 is hard to understand.

A: (a) Right after the citation of Zhu et al. (2026), the introduction describes the forcing operation we take from them (Lines 70–73), and Section 2 describes their forcing variants. (b) GRIP is not an expert-selection method: it constrains how the router's parameters may change during or after unlearning. In our comparison only the selection rule varies and the router is frozen in every arm, so GRIP does not fit, and we did not run it. (c) The caption of Figure 1 now explains its three panels and points to Section 4.1 and Table 3.

4. Related work: the SEUF description is not correct; ESFT is not originally for unlearning; TRACE is a withdrawn arXiv paper.

A: SEUF. We believe this sentence is correct. SEUF's affinity score (Zhuang et al., 2025, Eq. 2) averages the router score g over the tokens of forget-set sequences, and their Section 3 sets g to zero for experts outside the top-K. ESFT. You are right; the revised text states that it was proposed for expert-specialized fine-tuning (Line 124). TRACE. You are right, and thank you. arXiv:2606.10338 was withdrawn by its authors on 23 September 2026, with a note reporting an error in its per-expert gradient decomposition; we had missed this. The revised PDF marks the reference as withdrawn (Lines 125–127), removes it from the introduction, and drops the sentence of Section 5.3 that cited that decomposition; the argument there now rests on our Section 3.2. We keep the TRACE arm because its signal does not depend on that derivation: it is the frequency with which the router selects an expert on forget data. Its numbers do not change.

AuthorsOpenReview ↗

Response to Reviewer YEEY (2/2): points 5 to 8

5. Section 3 (Preliminaries) repeats prior work and does not need a whole section.

A: We agree that the MoE layer and the two unlearning objectives come from prior work, and Section 3 cites them as such. We keep the section, at about one page, because three later steps rest on it:

  1. Eq. 1 separates a gated routed term from an ungated shared term. The gated term is why changing the routing alone can move forget accuracy (Section 4.1). The ungated term is why no routing policy reaches the shared experts, which gives one of the two reference lines we compare against (Sections 4.2 and 5.1).
  2. It defines an arm, a training setting in which only the selection rule changes; every comparison in Section 5 is between arms.
  3. Its last paragraph shows that the gradient reaching a routed expert is scaled by its gate weight, so under a likelihood-based objective such as NPO one learning rate is not the same intervention for different arms. This is why we match forgetting depth explicitly, and it is the mechanism Section 5.3 uses for the NPO result (see point 4).

6. Section 4: is it a method or an analysis section? The insights are hard to follow and lack evidence.

A: Section 4 is the method: 4.1 establishes the phenomenon with a control, 4.2 defines the evaluation, and 4.3 the selection signal. Each insight points to its evidence:

  • Insight 1: Figure 4 and Table 8. With no parameter changed, masking lowers natural accuracy at every non-zero k, while forced accuracy rises and retain accuracy does not move.
  • Insight 2 follows from the definitions, since the probe searches a subset of the policies the threat model allows; Appendix E.3 checks it against a larger family.
  • Insight 3: Figure 5. The three affinity rules share 17–19 of their 24 experts with one another, and 1–2 with ours.
  • Insight 4: Tables 3 and 4.

7. Section 5: only one model and one benchmark; the settings and metrics differ from the baselines.

A: The routing-only control is now measured on five backbones and appears on one (Appendices B.2, E.1). On a second forget domain it is exploratory (Appendix E.2). The trained comparison is repeated on a second backbone, where the two selections do not separate (Appendix E.9). The revised paper states each result within this scope. On comparability: natural accuracy on WMDP is the metric SEUF reports; we read the forced accuracy next to it, and we retrain all baselines with one recipe in which only the selection rule varies.

8. The benefit of the method is unclear: in Table 3 it does not beat the baselines on all three metrics.

A: Table 3 compares the arms at an equal training budget, not at equal depth. Its point is that the two readings rank the same models in opposite directions (Insight 4):

Natural FAForced FARetain acc.
Ours0.3400.2850.616
Three affinity arms0.255–0.2590.342–0.3560.610–0.617

Under natural routing our arm forgets least; under forcing it leaks least; retain accuracy is in the same range. The comparison at matched depth is Table 4(a): on Moonlight our arm leaves less recoverable, at a cost of 0.006 in retain accuracy over ten seed pairs. Under a free-form scorer and on a second backbone this ordering does not hold, and the revised paper says so.

We hope these answers address your concerns, and we are happy to discuss further.

Thank you for the response. Given the limited novelty and unclear practical benefits, I believe the current evidence remains insufficient to justify acceptance at ICLR. Furthermore, I think ICLR papers should be human-readable and allow a person to get the gist in short order. Therefore, I maintain my recommendation to clearly reject the paper.

AuthorsOpenReview ↗

Response to Reviewer YEEY

Dear Reviewer YEEY,

Thank you for reading our response and for letting us know where you stand. We take your points on novelty, practical benefit and readability seriously, and we will use them to make the main point of the paper clear from the first page.

Best regards,

The authors

Reviewer_wrNkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates MoE unlearning under both natural routing and a forced policy that activates highly scored experts outside the natural top-k set. It also proposes LEAK, which selects trainable experts by single-expert forced accuracy rather than routing affinity. The main study updates 24 experts across three layers of Moonlight-16B-A3B-Instruct using RMU, with 307,200 forget-corpus tokens and 450 training steps. A 900-step continuation and an NPO comparison provide additional checks.

The routing-only control shows that forget accuracy can change without expert-weight updates. Under RMU, LEAK reduces forced accuracy relative to affinity selection at matched natural accuracy. However, the advantage weakens under stricter coverage and probe-budget comparisons, does not reproduce under NPO, and is not validated by trained-model experiments on a second backbone.

Critical strengths
  1. Evaluating beyond the deployed routing policy addresses an important weakness of natural-routing forget accuracy. The no-expert-update control directly demonstrates policy dependence, and the paper correctly treats each forced reading as a lower bound on recoverable capability rather than a certificate of removal.

  2. The paper reports substantial unfavorable evidence: the second-backbone control failure, NPO's negative result, the unreliable and expensive selector, incomplete depth coverage, and the small worst-case separation. Calibration/evaluation separation, random-expert controls, and untargeted-domain checks make the study more informative than a single favorable endpoint comparison.

Critical weaknesses
  1. Freezing router weights does not establish the claimed separation from rerouting. Section 5.1 infers that no arm can reduce natural accuracy by changing routing decisions because the router is frozen. Updating experts at layer 5 can nevertheless change the inputs to the frozen routers at layers 6 and 7, altering expert selection and weights. The routing-only control does not exclude this mechanism during training or localize recovered capability to untouched experts. Token-level route comparisons and replayed fixed routes are needed. This is especially consequential because GRIP already identifies expert-forcing recovery and explicitly discusses representation-induced routing drift; the present contribution must therefore rest on a sound additional diagnostic, not a newly identified failure mode. See GRIP, Sections 5.3–5.4.

  2. The selection advantage is not established under a jointly matched and sufficiently broad evaluation. The trained-model evidence covers one backbone, one forget domain, and a 307,200-token forget corpus. At matched RMU depth, the reported 0.303 versus 0.353 uses m = 8; restricting to fully covered depths loses agreement across seeds. The worst-budget sweep is instead available at unmatched-depth endpoints, where the closest cross-arm run separation is only 0.0037, roughly two questions. Thus depth matching and worst-budget robustness are not demonstrated together. NPO does not reproduce the advantage and costs retain utility; the second backbone has no trained-model comparison. These limitations leave a useful case study, but insufficient evidence for a dependable alternative expert-selection principle.

Questions for authors
  1. Do selected experts and gate weights remain stable after unlearning at every intervened layer? Please compare natural routing with replayed pre-unlearning routes to separate representation-induced rerouting from changes under a fixed computation path.

  2. Does LEAK outperform affinity selection when all seeds share the same forgetting-depth range and the maximum over probe budgets is measured at those same checkpoints? Please include item-level uncertainty and a trained-model replication on the second backbone or another forget domain.

5 comments
AuthorsOpenReview ↗

Response to Reviewer wrNk (1/2): W1 and Q1

Thank you for the detailed review and for the concrete experiments you suggested. We ran them and report the results in a new Appendix E of the revised PDF; changed text in the main body and in Appendices A and D is shown in blue. Each criterion was fixed before the run, and we report the results whether or not they support our claims. Due to the character limit, we split our response into two comments: this one answers W1 and Q1, and the next one answers W2 and Q2.

W1 / Q1: Freezing the router weights does not rule out rerouting. Do the selected experts and gate weights stay stable, and how does natural routing compare with replayed pre-unlearning routes?

A: We agree. Updating experts at layer 5 changes the inputs to the routers at layer 6 and later, so routing can change even though the router weights are frozen. We have withdrawn the sentence in Section 5.1 that said otherwise.

Following your suggestion, we measured routing on all 28 depth-cell checkpoints of the new-seed runs (13 of our arm, 15 of SEUF), on the 537 items of Table 4(a) (Appendix E.8, Table 29).

Stability. Layer 5 does not change, because its router reads its input before any updated expert. Jaccard overlap of each token's top-6 set with the unmodified model's:

Layer 6Layer 7Layers 8–26
Ours0.440.330.35
SEUF0.600.510.38
Held-out WikiText (both)0.990.990.98

The gate weights also move: the total-variation distance at layers 6 and 7 is 0.50 and 0.65 under our arm, and 0.31 and 0.54 under SEUF. So routing changes substantially on the forget domain and hardly on general text.

Replay. Under our arm and under SEUF, replaying the unmodified model's expert selections recovers 21.4 and 11.2 items (5 of 5 runs each): representation-induced rerouting hides part of what natural accuracy counts as removed, the phenomenon of Section 4.1 arising through training rather than through a mask.

Items recovered by replay (of 537)OursSEUF
Expert sets at every MoE layer, weights from the current gate (GRIP, Algorithm 2)21.4 (95% CI 10.7–32.0)11.2 (3.5–19.0)
Expert sets at layers 5–7 only17.15.1
Sets and weights together (fixed computation path)21.525.8

The last row is measured against the natural reading of its own pass. For SEUF, 11.2 is only 0.2 items above the threshold of 11 items we fixed before the run.

Two limits. Replay needs the unmodified model's routes, which the adversary of Section 4.2 does not have, so it is a diagnostic for an auditor. And like every forced reading, a replayed reading is a lower bound, not the amount removed.

W1 (last part): GRIP already identifies expert-forcing recovery and representation-induced routing drift.

A: We agree, and the revision no longer presents either as a new failure mode (GRIP, Sections 5.3–5.4). What we add is the diagnostic around the forcing operator:

  1. a routing-only control on the unmodified model, which lowers natural accuracy with no parameter changed, so nothing can have been removed (Section 4.1);
  2. forcing applied to every item and compared with the unmodified model under the same forcing, with the budget swept and its worst case reported (Section 4.2);
  3. comparisons only at matched depth (Corollary 2);
  4. now, on trained models, a measurement of how much rerouting hides under each selection rule (above).

In our runs every one of the 537 items has a rerouted token at layers 6–26 on every checkpoint, so GRIP's restriction to queries whose routing shifted would keep all items.

AuthorsOpenReview ↗

Response to Reviewer wrNk (2/2): W2 and Q2

W2 / Q2: Depth matching and worst-budget robustness are not shown together; please add item-level uncertainty and a trained-model replication on the second backbone or another forget domain.

A: Thank you for spelling out the comparison. We ran it as you describe: all 15 depth-cell checkpoints of SEUF's five new runs, read under all nine probe budgets, with the cells, the pairing and the criterion fixed before the run (Appendix E.4, Table 24). Depth matching and the worst probe budget hold together against all three affinity baselines (App. E.4). Against SEUF as well, reading every depth-cell checkpoint under all nine probe budgets and taking each checkpoint's worst case, our arm is lower in all 5 seed pairs (mean margin 0.020; p = 0.031, the smallest value 5 pairs allow; item-bootstrap 95% interval −0.008 to 0.047). At the one depth cell all ten runs reach (natural accuracy 0.325), our arm's worst case is lower in 5 of 5 seed pairs (mean difference 0.026).

Item-level uncertainty. The direction agrees across seeds, but it is not resolved at the item level. Every interval includes zero:

Worst-case margin (positive = ours lower)Item-bootstrap 95% interval
vs SEUF, all depth cells−0.008 to 0.047
vs SEUF, 0.325 cell−0.0004 to 0.059
vs retrained ESFT−0.006 to 0.044
vs retrained TRACE−0.003 to 0.054

The margin against SEUF comes mostly from seeds 5 to 7; seeds 8 and 9 are 5.3 and 2.5 items apart. This is the third comparison of this kind, and the three p values are uncorrected. At the shared cells, retain accuracy is 0.009 lower under our arm.

Second backbone. We trained both arms on Ling-mini-2.0, whose router is in the same family as Moonlight's (sigmoid scores with a per-expert correction bias, renormalised and scaled). We used the recipe of Section 5.1, five seeds, the same eighth of the routed experts per layer, and a depth grid derived by a rule fixed before training (Appendix E.9). On Ling-mini-2.0 the two selections do not separate at matched depth (mean margin −0.010); the ordering of Table 4(a) is specific to our backbone. One of the five seed pairs is positive (p = 0.97, one-sided). As a description only: at the final checkpoint, natural accuracy there is near chance (0.264 ours, 0.289 SEUF; chance is 0.25), while forced accuracy at m = 8 stays at 0.411 and 0.410, against 0.425 for the unmodified model. We did not train on a second forget domain or vary the forget corpus. On WMDP-Bio the routing-only control is exploratory (Appendix E.2), and on three further backbones it does not reproduce (Appendix E.1).

Results that go against the ordering. Under route replay our arm reads 16.4 items higher than SEUF at matched natural accuracy, which works against the ordering of Table 4(a) (item-bootstrap 95% interval for SEUF minus ours, −30.0 to −3.1 items; Appendix E.8). Under a free-form likelihood scorer the ordering reverses (Appendix E.5). Under NPO it does not hold, and our arm loses more retain accuracy (Section 5.3).

W2 (conclusion): a useful case study, but not a dependable expert-selection principle.

A: We agree. The revised paper states the selection result as a case study on Moonlight, under RMU and multiple-choice scoring, and rests its main contribution on the evaluation: the routing-only control and the forced-routing probe. The replay results above now support that part on trained models as well.

We hope this answers your questions, and we are happy to discuss any of them further.

Dear Reviewer wrNk,

Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.

The discussion phase closes on November 18 at 11:59 pm AoE.

Thank you for your attention and contribution.

Best regards,

Area Chair

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and new experiments

Thank you all for the careful reviews. We have uploaded a revised PDF; the main text still ends on page 10. Changed text in the main body and in Appendices A and D is shown in blue, and Appendix E is new. We reply to each reviewer separately below.

Main changes

  1. Scope of the claims. The abstract, the introduction, Sections 4.1, 4.3 and 5.2, the conclusion and the limitations now state each result where it was measured: the routing-only control on Moonlight, at the three layers the RMU configuration edits; the selection comparison on Moonlight, under representation-based unlearning (RMU) and multiple-choice scoring. "Comparable retain utility" is replaced by the measured difference (0.006).
  2. Section 5.1. We withdrew the sentence that inferred, from the frozen router, that no arm can lower natural accuracy by changing routing decisions. It is replaced by the routing change we measured after training and its effect under route replay (Appendix E.8).
  3. Section 5.2 adds the replication of Table 4(a) on five new seeds, the worst probe budget read at matched depth against all three affinity baselines, the trained comparison on a second backbone (where the two selections do not separate), and the route-replay result, which goes against the ordering of Table 4(a). It states that the ordering is read under the backup policy and holds only under multiple-choice scoring.
  4. Presentation and references. The introduction defines the natural and forced readings at first use, and the caption of Figure 1 explains its three panels. Section 2 states that ESFT was proposed for fine-tuning rather than for unlearning, and that the TRACE preprint has been withdrawn by its authors; the sentence of Section 5.3 that cited its gradient decomposition is removed, as is its citation in the introduction. Appendix A.1 states the weights of the backup policy as implemented, and Appendices D.1 and D.4 point to the new results.
  5. Limitations moved from the appendix into Section 7 of the main text.

New experiments (Appendix E)

All are on Moonlight-16B-A3B-Instruct, our backbone, except where another backbone is named. Each criterion was fixed before the run, and we report every result, including those that go against our claims.

  • [wrNk, 8yeh] Token-level routing comparison and route replay on all 28 depth-cell checkpoints (E.8, Table 29).
  • [wrNk] The worst probe budget at depth-matched checkpoints against SEUF (Table 24) and against the retrained ESFT and TRACE (Table 22), with item-level uncertainty (E.4).
  • [wrNk, 8yeh] The trained comparison on a second backbone, Ling-mini-2.0 (E.9, Table 30).
  • [8yeh] Random non-selected forcing (E.8), a larger family of routing policies (E.3, Table 18), and the control at other layers (E.7, Table 28).
  • [8yeh, YEEY] The routing-only control on DeepSeek-V2-Lite, OLMoE-1B-7B and Qwen3-30B-A3B (E.1, Tables 11–16) and on a second forget domain, WMDP-Bio (E.2, Table 17).
  • The replication of Table 4(a) on five new seeds (E.4, Table 19), free-form scoring (E.5, Table 27), and the stability of the selection (E.6).
  • Further controls fixed before the runs (E.4, Tables 20, 21, 23, 25 and 26).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.