- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether routing drift diagnoses routing failure in merged MoE language models. The authors cross source and merged router inputs and parameters to identify local sources of route changes, then use fixed-expert output comparisons, token-likelihood analyses, and task-level routing interventions to test whether those changes correspond to recoverable performance loss.
Across DeepSeekMoE, OLMoE, and Qwen3-MoE, most expert reassignments are locally attributed to representation changes, while structural drift poorly predicts gains from source-route restoration. Controlled router corruption produces detectable recoverable loss, whereas the tested natural routing alternatives do not establish reliable task benefits. SRR is evaluated as a case study of source-informed repair, but its local supervision and task evaluation do not establish reliable improvement. The main contribution is the distinction between structural drift and intervention-supported task harm, together with a diagnostic toolkit and supporting empirical evidence.
Critical strengths
-
The paper introduces a useful and operational diagnostic distinction. Rather than treating source/merged routing disagreement as an error label, it defines failure through task loss recoverable by a specified routing intervention with non-routing parameters fixed. Analyses across three architectures show that structural drift metrics poorly predict the tested restoration gains. This can help subsequent work avoid optimizing source agreement without evidence of task utility and is the main contribution supporting acceptance.
-
The negative findings have meaningful controls and statistical support. Matched random-route controls make the mixture-output similarity result more informative, and deliberate router corruption demonstrates that the task interventions can detect substantial recoverable loss. Paired task evaluation, prompt-level resampling that preserves token/layer dependence, and multiplicity correction strengthen the evidence. The authors also clearly distinguish fitting SRR targets from demonstrating actual recovery, giving the negative results independent scientific value.
Critical weaknesses
-
The natural task tests do not tightly bound possible recovery. Some intervals in Table 1 still allow accuracy improvements of several percentage points; for example, OLMoE–Average has an interval of −1.56 to +3.91 pp. The corresponding Qwen3 tests use only 64 ARC items. An interval containing zero establishes a lack of demonstrated improvement, but does not exclude practically useful recovery. The paper acknowledges this limitation, yet should more explicitly state what effect sizes the existing tests rule out so readers can assess the strength of the negative evidence.
-
The evidence is conditional on the reference intervention and fixed candidate construction. Source routes need not be suitable for merged experts, and source or frozen LC schedule replay differs from an online policy recomputing routes on intervened hidden states. SRR results are conditional on a specified construction protocol, update scale, and fixed checkpoints. Five deterministic evaluations do not quantify variation across repair samples or seeds. The experiments therefore support limitations of the tested metrics and candidates; conclusions about other routing policies or source-informed repair should remain appropriately scoped.
Questions for authors
The following clarification requests could materially affect my assessment:
-
What recovery magnitudes do the existing natural-task intervals rule out, and what effect size is considered practically meaningful? Can the 64-item Qwen3 test be supplemented with an existing larger matched evaluation, or explicitly presented as low-resolution evidence?
-
Please summarize which conclusions come from fixed-state router swaps, route-schedule replay, and online/end-to-end execution. To what extent does frozen LC schedule replay test the usefulness of calibrated routing? Are there existing task-level interventions with an online or calibrated policy that help distinguish an unsuitable reference from a lack of diagnostic value in drift metrics?
-
Are SRR results available across different construction samples or repair seeds? If not, please emphasize that the negative result is conditional on the fixed candidates and configurations, rather than interpreting repeated deterministic evaluation as evidence of repair-process stability.
3 comments
W2 & Q2:
Thank you for highlighting the importance of distinguishing different intervention types.
The experiments address complementary questions:
- Fixed-state interventions (Section 3.1): Crossing source/merged router inputs and parameters identifies the local contributions of representation and router-parameter changes to expert reassignment. This establishes local attribution, not task-level recovery.
- Token-local and route-schedule interventions (Sections 3.2 and 4.2): Token-local replacements measure the effects of specified routing changes, with subsequent computation allowed to respond. Task-level source/LC replay evaluates cached routing schedules while keeping non-routing parameters fixed. These experiments assess the specified alternatives, not arbitrary online routing policies.
- End-to-end router execution (Sections 5–6): Table 2 evaluates the SRR checkpoints, alongside HARC, through complete model execution, with routing recomputed on the hidden states produced during inference. These results therefore assess actual checkpoint-level task effects rather than frozen route replay.
Importantly, frozen LC schedule replay is not equivalent to executing the LC-calibrated router online, since the latter would recompute routing decisions on the intervened model's hidden states. Accordingly, the LC results only establish the effects of the specified frozen schedule. Although SRR and HARC provide end-to-end task evaluations, they do not constitute a matched online-versus-frozen comparison of the same LC policy. We therefore agree that the current experiments do not fully disentangle an unsuitable source reference from the potential diagnostic value of drift metrics for other routing-policy families. Our conclusion is specifically that the evaluated structural drift metrics poorly predict the tested source-restoration gains, not that routing drift is uninformative for every possible intervention. We will make these intervention-specific distinctions more explicit in the manuscript.
I will remain my rating.
We sincerely thank you for the thoughtful and constructive review.
We particularly appreciate the recognition of our intervention-based diagnostic framework, controlled experiments, and statistical analyses. We agree that the interpretation of negative results should carefully distinguish the absence of demonstrated recovery from evidence that recovery is impossible. We address the concerns below and will clarify the corresponding statements in the revised manuscript.
The revised manuscript has been submitted.
W1 & Q1:
We agree that confidence intervals containing zero do not establish equivalence to native routing or exclude potentially useful recovery. Our conclusion is therefore limited to the absence of demonstrated improvement under the evaluated routing interventions, rather than the absence of recoverable routing loss.
To clarify the effect sizes supported by the existing evidence, the upper endpoints of the reported 95% confidence intervals for science-domain source-route replay are +1.56/+2.34 percentage points (pp) for DeepSeekMoE and +3.91/+3.52 pp for OLMoE, under Average/Task Arithmetic merging, respectively (Table 1). Thus, the intervals remain compatible with a +1 pp improvement, which we consider a useful illustrative reference for practical relevance rather than a universal threshold across tasks. Some settings remain compatible with improvements of several percentage points. The broader domain results in Appendix A.7 likewise do not support a general claim of negligible recovery.
We also acknowledge the limited statistical resolution of the 64-item Qwen3 source-route replay. We will explicitly characterize these results as low-resolution supporting evidence, rather than use them to claim that practically meaningful recovery has been ruled out. Overall, our findings indicate that the tested natural routing references do not establish reliable task recovery; they do not imply that other routing policies cannot recover task performance.