acceptodds
Under review as a conference paper at ICLR 2027

Expert-Guided Router Recovery for Stable Mixture-of-Experts Pre-training

11 views

Abstract

Mixture-of-experts (MoE) language models expand model capacity while limiting per-token computation through sparse expert activation. However, loss spikes during from-scratch MoE pre-training remain a critical yet underexplored failure mode. To address this, we propose Expert-Guided Router Recovery (EGRR), a pre-training recovery method that bridges the gap between router decision-making and expert execution by enabling the router to assign tokens to experts according to capability information derived from the experts themselves. EGRR is based on the insight that around loss spikes, router-selected experts exhibit unusually low activation magnitudes, suggesting limited capability for processing the current representations. In EGRR, once a loss spike is detected, EGRR rolls training back to the latest stable checkpoint and densely activates all experts to guide router tuning with expert-derived capability information, while all non-router parameters are frozen. After 100M tokens recovery period, dense expert activation is removed and standard MoE pre-training resumes. EGRR therefore introduces additional computation only during recovery, with no additional inference cost. We evaluate EGRR in a 100B tokens from-scratch pre-training run of a 700M parameters MoE language model, demonstrating improved training stability and downstream performance over standard MoE training. The code is available at https://github.com/saz97/Expert-Guided-Router-Recovery.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Training Stability, Large Language Models, Pre-training

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Y95eOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies the issue of loss spikes in MoE training from scratch. They attribute loss spikes to poor router selections and show that, when encountering a spike, the selected experts exhibit unusually low activation magnitude. To mitigate the spikes during training, they propose rolling-back to a previously saved checkpoint and update only the router parameters within a few iterations (100M tokens). To supervise the router, they densely apply all experts to every token and build a teacher distribution from expert output norms. Using KL they match the router output distribution to that of the teacher. After training on 100M recovery token, the learned router is restored and updated together with the rest of the model parameters. The paper evaluates this method on two MoE configurations: 700M total / 247M active params, and 4B total / 1.18B active params both in a non-granular top-2 out of 8 expert selection setting. The results show improved performance across 8 downstream benchmarks.

Critical strengths
  • The paper is easy to follow and Figure 1 & 3 clearly demonstrate the method
  • The observation that the selected experts exhibit unusually low activation magnitude during spikes is valuable and informative
  • The evaluation while limited shows clear gains compared to the baseline and anticipatory routing technique across all the 8 considered baselines.
  • The authors provide the code which helps clarify some of the missing details in the paper
Critical weaknesses

My major concerns are as follows:

  1. While at the high level the method seems simple, the authors do not provide many important details. Throughout the paper and in Algorithm 2, they refer to a loss spike detection, but the manuscript does not specify the detection procedure or its hyperparameters. How robust is this selection across models and settings? The authors should explain and quantify how detector settings may affect recovery frequency, training stability, and any other computational overhead.

  2. Similarly, there is no information regarding the frequency of saving a new checkpoint. Does the method save a new checkpoint after each recovery phase, and how are repeated spikes before the next checkpoint handled?

  3. The authors provide preliminary empirical motivation for norm-based routing, but do not sufficiently establish why expert output norm is a reliable supervision signal for router recovery. The usefulness of an expert depends on the expert output’s direction relative to the loss gradient, as well as its magnitude. E.g. A large output in the wrong direction increases loss; a small correction in the right direction can help. If the intention for this supervision is to simply avoid experts with lower-norm outputs during the recovery phase, a teacher selecting randomly among experts above a norm threshold tests whether just avoiding low-response experts explains the benefit.

  4. Despite the small scale of models and experiments, the authors still do not provide seed variance or confidence intervals.

  5. There are many obvious ablations missing from the paper such as freezing vs non-freezing of experts during the recovery phase, random teachers, sensitivity of the spike detection approach, etc.

  6. There are obvious baselines that are missing from the comparisons: the router z-loss method and other conventional methods used for improving the training stability of regular dense LLMs (your own related work). The authors should demonstrate if their MoE-specific method is superior to these obvious baselines.

  7. The paper only evaluates slightly outdated MoE settings (top-2 of 8 experts), while many modern MoEs use extreme sparsity with fine-grained experts, which can render this method impractical: gathering expert weights or dispatching every token to every expert can become prohibitively expensive at those scales.

Questions for authors
  • What are the training details for the recovery phase? Is the same optimizer/ hyper-parameters e.g. learning rate used for the router-only training?

  • How does your method compare to the Anticipatory routing baseline in terms of the number of routing recovery instances. Does early prevention of spikes result in fewer future spike occurrences hence reducing the overhead for recovery or the number of spikes remains steady across the methods?

Reviewer_NgGeOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I notice that the abstract of this submission provides a GitHub link under name saz97, an account with other public repositories and social connections (e.g., followers and stars), raising potential anonymity concerns. The repository also shows an update dated September 28 2026, after the September 25 AoE submission deadline. These observations are based on information publicly visible on the repository and its associated GitHub profile. I did not established the authors' identities.

I believe these observations raise potential policy concerns and recommend rejection.

Critical strengths

See summary.

Critical weaknesses

See summary.

Questions for authors

See summary.

Reviewer_9LfPOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces loss spikes during MoE training as a common problem, and proposes a method for more stable training by introducing a "recovery procedure" such that whenever loss spikes occur in training they rollback to an earlier checkpoint, freeze backbone LLM weights and only train router weights using their recovery procedure, and then resume normal training after 100M tokens. Their recovery procedure introduces a new loss based on doing dense routing (all routers engaged) during recovery period.

Overall the idea is innovative, has good motivation, and shows interesting results.

Critical strengths

The paper presents a clearly motivated reason for focusing on loss-spikes, and an easy-to-follow examination of router behavior under different expert-selection mechanisms guided by loss spikes & activations magnitude to motivate their main EGRR algorithm.

The paper's experimental methodology and ablations are well founded.

Critical weaknesses

It remains to be seen how well this mechanism scales to larger MoE models.

Questions for authors

n/a

Comments on the paper

1 comment
Area_Chair_GA7YOpenReview ↗

Please respond to the rebuttal

Dear Reviewers,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.