- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies the issue of loss spikes in MoE training from scratch. They attribute loss spikes to poor router selections and show that, when encountering a spike, the selected experts exhibit unusually low activation magnitude. To mitigate the spikes during training, they propose rolling-back to a previously saved checkpoint and update only the router parameters within a few iterations (100M tokens). To supervise the router, they densely apply all experts to every token and build a teacher distribution from expert output norms. Using KL they match the router output distribution to that of the teacher. After training on 100M recovery token, the learned router is restored and updated together with the rest of the model parameters. The paper evaluates this method on two MoE configurations: 700M total / 247M active params, and 4B total / 1.18B active params both in a non-granular top-2 out of 8 expert selection setting. The results show improved performance across 8 downstream benchmarks.
Critical strengths
- The paper is easy to follow and Figure 1 & 3 clearly demonstrate the method
- The observation that the selected experts exhibit unusually low activation magnitude during spikes is valuable and informative
- The evaluation while limited shows clear gains compared to the baseline and anticipatory routing technique across all the 8 considered baselines.
- The authors provide the code which helps clarify some of the missing details in the paper
Critical weaknesses
My major concerns are as follows:
-
While at the high level the method seems simple, the authors do not provide many important details. Throughout the paper and in Algorithm 2, they refer to a loss spike detection, but the manuscript does not specify the detection procedure or its hyperparameters. How robust is this selection across models and settings? The authors should explain and quantify how detector settings may affect recovery frequency, training stability, and any other computational overhead.
-
Similarly, there is no information regarding the frequency of saving a new checkpoint. Does the method save a new checkpoint after each recovery phase, and how are repeated spikes before the next checkpoint handled?
-
The authors provide preliminary empirical motivation for norm-based routing, but do not sufficiently establish why expert output norm is a reliable supervision signal for router recovery. The usefulness of an expert depends on the expert output’s direction relative to the loss gradient, as well as its magnitude. E.g. A large output in the wrong direction increases loss; a small correction in the right direction can help. If the intention for this supervision is to simply avoid experts with lower-norm outputs during the recovery phase, a teacher selecting randomly among experts above a norm threshold tests whether just avoiding low-response experts explains the benefit.
-
Despite the small scale of models and experiments, the authors still do not provide seed variance or confidence intervals.
-
There are many obvious ablations missing from the paper such as freezing vs non-freezing of experts during the recovery phase, random teachers, sensitivity of the spike detection approach, etc.
-
There are obvious baselines that are missing from the comparisons: the router z-loss method and other conventional methods used for improving the training stability of regular dense LLMs (your own related work). The authors should demonstrate if their MoE-specific method is superior to these obvious baselines.
-
The paper only evaluates slightly outdated MoE settings (top-2 of 8 experts), while many modern MoEs use extreme sparsity with fine-grained experts, which can render this method impractical: gathering expert weights or dispatching every token to every expert can become prohibitively expensive at those scales.
Questions for authors
-
What are the training details for the recovery phase? Is the same optimizer/ hyper-parameters e.g. learning rate used for the router-only training?
-
How does your method compare to the Anticipatory routing baseline in terms of the number of routing recovery instances. Does early prevention of spikes result in fewer future spike occurrences hence reducing the overhead for recovery or the number of spikes remains steady across the methods?
Please respond to the rebuttal
Dear Reviewers,
The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.
Thank you, Area Chair