- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work concentrates on rethinking recent work that enhance trained MoE routers via “hindsight choices” and “search gain”. The authors propose a paired-answer measurement, and a controlled environment that could get both the hindsight value and the best input-only value. They claimed that nearly 70% of the search gain requires the information of the answer, showing why better expert choices based on the answer do not always result in a better router.
Critical strengths
- Overall, the proposed idea is sound, and the authors have conducted sufficient experiments to draw their conclusion.
Critical weaknesses
- This paper focuses on rethinking methods that adopt “hindsight choices” to supervise router training, which is not a mainstream approach in the broader MoE field. Its effectiveness has not been widely acknowledged, and we have not seen industrial-scale models deploying similar strategies. This weakens the contribution of this paper. In addition, it is not entirely clear what insights this study can offer for MoE in pre-training — for instance, regarding dynamic MoE routing strategies in pre-training.
- The 70% figure claimed in the paper is derived solely from limited experimental results on OLMoE, which cannot establish that this ratio still holds in real-world scenarios. Whether the same conclusion applies to other backbones (with different MoE routing strategies) also warrants further experimental verification. Moreover, would the task type, difficulty, and context length of different evaluation benchmarks affect the validity of the experimental conclusions?
Questions for authors
- Is the single-expert swap setting overly restrictive, thereby affecting the conclusions?