- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether dynamic restructuring of routed experts in small (<10B parameter) MoE architectures via pruning, merging, or a composite of these two operations can improve validation loss after a small number of recovery training steps compared to the unadulterated baseline.
Critical strengths
None
Critical weaknesses
- The paper’s core motivation is poorly aligned with the empirical evidence presented. The authors cite works such as DynMoE, EPnG, and LAEP and claim that these works have an “implicit quality premise” of improving loss. However, in each of these three cases, the motivation was largely related to pre-training or PEFT efficiency gains, not necessarily improved validation loss. The efficiency gains of these methods are not considered in this work. Further, many of the cited methods are intended to be applied during training end-to-end training, rather than local interventions coupled with only local recovery. So the claims refuting the value of these adaptive MoE methods fail to consider their intended use-case of supporting pre-training or fine-tuning where the majority of parameters are unfrozen. The authors present expert merging and pruning as MoE adaptation techniques; however, these are first and foremost model compression techniques. Any intervention that removes parameters is likely to increase validation loss, especially with very few optimizer steps and a limited hyperparameter sweep as described here. The claims made here with respect to the utility of MoE restructuring are not well supported by the empirical results.
- The paper’s writing is poor. Results are presented in a verbose manner with excessive enumeration of results without a coherent structure. Unspecific and colloquial language is used in many locations., eg.: “Destroying a layer’s busiest expert…”, “near-dead experts…”, “trigger fires the hardest…”, “Buy control streams, not evaluation data”, “Common random numbers cancel the data draw but not the trajectory.” Introduction of terms such as “Structural Utility” which obfuscate simple concepts rather than elucidating. Terms are swapped in various sections “prune vs. retire experts”.
Questions for authors
None