- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper theoretically analyzes how to approximate a hard-routed Mixture of Experts (MoE) model using a function that changes gradually (in a Lipschitz sense) instead of jumping discontinuously across routing boundaries. It does so with the target function and the router's expert selection choices frozen for each input rather than evolving with training. Under this model, the authors show that making neighboring experts "agree" on decision boundaries is not enough to satisfactorily approximate the hard-routed MoE. Instead, they show that achieving a near-optimal approximation in two or more dimensions forces a specific transition region between experts, which requires either an extra network layer or a growing number of hidden units in a single-layer network. The paper also gives explicit clipping methods to bridge jumps across one-dimensional and multi-region routers under their setup.
Critical strengths
This paper is honestly written. It explicitly states the assumptions and computational model under which its theory is conducted. It also supplements its mathematical proofs with deterministic numerical and symbolic checks.
Critical weaknesses
This paper is limited to a theoretical analysis of hard-routed MoEs via fixed-function approximation. There are no experiments on trained MoE checkpoints or real datasets to demonstrate whether the insights hold in practice. Because of the heavy mathematical notation and strong technical assumptions (such as enforcing a frozen discontinuous routing function), it is unclear how directly these theoretical bounds translate into practical design guidance for real-world MoEs.
Questions for authors
It is difficult for me to give this paper a positive recommendation given the lack of experiments and notational density of the results. However, I am open to considering any perspectives that I missed. In particular,
- What is the practical takeaway for used-in-practice MoE architecture designs? As in, what design change (or re-enforcement of current design beyond just using multi-layers and many parameters) does the theory imply?
- Why would we expect the Lipschitz bounds to retain their insight for real-world MoEs when key assumptions (frozen routing, shallow scalar experts, no nonlinear outputs, etc) no longer hold in practice? Especially when no empirical validation is provided?