- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper establishes convergence of population gradient EM for a Gaussian-gated mixture of linear experts under overparameterization, strong cluster separation, and least-squares initialization. Experiments investigate the effects of initialization, the number of experts, and load balancing.
Critical strengths
-
I find the joint-learning result interesting and technically meaningful. The analysis addresses the interactions between the gate and the experts, while also allowing the mixture weights to be learned.
-
It is also appealing that the analysis handles a softmax gate induced by Gaussian clusters, providing a connection to routing mechanisms used in practical MoEs.
Critical weaknesses
-
The role of least-squares initialization could be clarified further. Since Lemma B.1 already provides highly accurate expert coefficients under strong separation, I would appreciate a clearer explanation of the optimization challenges that remain after initialization and how the joint-training analysis relates to clustering followed by per-cluster regression.
-
The finite-sample implications would benefit from further discussion. It would be useful to understand whether the population setting was chosen mainly to limit the scope, or whether a finite-sample extension faces additional technical obstacles, particularly in controlling expert errors along the training trajectory.
Questions for authors
-
Since Lemma B.1 already gives highly accurate expert coefficients after initialization, could you clarify the main optimization challenges that remain under the theorem’s assumptions? In particular, how does the joint-learning guarantee relate to a simple approach that clusters the inputs and then fits each cluster by least squares?
-
Is the restriction to population gradients primarily a scope choice, or does the current proof face a specific obstacle in the finite-sample setting? In particular, do you expect the expert-error control along the trajectory to extend through standard concentration arguments, or would it require additional ideas? This distinction would help assess the scope of the guarantee.
2 comments
Thank you for the response and revisions. The clarification of the joint-training stability guarantee and the added finite-sample analysis address my main questions. I will maintain my weak accept rating.
Response to Reviewer 5Qx4
We thank the reviewer for the positive and focused review, and for finding the joint-learning result interesting and technically meaningful. The two weaknesses match the two questions, so we answer them together.
W1 and Q1. Role of the least-squares initialization, and relation to cluster-then-regress
We reiterate from the general response that experts need only to be polynomially close for adequate initialization. Even with accurate experts, the analysis has to handle the following, among others.
At Δ=8 and 12 the inputs alone form a well-separated Gaussian mixture, the setting of Zhou et al. (2026), so clustering first recovers the gates, as expected, and least squares on the recovered clusters gives accurate experts. At Δ=2 the input clusters overlap, and only joint training and closed-form EM recover them.
W2 and Q2. Finite samples
There was no fundamental hurdle in obtaining a finite-sample extension, and the result was not initially provided because it could not fit in the initial page limit. The expert-error control extends through standard uniform concentration, and only the weight step needed a new device. We now give the finite-sample analysis (Lemma 4.3, Theorem 4.4, Appendix F).
We hope these replies address the reviewer's questions.
Reference. Zhou, Xu, Fazel, Du. Global convergence of gradient EM for over-parameterized Gaussian mixtures. arXiv:2506.06584, 2026.