acceptodds
Under review as a conference paper at ICLR 2027

Provably Learning Mixtures of Linear Experts

Abstract

Mixture-of-Experts (MoE) models are widely deployed in modern machine learning. However, few convergence guarantees exist for gradient-based training of MoEs: the objective is nonconvex, and even simple mixtures trap first-order methods in spurious local optima. We analyze a canonical gated mixture featuring linear experts and a gate induced by Gaussian clusters in the input space, and give, to our knowledge, the first global convergence guarantee for jointly training the gate and the experts of such a model by gradient EM on its likelihood. Given a logarithmic overparameterization relative to the true number of experts , gradient EM converges from randomly initialized gates and least-squares-initialized experts to any prescribed accuracy in polynomially many iterations, under a separation condition on the gate centers. Our analysis establishes a gradient-dominance inequality for the coupled dynamics by decoupling the loss into per-cluster subproblems. Empirical results validate the predicted phase transition and suggest that the least-squares warm start required by our analysis is not needed in practice: randomly initialized experts perform nearly as well. We also find that leaving the mixing weights unconstrained concentrates routing onto the true number of experts, whereas load-balancing penalties spread it, so that the fitted model can be pruned to a fraction of its width without loss of accuracy.

Primary area:
optimization
Keywords:
Mixture of Experts, optimization, convergence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5Qx4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper establishes convergence of population gradient EM for a Gaussian-gated mixture of linear experts under overparameterization, strong cluster separation, and least-squares initialization. Experiments investigate the effects of initialization, the number of experts, and load balancing.

Critical strengths
  1. I find the joint-learning result interesting and technically meaningful. The analysis addresses the interactions between the gate and the experts, while also allowing the mixture weights to be learned.

  2. It is also appealing that the analysis handles a softmax gate induced by Gaussian clusters, providing a connection to routing mechanisms used in practical MoEs.

Critical weaknesses
  1. The role of least-squares initialization could be clarified further. Since Lemma B.1 already provides highly accurate expert coefficients under strong separation, I would appreciate a clearer explanation of the optimization challenges that remain after initialization and how the joint-training analysis relates to clustering followed by per-cluster regression.

  2. The finite-sample implications would benefit from further discussion. It would be useful to understand whether the population setting was chosen mainly to limit the scope, or whether a finite-sample extension faces additional technical obstacles, particularly in controlling expert errors along the training trajectory.

Questions for authors
  1. Since Lemma B.1 already gives highly accurate expert coefficients after initialization, could you clarify the main optimization challenges that remain under the theorem’s assumptions? In particular, how does the joint-learning guarantee relate to a simple approach that clusters the inputs and then fits each cluster by least squares?

  2. Is the restriction to population gradients primarily a scope choice, or does the current proof face a specific obstacle in the finite-sample setting? In particular, do you expect the expert-error control along the trajectory to extend through standard concentration arguments, or would it require additional ideas? This distinction would help assess the scope of the guarantee.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5Qx4

We thank the reviewer for the positive and focused review, and for finding the joint-learning result interesting and technically meaningful. The two weaknesses match the two questions, so we answer them together.

W1 and Q1. Role of the least-squares initialization, and relation to cluster-then-regress

Since Lemma B.1 already gives highly accurate expert coefficients after initialization, could you clarify the main optimization challenges that remain under the theorem's assumptions? In particular, how does the joint-learning guarantee relate to a simple approach that clusters the inputs and then fits each cluster by least squares?

We reiterate from the general response that experts need only to be polynomially close for adequate initialization. Even with accurate experts, the analysis has to handle the following, among others.

  • Gates and weights. The gates start up to ρ0≍d+log⁡(n/δ)\rho_0\asymp\sqrt{d+\log(n/\delta)} from their centers, the initial loss is of order d+log⁡nd+\log n, and n−mn-m surplus components must settle with the right group masses. This is the over-specified regime, in which EM is only sublinear.
  • Stability of the experts. The experts are trained throughout and may move anywhere within the budget BθB_\theta, yet their residuals enter every gate gradient, the per-cluster comparison and the cross-cluster ratios. The proof has to keep them inside the budget along the whole trajectory, and Theorem 4.2 is best read as this stability result. In the regime of the theorem a two-stage method should also succeed, and our result shows that the single joint gradient method, the procedure used in practice, does too. Outside that regime the methods differ. New Table 1 starts each method from the same initialization and reports the fraction of runs that recover the clusters (see Table 1 generated in response above).

At Δ=8\Delta=8 and 1212 the inputs alone form a well-separated Gaussian mixture, the setting of Zhou et al. (2026), so clustering first recovers the gates, as expected, and least squares on the recovered clusters gives accurate experts. At Δ=2\Delta=2 the input clusters overlap, and only joint training and closed-form EM recover them.

W2 and Q2. Finite samples

Is the restriction to population gradients primarily a scope choice, or does the current proof face a specific obstacle in the finite-sample setting? In particular, do you expect the expert-error control along the trajectory to extend through standard concentration arguments, or would it require additional ideas?

There was no fundamental hurdle in obtaining a finite-sample extension, and the result was not initially provided because it could not fit in the initial page limit. The expert-error control extends through standard uniform concentration, and only the weight step needed a new device. We now give the finite-sample analysis (Lemma 4.3, Theorem 4.4, Appendix F).

  • The population proof tolerates gradient errors of order εBθ/B2\varepsilon B_\theta/B^2 and weight-step errors of order ηε2/B2\eta\varepsilon^2/B^2 (Theorem F.2).
  • Sample averages achieve these uniformly over the bounded region that the iterates never leave, since the integrands are posteriors in [0,1][0,1] times polynomials in Gaussian variables (Lemma F.3).
  • The expert errors along the trajectory are controlled as in the population proof, with the accumulated gradient error added to the drift bound, which is what sets the required accuracy.
  • The weight certificate involves φi/p≤1/πi\varphi_i/p\le1/\pi_i, which is unbounded as surplus weights vanish, so the sample algorithm minimizes the sample loss over weights at least γ/n\gamma/n. This floor costs at most 2γ2\gamma in the certificate (Lemma F.1). The warm start needs N≳nB2Λ(1+Dθ)(d+log⁡(n/δ))/(πmin⁡ε2)N\gtrsim nB^2\Lambda(1+D_\theta)(d+\log(n/\delta))/(\pi_{\min}\varepsilon^2) samples. The overall sample complexity is polynomial, of order ε−8\varepsilon^{-8} up to logarithmic factors, and we have not optimized this exponent.

We hope these replies address the reviewer's questions.

Reference. Zhou, Xu, Fazel, Du. Global convergence of gradient EM for over-parameterized Gaussian mixtures. arXiv:2506.06584, 2026.

Thank you for the response and revisions. The clarification of the joint-training stability guarantee and the added finite-sample analysis address my main questions. I will maintain my weak accept rating.

Reviewer_DxGgOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a Gaussian-gated mixture of linear experts and analyzes joint training of the gate, experts, and mixing weights by population gradient EM on the likelihood. Under a separation condition on the gate centers, logarithmic overparameterization relative to the true number of experts, randomly initialized gates, and least-squares-initialized experts, the main result establishes convergence to a prescribed population KL accuracy above an exponentially small threshold in polynomially many iterations. The analysis establishes a gradient-dominance inequality by decoupling the loss into per-cluster subproblems, applying the Gaussian log-Sobolev inequality, and controlling expert drift within a shrinking error budget. Synthetic experiments investigate the scaling with the number of experts, the necessity of the least-squares warm start, and the effects of freely learned mixing weights and load-balancing penalties on router utilization and expert pruning.

Critical strengths
  • The paper studies joint training of the gate, experts, and mixing weights on the likelihood itself, addressing a relevant optimization question for mixture-of-experts models.
  • The comparison between least-squares and random expert initialization is a useful ablation of a key assumption in the analysis.
Critical weaknesses

The claim in lines 209–210 that the loss and updates are translation equivariant does not hold for the stated model. If x′=x−ax'=x-a, then an expert becomes y=x′⊤θℓ⋆+a⊤θℓ⋆+ϵy=x'^\top\theta_\ell^\star+a^\top\theta_\ell^\star+\epsilon.

The additional intercept depends on the expert and is not included in the model. Thus the authors cannot assume max⁡ℓ∥μℓ⋆∥≤Dmax⁡\max_\ell\|\mu_\ell^\star\|\le D_{\max} without imposing another restriction. This assumption is used to control moments in Lemma A.2, regression terms in Lemma C.4, and smoothness in Lemma D.1, so the problem affects the convergence argument.

For example, consider a balanced one-dimensional mixture with centers MM and M+ΔM+\Delta and both true expert coefficients equal to zero. Duplicate each true center equally among an even number nn of fitted experts, assign uniform weights, and set every fitted coefficient to b/Mb/M. The input marginal is fitted exactly and the weight dual gap is zero. As MM increases, the KL loss approaches b2/2b^2/2, the expert potential approaches zero, but the squared gradient norm grows as b2M2/nb^2M^2/n. The constant AA in the paper depends on Δ\Delta, not MM, so Eq. (49), ∥∇L∥F2≤2AL\|\nabla L\|_F^2\le 2AL, cannot hold uniformly over this family. The proof needs a bound on the absolute locations of the centers or a revised model and analysis.

Lemma B.1 already places the experts exponentially close to their true coefficients, and the analysis keeps them within a neighborhood whose radius shrinks with the target accuracy. The gate groups also remain unchanged throughout the analyzed trajectory. This leaves rather limited evidence about learning the gate and experts jointly from scratch. A comparison with clustering followed by regression, or with training only the gates while freezing the initialized experts, would help clarify what joint optimization contributes here. The weight updates introduce another restriction: Lemmas C.2 and C.4 require a prescribed dual-gap certificate, whereas the experiments perform only four inner EM updates. No gap measurements are reported, and the iteration bound does not account for the work required to obtain the theoretical certificate.

I also have difficulty interpreting Figure 1 as evidence for population global convergence. Its success criterion is evaluated on the training observations used to fit the model. This empirical log-density ratio has fitting bias and can be negative, and changing the fitted width changes that bias. The heatmap should be evaluated on independent data. A fixed budget of 10,000 updates also does not distinguish slow optimization from convergence to a suboptimal solution, so the plotted boundary provides limited evidence that mlog⁡mm\log m overparameterization is necessary.

The coverage explanation needs closer examination as well. Under the stated uniform six-component mixture and independent initialization, eight gate draws cover all six latent clusters with probability about 11.4%. Figure 2 nevertheless reports approximately 50–60% success at n=8n=8 under strong separation. I would need to see the actual initial coverage and subsequent cluster reassignment before accepting the explanation that a gate cannot cross a wide gap and that initial coverage is decisive.

The interpretation of the balancing and pruning results also seems too broad. With six equally weighted true components and eighteen fitted experts, copying each true expert three times and assigning weight 1/181/18 to each copy gives exact fitting, uniform average router usage, and zero balancing penalty simultaneously. This shows that accurate fitting and balanced routing are compatible, while free weights alone do not determine the effective expert count. Testing widths divisible by mm would help separate this issue from the chosen n=16,m=6n=16,m=6 configuration. Pruning only by the largest weights also favors concentrated solutions; merging similar experts or retaining representatives could give a different comparison. Finally, the skewed setting assigns one component weight 10−610^{-6}, making its expected training count only 0.01. Although acknowledged in the appendix, this makes that setting a weak test of the relationship between the fitted expert count and the true width.

Questions for authors

Please see the Critical Weaknesses section for my questions concerning the proof and experimental results.

How essential is the input-marginal likelihood p(x)p(x) to the convergence argument? Since the inputs already have a well-separated cluster structure, I would like to understand which conclusions depend on this additional clustering signal and whether training on the conditional likelihood p(y∣x)p(y\mid x) exhibits similar behavior.

Does overparameterization still improve optimization when initial cluster coverage is controlled? Comparing n=mn=m with larger widths while ensuring coverage in both cases would help distinguish the benefit of additional experts from the benefit of hitting every cluster at initialization.

How does the method compare with standard EM under the same initialization and fitted width? Since the center and linear-expert M-steps have closed-form solutions, this comparison would help clarify the computational and optimization tradeoffs of using gradient updates.

Could the authors make the resulting dependence on dimension explicit, including the restrictions imposed by AA and BθB_\theta? All experiments fix d=10d=10. Does the reported behavior persist as dimension increases when separation is scaled according to Assumption 1 and expert norms are controlled?

5 comments
AuthorsOpenReview ↗

Response to Reviewer DxGg (Part 1/4)

We thank the reviewer for a detailed and technically careful review, and for noting that the comparison of least-squares and random expert initialization is a useful ablation. Each point in the review led to a change in the paper. We quote each point and reply below.

W1. Translation equivariance

The claim in lines 209–210 that the loss and updates are translation equivariant does not hold for the stated model. [...] The proof needs a bound on the absolute locations of the centers or a revised model and analysis.

You are right; translating the inputs changes the model class. The claim was used only to bound ∣μℓ∗∣≤Dmax⁡\\|\mu^\ast_\ell\\|\le D_{\max}. We removed it and now track M∗=max⁡ℓ∣μℓ∗∣M_\ast=\max_\ell\\|\mu^\ast_\ell\\| through Λ=d+M∗2\Lambda=d+M_\ast^2. Every step holds as written, and each use of M∗M_\ast is highlighted in red in the paper. Your one-dimensional family is consistent with the corrected constants, since along it ∣∇L∣2\\|\nabla\mathcal L\\|^2 grows like s2s^2, while A≥(1+Λ)2A\ge(1+\Lambda)^2 grows like s4s^4. The experiments are unaffected, since their true centers have mean zero and hence M∗≤Dmax⁡M_\ast\le D_{\max}.

W2. Experts start near the truth, and what joint training adds

Lemma B.1 already places the experts exponentially close to their true coefficients [...]. A comparison with clustering followed by regression, or with training only the gates while freezing the initialized experts, would help clarify what joint optimization contributes here.

We affirm that the main result requires that experts stay near the truth, and reiterate the perspective of Theorem 4.2 as a stability result. As stated in the general comment, the proof needs only Uθ(0)≤Bθ2/4U_\theta^{(0)}\le B_\theta^2/4, which is polynomial in ε\varepsilon, and does not require exponentially close experts. To address the baselines mentioned, a new Table 1 (Section 5.2) starts every method from the same gates and weights (m=6m=6, n=16n=16, 20 runs). It reports the fraction of runs that recover the clusters. The failure of cluster recovery (Experts frozen) at closer separation (Δ=2\Delta=2 and 88), and a degradation in recovery performance at greater separation (Δ=12\Delta=12), further suggest that stability, even when experts are initialized close, is non-trivial.

MethodRecovery at Δ=2/8/12\Delta=2/8/12
Joint training, least-squares experts1.00/1.00/0.951.00/1.00/0.95
Joint training, random experts1.00/1.00/0.801.00/1.00/0.80
Cluster the inputs, then regress0.00/1.00/0.900.00/1.00/0.90
Experts frozen (gates and weights trained)0.00/0.00/0.650.00/0.00/0.65
Gates frozen (experts and weights trained)0.00/0.55/0.700.00/0.55/0.70
Closed-form EM, least-squares experts1.00/1.00/0.951.00/1.00/0.95

The theorem does not cover learning experts from far away. We name this as an open direction, and Figure 1(b) suggests that it is possible in practice.

AuthorsOpenReview ↗

Response to Reviewer DxGg (Part 2/4)

W2 (continued). Dual-gap certificate

Lemmas C.2 and C.4 require a prescribed dual-gap certificate, whereas the experiments perform only four inner EM updates. No gap measurements are reported, and the iteration bound does not account for the work required to obtain the theoretical certificate.

Correct. We now log the sample dual gap G^π=max⁡iψ^i/πi−1\hat G_\pi=\max_i\hat\psi_i/\pi_i-1, which uses the same posteriors as the EM update. In the joint-training runs of Table 1, after the four inner updates, the median over runs of its median over outer iterations is 3.6×10−53.6\times10^{-5}, and that of its 99th percentile 2.4×10−22.4\times10^{-2} (Appendix G). The weight subproblem is convex in π\pi, and the gap is a stopping rule that can be checked at run time. We now report that TT counts outer iterations with each inner step costing one E-step. Bounding the total number of inner steps is outside the focus of the paper.

W3. Figure 1 (training data, fixed budget)

Its success criterion is evaluated on the training observations used to fit the model. [...] The heatmap should be evaluated on independent data. A fixed budget of 10,000 updates also does not distinguish slow optimization from convergence to a suboptimal solution, so the plotted boundary provides limited evidence that [...] overparameterization is necessary.

We agree with the first point on the evaluation on independent data. To the second point, we made the adjustment and observed no qualitative change in our experiments. We reran Figure 1(a) on held-out data with nn up to 200, 30 trials per cell and twice the budget (17,25017{,}250 runs; new Appendix H.5). We acknowledge that any fixed budget is arbitrary, and computing limitations necessitated picking one. We provide a few more comments on the heat-map experiment for completeness:

  • Training-set bias. The training-set KL is biased downward by about k/Nk/N, k=m(2d+1)−1k=m(2d+1)-1 (median −0.121-0.121 at m=50m=50, against −0.105-0.105). On held-out data a fixed tolerance meets the estimation floor k/(2N)k/(2N), which exceeds 0.020.02 for m≥20m\ge20, so a run now succeeds when its held-out KL is at most 0.02+k/(2N)0.02+k/(2N).
  • Boundary. No run with n=mn=m succeeds for m≥10m\ge10. At the grid width closest to mlog⁡mm\log m, 1717\\% to 8787\\% of the runs succeed for 4≤m≤504\le m\le50, so n=mlog⁡mn=m\log m lies within the transition from failure to success. Where the initial gates cover every cluster with probability 1/21/2, 8080\\% to 9797\\% of the runs already succeed. The paper no longer describes the transition as sharp.
  • Slow or stuck. The budget binds for m≥16m\ge16, where every failed run that ends with all clusters covered stops at the cap. However, 8787\\% of the failed runs with n≥mn\ge m end with a true cluster that owns no gate (median held-out KL 4.04.0), and none of these succeeds. These runs are stuck rather than slow.
  • Necessity. Our result is that n≳mlog⁡mn\gtrsim m\log m suffices, and we did not claim that it is necessary. The sentence suggesting necessity was a conjecture, and we removed it.
AuthorsOpenReview ↗

Response to Reviewer DxGg (Part 3/4)

W4. Coverage

Under the stated uniform six-component mixture and independent initialization, eight gate draws cover all six latent clusters with probability about 11.4%. [...] I would need to see the actual initial coverage and subsequent cluster reassignment before accepting the explanation that a gate cannot cross a wide gap and that initial coverage is decisive.

We agree and thank the reviewer for suggesting exploration of cluster reassignment. New Appendix H.1 logs initial coverage and gate reassignments at Δ=12\Delta=12, with 4040 runs per width with independent draws and 20 further runs with one gate drawn from each cluster. The independent runs split by chance into those that start with every cluster covered and those that do not. The entries are recovered runs over runs.

Independent draws (40 per width)Coverage guaranteed
nnP(cover)covered at startuncovered at start(20 per width)
60.0150.015none6/406/4020/2020/20
80.1140.1147/77/719/3319/3320/2020/20
120.4380.43814/1414/1420/2620/2620/2020/20

No gate changed cluster when all were covered at initialization, as in our analysis. In the uncovered runs a gate changed cluster in 0.820.82 of cases, and every uncovered run that recovered did so through such a change. We removed the claims that a gate cannot cross a wide gap and that coverage is necessary, and gate migration is now listed as an interesting open direction.

W5. Balancing and pruning

With six equally weighted true components and eighteen fitted experts, copying each true expert three times [...] gives exact fitting, uniform average router usage, and zero balancing penalty simultaneously. [...] Testing widths divisible by [...] would help separate this issue [...]. Pruning only by the largest weights also favors concentrated solutions; merging similar experts or retaining representatives could give a different comparison. Finally, the skewed setting assigns one component weight [...], making its expected training count only 0.01.

Agreed on the need to distinguish between similar experts as well. The table summarizes the 20 runs per setting at n=16n=16, Δ=8\Delta=8, with mean effective counts (Table 4 of the paper).

uniform, λ=0\lambda=0uniform, λ=1\lambda=1skewed, λ=0\lambda=0skewed, λ=1\lambda=1
neffn_{\mathrm{eff}}6.036.0311.2911.293.863.8612.8012.80
neffZn^Z_{\mathrm{eff}}6.016.016.396.393.753.754.044.04
recovery1.001.000.900.900.750.750.550.55
success (held-out KL ≤0.02\le0.02)1.001.000.250.250.700.700.200.20
median held-out KL0.0060.0060.0220.0220.0070.0070.0230.023
  • Free weights. They do not determine neffn_{\mathrm{eff}} at the optimum. The paper now says so and calls the concentration at λ=0\lambda=0 an implicit bias of gradient EM. At n=16n=16 a nearly exact balanced fit exists (joint KL at most 0.00430.0043).
  • Duplicated experts. Your example shows that neffn_{\mathrm{eff}} counts copies of one expert separately. We added a similarity-sensitive extension (Leinster and Cobbold, 2012), neffZ=1/(ρˉ⊤Zρˉ)n^Z_{\mathrm{eff}}=1/(\bar\rho^\top Z\bar\rho), where Zij=∫φiφjZ_{ij}=\int\sqrt{\varphi_i\varphi_j} is the overlap of the joint densities of components ii and jj. It equals neffn_{\mathrm{eff}} when no components overlap and does not change when a component is split into copies, so it takes the same value for your exact balanced fit and for one expert per cluster. Widths divisible by mm are therefore not needed to separate the two effects. At n=16n=16, neffZn^Z_{\mathrm{eff}} stays near the true counts 66 (uniform) and 3.763.76 (skewed), so the balanced models mostly hold near-copies of the same experts.
  • Accuracy. The median held-out KL of the balanced models is close to the statistical error n(2d+1)/(2N)n(2d+1)/(2N) of 16 rather than 6 components, so their lower fixed-threshold success mostly reflects this error.
  • Pruning. Merging components with overlap at least 1/21/2 before pruning changes the held-out conditional KL at k=mk=m, λ=1\lambda=1, from 5.4425.442 to 0.0170.017 (uniform) and from 7.3177.317 to 0.0510.051 (skewed), the values at k=6k=6 in Figure 2(b, c). The text now says that a balanced model spreads each cluster's mass over near-copies, which pruning by weight alone cannot exploit, and that merging first recovers a compact model.
  • Skewed prior. Figure 2 is replaced by this experiment, with the skewed prior (1,2,3,5,8,13)/32(1,2,3,5,8,13)/32, whose smallest expected count is 312. Reference. Leinster, Cobbold. Measuring diversity: the importance of species similarity. Ecology 93(3), 2012.
Reviewer_mDCxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies joint likelihood training for a mixture of linear experts whose router comes from Gaussian input clusters. By analyzing a setup where data naturally forms Gaussian clusters and each expert solves a linear regression task, the authors prove that standard gradient-based training reliably finds the true distribution provided you over-provision the model with roughly mlog⁡mm \log m experts, give the experts a local regression head start, and the true data clusters are well-separated. The extra experts ensure that random initialization lands at least one gate near every true cluster, while the proof decouples the clusters and ensures expert errors do not destabilize the router during training. Finally, they have some experiments to show that experts actually learn fine from scratch without the theoretical warm start.

Critical strengths
  • Seems like there's an elegant new theory result (theorem B2)
  • Mixing weights are optimized: brings learnable mixing weights / router biases into the analysis. Fixed in other works as they claim
  • Treating gradient EM as gradient descent on the KL loss, and linking the n>mlogmn > m log m over-parameterization to coverage at initialization, elegant way of explaining why over-provisioning helps
Critical weaknesses
  • Connection to modern MoE seems overstated: The abstract and intro motivate the work with deployed sparse MoEs. But router weights and biases are tied to the cluster means and mixing weights, routing is dense rather than top-k, and the model is trained on the joint likelihood.
  • Lemma 4.1 (informal) lists three items, but the text refers to items 1–4, and item 3 cites itself ("using item 3").
  • Not sure if I follow if the experiment actually tests the theorem (can you clarify if Δ\Delta tested is satisfied by your theory)
Questions for authors
  • What is technically new relative to Zhou et al. (2026), beyond (a) the regression factor in the cross-group ratio bound and (b) the dual-gap handling of π\pi? A short comparison table would help.
  • Could the result hold with experts initialized only within a constant (ϵ\epsilon-independent) radius of the truth?
  • Under Assumption 1, the initial expert error is e−Ω(Δ2)e^{-\Omega(\Delta^2)}. Is it fair to say the expert parameters are effectively fixed near the truth during the analyzed trajectory? If not, could you point to where the analysis handles experts that move a non-negligible distance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mDCx (Part 1/2)

We thank the reviewer for the careful reading and precise pointers, and for noting the new theory result, the learnable mixing weights, and the link between over-parameterization and coverage at initialization. We quote each point and reply below.

W1. Connection to modern MoEs

The abstract and intro motivate the work with deployed sparse MoEs. But router weights and biases are tied to the cluster means and mixing weights, routing is dense rather than top-k, and the model is trained on the joint likelihood.

We agree that the initial framing overstated the connection to sparse-gate MoEs, and we have updated the abstract to explicitly state that the analyzed model is dense and trained on the joint likelihood. However, we maintain that our formulation captures essential mechanics of modern MoEs. As shown in the Introduction, our gate model equates to a dense softmax gate with a bias. We draw the connection to sparsity primarily to demonstrate how mixing coefficients (gate biases) can guide model pruning—a property highly relevant to practitioners and analogous to sparse MoE inference. Extending this analysis to the conditional likelihood setting remains an important open question, which we now note in the text.

W2. Lemma 4.1 numbering

Lemma 4.1 (informal) lists three items, but the text refers to items 1–4, and item 3 cites itself ("using item 3").

A missing item merged items 2 and 3. This is fixed.

W3. Do the experiments satisfy the separation condition?

Not sure if I follow if the experiment actually tests the theorem (can you clarify if [...] tested is satisfied by your theory)

No, as addressed in the general comment. With Csep=1C_{\mathrm{sep}}=1 and δ=0.1\delta=0.1, Assumption 1 requires Δ≥15.5\Delta\ge15.5 at n=16n=16 and Δ≥59\Delta\ge59 at n=200n=200 in d=10d=10 (Table 3), and the proof needs Csep≥32C_{\mathrm{sep}}\ge32. This is by design. The experiments probe behavior outside the theory, to identify which assumptions future analysis could relax (separation, warm start, gate migration), rather than to verify the constants. Section 5 now says this.

AuthorsOpenReview ↗

Response to Reviewer mDCx (Part 2/2)

Q1. Novelty relative to Zhou et al. (2026)

What is technically new relative to Zhou et al. (2026), beyond (a) the regression factor in the cross-group ratio bound and (b) the dual-gap handling of [...]? A short comparison table would help.

Since the joint distribution for a component φ(z)\varphi(z) is a Gaussian distribution, we have slightly generalized the Gaussian Mixture Model result of Zhou et al. (2026), although it is not emphasized. We also lay out the differences throughout the paper (Section 2, Sections 4.3 and 4.4, Appendices B to D), so we did not add a separate table. Specifically, we inherit the random gate initialization, the nearest-center projection, the gate potential, the form of the separation condition, and the dual-gap certificate for the weights; your (b) is theirs. New are the expert block and its coupling to the gates.

  • the expert residuals in every gate gradient (Stein form, eq. 21);
  • the joint per-cluster comparison (Lemma C.3), which handles the regression coordinate and produces the deficit that forces the expert budget;
  • the regression factors in the cross-cluster ratio bounds (Lemma D.3; your (a));
  • the smoothness of the expert block (Lemma D.1);
  • the drift control that keeps the experts inside the budget (Lemma B.3). Zhou et al. also have a local phase, which we do not. Like them, we now have a finite-sample guarantee (Theorem 4.4).

Q2. Constant radius

Could the result hold with experts initialized only within a constant ([...]-independent) radius of the truth?

Partly (see general response). The cross-group bounds need only the constant accuracy c/((1+Dθ)Λ)c/((1+D_\theta)\Lambda), but in the per-cluster comparison (Lemma C.3) the expert errors appear as a deficit that must stay below cεc\varepsilon, which forces Bθ≲ε/BB_\theta\lesssim\varepsilon/B. Empirically, a constant radius often suffices. Joint training from random N(0,Id)N(0,I_d) experts recovers the clusters in 1.00/1.00/0.801.00/1.00/0.80 of runs at Δ=2/8/12\Delta=2/8/12 (Table 1; see also Figure 1(b)).

Q3. Are the experts effectively fixed?

Is it fair to say the expert parameters are effectively fixed near the truth during the analyzed trajectory? If not, could you point to where the analysis handles experts that move a non-negligible distance?

In the regime of the theorem they stay within BθB_\theta of the truth. That is the stability result; they are not held fixed, however. They are trained, may move anywhere within BθB_\theta, and Lemma B.3 proves that they stay within it despite the coupling to the gates described above. Outside that regime, freezing the warm-started experts lowers recovery at Δ=2/8/12\Delta=2/8/12 from 1.00/1.00/0.951.00/1.00/0.95 to 0.00/0.00/0.650.00/0.00/0.65 (Table 1).

We hope these replies address the reviewer's concerns.

Reference. Zhou, Xu, Fazel, Du. Global convergence of gradient EM for over-parameterized Gaussian mixtures. arXiv:2506.06584, 2026.

I thank the authors for their detailed and careful responses. The revised abstract now accurately describes the analysed model as a dense, jointly trained mixture, and the clarification about the experiments deliberately probing outside the theory. My remaining reservations are that the analysed setting is some distance from practical sparse MoEs, and that the separation constants the theory requires are far from those tested. On balance I continue to keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes to the revised manuscript

We thank the three reviewers for their careful and constructive reviews. Across the reviews, the main concerns the reviewers raised were

(1) the benefit of joint training compared to simply clustering followed by regression

(2) an error in the translation-equivariance claim used to bound the center norms

(3) how the experiments relate to the theory, and the evaluation of Figure 1

(4) the finite-sample setting and the connection to deployed MoEs

The revised PDF marks every change in red, and the main text stays within 10 pages. All section, lemma, figure and table numbers in our replies refer to the revised PDF.

1. Theory

  • Main result. Our main Theorem 4.2 can be better interpreted as a stability result. To recount, the experts start within an error budget BθB_\theta and stay there. Also, no gate changes cluster, while gates, experts and weights are all jointly trained. This is not automatic, since by Stein's lemma every gate gradient contains the residuals of all experts, and the expert errors enter the cross-cluster likelihood ratios (Lemma D.3). The proof controls this coupling along the whole trajectory (Lemma B.3).
  • Initialization. Reviewers were concerned that the expert initialization seemed to require exponentially close experts. We acknowledge this concern and clarify in our presentation that exponentially close experts are not actually necessary for our theory. Algorithm 1 now requires only new Assumption 2, Uθ(0)≤Bθ2/4U_\theta^{(0)}\le B_\theta^2/4. This is polynomial in ε\varepsilon and, for small ε\varepsilon, independent of the separation Δ\Delta. Least squares meets it with a margin of order ε/B2\varepsilon/B^2.
  • Finite samples. New Lemma 4.3 and Theorem 4.4 (Appendix F) show that sample-based gradient EM, including the warm start, satisfies the conclusions of Theorem 4.2 with polynomially many samples.
  • Corrections. The claim made of translation-equivariance was not correct. However, the constants now depend on M∗=max⁡ℓ∣μℓ∗∣M_\ast=\max_\ell\\|\mu^\ast_\ell\\| through Λ=d+M∗2\Lambda=d+M_\ast^2, and the rest of the argument goes unchanged. The item numbering of Lemma 4.1 is fixed.

2. Experiments. We have added additional experiments and highlight that we intended to probe how gradient EM behaves outside the regime of the theorem and explore assumptions that future analysis could relax (warm start, separation, gate migration). The new experiments use N=10,000N=10{,}000 training and 20,00020{,}000 held-out points and at least 20 runs per setting.

3. Positioning. The abstract now describes the gate as a dense softmax gate trained on the joint likelihood, and the introduction says that the model has the form of a softmax-gated MoE and states how it differs from deployed sparse MoEs.

We reply to each reviewer point by point below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.