- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes nGEM, an optimizer for mixture density networks (MDNs) that addresses their slow convergence and mode collapse under negative log-likelihood (NLL) training. The authors view MDNs as latent-variable models and combine three facts: NLL and the generalized-EM objective share gradients, exact EM corresponds to natural gradient descent with the complete-data Fisher, and this Fisher is block-diagonal for Gaussian mixtures. The result is a closed-form preconditioner on the network outputs, with diagonal scalings for the Gaussian components and a pseudo-inverse of the categorical Fisher for the mixture logits, applied before backpropagation at negligible cost. Experiments on synthetic tasks, inverse MNIST, UCI regression, simulation-based inference and distributional RL show faster convergence and markedly reduced mode collapse.
Critical strengths
The paper tackles a practical problem, the slow convergence and mode collapse of MDNs trained with NLL, with a principled and simple solution. Recasting training as EM and exploiting the block-diagonal complete-data Fisher yields a closed-form natural-gradient preconditioner that is easy to implement, adds negligible cost and works with standard optimizers. The authors identify the mechanism behind its effectiveness: inverse weighting by the mixture weights lets collapsed components recover. Experiments across diverse tasks, including high-dimensional outputs, consistently show faster convergence and better mode coverage, and the paper is well written.
Critical weaknesses
Some points need clarification. Theorem 3.3 relates EM and natural gradient descent for the mixture parameters, but this does not carry over to the network weights through backpropagation. Moreover, Eq. (35) equates a data average of posterior expectations with an expectation under the model’s joint distribution, which holds only when the model matches the data; the equivalence is therefore approximate, and the weighting in nGEM approximates the responsibility weighting of exact EM. Stating this would make the theory more accurate without changing the method.
On the empirical side, the mode-collapse experiments compare nGEM only with plain NLL; the Dirichlet-prior baseline appears only in the regression experiments, and other remedies such as evolving winner-takes-all are not compared.
Minor: From my understanding, Eq. (60) should read (the reference code uses the correct sign already)
Questions for authors
Mode-collapse baselines: How does nGEM compare with methods designed against mode collapse, such as EWTA or entropy regularization of the mixture weights?
Regression evaluation: Do the regression gains hold with learning rates tuned per method and with mixture-mean RMSE or CRPS instead of the best-component RMSE?
2 comments
I thank the authors for the detailed response and the additional experiments.
Theoretical concerns.
- Corrected proof of Theorem 3.3. The exponential-family assumption makes the complete-data Hessian data-independent only in natural coordinates. In nGEM's coordinates, the M-step Hessian of the component blocks depends on the data through (and, for , through ), whereas the Fisher uses their model expectations, so the two agree only in expectation under the model. Under nonlinear reparameterizations, the Hessian of also gains a term linear in , so exactness does not extend to general coordinates.
- Scope of Theorem 3.3. The equivalence holds in output space only, so no EM or natural-gradient property is established for the actual network update.
- Overall. The central theoretical result was initially incorrect and is now described by the authors as a motivating heuristic, so the paper's framing as natural-gradient EM is not supported by the theory.
Experiments. Several points remain open:
- The mixture-mean RMSE results are not yet reported
- Learning rates appear not to be tuned separately for NLL and nGEM, although nGEM rescales gradients by and , which changes the effective step size.
- The added EWTA and entropy-regularization baselines cover only regression; the mode-collapse experiments, which support the central claim, still compare against plain NLL only.
- Figure 1 shows a selected failure case and does not seem representative of typical NLL behavior, as the authors acknowledge that the catastrophic failure does not persist across seeds.
Assessment. Given that the empirical contribution is limited and the practical advantage of the method is not clearly established, the theory is crucial for establishing the paper's contribution. The concerns raised by the other reviewers, together with the response, made me realize that the theoretical issues are more significant than I initially assessed. I therefore lower my score to 2.
We thank the reviewer for the constructive feedback.
We are very grateful to the reviewer for flagging this error. We want to acknowledge that all other reviewers have raised the same concern regarding the correctness of the proof for Theorem 3.3, in particular Equation 35-36 in Appendix B.3.
We would like to first clarify that this erroneous derivation does not invalidate our original claim in Theorem 3.3. Our error is that to compute the complete-data FIM F^(θ), the expectation should be taken under the model distribution p(x∣θ) rather than the data distribution p(x).
We have hence updated our proof for Theorem 3.3 in Appendix B.3, with an additional assumption that the complete-data probabilistic model p(x,z∣θ) belongs to the exponential family (EF) and can be written as p(x,z∣η)=h(x,z)expη⊤T(x,z)−A(η).
With the exponential-family assumption, the Hessian ∇η2logp(x,z∣η)=∇η2A(η) is independent of x and remains unchanged under the expectation w.r.t. either p(x) or p(x∣θ). Please kindly refer to the updated manuscript (Appendix B.3) for detailed derivations.
Yes, the EM-NGD equivalance (up to aforementioned assumptions) only holds in the mixture parameter space. We have briefly clarified in Section 5 that our method computes the natural gradient in the network output ϕ-space, not in the network weight θ-space.
Consider a neural network ϕ=f(x;θ) with parameter Jacobians J=∂ϕ/∂θ. Let g_\boldsymbol{\phi} = \nabla_\boldsymbol{\phi} L and g_\boldsymbol{\theta} = \nabla_\boldsymbol{\theta} L where L is the loss function.
In general, we have \Delta_\boldsymbol{\theta}^{(1)} \neq \Delta_\boldsymbol{\theta}^{(2)}. A sufficient condition for equality is JJ⊤=I, which is generally not true for arbitrary networks.
In conclusion, Theorem 3.3 mainly serves as a motivating heuristics rather than exact theoretical characterization for the nGEM algorithm. We trade off theoretical exactness for practical tractability.
We are happy to include this justification in the updated manuscript if the reviewer found it reasonable.
Thanks and we have fixed the typo.
We tested EWTA (#winners annealed from K=3 to 1) and entropy regularization on regression tasks, with RMSE still computed as the best-of-all RMSE. Please see Table 3 for full comparison with other baselines.
kin8nmenergyhousingkin8nmenergyhousingWe understand the confusion around best-of-all RMSE. The rationale behind the choice is that in many cases (e.g., inverse modeling), it is often desirable if one of the candidates is sufficiently accurate when the model could generate multiple candidates.
Nevertheless, we computed the weighted-mean RMSE RMSE(∑kπkμk,y), and the reported metrics is generally consistent with NLL and do not contradict our current conclusion. We will attach the full results in the appendix when updating the manuscript.