acceptodds
Under review as a conference paper at ICLR 2027

Learning Mixture Density via Natural Gradient Expectation Maximization

Abstract

We present *natural gradient expectation maximization* (nGEM), a novel approach to learning mixture density networks for conditional density estimation. Mixture density networks are conventionally optimized via the negative log-likelihood (NLL) objective, which suffers from slow convergence and mode collapse. We instead cast mixture density networks as latent-variable models that can be optimized in an expectation maximization framework, and derive tractable factorized natural gradient updates drawing upon theoretical ties between expectation maximization and natural gradient descent. Empirically, we show that nGEM achieves up to 10 faster convergence with almost zero extra computational overhead, yet scales well to practical and challenging tasks where NLL otherwise fails for mode collapse.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Gaussian mixture model, mixture density network, natural gradient descent, expectation maximization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vVz4OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes nGEM, an optimizer for mixture density networks (MDNs) that addresses their slow convergence and mode collapse under negative log-likelihood (NLL) training. The authors view MDNs as latent-variable models and combine three facts: NLL and the generalized-EM objective share gradients, exact EM corresponds to natural gradient descent with the complete-data Fisher, and this Fisher is block-diagonal for Gaussian mixtures. The result is a closed-form preconditioner on the network outputs, with diagonal σ2/πk\sigma^2/\pi_k scalings for the Gaussian components and a pseudo-inverse of the categorical Fisher for the mixture logits, applied before backpropagation at negligible cost. Experiments on synthetic tasks, inverse MNIST, UCI regression, simulation-based inference and distributional RL show faster convergence and markedly reduced mode collapse.

Critical strengths

The paper tackles a practical problem, the slow convergence and mode collapse of MDNs trained with NLL, with a principled and simple solution. Recasting training as EM and exploiting the block-diagonal complete-data Fisher yields a closed-form natural-gradient preconditioner that is easy to implement, adds negligible cost and works with standard optimizers. The authors identify the mechanism behind its effectiveness: inverse weighting by the mixture weights lets collapsed components recover. Experiments across diverse tasks, including high-dimensional outputs, consistently show faster convergence and better mode coverage, and the paper is well written.

Critical weaknesses

Some points need clarification. Theorem 3.3 relates EM and natural gradient descent for the mixture parameters, but this does not carry over to the network weights through backpropagation. Moreover, Eq. (35) equates a data average of posterior expectations with an expectation under the model’s joint distribution, which holds only when the model matches the data; the equivalence is therefore approximate, and the πk\pi_k weighting in nGEM approximates the responsibility weighting ρk\rho_k of exact EM. Stating this would make the theory more accurate without changing the method.

On the empirical side, the mode-collapse experiments compare nGEM only with plain NLL; the Dirichlet-prior baseline appears only in the regression experiments, and other remedies such as evolving winner-takes-all are not compared.

Minor: From my understanding, Eq. (60) should read I−1K11⊤I - \tfrac{1}{K}\mathbf{1}\mathbf{1}^\top (the reference code uses the correct sign already)

Questions for authors

Mode-collapse baselines: How does nGEM compare with methods designed against mode collapse, such as EWTA or entropy regularization of the mixture weights?

Regression evaluation: Do the regression gains hold with learning rates tuned per method and with mixture-mean RMSE or CRPS instead of the best-component RMSE?

2 comments

We thank the reviewer for the constructive feedback.

Eq. (35) equates a data average of posterior expectations with an expectation under the model’s joint distribution, which holds only when the model matches the data ...

We are very grateful to the reviewer for flagging this error. We want to acknowledge that all other reviewers have raised the same concern regarding the correctness of the proof for Theorem 3.3, in particular Equation 35-36 in Appendix B.3.

We would like to first clarify that this erroneous derivation does not invalidate our original claim in Theorem 3.3. Our error is that to compute the complete-data FIM F^(θ)\hat{F}(\boldsymbol{\theta}), the expectation should be taken under the model distribution p(x∣θ)p(\boldsymbol{x}|\boldsymbol{\theta}) rather than the data distribution p(x)p(\boldsymbol{x}).

We have hence updated our proof for Theorem 3.3 in Appendix B.3, with an additional assumption that the complete-data probabilistic model p(x,z∣θ)p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\theta}) belongs to the exponential family (EF) and can be written as p(x,z∣η)=h(x,z)exp⁡η⊤T(x,z)−A(η).p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\eta}) = h(\boldsymbol{x}, \boldsymbol{z}) \exp\\{\boldsymbol{\eta}^\top T(\boldsymbol{x}, \boldsymbol{z}) - A(\boldsymbol{\eta})\\}.

With the exponential-family assumption, the Hessian ∇η2log⁡p(x,z∣η)=∇η2A(η)\nabla^2_{\boldsymbol{\eta}} \log p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\eta}) = \nabla^2_{\boldsymbol{\eta}} A(\boldsymbol{\eta}) is independent of x\boldsymbol{x} and remains unchanged under the expectation w.r.t. either p(x)p(\boldsymbol{x}) or p(x∣θ)p(\boldsymbol{x}|\boldsymbol{\theta}). Please kindly refer to the updated manuscript (Appendix B.3) for detailed derivations.

Theorem 3.3 relates EM and natural gradient descent for the mixture parameters, but this does not carry over to the network weights through backpropagation.

Yes, the EM-NGD equivalance (up to aforementioned assumptions) only holds in the mixture parameter space. We have briefly clarified in Section 5 that our method computes the natural gradient in the network output ϕ\boldsymbol{\phi}-space, not in the network weight θ\boldsymbol{\theta}-space.

Consider a neural network ϕ=f(x;θ)\boldsymbol{\phi} = f(\boldsymbol{x}; \boldsymbol{\theta}) with parameter Jacobians J=∂ϕ/∂θJ = \partial\boldsymbol{\phi} / \partial\boldsymbol{\theta}. Let g_\boldsymbol{\phi} = \nabla_\boldsymbol{\phi} L and g_\boldsymbol{\theta} = \nabla_\boldsymbol{\theta} L where LL is the loss function.

  1. Our approach computes the natural gradient w.r.t. ϕ\boldsymbol{\phi} and back-propagate to θ\boldsymbol{\theta} via Jacobians JJ, producing the following parameter update: \Delta_\boldsymbol{\theta}^{(1)} = J^\top F(\boldsymbol{\phi})^{-1} g_\boldsymbol{\phi}.
  2. The standard natural gradient w.r.t. θ\boldsymbol{\theta} requires computing the FIM F(θ)F(\boldsymbol{\theta}), which is often intractable, and produces the following parameter update: \Delta_\boldsymbol{\theta}^{(2)} = F(\boldsymbol{\theta})^{-1} g_\boldsymbol{\theta} = (J^\top F(\boldsymbol{\phi}) J)^{-1} (J^\top g_\boldsymbol{\phi}).

In general, we have \Delta_\boldsymbol{\theta}^{(1)} \neq \Delta_\boldsymbol{\theta}^{(2)}. A sufficient condition for equality is JJ⊤=IJJ^\top = I, which is generally not true for arbitrary networks.

In conclusion, Theorem 3.3 mainly serves as a motivating heuristics rather than exact theoretical characterization for the nGEM algorithm. We trade off theoretical exactness for practical tractability.

We are happy to include this justification in the updated manuscript if the reviewer found it reasonable.

Eq. (60) should read I−1K11⊤I - \frac{1}{K} \mathbf{1}\mathbf{1}^\top ...

Thanks and we have fixed the typo.

How does nGEM compare with methods designed against mode collapse, such as EWTA or entropy regularization ... ?

We tested EWTA (#winners annealed from K=3K=3 to 11) and entropy regularization on regression tasks, with RMSE still computed as the best-of-all RMSE. Please see Table 3 for full comparison with other baselines.

MethodNLLRMSE
kin8nmenergyhousingkin8nmenergyhousing
ETWA-1.61 ± 0.04-2.65 ± 0.21-0.27 ± 0.100.29 ± 0.000.75 ± 0.120.92 ± 0.16
Ent-Reg-1.60 ± 0.03-3.81 ± 0.14-0.59 ± 0.190.35 ± 0.021.09 ± 0.040.98 ± 0.17

Do the regression gains hold ... with mixture-mean RMSE instead of the best-component RMSE?

We understand the confusion around best-of-all RMSE. The rationale behind the choice is that in many cases (e.g., inverse modeling), it is often desirable if one of the candidates is sufficiently accurate when the model could generate multiple candidates.

Nevertheless, we computed the weighted-mean RMSE RMSE(∑kπkμk,y)\mathrm{RMSE}(\sum_k \boldsymbol{\pi}_k\boldsymbol{\mu}_k, \boldsymbol{y}), and the reported metrics is generally consistent with NLL and do not contradict our current conclusion. We will attach the full results in the appendix when updating the manuscript.

I thank the authors for the detailed response and the additional experiments.

Theoretical concerns.

  1. Corrected proof of Theorem 3.3. The exponential-family assumption makes the complete-data Hessian data-independent only in natural coordinates. In nGEM's coordinates, the M-step Hessian of the component blocks depends on the data through rikr_{ik} (and, for σ\sigma, through (y−μk)2(y - \mu_k)^2), whereas the Fisher uses their model expectations, so the two agree only in expectation under the model. Under nonlinear reparameterizations, the Hessian of QQ also gains a term linear in ∇Q\nabla Q, so exactness does not extend to general coordinates.
  2. Scope of Theorem 3.3. The equivalence holds in output space only, so no EM or natural-gradient property is established for the actual network update.
  3. Overall. The central theoretical result was initially incorrect and is now described by the authors as a motivating heuristic, so the paper's framing as natural-gradient EM is not supported by the theory.

Experiments. Several points remain open:

  • The mixture-mean RMSE results are not yet reported
  • Learning rates appear not to be tuned separately for NLL and nGEM, although nGEM rescales gradients by σ2\sigma^2 and 1/πk1/\pi_k, which changes the effective step size.
  • The added EWTA and entropy-regularization baselines cover only regression; the mode-collapse experiments, which support the central claim, still compare against plain NLL only.
  • Figure 1 shows a selected failure case and does not seem representative of typical NLL behavior, as the authors acknowledge that the catastrophic failure does not persist across seeds.

Assessment. Given that the empirical contribution is limited and the practical advantage of the method is not clearly established, the theory is crucial for establishing the paper's contribution. The concerns raised by the other reviewers, together with the response, made me realize that the theoretical issues are more significant than I initially assessed. I therefore lower my score to 2.

Reviewer_UmksOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes natural gradient expectation maximization (nGEM) for training Gaussian mixture density networks. It reformulates likelihood-based training using latent-component responsibilities and applies blockwise preconditioning based on the complete-data Fisher information of the mixture parameters. The resulting output-space gradients are backpropagated through the neural network. The main methodological contribution is the particular output-gradient preconditioning scheme, rather than a new density model or the introduction of EM to mixture models.

Critical strengths

Simple and computationally inexpensive modification of MDN training. The component-wise Fisher structure allows the proposed preconditioning to be implemented without changing the MDN architecture or constructing a large network-level Fisher matrix. For diagonal Gaussian components, the required operations are particularly inexpensive, and Table 4 supports the claim of small additional runtime in the evaluated settings.

Critical weaknesses
  1. Limited methodological novelty and insufficiently established significance

EM for Gaussian mixtures and connections between EM and natural-gradient methods are established starting points, as the paper itself acknowledges. Introducing latent component assignments into an MDN is therefore not, by itself, a substantive contribution. Moreover, Corollary 3.2 shows that the gradient-based EM reformulation alone produces the same gradient as direct NLL training. The potentially new element is the specific output-space preconditioner. I do not find its tractability, together with the current evidence, sufficient to establish a substantial methodological advance beyond combining familiar ingredients. The presentation does not sufficiently clarify the boundary between classical GMM estimation, gradient-based MDN training, and genuinely new results.

The empirical motivation does not adequately compensate for this limited conceptual contribution. The striking failure in Figure 1 concerns a simple two-component GMM, but the reported setup uses SGD with batch size one and does not establish whether the failure persists under stronger conventional optimization procedures. Furthermore, the regression RMSE uses target-dependent component selection, making its improvements difficult to interpret as improvements in ordinary predictive performance. Consequently, neither the necessity of the proposed intervention nor the breadth of its practical advantage is convincingly established.

  1. The central EM--NGD equivalence is not justified by the provided proof.

In Appendix~B.3, the transition from Eq (34) to Eq (35) identifies an averaged M-step Hessian with the model-based complete-data Fisher. However, averaging the posterior-weighted Hessian over an arbitrary distribution q(x)q(x) produces an expectation under $ q(x)p_{\theta_t}(z\mid x), $ not under pθt(x,z)p_{\theta_t}(x,z). The identification therefore requires additional conditions that are not stated. Interpreting the averaging distribution as pθt(x)p_{\theta_t}(x) does not resolve the derivation: the marginal-score expectation then also vanishes, and the averaged equation does not yield the claimed nontrivial update for the observed data.

This is a separate issue from the stated second-order Taylor approximation: the proof changes the curvature matrix without establishing that the replacement is valid. Since Theorem 3.3 provides the central theoretical justification connecting EM to the proposed preconditioning, this is a material soundness concern rather than a minor notational error. Without a corrected derivation and a precise statement of its scope, I do not consider the theoretical foundation adequate for acceptance.

Questions for authors
  1. Theorem 3.3: clarification and correction.

What precisely is the averaging distribution p(x)p(x) in Eq (35)? Please provide a corrected derivation explaining when the averaged M-step Hessian equals the model-based complete-data Fisher, and how this yields Eq (36). Is Theorem 3.3 valid under its current generality, or does it require additional assumptions, a restricted parameterization, or a separate Fisher approximation?

  1. Scope of the contribution and the actual neural update: clarification.

Which results constitute new methodological insights, rather than specializations of the cited EM and natural-gradient formulations? In particular, the paper explicitly applies Fisher preconditioning in output space rather than network-parameter space. Which properties of EM or natural-gradient descent actually carry over to the backpropagated update in Eq (12)? Is the EM--NGD connection a guarantee for the proposed algorithm, or only motivation for choosing its preconditioner? This distinction is important for assessing the claimed contribution.

  1. Figure 1 and the NLL baseline: experimental clarification and supporting evidence.

For the Two-Gaussians experiment, please specify the data-generating parameters, mixture initialization, covariance parameterization and constraints, and whether both methods start from identical initial parameters. Does the reported failure persist with appropriately tuned NLL optimization, larger batches or full-batch optimization, and standard EM? NLL is an appropriate conventional MDN objective; my concern is whether the particular baseline configuration makes its failure appear more general than the evidence supports.

  1. Meaning of mode collapse: clarification of the central claim

How does the paper distinguish harmful failure to recover target modes from legitimate redundancy in an over-specified mixture? For example, a single-Gaussian target can be represented correctly by a K=2K=2 model with coincident components or an unused component. Conversely, identical components can have high mixture-weight entropy without representing distinct modes. What evidence establishes that nGEM improves density recovery, rather than merely increasing component utilization, and how does the proposed explanation account for identifiability and over-specification?

  1. Regression RMSE: clarification and re-evaluation

Is the minimum in $ \min_k \mathrm{RMSE}(\mu_k,y) $ taken separately for each evaluation sample, or after computing each component's dataset-level RMSE? Please provide the exact evaluation formula or code. This metric directly uses only the component means and selects components using the targets; it does not evaluate the point prediction implied by the fitted mixture weights. Do the improvements persist when reporting the RMSE of $ \sum_{k=1}^{K} \pi_k(x)\mu_k(x)? $ NLL already evaluates the full predictive distribution, so the concern here is the comparability and interpretation of the additional regression-accuracy claims.

  1. Consistency of the empirical claims: clarification

Section 4.4 states that nGEM improves RMSE on all tasks when using the same optimizer. However, for K-FAC on kin8nm, Table~3 reports worse NLL and worse RMSE with nGEM. How do the authors explain this degradation, and were learning rates and other relevant optimizer settings tuned separately for each method? Please reconcile the textual claims with these results and clarify the conditions under which the proposed method provides a reliable improvement.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the constructive feedback.

Please provide a corrected derivation explaining when the averaged M-step Hessian equals the model-based complete-data Fisher ...

Is Theorem 3.3 valid under its current generality, or does it require additional assumptions ... ?

We are very grateful to the reviewer for flagging this critical error. We want to acknowledge that all other reviewers have raised the same concern regarding the correctness of the proof for Theorem 3.3, in particular Equation 35-36 in Appendix B.3.

We have updated our proof for Theorem 3.3 in Appendix B.3, with an additional assumption that the complete-data probabilistic model p(x,z∣θ)p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\theta}) belongs to the exponential family (EF) and can be written as p(x,z∣η)=h(x,z)exp⁡η⊤T(x,z)−A(η),p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\eta}) = h(\boldsymbol{x}, \boldsymbol{z}) \exp\\{\boldsymbol{\eta}^\top T(\boldsymbol{x}, \boldsymbol{z}) - A(\boldsymbol{\eta})\\}, where η\boldsymbol{\eta} is the natural parameters and A(η)A(\boldsymbol{\eta}) is the log-partition function. With the exponential-family assumption, the Hessian ∇η2log⁡p(x,z∣η)=∇η2A(η)\nabla^2_{\boldsymbol{\eta}} \log p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\eta}) = \nabla^2_{\boldsymbol{\eta}} A(\boldsymbol{\eta}) is independent of x\boldsymbol{x} and remains unchanged under the expectation w.r.t. either p(x)p(\boldsymbol{x}) or p(x∣θ)p(\boldsymbol{x}|\boldsymbol{\theta}).

This exponential-family assumption easily holds for Gaussian mixtures, and is generally true for finite mixtures of EFs. We have also proved for other parametrization θ=f(η)\boldsymbol{\theta} = f(\boldsymbol{\eta}) where ff is a smooth and invertible change of coordinates. Please kindly refer to the updated manuscript (Appendix B.3) for detailed derivations.

Regarding the reviewer's concern, the empirical distribution p(x)p(\boldsymbol{x}) and model distribution p(x∣θ)p(\boldsymbol{x}|\boldsymbol{\theta}) are indeed not interchangeable in general, and our previous derivation is hence flawed. We have imposed an additional exponential-family assumption on p(x,z∣θ)p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\theta}) and updated the proof accordingly. Please let us know if further concerns arise.

Which properties of EM or natural-gradient descent actually carry over to the backpropagated update ... ? Is the EM-NGD connection a guarantee for the proposed algorithm, or only motivation for choosing its preconditioner?

We have briefly clarified in Section 5 that our method computes the natural gradient in the network output ϕ\boldsymbol{\phi}-space, not in the network weight θ\boldsymbol{\theta}-space. The EM-NGD equivalance (up to aforementioned assumptions) only holds in the ϕ\boldsymbol{\phi}-space, and does not automatically carry over to the back-propagated update.

Previously we did not expand on the difference between these two approaches due to space constraint. Now consider a neural network ϕ=f(x;θ)\boldsymbol{\phi} = f(\boldsymbol{x}; \boldsymbol{\theta}) with parameter Jacobians J=∂ϕ/∂θJ = \partial\boldsymbol{\phi} / \partial\boldsymbol{\theta}. Let g_\boldsymbol{\phi} = \nabla_\boldsymbol{\phi} L and g_\boldsymbol{\theta} = \nabla_\boldsymbol{\theta} L where LL is the loss function.

  1. Our approach computes the natural gradient w.r.t. ϕ\boldsymbol{\phi} and back-propagate to θ\boldsymbol{\theta} via Jacobians JJ, producing the following parameter update: \Delta_\boldsymbol{\theta}^{(1)} = J^\top F(\boldsymbol{\phi})^{-1} g_\boldsymbol{\phi}.
  2. The standard natural gradient w.r.t. θ\boldsymbol{\theta} requires computing the FIM F(θ)F(\boldsymbol{\theta}), which is often intractable, and produces the following parameter update: \Delta_\boldsymbol{\theta}^{(2)} = F(\boldsymbol{\theta})^{-1} g_\boldsymbol{\theta} = (J^\top F(\boldsymbol{\phi}) J)^{-1} (J^\top g_\boldsymbol{\phi}).

In general, we have \Delta_\boldsymbol{\theta}^{(1)} \neq \Delta_\boldsymbol{\theta}^{(2)}. A sufficient condition for equality is JJ⊤=IJJ^\top = I, which is generally not true for arbitrary networks.

In conclusion, Theorem 3.3 mainly serves as a motivating heuristics rather than exact theoretical characterization for the nGEM algorithm. We made this design choice to trade off theoretical exactness for practical tractability.

We are happy to include this justification in the updated manuscript if the reviewer found it reasonable.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

For the Two-Gaussians experiment, please specify the data-generating parameters, mixture initialization, covariance ...

The data-generating distribution is p(x)=0.5N(μ⋆,Σ⋆)+0.5N(−μ⋆,Σ⋆)p(\boldsymbol{x}) = 0.5\mathcal{N}(\boldsymbol{\mu}^\star, \boldsymbol{\Sigma}^\star) + 0.5\mathcal{N}(-\boldsymbol{\mu}^\star, \boldsymbol{\Sigma}^\star), where μ⋆=[2,2]⊤\boldsymbol{\mu}^\star = [2, 2]^\top and Σ⋆=12I\boldsymbol{\Sigma}^\star = \frac12I. Parameters are chosen such that two Gaussians are well-separated.

Given a fixed random seed, the synthetic dataset and the initial network parameters are exactly identical regardless of the objective (NLL / nGEM) and other hyperparameters (e.g., batch size, learning rate).

Does the reported failure persist with appropriately tuned NLL optimization, larger batches or full-batch optimization, and standard EM?

Figure 1(a) is a deliberately chosen failure example for illustration. For Two-Gaussians experiments, such catastrophic failure does not persist across all seeds, but still NLL shows consistently slower convergence and more mode collapse (Figure 1(c), Table 1).

Standard EM is generally free of mode collapse, unless the centroids are intialized closely to each other. However, if one minimizes NLL using SGD, the illustrated failure mode can be observed across different learning rates (Figure 3) and batch sizes (and full-batch descent).

How does the paper distinguish harmful failure ... from legitimate redundancy in an over-specified mixture?

To answer this question, we have conducted additional experiments. We generate a synthetic linear regression dataset y=Ax+b+ϵy = Ax + b + \epsilon where ϵ∼N(0,I)\epsilon \sim \mathcal{N}(0, I), and optimize MDNs with K=100K=100 components to fit the data. The data is perfectly unimodal, and the mixture model is highly over-specified. Training till convergence leads to one of the followings:

  1. Collapsed weights: ∃k πk≈1\exists k~\pi_k \approx 1. One component takes all the weights πk\pi_k, and the entropy H(π)H(\pi) is near zero.
  2. Collapsed components: ∀k μk(x)≈Ax+b\forall k~\mu_k(x) \approx Ax+b. All components converge to the ground-truth Gaussian, and the weights are almost equal πk≈1/K\pi_k \approx 1/K.

Our of 10 random seeds, we have 7 of the first scenario and 3 of the second scenario, with an average entropy H(π)H(\pi) of 0.11±0.010.11 \pm 0.01 and 4.39±0.214.39 \pm 0.21 respectively. In conclusion, we have observed no empirical evidence that nGEM will prevent mode collapse when collapsing is actually the correct behavior.

Is the minimum in min⁡kRMSE(μk,y)\min_k \mathrm{RMSE}(\boldsymbol{\mu}_k,\boldsymbol{y}) taken separately for each evaluation sample, or ... ?

Yes, the minimum is taken for each evaluation sample (x,y)(\boldsymbol{x},\boldsymbol{y}) individually. The dataset-level RMSE is the average of the best RMSE for all evaluation samples.

Do the improvements persist when reporting the RMSE of ∑k=1Kπk(x)μk(x)\sum_{k=1}^{K} \boldsymbol{\pi}_k(\boldsymbol{x})\boldsymbol{\mu}_k(\boldsymbol{x})?

We understand the confusion around using best-of-all RMSE. The rationale behind the choice are as follows:

  1. In many cases (e.g., inverse modeling), it is often desirable if one of the candidates is sufficiently accurate when the model generates multiple candidates.
  2. As the reviewer noted, NLL already "evaluates the full predictive distribution" and represent the average-case performance. We therefore also reports the minimum RMSE which represents the best-case performance

Nevertheless, we have re-computed the weighted-mean RMSE RMSE(∑kπkμk,y)\mathrm{RMSE}(\sum_k \boldsymbol{\pi}_k\boldsymbol{\mu}_k, \boldsymbol{y}), and the resulting metrics is generally consistent with NLL and do not contradict our current conclusion. We will attach the full results in the appendix when updating the manuscript.

For K-FAC on kin8nm, Table 3 reports worse NLL and worse RMSE with nGEM ... were learning rates and other relevant optimizer settings tuned separately for each method?

K-FAC indeed does not perform as well with nGEM. A potential reason is that (we observed during experiments) the matrices to be inverted is often more ill-conditioned with K-FAC and nGEM. We have already faithfully acknowledged that "their integration yields additive performance gain ... in some cases" in Section 4.4.

In Table 12, batch sizes are tuned for each dataset with multiple seeds. Learning rates are only tuned for each combination of dataset / optimizer using one seed due to the multiplicative costs. Other optimizer settings (e.g., weight decay) are all kept default.

Thank you for the additional experiments and clarifications. I appreciate the authors’ efforts to address the concerns raised in my review. However, several important concerns remain.

Theoretical justification. The proposed correction to Theorem 3.3 relies on an additional exponential-family assumption, which narrows the scope of the original claim. Even accepting the revised derivation, I find its role in justifying the proposed algorithm considerably more limited than the original presentation suggested. More importantly, the authors now acknowledge that the EM–NGD connection serves primarily as motivation and does not automatically extend to the actual neural-network update.

Experimental comparisons. Demonstrating reduced mode collapse relative to plain NLL optimization is not sufficient to establish a substantial methodological advance. I remain unconvinced that the particular NLL optimization setup used in the motivating experiments provides a sufficiently strong baseline. Although the response clarifies the experimental settings, it does not provide sufficient comparative evidence against more competitive, appropriately tuned optimization procedures or existing methods specifically designed to mitigate component collapse. This limitation makes it difficult to assess the practical advantage of nGEM.

Methodological novelty. This remains my main concern. EM is a standard approach to fitting Gaussian mixtures, and natural-gradient methods are well established. With the theoretical claims now qualified, I still view the core contribution as adapting these familiar ingredients into a tractable output-space preconditioner for MDNs. The response does not clearly explain what substantive methodological insight this adaptation provides beyond the established EM and natural-gradient formulations. While computational efficiency and ease of implementation are useful practical strengths, I do not find these benefits, together with the current theoretical and empirical evidence, sufficient to address my concern about the limited novelty.

For these reasons, I will maintain my original rating.

Reviewer_4LCqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Natural Gradient Expectation Maximization (nGEM), a method for training Mixture Density Networks (MDNs) using natural gradients with respect to the parameters of the Gaussian mixture. The authors show a connection between minimizing the negative log-likelihood (NLL) and generalized Expectation Maximization (EM). In particular, they show that the gradient of the NLL is equal to the gradient of the negative M-step objective of EM at the current parameters, and use this connection to motivate their method. Furthermore, the authors establish a connection between EM and natural gradient descent and derive the block-diagonal complete-data Fisher information matrix for Gaussian mixtures, which allows for decoupled preconditioning of each mixture component. The algorithm is then evaluated on multiple benchmarks, with results suggesting faster convergence, reduced Gaussian component collapse, and little computational overhead compared to standard NLL training.

Critical strengths
  • Natural gradient descent is often computationally expensive because it requires inverting Fisher information matrices. Applying it directly to the parameters of a neural network would be particularly costly. By instead applying natural gradient preconditioning to the output parameters of the network, corresponding to the Gaussian mixture parameters, the problem becomes much more tractable. Furthermore, by using the complete-data Fisher information matrix, the authors obtain a block-diagonal structure, which makes the inversion considerably easier and avoids substantial computational overhead.

  • Although the synthetic experiments are relatively simple, they convincingly demonstrate both faster convergence and improvements in preventing Gaussian component collapse.

  • The method is relatively straightforward to implement and could potentially benefit several areas. Since Gaussian mixture models are widely used, particularly in reinforcement learning, algorithms that accelerate convergence and prevent mode collapse could be useful across multiple applications

Critical weaknesses
  • The authors claim that exact EM is equivalent to natural gradient descent up to a second-order approximation. However, in the derivation of Theorem 3.3, the expectations of the Hessian of the complete-data log-likelihood are taken under different distributions. In particular, the Hessian of the EM objective is averaged over the empirical data distribution, whereas the complete-data Fisher information matrix is defined using an expectation under the joint model distribution. These two quantities are not generally equal. Although I do not think this invalidates the proposed optimization method, I believe the connection between EM and natural gradient descent is not as direct as the authors claim.

  • While the synthetic experiments show substantial improvements, these improvements do not seem to transfer as strongly to more realistic tasks. For example, in simulation-based inference, the results are very close. On the Hyperboloid benchmark, the proposed method achieves an MMD of 2.86 ± 0.30, compared to 2.90 ± 0.29 for NLL. Similar observations can be made in the reinforcement learning experiments, where the improvements are considerably smaller than in the synthetic tasks. This does not invalidate the experimental results, but it weakens the claim of broad superiority over standard NLL training.

Questions for authors
  • How do the authors justify replacing the empirical expectation of the complete-data log-likelihood Hessian with an expectation under the joint model distribution in Equation 35? Under which assumptions are these two quantities equivalent?

  • The ablation results suggest that inverse mixture-weight scaling plays an important role in preventing mode collapse. How much of the improvement achieved by nGEM could be reproduced by simply applying inverse mixture-weight scaling to the standard NLL gradients, without the remaining natural gradient preconditioning?

2 comments

We thank the reviewer for the constructive feedback.

How do the authors justify replacing the empirical expectation of the complete-data log-likelihood Hessian with an expectation under the joint model distribution in Equation 35? Under which assumptions are these two quantities equivalent?

We are very grateful to the reviewer for flagging this critical error. We want to acknowledge that all other reviewers have raised the same concern regarding the correctness of the proof for Theorem 3.3, in particular Equation 35-36 in Appendix B.3.

We have updated our proof for Theorem 3.3 in Appendix B.3, with an additional assumption that the complete-data probabilistic model p(x,z∣θ)p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\theta}) belongs to the exponential family (EF) and can be written as p(x,z∣η)=h(x,z)exp⁡η⊤T(x,z)−A(η),p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\eta}) = h(\boldsymbol{x}, \boldsymbol{z}) \exp\\{\boldsymbol{\eta}^\top T(\boldsymbol{x}, \boldsymbol{z}) - A(\boldsymbol{\eta})\\}, where η\boldsymbol{\eta} is the natural parameters and A(η)A(\boldsymbol{\eta}) is the log-partition function.

With the exponential-family assumption and under natural parametrization η\boldsymbol{\eta}, the Hessian ∇η2log⁡p(x,z∣η)=∇η2A(η)\nabla^2_{\boldsymbol{\eta}} \log p(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{\eta}) = \nabla^2_{\boldsymbol{\eta}} A(\boldsymbol{\eta}) is independent of x\boldsymbol{x} and remains unchanged under the expectation w.r.t. either p(x)p(\boldsymbol{x}) or p(x∣θ)p(\boldsymbol{x}|\boldsymbol{\theta}).

This exponential-family assumption easily holds for Gaussian mixtures, and is generally true for finite mixtures of EFs. We have also proved for other parametrization θ=f(η)\boldsymbol{\theta} = f(\boldsymbol{\eta}) where ff is a smooth and invertible change of coordinates. Please kindly refer to the updated manuscript (Appendix B.3) for detailed derivations.

We would like to finally clarify that this erroneous derivation does not invalidate our original claim in Theorem 3.3. Regarding the reviewer's concern, the empirical distribution p(x)p(\boldsymbol{x}) and model distribution p(x∣θ)p(\boldsymbol{x}|\boldsymbol{\theta}) are indeed not equal in general, and our previous derivation is hence flawed.

How much of the improvement achieved by nGEM could be reproduced by simply applying inverse mixture-weight scaling to the standard NLL gradients, without the remaining natural gradient preconditioning?

Please kindly refer to Table 5 in Appendix D.1. With only the inverse weighting πk−1\boldsymbol{\pi}_k^{-1}, nGEM can still prevent mode collapse but convergence could be slower under the same computation budget.

Intuitively, consider the gradient of NLL w.r.t. the the mean parameter μk\boldsymbol{\mu}_k, denoted as ∇∝(x−μk)/σk2\nabla \propto (\boldsymbol{x}-\boldsymbol{\mu}_k) / \boldsymbol{\sigma}_k^2. The gradient ∇\nabla could induce unstable updates depending on the scale of σk2\boldsymbol{\sigma}_k^2.

  1. Large σk2\boldsymbol{\sigma}_k^2 is equivalent to a small step size, leading to slow convergence.
  2. Small σk2\boldsymbol{\sigma}_k^2 is equivalent to a large step size, leading to training instability in neural networks.

With only the Gaussian FIM Fk−1F_k^{-1} preconditioning, the natural gradient update becomes ∇^∝(x−μk)\hat{\nabla} \propto (\boldsymbol{x}-\boldsymbol{\mu}_k), which moves μk\boldsymbol{\mu}_k towards the target x\boldsymbol{x} stably and consistently.

I thank the authors for the response. I believe Theorem 3.3 is now correct, and Table 5 answers my other question, but my concern about the modest gains on realistic tasks remains, so I retain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.