- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper explores whether switching between Wasserstein and Fisher–Rao geometry is useful for Gaussian and Gaussian-mixture-based variational inference with gradient flows. The paper's main contribution is an algorithm for inference with Gaussian mixtures. The scheme selects the per-component geometry adaptively. The approach is mainly analytical, giving exponential convergence rates for their splitting scheme. The method is also validated empirically on a high-dimensional mixture, density approximation for a normalizing flow, and inverse problems, showing superior performance in terms of ELBO and total variation over three baselines.
Critical strengths
The derivation of a switching rule from Wasserstein to Fisher–Rao in the Gaussian case is well motivated by the analysis in Theorem 3.1, which covers the regimes where the mean is not recovered and the covariance is incorrectly scaled. Carrying this to the mixture case, they show that switching still minimizes the KL even with component-wise geometries, which was surprising. The analysis is mostly convincing (see weaknesses), and the empirical results support that their method is competitive in terms of ELBO and total variation. Altogether, I lean towards acceptance.
Critical weaknesses
Theorem 3.1 does not cover the non-scaled regime for the covariance, that is, for any . Without this case, the analysis of Theorem 3.1 seems partial (see question below), because we rarely know the covariance a priori. With theory covering this regime, or a convincing argument for why it is not relevant, I would move this paper from a weak accept to a clear accept.
Questions for authors
- How is the higher MC variance accounted for when using Stein's identity to avoid Hessian computations?
- Does the covariance recovery (Theorem 3.1.1) hold for mixture components? If not, can you say anything about the covariance?
- The practical switching rule for Gaussian mixtures is intuitive in the Gaussian setting, but the mixture entropy couples the components. Is there a stronger motivation for why a one-time switch is a good choice? Or do you expect other schemes could be better (either in theory or in practice)?
- Can you say anything about recovery in the regime where for any , in either the Gaussian or the Gaussian mixture case? If not, is there an argument for why this regime can be safely ignored?
3 comments
Response to Reviewer fTa7 — Part 2/2
Covariance recovery for mixture components. We thank the reviewer for this question, which led us to extend Section 3 to Gaussian mixtures (Proposition 4.5 in the revision). For every component taking a Wasserstein step of size , the projected forward–backward covariance satisfies regardless of the overlap among components, where are the covariance bounds of the projection. For a Fisher–Rao step, the effect of the mixture entropy is controlled by , where is the component responsibility defined in Section 4. If , then Therefore, Wasserstein provides an immediate covariance floor, whereas Fisher–Rao cannot enlarge an arbitrarily small eigenvalue by more than a bounded factor per step when the weight is bounded away from zero. This result requires only mild additional assumptions, and further guarantees beyond Theorem 4.3 are an interesting direction for future work.
Motivation for a one-way switch. In the setting of Theorem 4.4 (single-Gaussian VI with a smooth, strongly log-concave, radial target), the one-way structure is a consequence of the analysis rather than an extra restriction. For mixtures, we do not prove this optimality; Proposition 4.5 shows that the reason to start with Wasserstein persists. Once the covariance is reasonably scaled, Fisher–Rao benefits from affine invariance, which has also been observed empirically, so we design the algorithm to stay with Fisher–Rao after switching. Other adaptive schemes may help in some cases, and Theorem 4.3 guarantees descent and stationarity for any admissible scheme.
We hope these responses address the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating the score, and we would be glad to clarify anything further. Thank you again for your time and thoughtful feedback.
Response to rebutal
I thank the authors for their clarifications. I appreciate the addition of Proposition 4.5, which strengthens the case for switching by showing that the Wasserstein update recovers from collapse in a single step, whereas the Fisher–Rao update may require multiple steps. However, I share the concern raised by Reviewer y81C that the paper neither establishes the optimality of the switching rule nor compares it empirically against alternative approaches. I will therefore maintain my current score.
Response to Reviewer fTa7 — Part 1/2
We thank the reviewer for the positive assessment, for identifying general initial covariances as an important theoretical gap, and for the constructive questions. The revision addresses this gap with new results, as described below.
General initial covariances. The restriction C0=εC⋆ in the original Theorem 3.1 was made to isolate the two phases cleanly. The revised manuscript removes it: Theorem B.1 and the new Proposition B.2 allow arbitrary C0≻0. For the Gaussian target N(m⋆,C⋆), let μ=μmin(C⋆) and c0=μmin(C0)<μ/2. Then
(C_t^{\mathrm{FR}})^{-1}=(1-e^{-t}) C_\star^{-1}+e^{-t} C_0^{-1}.$$ Consequently, the first times to reach the covariance floor $\mu I/2$ satisfy $$T_{\mathrm W}\le\tfrac12\log2,\qquad T_{\mathrm{FR}}\ge\log\Bigl(\frac{\mu}{2c_0}\Bigr).$$ The Wasserstein time is thus independent of the initial eigenvectors and of the smallest initial eigenvalue, whereas Fisher–Rao incurs a delay logarithmic in $1/c_0$. Proposition B.3 proves the discrete analogue for general smooth, log-concave targets: one Wasserstein forward–backward step with step size $h$ gives $C_1\succeq hI$, while pure Fisher–Rao needs logarithmically many fixed-size steps as $\beta c_0\to0$. These results cover the initial-covariance regime identified by the reviewer. We also have new sharp, affine-invariant convergence results for the Fisher–Rao gradient flow in Gaussian VI with general, non-Gaussian targets, which are being prepared as a separate paper. If the reviewer finds it valuable, we would be happy to include these techniques in this paper to extend the analysis beyond Gaussian targets. **Monte Carlo variance of the Stein estimator.** Estimators based on Stein's identity can indeed have higher Monte Carlo variance; this is the price of avoiding Hessian evaluations. To reduce unnecessary noise, we subtract the sample mean of the scores, which removes constant-score contributions while preserving unbiasedness (Appendix E). In our experiments, the method remains stable with moderate per-component sample sizes, which suggests that it is robust to Monte Carlo noise at the reported sampling budgets.