acceptodds
Under review as a conference paper at ICLR 2027

Fixed-point acceleration on Neural Wasserstein manifolds

Abstract

We propose a fixed-point acceleration method for optimization over probability measures, applicable to a variety of machine learning problems. We represent probability measures as pushforwards of a reference distribution through a neural network map and equip the parameter space with the induced pullback Wasserstein metric. This geometric structure allows natural gradient descent to be interpreted as a fixed-point iteration on a Riemannian manifold. We accelerate this iteration using Riemannian Anderson mixing method, which exploits historical residuals to capture approximate second-order information without forming Hessians. In the present parametric setting, the manifold is represented by Euclidean coordinates endowed with a point-dependent metric, simplifying retraction and vector transport. We evaluate the method on multimodal sampling, PDE-based inverse problems, aggregation dynamics, Bayesian logistic regression, and generative modeling. Across the tested tasks, Anderson mixing reduces the number of natural-gradient iterations and achieves competitive performance relative to SGD and Adam in terms of both loss-gradient evaluations and wall-clock time. These results suggest that fixed-point acceleration is an efficient tool for improving the computational efficiency of training on parametric Wasserstein methods

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
natural gradient, anderson acceleration, gradient flow in dennsity space

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3PXzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a fixed-point acceleration method for optimization over probability measures. Distributions are represented as pushforwards of a reference distribution through a neural network, and the parameter space is equipped with an induced pullback Wasserstein metric. Natural-gradient descent is formulated as a fixed-point iteration and accelerated using Riemannian Anderson mixing, which exploits historical residuals without explicitly forming Hessians. The parameter-space formulation simplifies retraction and vector transport. Experiments cover multimodal sampling, PDE-based inverse problems, aggregation dynamics, Bayesian logistic regression, and generative modeling. The reported results show fewer natural-gradient iterations and competitive performance relative to SGD and Adam in loss-gradient evaluations and wall-clock time.

Critical strengths

In the PDE-based inverse problem, RAM achieves acceleration with little additional overhead relative to NGD. Their iteration rates differ by approximately 2%, and the best-performing RAM configuration is 3.6 times faster than the best Adam configuration in wall-clock convergence time (Table 1).

The generative-modeling experiments demonstrate that acceleration can be achieved without over-relaxation β=1\beta = 1. Table 2 reports fewer iterations to reach the specified MMD thresholds on all three distributions: checkerboard, two spirals, and eight Gaussians.

Critical weaknesses

The convergence analysis needs to specify its valid relaxation range. Proposition 3.2 gives a residual estimate with the factor 1−βk+κβk1-\beta_k+\kappa\beta_k. However, the cited RAM proof explicitly uses βk≤1\beta_k\le1, whereas Algorithm 1 permits any βk>0\beta_k>0, and the experiments include β=3\beta=3. The estimate therefore cannot be directly extended to all allowed over-relaxed settings. The authors should state the required restriction on βk\beta_k and distinguish the configurations covered by the analysis from those supported empirically. If the guarantee is intended to cover over-relaxation, the relevant estimates require additional justification.

The empirical evaluation needs stronger ablations and sensitivity analysis. Comparisons with NGD, SGD, and Adam demonstrate the effectiveness of the proposed RAM implementation, but do not isolate the additional benefit of GkG_k-norm mixing over Euclidean Anderson mixing using the same natural-gradient residual definition.

Questions for authors

What relaxation range is required for Proposition 3.2? The cited proof uses βk≤1\beta_k\le1, whereas Algorithm 1 permits any βk>0\beta_k>0. Please distinguish the experimental configurations covered by the analysis from those supported empirically. If the guarantee extends to over-relaxation, how are the proof steps relying on βk≤1\beta_k\le1 modified?

How does performance change when the GkG_k-norm in the mixing least-squares problem is replaced with the Euclidean norm, while keeping the natural-gradient residual definition, step size, and history length unchanged and using comparable tuning budgets? Could this comparison help isolate the additional benefit of the point-dependent metric in the mixing step?

2 comments

We would like to thank the reviewer for the constructive criticism and pointing out the paper's strengths, especially the acceleration for the Bayesian inverse problem. In response to the weaknesses you have mentioned.

Distinction between over-relaxed (βk>1\beta_k > 1) and standard (βk∈[0,1]\beta_k \in [0, 1]) settings

We thank the reviewer for identifying this missing restriction. We agree that the cited proof underlying Proposition 3.2 requires 0<βk≤10<\beta_k\leq1. We have explicitly stated this condition in the main convergence statement (proposition 3.2) and revised Remark 3.1 to distinguish the range covered by the analysis.

Nevertheless, we believe that the over-relaxation technique is worth exploring numerically. It has been explored in the relevant literature on AM. Potra and Engler [1] analyze Anderson acceleration for linear problems with arbitrary nonzero mixing parameters, including values greater than one. In nonlinear applications, Storvik et al. [2] demonstrate the usefulness of combining Anderson acceleration with over-relaxation in phase-field fracture simulations, recommending a relaxation parameter of 1.6 based on their experiments. We have added the corresponding discussion in Remark 4.1

We would also like to note that over-relaxation is just one of the exploratory directions in the experiments, and is shown not to be necessary for the acceleration. For example, for the Inverse problem and MMD generative modeling, the noticeable acceleration is achieved solely for β=1\beta = 1.

Isolation of the effect of Riemannian scalar product

We thank the reviewer for the thoughtful suggestion.

We have performed an additional experiment according to your suggestion. The results can be found in Appendix C.3.

We focus on the problem of sampling from the Styblinsky-Tang distribution in d=50d=50. This is one of the hardest problems in the paper, due to the high dimension of the distribution, multimodal posterior and large number of iterations needed for convergence. The usage of natural gradient iteration is strongly justified in this case as it allows to avoid mode collapse, to which the methods, based on the Euclidean gradient (SGD, Adam) were prone.

We fix the hyperparameters of the model, the underlying natural gradient operator and the number of history vectors in RAM. We use adaptive regularization and only vary the regularization factor λ\lambda. The result is given on Figure 10. In the table below, we show the minimal amount of iterations, needed to reach three threshold of the value if target functional, by both Riemannian and Euclidean AM with 3 best values of the regularization factor.

method setupE=−726.0\mathcal{E} = -726.0E=−1104.0\mathcal{E} = -1104.0E=−1482.0\mathcal{E} = -1482.0E=−1860.0\mathcal{E} = -1860.0
riemannian, λ=10−7\lambda = 10^{-7}21974411849509
riemannian, λ=10−6\lambda = 10^{-6}225730115611840
riemannian, λ=10−5\lambda = 10^{-5}231743116011293
euclidean, λ=10−7\lambda = 10^{-7}6371053137018910
euclidean, λ=10−6\lambda = 10^{-6}16763766535524940
euclidean, λ=10−5\lambda = 10^{-5}63310481367>25 000

One can clearly see that the amount of iterations is smaller for RAM on all levels. The best performing RAM requires approximately 22 times less iterations than the best Euclidean AM to reach the smallest loss threshold. Moreover, from Figure 10, RAM is more stable and its performance of depends less on the particular value of the regularization factor than in the Euclidean case, meaning that the adaptive regularization is more efficient if computed in the right metric. We believe this experiment clearly isolates the benefit of using the Riemannian scalar product.

[1] Potra, F. A., and Engler, H. (2013). A characterization of the behavior of the Anderson acceleration on linear problems. Linear Algebra and its Applications, 438, 1002–1011.

[2] Storvik, E., Both, J. W., Sargado, J. M., Nordbotten, J. M., and Radu, F. A. (2021). An accelerated staggered scheme for variational phase-field models of brittle fracture. Computer Methods in Applied Mechanics and Engineering, 381, 113822.

Reviewer_3PXzOpenReview ↗

Thanks for replying

Thanks for replying, I will maintain my score.

Reviewer_1pVMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes an alternative approach to using natural gradient descent (NGD) on the Wasserstein manifold. This essentially consists of accelerating NGD by Anderson acceleration, incorporating previous iterations with some particular mixing coefficients computed using an inner optimization subroutine. Some basic convergence analysis connected to existing work is given under abstract assumptions on the operators and the approximation error of the parameterizations. The method is then applied to train flow operators from a Gaussian to some target distributions, and is competitive/outperforming standard Adam/SGD optimizers as well as the unaccelerated NGD method.

Critical strengths
  • Significant acceleration is achieved across different experiments.
Critical weaknesses
  • Optimizing KL in this case seems quite arbitrary, and the surrounding theory does not have any direct relation to probability measures. Plus, computing this NGD is quite expensive as seen in Appendix C1. Is this more stable compared to e.g. normalizing flows? Wouldn't this also apply to more general spaces that have some finite dimensional approximate parameterization? If the point of the work is to apply this acceleration to Wasserstein space, this has to be made clearer
  • The discussion on actually adding Anderson acceleration is a bit confusing. Why can one directly use the the mappings in Proposition 3.1? Once the metric is removed, would this not reduce directly to standard Anderson acceleration? Is it simply because one wants a descent direction?
Questions for authors
  • Prop. 3.2: what is "the operator" that is locally contractive? Is it Φh\Phi_h?
  • Why is RAM faster than NGD in Table 1?
  • Figures need work: e.g. figure 6 has no y-labels, figure 5b appears to be missing something in the sentence, figure 4 is essentially unreadable (why do some lines disappear at different iterations, and should this be the relative entropy instead of just entropy?)
  • (Suggestion) Would be interesting to see what the flow maps are at intermediate times.
2 comments
AuthorsOpenReview ↗

We thank the reviewer for your constructive criticism and for recognizing the strengths of the paper such as the acceleration achieved across the experiments.

Choice of KL and connection to optimization over probability measures.

KL is a standard objective for posterior approximation: for a target density ρ∗∝exp⁡(−V)\rho_* \propto \exp(-V), minimizing KL(ρθ∣ρ∗)\mathrm{KL}(\rho_\theta \mid \rho_*) does not require knowing the normalization constant, as detailed in Appendix C.1. Note that the method is not limited to KL divergence, but applies to any tractable objective functional, which is demonstrated by the aggregation-energy and MMD minimization experiments.

The connection between the theory and probability measures is given explicitly in Proposition 2.1, proved in Appendix B.3. Under the gradient-projection error assumption,

\leq \frac{\\|r_h(\theta)\\|_\theta^2}{h^2}+\delta_0^2.$$ Thus, the parametric fixed-point residual controls the stationarity up to the error in approximating the Wasserstein gradient by the parametric tangent space. Under the Assumption 2.2, Eq. (2.2.5) further gives $$\mathcal E(\rho\theta)-\mathcal E(\rho_*) \leq \frac{1}{\mu} \left( \frac{\\|r_h(\theta)\\|_{\theta}^2}{h^2}+\delta_0^2 \right).$$ Assumption 2.2 is standard in theory of Wasserstein gradient flows and is a generalization of PL inequality. It can be established for various functionals over the Wasserstein space such as KL, aggregation energy and MMD. We will make this connection and its assumptions more prominent and correct the inconsistent constant in Eq. (2.2.5). # Generality of the approach and comparison with normalizing flows. The approach can be applied to different spaces, such as $L_2$ or Sobolev $W^{k,2}$, and to other neural network parameterizations, provided the Riemannian metric is adjusted. As stated in the paper's title and abstract, we focus on applying it to the Wasserstein space. Firstly, this space is relevant to a multitude of relevant ML problems, including generative modeling, Bayesian logistic regression, Bayesian inverse problems etc., see Section 4. Secondly, NGD (without acceleration) for neural parameterizations of $L_2$ or Sobolev $W^{k,2}$ was already studied in [1]. It is interesting to explore the approach in the setting where the parameterized space is more complex, e.g. not a Hilbert space. Our experiments already use neural-ODE flow maps as the parametrization, **as in continuous normalizing flows**. Normalizing flows specify the representation, whereas methods such as Adam, NGD and RAM specify how the parameters are optimized. Our numerical experiments primarily concern optimization performance for the chosen representation. We also report that on KL optimization experiments with multimodal targets our methods is not prone to mode collapse, than SGD and Adam in the same cases. # Proposition 3.1 and the role of the metric. The coordinate formulas in Proposition 3.1 do not remove the metric. On the open parameter domain, addition satisfies the retraction identities $$R_\theta(0)=\theta, \qquad DR_\theta(0)=I,$$ and identity transport is a valid vector transport under the canonical identification of the coordinate tangent spaces. Propositions B.4 and B.5 establish the additional local bounds required by the convergence analysis. We've added references to these results next to Proposition 3.1. The metric remains present in the NG residual, the least-squares mixing problem, regularization factor in case of adaptive regularization, and the descent safeguard. If the inner product is replaced by the Euclidean one, the construction becomes Euclidean Anderson mixing applied to the NGD step $$g: \mathbb{R}^D \to  \mathbb{R}^D$$ $$g(\theta):= \theta -hG(\theta)^{+}\nabla_\theta E(\theta)$$  as the operator (eq. 2.2.1), with the corresponding regularization and safeguard. # Contractive operator in Proposition 3.2. The contractivity assumption in Appendix A is for the operator $$g(\theta) := \operatorname{Exp}_\theta \bigl(-h,\operatorname{grad}_G E(\theta)\bigr).$$ # Computational cost and Table 1. RAM is slightly slower per iteration than NGD in Table 1: 1.79 versus 1.83 iterations per second. Its lower total runtime comes from requiring fewer iterations. The best reported RAM configuration requires 242 iterations and 135 seconds, compared with 4037 iterations and 2226 seconds for NGD. We acknowledge the cost of computing the natural gradient. In this experiment, loss-gradient evaluation is the main bottleneck, and the additional mixing computations have a small measured overhead, as demonstrated in Table 1 and surrounding discussion. # Figures. We agree that the figures need revision. We will improve the readability of Figure 4, clarify the plotted objective and the reasons for differing curve lengths, correct the incomplete caption in Figure 5b, and add the missing axis labels in Figure 6. # References [1] Müller, J., & Zeinhofer, M. (2023, July). Achieving high accuracy with PINNs via energy natural gradient descent. International Conference on Machine Learning PMLR.
AuthorsOpenReview ↗

Comparison to standard Anderson Mixing

The reviewer has asked

Once the metric is removed, would this not reduce directly to standard Anderson acceleration?

As noted before, even with Euclidean retraction and vector transport, the point-dependent metric still plays the role in the determination of the mixing coefficients, adaptive regularization and descent safeguard. To emphasize this distinction, we have run additional ablation experiments.

The detailed description can be found in Appendix C.3 of the updated paper.

We focus on the problem of sampling from the Styblinsky-Tang distribution in d=50d=50. We fix the hyperparameters of the model, the underlying natural gradient operator and the number of history vectors in RAM. We use adaptive regularization and only vary the regularization factor λ\lambda. The result is given on Figure 10. In the table below, we show the minimal amount of iterations, needed to reach three threshold of the value if target functional, by both Riemannian and Euclidean AM with 3 best values of the regularization factor.

method setupE=−726.0\mathcal{E} = -726.0E=−1104.0\mathcal{E} = -1104.0E=−1482.0\mathcal{E} = -1482.0E=−1860.0\mathcal{E} = -1860.0
riemannian, λ=10−7\lambda = 10^{-7}21974411849509
riemannian, λ=10−6\lambda = 10^{-6}225730115611840
riemannian, λ=10−5\lambda = 10^{-5}231743116011293
euclidean, λ=10−7\lambda = 10^{-7}6371053137018910
euclidean, λ=10−6\lambda = 10^{-6}16763766535524940
euclidean, λ=10−5\lambda = 10^{-5}63310481367>25 000

One can clearly see that the amount of iterations is smaller for RAM on all levels. The best performing RAM requires approximately 22 times less iterations than the best Euclidean AM to reach the smallest loss threshold. Moreover, from Figure 10, RAM is more stable and its performance of depends less on the particular value of the regularization factor than in the Euclidean case, meaning that the adaptive regularization is more efficient if computed in the right metric. We believe this experiment clearly show that our method does not reduce directly to standard Anderson mixing.

Reviewer_DMPjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes to adapt a Riemannian version of the classical Anderson Mixing fixed point acceleration to compute Wasserstein-stationary points of functionals defined on probability densities. The authors first reformulate the Wasserstein natural gradient descent (of pushforward-parametrized densities) as a fixed point iteration in the parameter space with the corresponding metric tensor and bound the Wasserstein stationarity in terms of stationarity in parameters (Proposition 2.1). Then they propose they apply the RAM from [1] with both ℓ2\ell_2- and adaptive regularization with the Euclidean retraction and vector transport maps. Under an assumption on the conditioning of the metric tensor (Assumption 3.1) and contractivity the work then shows a next-residual bound in terms of the previous residuals and relaxation parameters. Finally the authors provide some numerical results showing that RAM provides acceleration in certain cases.

Critical strengths

Applying techniques from Riemannian optimization to Wasserstein manifolds (which can formally be seen as Riemannian manifolds).

Critical weaknesses

The reviewer has concerns about the novelty of this work in combination with the numerical evidence/improvement for downstream applications. The core ingredients of this paper such as parametrizing pushforward distributions with neural networks (including neural ODE), viewing Wasserstein natural gradient descent as a fixed-point iteration and applying Anderson Mixing to this iteration have all appeared in the literature before (see [2], [3] (for Gaussians)) and have also been cited by this work. The combination provided in this paper seems not to introduce any technical novelty or practical insight. In particular the geometric manifold operations (retraction and vector transport) as just the Euclidean ones (independent of the metric tensor) and some assumptions are strong (e.g. contractivity in for neural parametrizations in Proposition 3.2, also admitted by the authors). For this it seems that the method is a straightforward application of RAM to Wasserstein GF with neural parametrizations. Such novelty or insight would be unnecessary/less important if the method showed strong empirical performance on downstream applications. However, the evaluation combines synthetic problems with standard Bayesian logistic-regression benchmarks. These experiments do not provide sufficient evidence of performance on larger-scale downstream applications. Therefore the reviewer cannot advocate that the paper provides a sufficient contribution for publication.

[1] Zanyu Li and Chenglong Bao. Riemannian Anderson mixing methods for minimizing C2-functions on Riemannian manifolds. arXiv preprint arXiv:2309.04091, 2023. [2] Jin et al., Parameterized Wasserstein Gradient Flow - 2024 [3] Aksenov, Eigel and Oster, Anderson Mixing in Bures Wasserstein Space of Gaussian Measures - 2026

Questions for authors

The authors should either explain and emphasize their theoretical novelty or provide higher-dimensional non-toy experiments.

1 comment

We thank the reviewer for the careful assessment. We agree that the paper should distinguish more clearly what we build on from what we contribute. Our contribution is to adapt Riemannian Anderson mixing to optimization over neural pushforward families, relate the fixed-point residual to stationarity of the underlying distributional objective, and quantify the computational benefit of reusing residuals when natural-gradient steps are expensive.

Relation to prior work.

Jin et al. [2] develop a parametrization for approximating the evolution of a Wasserstein gradient flow. We use their natural-gradient step to address a different computational question: when the goal is a stationary distribution, can the history of these steps reduce the number of gradient evaluations needed to reach it? Aksenov et al. [3] study Anderson mixing on the Bures–Wasserstein manifold, where every iterate is Gaussian. Our formulation allows each iterate to be a nonlinear pushforward of the reference measure, so the same algorithm can fit non-Gaussian distributions, including multimodal targets.

What the analysis establishes.

For a neural pushforward family, a parameter update can access only the velocity fields generated by variations of the transport map. Building on the projection framework of [2], Proposition 2.1 relates the natural-gradient fixed-point residual rh(θ)r_h(\theta) to the full Wasserstein gradient:

\leq h^{-2}\|r_h(\theta)\|_\theta^2+\delta_0^2.$$ The two terms distinguish what optimization can improve from what depends on the parametrization. Reducing the residual controls the component of the gradient accessible to the model. The projection error $\delta_0$ accounts for directions outside its tangent space. Under the additional Wasserstein PL assumption, the bound also controls the objective gap. The reviewer is correct that retraction and vector transport have simple coordinate formulas. Their practical value is that we can reuse historical directions through addition and identity transport while retaining the point-dependent metric in the natural-gradient residual and the mixing least-squares problem. Proposition 3.1 and Appendix B justify these choices under the stated assumptions. Proposition 3.2 specializes the local RAM analysis of [1] and assumes contractivity. Establishing contractivity for general neural parametrizations remains open. We will make this distinction explicit in the revision. # Practical benefit and experimental scope. The most direct comparison is with NGD on the PDE inverse problem in Table 1, where loss-gradient evaluations require PDE solves. At the same convergence threshold, NGD takes 4,037 iterations and 2,226 seconds. The best tested RAM configuration takes 242 iterations and 135 seconds. This gives a **16.5-fold speedup over NGD** despite a slightly lower iteration rate (1.79 versus 1.83 iterations per second). The wall-clock gain comes from avoiding expensive gradient evaluations. The same configuration reaches the threshold 3.6 times faster than the best tested Adam configuration. Table 2 provides complementary evidence on non-Gaussian targets. With $\beta=1$, RAM reaches the specified MMD thresholds in fewer iterations than NGD on all three distributions: checkerboard, two spirals, and eight Gaussians. We agree that larger downstream experiments would strengthen the evidence on scalability. The practical claim supported by the present results is that, on a PDE-constrained posterior inference problem, adding residual mixing to the same natural-gradient solver can substantially reduce the time required to fit the posterior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.