- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a fixed-point acceleration method for optimization over probability measures. Distributions are represented as pushforwards of a reference distribution through a neural network, and the parameter space is equipped with an induced pullback Wasserstein metric. Natural-gradient descent is formulated as a fixed-point iteration and accelerated using Riemannian Anderson mixing, which exploits historical residuals without explicitly forming Hessians. The parameter-space formulation simplifies retraction and vector transport. Experiments cover multimodal sampling, PDE-based inverse problems, aggregation dynamics, Bayesian logistic regression, and generative modeling. The reported results show fewer natural-gradient iterations and competitive performance relative to SGD and Adam in loss-gradient evaluations and wall-clock time.
Critical strengths
In the PDE-based inverse problem, RAM achieves acceleration with little additional overhead relative to NGD. Their iteration rates differ by approximately 2%, and the best-performing RAM configuration is 3.6 times faster than the best Adam configuration in wall-clock convergence time (Table 1).
The generative-modeling experiments demonstrate that acceleration can be achieved without over-relaxation . Table 2 reports fewer iterations to reach the specified MMD thresholds on all three distributions: checkerboard, two spirals, and eight Gaussians.
Critical weaknesses
The convergence analysis needs to specify its valid relaxation range. Proposition 3.2 gives a residual estimate with the factor . However, the cited RAM proof explicitly uses , whereas Algorithm 1 permits any , and the experiments include . The estimate therefore cannot be directly extended to all allowed over-relaxed settings. The authors should state the required restriction on and distinguish the configurations covered by the analysis from those supported empirically. If the guarantee is intended to cover over-relaxation, the relevant estimates require additional justification.
The empirical evaluation needs stronger ablations and sensitivity analysis. Comparisons with NGD, SGD, and Adam demonstrate the effectiveness of the proposed RAM implementation, but do not isolate the additional benefit of -norm mixing over Euclidean Anderson mixing using the same natural-gradient residual definition.
Questions for authors
What relaxation range is required for Proposition 3.2? The cited proof uses , whereas Algorithm 1 permits any . Please distinguish the experimental configurations covered by the analysis from those supported empirically. If the guarantee extends to over-relaxation, how are the proof steps relying on modified?
How does performance change when the -norm in the mixing least-squares problem is replaced with the Euclidean norm, while keeping the natural-gradient residual definition, step size, and history length unchanged and using comparable tuning budgets? Could this comparison help isolate the additional benefit of the point-dependent metric in the mixing step?
2 comments
Thanks for replying
Thanks for replying, I will maintain my score.
We would like to thank the reviewer for the constructive criticism and pointing out the paper's strengths, especially the acceleration for the Bayesian inverse problem. In response to the weaknesses you have mentioned.
Distinction between over-relaxed (βk>1) and standard (βk∈[0,1]) settings
We thank the reviewer for identifying this missing restriction. We agree that the cited proof underlying Proposition 3.2 requires 0<βk≤1. We have explicitly stated this condition in the main convergence statement (proposition 3.2) and revised Remark 3.1 to distinguish the range covered by the analysis.
Nevertheless, we believe that the over-relaxation technique is worth exploring numerically. It has been explored in the relevant literature on AM. Potra and Engler [1] analyze Anderson acceleration for linear problems with arbitrary nonzero mixing parameters, including values greater than one. In nonlinear applications, Storvik et al. [2] demonstrate the usefulness of combining Anderson acceleration with over-relaxation in phase-field fracture simulations, recommending a relaxation parameter of 1.6 based on their experiments. We have added the corresponding discussion in Remark 4.1
We would also like to note that over-relaxation is just one of the exploratory directions in the experiments, and is shown not to be necessary for the acceleration. For example, for the Inverse problem and MMD generative modeling, the noticeable acceleration is achieved solely for β=1.
Isolation of the effect of Riemannian scalar product
We thank the reviewer for the thoughtful suggestion.
We have performed an additional experiment according to your suggestion. The results can be found in Appendix C.3.
We focus on the problem of sampling from the Styblinsky-Tang distribution in d=50. This is one of the hardest problems in the paper, due to the high dimension of the distribution, multimodal posterior and large number of iterations needed for convergence. The usage of natural gradient iteration is strongly justified in this case as it allows to avoid mode collapse, to which the methods, based on the Euclidean gradient (SGD, Adam) were prone.
We fix the hyperparameters of the model, the underlying natural gradient operator and the number of history vectors in RAM. We use adaptive regularization and only vary the regularization factor λ. The result is given on Figure 10. In the table below, we show the minimal amount of iterations, needed to reach three threshold of the value if target functional, by both Riemannian and Euclidean AM with 3 best values of the regularization factor.
One can clearly see that the amount of iterations is smaller for RAM on all levels. The best performing RAM requires approximately 2 times less iterations than the best Euclidean AM to reach the smallest loss threshold. Moreover, from Figure 10, RAM is more stable and its performance of depends less on the particular value of the regularization factor than in the Euclidean case, meaning that the adaptive regularization is more efficient if computed in the right metric. We believe this experiment clearly isolates the benefit of using the Riemannian scalar product.
[1] Potra, F. A., and Engler, H. (2013). A characterization of the behavior of the Anderson acceleration on linear problems. Linear Algebra and its Applications, 438, 1002–1011.
[2] Storvik, E., Both, J. W., Sargado, J. M., Nordbotten, J. M., and Radu, F. A. (2021). An accelerated staggered scheme for variational phase-field models of brittle fracture. Computer Methods in Applied Mechanics and Engineering, 381, 113822.