- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the statistical inference of randomized newton iterates for estimating parameters which exist on a Reimanian manifold. In particular, the algorithm in question is Polyak-Rupert averaging of the following sequence: The sequence of randomized Newton iterates (Hessian-conditioned SGD on the manifold). The randomization comes from solving a random projection of the linear system required to estimate the inverse of the Hessian. The main results of this work is not just establishing the asymptotic variance for the CLT of the estimator, but demonstrating that when the estimate of the random solve inverted Hessian is a fresh random estimate at each iteration that this introduces a lasting term on the asymptotic variance and if the random solve inverted Hessian is replaced with a running average of all the past estimates at each step that the estimator remains asymptotically unbiased and the additional term in the CLT variance arising from the random solve disappears. The paper demonstrates the proposed algorithm on a few numerical examples, one toy and one real world dataset without an existing ground truth (the ladder is only in the appendix).
It is my understanding that the contribution of demonstrating how the Hessian averaging step can remove the additional term from the asymptotic variance in contrast to the 'fresh random estimate' version of the algorithm is not new and is part of a recent line of work. The novelty here is adapting the same idea to the Riemannian Manifold setting.
I emphasize here that I do not have the sufficient background to understand any of the low level details of this paper and certainly not to check for correctness. That said my overall impressions (which were in general positive) are based entirely on my (hopefully correct) high level understanding of what this paper accomplished and where it stands relative to existing work.
Critical strengths
The paper provides a CLT for Polyak Rupert averaged Randomized Newton Iterates for estimating parameters existing on Riemannian manifolds. Further they demonstrate a scheme for the randomized solve step that does not leave lasting variation in the CLT, in contrast to the default randomized solve at each iterate which they also demonstrate leaves an additional covariance term in the corresponding CLT. While the idea for the randomization mechanism which does not leave a lasting effect is not new, demonstrating the results on Reimannian Manifolds in contrast to euclidean space seems to be a solid contribution.
Critical weaknesses
The experimental section is perhaps a bit weak. The examples are limited and it seems the authors admit that the Taurus example is perhaps somewhat simple and not substantially differentiated from the Euclidean case.
Questions for authors
Could you provide some more examples of applications where one might use this algorithm?
Does the Taurus example differentiate the manifold version of the algorithm from the euclidean version?
5 comments
Part 2/2 of our response to Reviewer iyNJ, continued from Part 1/2. We address Q1–Q2 below, followed by the references.
Q1. To demonstrate broader geometric applicability, Corollaries B.21–B.24 provide explicit specializations for intrinsic Fréchet means, directional sphere means, affine-invariant covariance centers on the SPD manifold, and Grassmann principal subspaces for streaming dimension reduction (illustrated via the WDBC data). Furthermore, the two-dimensional sphere benchmark in W1 of Part 1/2 evaluates the algorithm on a curved Riemannian state space: direction-based studentization raises coverage from 90.6% to 93.6% for nominal 95% regions.
Q2. The product torus was chosen as a controlled benchmark to isolate transient conditioning and solver-noise studentization within a chart-linear coordinate system, deliberately decoupling these effects from non-zero curvature.
To demonstrate broader geometric applicability, Corollaries B.21–B.24 provide explicit specializations for intrinsic Fréchet means, directional sphere means, affine-invariant covariance centers on the SPD manifold, and Grassmann principal subspaces for streaming dimension reduction (illustrated via the WDBC data). Furthermore, the two-dimensional sphere benchmark in W1 of Part 1/2 evaluates the algorithm on a curved Riemannian state space: direction-based studentization raises coverage from 90.6% to 93.6% for nominal 95% regions.
References.
- X. Du, W. Zhu, W. B. Wu, and S. Na (2025), Online Statistical Inference of Constrained Stochastic Optimization via Random Scaling, arXiv:2505.18327.
- X. Du, E. Rebrova, M. Dereziński, and S. Na (2026), Inference for Newton Methods with Accelerated Sketch-and-Project via Random Scaling, arXiv:2609.12421.
- H. Wang, X. Du, and S. Na (2026), Inference of Online Newton Methods with Nesterov's Accelerated Sketching, ICML 2026.
- N. Tripuraneni, N. Flammarion, F. Bach, and M. I. Jordan (2018), Averaging Stochastic Gradient Descent on Riemannian Manifolds, COLT 2018.
Thank you again for your valuable feedback. We hope our responses address your concerns and welcome any further questions or discussion.
Thank you again for your careful review and constructive feedback. We hope that our responses and revisions have addressed your concerns and questions. If you find that they resolve your main concerns, we would be grateful if you would reconsider your assessment and score in light of these changes. We would be happy to clarify any outstanding points.
Thank you for your time and valuable comments.
Note: Because of the 5000-character limit, we divided our response into two comments.
The distinction between predictable gains and fresh randomization is established in stochastic approximation. A fresh solver can retain random variation after the Hessian estimate has stabilized.
Under Theorem 3.4's independence, finite second moment and invertibility conditions, write ΔP=P−Pˉ and
\qquad V-V_{\mathrm{sand}} =H^{-1}\bar P^{-1}R(\bar P^{\ast})^{-1}(H^{\ast})^{-1}.$$ The extra covariance is positive semidefinite. Moreover, $$\operatorname{tr}(R) =\mathbb E\\|\Delta P S^{1/2}\\|_F^2,$$ so it vanishes exactly when $\Delta P S^{1/2}=0$ almost surely. It is nonzero when the solver fluctuates on the score covariance support, although individual contrasts can still have zero inflation. In particular, positive-definite $S$ and a nonconstant solver imply a nonzero extra covariance. Theorem 3.6 instead treats predictable gains under its inverse gain Abel-summation and weighted remainder conditions. Its centered conclusion is first-order asymptotic and uses the stated accumulated-bias conditions. **W1.** We have clarified the local CLT, bias-correction scope, and covariance tracking, and added the curved-sphere comparison below. This experiment measures how including solver variation changes coverage under non-zero curvature. The product torus was chosen as a controlled benchmark to isolate transient conditioning and solver noise studentization within a chart linear coordinate system, deliberately decoupling these effects from non-zero curvature. To directly investigate the combined interaction of nonlinear manifold curvature, learned curvature, and randomized solves, we conducted a dedicated benchmark on the two dimensional unit sphere $\mathbb S^2$. Observations were generated via the exponential map from clipped anisotropic tangent Gaussian draws (coordinate standard deviations $0.30$ and $0.18$, radius cap $0.85$). From a fixed non-zero initial offset, we compared RSGD, Predictable Newton (sample-Hessian Newton), and Proposed ($m=4$, fresh coordinate sketch-and-project with learned Hessian). Each configuration used $n=1024$ observations and a 128-step burn-in ($c=0.4$). Observation streams were paired and solver seeds generated independently. | Method | Reported coverage (%) | Score-only coverage (%) | Geodesic RMSE | |---|---:|---:|---:| | RSGD | 96.2 | 96.2 | 0.01171 | | Predictable Newton | 96.2 | 96.2 | 0.01171 | | Proposed ($m=4$) | **93.6** | **90.6** | 0.01350 | The 95% Wald regions are joint in two dimensions. In a low dimensional setting ($d=2$), sketching introduces solver variance without offering the high dimensional $O(md^2)$ versus $O(d^3)$ computational advantage, leading to a mild finite-sample variance penalty (RMSE $0.01350$ vs. $0.01171$). Crucially, this benchmark serves as a diagnostic for covariance estimation under non-zero curvature: 1. Under fresh solver randomization, standard score-only studentization underestimates variance, resulting in under coverage ($90.6\\%$). 2. Direction-based studentization increases coverage from 90.6% to 93.6%, a gain of 3.0 percentage points, by including solver variation in the Wald region. The two predictable methods exhibit nearly identical aggregate errors, with negligible paired mean squared-error difference, though their pathwise trajectories and Wald statistics differ. The sphere experiment shows the coverage gain from direction-based covariance correction alongside the larger RMSE caused by solver randomization. *Continued in Part 2/2 with Q1–Q2 and the references.*