acceptodds
Under review as a conference paper at ICLR 2027

Bootstrap Inference for Riemannian Optimization Problems

Abstract

We develop an online multiplier bootstrap for manifold-valued parameters estimated by averaging the iterates of Riemannian stochastic approximation. The method runs weighted iterations alongside the original algorithm, with each path using the same observations while updating its own state and intrinsic average. The weighted averages provide uncertainty assessments without a plug-in asymptotic covariance estimator. Under local stability and regularity conditions, we establish bootstrap consistency and show that a fixed number of paths suffices for asymptotically valid Student intervals and intrinsic confidence regions. The theory accommodates samplewise nonsmooth scores and identifies a critical oracle-bias regime in which the estimator’s limiting distribution shifts but the centered bootstrap distribution does not. Applications include spherical means, positive-definite matrix means and medians, and principal-subspace estimation. Simulations show coverage close to nominal in regular settings, and an analysis of North Atlantic storm locations illustrates descriptive online inference for a spherical center.

Primary area:
optimization
Keywords:
Bootstrap, Manifold data, Riemannian SGD, Statistical inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2GxDOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper deals with bootstrap estimation of the asymptotic covariance of Riemannian stochastic approximation iterates.

It deals with this problem by running independent copies of the stochastic approximation algorithm (with the same noise and inputs) but with random weights.

Under general assumptions, it obtains the asymptotic joint distribution of comparison vectors and the Student and Hotelling confidence regions.

The paper features numerical validation on real data, but this is limited to just one (rather basic) manifold, the sphere.

Critical strengths

A well written paper, which clearly defines its problem and contribution. It deals with a specialised issue, but manages to remain accessible.

The main theorems are obtained under general (less restrictive) assumptions and are clearly stated.

These theorems are easy to understand and useful even to people who do not care about their conditions/proofs.

Finally, the results in this paper are both mathematically motivated and of clear practical usefulness.

Critical weaknesses

I did not identify any strong weaknesses in this work. I have to say that I did not read much of the supplementary material (in particular the proofs). However, I am familiar with the proof techniques for this kind of problem, and the hypotheses and theorems were formulated in such a way that already make it clear the proof techniques are well understood by the authors. Perhaps a more in depth verification of the proofs will be possible if this work is submitted to a journal, later on.

Questions for authors

This submission tackles an apparently simple, yet in fact quite tricky problem : estimating the asymptotic covariance of Riemannian stochastic approximation iterates. The overal idea is to use a bootstrap strategy by running independent copies of the (..stochastic approximation) algorithm, with random weights. These copies produce averages (estimates) that are compared in the tangent space at the observed average.

Four contributions are put forward in the introduction. All of these are well formulated and very interesting to anyone who works on Riemannian stochastic approximation (in the board sens of this term).

Questions/remarks about Section 2 :

  • why can't alpha be equal to 1/2 or 1 (in equation (1))

  • equation (2) is correct, by possibly problematic in practice : the inverse of a retraction is very rarely defined throughout the manifold. It is usually defined in some sphere of radium strictly less than the radius of the manifold at the base point

  • equation (7) is perfectly fine, but it took me a while to realise I is an interval and not a number. You should say it is an interval.

Section 3 :

I appreciate the conditions/assumptions were stated in the most general form possible. This is clearly a strong point ... Moreover, the main results in Theorems 3.6 and so on are quite compelling.

When stuyding a "Riemannian extension" of a problem that is well known in Euclidean space, I always ask : how is the curved setting different from the flat one ? Are the results ultimately an extension of Euclidean one (with no "surprises") or is there a really new behavior due to curvature.

I think not discussing this issue is a weak point of this submission, but it is otherwise quite solid and well written.

4 comments

Thank you for these helpful questions about the assumptions, implementation, and geometric interpretation.

Note: Because of the 5000-character limit, we divided our response into two comments.

Reviewer concern: Why are the step-size endpoints excluded, how are local inverse retractions handled, can the interval notation be made explicit, and what changes because of curvature?

We address the four points in turn.

Step-size endpoints.

The manuscript discusses the step size range on p. 22. In the proofs of Lemmas C.17 and C.18, the Abel boundary is bounded by O(n(α−1)/2)O(n^{(\alpha-1)/2}) and the intrinsic-average remainder by O(n1/2−α)O(n^{1/2-\alpha}). Both vanish for α∈(1/2,1)\alpha\in(1/2,1). At α=1/2\alpha=1/2, the displayed averaging bound need not vanish; at α=1\alpha=1, the displayed Abel bound need not vanish. These estimates explain the range used by the present argument and do not establish impossibility at either endpoint. In particular, an analysis at α=1\alpha=1 requires separate control involving the step constant and the spectrum of the population derivative.

Local inverse retractions.

Section 2.3 and Assumption 3.3 require the inverses used by the state and intrinsic-average comparisons to be defined along the implemented trajectory, together with eventual localization in a common comparison neighborhood. The final reporting statistic uses a measurable completion when its inverse is unavailable, and the theory makes that reporting event asymptotically negligible. This completion does not supply a missing inverse needed earlier by an iterative-average update. The state safeguard, state-rate verification, and full-average localization are separate parts of the sufficient conditions.

Explicit confidence-interval notation.

Equation (7) displays the endpoints, and Corollary 3.7(i) identifies the result as an interval. The revised first definition now explicitly says "confidence interval." For a differentiable scalar functional ϕ\phi, fixed B≥1B\ge1, and confidence error probability τ∈(0,1)\tau\in(0,1), the construction is

s_{\phi,n,B}^{2} &=\frac1B\sum_{b=1}^{B} \bigl\\{\phi(\bar x_n^{\star(b)})-\phi(\bar x_n)\bigr\\}^{2},\\\\ I_{\phi,n,B}(1-\tau) &=\left[ \phi(\bar x_n)-t_{B,1-\tau/2}s_{\phi,n,B},\\; \phi(\bar x_n)+t_{B,1-\tau/2}s_{\phi,n,B} \right]. \end{aligned}$$ Here $t_{B,1-\tau/2}$ is the Student quantile with $B$ degrees of freedom. The scale is already computed from unscaled estimator differences, so the endpoints have no additional factor of $n^{-1/2}$. Centering is at the original estimate, which gives $B$ rather than $B-1$ degrees of freedom. The stated fallback is $\mathbb R$ when $s_{\phi,n,B}=0$; under the nonzero limiting-variance condition, this event is asymptotically negligible. *Continued in Part 2/2, which addresses curvature and comparison at the observed base point.*

Part 2/2 of our response to Reviewer 2GxD, continued from Part 1/2. This final comment addresses curvature and comparison at the observed base point.

Curvature and comparison at the observed base point.

Our results concern first-order inference in a regular, locally stable regime. Under the assumptions, the original and multiplier states eventually localize in a common neighborhood of an isolated target. The population derivative is nonsingular, and the states satisfy the second-moment bound E∣Rx⋆−1(xk)∣2=O(k−α)\mathbb E\\|R_{x_\star}^{-1}(x_k)\\|^2=O(k^{-\alpha}), with α∈(1/2,1)\alpha\in(1/2,1). These conditions together with the geometric comparison bounds make the additional errors from retraction, intrinsic averaging, and changing tangent base points negligible at the n\sqrt n scale.

Likewise, once the original and multiplier averages have errors of order n−1/2n^{-1/2}, the difference between their inverse-retraction comparison and their difference in target coordinates is of order n−1n^{-1}. Consequently, the leading asymptotic behavior is governed by a linear stochastic approximation in Tx⋆MT_{x_\star}M, giving the covariance

V=H−1Q(H−1)†.V=H^{-1}Q(H^{-1})^\dagger.

Thus, the familiar form of the limit follows from localization, regularity, and the scale of approximation.

Nevertheless, this does not make the limiting covariance independent of geometry. The operator HH is the derivative of the Riemannian population field and QQ is the covariance with respect to the tangent space. Both can depend on the geometry of the problem. For instance, for the spherical intrinsic mean in Section C.3.1, writing v=Log⁡x⋆(Y)v=\operatorname{Log}_{x_\star}(Y), r=∣v∣r=\\|v\\|, and e=v/re=v/r, we have

e\otimes e+r\cot(r)(I-e\otimes e) \right],$$ with the continuous extension at $r=0$. The factor $r\cot r$ explicitly reflects spherical geometry. Therefore, geometry can already change the leading uncertainty even though the covariance retains the same form. What vanishes under our assumptions is the additional first-order contribution from the nonlinear averaging and comparison operations. More explicit geometric effects would be relevant in several extensions. First, finite sample or higher order inference would retain geometric terms that are negligible in our first order analysis. Curvature, admissible neighborhood radii, and retraction properties could then enter approximation and coverage error bounds. Establishing such results would require quantitative control of the probabilistic terms, while separating geometric effects from step size and nonlinear drift effects. Second, cut locus interactions, singular population derivatives, or failure to converge to a common attracting solution can require a different analysis. Cut locus effects may alter the effective linearization, while degenerate targets may exhibit nonstandard rates or non-Gaussian limits. In these settings, one would need to establish the appropriate asymptotic behavior before determining whether the present bootstrap remains valid or requires modification. Thus establishing quantitative geometric effects and inference beyond this regime are important directions for future work.

Thank you again for your careful review and constructive feedback.

Thank you for answering my questions. In the first place, my decision was "clear accept", and it (of course) still holds.

Reviewer_pyETOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops an online multiplier bootstrap framework for inference in Riemannian stochastic optimization. By maintaining weighted stochastic approximation trajectories and their intrinsic averages, the method avoids explicit estimation of the asymptotic covariance. The authors establish bootstrap consistency, derive confidence intervals and intrinsic confidence regions with a fixed number of bootstrap replicates, and extend the theory to samplewise nonsmooth stochastic oracles. Numerical experiments are conducted on spheres, positive-definite matrices, and Grassmann manifolds.

Critical strengths
  1. The paper presents a unified framework for bootstrap inference on Riemannian manifolds, with detailed theoretical analysis of intrinsic averaging, tangent-space comparisons, and bootstrap consistency.

  2. The framework covers several important manifold optimization problems, including spherical means, positive-definite matrix means and medians, and principal-subspace estimation. The extension to samplewise nonsmooth stochastic oracles is also valuable.

  3. The fixed-replication approach enables asymptotically valid scalar confidence intervals using only a few additional trajectories, without explicitly estimating Hessians or covariance matrices.

Critical weaknesses
  1. The core algorithm and fixed-replication Student calibration are inherited from the Euclidean Cheap Online Bootstrap (COnB) framework of Lam and Wang (2026). Although the extension to Riemannian manifolds requires nontrivial geometric analysis, the underlying methodological principles remain largely unchanged. The conceptual advances beyond this generalization are not sufficiently clear.

Lam and Wang, Cheap Bootstrap for Fast Uncertainty Quantification of Stochastic Gradient Descent. JMLR 2026

  1. The theoretical guarantees require joint localization and stability of the original and weighted trajectories, including their full intrinsic averages. These conditions are not rigorously verified for the implementations used in the numerical experiments. In particular, the authors acknowledge that tangent-step rescaling alone does not guarantee the required localization, leaving a gap between the theoretical assumptions and the implemented algorithms.

  2. The experiments do not systematically examine challenging geometric regimes, such as small eigengaps in Grassmann PCA or ill-conditioned positive-definite matrices. Although interval lengths are reported in the appendix, the evaluation primarily emphasizes coverage without a systematic assessment of the trade-off between interval width and computational cost. This makes it difficult to assess the practical advantages of the proposed method.

  3. The approximation-bias analysis identifies an interesting critical regime, but no numerical experiments investigate this phenomenon. Moreover, despite establishing theoretical guarantees for full intrinsic confidence regions, the experiments evaluate only scalar confidence intervals, leaving the practical performance of the multivariate inference procedure unexplored.

Questions for authors
  1. What are the main conceptual contributions beyond extending the existing Euclidean COnB framework to Riemannian manifolds?

  2. Can the authors verify that the algorithms implemented in the experiments satisfy the localization and stability assumptions required by the theoretical results?

  3. Can the authors provide more comprehensive experiments evaluating computational efficiency, challenging geometric regimes, and the additional theoretical results on approximation bias and intrinsic confidence regions?

6 comments

We thank the reviewer for the detailed assessment and review feedback.

Note: Because of the 5000-character limit, we divided our response into three comments.

Comment 1

What are the main conceptual contributions beyond extending the existing Euclidean COnB framework to Riemannian manifolds?

We acknowledge the existence of prior work in the Euclidean setting. However, beyond extending Euclidean COnB to manifolds, our work makes three main conceptual contributions:

1. Observable intrinsic bootstrap across changing tangent spaces.

In Euclidean space, all points share the same vector space and averaging is linear. On a curved manifold, both linearity and coordinate alignment break down. Connecting an observable statistic to the sampling distribution requires establishing three theoretical links:

  • Controlling geometric updates: Tangent-space expansions must bound curvature nonlinearities and retraction errors for both the original and weighted recursions.
  • Linearizing intrinsic averages: The recursive intrinsic averages must be shown to equal coordinate arithmetic averages up to first order along both trajectories.
  • Bridging moving tangent spaces: The implementable statistic T~n⋆=n,Rxˉn−1(xˉn⋆)\widetilde T_n^\star = \sqrt n\\,R_{\bar x_n}^{-1}(\bar x_n^\star) lives in the random tangent space TxˉnMT_{\bar x_n}M, whereas the population quantities (HH and innovations ξk\xi_k) live in Tx⋆MT_{x_\star}M. We establish their conditional equivalence via parallel transport:

Pxˉn→x⋆T~n⋆=−H−11n∑k=1n(Wk−1)ξk+oP⋆(1).P_{\bar x_n\to x_\star}\widetilde T_n^\star = -H^{-1}\frac1{\sqrt n} \sum_{k=1}^n(W_k-1)\xi_k +o_{P^\star}(1).

Crucially, stability of the state iterates alone does not guarantee localization of their full recursive averages, nor does an unweighted Riemannian CLT imply conditional validity for a weighted comparison at a random base point. This link enables valid, derivative-free confidence regions centered directly at the observed estimate xˉn\bar x_n.

2. A relaxed framework for nonsmooth geometric M-estimation.

Standard Euclidean COnB requires twice-differentiable sample losses. We relax this requirement by conceptually decoupling the smoothness of the population field from the local L2L^2 behavior of the stochastic score, dispensing with samplewise smoothness and symmetric population derivatives.

This is essential for intrinsically nonsmooth manifold problems. For example, the geometric median on the positive-definite matrix manifold SPD(2)\mathcal{SPD}(2) (Proposition C.15) uses the normalized score

Ψ(X;Y)=−Log⁡X(Y)dAI(X,Y),\Psi(\mathbf X;\mathbf Y) =-\frac{\operatorname{Log}_{\mathbf X}(\mathbf Y)} {d_{\mathrm{AI}}(\mathbf X,\mathbf Y)},

which is samplewise nonsmooth at X=Y\mathbf X = \mathbf Y. Despite this singularity, our framework guarantees state stability and full-average localization, establishing bootstrap validity where standard smooth COnB theory fails.

3. Delineating the approximation-bias boundary (Proposition 3.8).

We explicitly characterize when the online bootstrap provides valid inference directly for the target x⋆x_\star. Proposition 3.8 identifies a critical regime where the estimator experiences an asymptotic bias shift while the centered multiplier law remains centered. This defines the precise theoretical boundary where the bootstrap applies directly and where additional bias correction is necessary.

Continued in Part 2/3, which addresses Comments 2 and 3 on localization, stability, and full intrinsic confidence regions.

Part 2/3, continued from Part 1/3. We address Comments 2 and 3 below.

Comment 2

Can the authors verify that the algorithms implemented in the experiments satisfy the localization and stability assumptions required by the theoretical results?

The theory requires both original and weighted state stability along with localization of the complete intrinsic averages in a common chart. Appendix A.1 clarifies these theoretical requirements and distinguishes them from operational tangent-step rescaling. Propositions C.7–C.8 and Lemma C.9 establish a rigorous sufficient verification route: an invariant safeguard and drift/moment bounds establish state contraction rates, and an invariant averaging basin controls the full recursive averages. Proposition C.15 carries out this end-to-end verification for the compact-density Riemannian median model.

Empirically, the step safeguard operates precisely to prevent trajectory excursions and maintain updates within a controlled comparison chart where local retractions and their inverses are well-defined. For the Figure 1 designs at n=5000n=5000 and B=10B=10, recorded counters across all 500 independent streams yield base rescaling counts of 8,0,288, 0, 28 out of 2,500,0002{,}500{,}000 updates, and weighted counts of 1151,96,9211151, 96, 921 out of 25,000,00025{,}000{,}000 updates across the sphere (q=2q=2), positive-definite mean (m=2m=2), and Grassmann (r=1,p=10r=1,p=10) models, respectively.

These vanishingly small rescaling frequencies (e.g., zero rescalings across 2.5 million updates for the positive-definite model, and below 0.0050.005\\% for the Grassmann model) demonstrate empirically that both the base and weighted trajectories remain safely within the regular domain throughout the iterations. Together with the analytical guarantees in Propositions C.7–C.8 and Lemma C.9, this confirms that the implemented algorithms satisfy the stability and localization behavior required for valid asymptotic inference.

Comment 3

Can the authors provide more comprehensive experiments evaluating computational efficiency, challenging geometric regimes, and the additional theoretical results on approximation bias and intrinsic confidence regions?

We address these components below; full intrinsic confidence regions are discussed in Comment 4.

Weak identification and positive-definite conditioning.

For the weak-eigengap elliptical Grassmann stress cells at gap 0.250.25, n=5000n=5000, and B=10B=10, scalar coverage is 0.8480.848, 0.8680.868, and 0.8540.854 for (r,p)=(1,10),(2,20),(3,30)(r,p)=(1,10),(2,20),(3,30), respectively. These results show that coverage in the regular designs does not transfer automatically to weak identification.

In our positive-definite simulations (Appendix A.2), target eigenvalues range within [e−0.3,e0.3][e^{-0.3}, e^{0.3}] (condition number ≈1.82\approx 1.82), focusing on nominal geometric performance. Systematically stressing condition numbers toward infinity would primarily test the numerical stability of matrix inversions and log-maps. We have clarified this scope in the revised discussion.

Computational cost.

The original block-level measurements are:

Model blockMedian seconds per replicationPeak worker working set (MB)
Sphere0.29106
Positive-definite0.98140
Grassmann0.27164

These measurements reflect whole-block pipeline execution. While exact wall-clock speedups depend on implementation backends, the decisive computational advantage of our online procedure is structural: it processes data in a single streaming pass with O(Bd)O(Bd) storage, completely avoiding the O(n)O(n) data storage and repeated full-dataset passes inherent to offline bootstrap replay. The method exchanges additional path evaluations for an inference procedure without online covariance-transport updates or batch-length tuning.

Approximation bias and further vector-field generality.

The primary conceptual role of Proposition 3.8 is to serve as an analytical demarcation result: it establishes the sharp O(k−1/2)O(k^{-1/2}) cumulative bias boundary, clarifying precisely when online multiplier resampling preserves validity and when systematic bias invalidates target inference. For a leading bias bk=k−1/2b0b_k=k^{-1/2}b_0, the estimator's limit shifts to N(−2H−1b0,V)N(-2H^{-1}b_0,V) while the centered multiplier law remains centered at N(0,V)N(0,V).

In Proposition C.15, we prioritized providing a complete, end-to-end verification for the challenging setting of samplewise nonsmoothness, where the oracle is exact and the population Hessian is naturally self-adjoint. Beyond this specific application, our general framework is deliberately formulated without enforcing symmetry or oracle exactness to maintain foundational generality for Riemannian stochastic approximation. We have clarified the precise scope and theoretical role of these general conditions in the revision.

Continued in Part 3/3, which addresses Comment 4 on full intrinsic confidence regions.

Part 3/3, continued from Part 2/3. This final comment addresses Comment 4 on full intrinsic confidence regions.

Comment 4

Moreover, despite establishing theoretical guarantees for full intrinsic confidence regions, the experiments evaluate only scalar confidence intervals, leaving the practical performance of the multivariate inference procedure unexplored.

We added a comprehensive fixed-law Grassmann experiment with (p,r,d)=(10,1,9)(p,r,d)=(10,1,9) and eigengap 0.60.6 evaluated across sample sizes n∈5000,10000,20000,50000n\in\\{5000, 10000, 20000, 50000\\} and budgets B∈9,10,20,40B\in\\{9, 10, 20, 40\\}. The 4-budget results at n=5000n=5000 (500 independent streams) are summarized below:

nnBBB/dB/dCoverage95% Wilson MC intervalMedian tangent-equivalent radius
5,00091.000.944[0.920, 0.961]3.637
5,000101.110.934[0.909, 0.953]0.889
5,000202.220.932[0.906, 0.951]0.255
5,000404.440.912[0.884, 0.934]0.212

At B=d=9B=d=9, the single denominator degree of freedom (B−d+1=1B-d+1=1) in the Hotelling critical value leads to an inflated radius (3.6373.637). Increasing BB slightly to B=10B=10 or 2020 drastically contracts the region (by 7575\\% and 9393\\%, respectively). While finite-sample dispersion at n=5000n=5000 yields 91.291.2\\% coverage at B=40B=40, expanding the horizon to n=20,000n=20{,}000 and 50,00050{,}000 brings coverage at B=40B=40 to 94.494.4\\%, aligning closely with nominal 9595\\% coverage while maintaining tight regions (median radius 0.0670.067 at n=50,000n=50{,}000).

Full-region inference requires B≥dB\ge d, a d×dd\times d reporting matrix, and its associated linear algebra. By contrast, scalar inference requires only pathwise scalar displacements at reporting (B≥1B\ge 1), avoiding the d×dd\times d matrix inversion.

Reviewer_MTbzOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops an online multiplier bootstrap for manifold-valued parameters estimated by recursively averaging Riemannian stochastic approximation iterates. Each bootstrap trajectory shares the observations and auxiliary oracle randomness with the original trajectory, evaluates the oracle at its own state, and maintains its own intrinsic average. Under local stability, regularity, and martingale innovation conditions, the authors establish an asymptotic linear representation for the original and weighted averages and prove conditional bootstrap consistency. A geometric comparison justifies evaluating the bootstrap difference in the tangent space at the observed average rather than at the unknown target.

Critical strengths
  • The paper addresses a useful inference problem with a clear computational motivation. Estimating the sandwich covariance of a manifold-valued stochastic approximation estimator can require population derivative information and compatible representations of quantities belonging to different tangent spaces. Maintaining a small collection of perturbed trajectories offers a practical route to scalar uncertainty quantification without explicitly estimating the population derivative or a full asymptotic covariance matrix.

  • The geometric and conditional-probability arguments are handled carefully. The paper explicitly distinguishes the target-tangent influence representation from the observable bootstrap comparison at the estimated base point. It also treats localization of the complete intrinsic averages separately from convergence of the state iterates. The arguments concerning finite exceptional prefixes, conditional negligibility, and asymptotic independence under shared observations address substantive technical details rather than relying on a formal subtraction of manifold-valued estimates.

Critical weaknesses
  • The methodological novelty is concentrated in a local geometric extension, and its significance needs a stronger demonstration. The main algorithmic components have close precedents. Fang, Xu, and Yang (2018) already use complete perturbed SGD trajectories evaluated at their own states, including applications with samplewise nonsmooth scores. Lam and Wang (2026) provide the fixed-replication COnB construction and Student calibration. Tripuraneni et al. (2018) develop the recursive intrinsic averaging rule and the unweighted manifold central limit theorem.

  • The connection between the theoretical path conditions and the experimental implementations is incomplete. Assumptions 3.2–3.3 require O(ηk)O(\eta_k) second-moment state bounds for both the original and weighted trajectories, as well as eventual localization of their full intrinsic averages in a common comparison chart. These are substantive conditions for a method that perturbs entire trajectories. Propositions C.7–C.8 and Lemma C.9 provide a sufficient route through invariant safeguards and an appropriate averaging basin, and Proposition C.15 establishes a fully verifiable median model.

However, the experimental descriptions in Appendix A.2 mainly specify tangent-step rescaling. As the paper itself notes on p. 13, rescaling alone does not establish the spherical cap-preservation conditions. Similar questions arise for the Grassmann implementation and for the state and population-regularity conditions retained in the log-normal positive-definite model. It remains unclear which reported configurations satisfy the complete hypotheses for the algorithms actually run.

Questions for authors
  1. What is the decisive new capability beyond existing COnB and Riemannian averaging? Which primitive geometric or stability result is essential to an application that cannot be addressed directly using the existing frameworks? A concrete example would help assess the significance of the extension.

  2. How do the intrinsic joint regions perform? Please report their coverage and size as B/dB/d varies, including a Grassmann example. What replication budget and computational cost are required for useful full-subspace inference rather than a single projector contrast?

  3. Can an application directly validate the additional oracle or vector-field generality? A concrete approximate oracle with jointly verified bias, variance, and stability conditions, accompanied by a critical-bias experiment, would be especially useful. Alternatively, can the authors provide an application with a nonsymmetric population derivative that satisfies the stated conditions?

  4. Can the numerical evidence include uncertainty and resource measurements? Please provide paired uncertainty estimates for important coverage differences, fixed-distribution sample-size curves, and runtime and memory measurements alongside interval lengths. Diagnostics of step rescaling and inverse-map availability would also clarify finite-sample reliability.

6 comments

We thank the reviewer for the careful and constructive assessment and address the four questions below.

Note: Because of the 5000-character limit, we divided our response into four comments.

Question 1:

What is the decisive new capability beyond existing COnB and Riemannian averaging? Which primitive geometric or stability result is essential to an application that cannot be addressed directly using the existing frameworks? A concrete example would help assess the significance of the extension.

We acknowledge that there is prior work in the corresponding Euclidean setting. Nevertheless, we respectfully disagree with the assessment that these connections diminish the novelty and significance of our contributions. Our core contribution is a derivative-free bootstrap framework for recursively averaged manifold-valued estimator xˉn\bar x_n. It achieves valid inference without requiring knowledge of x⋆x_\star or explicit estimation of HH and the asymptotic covariance. This is far from a simple combination of Euclidean bootstrap and a Riemannian central limit theorem. In particular, manifold curvature introduces three distinct hurdles:

First, the coordinate representation of a Riemannian stochastic update contains geometric nonlinearities and retraction errors. These must be controlled for both the original and weighted recursions.

Second, the recursively computed intrinsic average is generally not equal to the arithmetic average of the iterates in local coordinates. Its first-order equivalence to the coordinate average must be established for the complete original and weighted trajectories.

Third, the implementable comparison

T~n⋆=n,Rxˉn−1(xˉn⋆)\widetilde T_n^\star = \sqrt n\\,R_{\bar x_n}^{-1}(\bar x_n^\star)

belongs to the random tangent space TxˉnMT_{\bar x_n}M, whereas the population derivative and asymptotic innovations are defined in Tx⋆MT_{x_\star}M. Theorem 3.5 and Lemma C.20 establish the connection

Pxˉn→x⋆T~n⋆=−H−11n∑k=1n(Wk−1)ξk+oP⋆(1).P_{\bar x_n\to x_\star}\widetilde T_n^\star = -H^{-1}\frac1{\sqrt n} \sum_{k=1}^n(W_k-1)\xi_k +o_{P^\star}(1).

Establishing the above conditional approximation for the implemented intrinsic estimator is the central additional guarantee.

A concrete example illustrating this setting is provided in Proposition C.15 of our Theory Supplement. Consider the affine-invariant geometric median of 2×22\times2 positive-definite matrices, a manifold of dimension d=3d=3. Proposition C.15 specifies observations with density

=c_R h\\!\left(d_{\mathrm{AI}}(\mathbf A,\mathbf Y)^2\right), \qquad h(s)= \begin{cases} \exp\\{-1/(R^2-s)\\}, & s < R^2,\\\\ 0, & s\ge R^2, \end{cases}$$ with respect to Riemannian volume, where $R > 0$ and $c_R$ is the normalizing constant. Its unique geometric median is $\mathbf A$. The stochastic score $$\Psi(\mathbf{X};\mathbf{Y}) =-\frac{\operatorname{Log}_{\mathbf{X}}(\mathbf{Y})} {d_{\mathrm{AI}}(\mathbf{X},\mathbf{Y})}, \qquad \mathbf{Y}\ne\mathbf{X}.$$ is bounded but samplewise nonsmooth at $\mathbf X=\mathbf Y$, so the smooth sample-loss assumptions of COnB do not directly apply. This example highlights the essential role of our primitive results: Proposition C.7 establishes the required local expansion around the non-flat geometry; Proposition C.8 guarantees weighted state stability under the specified safeguard; Lemma C.9 controls the complete intrinsic averages; and Lemma C.20 justifies the observable comparison. *Continued in Part 2/4, which addresses Question 2 on intrinsic joint regions, coverage, and finite-sample diagnostics.*

Part 2/4 of our response, continued from Part 1/4. We now address Question 2; its replication and computational requirements continue in Part 3/4.

Question 2:

How do the intrinsic joint regions perform? Please report their coverage and size as BB varies, including a Grassmann example. What replication budget and computational cost are required for useful full-subspace inference rather than a single projector contrast?

The fixed-replication scalar intervals are valid for every fixed B≥1B\ge1 under the stated conditions. For a dd-dimensional intrinsic region, our fixed-BB construction is asymptotically valid for every fixed B≥dB\ge d under the stated assumptions, with positive definite limiting covariance. It uses the uncentered second moment of the bootstrap displacements and the critical value

cd,B,0.05=dBB−d+1Fd,B−d+1;0.95.c_{d,B,0.05}=\frac{dB}{B-d+1}F_{d,B-d+1;0.95}.

Although B=dB=d is admissible, its single denominator degree of freedom makes it an overly conservative practical baseline: for d=9d=9, the critical value is approximately 19,48419{,}484, compared with 41.9541.95 at B=20B=20 and 24.6224.62 at B=40B=40.

We expanded the supplementary Gaussian Gr⁡(1,10)\operatorname{Gr}(1,10) experiment (d=9d=9, eigengap 0.60.6) to n∈5000,10000,20000,50000n\in\\{5000,10000,20000,50000\\}. Each row below uses 500 independent data streams. The data streams are nested across sample sizes, and the first BB of the same 40 independently weighted paths are used across budgets. The original initialization, 0.1k−0.70.1k^{-0.7} step sizes, step safeguard, exponential multipliers, and full intrinsic averages are unchanged. All 500 original n=5000n=5000 results were reproduced before extension.

nnBBB/dB/dCoverage95% Wilson MC intervalMedian tangent-equivalent radius
5,00091.000.944[0.920, 0.961]3.637
5,000101.110.934[0.909, 0.953]0.889
5,000202.220.932[0.906, 0.951]0.255
5,000404.440.912[0.884, 0.934]0.212
10,00091.000.952[0.930, 0.968]2.772
10,000101.110.940[0.916, 0.958]0.661
10,000202.220.938[0.913, 0.956]0.192
10,000404.440.932[0.906, 0.951]0.159
20,00091.000.958[0.937, 0.972]1.952
20,000101.110.948[0.925, 0.964]0.472
20,000202.220.940[0.916, 0.958]0.136
20,000404.440.944[0.920, 0.961]0.113
50,00091.000.958[0.937, 0.972]1.161
50,000101.110.940[0.916, 0.958]0.281
50,000202.220.940[0.916, 0.958]0.081
50,000404.440.944[0.920, 0.961]0.067

At B=40B=40, coverage increases from 0.9120.912 at n=5000n=5000 to 0.9320.932 at n=10000n=10000 and stabilizes at 0.9440.944 for n≥20000n \ge 20000. The observed coverage range across budgets narrows from 3.23.2 percentage points at n=5000n=5000 to 1.81.8 points at n=20000n=20000 and 5000050000. While B=40B=40 exhibits a minor finite-sample dip at n=5000n=5000 (0.9120.912), this discrepancy diminishes with sample size, aligning closely with nominal 9595\\% coverage across all budgets for n≥20000n \ge 20000.

Larger budgets consistently reduce the median size statistic. At n=20000n=20000, increasing BB from 20 to 40 reduces it from 0.1360.136 to 0.1130.113, with coverage 0.9400.940 and 0.9440.944, respectively. At n=50000n=50000, the corresponding radii are 0.0810.081 and 0.0670.067. This statistic is the radius of a Euclidean ball with the same volume as the tangent-coordinate ellipsoid before restriction to the local retraction chart. The much larger radii at B=dB=d illustrate why theoretical admissibility alone does not yield a useful local region.

Independent displacement-SVD calculations reproduce all 8000 coverage decisions; no singular-matrix fallback is triggered. A separate 100,000-stream Gaussian normal-Wishart benchmark gives coverage 0.94900.9490 to 0.95030.9503 across these budgets. For B=40B=40, the empirical 95th percentile of the actual region statistic decreases from 29.7929.79 at n=5000n=5000 to 24.8124.81 at n=50000n=50000, approaching its unchanged critical value 24.6224.62.

Common-target tangent diagnostics show finite-sample bias and heavier-than-Gaussian centered estimator tails. At n=5000n=5000, the mean centered bootstrap covariance trace is 0.9320.932 times the empirical sampling covariance trace (pointwise 95% MC interval [0.868,0.995][0.868,0.995]); the corresponding uncentered second-moment ratio used by the region is 0.9490.949. As nn increases to 2000020000, this ratio improves to 0.9810.981, confirming that finite-sample dispersion rapidly converges to the asymptotic target.

Continued in Part 3/4, which completes Question 2 with replication and computational requirements, then addresses Question 3.

Part 3/4 of our response, continued from Part 2/4. We complete Question 2 below, followed by our response to Question 3.

Question 2 (continued)

For Gr⁡(r,p)\operatorname{Gr}(r,p), d=r(p−r)d=r(p-r), so the minimum budgets for (r,p)=(1,10),(2,20),(3,30)(r,p)=(1,10),(2,20),(3,30) are 9,36,819,36,81, respectively. In the present rank-one example, n=20000n=20000 with B=20B=20 to 40 (B/d=2.22B/d=2.22 to 4.44) provides an empirically supported balance: coverage is close to nominal, and regions are substantially smaller than at B=dB=d. Increasing nn to 50000 further reduces region size while maintaining stable, calibrated coverage (0.9440.944). Single-worker replays of five existing streams took median trajectory times of 0.900.90 and 1.831.83 seconds at n=20000n=20000 for B=20B=20 and 40, and 2.302.30 and 4.614.61 seconds at n=50000n=50000; these timings exclude data and multiplier generation and compilation.

Each observation requires BB additional oracle evaluations and state retractions. After the first observation, maintaining the weighted full averages also requires BB inverse-retraction/retraction pairs. Storage for the additional paths is O(Bd)O(Bd) when states admit an O(d)O(d) representation. With Grassmann QR representatives, states and averages are p×rp\times r matrices, giving O(Bpr)O(Bpr) path storage. Full-region reporting additionally forms a d×dd\times d second-moment matrix and performs the corresponding linear algebra. A single projector contrast uses scalar path displacements for calibration and avoids this full reporting matrix, while still maintaining the manifold-valued trajectories. Consequently, while scalar inference is exceptionally economical (B≥1B \ge 1), joint full-subspace inference naturally benefits from a moderate budget (B≥dB \ge d) to properly estimate the full tangent-space covariance geometry.

Question 3:

Can an application directly validate the additional oracle or vector-field generality? A concrete approximate oracle with jointly verified bias, variance, and stability conditions, accompanied by a critical-bias experiment, would be especially useful. Alternatively, can the authors provide an application with a nonsymmetric population derivative that satisfies the stated conditions?

The result directly relevant to approximate oracles is Proposition 3.8, which identifies the cumulative bias boundary under the remaining innovation, state rate, localization, and remainder conditions.

For a fixed nonzero b0∈Tx⋆Mb_0\in T_{x_\star}\mathcal M and bk=k−γb0b_k=k^{-\gamma}b_0, the accumulated bias n−1/2∑k=1nbkn^{-1/2}\sum_{k=1}^n b_k vanishes when γ>1/2\gamma > 1/2. At the critical rate bk=k−1/2b0b_k=k^{-1/2}b_0, the proposition gives

\rightsquigarrow N(-2H^{-1}b_0,V), \qquad \mathcal L^\star(T_n^\star)\rightsquigarrow N(0,V) \quad\text{in probability},$$ where $V=H^{-1}Q(H^{-1})^\dagger$, $Q$ is the limiting innovation covariance, and $\dagger$ denotes the adjoint. The centered multiplier law misses the deterministic shift because, for the independent mean-one, variance-one multipliers, $$\mathbb E^\star \left\\|\frac{1}{\sqrt n} \sum_{k=1}^n(W_k-1)b_k\right\\|^2 =\frac{1}{n}\sum_{k=1}^n\\|b_k\\|^2 =\frac{\\|b_0\\|^2}{n}\sum_{k=1}^n\frac1k \longrightarrow0.$$ Increasing the number of multiplier paths does not remove this shared systematic bias. This supplies an oracle-accuracy criterion for inference about the intended target, conditional on the other hypotheses of the proposition. In Proposition C.15, we focus on fully verifying the samplewise nonsmooth setting. In this problem, the oracle is exact, and the Hessian is naturally self-adjoint. Beyond this specific application, our general theory deliberately does not require symmetry or oracle exactness. This keeps our framework broadly applicable to general Riemannian stochastic approximation. In particular, Proposition 3.8 establishes a sharp theoretical boundary at the $O(k^{-1/2})$ cumulative bias rate. It clarifies exactly when the online bootstrap remains valid, and when systematic bias invalidates inference about the target. We have revised the manuscript to make these distinctions and the role of our general conditions explicit. *Continued in Part 4/4, which addresses Question 4 on uncertainty, interval lengths, resources, and finite-sample diagnostics.*

Comments on the paper

1 comment

Thank you for your time and valuable comments.

We clarified the observable random-base comparison, the separate roles of state stability and full-average localization, and the fixed-replication calibration for scalar intervals and intrinsic regions. The Grassmann study now includes the completed nested sample-size and replication-budget comparison, with coverage, region size, tangent diagnostics and computational costs. Representative full-region results appear in the main text, with the complete comparison and diagnostics in the numerical supplement. The geometric-median example and approximation-bias result retain their stated assumptions.

These changes have been incorporated into the article, with the additions and modifications highlighted in blue. We hope the responses address your concerns and welcome further questions or discussion. Thank you again for your valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.