acceptodds
Under review as a conference paper at ICLR 2027

Inverse-Free Second-Order Momentum for Decentralized Riemannian Stochastic Optimization on Compact Submanifolds

Abstract

We study decentralized stochastic optimization on compact embedded submanifolds and propose decentralized projected Riemannian stochastic Hessian-corrected momentum (DPRSHCM). The method pairs a correction for the full ambient variation of the Riemannian gradient with a projected iterate difference, yielding a quadratic remainder as tangent spaces change. It combines this correction with gradient tracking and metric projection, using one stochastic gradient and one Hessian–vector product per node per iteration. With a diminishing stepsize and momentum parameters independent of the iteration budget, DPRSHCM achieves per-node oracle complexity for reducing both the expected squared Riemannian gradient norm and mean squared consensus error below . A constant stepsize chosen for a prescribed budget removes the logarithmic factor. With exact local gradients and stochastic Hessian–vector products, fixed parameters achieve the same joint criterion in iterations. Experiments on PCA and matrix completion demonstrate the method's effectiveness and the computational benefits of reusing gradient computations in matrix completion.

Primary area:
optimization
Keywords:
decentralized stochastic optimization, compact submanifolds, Hessian-corrected momentum, inverse-free method, second-order information

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_tcjWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies decentralized stochastic optimization on compact embedded submanifolds and proposes DPRSHCM, which estimates gradients within a gradient-tracking and metric-projection framework using Hessian-corrected momentum. The correction term is the full directional derivative of the Riemannian gradient, viewed as an ambient vector field. It is applied to the iterate difference after projection onto the current tangent space. The theoretical results are Theorem 6.1, Theorem 6.2 and Theorem 6.3. The experiments cover synthetic PCA and LRMC, PCA on Ionosphere and MNIST, and LRMC on MovieLens-100K.

Critical strengths
  1. The diminishing schedule does not depend on the total number of iterations. Runs can therefore be extended without retuning.
  2. The proofs are complete and checkable, and the geometric correction is well motivated. I checked the main derivations of Lemma 5.1, Lemmas E.2–E.18 and Theorems 6.1–6.3 and found no errors.
  3. The experimental reporting is also careful.
Critical weaknesses
  1. The theory does not cover the experiments. LRMC falls outside Assumption 4.1, the sampling and output rules differ from those assumed in the theory, and even in the most favorable case the stepsize used exceeds the theoretical bound.
  2. The complexity results do not improve on prior work.
  3. The benefit of the Hessian correction itself is not isolated. There is neither a diminishing-stepsize DPRSRM baseline nor an exact-gradient baseline without HVP, and the constant configuration shows no consistent advantage over DPRSRM. Moreover, the experiments are small, and the real-data runs are repeated only two or three times.
Questions for authors
  1. Could you add a diminishing-stepsize DPRSRM baseline and a separately tuned exact-gradient baseline without HVP?
  2. Are the guarantees meaningful at the values of L_d and the stepsizes actually used? Could you provide a tighter local bound?
  3. What exactly does "Inverse-Free" in the title refer to?
3 comments

Thank you for checking the main derivations and recognizing the horizon-independent schedule.

W1. Relation between theory and experiments

  • The revised objective meets the smoothness conditions. Appendix A verifies regularized LRMC, while Theorem 6.3 adds reshuffling analysis for complete equal-sized batches and the specified schedules. Appendix G.1 identifies the per-iteration schedule and truncated MovieLens epochs used in the reported runs.

  • Each output criterion has a stated role. The theorem bounds a uniformly selected iterate from the second half of the run; the tables report terminal performance. The theoretical guarantee is attached to the former criterion.

  • The parameter discussion separates sufficient conditions from empirical tuning. The reported stepsizes exceed the theorem's sufficient thresholds. Remark E.20 adds a computable PCA clipping bound and states the remaining restrictions; the existing sweeps assess stability at the tested parameter values.

W2. Contribution beyond the complexity exponent

The advance is the correction mechanism and its guarantee for decentralized manifold optimization. DPRSHCM uses stochastic information at the current point and admits a schedule independent of the iteration budget.

  • Geometry determines the correction. Operator (5.1) accounts for both tangent and normal gradient variation, yielding the quadratic remainder of Lemma 5.1.

  • Network motion changes the error analysis. Lemmas E.9 and E.15 bound displacement moments jointly with consensus and tracking errors. This supplies the control needed for the horizon-independent schedule.

Appendix B compares these requirements alongside the shared ϵ−3/2\epsilon^{-3/2} exponent and the respective oracle models. Theorem 6.3 further provides a complete-epoch reshuffling guarantee.

W3 and Q1. Correction ablations and baselines

  • The diminishing-step comparison isolates the correction from the schedule. Appendix G.5 uses the same parameters, PCA instance and ten sampling streams. The table gives the first epoch-end checkpoint at which the mean squared estimator error reaches the target (Figure 18(c)).
Error targetUncorrectedDPRSRMDPRSHCM
113269955995499
10−110^{-1}5969990998999
10−210^{-2}Not reached1429914199

The checkpoints are k=99,199,…k=99,199,\ldots; “Not reached” means no qualifying checkpoint within 10510^5 iterations. Both corrected estimators reach these accuracy levels far sooner than the uncorrected recursion. Their close results support comparable estimator accuracy.

  • The component ablation identifies the role of each correction term. Table 14 gives the following terminal PCA measures at the same constant parameters (α,τ)=(2.4×10−5,0.001)(\alpha,\tau)=(2.4\times10^{-5},0.001); entries are mean ±\pm standard deviation over ten sampling streams.
EstimatorObjective gapGradient norm GKG_K
Full DPRSHCM(3.7±1.5)×10−5(3.7\pm1.5)\times10^{-5}0.206±0.0520.206\pm0.052
Without projector derivative (DPRSHCM-C)0.024±0.0440.024\pm0.0442.13±12.13\pm1
Without HVP (DPRSHCM-R)0.407±0.970.407\pm0.975.95±6.45.95\pm6.4
DPRSRM(3.7±1.4)×10−5(3.7\pm1.4)\times10^{-5}0.208±0.0480.208\pm0.048

Removing either term increases the mean objective gap and gradient norm. The full correction attains accuracy comparable to DPRSRM while using gradient information at one point.

  • The LRMC comparison measures the computational benefit. With matching diminishing parameters, DPRSHCM uses approximately 37% less total MPI time over 20,000 iterations, with comparable residuals (Table 12). Real-data comparisons use ten sampling streams per setting on a fixed partition. The paper now focuses on fully stochastic optimization; the exact-gradient specialization and its experiments have been removed.

Q2. Clipping radius and admissible stepsizes

  • Remark E.20 gives a computable clipping radius for PCA. With fi(X)=−12tr⁡(X⊤CiX)f_i(X)=-\tfrac12\operatorname{tr}(X^\top C_iX), tangent projection gives

    ∥grad⁡fi(X)∥F≤∥CiX∥F≤r∥Ci∥2.\|\operatorname{grad}f_i(X)\|_F\le\|C_iX\|_F\le\sqrt r\|C_i\|_2.

    Thus Ld=crmax⁡i∥Ci∥2L_d=c\sqrt r\max_i\|C_i\|_2, c≥1c\ge1, satisfies the clipping assumption and makes the radius explicitly data-dependent.

  • The theorem quantifies sufficient stepsize conditions. The experimental stepsizes are larger than these thresholds. The trajectory-based estimate in Remark E.20 bounds the tracker a posteriori; a local convergence theorem would also need an invariant region and verification of the remaining parameter conditions.

Q3. Meaning of “Inverse-Free”

“Inverse-Free” describes the outer optimization update. The method uses HVPs and the tangent-projector derivative without assembling the outer objective Hessian or solving a Newton system. Appendix F now states this meaning explicitly. For reduced objectives such as LRMC, the inner least-squares factorization remains part of objective evaluation and is reused by the correction.

After considering the author's response and other comments, I keep the score unchanged.

Thank you for reconsidering our submission. We appreciate your careful assessment.

Regarding your concern about isolating the correction’s benefit, the additional ten-node MPI comparison uses matching diminishing-schedule parameters for DPRSHCM and DPRSRM. DPRSHCM reduces time to the same joint-residual target by 35.5%–36.6% across four thresholds, from 100100 to 0.10.1. At 0.050.05, it succeeds in 10/10 runs, compared with 9/10 for DPRSRM. Together with the component ablations, these results provide direct evidence of the correction’s computational benefit.

Could you clarify which concern remains decisive for your assessment? This would help us address the remaining issue precisely.

Thank you again for your time and thoughtful feedback.

Reviewer_sd6qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies decentralized stochastic optimization on compact embedded submanifolds. Instead of using momentum with gradients at two points evaluated with the same sample, the paper uses a Hessian-vector product at the current point as an estimate, leading to a quadratic remainder in the ambient norm. The resulting algorithm, called the decentralized projected Riemannian stochastic Hessian-corrected momentum (DPRSHCM) method, is shown to achieve O~(ϵ−3/2)\widetilde{O}(\epsilon^{-3/2}) per-node oracle complexity to attain a stochastic ϵ\epsilon-stationary point when using diminishing stepsize and momentum parameters independent of the iteration budget. Using constant stepsize for a prescribed budget removes the logarithmic factor, and O(ε−1)O(\varepsilon^{-1}) complexity is attained when using exact local gradients and stochastic Hessian-vector products. Experiments on PCA and LRMC demonstrate the proposed methods' performance.

Critical strengths
  1. The paper proposes a well-motivated algorithm that combines gradient tracking and metric projection with a correction that takes into account changing tangent spaces. This avoids the two-point correction in prior works (Deng & Hu, 2025b; 2026), in particular the calculation of gradients at two points evaluated with the same sample. The idea is natural but is naturally motivated by the second-order expansion lemma (Lemma 5.1).

  2. The paper establishes three complexity results under various settings (Theorems 6.1 to 6.3) for the studied Riemannian setting.

Critical weaknesses
  1. Numerical comparisons are mainly in terms of the iteration counts. It would be more meaningful to compare the algorithms in terms of running time, as this will illustrate better the numerical benefits of the proposed second-order momentum over existing algorithms. While some comparisons on running time are provided in the appendix, there are no comparisons on the total running time of the algorithms to reach a common measure of accuracy.

  2. While theoretically interesting, one main practical issue is the range of class of problems where doing the calculations in DPRSHCM is more beneficial than simply doing one additional stochastic gradient calculation. DPRSHCM involves several additional calculations namely (i) the projection of the iterate difference onto the current tangent space in (5.3), (ii) the Hessian-vector product in (5.6), and (iii) the derivative of the tangent projector in (5.1).

Questions for authors
  1. Do the test problems satisfy the assumptions required in the convergence analysis? In Appendix A, it mentions that LRMC objective is only locally smooth, which is different from the requirement in Assumption 4.1.

  2. Can the authors propose concrete examples of classes of functions where the additional calculations required in DPRSHCM implementation can be efficiently done?

  3. Can the authors provide more detailed comparisons with other existing second-order methods in both Euclidean and Riemannian settings? Perhaps, a table comparing existing works in terms of problem settings, oracle assumptions, and convergence rates would be helpful to see the positioning of this work. Can the authors also clarify whether the additional theoretical contributions mainly concern extending Euclidean optimization results to Riemannian setting?

4 comments

Thank you for recognizing the motivation and geometric correction. We address running time, computational cost and the relation to existing methods below.

W1. Running time to a common accuracy

  • DPRSHCM reaches the common accuracy in less recorded time in both comparisons below. The target is the joint residual Gk2+ck2≤ϵG_k^2+c_k^2\le\epsilon; entries are mean times over ten runs, with 10/10 successful runs for each method in each row.
ProblemTarget ϵ\epsilonDPRSHCM time (s)DPRSRM time (s)
Synthetic PCA0.10.14.545.15
MovieLens-100K5510.2111.52

DPRSHCM uses diminishing steps in both rows. DPRSRM uses the budget-tuned constant setting on PCA and the diminishing setting on MovieLens. These are recorded cumulative times to the first qualifying checkpoint (Appendix G.4.2 and Table 11); Appendix G.1 states the threshold-selection rule.

  • Fixed-budget timings separately measure total computational cost. On ten-node synthetic LRMC, 20,000 iterations take 24.47 s versus 38.63 s, approximately 37% less time, with comparable final residuals (Table 12). The separate eight-node LRMC crossing times in Table 11 are estimates from total runtime.

W2. When the correction saves computation

  • Factorization reuse gives a concrete setting where the correction saves work. In variable-projection least squares, the gradient and correction share the current inner solution and factorization. A gradient at a second point requires a new factorization. The benefit follows when this saved work exceeds the HVP and geometric costs.

  • The measurements include the complete correction. On ten-node synthetic LRMC with matching diminishing parameters:

OperationTime per iteration (μ\mus)
DPRSHCM: complete correction140
DPRSRM: second gradient808

The complete correction includes the HVP and geometric operations. These figures compare the correction computations; total algorithm time is reported separately in Table 12. Appendix F and Tables 4 and 5 give the objective and geometric cost accounting.

Q1. Smoothness of the test objectives

The revised LRMC objective satisfies the stated smoothness assumptions. Section 7.2 uses Tikhonov-regularized variable projection, whose inner matrix satisfies XΩ⊤XΩ+λI⪰λIX_\Omega^\top X_\Omega+\lambda I\succeq\lambda I for every ambient XX, with λ>0\lambda>0. The reduced objective is smooth throughout the ambient space; its second and third derivatives are bounded on the compact convex hull of the Stiefel manifold. Appendix A verifies Assumptions 4.1 and 4.3 for this model and for PCA.

Q2. Objectives with efficient corrections

  • Variable-projection least squares permits reuse of the main gradient computation. The HVP reuses the inner solution and factorization. On synthetic LRMC, the measured complete correction costs 0.17 gradient evaluations.

  • Other standard objectives also admit inexpensive HVPs. For generalized linear and quadratic objectives, HVPs have approximately the same leading-order arithmetic cost as gradients; reverse-mode differentiation gives HVPs at a small constant multiple of gradient cost. Appendix F.1 and Table 4 describe these objective-dependent operations. Table 5 supplies the geometric costs needed to compare the complete correction with a second gradient.

Q3. Related methods and technical contribution

  • Appendix B now compares the relevant oracle models and guarantees. Table 2 covers decentralized methods; a further comparison includes Euclidean SGD-Hess and centralized Riemannian methods, with their accuracy criteria and oracle assumptions.

  • The geometric contribution is control of the full gradient variation. The Riemannian Hessian captures tangent variation; operator (5.1) also accounts for normal variation and yields the quadratic remainder in Lemma 5.1.

  • Decentralization requires joint control of correction, consensus and tracking. Neighbor mixing enters the correction displacement. The weighted second- and fourth-moment bounds in Lemmas E.9 and E.15 control this coupling and support the horizon-independent schedule. These arguments explain the additional contribution beyond the shared complexity exponent.

Thank you for the additional efforts for addressing my concerns. The following are further comments.

  1. The additional running time comparison to a common accuracy is helpful. My remaining concerns here include the selection of accuracy threshold and the seemingly modest improvement, which makes me doubt the practical significance. Can the authors comment on the choice of ε\varepsilon? Especially, ε\varepsilon is the set threshold for Gk2+ck2G_k^2+c_k^2, so the chosen values seem large. Can the authors comment on the runtime advantage over a range of ε\varepsilon’s with possibly smaller values?
  2. In relation to W2, the LRMC application is convincing. However, it is still not clear what broader problem classes the computational advantage of the HVP correction applies.

Thank you for your follow-up and for recognizing the LRMC application.

1. Accuracy thresholds and practical runtime gains

DPRSHCM reduces time to a common accuracy by 35.5%–36.6% across four thresholds. The additional ten-node synthetic LRMC comparison uses ten runs and K=20,000K=20{,}000 iterations per run, with the criterion Gk2+ck2≤εG_k^2+c_k^2\le\varepsilon.

Methodε=102\varepsilon=10^210101110−110^{-1}5×10−25\times10^{-2}
DPRSHCM (diminishing)5.9 s7.9 s11.1 s17.1 s21.3 s
DPRSRM (diminishing)9.3 s12.3 s17.2 s26.9 s9/10 runs
DPRSRM16.0 s18.9 s22.1 s3/10 runs0/10 runs
S-DPRGD11.3 s13.4 s15.9 s0/10 runs0/10 runs
DRSGD11.1 s13.2 s15.7 s0/10 runs0/10 runs

Times are means to the first qualifying checkpoint, including MPI communication, for targets reached in all ten runs. Other entries give success counts within the budget.

  • Runtime advantage across thresholds. The 35.5%–36.6% savings hold against diminishing DPRSRM with matching schedule parameters from ε=100\varepsilon=100 to 0.10.1. At 0.050.05, DPRSHCM succeeds in 10/10 runs, averaging 21.3 s, versus 9/10 runs for diminishing DPRSRM. This complements the earlier MovieLens comparison with evidence across multiple accuracy levels.

  • Threshold selection. The original threshold was the smallest value of the form a×10pa\times10^p, a∈{1,2,5}a\in\{1,2,5\}, reached by diminishing DPRSHCM in all ten runs within the budget. The table now shows performance across the threshold range.

  • Meaning of the absolute values. The thresholds reflect the normalization of the objective. In the ten-node synthetic LRMC experiment above, each node holds 96 columns. Equation (A.2) averages over nodes and sums the column losses within each node:

    \qquad \bar F(X)=\frac1{960}\sum_{i=1}^{10}\sum_{t=1}^{96}\phi_{i,t}^{\lambda}(X)=\frac{F(X)}{96},$$ where $\bar F$ is the mean loss per column. At the same iterate, ${grad}\bar F={grad}F/96$, so the squared gradient norm scales by $96^2$. Consequently, $G_k^2+c_k^2\le0.05$ guarantees $$\|\operatorname{grad}\bar F(\bar X_k)\|^2 \le\frac{0.05}{96^2}\approx5.43\times10^{-6}.$$ This quantifies the gradient accuracy on the mean-loss scale. All reported times use the original objective and joint stopping criterion, including the consensus error.

2. Broader problem classes with a computational advantage

The same reuse mechanism applies to weighted, logistic, and Poisson low-rank models. These fit the smooth reduced formulation

\left( \sum_{j\in\Omega_t}\ell(x_j^\top v;a_{jt}) +\frac{\lambda}{2}\lVert v\rVert^2 \right),$$ $$X\in\mathrm{St}(d,r),\qquad \lambda>0.$$ where $x_j^\top$ is row $j$ of $X$ and $\ell$ is smooth and convex in its first argument. The ridge term makes the inner problem strongly convex. - **Applications.** Weighted squared loss gives regularized weighted low-rank approximation [1]. Logistic loss $\log(1+e^\theta)-a\theta$ models binary observations; Poisson loss $e^\theta-a\theta$ models counts. The latter two give regularized models within the exponential-family and generalized low-rank frameworks [2,3]. - **Why computation can be saved.** The gradient requires the current inner solution. Implicit differentiation obtains the HVP through one linear solve and derivative products, reusing the solution and, when available, its Hessian factorization. This can replace another iterative inner optimization for the second gradient in logistic and Poisson models. Appendix F.1 of the revised manuscript, under “Gradients with an inner solve,” gives the general derivative formula in Eq. (F.2). - **Computational criterion and further experiments.** Savings arise when the incremental HVP and geometric costs are below the second-gradient cost. On LRMC, the complete correction takes **140 microseconds**, versus **808 microseconds** for the second gradient. We plan to evaluate weighted, logistic, and Poisson low-rank models using component timings and time to common accuracy to quantify the benefit in these applications. ### References [1] J. H. Manton, R. Mahony, and Y. Hua. The Geometry of Weighted Low-Rank Approximations. IEEE Transactions on Signal Processing, 51(2):500–514, 2003. [2] M. Collins, S. Dasgupta, and R. E. Schapire. A Generalization of Principal Component Analysis to the Exponential Family. NeurIPS, 2001. [3] M. Udell, C. Horn, R. Zadeh, and S. Boyd. Generalized Low Rank Models. Foundations and Trends in Machine Learning, 9(1):1–118, 2016.

I'm satisfied with all their responses now. I'll raise the score accordingly.

Comments on the paper

2 comments

Thank you for your questions on regularity, sampling and computational cost.

W1 and Q1. LRMC regularity and conditioning

  • Regularization supplies uniform smoothness throughout the required domain. The revised LRMC model uses λ=10−3\lambda=10^{-3} on synthetic data and λ=10−2\lambda=10^{-2} on MovieLens. Its inner matrix is uniformly positive definite for every ambient XX. Appendix A verifies the objective assumptions and resolves the rank issue.

  • Recorded singular values provide a conditioning check. On MovieLens, the smallest observed-row singular value exceeds 0.18 at every recorded checkpoint through step 2000 (Appendix G.3.2). The guarantee concerns the regularized objective. These checkpoints describe computed trajectories; an invariant-region theorem for the unregularized objective requires additional conditions.

W2 and Q2. Convergence under random reshuffling

  • Theorem 6.3 guarantees convergence under reshuffling. It gives O~(K−2/3)\widetilde{\mathcal O}(K^{-2/3}) for the prescribed epoch-wise diminishing schedule and O(K−2/3)\mathcal O(K^{-2/3}) for the horizon-dependent constant schedule. Gradient and correction use the same batch. Appendix H uses cancellation over complete epochs of equal-sized batches, under the stated regularity and parameter conditions.

  • Appendix G.1 specifies the experimental protocol alongside these conditions. The main diminishing runs update parameters each iteration; MovieLens uses 30 of 38 local columns before reshuffling. The theorem uses a uniform output index in the second half, whereas the tables report the last iterate. The guarantee covers the complete-epoch protocol; experiments assess the documented implementation.

W3 and Q3. Geometric information and cost

  • The geometric cost is explicit for six classical compact manifolds. Appendix F.2 and Table 5 give the projector derivatives and arithmetic counts. On Stiefel, the derivative costs O(dr2)O(dr^2) and reuses tangent-projection intermediates. The measured correction ratio κ\kappa includes these geometric operations; Table 2 lists them separately from stochastic objective-oracle calls.

  • For general manifolds, requirements depend on their representation. A smooth equality-constraint representation with full-rank Jacobian can require the constraint Jacobian, its directional derivative and linear solves.

W4 and Q4. Scope of the exact-gradient comparison

  • The paper focuses on stochastic gradient estimation, where the correction is useful. With exact local gradients, DPRGT is the natural baseline. The original exact-gradient result was a specialization with the same complexity order as deterministic gradient tracking. We have removed that result, its experimental configuration and the associated contribution claims.

  • The revised comparison tests the two stochastic correction mechanisms. With matching diminishing parameters on ten-node LRMC, DPRSHCM and DPRSRM attain comparable residuals in 24.47 s and 38.63 s, respectively, over 20,000 iterations with MPI communication included. Tables 2 and 13 report stochastic oracle and component-evaluation costs.

W5. Runtime gains and tuning protocol

  • The revised MPI comparison shows lower total time at comparable residuals. Both methods use matching diminishing parameters on ten-node synthetic LRMC for K=20000K=20000. Entries are mean ±\pm sample standard deviation over ten runs, with communication included (Table 12).
MethodFinal GK2+cK2G_K^2+c_K^2Total time (s)
DPRSHCM (diminishing)0.0379±0.00340.0379\pm0.003424.47±1.65\mathbf{24.47}\pm1.65
DPRSRM (diminishing)0.0405±0.00550.0405\pm0.005538.63±5.1338.63\pm5.13

This is approximately 37% less total time at the same iteration budget. At common accuracy on MovieLens, both methods reach residual 5 in all ten runs, taking 10.21 s and 11.52 s on average (Table 11).

  • The cost analysis explains when this advantage arises. LRMC permits reuse of the inner factorization; PCA has a different gradient-to-correction cost ratio. See Appendix F. Appendix G.1 identifies independently tuned configurations and the LRMC comparison with shared diminishing parameters.

Q5. Parameter selection and transfer

  • Parameter selection uses two quantities, (α0,τ0)(\alpha_0,\tau_0), with KHK_H determined by the schedule. We set KH=τ0/(2α0α1)K_H=\tau_0/(2\alpha_0\alpha_1), avoiding a separate KHK_H search. Appendix G.1 documents the pilot procedure and the comparisons with shared parameters.

  • PCA admits an explicit clipping radius. For fi(X)=−12tr⁡(X⊤CiX)f_i(X)=-\tfrac12\operatorname{tr}(X^\top C_iX), with loss normalization absorbed in CiC_i, Remark E.20 gives Ld≥rmax⁡i∥Ci∥2L_d\ge\sqrt r\max_i\|C_i\|_2.

  • The sensitivity tests quantify transfer within PCA. Appendix G.4.1 varies node counts, batch sizes, spectra and topologies with fixed parameters. The spectrum test identifies a setting requiring recalibration. Parameter selection retains the pilot procedure; transfer evidence concerns these PCA settings.

Dear Reviewers,

Thank you for your constructive feedback. Our revision addresses the main concerns by:

  • Establishing regularity for the regularized LRMC objective and adding convergence analysis under random reshuffling, with explicit sampling and parameter conditions.
  • Clarifying the geometric correction, its computational cost, and the contribution beyond the convergence rate.
  • Providing matched-schedule comparisons and component ablations. On synthetic LRMC, DPRSHCM achieves comparable residuals with approximately 37% less total MPI runtime.
  • Focusing the paper on fully stochastic optimization and removing the exact-gradient specialization.

We would appreciate your assessment of these revisions. If they resolve your concerns, we kindly ask you to reconsider your scores accordingly. We welcome further questions and would be happy to discuss any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.