- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper investigates the stability and generalization properties of decentralized projected Riemannian stochastic gradient descent (DPRGD) on compact smooth embedded manifolds.
The authors derive stability-based generalization bounds by relating the expansion induced by manifold projection to consensus errors and showing that the additional geometric factor can be controlled through the accumulated squared step sizes.
Motivated by this analysis, they introduce DPRGD-ATC, an adapt-then-combine variant that averages locally updated models before projecting them onto the manifold. They establish tighter stability and generalization upper bounds than for DPRGD under identical assumptions and communication budgets. A local geometric construction further demonstrates that DPRGD-ATC can achieve strictly lower sample sensitivity. The authors also establish empirical stationarity guarantees for both algorithms.
The theoretical results are complemented by numerical experiments involving synthetic PCA problems on sphere, Stiefel, and Grassmann manifolds, together with an application to subspace learning on Fashion-MNIST.
Critical strengths
The paper addresses an important difficulty in extending Euclidean stability results to manifold-constrained optimization, namely the possible amplification of perturbations by the projection operator. Relating this amplification to consensus errors and controlling its accumulation through squared step sizes provides an interesting theoretical contribution. The analysis also makes explicit the respective influences of manifold geometry and network topology on generalization guarantees.
The proposed DPRGD-ATC algorithm is a natural extension of the adapt-then-combine strategy to manifold-constrained optimization. The authors establish that changing the order of local adaptation and combination reduces the consensus-dependent terms in the stability upper bound without increasing the per-iteration communication budget. The local construction relating the improvement in sample sensitivity to manifold curvature is also of interest, as it demonstrates that the benefit can extend beyond the comparison of theoretical upper bounds.
The additional stationarity analysis and the numerical experiments across several manifold geometries further strengthen the theoretical investigation.
Critical weaknesses
The main theoretical result establishes that DPRGD-ATC admits a tighter stability-based generalization upper bound than DPRGD. However, improving an upper bound does not necessarily imply a corresponding improvement in actual algorithmic stability or generalization performance.
This distinction is particularly important in view of the numerical results. Although the reported stability upper bounds decrease substantially in the synthetic experiments, the observed differences in sample sensitivity and generalization are much smaller and generally not statistically significant. On Fashion-MNIST, the experiments with identical parameters also show nearly indistinguishable generalization performance between the two algorithms. The additional experiments with a fixed DPRGD baseline provide statistically significant improvements in the pooled absolute generalization gap, but their practical magnitude remains extremely small.
Overall, although the theoretical improvement is mathematically interesting, the numerical evidence does not convincingly establish that it translates into a practically meaningful improvement in generalization performance.
--
The main comparison between DPRGD and DPRGD-ATC relies on relatively restrictive assumptions, including sufficiently small deterministic step sizes and a strong mixing condition requiring the network contraction factor to satisfy (sigma<1/2).
The latter condition may entail substantial communication costs on poorly connected networks, raising questions about the practical scalability of the theoretical guarantees.
Furthermore, the Fashion-MNIST experiments use step sizes that fall outside the sufficient conditions of the theoretical analysis, while several network configurations do not satisfy the strong mixing requirement. Although these experiments demonstrate that the algorithms can operate beyond the theoretically covered regime, they do not establish whether the guaranteed ordering of the stability upper bounds remains valid or informative in these more practical settings.
A more systematic investigation of the influence of step size, network connectivity, and communication costs would help clarify the conditions under which DPRGD-ATC provides a significant advantage.
Questions for authors
The theoretical stability bounds show a substantial improvement for DPRGD-ATC, whereas the empirical differences in sample sensitivity and generalization are generally very small. Could the authors explain this discrepancy and provide further insight into the tightness of the bounds? Under which conditions should one expect the theoretical advantage to translate into a meaningful improvement in generalization performance?
2 comments
Part 2/2
5. Weakness: Fashion-MNIST and theorem conditions
- Fashion-MNIST evaluates empirical performance beyond the sufficient step thresholds, using fixed multi-round communication. It retains all 784 pixels, , 4096 training images and . Separate development data select . We compute , charge neighbor exchanges, and check projection validity numerically.
| Agents | Rounds | Main step threshold | |
|---|---|---|---|
| 4 | 2 | 0.2500 | |
| 8 | 2 | 0.7286 | mixing fails |
| 16 | 2 | 0.9253 | mixing fails |
| 32 | 2 | 0.9809 | mixing fails |
| 32 | 144 | 0.2490 |
- The step exceeds both main and weaker individual thresholds. Appendix G.3 gives the status; the experimental-details attachment retains weaker-mixing thresholds. These runs test high-dimensional learning and network effects. Stiefel and circle experiments separately verify bounds and local loss responses within the certified regime. Steps and communication remain fixed during each run.
6. Weakness: Parameter, network and resource assessment
- Common-parameter comparisons quantify network dependence; separate tuning uses independent validation. At , , twenty shared Stiefel pairs give (Table 19):
| B bound | A bound | Reduction | Total rounds | ||
|---|---|---|---|---|---|
| 2 | 1/4 | 0.0310 | 0.0228 | 26.52% | 1024 |
| 3 | 1/8 | 0.0258 | 0.0221 | 14.19% | 1536 |
| 4 | 1/16 | 0.0248 | 0.0220 | 11.22% | 2048 |
All satisfy the conditions. More communication improves both bounds and reduces ATC's relative advantage. ATC has lower consensus error; sensitivity and absolute-gap intervals contain zero. Constant and square-summable comparisons share data, sampling prefixes and initialization.
- Independent tuning quantifies resource savings and statistical changes. The added synthetic search evaluates 50 configurations (steps up to 10 and deterministic decays), screens eight on fresh data, and freezes one before 100 untouched validation pairs. Appendix G.4 retains selection rules, results and costs; the attachment gives all search records.
| Evaluation | Gap reduction | Difference A-B | Paired 95% interval |
|---|---|---|---|
| Fashion-MNIST, pooled | 0.029% | ||
| Synthetic, 100 pairs | 0.5267% |
- Both configurations halve communication. Fashion-MNIST retains baseline steps and length; over 50 new pairs per node count, the smallest mean progress ratio across both risks and all node counts is 100.001% (Appendix F). Synthetic progress is 99.476%: its 0.524% decrease accompanies a population-risk increase of . This tuned configuration has and is evaluated empirically. These results describe joint tuning effects on resources, learning and statistics.
7. Question: Tightness and conditions for actual gains
-
Proposition 4 gives sufficient conditions for strict local sensitivity improvement; diagnostics quantify conservatism on tested trajectories. Worst-case tightness requires a matching lower bound for the same uniform quantity. Projection and loss bounds cover all perturbation directions.
-
Geometry and data jointly yield the local effect. With nonzero normal curvature, opposing outer-node samples, a middle-node replacement along a unit tangent , common initialization and exact averaging,
=\frac{\eta^2}{3}\|\mathrm{II}_x(v,v)\|^2+o(\eta^2).$$ The second fundamental form $\mathrm{II}$ gives a positive coefficient. Small admissible steps and replacements yield strict finite-loss ordering (E1). -
Larger average benefits would require these effects to occur often under the data distribution. Different local gradients, larger projection distances and replacements along expanding directions are hypotheses to examine at comparable learning quality; Proposition 4 proves the local effect.
Part 1/2
Thank you for asking about statistical gains, assumptions and costs.
1. Weakness: Bounds and actual statistical performance
Theorem 3 orders uniform bounds; Proposition 4 proves a strict actual local effect under specified geometry and data conditions. For each manifold in the stated class, it constructs an instance with lower ATC replacement sensitivity (Section 4.1).
Both orderings agree in an admissible circle experiment. At η=η∗=0.004238021180, replacement size 10−3, Appendix G.2 gives:
Summation uses 80-digit arithmetic. The ordering agrees with the derivative ratio 3/(1+2/1+η2)>1 and its quadratic step scale.
2. Weakness: Magnitude and practical significance
Geometric improvements are clear at comparable learning quality; matched-parameter average-gap intervals contain zero. Common decaying steps satisfy the theorem, with complete bounds 0.0133 versus 0.0117, a 12.22% reduction below the loss range. Consensus error falls 78.98% and maximum projection distance 56.13%, with essentially equal risk decreases. The absolute-gap difference interval is [−8.392,2.260]×10−9 (Appendix G.1).
Observed distances explain one source of the difference in effect sizes.
Observed products are close to one; actual sensitivity also depends on direction, loss and data distribution. Fashion-MNIST consensus ratios A/B are 0.208, 0.499, 0.686, 0.803 at 4, 8, 16, 32 agents, with comparable risks. E6 reports separate tuning results.
3. Weakness: Small steps and strong mixing
Both DPRGD and ATC use small-step and mixing safeguards to control manifold projection. These keep update inputs and final means in a tube with unique projection and controlled expansion. Baseline DPRGD has analogous safeguards in Lemmas 5.5-5.6, with contraction 2λk. We control both neighboring-data trajectories for every sampling sequence; ATC uses the same admissible configuration and budget. Deng and Hu, Section 5.2
The main threshold also gives bound ordering. For σ<1/2,
=a(1-\sigma^2)\delta_t^2+C_1(1-2\sigma)\delta_t\eta_tG\ge0, \quad a=\frac1{2\tau}+\frac{C_1}{2}.$$ Here $\delta_t$ bounds node distance, $G$ bounds gradients, and $\tau,C_1$ are geometric constants.Individual guarantees allow weaker mixing: Appendix A.10 uses Lhσ<1, Lh=(1−h/(4τ))−1, and its own step threshold. Every fixed σ<1 permits sufficiently small h and steps. Main bound ordering additionally uses the nonnegative difference above; decreasing steps alone leaves σ unchanged.
4. Weakness: Communication on poorly connected networks
The minimum is 72 rounds, an analytical calculation; training uses 2 or 144. The stronger settings cost 36 or 72 times the two-round case. More mixing permits a larger certified step; fixed target contraction on this ring requires k=Θ(m2).
ATC adds no communication within a setting. Both methods exchange equal-sized vectors and use equal gradient counts. Table 13 gives costs; Table 20 gives conditions. Single-server simulations report algorithmic costs, not measured distributed speedup.