- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies communication-efficient distributed inference for manifold-valued parameters. An independent initial estimate establishes a common local coordinate system, in which workers compute scores, score second moments, and optionally Hessians. These summaries support an aggregated Newton estimator and a master-worker surrogate, with confidence regions constructed using sandwich covariance estimators. The theory separates the linearization remainder associated with the initial estimate from the additional error introduced by a single worker's Hessian, and establishes conditions for first-order equivalence to centralized estimation on the worker data. For PCA, the authors further recover the exact centralized worker-sample principal subspace from aggregated summaries. Experiments cover spherical intrinsic means, Grassmann subspaces, and the Dry Bean data.
Critical strengths
The theory clearly distinguishes statistical error, distributed approximation error, and covariance-estimation consistency, clarifying the requirements for valid inference.
The exact PCA recovery result has technical value and demonstrates how special algebraic structure can relax the initial-accuracy requirement of general one-step estimation.
The experimental comparisons are transparent: they use the same worker data, count no-report events as noncoverage, and discuss initialization costs and complete communication paths.
Critical weaknesses
-
The methodology builds substantially on classical one-step estimation, surrogate likelihood, and sandwich inference. The additional difficulties introduced by the common random chart, and the substantive advances beyond existing Euclidean results, could be articulated more sharply. Practical diagnostics indicating when the asymptotic approximation is adequate would also strengthen the contribution.
-
Finite-sample inferential reliability is a substantial concern. In one Gaussian setting in Table 1, nominal 95% coverage corresponds to empirical coverage of 95.6%, 84.6%, and 3.8% for the centralized, Newton, and surrogate procedures. These results do not contradict the asymptotic theorems, but show that the reporting checks do not ensure reliable coverage.
-
Practical efficiency depends on initial-sample allocation, parameter dimension, and the communication environment. Let denote the total sample size across all workers and the independent initial-sample size, with . The initial-sample observations are not reused in the pooled estimating equations, yielding a first-order variance factor of relative to an all-data benchmark. This factor approaches one when . Although the paper investigates allocation trade-offs, a practically implementable rule for selecting the initial-sample size remains unclear. Furthermore, transmitting Hessians and score second-moment matrices can make one-round procedures more expensive in bytes than multi-round alternatives, and the fixed-dimensional, i.i.d., local-regularity assumptions restrict applicability.
Questions for authors
-
Beyond the distinctions already discussed, can the authors identify the specific new inferential results relative to Jordan et al. (2019), the distributed PCA results relative to Fan et al. (2019), and the advances beyond manifold optimization in Saparbayeva et al. (2018)? Which steps require arguments beyond existing theory and a local coordinate transformation?
-
Can the worker summaries be used to diagnose one-round approximation error and indicate when additional communication is necessary or a confidence region should be withheld? In particular, can the adaptive covariance update in the appendix identify or mitigate the severe failures in Table 1?
3.Under a fixed total sample size , how should the initial-sample size be chosen? Is there a theoretically justified, practically implementable rule that does not require unknown population quantities?
- For the same inferential target, and accounting for initialization, transmitted bytes, network latency, and computation, under what conditions does the one-round approach outperform dedicated multi-round procedures?
5 comments
Response to Reviewer J3Qe (2/3)
Q1. The mathematical targets and communication protocols differ as follows.
- Jordan, Lee and Yang (2019), Theorems 1 and 3 and Corollary 4, establish Euclidean surrogate/one-step approximation and sandwich inference. Our conditional uniform Hessian and centered score-transfer lemmas verify the changing independent chart; the resulting Hölder rates control its approximation error, and the inference theorem maps studentized regions to the manifold.
- Fan, Wang, Wang and Zhu (2019) analyze aggregation of local eigenspace projectors, including variance and aggregation bias and unbiasedness under symmetric innovations. Our PCA result uses a different summary protocol to reconstruct the exact empirical worker-sample subspace and support central-centered covariance selection. Direct ambient second-moment aggregation is the exact point benchmark.
- Saparbayeva, Zhang and Lin (2018), Algorithm 1 and Theorem 2, study manifold surrogate optimization and distance to a centralized empirical minimizer under transported-Hessian regularity. Our target is the population parameter and its studentized mapped confidence region, with separate point-approximation and covariance-consistency conditions.
The random-chart verification supplies the additional uniformity needed for these inferential conclusions.
Q2. For PCA, the communicated summaries reconstruct the exact centralized worker-sample point . Whenever the reconstruction and chart checks succeed, they give the observable standardized discrepancy
The covariance-update rule centers the region at this recovered point. It retains the initial covariance when and the chart check passes, and otherwise requests score second moments at the recovered fit. With the prespecified , every path in the severe Gaussian and comparison triggers that additional round, giving coverage 0.956 and 0.924. For comparison, exact recentring with initial covariance gives 0.952 and 0.874. Keeping the original surrogate point with pooled covariance gives 0.038 and 0.014. The observed improvement thus involves correcting the center; heavy-tailed fitted-covariance coverage still falls below nominal.
For a general loss, a computable conservative bound is available when valid numerical Hessian Hölder envelopes are known throughout a declared chart ball . Set , , , , and
when , using otherwise. The check yields a unique anchor-local empirical minimizer and
=\frac{\widehat Lq^{1+\rho}}{(1+\rho)\lambda^{2+\rho}},\qquad \eta_{N,\mathrm{local}}\le\frac{\sqrt N b_N}{\sqrt{\lambda_{\min}(\widehat\Sigma_N)}}.$$ The full-chart check $a-\widehat Lr_0^\rho > 0$, or another localization argument for the designated benchmark, identifies this root with the centralized fit. Each worker supplies one additional scalar envelope average; scores/Hessians must be exact or accompanied by derivative-error control. The bound controls center approximation conditional on these checks. Its inputs include a known valid envelope evaluator, beyond merely assuming an envelope exists. The certificate follows because the Hessian is at least $\lambda I$ throughout the closed $r$-ball. On the sphere of radius $2q/\lambda$, its gradient points outward, so the empirical loss has an interior stationary minimizer; strong convexity gives $\\|u_N\\|\le q/\lambda$. Taylor expansion gives a remainder at most $\widehat L\\|u_N\\|^{1+\rho}/(1+\rho)$; multiplication by $A_N^{-1}$ yields a bound $(2/3)b_N\le b_N$. The full-chart inequality ensures uniqueness on the entire declared chart. If $q=0$ both points are zero; if $\widehat L=0$ Newton is exact. Asymptotic validity of adaptive PCA covariance selection follows from coupling: the two central-centered Wald statistics satisfy $W_A-W_C=o_p(1)$ and share a continuous chi-square limit. Their coverage indicators therefore differ with probability $o(1)$, so observable selection preserves the limiting coverage under the paper's assumptions.Response to Reviewer J3Qe (3/3)
Q3. A population-free sufficient allocation is
n=\lfloor(T-h_T)/m\rfloor,\qquad n_0=T-mn.$$ Choose the independent split before observing the data and require $n\ge1$. For fixed $\rho > 0$, $m=o(T)$ and a root-$n_0$ independent pilot, the rule gives $h_T\le n_0 < h_T+m$, $N\sim T$ and $$\frac{N}{n_0^{1+\rho}}\le\frac{(\log T)^{1+\rho}}{T^\rho}\to0,\qquad \frac{m}{n_0}\le\sqrt{m/T}\to0.$$ These imply the Newton and surrogate remainder conditions while $T/N\to1$. The rule is a sufficient asymptotic allocation using only $T,m$; finite-budget choices also depend on conditioning and the target basin. At $T=19200,m=32$, the formula gives $n_0=1952$ and $n=539$. This exact allocation was not simulated. The nearest evaluated allocation, $n_0=1920$, gives Newton/surrogate coverage 0.892/0.162; thus the asymptotic allocation rule is not a finite-budget coverage prescription. The fixed-budget Gaussian results are: | Independent pilot $n_0$ | Newton | Surrogate | Worker-centralized PCA | | ---: | ---: | ---: | ---: | | 960 | 0.804 | 0.040 | 0.944 | | 1920 | 0.892 | 0.162 | 0.940 | | 3840 | 0.924 | 0.382 | 0.938 | A larger pilot substantially improves approximation in this setting, while the worker sample shrinks. The surrogate remains sensitive to the master-worker Hessian even at the largest pilot. **Q4.** The comparison includes pilot fitting/collection, broadcasts, worker score-moment computation, coordinator factorization, and iterative surrogate solves. For symmetric 64-bit payloads with the master colocated, $D=20$, $k=3$, $m=32$ and $d=51$, the dense one-round, encoded one-round and dedicated two-round payloads are respectively 685,224, 395,808 and 395,808 bytes. A remote pilot adds $8n_0D$ bytes; this is 160,000 bytes for $n_0=1000$ and 614,400 for $n_0=3840$. Under the additive cost model $K=rL_u+bL_b+B/\beta+C$, where $L_u$ is upload latency, $L_b$ broadcast latency, $\beta$ bandwidth and $C$ computation time, a local-pilot encoded one-round method is faster when $$L_u > C_{\mathrm{encoded}}-C_{\mathrm{dedicated}}.$$ For the dense implementation, the condition becomes $$L_u > 289416/\beta+C_{\mathrm{dense}}-C_{\mathrm{dedicated}}.$$ With remote collection of the $n_0=3840$ pilot, the encoded cost difference is $614400/\beta+C_{\mathrm{encoded}}-C_{\mathrm{dedicated}}$; the saved worker-upload stage is offset by pilot collection. These cost comparisons use the same $N$ worker observations and the specified latency model. At fixed total $T$, dedicated PCA can also use observations otherwise allocated to the pilot. End-to-end distributed runtime and network latency were not measured, so the crossover depends on the deployment-specific computation costs $C$. Computation includes dense score-moment accumulation $O(nd^2)$, coordinator factorization $O(d^3)$ and pilot PCA $O(n_0D^2+D^3)$. The severe-design surrogate uses a pooled-Hessian initialization fallback that requires the corresponding messages or another request; the fixed-total comparison initializes from the master Hessian or zero. The matched statistical comparison informs whether these costs buy adequate inference: Gaussian exact-recentered initial covariance covers 0.952 versus fitted covariance 0.956, whereas the $t_8$ values are 0.874 versus 0.924. For a general nonlinear loss, the fixed-total sphere comparison at $T=4000,m=20$ gives one-/two-round surrogate coverage 0.948/0.960 and 0.972/0.970 for the isotropic design at initial sizes 20 and 400; the asymmetric values are 0.944/0.950 and 0.930/0.934. The second round adds a worker pass and an anchor broadcast, while these coverage differences are small. Thank you again for your valuable feedback. We hope our responses address your concerns and welcome any further questions or discussion.
Response to Reviewer J3Qe (1/3)
Thank you for your time and valuable comments.
W1. The contribution combines the conditional uniform Hessian law and centered score-transfer argument at the independent random chart, using an independent initial sample D0 (pilot), with the separate Hölder and master-Hessian error rates and mapped Wald inference. We also distinguish two observable diagnostics. For PCA, the existing summaries recover the exact worker-sample subspace and hence the point-approximation discrepancy. For a general loss with known domain-wide Hessian Hölder envelopes, a conservative local Newton-error bound follows from an empirical curvature and localization check, as detailed in Q2. These diagnostics assess center approximation; covariance calibration remains a separate requirement.
W2. The additional paired comparison applies exact recentring and covariance updating to the same severe design highlighted in the review.
The additional paired comparison uses the submitted severe Grassmann setting: D=20, k=3, eigengap 0.5, N=16000, m=32, and an independent pilot of size n0=1000. The 500 submitted seeds per law are reused, with common initial and worker observations for all methods. Coverage is for nominal 95% joint regions.
Exact recentring raises Gaussian coverage from 0.846 to 0.952 while retaining the initial covariance. In the heavy-tailed setting, moving to the fitted-chart covariance protocol raises coverage from 0.874 to 0.924. The adaptive rule requests the fitted-covariance round on every path in this severe setting. The dedicated and adaptive rows therefore coincide with the worker-centralized Wald region. The initial-covariance comparison isolates the center correction; the fitted-chart protocol additionally changes covariance evaluation and coordinates. The t8 fitted-covariance region still undercovers, and retaining the surrogate center while replacing its covariance gives coverage only 0.014.
The reporting checks assess numerical feasibility. Among returned severe Gaussian regions, coverage is 0.870 for Newton and 0.083 for the surrogate; reliable coverage also requires a sufficiently accurate center and consistent covariance. A numerical fit inherits the exact-minimizer theorem under local strong convexity and an optimization residual of order op(N−1/2). Numerical inheritance additionally requires derivative and linear-solve errors to preserve this accuracy and the reporting gates to succeed with probability tending to one. Fixed spectral thresholds require separation of the limiting spectra from those thresholds.
The submitted comparison at the different allocation T=19200, n0=3840, N=15360 also shows the finite-sample cost of updating: initial-covariance recentring covers 0.938 (Gaussian) and 0.906 (t5), versus 0.932 and 0.890 for the adaptive update. Initial-chart and fitted-chart regions differ in both covariance evaluation and coordinate representation. These results compare complete protocols across the stated settings.
W3. The allocation rule below uses only T,m. The cost comparison separates local and remotely collected initial samples, as in the submitted protocol comparison, and distinguishes dense from structure-aware messages. With a root-n0 independent pilot, the sufficient conditions are N=o(n01+ρ) and, for the surrogate, m=o(n0). The rule in Q3 satisfies both with T/N→1 under m=o(T) and fixed ρ>0. At finite budget, the worker-only first-order variance factor is T/N.
In the fixed-budget Gaussian comparison at T=19200,m=32, increasing the independent pilot from 960 to 3840 raises Newton coverage from 0.804 to 0.924 and surrogate coverage from 0.040 to 0.382. Worker-centralized coverage changes from 0.944 to 0.938 as fewer observations remain with the workers. This illustrates the competing approximation and sample-allocation effects.
For PCA, the structure-aware one-round payload is 395,808 bytes at D=20, k=3, m=32, equal to the dedicated two-round payload; the dense implementation uses 685,224 bytes. A remotely collected pilot adds 8Dn0 bytes and a collection stage; a pooled-Hessian initialization fallback requires those Hessian messages or another request. The statistical guarantees retain the stated fixed-dimensional, iid, local model.