- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SPADE-DFL, a derivative-free decentralized learning method combining component-memory single-point estimation with local linearized ADMM updates. Under uniform query-moment assumptions, a prescribed nonprivate schedule preserves an stationarity--consensus bound while using communication rounds. A separate private release mechanism clips accumulated local increments and adds Gaussian noise, yielding client-level privacy for the interactive transcript and a finite-horizon optimization bound. Experiments evaluate classification accuracy and communication costs.
Critical strengths
The paper connects local derivative-free updates, explicit network stability, and transcript-level privacy within a unified analysis that accounts for communication and function-query costs.
Critical weaknesses
W1. The communication experiment does not establish savings at matched optimization accuracy. Section 5.5 compares schedules at the same local-update budget, making the reduction in communication largely follow from . At the largest budget, the reported 96.8% saving accompanies a stationarity--consensus residual of 0.010392 rather than 0.000372, approximately 28 times larger. This does not contradict the asymptotic order result, but it limits the empirical evidence for communication efficiency at a fixed target accuracy. Please report transmitted scalars required to reach common residual thresholds, along with the associated query costs.
W2. The classification comparisons need stronger resource and privacy controls. Section 5.2 fixes 50 communication rounds but combines single-point and symmetric two-point estimators, while SPADE-DFL also reinitializes its component memory each round. Section 4.4 correctly counts these initialization queries, yet the main comparison does not establish matched total query budgets or comparable privacy levels across the methods. The centralized DP-SGD baseline additionally uses a different communication setting. Comparisons at matched function-query budgets and, where applicable, matched privacy definitions and budgets would better isolate the algorithmic improvement.
W3. The scope of the privacy guarantee for the experimental pipeline should be made more explicit. The theorem protects replacement of a fixed-size client dataset under public, data-independent initialization and schedules, whereas the classification pipeline includes PCA fitted on the training split. Section 5.3 explicitly conditions the reported privacy guarantee on frozen preprocessing and partitioning. If the PCA transform is learned from the same protected data, holding it fixed across neighboring datasets requires additional justification and does not automatically provide privacy for the full pipeline. Please specify whether preprocessing uses public or disjoint data, is separately privatized, or lies outside the protected-data definition.
Questions for authors
Q1. Corollary 1 requires the oracle moment bound to be uniform over the prescribed family of runs as the smoothing radius decreases. How is this verified for the classification losses and query noise used in the experiments? Please distinguish a theoretical assumption from a property established for the evaluated models.
Q2. The communication schedule in Section 5.5 refers back to Corollary 1, which gives a nonprivate rate, while the method also has a private variant. Please state explicitly in the experiment and Figure 10 caption whether clipping and release noise are disabled; if not, provide their settings and the applicable finite-horizon bound.
4 comments
Part 2/2
W3. PCA and the protected-data definition.
The reported guarantee covers the interactive training transcript conditional on a fixed preprocessing map and client partition; it does not establish end-to-end privacy for raw data.
(1) Status of preprocessing. Brain Tumor MRI uses PCA fitted on the training split. This fit has not been established as public, independent of the protected data, or separately private, and its privacy cost is outside the reported training-stage accounting. Freezing the fitted transform does not resolve this issue: replacing a raw client dataset could change the PCA basis and hence the representations of other clients' data.
(2) Supported privacy statement. Section 5.3 now explicitly limits the guarantee to replacement of one fixed-size preprocessed client dataset while the preprocessing map and partition remain fixed. An end-to-end guarantee would require independently fixed preprocessing or a private preprocessing mechanism whose privacy cost is composed with training, including any data-dependent normalization and partitioning.
Q1. Uniformity across smoothing radii and runs.
Appendix B now explicitly labels its bounded-loss example as sufficient, rather than verification of the experimental objectives, and points to the original Assumption 3 instead of restating it. If losses satisfy and centered query noise has conditional variance at most , with both constants independent of , then is uniform. Unconstrained cross-entropy does not automatically meet this condition.
For the main classification comparisons, MNIST and Fashion-MNIST use a sigmoid loss with quadratic regularization, Alzheimer's uses logistic loss with bounded regularization, and Brain Tumor MRI uses softmax cross-entropy with quadratic regularization. These objectives do not provide the global bounded-loss justification used in Appendix B, and we have not established a uniform query-moment bound across the prescribed runs.
The Alzheimer's oracle evaluates the loss directly. The other three implementations apply a mean-one Gaussian multiplier to the margin or logits and add centered Gaussian noise. Because the multiplier acts inside a nonlinear loss, the resulting perturbation is not generally conditionally centered relative to the nominal objective. Thus, both the required moment uniformity and the applicability of the centered-noise model need to be distinguished from the implementation settings.
By contrast, Figure 10 evaluates the synthetic squared sigmoid loss directly. It lies in at every queried point, giving a second-moment bound of independently of and . The synthetic study therefore supplies an instance satisfying the query-moment condition; the classification studies are empirical evaluations.
Q2. Whether Figure 10 is nonprivate.
Yes. Both Figure 10 schedules use the nonprivate setting of Corollary 1: clipping and Gaussian release noise are disabled, and the synthetic component loss is evaluated without query noise. Thus, the experiment concerns the nonprivate communication schedule rather than the private finite-horizon guarantee.
The new limitations paragraph records the remaining empirical gaps; no additional experiments or stronger pipeline privacy are claimed.
Thank you for the clarification. I will increase my score by one point.
We sincerely thank the reviewer for reconsidering our responses and updating their assessment. We appreciate your thoughtful feedback and constructive suggestions, which have helped us clarify the methodological contributions, strengthen the theoretical analysis, and improve the quality of the manuscript.
Thank you again for your time and consideration throughout the discussion.
Part 1/2
Thank you for distinguishing the communication theorem from the empirical evidence and identifying the preprocessing boundary of the privacy claim.
W1. Communication at matched optimization quality.
Your interpretation of Figure 10 is correct. At T=29,791, the schedule with τ=31 transmits 768,800 scalars versus 23,832,800 for τ=1, but its mean residual is 0.010392 versus 0.000372, approximately 27.94 times larger. The 96.8% saving is therefore a matched-update-budget result, not a matched-residual result.
Corollary 1 establishes SK=O(T−1/3+τ2/T) with uniform constants under its assumptions. It gives sufficient asymptotic round bounds of O(εstat−2) for the prescribed schedule and O(εstat−3) for its τ=1 specialization. Equal asymptotic order does not imply equal residuals at the tested budgets. We have revised the Figure 10 caption and Section 5.5 to identify this communication-residual trade-off and report the 27.94 residual ratio.
An exploratory post hoc comparison of existing 30-seed runs gives the following nearby residual levels.
Between these two measured points, the longer local stage uses approximately 3.9% fewer transmitted scalars and 3.1 times as many function queries. These are separate runs with T-dependent parameter schedules, and their residuals are not identical. The comparison is therefore descriptive and does not determine the costs of attaining a common threshold.
Section 5.5 and the revised Figure 10 caption identify the matched-update-budget comparison, and the limitations paragraph states that matched-residual comparisons remain unestablished.
W2. Resource and privacy controls.
We agree that 50 rounds are insufficient to equalize computation. The Table 1 SPADE-DFL implementation retains memory across rounds. The per-round initialization cost in Section 4.4 applies to the analyzed algorithm. Two-point estimates require two function calls per sampled component, and any baseline initialization/refresh calls must also be counted. Section 5.2 and the Table 1 caption now explicitly restrict the comparison to its method-specific query and privacy configurations.
Controlled comparisons require matched queries, partitions, topology, tuning, and privacy unit, adjacency, observation model and total (ϵ,δ). Record-level and client-replacement guarantees differ. Pooled-data DP-SGD remains a centralized reference.
The table reports recorded training function evaluations per run, summed across clients and including initialization and refresh calls. Ranges reflect fold-dependent counts for Alzheimer's disease and seed-dependent Poisson sampling for Brain Tumor DP-SGD; they are not confidence intervals. These counts make the computational differences explicit and do not imply matched-query comparisons.
DP-SGD and DP-FedAvg both target (8,10−5), but their accounting protects different units: individual records for DP-SGD and clients for DP-FedAvg under a trusted-server model. The SPADE-DFL budgets reported in Section 5.3 apply to the training transcripts from those runs, conditional on fixed preprocessing and partitioning. The privacy budgets in Section 5.3 are specific to those runs, so we report no DP budget for the SPADE-DFL entries in Table 1. The recorded counts show differences in both directions. For MNIST, SPADE-DFL uses 61,783 function calls per run, compared with 1,581 for each of the two single-point baselines and 1,224,500 for the LT-ADMM-VR adaptation. Thus the relative query cost depends on the reference method, even when all three decentralized comparisons use the same number of rounds. The accuracy values in Table 1 characterize the tested configurations, and the query totals make the cost differences explicit.