- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In decentralized federated learning, clients train on their own data and periodically average their models with their neighbours, without a central server; this submission asks when such an averaging step may hurt the receiving client under heterogeneous data. It proposes NoRA, which injects Gaussian probe noise during local training to estimate each client's sensitivity. It then scores each neighbour edge by "sensitivity × model distance" (NPAR) and down-weights high-risk edges in the mixing matrix, moving the removed weight to the diagonal so that the matrix stays symmetric and doubly stochastic. The paper gives a convergence guarantee and reports results on MNIST, CIFAR-10 and CIFAR-100 under several data partitions and network topologies. Disclaimer: decentralized optimization is outside my main area of expertise. I could not assess the theoretical analysis or the positioning against related work in depth. My comments therefore focus on the experimental evidence that can be checked directly from the tables, and my confidence is low.
Critical strengths
- The mechanism is simple and cheap: each client sends only one extra scalar per round. The modified mixing matrix remains symmetric and doubly stochastic (Prop. 2), so the network average is preserved.
- NoRA has the best mean accuracy in all 12 settings of Table 1, and the numbers quoted in the text match the table.
- The paper is clearly written and states some of its limitations explicitly.
Critical weaknesses
- The ablation does not clearly isolate the contribution of NPAR: noise injection alone already accounts for most of the gain over DFedAvg (68.62 vs. 67.00), and NoRA is within one standard deviation of Random mask (69.40±0.96 vs. 68.73±0.51).
- The motivating diagnostic uses a different setup from the main experiments. Appendix A uses a static 10-regular graph, learning rate 0.1 and no injected noise (l.597–602), while the main experiments use time-varying random graphs, learning rate 0.065 and noise. It is unclear whether Fig. 1(b,c) carries over to the setting in which NoRA is actually evaluated.
Questions for authors
- How much of NoRA's advantage remains if DFedAvg, DFedSAM and DFedADMM are trained with the same injected noise?
- Corollary 2 suggests that attenuating edges worsens the mixing constant. Could the authors explain, intuitively or empirically, why attenuating high-risk edges nevertheless improves accuracy?
- What is the per-round computational overhead of the noise probing?
4 comments
R2. Diagnostic protocol and the trade-off between parameter mixing and aggregation risk (Critical Weakness 2; Question 2)
Diagnostic under the main training protocol (W2). Appendix A uses a static 10-regular graph to control for differences in client degree. We clarified that these diagnostic trajectories were generated without probe-noise injection into local updates; Gaussian probes were used only offline to estimate sensitivity for the edge-level risk analysis in Fig. 1(b,c).
To test the effect of risk-aware mixing under the main training protocol, we added Appendix D.6 and Fig. 8. The new experiments use CIFAR-10 with the Pathological partition, 100 clients, time-varying random graphs, an initial learning rate of 0.065 with a per-round decay factor of 0.9995, and probe-induced noise during local training. At each checkpoint along a NoRA trajectory, we apply base gossip and NPAR-guided mixing to the same locally updated models on the same graph and evaluate both on the same held-out local sets. The table below reports the mean positive increase in held-out local cross-entropy (nats), shown as mean standard deviation across three independent runs (Appendix D.6, Fig. 8(b)).
| Mixing rule | Round 50 | Round 100 | Round 200 | Round 500 |
|---|---|---|---|---|
| Base gossip | 0.77 0.11 | 0.57 0.02 | 0.49 0.02 | 0.43 0.08 |
| NPAR-guided mixing | 0.77 0.11 | 0.54 0.01 | 0.43 0.02 | 0.12 0.01 |
Both rules yield the same mean at round 50. NPAR-guided mixing yields smaller immediate loss increases at the three later checkpoints, with the largest reduction at round 500. This comparison shows that NPAR-guided mixing can reduce immediate aggregation harm under the main training protocol. The edge-level adverse averaging rates and AUROC in Fig. 1(b,c) were measured under the controlled protocol in Appendix A.
Why accuracy can improve with weaker parameter mixing (Q2). Corollary 2 gives a worst-case guarantee for the contraction of parameter disagreement. In the revision, we added Corollary 3 (Appendix C.2), which bounds the finite-window contraction factor under risk-aware mixing in terms of the base gossip contraction factor and the attenuation strength. In NoRA, attenuating a high-risk edge shifts weight back to the receiving model, reducing that neighbor’s contribution to the aggregation-induced displacement.
The new diagnostics in Appendix D.6 examine the trade-off between parameter consensus and the risk of adverse aggregation. At round 500, NoRA exhibits greater parameter disagreement than Noise only ( vs. , Fig. 8(a)). This indicates that NoRA does not necessarily promote stronger parameter consensus; instead, it prioritizes limiting the influence of potentially harmful neighbors. As shown in Fig. 8(b), risk-aware mixing reduces immediate aggregation-induced loss increases relative to base gossip. This observation is consistent with NoRA's higher mean test accuracy than Noise only (69.40% vs. 68.62%, Table 3), despite its greater parameter disagreement.
R3. Per-round computational overhead of noise probing (Question 3)
We added a component-level FLOP breakdown in Appendix D.2 using the same CIFAR-10 Dirichlet () setting as Table 5.
| NoRA component | Mean TFLOPs per round |
|---|---|
| Standard local training | 20.19 |
| Noise probing | 12.83 |
| NPAR computation and risk-aware mixing | 0.0081 |
| Total | 33.03 |
Noise probing adds 12.83 TFLOPs per round, equivalent to 63.5% of the standard local training cost and 38.8% of NoRA’s total per-round FLOPs. NoRA computes one vector–Jacobian product per local step without explicitly forming the full Jacobian. The resulting probe response is used in the local update (Eq. (3)) and reused for sensitivity estimation (Eq. (6)), requiring no additional product computation.
Thank you again for your careful assessment. We would like to provide two additional clarifications regarding the theoretical interpretation of our aggregation mechanism and its practical applicability.
Further clarification on Q2: Parameter consensus versus predictive performance.
A further distinction is important when interpreting the relationship between parameter consensus and predictive performance. Corollary 2 characterizes the contraction of inter-client parameter disagreement, rather than predictive accuracy under heterogeneous local objectives. A weaker contraction guarantee therefore does not imply lower predictive accuracy. Under data heterogeneity, stronger parameter consensus does not necessarily lead to more beneficial aggregation, since averaging with a dissimilar neighbor may increase the receiving model's local loss. NoRA instead prioritizes limiting such harmful interactions, as illustrated by the diagnostics in Fig. 8.
Further clarification on Q3: The computation–communication trade-off.
Beyond quantifying the additional probing cost, we would like to clarify the settings in which this computational trade-off can be beneficial. It is particularly relevant to communication-constrained decentralized networks, where model exchange can be a major system bottleneck.
Table 2 demonstrates substantial communication savings under heterogeneous data. With Dirichlet , NoRA reaches the target accuracy in 297 rounds, compared with 420 for DFedADMM, while reducing the relative communication cost from 0.90 to 0.66. Under the Pathological partition, NoRA requires only 89 rounds, compared with 116 for DFedAvg, reducing the relative communication cost from 1.00 to 0.81. These results correspond to 26.7% and 19.0% reductions in cumulative communication volume, respectively, relative to the fastest competing baseline in each setting.
In such settings, additional local computation may be preferable to repeated model communication, especially when communication is more costly than local processing. We acknowledge that this trade-off may be less favorable for compute- or energy-constrained devices, where reducing the probing overhead remains an important direction for future work.
We hope these additional clarifications address the remaining conceptual and practical concerns. Given the additional experimental evidence and explanations provided in the revision, we would greatly appreciate your reconsideration of the overall rating.
R1. Contribution of NPAR and comparison with baselines using local noise injection (Critical Weakness 1; Question 1)
We added the Base reference to Table 3 (Sec. 5.5) to measure the effect of noise injection under the same ablation protocol. We also evaluated variants of DFedAvg, DFedSAM, and DFedADMM with NoRA's probe-induced perturbation incorporated into their local updates (Appendix D.4, Table 7), examining how much of NoRA's accuracy advantage remains when the same perturbation is introduced into competing methods.
Controlled component comparisons (W1). Base uses NoRA’s local SGD configuration and fixed gossip, with noise injection and NPAR-guided aggregation disabled. Noise-only adds probe-induced noise to this reference and improves mean test accuracy from 67.96 ± 0.64% to 68.62 ± 0.78%, a gain of 0.66 percentage points. As stated in the original submission, we follow the original method designs and recommended hyperparameters where applicable, with necessary adaptations to our common experimental setup. To make these settings explicit, the revision lists the baseline optimizer hyperparameters in Appendix D.1 (Table 4) and specifies NoRA’s momentum in Sec. 5.1. In particular, DFedAvg uses zero momentum and an initial learning rate of 0.1 with a per-round decay factor of 0.998, whereas Base and Noise-only use NoRA’s configuration: momentum 0.5 and an initial learning rate of 0.065 with a decay factor of 0.9995. Thus, comparing Noise-only with DFedAvg’s 67.00% also reflects differences in optimizer settings. The matched Base comparison isolates the effect of enabling noise injection under the same optimizer and gossip settings.
With noise injection retained, NoRA reaches 69.40 ± 0.96%, compared with 68.73 ± 0.51% for Random mask, which matches the attenuation strength and the number of attenuated edges. The 0.67-percentage-point difference favors NPAR in mean accuracy, although we do not interpret it as evidence of statistical significance given the variability across runs. The extended ablations in Appendix D.3 (Table 6) also show that NoRA has higher mean accuracy than either single-signal variant in both new Dirichlet settings.
Noise injection in competing baselines (Q1). On CIFAR-10 with the Pathological partition, we add NoRA's probe-induced perturbation from Eq. (3) to the stochastic-gradient component at each local step, using the same scale (ρ=0.1). Each variant retains its baseline’s optimizer settings, aggregation rule, three seeds, and communication graph sequences. All methods use 100 clients and 500 communication rounds. Test accuracy is reported as mean ± standard deviation.
Noise injection improves the mean accuracy of all three baselines. Nevertheless, NoRA's 69.40% mean accuracy remains above all three variants with local noise injection. Table 7 quantifies these remaining gaps, while the controlled comparisons in Table 3 further distinguish the contributions of noise injection and NPAR-guided aggregation.