acceptodds
Under review as a conference paper at ICLR 2027

Neighbor Averaging as Perturbation: Risk-Aware Decentralized Federated Learning via Noise-Probed Sensitivity

Abstract

Decentralized federated learning (DFL) enables collaborative training by repeatedly exchanging and averaging locally updated models. Under heterogeneous data, however, an update beneficial to its local client may not benefit a receiving neighbor when averaged. Existing topology-driven or learning-adaptive schemes regulate collaboration structure or influence, while the state-dependent effect of a specific averaging operation on the receiving model remains insufficiently characterized. We introduce a perturbation view of neighbor averaging, treating each candidate average as a finite parameter perturbation along the neighbor-model difference. We derive a local output-drift bound showing that the resulting effect depends jointly on the receiving model's sensitivity and the induced displacement. Motivated by this bound, we propose NoRA (Noise-Probed Risk-Aware Averaging) and the NPAR score, which combines noise-probed sensitivity with pairwise model displacement to estimate averaging risk. NoRA attenuates high-risk interactions with only one additional scalar per client beyond standard model exchange. NoRA preserves symmetric doubly stochastic mixing under state-dependent, time-varying weights and retains the standard leading convergence rate under finite-window spectral contraction. Experiments across diverse datasets, heterogeneity settings, and network topologies validate its effectiveness. On CIFAR-10 with Dirichlet , NoRA improves mean global test accuracy by 1.7% over the strongest baseline and reaches the target accuracy with 29.3% fewer communication rounds than the fastest competing method.

Primary area:
optimization
Keywords:
decentralized federated learning, decentralized optimization, data heterogeneity, adaptive model averaging, risk-aware aggregation, time-varying networks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xFwTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In decentralized federated learning, clients train on their own data and periodically average their models with their neighbours, without a central server; this submission asks when such an averaging step may hurt the receiving client under heterogeneous data. It proposes NoRA, which injects Gaussian probe noise during local training to estimate each client's sensitivity. It then scores each neighbour edge by "sensitivity × model distance" (NPAR) and down-weights high-risk edges in the mixing matrix, moving the removed weight to the diagonal so that the matrix stays symmetric and doubly stochastic. The paper gives a convergence guarantee and reports results on MNIST, CIFAR-10 and CIFAR-100 under several data partitions and network topologies. Disclaimer: decentralized optimization is outside my main area of expertise. I could not assess the theoretical analysis or the positioning against related work in depth. My comments therefore focus on the experimental evidence that can be checked directly from the tables, and my confidence is low.

Critical strengths
  1. The mechanism is simple and cheap: each client sends only one extra scalar per round. The modified mixing matrix remains symmetric and doubly stochastic (Prop. 2), so the network average is preserved.
  2. NoRA has the best mean accuracy in all 12 settings of Table 1, and the numbers quoted in the text match the table.
  3. The paper is clearly written and states some of its limitations explicitly.
Critical weaknesses
  1. The ablation does not clearly isolate the contribution of NPAR: noise injection alone already accounts for most of the gain over DFedAvg (68.62 vs. 67.00), and NoRA is within one standard deviation of Random mask (69.40±0.96 vs. 68.73±0.51).
  2. The motivating diagnostic uses a different setup from the main experiments. Appendix A uses a static 10-regular graph, learning rate 0.1 and no injected noise (l.597–602), while the main experiments use time-varying random graphs, learning rate 0.065 and noise. It is unclear whether Fig. 1(b,c) carries over to the setting in which NoRA is actually evaluated.
Questions for authors
  1. How much of NoRA's advantage remains if DFedAvg, DFedSAM and DFedADMM are trained with the same injected noise?
  2. Corollary 2 suggests that attenuating edges worsens the mixing constant. Could the authors explain, intuitively or empirically, why attenuating high-risk edges nevertheless improves accuracy?
  3. What is the per-round computational overhead of the noise probing?
4 comments

R1. Contribution of NPAR and comparison with baselines using local noise injection (Critical Weakness 1; Question 1)

We added the Base reference to Table 3 (Sec. 5.5) to measure the effect of noise injection under the same ablation protocol. We also evaluated variants of DFedAvg, DFedSAM, and DFedADMM with NoRA's probe-induced perturbation incorporated into their local updates (Appendix D.4, Table 7), examining how much of NoRA's accuracy advantage remains when the same perturbation is introduced into competing methods.

Controlled component comparisons (W1). Base uses NoRA’s local SGD configuration and fixed gossip, with noise injection and NPAR-guided aggregation disabled. Noise-only adds probe-induced noise to this reference and improves mean test accuracy from 67.96 ±\pm 0.64% to 68.62 ±\pm 0.78%, a gain of 0.66 percentage points. As stated in the original submission, we follow the original method designs and recommended hyperparameters where applicable, with necessary adaptations to our common experimental setup. To make these settings explicit, the revision lists the baseline optimizer hyperparameters in Appendix D.1 (Table 4) and specifies NoRA’s momentum in Sec. 5.1. In particular, DFedAvg uses zero momentum and an initial learning rate of 0.1 with a per-round decay factor of 0.998, whereas Base and Noise-only use NoRA’s configuration: momentum 0.5 and an initial learning rate of 0.065 with a decay factor of 0.9995. Thus, comparing Noise-only with DFedAvg’s 67.00% also reflects differences in optimizer settings. The matched Base comparison isolates the effect of enabling noise injection under the same optimizer and gossip settings.

With noise injection retained, NoRA reaches 69.40 ±\pm 0.96%, compared with 68.73 ±\pm 0.51% for Random mask, which matches the attenuation strength and the number of attenuated edges. The 0.67-percentage-point difference favors NPAR in mean accuracy, although we do not interpret it as evidence of statistical significance given the variability across runs. The extended ablations in Appendix D.3 (Table 6) also show that NoRA has higher mean accuracy than either single-signal variant in both new Dirichlet settings.

Noise injection in competing baselines (Q1). On CIFAR-10 with the Pathological partition, we add NoRA's probe-induced perturbation from Eq. (3) to the stochastic-gradient component at each local step, using the same scale (ρ=0.1\rho=0.1). Each variant retains its baseline’s optimizer settings, aggregation rule, three seeds, and communication graph sequences. All methods use 100 clients and 500 communication rounds. Test accuracy is reported as mean ±\pm standard deviation.

MethodOriginal accuracy (%)With noise (%)Gain from noise (percentage points)NoRA’s lead after adding noise (percentage points)
DFedAvg67.00 ±\pm 0.9267.58 ±\pm 0.48+0.581.82
DFedSAM60.22 ±\pm 0.9760.97 ±\pm 0.67+0.758.43
DFedADMM54.96 ±\pm 0.6255.59 ±\pm 0.34+0.6313.81

Noise injection improves the mean accuracy of all three baselines. Nevertheless, NoRA's 69.40% mean accuracy remains above all three variants with local noise injection. Table 7 quantifies these remaining gaps, while the controlled comparisons in Table 3 further distinguish the contributions of noise injection and NPAR-guided aggregation.

R2. Diagnostic protocol and the trade-off between parameter mixing and aggregation risk (Critical Weakness 2; Question 2)

Diagnostic under the main training protocol (W2). Appendix A uses a static 10-regular graph to control for differences in client degree. We clarified that these diagnostic trajectories were generated without probe-noise injection into local updates; Gaussian probes were used only offline to estimate sensitivity for the edge-level risk analysis in Fig. 1(b,c).

To test the effect of risk-aware mixing under the main training protocol, we added Appendix D.6 and Fig. 8. The new experiments use CIFAR-10 with the Pathological partition, 100 clients, time-varying random graphs, an initial learning rate of 0.065 with a per-round decay factor of 0.9995, and probe-induced noise during local training. At each checkpoint along a NoRA trajectory, we apply base gossip and NPAR-guided mixing to the same locally updated models on the same graph and evaluate both on the same held-out local sets. The table below reports the mean positive increase in held-out local cross-entropy (nats), shown as mean ±\pm standard deviation across three independent runs (Appendix D.6, Fig. 8(b)).

Mixing ruleRound 50Round 100Round 200Round 500
Base gossip0.77 ±\pm 0.110.57 ±\pm 0.020.49 ±\pm 0.020.43 ±\pm 0.08
NPAR-guided mixing0.77 ±\pm 0.110.54 ±\pm 0.010.43 ±\pm 0.020.12 ±\pm 0.01

Both rules yield the same mean at round 50. NPAR-guided mixing yields smaller immediate loss increases at the three later checkpoints, with the largest reduction at round 500. This comparison shows that NPAR-guided mixing can reduce immediate aggregation harm under the main training protocol. The edge-level adverse averaging rates and AUROC in Fig. 1(b,c) were measured under the controlled protocol in Appendix A.

Why accuracy can improve with weaker parameter mixing (Q2). Corollary 2 gives a worst-case guarantee for the contraction of parameter disagreement. In the revision, we added Corollary 3 (Appendix C.2), which bounds the finite-window contraction factor under risk-aware mixing in terms of the base gossip contraction factor and the attenuation strength. In NoRA, attenuating a high-risk edge shifts weight back to the receiving model, reducing that neighbor’s contribution to the aggregation-induced displacement.

The new diagnostics in Appendix D.6 examine the trade-off between parameter consensus and the risk of adverse aggregation. At round 500, NoRA exhibits greater parameter disagreement than Noise only (6.49×10−76.49\times10^{-7} vs. 2.97×10−72.97\times10^{-7}, Fig. 8(a)). This indicates that NoRA does not necessarily promote stronger parameter consensus; instead, it prioritizes limiting the influence of potentially harmful neighbors. As shown in Fig. 8(b), risk-aware mixing reduces immediate aggregation-induced loss increases relative to base gossip. This observation is consistent with NoRA's higher mean test accuracy than Noise only (69.40% vs. 68.62%, Table 3), despite its greater parameter disagreement.

R3. Per-round computational overhead of noise probing (Question 3)

We added a component-level FLOP breakdown in Appendix D.2 using the same CIFAR-10 Dirichlet (α=0.3\alpha=0.3) setting as Table 5.

NoRA componentMean TFLOPs per round
Standard local training20.19
Noise probing12.83
NPAR computation and risk-aware mixing0.0081
Total33.03

Noise probing adds 12.83 TFLOPs per round, equivalent to 63.5% of the standard local training cost and 38.8% of NoRA’s total per-round FLOPs. NoRA computes one vector–Jacobian product per local step without explicitly forming the full Jacobian. The resulting probe response is used in the local update (Eq. (3)) and reused for sensitivity estimation (Eq. (6)), requiring no additional product computation.

Thank you again for your careful assessment. We would like to provide two additional clarifications regarding the theoretical interpretation of our aggregation mechanism and its practical applicability.

Further clarification on Q2: Parameter consensus versus predictive performance.

A further distinction is important when interpreting the relationship between parameter consensus and predictive performance. Corollary 2 characterizes the contraction of inter-client parameter disagreement, rather than predictive accuracy under heterogeneous local objectives. A weaker contraction guarantee therefore does not imply lower predictive accuracy. Under data heterogeneity, stronger parameter consensus does not necessarily lead to more beneficial aggregation, since averaging with a dissimilar neighbor may increase the receiving model's local loss. NoRA instead prioritizes limiting such harmful interactions, as illustrated by the diagnostics in Fig. 8.

Further clarification on Q3: The computation–communication trade-off.

Beyond quantifying the additional probing cost, we would like to clarify the settings in which this computational trade-off can be beneficial. It is particularly relevant to communication-constrained decentralized networks, where model exchange can be a major system bottleneck.

Table 2 demonstrates substantial communication savings under heterogeneous data. With Dirichlet α=0.3\alpha=0.3, NoRA reaches the target accuracy in 297 rounds, compared with 420 for DFedADMM, while reducing the relative communication cost from 0.90 to 0.66. Under the Pathological partition, NoRA requires only 89 rounds, compared with 116 for DFedAvg, reducing the relative communication cost from 1.00 to 0.81. These results correspond to 26.7% and 19.0% reductions in cumulative communication volume, respectively, relative to the fastest competing baseline in each setting.

In such settings, additional local computation may be preferable to repeated model communication, especially when communication is more costly than local processing. We acknowledge that this trade-off may be less favorable for compute- or energy-constrained devices, where reducing the probing overhead remains an important direction for future work.

We hope these additional clarifications address the remaining conceptual and practical concerns. Given the additional experimental evidence and explanations provided in the revision, we would greatly appreciate your reconsideration of the overall rating.

Reviewer_fcL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates decentralized federated learning (DFL) under heterogeneous data distributions, focusing on the potential adverse effects of neighbor model averaging on receiving clients. The authors introduce a perturbation-based perspective that interprets neighbor averaging as a parameter displacement and analyzes its impact on model outputs through Jacobian sensitivity. Motivated by this analysis, the paper proposes NoRA (Noise-Probed Risk-Aware Averaging), which estimates local model sensitivity using Gaussian noise probing and constructs the NPAR score by combining receiver sensitivity with pairwise model displacement. Based on this score, NoRA dynamically attenuates potentially harmful neighbor interactions while preserving symmetric doubly stochastic mixing and the network-wide model average. The authors further establish non-convex convergence guarantees under state-dependent, time-varying communication weights, retaining the standard leading convergence rate under suitable spectral contraction conditions. Finally, experiments on MNIST, CIFAR-10, and CIFAR-100 across different data heterogeneity settings and network topologies are conducted to evaluate predictive accuracy, convergence behavior, communication efficiency, and the contributions of individual algorithmic components. The reported results show improvements over several representative decentralized learning baselines.

Critical strengths

1. Well-Motivated Algorithm Design with Theoretical Support.

The paper presents a coherent algorithmic framework for addressing potentially harmful neighbor averaging in decentralized federated learning. By interpreting neighbor averaging as a parameter perturbation, the authors establish a connection between model sensitivity, pairwise parameter displacement, and aggregation-induced output changes (Proposition 1, Sec. 3.3). This analysis provides a reasonable theoretical motivation for the proposed NPAR score and its integration into risk-aware mixing. Furthermore, the algorithm preserves symmetric doubly stochastic aggregation (Proposition 2), while Theorem 1 and Corollary 1 establish non-convex convergence guarantees under state-dependent, time-varying mixing. The theoretical analysis is systematically developed, with detailed proofs provided in Appendices B and C. These contributions are valuable because they demonstrate that the proposed adaptive aggregation mechanism is supported by a clear mathematical formulation and can retain standard convergence guarantees rather than relying solely on empirical observations.

2. Comprehensive Experimental Evaluation and Consistent Empirical Improvements.

The paper demonstrates considerable experimental effort through evaluations across three benchmark datasets (MNIST, CIFAR-10, and CIFAR-100), multiple IID and non-IID data partitions, and several communication topologies. Table 1 reports consistent improvements in mean test accuracy over six representative decentralized learning baselines across all twelve evaluated settings. Table 2 further demonstrates reductions in communication rounds and cumulative communication costs, particularly under heterogeneous data distributions. Beyond overall performance comparisons, the authors provide component ablations (Table 3), hyperparameter sensitivity analyses (Fig. 5 and Appendix D.2), and an offline diagnosis of the proposed NPAR risk indicator (Fig. 1 and Appendix A). These evaluations provide useful empirical evidence for the effectiveness of the proposed method and demonstrate a substantial effort to examine its behavior from multiple perspectives.

Critical weaknesses

1. Limited Methodological Novelty and Theoretical Distinctiveness.

Although the paper presents a coherent method supported by theoretical analysis, its core contribution appears relatively incremental given the extensive research on adaptive aggregation in federated learning. Two concerns particularly limit its methodological and theoretical significance:

  • Insufficient differentiation from existing aggregation approaches. Adaptive aggregation has been extensively investigated in both centralized and decentralized federated learning, with existing methods adjusting aggregation weights based on model similarity, parameter divergence, gradient relationships, or learning-dependent signals. However, the paper does not sufficiently explain which fundamental limitations of these approaches motivate the proposed NPAR formulation, or why combining Jacobian sensitivity with pairwise model displacement provides a substantially different solution. Moreover, the related work discussion focuses primarily on decentralized approaches while overlooking relevant centralized FL research, such as L-DAWA [1], which also considers divergence-aware model aggregation under heterogeneous data. Although these methods operate under different communication architectures, their underlying motivations and aggregation principles remain relevant. Without a clearer comparison with existing aggregation strategies, the novelty and necessity of the proposed approach remain insufficiently established.

  • Limited theoretical characterization beyond standard convergence guarantees. Proposition 1 provides a mathematical motivation for NPAR through an upper bound on aggregation-induced output drift, while Theorem 1 and Corollary 1 establish non-convex convergence under suitable spectral contraction conditions. These results are useful, but the theoretical contribution primarily establishes that NoRA retains the standard convergence rate rather than demonstrating improved optimization properties. In particular, the paper does not provide theoretical comparisons with representative existing aggregation methods or characterize whether NPAR-guided mixing leads to tighter convergence bounds, reduced heterogeneity-induced errors, or improved consensus behavior. Furthermore, the output-drift bound does not directly establish that larger NPAR values correspond to harmful averaging in terms of local loss. Consequently, the current theoretical analysis does not sufficiently explain why the proposed risk-aware mechanism should outperform existing aggregation strategies.

2. Insufficient Experimental Evidence for Practical Efficiency and Mechanistic Understanding.

Although the paper includes extensive benchmark comparisons, several important evaluations are still missing. These limitations are particularly relevant because stronger empirical evidence is needed to establish the practical value and necessity of the proposed design.

  • Incomplete evaluation of computational overhead and end-to-end efficiency. Table 2 demonstrates reductions in communication rounds and cumulative communication volume, but NoRA additionally computes Jacobian-related probe responses during local optimization (Eq. (3)). This introduces computational operations beyond conventional gradient updates, yet the paper does not quantify the additional computation, per-round runtime, GPU memory overhead, or total wall-clock training time. Therefore, it remains unclear whether the reported communication savings translate into improved overall training efficiency, particularly when local computation is a significant system bottleneck.

  • Insufficient ablation evidence for the necessity of the core components. Although Table 3 includes several component ablations, the improvements over simpler alternatives are relatively modest. For example, NoRA exceeds Sensitivity Only and Displacement Only by only 0.44 and 0.58 percentage points, respectively, with standard deviations of comparable magnitude. Furthermore, the experiments do not clearly isolate the effects of estimating sensitivity through Gaussian probing from those of injecting the resulting noise into local SGD updates. A direct comparison between sensitivity probing with and without gradient perturbation, together with broader component ablations across different data heterogeneity settings, would help clarify whether the full NoRA design consistently provides meaningful benefits.

  • Limited analysis of how risk-aware perturbation affects learned representations. The experimental evaluation primarily focuses on final test accuracy, convergence curves, communication efficiency, and parameter sensitivity. However, the central motivation of NoRA is that neighbor averaging acts as a parameter perturbation whose impact depends on the receiving model's sensitivity. The paper does not sufficiently investigate how the proposed risk-aware mechanism changes the learned feature representations, inter-client model divergence, or the evolution of consensus under heterogeneous data. For example, representation-space visualizations or analyses of inter-client feature alignment could help illustrate whether NoRA actually mitigates harmful model interactions rather than simply improving predictive performance through altered aggregation weights. Such evidence would provide a more direct understanding of the proposed mechanism and strengthen its empirical justification.

[1] Rehman et al., "L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learning," ICCV, 2023.

Questions for authors
  1. What fundamental limitations of existing adaptive aggregation methods motivate the proposed NPAR formulation? How does NoRA differ from prior similarity- or divergence-aware aggregation approaches in centralized and decentralized federated learning, beyond introducing a different weighting criterion?

  2. Theorem 1 and Corollary 1 establish the standard non-convex convergence rate. To what extent does the theoretical analysis explain the advantages of NPAR-guided mixing? Could the authors provide further analysis of its effects on convergence bounds, consensus error, or robustness to data heterogeneity compared with conventional aggregation?

  3. Table 3 reports relatively modest improvements over simpler variants. How consistently does combining sensitivity and displacement improve performance across different datasets and non-IID settings? In particular, would sensitivity probing without injecting noise into SGD achieve comparable results to the full NoRA design?

  4. Although Table 2 demonstrates communication savings, NoRA introduces additional Jacobian-related computations during local training. What are the associated computational overhead, per-round runtime, and total wall-clock training costs compared with representative baselines?

  5. How does NPAR-guided averaging affect inter-client model divergence, learned representations, or consensus dynamics during training? Additional quantitative analysis or visualizations under heterogeneous data would help clarify whether NoRA effectively mitigates harmful neighbor interactions beyond improving final test accuracy.

6 comments
AuthorsOpenReview ↗

Response to Reviewer fcL1 on NoRA’s Methodological Novelty and Theoretical Analysis

R1. Differentiation from existing adaptive aggregation (Critical Weakness 1.1; Question 1).

NoRA explicitly accounts for receiver sensitivity to the displacement induced by candidate neighbor averaging. DecFus combines neighborhood training-loss variance with layer-wise cosine similarity to guide layer exchange and averaging, whereas L-DAWA uses layer-wise angular divergence from the previous global model to weight server-side aggregation. These measures characterize parameter relationships but do not explicitly capture the receiver-specific output response. Consequently, identical displacement magnitudes may induce different output changes at receivers with different sensitivities.

This distinction motivates NPAR, which combines pairwise model displacement with a noise-probed sensitivity proxy, as motivated by the output-drift analysis in Sec. 3.3. This combination directly affects aggregation: under a fixed risk threshold, higher maximum endpoint sensitivity implies a smaller admissible displacement before attenuation. The resulting mixing rule preserves the network average (Sec. 3.4).

We revised Sec. 2.2 (p. 3, lines 127–144) to clarify the distinct signals used by existing adaptive collaboration methods and include L-DAWA as a related centralized approach. A dedicated paragraph now discusses DecFus and L-DAWA, highlighting the distinction between parameter-space relationships and receiver-specific output responses that motivates NoRA.

R2. Theoretical characterization of NPAR-guided mixing (Critical Weakness 1.2; Question 2).

Theorem 1 and Corollary 1 establish that NoRA retains the standard leading convergence rate under finite-window spectral contraction. To quantify the effect of risk attenuation relative to conventional gossip, we added Corollary 3 in Appendix C.2. If the base gossip sequence has a contraction factor bounded by βW\beta_W over each BB-round window, the corresponding BB-round risk-aware mixing satisfies

∥At:t−B+1−1P∥2≤1−(1−ψ)B(1−βW).\lVert {A}_{t:t-B+1}-1{P}\rVert _2\le 1-(1-\psi)^B(1-\beta_W).

This yields a valid choice of CB=B/[(1−ψ)B(1−βW)]{C}_B=B/[(1-\psi)^B(1-\beta_W)] for the convergence analysis.

In the revised discussion after Corollary 1 (Sec. 4), we now explicitly connect this comparison to the consensus error bound and the convergence bound. Lemma 2 in the original submission already includes the heterogeneity term 32η2K2CB2σg232\eta^2K^2{C}_B^2\sigma_g^2 in the consensus error bound. Substituting the bound on ETE_T into the 20L2ET20L^2E_T term in Eq. (45) of Appendix C.5 explains how CB2{C}_B^2 enters the higher-order heterogeneity term of the convergence bound in Eq. (18). This comparison concerns worst-case theoretical guarantees, while realized consensus and convergence also depend on local update trajectories.

Proposition 1 bounds the magnitude of output drift and leaves the sign of the receiver’s loss change undetermined, as noted in Sec. 3.3. We therefore examine the association between NPAR and adverse averaging empirically in Fig. 1(b,c) and Appendix A. We label an edge adverse if candidate pairwise averaging increases held-out local loss at either endpoint. The adverse averaging rate rises from 41% in the lowest NPAR quintile to 71% in the highest. NPAR achieves an AUROC of 0.70, compared with 0.60 for displacement alone and 0.63 for sensitivity alone. These results support NPAR as an empirical surrogate for one-step adverse averaging under this diagnostic protocol.

AuthorsOpenReview ↗

Response to Reviewer fcL1 on Component Ablations and the Role of Noise Injection

R3. Component ablations and the role of noise injection (Critical Weakness 2.2; Question 3)

We made two revisions in response to these concerns. First, we clarified the Risk only ablation in Sec. 5.5 to distinguish Gaussian probing for sensitivity estimation from injecting probe noise into local SGD updates. Risk only retains Gaussian probing and NPAR-guided mixing and disables noise injection into local updates. Its comparison with full NoRA therefore isolates the contribution of noise injection while keeping the probing and mixing procedures unchanged.

Second, to examine whether the gains in the original CIFAR-10 Pathological ablation generalize across datasets and non-IID settings, we extended the same comparisons to CIFAR-10 and CIFAR-100 with Dirichlet partitions (α=0.3\alpha=0.3). Appendix D.3 and Table 6 report the two additional settings alongside the original one. Displacement only and Sensitivity only retain noise injection but replace the joint NPAR score with pairwise model displacement or receiver sensitivity, respectively, thereby isolating the benefit of combining both signals for mixing.

Global test accuracy (%) is reported as mean ±\pm standard deviation over three independent runs.

MethodCIFAR-10 Patho. (original)CIFAR-10 Dir0.3 (new)CIFAR-100 Dir.0.3 (new)
Risk only68.79 ±\pm 0.4775.47 ±\pm 0.2436.17 ±\pm 0.23
Displacement only68.82 ±\pm 0.5375.68 ±\pm 0.1336.13 ±\pm 0.27
Sensitivity only68.96 ±\pm 0.6675.49 ±\pm 0.1836.22 ±\pm 0.41
NoRA69.40 ±\pm 0.9676.14 ±\pm 0.2236.61 ±\pm 0.44

Combining sensitivity and displacement. NoRA achieves the highest mean accuracy across all three settings. Compared with Displacement only and Sensitivity only, it improves mean accuracy by 0.46%-0.58% and 0.39%-0.65%, respectively. These consistent gains across datasets and partition settings support the complementary roles of receiver sensitivity and pairwise model displacement in NPAR.

Probing without noise injection. NoRA also outperforms Risk only by 0.44%-0.67% across all three settings. Since both methods retain Gaussian sensitivity probing and NPAR-guided mixing, this comparison isolates the additional contribution of injecting probe noise into local SGD updates. The consistent gains indicate that sensitivity probing alone does not fully account for NoRA's performance improvements.

R4. Computational overhead and wall-clock efficiency (Critical Weakness 2.1; Question 4)

We have added Appendix D.2 and Table 5, with a pointer in Sec. 5.3, to quantify NoRA's computational and memory overhead and evaluate its end-to-end training efficiency. We compare NoRA with DFedAvg, DFedSAM, and DFedADMM on CIFAR-10 with a Dirichlet partition (α=0.3\alpha=0.3), using the same 74% test accuracy target as in Table 2. We report per-round FLOPs and runtime, peak GPU memory, the wall-clock time for the full 500-round budget, and the time to reach the target accuracy. The 500-round time reports the full-budget wall-clock time, whereas time to target is measured until the first evaluation reaching or exceeding 74%. Per-round runtime is obtained by dividing the full-budget time by 500. FLOP counts include the additional probe computation in Eq. (3) and are reported as means because their standard deviations (0.06-0.09 GFLOPs/round) are below the reporting precision (0.01 TFLOPs/round). Other measurements are reported as mean ±\pm standard deviation.

MethodTFLOPs/roundRuntime (s/round)Peak GPU memory (MiB)500-round time (min)Time to 74% (min)
DFedAvg20.2110.63 ±\pm 0.107052.46 ±\pm 29.5488.58 ±\pm 0.8565.87 ±\pm 3.05
DFedSAM40.4014.06 ±\pm 1.917044.62 ±\pm 35.70117.17 ±\pm 15.9291.40 ±\pm 2.37
DFedADMM20.2224.14 ±\pm 1.852723.06 ±\pm 4.36201.17 ±\pm 15.39145.17 ±\pm 25.56
NoRA33.0321.61 ±\pm 0.751708.62 ±\pm 11.31180.08 ±\pm 6.22107.80 ±\pm 7.36

Computational cost. NoRA requires 33.03 TFLOPs per round, 63.4% more than DFedAvg and DFedADMM but 18.2% less than DFedSAM. Noise probing in Eq. (3) accounts for 12.83 TFLOPs per round, or 38.8% of NoRA's total per-round FLOPs. The resulting probe response is reused for sensitivity estimation in Eq. (6), avoiding additional computation of this product and explicit Jacobian construction.

GPU memory. NoRA achieves the lowest peak GPU memory consumption (1708.62 MiB), reducing memory usage by 37.3% relative to DFedADMM and approximately 75.8% relative to DFedAvg and DFedSAM.

Per-round runtime. NoRA requires 21.61 s per round, 10.5% less than DFedADMM (24.14 s), although more than DFedAvg (10.63 s) and DFedSAM (14.06 s). Notably, despite requiring fewer FLOPs than DFedSAM, NoRA has a higher per-round runtime, indicating that FLOP counts alone do not fully characterize practical execution efficiency.

End-to-end training time. Over the full 500-round budget, NoRA takes 180.08 min, compared with 201.17 min for DFedADMM, 117.17 min for DFedSAM, and 88.58 min for DFedAvg. More importantly, NoRA reaches the 74% accuracy target in 107.80 min, 25.7% sooner than DFedADMM (145.17 min). However, DFedAvg and DFedSAM reach the target in 65.87 and 91.40 min, respectively. Thus, the communication savings translate into shorter time to target relative to DFedADMM, but not relative to all competing methods. This comparison shows that the wall-clock benefit of reduced communication depends on the per-round computational cost of each method.

Reviewer_vjhnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the aggregation problem in decentralized federated learning. It develops a specific weighting strategy by analyzing the Jacobian sensitivities of the models. The idea is not particularly novel, but the paper is technically solid and the engineering work is well executed. It provides a nonconvex convergence analysis under time-varying adaptive mixing and evaluates the method on three image-classification datasets, using multiple data partitioning schemes and communication topologies.

Critical strengths
  1. A particularly solid design: The same neighboring model may have different effects on aggregation depending on the recipient’s training state. Combining receiver-side sensitivity with model distance is a strong design choice.

  2. The method is simple and efficient.

  3. I did not find any significant errors in the convergence analysis.

Critical weaknesses

1.The paper’s overall novelty appears limited. The authors should discuss the theoretical motivation and the underlying mechanism more thoroughly.

2.The theoretical exposition is quite unclear; for example, the properties are not stated clearly. The constant in Theorem 1 also seems readily improvable.

Questions for authors

Overall, novelty is my main concern. I hope the authors can clearly articulate the deeper motivation and insights behind the work to convince me to raise my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer vjhn on NoRA’s Motivation, Mechanism, and Convergence Bound

R1. Novelty, theoretical motivation, and underlying mechanism (Critical Weakness 1).

Thank you for the opportunity to clarify the motivation and mechanism. Our key insight is to interpret a candidate neighbor average as a finite parameter perturbation of the receiving model. The resulting output drift is bounded in terms of both the displacement magnitude and the receiver's local sensitivity along the perturbation path. This motivates NPAR, which combines pairwise model displacement with a noise-probed sensitivity proxy.

In the Introduction (p. 2), we begin a separate paragraph with “To characterize this state-dependent effect before averaging,” to distinguish the problem formulation and offline diagnostic from the proposed perturbation perspective. We further emphasize that parameter displacement alone does not determine the resulting output change, and clarify the complementary roles of displacement magnitude and receiver sensitivity.

In Sec. 3.3 (p. 4, after Proposition 1), we explicitly explain this complementarity: identical displacement magnitudes may yield different output-drift bounds under different sensitivity levels, whereas sensitivity alone cannot distinguish neighbors at different distances.

In Sec. 3.4 (p. 5, after Eq. (11)), we clarify that, for a fixed risk threshold, the admissible displacement decreases with the maximum endpoint sensitivity. Following Proposition 2, we further explain the effect of risk-aware mixing on subsequent optimization: for fixed locally updated models, the base and risk-aware mixing matrices preserve the same network mean but yield potentially different client-level aggregates. These aggregates initialize the next local-training round and may therefore induce different local gradients.

R2. Theoretical exposition and constants in Theorem 1 (Critical Weakness 2).

Thank you for pointing out the opportunity to tighten the constants. We have sharpened the finite-time convergence bound in Theorem 1 and revised its derivation in Appendices C.5–C.6, without changing the assumptions or step-size condition.

The refinement involves two improvements:

  1. Retaining variance reduction from client averaging. Eq. (39) establishes that E∥e‾t∥2≤Kσρ2/m{E}\lVert \overline{{e}}_t\rVert ^2\le K\sigma_\rho^2/m. The original proof relaxed this bound to Kσρ2K\sigma_\rho^2. We now retain the 1/m1/m factor throughout Eqs. (42)–(45), while accounting separately for the noise contribution arising from local drift.

  2. Tightening the final absorption. Under the original step-size condition, the coefficient of GTG_T on the right-hand side is at most 5/325/32. Absorbing this term into the left-hand side therefore incurs a factor of at most 32/2732/27, improving upon the previous factor of 22. We explicitly derive the resulting coefficient bounds.

Letting Δ0=f(x‾0)−f⋆\Delta_0=f(\overline{{x}}^0)-f^\star and GT=T−1∑t=0T−1E∥∇f(x‾t)∥2G_T=T^{-1}\sum_{t=0}^{T-1}{E}\lVert \nabla f(\overline{{x}}^t)\rVert ^2, the revised Eq. (17) becomes

\frac{5\Delta_0}{\eta KT} +\frac{5L\eta}{m}\sigma_\rho^2 +760L^2\eta^2K(1+{C}_B^2) (\sigma_\rho^2+K\sigma_g^2).$$ Corollary 1 and its proof have been updated accordingly. All revisions are highlighted in blue in the revised manuscript. The original inequalities remain valid but yield more conservative constants; hence, the original convergence guarantee is unaffected. The revised analysis sharpens the finite-time bound without changing its asymptotic dependence on $T$ and $K$, retaining the standard ${O}(1/\sqrt{KT})$ leading term under the original step-size scaling.

To further clarify the novelty of NoRA, we highlight its distinction from closely related adaptive aggregation approaches. As discussed in the revised Sec. 2.2, DecFus combines neighborhood training-loss variance with layer-wise cosine similarity to adapt the balance between layer exchange and model averaging, while L-DAWA uses layer-wise angular divergence to adjust server-side aggregation in federated self-supervised learning. Although these approaches exploit parameter-space relationships to guide layer-wise fusion or aggregation, they do not explicitly account for how the receiving model's sensitivity affects the output drift induced by a candidate averaging perturbation.

NoRA addresses this complementary aspect. Specifically, Proposition 1 establishes an output-drift bound in terms of pairwise parameter displacement and receiver-side Jacobian sensitivity. Since evaluating sensitivity along each candidate averaging path is costly, NoRA uses Gaussian probing to obtain a lightweight, trajectory-level sensitivity proxy (Sec. 3.2). Combining this proxy with pairwise displacement yields NPAR, which guides the attenuation of high-risk neighbor interactions (Secs. 3.3–3.4). We emphasize that NPAR is a theoretically motivated risk surrogate rather than a direct evaluation of the pathwise bound or a guarantee of adverse averaging.

The additional diagnostics in Appendix D.6 provide empirical support for this mechanism. At round 500, applying base gossip and NPAR-guided mixing to the same locally updated models on the same communication graph reduces the mean positive increase in held-out local loss from 0.43 to 0.12 nats. Meanwhile, Fig. 8(a) shows that NoRA retains greater inter-client parameter disagreement than Noise only (6.49×10−76.49\times10^{-7} vs. 2.97×10−72.97\times10^{-7}). These results suggest that NoRA's benefit does not rely on enforcing stronger parameter consensus, but is consistent with its ability to reduce potentially harmful neighbor influence through receiver-aware risk modulation.

We hope these clarifications and additional results address your concerns regarding the novelty and underlying mechanism of NoRA. If these points resolve your remaining concerns, we would greatly appreciate your reconsideration of the rating.

I took some extra time to go through the theoretical section carefully, which delayed my follow-up. Overall, most of my concerns have been addressed. The performance overhead may still be worth discussing, though I see it as secondary to the paper’s main contribution. It’s hard for any single paper to excel on every front. I’m willing to raise my score by one point. Good luck!

We sincerely thank the reviewer for taking the time to further examine the theoretical section and for their thoughtful feedback. We are pleased that our responses have addressed most of your concerns, and we greatly appreciate your willingness to reconsider and raise your score. We also acknowledge the performance overhead as an important trade-off and appreciate your understanding of its secondary role relative to the paper's main contribution. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.