acceptodds
Under review as a conference paper at ICLR 2027

How Communication Shapes Convergence and Generalization in Decentralized Training: A Feature Learning Perspective

Abstract

As data, model, and client populations continue to grow, decentralized learning has emerged as an important paradigm for training without a central server. Despite its practical relevance, existing theory has largely focused on optimization and consensus, with generalization typically studied through algorithmic stability. It remains unclear how decentralized communication shapes feature learning in neural networks and, in turn, affects the generalization of individual client models. To address this gap, we analyze decentralized gradient descent for two-layer convolutional neural networks from a feature learning perspective. The derived dynamics of signal and noise coefficients characterize how communication propagates predictive signals and sample-specific noise across clients. The corresponding topology-aware quantities yield early generalization bounds and reveal client-placement effects that graph spectra and feature similarity alone cannot capture. The analysis further establishes training convergence and client-level risk guarantees at a common fitted iterate. An extension to periodic local updates identifies a sufficient local-update budget under which the early generalization guarantee retains the well-mixed scale. Experiments on synthetic and real-world datasets under IID and non-IID settings support the predicted effects of communication topology, client placement, and communication frequency.

Primary area:
learning theory
Keywords:
decentralized training, feature learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_t5XqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The work in decentralized learning has historically been on its convergence and consensus analysis. This work instead studies how the consensus-driven decentralized optimization affects the generalization capabilities of the participating clients. To achieve this, the paper studies a simple two-layer CNN model through the lens of feature learning theory, where the features are modeled as signal vectors mixed with noise terms.

Critical strengths

As the convergence and consensus results has matured in the decentralized learning literature, this paper opens a new door to additional analysis of decentralized gradient descent algorithms. The mathematical framework of feature learning chosen to study generalization seems suitable, carrying interesting intuitions.

Critical weaknesses

The mathematical framework used in this paper closely follows Huang et al., 2024, where signal and noise trajectory tracking is used to analyze generalization for federated learning. In its current version, it is unclear what are the key theoretical hurdles that arise when moving to decentralized gradient descent. The paper could put emphasis on this to help better appreciate the results in the main text and the appendix relative to prior work (especially Huang et al., 2024).

Questions for authors
  1. It is stated in Sec. 3 that the signal vectors {μ(c)}c=1C\{\mu(c)\}_{c=1}^C are assumed to be mutually orthogonal. This would make χk\chi_k defined in Sec. 4.1 always be χk=1\chi_k=1, and affect all the subsequent results involving χk\chi_k.

  2. Eq. (15) in Sec. 4.3 should explicitly be written in an Assumption block for clarity, similar to Assumptions 3.1 and 4.1.

4 comments

We thank the reviewer for the constructive feedback and for recognizing the relevance of our work. Below, we clarify our contributions relative to prior work and address the two questions.

W1: The mathematical framework used in this paper closely follows Huang et al., 2024, where signal and noise trajectory tracking is used to analyze generalization for federated learning. In its current version, it is unclear what are the key theoretical hurdles that arise when moving to decentralized gradient descent. The paper could put emphasis on this to help better appreciate the results in the main text and the appendix relative to prior work (especially Huang et al., 2024).

A: Our work establishes early-risk bounds that capture both communication topology and client placement, together with convergence and client-level risk guarantees at a common fitted iterate. We also derive an explicit sufficient bound on the number of local updates between communication rounds that preserves the well-mixed scale of the early generalization certificate.

The signal–noise framework developed in prior feature-learning studies, including Cao et al. (2022), Kou et al. (2023), and Huang et al. (2024), provides the starting point for our analysis. Establishing these results without exact global synchronization requires additional tools. We develop new analytical techniques to track feature propagation, control coefficient growth, and prove convergence under decentralized mixing.

First, uniform server averaging in the analysis of Huang et al. yields the 1/K1/K noise factor and the χk/K\chi_k/K signal-sharing factor. In decentralized training, each gradient contribution propagates with weights that depend on its source, receiver, and update time. We retain these weights in Lemma 4.3 while controlling local logits and activations. Signal contributions accumulate through Sk(T)S_k(T), while a Gram-matrix estimate controls noise energy through the squared propagation weights in Qk(T)Q_k(T). Together, these estimates yield the placement-sensitive early-risk certificate.

Second, server aggregation produces common coefficients across clients, a property used in Huang et al.’s coefficient induction. During subsequent local updates, new contributions from a training sample arise only at its source client. Under decentralized training, a propagated coefficient can be larger at another receiver, while its new gradient contribution remains determined by the source model. A large coefficient at the receiver therefore does not directly imply a small loss-derivative magnitude at the source. Lemma A.7 resolves this mismatch by tracking the historical maximum of the coefficients. Mixing alone cannot increase this maximum, and a source that raises it must already have a local coefficient close to that maximum. The source margin can then control further coefficient growth, closing the induction for the coefficient bounds and activation persistence.

Finally, Lemma A.12 establishes convergence using the average of the local distance potentials. By completing the square in each eigenmode of the mixing matrix, we control the interaction between mixing and gradients evaluated at different local models and derive a descent inequality for decentralized gradient descent. We will highlight these technical contributions and explain more explicitly how they extend beyond Huang et al.’s analysis.

Q1: It is stated in Sec. 3 that the signal vectors {μ(c)}c=1C\{\mu^{(c)}\}_{c=1}^{C} are assumed to be mutually orthogonal. This would make χk\chi_k defined in Sec. 4.1 always be χk=1\chi_k=1, and affect all the subsequent results involving χk\chi_k.

A: Orthogonality applies to the distinct signal types, while multiple clients may share the same type. Thus, χk\chi_k counts the clients sharing client kk’s signal. It remains fixed throughout training but may vary across clients and signal assignments. It equals one only when client kk is the sole client assigned to its type. In the four-client example in Section 4.2, each type is shared by two clients, giving χk=2\chi_k=2. We will clarify this distinction in Section 3.

Q2: Eq. (15) in Sec. 4.3 should explicitly be written in an Assumption block for clarity, similar to Assumptions 3.1 and 4.1.

A: We will state Eq. (15) in a separate assumption block and explicitly identify it as an additional condition for the fitting analysis. This additional condition supports activation persistence in the fitting analysis and is not required for Lemma 4.3 or Eq. (39). Assumptions 3.1 and 4.1 provide the common conditions for both the early and fitting analyses.

Thank you again for your time and constructive feedback. In our responses, we explain the additional analytical challenges beyond server averaging and how we address them through weight tracking, historical maximum coefficient bounds, and an average local-distance potential for convergence. We also clarify that distinct signal types are orthogonal while multiple clients may share a type, and explain the role of the additional fitting assumption.

As the discussion deadline approaches, please let us know if you have any remaining questions or concerns. We would be happy to address them. If our clarifications have resolved your concerns, we would be grateful if you would consider revising your score upward.

Thank you for your responses. Regarding the critical weakness I mentioned in my initial review, I do not see the corrections and clarification applied to the paper in the revised version. That still makes it hard to appreciate the novelty of this work compared to Huang et al., 2024.

Thank you for pointing this out. Our previous revision included a brief technical summary after Assumption 4.1. We have expanded it into a dedicated remark, “Technical challenges beyond the federated setting,” highlighted in blue. The remark clarifies which steps of Huang et al.’s (2024) analysis require new arguments in decentralized training and how our proofs address them.

More broadly, our work shows how peer-to-peer communication shapes client-level generalization. Under limited training and communication budgets, our early-risk bounds distinguish client placements even when the graph spectrum and aggregate signal similarity are identical. Stability-based analyses that rely only on the graph spectrum and aggregate signal similarity cannot distinguish these placements. We also establish convergence and client-level risk guarantees at a common fitted iterate. Finally, we give an explicit sufficient bound on local updates between communication rounds that preserves the well-mixed scale of the early-risk guarantee.

We thank the reviewer again for the constructive feedback, which helped us clarify our contributions and their significance for decentralized learning.

Reviewer_YTLoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies decentralized training of two-layer ReLU CNNs under a heterogeneous signal–noise model. It tracks how predictive signals and sample-specific noise spread across clients. The quantities Sk(T)S_k(T) and Qk(T)Q_k(T) give an early local-risk bound through Sk(T)2/Qk(T)S_k(T)^2/Q_k(T). They capture client-placement effects beyond graph spectra and aggregate feature similarity. The paper also gives training and risk guarantees at a common fitted iterate. A periodic-update extension gives a sufficient bound on the local-update interval. Synthetic and small real-data experiments support the proposed mechanisms.

Critical strengths

This paper is well motivated and well written.

The graph-specific insight is useful. Earlier feature-learning work already separates signal learning from noise memorization. This paper tracks where information comes from, how it travels, and when it reaches each client. Even with the same graph and type counts, different placements can change signal acquisition. This is an interesting contribution to the understanding of decentralized learning.

Critical weaknesses
  • The main generalization theory is "local". Equation (3) evaluates each client's model on its own distribution DkD_k. The appendix also tests the models on a global mixture. Yet the main theorems do not give a corresponding explicit global-risk guarantee under heterogeneous data. Lemma A.4 provides a cross-distribution coefficient certificate, but its global implications are not developed. It remains unclear how local learning transfers to the global distribution. This limits the scope of the generalization results.
  • The practical guidance on topology design is still limited. The results offer several useful insights. It is less clear how to use them to choose a topology that balances convergence and generalization. The broad principle of encouraging communication among similar clients is already familiar. A concrete example of how Sk(T)S_k(T) and Qk(T)Q_k(T) change a design choice would help clarify the added guidance.

Additional minor limitations

  • The analysis uses a fixed symmetric mixing matrix. The periodic-update extension changes when communication occurs. It does not study adaptive neighbor selection or asymmetric communication. Both matter in practical decentralized systems.
  • The real-data tests use small rotated MNIST and Digits-5 setups. They illustrate the theory, but provide limited evidence for larger systems. I see this as a minor weakness for a theory paper.
Questions for authors
  1. Which topology and placement effects should persist after fitting? Theorem 4.7 gives a fitted-iterate guarantee, but the detailed Sk(T)2/Qk(T)S_k(T)^2/Q_k(T) risk dependence is an early-time result. What does the current theory tell us about later training? This would also help distinguish its scope from stochastic long-time effects such as those in [1].
  2. How to improve topology or communication choice follows from the analysis? A small design example would help show what is gained beyond the general principle of favoring "similar clients".
  3. A potential contradiction. Equation (11) and the early horizon require σ0≤cTnˉ/(σpd)\sigma_0\le c_T\sqrt{\bar n}/(\sigma_p d). Equations (15) and (48) require σ0=Ω~(Knˉ/(σpd))\sigma_0=\widetilde{\Omega}(K\bar n/(\sigma_p d)). Table 2 clearly separates these regimes. Can Equation (39) give a nontrivial early-risk guarantee under the initialization used for fitting?

References

[1] Gu et al. Why (and When) does Local SGD Generalize Better than SGD? ICLR 2023.

4 comments

We thank the reviewer for the positive feedback and the thoughtful questions about the significance and scope of our results.

W1: The main generalization theory is "local". It remains unclear how local learning transfers to the global distribution. This limits the scope of the generalization results.

A. Our main objective is to understand how decentralized training shapes feature learning and generalization at individual clients. We therefore evaluate each receiving model on fresh data from its own distribution. This choice allows us to characterize how clients benefit from information propagated through the network. Mixture evaluation addresses the complementary question of how each model performs across distributions.

Nevertheless, our framework can also be extended to broader evaluation settings, such as Dmix=∑qπqDq\mathcal D_{\mathrm{mix}}=\sum_q\pi_q\mathcal D_q. Under the conditions of Lemma A.4 and with vk(t)>0v_k(t)>0,

\le \sum_q \pi_q \exp\!\left[ -\frac12 \left( \frac{s_{kq}(t)}{\sigma_p v_k(t)} -\frac{1}{\sqrt{2\pi}} \right)_+^2 \right].$$ This bound connects mixture performance to the receiving model’s learned margins across signal types. Controlling its own-domain margin alone does not ensure a small mixture risk. We will explicitly state this corollary and its interpretation in the appendix, while retaining client-level generalization as the main focus. > Q1: Which topology and placement effects should persist after fitting? What does the current theory tell us about later training? This would also help distinguish its scope from stochastic long-time effects such as those in [1]. A: Our theory distinguishes finite-time placement sensitivity from generalization at a fitted iterate. Theorem 4.7 provides client-level test-risk bounds at an iterate with small training loss and identifies a regime in which fitting coexists with constant test error. These results address whether the features acquired through communication support generalization once the training data are fitted. To examine placement effects after fitting, we extended the ring experiments in Appendix C.5 to $73728$ updates. At the mean training loss $\le0.1$, Clustered and Alternating achieve mean own-type errors of $1.699\%$ and $1.702\%$, respectively. At the common final budget, both achieve training loss around $0.072$ and own-type error around $1.62\%$. The small placement gap at this fitted regime is consistent with the placement-independent limit of the propagation factor in Proposition 4.5. We will include these extended results in the revised Appendix C.5. Gu et al. explain generalization through drift induced by stochastic gradients near a manifold of minima, showing how it shapes the implicit bias of optimization after fitting. Our analysis instead tracks how predictive signals and sample-specific training noise propagate across heterogeneous clients and shape the features each client acquires. This yields placement-dependent early-risk bounds even under full-batch updates where stochastic-gradient noise is absent.

Q2&W2: The practical guidance on topology design is still limited. How does the analysis help choose a topology or communication scheme that balances convergence and generalization? A concrete example of how Sk(T)S_k(T) and Qk(T)Q_k(T) change a design choice would clarify what is gained beyond the familiar principle of favoring similar clients.

A: Our analysis provides a generalization criterion for communication design: among schemes that satisfy the conditions, compare Sk(T)2/Qk(T)S_k(T)^2/Q_k(T) at the target training horizon. For example, consider four clients with signal types (a,a,b,b)(a,a,b,b). Full averaging mixes faster than a lazy ring but reduces both useful signal exposure S1(T)S_1(T) and noise concentration Q1(T)Q_1(T). At T=3T=3, the ring yields a larger S1(T)2/Q1(T)S_1(T)^2/Q_1(T) (1.3991.399 versus 1.3331.333). Thus, selecting a topology solely for faster mixing need not give the strongest finite-time generalization certificate.

The guidance also goes beyond favoring similar neighbors. Section 5.1 compares placements with the same graph, signal-type counts, and distribution of same-type neighbor counts. Their different multi-hop exposures produce different Sk(T)S_k(T) with identical Qk(T)Q_k(T), allowing the criterion to distinguish choices that these local similarity statistics cannot.

For communication frequency, Eq. (19) provides a sufficient budget for preserving the well-mixed generalization scale. We conducted new experiments on the 16-client ring with χk=8\chi_k=8 at T=1536T=1536, where Eq. (19) permits h≤5.167h\le5.167. Choosing h=4h=4 instead of h=1h=1 reduces communication rounds by 75%75\%, while yielding comparable mean own-type errors for Clustered placement (3.249%3.249\% versus 3.381%3.381\%). These results illustrate how the analysis can guide the choice of communication frequency to reduce communication while maintaining comparable prediction quality.

Q3: A potential contradiction. Equation (11) and the early horizon require σ0≤cTnˉ/(σpd)\sigma_0\le c_T\sqrt{\bar n}/(\sigma_p d). Equations (15) and (48) require σ0=Ω~(Knˉ/(σpd))\sigma_0=\widetilde{\Omega}(K\bar n/(\sigma_p d)). Table 2 clearly separates these regimes. Can Equation (39) give a nontrivial early-risk guarantee under the initialization used for fitting?

A: Equation (39) can yield a nontrivial early-risk guarantee under the initialization used for fitting. Our fitting analysis imposes two-sided initialization bounds, as in Huang et al. (2024, Assumption 4.2(3)). Lemma 4.3 and Eq. (39) explicitly retain the initialization contribution without requiring the fitting lower bound. Equation (11) provides additional sufficient conditions for absorbing the initialization terms in Eqs. (37)–(38) into the learned signal and noise contributions, yielding the simplified Sk(T)2/Qk(T)S_k(T)^2/Q_k(T) certificate in Appendix A.6. These conditions are not required by Eq. (39) or the fitting theorems. Table 2 summarizes this distinction.

To demonstrate compatibility, fix KK, a mixing matrix satisfying Assumption 3.1, and δ,ϵ∈(0,1)\delta,\epsilon\in(0,1). Set σp=1\sigma_p=1 and consider nˉ=m=N,d=N6,∥μ∥22=N4,\bar n=m=N,\qquad d=N^6,\qquad \|\mu\|_2^2=N^4, η=aN−5,σ0=bN−5log⁡N,\eta=aN^{-5},\qquad \sigma_0=bN^{-5}\sqrt{\log N}, where a>0a>0 is sufficiently small and b>0b>0 is sufficiently large. For sufficiently large NN, these choices satisfy the assumptions of both the general early analysis and the fitting analysis. At T=T0=Θ(N)T=T_0=\Theta(N), the normalized margin in Eq. (39) is Θ(N/log⁡N)\Theta(N/\sqrt{\log N}), giving

\le \exp\left[-\Omega\left(\frac{N^2}{\log N}\right)\right].$$ Thus, the same initialization can support both a nontrivial early-risk bound and the fitting guarantees, even when the additional absorption conditions in Eq. (11) are not satisfied. The fitting assumptions alone do not ensure a nontrivial early bound, which also requires sufficient signal relative to initialization and noise. We will include this compatibility example alongside Eq. (39).

Thank you again for your thoughtful feedback. In our responses, we provide a mixture-risk extension, clarify the distinction between early placement effects and fitted-stage guarantees, and report additional experiments through fitting. We also gave concrete examples of topology and communication-frequency choices, together with an explicit initialization example showing that the general early-risk bound can be nontrivial under the fitting assumptions.

As the discussion deadline approaches, please let us know if you have any remaining questions or concerns. We would be happy to address them. If our responses and additional results have resolved your concerns, we would be grateful if you would consider revising your score upward.

Thank you for the response, which addresses part of the concerns. The paper provides some insights into how communication shapes feature learning, but the scope remains centered on client-local generalization. Overall, the reviewer maintains the current positive rating.

Reviewer_XLSPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how exchanging models between clients affects what a two-layer CNN learns. It tracks useful signal and training noise, relates them to test error, and studies how often clients should communicate. Experiments use synthetic data and digit images.

Critical strengths
  1. The ratio S_k(T)^2/Q_k(T) captures differences that graph connectivity alone misses. It distinguishes different placements of clients on the same graph, even when the same number of clients share each signal.

  2. The synthetic experiments carefully separate signal learning from noise learning. Some comparisons keep the graph and the number of similar neighbors fixed. Using five paired random seeds also makes the comparisons more convincing.

Critical weaknesses
  1. A needed assumption might be missing. Lemma A.2 uses rank(H) >= d/2, but Section 3 does not require it. If C = d, the noise disappears (H = 0), so Theorem 4.7's nonzero test-error lower bound cannot hold. Please state the needed restriction on C.

  2. Two proofs are missing. Lemmas A.13 and A.14 support Theorem 4.7, but are stated without proofs.

  3. The model is very simple. Each example has a perfect label signal in one patch and separate Gaussian noise orthogonal to all signals. Clients' signals are either identical or orthogonal. The theory also covers only binary labels, logistic loss, and a two-layer CNN.

  4. The new result about client placement only covers early training. The bound after fitting does not distinguish placements. The experiments also leave that stage largely untested: Table 5 reports loss near 0.69 at the main checkpoint, and the ring runs in Appendix C.5 never reach loss 0.1 within 8192 updates.

  5. The communication-interval experiment does not directly test the proposed guarantee. Appendix C.5 says that the condition in Eq. (19) is not satisfied at the reported early checkpoint.

  6. Real-data tests are limited. Rotated MNIST uses only digits 0 and 1. Digits-5 uses 100 updates and three seeds. Appendix C.8 also omits the full CNN layer details, making these experiments harder to reproduce.

  7. The real-data tests do not check the proposed explanation. They report prediction errors without measuring signal and noise learning. It is therefore unclear whether this mechanism explains the observed effects of client placement.

  8. The benefit depends on the test distribution. In Table 9 at h = 16, clustering improves own-domain error from 0.300% to 0.218%, but error across both domains rises from 0.303% to 7.214%. This tradeoff limits the practical benefit.

Questions for authors
  1. Is C <= d/2 intended? Please clarify this assumption and provide proofs of Lemmas A.13 and A.14.

  2. Can you report signal and noise measurements after the models reach small training loss, and test a communication interval satisfying Eq. (19)?

  3. What are the exact CNN architectures? What evidence connects the real-data results to the proposed signal/noise explanation?

5 comments

We thank the reviewer for the constructive feedback and for recognizing the strengths of our work. We address the concerns below.

Q1&W1&W2: Is C≤d/2C\le d/2 intended? Please clarify the assumption and provide the missing proofs of Lemmas A.13 and A.14, which support Theorem 4.7.

A: The condition C≤d/2C\le d/2 ensures rank⁡(H)=d−C≥d/2\operatorname{rank}(H)=d-C\ge d/2, as used in Lemma A.2 and the test-error lower bound. This restriction is mild in the intended regime, where the number of signal types is much smaller than the sample size and d=Ω~(n2)d=\widetilde{\Omega}(n^2). We will state this condition explicitly in Section 3 and include the full proofs of Lemmas A.13 and A.14 in the revised appendix.

W3: The model is very simple. Each example has a perfect label signal in one patch and separate Gaussian noise orthogonal to all signals. Clients' signals are either identical or orthogonal. The theory also covers only binary labels, logistic loss, and a two-layer CNN.

A: Our data model and two-layer CNN build on the feature-learning framework studied by [1,2,3]. This setup allows us to track signal learning and noise memorization separately and examine how decentralized communication affects both. Beyond this theoretical setting, our experiments on Rotated MNIST and Digits-5, including deeper networks and multiclass classification, support the predicted effects of client placement in more complex settings. We view this framework as a foundation for extending the theory to more complex data distributions and deeper architectures in future work.

[1] Cao et al. (2022). Benign Overfitting in Two-layer Convolutional Neural Networks. NeurIPS.

[2] Kou et al. (2023). Benign Overfitting in Two-layer ReLU Convolutional Neural Networks. ICML.

[3] Huang et al. (2024). Understanding Convergence and Generalization in Federated Learning through Feature Learning Theory. ICLR.

Q2&W4&W5: The placement-sensitive result covers early training, while the fitted-iterate bound does not distinguish placements. Can you report signal and noise measurements after the models reach small training loss and test a communication interval satisfying Eq. (19)?

A. Our placement-sensitive results address feature learning and generalization within a finite training budget. The original experiments therefore focused on this regime. We appreciate the suggestion to further examine how these dynamics evolve after fitting and have conducted additional experiments.

We extended the ring experiments in Table 7 from 81928192 to 7372873728 updates. Both layouts reach mean training loss ≤0.1\le0.1 after approximately 5290052900 updates, with mean signal margins around 0.7140.714, held-out noise second moments around 0.0960.096, and own-type errors of 1.699%1.699\% for Clustered and 1.702%1.702\% for Alternating. At the common final budget, both achieve training loss around 0.0720.072 and own-type error around 1.62%1.62\%. The small fitted-stage difference is consistent with the placement-independent limit of the propagation factor under prolonged mixing in Prop.4.5.

To complement the early communication-interval experiments in Table 6, we added training runs with h=4h=4 at T=1536T=1536, where Eq. (19) permits h≤5.167h\le5.167. Clustered achieves a larger signal margin than Alternating (1.353×10−41.353\times10^{-4} versus 1.282×10−41.282\times10^{-4}), nearly identical held-out noise second moments (4.921×10−94.921\times10^{-9}), and lower mean own-type error (3.249%3.249\% versus 3.841%3.841\%). These experiments directly address the requested communication condition. We will include the full measurements and condition audit in the revised appendix.

Q3&W6&W7: The real-data tests are limited: Rotated MNIST uses only digits 0 and 1, and Digits-5 uses 100 updates and three seeds. Appendix C.8 omits the full CNN layer details. What are the exact architectures? The real-data results report prediction errors without measuring signal and noise learning; what evidence connects the observed placement effects to the proposed signal/noise explanation?

A: Rotated MNIST uses two convolutional layers (16 and 32 channels), followed by fully connected layers of widths 120 and 84 and a two-class output, without BatchNorm or dropout. Digits-5 uses three convolutional layers (64, 64, and 128 channels), followed by fully connected layers of widths 2048 and 512 and a 10-class output. Both networks use 5×55\times 5 convolutions with stride 1, ReLU activations in hidden layers, and 2×22\times 2 max pooling after the first two convolutional layers; convolutional padding is 0 for Rotated MNIST and 2 for Digits-5. For Digits-5, we evaluate a BatchNorm baseline and a no-BatchNorm with Kaiming initialization. We will include the complete architectures in Appendix C.8.

We also quantify communication exposure in the real-data experiments. Qk(T)Q_k(T) is computed directly from the mixing matrices and communication schedule, while Sk(T)=∑q[AT]kqI[c(q)=c(k)]S_k(T)=\sum_q [A_T]_{kq}\mathbb{I}[c(q)=c(k)] measures same-domain exposure, using domain identity as a proxy for signal similarity. On Digits-5, switching from Alternating to Clustered increases the client-averaged ratio (Sk(T))2/Qk(T)(S_k(T))^2/Q_k(T) from 0.754 to 1.809 while leaving Qk(T)Q_k(T) unchanged, accompanied by lower own-domain error in both models. On Rotated MNIST, the gap between placements in this ratio increases with less frequent communication, consistent with the larger observed placement effect. These exposure measures provide a quantitative connection between the theoretical mechanism and the real-data results, although they do not directly measure learned signal and noise energies. We will add these calculations and their interpretation to Appendix C.8.

W8: The benefit depends on the test distribution. In Table 9 at h=16h=16, clustering improves own-domain error from 0.300%0.300\% to 0.218%0.218\%, but error across both domains rises from 0.303%0.303\% to 7.214%7.214\%. This tradeoff limits the practical benefit.

A: Our goal is to understand how decentralized communication shapes feature learning and generalization, including how its effects depend on client placement and the prediction objective. Our theoretical analysis evaluates each client on its own distribution, as defined in Eq. (3), and mean own-domain error is the corresponding experimental metric. We additionally report mixture error to examine performance across domains.

Table 9 highlights an important distinction between these objectives: clustering benefits own-domain prediction in this setting but incurs a substantial cost when each client must predict across both domains. This tradeoff clarifies the practical implication of our findings that placement should be chosen according to the distribution each client is expected to serve. We will make this interpretation more explicit.

Thank you again for your time and constructive feedback. In our responses, we clarify the required signal-type assumption and add the missing proofs in the revised appendix. We also report additional signal and noise measurements after fitting, test a communication interval satisfying Eq. (19), provide architecture details and real-data communication analysis, and clarify the tradeoff between own-domain and mixture performance.

As the discussion deadline approaches, please let us know if you have any remaining questions or concerns. We would be happy to address them. If our responses and additional results have resolved your concerns, we would be grateful if you would consider revising your score upward.

Comments on the paper

1 comment

We thank all reviewers for their time and constructive feedback. As the discussion deadline approaches, we would like to briefly summarize the main clarifications and additional results in our responses:

  • Theoretical contributions and assumptions. We explain the additional analytical challenges beyond server averaging (Reviewer t5Xq, W1), clarify the signal-type definition and the additional fitting assumption (Reviewer t5Xq, Q1–Q2), and provide an explicit example showing that the general early-risk bound can be nontrivial under the fitting assumptions (Reviewer YTLo, Q3).

  • Additional experimental evidence. We report signal and noise measurements after fitting and test a communication interval satisfying Eq. (19) (Reviewer XLSP, Q2/W4–W5). We also provide real-data communication analysis and architecture details (Reviewer XLSP, Q3/W6–W7).

  • Scope and practical implications. We clarify the distinction between early placement effects and fitted-stage guarantees (Reviewer YTLo, Q1), provide a mixture-risk extension (Reviewer YTLo, W1), and illustrate how the analysis informs topology and communication-frequency choices (Reviewer YTLo, Q2/W2).

Please let us know if you have any remaining questions or concerns. We would be happy to address them before the discussion closes. If our responses and additional results have resolved your concerns, we would be grateful if you would consider revising your scores upward.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.