- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The work in decentralized learning has historically been on its convergence and consensus analysis. This work instead studies how the consensus-driven decentralized optimization affects the generalization capabilities of the participating clients. To achieve this, the paper studies a simple two-layer CNN model through the lens of feature learning theory, where the features are modeled as signal vectors mixed with noise terms.
Critical strengths
As the convergence and consensus results has matured in the decentralized learning literature, this paper opens a new door to additional analysis of decentralized gradient descent algorithms. The mathematical framework of feature learning chosen to study generalization seems suitable, carrying interesting intuitions.
Critical weaknesses
The mathematical framework used in this paper closely follows Huang et al., 2024, where signal and noise trajectory tracking is used to analyze generalization for federated learning. In its current version, it is unclear what are the key theoretical hurdles that arise when moving to decentralized gradient descent. The paper could put emphasis on this to help better appreciate the results in the main text and the appendix relative to prior work (especially Huang et al., 2024).
Questions for authors
-
It is stated in Sec. 3 that the signal vectors are assumed to be mutually orthogonal. This would make defined in Sec. 4.1 always be , and affect all the subsequent results involving .
-
Eq. (15) in Sec. 4.3 should explicitly be written in an Assumption block for clarity, similar to Assumptions 3.1 and 4.1.
4 comments
Thank you again for your time and constructive feedback. In our responses, we explain the additional analytical challenges beyond server averaging and how we address them through weight tracking, historical maximum coefficient bounds, and an average local-distance potential for convergence. We also clarify that distinct signal types are orthogonal while multiple clients may share a type, and explain the role of the additional fitting assumption.
As the discussion deadline approaches, please let us know if you have any remaining questions or concerns. We would be happy to address them. If our clarifications have resolved your concerns, we would be grateful if you would consider revising your score upward.
Thank you for your responses. Regarding the critical weakness I mentioned in my initial review, I do not see the corrections and clarification applied to the paper in the revised version. That still makes it hard to appreciate the novelty of this work compared to Huang et al., 2024.
Thank you for pointing this out. Our previous revision included a brief technical summary after Assumption 4.1. We have expanded it into a dedicated remark, “Technical challenges beyond the federated setting,” highlighted in blue. The remark clarifies which steps of Huang et al.’s (2024) analysis require new arguments in decentralized training and how our proofs address them.
More broadly, our work shows how peer-to-peer communication shapes client-level generalization. Under limited training and communication budgets, our early-risk bounds distinguish client placements even when the graph spectrum and aggregate signal similarity are identical. Stability-based analyses that rely only on the graph spectrum and aggregate signal similarity cannot distinguish these placements. We also establish convergence and client-level risk guarantees at a common fitted iterate. Finally, we give an explicit sufficient bound on local updates between communication rounds that preserves the well-mixed scale of the early-risk guarantee.
We thank the reviewer again for the constructive feedback, which helped us clarify our contributions and their significance for decentralized learning.
We thank the reviewer for the constructive feedback and for recognizing the relevance of our work. Below, we clarify our contributions relative to prior work and address the two questions.
A: Our work establishes early-risk bounds that capture both communication topology and client placement, together with convergence and client-level risk guarantees at a common fitted iterate. We also derive an explicit sufficient bound on the number of local updates between communication rounds that preserves the well-mixed scale of the early generalization certificate.
The signal–noise framework developed in prior feature-learning studies, including Cao et al. (2022), Kou et al. (2023), and Huang et al. (2024), provides the starting point for our analysis. Establishing these results without exact global synchronization requires additional tools. We develop new analytical techniques to track feature propagation, control coefficient growth, and prove convergence under decentralized mixing.
First, uniform server averaging in the analysis of Huang et al. yields the 1/K noise factor and the χk/K signal-sharing factor. In decentralized training, each gradient contribution propagates with weights that depend on its source, receiver, and update time. We retain these weights in Lemma 4.3 while controlling local logits and activations. Signal contributions accumulate through Sk(T), while a Gram-matrix estimate controls noise energy through the squared propagation weights in Qk(T). Together, these estimates yield the placement-sensitive early-risk certificate.
Second, server aggregation produces common coefficients across clients, a property used in Huang et al.’s coefficient induction. During subsequent local updates, new contributions from a training sample arise only at its source client. Under decentralized training, a propagated coefficient can be larger at another receiver, while its new gradient contribution remains determined by the source model. A large coefficient at the receiver therefore does not directly imply a small loss-derivative magnitude at the source. Lemma A.7 resolves this mismatch by tracking the historical maximum of the coefficients. Mixing alone cannot increase this maximum, and a source that raises it must already have a local coefficient close to that maximum. The source margin can then control further coefficient growth, closing the induction for the coefficient bounds and activation persistence.
Finally, Lemma A.12 establishes convergence using the average of the local distance potentials. By completing the square in each eigenmode of the mixing matrix, we control the interaction between mixing and gradients evaluated at different local models and derive a descent inequality for decentralized gradient descent. We will highlight these technical contributions and explain more explicitly how they extend beyond Huang et al.’s analysis.
A: Orthogonality applies to the distinct signal types, while multiple clients may share the same type. Thus, χk counts the clients sharing client k’s signal. It remains fixed throughout training but may vary across clients and signal assignments. It equals one only when client k is the sole client assigned to its type. In the four-client example in Section 4.2, each type is shared by two clients, giving χk=2. We will clarify this distinction in Section 3.
A: We will state Eq. (15) in a separate assumption block and explicitly identify it as an additional condition for the fitting analysis. This additional condition supports activation persistence in the fitting analysis and is not required for Lemma 4.3 or Eq. (39). Assumptions 3.1 and 4.1 provide the common conditions for both the early and fitting analyses.