- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides a theoretical analysis of decentralized SGD type algorithms in hierarchical networks specifically of 2 layers. They propose a different H0,H1 generalized smoothness analysis which is novel in this setting in the existing literature and propose a gradient clipping based algorithm (clipping client gradients before aggregation and clipping hub steps) and show convergence guarantees across convex and non convex objectives.
Critical strengths
- The authors provide substantial mathematical rigor to a paper in a complex setup with both convex and non-convex formulations.
- Their H0,H1 smoothness is perhaps a novelty and an interesting analysis component. Clipping is also shown to be necessary, which is an interesting observation.
Critical weaknesses
-
I understand that the paper is a theory/optimization paper , but the practical takeaways are very limited relative to the baselines the paper positions it against. The paper poses itself and compares itself with local sgd, decentralized sgd type methods where, clipping and hierarchical aggregation has been done in many ways in literature (even in the paper's own related works cite them). This broader prior work also also consider more fundamental complexities such as non full participation, additive noise, adversarial models (label flip is not really adversarial), non-exact gradient, and also gradient clipping and variance reduction methods, and then go on to show that those algorithms have worked on training anywhere between 100,s 1000s of extremely non-iid scenarios to training (small) language models even in practice. By comparison, the setting in this paper is considerably constrained by its own assumptions and the experiments are not detailed or sufficient enough for us readers to extract any practicial lesson. 1.a. The paper starts with motivations about non-iid and transformers etc, the experiments are convex (logistic,exponential,quadratic) with only two toy examples. Without larger scale experiments with convex and non convex, larger in size of distributed network, non iid-ness, model trained, skewed distribution, etc. in 2026 it is difficult to justify. 1.b. clipping at many levels has been prevalent in federated learning literature, this is not the novel contribution. i think the novelty is around the H0,H1 smoothness class and the specific setting of two layered hierarchy with clipping before aggregation. This can be made much more clear perhaps.
-
The paper does compare itself with local sgd, but the 3 algorithms in the paper (HUBCLIP, GT-CLIP, H2GT-CLIP) fundamentally do not do any local sgd or local steps which may lead to additive errors (e.g. could be because of client drift and or gradient compression etc.) and this regime does not appear to be modeled here which is leaving a gap. How would the analysis and experimental behavior change if the clients are taking local steps?
-
As a decentralized algorithm paper, communication experiments analysis is limited. No ablation over the topology or connectivity or benchmarks against communication against local sgd baselines. fastmix does require gossip communication for every single gradient step, compared tofederated learning/ SGD algorithms, the communication impacts of running such an algorithm in practice is not understandable. The authors mention about 38x message saving against some flat baseline but that is not clear in the context of existing methods of decentralized algorithms.
-
The paper does not go into details about how we are supposed to get the optimal values of all the variables, for example the step size, clipping radius, even constants like H0, H1, the spectral gap etc. These gets even worse if we consider training anything more complex than a logistic regression such as a MLP, a CNN/LSTM or even a transformer. Their own experiments show that assumptions in their theory may or may not even hold in their own toy experiments (e.g. dissimilarity) and the method is not parameter free itself.
-
The paper reads very difficult. It assumes familiarity with subjects such as generalized smoothness etc. and several notations are being deferred to appendix or not defined before use.
Questions for authors
See weaknesses.
2 comments
Dear Reviewer GDhi,
Thank you again for reviewing our paper. Before the discussion closes, could you let us know whether our rebuttal addresses your main concerns? Even a brief reply would be appreciated.
Thank you for the detailed review. Your request for neural networks led to the CNN and transformer experiments now in §7, and while re-checking the communication cost for your third point we found and corrected an undercount of the messages of tracking methods.
Weakness #1 (practical lessons). §7 now ends with four. Clip before aggregation when the start is far from the solution or some clients send large gradients; without clipping the step must be 4 to 32 times smaller, or the method diverges (Figs. 4a, 5, 9a). The product γτ bounds the clipped step and locates the stability boundary for small radii, but γ and τ still need separate tuning (Fig. 9c). One gossip step per round (Kin=1) converged on every topology we tried (Fig. 8). Tracking pays when hubs differ, gossip is cheap and high accuracy is needed (Figs. 3c, 4b, 4c, 9a); it costs a second gossip exchange per round and gave no measurable gain on the CNN and the transformer.
Weakness #1 (settings of prior work). The paper covers partial participation and stochastic gradients at the federated level under strong growth (Theorem 6), heavy-tailed noise at that level (Theorem 2), per-round differential privacy (Proposition 4), and stochastic clients in the hierarchy with full participation and a growing batch (Corollary 1). Additive noise at the decentralized and hierarchical levels and Byzantine clients are not covered, and the Limitations paragraph says so.
Weakness #1.a (convex experiments). The convex runs let us compute f⋆ and R and test the theorems. We added three nonconvex experiments in hierarchies of 8 hubs and 8 clients per hub with heterogeneous splits: a two-layer network (App. H) and, in §7 (Figure 5), a CNN on Fashion-MNIST and a GPT-style transformer with 3.3M parameters on WikiText-2, both with mini-batches and three seeds:
Clipping before aggregation wins on every seed. HD-SGD with our client clipping is our method without tracking, so on these tasks the gain comes from clipping.
Weakness #1.b (novelty). Agreed: §1 now says that clipping, tracking and Chebyshev gossip are known tools and that the contribution is the analysis (see the reply to e3rH). One correction to the strengths: Proposition 3 shows that unclipped tracking with a constant step can diverge in this class; it does not show that clipping is the only remedy.
Weakness #2 (local steps). L local clipped steps keep the analysis within the locality radius when Lγℓτ≤1/H1, but the accumulated direction also needs alignment and tracking arguments that we have not proved; §6 says so. Fig. 9(b) tests one variant, in which clients return the normalized displacement: rounds to 10−2 are 35 or 36 for every L∈{1,2,5,10}, and grow from 149 to 181 at 10−4. Other designs are not ruled out.
Weakness #3 (communication). Fig. 8 varies n∈{4,9,16,25} and the topology, and counts hub messages to 10−2 against hierarchical local GD, the structure of SD-FEEL and multi-level local SGD. A round of our method costs 4∣E∣Kin messages, since the tracker exchange uses gradients at the newly mixed models. With this count our method needs 1.3 to 2.3 times fewer messages for n≥16 on the torus and the matching graph and on the ring with n=9, 1.7 times more on the torus with n=9, and about twice as many at n=4; the baseline misses the target on the ring with n≥16. The 38× of §7 is 50× with the corrected count.
Weakness #4 (parameters). The method is not parameter-free. Theorem 5 prescribes γ=1/(180H^0), τ=H^0/(H1R~) and Kin=⌈log(2048nΓ)/2(1−λ)⌉, and the run of Fig. 3(b) uses these values. On real data we tune γ and τ on a grid, and Kin=4 already matches the prescribed 18 (Fig. 8c). The dispersion thresholds are violated on real data, and the method still converges (App. F).
Weakness #5 (readability). δ is defined at first use, and Proposition 1 no longer refers forward to Lemma 6.