- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes GASLoC, a decentralized framework for LLM pretraining that combines blocks of local optimization steps, randomized exchanges with one or two peers, and an outer optimizer applied to a pseudo-gradient constructed from the received parameters. The framework permits worker-specific numbers of local steps to accommodate system heterogeneity. The paper claims a nonconvex convergence guarantee under a shared objective and interprets outer momentum as a mechanism for accelerating consensus. Evaluation covers 134M and 551M dense Transformers and OLMoE-1B-7B, with comparisons of validation loss, downstream performance, and training time under limited and heterogeneous bandwidth.
Critical strengths
-
The paper provides useful empirical evidence that the proposed outer update can improve training with sparse communication. The evaluation includes both dense and MoE language models. In particular, Table 10 reports four-seed results for the 134M model with 16 replicas: with H=1 and the same randomized 2-Peer topology, GASLoC achieves a validation loss of 3.219 ± 0.008, compared with 3.277 ± 0.007 for DAdam. This is a concrete, controlled improvement, and the multi-seed evaluation strengthens the empirical case for the method in this setting.
-
The framework addresses a practically relevant communication constraint and includes measured system-level validation. Allowing different local step counts while using sparse peer exchanges provides a potentially useful way to accommodate bandwidth heterogeneity. Section 4.3 reports an approximately 16% reduction in end-to-end training time for GASLoC-1-Peer relative to DiLoCo in an eight-GPU bandwidth-straggler experiment at a matched token budget.
Critical weaknesses
The current proof does not establish Proposition 2.
1.The eigenvalue bound does not imply the claimed norm contraction.
On p. 22, lines 1160-1163, the proof concludes that ||A X_t|| <= q ||X_t|| because the eigenvalues have modulus at most q = 1 - sqrt(alpha/chi). This implication is not valid for the Euclidean norm used here.
For example, set chi = 2, alpha = 1/2, and lambda = 1/2. Then q = 1/2 and gamma = 1/4. The corresponding 2-by-2 block is B = [[1, -1/4], [1, 0]]. Its eigenvalues are both 1/2, but it maps the unit column vector (1, 0) to (1, 1), whose norm is sqrt(2). The claimed bound would require a norm of at most 1/2.
This contradicts the stated contraction. The accelerated-consensus bound therefore needs a corrected proof that also accounts for local updates.
2.The stochastic disagreement bound omits a noise contribution.
On p. 18, line 918, G_t averages true gradients along local trajectories, whereas g_t in Eq. (8) uses stochastic gradients. Yet the disagreement bound in lines 933-935 uses ||Pi G_t||^2 without accounting for the noise in g_t. The authors should explain how this contribution is controlled and whether the correction changes Proposition 2.
Questions for authors
-
Can the authors correct the contraction and stochastic-noise issues in W1 and clarify whether Proposition 2, including the claimed acceleration, still holds?
-
Which update is used in the experiments, and how does it correspond to Algorithm 1, Eqs. (1)-(2), and the recurrence analyzed in Lemma 1(iii)?
3 comments
Q1 - Do the results survive the corrections?
Revised Proposition 2 covers zero momentum. Proposition 3 covers the accelerated recurrence on a fixed connected graph and bounds the powers of A directly (see W1). The stochastic-noise correction adds terms to both finite-time bounds (see W2), and the tuned rate in Eq. 36 remains for .
The acceleration still holds. Eq. 39 gives , where . Thus communication rounds suffice to reduce the disagreement state norm by a factor . Standard gossip gives instead rounds. The accelerated consensus bound therefore retains its square root dependence on .
Q2 - Update used in the experiments
The experiments implement Algorithm 1, using PyTorch AdamW for the local steps and PyTorch SGD with nesterov=True for the outer step. The outer optimizer takes a pseudo-gradient averaged over the current neighborhood as described in line 13 of the Algorithm.
Eq. 1 presents the inner loop with an SGD optimizer. Eq. 2 and 34 present the outer loop using extrapolations with momentum, a simplified model of the empirical approach. Proposition 3 analyzes recurrence defined by Eqs. 33 and 34.
Dear reviewer, we are following up before the discussion period closes on Nov 18. We would be grateful to know whether our reply addresses your concerns. If it does, we would appreciate it if you would consider updating your score.
We thank the reviewer for taking the time to review our work. We acknowledge that our proof incorrectly assumed that the spectral norm equals the spectral radius, an identity that holds for symmetric matrices. A spectral radius below one guarantees eventual geometric decay of the matrix powers, even if their norms initially grow. Proposition 3 replaces the accelerated case of Lemma 1(iii) and gives an explicit bound on the matrix powers to control the cumulative effect of local updates. We have highlighted the changes in cyan in the revised manuscript.
In our view, the paper’s primary contribution is empirical. Our experiments show that decentralized LLM pretraining with sparse communication and multiple local steps remains competitive with DiLoCo, and that adapting local step counts to heterogeneous bandwidth reduces end-to-end training time. The theoretical analysis is a secondary contribution. We use it to highlight a connection between outer momentum in local optimization and classical Chebyshev acceleration in gossip, which is often overlooked.
W1 - Euclidean contraction and local updates
The updated accelerated guarantee in Proposition 3 (Appendix A) bounds the powers of matrix A directly. With q=χ+1χ−1, Cayley-Hamilton gives Eq. 39: ∥Ak∥≤4kqk−1, k≥1. Eq. 38 includes the local updates as inputs at each round. Unrolling the recurrence and applying Young’s convolution inequality gives the cumulative disagreement bound in Eq. 40.
W2 - Stochastic gradient noise
The updated disagreement bound uses the stochastic update gt, not Gt. Since gt=−β(Gt+Nt), where Nt is the averaged gradient noise. Using πNt∥≤∥Nt∥ and ∥a+b∥2≤2∥a∥2+2∥b∥2 gives
\le 2\beta^2 E\|\pi G_t\|^2 +2\beta^2\sigma^2\sum_{i=1}^n\frac{1}{H_i}$$ (Appendix A, lines 1018–1021). In Proposition 2, the correction adds $12e^2\left(\frac{\rho\eta\beta L}{\alpha}\right)^2\sigma^2 h$ to the bound. For fixed $\eta$ and gossip parameters, the added term is $O(T^{-1})$ with $\beta\propto T^{-1/2}$, so the submission's rate is preserved. In Proposition 3, the corresponding term in Eq. 35 scales as $\chi\sigma^2h/T$ under the tuning in Eq. 36. Since $\chi/T\le\sqrt{\chi}/\sqrt{T}$ for $T\ge\chi$, the rate state in the submission is preserved.