acceptodds
Under review as a conference paper at ICLR 2027

Unifying Local Communication and Local Updates for LLM Pretraining

Abstract

Several methods reduce the communication cost of distributed LLM pre-training by performing multiple local optimizer steps between global aggregations. However, these methods still rely on global collectives, whose duration is limited by the available bandwidth and the slowest worker. We introduce GASLoC, a decentralized method in which each worker performs one or more local optimizer steps, exchanges the resulting parameters with one or two randomly selected peers, and applies an outer optimizer to the exchanged update. This structure avoids global collectives and allows the number of local steps to vary across workers. We establish convergence under different local step counts and show that outer momentum can accelerate consensus. Empirically, at token budgets selected according to compute optimal scaling, the strongest GASLoC variant achieves lower validation loss than the evaluated decentralized baselines when communicating after every local step. With multiple local steps between communications, it remains competitive with DiLoCo while avoiding global collectives. Under heterogeneous bandwidth, adapting the number of local steps across workers reduces end-to-end training time relative to DiLoCo.

Primary area:
optimization
Keywords:
decentralized learning, gossip, outer momentum

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Q7bLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GASLoC, a decentralized framework for LLM pretraining that combines blocks of local optimization steps, randomized exchanges with one or two peers, and an outer optimizer applied to a pseudo-gradient constructed from the received parameters. The framework permits worker-specific numbers of local steps to accommodate system heterogeneity. The paper claims a nonconvex convergence guarantee under a shared objective and interprets outer momentum as a mechanism for accelerating consensus. Evaluation covers 134M and 551M dense Transformers and OLMoE-1B-7B, with comparisons of validation loss, downstream performance, and training time under limited and heterogeneous bandwidth.

Critical strengths
  1. The paper provides useful empirical evidence that the proposed outer update can improve training with sparse communication. The evaluation includes both dense and MoE language models. In particular, Table 10 reports four-seed results for the 134M model with 16 replicas: with H=1 and the same randomized 2-Peer topology, GASLoC achieves a validation loss of 3.219 ± 0.008, compared with 3.277 ± 0.007 for DAdam. This is a concrete, controlled improvement, and the multi-seed evaluation strengthens the empirical case for the method in this setting.

  2. The framework addresses a practically relevant communication constraint and includes measured system-level validation. Allowing different local step counts while using sparse peer exchanges provides a potentially useful way to accommodate bandwidth heterogeneity. Section 4.3 reports an approximately 16% reduction in end-to-end training time for GASLoC-1-Peer relative to DiLoCo in an eight-GPU bandwidth-straggler experiment at a matched token budget.

Critical weaknesses

The current proof does not establish Proposition 2.

1.The eigenvalue bound does not imply the claimed norm contraction.

On p. 22, lines 1160-1163, the proof concludes that ||A X_t|| <= q ||X_t|| because the eigenvalues have modulus at most q = 1 - sqrt(alpha/chi). This implication is not valid for the Euclidean norm used here.

For example, set chi = 2, alpha = 1/2, and lambda = 1/2. Then q = 1/2 and gamma = 1/4. The corresponding 2-by-2 block is B = [[1, -1/4], [1, 0]]. Its eigenvalues are both 1/2, but it maps the unit column vector (1, 0) to (1, 1), whose norm is sqrt(2). The claimed bound would require a norm of at most 1/2.

This contradicts the stated contraction. The accelerated-consensus bound therefore needs a corrected proof that also accounts for local updates.

2.The stochastic disagreement bound omits a noise contribution.

On p. 18, line 918, G_t averages true gradients along local trajectories, whereas g_t in Eq. (8) uses stochastic gradients. Yet the disagreement bound in lines 933-935 uses ||Pi G_t||^2 without accounting for the noise in g_t. The authors should explain how this contribution is controlled and whether the correction changes Proposition 2.

Questions for authors
  1. Can the authors correct the contraction and stochastic-noise issues in W1 and clarify whether Proposition 2, including the claimed acceleration, still holds?

  2. Which update is used in the experiments, and how does it correspond to Algorithm 1, Eqs. (1)-(2), and the recurrence analyzed in Lemma 1(iii)?

3 comments

We thank the reviewer for taking the time to review our work. We acknowledge that our proof incorrectly assumed that the spectral norm equals the spectral radius, an identity that holds for symmetric matrices. A spectral radius below one guarantees eventual geometric decay of the matrix powers, even if their norms initially grow. Proposition 3 replaces the accelerated case of Lemma 1(iii) and gives an explicit bound on the matrix powers to control the cumulative effect of local updates. We have highlighted the changes in cyan in the revised manuscript.

In our view, the paper’s primary contribution is empirical. Our experiments show that decentralized LLM pretraining with sparse communication and multiple local steps remains competitive with DiLoCo, and that adapting local step counts to heterogeneous bandwidth reduces end-to-end training time. The theoretical analysis is a secondary contribution. We use it to highlight a connection between outer momentum in local optimization and classical Chebyshev acceleration in gossip, which is often overlooked.

W1 - Euclidean contraction and local updates

The updated accelerated guarantee in Proposition 3 (Appendix A) bounds the powers of matrix A directly. With q=χ−1χ+1q = \frac{\sqrt{\chi}-1}{\sqrt{\chi}+1}, Cayley-Hamilton gives Eq. 39: ∥Ak∥≤4k qk−1, k≥1.\|A^k\| \le 4k\,q^{k-1}, \ k \ge 1. Eq. 38 includes the local updates as inputs at each round. Unrolling the recurrence and applying Young’s convolution inequality gives the cumulative disagreement bound in Eq. 40.

W2 - Stochastic gradient noise

The updated disagreement bound uses the stochastic update gtg_t, not GtG_t. Since gt=−β(Gt+Nt)g_t = -\beta(G_t + N_t), where NtN_t is the averaged gradient noise. Using πNt∥≤∥Nt∥\pi N_t\|\le\|N_t\| and ∥a+b∥2≤2∥a∥2+2∥b∥2\|a+b\|^2\le2\|a\|^2+2\|b\|^2 gives

\le 2\beta^2 E\|\pi G_t\|^2 +2\beta^2\sigma^2\sum_{i=1}^n\frac{1}{H_i}$$ (Appendix A, lines 1018–1021). In Proposition 2, the correction adds $12e^2\left(\frac{\rho\eta\beta L}{\alpha}\right)^2\sigma^2 h$ to the bound. For fixed $\eta$ and gossip parameters, the added term is $O(T^{-1})$ with $\beta\propto T^{-1/2}$, so the submission's rate is preserved. In Proposition 3, the corresponding term in Eq. 35 scales as $\chi\sigma^2h/T$ under the tuning in Eq. 36. Since $\chi/T\le\sqrt{\chi}/\sqrt{T}$ for $T\ge\chi$, the rate state in the submission is preserved.

Q1 - Do the results survive the corrections?

Revised Proposition 2 covers zero momentum. Proposition 3 covers the accelerated recurrence on a fixed connected graph and bounds the powers of A directly (see W1). The stochastic-noise correction adds terms to both finite-time bounds (see W2), and the tuned rate in Eq. 36 remains O(χ/T)O(\sqrt{\chi}/\sqrt{T}) for T≥χT\ge\chi.

The acceleration still holds. Eq. 39 gives ∥Xk∥≤4kqk−1∥X0∥\|X_k\|\le 4kq^{k-1}\|X_0\|, where q=(χ−1)/(χ+1)q=(\sqrt{\chi}-1)/(\sqrt{\chi}+1). Thus O(χlog⁡(χ/ε))O(\sqrt{\chi}\log(\chi/\varepsilon)) communication rounds suffice to reduce the disagreement state norm by a factor ε\varepsilon. Standard gossip gives instead O((χ/α)log⁡(1/ε))O((\chi/\alpha)\log(1/\varepsilon)) rounds. The accelerated consensus bound therefore retains its square root dependence on χ\chi.

Q2 - Update used in the experiments

The experiments implement Algorithm 1, using PyTorch AdamW for the local steps and PyTorch SGD with nesterov=True for the outer step. The outer optimizer takes a pseudo-gradient averaged over the current neighborhood as described in line 13 of the Algorithm.

Eq. 1 presents the inner loop with an SGD optimizer. Eq. 2 and 34 present the outer loop using extrapolations with momentum, a simplified model of the empirical approach. Proposition 3 analyzes recurrence defined by Eqs. 33 and 34.

Dear reviewer, we are following up before the discussion period closes on Nov 18. We would be grateful to know whether our reply addresses your concerns. If it does, we would appreciate it if you would consider updating your score.

Reviewer_bu5SOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GASLoC, a decentralized method for LLM pretraining that combines blocks of local optimization with sparse peer communication and an outer optimizer. Workers exchange models with one or two randomly selected peers, and the number of local steps may differ across workers. The paper studies the mixing properties of these communication schemes and states a nonconvex convergence guarantee.

Critical strengths

The paper studies an important and potentially impactful problem: reducing communication and synchronization overhead in LLM pretraining, particularly under limited or heterogeneous connectivity.

Critical weaknesses

1. The accelerated-gossip proof contains an error.

On page 22, the proof infers the Euclidean contraction ∣AX∣≤q∣X∣\\|AX\\|\le q\\|X\\|, where q=1−α/χ<1q=1-\sqrt{\alpha/\chi}<1, from the eigenvalues of the augmented recurrence matrix AA. This implication is invalid for the stated matrix. Taking X=(u,0)X=(u,0) for a nonzero disagreement vector uu gives AX=(((1+γ)I−αΛ)u,u)AX=(((1+\gamma)I-\alpha\Lambda)u,u), hence ∣AX∣≥∣X∣\\|AX\\|\ge \\|X\\|, contradicting the claimed contraction.

2. The theoretical contribution is insufficiently positioned against existing results.

The most direct missing comparison is OLDSGD (Zhou and Pu, 2026), which provides convergence guarantees for decentralized local updates with communication–computation overlap. SwarmSGD (Nadiradze et al., 2021) also analyzes asynchronous pairwise communication combined with local updates. More broadly, Cooperative SGD (Wang and Joshi, 2021) and Koloskova et al. (2020) already provide frameworks covering local updates and decentralized communication. The paper needs an explicit comparison of assumptions, convergence rates, and oracle/communication complexities with the closest results to establish what its guarantee adds.

In particular, Proposition 2 assumes a common objective and samplewise LL-smoothness, whereas OLDSGD and the nonconvex analysis of Koloskova et al. accommodate heterogeneous local objectives and require smoothness only of the expected local objectives. The paper should explain what improvement or additional capability its guarantee provides under these stronger restrictions.

Other relevant theoretical work includes PowerGossip (Vogels et al., 2020), which studies low-rank compression of neighbor exchanges, and LOSCAR-SGD (Maziane et al., 2026), which analyzes overlapping communication with local computation in a server-based setting. Their different mechanisms and communication models should be discussed when positioning the contribution.

3. Important experimental baselines are missing or only partially evaluated.

The evaluation omits OLDSGD (Zhou and Pu, 2026), the most important missing baseline given its combination of decentralized local updates and communication–computation overlap. Factored Gossip DiLoCo (Koneputugodage et al., 2026), which includes a peer-only variant with overlapping communication, is another close comparator that is not evaluated. NoLoCo (Kolehmainen et al., 2025) is evaluated only in a subset of settings.

Without these comparisons, the experiments do not establish whether GASLoC offers an advantage over the closest existing decentralized methods.

Questions for authors

See weaknesses

3 comments

We thank the reviewer for taking the time to review our work. We acknowledge that our proof incorrectly assumed that the spectral norm equals the spectral radius, an identity that holds for symmetric matrices. A spectral radius below one guarantees eventual geometric decay of the matrix powers, even if their norms initially grow. Proposition 3 replaces the accelerated case of Lemma 1(iii) and gives an explicit bound on the matrix powers to control the cumulative effect of local updates. We have highlighted the changes in cyan in the revised manuscript.

In our view, the paper’s primary contribution is empirical. Our experiments show that decentralized LLM pretraining with sparse communication and multiple local steps remains competitive with DiLoCo, and that adapting local step counts to heterogeneous bandwidth reduces end-to-end training time. The theoretical analysis is a secondary contribution. We use it to highlight a connection between outer momentum in local optimization and classical Chebyshev acceleration in gossip, which is often overlooked.

W1 - Euclidean contraction

The updated accelerated guarantee in Proposition 3 (Appendix A) bounds the powers of matrix A directly. With q=χ−1χ+1q = \frac{\sqrt{\chi}-1}{\sqrt{\chi}+1}, Cayley-Hamilton gives Eq. 39: ∥Ak∥≤4k qk−1, k≥1.\|A^k\| \le 4k\,q^{k-1}, \ k \ge 1. Applying Young’s convolution inequality to Eq. 39 gives the cumulative disagreement bound in Eq. 40, which replaces the one-step contraction in the proof.

W2 - Theoretical contribution is insufficiently positioned against the literature

Assumptions. The assumptions in Proposition 2 and 3 were relaxed to require L-smoothness of the expected objective f. This is the same strength level at which OLDSGD, SwarmSGD, Cooperative SGD, and the nonconvex analysis of Koloskova et al. impose smoothness. SwarmSGD additionally assumes a uniformly bounded second moment of the stochastic gradients, E∣g~i(x)∣2≤G2\mathbb E|\tilde g_i(x)|^2\le G^2, which Propositions 2 and 3 do not require. GASLoC retains the common objective assumption as its targets are LLM pretraining tasks, where all workers sample iid shards from a shared corpus. This guarantees:

  • Proposition 2 allows each worker to take a different number of local steps HiH_i, while OLDSGD and Cooperative SGD require a common local-step interval, Koloskova et al. a shared communication schedule, and SwarmSGD uses random local step counts with common mean.
  • GASLoC gives explicit contraction constants for the random 1-Peer and 2-Peer schemes (Proposition 1 and Lemma 1(ii)). These constants remain bounded as the worker count grows at fixed α\alpha.
  • GASLoC analyses the use of an outer optimizer with and without momentum (Proposition 2 and 3).

The table compares leading terms for reaching an average expected squared gradient norm of at most ϵ\epsilon. Here h=1n∑i1/Hih=\frac1n\sum_i1/H_i, and n is the worker count. The leading terms target an average expected squared gradient norm of at most ϵ\epsilon.

Leading complexityUnit
OLDSGDO((Δ+Lσ2)2nϵ2)O\left(\frac{(\Delta+L\sigma^2)^2}{n\epsilon^2}\right)Local steps per worker
Cooperative SGDO((LΔ+σ2)2nϵ2)O\left(\frac{(L\Delta+\sigma^2)^2}{n\epsilon^2}\right)Local steps per worker
Koloskova et al.O(LΔσ^2nϵ2)O\left(\frac{L\Delta\hat\sigma^2}{n\epsilon^2}\right)Local steps per worker
SwarmSGDO((Δ/H+σ2+Hς2)2ϵ2)O\left(\frac{(\Delta/H+\sigma^2+H\varsigma^2)^2}{\epsilon^2}\right)Pairwise interactions
GASLoCO(LΔσ2hnϵ2)O\left(\frac{L\Delta\sigma^2 h}{n\epsilon^2}\right)Gossip rounds

GASLoC uses TT rounds and THiTH_i gradient calls on worker ii. For Hi=HH_i=H, its leading term is O(σ2/(nHϵ2))O(\sigma^2/(nH\epsilon^2)) rounds and O(σ2/(nϵ2))O(\sigma^2/(n\epsilon^2)) calls per worker, matching the leading oracle scaling of Koloskova et al.

We have now included this comparison in the text as Appendix F.

Communication mechanism. PowerGossip compresses the model differences exchanged between neighbours into low-rank messages. It assumes a fixed mixing matrix, and its compressor is warm-started from state kept on each connection. It therefore does not directly apply to the fresh random peers that GASLoC samples each round. On the other hand, LOSCAR-SGD relies on a central server to average shared random subset of coordinates. GASLoC is instead a fully decentralized approach.

W3 - OLDSGD and Factored DiLoCo

OLDSGD is a special case of our Local‑DAdam baseline (Appendix C) with SGD inner steps, outer SGD with fixed learning rate 1. To provide a fair comparison with other methods in the LLM pretraining task considered in the paper, we tested this method with local AdamW. Its performance is worse than GASLoC in the tested settings. Factored Gossip DiLoCo differs from traditional gossip methods because it exchanges pseudo-gradients over a gossip graph. Mixing pseudo-gradients alone does not drive the models toward consensus, so the method needs also to mix round-start models. GASLoC, instead, exchanges only post-local models once per round and computes the outer pseudo-gradient locally. This follows standard gossip in mixing model parameters. On matched topologies, GASLoC achieves comparable validation loss with half the communication volume.

We included these results in Table 3.

Dear reviewer, we are following up before the discussion period closes on Nov 18. We would be grateful to know whether our reply addresses your concerns. If it does, we would appreciate it if you would consider updating your score.

Reviewer_eEhMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes GASLoC for decentralized LLM pretraining. Each worker performs local updates, exchanges parameters with a small set of peers, and applies an outer optimizer to the resulting pseudo-gradient. The framework recovers DiLoCo under complete communication and permits different local step counts across workers. The paper relates outer momentum to faster consensus, analyzes convergence for a shared nonconvex objective, and evaluates dense and MoE language models. The strongest variants achieve lower validation loss than the evaluated gossip baselines. A bandwidth experiment shows reduced training time.

Critical strengths
  1. Clear design and an insightful motivation. The reviewer finds the interpretation of outer momentum as a way to reduce information-propagation delay useful. The inner/outer optimizer structure makes the connection between local updates and sparse communication easy to follow. This gives a concrete perspective on why optimizer memory may matter in decentralized training.

  2. Promising results against the evaluated gossip baselines. Table 3 shows consistent gains across topologies and worker counts.

Critical weaknesses
  1. Two gaps in the momentum convergence proof.

    The consensus bound in Lemma 1(iii) does not hold in the standard Euclidean norm. In Appendix A, p. 22, lines 1142–1163, the proof defines Xt=(πxt,πxt−1)⊤X_t=(\pi x_t,\pi x_{t-1})^\top and considers

    A=((1+γ)I−αΛ−γII0).A=\begin{pmatrix}(1+\gamma)I-\alpha\Lambda&-\gamma I\\\\ I&0\end{pmatrix}.

    Here q=1−α/χ,γ=q2q=1-\sqrt{\alpha/\chi},\qquad\gamma=q^2. The proof bounds the eigenvalue moduli, then concludes that ∣AX∣≤q∣X∣\\|AX\\|\le q\\|X\\|. A spectral-radius bound does not imply this Euclidean contraction, and no alternative state norm is defined. For nonzero uu with 1⊤u=0\mathbf1^\top u=0, the state X=(u,0)⊤X=(u,0)^\top gives

    \\|AX\\|_2^2 &=\\|((1+\gamma)I-\alpha\Lambda)u\\|_2^2\\\\ &\quad+\\|u\\|_2^2\ge\\|X\\|_2^2, \end{aligned}$$ which contradicts contraction for $0 < q < 1$. Common initialization does not resolve the issue in the lemma's perturbed system. With $\Lambda=\pi$, $\chi=1$, $\alpha=1/2$, $X_0=0$, $\pi y_0=2u$ and $\pi y_1=0$, unit $u$ gives an energy increment $6-4\sqrt2 > 0$, while the lemma requires at most $-1/2$. Changing to another fixed norm does not recover the exact factor $q$ either: the slowest mode has $B=qI+N$, $N\ne0$, $N^2=0$, so $B^t=q^tI+tq^{t-1}N$. A weighted norm may give a slower contraction, but its relation to actual disagreement and its amplification of local updates still need to be established. **The descent proof uses the mean update without momentum.** Proposition 2 assumes Eq. (1), which has no momentum, but also lists an accelerated branch with nonzero momentum. Appendix Eqs. (10)–(15) use $\bar x_{t+1}=\bar x_t+\eta\bar g_t$. In contrast, Eq. (2), with $z_t=x_t+\eta g_t$, gives the mean update $$\bar x_{t+1}=\bar x_t+\eta\bar g_t+\gamma(\bar z_t-\bar z_{t-1}).$$ Averaging removes the Laplacian term but retains momentum. Eq. (8) defines $g_t$ from the current local gradients, without the explicit outer-momentum increment. The variable $\bar x_t$ is the actual parameter average, not an auxiliary iterate. These definitions therefore do not account for the missing term. The disagreement recurrence also omits the $\gamma\eta\pi(g_t-g_{t-1})$ term generated by Eq. (2). The mismatch remains even when all workers agree exactly. Consider $f(x)=x^2/2$, identical deterministic workers, $H_i=1$, and parameters that satisfy the listed step-size bounds: $L=1$, $\alpha=0.5$, $\chi=100$, $\beta=0.05$, $\eta=0.16$, with the prescribed $\gamma$. A normalized four-node cycle with alternating edge weights $0.99$ and $0.01$ gives $\chi=100$. Initialize at $x_0=1000$ and use Eq. (1) to obtain $x_1=992$ before applying Eq. (2). The resulting scalar recurrence is $x_{t+1}=(1-0.008)((1+\gamma)x_t-\gamma x_{t-1})$. This gives $f(x_{49})\approx0.00170$ and $f(x_{50})\approx1.64955$, an increase of about $1.648$. With zero noise and disagreement, however, the descent inequality at lines 984–985 would require $\Delta f\le-\eta\beta\\|\nabla f(x_{49})\\|^2/4 < 0$. This failure is therefore independent of the consensus-bound error. Replacing the reference to Eq. (1) with Eq. (2) alone would not address it. Both gaps affect the claimed $\chi\to\sqrt\chi$ improvement in the full training guarantee. At lines 933–938, the proof uses Lemma 1 to supply a negative disagreement term that cancels the positive one in the objective analysis. The value of $\rho$ also controls the allowed step size, $\eta\beta\le\alpha/(3\sqrt3\\,\rho L)$. A valid proof needs to control transient behavior and the amplification of local updates. It also needs to analyze the actual mean update with momentum, for example through a velocity-dependent Lyapunov function or a justified auxiliary iterate. The step-size and graph dependence then need to be checked again to determine whether $\rho=\sqrt\chi$ still holds. These are substantive proof gaps; changing an equation reference or a constant would not resolve them. The counterexamples disprove the stated Euclidean lemma and the one-step descent when applied to Eq. (2). They do not disprove the literal Eq. (1) theorem or its final time-averaged bound, classical consensus acceleration, the zero-momentum branches, or the empirical observations.
  2. The momentum gains and the time-to-quality advantage need a clearer explanation. Figure 8 shows that outer momentum improves validation loss under sparse communication. But loss alone does not show how much comes from better information mixing, how much also occurs with complete communication, or how these effects interact. The same optimizer ordering is reported under complete communication. Figure 9 also compares hyperparameter sensitivity for sparse and complete communication. These are useful checks, but they do not directly test the proposed consensus explanation. The link from this fixed-graph interpretation to randomized-peer training with local AdamW and outer Nesterov momentum also needs to be clearer.

Questions for authors
  1. Why should “momentum helps” imply that the benefit comes from consensus? Can the authors separate the consensus effect from the optimization effect, for example by tracking disagreement alongside loss under matched optimizer settings?
  2. After tuning each method's communication interval and schedule, does GASLoC still reach the same validation quality in less wall-clock time than DiLoCo? A time-to-validation-quality comparison would directly address the practical claim.
3 comments

We thank the reviewer for taking the time to review our work. We acknowledge that our proof incorrectly assumed that the spectral norm equals the spectral radius, an identity that holds for symmetric matrices. A spectral radius below one guarantees eventual geometric decay of the matrix powers, even if their norms initially grow. Proposition 3 replaces the accelerated case of Lemma 1(iii) and gives an explicit bound on the matrix powers to control the cumulative effect of local updates. We have highlighted the changes in cyan in the revised manuscript.

In our view, the paper’s primary contribution is empirical. Our experiments show that decentralized LLM pretraining with sparse communication and multiple local steps remains competitive with DiLoCo, and that adapting local step counts to heterogeneous bandwidth reduces end-to-end training time. The theoretical analysis is a secondary contribution. We use it to highlight a connection between outer momentum in local optimization and classical Chebyshev acceleration in gossip, which is often overlooked.

W1 - Momentum accelerated proof

Consensus bound. The updated accelerated guarantee in Proposition 3 (Appendix A) bounds the powers of matrix A directly. With q=χ−1χ+1q = \frac{\sqrt{\chi}-1}{\sqrt{\chi}+1}, Cayley-Hamilton gives Eq. 39: ∥Ak∥≤4k qk−1, k≥1.\|A^k\| \le 4k\,q^{k-1}, \ k \ge 1. The factor kk accounts for the transient behavior at repeated roots. Eq. 38 includes the local updates as inputs to the recurrence. Unrolling it and applying Young’s convolution inequality gives Eq. 40, which bounds cumulative squared model disagreement by 16η2χ216\eta^2\chi^2 times the cumulative squared disagreement in the local updates.

Mean update. Proposition 3 uses the mean update with momentum, xˉt+1=xˉt+ηgˉt+γsˉt\bar x_{t+1} = \bar x_t + \eta \bar g_t + \gamma \bar s_t, where sˉt=yˉt−yˉt−1\bar s_t = \bar y_t - \bar y_{t-1} and yt=xt+ηgty_t = x_t + \eta g_t. The shifted average x^t=xˉt+γ21−γsˉt−1\hat x_t = \bar x_t + \frac{\gamma^2}{1-\gamma}\bar s_{t-1} satisfies x^t+1=x^t+η1−γ gˉt\hat x_{t+1} = \hat x_t + \frac{\eta}{1-\gamma}\,\bar g_t \quad by Eq. 37. The proof then applies the smoothness inequality to f(x^t)f(\hat x_t) in Eq. 42 and controls the shift using Eq. 41. The revised disagreement recurrence, written in terms of πyt\pi y_t, retains the contribution γηπ(gt−gt−1)\gamma\eta\pi(g_t-g_{t-1}) from Eq. 2. The proof sums the smoothness inequality along the shifted iterates x^t\hat x_t to control the shift back to xˉt\bar x_t.

Guarantee. With the tuning in Eq. 36 and T≥χT\ge\chi, Proposition 3 bounds the time-averaged expected squared gradient norm at xˉt\bar x_t by O(χ/T)O(\sqrt{\chi}/\sqrt{T}).

W2 and Q1 - Momentum effect

We compared outer Nesterov momentum (γ=0.7\gamma=0.7) with outer SGD (γ=0\gamma=0) on a fixed ring and with complete averaging. Figure 10b shows training loss for all four runs. The final validation losses were

TopologyWith momentumWithout momentumLoss reduction
Ring3.423.660.24
Complete3.403.600.20

Optimization. With complete averaging, all replicas share the same model after each outer round, so the consensus distance Dt=1n∑i∥xi,t−xˉt∥2D_t=\frac1n\sum_i\|x_{i,t}-\bar x_t\|^2 is zero at that point. The 0.20 loss reduction shows that outer momentum improves optimization even with zero post-round disagreement.

Consensus. On the 16-worker ring with H=30H=30, Figure 10a shows that momentum raises and advances the peak in DtD_t, followed by faster decay late in training. The final DtD_t with momentum is about 42% of the value with outer SGD. Under the tuning in Proposition 3, momentum can initially amplify disagreement from a single local update. With no further inputs, Eq. 39 bounds the rounds needed to reduce that disagreement to tolerance ϵ\epsilon by O(χlog⁡(χ/ϵ))O(\sqrt{\chi}\log(\chi/\epsilon)), compared with O((χ/α)log⁡(1/ϵ))O((\chi/\alpha)\log(1/\epsilon)) for standard gossip.

Momentum narrows the final validation-loss gap between the ring and complete averaging from 0.06 to 0.02. The additional 0.04 gain on the ring can be attributed to a contribution of faster mixing.

Appendix E.1.3 reports this ablation.

Link to the training setting.

Local AdamW. For the fixed-graph outer recursion analyzed in Proposition 3, the local optimizer enters Eq. (38) through the net local updates gtg_t. The matrix-power bound in Eq. 39 is independent of how these updates are computed, so, if all workers start from the same model, the bound in Eq. 40 applies to any sequence of local updates, including those from AdamW.

Outer Nesterov. In training, we apply PyTorch’s Nesterov momentum to the difference between each worker’s starting model and the peer average of the post local updates models (Algorithm 1). Eq. 2 and 34 use extrapolations with momentum, a simplified model of the empirical approach.

Randomized peers. Table 3 includes several fixed topologies, including rings and k-regular graphs, so the analysis on a fixed graph is directly relevant to our evaluation, especially the challenging ring graph. In 2-Peer, workers communicate along a ring that is resampled each round. Each pair of workers is equally likely to be neighbors, and therefore the expected Laplacian is proportional to that of the complete graph (Proposition 1).

Q2 - Time to the same validation quality

We measured time to DiLoCo’s validation loss for the 134M model on eight A6000 GPUs with links capped at 2 Gbps. We used the interval sweep in Figure 5 to choose H=10 for 1-Peer and H=15 for 2-Peer. Both configurations finish at 3.29 just below the validation loss of 3.30 achieved by DiLoCo at H=30.

MethodHStepsTimeTime saved
DiLoCo301,2906007 s—
1-Peer101,2305872 s2.2%
2-Peer151,2005795 s3.5%

The shorter intervals give both sparse variants more communication rounds before reaching 3.30 than DiLoCo. Both reach the target loss in fewer steps, saving up to 3.5% wall-clock time.

These observations were added in the paragraph “Time to validation quality.” in Section 4.4.

Dear reviewer, we are following up before the discussion period closes on Nov 18. We would be grateful to know whether our reply addresses your concerns. If it does, we would appreciate it if you would consider updating your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.