acceptodds
Under review as a conference paper at ICLR 2027

PACE: Parameter Averaging with Consensus Extrapolation for Distributed Training

Abstract

Existing methods for training language models across slow, heterogeneous, or unreliable networks utilize infrequent or compressed gradient exchanges to reduce communication traffic. However, this creates two problems. a) They still pause computation and b) work best only when every worker remains available. Asynchronous sparse parameter averaging avoids both constraints: workers exchange a random subset of model weights in the background, absent workers contribute nothing, and stale workers are naturally pulled back when they return. Its weakness is optimization quality. We show that the missing ingredient is momentum on the mean motion of the worker population. That motion is already present in each sparse average, so Pace recovers it as a shared consensus-velocity buffer without sending any additional data. Every worker reconstructs the same buffer and uses it in a Nesterov-style update. We also propose accumulate-to-match (ATM) for handling heterogeneous hardware. ATM lets each worker take an optimizer step only after processing the same number of samples, so slower workers become stale instead of providing noisy updates due to lower batch sizes. Sparse averaging handles this bounded staleness much better than such gradient noise. Together, these changes turn sparse parameter averaging from a weaker optimizer into the strongest communication-efficient method in our heterogeneous and unreliable settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Distributed training, communication compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BZLeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies distributed LM training on heterogeneous, unreliable workers, building on top of async SPARTA (background averaging of a random 5% of parameters). What is new is that it adds: (1) momentum from consecutive sparse averages (the fleet’s mean motion), with “echo cancellation” of each worker’s own contribution, used as a Nesterov look-ahead; (2) accumulate-to-match: slow workers accumulate gradients to the full batch.

Critical strengths
  1. Clever reuse of existing communication. Async SPARTA already averages the same parameters again and again, so the change between two averages shows where the whole fleet is moving. PACE turns this into momentum at no extra communication cost, and subtracts each worker’s own contribution so the momentum doesn’t amplify itself.

  2. Practical. There is no coordinator and no global synchronization, so slow or missing workers don’t block anyone; slow GPUs simply take fewer, full-batch steps. In a real 8-region WAN run on mixed GPUs, it finishes in 17.4 h versus 27.8 h for async DiLoCo.

Critical weaknesses
  1. Every result is a single run, and in the main heterogeneous setting the gains are within the paper’s own noise threshold.

  2. The algorithm doesn’t match the claims. The paper says every worker holds the same momentum buffer, but as written each worker subtracts its own, different contribution, which would push the buffers apart; yet the paper reports them converging. So the actual implementation must differ from the description, and the theory covers only a simplified version.

Questions for authors
  1. What exactly is the implemented momentum update, and how do the workers’ buffers stay in agreement?
  2. How much of the gain comes from the new momentum alone, versus accumulate-to-match and throughput weighting? How does it relate to SlowMo and quasi-global momentum?
1 comment

Thank you for the review and for highlighting that the reusing of existing communication and the practical results is interesting. We clarify how the velocity buffers evolve, separate the contributions of velocity, ATM, and weighting, and provide more details on communication overlap.

Q1 / W2: The implemented momentum update and agreement between workers' buffers.

The key distinction here is between observing the same averaged parameters and maintaining exactly identical velocity buffers. Workers with the same observation history share the former. However, their buffers also depend on their local echo ledgers.

The implementation follows Algorithm 1 and Eq. (4). Worker rr maintains a coordinate-wise ledger Qr,iQ_{r,i} of the displacement actually applied by its assist since coordinate ii was last observed. At each optimizer step, it records

dr=AλMr(Xr)−Xr.d_r=\mathcal A_{\lambda M_r}(X_r)-X_r.

This is the realized displacement, including the effect of the norm-preserving transformation. When coordinate ii is observed again, the worker computes

v^r,i=Wˉi−Wˉi−−Qr,iΔti,\hat v_{r,i} = \frac{\bar W_i-\bar W_i^{-}-Q_{r,i}}{\Delta t_i},

then updates

Mr,i+=μMr,i+(1−μ)v^r,i,μ=0.5,M_{r,i}^{+} =\mu M_{r,i}+(1-\mu)\hat v_{r,i}, \qquad \mu=0.5,

and resets Qr,iQ_{r,i}. Here Δti\Delta t_i is the elapsed interval measured in averaging windows, as in Algorithm 1.

For two workers with the same observation history, Wˉi\bar W_i, Wˉi−\bar W_i^{-}, and Δti\Delta t_i are shared. Their buffer difference therefore evolves as

Mr,i+−Ms,i+=μ(Mr,i−Ms,i)−1−μΔti(Qr,i−Qs,i).M_{r,i}^{+}-M_{s,i}^{+} = \mu(M_{r,i}-M_{s,i}) - \frac{1-\mu}{\Delta t_i}(Q_{r,i}-Q_{s,i}).

Note that the first term contracts previous disagreement. The second introduces disagreement through differences in the applied assists. Since the ledgers themselves depend on the buffers, stability requires considering the coupled system (the contraction term alone does not establish it). This explains how the stated implementation can exhibit buffer recovery without maintaining exact equality at every observation. We will make this more clear in the revised version. Thank you for raising this important point.

Q2: Contributions of consensus velocity, ATM, and throughput weighting.

At 500M, the C4 contribution of consensus velocity is:

RegimeC4 improvement from consensus velocity
HOMOGENEOUS0.068±0.0010.068 \pm 0.001
OUTAGE0.078±0.0020.078 \pm 0.002
SESSION0.078±0.0020.078 \pm 0.002
SESS-HET0.039±0.0020.039 \pm 0.002
MIXED-SPEED0.0340.034 / 0.0180.018 across two baseline runs

As shown, the FineWeb-Edu results follow the same pattern. Velocity gives a consistent benefit in different settings.

Table 15 shows the effect of ATM in MIXED-SPEED:

Change from full PACEC4FineWeb-Edu
Replace ATM with naive batch cutoff, retain weighting+0.130±0.004+0.130 \pm 0.004+0.116±0.003+0.116 \pm 0.003
Replace ATM with naive batch cutoff and use uniform averaging+0.377±0.005+0.377 \pm 0.005+0.361±0.004+0.361 \pm 0.004
Retain ATM and use uniform averaging+0.002±0.002+0.002 \pm 0.002+0.010±0.001+0.010 \pm 0.001

ATM keeps the target sample count per optimizer update fixed, so slower workers take fewer full-size updates. It supplies the larger quality improvement in this heterogeneous setting. Capacity weighting helps substantially under naive batch cutoff, recovering about 0.2470.247 nats on C4, but adds little once ATM is enabled.

Q2: Relation to SlowMo and quasi-global momentum.

The main distinction here is how the motion estimate is formed and used while training continues. Standard SlowMo averages workers' models at an outer update boundary and applies momentum before the next local training phase. Quasi-global momentum is closer to PACE, and constructs local momentum estimates from consecutive model differences after neighbor mixing, without additional momentum communication. It therefore should not be grouped with methods that require a global outer momentum synchronization round.

PACE estimates motion from sparse, delayed observations of the participating workers' mean parameters. Observations arrive irregularly for each coordinate, while the resulting velocity is used at local optimizer steps between exchanges. The estimator consequently observes the effects of its own previous assists, creating the feedback loop addressed by echo cancellation.

PACE preserves the communication overlap. Local computation continues while the communication is happening, and velocity reconstruction doesn’t add any communication payload or a separate momentum synchronization round. Note that the cancellation ablation and the velocity gains on the same communication substrate show that both the feedback treatment and the recovered signal have benefits.

Reviewer_K1DBOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACE, a distributed training method designed for heterogeneous hardware, low-bandwidth networks, and worker failures. PACE estimates a shared, fleet-level velocity from delayed sparse parameter averages and uses echo cancellation to prevent self-reinforcing feedback in the velocity estimator. The authors also introduce accumulate-to-match (ATM), which keeps the number of samples per optimizer update approximately constant across workers, converting hardware speed differences into differences in update frequency and parameter staleness.

The paper evaluates PACE on models with 226M, 500M, and 1B parameters under heterogeneous speeds, worker outages, session churn, and a real eight-region WAN deployment. It also includes ablations and theoretical analyses.

Critical strengths
  • Practical problem setting. PACE addresses both communication blocking and global synchronization barriers, making it relevant to training environments with high network latency, heterogeneous hardware, or intermittent worker availability.
  • Interesting optimization mechanism. The method extracts fleet-level motion from successive sparse parameter averages and uses echo cancellation to address self-reinforcing feedback in the velocity estimator.
  • Broad experimental coverage. The authors evaluate multiple model sizes, heterogeneous worker speeds, outages, and session churn, and provide ablations of key components, including shared velocity, coupling schedules, and ATM.
  • Promising real-WAN results. In an eight-region AWS deployment, PACE completes training in 17.4 hours, with communication blocking accounting for only 0.9% of wall-clock time. It outperforms the asynchronous DiLoCo configuration evaluated in the paper, demonstrating potential practical benefits in high-latency networks.
Critical weaknesses
  • The novelty relative to recent work needs to be better established. AsyncMesh has studied asynchronous optimization and delay correction, LOSCAR-SGD combines sparse parameter averaging, communication-computation overlap, and heterogeneous local updates, and Factored Gossip DiLoCo also targets low-blocking distributed training. PACE's main additions appear to be shared-velocity estimation and ATM, but their independent advantages and conditions of applicability are not yet sufficiently established relative to the closest methods.
  • There is a gap between the theoretical analysis and the actual implementation. The convergence theorem analyzes a PACE-SGD variant with a hard-clipped additive assist, whereas the main experiments use AdamW and a norm-preserving transformation for hidden-layer weights. The appendix also acknowledges that the simplified theoretical model does not fully capture the skipped-update clock used by ATM. Therefore, the current theoretical results should not be interpreted as a convergence guarantee for the complete PACE+ATM implementation.
  • The contributions of individual mechanisms are not fully isolated. The paper provides useful ablations, but shared velocity, echo cancellation, norm preservation, and ATM interact with one another. For example, removing echo cancellation leads to divergence, whereas removing consensus velocity causes a comparatively modest loss increase. The authors could better clarify which components are necessary for stability, which primarily improve optimization quality, and whether these effects remain consistent across different operating regimes.
  • The experimental budget and system implementation need clarification. The reported sequence length, batch size, number of training steps, and total token count for the 500M experiments appear inconsistent under the conventional interpretation of global batch size. In addition, although the paper claims to require neither a coordinator nor a global quorum, the implementation uses weighted All-Reduce. The handling of dynamic participant sets, communication during worker failures, and state synchronization when workers rejoin should be described more clearly.
Questions for authors
  • What are the key algorithmic differences between PACE and AsyncMesh, LOSCAR-SGD, and Factored Gossip DiLoCo? Can the authors clarify which limitations of existing methods are specifically addressed by shared-velocity estimation and ATM?
  • Can the authors clarify the scope of the theoretical results relative to the actual AdamW implementation, and explain how the ATM analysis accounts for skipped optimizer updates by slower workers?
  • Can the authors clarify the batch-size definition, total token budget, and dynamic-participant communication protocol in the 500M experiments? Can they also further isolate the roles of shared velocity and ATM across the different operating regimes?
4 comments

Thank you for the detailed review and for highlighting the practical setting, the velocity mechanism, and the WAN results. We clarify the algorithmic contributions, the scope of the analysis, and the experimental protocol below. We will also distinguish communication overlap from the participation requirements of the underlying collective.

W1 / Q1: Differences from AsyncMesh, LOSCAR-SGD, and Factored-Gossip DiLoCo.

PACE builds on asynchronous sparse parameter averaging. Its contribution is to recover an optimization signal from the averages already being exchanged and use it during local training, without adding communication payload or a separate momentum synchronization round. ATM complements this by addressing the gradient noise introduced when slower workers use smaller batches.

AsyncMesh. AsyncMesh introduced the delay-corrected sparse averaging substrate that we use, including its overlap of communication and computation. PACE preserves that exchange protocol and additionally estimates the motion of the participating workers' mean parameters from consecutive observations of each selected coordinate. This velocity supplies a look-ahead and an assist at each local optimizer step.

Using the signal this way introduces a feedback problem: previous assists change the parameters, and those changes later appear in the averages used to estimate future velocity. PACE tracks the displacement actually applied by the assist and deducts it from the observed motion. This echo cancellation is necessary at the coupling used in our experiments; removing it causes divergence in Table 3.

LOSCAR-SGD. We would like to clarify that LOSCAR-SGD is not exactly a standalone algorithm, but a special case of AsyncMesh. Its delay-corrected sparse merge is basically the AsyncMesh's unsmoothed correction, obtained by setting t=1 in the EMA update. Therefore, LOSCAR-SGD uses the same delay-corrected merge (as us and asyncmesh) and allows worker-specific amounts of local computation. This is an important precedent for the principle behind ATM: slower workers can take fewer full-size updates. LOSCAR-SGD uses uniform averaging at a server and does not extract momentum from successive parameter averages.

PACE combines it with the velocity mechanism, and holds the velocity look-ahead fixed within each accumulation window. We also provide a controlled comparison against batch scaling and capacity weighting. At 500M, replacing ATM with naive batch scaling increases C4 loss by 0.1300.004 with capacity weighting retained. Removing both ATM and weighting increases it by 0.3770.005. At 226M, ATM improves C4 loss by 0.330 nats under uniform averaging. We will make these distinctions clearer in the revision.

Factored-Gossip DiLoCo. Factored-Gossip DiLoCo overlaps parameter mixing with computation but retains a smaller blocking mixing step. In contrast, PACE derives its velocity from sparse, delayed state averages while local computation continues (completely non blocking) between launching and applying those exchanges. This adds no additional collective or outer synchronization boundary. Note that ATM and consensus velocity two additions address different limitations: consensus velocity recovers motion information that ordinary state averaging discards, while ATM reduces the speed-dependent noise caused by smaller local batches.

We will make the distinctions more clearer in the revised version.

W2/Q2: Scope of the theory and the skipped-update clock.

We will make the scope of each theoretical result explicit below.

  • Lemma 1 establishes mean preservation and removal of snapshot disagreement for the sparse correction. These properties are independent of the local optimizer.

  • Lemma 2 analyzes the velocity estimator under bounded optimizer displacements. It identifies the feedback instability and the role of echo cancellation. Recording the realized displacement A(X)−X\mathcal A(X)-X accommodates the norm-preserving assist, however, exact cancellation requires the deducted echo to match the observed fleet-average echo. Heterogeneous update frequencies and changing participation can introduce a mismatch.

  • Proposition 1 bounds consensus error under a second-moment assumption on local displacements. When calibrated with the AdamW steps, the bound covers the measured consensus error at all 295 sampled steps, with a peak measured-to-bound ratio of 0.61. This is an empirical check of the bound in the implemented setting.

  • Theorem 1 gives a stationarity guarantee for PACE-SGD with a hard-clipped additive assist. It does not establish convergence or acceleration for the full AdamW implementation.

For ATM, per-update variance and update frequency need to be considered separately. Each worker accumulates approximately the same number of samples before an optimizer update, reducing the speed dependence of that update's variance. A slower worker also takes fewer updates per unit time.

The reviewer is correct about the skipped-update clock. Proposition 2's staleness analysis uses a bounded-delay model and does not capture this. An event-time analysis would be needed to account for the changing frequency of each worker's contribution. We will move this limitation from Remark 8 into the result statement. Table 15 evaluates the net effect under the implemented clock through a matched wall-clock comparison.

We also agree with the reviewer that we have used a simplified SGD variant to isolate the effects of sparse communication, delayed observations, and the velocity assist for ease of analysis. However, we would like to mention that this has been a standard practice in the field, specifically with experiment-heavy works. Such as,

  • NoLoCo analyzes its outer update with an SGD inner optimizer on a stochastic quadratic objective, and uses Adam for experiments.

  • Theory works on DiLoCo. In Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration, Khaled et al. (NeurIPS 2025) analyze SGD inner updates with different outer optimizers, including Nesterov acceleration, while using AdamW as the inner optimizer in experiments.

  • Moshpit SGD (Ryabinin et al., NeurIPS 2021) shows convergence guarantees for its SGD formulation, while its pretraining experiments use the adaptive LAMB optimizer.

  • Quasi-global momentum (Lin et al., ICML 2021) provides a convergence theorem for QG-DSGDm and also use Adam for evaluation.

Our approach is similar. It quantifies the effects of sparse refreshes and delay on disagreement and establishes a stationarity bound for the clipped SGD variant. These results help explain the behavior of the communication and assist mechanisms, while the experiments evaluate their effectiveness with AdamW. We agree with the reviewer that an analysis with Adam would be stronger and leave it for future work.

W3 / Q3: Which components account for the gains?

The ablations show distinct roles for the components (500M experiments). Positive differences mean higher loss after the stated change.

Change from PACERegimeC4FineWeb-Edu
Remove echo cancellationMIXED, 226MDiverges—
Keep 4 throughout trainingMIXED-SPEED+0.4530.008+0.4470.008
Remove consensus velocityHOMOGENEOUS+0.0680.001+0.0720.001
Remove consensus velocityOUTAGE+0.0780.002+0.0800.002
Remove consensus velocitySESSION+0.0780.002+0.0800.001
Remove consensus velocitySESS-HET+0.0390.002+0.0400.002
Remove consensus velocityMIXED-SPEED, two baseline runs+0.034 / +0.018+0.038 / +0.020
Remove consensus velocityONE-FAST-0.0050.002-0.0010.002
Remove ATM, retain velocity and weightingMIXED-SPEED+0.1300.004+0.1160.003
Remove ATM and weighting, retain velocityMIXED-SPEED+0.3770.005+0.3610.004
Remove weighting, retain ATMMIXED-SPEED+0.0020.002+0.0100.001
Remove norm preservationMIXED, 226M+0.021—

Consensus velocity consistently improves quality in the equal-speed regimes, including outages and churn. Different update frequencies produce differences between workers' velocity buffers, which may help explain this pattern. ATM supplies the larger gain in MIXED-SPEED. Capacity weighting has little additional effect once ATM is enabled at 500M, but helps substantially when workers use naive batch scaling.

W4 / Q3: Batch size and token budget.

Below we clarify the batch and schedule descriptions. We will make it clearer in the revised version.

The 500M batch specification is 320 sequences of length 1,024, corresponding to 327,680 tokens per full global batch. Over 30,000 full-batch steps, this gives a reference budget of 9.83B tokens, approximately 20 tokens per parameter for the 490.7M-parameter model.

With eight replicas, the target batch for each local optimizer update is B=40 sequences, formed from 20 microbatches of two sequences. Under ATM, this is the batch per local optimizer update; slower workers take those updates less frequently.

The remaining settings are:

  • Peak learning rate: 510-4.
  • Warmup: 4,000 steps.
  • Linear decay to 1%1\% of the peak learning rate.
  • Weight decay: 0.1.
  • Gradient clipping: 1.0.

The 1B configuration behind Tables 5 and 10 uses a global batch of 1,024 sequences of length 1,024 for 20,000 steps, corresponding to a full-batch reference budget of approximately 21.0B tokens. Its peak learning rate is 410-4, with 4,000 warmup steps. Appendix M uses the same batch specification for 8,000 steps. Table 4 gives the communication-budget comparison.

W4 / Q3: Communication overlap, quorum requirements, and rejoining workers.

We would like to highlight two distinguished properties below:

  • Communication overlap: whether local computation continues while an exchange is in flight.
  • Quorum blocking: whether completing an exchange requires contributions from a prescribed set of workers.

PACE overlaps sparse exchanges with local computation. Every R steps, a worker snapshots the selected coordinates and launches a non-blocking weighted all-reduce of

(ρrWrsnap,ρr).\left(\rho_r W_r^{\mathrm{snap}},\rho_r\right).

Training continues while the exchange runs, and the result is applied after D steps. The velocity estimator reuses this result and doesn’t introduces any additional collective or momentum synchronization round. In the WAN deployment, communication blocking accounts for 0.9%0.9\% of wall-clock time, so the measured benefit is substantial overlap. The collective is peer-to-peer. That is, no process owns the model, optimizer state, or velocity buffer on behalf of the other workers. This coordinator-free organization does not, by itself, remove the collective's participation requirements.

In the outage and churn experiments, absences are emulated within a fixed process group. An absent worker does not contribute to the fleet. Although they are absent, it continues with plain AdamW steps without the assist.

A returning worker resumes from its own parameters, optimizer state, and buffer. It needs the current shared tick index to determine the mask and schedules, but receives no checkpoint or optimizer-state transfer. Subsequent sparse exchanges repair its state. Figure 5 shows parameter recovery over roughly 400 steps, and Appendix C.1 shows recovery of the velocity buffer.

At the algorithm level, the averaging rule requires only

\quad\text{and}\quad \sum_{r\in\mathcal P}\rho_r$$ for a nonempty participating set $\mathcal P$. It is therefore mathematically compatible with changing participant sets. Thus, the experiments demonstrate communication overlap, no additional momentum synchronization round, and recovery from stale local states under absences. We will revise the presentation to make the above points clearer.

Thank you for the detailed responses and additional experiments. The rebuttal has addressed some of my concerns, but I still have the following two reservations:

  • Worker outages are simulated within a fixed process group. Therefore, the distinction between algorithm-level support for dynamic worker participation and the communication capabilities of the actual system should be clarified, and the corresponding claims should be appropriately qualified.
  • The ablation results suggest that consensus velocity provides relatively limited gains in heterogeneous settings. As one of the main contributions of this paper, its effectiveness and stability warrant further investigation.

Given these remaining concerns, I have decided to maintain my original rating.

Reviewer_ZHpMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACE, an asynchronous distributed training method that extends sparse parameter averaging with momentum reconstructed from successive averaged parameter states. The method additionally introduces accumulate-to-match (ATM), which equalizes the number of samples processed per optimizer step across heterogeneous workers. The authors provide theoretical analysis and experiments across heterogeneous hardware, worker outages, and an eight-region deployment. The contribution is a fairly natural extension of momentum to the sparse averaging setting, and thus does not offer a great degree of novelty. Similarly, accumulate-to-match resembles an extension of staleness-aware async federated learning methods. Despite somewhat limited novel insights, the work is generally well-executed from an algorithmic, systems, theoretical, and experimental perspective.

Critical strengths
  • The paper addresses an important practical problem, particularly for distributed training across heterogeneous hardware and unreliable networks without requiring a fixed communication interval.
  • Applying momentum to successive parameter-average deltas is an intuitive idea that appears to work well in practice.
  • The experimental evaluation is convincing, with substantial wall-clock improvements in geographically distributed training.
  • The method integrates naturally with existing sparse parameter-averaging frameworks.
Critical weaknesses

1.Limited conceptual novelty.

The underlying principle of applying momentum to aggregated parameter updates is well established, including in SlowMo, FedOPT, and BMUF. Nevertheless, adapting this principle to sparse asynchronous averaging is a reasonable contribution, particularly given the encouraging experimental results.

2.No evaluation of Muon-like optimisers, which are generally replacing AdamW in pre-training.

3.Limited theoretical guarantees.

The theoretical analysis concerns a simplified SGD variant rather than the AdamW-based algorithm evaluated experimentally. The results are reassuring but provide limited guarantees for the actual implementation.

Questions for authors

1.Have the authors experimented with applying PACE to Muon? Given Muon's distinctive momentum and orthogonalisation mechanisms, it would be interesting to establish whether the proposed consensus-velocity correction remains beneficial. Previous work (MuLoCo) has found that using Muon provides some distinct advantages over AdamW.

2.How sensitive is the consensus-velocity estimate to changes in worker participation between averaging rounds? One would expect a round with fewer workers to have a higher variance.

3 comments

Thank you for the thoughtful review and for highlighting the practical motivation and experimental results.

W1: Novelty relative to SlowMo, FedOpt, and BMUF.

The distinction we would like to emphasize is how PACE obtains and uses momentum while communication overlaps with training. In contrast, SlowMo and BMUF aggregate workers' models at outer update boundaries, and the next local training phase depends on completing that aggregation and momentum update. FedOpt similarly uses server aggregation rounds, although participation comes from a selected client cohort instead of the entire client population.

Note that PACE extracts a velocity signal from the sparse, delayed parameter averages already exchanged. Workers continue local computation while the communication happens, and use the reconstructed velocity for a look-ahead and an assist at each local optimizer step. This doesn’t add any communication payload or a separate blocking momentum synchronization round.

We also handle a feedback problem that comes with this. The assist changes the parameters, and those changes later appear in the averages used to estimate the next velocity. With sparse, irregular observations and delayed communication, the estimator can amplify its own previous output. PACE addresses this through echo cancellation, by keeping track of the displacement actually applied by the assist, including the norm-preserving transformation. Removing this mechanism causes divergence at the coupling used in our experiments (Table 3). At 500M, consensus velocity improves validation loss by 0.068–0.080 nats across the equal-speed regimes, including outages and session churn. In HOMOGENEOUS, it reduces C4 loss from 3.731 to 3.662, compared with 3.618 for DiLoCo. Thus, it recovers much of the optimization gap of sparse state averaging while preserving its communication pattern.

We will make this combination of signal reconstruction, feedback treatment, and communication overlap clearer in the introduction and related work.

W2 / Q1: Muon.

Following your suggestion, we ran the 500M MIXED-SPEED comparison with Muon on the eligible 2-D weight matrices. We used learning rate 0.02, Nesterov momentum 0.95, and five Newton–Schulz steps, with the orthogonalized update scaled by

max⁡(1,nrowsncols).\sqrt{\max\left(1,\frac{n_{\mathrm{rows}}}{n_{\mathrm{cols}}}\right)}.

Embeddings, the output head, and 1-D parameters continued to use AdamW.

RegimeMethodC4, AdamWC4, MuonFineWeb-Edu, AdamWFineWeb-Edu, Muon
SESSIONPACE3.6423.6483.0113.006
SESSIONasync DiLoCo (DN+DyLU)4.0093.9983.3543.361
SESSIONSparseLoCoDivergedDivergedDivergedDiverged
SESSIONasync SPARTA3.7263.7313.0993.093
SESS-HETPACE3.7553.7493.1313.138
SESS-HETasync DiLoCo (DN+DyLU)3.9813.9903.3423.335
SESS-HETSparseLoCoDivergedDivergedDivergedDiverged
SESS-HETasync SPARTA3.8043.8113.1793.172

Scope and purpose of the theoretical analysis.

We agree with the reviewer that we have used a simplified clipped PACE-SGD variant to isolate the effects of sparse communication, delayed observations, and the velocity assist. This makes it possible to study these mechanisms without also having to analyze AdamW's evolving adaptive preconditioner. However, we would like to mention that this has been a standard practice in the field, specifically with experiment-heavy works, which grants ease of analysis. Below we mention a few:

  • NoLoCo (Kolehmainen et al., 2025) analyzes its outer update with an SGD inner optimizer on a stochastic quadratic objective, while using Adam in its experiments.

  • Later theory works connected to DiLoCo follows a similar approach. In Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration, Khaled et al. (NeurIPS 2025) analyze SGD inner updates with different outer optimizers, including Nesterov acceleration, while using AdamW as the inner optimizer throughout their experiments. This provides insight into the outer optimization mechanism under a simplified inner update.

  • Moshpit SGD (Ryabinin et al., NeurIPS 2021) establishes convergence guarantees for its SGD formulation, while its ALBERT pretraining experiments use the adaptive LAMB optimizer.

  • Quasi-global momentum (Lin et al., ICML 2021) provides a convergence theorem for QG-DSGDm and also evaluates an Adam extension.

PACE's theory serves a similar purpose. It quantifies the effects of sparse refreshes and delay on disagreement and establishes a stationarity bound for the clipped SGD variant. These results help explain the behavior of the communication and assist mechanisms, while the experiments evaluate their effectiveness with AdamW. However, we agree with the reviewer that analysis with Adam would make the contribution much stronger and leave it for future work.

Q2: Sensitivity to changes in worker participation.

Participation changes affect both the variance of the average and the motion observed by the estimator. There are three relevant effects.

  1. Variance of the average. Under independent noise with equal per-update variance, the variance bound for a weighted average scales with

    \qquad \tilde w_r=\frac{\rho_r}{\sum_{s\in\mathcal P}\rho_s},$$ where $\mathcal P$ is the participating set and $\rho_r$ is worker $r$'s capacity weight. For eight equal workers, losing one increases this factor from $1/8$ to $1/7$, a relative increase of about $14.3\%$. With unequal weights, the effect depends on which workers are absent and how the remaining weights are renormalized. This describes the sampling-noise contribution, which is that the velocity estimate also depends on the following two effects.
  2. Changes in the population being observed. If the participant set changes between observations, Wˉ−Wˉ−\bar W-\bar W^{-} includes a composition change as well as model motion. The size of this additional term depends on the disagreement of the workers entering or leaving the average. Sparse parameter mixing reduces that disagreement, and the velocity EMA, with μ=0.5\mu=0.5 per observation, downweights isolated perturbations over subsequent observations.

  3. Echo mismatch. Note that each worker deducts its own applied assist and the observed mean contains the assists of the participating workers. Therefore, changes in participation or update frequency can make the deduction imperfect. Workers can consequently maintain different velocity buffers even when they receive the same current average.

Note that the outage and session-churn experiments test the combined effect of these changes. In SESSION and SESS-HET, with 5%5\% stationary absence and at most two workers absent, PACE remains stable and improves loss over the evaluated baselines by approximately 0.040.04–0.320.32 nats, depending on the comparison. In the outage experiment, buffer disagreement falls below its pre-outage value after the worker returns (Appendix C.1). These results demonstrate recovery and stable training under the tested participation patterns. We will make the dependence on participant composition and echo mismatch clearer in the revised version.

Thank you for the rebuttal as it has provided useful additional insight. I believe the work is of high quality, albeit incremental. Thus I maintain my original positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.