- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies distributed LM training on heterogeneous, unreliable workers, building on top of async SPARTA (background averaging of a random 5% of parameters). What is new is that it adds: (1) momentum from consecutive sparse averages (the fleet’s mean motion), with “echo cancellation” of each worker’s own contribution, used as a Nesterov look-ahead; (2) accumulate-to-match: slow workers accumulate gradients to the full batch.
Critical strengths
-
Clever reuse of existing communication. Async SPARTA already averages the same parameters again and again, so the change between two averages shows where the whole fleet is moving. PACE turns this into momentum at no extra communication cost, and subtracts each worker’s own contribution so the momentum doesn’t amplify itself.
-
Practical. There is no coordinator and no global synchronization, so slow or missing workers don’t block anyone; slow GPUs simply take fewer, full-batch steps. In a real 8-region WAN run on mixed GPUs, it finishes in 17.4 h versus 27.8 h for async DiLoCo.
Critical weaknesses
-
Every result is a single run, and in the main heterogeneous setting the gains are within the paper’s own noise threshold.
-
The algorithm doesn’t match the claims. The paper says every worker holds the same momentum buffer, but as written each worker subtracts its own, different contribution, which would push the buffers apart; yet the paper reports them converging. So the actual implementation must differ from the description, and the theory covers only a simplified version.
Questions for authors
- What exactly is the implemented momentum update, and how do the workers’ buffers stay in agreement?
- How much of the gain comes from the new momentum alone, versus accumulate-to-match and throughput weighting? How does it relate to SlowMo and quasi-global momentum?
Thank you for the review and for highlighting that the reusing of existing communication and the practical results is interesting. We clarify how the velocity buffers evolve, separate the contributions of velocity, ATM, and weighting, and provide more details on communication overlap.
Q1 / W2: The implemented momentum update and agreement between workers' buffers.
The key distinction here is between observing the same averaged parameters and maintaining exactly identical velocity buffers. Workers with the same observation history share the former. However, their buffers also depend on their local echo ledgers.
The implementation follows Algorithm 1 and Eq. (4). Worker r maintains a coordinate-wise ledger Qr,i of the displacement actually applied by its assist since coordinate i was last observed. At each optimizer step, it records
dr=AλMr(Xr)−Xr.
This is the realized displacement, including the effect of the norm-preserving transformation. When coordinate i is observed again, the worker computes
v^r,i=ΔtiWˉi−Wˉi−−Qr,i,
then updates
Mr,i+=μMr,i+(1−μ)v^r,i,μ=0.5,
and resets Qr,i. Here Δti is the elapsed interval measured in averaging windows, as in Algorithm 1.
For two workers with the same observation history, Wˉi, Wˉi−, and Δti are shared. Their buffer difference therefore evolves as
Mr,i+−Ms,i+=μ(Mr,i−Ms,i)−Δti1−μ(Qr,i−Qs,i).
Note that the first term contracts previous disagreement. The second introduces disagreement through differences in the applied assists. Since the ledgers themselves depend on the buffers, stability requires considering the coupled system (the contraction term alone does not establish it). This explains how the stated implementation can exhibit buffer recovery without maintaining exact equality at every observation. We will make this more clear in the revised version. Thank you for raising this important point.
Q2: Contributions of consensus velocity, ATM, and throughput weighting.
At 500M, the C4 contribution of consensus velocity is:
As shown, the FineWeb-Edu results follow the same pattern. Velocity gives a consistent benefit in different settings.
Table 15 shows the effect of ATM in MIXED-SPEED:
ATM keeps the target sample count per optimizer update fixed, so slower workers take fewer full-size updates. It supplies the larger quality improvement in this heterogeneous setting. Capacity weighting helps substantially under naive batch cutoff, recovering about 0.247 nats on C4, but adds little once ATM is enabled.
Q2: Relation to SlowMo and quasi-global momentum.
The main distinction here is how the motion estimate is formed and used while training continues. Standard SlowMo averages workers' models at an outer update boundary and applies momentum before the next local training phase. Quasi-global momentum is closer to PACE, and constructs local momentum estimates from consecutive model differences after neighbor mixing, without additional momentum communication. It therefore should not be grouped with methods that require a global outer momentum synchronization round.
PACE estimates motion from sparse, delayed observations of the participating workers' mean parameters. Observations arrive irregularly for each coordinate, while the resulting velocity is used at local optimizer steps between exchanges. The estimator consequently observes the effects of its own previous assists, creating the feedback loop addressed by echo cancellation.
PACE preserves the communication overlap. Local computation continues while the communication is happening, and velocity reconstruction doesn’t add any communication payload or a separate momentum synchronization round. Note that the cancellation ablation and the velocity gains on the same communication substrate show that both the feedback treatment and the recovered signal have benefits.