acceptodds
Under review as a conference paper at ICLR 2027

Nesterov Acceleration in Local Step Methods

Abstract

Local-step methods reduce communication in distributed training by allowing workers to perform multiple optimizer steps between synchronizations. Prior work has shown that incorporating Nesterov acceleration can improve local-step training, but how it should interact with both periodic communication and updates within each local block remains less understood. We identify four constructions for such interaction, each restricted to communicating one model-size vector between workers per round. We analyze classical, modern, and primal formulations across these constructions, including generalized variants with independent control of smoothing and look-ahead, and establish when they are equivalent. Our language-modeling experiments identify two ways to improve DiLoCo-style training: tuning generalized retention and extrapolation coefficients, and reporting a smoothed iterate. They also identify a construction that performs well with frequent communication, and in what settings Nesterov acceleration does not help.

Primary area:
optimization
Keywords:
Distributed training, Local steps, Nesterov Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h2WmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how Nesterov acceleration interacts with local-step training under a one-model-vector communication constraint. It organizes four constructions based on the initial algorithmic analysis, derives equivalences and differences among classical, modern, and primal formulations, and studies independent retention and extrapolation coefficients. Experiments on 124M and 1B language models examine coefficient choices and reported iterates, including settings where acceleration does not help.

Critical strengths

I appreciate the framing of the question around how per-step Nesterov acceleration should use shared and local history under periodic communication. The discussion of when shared history helps and the experimental takeaways are useful, but the narrative would benefit from clearer links between the proposed intuitions and the observed results.

Critical weaknesses

W1. The single-seed comparisons leave the reliability of the reported method differences unresolved. The paper acknowledges this limitation, but without repeated experiments, particularly at smaller scales, it is difficult to assess how much of the reported contribution and impact extends beyond the observed runs.

W2. The justification for transferring hyperparameters from 124M to 1B is insufficient for my understanding of the topic, reducing the perceived contribution and impact of the larger-scale results. I would expect the results to remain valid under an appropriate hyperparameter-transfer framework, but the paper needs clearer support for this transfer given the model scale, batch size, learning rate, and training horizon. Hyperparameter transfer frameworks are strongly recommended here.

Questions for authors

Q1. Please provide results from multiple seeds, especially at smaller scales, to assess the stability of the reported differences.

Q2. Please justify and support transferring the selected 124M coefficients to the 1B experiments, given changes in model scale and the roles of batch size, learning rate, and training horizon.

1 comment

We thank the reviewer for their time and review, and respond to their concerns below:

W1/Q1: "The single-seed comparisons leave the reliability of the reported method differences unresolved. The paper acknowledges this limitation, but without repeated experiments, particularly at smaller scales, it is difficult to assess how much of the reported contribution and impact extends beyond the observed runs."

  • We agree that the single-seed design leaves the variability of the reported performance differences unresolved. The hyperparameter search examines dependence on coefficients and training settings, but does not establish robustness across training seeds. However the hyperparameter search is already very costly (we ran 853 completed runs across the eleven settings and 25 early-stopped runs), and repeating with multiple seeds would further increase this cost. While this would affect exact rankings, we can still conclude general trends, such as RCN-P performs well in most settings, FedAvg only works well in the very large batch setting, and RCN-C seems be competitive when the batch size is large and the number of local steps is small.

W2/Q2 "The justification for transferring hyperparameters from 124M to 1B is insufficient for my understanding of the topic, reducing the perceived contribution and impact of the larger-scale results. I would expect the results to remain valid under an appropriate hyperparameter-transfer framework, but the paper needs clearer support for this transfer given the model scale, batch size, learning rate, and training horizon. Hyperparameter transfer frameworks are strongly recommended here."

  • The 1B experiments retain the 124M donor setting S9’s worker count (K=8K=8), global batch (G=512G=512), synchronization interval (H=128H=128), and sequence length (1,024). They also retain the architecture family and AdamW configuration. Each method inherits its selected Nesterov coefficients and outer scale, while its learning rate is selected through a two-candidate validation pilot at 1B. Thus, this is coefficient and outer-scale transfer with limited learning-rate adaptation, rather than zero-shot transfer of all hyperparameters. The 20B experiments then reuse those selected recipes without further tuning, starting fresh runs with a schedule defined for the longer horizon.
  • Our objective is to test whether the selected recipes remain useful at larger model scale and longer training horizons. We do not establish that these coefficients remain optimal, that the methods are independently optimized at 1B, or that a general hyperparameter-transfer law holds. We will make this scope and the distinction between transferred and retuned quantities more explicit in the revised version.
Reviewer_RQMXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study how Nesterov acceleration can be incorporated in the distributed training. There are a few unknowns and less understood parts in the distributed training communication and updates even though Nesterov updates has been shown to be effective. The authors develop and identify 4 constructions:

  1. Step -H Outer Nesterov (SHN)
  2. Round Conditioned Local Nesterov (RCN)
  3. Per Local Step Newton (PLN) and
  4. Two Time scale Nesterov (TTN)

and then analyze them against the classical, modern and primal formulation. Finally authors experimentally compare the methods discussed and developed in the paper with FedAvg.

Critical strengths

The paper delves into the formulation of Nesterov acceleration method in the distributed training. The authors discuss DiLoCo-style training that found Nesterov style update to be the effective scheme in the language model training. In the effort to better understand what part should go in the local update and how the synchronization should happen are very much less understood. This paper understand the gap and then identify 4 different schemes for 3 different formulation that fit into the distributed training where the communication is one model size vector.

The strengths are

  1. The identification of the strategies (Step -H Outer Nesterov, Round Conditioned Local Nesterov , Per Local Step Newton and Two Time scale Nesterov) .
  2. The table 1 shows a way to understand 3 different formulation (classical, modern and primal) and what are the properties that are important in the design of the distributed training.
Critical weaknesses

The main problem that I have with the paper is its presentation. The paper is missing some key sections to explain why and how. The paper requires a bit of changes.

The paper lacks a proper contribution section. Without it, severe difficult arise to point out the importance, novelty and context of the paper. Also the paper depends a lot on prior work and then explain what it is for on top of it. In my opinion, the paper should stands on the ground of the distributed training with Nesterov style update. The paper talks about DiLoCo-style and GPA and then move to the methods. A better approach would be to establish the base of Federated learning and the need of better training with Nesterov. A simple contribution section and introduction would do.

The explanation of the paper is also not good for me. Be better at explaining what the state and the update and the outer loop and wrapper is. The paper depends too much on the previous literature. The reader should be able to understand the context and problem and the setup with the paper standalone.

The algorithms and the updates that are in the table form needs to be better written. Going back and forth the appendix and the main paper is not easy and sometimes they do not convey clarity or the full message. The algorithms that are presented in the Appendix provides more clarity than the tables in the main paper. Bringing the algorithms to the main paper would serve better.

Questions for authors

What is the motivation for having 2 different coefficient tied and generalized?

For the SHN : It is not clear (even in the Appendix) what is the local block and what is being updated.

What is the intended purpose of the GPA? It is not clear what is the outcome that is trying to establish in the paper.

The paper should also explain the goal for the algorithms. All of the explanations are done with experiments. A simple strategy to know what is better method and the need for such methods would provide completeness to the development of such algorithms.

3 comments

We thank the reviewer for their time and review, and respond to their concerns below:

"The main problem that I have with the paper is its presentation. The paper is missing some key sections to explain why and how. The paper requires a bit of changes."

  • We appreciate the reviewer’s specific feedback on presentation. We agree that the paper is brief in its explanation in many areas, there is a lot of background material and investigation into the constructions to cover in nine pages. We made a deliberate choice to explain the background for a reader familiar with multi-worker local step methods (e.g., federated learning) but unfamiliar with DiLoCo, GPA, and different formulations of Nesterov momentum. Thus, at the beginning of the background section (line 79) we mention that we give a broader related work section at the start of the appendix, where we properly explain the context within federated learning. However, we agree that we should (1) make it clear that those unfamiliar with multi-worker local step methods or federated learning should read that appendix section, and (2) do a better job of explaining the setting in that appendix section, not just related work. We will make these changes in the revised version.

"... A better approach would be to establish the base of Federated learning and the need of better training with Nesterov. ..."

  • That does not capture our motivation/contribution, as discussed in paragraphs 2-4 of the introduction, our motivation is
    • DiLoCo already shows that incorporating outer Nesterov provides improvements in the multi-worker local step setting, and part of their explanation for this is their observation that outer Nesterov even gives improvement in the single-worker case. However, GPA then showed that even better performance can be gained in the single worker case by applying Nesterov at each step based on interesting new components (different Nesterov formulations applied as generic wrappers and generalized coefficients). Thus, our motivation in this paper is to investigate whether these components would also give benefits in the original multi-worker setting, where local steps are required so that communication is infrequent (unlike the single-worker setting). Thus, this setting is highly relevant for distributed training and federated learning.
  • As we explain in the remaining paragraphs in the introduction, and in our conclusion, our contribution is
    • We show that it is not clear how to integrate these components into this setting, there are multiple choices. Thus, we organize these choices into four constructions based on one key axis: on what scale to use global (cross-worker) information via Nesterov. Within these constructions we investigate and analyze various possible methods, what optimization settings each would be best suited for, and their relationships to each other. Our experiments then identify how these choices affect optimization.
  • We agree our introduction could have explained the contributions more clearly, we will make sure to do that in a revised version.

"... Be better at explaining what the state and the update and the outer loop and wrapper is. The paper depends too much on the previous literature. ..."

  • By the base update, we mean the parameter increment produced by an optimizer such as AdamW at the specified evaluation point. Its internal optimizer state is distinct from the additional Nesterov state. A Nesterov wrapper applies a Nesterov formulation to any type of update (this can be a single base update, or a block of multiple sequential base updates), and its state contains all variables required to continue applying steps of the formulation. We never discuss an outer loop. We will make these definitions more clear in the revised version.
  • In Table 1, for each Nesterov formulation we explain what the lookahead point, reported iterate, state, and init are on the right, and the formulation's steps (which is the wrapper around an update U(⋅)U(\cdot)) are on the left. We explained the relevant previous literature in Section 2, giving their algorithms in Table 2.

"... The algorithms that are presented in the Appendix provides more clarity than the tables in the main paper. Bringing the algorithms to the main paper would serve better."

  • Yes, we agree that they provide a lot of clarity, however we cannot bring them into the main paper given the space restrictions. We already have 8 shorter algorithms in the main paper, the 9 of the 10 algorithms in the appendix are extensions of them (adding generalized coefficients, like in GPA as shown in Table 2) with an explicit loop structure, and the other one is a simple explanation of how local steps work (assumed to be background). We will revise the main-text presentation to make the local block and synchronization update easier to follow, with direct references to the corresponding detailed algorithms.

"What is the motivation for having 2 different coefficient tied and generalized?"

  • Original Nesterov has a single coefficient μ\mu that controls both the amount of smoothing/retention in the momentum update and the lookahead extrapolation/correction, we call this tied coefficients. We use generalized for when we have two separate coefficients for these two purposes, μx\mu_x and μy\mu_y. As explained in Section 4, μx\mu_x controls momentum retention, and μy\mu_y controls the lookahead extrapolation/correction. Often we want to have larger extrapolation/correction but smaller retention. DiLoCo uses μ=0.9\mu=0.9, favouring high correction at the cost of high retention, while our experiments show that when we use generalized coefficients, μx∈[0.25,0.75]\mu_x\in[0.25,0.75] and μy≥0.9\mu_y\geq 0.9 work well (chosen in most settings). This is discussed in lines 376-408.

"For the SHN : It is not clear (even in the Appendix) what is the local block and what is being updated."

  • A local block consists of HH consecutive base-optimizer updates on each worker, which is informally introduced on line 41. We will make this more clear in the revised version. We treat each Nesterov formulation as a wrapper over any type of update U(⋅)U(\cdot), and thus such an update can be a single base update or a local block consisting of many base updates. The steps of each Nesterov formulation update the variables in the Nesterov state (see Table 1).

"What is the intended purpose of the GPA? It is not clear what is the outcome that is trying to establish in the paper."

  • We first discuss this in lines 41-44: in the single worker setting, GPA wraps every local step rather than entire local blocks, which gives improved performance. We further expand upon this in lines 141-148: the method can be considered a smoothed alternative to single worker DiLoCo.

"The paper should also explain the goal for the algorithms. ..."

  • The goal is better optimization performance for multi-worker local step methods. In this setting, each worker takes steps locally using local batch gradients, but these do not estimate the true gradient direction as well as if all workers combined their local batch gradients into a single global batch gradient. Note that the latter is what standard distributed data-parallel training does, but is expensive as it requires frequent communication (every update step). This is why global momentum helps, it moves the parameters in the recent combined direction from all workers (computed from averaging events of previous rounds), however this information is not as recent as local updates from the current round. Our constructions investigate different ways to combine these sources of local and global history while staying within a Nesterov framework. Their usefulness depends on the training setting and on the memory already supplied by the base optimizer.
Reviewer_RQMXOpenReview ↗

Response

The problem that I have with the paper is the writing. It needs a full rewrite where the authors needs to properly describe the problem setup and then explain the contributions and the algorithms and related explanation. So I will keep my score and urge the authors to resubmit the paper.

Reviewer_MQGpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work studies different approaches to apply Nesterov to multi-GPU scenarios, based on single-GPU Nesterov with DiLoCo. Specifically, four settings: 1. Outer Nesterov once per round (SHN, which includes DiLoCo); 2. A fixed shared momentum applied at every local step (RCN); 3. Per-local-step Nesterov with worker-local state (PLN); and 4. A combination of the two (TTN). Each setting uses classical, modern, and primal forms. Experiments first use a 128M model to test different settings, and the selected recipes are used to evaluate on the 1B model. The findings of this work include: 1. Decoupling momentum retention from look-ahead beats tuned DiLoCo in most of the evaluated settings. 2. RCN-C is competitive with frequent synchronization, while FedAvg outperforms others at the largest global batch size of 1024. 3. PLN/TTN does not help on top of AdamW, and TTN selects zero local momentum in nine of the eleven settings.

In general, my concerns are that this work has limited innovations and unclear contributions, which is beyond a significant revision or restructuring of this paper.

Critical strengths

With a fixed communication budget, the approach preserves test/validation loss.

Critical weaknesses
  1. Limited novelty. The gains of this work come from existing ideas. First, decoupling retention and look-ahead is GPA's innovation. Second, the mooted-iterate reporting idea is from primal averaging/Schedule-Free. Third, in the four settings, three of them are from known methods or equivalent to each other. For example, SHN is DiLoCo, PLN is a variant of Nesterov FedAvg/GPA, and RCN-P is equivalent to SHN. TTN is equivalent to RCN-C in nine of the eleven settings. While RCN-C is the new variant, it only has \textbf{three} of eleven settings that outperform with small gains.

  2. In addition to the first weakness, unclear contribution. Gaps between Nesterov update settings are as small as noise. Therefore, the evaluation is not convincing. For example, on the 124M, most settings have a difference of around 0.003 to 0.01 nats. For the 1B model at 10B and 20B horizons, the difference is less than 0.1. Without evaluation on downstream tasks, the contribution is unclear.

  3. No system-level evaluations. This work is motivated by achieving communication efficiency by reducing the communication frequency. However, the system performance is not reported. For example, step time and communication throughput. Without system-level evaluations, it is unclear whether the approaches are practical.

  4. Lack of downstream task evaluation. All results reported are FineWeb validation or test loss. The evaluation should include benchmarking tasks such as PIQA, MMLU, HellaSwag, etc to present its performance in practice.

Questions for authors
  1. How does this work differ from previous works? Additionally, the introduction should clearly list the novelty and contributions of this paper.

  2. How does each approach perform on the downstream task? Or how do the validation/test losses reflect actual performance in completing real-world tasks?

  3. What is the system setting? For example, for eight H100 GPUs, are they within a single node with NVLINK? Or four GPUs per node, and what are the inter-node connections (e.g., InfiniBand, RoCEv2)? This is critical for communication-efficient innovations.

  4. What is the step time or throughput in tokens per second for this work, compared to existing work?

3 comments

We thank the reviewer for their time and review, and respond to their concerns below:

W1: "Limited novelty. The gains of this work come from existing ideas. ... Third, in the four settings, three of them are from known methods or equivalent to each other. ..."

  • We are not claiming novelty in the components themselves, we explicitly note that our main components are from GPA (lines 41-44), but claim novelty in exploring how those components can be used for the multi-worker local step case. Part of DiLoCo's evidence for its good performance in the multi-worker local step setting is that it even provides improvements in the single worker case where local steps are not needed, but then GPA shows that even better performance can be gained in the single worker setting by disregarding local steps and applying Nesterov every step (they specifically introduce the components of different Nesterov formulations applied as generic wrappers and generalized coefficients). We instead investigate the original multi-worker case, where local steps are needed (as communication should be infrequent), which is novel as no one has investigated GPA's components in this setting. In fact, there is no clear answer to what is the best way to utilize those components in this new setting, there are multiple choices. Our four constructions organize these choices, they are not presented as four wholly unrelated inventions.

  • The equivalences mentioned are either not correct, or are actually key findings in our paper:

    • It is not as simple as SHN is DiLoCo: DiLoCo is tied SHN-M (SHN with modern Nesterov and tied coefficients), and we specifically find that reporting the averaged iterate in the modern version (or equivalently using classical or primal SHN) and using decoupled (generalized) coefficients provide significant improvements over DiLoCo. Thus our construction of SHN and our exploration of possible choices to make within this construction directly find improvements over DiLoCo.
    • PLN is not simply a variant of GPA, it generalizes it to the multi-worker case with a non-trivial extension (restricting communication to averaging a single state). It is similar to Nesterov FedAvg, but again differs because of our strict communication constraint of averaging a single state.
    • Our finding that generalized RCN-P is equivalent to generalized SHN is unexpected and non-trivial. The explicit conversion between their coefficients uncovers many interesting observations for tuning them, such as which ranges are reasonable and how finely the coefficients should be searched (Section 5 and Appendix B.4).
    • TTN is much more expressive than RCN-C, and the fact that hyperparameter search found it better to zero out that extra expressiveness is a key finding, showing that RCN-C is sufficient.
    • RCN-C is indeed the most novel variant, it has a clear intuitive motivation and works well in three of the settings, specifically in a regime consistent with our intuition about the benefits of RCN-C. Furthermore, the regime where it outperforms other methods is quite practical: the large local batch and few local step regime, which is the situation for cross-datacentre training.

W2: "Gaps between Nesterov update settings are as small as noise ... on the 124M, most settings have a difference of around 0.003 to 0.01 nats"

  • We agree that several comparisons between leading methods are close. However, between key lines of investigation in the paper, the difference is much more significant, for example:
    • Nesterov acceleration improves over FedAvg (plain local steps, no acceleration) by 0.035–0.132 nats/token in ten settings.
    • The improvement of generalized SHN-M over DiLoCo (tied SHN-M) is 0.041 nats/token on average over all settings, reaching 0.154 on S4.
    • At 1B, RCN-P improves over FedAvg by approximately 0.084 and 0.059 nats/token at the 10B and 20B token horizons, respectively.
  • Thus while several leading-method comparisons are close and require repeated-seed evaluation, there are clear differences for many major comparisons.

W3: "This work is motivated by achieving communication efficiency by reducing the communication frequency.."

  • Within each setting, all methods use the same synchronization interval and communicate one FP32 model-sized optimization vector per round. Our question is which update construction gives better optimization performance under this common communication constraint. We do not claim an additional communication reduction relative to DiLoCo or an end-to-end throughput improvement, and thus do not need systems evaluation.

W4: "Lack of downstream task evaluation. ..."

  • Held-out loss (test and val) directly evaluates generalization on the language-modeling objective and is therefore directly relevant to our optimization question, and is standard in the optimization literature. We agree that downstream evaluation would provide complementary evidence about practical consequences, but it is not necessary to distinguish optimization performance improvements.

Q1: "How does this work differ from previous works? .."

  • Most of the components we introduce into the setting originate in prior work, specifically from GPA, which we are very clear about (see lines 41-44). Our work differs from any previous work by explicitly bringing these components into the multi-worker local step setting, which we note is not clear how to do, and thus we investigate many possible approaches. Specifically, we do a systematic investigation of where Nesterov state can enter in relation to local blocks and synchronization. Our contribution is (1) identifying this line of investigation (how to incorporate Nesterov state), (2) analyzing the possible resulting methods, what optimization settings each would be best suited for, and their relationships to each other, and (3) identifying which choices affect the observed optimization outcomes. This contribution does not depend solely on small performance differences between the best-performing formulations.

Q2: "How does each approach perform on the downstream task? ..."

  • See W4.

Q3: "What is the system setting? ..."

  • The setting is K>1 GPUs/nodes (workers) that communicate infrequently, every H local steps (the multi-worker local step literature, which includes federated learning). This setting targets the geographically distributed setting where links between workers are slow, typically consumer internet links. Thus, while experiments are done in a single node setting, their aim is for deploying these methods on multiple nodes that are geographically separated. We will make this setting from the literature more clear in the revised version.

Q4: "What is the step time or throughput in tokens per second for this work, compared to existing work?"

  • We did not benchmark step time or tokens per second. Within each experimental setting, all methods perform the same number of local base-optimizer updates and communicate one FP32 model-sized optimization vector every HH steps. The comparison therefore measures optimization quality under matched training-token and optimization-communication budgets. The methods differ in additional local state updates, which may increase the step time slightly, but this is negligible compared to the communication time.
Reviewer_MQGpOpenReview ↗

Reviewer's Response

The rebuttal listed deployment efforts into multi-worker setting. The findings and observations may require some empirical effort. However, the innovations are not sufficient for a top-tier research paper such as ICLR. Additionally, the rebuttal refused to measure the downstream tasks (W4). Combining with W2, I was not convinced about the work's effectiveness, and how practical this work would be, to contribute to the community. Therefore, I keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.