- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how Nesterov acceleration interacts with local-step training under a one-model-vector communication constraint. It organizes four constructions based on the initial algorithmic analysis, derives equivalences and differences among classical, modern, and primal formulations, and studies independent retention and extrapolation coefficients. Experiments on 124M and 1B language models examine coefficient choices and reported iterates, including settings where acceleration does not help.
Critical strengths
I appreciate the framing of the question around how per-step Nesterov acceleration should use shared and local history under periodic communication. The discussion of when shared history helps and the experimental takeaways are useful, but the narrative would benefit from clearer links between the proposed intuitions and the observed results.
Critical weaknesses
W1. The single-seed comparisons leave the reliability of the reported method differences unresolved. The paper acknowledges this limitation, but without repeated experiments, particularly at smaller scales, it is difficult to assess how much of the reported contribution and impact extends beyond the observed runs.
W2. The justification for transferring hyperparameters from 124M to 1B is insufficient for my understanding of the topic, reducing the perceived contribution and impact of the larger-scale results. I would expect the results to remain valid under an appropriate hyperparameter-transfer framework, but the paper needs clearer support for this transfer given the model scale, batch size, learning rate, and training horizon. Hyperparameter transfer frameworks are strongly recommended here.
Questions for authors
Q1. Please provide results from multiple seeds, especially at smaller scales, to assess the stability of the reported differences.
Q2. Please justify and support transferring the selected 124M coefficients to the 1B experiments, given changes in model scale and the roles of batch size, learning rate, and training horizon.
We thank the reviewer for their time and review, and respond to their concerns below:
W1/Q1: "The single-seed comparisons leave the reliability of the reported method differences unresolved. The paper acknowledges this limitation, but without repeated experiments, particularly at smaller scales, it is difficult to assess how much of the reported contribution and impact extends beyond the observed runs."
W2/Q2 "The justification for transferring hyperparameters from 124M to 1B is insufficient for my understanding of the topic, reducing the perceived contribution and impact of the larger-scale results. I would expect the results to remain valid under an appropriate hyperparameter-transfer framework, but the paper needs clearer support for this transfer given the model scale, batch size, learning rate, and training horizon. Hyperparameter transfer frameworks are strongly recommended here."