- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how the source model’s learning-rate schedule affects the training progress retained after model growth and a learning-rate reset. It compares growth and no-growth runs from flat and annealed sources, measuring post-event loss differences at equal FLOPs. Using a river-valley approximation, it separates persistent progress, represented by the accumulated normalized learning rate, from a temporary loss reduction maintained by a low learning rate. This gives a parameter-free prediction for a shorter flat-trained source that should match an annealed source after reset. For their BERT experiment, the predicted flat-source event is at step 93027 rather than 165000. The experiment results support an approximate relationship between accumulated learning rate and reusable source training.
Critical strengths
-
This paper identifies a practically relevant aspect of growth evaluation and evlauates it in compute units. A function-preserving expansion does not guarantee that subsequent training preserves the source checkpoint’s low loss when the learning rate is reset. The explicit comparison of flat and annealed sources, with and without growth, makes this issue visible.
-
The river-matching rule makes a quantitative prediction before observing the continuation results. The matching point is derived from the schedule, without fitting the post-reset loss curves. In the main experiment, the rule predicts that a flat source trained for 93027 steps should approximately match a 165000-step annealed source after reset. The agreement is examined without growth and under both LEMON and AKI.
Critical weaknesses
Limited generalizability to more complex model growth: The proposed river–hill budget is derived from an idealized SGD analysis with a fixed model architecture, and its applicability across architectural changes is primarily supported by empirical observations rather than theoretical guarantees. Moreover, the experiments are restricted to a single width-expansion setting with BERT, without considering depth expansion or multi-stage progressive growth. In parctice, architectural changes may substantially alter the loss landscape, curvature, and optimization dynamics, potentially invalidating the assumption that reusable training progress can be characterized solely by accumulated learning rate. Consequently, it remains unclear whether the proposed budget and the resulting quantitative conclusions generalize beyond the relatively simple growth setting studied in this paper.
Questions for authors
-
Could the authors further clarify under what conditions the river–hill budget and reset-equivalence prediction are expected to remain valid when model growth involves increasing depth or multiple growth stages? In particular, how might changes in the loss landscape induced by these operations affect the assumption that reusable progress is proportional to accumulated learning rate (I dont think this assumption is valid in complex cases)? Is there any theoretical justification or existing empirical evidence supporting such extensions, or should the current conclusions be interpreted as primarily applicable to the single-stage width-expansion setting considered in this paper?
-
Could the authors explain more detailed the meaning of the 45% claim? Should this be interpreted as the forgone fraction of peak-rate-equivalent steps within the annealing segment, rather than a fraction of achieved loss reduction or total pipeline compute?
We thank you for your time and thoughtful engagement with our paper.