acceptodds
Under review as a conference paper at ICLR 2027

River-Hill Budgets: Understanding the Effects of Annealing in Model Growth

Abstract

Model growth initializes a larger model from a smaller source model that has already been trained, but saves compute only if the grown model properly reuses that source’s training. Work on growth has focused on the growth operator, which is in charge of preserving the source model’s function across the transfer while ensuring the newly added parameters are properly made use of. However, when param eters are transferred and the learning-rate schedule is reset, the sudden learning-rate increase creates a training offset on top of the parameter transfer. We separate the impact of the growth operator from the learning-rate increase and find that the newly expanded parameter space does not necessarily help recover from this schedule offset. Using the river-valley view of the loss landscape, we explain and quantify this offset. For a standard BERT configuration, the river-valley view predicts that 45% of progress made during the decay to 10% peak learning rate is lost upon reset, and up to 50% for a decay to zero. The same model trained with 45% fewer anneal steps, making only reset-surviving progress, closely matches the annealed run’s loss trajectory. These results show that the source model’s learning-rate schedule severely limits how much training survives model growth.

Primary area:
optimization
Keywords:
model growth, learning rate schedules, annealing, loss landscape, efficient training, progressive training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ng4UOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the source model’s learning-rate schedule affects the training progress retained after model growth and a learning-rate reset. It compares growth and no-growth runs from flat and annealed sources, measuring post-event loss differences at equal FLOPs. Using a river-valley approximation, it separates persistent progress, represented by the accumulated normalized learning rate, from a temporary loss reduction maintained by a low learning rate. This gives a parameter-free prediction for a shorter flat-trained source that should match an annealed source after reset. For their BERT experiment, the predicted flat-source event is at step 93027 rather than 165000. The experiment results support an approximate relationship between accumulated learning rate and reusable source training.

Critical strengths
  1. This paper identifies a practically relevant aspect of growth evaluation and evlauates it in compute units. A function-preserving expansion does not guarantee that subsequent training preserves the source checkpoint’s low loss when the learning rate is reset. The explicit comparison of flat and annealed sources, with and without growth, makes this issue visible.

  2. The river-matching rule makes a quantitative prediction before observing the continuation results. The matching point is derived from the schedule, without fitting the post-reset loss curves. In the main experiment, the rule predicts that a flat source trained for 93027 steps should approximately match a 165000-step annealed source after reset. The agreement is examined without growth and under both LEMON and AKI.

Critical weaknesses

Limited generalizability to more complex model growth: The proposed river–hill budget is derived from an idealized SGD analysis with a fixed model architecture, and its applicability across architectural changes is primarily supported by empirical observations rather than theoretical guarantees. Moreover, the experiments are restricted to a single width-expansion setting with BERT, without considering depth expansion or multi-stage progressive growth. In parctice, architectural changes may substantially alter the loss landscape, curvature, and optimization dynamics, potentially invalidating the assumption that reusable training progress can be characterized solely by accumulated learning rate. Consequently, it remains unclear whether the proposed budget and the resulting quantitative conclusions generalize beyond the relatively simple growth setting studied in this paper.

Questions for authors
  1. Could the authors further clarify under what conditions the river–hill budget and reset-equivalence prediction are expected to remain valid when model growth involves increasing depth or multiple growth stages? In particular, how might changes in the loss landscape induced by these operations affect the assumption that reusable progress is proportional to accumulated learning rate (I dont think this assumption is valid in complex cases)? Is there any theoretical justification or existing empirical evidence supporting such extensions, or should the current conclusions be interpreted as primarily applicable to the single-stage width-expansion setting considered in this paper?

  2. Could the authors explain more detailed the meaning of the 45% claim? Should this be interpreted as the forgone fraction of peak-rate-equivalent steps within the annealing segment, rather than a fraction of achieved loss reduction or total pipeline compute?

1 comment

We thank you for your time and thoughtful engagement with our paper.

  1. We have extended the experiments as detailed in our general comment, and our results hold for depth growth and multi-growth. Our model explains what happens at the moment of reset. The hill reduction is rapidly lost, while the river progress is kept and carried into the grown model by the operator. What we measure is what the source loses, the hill term. How the grown model then interacts with the river-valley structure it inherits, and how well the river progress is preserved, remains unknown. The current river-valley perspective does not describe how this structure transforms across an architectural change (Sec. 6, "The river valley under growth"), and we see this as a next step worth exploring.
  2. We agree that the claim was unclear. Your reading is correct, it is the share of steps. 45% is the share of the decay phase steps only (excluding warmup). We've updated Table 1 to use the total number of source steps (including warmup) as the denominator for ease of comparison, and now show the respective shares for each of the new schedules.
Reviewer_JT9vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies model growth. The paper argues that resetting the learning rate schedule can contribute to degradation of the model's loss after a growth operator is applied. The paper separate the contribution of the operator itself and the learning rate schedule, and argues that the river-valley landscape view of the loss from Wen et al. (2025) can explain why performance spikes after a growth and reset.

Critical strengths

Clarity: The paper is reasonable well written.

Impact/Informativeness/Novelty: The paper is studying a problem for which there is not a lot of prior work, which is that of the interaction between model growth and the learning rate schedule. The paper also argues that resetting the learning rate results substantial amount of the training (during the annealing) being effectively thrown away, which has implications for efficiency during training.

Soundness: The paper seems sound.

The river-valley intuition is interesting.

Critical weaknesses

Completeness: I think the main weakness for this paper is that it is missing an experimental breadth that would be required to fully understand the relationship between the learning rate schedule and model growth. For example:

  • The main claims made by this paper, relating the annealing vs not annealing and growth, many depend substantially on the quality and properties of the original model before growth including all of the usual training hyperparameters, the number of training steps, the size of the model, perhaps even the architecture of the model. Generally, tracking the effect as a function of each of these hyperparameters and also as a function of e.g. the original base model loss, would improve the completeness of this paper.
  • The primary purpose of this paper is to study the effect of learning rate schedules, but only cosine, with a few different min learning rates (r=1, e.g., constant, r=0.1, and r=0) were studied. A complete paper should also consider other learning rate schedules, attempt to quantify the effect of the minimum learning rate r, compare across different warmup strategies for the new model, compare different learning rates, etc.
  • Only one growth operator was studied. It would make a much stronger paper to consider multiple types of growth operators, and try other approaches that intuitively would limit the amount of degradation that is incurred by growing the model.
Questions for authors

See weaknesses.

2 comments

Thank you for your time and for recognizing the novelty and implications of our work.

  1. We have increased the breadth of the river-matching experiments as detailed in our general comment. As for the other variables (hyperparameters, size, architecture, source model loss at growth, etc.), we have chosen to focus on investigating how much of the loss offset at growth can be attributed to the learning rate schedule across the source and grown models. We have clarified that the remainder can only be attributed to everything around the growth operation rather than specifically to the growth operator. The Limitations now state that we use one model scale per setting and one peak learning rate. We agree that considering the other variables would help complete the picture, and we encourage future work to investigate how these interact with the learning rate schedule and growth. In the meantime, we believe that this work offers an intuitive explanation for how much compute can be reused by growing the model given a learning rate schedule. We have added additional schedules and ratios, notably r=0r=0 (Fig. 3). App. E explains our analysis of warmup. Measuring the offset rather than the instantaneous loss spike, we find that a re-warmup lowers and delays the spike, and delays the recovery with it, so the offset is shifted in time rather than reduced. A warmup strategy that mitigates this may exist. Our analysis applies directly when only an annealed checkpoint is available, since the budget gives how much of its training will be reused, and we leave such strategies to future work.
  2. In addition to the non-function-preserving operator (AKI) of the original submission, we now include depth growth and multi-growth, and have found that our river-matching prediction holds in each. As for other approaches to limit the amount of degradation, we encourage future work to measure degradation in terms of a loss offset vs. FLOPs rather than focusing primarily on reducing the loss spike or final performance.

I appreciate that you are adding the potential lack of breadth of experiments to the limitations sections, but this doesn't address the main concern I have, which is that for a paper with the goal of understanding the interaction between the learning rate schedule and model growth, the paper doesn't cover sufficient breadth in the learning rate schedule and associated hyperparameters to be able to make a strong claim. I would still lean reject.

Reviewer_BRkdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes the learning rate decay in the context of the model growing. The central question can be formulated as "how much of a source model's training is lost when the source is grown and its learning rate is reset to peak?" Separating the growth-operator perturbation from the schedule reset at equal FLOPs, the authors show the schedule offset is the larger and more persistent of the two, and that added parameters do not help absorb it. They then apply the river–valley loss-landscape model to predict the offset from the schedule alone: a cosine anneal to 0.1 η_max retains only 55% of its steps as river progress after reset, discarding 45%. A "river-matched" flat source trained for exactly that much progress becomes indistinguishable from the annealed source after the reset, confirming the prediction with and without growth.

Critical strengths
  • The authors define a clean framework to separate the loss spike into two parts: growth-induced and LR-induced. Later, they show that LR-induced spike is the biggest contributor. I see this as the main not-obvious outcome of the paper.
  • The authors provide a deep exploration into the "river-matched" approach to justify the proposed growth without annealing.
  • The authors frame the model growth in the broader scale: the pre-growth model matters beyond compute spend on it but also HOW the compute was spent.
Critical weaknesses
  • The main outcome of the work can be formulated more broadly: "LR annealing wastes compute if training is continued." This phenomenon has already been analyzed in prior work, including in exact or similar settings [1,3], as a corollary of broader cooldown analyses [2], and as a consequence of the river-valley view of training (Wen et al., 2025). Please clarify what is novel in your findings beyond demonstrating known behavior in a new setting (model growth). Consider reframing the conclusion—"These results show that the source model’s learning-rate schedule severely limits how much training survives model growth"—as it currently overstates the contribution. Rather, the primary contribution lies in the decomposition of the performance drop. Overall, the narrative framing remains ambiguous: are you investigating cooldown resumption or exploring model growth?
  • Figure 2 (middle and right) compares models over only a small fraction of the FLOPs shown in the bottom-left panel. Please consider running LflatL_\text{flat} and LresetL_\text{reset} longer to provide a complete picture of post-event dynamics; the current window is too narrow to draw solid conclusions.
  • The absolute values of ΔL\Delta L in Figure 3 are uninformative, as any variance can appear significant through y-axis scaling. Please use a relative metric instead (e.g., normalized by a reference loss). The same issue applies to the formulation of I(c)I(c); consider updating its definition with a baseline-loss normalization.
  • The main text does not specify which model architecture is used in each experiment or its size.

Minor

  • Section 2 introduces a rather unusual setup: increasing the LR to its maximum without re-warming up. Consider citing the appendix here to maintain flow while justifying this learning-rate schedule.
  • Please standardize the notation across variants such as Lreset + growL_{\text{reset + grow}} and Lreset,GL_{\text{reset}, G}.
  • When stating training step counts, please include the corresponding token counts in parentheses.
  • Please consider creating your own descriptive Figure 1 rather than reusing a visualization from a different paper.

[1] How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining; Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu, Wenguang Chen; ICLR 2026

[2] Training dynamics of the cooldown stage in warmup-stable-decay learning rate scheduler; Aleksandr Dremov, Alexander Hägele, Atli Kosson, Martin Jaggi; TMLR, ICLR 2026

[3] Simple and Scalable Strategies to Continually Pre-train Large Language Models; Ibrahim, A., Thérien, B., Gupta, K., et al.; TMLR

Questions for authors
  • Is there a specific reason for why you use BERT and MLM as the objective for the experiments?
1 comment

Thank you for your time and helpful comments, which have contributed to improving our submission.

  1. We agree that the effect of resetting / rewarming the LR after an anneal is already demonstrated in prior work. The introduction now cites [3] and Wen et al. as establishing the issue, and [1] and [2] are discussed in Related Work. We have adjusted the framing to focus on the decomposition and its implications for model growth, so that the paper focuses on exploring what the wasted compute represents (i.e the compute the source spent reducing the hill term). Our core contribution is this decomposition and its explanation through the river-hill budget, which river matching then tests. Together, they explain how much of the source compute can be reused under a given LR schedule, which goes beyond analyzing the magnitude of the loss spike or the recovery time. We reworded the conclusion to make clear that our contribution is measuring how much training survives growth, through the decomposition and the river-hill budget.
  2. Since we compare the effect of schedules, we matched runs by step count so that every run is at the same point of the same schedule and has seen the same data. Matching on FLOPs instead would mean continuing the runs that are not grown past their schedule, either at the floor or with a stretched cosine, and adding several passes over the repeated data. Both would have introduced differences that the grown runs do not have. To make the difference in compute in the comparison clearer, we have adjusted the figures to show more of the grown runs.
  3. We agree that the scale was uninformative. Fig. 2 in Sec 2 now shows Δsched\Delta_{sched} and Δgrow\Delta_{grow} for scale. For Sec. 4, II was not the most appropriate metric as Δsched\Delta_{sched} should vanish. We instead plot the offsets Δsched\Delta_{sched} and Δgrow\Delta_{grow} which show that Δsched\Delta_{sched} hovers near zero and is small relative to Δgrow\Delta_{grow}. We also added a plot that aligns the two loss curves to show the post-growth similarity. We include the no-growth controls as dashed lines.
  4. Experiments are now more clearly defined. As for the question, we use BERT and MLM to follow the configuration of LEMON, the primary operator we studied, so that it runs in the setting it was designed for (Sec. 2, Setup).
  5. Minor changes were all addressed:
    1. Warmup is now more explicitly discussed, and the appendix graphs are made clearer.
    2. Notation was standardized and now properly translates across experiments.
    3. Token counts were added alongside step counts.
    4. The original caption did not make the figure's origin clear. Fig. 1 is our own figure. We reused the river-valley landscape of Wen et al. on purpose, so that readers already familiar with it have a common starting point from which we develop the intuition for annealing and growth. The updated caption makes this explicit.

Comments on the paper

1 comment

We want to thank all three reviewers for their time and constructive feedback, and apologize for the delayed response. We ran a significant number of new experiments and revised much of the text, so we summarize the main changes here.

Framing. The revision makes our contribution clearer. Prior work shows that resetting an annealed model has a cost, and the introduction now presents this as our starting point. Our contribution is an account of what that cost consists of and how large it is at growth. The offset after growth separates into a growth part and a schedule part (Sec. 2). The river-valley view explains the schedule part as the anneal's gain, which the reset discards (Sec. 3). River matching tests this explanation (Sec. 4). The abstract, introduction and conclusion were rewritten accordingly, and claims are scoped to the settings we tested.

New experiments. We added four river-matched pairs (Table 1, Figs. 3 and 4): a cosine to 0 without warmup (Fig. 3), a Noam schedule with a 0.1 floor (Fig. 4c), depth growth from 6 to 12 layers (Fig. 4a), and multi-growth with two successive events, matched at each (Fig. 4d). Every pair runs on three seeds and includes no-growth controls. In each, the matched sources continue to follow nearly the same trajectory after the reset.

Presentation. Figures show more of the grown runs, and the river-matched figures plot Δsched\Delta_{sched} against Δgrow\Delta_{grow}. Each experiment's architecture and size are given in the main text, notation is standardized in the text, and step counts come with token counts.

NOTE: Two runs of the depth-growth and multi-growth pairs are still finishing. We will update Figs. 4a and 4d with the complete results once they end.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.