- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript presents a study that analyzes and tests whether learn rate transfer works in RL post-training of LLMs. From the perspective of maximum update, the transfer may not work in RL post-training. However, in empirical studies, the scaled parameterization works well when transferring among different model widths and it also exhibits a wider range of high performing learning rates.
Critical strengths
The work includes both theoretical analysis and empirical studies, investigating the learning rate transfer problem from different perspectives allows for a more comprehensive understanding.
Critical weaknesses
- The model sizes tested in this work are up to 1 billion, which is a bit too small to provide solid support that the learning rate can be transferred to larger-size LLMs.
- There is no new method or improvements proposed in this work. Simply comparing standard parameterization with scaled parameterization limits the original contribution of the work.
Questions for authors
- Why only experiments on varying width were performed? How about the effect of increased depth in learning transfer transfer?
- Do you think your current conclusion can be extended to cases where models have significantly more parameters (>>1B)?
3 comments
Additional empirical evidence
Q Depth scaling?
Thanks for raising this point. We conducted new depth transfer experiments in a more systematic setting, at depths 8, 16, 32, and 48, all with width 256. This is a 3X wider range of depths studied than in Figure 8 in our original paper, as well as having greater depth-to-width ratios to force the models to rely on depth more effectively. As seen, the collapse boundary is much more consistent for scaled than for standard parameterization, and the region of optimal learning rate is the same for scaled across depths, while standard parameterization exhibits larger variations.
In a realistic setting, note that our 3.6 billion parameter scale experiments use the official Llama 3B architecture, hence the transfer behavior exhibited there also demonstrates depth transfer, in addition to width transfer, from the smaller models.
Thanks for the responses and new experiments. I will increase my rating from 2 to 3.
Extensive additional empirical evidence.
Thank you for your helpful review!
We link to anonymized plots (allowed by ICLR) to make visualization easier.
W limited model size and empirical evidence.
Indeed a larger scale study would further clarify any transfer difference between scaled and baseline parameterizations. Towards this, we conducted extensive additional experiments, at the 3.6B parameter scale. We used the same configuration as the official Llama3 3B model (except the embedding and unembedding weights are not tied to allow their parameterizations to be scaled independently). This practical configuration was chosen to allow comparisons in a realistic setting in terms of the width, depth, and number of heads ratio.
Furthermore, we ran numerous additional experiments with more seeds to better separate real transfer from noise. As seen on tasks KodCode, Numina, and GSM-Aug, the transfer behavior is much more consistent for scaled models than baseline, staying well outside variations from noise. The standard deviation only widens near the collapse boundary in the high learning rate region. Note that the evaluation sampling seeds remain the same for different training seeds, to clearly isolate the hyperparameter transfer effects during training from evaluation noise.
In addition, following a suggestion by Reviewer fM8k, we measured the regret when transferring from smaller to larger models, defined as the loss in reward from the optimum at the larger width when using a given learning rate selected at the smaller width. This regret can be seen in Figures Numina, GSM8k-Aug, and KodCode.
In these plots, points above the diagonal line indicate positive reward gain when transferring the learning rate from smaller to larger widths. As seen, scaled parameterization generally transfers more consistently across widths at almost all learning rates. Furthermore, transfer failures become more prominent at larger model sizes for the standard parameterization, when the difference in model topology from the smallest model becomes larger. This is exactly what proper scaling is designed to fix.
W The proposed scaling is the same as ones proposed before. Novelty.
Our novelty lies in the fact that we answer the previously unanswered question: when can hyperparameter transfer for LLM posttraining with GRPO? We prove that proper scaling indeed is compatible with hyperparameter transfer (Prop. 4.1), in particular the forward and backward passes under GRPO do not introduce any width- or depth-dependence. Furthermore, we empirically show that the transfer is much more consistent for scaled models. When we first set out to derive the correct scaling from the GRPO objective, there was a priori no knowledge that the correct scaling happens to be the same as one introduced before.
Please also note that as Reviewer fM8k pointed out, our original implementation used a fixed MLP intermediate size of 8192, regardless of width. While this produced valid models, it is not standard and did lower the parameter count gap across widths and increased MLP hidden dimension consistency for all models. These new results ran the entire pretrain-SFT-GRPO pipeline using the width-dependent intermediate size, which indeed increased the transfer gap between scaled models and baseline. This makes sense, as the topology of the models is now less consistent across widths.