- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript presents an empirical study that establishes an effective yet simple solution to the training-inference mismatch problem of RL fine-tuning of LLMs. The main discovery is that using FP16 instead of BF16 can significantly improve the stability and performance of RL fine-tuning.
Critical strengths
The main strength of the work is the clear message conveyed by the experiments: changing BF16 to FP16 could be an easy but very effective solution to the training-inference mismatch problem of RL. BF16 is currently the dominant precision format used by the community, so this work should be inspiring and potentially have a large impact.
Critical weaknesses
The current conclusion is mainly based on end-point performances within a small number of setups. Though it is clear that FP16 outperforms BF16 consistently in most cases, the current investigation is still a bit superficial. In order to prove that the performance improvement is mainly due to a better matching between training and inference, more in-depth investigations are needed. For example, it is also possible that the performance gain is mainly due to the higher precision of gradient calculation or a more consistent inference results because of the improved precision. How to detach the effect of improved precision from the effect of better training-inference match is critical to further understand the phenomenon.
Questions for authors
- Do you think it is feasible to further explore the best tradeoff between exponent bits and mantissa bits. It is possible that there exists a better bit allocation other than both BF16 and FP16 for RL fine-tuning. This could be an interesting future direction.
- The improvement of using FP16 is relatively smaller in MOE models. What could be the possible causes?
2 comments
Thanks for your response. I will maintain my rating.
We thank the reviewer for the insightful and valuable feedback. We respond to individual points from your review below.
Section 4.3 was actually intended to conduct this ablation. In Figure 5, FP16 for training/inference only collapse eventually, while FP16 for both training and inference improves stably. Additionally, Figure 3 shows that training-inference mismatch is an important indicator for training collapse. So we put most of our focus on training-inference mismatch.
Thanks for this interesting question. We think it is possible to have a better bit allocation. However, it requires hardware support for such precision formats, which is beyond the scope of this work and could be an interesting future direction.
For MoE models, the top-k router for expert selection is discontinuous, so even small numerical errors can result in different activated experts, which amplifies the training-inference mismatch. In such scenarios, FP16 is still helpful because improved precision can make the router more consistent.
A common strategy for MoE RL training is rollout routing replay (aka R3), which reuses the same experts activated during inference. With R3, the numerical behavior is identical to a dense model because the discontinuity is eliminated. In our experiments, we did not use R3, which might be one reason why the improvement is relatively smaller (but still noticeable).