- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper provides an empirical study of the error from FP4 training. It breaks down the loss gap from FP4 training into error from the forwards pass, backwards pass, and "excess" error caused by correlations between quantizing both the FW and BW pass. The paper studies RTN W4A4 FW passes and unbiased backwards passes, as has been adopted in the FP4 training literature.
The main findings of the paper are that the relative effect of forwards pass quantization goes down with scale, whereas backwards pass quantization does not. Gradient variance from stochastic rounding adds a constant/ slightly growing loss gap as training progresses.
The paper claims to study FP4 training across different scales, but all models in the paper are tiny and much smaller than models one would reasonably train in FP4 to get real-world speedups. It is not clear to me if the trends in the paper hold at actual LLM-sized models (eg 10 or 100B parameters), or if they reverse at some point since the paper only studies ~100M to 1B parameter models.
This paper does not contain any real theory to explain the effects they observed and is largely just an empirical study on various FP4 training configs. The writing in the paper also seems heavily LLM edited / generated and is hard to follow. There are a lot of unnatural word and phrase choices that native English speakers would not use.
Critical strengths
The findings in this paper are interesting and are "new" in the sense that I am not aware of other papers that have looked into this for FP4 training. However, given the small scale of the models and limited analysis, it is not clear to me how valuable these results are vs the same experiments run on larger models.
Critical weaknesses
See above. The models used are tiny and there is limited theoretical analysis on why these behaviors might be happening. I also found it odd that NVFP4 clipping did not hurt the model, since this makes the gradient biased.
Questions for authors
See above.