- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates when scaling the weights keeps the same hidden neurons in different substructures with Normalizations, such as MLPs and Attentions. Under this condition, weight decay can be ignored to simplify the training settings. The authors also discussed the convergence rates under this setting with detailed mathematical analyses. Experiments are also conducted to identify the optimization methods.
Critical strengths
The mathematical analyses are detailed and meaningful.
The article is well-organized and easy to follow.
Critical weaknesses
Confusion on the title. I did not get what the title tells us. In the main context, I mainly see the discussion on the authors conduct researches on weight decay. I did not get what is normalized from the main context related to the title.
Limited experiments. In experiments, this paper shows the acceleration using NorSODA. However, the improvement is limited, and the final performance is not recorded. It is better if the authors test their methods on additional tasks like image-classifications and so on.
Questions for authors
- Can the authors further explain what the meaning of the title is, and its relationship to the research topic of this paper?
- What is the final performance of Figure 2? Are there other indexes which can test the performance except the validation loss? Note that the lower loss does not always obtain the better performance.
- Can the method applied on the ViTs for image-classification tasks? What are the training/validation accuracy? Will we get the similar conclusions as the current experiments?
2 comments
Thanks for your detailed reply. All my concerns are addressed. I will maintain my score.
We thank the reviewer for the positive assessment. Following the comments, we have clarified the title and added a new image-classification experiment on CIFAR-10.
The title is meant as a play on words, conveying that fact that normalization of weight can safely be used by the optimizer for weights that are followed by normalization layer. We have change the title to "Normalize your normalized layers: acceleration under scale-invariance" to specify more precisely the main theoretical contribution.
We confirm with new experiments that the improvements of NorSODA carry over to image classification on CIFAR-10 without any retuning. Specifically, we train a convolutional network with batch normalization on CIFAR-10 (Appendix E.3), which differs from the nanoGPT experiments in almost every respect, as it uses:
The results are in Figure 6, with an accompanying table that reports the final test accuracy at every horizon.
Questions
See the response above regarding the title.
The final performance of the two methods is stated in the plot (see the zoomed-in panel). For pre-training of LLMs we are not sure of a better measure. However, for the new CIFAR-10 experiments we are reporting the test accuracy (Figure 6).
Yes, we obtain similar results on CIFAR-10 without any retuning as mentioned above.