- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In this paper, the authors explore the condensation phenomenon in two-layers ReLU networks when weight decay is employed. They separate training into three stages: fitting, compression, and condensation. Through an evaluation of the evolution of the weights’ amplitude and angle, they show mostly theoretically how weight decay with high initialisations is usually linked with higher levels of condensation.
Critical strengths
- S1 - Theoretical derivation: Most of the results are strongly backed by theoretical derivations and proofs. I have not read them in detail, in particular the parts in the Appendix, but everything I have read made sense. The main claims in the paper are appropriately backed by the theoretical results.
- S2 - Clarity and presentation: Although the paper mostly consists of theoretical derivations, the explanation and discussion of the research performed is very clear. The goal of the paper is easy to understand.
Critical weaknesses
- W1 - Significance: This paper focuses on the simplest example of two-layers ReLU neural networks representing very simple piecewise linear functions. In the Appendix, the authors explore a few more settings experimentally. The Transformer experiment somewhat addresses this issue and demonstrates that the authors’ results are empirically verified, but to do so they have to use a very large weight decay factor of $0.8$. All things considered, these results are interesting, but it is unclear how strong the condensation effect of weight decay would be in more realistic settings.
Questions for authors
- Why have the authors used such a high $L_2$ weight decay factor in their Transformer experiment? What would their results look like with more common values that are closer in order of magnitude to the learning rate?
- I have not clearly understood where the different stages, and the boundaries between them, are precisely defined. In Section 4, how exactly is it defined when to go from one stage to the next one? In particular, between stages II and III.
3 comments
Response to the authors' rebuttal
I would like to thank the authors for their rebuttal. They have cleared up the questions I raised, and after reading through the other reviews and rebuttals, I believe that the critical weaknesses have been addressed. I will raise my score.
Thank you for taking the time to reconsider our work. We are glad that our rebuttal has addressed your concerns. We sincerely appreciate your careful evaluation and constructive feedback.
We sincerely thank you for your careful review and constructive feedback. We have made the corresponding revisions in the manuscript. Below, we address your concerns point by point.
1. Choice of the weight decay coefficient (Weakness 1, Question 1)
We would like to clarify that the weight decay coefficients adopted in large language model training do not need to share the same magnitude as the learning rate. For example, the DeepSeek-V3 (https://arxiv.org/pdf/2412.19437) uses a weight decay coefficient of 0.1. If the weight decay coefficient is set to the same order of magnitude as the learning rate, substantially longer training may be needed before condensation becomes observable.
We have added an experiment with a weight decay coefficient of 0.1 in Appendix H.6, which is closer to values used in practical training settings. With the training duration extended to 1,200 epochs, we observe condensation in this setting as well.
2. Boundaries between the three stages (Question 2)
We do not specify sharp boundaries between the three stages in the paper. Rather, the stages describe changes in the dominant behavior of the training dynamics.Theorem 5 shows that, once neuron amplitudes stabilize, the extrema of qe(θ) also stabilize. The subsequent angular dynamics then drive neuron directions to condense around these extrema. Accordingly, we use when neuron amplitudes reach a plateau as an empirical indication of the transition into Stage III.