- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper constructs a counterexample showing that Gaussian activation tails can affect the asymptotic training dynamics of neural networks, even though they vanish under ReLU homogenization. They further extend the result to gradient descent and SGD, and show that adding a trainable output bias can eliminate the classification failure in their construction.
Critical strengths
Not applicable.
Critical weaknesses
The motivation of this paper is unclear. Why should we expect the homogenization of an activation function to determine the asymptotic parameter profile of gradient flow, particularly when training fails to fit the data? I am not convinced that the paper addresses an important or well-motivated question.
The presentation is unnecessarily complicated and overly defensive. Even the opening sentence of the abstract introduces several technical concepts without explaining the underlying problem. The extensive mathematical machinery makes it difficult to identify the central insight and why it matters.
Questions for authors
What is the main motivation for studying this particular failure case? What broader insight about neural network training can we learn from such a specialized construction?