- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies implicit bias in two-layer ReLU neural networks and investigates whether neurons will stay on optimal features or leave them later in training. The authors find the phenomenon of leaving optimal features both theoretically and empirically.
Critical strengths
The paper provides both theoretical and empirical evidence.
Critical weaknesses
The example with specific numbers in Section 3 is a good motivating example. However, it would be helpful to provide results beyond this example under some reasonable assumptions that do not restrict the claim to specific numerical values.
Questions for authors
If the authors think their example with specific numbers in Section 3 can be directly extended beyond those specific values, an explanation would be helpful.
Thank you to the authors for the effort of extending the result. However, I think the authors' result is still far from a generic characterization of the condition. While some parts became better using the variable q, other parts (e.g., y=(1,0,2,1), (1,0), (0,1) in X_q) still remain as specific values. Furthermore, the interval [0.95, 1.05] for q is quite short. Therefore, I will keep my score.