- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper analyzes the learning dynamics of two-layer ReLU networks in the mean-field limit. While prior works have established exponential convergence guarantees for bounded and smooth activation functions, the quantitative convergence guarantee for ReLU networks remains still open. The paper have developed novel techniques to handle non-smoothness and unboundedness of ReLU activation and derived exponential convergence rate to the global optimum.
Critical strengths
- Quantitative convergence guarantees for ReLU networks in the mean-field regime is an important open question and the paper have addressed this by demonstrating the exponential convergence guarantee under mild assumptions. The dependence of the convergence rate on the inverse temperature is polynomial, which is milder than that established in the other setup.
- The paper have introduced several techniques to resolve technical difficulties posed by the non-smoothness and unboundedness of ReLU.
Critical weaknesses
- The obtained PL constant has exponential dependence on the dimension while the dependence on the inverse templerature is polynomial. Thus, the obtained convergence guarantee is not effective for high-dimensional problems.
- The convergence analysis heavily depends on the 2-homogeneity of ReLU activation. Thus, it is not straightforward to extend it to other unbounded activation functions.
Questions for authors
- Could you elaborate how the PL constant depends on the dimension? Is it milder than that established in the previous works?
- Is it possible to extend the analysis to other activation functions like GeLU?