- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper provides the generalization bounds for the homogeneous neural network regimes and proves that this regime enables a slower step-size decay under mild assumptions. This paper also extends the theoretical results in several aspects, such as other loss functions, the specific case of three-homogeneous networks, and non-Lipschitz regimes.
Critical strengths
This paper establishes generalization bounds for -homogeneous neural networks via algorithmic stability, and then it specializes these results to the case of three-homogeneous neural networks. Generalization bounds without the Lipschitz assumption are also derived based on the concept of conditional on-average stability.
Critical weaknesses
-
The experiments evaluate SGD with various step sizes. For validation of theoretical results, experiments are validated only for the three-layer ReLU network. The experiments may not be sufficient to support the claimed broad applicability of the theoretical findings. It is also unclear how those assumptions can be verified.
-
It remains unclear how the current theoretical analysis could be extended to more general settings or how some assumptions could be relaxed (see Questions).
Questions for authors
-
Can the theoretical results of this paper be extended to practical scenarios beyond homogeneous neural networks? For example, transformer or large language models?
-
Can this paper be extended to more general multi-class classification tasks?
-
Experimental validation on more practical networks is needed, also for those models mentioned in the paper, including VGG-19?
-
Could the authors explain why Assumption 1 (lower bounded loss) is mild?