- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how the learnability of teacher-student neural networks depends on the geometry of the teacher neurons. Both the teacher and the student are single-layer neural networks with the same input dimension. The student network may have more neurons that the teacher. When both teacher and student have only a single neuron, the authors show that the initial dissimilarity between the teacher- and student-neurons could lead to divergence of learning. When both teacher and student have two neurons, the authors show that a teacher with maximally dissimilar neurons can allow easier learning. Similar observations are made when the teacher has an even a larger number of neurons. Finally, the paper shows that increasing the learning rate of the readout layer while decreasing that of the hidden biases can improve learning success. Most of the results are obtained through numerical simulations of these networks.
Critical strengths
- Studying how the similarity or dissimilarity between teacher neurons can affect the learnability of the student seems to be a new angle.
Critical weaknesses
-
The significance of the main findings for larger networks seems questionable. Specifically, if the teacher network has more than two layers, it is unclear how one can still determine whether neurons are similar or dissimilar. Generalization to real networks (e.g., transformers) seem to be even more difficult.
-
The experimental setup of this paper has little connection to practice. For theory, it is understandable that we study a simple model. However, the experiment in the paper also uses a NN with one hidden layer, which is rarely used in practice.
-
Most of the results for two or more neurons are based on experimental observations (except Lemma 1, but it is unclear to the reader what its significance is). In this sense, the technical depth of the paper is somewhat limited.
Questions for authors
-
Does the main conclusion of this paper provide any insight when the teacher network is any real networks (such as a well-optimized language transformer or diffusion transformer)?
-
Where is Figure 2E that the paper refers to at the end of Section 3?
2 comments
I thank the authors for the response. However, it does not seem to address my concerns on the technical depth and on the relevance to more complex networks. I wish to keep my score.
Thank you for recognizing the novelty of our research question, and the soundness of our study. Below, we address your questions.
Question 1
Our analyses allow us to make predictions about which networks would be easier to fit than others based on their internal geometry. This should be possible to do in deep networks by considering the similarity of the different neurons in each layer separately. Connecting these results to transformers is an interesting question for future research!
Question 2
Thanks for raising this point. This was actually supposed to be Figure 2D. We have updated it in the manuscript.
If you have any more questions, please let us know. If not, we would be grateful if you would consider raising your score!