- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how the specialization of hidden neurons is impacted by the imbalance in the input weight and output weight scales. The general finding is that the larger the output weight is than the input weight, this student tends to specialize to the teacher and otherwise, the student’s functional coefficient will vanish. The theoretical result is for teacher with one hidden direction and simulation extends this to teacher with two directions. This is an interesting question, but the manuscript requires substantial revision. The most serious issue is that most parts of the paper is not readable and understandable by humans. Many terms are not clearly defined and mathematical notations are cumbersome.
Critical strengths
*Understanding how the initial weight scale of each hidden neuron controls its specialization is an interesting question. This paper studies the case where the initial state has the same functional scale but different weight imbalance is a nice setting to separate the total functional contribution of the student vs. the alignment learned by the student.
*Results in Section 3.1-3.2 provides clear intuitions about why positive imbalance leads to specialization and negative imbalance does not. This is due to their different learning dynamics shown in Figure.D3, at least in the setting with only one hidden neuron.
- Figure 1 nicely demonstrates the idea that positive imbalance leads to specialization and negative imbalance tends to generate vanishing weights.
Critical weaknesses
*Results in Section 3.4 and the later sections are not readable. Many terms are not defined when first using it. This happens already at the end of section 3.3 around Eq.9.
First of all, the manuscript does not say how Eq.9 is derived.
Second, the sentence “Kernel inequalities then exclude every stationary state except (1,0,0) in an invariant region containing the initialization.” Which kernel inequality does this refer to? There are many sentences like this that are not sufficiently explained. The word “limiting dynamics” is used multiple times but it is unclear to me what it means.
Finally, the experiment section has the same issue. In line 414, the notation Psin\psi is not defined anywhere in the main text. In line 467, what are configuration A and B?
The manuscript also seems to use “student” to refer a single hidden neuron, which could be confusing. “Student” typically refers to the whole student network in the literature. The authors may consider changing this.
*How weight imbalance determines the final solution has been studied in earlier works. One missing reference here is “Jarvis et al. (2025), “A Theory of Initialisation’s Impact on Specialisation.” Tian (2020) also studies when a student weight vanishes. The author should explicitly discuss what is novel in their work compared to these ones.
Questions for authors
-
Could the authors rewrite section 3.4 and the later parts for better human readability and understanding their results?
-
Could the authors discuss the novelty of their work compared to Jarvis et al. 2025 and Tian 2020?