acceptodds
Under review as a conference paper at ICLR 2027

Relative Weight Scales Control Feature Specialization in ReLU Networks

Abstract

The success of deep learning depends on learning useful representations, yet predicting how training organizes these representations across neurons remains difficult. In this work, we show that changing the scale of initial weights can determine which neurons learn a feature without altering any neuron’s initial contribution. We construct ReLU networks with identical initial features and predictions that reach the same final predictions with different roles for their neurons. In one, all neurons share the learned feature. In the other, one neuron acquires it while every other neuron’s contribution vanishes. The only change is the relative scale of each neuron’s input and output weights. Our analysis explains how an initial learning advantage persists through convergence: as one neuron learns the target, it reduces the error driving the others and limits their subsequent adaptation. We prove this outcome in a nonlinear model under gradient flow and small-step gradient descent, and quantify how scale changes the speed and path of feature learning. Experiments verify the predicted dynamics and show that scale also changes feature assignment when two features compete. These results reveal how initialization can control the organization of a learned representation without changing what the network initially represents.

Primary area:
learning theory
Keywords:
Learning Theory, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zf2nOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how the specialization of hidden neurons is impacted by the imbalance in the input weight and output weight scales. The general finding is that the larger the output weight is than the input weight, this student tends to specialize to the teacher and otherwise, the student’s functional coefficient will vanish. The theoretical result is for teacher with one hidden direction and simulation extends this to teacher with two directions. This is an interesting question, but the manuscript requires substantial revision. The most serious issue is that most parts of the paper is not readable and understandable by humans. Many terms are not clearly defined and mathematical notations are cumbersome.

Critical strengths

*Understanding how the initial weight scale of each hidden neuron controls its specialization is an interesting question. This paper studies the case where the initial state has the same functional scale but different weight imbalance is a nice setting to separate the total functional contribution of the student vs. the alignment learned by the student.

*Results in Section 3.1-3.2 provides clear intuitions about why positive imbalance leads to specialization and negative imbalance does not. This is due to their different learning dynamics shown in Figure.D3, at least in the setting with only one hidden neuron.

  • Figure 1 nicely demonstrates the idea that positive imbalance leads to specialization and negative imbalance tends to generate vanishing weights.
Critical weaknesses

*Results in Section 3.4 and the later sections are not readable. Many terms are not defined when first using it. This happens already at the end of section 3.3 around Eq.9.

First of all, the manuscript does not say how Eq.9 is derived.

Second, the sentence “Kernel inequalities then exclude every stationary state except (1,0,0) in an invariant region containing the initialization.” Which kernel inequality does this refer to? There are many sentences like this that are not sufficiently explained. The word “limiting dynamics” is used multiple times but it is unclear to me what it means.

Finally, the experiment section has the same issue. In line 414, the notation Psin\psi is not defined anywhere in the main text. In line 467, what are configuration A and B?

The manuscript also seems to use “student” to refer a single hidden neuron, which could be confusing. “Student” typically refers to the whole student network in the literature. The authors may consider changing this.

*How weight imbalance determines the final solution has been studied in earlier works. One missing reference here is “Jarvis et al. (2025), “A Theory of Initialisation’s Impact on Specialisation.” Tian (2020) also studies when a student weight vanishes. The author should explicitly discuss what is novel in their work compared to these ones.

Questions for authors
  1. Could the authors rewrite section 3.4 and the later parts for better human readability and understanding their results?

  2. Could the authors discuss the novelty of their work compared to Jarvis et al. 2025 and Tian 2020?

Reviewer_zygGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the relative initialization scale (the ratio of input-to-hidden vs hidden-to-output weights) affects the ultimate learned solution in a ReLU network setting. The core finding is that even when two networks have identical initial outputs, if they differ in relative scale they can learn different internal representational structures. In particular either the network learns one neuron with many nearly dead neurons, or many neurons that copy the same learned feature. The theoretical results prove this structure in the setting considered.

Critical strengths

The paper does a fine job of discussing the impact of relative scale, extending analyses to a nonlinear setting. The result is intuitive in light of related literature, but the formal proof is a contribution.

Critical weaknesses

Overall the results are interesting but limited in scope and the broader significance is uncertain. These results seem to be a modest delta over prior work, namely Kunin et al.

The paper could be improved by citing related work on ReLU network specialization, such as Jarvis et al., 2025. A theory of initialisation’s impact on specialisation, which also studies ReLU specialization under weight imbalance (their terminology for different relative scales).

Questions for authors

What implications does this analysis hold for AI systems more broadly?

What conceptual confusions or debates does the theoretical analysis resolve?

Reviewer_KpsiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the relative scales of input and output weights affect which neurons learn a target feature. It constructs two-layer ReLU networks with identical initial predictions and neuron contributions but different subsequent learning dynamics. In a Gaussian single-teacher setting, an appropriate rescaling makes one neuron learn the target while the other neurons’ contributions vanish, whereas a symmetric initialization leads to shared learning.

Critical strengths
  • Holding the initial predictor and individual neuron contributions fixed provides a useful way to isolate the effect of parameter scaling.
  • The analysis follows the dynamics through convergence. In particular, it explains why an early learning advantage can persist after the selected neuron reduces the residual driving the other neurons.
Critical weaknesses
  • Limited explanation of novelty: The dependence of feature learning on relative layer scales, and the emergence of specialized and redundant neurons, already appear in existing two-layer analyses. The paper acknowledges several of these connections, including Kunin et al., Williams et al.. However, it does not explain sufficiently clearly what new understanding follows from the particular construction beyond making a familiar competition mechanism precise.
  • Unclear consequences for learning: It is not clear why having one neuron learn the target while the others vanish is itself an interesting learning outcome. Both constructions reach the same final predictor. The paper would benefit from explaining whether the different allocation of features has consequences for generalization, subsequent learning, robustness, or pruning.
  • Restricted setting: The main convergence result concerns one teacher feature, Gaussian inputs, identical initial student directions and contributions, and sufficiently large scale imbalance. The extension to nearby initializations establishes finite-time selection, while eventual pruning requires additional conditions. The experiments with multiple features are interesting, but the extent to which the main mechanism persists in more general learning problems remains unclear.
  • Presentation of the main insight: The distinction between faster initial alignment, finite-time selection, and eventual specialization should be made more transparent. These are different conclusions, but their assumptions and proof requirements are spread across several results. A shorter explanation of the main mechanism would make the contribution easier to assess.
Questions for authors
  • How does the proposed mechanism differ from the separation of learning times and subsequent reduction of residual signal familiar from saddle-to-saddle dynamics and analyses of two-layer and multi-index models?
  • What consequence for learning follows from selecting one neuron rather than distributing the same feature across several neurons?
  • Which parts of the convergence result persist for nonduplicate initializations and multiple teacher features without additional separation assumptions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.