acceptodds
Under review as a conference paper at ICLR 2027

SGD Selects Initialization-Dependent Representation in Overparameterized ReLU Networks

Abstract

Overparameterized neural networks can represent the same function through many different hidden configurations, raising the question of which representation is favored by a given training algorithm. We study how stochastic gradient descent (SGD), jointly training both layers, resolves this ambiguity in finite-width, bias-free, two-layer ReLU networks trained to learn a linear teacher under Gaussian inputs. At width two, exact fitting leaves no choice: the neurons must align with the teacher directions. At every width , however, infinitely many non-aligned exact fits are possible. Despite this ambiguity, we prove that joint batch-size-one SGD from sufficiently small initialization drives every neuron arbitrarily close to one of the two teacher directions while reaching arbitrarily small population error in finite time. This occurs without explicit regularization, layerwise training, gradient clipping, or a learning-rate schedule. The dynamics additionally select an initialization-dependent allocation of the fitted function across neurons, providing a finer characterization of the representation selected by SGD. Numerical experiments reproduce the predicted alignment, fitting, and initialization-dependent allocation. Our results identify sufficient conditions under which standard SGD exhibits a strong implicit bias toward a highly structured internal representation, even when overparameterization creates infinitely many qualitatively different exact representations.

Primary area:
optimization
Keywords:
optimization dynamics, stochastic gradient descent, neural network alignment, two-layer ReLU networks, implicit bias, feature learning, overparameterization, learning theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AwTmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies two-layer, bias-free ReLU networks learning a linear teacher under Gaussian inputs. Under structured small initialization, it establishes finite-time all-neuron alignment and population fitting for joint SGD at arbitrary finite width. It characterizes minimum-mass/minimum-norm exact representations and initialization-dependent neuronwise allocations. Synthetic experiments illustrate the predicted dynamics.

Critical strengths

Preserving every neuron’s alignment through fitting at arbitrary finite width is a genuine technical contribution. The mass-weighted angular analysis controls coupled neuron interactions, yielding representation-level guarantees beyond successful fitting or early alignment alone.

Critical weaknesses

Limited scientific significance of the problem addressed. I acknowledge the minimum-mass/minimum-norm interpretation in Proposition 2 and the technical difficulty of the arbitrary-width analysis. Nevertheless, the main payoff is a finer description of how SGD parameterizes a single linear function; Theorem 2 further determines coefficient sharing among copies of the same aligned feature.

These are genuine contributions, but I find their conceptual significance limited. The work strengthens the familiar small-initialization alignment picture without establishing a comparably substantial advance in our understanding of learning. My concern is not that a stylized model requires immediate applications, but that the specific question resolved offers insufficient scientific insight to support acceptance.

Questions for authors

Clarification: Beyond the minimum-norm interpretation, what substantive insight makes the finer representation-selection question important? Please identify a consequence or general principle supported by the current results, distinguishing it from the technical difficulty and precision of the guarantees.

2 comments

We thank the reviewer for the careful review and for recognizing that preserving every neuron’s alignment through fitting is a genuine technical contribution. Below we address the reviewer’s comments and questions, and we would be happy to provide further clarification if needed.

\newline

Q1. Limited scientific significance [...].\textbf{Q1. Limited scientific significance [...].}

A1.\textbf{A1.} Thank you for acknowledging the technical contribution. Our view is that the significance lies in a basic question about joint SGD training: if many hidden representations compute the same function, which one does SGD pick? For the linear target this question can be answered precisely.

Tinaz et al. (COLT 2026) study joint training on the same linear target with two hidden neurons, and obtain stronger convergence guarantees in that setting. At width two, however, an exact fit determines the neurons' directions and effective masses, so there is nothing left to select. This changes at width m≥4m \geq 4. By Proposition 1 and Corollary 2 there are infinitely many exact fits, including ones with non-aligned neurons, and all are stationary under every SGD update.

We agree that the alignment phenomenon in Theorem 1 is consistent with the known small-initialization picture (Maennel et al., 2018; Min et al., 2024; Boursier et al., 2022). Those results concern gradient flow, either under special data assumptions or during the early phase of training. Theorem 1 establishes alignment for single-example joint SGD on Gaussian data and controls it through the fitting phase.

Theorem 2 goes further. It predicts which aligned representation is reached. Each neuron's share at fitting time is determined by its initial size and its alignment with its teacher pole. In particular the representation is dense (all neurons keep some mass), although the sparse fit with one neuron per pole has exactly the same norm. Also, the neurons are not identical copies to begin with. They start in different directions, mostly far from the teacher, become aligned during training, and their final shares still depend on where they started.

We think understanding plain joint training, without staged updates, modified algorithms or explicit regularization, brings theory closer to SGD in practice, and this is why we find analyses such as Tinaz et al.'s and ours useful. In our case the proof also explains why all neurons contribute (under our initialization assumptions). Early on, the linear teacher pulls each neuron toward its pole independently of its angle. After alignment, the neurons grow at almost the same rate, and their relative shares barely move. For more general nonlinear targets the pull depends on the direction, so a neuron that starts in a favorable direction by chance might grow faster than the others. Whether this leads to winner-takes-all selection is an open question.

\newline

Q2. Clarification [...].\textbf{Q2. Clarification [...].}

A2.\textbf{A2.} The main principle we find interesting is that initialization can determine which hidden representation SGD learns, even when many different representations give the same function and parameter norm. So neither the loss nor minimum norm can explain which particular representation SGD learns. Theorem 2, however, shows that this choice is actually predictable from initialization. Even though the neurons largely change directions during training, their final shares still depend on where they started. In some sense, the initialization leaves a trace that survives the entire training process up to fitting.

We were also curious whether this is something specific to the linear teacher, so we ran additional experiments with simple nonlinear odd teachers, such as tanh⁡\tanh. The prediction still works very well (around 5% median error with fixed-dataset SGD). While we do not have a theoretical guarantee for these cases, the experiments suggest that the importance of individual neurons may be determined early in training more generally. We will include these results in the appendix.

\newline

We thank the reviewer again for their thoughtful comments and would be happy to clarify any remaining questions.

Thank you again for you review. We have uploaded the revised pdf. Please see Appendix I for the additional experiments, including the nonlinear odd teacher experiments.

We remain happy to answer any further questions.

Reviewer_4jDeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies a bias-free two-layer ReLU network trained jointly on both layers by single-example SGD, sampled with replacement from one fixed Gaussian dataset. The target is a linear teacher x↦β θ⋆⊤xx \mapsto \beta\,\theta_\star^\top x, training starts from small initialization, and there is no explicit regularization. The paper makes four contributions.

  1. Zero-loss set (Prop. 1 / Cor. 2): Prop. 1 characterizes the exact zero set. At width 22 exact fits are rigid (up to the split of each neuron's mass between the two layers); at width 33 they stay on the teacher line but need not sit at the output-sign-selected pole; and at every width m≥4m \ge 4 there is a continuum of exact fits that are not aligned with ±θ⋆\pm\theta_\star, built from antipodal pairs.
  2. Alignment and fitting (Thm. 1): under Assumption 1, every neuron ends within O(εlog⁡(1/ε))O(\varepsilon\log(1/\varepsilon)) of its output-sign-selected pole ±θ⋆\pm\theta_\star, and the population L2L^2 error reaches O(βε)O(\beta\varepsilon) in finite time.
  3. Optimality (Prop. 2): among exact fits, aligned fits minimize effective mass, and balanced aligned fits minimize parameter norm.
  4. Allocation (Thm. 2): the normalized squared growing mode λj=(Gj0)2/∑i∈Ssj(Gi0)2\lambda_j = (G_j^0)^2/\sum_{i\in S_{s_j}}(G_i^0)^2 predicts each neuron's share of its class mass within a factor e±Cεe^{\pm C\varepsilon}, and the final point is within relative distance O(ε)O(\varepsilon) of an explicit balanced aligned representation Ψsel\Psi^{\mathrm{sel}}.
Critical strengths
  1. Well-posed question. Prop. 1 and Cor. 2 show convincingly that once m≥4m \ge 4 zero loss does not determine the representation, and the paper notes (Sec. 3.2) that function-level descriptors such as the EGOP cannot detect which one is reached. This makes "which representation does training select?" a meaningful question.
  2. Unusually complete analysis. Both layers are trained jointly by single-example SGD on a reused finite dataset, at arbitrary width and without a regularizer. Errors are controlled for each neuron relative to its current size through the whole fitting time, not just an early phase.
  3. Theorem 2 goes beyond alignment and low rank: it addresses which parameterization is selected among many that represent the same function, and the experiments track it closely.
  4. Honest positioning. The paper states plainly where Tinaz et al. is stronger: initialization, asymptotic convergence and sample efficiency
Critical weaknesses
  1. A close predecessor is missing. Chistikov, Englert & Lazić (NeurIPS 2023, "Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs") analyze arbitrary-width ReLU networks from small initialization, with a neuron-level non-asymptotic analysis of alignment followed by growth. Their setting differs (a single-ReLU teacher, gradient flow, and correlated training inputs), but leaving them out makes the account of methodological novelty incomplete.
  2. Restrictive initialization. Assumption 1 ties the reference mass scale to the target accuracy, p0=c0βε4/mp_0 = c_0\beta\varepsilon^4/\sqrt{m}, takes both layer magnitudes in [p0,54p0][\sqrt{p_0}, \tfrac54\sqrt{p_0}], requires sign-class balance m/4≤∣Ss∣≤3m/4m/4 \le |S_s| \le 3m/4, and imposes the angular condition sj⟨uj0,θ⋆⟩≥−1/2s_j\langle u_j^0,\theta_\star\rangle \ge -1/2. Corollary C.1 gives a random initialization that does not use the teacher and satisfies these conditions with high probability when d≳log⁡(m/γ)d \gtrsim \log(m/\gamma) and m≳log⁡(1/γ)m \gtrsim \log(1/\gamma), so the restriction is not knowledge of θ⋆\theta_\star. As the paper itself notes , the magnitude interval is a sufficient rather than optimized condition, independent Gaussian output weights are not covered, and the small overall scale is essential to the proof. It remains unclear whether alignment and the allocation law persist outside these conditions, for example at larger scale or with Gaussian output weights.
  3. Finite-time scope. The selection guarantee holds only up to a finite fitting time. Given the cubic flatness near aligned fits (Prop. G.1), the paper leaves open whether the same run converges to, or stays near, the selected representation. The paper acknowledges this and the initialization-scale limit, but they bound how far the main claims reach.
  4. The experiments do not test the theorems' setting. Both experiments use freshly drawn Gaussian examples (App. H; the allocation experiment reuses this protocol) and balanced initial magnitudes ∣aj(0)∣=∥wj(0)∥=p0|a_j(0)| = \|w_j(0)\| = \sqrt{p_0}, whereas Theorems 1–2 concern repeated sampling from one fixed dataset and allow unbalanced magnitudes.
Questions for authors
  1. Tinaz et al. (COLT 2026, Sec. 4) report that at widths above 22 no individual neuron aligns with ±a\pm a under random-sign Xavier initialization. Could you run an initialization sweep under the fixed-dataset sampling model that tests the allocation law directly and separates the roles of (a) the initialization scale, (b) the angular condition sj⟨uj0,θ⋆⟩≥−1/2s_j\langle u_j^0,\theta_\star\rangle \ge -1/2, and (c) the layer-magnitude distribution?
2 comments

We thank the reviewer for taking the time to review our paper and for acknowledging the unusually complete analysis. Below we address each of the reviewer’s comments and questions.

\newline

Q1. A close predecessor is missing. [...].\textbf{Q1. A close predecessor is missing. [...].}

A1.\textbf{A1.} Thank you for identifying this omission. Chistikov, Englert, and Lazić study a different problem, but their work falls within the broader area of training dynamics and implicit bias in ReLU networks. We have added the reference and explicitly discussed the similarities and differences between the two works.

\newline

Q2. Restrictive initialization. [...]. + Finite-time scope. [...].\textbf{Q2. Restrictive initialization. [...]. + Finite-time scope. [...].}

A2.\textbf{A2.} We would like to emphasize that our assumptions are sufficient conditions for our theoretical guarantees, and we do not claim they are necessary. Our new experiments show that the main behavior persists beyond several of these conditions (e.g. unbalanced magnitudes, i.i.d. Gaussian output weights, and violations of the angular condition). We will include these experiments in the appendix, together with a discussion of the role of our assumptions. We agree that understanding the dynamics beyond fitting time is an interesting open question.

\newline

Q3. The experiments do not test [...].\textbf{Q3. The experiments do not test [...].}

A3.\textbf{A3.} We have now run additional experiments using batch-size-one SGD on a fixed dataset, as in our theorems. We also tested different layer-magnitude distributions. The results are consistent with our theoretical predictions, for more details see our A4 response to reviewer 6a4m. We will include these experiments in the revised paper.

\newline

Q4. Tinaz et al. (COLT 2026, Sec. 4) [...].\textbf{Q4. Tinaz et al. (COLT 2026, Sec. 4) [...].}

A4.\textbf{A4.} We ran additional experiments on a fixed dataset, varying the initialization scale, the angular condition, and the layer magnitudes separately. Alignment and the allocation prediction still work well under unbalanced layer magnitudes, i.i.d. Gaussian output weights, and even when the angular condition is violated, as long as the initialization is sufficiently small.

What seems to matter most is the initialization scale. When it is small, the teacher dominates the early dynamics and drives the neurons toward alignment before they grow. At larger scales (20 times our original scale), this is no longer guaranteed, and individual neurons may fail to align. With Xavier initialization, both alignment and the allocation prediction break down, consistent with what Tinaz et al. observed. We will include these experiments in the revised paper.

\newline

We thank the reviewer again for their thoughtful comments and would be happy to clarify any remaining questions.

Thank you again for your helpful comments. We have now uploaded the revised pdf. Please see Appendix I for the additional experiments discussed in our response.

We remain happy to answer any further questions.

Reviewer_6a4mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies representation selection in finite-width, two-layer ReLU networks trained jointly by SGD on a noiseless linear teacher with isotropic Gaussian inputs. It proves that sufficiently small initialization leads to neuronwise alignment with the two teacher directions and characterizes an initialization-dependent allocation of effective neuron masses. The studied regime is distinct from lazy/NTK training, so existing NTK guarantees do not directly imply its neuronwise alignment and allocation results.

Critical strengths
  • This paper considers a very interesting and deep foundational question of which representation is favored by training algorithms. For tractability, this paper tackles this problem from the widely-adopted (in theoretical works) stochastic gradient descent and two-layer ReLU networks. Such focus is neat, and provides sufficient tractability for insightful results. 

  • The findings on the sharp difference between width two and every integer larger than and equals 4 are interesting. 

  • The characterizations about the aligned family and the corresponding dynamics are interesting. 

  • (Theorem 2) The specification on the initialization of the neural networks is reasonable and the choice of focused initialization makes sense. 

  • Theorem 1: Both the message -- aligns then fits -- is insightful and the analysis is solid and novel. 

  • Overall, this paper is well-written and the proofs are solid. Nevertheless, there are multiple places that need further clarifications and justification such as the analysis on the transfer from blockwise fixed-dataset SGD estimates to the deterministic phase analysis is a bit compressed, and that imported uniform concentration lemma should also be stated with fully traceable assumptions.

Critical weaknesses
  •  Gaussian inputs provide explicit ReLU-kernel formulas and simplify angular interaction estimates. I totally understand that the Gaussianity assumption has been widely adopted in the theoretical analysis of neural networks, particularly the dynamics of neural network training. However, the teacher-force identity on the early alignment mechanism may also hold for symmetric isotropic distributions. It would be great if the authors could provide clarification and discussion (or experiments) on the possibility of extending the work to slightly more general settings.  
  • It would be great if the authors could elaborate or discuss whether and how representation selection survives label noise or the final model predication accuracy. Even a controlled numerical investigation would help assess the robustness of the mechanism.
Questions for authors

none

3 comments

We thank the reviewer for taking the time to review our paper and for positively grading it. Below we address each of the reviewer’s comments and questions.

\newline

Q1. A noiseless linear teacher [...].\textbf{Q1. A noiseless linear teacher [...].}

A1.\textbf{A1.} The two-pole structure is specific to this setting. What we find potentially more general is how small initialization allows the teacher to shape the neurons early in training, and how this influences their relative contributions later. We will clarify this distinction and the limitations of our setting.

\newline

Q2. Gaussian inputs [...].\textbf{Q2. Gaussian inputs [...].}

A2.\textbf{A2.} We would like to emphasize that our assumptions, such as Gaussianity, are sufficient conditions for our theoretical guarantees, but do not appear necessary for the observed behavior. In fact, as you mention the identity driving the early alignment phase also holds for centrally symmetric isotropic distributions. Our new experiments with isotropic Laplace and uniform inputs show similar alignment and allocation behavior. We also tested anisotropic Gaussian inputs, which show how the covariance influences the neurons' early alignment. We will include these experiments in the appendix and clarify where Gaussianity is needed in the proof.

\newline

Q3. Independent mean-zero observation [...].\textbf{Q3. Independent mean-zero observation [...].}

A3.\textbf{A3.} We conducted experiments with noisy labels and found that both alignment and the initialization-dependent allocation remain robust to moderate noise at fitting. We will include these robustness experiments in the revised paper.

\newline

Q4. Appendix H describes [...].\textbf{Q4. Appendix H describes [...].}

A4.\textbf{A4.} We ran additional experiments using batch-size-one SGD on a fixed dataset, exactly as in our theoretical setting. Across widths 4, 16, and 64, all 20 runs successfully fit the target and align before fitting, while the initialization-dependent allocation remains well predicted. We also found that the prediction becomes more and more accurate with smaller learning rates and with larger datasets, consistent with our theory. We also performed a sweep on initialization scales and learning rates, confirming that alignment and fitting are robust across a range of sufficiently small initializations and learning rates. We will include these results in the revised paper.

\newline

We thank the reviewer again for their thoughtful comments and would be happy to clarify any remaining questions.

Thanks for the response. I appreciate it. It would be great if the authors can provide preliminary revision of the paper, by adding the experiments and explaination. Maybe I did not check the correct place, the submission is still the unchanged. Nevertheless, I think this is an interesting and solid paper, and my rating of this paper is still positive.

Thanks for your response, we appreciate your positive rating. We have now uploaded the revised pdf. Appendix I contains the additional experiments discussed in our responses. We slightly revised Appendix H and Section 6 to explicitly state the fixed-dataset sampling protocol. As noted in our earlier response, we observed only small differences in alignment and allocation between online and fixed-dataset SGD at the tested settings.

We remain happy to answer any further questions.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. The reviewers highlighted the following strengths:

 ~

  • Reviewer 6a4m: guarantees that "concern every neuron and maintain alignment through the fitting phase", a mass formula giving "a concrete, testable prediction beyond directional alignment", and proofs that are "overall well written and clear". After the discussion: "an interesting and solid paper".

 ~

  • Reviewer 4jDe: a well-posed question, an "unusually complete analysis" (joint training of both layers by single-example SGD on a reused dataset, at arbitrary width, without regularization), and a Theorem 2 that "goes beyond alignment and low rank".

 ~

  • Reviewer AwTm: preserving every neuron's alignment through fitting at arbitrary width is "a genuine technical contribution", yielding "representation-level guarantees beyond successful fitting or early alignment alone".

 ~

Revisions. As requested by the reviewers, we ran new experiments (Appendix I) showing that alignment and the predicted allocation remain robust well beyond our assumptions.

 ~

A general principle (AwTm). Reviewer AwTm asked for a general principle, beyond minimum norm and the precision of our guarantees. Our answer: which hidden representation SGD learns is decided early in training by the initialization, not at the end by the loss or the norm. Many exact fits share the same function and the same norm, so neither criterion can choose among them, yet Theorem 2 predicts SGD's choice before training. This principle is not tied to the linear teacher. For centrally symmetric inputs and any odd teacher, the early pull on each neuron has the same form as for a linear one, and our new experiments with tanh teachers confirm that the same prediction holds (Appendix I).

 ~

We are happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.