- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies two-layer, bias-free ReLU networks learning a linear teacher under Gaussian inputs. Under structured small initialization, it establishes finite-time all-neuron alignment and population fitting for joint SGD at arbitrary finite width. It characterizes minimum-mass/minimum-norm exact representations and initialization-dependent neuronwise allocations. Synthetic experiments illustrate the predicted dynamics.
Critical strengths
Preserving every neuron’s alignment through fitting at arbitrary finite width is a genuine technical contribution. The mass-weighted angular analysis controls coupled neuron interactions, yielding representation-level guarantees beyond successful fitting or early alignment alone.
Critical weaknesses
Limited scientific significance of the problem addressed. I acknowledge the minimum-mass/minimum-norm interpretation in Proposition 2 and the technical difficulty of the arbitrary-width analysis. Nevertheless, the main payoff is a finer description of how SGD parameterizes a single linear function; Theorem 2 further determines coefficient sharing among copies of the same aligned feature.
These are genuine contributions, but I find their conceptual significance limited. The work strengthens the familiar small-initialization alignment picture without establishing a comparably substantial advance in our understanding of learning. My concern is not that a stylized model requires immediate applications, but that the specific question resolved offers insufficient scientific insight to support acceptance.
Questions for authors
Clarification: Beyond the minimum-norm interpretation, what substantive insight makes the finer representation-selection question important? Please identify a consequence or general principle supported by the current results, distinguishing it from the technical difficulty and precision of the guarantees.
2 comments
Thank you again for you review. We have uploaded the revised pdf. Please see Appendix I for the additional experiments, including the nonlinear odd teacher experiments.
We remain happy to answer any further questions.
We thank the reviewer for the careful review and for recognizing that preserving every neuron’s alignment through fitting is a genuine technical contribution. Below we address the reviewer’s comments and questions, and we would be happy to provide further clarification if needed.
Q1. Limited scientific significance [...].
A1. Thank you for acknowledging the technical contribution. Our view is that the significance lies in a basic question about joint SGD training: if many hidden representations compute the same function, which one does SGD pick? For the linear target this question can be answered precisely.
Tinaz et al. (COLT 2026) study joint training on the same linear target with two hidden neurons, and obtain stronger convergence guarantees in that setting. At width two, however, an exact fit determines the neurons' directions and effective masses, so there is nothing left to select. This changes at width m≥4. By Proposition 1 and Corollary 2 there are infinitely many exact fits, including ones with non-aligned neurons, and all are stationary under every SGD update.
We agree that the alignment phenomenon in Theorem 1 is consistent with the known small-initialization picture (Maennel et al., 2018; Min et al., 2024; Boursier et al., 2022). Those results concern gradient flow, either under special data assumptions or during the early phase of training. Theorem 1 establishes alignment for single-example joint SGD on Gaussian data and controls it through the fitting phase.
Theorem 2 goes further. It predicts which aligned representation is reached. Each neuron's share at fitting time is determined by its initial size and its alignment with its teacher pole. In particular the representation is dense (all neurons keep some mass), although the sparse fit with one neuron per pole has exactly the same norm. Also, the neurons are not identical copies to begin with. They start in different directions, mostly far from the teacher, become aligned during training, and their final shares still depend on where they started.
We think understanding plain joint training, without staged updates, modified algorithms or explicit regularization, brings theory closer to SGD in practice, and this is why we find analyses such as Tinaz et al.'s and ours useful. In our case the proof also explains why all neurons contribute (under our initialization assumptions). Early on, the linear teacher pulls each neuron toward its pole independently of its angle. After alignment, the neurons grow at almost the same rate, and their relative shares barely move. For more general nonlinear targets the pull depends on the direction, so a neuron that starts in a favorable direction by chance might grow faster than the others. Whether this leads to winner-takes-all selection is an open question.
Q2. Clarification [...].
A2. The main principle we find interesting is that initialization can determine which hidden representation SGD learns, even when many different representations give the same function and parameter norm. So neither the loss nor minimum norm can explain which particular representation SGD learns. Theorem 2, however, shows that this choice is actually predictable from initialization. Even though the neurons largely change directions during training, their final shares still depend on where they started. In some sense, the initialization leaves a trace that survives the entire training process up to fitting.
We were also curious whether this is something specific to the linear teacher, so we ran additional experiments with simple nonlinear odd teachers, such as tanh. The prediction still works very well (around 5% median error with fixed-dataset SGD). While we do not have a theoretical guarantee for these cases, the experiments suggest that the importance of individual neurons may be determined early in training more generally. We will include these results in the appendix.
We thank the reviewer again for their thoughtful comments and would be happy to clarify any remaining questions.