- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the early-stage alignment dynamics of two-layer neural networks under small random initialization. In a setting where a teacher network with Hermite-type neurons and a student network with ReLU neurons, the authors demonstrate that the student network can recover the teacher's subspace with the loss remaining nearly constant. And the paper establishes improvements in refitted prediction error and a subsequent decrease in training loss. The paper also includes results for SwiGLU networks.
Critical strengths
The paper provides theoretical guarantees for early-stage subspace alignment under small random initialization.
Critical weaknesses
The phenomenon of early alignment and the separation of timescales between directional alignment, parameter growth, and loss reduction has been investigated in existing literature [1–6]. In particular, they explained why, under small-initialization conditions, the alignment of neuron directions precedes both the growth of parameter norms and the decrease in loss, provided a rigorous analysis of this multi-stage learning dynamic within the context of teacher–student models.
The theoretical results rely on constructed teacher hermite functions, Gaussian input distributions, and population gradient descent, with conditions regarding initialization scale, network width, and learning rate. These assumptions limit the generality and practical relevance of the theoretical results. It is unclear whether the results here provide new insights beyond the existing literature.
The theoretical analysis in this paper focuses on the initial subspace alignment phase. The authors prove that the training loss will decrease thereafter; they do not prove the subsequent optimization dynamics and provide a theoretical guarantee for convergence to the teacher function.
Overall, the main results of this paper are better characterized as an incremental extension and reformulation of existing early-alignment theories within a specific setting; they do not propose, prove, or reveal the fundamentally new mechanisms underlying the learning dynamics of neural networks. The novelty and significance of the theoretical contributions presented in this paper are relatively limited.
[1] Maennel, H., Bousquet, O., & Gelly, S. (2018). Gradient Descent Quantizes ReLU Network Features. arXiv:1803.08367.
[2] Atanasov, A., Bordelon, B., & Pehlevan, C. (2022). Neural Networks as Kernel Learners: The Silent Alignment Effect. ICLR 2022.
[3] Min, H., Mallada, E., & Vidal, R. (2024). Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization. ICLR 2024.
[4] Boursier, E., & Flammarion, N. (2025). Early Alignment in Two-Layer Networks Training Is a Two-Edged Sword. Journal of Machine Learning Research.
[5] Xu, W., & Du, S. S. (2023). Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron. COLT 2023.
[6] Zhu, Z., Liu, F., & Cevher, V. (2025). How Gradient Descent Balances Features: A Dynamical Analysis for Two-Layer Neural Networks. ICLR 2025.
Questions for authors
Could the authors clarify what fundamentally new theoretical contributions this paper offers when compared to existing work?
9 comments
Continuation of our response.
Second, subspace recovery alone need not give a useful bounded readout of the actual nonlinear features. Acquired biased ridge profiles give an approximation under the same initialization/acquisition caps: , . The full prescription gives and , hence gain above 0.399 (p. 6, ll. 316 to 320). Caps use normalized-feature coordinates; equivalent raw coefficients may grow. The diagnostic head never enters training. These steps go beyond neuron alignment here; we do not claim first use of their methods.
The submission discusses [1] to [4] (p. 2, ll. 083 to 091); Table Q2 adds [5] and [6]. The compared alignment/growth analyses in [1], [3], [4] concern finite datasets; [2] uses a linear teacher on a finite training set, and [5], [6] use population teacher–student models. In [2], analytical dynamics concern linear networks, alongside nonlinear empirical evidence. Our proof uses the earlier directional mechanism; its additional claim is Table Q1's predictor-level conjunction.
Table Q2. Comparison with all six references supplied by the reviewer. Numbering is retained in the reference block below.
| Reference | Contribution we acknowledge | Distinction from Table Q1 |
|---|---|---|
| [1] Maennel, Bousquet and Gelly | Small-initialization directions concentrate near data-dependent extrema, or can freeze; biases and both layers are trained. | A direct precursor to the directional mechanism, without the present full-AGOP, every-direction and identically capped-refit conjunction. The bias-free obstruction below does not apply to their biased units. |
| [2] Atanasov, Bordelon and Pehlevan | Silent alignment, linear-network dynamics, an early-time scaling argument for homogeneous networks, a stylized predictive-probe analysis, and nonlinear empirical evidence. | The analyzed linear predictor cannot exploit our core teacher's zero-linear higher-order signal. This does not negate their nonlinear experiments or predictive evidence. |
| [3] Min, Mallada and Vidal | Early bias-free ReLU alignment into activation cones and subsequent optimization under finite binary-data separability assumptions. | Classification and cone alignment differ from population regression and every-direction predictor-AGOP recovery. The core-class obstruction below applies to the corresponding bias-free regression model. |
| [4] Boursier and Flammarion | Early alignment and a three-point example converging to a suboptimal stationary point at every width. | Their initialization assumes the magnitude of each head is at least its hidden-row norm. Our unscreened independent Gaussian initialization does not enforce this condition, which fails with high probability. Their example demonstrates that alignment need not imply global optimality. Without this condition, their three-point example converges to a global minimum (Section 6), while their unbalanced Gaussian run does not (Appendix B.2). |
| [5] Xu and Du | Alignment and convergence for a single nonlinear ReLU teacher under population GD. | The ReLU teacher has nonzero linear signal; the student is bias-free with unit output weights. On the teachers of Theorem 4.1, this two-layer student class cannot beat the zero predictor at any finite width. Their convergence guarantee is stronger on the optimization axis, for a different target/model class. |
| [6] Zhu, Liu and Cevher | Many-to-one alignment, feature balancing and convergence for orthogonal ReLU teachers. | Their ReLU teachers have linear signal, and weak recovery is an initialization assumption. On the teachers of Theorem 4.1, their bias-free two-layer student class cannot beat the zero predictor at any finite width. Our acquisition statement starts from unscreened IID Gaussian parameters. |
Earlier small-output estimates can already imply little loss change, and several references prove stronger convergence in their settings. None of the six compared statements supplies Table Q1's conjunction for this teacher/model class.
The submission discusses gradient-outer-product precedents, including AGOP progress during flat loss in recursive feature machines (p. 4, ll. 167 to 175). We add Trivedi et al. (2014), on estimating the expected gradient outer product, to this comparison; Table B5 covers BBJL's additional precedents.
Continuation of our response.
4. Recovery separated from neuron specialization, with a gated counterpart. The target influences the route to recovery. [5] and [6] align students with teacher neurons, assuming weak recovery in [6]; [1] and [4] identify zero-output correlation extrema. Our cubic proof uses axis-directed rows (p. 31, ll. 1626 to 1629). Submitted Proposition A.1 separates recovery and specialization as approximation objectives, not dynamics. Additional analysis and saved runs distinguish three routes:
- Cubic-led teachers specialize. Including the theorem's family, approximately 93 to 100% of seed-averaged importance lies within of a teacher axis at plateau end. Answer 2's fixed-size analysis gives importance concentration on teacher-axis rows on its stated event as initialization vanishes under flow.
- Interaction targets remain mixed. For and , Gaussian-input biased ReLU/leaky-ReLU gradient flow is tracked as . Under negative-rescaled-balance and growing-neuron two-line-coverage events, at , , importance concentrates on rows from and from , or from both for . The full top-two AGOP projector converges to with a strict gap. Event probability is at least 0.977 at , without a numerical resolution guarantee. In the six original ReLU interaction cohorts, 299/300 runs cross resolved before loss-window exit; endpoint importance within of a prescribed axis is at most . These are geometry/angle measurements; bounded-refit events are separate. Not every neuron is asserted mixed.
- The rotation-invariant quadratic needs no preferred axis. Target/initialization rotational invariance within and GD equivariance preclude a preferred axis in distribution. Small-initialization gradient-flow analysis shows that growing neurons retain their initial in-plane directions while AGOP converges to , provided those directions span , below the leading plateau length. The main-study joint event holds inside the plateau in 50/50 runs; each run's importance-weighted cosine with its initial direction is at least 0.999.
Teacher, model and input law determine the zero-output correlation and predicted directions. Seed-averaged modal bins lie within of predicted axis, mean or mixed angles for 19 of 21 teachers. The quadratic has no preferred axis; 's short plateau mainly fits the mean, making endpoint angles uninformative. All 21 registered tests pass, including all 12 teachers whose angle distributions are absent from the paper. Predictions were frozen before this read-only analysis, not before training. Flow limits give no quantitative claim at or empirical plateau end. Angle labels refer to the chosen teacher basis, not human-semantic interpretability; we do not claim general failure to specialize.
The proposed appendix short_polysemanticity.pdf derives the interaction/rotation-invariant flow results from zero-output correlation, with below-leading-plateau tracking and full-predictor AGOP rank under the stated coverage/spanning conditions; its supplementary note gives the complete proofs. Paper inclusion requires reviewers' agreement.
Timing provides a further quantitative check. The logarithmic release-lag form has precedents in [1] and [4]; we do not claim it. In the fixed-step population-GD sweep, is the force step (i7sf, Answer 2). For largest teacher correlation , predicted exit delay per is : 383.24 updates for , 316.14 for ; observed values are 383.241 and 316.138. The first checkpoint gaining at least 0.5 alignment is unchanged across to in 20/20 seeds per teacher. Both observations were already tested for across to in the earlier matched-scale study reported to i7sf, before the sweep's predictions were frozen. The sweep extends scales and adds : 43/44 evaluable frozen predictions pass; two additional predictions were withdrawn before training as unevaluable. Outside Theorem 4.1's hypotheses, these runs mirror its scale-independent acquisition checkpoint and release lag (eq. (42), p. 30), without claiming its constants.
Response to Reviewer q7F3
We thank the reviewer for their comments and thoughtful questions. We provide anonymous link containing two proposed appendices,
short_finite_sample.pdfandshort_polysemanticity.pdf, and a short note on other activations,short_other_activations.pdf. The supplementary notes they cite for complete proofs,supplementary_note_finite_sample.pdfandsupplementary_note_flow_directions.pdf, are in the same folder for verification; they are not proposed for the paper. Code for the experiments reported in this response is inrebuttal_code/in the same folder. Submitted-study code remains in the anonymous repository linked in the paper’s Reproducibility statement. Response tables are B1–B5, I1–I5 and Q1–Q3. Other manuscript references refer to the submitted PDF unless marked revised. We have uploaded a revised PDF; its new text is in violet, and existing section, statement, equation and appendix numbers are unchanged. Concern summaries paraphrase reviews; W/Q order is unchanged. The index at the link separates accompanying files from proposed paper additions.1. Novelty beyond early alignment
Concern (W1, W4; Q1). Prior work separates early alignment from parameter growth and loss reduction. Is this an incremental reformulation, or a distinct theoretical contribution?
Response. We respectfully disagree with the characterization as a reformulation. Our emphasis is “the joint dynamical and spectral guarantee under simultaneous training” (p. 2, ll. 105 to 106): separating representation acquisition from current-readout performance under explicit assumptions, not proposing a universal training mechanism.
1. A joint predictor-level guarantee on one unchanged trajectory. Theorem 4.1 jointly gives every-direction recovery with a gap in the full predictor's input AGOP and predictive improvement from its frozen normalized features under unchanged coefficient caps. Acquisition occurs at a pre-initialization-fixed checkpoint within a quantified high-loss interval; the original GD trajectory later reduces loss without replacing its head (Table Q1).
Table Q1. Joint statement of submitted Theorem 4.1. Assumptions include Gaussian inputs, biased ReLU/leaky-ReLU students, the specified positive-mixture teacher family, IID Gaussian initialization and the complete parameter prescription. All four conclusions hold on one event of probability at least 1−7δ/8; N<Nc are fixed before initialization.
2. A higher-order teacher regime with a model-class distinction. The infinite-dimensional positive-mixture family has no signal below degree three and generally has infinite Hermite expansions. Linear and bias-free two-layer ReLU/leaky-ReLU predictors cannot beat the zero predictor on Theorem 4.1's teachers at any finite width (Answer 2). Thus the corresponding models in [2], [3], [5] and [6] cannot yield this guarantee. Gaussian inputs, structured teachers and impractical sufficient scales remain limitations.
3. Two mathematical steps connect early alignment to the predictor level. First, GN=E[∇xfN∇xfNT] includes current signed heads and all cross-neuron terms. Outside-U gradient suppression alone does not ensure recovery of every direction inside U. The complete input gradient's second-Hermite matrix D gives UTGNU⪰DDT/2; its positive weakest-direction bound and outside-gradient control give the gap and full-subspace recovery (p. 6, ll. 306 to 314).