acceptodds
Under review as a conference paper at ICLR 2027

Subspace Learning During Population-Loss Plateaus

Abstract

A nearly constant loss can conceal substantial feature learning. We prove this for two-layer ReLU and leaky-ReLU networks under simultaneous fixed-step population gradient descent from small IID Gaussian initialization. For positive mixtures of damped cubic links in Gaussian Sobolev space, loss stays near one while minimum alignment of the predictor's top- average gradient outer product (AGOP) subspace with the teacher subspace improves by at least . Identically budgeted refitting improves population MSE by more than , and the unchanged trajectory later reduces its own loss. A complementary theorem covers small additive Sobolev perturbations of unequal-weight cubic teachers using projected features. For SwiGLU, at fixed width and dimension as initialization vanishes, we prove leading-AGOP-direction alignment during a loss plateau for square-integrable teachers with a nonzero Hermite component of degree one, two, or three; unrestricted rank-one cubic refit gains hold at a specified width. These guarantees require structured teachers and restrictive parameters. An approximation result further distinguishes subspace recovery from neuron specialization: some interaction targets retain unavoidable prediction error when ridge neurons specialize along shared orthogonal axes within the teacher subspace. Population-moment experiments across 21 teacher links and 50 initializations compare alignment, refit improvement, and trained-loss reduction.

Primary area:
optimization
Keywords:
Neural Networks, Subspace Learning, Loss Minimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q7F3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the early-stage alignment dynamics of two-layer neural networks under small random initialization. In a setting where a teacher network with rr Hermite-type neurons and a student network with mm ReLU neurons, the authors demonstrate that the student network can recover the teacher's subspace with the loss remaining nearly constant. And the paper establishes improvements in refitted prediction error and a subsequent decrease in training loss. The paper also includes results for SwiGLU networks.

Critical strengths

The paper provides theoretical guarantees for early-stage subspace alignment under small random initialization.

Critical weaknesses

The phenomenon of early alignment and the separation of timescales between directional alignment, parameter growth, and loss reduction has been investigated in existing literature [1–6]. In particular, they explained why, under small-initialization conditions, the alignment of neuron directions precedes both the growth of parameter norms and the decrease in loss, provided a rigorous analysis of this multi-stage learning dynamic within the context of teacher–student models.

The theoretical results rely on constructed teacher hermite functions, Gaussian input distributions, and population gradient descent, with conditions regarding initialization scale, network width, and learning rate. These assumptions limit the generality and practical relevance of the theoretical results. It is unclear whether the results here provide new insights beyond the existing literature.

The theoretical analysis in this paper focuses on the initial subspace alignment phase. The authors prove that the training loss will decrease thereafter; they do not prove the subsequent optimization dynamics and provide a theoretical guarantee for convergence to the teacher function.

Overall, the main results of this paper are better characterized as an incremental extension and reformulation of existing early-alignment theories within a specific setting; they do not propose, prove, or reveal the fundamentally new mechanisms underlying the learning dynamics of neural networks. The novelty and significance of the theoretical contributions presented in this paper are relatively limited.

[1] Maennel, H., Bousquet, O., & Gelly, S. (2018). Gradient Descent Quantizes ReLU Network Features. arXiv:1803.08367.

[2] Atanasov, A., Bordelon, B., & Pehlevan, C. (2022). Neural Networks as Kernel Learners: The Silent Alignment Effect. ICLR 2022.

[3] Min, H., Mallada, E., & Vidal, R. (2024). Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization. ICLR 2024.

[4] Boursier, E., & Flammarion, N. (2025). Early Alignment in Two-Layer Networks Training Is a Two-Edged Sword. Journal of Machine Learning Research.

[5] Xu, W., & Du, S. S. (2023). Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron. COLT 2023.

[6] Zhu, Z., Liu, F., & Cevher, V. (2025). How Gradient Descent Balances Features: A Dynamical Analysis for Two-Layer Neural Networks. ICLR 2025.

Questions for authors

Could the authors clarify what fundamentally new theoretical contributions this paper offers when compared to existing work?

9 comments
AuthorsOpenReview ↗

Response to Reviewer q7F3

We thank the reviewer for their comments and thoughtful questions. We provide anonymous link containing two proposed appendices, short_finite_sample.pdf and short_polysemanticity.pdf, and a short note on other activations, short_other_activations.pdf. The supplementary notes they cite for complete proofs, supplementary_note_finite_sample.pdf and supplementary_note_flow_directions.pdf, are in the same folder for verification; they are not proposed for the paper. Code for the experiments reported in this response is in rebuttal_code/ in the same folder. Submitted-study code remains in the anonymous repository linked in the paper’s Reproducibility statement. Response tables are B1–B5, I1–I5 and Q1–Q3. Other manuscript references refer to the submitted PDF unless marked revised. We have uploaded a revised PDF; its new text is in violet, and existing section, statement, equation and appendix numbers are unchanged. Concern summaries paraphrase reviews; W/Q order is unchanged. The index at the link separates accompanying files from proposed paper additions.

1. Novelty beyond early alignment

Concern (W1, W4; Q1). Prior work separates early alignment from parameter growth and loss reduction. Is this an incremental reformulation, or a distinct theoretical contribution?

Response. We respectfully disagree with the characterization as a reformulation. Our emphasis is “the joint dynamical and spectral guarantee under simultaneous training” (p. 2, ll. 105 to 106): separating representation acquisition from current-readout performance under explicit assumptions, not proposing a universal training mechanism.

1. A joint predictor-level guarantee on one unchanged trajectory. Theorem 4.1 jointly gives every-direction recovery with a gap in the full predictor's input AGOP and predictive improvement from its frozen normalized features under unchanged coefficient caps. Acquisition occurs at a pre-initialization-fixed checkpoint within a quantified high-loss interval; the original GD trajectory later reduces loss without replacing its head (Table Q1).

Table Q1. Joint statement of submitted Theorem 4.1. Assumptions include Gaussian inputs, biased ReLU/leaky-ReLU students, the specified positive-mixture teacher family, IID Gaussian initialization and the complete parameter prescription. All four conclusions hold on one event of probability at least 1−7δ/81-7\delta/8; N<NcN < N_c are fixed before initialization.

QuantitySimultaneous conclusion
Trained lossThroughout 0≤n≤Nc0\le n\le N_c, 1−ϵL/16≤Ln≤1+ϵL/161-\epsilon_L/16\le L_n\le1+\epsilon_L/16, with max/min ratio at most 1+ϵL/41+\epsilon_L/4.
Full-predictor AGOPPositive rank-rr cutoff gaps at initialization and NN; Amin⁡,N≥1−ϵGA_{\min,N}\ge1-\epsilon_G, 0<ϵG≤1/40 < \epsilon_G\le1/4, and minimum-alignment gain at least 1/21/2.
Identically bounded refitR(0)−R(N)>0.399\mathcal R(0)-\mathcal R(N) > 0.399, using each checkpoint’s full feature bank, with the same normalization and fixed ℓ1,ℓ2\ell_1,\ell_2 coefficient caps.
Unchanged training trajectoryA finite N2>NcN_2 > N_c satisfies LN2≤1−2GcL_{N_2}\le1-2G_c, at least Gc>0G_c > 0 below every plateau loss.

2. A higher-order teacher regime with a model-class distinction. The infinite-dimensional positive-mixture family has no signal below degree three and generally has infinite Hermite expansions. Linear and bias-free two-layer ReLU/leaky-ReLU predictors cannot beat the zero predictor on Theorem 4.1's teachers at any finite width (Answer 2). Thus the corresponding models in [2], [3], [5] and [6] cannot yield this guarantee. Gaussian inputs, structured teachers and impractical sufficient scales remain limitations.

3. Two mathematical steps connect early alignment to the predictor level. First, GN=E[∇xfN∇xfNT]\mathsf G_N=\mathbb E[\nabla_x f_N\nabla_x f_N^T] includes current signed heads and all cross-neuron terms. Outside-UU gradient suppression alone does not ensure recovery of every direction inside UU. The complete input gradient's second-Hermite matrix DD gives UTGNU⪰DDT/2U^T\mathsf G_NU\succeq DD^T/2; its positive weakest-direction bound and outside-gradient control give the gap and full-subspace recovery (p. 6, ll. 306 to 314).

Continuation of our response.


Second, subspace recovery alone need not give a useful bounded readout of the actual nonlinear features. Acquired biased ridge profiles give an approximation under the same initialization/acquisition caps: ∥v∥2≤32/(1−α)\Vert v\Vert _2\le32/(1-\alpha), ∥v∥1≤64r/(1−α)\Vert v\Vert _1\le64\sqrt r/(1-\alpha). The full prescription gives R(0)≥1−10−4/8\mathcal R(0)\ge1-10^{-4}/8 and R(N)<(3/5+10−4/64)2\mathcal R(N) < (\sqrt{3/5}+10^{-4}/64)^2, hence gain above 0.399 (p. 6, ll. 316 to 320). Caps use normalized-feature coordinates; equivalent raw coefficients may grow. The diagnostic head never enters training. These steps go beyond neuron alignment here; we do not claim first use of their methods.

The submission discusses [1] to [4] (p. 2, ll. 083 to 091); Table Q2 adds [5] and [6]. The compared alignment/growth analyses in [1], [3], [4] concern finite datasets; [2] uses a linear teacher on a finite training set, and [5], [6] use population teacher–student models. In [2], analytical dynamics concern linear networks, alongside nonlinear empirical evidence. Our proof uses the earlier directional mechanism; its additional claim is Table Q1's predictor-level conjunction.

Table Q2. Comparison with all six references supplied by the reviewer. Numbering is retained in the reference block below.

ReferenceContribution we acknowledgeDistinction from Table Q1
[1] Maennel, Bousquet and GellySmall-initialization directions concentrate near data-dependent extrema, or can freeze; biases and both layers are trained.A direct precursor to the directional mechanism, without the present full-AGOP, every-direction and identically capped-refit conjunction. The bias-free obstruction below does not apply to their biased units.
[2] Atanasov, Bordelon and PehlevanSilent alignment, linear-network dynamics, an early-time scaling argument for homogeneous networks, a stylized predictive-probe analysis, and nonlinear empirical evidence.The analyzed linear predictor cannot exploit our core teacher's zero-linear higher-order signal. This does not negate their nonlinear experiments or predictive evidence.
[3] Min, Mallada and VidalEarly bias-free ReLU alignment into activation cones and subsequent optimization under finite binary-data separability assumptions.Classification and cone alignment differ from population regression and every-direction predictor-AGOP recovery. The core-class obstruction below applies to the corresponding bias-free regression model.
[4] Boursier and FlammarionEarly alignment and a three-point example converging to a suboptimal stationary point at every width.Their initialization assumes the magnitude of each head is at least its hidden-row norm. Our unscreened independent Gaussian initialization does not enforce this condition, which fails with high probability. Their example demonstrates that alignment need not imply global optimality. Without this condition, their three-point example converges to a global minimum (Section 6), while their unbalanced Gaussian run does not (Appendix B.2).
[5] Xu and DuAlignment and convergence for a single nonlinear ReLU teacher under population GD.The ReLU teacher has nonzero linear signal; the student is bias-free with unit output weights. On the teachers of Theorem 4.1, this two-layer student class cannot beat the zero predictor at any finite width. Their convergence guarantee is stronger on the optimization axis, for a different target/model class.
[6] Zhu, Liu and CevherMany-to-one alignment, feature balancing and convergence for orthogonal ReLU teachers.Their ReLU teachers have linear signal, and weak recovery is an initialization assumption. On the teachers of Theorem 4.1, their bias-free two-layer student class cannot beat the zero predictor at any finite width. Our acquisition statement starts from unscreened IID Gaussian parameters.

Earlier small-output estimates can already imply little loss change, and several references prove stronger convergence in their settings. None of the six compared statements supplies Table Q1's conjunction for this teacher/model class.

The submission discusses gradient-outer-product precedents, including AGOP progress during flat loss in recursive feature machines (p. 4, ll. 167 to 175). We add Trivedi et al. (2014), on estimating the expected gradient outer product, to this comparison; Table B5 covers BBJL's additional precedents.

Continuation of our response.


4. Recovery separated from neuron specialization, with a gated counterpart. The target influences the route to recovery. [5] and [6] align students with teacher neurons, assuming weak recovery in [6]; [1] and [4] identify zero-output correlation extrema. Our cubic proof uses axis-directed rows (p. 31, ll. 1626 to 1629). Submitted Proposition A.1 separates recovery and specialization as approximation objectives, not dynamics. Additional analysis and saved runs distinguish three routes:

  • Cubic-led teachers specialize. Including the theorem's family, approximately 93 to 100% of seed-averaged importance lies within 10∘10^\circ of a teacher axis at plateau end. Answer 2's fixed-size h3h_3 analysis gives importance concentration on teacher-axis rows on its stated event as initialization vanishes under flow.
  • Interaction targets remain mixed. For h2(S)Th_2(S)T and STST, Gaussian-input biased ReLU/leaky-ReLU gradient flow is tracked as s→0s\to0. Under negative-rescaled-balance and growing-neuron two-line-coverage events, at ts=alog⁡(1/s)t_s=a\log(1/s), 0<a<1/C\*0 < a < 1/C_\*, importance concentrates on rows 35.26∘35.26^\circ from u1u_1 and 54.74∘54.74^\circ from u2u_2, or 45∘45^\circ from both for STST. The full top-two AGOP projector converges to UUTUU^T with a strict gap. Event probability is at least 0.977 at d=16,m=32d=16,m=32, without a numerical 10−910^{-9} resolution guarantee. In the six original ReLU interaction cohorts, 299/300 runs cross resolved Amin⁡≥0.9A_{\min}\ge0.9 before loss-window exit; endpoint importance within 10∘10^\circ of a prescribed axis is at most 3.6508×10−73.6508\times10^{-7}. These are geometry/angle measurements; bounded-refit events are separate. Not every neuron is asserted mixed.
  • The rotation-invariant quadratic needs no preferred axis. Target/initialization rotational invariance within UU and GD equivariance preclude a preferred axis in distribution. Small-initialization gradient-flow analysis shows that growing neurons retain their initial in-plane directions while AGOP converges to UU, provided those directions span UU, below the leading plateau length. The main-study joint event holds inside the plateau in 50/50 runs; each run's importance-weighted cosine with its initial direction is at least 0.999.

Teacher, model and input law determine the zero-output correlation and predicted directions. Seed-averaged modal 5∘5^\circ bins lie within 2.5∘2.5^\circ of predicted axis, mean or mixed angles for 19 of 21 teachers. The quadratic has no preferred axis; ∣z∣|z|'s short plateau mainly fits the mean, making endpoint angles uninformative. All 21 registered tests pass, including all 12 teachers whose angle distributions are absent from the paper. Predictions were frozen before this read-only analysis, not before training. Flow limits give no quantitative claim at s=10−4s=10^{-4} or empirical plateau end. Angle labels refer to the chosen teacher basis, not human-semantic interpretability; we do not claim general failure to specialize.

The proposed appendix short_polysemanticity.pdf derives the interaction/rotation-invariant flow results from zero-output correlation, with below-leading-plateau tracking and full-predictor AGOP rank under the stated coverage/spanning conditions; its supplementary note gives the complete proofs. Paper inclusion requires reviewers' agreement.

Timing provides a further quantitative check. The logarithmic release-lag form has precedents in [1] and [4]; we do not claim it. In the fixed-step population-GD sweep, h=0.05h=0.05 is the force step (i7sf, Answer 2). For largest teacher correlation C\*C_\*, predicted exit delay per ln⁡(1/s)\ln(1/s) is 1/ln⁡(1+hC\*)1/\ln(1+hC_\*): 383.24 updates for h3h_3, 316.14 for g2/3g_{2/3}; observed values are 383.241 and 316.138. The first checkpoint gaining at least 0.5 alignment is unchanged across s=10−4s=10^{-4} to 10−1210^{-12} in 20/20 seeds per teacher. Both observations were already tested for h3h_3 across 10−410^{-4} to 10−810^{-8} in the earlier matched-scale study reported to i7sf, before the sweep's predictions were frozen. The sweep extends scales and adds g2/3g_{2/3}: 43/44 evaluable frozen predictions pass; two additional predictions were withdrawn before training as unevaluable. Outside Theorem 4.1's hypotheses, these runs mirror its scale-independent acquisition checkpoint and log⁡(1/s)\log(1/s) release lag (eq. (42), p. 30), without claiming its constants.

Reviewer_i7sfOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether neural networks can learn useful feature representations while their population loss remains nearly constant. For two-layer ReLU and leaky-ReLU networks trained by simultaneous population gradient descent from small Gaussian initialization, the authors establish that, under structured teacher and parameter assumptions, the predictor's average gradient outer product (AGOP) can recover the full teacher subspace during a high-loss plateau. They further prove that the learned features support a substantially better bounded-readout refit and that the original training trajectory subsequently reduces its loss. Complementary results characterize leading-direction alignment for SwiGLU networks and refitting improvements under additional restrictions. Population-gradient experiments across multiple synthetic teacher functions and random initializations provide numerical illustrations of these phenomena.

Critical strengths
  1. The paper provides a nontrivial theoretical result showing that full teacher-subspace recovery, substantial bounded-readout refitting improvement, and subsequent loss reduction can occur along the same simultaneous gradient-descent trajectory while the original population loss remains nearly constant. This goes beyond demonstrating early neuron alignment alone and provides a stronger connection between representation learning and predictive information.

  2. The paper distinguishes AGOP-based subspace recovery from the predictive usefulness of learned features through controlled readout refitting. The use of identical normalization and coefficient budgets makes the comparison between initialization and learned representations more meaningful. The extensions to additional teacher families and SwiGLU, together with extensive population-gradient experiments, further illustrate the relevance of this distinction.

Critical weaknesses
  1. The central phenomenon that useful representation learning can precede noticeable loss reduction has already been studied in prior work on early alignment, silent alignment, and hidden learning progress. Although this paper establishes stronger joint guarantees, they require highly structured teacher functions, Gaussian inputs, exact population gradients, extremely small initialization and step sizes, and potentially impractical width and time requirements. Consequently, the extent to which these results provide substantially new insight into general neural-network training, beyond a specialized theoretical existence guarantee, remains unclear.

  2. The experiments remain confined to synthetic population teacher–student settings, despite the broad motivation concerning neural-network learning progress. Moreover, AGOP alignment measures directional geometry rather than predictive performance, while refitting measures potential predictive capability after additional readout optimization rather than the actual predictor's performance. Although the main ReLU theorem establishes eventual loss reduction, it does not provide a practical quantitative relationship between early diagnostic improvements and subsequent training progress. The lack of finite-sample or realistic-network experiments limits the demonstrated utility and generality of these findings.

Questions for authors
  1. Previous studies have already established early alignment, silent alignment, and hidden learning progress before substantial loss reduction. Could the authors clarify which new conceptual insights follow specifically from their joint AGOP recovery, bounded-refitting, and later-loss guarantees, beyond strengthening existing results under additional assumptions? In particular, how do these guarantees compare with the closest theoretical analyses of multi-directional feature learning?

  2. The main ReLU theorem requires restrictive teacher structure and potentially extremely small initialization, learning rates, and large network widths or acquisition horizons. Could the authors quantify how these sufficient parameter requirements scale in representative settings and clarify whether the same mechanisms are expected to persist under substantially less restrictive training conditions? This would help assess the practical relevance of the theoretical guarantees.

  3. The experimental evaluation focuses on synthetic population-learning problems, with several configurations outside the theoretical guarantees and varying levels of success across teacher functions. Could the authors provide evidence from finite-sample SGD or more realistic neural architectures showing that representation improvements reliably precede loss reduction? Additionally, how should the limited joint-success rates in some appendix experiments be interpreted, and what determines when the proposed phenomenon occurs?

  4. AGOP alignment can improve even when the predictor's output remains extremely small, while refitting evaluates a separately optimized readout rather than the actual trained predictor. Could the authors clarify whether early AGOP alignment or refitting improvement can quantitatively predict the magnitude or timing of subsequent loss reduction? How stable and informative are these diagnostics under finite-sample estimation, especially when the AGOP eigenvalues are very small or unrestricted refitting requires large coefficients?

9 comments
AuthorsOpenReview ↗

Response to Reviewer i7sf

We thank the reviewer for their detailed comments and thoughtful questions. We provide anonymous link, which contains two proposed appendices, short_finite_sample.pdf and short_polysemanticity.pdf, and a short note on other activations, short_other_activations.pdf. The supplementary notes they cite for complete proofs, supplementary_note_finite_sample.pdf and supplementary_note_flow_directions.pdf, are in the same folder for verification; they are not proposed for the paper. Code for the experiments reported in this response is in rebuttal_code/ in the same folder. Submitted-study code remains in the anonymous repository linked in the paper’s Reproducibility statement. Response tables are B1–B5, I1–I5 and Q1–Q3. Other manuscript references refer to the submitted PDF unless marked revised. We have uploaded a revised PDF; its new text is in violet, and existing section, statement, equation and appendix numbers are unchanged. Concern summaries paraphrase reviews; W/Q order is unchanged. The index at the link separates accompanying files from proposed paper additions.

1. Novelty

Concern (W1; Q1). Beyond prior early-alignment and hidden-progress results, what does the joint guarantee add to multi-directional feature-learning theory under structured teachers, Gaussian inputs, and exact population gradients?

Response. Theorem 4.1 (p. 6, clauses (9) to (12)) connects representation learning and predictive information beyond early alignment. Leading-direction alignment need not recover the full subspace; geometry need not yield useful readouts; useful refits need not imply subsequent training improvement. On one trajectory, Theorem 4.1 combines near-complete recovery of every direction, bounded-refit gain greater than 0.399 during the plateau, and later trained-loss decrease. Its AGOP includes the complete predictor's signed heads and cross terms. The finite decrease implies neither convergence nor attainment of refit risk.

Loss need not reveal learned-direction counts or feature predictiveness. The closest square-loss staircase analyses recover each direction alongside its loss decrease (Ren, Nichani, Wu and Lee, 2025, Theorem 2.1; Ben Arous, Erdogdu, Vural and Wu, 2025, Theorem 1). Zero-output-correlation analysis identifies candidate plateau directions: under stated small-initialization gradient-flow conditions, subspace recovery can involve mixed neurons for two interaction teachers, or growing neurons retaining initial in-subspace directions for the additive quadratic. Axis-specialization statistics can miss what the trained predictor's AGOP records.

Table B5 compares subspace-contraction, silent-alignment, and predictive-refitting precedents. Damian et al. (2022) assume expected target Hessian rank rr, versus zero for our central odd teachers (p. 2, ll. 099 to 100). Dandi et al. (2024, Theorem 5) obtain weak Hermite-tensor-subspace recovery after a large hidden-layer step under polynomial activation and link assumptions. Both give stronger sample and width guarantees and fit readouts later. Our simultaneous-training result joins feature acquisition, trained-predictor loss and subsequent decrease for ReLU/leaky-ReLU and nonpolynomial damped-cubic links.

2. Sufficient parameter scales

Concern (W1; Q2). How restrictive are the sufficient initialization, step-size, width and acquisition-time requirements? Does the mechanism persist under less restrictive conditions?

Response. The sufficient prescription is impractical: for ReLU, r=2r=2, δ=0.1\delta=0.1, and the pure cubic teacher h3h_3, it gives width approximately 8.78×10128.78\times10^{12} and dimension 1.64×1071.64\times10^7. Width populates rare Gaussian categories; the acquisition clock includes 4R/k4R/k, where k=(1−α)λ\*/(4096d2)k=(1-\alpha)\lambda_\*/(4096d^2). Subsequent exponential bounds force extremely small initialization and steps. These sufficient, not necessary, conditions give no practical complexity or sample-size bound, as stated in the submission (p. 5, l. 269 to p. 6, l. 273). Evaluated at rank two, the prescription of Theorem E.1 (pp. 32 to 34) gives width about 7.1×1077.1\times10^7 and dimension 3282, with a different, projected refit.

Accessible-size experiments test two theorem-family targets, h3h_3 and g2/3g_{2/3}. Smallest tested widths meeting the declared 18/20 joint-success rule were 32 at rank two, 128 at rank four. At rank two, width 32, largest tested qualifying initialization scales were 0.1 for h3h_3 and 0.01 for g2/3g_{2/3}. These results do not certify theorem conditions.

Continuation of our response.


Scale dependence at fixed step. In a population sweep (r=2,d=16,m=32r=2,d=16,m=32, force step h=0.05h=0.05, 20 seeds per setting), each seed first reaches alignment gain 0.5 at the same saved checkpoint across five scales, 10−410^{-4} to 10−1210^{-12}. A force step is θ+=θ−h(m/2)∇L\theta^+=\theta-h(m/2)\nabla L, where LL is full mean squared error. Measured exit delays are 383.24 (h3h_3) and 316.14 (g2/3g_{2/3}) updates per unit ln⁡(1/s)\ln(1/s). Zero-output analysis predicts ln⁡(100)/ln⁡(1+hC\*)\ln(100)/\ln(1+hC_\*) updates per factor 100, with teacher-correlation-derived C\*C_\*: about 1,765 for h3h_3, versus 1,764 or 1,765 observed in every seed; corresponding observed g2/3g_{2/3} delays are 1,455 or 1,456. This finite-loss-window prediction is empirically supported, not an exit-controlling theorem. Its logarithmic lag mirrors submission equation (42) (p. 30), without identifying measured slopes with its constants.

Of 44 evaluable frozen predictions, 43 pass; two additional predictions were withdrawn before training as unevaluable. Predictions preceded the sweep; scorer freezing followed outcomes. Earlier scales tested the h3h_3 slope and fixed checkpoint; blind scope covers g2/3g_{2/3} and added scales.

A fixed-size additive-h3h_3 result concerns gradient flow as s→0s\to0, at fixed d>rd > r and width mm, for example d=16,m=32d=16,m=32. Conditional on negative-balance, axis-coverage and initial-refit events, it yields flat loss, full AGOP recovery and bounded-refit gain above 0.399 before the leading-order plateau end. It gives no usable finite-ss cutoff, complete fixed-step-GD counterpart, or later-loss clause. Our response to Reviewer q7F3 gives the full statement.

Matched scale comparison. A second finite-sample study fixes outcomes and comparisons in advance for s=10−4,10−6,10−8s=10^{-4},10^{-6},10^{-8}, widths 32 and 64, and 18 confirmation seeds per cell. Of 594 same-seed scale pairs, 584 reach the alignment threshold in both runs, and all 584 first reach it at the same saved checkpoint. For population h3h_3, width 64, h=0.05h=0.05, each factor-100 reduction delays exit 1,764 to 1,765 updates; median ratios of alignment time to window end are 0.20, 0.15, 0.12 at the three scales. First-study comparisons agree in 256/260 eligible pairs.

In a selected setting (m=64m=64, fresh batch 64, h=0.025,s=10−8h=0.025,s=10^{-8}, 40,000 updates), the original 0.399 endpoint event occurs in 18/18 cubic and 17/18 quadratic-product runs, versus matched population GD's 18/18 and 18/18. Our records retain complete comparisons, shared controls, and first-study seed exposure.

3. Finite-sample and architecture evidence

Concern (W2; Q3). Does representation improvement precede loss reduction under finite-sample SGD and richer architectures? How should varying joint-success rates, including outside-theory settings, be interpreted?

Response. The phenomenon occurs in the finite-sample and additional settings below; joint-success rates vary by setting and event definition. Section 2's matched SGD and population comparison uses the original 0.399 event; Table I1 lowers the refit threshold; Table I2 reports separate analogous events.

Finite-sample training. Four teachers ×\times eight arms ×\times 20 paired seeds (1002 to 1021) use d=16,r=2,m=32,s=10−4d=16,r=2,m=32,s=10^{-4}, 20,000 updates. The window ends immediately before normalized population loss leaves [0.95,1.05][0.95,1.05] or running maximum/minimum exceeds 1.05. The original same-checkpoint event requires resolved initial and current scaled AGOPs at the 10−910^{-9} relative-gap and rank-rr eigenvalue cutoff, minimum-alignment gain at least 0.5, and conservative bounded-refit gain at least 0.399: initial risk lower bound minus current feasible risk and a 10−510^{-5} guard. Feature normalization and coefficient caps stay fixed.

Continuation of our response.


Table I1. Selected conditions: retrospective refit-effect sensitivity. Six arms and three teachers are shown from the full four-teacher, eight-arm design. Only the refit threshold changes to 0.20. Each entry is “within” and “end” counts, each out of 20, in the unchanged window. The adaptive-step population arm matches the displayed population row.

Training armh3h_3Quadratic productSiLU product
Population GD20 within; 20 end20 within; 20 end20 within; 18 end
Fresh batch 1,02420 within; 17 end20 within; 18 end20 within; 18 end
Fresh batch 6418 within; 15 end17 within; 16 end20 within; 18 end
Fixed pool 65,536, full batch19 within; 18 end17 within; 17 end20 within; 18 end
Fixed pool 16,384, full batch16 within; 16 end18 within; 18 end20 within; 17 end
Batch 64 from pool 16,38418 within; 17 end19 within; 19 end20 within; 18 end

Under the original 0.399 threshold, h3h_3 qualifies within-window in 20/20 population and 19/20 fresh-batch-1,024 runs. Retrospective 0.20 sensitivity does not establish the 0.399 event. For additive ReLU, initial optimal refit risk is at most feasible normalized-MSE risk, below 0.266 in all 20 starts. Nonnegative risk makes absolute 0.399 improvement impossible here. Our records retain original counts and the complete four-teacher grid, including this boundary case. Tables B1–B3 concern separate submitted-study cohorts.

Additional settings. Table I2 uses fresh-sample SGD, held-out loss windows, and two-fold unrestricted refits for non-Gaussian and depth experiments or capped refits for MNIST. These analogous joint events differ from the first finite-sample study.

Table I2. Selected additional settings: within-window joint events. Each count is out of 10 seeds per condition.

SettingJoint counts and qualification
Uniform / Laplace inputs, affine-corrected cubic targets8/10 / 10/10; Gaussian comparison 10/10
Exploratory two-hidden-layer network, width 32 per layer, quadratic productOriginal guard: 8/10; amended initial guard, frozen before the scored runs: 9/10
Paired MNIST images, planted product / binary-label productCapped joint event: 10/10 / 10/10; all subsequently reach held-out loss at most 0.9 of its start

These paired-image MNIST tasks use eight whitened principal components per image and a declared reference subspace, not standard classification. Our records retain the full extension grid.

Depth at three scales. A follow-up with predictions frozen before its runs repeats Table I2's two-hidden-layer student, with a one-hidden-layer control of the same width, at s=0.03s=0.03, 0.010.01 and 0.0030.003 (quadratic product; 10 paired seeds; batch size proportional to ss). All three predictions pass. A tenfold smaller ss lengthens the two-layer plateau 8.7 to 22.5 times (median 24,160 to 358,540 updates), as the frozen 1/s1/s prediction required, but the one-layer plateau only 1.25 to 1.35 times (median 2,895 to 3,855), and the alignment time moves with the plateau. Under the amended guard, Table I2's joint event occurs by the plateau's last update in 9/10, 9/10 and 10/10 runs. One width does not establish a general deep-network result.

Additional gated students. In a separate population experiment, we compared SwiGLU, the submitted paper's gated student, with GeGLU and ReGLU on four rank-one teachers at d=16,m=64,ε=0.05d=16,m=64,\varepsilon=0.05, using 20 seeds per condition. In selected ReLU- and tanh-teacher conditions, the within-plateau joint event occurred in 20/20 and 19/20 SwiGLU runs, and 19/20 and 18/20 GeGLU runs, respectively. These conditions meet the frozen 16/20 criterion. The event requires leading-AGOP-direction alignment gain at least 0.5 and unrestricted-refit improvement at least 0.399 of target variance at the same resolved saved checkpoint. The plateau is the maximal initial prefix whose variance-normalized loss stays within 10−310^{-3} of its initial value. For ReGLU, both h3h_3 and h3+0.3h2h_3+0.3h_2 have 20/20 runs with endpoint alignment gain at least 0.5 and leading alignment above 0.95. These ReGLU counts establish alignment only. The results are empirical support under the stated protocol; full-network extensions of the theorem to the additional gates remain open.

Reviewer_BBJLOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a nearly flat population loss can hide substantial feature learning, and measures learning progress by four quantities: the trained population loss; AGOP alignment with the teacher subspace (Amin⁡A_{\min}, the worst-direction squared cosine; AmeanA_{\mathrm{mean}}; and AtopA_{\mathrm{top}} for SwiGLU); refit error of a fresh budgeted linear readout on the frozen normalized features (plus projected and unrestricted variants); and, in experiments, axis-snapped refits and neuron-angle histograms that test whether neurons specialize to teacher axes.

For two-layer ReLU/leaky-ReLU networks trained by population GD on all parameters from small initialization, on positive mixtures of damped cubic multi-index teachers, Theorem 4.1 shows that with high probability the same trajectory simultaneously keeps the loss near one, raises Amin⁡A_{\min} by ≥1/2\ge 1/2, improves the refit by >0.399> 0.399, and later lowers its loss. A weaker asymptotic result (Theorem 5.1) gives leading-direction AGOP alignment for SwiGLU as initialization vanishes. Population-moment experiments on 21 ReLU and 4 SwiGLU teachers illustrate the pattern, and an approximation result shows subspace recovery does not require neurons to align with teacher axes.

Critical strengths
  • Sound and Well-measured. The separation of loss, geometry, and predictive information is a useful lens; the results that I used AI to check are correct.
  • Honest and reproducible. The main text is upfront that the theorems give no practical bounds, that the experiments are run at scales the theorems do not cover, and that the SwiGLU result is only asymptotic; code, seeds, and per-run diagnostics are released.
Critical weaknesses
  • The theorems prove much less than the framing suggests. For example, Theorem 4.1 holds only at scales that cannot be instantiated (width ∼1013\sim10^{13}, dimension ∼107\sim10^{7} ).
  • The experiments do not support the headline claims. The claim is that alignment and refit improvement happen together during the plateau on the same run. The paper defines exactly this as its success criterion, but never reports how often it occurs in the main study, and the one example it showcases does not meet its own refit threshold. What the main text shows is three favorable teachers; the appendix reveals that many teachers fall short and some show the opposite pattern — loss drops without the subspace being recovered. In numbers: the showcased refit gain is 0.374 against a 0.399 threshold (Fig. 1b); where the success criterion is reported at all, it is met in 2 of 200 runs (Table 2); and about half of the 21 teachers miss the refit threshold at the plateau's end (Table 1).
Questions for authors

What meaningful insight do the results provide over the existing literature?

  • The first paragraph's motivating question decomposes into three parts, each already answered: full subspace recovery for multi-index teachers (Mousavi-Hosseini et al., ICLR 2023); representation change before the loss moves (silent alignment, Atanasov et al. 2022); and refit on frozen features showing they are already predictive (Ba et al., NeurIPS 2022).
  • Given that the theorem holds only at scales that cannot be instantiated, what should a reader take away from this paper that they could not take away from the prior work, and which results substantiates it?
  • Highlighting the meaning of this paper will be necessary for a improvement of assessment.
9 comments
AuthorsOpenReview ↗

Response to Reviewer BBJL

We thank the reviewer for the careful review and the thoughtful criticism of the paper. We provide anonymous link containing two proposed appendices, short_finite_sample.pdf and short_polysemanticity.pdf, and a short note on other activations, short_other_activations.pdf. The supplementary notes they cite for complete proofs, supplementary_note_finite_sample.pdf and supplementary_note_flow_directions.pdf, are in the same folder for verification; they are not proposed for the paper. Code for the experiments reported in this response is in rebuttal_code/ in the same folder. Submitted-study code remains in the anonymous repository linked in the paper’s Reproducibility statement. Response tables are B1–B5, I1–I5 and Q1–Q3. Other manuscript references refer to the submitted PDF unless marked revised. We have uploaded a revised PDF; its new text is in violet, and existing section, statement, equation and appendix numbers are unchanged. Concern summaries paraphrase reviews; W/Q order is unchanged. The index at the link separates accompanying files from proposed paper additions.

1. Sufficient scales and the practical takeaway

Concern (W1; question, second bullet). Sufficient width and dimension are impractically large. What do the theorem and accessible-scale experiments establish?

Response. The submission states that guarantees require structured teachers and restrictive parameters (abstract, p. 1, ll. 022 to 023), are sufficient existence bounds with no practical complexity or sample-size bound (p. 2, ll. 105 to 107; p. 6, l. 273), and are not instantiated by the experiments (p. 7, ll. 357 to 358). The complete rank-two ReLU prescription at δ=0.1\delta=0.1 gives approximately m=8.78×1012m=8.78\times10^{12}, d=1.64×107d=1.64\times10^7: sufficient, not necessary, sizes.

The guarantee is stronger than individual neuron alignment. At one acquisition checkpoint NN during the high-loss plateau, Theorem 4.1 states

A_{\min}(N)-A_{\min}(0)\ge\tfrac34-\epsilon_G\ge\tfrac12,\qquad \mathcal R(0)-\mathcal R(N) > 0.399,$$ with $0 < \epsilon_G\le1/4$, on the same event of probability at least $1-7\delta/8$. Here the AGOP belongs to the **full trained predictor**, and the refit uses identical normalized-feature coefficient constraints at initialization and acquisition. A subsequent positive loss decrease occurs along the unchanged training trajectory. These conclusions hold under the complete teacher and parameter assumptions. At $d=16,m=32,r=2$, 266/300 theorem-family runs meet the original joint criterion within their plateaus; 295/300 meet its geometry criterion with conservative refit gain $\ge0.20$ (Table B1). A separate two-teacher study finds widths 32 at rank two and 128 at rank four meet its declared 18/20 reliability bar, without sharpening the theorem’s worst-case bound. Fixed-size analysis gives plateau, every-direction recovery and refit-gain counterparts, but no later-loss clause, for additive $h_3$ at $d=16,m=32$ in the small-initialization gradient-flow limit. Its event-probability estimate has a 95% confidence lower bound of 0.9624, incorporating a simulated initial-refit tail; this is not an unconditional rigorous probability bound. Our response to q7F3 gives the initialization conditions and remaining limits. The practical takeaway: high, nearly unchanged loss can conceal every-direction acquisition and substantial bounded-readout information. Table B1 measures this conjunction at accessible sizes; B2–B3 distinguish outside-family partial learning, B4 gives separate higher-rank examples, and our response to i7sf tests finite-sample/deeper regimes. These observations distinguish empirical scope from the theorem’s existence guarantee. ### 2. Same-run joint acquisition, Figure 1 and teacher scope **Concern (W2).** Joint-success frequencies are missing; Figure 1’s refit gain is 0.374, below 0.399; submitted Table 2 reports 2/200 successes; many teachers miss endpoint thresholds or reduce loss without subspace recovery. Do the results support the claims? **Response.** The paper defines the per-run event and warns that separate ensemble means do not establish it (p. 7, l. 377 to p. 8, l. 378; p. 15, ll. 779 to 782 and 805 to 808), but omitted the main joint count: **266/300** theorem-family runs qualify inside their plateaus (Table B1).

Continuation of our response.


SiLU product is an interaction target outside the plateau theorems (p. 2, ll. 076 to 077; p. 7, ll. 365 to 366). The 2/200 comes from a separate four-teacher follow-up outside the positive-family theorem (p. 17, ll. 900 to 901), illustrating partial improvement without uniform recovery (p. 18, ll. 935 to 936), not the 21-teacher study. Submitted Table 1’s ten teacher-mean endpoint refit gains below 0.399 are all outside the family; the six family teachers’ means are 0.473–0.808 (p. 16).

Protocol. Main study: 21 normalized teachers, 50 seeds each, r=2,d=16,m=32,s=10−4r=2,d=16,m=32,s=10^{-4}, 20,000 population-training updates, jointly trained heads, weights and biases. The plateau is the maximal initial prefix with Ln∈[0.95,1.05]L_n\in[0.95,1.05] and running maximum/minimum ≤1.05\le1.05, checked every update. Refits use unprojected normalized features, no intercept, unchanged ∥v∥2≤32\Vert v\Vert _2\le32, ∥v∥1≤642\Vert v\Vert _1\le64\sqrt2. At one saved checkpoint, the original joint event requires resolved initial/current spectra, minimum-alignment gain ≥0.5\ge0.5, and conservative refit decrease ≥0.399\ge0.399: initial numerical lower risk minus current feasible upper risk, with a 10−510^{-5} margin. Every run stays in the denominator.

Table B1. Original joint event and retrospective refit sensitivity. Cells give within-plateau and endpoint counts; denominators appear in headings. “Within” requires one qualifying saved checkpoint; “endpoint” uses the last in-plateau state. Only the refit threshold varies; geometry, coefficient budgets and numerical screens are fixed.

Required conservative refit decreaseAll 21 teachers, /1,050Six family teachers, /300Other 15 teachers, /750
0.399 (original)603 within; 495 end266 within; 263 end337 within; 232 end
0.30726 within; 576 end292 within; 289 end434 within; 287 end
0.20763 within; 706 end295 within; 295 end468 within; 411 end
0.10820 within; 767 end295 within; 295 end525 within; 472 end
0.05841 within; 794 end295 within; 295 end546 within; 499 end

The 295/300 family runs meet minimum-alignment gain ≥0.5\ge0.5 and refit gain ≥0.20\ge0.20 both within the plateau and at its endpoint. This retrospective effect-size comparison retains 0.399 as the original reference. These small-width runs share the teacher family, not the sufficient parameter regime. The event is not confined to Figure 1’s three teachers (p. 7, ll. 369 to 370): 15/21 teachers have 47–50 qualifying runs out of 50 within the plateau at the retrospective refit threshold 0.20, with the alignment criterion unchanged. Our records retain the full per-teacher table.

Figure 1. The SiLU-product value 0.374 is the 50-seed mean at update 200, below 0.399; same-run counts follow.

Table B2. Selected checkpoint comparisons, SiLU-product Figure 1 cohort, all 50 seeds: original criterion and retrospective 0.20 comparison.

Reference event or sensitivityJoint qualifying runs
Original criterion at update 30035/50
Original criterion somewhere within plateau36/50
Minimum gain ≥0.5\ge0.5, refit gain ≥0.20\ge0.20, somewhere within plateau50/50
Minimum gain ≥0.5\ge0.5, refit gain ≥0.20\ge0.20, at plateau endpoint42/50

Refit improvement is nonmonotone (Figure 1 caption, p. 3, l. 154). Submitted Table 2 counts any saved in-plateau event, not persistence to the endpoint (p. 18). Figure 1’s additive cubic is in-family, with mean endpoint refit gain 0.601 (Table 1, p. 16).

Submitted Table 2. The follow-up tests raw/centered additive SiLU, additive ReLU and additive sine: d=64,m=32,r=2,s=0.01d=64,m=32,r=2,s=0.01, 50 seeds/teacher, 20,000 updates, denser diagnostics, same capped readout. All 200 initial/in-plateau spectra are resolved. A disclosed pilot selected this setting for spectral resolution, not guaranteed full recovery.

Table B3. Original same-checkpoint event and selected retrospective comparison in the separate 200-run follow-up.

Required alignment gainRequired conservative refit decreaseWithin plateauAt endpoint
Minimum ≥0.50\ge0.50 (original)0.3992/2002/200
Mean ≥0.30\ge0.300.20200/200200/200

Continuation of our response.


The original 2/200 strong-recovery event is unchanged. The retrospective row shows simultaneous partial geometric and predictive learning in all 200 runs. Mean alignment averages within-run squared principal-angle cosines, not full-subspace recovery. These are effect sizes, not redefined success; our complete teacher-level/sensitivity records retain all cases.

These outside-family illustrations show why geometry and prediction must be measured jointly: prediction can improve without every-direction recovery, while recovered geometry need not supply sufficiently expressive features under fixed readout constraints.

Submitted Table 1 and the “opposite pattern.” The theorem’s 0.399 is not calibrated for all teachers; missing it can coexist with useful learning when initial refit risk is small (p. 15, ll. 808 to 809). The five loss-before-recorded-recovery cohorts have dominant linear signal or nonzero mean, excluded by equation (6). In the follow-up cohorts, mean/linear components explain median 90–98% of loss decrease (p. 18, ll. 939 to 941). Numerical zero-output correlation peaks at the mean teacher direction for dominant-linear cases, giving a model-based explanation for concentration there; the nonzero-mean case can fit its mean during its short plateau. Among the other 16 cohorts, 794/800 runs cross resolved Amin⁡≥0.9A_{\min}\ge0.9 within the plateau: geometry, not joint-refit counts.

Separate higher-rank evidence. ReLU study: d=64,m=256,s=0.01d=64,m=256,s=0.01, ten seeds/setting, normalized-feature refit with profiled intercept, ∥v∥2≤32\Vert v\Vert _2\le32, ∥v∥1≤64r\Vert v\Vert _1\le64\sqrt r. Qualifying sequences require minimum-alignment gain ≥0.5\ge0.5 and refit improvement ≥0.1Var⁡(Y)\ge0.1\operatorname{Var}(Y) at one in-plateau checkpoint, then trained-loss reduction ≥0.1Var⁡(Y)\ge0.1\operatorname{Var}(Y) from that checkpoint. The strict plateau bounds every preceding update’s absolute variance-normalized loss change by 0.001.

Table B4. Higher-rank ReLU sequence results under the strict 0.001 window.

TeacherRankQualifying sequences
Additive quadratic210/10
Additive quadratic410/10
Additive quadratic810/10
Additive quadratic1610/10
Absolute value810/10
Gaussian bump810/10

All 60 also qualify at refit threshold 0.399Var⁡(Y)0.399\operatorname{Var}(Y): 0.399 raw MSE for unit-variance quadratic targets; absolute-value/Gaussian-bump variances are approximately 0.06660/0.01897. These separate illustrations differ in width, initialization, refit convention and horizons; they do not replace the original cohort. Four rank-16 trajectories were reused, not rerun. Observed ordering alone does not establish causality.

3. Novelty relative to the cited precedents

Concern (question and its first/third bullets). Beyond multi-index subspace learning, silent alignment and predictive frozen features, what conceptual insight and significance does this paper add?

Response. Those precedents do not already imply the joint statement here; we claim no priority for their ingredients. Our contribution connects every teacher direction in the full predictor’s AGOP, predictive acquisition under unchanged coefficient budgets, and later progress of the original predictor on one trajectory, for nonlinear targets without Hermite signal below degree three.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Official Comments by Authors

We thank all reviewers for their careful reading and constructive comments. We have uploaded a revised PDF, with all new text in violet. Existing section, statement, equation and appendix numbers are unchanged, so every pointer in our responses still applies. The main changes are:

Theory

  • Section 3 now opens the teacher family with its simplest member, the additive cubic teacher. It also shows that on the teachers of Theorem 4.1, a two-layer ReLU or leaky-ReLU network without hidden biases cannot improve on the zero predictor, at any width. Nonzero hidden biases remove this obstruction. A new proposition in Appendix C gives the proof.
  • Section 4 now states the smallest sizes that the sufficient prescriptions of Theorem 4.1 and Theorem E.1 allow at rank two. The new Appendix B.3 gives the computation.
  • Appendix A.1 adds two propositions, with proofs, on the neuron directions that the teacher favors while the output is small. For the two interaction teachers, these directions mix the teacher axes (35.26∘35.26^\circ and 45∘45^\circ from the first axis). A rotation-invariant target prefers no teacher axis in distribution. Both results complement Proposition A.1, which separates subspace recovery from neuron specialization.
  • Appendix A.1 also accounts for the order of events in the five baseline teachers outside the family of Theorem 4.1: each has a linear component or a nonzero mean, both excluded by equation (6).

Related work

  • Section 2 adds ten references and places each relative to Theorem 4.1: Xu and Du (2023), Zhu et al. (2025), Mousavi-Hosseini et al. (2023), Ba et al. (2022), Abbe et al. (2022), Glasgow (2024), Oko et al. (2024), Ren et al. (2025), Ben Arous et al. (2025), and Ren and Lee (2025).
  • Section 2 now credits Maennel et al. (2018) and Boursier and Flammarion (2025) for early directional alignment and logarithmic time scales. It also states the combination that none of the compared statements provides: a flat loss, every-direction recovery, a refit gain and a later loss decrease on one trajectory, for our teacher and model class.

Presentation and scope

  • The abstract states that the result holds under explicit sufficient conditions on width, dimension, initialization scale and step size. The Introduction now states what Theorem 4.1, the parity statement of Section 3, Theorem 5.1, Proposition A.1 and the experiments each establish.
  • The captions of Tables 1 and 2 now state the rule that defines the joint reference event.
  • Section 6 has a new paragraph, "Scope and limitations", on the setting of Theorem 4.1 and of the experiments. One Discussion paragraph on extensions moved, unchanged, to the new Appendix T.9.

Further revisions, with your agreement (anonymous link)

Our responses report new theory and experiments that are not yet in the paper. We would add any of the following that reviewers find useful, in the form they prefer:

  • Two short theory appendices, both at the anonymous link: finite-sample training (the exact update for fresh batches, fixed pools and minibatches, with tracking bounds) and polysemanticity (flow results for interaction and rotation-invariant teachers). A short note outlines how the analysis extends to GELU, SiLU, GeGLU and ReGLU students.
  • Joint-event counts and refit sensitivity (Tables B1 to B3), higher-rank results (B4), and the comparison tables B5 and Q2, as appendix tables next to the Section 2 discussion.
  • The experimental studies in our responses on finite-sample, non-Gaussian, MNIST and depth settings (Tables I1 and I2, and the depth follow-up at three scales), on width and initialization scale, and on neuron specialization.
  • The gated-student comparison (SwiGLU, GeGLU, ReGLU), the timing predictions (Table I3) and the diagnostic-estimation results (Tables I4 and I5).
  • The conditional fixed-size result for the additive cubic teacher.

We would include new theory only with the reviewers' agreement.

AuthorsOpenReview ↗

Final Comments from the Authors

We thank all the reviewers, and here we provide our final comments on the key concerns, which we tried to address. The reviews and discussion helped us narrow down these concerns and substantially improve the paper.

Contribution. To our knowledge, this is the first proof that a network trained by gradient descent from small initialization learns every direction of a multi-index subspace, with features that are already predictive at a fixed readout budget, while the loss stays flat, and that the same run then lowers its loss. It holds for ReLU-type networks, with a leading-direction version for SwiGLU, and is stated for two teacher-free diagnostics: the AGOP of the full predictor and a fixed-budget refit of the learned features.

1. Relation to early alignment, and the assumptions (all reviewers). The main concern was our assumptions and setting. Two tables (Q4, Q5) in our reply to Reviewer q7F3, also shown below, compare our work with the early-alignment results [1]-[6].

  • Table Q4 (assumptions) shows that our setting is broader in general. We train all parameters, with biases, by fixed-step GD from IID Gaussian initialization, without the gradient flow, balanced or warm-started\color{red}{\text{gradient flow, balanced or warm-started}} initialization, or fixed output weights\color{red}{\text{initialization, or fixed output weights}} that several of these works need. The rebuttal adds leading-order analyses for GELU, SiLU, GeGLU and ReGLU.
  • Table Q5 (feature statistics) shows how they treat polysemanticity. [5] and [6] measure each neuron's angle to a teacher neuron, which misses mixed (polysemantic) neurons\color{red}{\text{misses mixed (polysemantic) neurons}}; [1]-[4] do not address mixing ([1], [3], [4] have no teacher; [2] studies linear networks). The AGOP of the full predictor does not depend on how features are split across neurons and needs no teacher.

None of [1]-[6] reaches our joint conclusion, and their convergence results rely on separable data, one teacher neuron, or warm-started orthogonal teacher neurons.\color{red}{\text{None of [1]-[6] reaches our joint conclusion, and their convergence results rely on separable data, one teacher neuron, or warm-started orthogonal teacher neurons.}}

2. Finite samples (i7sf). A proposed appendix (anonymous link) derives the exact dynamics for fresh batches, fixed pools and minibatches. In the small-output phase, SGD tracks the small-output model under stated conditions; the sampling noise is centered and independent of the initialization scale. In a matched setting, SGD with fresh batches reproduces the population joint event in 18/18 and 17/18 runs; fixed pools and minibatches also show plateau-phase alignment and refit gains (Table I1), and the diagnostics are stable when estimated from finite samples (Tables I4 and I5). A full finite-sample version of the main theorem remains open.

3. Polysemanticity. A second proposed appendix shows that whether the learned neurons are monosemantic or polysemantic depends on the teacher, and predicts which. For additive cubic teachers the candidate directions are the teacher axes, and neurons specialize. For interaction teachers, we prove that in the small-initialization limit the subspace is recovered by mixed neurons while the loss stays flat and the AGOP records it; in 299 of 300 runs the AGOP recovers the subspace during the plateau.

4. Other concerns.

  • The later loss decrease (i7sf). The guaranteed decrease is a worst-case constant. In experiments the loss falls to mean MSE at most 0.20, the exit time is predicted from the learned direction (1,765 predicted, 1,764 or 1,765 observed), and adding the diagnostics improves held-out prediction of when the loss falls (concordance 0.628 to 0.823).
  • Generality (i7sf, BBJL). The effect appears beyond the theorem's family: in most of the 21 teachers, at ranks up to 16 with non-polynomial links, with non-Gaussian and MNIST inputs, and in deeper and other gated networks.
  • Significance (BBJL). The AGOP is a widely used feature statistic, from understanding how neural networks learn to steering and monitoring large models. Ours is, to our knowledge, the first guarantee for the AGOP of a network trained by gradient descent, and it shows that a flat loss can hide strong feature learning.

Further revision.

  • Reviewer q7F3 pointed to [1]-[6], the assumptions and the missing convergence proof, so we will add Tables Q2, Q4 and Q5, the polysemanticity appendix, and a limitations paragraph stating that convergence is not proved.
  • Reviewer i7sf asked for finite-sample evidence and for the additional experimental results to be included, so we will add the finite-sample appendix and all additional results, with one table giving every setting's criterion and counts.
  • Reviewer BBJL pointed out the scale of Theorem 4.1 and the missing joint rate, and asked us to highlight the paper's meaning, so we will frame Theorem 4.1 as an existence result, report the joint rates, and state up front what the paper adds.

We thank the AC for their time and care.

Tables Q4 and Q5 from our reply to Reviewer q7F3, referred to in the comment above. Violet\color{purple}{\text{Violet}}: added in the rebuttal.


Table Q4. Assumptions.

[1], [3], [4][2][5][6]\bbox[lightgray,3pt]Ours\bbox[lightgray,3pt]{\textbf{Ours}}Ours vs. [1]-[6]
Inputsfinite set; orthogonally separable in [3]finite setGaussian, populationGaussian, populationGaussian, population; finite samples;\color{purple}{\text{finite samples;}} non-Gaussian and\color{purple}{\text{non-Gaussian and}} MNIST inputs\color{purple}{\text{MNIST inputs}}broader\color{green}{\textbf{broader}} than [5], [6] (finite samples)\color{purple}{\text{(finite samples)}}; not comparable* with [1]-[4]
Teachernonelinearone ReLU neuronorthogonal, equal-norm ReLU neuronsdamped cubic links; SwiGLU: any teacher with low-degree signal; interaction\color{purple}{\text{interaction}} targets; ranks\color{purple}{\text{targets; ranks}} up to 16;\color{purple}{\text{up to 16;}} non-polynomial\color{purple}{\text{non-polynomial}} links\color{purple}{\text{links}}broader\color{green}{\textbf{broader}} than [2], [5], [6] (SwiGLU); not comparable* with [1], [3], [4]
NetworkReLU or leaky ReLUlinearReLUReLUleaky ReLU; SwiGLU; GELU, SiLU,\color{purple}{\text{GELU, SiLU,}} GeGLU, ReGLU\color{purple}{\text{GeGLU, ReGLU}} (leading order)\color{purple}{\text{(leading order)}}broader\color{green}{\textbf{broader}} than [1], [3]-[6]
Trained parametersall; no biases in [3]all, no biaseshidden only, no biaseshidden only, no biasesall, with biasesbroader\color{green}{\textbf{broader}} than [2], [3], [5], [6]; same as [1], [4]
Traininggradient flowgradient flowGDGDfixed-step GD; online SGD\color{purple}{\text{online SGD}}broader\color{green}{\textbf{broader}} than [1]-[4]; same as [5], [6]
Initializationsmall; balanced or output-dominatedsmallσ=O(n−4226)\sigma = O(n^{-4226}) at width nnsmall, warm startsmall IID Gaussian, signed heads; scales up to 0.1\color{purple}{\text{scales up to 0.1}}broader\color{green}{\textbf{broader}} than [1], [3], [4], [6]; same as [2], [5]
Widthanyanyanyconstant, balanced countssufficient lower bound; any for SwiGLU; widths from 32;\color{purple}{\text{widths from 32;}} conditional\color{purple}{\text{conditional}} fixed-width result\color{purple}{\text{fixed-width result}}not comparable*; same for SwiGLU

*Not like for like: [1]-[4] fix one finite dataset, and [1], [3], [4] have no teacher. Our width bound ensures suitable starting neurons for every teacher direction, which [6] assumes (a warm start) and which [1]-[5], with at most one teacher direction, do not need.

  • Our teachers have no linear or quadratic Hermite component, unlike the ReLU teachers of [5] and [6], whose low-order signal drives alignment.

Table Q5. Feature statistics.

StatisticComputed fromComputable without the teacherMixed (polysemantic) neurons
[1], [3], [4]neuron directionseach neuronyesnot addressed\color{red}{\text{not addressed}}
[2]tangent kernelparameter gradientsyeslinear network\color{red}{\text{linear network}}
[5], [6]angles to teacher neuronseach neuronno\color{red}{\text{no}}no\color{red}{\text{no}}: each aligns with a teacher
\bbox[lightgray,3pt]Ours\bbox[lightgray,3pt]{\textbf{Ours}}AGOP; fixed-budget refit (prediction)input gradients of the predictoryesyes; proved for\color{purple}{\text{proved for}} interaction targets\color{purple}{\text{interaction targets}}
  • Neuron-level statistics depend on how features are split across neurons; the AGOP does not. For interaction targets, axis-aligned networks cannot beat MSE 1/4 on one such target (Proposition A.1), so mixed neurons are needed.

References

  • [1] Maennel, H., Bousquet, O., and Gelly, S. (2018). Gradient descent quantizes ReLU network features. arXiv:1803.08367.
  • [2] Atanasov, A., Bordelon, B., and Pehlevan, C. (2022). Neural networks as kernel learners: The silent alignment effect. ICLR.
  • [3] Min, H., Mallada, E., and Vidal, R. (2024). Early neuron alignment in two-layer ReLU networks with small initialization. ICLR.
  • [4] Boursier, E., and Flammarion, N. (2025). Early alignment in two-layer networks training is a two-edged sword. Journal of Machine Learning Research.
  • [5] Xu, W., and Du, S. S. (2023). Over-parameterization exponentially slows down gradient descent for learning a single neuron. COLT.
  • [6] Zhu, Z., Liu, F., and Cevher, V. (2025). How gradient descent balances features: A dynamical analysis for two-layer neural networks. ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.