- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies feature learning in the setting of a binary classification in which the two class-conditional distributions are supported on distinct linear subspaces, including the two cases of (1) two lines and (2) two hyperplanes. In this settings, the optimal weights would align with the subspace normal, hence, the paper studies whether gradient flows can learn these normals, as well as the training dynamics. The paper main claim is that in this setting, the training dynamics include 3 phases: (1) early alignment to these normals, (2) depart from the alignment, but still within the cone invariance, (3) re-align to these normals. The paper supports this claim by (1) theoretical results (page 5, Theorem 3.1 and 3.2) with assumption of class balance, subspace angle (Assumption 2) and continuous gradient flow and (2) empirical results on a small 2-layer ReLU networks (32 units, full-batch gradient descent) in Fig. 3, with noiseless (panel A) and noisy setting (panel B).
Critical strengths
Clear theoretical results (backed by numerical verification) in the binary classification on linear subspaces setting (in two sub-cases of lines and hyperplanes)
The paper identifies a specific setting that previous works on feature learning has not characterized the training dynamics: binary classification in which each class lie on a linear subspace. The paper then characterizes the optimal weight solution in this setting (four neurons supported entirely on the class normals), and ask a well-posed question of how the weight neurons evolve which respect to these optimal solutions (page 2).
The theoretical results (Theorem 3.1 (for lines) and Theorem 3.2 (for hyperplanes)) answers the above questions, which characterize the dynamics as 3 phases: early alignment, departure, and re-alignment.
The numerical verification (Fig. 6) on a small 2-layer ReLU networks (32 units, full-batch gradient descent) empirically validates the theorem by showing the early alignment (measured via "Worst normal error") drops around epoch 100, then raise again (departure phase), and then re-align again around epoch 1000. The empirical results also include the setting with off-subspace noise, and show that the theory holds until a noise threshold () (no theoretical results is shown for the noisy setting).
Critical weaknesses
The setting, while admits nice analytical results, is only applied on sub-cases (lines and hyperplanes) and synthetic data with quite restrictive assumptions
The paper setting is quite restrictive and the results (both theory and empirical) is only applied on a small set of synthetic data (no results is found on any real dataset). The paper claimed that the weakness of the current analysis on feature learning dynamics is "strong assumption on the data" (L46), but the settings and results in this paper seems to also fall into the same category. The current setting is too limited (binary classification on class that lie on linear subspace with dimension with no noise perturbation and assumed class balance), which makes the results on the training dynamics, while interesting, is unclear whether it could be applied to any realistic settings. In fact, in the empirical results in Fig. 3 the right panel, the training dynamics does not follow the pattern of "align-departure-realign" even for synthetic data with each class lie on intermediate subspace dimensions.
Please note that although works in the NTK / feature learning field is quite theoretical, the impactful works (Jacot et al Neural Tangent Kernel and Chizat et al Lazy Training on Differentiable Programming) usually have verification on real dataset.
Questions for authors
- Sensitivity to Assumption 2. The active-moment balance window becomes increasingly restrictive as the subspace angle decreases. While Appendix B.5 shows that violating this condition can produce interior attractors, how sensitive are the results to approaching or crossing the balance boundary? In particular, does violating the assumption impair classification performance, or primarily change the learned feature directions?
- Robustness to off-subspace noise. In Fig. 3 (middle), early alignment appears robust to the tested perturbations, but the normal error subsequently increases at (\sigma=0.01). Does this reflect a noise-dependent deviation from the noiseless target normals, or a qualitative change in the training dynamics? Reporting classification error alongside alignment error, with noise measured relative to the data scale, would help establish the practical significance of this departure.
- Extension beyond lines and hyperplanes. Fig. 3 (right) provides empirical evidence for intermediate subspace dimensions, although alignment varies across configurations. What are the main obstacles to extending the theoretical characterization to these settings, and what replaces the four normal directions as the target feature set? Could the authors also test whether the predicted dynamics persist on real-data representations with approximately linear class structure?
3 comments
Answering the reviewer's questions
Q1. Sensitivity to Assumption 2. In short, violating Assumption 2 changes the early features, not the final classifier.
- Approaching the boundary slows alignment: Theorem 3.1 gives T_align = O((π−θ)/Δ), where Δ is the margin by which Assumption 2 holds, so a smaller initialization is needed. Appendix E.3 (Figure 6) confirms this angle dependence.
- Crossing the boundary changes which direction neurons select early: a stable interior direction appears, and neurons align with it rather than with the normal (Appendix B.5, Propositions 10–11, Theorem B.4).
- Classification is not impaired. For two lines, these interior directions lie inside the neurons' late cones (Lemma 13), and the late-phase theorem needs no balance condition (Theorem B.3). So, provided all four groups are populated, the loss still vanishes and the neurons still return to the normals asymptotically (Corollary B.5). Empirically, this return becomes slower and less uniform as the window narrows at small angles (Appendix E.4, Figures 7–8).
Q2. Off-subspace noise. Classification accuracy is already reported alongside normal error and loss in Figure 4 (Appendix E.1, bottom row). The noise level σ is indeed the standard deviation per orthogonal direction, relative to unit variance per in-support direction. We believe the two noise levels in question behave differently.
- Moderate noise (σ = 10⁻²): a deviation, not a change of regime. Early alignment and the overall trajectory are unchanged, and classification remains essentially perfect. The rise in normal error reflects a shift of the target: with noise, a normal no longer annihilates a class exactly, so the clean normals are no longer exactly optimal.
- Large noise (σ = 10⁻¹): a qualitative change. The loss stops decreasing and accuracy drops. We conjecture that the noisy data become non-separable for the network. Our analysis of the fitting and late phases relies on the network fitting the data perfectly, with the loss tending to zero, which is what drives realignment. Without a perfect fit, the late-phase dynamics change, and the neurons need not return to the normals.
Q3. Extension beyond lines and hyperplanes. The normal alignment mechanism extends directly: a neuron orthogonal to a class subspace is inactive on all of that class, whatever its dimension. What replaces the four normals is the orthogonal subspace of a class. For a line, restricted to the data plane, or for a hyperplane, this space is one-dimensional. For intermediate dimensions it is higher-dimensional, so normal alignment no longer yields a single feature direction. Our preliminary experiments show that a second selection takes place within that space, governed by the several principal angles between the subspaces. This suggests a two-stage picture: first approach the orthogonal subspace, then select within it. The late phase also presents additional challenges: our cone invariance and four-score reduction rely on this space being a single line. We have added a new appendix (Appendix G) that develops this extension and its obstacles in detail.
Thank you for the detailed response and additional experiments! About the experiments on real dataset, I did take a look and it seems that the results hold on the CIFAR-10 neural-collapse features but it was mixed in the Extended Yale B faces. I still think the settings, the findings and the scope is currently quite limited (only applying on clean assumptions of sub-cases (lines and hyperplanes)), therefore I'll keep my original score.
Addressing the weaknesses
Thank you for the detailed questions, and for recognizing the clarity of the theoretical results and their numerical verification. We first address the concern about scope and real data, then the three questions.
Weakness: restrictive setting and absence of real data.
We respectfully disagree with this assessment, which we believe rests on a misreading of how we discuss prior work.
We do not argue that assumptions are a weakness. The paragraph at L46 describes two complementary approaches. Analyses under structured data assumptions identify the directions selected by individual neurons and follow their evolution through training. Analyses of generic data, through extrema or KKT points of auxiliary objectives, enlarge the scope but leave the geometry and dynamical relevance of these points unresolved (pp. 1–2). We present the precision of the first approach as a strength, and our work belongs to it. Our point is that the structures analyzed so far (orthogonality, separability, mean-separated mixtures) exclude data whose classes differ in the geometry of their supports rather than in their means. We study unions of subspaces because they are a canonical model of structured real data (Vidal, 2011), and so bring this line of theory closer to realistic settings.
Rejecting the paper on this basis would be unfair to how the field progresses. Feature-learning theory, and deep learning theory more broadly, have advanced by resolving structured settings before relaxing them: orthogonal inputs, linearly separable data, XOR, Gaussian mixtures, and, more broadly, deep linear networks (Saxe et al., 2014). Moreover, many influential works in this line contain no real-data experiments: Glasgow (2024) has no experiments, Boursier et al. (2022) illustrate their results on a two-point dataset, and Boursier & Flammarion (2025) on synthetic teacher–student data. The NTK and lazy-training papers the reviewer cites make claims about a training regime, which can be tested on any dataset. Our claims concern which directions neurons learn relative to the data geometry, and are not expected to hold for generic datasets with more complex geometries.
Nonetheless, we have added real-data experiments (new Appendix F). When the classes of a real dataset lie approximately on subspaces, the supports can be estimated and the predicted features checked directly.
In the camera-ready version, we will use the additional page to present these results in the main body.