Classifying Subspaces with Two-Layer ReLU Networks: Align, Depart, and Realign

High-dimensional data and the representations neural networks learn from it typically concentrate near low-dimensional structures, of which a union of linear subspaces is the canonical model. We give the first analysis of feature learning in this setting, for two-layer ReLU networks trained by gradient flow from small initialization. Existing results characterize feature learning under data assumptions—Gaussian, orthogonal, or $\mu$-separable—that a union of subspaces may not satisfy. We treat the two extremes of subspace dimension, classes supported on two lines and on two hyperplanes, and establish an align, depart, and realign pattern: neurons first select directions orthogonal to one of the two subspaces, may temporarily depart from them during fitting, and ultimately return to them as the loss converges to zero. These directions define features that vanish on one of the classes, revealing an implicit bias toward representations based on distance to subspaces. We characterize how class balance and subspace separation govern early alignment, prove geometric constraints on subsequent feature evolution, and establish convergence guarantees. For hyperplanes, we additionally identify a mechanism that can make early alignment arbitrarily slow. Numerical experiments illustrate the predicted dynamics. Together, our results explain how data geometry shapes both early feature selection and the final learned representation.