acceptodds
Under review as a conference paper at ICLR 2027

Classifying Subspaces with Two-Layer ReLU Networks: Align, Depart, and Realign

Abstract

High-dimensional data and the representations neural networks learn from it typically concentrate near low-dimensional structures, of which a union of linear subspaces is the canonical model. We give the first analysis of feature learning in this setting, for two-layer ReLU networks trained by gradient flow from small initialization. Existing results characterize feature learning under data assumptions—Gaussian, orthogonal, or -separable—that a union of subspaces may not satisfy. We treat the two extremes of subspace dimension, classes supported on two lines and on two hyperplanes, and establish an align, depart, and realign pattern: neurons first select directions orthogonal to one of the two subspaces, may temporarily depart from them during fitting, and ultimately return to them as the loss converges to zero. These directions define features that vanish on one of the classes, revealing an implicit bias toward representations based on distance to subspaces. We characterize how class balance and subspace separation govern early alignment, prove geometric constraints on subsequent feature evolution, and establish convergence guarantees. For hyperplanes, we additionally identify a mechanism that can make early alignment arbitrarily slow. Numerical experiments illustrate the predicted dynamics. Together, our results explain how data geometry shapes both early feature selection and the final learned representation.

Primary area:
optimization
Keywords:
relu networks, gradient dynamics, union of subspaces, feature learning, implicit bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o3F3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies feature learning in the setting of a binary classification in which the two class-conditional distributions are supported on distinct linear subspaces, including the two cases of (1) two lines and (2) two hyperplanes. In this settings, the optimal weights would align with the subspace normal, hence, the paper studies whether gradient flows can learn these normals, as well as the training dynamics. The paper main claim is that in this setting, the training dynamics include 3 phases: (1) early alignment to these normals, (2) depart from the alignment, but still within the cone invariance, (3) re-align to these normals. The paper supports this claim by (1) theoretical results (page 5, Theorem 3.1 and 3.2) with assumption of class balance, subspace angle (Assumption 2) and continuous gradient flow and (2) empirical results on a small 2-layer ReLU networks (32 units, full-batch gradient descent) in Fig. 3, with noiseless (panel A) and noisy setting (panel B).

Critical strengths

Clear theoretical results (backed by numerical verification) in the binary classification on linear subspaces setting (in two sub-cases of lines and hyperplanes)

The paper identifies a specific setting that previous works on feature learning has not characterized the training dynamics: binary classification in which each class lie on a linear subspace. The paper then characterizes the optimal weight solution in this setting (four neurons supported entirely on the class normals), and ask a well-posed question of how the weight neurons evolve which respect to these optimal solutions (page 2).

The theoretical results (Theorem 3.1 (for lines) and Theorem 3.2 (for hyperplanes)) answers the above questions, which characterize the dynamics as 3 phases: early alignment, departure, and re-alignment.

The numerical verification (Fig. 6) on a small 2-layer ReLU networks (32 units, full-batch gradient descent) empirically validates the theorem by showing the early alignment (measured via "Worst normal error") drops around epoch 100, then raise again (departure phase), and then re-align again around epoch 1000. The empirical results also include the setting with off-subspace noise, and show that the theory holds until a noise threshold (σ<0.01\sigma < 0.01) (no theoretical results is shown for the noisy setting).

Critical weaknesses

The setting, while admits nice analytical results, is only applied on sub-cases (lines and hyperplanes) and synthetic data with quite restrictive assumptions

The paper setting is quite restrictive and the results (both theory and empirical) is only applied on a small set of synthetic data (no results is found on any real dataset). The paper claimed that the weakness of the current analysis on feature learning dynamics is "strong assumption on the data" (L46), but the settings and results in this paper seems to also fall into the same category. The current setting is too limited (binary classification on class that lie on linear subspace with dimension d−1d-1 with no noise perturbation and assumed class balance), which makes the results on the training dynamics, while interesting, is unclear whether it could be applied to any realistic settings. In fact, in the empirical results in Fig. 3 the right panel, the training dynamics does not follow the pattern of "align-departure-realign" even for synthetic data with each class lie on intermediate subspace dimensions.

Please note that although works in the NTK / feature learning field is quite theoretical, the impactful works (Jacot et al Neural Tangent Kernel and Chizat et al Lazy Training on Differentiable Programming) usually have verification on real dataset.

Questions for authors
  1. Sensitivity to Assumption 2. The active-moment balance window becomes increasingly restrictive as the subspace angle decreases. While Appendix B.5 shows that violating this condition can produce interior attractors, how sensitive are the results to approaching or crossing the balance boundary? In particular, does violating the assumption impair classification performance, or primarily change the learned feature directions?
  2. Robustness to off-subspace noise. In Fig. 3 (middle), early alignment appears robust to the tested perturbations, but the normal error subsequently increases at (\sigma=0.01). Does this reflect a noise-dependent deviation from the noiseless target normals, or a qualitative change in the training dynamics? Reporting classification error alongside alignment error, with noise measured relative to the data scale, would help establish the practical significance of this departure.
  3. Extension beyond lines and hyperplanes. Fig. 3 (right) provides empirical evidence for intermediate subspace dimensions, although alignment varies across configurations. What are the main obstacles to extending the theoretical characterization to these settings, and what replaces the four normal directions as the target feature set? Could the authors also test whether the predicted dynamics persist on real-data representations with approximately linear class structure?
3 comments
AuthorsOpenReview ↗

Addressing the weaknesses

Thank you for the detailed questions, and for recognizing the clarity of the theoretical results and their numerical verification. We first address the concern about scope and real data, then the three questions.

Weakness: restrictive setting and absence of real data.

We respectfully disagree with this assessment, which we believe rests on a misreading of how we discuss prior work.

We do not argue that assumptions are a weakness. The paragraph at L46 describes two complementary approaches. Analyses under structured data assumptions identify the directions selected by individual neurons and follow their evolution through training. Analyses of generic data, through extrema or KKT points of auxiliary objectives, enlarge the scope but leave the geometry and dynamical relevance of these points unresolved (pp. 1–2). We present the precision of the first approach as a strength, and our work belongs to it. Our point is that the structures analyzed so far (orthogonality, separability, mean-separated mixtures) exclude data whose classes differ in the geometry of their supports rather than in their means. We study unions of subspaces because they are a canonical model of structured real data (Vidal, 2011), and so bring this line of theory closer to realistic settings.

Rejecting the paper on this basis would be unfair to how the field progresses. Feature-learning theory, and deep learning theory more broadly, have advanced by resolving structured settings before relaxing them: orthogonal inputs, linearly separable data, XOR, Gaussian mixtures, and, more broadly, deep linear networks (Saxe et al., 2014). Moreover, many influential works in this line contain no real-data experiments: Glasgow (2024) has no experiments, Boursier et al. (2022) illustrate their results on a two-point dataset, and Boursier & Flammarion (2025) on synthetic teacher–student data. The NTK and lazy-training papers the reviewer cites make claims about a training regime, which can be tested on any dataset. Our claims concern which directions neurons learn relative to the data geometry, and are not expected to hold for generic datasets with more complex geometries.

Nonetheless, we have added real-data experiments (new Appendix F). When the classes of a real dataset lie approximately on subspaces, the supports can be estimated and the predicted features checked directly.

  • Extended Yale B faces, a classical union-of-subspaces dataset: each person's images lie near a low-dimensional subspace (Basri & Jacobs, 2003). Although these classes are far from spherically symmetric, neurons still align with the normals early, and the alignment becomes much sharper when each class is made isotropic within its subspace.
  • CIFAR-10 neural-collapse features. In the penultimate features of a trained ResNet-18, each class concentrates near a line (Papyan et al., 2020). Across all class pairs, neurons align with class normals, depart slightly once fitting begins, and return, consistently with what our theory predicts. With the less collapsed features of an earlier checkpoint, cat vs. dog shows the full align–depart–realign trajectory on a large scale.

In the camera-ready version, we will use the additional page to present these results in the main body.

AuthorsOpenReview ↗

Answering the reviewer's questions

Q1. Sensitivity to Assumption 2. In short, violating Assumption 2 changes the early features, not the final classifier.

  • Approaching the boundary slows alignment: Theorem 3.1 gives T_align = O((π−θ)/Δ), where Δ is the margin by which Assumption 2 holds, so a smaller initialization is needed. Appendix E.3 (Figure 6) confirms this angle dependence.
  • Crossing the boundary changes which direction neurons select early: a stable interior direction appears, and neurons align with it rather than with the normal (Appendix B.5, Propositions 10–11, Theorem B.4).
  • Classification is not impaired. For two lines, these interior directions lie inside the neurons' late cones (Lemma 13), and the late-phase theorem needs no balance condition (Theorem B.3). So, provided all four groups are populated, the loss still vanishes and the neurons still return to the normals asymptotically (Corollary B.5). Empirically, this return becomes slower and less uniform as the window narrows at small angles (Appendix E.4, Figures 7–8).

Q2. Off-subspace noise. Classification accuracy is already reported alongside normal error and loss in Figure 4 (Appendix E.1, bottom row). The noise level σ is indeed the standard deviation per orthogonal direction, relative to unit variance per in-support direction. We believe the two noise levels in question behave differently.

  • Moderate noise (σ = 10⁻²): a deviation, not a change of regime. Early alignment and the overall trajectory are unchanged, and classification remains essentially perfect. The rise in normal error reflects a shift of the target: with noise, a normal no longer annihilates a class exactly, so the clean normals are no longer exactly optimal.
  • Large noise (σ = 10⁻¹): a qualitative change. The loss stops decreasing and accuracy drops. We conjecture that the noisy data become non-separable for the network. Our analysis of the fitting and late phases relies on the network fitting the data perfectly, with the loss tending to zero, which is what drives realignment. Without a perfect fit, the late-phase dynamics change, and the neurons need not return to the normals.

Q3. Extension beyond lines and hyperplanes. The normal alignment mechanism extends directly: a neuron orthogonal to a class subspace is inactive on all of that class, whatever its dimension. What replaces the four normals is the orthogonal subspace of a class. For a line, restricted to the data plane, or for a hyperplane, this space is one-dimensional. For intermediate dimensions it is higher-dimensional, so normal alignment no longer yields a single feature direction. Our preliminary experiments show that a second selection takes place within that space, governed by the several principal angles between the subspaces. This suggests a two-stage picture: first approach the orthogonal subspace, then select within it. The late phase also presents additional challenges: our cone invariance and four-score reduction rely on this space being a single line. We have added a new appendix (Appendix G) that develops this extension and its obstacles in detail.

Thank you for the detailed response and additional experiments! About the experiments on real dataset, I did take a look and it seems that the results hold on the CIFAR-10 neural-collapse features but it was mixed in the Extended Yale B faces. I still think the settings, the findings and the scope is currently quite limited (only applying on clean assumptions of sub-cases (lines and hyperplanes)), therefore I'll keep my original score.

Reviewer_t5PHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors consider a learning problem where two distinct classes lies on two intersecting hyperplanes. The analyzed neural network is a biasless two-layer feedforward ReLU network initialized small, aka the parameter has low norm. Additionally the authors assume symmetries in the data and small, balanced initial parametrization. As learning algorithm gradient (Clarke subgradient) flow is considered. The authors focus is the dynamics of learning and they identify two distinct phases. Given these conditions gradient flow will in the first phase align the parameter to the normal vector of one of the hyperplane. Interestingly throughout in the first phase the norm of the parameter will remain small. This dynamic shifts after a while however the larger normed parameter will not leave its cone. The size of this cone is determined by the angle between the hyperplanes. The final classifier will be based on the distances from the subspaces.

I have found the paper consistent and mathematically sound and I did not find any major error in the presented theorems and proofs. The two main theorems (Theorem 3.1 and 3.2) on their own are valuable. Imo the main contributions is the interesting dynamic with concern about setups realism. Another major concern is the limited scope and limited setup, see weak points. The contributions claimed in the introduction are meaning and evolution of the learned features. I have found these a bit overestimated, especially the meaning part.

I found the literature cited in the paper sufficient. However it needs more discussion about the connection with the XOR results in (Glasgow, ICLR, 2024) and (Tsoy &Konstantinov, 2024) would increase the related work part. Regarding these I would not call the paper the "first analysis".

In my opinion the main contributions are the invariant cone and the realignment arguments.

Critical strengths
  1. The analysis of the dynamic of learning.
  2. Theorems and the proofs have interesting elements.
Critical weaknesses
  1. Limited scope: the arguments are heavily relying on the properties of the biasless shallow relu network with constrained initizalization (small, balanced) and loss.
  2. Limited setup: even thought the authors replace the conditions in previous works the presented result is also highly limited, e.g. symmetrical data alignment or the idea of data relies on two intersecting hyperplanes (noise?)
  3. Given in the paper cited results the literature needs to be improved.
  4. Figure 2 is not clearly described.
Questions for authors
  1. How stable are alignment, cone invariance and realignment if we violate the symmetry or add some noise to the data to off going from the subspaces?
3 comments

Thank you for the careful reading, for checking the theorems and proofs, and for identifying cone invariance and realignment as the main contributions. We address the four weaknesses in order; the question on symmetry violations and noise is answered under W2 and Q1.

On the "meaning" of the learned features. By meaning we refer to interpretability. Prior analyses typically identify a learned feature as the solution of an auxiliary problem, such as an extremum or KKT point of a data-dependent objective (Maennel et al., 2018; Kumar & Haupt, 2024; Boursier & Flammarion, 2025). This identifies a direction but not what it represents, a gap we point out in the introduction (p. 2). Here the learned features have a direct geometric interpretation: each is a normal to one class's subspace.

W1. Scope: architecture, initialization, and loss. Our assumptions are the standard ones of the feature-learning literature we build on. Bias-free two-layer ReLU networks with small balanced initialization are analyzed by Boursier et al. (2022), Min et al. (2024), and Glasgow (2024); Tsoy & Konstantinov (2024) use the same initialization with a smoothed ReLU. More broadly, small initialization is the standard regime for studying feature learning in deep learning theory (Maennel et al., 2018; Phuong & Lampert, 2021; Kumar & Haupt, 2024; Boursier & Flammarion, 2025), since it is where networks leave the lazy/NTK regime and learn data-dependent features (Chizat et al., 2019). These assumptions are what currently make the coupled neuron dynamics tractable. More generally, deep learning theory has typically progressed by resolving a mechanism exactly in such a regime before relaxing it, with exact solutions of deep linear networks (Saxe et al., 2014) a canonical example. Our paper is intended as that step for data supported on unions of subspaces.

W2. Limited setup: symmetry, exact subspaces, and noise.

Every theory of which features neural networks learn rests on a data model, and the field has advanced by resolving one geometry exactly before moving to the next: orthogonal inputs (Boursier et al., 2022), orthogonally separable data (Phuong & Lampert, 2021), linearly separable data (Lyu et al., 2021), XOR (Glasgow, 2024; Tsoy & Konstantinov, 2024), and Gaussian mixtures (Refinetti et al., 2021). Each of these makes assumptions; what matters is whether a new setting brings the theory closer to real data and reveals new behavior. Ours does both, so we respectfully submit that the presence of assumptions should not by itself be grounds for rejection. We would also like to emphasize the following points which supports this:

  • Our setting moves closer to real data and reveals new phenomena. Unions of subspaces are a canonical model of structured data (Vidal, 2011) that existing feature learning theory has not addressed (see our response to W3 and revised appendix for novelty and comparison with the XOR literature). In this setting we follow every neuron through the entire trajectory, and show that neurons can depart from their early features within invariant cones and then return; in prior work, early and final features either coincide throughout (Phuong & Lampert, 2021; Min et al., 2024) or diverge (Tsoy & Konstantinov, 2024; Boursier & Flammarion, 2025). We believe this to be a valuable insight to the community (see our discussion in section 7).

  • Our assumptions are milder than they may appear. For two lines, no symmetry is assumed: Theorem 3.1 covers arbitrary finite data. For hyperplanes, symmetry concerns only how each class is spread within its own hyperplane; radial laws and class probabilities are arbitrary and may differ between classes.

  • New experiments show the predicted dynamics on real data. We have added experiments on two real datasets whose classes lie approximately on subspaces: CIFAR-10 neural-collapse features (Papyan et al., 2020) and Extended Yale B faces (Basri & Jacobs, 2003). They show that the predicted features, the opposite-class normals, emerge on real data; that neurons align early, depart during fitting, and realign; and that alignment becomes sharper as the data approach the within-class symmetry our theory assumes. Details are in our response to Reviewer o3F3 and in the new Appendix F.

W3. Comparison with Glasgow (2024) and Tsoy & Konstantinov (2024). We appreciate the reviewer's comment. In the revision, we have expanded the comparison with both works in Appendix D (pp. 61–62), and in the camera-ready version we will also extend the discussion in the main text ("Connections with prior work", p. 6, lines 288–299).

On "first analysis". Neither XOR analysis covers our setting. Tsoy & Konstantinov study a smoothed approximation of ReLU, and Glasgow studies minibatch SGD, whereas we analyze the exact ReLU under gradient flow. Both assume that the classes are orthogonal (θ = π/2), a case in which our invariant cones collapse onto the normals, so the departure we characterize cannot occur. Real data, such in as the examples we included in our new experiment section, violate this condition and XOR theory does not apply. Moreover, Tsoy & Konstantinov's conditions (their Def. 4.3 and Assumption 5.1(4)) exclude data on the activation boundaries of the learned features, whereas our work allows an entire class to lie on the activation boundaries.

W4. Description of Figure 2. Figure 2 depicts the invariant cones of Eq. (1), explained in the surrounding text of Section 5 (p. 7, lines 373–377). Each shaded sector spans an opposite-class normal and the corresponding own-class ray, with angular width π/2 − θ; blue sectors correspond to positive-output neurons and red sectors to negative-output neurons. After alignment, a neuron may leave its normal but remains in its sector. In the camera-ready version we will use the additional space to expand the caption so that the figure is self-contained.

Q1. Stability under symmetry violations and off-subspace noise.

Noise. With off-subspace noise, a normal no longer annihilates a class exactly, so the exact statements cannot hold verbatim; a theory for noisy data is an important direction beyond the present scope. Empirically (Figure 3; Appendix E, Figures 4–5), the pattern persists under small noise; as the noise grows, the learned features move gradually away from the normals while classification remains accurate, and only large noise degrades both alignment and classification. The real-data experiments discussed in our response to W2 have substantial off-subspace residuals and still show alignment.

Symmetry. The line theorem already covers asymmetric data. The hyperplane experiments with 4,096 and 8,192 samples per class are not exactly spherically symmetric yet closely track the population approximation (Figure 3). Since these samples are paired with their reflections (p. 62), they preserve central symmetry. We have therefore added a controlled test (new Appendix E.5): two hyperplanes with anisotropic within-class distributions and no reflection pairing. The results show that alignment survives mild anisotropy and degrades gradually as the anisotropy grows, disappearing before fitting for strongly anisotropic classes. During fitting, most of the network's weight still moves toward the normals, although for strong anisotropy some neurons do not. Real data show the same effect: on Extended Yale B faces, whose classes are highly anisotropic, making each class isotropic within its subspace sharpens alignment substantially (Appendix F.1).

First of all I thank the authors to address most of my questions. I am partially agree with the setting and scope argument although I still think the setup is limited therefore I keep my score for now although I am on the edge to reconsider during the discussion.

Reviewer_zqe3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies two-layer ReLU networks trained by gradient flow from small initialization on data supported on (1) two lines or (2) two hyperplanes. Under suitable assumptions, for both geometries, the dynamics of neuron exhibit similar three-stage behavior

  1. neurons first align with subspace normals
  2. they may depart during fitting while remaining within assigned cones;
  3. they eventually realign as the exponential loss vanishes.
Critical strengths
  1. The paper is very well written and well organized. I really appreciate the considerable effort the authors have put into making the problem setup and results as accessible as possible through carefully designed figures and many remarks clarifying the notation.
  2. This paper provides a meaningful geometric interpretation of feature learning: neurons learn to annihilate the opposite-class support, rather than merely align with their own class. The late-time analysis of the neuron dynamics is particularly valuable.
Critical weaknesses
  1. The assumption on the two geometries are quite different. For the two-line case, the data distribution is atomic, while for the two-hyperplane case, the distribution is symmetric. Would be great if the authors can elaborate on the rationale of such assumptions. Also, I'm not certain how the analysis can be extended to subspaces with dimension in between 1 and d-1.
  2. It is not clear why and how departure from alignment can occur in the intermediate training regime. It would be interesting to see under what conditions such departures necessarily occur and whether this behavior is generic.
Questions for authors

Please see the second point of weakness.

2 comments

Thank you for the positive assessment, and in particular for highlighting the geometric interpretation of the learned features and the late-time analysis. We address the two weaknesses below.

W1. Why atomic data for lines and spherical symmetry for hyperplanes?

We discuss the reason in p. 3, lines 144–148. On two lines, every sample lies on one of four rays, so a neuron's activation pattern is described by finitely many ray masses. This allows a direct finite-sample analysis with arbitrary radii and weights on each ray, and no symmetry assumption. On two hyperplanes, a finite sample would induce a high-dimensional, sample-dependent activation arrangement with no comparable low-dimensional description, and it can introduce directional imbalances within each class. Spherical symmetry removes these effects at the population level, which makes the dynamics explicit and isolates the role of the subspace geometry. We also want to highlight that the assumption concerns only how each class is distributed within its own hyperplane: the radial law is arbitrary up to mild integrability, and it may differ between classes, as may the class probabilities. A finite-sample hyperplane theorem would additionally have to control how sampling effects interact with the dynamics. Empirically, finite samples of 4,096 and 8,192 points per class reproduce the align–depart–realign pattern and closely track the population dynamics (Figure 3; Appendix E).

Extension to intermediate dimensions. The mechanism behind normal alignment is dimension-free: a neuron orthogonal to a class subspace is inactive on that entire class, and can therefore respond to the other class without interference. What changes is how many directions achieve this. For a hyperplane, or for a line within the span of the data, the opposite-class normal space is a single direction, so suppressing the opposite class determines the feature. For intermediate dimensions this space is higher-dimensional, so a second selection takes place within it, governed by the several principal angles between the subspaces rather than a single θ. This suggests a two-stage picture: neurons first approach the opposite class's normal space, then select directions within it. Our early-phase argument gives local capture under a suitable balance condition. The main obstacle is the late phase: our planar cone invariance and four-score reduction rely on the normal space being one-dimensional, whereas in general neurons can also move within it. We have added a new appendix (Appendix G) that develops this extension and its obstacles in detail.

W2. Why can departure occur, and when is it forced?

After alignment, whether a normal attracts depends on the residual-weighted moments $R^τ_c(t) $(Eq. 12), which replace the data moments of Assumption 2. As fitting progresses, the class that is currently better fit has its residuals shrink faster, so the ratio of residual moments can leave the window that makes the normal attracting. The neuron may then leave the normal, but only within its invariant cone. Corollary A.2 (pp. 25–26) makes this precise. For a neuron in group (c, τ) at its normal, let dcτ(t)=cosθ⋅Rcτ(t)−R−cτ(t)d^τ_c(t) = cos θ · R^τ_c(t) − R^τ_{−c}(t). If dcτ<0d^τ_c < 0, the neuron remains at its normal locally. If dcτ>0d^τ_c > 0 on an interval, then no admissible Clarke continuation can remain at the normal throughout that interval, so departure is forced.

Is it generic? Not universally. Departure requires the residual condition above, and at θ = π/2 each cone collapses to its normal ray, so departure cannot occur. This is why the paper states that neurons may depart. For non-orthogonal supports, however, departure is typical in our experiments. At θ = 10°, 30°, and 60°, the residual ratios make their largest excursions around the onset of fitting, and the neurons depart from their normals at that time (Figure 5; Appendix E.2 and E.4, Figures 7–8).

Thank you for the detailed response. My positive rating remains.

Comments on the paper

1 comment

We thank the reviewers and the area chair for their time and comments, and we have responded to each review in detail. We have also added new experiments and analyses to the appendix. For convenience during the rebuttal period, the new material is confined to the appendix and marked in blue:

  • Appendix D (expanded): a more detailed comparison with prior analyses of XOR-type data (Tsoy & Konstantinov (2024), Glasgow (2024)), including a side-by-side table of assumptions and results.
  • Appendix E.5 (new): experiments that break within-class symmetry, showing how the early alignment to subspace normals is still present but degrades as the classes become more anisotropic, while the network still fits the data.
  • Appendix F (new): experiments on real data. On Extended Yale B faces, the predicted normal features emerge. On collapsed CIFAR-10 ResNet features, alignment, departure and realignment appear across class pairs (e.g., cat vs. dog).
  • Appendix G (new): a discussion of how some of the results can be extended to subspaces of intermediate dimension, and the obstacles to a full analysis.

We also added a description of the training protocol and the recorded quantities used in the experiments to Appendix E. In the camera-ready version, we will use the additional page to bring these results into the main text and to clarify the points raised in the reviews, including the expanded comparison with prior work and the interpretation of the learned features.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.