A classical result for linear networks trained with mean-squared-error loss is that gradient flow learns the singular modes of the data sequentially, in order of importance. The precise mechanics of such sequential emergence under cross-entropy loss remain largely unknown. We study this question in a minimal nonconvex setting: a two-layer linear network with orthogonal inputs and step-imbalanced classes, equivalent to the unconstrained feature model used in neural collapse analyses. First, we characterize the regularization path in spectral coordinates, showing sequential mode emergence with activation thresholds that, unlike in MSE, need not coincide with the corresponding data singular values. As regularization vanishes, active singular values diverge, while their normalized values converge at a logarithmic rate and can approach their limits from either side. Second, for gradient flow under suitable spectral initialization, we prove majority-first emergence and show that the majority-to-minority mode ratio converges to that of the regularization-path limit. Our analysis relies on a novel imbalance-adapted Hadamard basis in which we show that softmax preserves a diagonal-plus-rank-one structure.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.Reviewer_44zxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
This paper conducts a theoretical analysis of a two-layer linear network using the Unconstrained Features Model (UFM) under step-imbalanced cross-entropy classification. Building on prior work that introduced the UFM for balanced and imbalanced neural collapse (Mixon et al., 2020; Fang et al., 2021; Thrampoulidis et al., 2022; Hong & Ling, 2023) and spectral dynamics in linear networks (Saxe et al. 2014; Garrod et al. 2025; Zhao & Thrampoulidis 2025). Using a Hadamard reduction, they analyze the regularization path and spectrally initialized gradient flow. They derive the following phenomena: sequential mode emergence, directional overshoot, logarithmic growth, and convergence bounds. Rhetorically, this work is motivated both by the linear-network spectral dynamics literature (Saxe et al. 2014; Garrod et al. 2025) and by the neural collapse literature (Papyan et al. 2020) from which the UFM framework originated (Mixon et al. 2020; Fang et al. 2021). However, the neural collapse literature relies heavily on the premise that the empirical phenomenon is first observed in practical deep networks and the UFM serves to conceptually explain it: see Mixon et al. (2020), Fang et al. (2021), Han et al. (2021), and Thrampoulidis et al. (2022). Here, the authors validate their predictions only via numerical simulations of the stylized UFM itself, without testing whether these dynamical phenomena actually occur in practice.
Critical strengths
The paper conducts a rigorous analysis under their 2-layer theoretical model. They very clearly present their findings of the following theoretical predictions: sequential mode emergence, directional overshoot, logarithmic growth, and convergence bounds.
Critical weaknesses
The analysis is highly stylized. It draws inspiration from the neural collapse literature (which validates their analyzed phenomena on experiments using real datasets and AI architectures) to justify this new direction. However, unlike the neural collapse literature, the authors do not provide any experiments on real datasets or AI architectures to show that their predicted phenomena actually occur in practice.
Questions for authors
To keep the review process concise:
Why might we believe that their predicted behavior would occur in real AI training (on real architectures and datasets) rather than just this stylized model? Might the authors provide 1 or 2 experiments on commonly acknowledged datasets and architectures similar to Papyan et al. (2020)?
The existence of such an experiment would be the biggest factor for re-assessment for me.
Thank you for your thoughtful feedback. We do agree the paper benefits from such experiments and comparison, and have thus added experiments on nonlinear architectures and real data (MNIST, CIFAR-10) in the revision.
The detailed description and presentation of the results is in App. H.3, and we have included one of the figures in the main body (Fig. 4) for training 4 classes of MNIST with an imbalance ratio R=10 on a ReLU MLP with two hidden layers. The figure tracks the projections of the class-mean logits onto the directions of the SEL (centered one-hot) matrix along (i) the regularization strength λ and (ii) iterations of SGD, for CE (top) and MSE (bottom).
For (i), we regularize the loss with a ridge penalty on all parameters and trace the solutions over a grid of λ by continuation, i.e., we train with GD for a fixed number of iterations at each λ and use the solution as initialization for the next. For (ii), we run unregularized CE/MSE training well beyond the terminal phase of training (i.e., well after zero training error).
We thus produce the full 2x2 "landscape" that our theory brings together, CE vs. MSE and GF vs. RP, which are normally studied separately in prior work we are aware of.
Qualitatively, we see agreement with the theory developed for our minimal model (the two-layer linear network with whitened inputs): along both paths the majority, maj-min, and minority modes emerge sequentially, and under CE their normalized versions slowly approach the SEL limit from different sides, unlike the convergence to finite targets under MSE. For CIFAR-10, MSE behaves as for our theory model and MNIST, and under CE the minority mode again emerges last, but the faster emergence of majority versus the maj-min mode is now less visible, while also, at the end of training the majority lies below its SEL value and the maj-min above. We believe this is consistent with the nature of the theoretical model treating the penultimate features as freely optimizable (thus, the whitened inputs in our theory) and can thus be expected to describe the trained network more accurately for simpler data such as MNIST than for more complex data such as CIFAR-10. In spirit of the long line of theoretical work on linear networks, our results are meant as stylized predictions that provide intuition for practical training.
I feel the authors made a good-faith effort to address my concerns within realistic time constraints of the ICLR reviewing period. Thus, I raise my score to a 3. I do not raise it to a 4 because I still find the experiments rather over simplistic (such as using smaller datasets like CIFAR10 and MNIST and further reducing them to 4-class problems).
This paper studies the mechanics of feature learning for two-layer linear networks trained with cross-entropy losses and imbalanced data classes. The work shows that the sequential learning of singular data modes observed under MSE-based Gradient Flow (GF) partially transfers to this setting. In parallel, the authors refine existing results characterizing feature emergence along the Regularization Path (RP). In particular, the following results are given for the minimal nonconvex setting of an unconstrained feature model (UFM)
First, a technical result is established that a Sylvester-Hadamard parametrization of the centered label matrix's singular bases induces the softmax residual to preserve a diagonal-plus-rank-one structure of its argument. This result is an extension of Garrod et. al (2025)'s construction to imbalanced classes.
Second, it is shown that the minimizers of UFM-based λ-regularized problems belong to the diagonal-plus-rank-one spectral family preserved by the softmax, for any λ>0. Furthermore, it is shown that characterizing these minimizers amounts to solving a 4D convex optimization problem. Finally, the authors characterize the minimizers as functions of λ, showing distinct regimes of emergence for the majority/minority/majority-minority class singular modes along the RP, and establish that as λ→0 the singular value profile of the optimal logit matrix converges to that of the centered label matrix at rate O(1/log(1/λ)).
Third, it is shown that GF (λ=0) initialized at a point in the spectral subspace preserved by the softmax remains in that subspace. This yields a characterization of logit parameter dynamics under additionally balanced initialization. Finally, a majority-mode-first emergence is established under the further assumption of ϵ-small, positive initialization, along with some structural connections between the GF and the RP.
Finally, experiments are provided in support of the theory
Critical strengths
Feature learning order and its dynamics under cross-entropy losses and imbalanced classes is an important problem, and this paper rigorously addresses it to the extent allowed by existing mathematical tools
The paper is well structured and well written, and the arguments are mostly easy to follow thanks to per-section discussions contextualizing the technical arguments (though indeed the paper is dense!)
The results are related appropriately to existing literature, to the best of my knowledge
Experiments align well with the theory and cover most relevant settings
Critical weaknesses
Certain paper framing choices and discussions are vague
Questions for authors
Questions / concerns:
The RP results are very abruptly introduced in the paper, as the introduction emphasizes only singular mode emergence during training (GF). Since a large section of the paper is dedicated to the study of RP for UFM, the authors should provide more context/motivation for introducing the RP study alongside GF dynamics.
In addition to Figure 9, can the authors provide a similar experiment using a centered, ϵ2-variance Gaussian initialization of the factor matrices for several values of ϵ? It would be useful to know how much behavior departs from theory under practical initialization scenarios.
Can the authors discuss whether Thm. 6 can be extended to an imbalanced style of initialization analogous to that of [1] for the MSE case? It is not clear to me from the discussion in App. B.2.
Minor comments:
Line 262: Equation (4) has no "residual matrix Q" in its LHS
The authors refer to singular values as having direction in several places, which is inaccurate since they're scalars: line 312 "[...] three singular values diverge, but their direction converges [...]", line 327 "singular values [...] converge directionally". It would be clearer to consistently use the term "singular value profile" for directional convergence, since the authors already use it inside Thm. 3.
[1] Varre, et al. "On the spectral bias of two-layer linear networks." Advances in neural information processing systems 36 (2023): 64380-64414.
Thank you for your careful reading and positive feedback.
RP abruptly introduced
Thank you, this is indeed a fair point. We have added a paragraph to the Introduction (in blue) motivating the RP as the second standard view, alongside GF, on which direction CE training selects, and noting that it also carries its own λ-indexed notion of mode emergence. We also now state in the contributions that the approximate softmax diagonalization (Thm. 1) underlies both the RP and GF analyses.
Thm 6 under imbalanced init
Thank you for the question and for the reference, which we now cite. We agree that App. B.2 could cause confusion and have expanded it in the revision (in blue). Recall that the layer imbalance W⊤W−HH⊤, which is conserved along GF, in our spectral coordinates reduces to scalars c (maj-min mode) and c±,i (majority/minority modes). In short:
(i) Imbalance at the initialization scale. If the imbalance is of the same order as the initialization scale ε, the proof of Thm. 6 goes through essentially verbatim, using 2a≤4a2+c2≤2a+∣c∣ and Gronwall.
(ii) Larger but isotropic imbalance. More generally, what matters is not the size of the imbalance but how it compares across modes. If it is isotropic, i.e., ∣c±,i∣=∣c∣ for all modes, as in [1] (explicit in their Lemma 3.2, where the conserved imbalance is a multiple of the identity), we expect Thm. 6 to hold even when ∣c∣≫ε. To prove this, one can use the now standard change of variables to hyperbolic coordinates of Saxe et al. (2014, App. A), a=2∣c∣sinh(2s), in which the majority/minority dynamics become exactly s˙=b. Under isotropic imbalance, this is the hyperbolic-entropy mirror flow of [1] (their Thm. 3.2(a)/Lemma C.13), with the CE loss in place of the square loss. The only subtlety is the maj-min mode where as in the balanced case, its singular direction rotates, so its singular value does not obey a closed scalar equation (see the non-diagonal (α,a)-block of the preconditioner in Sec. 5.3). However, as in our proof of Thm. 6, a Cauchy–Schwarz bound suffices for emergence. Given the already dense results, we leave the details beyond the scope of this paper.
(iii) Strongly anisotropic imbalance. Under strongly anisotropic imbalance (e.g., ∣c−∣≫∣c+∣+a+(0)), the order may change and Thm. 6 may fail. This is what the original App. B.2 intended and we have now revised it to state its scope precisely.
Fig. 9 for Gaussian initialization
Thank you for the nice suggestion. We have done this and added the results as Fig. 3 (main text) and Fig. 12 in the revision, with App. H.2.2 describing the setup and results.
In short, we draw iid Gaussian W,H with standard deviation εinit and, for each draw, construct a matched Hadamard initialization with the same total energy and the same normalized initial logit spectrum. We then compare GD from the Gaussian initialization with the solution of our reduced ODE (Eq. 11) from the matched Hadamard one, tracking the sorted singular values of the centered logit matrix and their normalized versions, for two imbalance ratios R=5 and R=20 and εinit∈{10−3,10−2,10−1}.
For small εinit=10−3, the agreement between the Gaussian and Hadamard trajectories is good, with emergence occurring somewhat later under Gaussian initialization (since initial logits are smaller). For R=20, a gap appears at late times in the two largest singular values, while the smallest ones agree. As εinit increases, this late-time gap for R=20 becomes larger. For R=5, the late-time agreement remains good, and the main difference appears around emergence. In all cases, the sequential emergence of the singular values is qualitatively consistent between the two initializations.
Minor comments
Thanks again for your careful reading and for pointing these out. They have been fixed in the revision.
Thank you for your reply and addressing the comments
I acknowledge the authors' response. I have read the other reviewers' feedback and the authors' respective responses. My original evaluation remains --- this is a good paper with a substantial contribution.
The paper investigates how cross-entropy learns different spectral modes in a simplified linear network with step-imbalanced classes. The paper first introduces a tool to show that the softmax residual preserves a diagonal-plus-rank-one structure. This enables a low-dimensional analysis of both the regularization path and gradient-flow dynamics. Along the regularization path, the authors characterize sequential activation of majority, majority–minority, and minority modes, together with their directional convergence as regularization vanishes. Under structured initialization, they further prove majority-first emergence during gradient flow, and convergence of the majority/minority mode ratio to its regularization-path limit
Critical strengths
The paper develops an interesting diagonal-plus-rank-one construction that makes the coupled softmax dynamics tractable.
The theoretical contribution is thorough, the paper not only derives regularized optima, but also studies sequential mode emergence, directional convergence rates, and gradient-flow dynamics, etc.
The paper clearly contrasts CE with MSE. It shows that although CE admits a similar sequential learning structure as in MSE, a lot of aspects differs, including diverging singular values, and different directional convergence.
Critical weaknesses
The paper is comprehensive in its theoretical contributions, but the empirical evaluation could be strengthened in both presentation and scope.
Some of the important numerical results in Appendix H should be considered to place into the main manuscript. This could potentially help the readers to digest the paper better.
Beyond the simplified and synthetic setting evaluated in the paper, some real datasets and nonlinear networks could be checked to expand the coverage of the paper further.
Questions for authors
Given the theoretical results, is it possible to design some kind of spectral preconditioning tricks to better solve the imbalanced classification problem? (for example, by reweighting gradient components to balance the learning speeds of majority and minority modes?)
Thank you for your positive remarks about the paper and for the useful suggestions.
Numerical results in the main text
Thank you for the suggestion. We have done so by including in the revision one figure in each of the regularization path (RP) and gradient flow (GF) sections. Fig. 2 shows the spectral coordinates and normalized singular values along the RP as functions of the regularization weight λ, comparing our analytic solution with a convex solver and projected gradient descent, for K=4 classes and imbalance ratios R=5 and R=20. The numerical solutions match the analytic ones across the path. Fig. 3 compares our reduced GF dynamics under Hadamard initialization with gradient descent from a small Gaussian initialization matched in total energy, for the same values of R. Again, the two show qualitatively the same sequential emergence and (slow) convergence. We hope these help illustrate and contextualize the theoretical findings of the two sections.
Real datasets and nonlinear networks.
Thank you for raising this point. Followed by your and other reviewers' comments, we have added experiments on nonlinear architectures and real data (MNIST, CIFAR-10) in the revision.
The detailed description and presentation of the results is in App. H.3, and we have included one of the figures in the main body (Fig. 4) for training 4 classes of MNIST with an imbalance ratio R=10 on a ReLU MLP with two hidden layers. The figure tracks the projections of the class-mean logits onto the directions of the SEL (centered one-hot) matrix along (i) the regularization strength λ and (ii) iterations of SGD, for CE (top) and MSE (bottom).
For (i), we regularize the loss with a ridge penalty on all parameters and trace the solutions over a grid of λ by continuation, i.e., we train with GD for a fixed number of iterations at each λ and use the solution as initialization for the next. For (ii), we run unregularized CE/MSE training well beyond the terminal phase of training (i.e., well after zero training error).
We thus produce the full 2x2 "landscape" that our theory brings together, CE vs. MSE and GF vs. RP, which are normally studied separately in prior work we are aware of.
Qualitatively, we see agreement with the theory developed for our minimal model (the two-layer linear network with whitened inputs): along both paths the majority, maj-min, and minority modes emerge sequentially, and under CE their normalized versions slowly approach the SEL limit from different sides, unlike the convergence to finite targets under MSE. For CIFAR-10, MSE behaves as for our theory model and MNIST, and under CE the minority mode again emerges last, but the faster emergence of majority versus the maj-min mode is now less visible, while also, at the end of training the majority lies below its SEL value and the maj-min above. We believe this is consistent with the nature of the theoretical model treating the penultimate features as freely optimizable (thus, the whitened inputs in our theory) and can thus be expected to describe the trained network more accurately for simpler data such as MNIST than for more complex data such as CIFAR-10. In spirit of the long line of theoretical work on linear networks, our results are meant as stylized predictions that provide intuition for practical training.
Spectral preconditioning. Good question. Indeed, this type of spectral analysis can yield algorithmic insights. A natural example is loss reweighting, which spectral analysis can show it acts as a form of spectral preconditioning.
Concretely, consider the weighted CE loss ∑jωyjℓj, where examples from majority classes have weight 1 and examples from minority classes have weight ω (e.g., ω=R corresponds to inverse-frequency weighting). Its residual is that of standard CE with each column multiplied by the weight of the corresponding example, keeping it in the form required by Thm. 1. Thus, in our spectral coordinates the reweighting simply rescales the minority components b− by ω, leaves the majority components b+ unchanged, and mixes the maj-min components (β,b) through a fixed 2x2 matrix. As a consequence, we expect the linearized growth rates around the origin of the majority, maj-min, and minority modes change from R, (R+1)/2, and 1 (as in the proof of Thm. 6) to R, (R+ω2)/2, and ω. In particular, ω=R would then equalize all three; we leave formalization of this to future work. This spectral view of reweighting was recently established for the square loss in the UFM by Zhao et al. (2026), and our framework allows extending it to CE.
Beyond emergence, this can also be used to examine convergence. It is known, formally for linear models (Sagawa et al., 2020), that reweighting does not change the limiting direction of CE training. The intuition here is that a preconditioner changes how fast the modes grow but not where they end up. In general, understanding this better as well as the interplay of such classical reweightings with more modern logit-level reweightings (e.g., Menon et al., 2020) along the full trajectory both in terms of RP and GF is an interesting question. We believe the spectral framework provides the tools for such an analysis, which deserves a separate study (we have added an explicit remark that the analysis framework extends to reweighted and logit-adjusted losses in Sec. B.2).
This paper studies how a two-layer linear network learns to tell classes apart under class imbalance and cross-entropy (CE) loss. The central question in this article is which classes are learned first, and the evolution across training. Earlier mean-squared-error (MSE) analyses characterize this evolution under suitable assumptions. This article attempts to extend that analysis to the cross entropy loss commonly used to train neural networks in practice. The softmax operation within CE couples the class scores and thus makes the learning dynamics harder to analyze.
The authors consider the unconstrained features model (UFM) - a standard simplified setting in the neural collapse literature. The objective is learning a classifier weight W∈RK×d and free features H∈Rd×n for n examples, K classes and hidden width d≥K. This article assumes that the inputs are orthgonal vectors ei∈Rn and recast the regularized UFM optimization as minZL(Z)+∥∥Z∥∥⋆ where Z=WH∈RK×n, the scores before softmax. The authors choose a particular structured form of class imbalance: K=2m classes with M=K/2 classes each for the majority and minority group, and R>1 examples per majority class and one per minority class.
The target of the learning for the fixed dataset, is the simplex-encoding label matrix (SEL), Z∈RK×n, obtained by subtracting 1/K from each entry of the one-hot label matrix. Its singular modes are the rank-one components sjujvj⊤, with uj∈RK, vj∈Rn, and singular values sj>0. The learned amplitude aj=uj⊤Zvj∈R measures a singular mode's coefficient in Z. Under small, spectrally aligned initialization, the prior MSE analysis establishes that modes with larger target singular values sj are learned earlier.
Tracking the amplitudes across the learning trajectory requires careful account of the effect of the nonlinear softmax layer. Here, the authors build on Garrod et. al. 2025 who establish a Hadamard diagonalization of softmax for balanced classes (R=1). This diagonlization structure changes under class imbalance. The authors construct an imbalance-adapted singular bases U∈RK×(K−1) and V∈Rn×(K−1) using ΦM∈RM×(M−1), the Sylvester–Hadamard matrix with its all-ones column removed. The majority and minority columns of U are ΦM/M within their respective class groups and the pairing vj=Z⊤uj/sj incorporates the class counts.
Theorem 1 shows that adding a rank-one term ensures that the scores and their negative CE gradient have the same matrix form:
\quad\Longrightarrow\quad
Y-S(Z)=UBV^\top+\beta u_0\widetilde v^\top,$$
where $A,B\in\mathbb R^{(K-1)\times(K-1)}$ are diagonal and $\alpha,\beta\in\mathbb R$. Here, the diagonal entries of $A$ comprise the majority–minority coefficient $a\in\mathbb R$ and the majority and minority amplitude vectors $\mathbf a_+,\mathbf a_-\in\mathbb R^{M-1}$. Together with $\alpha$, this gives $K$ scalar coordinates. Here $u_0\in\mathbb R^K$ is the majority–minority column of $U$, and $\widetilde v\in\mathbb R^n$ is a unit vector orthogonal to $V$'s columns, with equal entries for examples in the same group. With $Y\in\mathbb R^{K\times n}$ the one-hot labels and $S$ columnwise softmax, $Y-S(Z)=-\nabla_Z\mathcal L(Z)$ for the summed CE loss. Thus, a gradient step directly on $Z$ changes only the coefficients in this representation. This result later enables the analysis of both the regularized optima and the suitably initialized gradient flow.
Theorem 1 showed that the gradients preserves the particular structure identified by the authors. In Theorem 2, the authors establish that the identified structure is also satisfied by the global optima of the regularized optimization for every regularization hyperparameter $\lambda>0$. Additionally, Theorem 2 shows that, at the unique regularized global optimum, all the amplitudes within a group are equal: $\mathbf a_+=a_+\mathbf1_{M-1}, \mathbf a_-=a_-\mathbf1_{M-1}.$ This way, Theorem 1's $K$ coordinates can be reduced to the four scalars $(\alpha,a,a_+,a_-)\in\mathbb R^4$, i.e. solving the resulting four-variable optimization can recover the global regularized score matrix, and varying $\lambda$ traces the regularization path (RP). The pair $(\alpha,a)$ determines the mixed-mode contribution $u_0(a v_0+\alpha\widetilde v)^\top$, where $v_0\in\mathbb R^n$ is the first column of $V$. The scalars $a_+$ and $a_-$ are the common majority and minority amplitudes, respectively. Proposition 1 connects these four scalars to when the modes enter the solution in the regularization path as the regularization hyperparameter weakens: majority first, mixed, then minority, (here "entering" meaning a mode's coefficient becomes nonzero). Note: the authors disclose that the phases and spectral formulas can also be recovered from Hong and Ling (2024), after a reparameterization. The contribution here is the derivation in the spectral coordinates used for the training analysis.
Next, the authors study the limiting behavior under the CE objective since the magnitudes of the singular values may diverge. For the RP solution at penalty $\lambda$, let $\boldsymbol\sigma_\lambda=(\sigma_{0,\lambda},\sigma_{+,\lambda},\sigma_{-,\lambda})=(\sqrt{a_\lambda^2+\alpha_\lambda^2},a_{+,\lambda},a_{-,\lambda})\in\mathbb R^3$, representing mixed, majority and minority singular magnitudes. Let $N_\lambda=\sigma_{0,\lambda}+(M-1)(\sigma_{+,\lambda}+\sigma_{-,\lambda})$. Although the magnitudes diverge as $\lambda\to0$, Theorem 3 proves that $$\bar{\boldsymbol\sigma}_{\lambda}:={\boldsymbol\sigma}_{\lambda}/N_{\lambda}=\bar{\boldsymbol\sigma}_{\infty}+O\left(\frac{1}{\log(\frac{1}{\lambda})}\right).$$
Theorem 4 also shows that at finite regularization penality: $\bar\sigma_{+,\lambda}\geq\bar\sigma_{+,\infty}$, $\bar\sigma_{0,\lambda}\leq\bar\sigma_{0,\infty}$, and $\bar\sigma_{-,\lambda}\leq\bar\sigma_{-,\infty}$ for $0<\lambda<\sqrt R$. Note: the limiting geometry and the $\frac{1}{log (1/\lambda)}$ rate is already established by Thrampoulidis et al. (2022) but for the simpler case of balanced class proportions.
The regularization path concern the sequence of optima. Theorems 5–6 concern the training trajectories under the unregularized gradient flow (GF) on $W,H$. Theorem 5 shows that under suitable initialization, these matrices retain the prescribed spectral form, extending Garrod et al. (2025). When the initial within-group amplitudes and the mixed coordinates $\gamma_0,\delta_0$ are both $O(\varepsilon)$ for a sufficiently small $\varepsilon$, Theorem 6 shows that the majority mode reaches a threshold of singular value magnitude first before the mixed or minority modes. Theorems 7-8 extend this observation and show that the relative amplitudes settle into the ground-truth proportions as training progresses to later stages.
In conclusion, this article contributes to the broader effort to understand the emergence of structure in neural representations.
Critical strengths
This article extends the balanced-class analysis to a simplified yet non-trival case of class imbalance. Although not the focus of the submitted article, the progression of ideas in this article suggests other interesting avenues for future inquiry, for e.g. the design of learning objectives and algorithms where the errors of misclassification are weighted non-uniformly across the set of class pairs. The emergence of modes tied to the regularization penalty suggests a potential way to encode additional preferences/non-uniform weighting to reorder the emergence of modes across regularization path or the training trajectories subject. Such a reordering could be adapted for compute constraints and hyper-parameter grid-searches for large-scale training.
The authors show that the spectral coefficients characterizing majority, minority and mixed modes provide a unifying starting point for investigated both the regularization path of optimal solutions and the training trajectory under gradient flow. Theorem 2's reduction shows that this analysis hinges on just the 4 scalars.
The expanded related-work discussion in Appendix B.1 is thorough and insighftul. It gives a clear narrative through the different strands of past literature and was critical to help me understand the broader context of this paper.
Critical weaknesses
My main concern is the writing/communication of results. The main text is dense, and I often needed the appendix to understand the implications of a statement or the motivations behind a definition. The main text would strongly benefit from more motivation and intuition that helps readers absorb the results without having to reconstruct it from the proofs in the appendix. Further the length and density of the appendix (although well organized) presents a real impediment to reliable verification. I have 'skimmed' the proofs multiple times, and their overall arguments 'appear' correct. However, a careful vetting of the 60+ pages of detailed algebraic calculations might take longer than a conference reviewing timeline feasibly permits. As such, I do not feel comfortable in certifying correctness even though I believe that the theoretical statements are sound.
The notation choices made it unnecessarily hard to parse the article. As an example: the authors use a∈R, a∈RK, a±∈RM−1, and later the common scalar entries a±∈R. The mixed-mode singular magnitude is a2+α2, so even here an amplitude coordinate and a singular magnitude are different objects. I strongly encourage the authors to disambiguate their notation.
If not for the writing choices and the density/length of the appendix I would have increased my score for the article.
Questions for authors
The construction of the adapted Hadamard bases and the identified spectral structure felt hand-crafted for the choice of CE and the particular form of class imbalance. Could the technical machinery here be repurposed for a more general form of non-linear smooth loss or a broader form of class imbalance?
We appreciate your careful read and constructive feedback.
Presentation and verification. Points well taken. We have tried to accompany each result in the main text with intuition rather than presenting it in isolation, e.g., the discussion after Thm. 1 on how to read the spectral coordinates, the comparison with MSE after Thms. 3–4, and the intuition after Thm. 6 on why the majority emerges first and why the maj-min mode is harder to control, as well as the proof sketch of Thm. 8.
Following your and other reviewers' comments, in the revision we have added (i) a motivation for studying the RP alongside GF in the Introduction, (ii) an explicit distinction between spectral coordinates and singular values after Thm. 1, and (iii) numerical illustrations of both the RP and GF results in the main text (Figs. 2 and 3), all in blue. Following your remarks, we have also revised some of the notation (see response below).
Regarding verification, we fully understand the concern given the length of the appendix. In part, this length is because we have tried to be detailed and exhaustive in the background discussion (App. A), related work (App. B), and numerical results (App. H), while the proofs are in App. C–G.
To further support the validity of our theoretical findings, we have validated them numerically wherever possible, and the new main-text figures now show such validations directly: Fig. 2 compares our closed-form RP solution with a direct convex solver (CVXPY) and projected gradient descent, and Fig. 3 compares the reduced GF dynamics with gradient descent from Gaussian initialization. Together with the intuitive discussions and proof sketches, we hope these help readers gain confidence in the results without having to consult the appendix.
Notation.
Thank you for the pointer. Your point is well-taken and we make two changes in the revision (in blue) to help parse the notation: (i) we now use the symbol σ0=α2+a2 for the maj-min singular value (rather than a0, used previously) throughout, in both the RP and GF sections. This helps separate the spectral coordinates(α,a) from the singular valueσ0. (ii) Right after Thm. 1, where the spectral coordinates are first introduced, we now explicitly explain the distinction between spectral coordinates and singular values: for the majority and minority modes, spectral coordinates and singular values coincide, whereas for the maj-min mode neither α nor a is itself a singular value. We have also unified the terminology to "spectral coordinates" throughout.
Technical machinery for other losses and imbalance styles
That is a good question, thank you. Indeed, the Hadamard-parameterized bases are not tailored to CE or to two groups. App. B.2 already anticipated the extension to more groups of imbalance but we have now added concrete details in the revision together with a remark on how to extend to other losses (see revised App. B.2, in blue).
For your convenience, we provide here a short summary of the two points:
Broader class imbalance. The natural extension is multi-step imbalance, with K=kM classes in k>2 groups and Ri samples per class in group i. Thm. 1 extends here because the bases U,V of the corresponding SEL matrix keep a similar form as in Sec. 3. In these bases, logits and residuals are diagonal on one Hadamard block per group, generalizing our majority and minority modes, plus a single (k−1)×k block coupling all groups, generalizing our maj-min pair (α,a) (for us, k=2). The change is a modified map F in Thm. 1. Now, making this form of approximate diagonalization useful requires simplifying the resulting equations as we do for two groups in Eqns. (5-6). This needs additional work beyond our scope, although we do expect the same ideas to apply. For long-tailed imbalance, on the other hand, we expect that fresh ideas are needed.
Other losses. A natural extension is to other smooth losses, such as reweighted or logit-adjusted CE. Again, Thm. 1 extends to these because its proof only uses that (a) the per-example loss gradient is equivariant under permutations of the classes within each group (a property that plays nicely with the phase-shift property of Hadamard matrices), and (ii) that the loss is invariant to constant shift. The logits and residuals then keep exactly the same structure, so the GF reduction of Thm. 5 also carries over, and by convexity we expect the same for the RP reduction of Thm. 2. As above, the change is a modified map F in Thm. 1, which to make useful requires analyzing its new closed form; this again again needs additional work beyond the scope of this paper, but we believe it strengthens the point that our framework, built on Hadamard-parameterized bases, admits several extensions that future work can study.
Experiments on real data and nonlinear networks
Thank you for your thoughtful feedback. We do agree the paper benefits from such experiments and comparison, and have thus added experiments on nonlinear architectures and real data (MNIST, CIFAR-10) in the revision.
The detailed description and presentation of the results is in App. H.3, and we have included one of the figures in the main body (Fig. 4) for training 4 classes of MNIST with an imbalance ratio R=10 on a ReLU MLP with two hidden layers. The figure tracks the projections of the class-mean logits onto the directions of the SEL (centered one-hot) matrix along (i) the regularization strength λ and (ii) iterations of SGD, for CE (top) and MSE (bottom).
For (i), we regularize the loss with a ridge penalty on all parameters and trace the solutions over a grid of λ by continuation, i.e., we train with GD for a fixed number of iterations at each λ and use the solution as initialization for the next. For (ii), we run unregularized CE/MSE training well beyond the terminal phase of training (i.e., well after zero training error). We thus produce the full 2x2 "landscape" that our theory brings together, CE vs. MSE and GF vs. RP, which are normally studied separately in prior work we are aware of.
Qualitatively, we see agreement with the theory developed for our minimal model (the two-layer linear network with whitened inputs): along both paths the majority, maj-min, and minority modes emerge sequentially, and under CE their normalized versions slowly approach the SEL limit from different sides, unlike the convergence to finite targets under MSE. For CIFAR-10, MSE behaves as for our theory model and MNIST, and under CE the minority mode again emerges last, but the faster emergence of majority versus the maj-min mode is now less visible, while also, at the end of training the majority lies below its SEL value and the maj-min above. We believe this is consistent with the nature of the theoretical model treating the penultimate features as freely optimizable (thus, the whitened inputs in our theory) and can thus be expected to describe the trained network more accurately for simpler data such as MNIST than for more complex data such as CIFAR-10. In spirit of the long line of theoretical work on linear networks, our results are meant as stylized predictions that provide intuition for practical training.
Thanks again for your suggestion.