acceptodds
Under review as a conference paper at ICLR 2027

What Does Neural Collapse Indicate for Generalization?: A Singular Learning Theory Perspective

Abstract

Neural Collapse (NC), which describes the emergence of highly structured class representations in the terminal phase of training, has been widely studied in connection with generalization. However, its practical role remains unclear, as several studies report that NC can hinder generalization in downstream tasks. Existing analyses have mainly focused on NC at the level of representation geometry, leaving unclear what this geometric simplification implies for the complexity of the underlying model. We study this question through Singular Learning Theory (SLT), which quantifies local singular complexity using the finite-temperature empirical Local Learning Coefficient (LLC). In this work, we establish a theoretical bridge between NC and SLT by characterizing local singular complexity through the LLC under exact and approximate NC. Empirically, we validate the predicted stability of local LLC under small deviations from exact-NC and the changes in local loss geometry near exact-NC in Polynomial Neural Networks, and observe similar complexity-related trends across broader architectures. This complexity-based view also offers a new perspective on in-domain generalization under NC. Furthermore, we show that deterministic NC statistics can serve as efficient sampling-free indicators of changes in model complexity during training.

Primary area:
learning theory
Keywords:
Singular Learning Theory, Geometric Deep Learning, Neuroalgebraic Geometry, Neural Collapse, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UjDNOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper study the connection between the neural collapse phenomenon and the generalization ability of the neural network by singular learning theory. The paper first consider exact NC and compute the corresponding local RLCT under certain assumptions, which lead to a generalization bound. Then, for non-exact NC, the authors show that the LLC of the model can be approximated by the LLC of the exact NC model. Then under the assumptions that the LLC is close to the RLCT, the authors derive Baysian generalization bounds using the RLCT.

Critical strengths

This paper studies the relation between the neural collapse phenomenon and the generalization ability of the neural networks, which I think is a crucial problem. This paper applies singular learning theory, which I think is a new angle for studying this problem.

Critical weaknesses
  1. As a reader that is familiar with neural collapse literature, but not singular learning theory, this paper is hard to understand.

    • The term RLCT is frequently used in the main text, but neither properly defined nor discussed before its usage.
    • In Theorem 4.5, it is not clear how the u,v,ξu,v,\xi and its dimension r,qr,q relate to the actual neural network parameters defined in section 3.1.
    • Theorem 4.9 "assume the statistical compatibility conditions of Appendix A.4", but these assumptions are not properly defined in the main text. While I understand the page limit, I suggest the authors consider re-structure the paper, and put necessary definitions and discussion in the main text.
  2. Several results in this paper relies on assumptions that are not properly justified. Thus, it is hard to understand the scope of the results of this paper.

    • Theorem 4.5 and 4.6 assume certain structure on the empirical loss (equivalently Kn\mathcal K_n), which is not justitfied. Could the authors explain more on: (1) are these conditions universally satisfied for the architecture in section 3.1 and all datasets? (2) if not, under what extra assumptions on the model architecture and datasets will these assumptions satisfied?
    • Theorem 4.9 "assume the statistical compatibility conditions of Appendix A.4". However, these assumptions are again not justified. Could the the authors again explain: (1) are the statistical compatibility conditions universally satisfied for the architecture in section 3.1 and all datasets? (2) if not, under what extra assumptions on the model architecture and datasets will these assumptions satisfied?
Questions for authors

Please also refer to the "Weaknesses" part.

  1. Could the authors explain more on the connection between Proposition 4.3 and Theorem 4.5?
6 comments
AuthorsOpenReview ↗

Main Comment: Clarifying Theoretical Assumptions and Statistical Justification

Thank you for your constructive feedback. We agree that the theoretical assumptions, their applicability, and the connection between the original Proposition 4.3 and the original Theorem 4.5 require clearer explanations. We have substantially strengthened our theoretical analysis to address these concerns.

  • Addressing Weakness 1 (Clarity of theoretical presentation): We will define RLCT and explain its role in our analysis before using it in the main text. We will also clarify the meaning of the coordinates (u,v,ξ)(u,v,\xi) and dimensions (r,q)(r,q) in the original Theorem 4.5, including their relationship to the neural network parameters. Furthermore, we will move the essential statistical compatibility conditions of Theorem 4.9 from the appendix to the main text.

  • Addressing Weakness 2 (Assumptions of Theorems 4.5 and 4.6): We agree that the original loss-structure assumption was not sufficiently justified by Proposition 4.3. Rather than assuming this structure, we now derive the local RLCT directly from the full network loss in an explicit, fully trainable PNN. This yields an exact local RLCT classification under specified architectural and data conditions, without relying on the original normal-form assumption.

  • Addressing Weakness 2 (Statistical assumptions of Theorem 4.9): We clarify that these assumptions are not universally guaranteed. We establish the population RLCT correspondence directly in our Gaussian PNN and separately examine the additional empirical LLC compatibility and scaling conditions required by Theorem 4.9. We will explicitly distinguish the conditions we have established from those that remain assumptions.

Follow-up 1 clarifies the original coordinates and the relationship between Proposition 4.3 and Theorem 4.5. Follow-up 2 addresses the loss-structure assumptions through a direct RLCT derivation, while Follow-up 3 explains the statistical assumptions and their justification in Theorem 4.9.

In the revision, we will clarify the definitions of RLCT and LLC, explain the original normal-form coordinates and their limitations, and present the necessary assumptions of Theorem 4.9 more explicitly in the main text.

Further details on the derivations, applicability conditions, and statistical justification are provided in the follow-up comments below.

AuthorsOpenReview ↗

Follow-up 1: Clarifying the Theoretical Presentation (Weakness 1 and Question 1)

We agree that several key concepts and assumptions were not sufficiently explained in the main text. We will reorganize the theoretical presentation to address the following points.

  • Definition of RLCT. We will introduce RLCT before its first use and explain its meaning as a measure of how the volume of low-loss parameter configurations scales near a solution. We will also clarify its role in connecting local loss complexity to Bayesian generalization.

  • Meaning of the coordinates in the original Theorem 4.5. The coordinates (u,v,ξ)(u,v,\xi) are intended to represent quadratic, higher-order, and locally flat directions in the assumed normal form, with rr and qq denoting the dimensions of the first two blocks. However, these coordinates were not explicitly constructed from the original network parameters θ=(W1,…,WL)\theta=(W_1,\ldots,W_L). We will clarify their intended interpretation and this limitation in the revised manuscript.

  • Connection between Proposition 4.3 and Theorem 4.5. The original Proposition 4.3 demonstrates higher-order loss behavior along a particular perturbation curve, but does not establish the separable normal form assumed in the original Theorem 4.5. We will make this relationship explicit and replace the assumed normal-form argument in the main text with a direct analysis of the network's prediction errors, as described in Follow-up 2.

We will also move the essential statistical compatibility conditions of Theorem 4.9 from Appendix A.4 to the main text. Their meaning and justification are addressed in Follow-up 3.

AuthorsOpenReview ↗

Follow-up 2: Justification and Scope of the Loss-Structure Assumptions (Weakness 2)

The original loss-structure assumptions were not shown to hold for all parameter configurations and datasets in the setting of Section 3.1. To address this concern, we replace the assumed normal form with a direct RLCT calculation under explicit architectural and data conditions.

  • Model and dataset conditions. We consider a fully trainable, one-hidden-layer cubic PNN, fθ(x)=Cσ(Wx)f_\theta(x)=C\sigma(Wx) with σ(t)=t3−3t\sigma(t)=t^3-3t and hidden width five. All 40 parameters are trainable. The inputs have the class-structured form xk,z=ak+Uzx_{k,z}=a_k+Uz, where [a1 a2 a3 U][a_1\ a_2\ a_3\ U] is invertible. We require positive sample weights and classwise degree-three polynomial unisolvence, which is satisfied by our Pascal Cloud construction and other datasets meeting the same rank condition.

  • Direct RLCT derivation. Instead of assuming a loss normal form, we express the complete prediction errors through the residual ideal

    I=⟨CH−P,  CV⟩,\mathcal I=\langle CH-P,\;CV\rangle,

    where CC is the classifier, PP is the centered target matrix, HH contains the hidden class means, and VV contains the within-class feature coefficients. The unisolvence condition makes this ideal equivalent to the empirical residual ideal. Analyzing the full 40-dimensional parameter space gives

    \begin{cases} 19/2,&\mathrm{NC1\ holds},\\\\ 13,&\mathrm{NC1\ fails}, \end{cases}$$ for every zero-residual solution sufficiently near our constructed exact-NC solution. This calculation does not require the original normal-form assumption or the width condition of Proposition 4.3.
  • Applicability beyond the constructed dataset. The same RLCT classification holds for classwise degree-three-unisolvent finite datasets and transfers to fixed, nondegenerate class-conditional Gaussian input distributions within the specified model family. We also establish strict RLCT separation near constructed solutions at widths five through nine, although the exact two-value classification is specific to width five.

Thus, we do not claim that the original normal-form assumption holds universally. Instead, the revised results establish the local RLCT directly under stated model and data conditions. We will present these conditions and the corresponding derivation in the revised main text, with technical details in the appendix.

Reviewer_Ak2nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Neural collapse (NC), discovered by Papyan et al. 2020, describes the phenomena by which last-layer features in a classification task converge to class means during the late phase of training (e.g. after the train accuracy has already reached 100%). However, the relationship between neural collapse and generalization remains unclear. This paper proposes using singular learning theory (SLT) to establish a link to generalization. The main theorem of SLT (see e.g. Watanabe 2009) says that the generalization error of a parametric Bayesian model (like a Bayesian neural network) is determined asymptotically by a quantity known as the local learning coefficient (LLC)[1], denoted by λ\lambda. This paper aims to determine, under several strong assumptions, the LLC for (polynomial) neural networks after NC, and thereby link NC to (Bayesian) generalization performance.

They also perform experiments attempting to empirically validate the main steps used in the paper’s argument, including: that approximately collapsed networks have complexity close to that of perfectly collapsed ones, that this complexity predicts generalization, and that breaking collapse can increase complexity. They additionally explore whether NC statistics can serve as cheap indicators of changes in complexity during training, via a transformer grokking experiment.

[1]: Note that throughout my review, I have elided the difference between the local learning coefficient (LLC), the global learning coefficient (often simply called “learning coefficient”), and the real-log canonical threshold (RLCT) as the distinction is unimportant for our purposes.

Critical strengths

The motivation behind the work seems strong. The connection between neural collapse and generalization remains unclear, and seems like it should be explored theoretically. Using singular learning theory to attack this question appears like a novel and interesting angle.

It is to the authors’ credit that they use singular learning theory here, as more simplistic generalization analyses often use tools like the loss Hessian which relies on regularity assumptions that do not hold for neural networks (see Lau et al. 2025). I am excited to see singular learning theory applied to investigate modern deep learning phenomena, as I believe this is an underexplored angle. Similarly, the use of polynomial neural networks is also an emerging but still undervalued tool in my opinion, and I am excited to see these used here.

Critical weaknesses

The paper attempts to connect neural collapse (NC) to singular learning theory (SLT). It remains unclear to me how much this bridge is actually established by the work; much of the connection relies on postulated assumptions rather than derivations or experimental results. More bluntly, it is not clear to me whether the theory gets out significantly more than is put in. This problem is exacerbated by the quality of the writing, which is dense and difficult to read.

For example, Theorem 4.5 appears to be a key result establishing the postulated learning coefficient of a solution under neural collapse, $\lambda_{NC}$. However, the result uses no property of NC, as far as I can tell. Instead, it assumes that the loss obeys a very particular “normal form” (a specific sum of monomials), at which point the LLC calculation is trivial and immediate. As far as I can tell, this normal form is supported only via a motivating sketch in Proposition 4.3 - but Proposition 4.3 itself requires incredibly unrealistic assumptions ($d_{L−1} \geq n$, i.e. penultimate-layer width larger than the dataset size), and depends only weakly on the NC assumptions. If this reading is correct, then it appears to me like there is no actual derivation of learning coefficients from neural collapse assumptions.

This is just one example; similar concerns appear to apply to other results throughout the paper, such as Theorem 4.8 and Theorem 4.9, which also appear to make questionable assumptions, though I invite the authors to defend them. Finding these issues is time-consuming, given the dense writing style and key assumptions which are relegated to the appendix.

Questions for authors
  • What does the framework predict about whether NC helps or hurts generalization? This is the question asked in the title of the paper. I may be able to infer this from the RLCT formula, but I struggle to find an answer explicitly stated anywhere.
  • Is there any network under NC for which the normal form assumed in Theorem 4.5 can be shown to hold, or at the very least checked numerically?
7 comments
AuthorsOpenReview ↗

Main Comment: Strengthening the Theoretical Connection between NC and Generalization

Thank you for your constructive feedback. As we understand it, the central concern raised in Weakness 1 is that Proposition 4.3 does not sufficiently justify the local normal form assumed in Theorem 4.5. To address this concern, we have substantially strengthened our theoretical analysis by deriving the relevant loss structure directly from the network's prediction errors, without relying on the previously assumed normal form.

Our new results establish three main conclusions:

  • NC can be associated with lower local model complexity. In a fully trainable, 40-parameter cubic PNN, we prove that all zero-residual solutions sufficiently near a constructed exact-NC solution have an RLCT of 19/219/2 when NC1 holds and 1313 when NC1 fails. We also construct NC1-breaking perturbations that preserve NC2 and NC3. These results are derived from the full network loss without requiring the hidden width to exceed the dataset size. Moreover, the RLCT classification extends beyond the finite Pascal Cloud to arbitrary classwise degree-three-unisolvent datasets and nondegenerate class-conditional Gaussian population distributions.

  • NC also shapes the finer structure of the local loss landscape. Our higher-width results show that classifier alignment (NC3) can affect the logarithmic factor of the local learning asymptotics, even when the RLCT remains unchanged. Moreover, solutions satisfying the same NC conditions can exhibit different local loss structures.

  • Implications for Bayesian generalization. In our explicit Gaussian PNN model, the population RLCT agrees with the computed full-loss RLCT. Under appropriate SLT regularity and prior-localization conditions, the smaller RLCT at NC1-preserving solutions translates into a smaller leading-order expected Bayesian predictive generalization error compared with the nearby NC1-breaking solutions.

Follow-ups 1 and 2 present the model construction and direct RLCT derivation that replace the original normal-form assumption. Follow-ups 3 and 4 separately address the assumptions and applicability of Theorems 4.8 and 4.9, respectively.

We would appreciate any further feedback on whether these results address your main theoretical concerns.

AuthorsOpenReview ↗

Follow-up 1: Model and Dataset for the Revised Proofs

To address the concern about the assumed loss structure, we introduce a model and dataset for which the full empirical loss can be analyzed directly.

  • We construct a synthetic dataset that separates class-specific structure from within-class variation, allowing us to analyze the relationship between NC and loss singularities explicitly.

  • We use three classes with 27 samples per class (n=81n = 81 total samples) and centered class targets. The synthetic dataset, called the Pascal Cloud, is constructed using a triangular lattice and Pascal-type multinomial coefficients.

  • The Pascal Cloud is generated as follows:

    • At refinement depth dd, we arrange coarse sites on a triangular lattice indexed by nonnegative integers (i,j,ℓ)(i,j,\ell) satisfying i+j+ℓ=di+j+\ell=d. Each site is assigned the multinomial coefficient

      wi,j,ℓ=d!i!j!ℓ!.w_{i,j,\ell}=\frac{d!}{i!j!\ell!}.

    • Each coarse site is replaced by wi,j,ℓw_{i,j,\ell} distinct sites from a smaller triangular lattice, with offsets centered to preserve the coarse-site mean and scaled by a small refinement factor. At depth d=3d=3, this produces 33=273^3=27 distinct samples per class in our chosen configuration.

  • The purpose of this construction is to obtain a dataset with degree-three polynomial unisolvence. In our chosen configuration, the polynomial evaluation matrix has full column rank, which we verify by exact rational computation. Importantly, our RLCT results do not depend on the particular coordinates of the Pascal Cloud. They extend to arbitrary classwise degree-three-unisolvent datasets with positive sample weights.

  • Our network is a fully trainable cubic PNN with hidden width five (dL−1=5≪n=81d_{L-1} = 5 \ll n = 81):

    fθ(x)=Cσ(Wx),σ(t)=t3−3t,f_\theta(x)=C\sigma(Wx),\qquad\sigma(t)=t^3-3t,

    where W∈R5×5W\in\mathbb R^{5\times5} is the hidden-layer weight matrix and C∈R3×5C\in\mathbb R^{3\times5} is the classifier matrix. All 40 parameters are trainable.

  • For the analysis, we use a coordinate system that separates class-specific and within-class components. Each input can be expressed as

    xk,z=ak+Uz∈R5,x_{k,z}=a_k+Uz\in\mathbb R^5,

    where ak∈R5a_k\in\mathbb R^5 is the class-specific component, z∈R2z\in\mathbb R^2 represents within-class variation, and U∈R5×2U\in\mathbb R^{5\times2} maps these variations into the input space.

    Assuming A=[a1 a2 a3 U]A=[a_1\ a_2\ a_3\ U] is invertible, we can equivalently write

    W=[T B]A−1,fθ(ak+Uz)=Cσ(Tek+Bz),W=[T\ B]A^{-1},\qquad f_\theta(a_k+Uz)=C\sigma(Te_k+Bz),

    where ek∈R3e_k\in\mathbb R^3 is the kk-th standard basis vector. This is an invertible reparameterization of the original network, so it preserves the local RLCT.

  • Since the classwise polynomial evaluation matrices have full column rank, the sample residual ideal coincides with the polynomial coefficient residual ideal. The corresponding squared losses are also comparable up to positive constants.

  • Consequently, we can compute the local RLCT of the original empirical squared loss through the full polynomial coefficient residual ideal, without restricting the trainable parameter space. This equivalence holds for arbitrary classwise degree-three-unisolvent datasets with positive sample weights.

  • Continuous population extension: For fixed Z∣k∼N(νk,Σk)Z\mid k\sim\mathcal N(\nu_k,\Sigma_k) with Σk≻0\Sigma_k\succ0, the classwise degree-three monomial moment matrices Mk=E[v(Z)v(Z)⊤∣k]M_k=\mathbb E[v(Z)v(Z)^\top\mid k] are positive definite. The Gaussian population loss is a positive-definite quadratic form in the same 90 coefficient residuals, so the analytic ideal, RLCT, and logarithmic multiplicity agree at any fixed parameter point. The class indicator eke_k remains part of the input. Although the residual ideal is invariant, NC2 and NC3 are evaluated from the actual class means under the chosen distribution.

AuthorsOpenReview ↗

Follow-up 2: Direct Derivation of NC-Dependent Local Learning Complexity

We derive the local loss structure from the prediction errors without assuming a normal form. We plan to move the original quartic-curve and conditional normal-form results to the appendix, replacing them in the main text with the results below.

Revised Proposition 4.3: Prediction-Error Representation

Consider the three-class cubic PNN fθ(ek,z)=Cσ(Tek+Bz)f_\theta(e_k,z)=C\sigma(Te_k+Bz) with hidden width hh. Let P=I3−11⊤/3P=I_3-\mathbf1\mathbf1^\top/3 be the centered target matrix, HH the hidden class means, and VV the 2727 nonconstant feature-coefficient vectors (99 bivariate monomials per class). The complete prediction-error ideal is

I=⟨CH−P, CV⟩.\mathcal I=\langle CH-P,\,CV\rangle.

The blocks CH−PCH-P and CVCV give 99 class-mean and 8181 within-class output equations. Under classwise degree-three unisolvence, they generate the empirical residual ideal.

NC1 gives V=0V=0. Let E∈R3×2E\in\mathbb R^{3\times2} satisfy EE⊤=PEE^\top=P, and set R=HER=HE and G=R⊤RG=R^\top R. NC2 holds exactly when G=s2I2G=s^2I_2 for some s>0s>0. On CH=PCH=P, where rank⁡R=2\operatorname{rank}R=2, the classifier decomposes uniquely as

C=EG−1R⊤+L,LR=0,C=EG^{-1}R^\top+L,\qquad LR=0,

Here, NC3 additionally requires G=s2I2G=s^2I_2 and L=0L=0. Moreover,

CV=EG−1R⊤V+LV.CV=EG^{-1}R^\top V+LV.

Thus NC1, NC2, and NC3 constrain a common residual system, whose local structure determines the RLCT.

Revised Theorem 4.5: Exact Local RLCT Classification

For a fully active exact-NC solution θ∗\theta_* of width five, analysis of the residual ideal over all 40 trainable parameters yields the following RLCT at every nearby zero-residual solution:

\begin{cases} 19/2,&B=0\quad(\mathrm{NC1}),\\ 13,&B\neq0\quad(\neg\mathrm{NC1}). \end{cases}$$ **Proof sketch.** We first remove eight independent mean-error constraints by an invertible local analytic coordinate change. For $B\neq0$, 18 further regular constraints remain, yielding $(\lambda,m)=(13,1)$. For $B=0$, direct analysis of the remaining higher-order residual interactions gives $\lambda=19/2$. This classifies the exponent across the nearby zero-residual set, not just along perturbation curves. The multiplicity at the exact-NC reference remains undetermined (though $m=1$ at other analyzed NC1-preserving points). ### Revised Theorem 4.6: NC1-Breaking and Strict RLCT Separation An analytic zero-residual deformation breaks NC1 while preserving NC2 and NC3. At width five, the RLCT rises from $19/2$ to $13$. The deformation extends to any finite width $h\ge5$ for class-dependent Gaussian means and common positive-definite covariance. At the constructed endpoints for widths five through nine, every sufficiently nearby NC1-breaking interpolant has larger RLCT than every nearby NC1-preserving interpolant. **Distributional robustness.** Classwise degree-three unisolvence identifies the finite-sample and coefficient residual ideals. Fixed nondegenerate class-conditional Gaussian inputs preserve the ideal, RLCT, and multiplicity at each parameter point, even with different positive-definite covariances across classes. Unlike this RLCT transfer, preserving NC2 and NC3 along the explicit deformation uses the common-covariance condition above. ### Additional Proposition: NC Conditions and Logarithmic Multiplicity - At width six, changing only the classifier breaks NC3 while preserving the hidden representation, NC1, NC2, and exact interpolation. Both solutions have $\lambda=29/2$, but their logarithmic multiplicities differ: $m=2$ versus $m=1$. - We also construct pairs at widths six and nine satisfying identical NC1, NC2, and NC3 conditions but having different logarithmic multiplicities. These results follow from the full local residual equations and will be unified in an appendix proposition. The NC conditions thus constrain but do not determine the full singularity. Changing multiplicity at fixed RLCT does not itself change the leading Bayesian coefficient $\lambda/n$.
Reviewer_T4DZOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the relationship between neural collapse, the loss landscape and local parameter complexity. The authors compare learned features with a reference obtained by projecting them to satisfy two neural-collapse properties. They derive bounds on how much this projection changes the loss and local learning coefficient and connect the analysis to Bayesian generalization. Experiments in standard neural networks illustrate these results.

Critical strengths

The paper offers an interesting perspective on neural collapse through singular learning theory, especially by studying how the collapsed arrangement relates to the surrounding loss landscape.

Critical weaknesses
  1. The presentation is a bit difficult to follow. Some technical terms, symbols, and acronyms are introduced with limited explanation, making the main results less accessible to readers without a background in singular learning theory. The connection between the training-transition experiment in Fig 4 and the main theoretical results also needs clearer explanation.

  2. It is somewhat unclear to me what the bounds and analysis explain about neural collapse in ordinary training. The projected features serve as an analytical reference, and the main conclusions require several additional assumptions. It would help to clarify what the analysis reveals about just ordinarily trained netowrk’s learned features and generalization beyond the comparison with the projected reference.

Questions for authors

It would be very helpful if the authors can clarify for weakness 2.

7 comments
AuthorsOpenReview ↗

Main Comment: The Core Conclusion of the Paper and Revision Direction

Thank you for taking the time to review our paper and for identifying the parts that were unclear. We will revise the manuscript with the following clarifications about how our results on the NC reference relate to ordinarily trained neural networks and their generalization.

To address Weakness 2, we clarify how our results on exact NC relate to the local complexity and generalization of ordinarily trained neural networks.

Here, we use the term "solutions" to refer to zero-training-loss parameter configurations of the explicit PNN considered in our theoretical analysis. Among these solutions, those satisfying within-class collapse have lower local complexity than nearby solutions without it. We also study networks exhibiting approximate, rather than exact, NC.

  • Why NC matters: Our analysis shows that exact NC has the lowest local learning complexity among the nearby solutions considered. Departures from exact NC can increase this complexity.
  • What this means for trained networks: Our key result shows that a trained network sufficiently close to NC can have local learning complexity close to that of the exact NC reference at finite temperature under the conditions of Theorem 4.8.
  • What this means for generalization: Under appropriate statistical conditions, this lower complexity can translate into better Bayesian generalization for the actual trained network, as established in Theorem 4.9.
  • Improving accessibility and presentation (Addressing Weakness 1): We are revising the manuscript with three goals: 1) explaining SLT, RLCT, LLC, and the Gibbs distribution intuitively for readers unfamiliar with SLT (see Follow-up 1), 2) clarifying why each concept is needed to connect representation space to parameter space, and 3) clearly stating the scope and assumptions of each theorem.
  • Figure 4 investigates whether this perspective can be useful for monitoring training transitions in Transformers. It presents an exploratory experiment that motivates the use of deterministic NC statistics as low-cost indicators of regime changes, rather than directly verifying our theorems on Transformers.(see Follow-up 4).

We provide more detailed explanations in the follow-up comments below. We would welcome any further questions or feedback on these clarifications.

AuthorsOpenReview ↗

Follow-up 1: Meaning of SLT, RLCT, LLC, and Bayesian Generalization

To address Weakness 1, we will revise the theoretical background to explain why SLT, RLCT, and LLC are needed for our analysis and how they connect NC geometry to generalization. The revised background will explain these concepts in the following order, emphasizing their roles in our argument.

  • SLT: Why is it necessary?

    NC describes geometric structure in representation space, but its connection to generalization remains unclear. Neural networks are often singular models because different parameter configurations can represent the same function (non-identifiability), making parameter dimension alone insufficient to characterize their complexity. SLT provides a framework for analyzing this complexity through local loss geometry, allowing us to investigate how NC in representation space relates to model complexity in parameter space.

  • RLCT: How does it characterize model complexity?

    The RLCT is a central quantity in SLT that characterizes how the volume of low-loss parameter regions scales near a loss minimum. For example, even with the same number of parameters, u2+v2u^2+v^2 and u2+v4u^2+v^4 have different RLCTs because their losses increase at different rates along parameter directions. Thus, the RLCT captures aspects of loss complexity that parameter dimension alone cannot describe.

    More precisely, a local learning integral can scale as n−λ(log⁡n)m−1n^{-\lambda}(\log n)^{m-1}. The RLCT λ\lambda determines the power of nn, whereas mm determines an additional logarithmic factor.

  • LLC: What if the RLCT is difficult to compute directly?

    In neural networks, directly computing the RLCT is difficult due to their high-dimensional parameter spaces and complex loss landscapes. The LLC provides a way to examine local loss complexity around trained parameters at a finite inverse temperature. Under appropriate conditions, the LLC converges to the corresponding RLCT as the inverse temperature increases toward infinity.

  • Gibbs distribution: How is the LLC computed?

    The Gibbs distribution assigns greater probability to parameters with lower loss within a local neighborhood. Here, inverse temperature is not a physical temperature but a parameter controlling how strongly the distribution concentrates on low-loss regions. The LLC uses the Gibbs-averaged excess loss, measured relative to the local minimum, to characterize local loss complexity.

  • Bayesian generalization: How does NC relate to generalization?

    Under appropriate statistical assumptions, SLT shows that the leading term of the expected Bayesian generalization error is given by λ/n\lambda/n, where λ\lambda is the population RLCT.

    In this work, we first show that deviations from NC geometry bound the difference between the empirical loss and its NC-projected counterpart (Proposition 4.7). We then extend this result to a bound on the LLC difference between the two losses (Theorem 4.8). Under additional assumptions connecting empirical loss complexity to the population RLCT, we derive a Bayesian generalization result for the actual network (Theorem 4.9).

Thus, our central goal is to establish how NC geometry relates to local loss complexity and, ultimately, to generalization.

AuthorsOpenReview ↗

Follow-up 2: The Relationship between the NC-Projected Loss and the Actual Network

  • The NC-projected loss is an analytical reference constructed from the features of the actual network, rather than a separate predictive model. In an ordinarily trained network, weights and features are coupled in a complex manner, making it difficult to isolate the effect of representation geometry on the parameter-space loss Ln(θ)L_n(\theta) directly.
  • The NC reference LnNC(θ)L_n^{\mathrm{NC}}(\theta) is constructed on the same parameter neighborhood UU by removing within-class feature variation and projecting the class means onto a scaled simplex ETF configuration while preserving the global mean.
  • The deviation of the actual learned features from this reference is quantified directly using measurable NC1 and NC2 deviations (ϵNC1,ϵNC2\epsilon_{\mathrm{NC1}}, \epsilon_{\mathrm{NC2}}).
  • Proposition 4.7 uses these deviations to bound the pointwise difference ∣Ln(θ)−LnNC(θ)∣|L_n(\theta) - L_n^{\mathrm{NC}}(\theta)| between the empirical loss and its NC-projected counterpart.
  • Theorem 4.8 shows that the finite-temperature LLCs of the two losses (λ^t\hat{\lambda}_t and λ^tNC\hat{\lambda}_t^{\mathrm{NC}}) remain close when the NC deviations are uniformly small over the parameter neighborhood and the stated Gibbs variance condition holds.
  • Therefore, our analysis is not limited to idealized exact-NC configurations. It quantifies how the representation geometry of an actually trained network constrains its local parameter-space complexity.
  • However, connecting the NC-projected loss to asymptotic Bayesian generalization requires additional statistical compatibility conditions. In particular, the projected loss and the actual population loss do not necessarily have the same RLCT without further structural conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.