- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper study the connection between the neural collapse phenomenon and the generalization ability of the neural network by singular learning theory. The paper first consider exact NC and compute the corresponding local RLCT under certain assumptions, which lead to a generalization bound. Then, for non-exact NC, the authors show that the LLC of the model can be approximated by the LLC of the exact NC model. Then under the assumptions that the LLC is close to the RLCT, the authors derive Baysian generalization bounds using the RLCT.
Critical strengths
This paper studies the relation between the neural collapse phenomenon and the generalization ability of the neural networks, which I think is a crucial problem. This paper applies singular learning theory, which I think is a new angle for studying this problem.
Critical weaknesses
-
As a reader that is familiar with neural collapse literature, but not singular learning theory, this paper is hard to understand.
- The term RLCT is frequently used in the main text, but neither properly defined nor discussed before its usage.
- In Theorem 4.5, it is not clear how the and its dimension relate to the actual neural network parameters defined in section 3.1.
- Theorem 4.9 "assume the statistical compatibility conditions of Appendix A.4", but these assumptions are not properly defined in the main text. While I understand the page limit, I suggest the authors consider re-structure the paper, and put necessary definitions and discussion in the main text.
-
Several results in this paper relies on assumptions that are not properly justified. Thus, it is hard to understand the scope of the results of this paper.
- Theorem 4.5 and 4.6 assume certain structure on the empirical loss (equivalently ), which is not justitfied. Could the authors explain more on: (1) are these conditions universally satisfied for the architecture in section 3.1 and all datasets? (2) if not, under what extra assumptions on the model architecture and datasets will these assumptions satisfied?
- Theorem 4.9 "assume the statistical compatibility conditions of Appendix A.4". However, these assumptions are again not justified. Could the the authors again explain: (1) are the statistical compatibility conditions universally satisfied for the architecture in section 3.1 and all datasets? (2) if not, under what extra assumptions on the model architecture and datasets will these assumptions satisfied?
Questions for authors
Please also refer to the "Weaknesses" part.
- Could the authors explain more on the connection between Proposition 4.3 and Theorem 4.5?
6 comments
Follow-up 1: Clarifying the Theoretical Presentation (Weakness 1 and Question 1)
We agree that several key concepts and assumptions were not sufficiently explained in the main text. We will reorganize the theoretical presentation to address the following points.
-
Definition of RLCT. We will introduce RLCT before its first use and explain its meaning as a measure of how the volume of low-loss parameter configurations scales near a solution. We will also clarify its role in connecting local loss complexity to Bayesian generalization.
-
Meaning of the coordinates in the original Theorem 4.5. The coordinates are intended to represent quadratic, higher-order, and locally flat directions in the assumed normal form, with and denoting the dimensions of the first two blocks. However, these coordinates were not explicitly constructed from the original network parameters . We will clarify their intended interpretation and this limitation in the revised manuscript.
-
Connection between Proposition 4.3 and Theorem 4.5. The original Proposition 4.3 demonstrates higher-order loss behavior along a particular perturbation curve, but does not establish the separable normal form assumed in the original Theorem 4.5. We will make this relationship explicit and replace the assumed normal-form argument in the main text with a direct analysis of the network's prediction errors, as described in Follow-up 2.
We will also move the essential statistical compatibility conditions of Theorem 4.9 from Appendix A.4 to the main text. Their meaning and justification are addressed in Follow-up 3.
Follow-up 2: Justification and Scope of the Loss-Structure Assumptions (Weakness 2)
The original loss-structure assumptions were not shown to hold for all parameter configurations and datasets in the setting of Section 3.1. To address this concern, we replace the assumed normal form with a direct RLCT calculation under explicit architectural and data conditions.
-
Model and dataset conditions. We consider a fully trainable, one-hidden-layer cubic PNN, with and hidden width five. All 40 parameters are trainable. The inputs have the class-structured form , where is invertible. We require positive sample weights and classwise degree-three polynomial unisolvence, which is satisfied by our Pascal Cloud construction and other datasets meeting the same rank condition.
-
Direct RLCT derivation. Instead of assuming a loss normal form, we express the complete prediction errors through the residual ideal
where is the classifier, is the centered target matrix, contains the hidden class means, and contains the within-class feature coefficients. The unisolvence condition makes this ideal equivalent to the empirical residual ideal. Analyzing the full 40-dimensional parameter space gives
\begin{cases} 19/2,&\mathrm{NC1\ holds},\\\\ 13,&\mathrm{NC1\ fails}, \end{cases}$$ for every zero-residual solution sufficiently near our constructed exact-NC solution. This calculation does not require the original normal-form assumption or the width condition of Proposition 4.3. -
Applicability beyond the constructed dataset. The same RLCT classification holds for classwise degree-three-unisolvent finite datasets and transfers to fixed, nondegenerate class-conditional Gaussian input distributions within the specified model family. We also establish strict RLCT separation near constructed solutions at widths five through nine, although the exact two-value classification is specific to width five.
Thus, we do not claim that the original normal-form assumption holds universally. Instead, the revised results establish the local RLCT directly under stated model and data conditions. We will present these conditions and the corresponding derivation in the revised main text, with technical details in the appendix.
Main Comment: Clarifying Theoretical Assumptions and Statistical Justification
Thank you for your constructive feedback. We agree that the theoretical assumptions, their applicability, and the connection between the original Proposition 4.3 and the original Theorem 4.5 require clearer explanations. We have substantially strengthened our theoretical analysis to address these concerns.
Addressing Weakness 1 (Clarity of theoretical presentation): We will define RLCT and explain its role in our analysis before using it in the main text. We will also clarify the meaning of the coordinates (u,v,ξ) and dimensions (r,q) in the original Theorem 4.5, including their relationship to the neural network parameters. Furthermore, we will move the essential statistical compatibility conditions of Theorem 4.9 from the appendix to the main text.
Addressing Weakness 2 (Assumptions of Theorems 4.5 and 4.6): We agree that the original loss-structure assumption was not sufficiently justified by Proposition 4.3. Rather than assuming this structure, we now derive the local RLCT directly from the full network loss in an explicit, fully trainable PNN. This yields an exact local RLCT classification under specified architectural and data conditions, without relying on the original normal-form assumption.
Addressing Weakness 2 (Statistical assumptions of Theorem 4.9): We clarify that these assumptions are not universally guaranteed. We establish the population RLCT correspondence directly in our Gaussian PNN and separately examine the additional empirical LLC compatibility and scaling conditions required by Theorem 4.9. We will explicitly distinguish the conditions we have established from those that remain assumptions.
Follow-up 1 clarifies the original coordinates and the relationship between Proposition 4.3 and Theorem 4.5. Follow-up 2 addresses the loss-structure assumptions through a direct RLCT derivation, while Follow-up 3 explains the statistical assumptions and their justification in Theorem 4.9.
In the revision, we will clarify the definitions of RLCT and LLC, explain the original normal-form coordinates and their limitations, and present the necessary assumptions of Theorem 4.9 more explicitly in the main text.
Further details on the derivations, applicability conditions, and statistical justification are provided in the follow-up comments below.