- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the stabilization of the top eigenspaces of the Hessian and gradient covariance matrix during neural network training. The authors first identify a limitation of the conventional subspace overlap metric and introduce a normalized instability metric to better characterize subspace stabilization. Through experiments, they observe that the top-() subspaces of the Hessian and gradient covariance matrix exhibit strong alignment and stabilization in certain classification settings. To explain this phenomenon, they identify a subspace spanned by centered class-wise gradients and analyze its evolution under gradient flow. The theoretical results relate the rate of subspace change to the Hessian curvature outside the identified subspace. The paper also examines a setting where stabilization fails and discusses possible explanations for this behavior.
Critical strengths
-
The paper studies an interesting and meaningful problem. Understanding why the Hessian's top subspace stabilizes during training could provide useful insights into neural network optimization and learning dynamics.
-
I like the motivation behind the proposed instability metric. The authors identify a limitation of the commonly used overlap metric and provide experimental evidence showing that the new metric can better distinguish different stabilization behaviors.
-
The empirical connection between the Hessian, gradient covariance matrix, and centered class-wise gradients is interesting. In particular, the observation that a relatively simple class-gradient subspace can closely align with the Hessian's top subspace is quite insightful.
Critical weaknesses
-
My main concern is that the theoretical analysis does not fully explain the stabilization of the Hessian's top subspace. The main results characterize the evolution of the centered class-gradient subspace, but the connection between this subspace and the actual Hessian top eigenspace is still largely supported by empirical observations. Thus, the theoretical explanation of the paper's central question remains incomplete.
-
The discussion of stabilization failure in Section 4.2 is somewhat limited. While the authors observe weaker subspace alignment and less separation between the top and bulk eigenvalues, it remains unclear why these changes occur in the third experimental setting.
-
The connection between the proposed instability metric and the theoretical analysis could be clearer. The metric measures subspace changes relative to the magnitude of parameter updates, whereas the theoretical results mainly bound the absolute rate of subspace change. It is therefore unclear whether the theoretical results directly explain the stabilization measured by the proposed metric.
Questions for authors
See weaknesses above.
2 comments
Thank you so much for the response. However, my main concerns remain unresolved for the following reasons:
- Theorem 5 assumes the -th singular value is very small (since is small), which conflicts with Theorem 4 where in the denominator must be large for the bound to hold. Furthermore, the link between and the actual Hessian top eigenspace remains empirical.
- Aside from a missing square on the left-hand side of your formula, dividing Theorem 1 by cancels out the small gradient norm in the numerator. Figure 4 (Right) still plots the unnormalized and , which are small largely because itself is small (Figure 4, Left), rather than validating the normalized bound for instability.
Please feel free to correct me if there is anything wrong in my response.
Thank you for the detailed review and constructive feedback. We are glad that you find our paper studying an important problem, identified limitation of existing metric, which motivated our new metric, and propose and interesting connection between the Hessian, gradient covariance matrix, and centered class-wise gradients. We respond to the weaknesses below:
The connection between this subspace and the actual Hessian top eigenspace is still largely supported by empirical observations. We agree that this is a major limitation of our paper. In the general response we have outlined which part is proved and which part is assumed. Although we did not provide a rigorous proof of the alignment, in Section 4 we try to justify this by using the existing result from Papyan (2020). In particular, Papyan (2020) attributes the top-C subspace of the Hessian to the G^ matrix defined in Section 4.1 of our paper. We justify that the centered-class gradient subspace is close to the subspace of G^ by gradually constructing approximations in Theorem 3 and 4. Admittedly, in the previous version our analysis stopped after showing span(g1,…,gC) is close to the span of G^, which implies that span(g1,…,gC) is close to the Hessian's top-C subspace, under the condition that Papyan 2020's setup holds. In the updated version, we strenghten the link to our identifier with the center-class gradient by proving a new theorem (Theorem 5) that bounds the overlap between the top-(C−1) subspace of the matrix with rows gc and our identifier span(g1−g,…,gC−g), which further strenghtens the connection between the centered-class gradient subspace that we show to have a slow movement, and the top Hessian subspace.
Discussion of stabilization failure is limited. Thank you for the suggestion to make further discussion of the stabilization failure. We believe that adding batch norm changed the learning behavior of the neural network by re-weighting the importance of different layers. Based on this hypothesis, we investigated the percentage of the top-C eigenvectors distributed on the last layer (experimental results provided in Appendix B.3 of our updated paper), and found that compared with the successful stabilization settings, CNN with BN's top eigenvector are less concentrated on the last layer, possibly due to the norm-growing property of the weight matrices in the scaling invariant layers before BN is applied. Although this may not fully explain the failure of the stabilization, we believe that it provides another clue of why stabilization failure occurs.
Connection between proposed instability metric and the theoretical analysis could be clearer. In our gradient flow analysis, we bound the magnitude of the time-derivative of the subspace ∣P˙(t)∣. This quantity can be seen as limη→0η∣P(θt+1)−P(θt)∣=2K∣∇L(θt)∣2⋅limη→0Instability(θt+1,θt). This gives us 2K∣∇L(θt)∣2∣P˙(t)∣=limη→0Instability(θt+1,θt). Therefore, Theorem 1 can be transformed into limη→0Instability(θt+1,θt)≤σ(G(t))22(∣g(t)∣22∣(I−P(t))H(t)g(t)∣22+C1∑c=1C−1∣g(t)∣22∣(I−P(t))Hc(t)g(t)∣22).