acceptodds
Under review as a conference paper at ICLR 2027

Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix

Abstract

The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top- subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.

Primary area:
optimization
Keywords:
Hessian, Subspace Stabilization, Gradient Covariance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dWWvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the stabilization of the top eigenspaces of the Hessian and gradient covariance matrix during neural network training. The authors first identify a limitation of the conventional subspace overlap metric and introduce a normalized instability metric to better characterize subspace stabilization. Through experiments, they observe that the top-(C−1C-1) subspaces of the Hessian and gradient covariance matrix exhibit strong alignment and stabilization in certain classification settings. To explain this phenomenon, they identify a subspace spanned by centered class-wise gradients and analyze its evolution under gradient flow. The theoretical results relate the rate of subspace change to the Hessian curvature outside the identified subspace. The paper also examines a setting where stabilization fails and discusses possible explanations for this behavior.

Critical strengths
  • The paper studies an interesting and meaningful problem. Understanding why the Hessian's top subspace stabilizes during training could provide useful insights into neural network optimization and learning dynamics.

  • I like the motivation behind the proposed instability metric. The authors identify a limitation of the commonly used overlap metric and provide experimental evidence showing that the new metric can better distinguish different stabilization behaviors.

  • The empirical connection between the Hessian, gradient covariance matrix, and centered class-wise gradients is interesting. In particular, the observation that a relatively simple class-gradient subspace can closely align with the Hessian's top subspace is quite insightful.

Critical weaknesses
  • My main concern is that the theoretical analysis does not fully explain the stabilization of the Hessian's top subspace. The main results characterize the evolution of the centered class-gradient subspace, but the connection between this subspace and the actual Hessian top eigenspace is still largely supported by empirical observations. Thus, the theoretical explanation of the paper's central question remains incomplete.

  • The discussion of stabilization failure in Section 4.2 is somewhat limited. While the authors observe weaker subspace alignment and less separation between the top and bulk eigenvalues, it remains unclear why these changes occur in the third experimental setting.

  • The connection between the proposed instability metric and the theoretical analysis could be clearer. The metric measures subspace changes relative to the magnitude of parameter updates, whereas the theoretical results mainly bound the absolute rate of subspace change. It is therefore unclear whether the theoretical results directly explain the stabilization measured by the proposed metric.

Questions for authors

See weaknesses above.

2 comments

Thank you for the detailed review and constructive feedback. We are glad that you find our paper studying an important problem, identified limitation of existing metric, which motivated our new metric, and propose and interesting connection between the Hessian, gradient covariance matrix, and centered class-wise gradients. We respond to the weaknesses below:

The connection between this subspace and the actual Hessian top eigenspace is still largely supported by empirical observations. We agree that this is a major limitation of our paper. In the general response we have outlined which part is proved and which part is assumed. Although we did not provide a rigorous proof of the alignment, in Section 4 we try to justify this by using the existing result from Papyan (2020). In particular, Papyan (2020) attributes the top-CC subspace of the Hessian to the G^\hat{\mathbf{G}} matrix defined in Section 4.1 of our paper. We justify that the centered-class gradient subspace is close to the subspace of G^\hat{\mathbf{G}} by gradually constructing approximations in Theorem 3 and 4. Admittedly, in the previous version our analysis stopped after showing span(g1,…,gC)\text{span}(\mathbf{g}_1,\dots,\mathbf{g}_C) is close to the span of G^\hat{\mathbf{G}}, which implies that span(g1,…,gC)\text{span}(\mathbf{g}_1,\dots,\mathbf{g}_C) is close to the Hessian's top-CC subspace, under the condition that Papyan 2020's setup holds. In the updated version, we strenghten the link to our identifier with the center-class gradient by proving a new theorem (Theorem 5) that bounds the overlap between the top-(C−1)(C-1) subspace of the matrix with rows gc\mathbf{g}_c and our identifier span(g1−g,…,gC−g)\text{span}(\mathbf{g}_1 - \mathbf{g},\dots,\mathbf{g}_{C} - \mathbf{g}), which further strenghtens the connection between the centered-class gradient subspace that we show to have a slow movement, and the top Hessian subspace.

Discussion of stabilization failure is limited. Thank you for the suggestion to make further discussion of the stabilization failure. We believe that adding batch norm changed the learning behavior of the neural network by re-weighting the importance of different layers. Based on this hypothesis, we investigated the percentage of the top-CC eigenvectors distributed on the last layer (experimental results provided in Appendix B.3 of our updated paper), and found that compared with the successful stabilization settings, CNN with BN's top eigenvector are less concentrated on the last layer, possibly due to the norm-growing property of the weight matrices in the scaling invariant layers before BN is applied. Although this may not fully explain the failure of the stabilization, we believe that it provides another clue of why stabilization failure occurs.

Connection between proposed instability metric and the theoretical analysis could be clearer. In our gradient flow analysis, we bound the magnitude of the time-derivative of the subspace ∣P˙(t)∣\\|\dot{\mathbf{P}}(t)\\|. This quantity can be seen as lim⁡η→0∣P(θt+1)−P(θt)∣η=2K∣∇L(θt)∣2⋅lim⁡η→0Instability(θt+1,θt)\lim_{\eta\rightarrow 0}\frac{\\|\mathbf{P}(\mathbf{\theta}_{t+1}) - \mathbf{P}(\mathbf{\theta}_t)\\|}{\eta} = \sqrt{2K}\\|\nabla \mathcal{L}(\mathbf{\theta}_t)\\|_2\cdot \lim_{\eta\rightarrow 0}\text{Instability}(\mathbf{\theta}_{t+1},\mathbf{\theta}_t). This gives us ∣P˙(t)∣2K∣∇L(θt)∣2=lim⁡η→0Instability(θt+1,θt)\frac{\\|\dot{\mathbf{P}}(t)\\|}{\sqrt{2K}\\|\nabla \mathcal{L}(\mathbf{\theta}_t)\\|_2} = \lim_{\eta\rightarrow 0}\text{Instability}(\mathbf{\theta}_{t+1},\mathbf{\theta}_t). Therefore, Theorem 1 can be transformed into lim⁡η→0Instability(θt+1,θt)≤2σ(G(t))2(∣(I−P(t))H(t)g(t)∣22∣g(t)∣22+1C∑c=1C−1∣(I−P(t))Hc(t)g(t)∣22∣g(t)∣22)\lim_{\eta\rightarrow 0}\text{Instability}(\mathbf{\theta}_{t+1},\mathbf{\theta}_t) \leq \frac{2}{\sigma(\mathbf{G}(t))^2}(\frac{\\|(\mathbf{I} - \mathbf{P}(t))\mathbf{H}(t)\mathbf{g}(t)\\|_2^2}{\\|\mathbf{g}(t)\\|_2^2} + \frac{1}{C}\sum_{c=1}^{C-1}\frac{\\|(\mathbf{I} - \mathbf{P}(t))\mathbf{H}_c(t)\mathbf{g}(t)\\|_2^2}{\\|\mathbf{g}(t)\\|_2^2}).

Thank you so much for the response. However, my main concerns remain unresolved for the following reasons:

  1. Theorem 5 assumes the CC-th singular value σC(Q)\sigma_C(\mathbf{Q}) is very small (since ∥g∥2\|\mathbf{g}\|_2 is small), which conflicts with Theorem 4 where σC(G′)2\sigma_C(\mathbf{G}')^2 in the denominator must be large for the bound to hold. Furthermore, the link between G^\hat{\mathbf{G}} and the actual Hessian top eigenspace remains empirical.
  2. Aside from a missing square on the left-hand side of your formula, dividing Theorem 1 by ∥g(t)∥2\|\mathbf{g}(t)\|_2 cancels out the small gradient norm ∥g(t)∥2\|\mathbf{g}(t)\|_2 in the numerator. Figure 4 (Right) still plots the unnormalized ∥(I−P)Hg∥2\|(\mathbf{I}-\mathbf{P})\mathbf{H}\mathbf{g}\|_2 and ∥(I−P)Hcg∥2\|(\mathbf{I}-\mathbf{P})\mathbf{H}_c\mathbf{g}\|_2, which are small largely because ∥g(t)∥2\|\mathbf{g}(t)\|_2 itself is small (Figure 4, Left), rather than validating the normalized bound for instability.

Please feel free to correct me if there is anything wrong in my response.

Reviewer_tHEwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the Hessian subspace stabilizes during training, and if so, why it happens. The authors report that the standard overlap metric goes to 1 whenever the step size is small. They propose an instability metric that divides the subspace change by the parameter displacement. They then observe that the top-(C−1) subspace of the gradient covariance Σ\Sigma closely matches the Hessian’s. Using a between-/within-class split, they identify Σ\Sigma's top subspace as the span of centered class gradients {g_c − g}. Under gradient flow, they bound how fast this span rotates by the Hessian’s norm in the bulk subspace (Theorems 1–2). They also connect the span to Papyan’s (2020) cross-class subspace (Theorems 3–4). Experiments cover an MLP on MNIST, a 2-layer CNN on a CIFAR-10 subset, and a 3-layer CNN with BatchNorm, where stabilization fails.

Critical strengths
  • The instability metric fixes a real confound. Eq. 2 shows clearly why overlap alone can’t separate a stable subspace from a slow optimizer. Plotting both metrics together is a sensible protocol.

  • {g_c − g} is a simple, interpretable identifier for the subspace. Fig. 3 shows near-perfect overlap with Σ\Sigma's top-(C−1) subspace in the stable settings. It is an exact rank-(C−1) characterization, not just a covering set.

  • The alignment between the Hessian’s and Σ\Sigma's top subspaces is a useful observation, since Σ\Sigma is much cheaper to estimate.

Critical weaknesses
  • Scale is very small. The experiments use two- and three-layer networks on 5k–10k-sample subsets, plain constant-LR SGD with no momentum or augmentation, C = 10, and 4 seeds.

  • Regarding the instability metric

     - It divides by the displacement between checkpoints 100–300 steps apart, which is not the same as path length
     - It is not scale-invariant. This matters for BN networks, whose weights are scale-invariant and whose effective learning rate drifts.
    
  • The explanation is partly circular and the Hessian link is heuristic:

     -   Theorem 1 bounds the rotation rate given a small bulk norm and a large $\Sigma_min(G)$. The outlier–bulk separation is assumed, not derived, and the authors list it as open question (iii)
    - The gradient-flow analysis also ignores SGD noise, even though the experiments use minibatch SGD
    

Minor typos and notation irregularities in theorems and an unfinished sentence at the end of the paragraph before Section 3.1

Questions for authors
  • How sensitive is the instability metric to checkpoint spacing and to parameter norm? Would a normalized version change the BN result?
  • Is the BN failure explained by BN suppressing Hessian outliers, as reported by Ghorbani et al. (2019)? Does removing BN from the 3-layer CNN restore stabilization, separating depth from normalization?
  • How tight is Theorem 1 in practice?
3 comments

Thank you for the detailed review and the constructive feedback. We are glad that you see our stability metric fixing a confound in previous study, that the centered class gradient subspace is a simple and interpretable identifier for the top Hessian subspace, and that the alignment between top Hessian and gradient covariance subspace is a useful observation. We respond to the weaknesses you raised below:

Scale is small. We acknowledge that the scale of the experiments we perform in the paper is quite small. To perform the detailed analysis (instead of statistical estimation) on the eigenvalue and eigenvectors of the Hessian, the computation cost will grow quadratically as the number of parameters, which makes it hard to perform large scale experiments. To make our result closer to practice, we included an additional experiment setting where we train a 4-layer nanoGPT model on the TinyShakespear dataset. The result further support our claim, and we have included the details in Appendix B.1.

Stability metric is not divided by path length but by end-point difference. Thank you for pointing out this potential ambiguity. We would like to clarify that we made the explicit choice to divide the subspace change by end-point difference. Consider two steps of updates θt+1=θt+v1\mathbf{\theta}_{t+1} = \mathbf{\theta}_t + \mathbf{v}_1 and θt+2=θt+1+v2\mathbf{\theta}_{t+2} = \mathbf{\theta}_{t+1} + \mathbf{v}_2. If, in an noisy oscillation scenario v1≈v2\mathbf{v}_1 \approx \mathbf{v}_2, then θt+2≈θt\mathbf{\theta}_{t+2} \approx \mathbf{\theta}_{t}. In this case, we should expect the subspace at t+2t+2 to be relatively close to tt, but due to the small difference between θt+2\mathbf{\theta}_{t+2} and θt\mathbf{\theta}_t. If we divide the subspace change by the path-length instead of the end-point difference, then the noisy oscillation magnitude ∣v1∣\\|\mathbf{v}_1\\| will be come a new confound in the metric.

Stability metric is not scale-invariant. We agree that our metric is not scale-invariant. However, we need to clarify that we design our metric to respect the influence from model architecture or initialization on the property of the Hessian. We believe that much of the value of our work lies in optimization. BN and its resulting effective learning rate naturally changes the optimization property, and we wish our metric can respond to that change instead of staying the same. From a practical perspective, the scale-invariant natural does not change the conclusion we obtain from our current metric: for parameter W\mathbf{W} in a scale-invariant layer, the corresponding Hessian scales according to ∣W∣−2\\|\mathbf{W}\\|^{-2} while gradient scales according to ∣W∣−1\\|\mathbf{W}\\|^{-1}. Therefore, the instability metric scales roughly as ∣W∣−1\\|\mathbf{W}\\|^{-1}. Throughout the training, ∣W∣\\|\mathbf{W}\\| in the scale-invariant layer can only grow, so any potential confounding factor caused by the scale-invariance should only make the instability smaller. However, we still observe great instability in this scenario, which means that the major disturbance force out-shadows the potential confounding factor. Nevertheless, we agree that when one aims to study the Hessian top subspace stability in a scale-invariant scenario, having metric that scale Hessian and gradient blocks accordingly can be a good option.

In Theorem 1, the outlier-bulk separation is assumed, not proved. We acknowledge this limitation of our work. However, we would like to point out that the outlier-bulk separation is better understood as a separate mechanism, while our paper's focus is to understand the subspace stabilization under such condition. This is because the outlier-bulk separation can be tied closely to the progressive sharpening behavior, which can be related to the model's feature learning dynamic. Therefore, justifying the outlier-bulk separation may require a separate set of assumptions and a more careful analysis of the initial dynamic of the optimizers. We agree that this is an interesting open problem, but might lie outside the focus of our paper.

The gradient-flow analysis also ignores SGD noise. Thank you for the suggestion for enhancing the analysis with stochastic gradient instead of vanilla gradient. We notice that our analysis can be easily extended to the case of stochastic gradient, and we have made the changes to include the stronger statement in the revised version. Since the Hessian top subspace also aligns tightly with the gradient covariance subspace, the projection (I−P(t))ξt(\mathbf{I} - \mathbf{P}(t))\mathbf{\xi}_t will also have a small magnitude due to separation between gradient covariance's top and bulk subspace eigenvalues. We have also added the corresponding discussion at the end of Section 3.

Below we respond to your questions:

Sensitivity of the metric. For small checkpoint spacing, the numerator ∣P(θ)−P(θ′)∣F\\|P(\theta) - P(\theta')\\|_F is first-order in ∣θ−θ′∣2\\|\theta - \theta'\\|_2. Dividing by the displacement therefore removes the leading dependence on spacing, so the metric approximates the rate of change of the projector along the direction of motion, although there might be numerical stability issues. For larger spacing it becomes an average over the window. We did not run a dedicated spacing ablation (we used 100 steps for MNIST and 300 steps for CIFAR-10). However, the overlap with the final checkpoint in Appendix B.2 involves neither a spacing choice nor a division by displacement, and it gives the same qualitative conclusion in all three settings. Regarding parameter norm, ss discussed in our reply on scale invariance above, the Hessian block of a scale-invariant layer scales as ∣W∣−2\\|W\\|^{-2} and its gradient as ∣W∣−1\\|W\\|^{-1}, so to first order our metric is scaled by ∣W∣−1\\|W\\|^{-1}. Measuring displacement in normalized coordinates would multiply the instability by roughly ∣W∣\\|W\\|, which grows during training. We would therefore expect a normalized version to report larger, not smaller, instability in the BN setting. We have not computed this variant, so this is an argument rather than a measurement.

Failture in the BN case. Our observations are consistent with that explanation. In the BN setting, the separation between the top-(C−1)(C-1) outliers and the bulk largely disappears for both HH and Σ\Sigma (Figure 2, third row), and this separation is what Theorems 1–2 rely on. Appendix B.3 adds that the top eigenvectors in this setting carry little mass on the last layer. However, we have not run the 3-layer CNN without BN. The 2-layer CNN that does stabilize differs from the 3-layer one in both depth and normalization, so our current experiments do not separate the two factors. We will state this explicitly in Section 4.2 and point to Ghorbani et al. (2019) as a possible explanation.

Tightness of Theorem 1. We did not plot ∣P˙(t)∣F\\|\dot{P}(t)\\|_F directly against the bound, so we cannot quote a tightness ratio. We expect the bound to be loose by constant factors, because the proof i). uses the triangle inequality on X+X⊤X + X^\top; ii). bounds ∣(GG⊤)−1G∣2\\|(GG^\top)^{-1}G\\|_2 by the worst direction, 1/σmin⁡(G)1/\sigma_{\min}(G); and iii). splits the (H−Hc)g(H - H_c)g terms without using possible cancellations. The theorem is meant to identify what controls the rotation rate (curvature outside the identifier relative to the spread of the class gradients), not to predict the exact rate.

Thank you so much for the response. The rebuttal does clarify the design of the metric and adds an experiment, but some of the concerns are still unexplained. The BN case has still been left unexplained without the no-BN ablation. The tightness of the bound is not measured and the outlier bulk separation explanation is based on an assumption. Please correct me if I have misread any of these points or missed something in the rebuttals.

Reviewer_TAp5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why the leading Hessian subspace can become stable early in training. The authors first point out a problem with the usual measure of stability, two consecutive subspaces can look almost identical simply because the parameter update is small. They propose an instability measure that divides the change in subspace by the distance moved in parameter space. For classification, the paper then studies the subspace formed by each class’s average gradient after subtracting the overall average gradient. In some of the tested models, this space closely matches the leading Hessian subspace. The authors prove bounds on how quickly the class gradient subspace changes under gradient flow, and relate those bounds to curvature outside that subspace.

Critical strengths
  • The paper identifies a real ambiguity in the usual consecutive checkpoint overlap measure. A high overlap can result from a small parameter update, so it does not by itself show that the subspace is insensitive to parameter movement. The proposed measure addresses this specific issue.

  • The paper combines this observation with conditional theory, and its 3 layer CNN example usefully shows that the proposed picture does not fit every training setting.

Critical weaknesses
  • Theorem 1 bounds the movement of the centered class gradient subspace, not the leading Hessian subspace itself. The empirical alignment in Figure 3 is informative and positive, but there isn't a bound on their difference over training, which makes it hard to tell how much of the Hessian stabilization the theory actually explains.

  • The evidence for calling the 9 dimensional subspace “stable” is mainly a comparison of curves in Figure 1. The paper doesn't give a numerical criterion for stability, and the measure describes change between sampled checkpoints rather than how far the subspace drifts over the whole training run. The experiments support a local stability claim in the MLP and 2 layer CNN, but the scope of that claim could be stated more clearly. I think a direct summary of longer range subspace movement in the existing runs would help interpret the central result.

Questions for authors
  • Can you quantify how close the centered class gradient subspace must remain to the leading Hessian subspace for the bound in Theorem 1 to explain the Hessian result? If such a guarantee is not available, could you state more clearly which part of the Hessian explanation is theoretical and which part is empirical?

  • What concrete criterion do you use to call the C-1 subspace stable in Figure 1? Could you also report its change from an early checkpoint to a much later one, alongside the consecutive-checkpoint measure?

1 comment

Thank you for the positive feedback. We really appreciate it that you find our paper identifying a real confounding factor in existing measure of subspace stabilization, and acknowledge our empirical and theoretical contributions. We respond to the weaknesses you raised below.

No bound on the difference between centered-class gradient subspace and the Hessian top subspace. We agree that this is a major limitation of our paper. Although we did not provide a rigorous proof of the alignment, in Section 4 we try to justify this by using the existing result from Papyan (2020). In particular, Papyan (2020) attributes the top-CC subspace of the Hessian to the G^\hat{\mathbf{G}} matrix defined in Section 4.1 of our paper. We justify that the centered-class gradient subspace is close to the subspace of G^\hat{\mathbf{G}} by gradually constructing approximations in Theorem 3 and 4. Admittedly, in the previous version our analysis stopped after showing span(g1,…,gC)\text{span}(\mathbf{g}_1,\dots,\mathbf{g}_C) is close to the span of G^\hat{\mathbf{G}}, which implies that span(g1,…,gC)\text{span}(\mathbf{g}_1,\dots,\mathbf{g}_C) is close to the Hessian's top-CC subspace, under the condition that Papyan 2020's setup holds. In the updated version, we strenghten the link to our identifier with the center-class gradient by proving a new theorem (Theorem 5) that bounds the overlap between the top-(C−1)(C-1) subspace of the matrix with rows gc\mathbf{g}_c and our identifier span(g1−g,…,gC−g)\text{span}(\mathbf{g}_1 - \mathbf{g},\dots,\mathbf{g}_{C} - \mathbf{g}), which further strenghtens the connection between the centered-class gradient subspace that we show to have a slow movement, and the top Hessian subspace.

No numerical criteria for stability. We agree that whether subspace stabilization occurs has a vague criteria. Our perspective on this is that, instead of classifying the behavior into a binary "stabilized-or-not-stabilized" scenario, it might be better to understand the stabilization as a continuous metric, which is measured by our proposed metric. In the meantime, we agree that providing data over a longer range of subspace movement can be beneficial. In Appendix B.2 of our updated paper, we plot the overlap between the subspace at each checkpoint and the subspace in the final checkpoint. The behavior aligns much with the observation based on our metric: in the MLP and 2-layer CNN case, the overlap with the final checkpoint reached 0.8 early in training, where in the 3-layer CNN case (where stabilization does not happen), we see a constant speed increase of the overlap value.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviews

We thank all reviewers for their constructive feedback. We are happy that the reviewers find that: our instability metric fixes a real confound in the overlap metric, which cannot tell a stable subspace from a small update (TAp5, tHEw, dWWv); the centered class-gradient span spangc−g\text{span}\\{g_c - g\\} is a simple, interpretable identifier and an exact rank-(C−1)(C-1) characterization (tHEw, dWWv); the alignment between the top subspaces of the Hessian and the gradient covariance is useful (tHEw, dWWv); and the 3-layer CNN shows that the picture does not fit every setting being a valuable observation (TAp5).

Regarding the shared concern that the theory bounds the identifier, not the Hessian’s top subspace (TAp5, dWWv). We agree that this is the main limitation of the paper. Following Reviewer TAp5’s suggestion, we state explicitly which parts of the argument are theoretical and which are empirical.

Theoretical (under the stated assumptions):

  • Theorems 1–2: under (stochastic) gradient flow, the rotation rate of spangc−g\text{span}\\{g_c - g\\} is controlled by HH and HcH_c outside this subspace.
  • Theorem 3: the subspace span(G^)\text{span}(\hat{G}) that Papyan (2020) associates with the Hessian outliers equals spang1′,…,gC′\text{span}\\{g'_1, \dots, g'_C\\}.
  • Theorem 4: spangc′\text{span}\\{g'_c\\} and spangc\text{span}\\{g_c\\} are close.
  • Theorem 5 (new): In the previous version, this chain stopped at the CC-dimensional spang1,…,gC\text{span}\\{g_1, \dots, g_C\\}, which left a dimension mismatch with our (C−1)(C-1)-dimensional identifier. Theorem 5 closes this step. Let QQ be the matrix whose rows are gcg_c. Assume an eigengap condition on QQ and a full gradient gg that is small relative to the class gradients. We show that the top-(C−1)(C-1) right singular subspace of QQ nearly coincides with spang1−g,…,gC−g\text{span}\\{g_1 - g, \dots, g_C - g\\}. In other words, centering removes only the weakest direction of spangc\text{span}\\{g_c\\}.

Empirical, or taken from prior work:

  • The Hessian–G^\hat{G} link that the Hessian’s top subspace is captured by span(G^)\mathrm{span}(\hat{G}) rests on the class/cross-class structure identified by Papyan (2020). Instead of proving it, we verify it in our settings (Figure 3, second column; Figure 7 for the new nanoGPT experiment).
  • The assumptions of Theorems 4–5. A small ∥g∥2\|g\|_2 relative to ∥gc∥2\|g_c\|_2 and a well-separated bulk are checked empirically.

Other new results in the revision:

  • Stochastic gradient flow: Theorem 1 now covers stochastic gradients. The end of Section 3 discusses why the noise term is small when the top subspaces of HH, HcH_c and Σ\Sigma align (tHEw). nanoGPT on TinyShakespeare (Appendix B.1): The same qualitative picture holds: stabilization, alignment between the Hessian and Σ\Sigma, and our identifier predicting both subspaces.
  • Overlap with the final checkpoint (Appendix B.2): The MLP and 2-layer CNN approach their final subspace much earlier than the 3-layer CNN with BN, consistent with the instability metric (TAp5).
  • Last-layer mass of the top eigenvectors (Appendix B.3): In the failure case, this mass decays well below the last layer’s share of parameters, unlike in the stable settings. We offer this as a clue, not a full explanation (dWWv).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.