It has been experimentally shown that the Hessian matrix of the training loss of a neural network is well-approximated by its diagonal blocks, but the reason why this occurs is not known. We prove a set of lemmas that establish that the sharpness of the Hessian is approximately equal to the sharpness of its block diagonal approximation provided that the forward projections of the eigenvectors are approximately colinear and their Gram matrix is low-rank. We show experimentally that these conditions hold and are to be expected given the structure of the Hessian matrix, establishing why the Hessian matrix is near-block-diagonal in practice.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.Reviewer_p57cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
The paper studies whether layerwise curvature captures the dominant Hessian direction. It derives the Gauss-Newton bound and argues that near equality supports approximating the full Hessian's top eigenvector by a weighted concatenation of the layerwise top eigenvectors.
Critical strengths
The layerwise bound is valid, and the second-moment formulation helps explain how the layers contribute to curvature.
The experiments cover several architectures and training settings. Table 3 also checks eigenvector stability across data subsets.
Critical weaknesses
The step from eigenvalues to eigenvectors needs justification. My main concern is the near-equality argument in Section 5 (p. 9). For 0<ε<1/2, take
1&0&0&0 \\
0&1-\varepsilon&0&1-\varepsilon\\
0&0&1&0\\
0&1-\varepsilon&0&1-\varepsilon
\end{pmatrix}.$$
Both diagonal blocks have top eigenvector $(1,0)^T$, so any weighted concatenation is $(c_1,0,c_2,0)^T$. However,
$$v_1(G_\varepsilon)=\frac{(0,1,0,1)^T}{\sqrt2},
\qquad
\frac{\lambda_1(G_\varepsilon)}{\sum_j\lambda_1(G_{jj})}=1-\varepsilon.$$
The bound can be arbitrarily tight while the concatenation is orthogonal to the true direction.
Lemma 3.2 (p. 4) also fails on this example. Let $a$ take $\sqrt3e_1$, $\sqrt3e_3$ and $\sqrt{3(1-\varepsilon)}(e_2+e_4)$ with equal probability, so $G_\varepsilon=\mathbb E[aa^T]$. The local top projections satisfy the lemma's pointwise collinearity condition, but the global top blocks point along $(0,1)^T$. The proof on p. 13 assumes that optimality in expectation gives a pointwise norm bound. This step is incorrect. Corollary 3.1 has an additional rank-one assumption, which this example does not satisfy.
The normalisation of ρ needs to be stated consistently. Section 3.1 normalises each block separately. Section 7.2.1 (p. 14) uses the restricted global eigenvector in sj=vjTGjjvj. For ∣v1(G)∣=1, this gives
\qquad \rho=\frac{\lambda_1(G)}{\sum_js_j}\ge1,$$
whereas Table 1 reports values below one. Please state the normalisation used in the code.
Questions for authors
How should Lemma 3.2 be corrected? For the direction claim, let UM contain the unit top eigenvectors of M's diagonal blocks, padded with zeros. Please report
SM=∥UMTv1(M)∥2
for M=G and the full Hessian on existing checkpoints. This is the best squared cosine over all block weights. Table 3 measures sampling stability, not this overlap. Use the corresponding top eigenspaces for tied eigenvalues.
What formulas are used for ρ and the component ratio, including block normalisation?
The paper studies why a block-diagonal approximation of the Hessian can give good results when optimizing neural architectures. In particular, the paper proposes a template for modeling neural architectures which is claimed to encompass multilayer perceptrons, convolutional neural networks, and transformers. Under this modeling choice, the paper studies the Hessian of nL-layered architectures through its classical Gauss-Newton-plus-residual decomposition. In particular, letting the Hessian be H, the authors use a decomposition H=G+H=G+Γ+V, where the non-caligraphic H denotes the residual term. They proceed to give an upper bound on λmax(G) in terms of the sum of top eigenvalues of individual diagonal blocks, and show that this bound is tight when the forward projections of the eigenvectors are approximately colinear and their Gram matrix is rank-one. Authors argue that under these conditions, the top eigenvector of the Hessian is approximately equal to the weighted concatenation of the eigenvectors of its diagonal blocks. A similar bound is derived for the Γ component. Numerical experiments are provided alongside the theory.
Critical strengths
None
Critical weaknesses
(see next section for an elaboration on the following points)
The paper is very poorly written (negligent language, mathematical notation and typesetting; idea presentation lacks structure; lack of results framing and discussion)
The paper's positioning within existing literature is essentially missing. The few citations serve only to motivate the broad question of block-diagonal Hessian approximation, but do not cover existing works in terms of their assumptions, precise results, respective limitations, and how all these aspects relate to the current work (what does the work improve on? are the assumptions more realistic here? If so, why? What is the main technical difficulty in establishing the accuracy of such an approximation for real-world models? etc).
The proofs are incomplete or contain mistakes
The reasoning behind the main claim is flawed
The experiments do not give sufficient weight to the theory
Questions for authors
All references cited below are at the end of this section.
Questions:
Why did the authors choose to further decompose H into Γ+V when only Γ is then analyzed?
The paper cited for the Hessian being primarily determined by G is old, and only claims this phenomenon near convergence. To justify your study of G throughout training, an argument has to be made about the evidence of G's accurate approximation along trajectories. In particular, [1], [2], and [3] porvide experiments/discussions around this theme, yet no mention is made of this (the works are cited, but for other resons).
Major concerns:
Technical correctness:
The eigenvector equality in Lemma 5.1 is not correct. The vectors v1(M) and v1(M) belong to spaces of different dimensions.
The proof of Lemma 3.1 is incorrect as written, and also incomplete. First off, the proof should discuss the corner case of one of the optimizers being null, i.e. ∃i such that Wi′=0. Secondly, eliminating opimization over W^i′ in the string of equalities in lines 624 --- 629 such as to arrive at the claim that d is the top eigenvector requires a careful argument which is currently missing. Those parameters cannot be fixed inside K′ without justifying that the optimum of the original problem does nto change. Finally, the claim in line 635 that W^i realize the maximum and the immediately following expression corresponding to componentwise maximums is incorrect, as no such thing was established or assumed.
One of the central claims of the paper is that "the top eigenvector of the Hessian is approximately equal to the weighted concatenation of the eigenvectors of its diagonal blocks provided that the forward projections of the eigenvectors are approximately colinear and their Gram matrix is rank-one". With the amendment that this claim in fact addresses only Gauss-Newton component of the Hessian decomposition, this claim is not formally proven but just stated as a mention following Corollary 3.1. Central theoretical claims must have formal proofs.
Methodology & claims:
The claim that the network model introduced at the beginning of section 2 encompasses multilayer perceptrons, convolutional neural networks, and transformers should be backed by proofs (can be deferred to the appendix). Firstly, this is because a priori it is not clear how all the architectural subtleties are modeled in this template, especially for transformers (e.g., LayerNorm). Secondly, the definition of a layer using this formalism is unclear (in particular, modeling a transformer layer requires several alternating linear/nonlinear compositions).
The main claim of this paper is that "We prove that the top eigenvector of the Hessian is approximately equal to the weighted concatenation of the eigenvectors of its diagonal blocks provided that the forward projections of the eigenvectors are approximately colinear and their Gram matrix is rank-one". This claim is not supported by the paper's developments, as the theoretical results and experiments concern the Gauss-Newton term G in the Hessian decomposition H=G+H. The authors say in Section 5 that "Since G dominates the Hessian, the top eigenvector of the Hessian is approximately equal to the weighted concatenation of the top eigenvectors of the block diagonals.". However, such results do not simply transfer from G to H, and nontrivial formal and empirical steps are required to support this, which are missing in the current work.
A close-to-one G component ratio does not imply, in the absence of angular measurements, top eigenvector alignment. A counterexample constructed with the help of GPT is the following. For 0<t<1/2 the following matrix can be split in 4 blocks of 2×2 elements
\begin{align}
G(t)=
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 1-t^2 & 0 & (1-t^2)\sqrt t\\
0 & 0 & t & 0\\
0 & (1-t^2)\sqrt t & 0 & t(1-t^2)
\end{pmatrix} = \begin{pmatrix}
G_1 & G_2 \\
G_3 & G_4
\end{pmatrix}
\end{align}
Denote the top eigenvector of a matrix A as v1(A) and the top eigenvalue of A as λ1(A) . Then, v1(G)=1+t1[0,1,0,t]⊤, v1(G1)=[1,0]⊤, v1(G4)=[1,0]⊤, λ1(G1)=1 and λ1(G4)=t. Letting v1(A)[i:j] denote the vector obtained by selecting the elements of v1(A) between positions i and j inclusively, observe that $ \langle v_1(G)[1:2] , v_1(G_1)\rangle = 0 $ and ⟨v1(G)[3:4],v1(G4)⟩. However, we have that
\begin{align}
\frac{v_1(G)[1:2]^\top G_1 v_1(G)[1:2] + v_1(G)[3:4]^\top G_4 v_1(G)[3:4]}{\lambda_1(G_1) + \lambda_1(G_4) } = \frac{(1-t^2)(1+t^2)}{(1+t)^2} \overset{t\downarrow 0}{\longrightarrow} 1
\end{align}
Therefore, for the empirical validation of the claim, the authors need to measure the angles between the vectors as these quantities can be deceiving.
Experimental setup:
The first round of experiments are claimed over 4 weight initialization seeds and 4 data samplings. However, the plots only present a single experimental instance (based on the authors' description), whereas mean + confidence band type plots would have been needed for all configurations.
The main finding in Figure 1 was already shown empirically (at least) by [1] in Appendix Figures 10 c) and 11 c) on the CIFAR10 dataset. The authors fail to acknowledge this.
The large model experiment essentially computes the target quantities only on a trained snapshot of the networks (to my understanding). Even if the quantities evaluated in the experiment would have been appropriate to sustain the main claim of the paper, this experiment would have failed to practically validate it because it shows nothing about G's behavior ** throughout ** training.
Transformers are claimed to be encompassed by the architectural template introduced in the paper (lines 051 -- 061). However, no experiment is presented for this very important architecture.
Notes for improvement:
General presentation:
The paper would benefit from a section clearly enumerating what the authors consider to be their contributions
The paper would benefit from a section describing an overview of the technical steps taken to prove the main claim
Typesetting:
typesetting both equations and references using round parentheses (⋅) is very confusing. Furthermore, the experimental section includes lists of hyperparameter values in the same format. This leaves it to the reader to differentiate while reading between (⋅) being a citation, an equation reference, or simply a list of numerical values. This makes for an unpleasant read.
* References are not hyperlinked. Please use the default format provided by the ICLR LaTex template. In particular, hyperlinking numerical references is crucial for easy navigation of the paper.
Currently, there is a lack of integration of the mathematical expressions into the text. The recurring structure of "[Introduce the next quantity] : [quantity] [no full stop punctuation] [next sentence]" is both grammatically incorrect and leaves a large amount of space unused. This space would have better served related literature discussions or results-related discussions.
Figure and table arrangement: page 8 of the manuscript shows poor integration of the results into the prose, and again, wastes space that could have been directed elsewhere.
Mathematical presentation:
Theoretical statement bodies poorly typeset in the detriment of space, and contain repetitive quantity definitions like that of Wi′ and max∥W1∥2+⋯+∥WnL∥2=1E[∑iMiα^i(Wi)Ui2] in both Lemma 3.1 and 3.2
Line 353: "EKTK is colinear" --- the expression is inadequate, a single object cannot be colinear
The body of Corollary 3.2 is just an inequality, no words or punctuation
The statement "G consistently dominates the Hessian matrix" is ambiguous, and the authors likely mean that G dominates the other terms in the decomposition of the Hessian. Furthermore, there are several ways in which a matrix can dominate another: by Loewner order, by norm, entrywise, matrix action on vectors, etc and this should be made precise according to the literature establishing it. Please make the wording mathematically precise and provide citations that substantiate the qualifier "consistently", accompanied by a discussion.
The authors use the notion of "principal values of the projected vectors" (line 040). “Principal value” is ** not ** a mathematical notion associated with individual vectors. Please use accurate, established terminology.
The authors should avoid re-proving well-established results, or immediate consequences thereof. The eigenvalue statement of Lemma 5.1 is an immediate consequence of Proposition 5.5.13 ii) in [4].
1] Cohen, Jeremy M., et al. "Gradient descent on neural networks typically occurs at the edge of stability." arXiv preprint arXiv:2103.00065 (2021).
[2] Papyan, Vardan. "Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians." arXiv preprint arXiv:1901.08244 (2019).
[3] Papyan, Vardan. "Traces of class/cross-class structure pervade deep learning spectra." Journal of Machine Learning Research 21.252 (2020): 1-64.
[4] Bernstein, Dennis S. Matrix mathematics. Vol. 2. Princeton, NJ: Princeton university press, 2011.
This paper attempts to explain why the diagonal blocks of the Hessian approximate its top eigenvalue / eigenvector. It shows that the top eigenvalue of the Gauss-Newton term is bounded by the sum of the top eigenvalues of its layer blocks, with equality under colinearity and rank-one conditions, and reports experiments on VGG and ResNets suggesting the bound is nearly tight. However the fact that this bound is tight means the direct opposite of block-diagonality, since, as proved in the paper’s own Lemma 5.1, a strictly block diagonal matrix shall have its eigenvalue equal to the maximum over the eigenvalues of the blocks instead of their sum. Therefore the paper has substantial soundness issues and hence I recommend for rejection.
Critical strengths
None
Critical weaknesses
The paper is self-contradictory in many places. The fact that the sum of maximum eigenvalues of the diagonal blocks well-approximate the maximum eigenvalue of the entire Hessian means the matrix is largely non-block-diagonal. A block-diagonal matrix shall have its largest eigenvalue equal to the maximum among the maximum eigenvalues of the diagonal blocks (as stated in the paper's own Lemma 5.1).
Besides, the citation format in this paper diverges from the official template, which I think is worth mentioning.
Questions for authors
Given the current state of the paper, no additional information may materially affect my assessment.
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.