Near Block-Diagonality in Hessian Matrices

It has been experimentally shown that the Hessian matrix of the training loss of a neural network is well-approximated by its diagonal blocks, but the reason why this occurs is not known. We prove a set of lemmas that establish that the sharpness of the Hessian is approximately equal to the sharpness of its block diagonal approximation provided that the forward projections of the eigenvectors are approximately colinear and their Gram matrix is low-rank. We show experimentally that these conditions hold and are to be expected given the structure of the Hessian matrix, establishing why the Hessian matrix is near-block-diagonal in practice.