- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the reparameterization dependence of Hessian-based sharpness (Dinh et al., 2017). It proposes the per-layer diagnostic and shows it is exactly invariant under the rescaling : , in positively homogeneous networks, since numerator and denominator both scale as . It further claims that this invariance cannot be extended beyond diagonal rescalings (Prop. 2), and introduces an inverse variant with the sign needed for a Laplace/PAC-Bayes bound. Experiments on small MLPs numerically confirm the invariances, and an exploratory experiment relates to the generalization gap across four widths.
Critical strengths
- Correct, clearly scoped core result. The scaling lemmas and the exact invariance of and under layer-wise rescaling (Theorems 1 and 4) are correct, and the paper is transparent that it establishes no generalization bound. This matters because the paper makes a precise, checkable claim rather than an overreaching one.
- Reproducible verification. Experimental settings are described in enough detail to reproduce, and the invariance is checked across several architectures, seeds and two real datasets, so the reported numbers can be trusted as a confirmation of the algebra.
Critical weaknesses
-
The main result is elementary and its key claims are overstated. For a bias-free ReLU network, for , so differentiating twice at gives : the same function-space quantity for every layer (for squared loss, simply ), not landscape curvature. is invariant because it is a ratio of two separately invariant quantities, and many other combinations are equally invariant, e.g. or , contradicting the uniqueness claim of Remark 1. Likewise, Prop. 2 and the Section 5 heading call the rescaling "non-diagonal", but is diagonal with unequal entries; "non-uniform" would be accurate. With this fixed, Prop. 2 shows is invariant only for , contradicting the claim that it holds for the full diagonal group. Finally, the abstract claims the diagnostic avoids explicit parameter norms, yet Definition 3 divides by . This is consequential because the paper's stated contribution rests on these claims.
-
Limited relevance to the sharpness/generalization question. Both and normalize by , which vanishes at minima, exactly where the question of Dinh et al. is posed. The Limitations argument that is zero away from ReLU boundaries is incorrect: there the loss Hessian equals the Gauss–Newton term , with , which is generically nonzero. The paper does not compare or empirically with comparable measures, e.g. those in the paper's own Table 1 (path norm, Fisher–Rao norm, relative flatness, Fisher-normalized Hessian), and the generalization evidence in Fig. 8 ( over three width means after excluding width 8) carries little weight. This matters because without such evidence the diagnostic's usefulness is not demonstrated.
Questions for authors
Clarification requests:
-
Can you confirm that is identical across layers in your networks? If so, what curvature information does carry beyond this quantity and the weight/gradient norms?
-
Related to the previous question: Proposition 2 states that no function of and alone can correct the Rayleigh quotient back to an invariant quantity under non-uniform diagonal rescaling. However, multiplying by gives , which by Theorem 2 is invariant under all positive diagonal ; similarly, is invariant under the full diagonal group. Thus the normalization by appears to be what restricts invariance to uniform scaling. What motivates using the Rayleigh quotient, and the specific normalizer in , given that alternatives are invariant under a strictly larger group?
-
How do and behave as training converges and , and how do they compare empirically with existing rescaling-invariant measures such as relative flatness or adaptive sharpness?