acceptodds
Withdrawn from ICLR 2027

A Scale Invariant Curvature Diagnostic from Coupled Derivative

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep neural networks possess reparameterization symmetries that systematically alter parameter space Hessian curvature despite the underlying function remaining entirely static. Prior work addresses this variance by proposing invariant flatness measures that rely on explicit weight magnitude scaling. We introduce a scale invariant diagnostic constructed entirely from coupled first and second order loss derivatives, bypassing explicit parameter norms. The diagnostic couples the layer restricted Hessian Rayleigh quotient, evaluated along the layer’s own parameter direction, with the squared gradient norm of that same layer. Under continuous node wise rescaling Wl → cWl , Wl+1 → c −1Wl+1 in a positively homoge￾neous network, both the Rayleigh quotient of the layer restricted Hessian and the squared gradient norm of that same layer scale by c −2 ; dividing the former by the latter cancels this common factor exactly, for every c > 0. We analyze the same construction under general reparameterizations, characterize exactly the symmetry group for which the cancellation remains exact, and prove that it cannot be extended further. We show that replacing the Hessian with its pseudoinverse in the same coupling yields a second exactly invariant diagnostic that decreases rather than increases with curvature, resolving a sign obstruction that otherwise blocks connecting either quantity to a Laplace based generalization bound. We verify exact invariance on trained networks of up to fourteen hidden layers and one hundred twenty six thousand parameters, on two real datasets, across multiple architectures and seeds, and we report an exploratory multi seed experiment in which the inverse coupled diagnostic tracks the generalization gap across network widths. We present this as a precise, narrow invariance result: we do not establish a generalization bound, and we do not evaluate the diagnostic beyond the small scale settings reported here.

Primary area:
learning theory
Keywords:
Scale invariant curvature, Hessian reparameterization, Cross layer gradient coupling, Loss landscape geometry, Second order diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Tv3kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the reparameterization dependence of Hessian-based sharpness (Dinh et al., 2017). It proposes the per-layer diagnostic Ql=θl⊤Hlθl∥θl∥2∥∇WlL∥2Q_l = \frac{\theta_l^\top H_l \theta_l}{\|\theta_l\|^2 \|\nabla_{W_l}\mathcal{L}\|^2} and shows it is exactly invariant under the rescaling ρc\rho_{c}: Wl→cWlW_l \to cW_l, Wl+1→c−1Wl+1W_{l+1} \to c^{-1}W_{l+1} in positively homogeneous networks, since numerator and denominator both scale as c−2c^{-2}. It further claims that this invariance cannot be extended beyond diagonal rescalings (Prop. 2), and introduces an inverse variant Rl=tr⁡(Hl+)∣∇WlL∥2R_l = \operatorname{tr}(H_l^+) |\nabla_{W_l}\mathcal{L}\|^2 with the sign needed for a Laplace/PAC-Bayes bound. Experiments on small MLPs numerically confirm the invariances, and an exploratory experiment relates RlR_l to the generalization gap across four widths.

Critical strengths
  1. Correct, clearly scoped core result. The scaling lemmas and the exact invariance of QlQ_l and RlR_l under layer-wise rescaling (Theorems 1 and 4) are correct, and the paper is transparent that it establishes no generalization bound. This matters because the paper makes a precise, checkable claim rather than an overreaching one.
  2. Reproducible verification. Experimental settings are described in enough detail to reproduce, and the invariance is checked across several architectures, seeds and two real datasets, so the reported numbers can be trusted as a confirmation of the algebra.
Critical weaknesses
  1. The main result is elementary and its key claims are overstated. For a bias-free ReLU network, f(x;tθl)=tf(x;θ)f(x; t\theta_l) = t f(x;\theta) for t>0t>0, so differentiating L(tθl)\mathcal{L}(t\theta_l) twice at t=1t=1 gives θl⊤Hlθl=E[ℓ′′(f,y)f(x)2]\theta_l^\top H_l \theta_l = \mathbb{E}[\ell''(f,y) f(x)^2]: the same function-space quantity for every layer (for squared loss, simply E[f2]\mathbb{E}[f^2]), not landscape curvature. QlQ_l is invariant because it is a ratio of two separately invariant quantities, and many other combinations are equally invariant, e.g. λmax⁡(Hl)∥θl∥2\lambda_{\max}(H_l)\Vert\theta_l\Vert^2 or tr⁡(Hl)∥θl∥2\operatorname{tr}(H_l)\Vert\theta_l\Vert^2, contradicting the uniqueness claim of Remark 1. Likewise, Prop. 2 and the Section 5 heading call the rescaling "non-diagonal", but Dl=diag⁡(c1,…,cn)D_l = \operatorname{diag}(c_1,\dots,c_n) is diagonal with unequal entries; "non-uniform" would be accurate. With this fixed, Prop. 2 shows QlQ_l is invariant only for Dl=cID_l = cI, contradicting the claim that it holds for the full diagonal group. Finally, the abstract claims the diagnostic avoids explicit parameter norms, yet Definition 3 divides by ∥θl∥2\Vert\theta_l\Vert^2. This is consequential because the paper's stated contribution rests on these claims.

  2. Limited relevance to the sharpness/generalization question. Both QlQ_l and RlR_l normalize by ∥∇WlL∥2\Vert\nabla_{W_l}\mathcal{L}\Vert^2, which vanishes at minima, exactly where the question of Dinh et al. is posed. The Limitations argument that HlH_l is zero away from ReLU boundaries is incorrect: there the loss Hessian equals the Gauss–Newton term E[Jl⊤ℓ′′Jl]\mathbb{E}[J_l^\top \ell'' J_l], with Jl=∂f/∂θlJ_l = \partial f/\partial \theta_l, which is generically nonzero. The paper does not compare QlQ_l or RlR_l empirically with comparable measures, e.g. those in the paper's own Table 1 (path norm, Fisher–Rao norm, relative flatness, Fisher-normalized Hessian), and the generalization evidence in Fig. 8 (r=−0.98r=-0.98 over three width means after excluding width 8) carries little weight. This matters because without such evidence the diagnostic's usefulness is not demonstrated.

Questions for authors

Clarification requests:

  1. Can you confirm that θl⊤Hlθl=E[ℓ′′(f,y)f(x)2]\theta_l^\top H_l \theta_l = \mathbb{E}[\ell''(f,y) f(x)^2] is identical across layers in your networks? If so, what curvature information does QlQ_l carry beyond this quantity and the weight/gradient norms?

  2. Related to the previous question: Proposition 2 states that no function of θl\theta_l and ∇WlL\nabla_{W_l}\mathcal{L} alone can correct the Rayleigh quotient ρl\rho_l back to an invariant quantity under non-uniform diagonal rescaling. However, multiplying ρl\rho_l by ∥θl∥2\Vert\theta_l\Vert^2 gives θl⊤Hlθl\theta_l^\top H_l \theta_l, which by Theorem 2 is invariant under all positive diagonal DlD_l; similarly, θl⊤Hlθl/(θl⊤∇WlL)2\theta_l^\top H_l \theta_l / (\theta_l^\top \nabla_{W_l}\mathcal{L})^2 is invariant under the full diagonal group. Thus the normalization by ∥θl∥2\Vert\theta_l\Vert^2 appears to be what restricts invariance to uniform scaling. What motivates using the Rayleigh quotient, and the specific normalizer in QlQ_l, given that alternatives are invariant under a strictly larger group?

  3. How do QlQ_l and RlR_l behave as training converges and ∥∇L∥→0\Vert\nabla\mathcal{L}\Vert \to 0, and how do they compare empirically with existing rescaling-invariant measures such as relative flatness or adaptive sharpness?

Reviewer_xBPPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I believe the paper should be desk-rejected because it violates the ICLR 2027 Author Guidelines, which limit the main text to 9 pages at initial submission. The paper's main text spans 10 pages, exceeding the permitted page limit.

Critical strengths

NA

Critical weaknesses

NA

Questions for authors

NA

Reviewer_hpDsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a layer-wise curvature diagnostic that is exactly invariant under positive rescaling in homogeneous neural networks. The authors establish its invariance theoretically, investigate its behavior under more general reparameterizations, and introduce a pseudoinverse-based variant motivated by PAC-Bayes analysis. Numerical experiments verify the invariance properties and explore the relationship between the proposed diagnostic and generalization.

Critical strengths
  • The paper addresses the well-known issue that Hessian-based sharpness measures are sensitive to function-preserving reparameterizations. The problem is relevant to understanding curvature and generalization in neural networks.
  • The central invariance result follows from clearly stated transformation properties of the gradient and Hessian. The main proof is concise and easy to follow.
  • The experiments illustrate the theoretical invariance across different architectures and scaling factors. The authors also acknowledge that they do not establish a generalization bound or evaluate the method at large scale.
Critical weaknesses

My primary concern is that several theoretical claims are incorrect or inconsistent with the paper's own results.

First, the uniqueness claim in Remark 1 seems to be false. For example, the simpler quantity Sl=θl⊤HlθlS_l=\theta_l^\top H_l\theta_l is already invariant under the more general neuron-wise diagonal rescaling, as established by the paper's own eq.(8). Similarly, λmax⁡(Hl)∣θl∣2\lambda_{\max}(H_l)|\theta_l|^2 is exactly invariant under uniform layer rescaling.

Second, the claimed characterization of the maximal invariance group is inconsistent with the analysis. Thm. 2 establishes the transformation of the Rayleigh quotient, not the invariance of QlQ_l under general diagonal rescaling. In fact, Fig. 5 demonstrates that QlQ_l changes under non-uniform diagonal scaling, directly contradicting the claimed extension to the full diagonal rescaling group. Furthermore, the impossibility statement in Prop. 2 is not justified, since the invariant quantity in eq.(8) already provides a counterexample.

Finally, Section 6 incorrectly states that the Hessian vanishes away from ReLU activation boundaries. While the second derivative of the network output with respect to a single layer vanishes almost everywhere, the Hessian of the loss generally contains a nonzero Gauss-Newton term.

Other weaknesses:

  • The main invariance result follows directly from the homogeneity properties of the gradient and Hessian, making the construction relatively straightforward.
  • Most experiments verify algebraic identities rather than demonstrate the practical usefulness of the proposed diagnostics. The reported generalization correlation relies on only three width-level observations after excluding one configuration, and the proposed connection to PAC-Bayes bounds remains insufficiently justified.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.