- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the geometric mechanism underlying Linear Mode Connectivity in wide neural networks. The authors decompose Fisher information geometry into two axes: Shape (local curvature/metric variation, captured by ) and Length (the Fisher quadratic form of the displacement between two trained minima). They prove that Shape flattens with width at dimension-free rates, derive a Fisher-barrier inequality bounding the interpolation barrier by Fisher length and predictive uncertainty , and validate both axes empirically across MLPs, teacher–student networks, and CNNs under NTK-lazy, Standard, and μP parameterizations. The central empirical/conceptual claim is that Shape flattening is universal and necessary but not sufficient for barrier collapse; collapse additionally requires Fisher length reduction, which in turn is only informative when predictive uncertainty is controlled.
Critical strengths
- S1. The central negative result is very interesting and well-supported. Across all three parameterizations and all architectures tested, the Shape axis flattens with width, yet the barrier behaves in different ways under the three regimes (grows under NTK-lazy, collapses under μP, non-monotone under Standard). This refutes the common intuition that "wider networks are flatter, hence more connectable," and the paper demonstrates it formally.
- S2. Fisher–Hessian identity () is a clean, useful bridge for the theoretical results. It gives an interpretable gating condition: Fisher-based analysis only tracks the true loss landscape when predictive uncertainty \rho^ is low, and this gating condition is what the paper later aims to show that governs barrier collapse.
Critical weaknesses
- W1. The theoretical flattening result covers a narrower scope than the claims suggest. Theorem 4.4's guarantee is conditional on Assumption 4.2, which is only proven for a simplified "frozen-softmax Fisher surrogate" under NTK parameterization (Appendix C), not for the true Fisher matrix, and not for Standard or μP training. For those cases (which correspond to the bulk of the experiments), flattening is only demonstrated empirically. The abstract and intro read as though "Shape flattens with width, dimension-free" is established in general; this should be qualified more clearly as proven-for-one-idealized-case, empirically-observed-elsewhere.
- W2. Weak motivation and intuition throughout the main text. The introduction does not explain why Shape and Length are the natural/correct axes to split Fisher geometry into, nor what specifically fails in prior (Euclidean Hessian-based) flatness analyses that this decomposition fixes. Central quantities (, , , and ) are introduced formally but rarely explained in words before being used in bounds. The paper is easier to follow once one has worked through the appendix proofs than it should be from the main text alone.
Questions for authors
Questions
- Q1. Could you provide a small, worked/intuitive example (even a toy 2–3 parameter model) illustrating concretely how Shape can be fully flat while the interpolation barrier remains large? This would substantially help readers build intuition before the formalism is introduced.
- Q2. What is the conceptual justification for decomposing Fisher geometry specifically into Shape and Length, as opposed to other possible splits? What would be gained or lost under an alternative decomposition?
- Q3. Assumption 4.2 / Theorem 4.4's rate is proven only for the frozen-softmax surrogate under NTK parameterization. Do you expect (or can you show) that an analogous flattening result provably holds for the true Fisher matrix, or under Standard/μP training — or is this inherently an empirical-only finding outside the NTK-surrogate case?
7 comments
[W2 / Q2] What is the conceptual justification for decomposing Fisher geometry specifically into Shape and Length, as opposed to other possible splits? What would be gained or lost under an alternative decomposition?
Response. Thank you for this conceptual question. In short, the split follows the order in which each object enters the loss along the segment.
At second order, , i.e., the Fisher energy (Length) plus a residual gated by . At third order, , with , and Shape enters only at this order. This is the standard Riemannian hierarchy: metric (, Length) connection (, ) curvature (, ).
Shape determines how geodesics bend relative to straight lines, and it touches the loss only through part of the cubic remainder. Flattening therefore leaves the second-order terms, namely the Fisher energy and the -gated residual, unconstrained. We split the Hessian instead of using it as a whole because and is label-free, while is indefinite and label-dependent. Along the segment, is the only source of the negative curvature that a barrier requires.
Two alternative splits. (i) The paper tests the split into direction and magnitude, . The direction alone explains of the barrier exponent, against for the full , and only under P (Fig. 5b). This split loses the scale of the displacement.
(ii) A single Fisher–Rao distance merges Shape and Length into one scalar and cannot express a straight path with a high barrier. With a constant metric, it reduces to , i.e., Length plus the damping term.
Revision. We will state this justification of the split into Shape and Length in the introduction, and explain , , and in words where each first appears.
[W1 / Q3] Assumption 4.2 / Theorem 4.4's rate is proven only for the frozen-softmax surrogate under NTK parameterization. Do you expect (or can you show) that an analogous flattening result provably holds for the true Fisher matrix, or under Standard/µP training — or is this inherently an empirical-only finding outside the NTK-surrogate case?
Response. Thank you for this thoughtful question. Outside the NTK-surrogate case, flattening is measured rather than proven, and the measurements are consistent across the grid. decreases with width in every Standard and P cell except one, as it does in all NTK-lazy cells (three architectures, four activations; table below). The exception is the CNN with softplus under Standard, which shows no trend. This scope is stated in §3.3 (proven for the frozen-softmax surrogate under NTK, observed empirically elsewhere), and the abstract and introduction will use the same wording.
Width exponent of , as estimated in the paper (range over activations):
| Architecture | NTK-lazy | Standard | P |
|---|---|---|---|
| MLP | 0.86–1.16 | 0.60–1.02 | 1.19–1.93 |
| Teacher–student | 1.08–1.37 | 1.32–1.68 | 1.53–2.55 |
| CNN | 0.50–1.36 | 0.59–1.39 | 1.51–1.76 |
On a proof. These measurements lead us to expect that an analogous statement holds under Standard and P. The argument of Appendix C relies on Jacobian-stability bounds from the NTK literature (), which require the weights to stay near initialisation. P is constructed so that features, and hence the tangent kernel, move by (Yang & Hu, 2021), so a proof in that regime has to take a different route. Our measurements are consistent with this. The relative change of the empirical tangent kernel over training, , grows with width under P, whereas under NTK-lazy it shrinks as , close to the rate those bounds assume (table below). Flattening under P therefore occurs while the kernel moves, and it is not a by-product of lazy training.
Relative change of the empirical tangent kernel, (median, MLP, GELU):
| 64 | 128 | 256 | 512 | 1024 | 2048 | |
|---|---|---|---|---|---|---|
| NTK-lazy | 7.6 | 5.2 | 3.6 | 2.6 | 1.7 | 1.1 |
| P | 22.6 | 34.0 | 46.7 | 82.6 | 115.7 | 165.0 |
For the true Fisher under NTK, Appendix C already gives , i.e., the lazy rate up to a term controlled by predictive uncertainty.
What this gap affects. Two points limit its impact. First, Theorem 4.4 is conditional: once Assumption 4.2 holds, in any regime, the Christoffel and curvature bounds follow with constants independent of . Second, the central negative result needs flattening only where the barrier persists, i.e., under NTK-lazy, which is the regime where the assumption is proven (for the surrogate). Empirically, flattening is observed in all three regimes, in all but one of the 36 cells (Figs. 1b, 6).
Revision. We will state the Standard and P case as a conjecture supported by the measurements above.
Thank you very much for your positive evaluation and for finding our central negative result interesting and well-supported and the Fisher–Hessian identity a clean, useful bridge for the theory. Below are our point-by-point responses to your questions.
[Q1] Could you provide a small, worked/intuitive example (even a toy 2–3 parameter model) illustrating concretely how Shape can be fully flat while the interpolation barrier remains large?
Response. Thank you for this constructive suggestion. Below, we give a one-parameter model in which Shape is exactly flat, yet the straight path between two global minima still has a barrier.
Example. Take one parameter w, two inputs and a Gaussian likelihood with unit variance: f(x1;w)=(cosw,sinw), f(x2;w)=21(cos2w,sin2w), with targets (1,0) and (−1,0). Both Jacobians have unit norm, so F≡1. The metric is therefore constant, Γ≡0, and every straight line is a geodesic. In other words, Shape is exactly flat. The loss is L(w)=0.8125−21cosw+41cos2w, which has two global minima at w=±π/3 (L=0.4375) that predict differently on x1. The straight line between them, which is also the geodesic, crosses w=0, where L=0.5625. This gives a barrier of B=1/8.
Numerical check. We also checked the example numerically. On the segment, L matches the closed form to machine precision, and ∂wF vanishes up to numerical error (so Γ=0). L attains its maximum on the segment at w=0, and the barrier is 0.1250. The table lists three points on the segment.
Where the barrier comes from. At the top of the barrier, the Fisher–Hessian identity identifies its source: ∂2L=−21 while F=1, so the residual is E=−23. The negative curvature that a barrier requires cannot come from F⪰0. It comes entirely from the uncertainty-weighted residual, which is large where the model fits poorly. The Fisher length of the displacement is LF=(2π/3)2≈4.39.
Revision. We will add this example as a new appendix, with a pointer from the opening of §4.