acceptodds
Under review as a conference paper at ICLR 2027

The Geometry Behind the Barrier: Fisher Geometry and Linear Mode Connectivity in Wide Neural Networks

Abstract

After permutation alignment, independently trained minima of wide neural networks often exhibit Linear Mode Connectivity (LMC), yet the geometric mechanism governing the interpolation loss barrier remains unclear. We show that Fisher information geometry explains the barrier through two decoupled axes: Shape, governing manifold curvature, and Length, governing the Fisher energy along the parameter displacement. Along the Shape axis, we show that the local variation of the Fisher metric vanishes in operator norm with width, driving Christoffel symbols and sectional curvature to zero with dimension-free bounds. However, this flattening is a local property of the metric: it neither implies that Fisher geodesics coincide with linear interpolations nor suffices for barrier collapse. Along the Length axis, we establish a Fisher-barrier inequality showing that the barrier is bounded by the Fisher quadratic form and predictive uncertainty. Experiments across architectures and training regimes show that while metric flattening occurs in every setting, geodesic-linear alignment is regime-dependent, and barrier collapse is driven by Fisher length reduction, most strongly when predictive uncertainty is controlled.

Primary area:
learning theory
Keywords:
linear mode connectivity, Fisher information geometry, loss landscape, wide neural networks, neural tangent kernel, maximal update parameterization, model merging

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_idLYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the geometric mechanism underlying Linear Mode Connectivity in wide neural networks. The authors decompose Fisher information geometry into two axes: Shape (local curvature/metric variation, captured by SF=sup⁡∣∂F∣op\mathcal{S}F = \sup|\partial F|{op}) and Length (the Fisher quadratic form LF=Δ⊤FΔ\mathcal{L}_F = \Delta^\top F \Delta of the displacement between two trained minima). They prove that Shape flattens with width at dimension-free rates, derive a Fisher-barrier inequality bounding the interpolation barrier by Fisher length and predictive uncertainty ρ∗\rho^*, and validate both axes empirically across MLPs, teacher–student networks, and CNNs under NTK-lazy, Standard, and μP parameterizations. The central empirical/conceptual claim is that Shape flattening is universal and necessary but not sufficient for barrier collapse; collapse additionally requires Fisher length reduction, which in turn is only informative when predictive uncertainty is controlled.

Critical strengths
  • S1. The central negative result is very interesting and well-supported. Across all three parameterizations and all architectures tested, the Shape axis flattens with width, yet the barrier behaves in different ways under the three regimes (grows under NTK-lazy, collapses under μP, non-monotone under Standard). This refutes the common intuition that "wider networks are flatter, hence more connectable," and the paper demonstrates it formally.
  • S2. Fisher–Hessian identity (∇2L=F+E\nabla^2 L = F + E) is a clean, useful bridge for the theoretical results. It gives an interpretable gating condition: Fisher-based analysis only tracks the true loss landscape when predictive uncertainty \rho^ is low, and this gating condition is what the paper later aims to show that governs barrier collapse.
Critical weaknesses
  • W1. The theoretical flattening result covers a narrower scope than the claims suggest. Theorem 4.4's guarantee is conditional on Assumption 4.2, which is only proven for a simplified "frozen-softmax Fisher surrogate" under NTK parameterization (Appendix C), not for the true Fisher matrix, and not for Standard or μP training. For those cases (which correspond to the bulk of the experiments), flattening is only demonstrated empirically. The abstract and intro read as though "Shape flattens with width, dimension-free" is established in general; this should be qualified more clearly as proven-for-one-idealized-case, empirically-observed-elsewhere.
  • W2. Weak motivation and intuition throughout the main text. The introduction does not explain why Shape and Length are the natural/correct axes to split Fisher geometry into, nor what specifically fails in prior (Euclidean Hessian-based) flatness analyses that this decomposition fixes. Central quantities (SF\mathcal{S}_F, MfM_f, MLM_L, and R(w0)R(w_0)) are introduced formally but rarely explained in words before being used in bounds. The paper is easier to follow once one has worked through the appendix proofs than it should be from the main text alone.
Questions for authors

Questions

  • Q1. Could you provide a small, worked/intuitive example (even a toy 2–3 parameter model) illustrating concretely how Shape can be fully flat while the interpolation barrier remains large? This would substantially help readers build intuition before the formalism is introduced.
  • Q2. What is the conceptual justification for decomposing Fisher geometry specifically into Shape and Length, as opposed to other possible splits? What would be gained or lost under an alternative decomposition?
  • Q3. Assumption 4.2 / Theorem 4.4's α=1/2\alpha=1/2 rate is proven only for the frozen-softmax surrogate under NTK parameterization. Do you expect (or can you show) that an analogous flattening result provably holds for the true Fisher matrix, or under Standard/μP training — or is this inherently an empirical-only finding outside the NTK-surrogate case?
7 comments

Thank you very much for your positive evaluation and for finding our central negative result interesting and well-supported and the Fisher–Hessian identity a clean, useful bridge for the theory. Below are our point-by-point responses to your questions.

[Q1] Could you provide a small, worked/intuitive example (even a toy 2–3 parameter model) illustrating concretely how Shape can be fully flat while the interpolation barrier remains large?

Response. Thank you for this constructive suggestion. Below, we give a one-parameter model in which Shape is exactly flat, yet the straight path between two global minima still has a barrier.

Example. Take one parameter ww, two inputs and a Gaussian likelihood with unit variance: f(x1;w)=(cos⁡w,sin⁡w)f(x_1;w) = (\cos w, \sin w), f(x2;w)=12(cos⁡2w,sin⁡2w)f(x_2;w) = \tfrac12(\cos 2w, \sin 2w), with targets (1,0)(1,0) and (−1,0)(-1,0). Both Jacobians have unit norm, so F≡1F \equiv 1. The metric is therefore constant, Γ≡0\Gamma \equiv 0, and every straight line is a geodesic. In other words, Shape is exactly flat. The loss is L(w)=0.8125−12cos⁡w+14cos⁡2wL(w) = 0.8125 - \tfrac12\cos w + \tfrac14\cos 2w, which has two global minima at w=±π/3w = \pm\pi/3 (L=0.4375L = 0.4375) that predict differently on x1x_1. The straight line between them, which is also the geodesic, crosses w=0w = 0, where L=0.5625L = 0.5625. This gives a barrier of B=1/8B = 1/8.

Numerical check. We also checked the example numerically. On the segment, LL matches the closed form to machine precision, and ∂wF\partial_w F vanishes up to numerical error (so Γ=0\Gamma = 0). LL attains its maximum on the segment at w=0w=0, and the barrier is 0.12500.1250. The table lists three points on the segment.

wwf(x1;w)f(x_1;w)f(x2;w)f(x_2;w)L(w)L(w)FF∂2L\partial^2 Lresidual E=∂2L−FE = \partial^2 L - F
−π/3-\pi/3 (minimum)(0.500,−0.866)(0.500, -0.866)(−0.250,−0.433)(-0.250, -0.433)0.43750.437511+0.75+0.75−0.25-0.25
00 (barrier top)(1,0)(1, 0)(0.500,0)(0.500, 0)0.56250.562511−0.50-0.50−1.50-1.50
+π/3+\pi/3 (minimum)(0.500,+0.866)(0.500, +0.866)(−0.250,+0.433)(-0.250, +0.433)0.43750.437511+0.75+0.75−0.25-0.25

Where the barrier comes from. At the top of the barrier, the Fisher–Hessian identity identifies its source: ∂2L=−12\partial^2 L = -\tfrac12 while F=1F = 1, so the residual is E=−32E = -\tfrac32. The negative curvature that a barrier requires cannot come from F⪰0F \succeq 0. It comes entirely from the uncertainty-weighted residual, which is large where the model fits poorly. The Fisher length of the displacement is LF=(2π/3)2≈4.39\mathcal{L}_F = (2\pi/3)^2 \approx 4.39.

Revision. We will add this example as a new appendix, with a pointer from the opening of §4.

[W2 / Q2] What is the conceptual justification for decomposing Fisher geometry specifically into Shape and Length, as opposed to other possible splits? What would be gained or lost under an alternative decomposition?

Response. Thank you for this conceptual question. In short, the split follows the order in which each object enters the loss along the segment.

At second order, Δ⊤∇2L,Δ=Δ⊤FΔ+Δ⊤EΔ\Delta^\top \nabla^2 L\\,\Delta = \Delta^\top F\Delta + \Delta^\top E\Delta, i.e., the Fisher energy (Length) plus a residual gated by ρ∗\rho^*. At third order, D3L=∂F+∂ED^3 L = \partial F + \partial E, with ∣Δ⊤(∂ΔF)Δ∣≤SF∣Δ∣3|\Delta^\top(\partial_\Delta F)\Delta| \le \mathcal{S}_F\\|\Delta\\|^3, and Shape enters only at this order. This is the standard Riemannian hierarchy: metric (FF, Length) →\to connection (∂F\partial F, Γ\Gamma) →\to curvature (∂2F\partial^2 F, Sec\mathrm{Sec}).

Shape determines how geodesics bend relative to straight lines, and it touches the loss only through part of the cubic remainder. Flattening therefore leaves the second-order terms, namely the Fisher energy and the ρ∗\rho^*-gated residual, unconstrained. We split the Hessian instead of using it as a whole because F⪰0F \succeq 0 and is label-free, while EE is indefinite and label-dependent. Along the segment, EE is the only source of the negative curvature that a barrier requires.

Two alternative splits. (i) The paper tests the split into direction and magnitude, LF=∣Δ∣2RF\mathcal{L}_F = \\|\Delta\\|^2\mathcal{R}_F. The direction alone explains R2=0.20R^2 = 0.20 of the barrier exponent, against 0.630.63 for the full LF\mathcal{L}_F, and only 0.090.09 under μ\muP (Fig. 5b). This split loses the scale of the displacement.

(ii) A single Fisher–Rao distance dF(wA,wB)d_F(w_A, w_B) merges Shape and Length into one scalar and cannot express a straight path with a high barrier. With a constant metric, it reduces to Δ⊤GFΔ\Delta^\top G_F\Delta, i.e., Length plus the damping term.

Revision. We will state this justification of the split into Shape and Length in the introduction, and explain SF\mathcal{S}_F, MfM_f, MLM_L and R(w0)R(w_0) in words where each first appears.

[W1 / Q3] Assumption 4.2 / Theorem 4.4's α=1/2\alpha = 1/2 rate is proven only for the frozen-softmax surrogate under NTK parameterization. Do you expect (or can you show) that an analogous flattening result provably holds for the true Fisher matrix, or under Standard/µP training — or is this inherently an empirical-only finding outside the NTK-surrogate case?

Response. Thank you for this thoughtful question. Outside the NTK-surrogate case, flattening is measured rather than proven, and the measurements are consistent across the grid. ∣∂F∣op\\|\partial F\\|_{\mathrm{op}} decreases with width in every Standard and μ\muP cell except one, as it does in all NTK-lazy cells (three architectures, four activations; table below). The exception is the CNN with softplus under Standard, which shows no trend. This scope is stated in §3.3 (proven for the frozen-softmax surrogate under NTK, observed empirically elsewhere), and the abstract and introduction will use the same wording.

Width exponent α\alpha of ∣∂F∣op∼n−α\\|\partial F\\|_{\mathrm{op}} \sim n^{-\alpha}, as estimated in the paper (range over activations):

ArchitectureNTK-lazyStandardμ\muP
MLP0.86–1.160.60–1.021.19–1.93
Teacher–student1.08–1.371.32–1.681.53–2.55
CNN0.50–1.360.59–1.391.51–1.76

On a proof. These measurements lead us to expect that an analogous statement holds under Standard and μ\muP. The argument of Appendix C relies on Jacobian-stability bounds from the NTK literature (∣Hk∣op≤CH/n\\|H_k\\|_{\mathrm{op}} \le C_H/\sqrt{n}), which require the weights to stay near initialisation. μ\muP is constructed so that features, and hence the tangent kernel, move by O(1)O(1) (Yang & Hu, 2021), so a proof in that regime has to take a different route. Our measurements are consistent with this. The relative change of the empirical tangent kernel over training, ∣KT−K0∣F/∣K0∣F\\|K_T - K_0\\|_F/\\|K_0\\|_F, grows with width under μ\muP, whereas under NTK-lazy it shrinks as n−0.55n^{-0.55}, close to the 1/n1/\sqrt{n} rate those bounds assume (table below). Flattening under μ\muP therefore occurs while the kernel moves, and it is not a by-product of lazy training.

Relative change of the empirical tangent kernel, ∣KT−K0∣F∣K0∣F\frac{\\|K_T - K_0\\|_F}{\\|K_0\\|_F} (median, MLP, GELU):

nn6412825651210242048
NTK-lazy7.65.23.62.61.71.1
μ\muP22.634.046.782.6115.7165.0

For the true Fisher under NTK, Appendix C already gives ∣∂F∣op≤c1′/n+3CJ3ρS\\|\partial F\\|_{\mathrm{op}} \le c_1'/\sqrt{n} + 3C_J^3\rho_S, i.e., the lazy rate up to a term controlled by predictive uncertainty.

What this gap affects. Two points limit its impact. First, Theorem 4.4 is conditional: once Assumption 4.2 holds, in any regime, the Christoffel and curvature bounds follow with constants independent of PP. Second, the central negative result needs flattening only where the barrier persists, i.e., under NTK-lazy, which is the regime where the assumption is proven (for the surrogate). Empirically, flattening is observed in all three regimes, in all but one of the 36 cells (Figs. 1b, 6).

Revision. We will state the Standard and μ\muP case as a conjecture supported by the measurements above.

Reviewer_7Gy2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why increasing network width flattens the Fisher geometry without universally eliminating linear interpolation barriers between permutation-aligned minima. To explain this divergence, the authors decompose Fisher information geometry into two decoupled axes:

  • Shape (Curvature and Path Geometry): Under stated derivative-decay and regularity assumptions, the authors derive dimension-free bounds relating metric derivatives to Christoffel symbols and sectional curvature. Empirically, metric-derivative decay occurs across the studied regimes, but this local flattening alone does not ensure a low-loss interpolation path.
  • Length (Fisher Energy and Predictive Uncertainty): Governed by the full Fisher quadratic form of the parameter displacement. Using a Fisher–Hessian identity, the authors derive an upper bound involving the endpoint Fisher energy, predictive uncertainty, and additional remainder terms.

Empirically, the authors evaluate these quantities across NTK-lazy, Standard, and Maximal Update (μ\muP) parameterizations using MLPs, Teacher–Student models, and CNNs across four smooth activations. The results confirm that while metric flattening and directional alignment (Rayleigh quotient RF\mathcal{R}_F) improve universally with width, barrier collapse tracks the full Fisher energy LF\mathcal{L}_F alongside controlled predictive uncertainty.

Critical strengths
  1. Conceptual Clarity of the Shape–Length Decomposition: Separating local metric flattening (Shape) from the energetic displacement scale (Length) provides a clean, mathematically grounded explanation for why width-induced landscape flattening and directional spectral alignment do not automatically guarantee linear mode connectivity.
  2. Systematic Empirical Validation: The experimental grid across parameterizations (NTK-lazy, Standard, μ\muP), widths, and seeds—including the CIFAR-10 extension—is carefully executed. Demonstrating that the full Fisher energy LF\mathcal{L}_F tracks barrier scaling substantially better than the normalized Rayleigh quotient RF\mathcal{R}_F provides convincing support for the paper's core Length-axis thesis within the tested settings.
Critical weaknesses
  1. Empirical Scope and Omission of Trained ReLU Models: The empirical evaluation is restricted to relatively small, residual-free MLPs and 3-layer CNNs on 10-class tasks, omitting modern architectural components that influence mode connectivity and parameter symmetries (such as residual connections or Transformer attention blocks). More importantly, while Appendix D notes that ReLU networks were trained across the grid, they are excluded from reported figures because the theoretical analysis assumes C3C^3 smoothness. Leaving the paper's core empirical claim untested on the already-trained piecewise-linear models limits the practical scope of the findings.
  2. Confounding Role of Data-Free L2L_2 Weight Matching: All experiments rely exclusively on data-free layerwise L2L_2 weight matching (Ainsworth et al., 2023) to align endpoints. It is unclear whether the persistent barrier and large Fisher energy in the NTK-lazy regime (or the barrier rebound at large widths in the Standard MLP) reflect an intrinsic geometric obstruction of the regime, or an artifact of data-free L2L_2 weight matching failing to identify functionally aligned permutations. A representative comparison with an applicable data-aware alignment method, such as activation matching from Ainsworth et al. (2023), would help assess how sensitive the reported regime contrast is to the chosen alignment procedure.

Ainsworth, Samuel K., Jonathan Hayase, and Siddhartha Srinivasa. "Git re-basin: Merging models modulo permutation symmetries, 2023." URL https://arxiv.org/abs/2209.04836 (2022).

Questions for authors
  1. Appendix D states that ReLU models were trained, but they are excluded from the main figures due to the C3C^3 requirement for ∂zF\partial_z F. Could you report the barrier and Fisher-energy trends (LF\mathcal{L}_F) for these ReLU models? This would help assess whether the Length-axis relationship holds for piecewise-linear networks, although these models fall outside the smoothness assumptions of the Shape analysis.
  2. Could you compare weight matching with an applicable data-aware alignment method on representative model pairs, or clarify which empirical conclusions are conditional on the chosen alignment? This would help assess the robustness of the reported regime contrast.
  3. Appendix E.3/Figure 10 reports that cells with relatively large predictive uncertainty can still exhibit barrier collapse, whereas the conclusion says collapse occurs only when Fisher-energy reduction accompanies controlled uncertainty. Could you reconcile these statements and distinguish conditions for a useful upper bound from necessary conditions for barrier collapse?
4 comments

Thank you very much for your detailed review and for your appreciation of the conceptual clarity of the Shape–Length decomposition and our systematic empirical validation. Below are our point-by-point responses to your comments.

[W1 / Q1] Appendix D states that ReLU models were trained, but they are excluded from the main figures due to the C3C^3 requirement for ∂zF\partial_z F. Could you report the barrier and Fisher-energy trends (LF\mathcal{L}_F) for these ReLU models?

Response. Thank you for this helpful suggestion. The Length-axis relation also holds on the ReLU networks. We excluded them from the figures because the Shape axis involves ∂F\partial F, which requires C3C^3 smoothness, whereas the Length axis does not. Since F=E[J⊤SJ]F = \mathbb{E}[J^\top S J] needs only first derivatives, which exist almost everywhere for ReLU, the Fisher length costs one JVP per batch.

Additional experiment. We measured the barrier and the Fisher length of the ReLU networks on the full grid, under weight matching. The barriers follow the same pattern as those of the smooth activations. They persist under NTK-lazy and collapse under μ\muP and Standard, except for the Standard MLP, which turns back up at large width (see our response to W2 / Q2 on this rebound). The barrier exponent tracks the Fisher-length exponent across the ReLU cells (first table). When these cells are added to Fig. 5, the Fisher length still explains the barrier exponent much better than the Rayleigh quotient (second table). Theorem 4.1 assumes C3C^3, so for ReLU these results test only the empirical Length–barrier relation.

Width exponents for ReLU:

ArchitectureRegimeαB\alpha_B (barrier)αLF\alpha_{\mathcal{L}_F} (Fisher length)
MLPNTK-lazy-0.19-0.11
MLPStandard+0.08+0.34
MLPμ\muP+1.19+0.82
TSNTK-lazy-0.16-0.02
TSStandard+0.65+1.15
TSμ\muP+1.19+1.35
CNNNTK-lazy+0.07+0.03
CNNStandard+0.85+2.18
CNNμ\muP+1.03+1.63

Fit of the barrier exponent in Fig. 5:

Cells in the fitR2R^2 with the Fisher lengthR2R^2 with the Rayleigh quotient
smooth activations only0.630.20
smooth activations ReLU cells0.640.22

[W1] The empirical evaluation is restricted to relatively small, residual-free MLPs and 3-layer CNNs on 10-class tasks, omitting modern architectural components that influence mode connectivity and parameter symmetries (such as residual connections or Transformer attention blocks).

Response. Thank you for raising the question of scope. Our claims concern width scaling under three parameterizations, and we built the grid for this comparison, using residual-free MLPs and a 3-layer CNN on 10-class tasks. Residual streams tie the permutation of every block to one shared permutation, and μ\muP for residual networks needs a depth scaling that we did not want to confound with the width comparison. We leave Transformers to future work.

Revision. We will state this scope in the abstract and in a limitations paragraph. In the new appendix with the ReLU results, we will also state that they test the Length–barrier relation and not the bound.

[W2 / Q2] Could you compare weight matching with an applicable data-aware alignment method on representative model pairs, or clarify which empirical conclusions are conditional on the chosen alignment?

Response. Thank you for this important question and for suggesting a concrete data-aware comparison. We ran the comparison you asked for. Before presenting the results, we first clarify which conclusions can depend on the alignment at all.

Which conclusions depend on the alignment. Any fixed alignment π\pi gives B(π)≥B∗:=min⁡πB(π)B(\pi) \ge B^* := \min_\pi B(\pi), the barrier modulo permutation. This has three consequences.

  • (a) Collapse under weight matching (μ\muP in every architecture, and Standard on the teacher–student networks and the CNN) implies collapse modulo permutation and is therefore alignment-robust.

  • (b) The persistent NTK-lazy barrier and the Standard-MLP rebound are upper bounds, so, as the reviewer suggests, they are conditional on the alignment.

  • (c) The Shape axis (∣∂F∣op\\|\partial F\\|_{\mathrm{op}}) and the endpoint uncertainty ρ∗(wA)\rho^*(w_A) are computed on single networks and are alignment-free. In contrast, LF\mathcal{L}_F, RF\mathcal{R}_F, DrelD_{\mathrm{rel}} and the coupling of Fig. 5 depend on Δ\Delta, and hence on π\pi.

Additional experiment (activation matching). As a second alignment, we used activation matching (Ainsworth et al., 2023), i.e., a Hungarian assignment on correlations of post-activation features, computed on training inputs outside the evaluation set. The table reports the median barrier BB on the MLP with GELU.

Regimennunalignedweight matchingactivation matching
NTK-lazy2561.170.590.19
NTK-lazy10241.180.820.15
NTK-lazy40961.201.070.17
Standard2560.390.0110.019
Standard10240.300.0140.014
Standard40960.450.1090.021
μ\muP2560.380.00700.013
μ\muP10240.260.00140.0028
μ\muP40960.280.00050.0012

Neither alignment dominates the other. Activation matching gives lower barriers under NTK-lazy and for Standard at n≥1024n \ge 1024, while weight matching gives lower barriers under μ\muP, and both are upper bounds on B∗B^*. The results lead to four findings.

First, the regime ordering is the same under both alignments. NTK-lazy > Standard > μ\muP holds at every width, and μ\muP collapses under both.

Second, the NTK-lazy barrier persists under activation matching. It is lower than under weight matching, but it stays nearly flat as width grows. Its Fisher length falls only weakly, while the alignment-free Shape scalar decreases. The decoupling between flattening and collapse therefore survives, and so does its Length-axis account. What does not survive is the growth of the NTK barrier with width. Weight matching barely aligns NTK-lazy networks. The hidden units it pairs are hardly more correlated than in unaligned networks, while the pairs found by activation matching are far more correlated. This is consistent with lazy weights staying close to their random initialisation.

Third, as the reviewer suspected, the Standard-MLP rebound is largely an effect of weight matching. At large width the barrier rises much more under weight matching than under activation matching (table above). This is the one case that the Fisher length does not account for. It changes little between the two alignments, while the uncertainty in the middle of the path drops several-fold.

Fourth, the Length-axis relation holds across alignments. When the alignment changes, the barrier and the Fisher length almost always move in the same direction, and log⁡B\log B and log⁡LF\log \mathcal{L}_F remain correlated within each regime.

Revision. We will state this classification in §5.1, describe the NTK-lazy barrier as persistent rather than growing, and qualify the Standard-MLP rebound accordingly.

[Q3] Appendix E.3/Figure 10 reports that cells with relatively large predictive uncertainty can still exhibit barrier collapse, whereas the conclusion says collapse occurs only when Fisher-energy reduction accompanies controlled uncertainty. Could you reconcile these statements and distinguish conditions for a useful upper bound from necessary conditions for barrier collapse?

Response. Thank you for the careful reading and for this precise question. The two statements are reconciled once the condition for the bound to be informative is separated from the condition for collapse. The sentence in §6 merges the two, and we will correct it. Below, we separate three conditions.

(i) When the bound is informative. The bound is informative when both ρ∗(w0)\rho^*(w_0) and ML∣Δ∣M_L\\|\Delta\\| are small. Relative to the Fisher term, the uncertainty term is proportional to the first and the cubic remainder to the second.

(ii) An empirically necessary condition. On our grid, the empirically necessary condition is Fisher-length reduction (in the table, small ρ∗\rho^* means ≤0.05\le 0.05). None of the 9 cells without length reduction collapses.

Fisher length fallsρ∗\rho^*cellscollapse (αB>0\alpha_B > 0)median αB\alpha_B
yessmall2119+0.84+0.84
yeslarge66+0.21+0.21
nosmall10−0.17-0.17
nolarge80−0.21-0.21

(iii) A theoretical necessary condition. Applying Lemma 4.5 to the loss along the segment, minus its linear interpolant, gives a version of the bound that needs only C2C^2 and has no gradient or cubic term. It shows that BB can exceed the endpoint floor 12∣LA−LB∣\tfrac12|L_A - L_B| only if ρ∗(γ(s)),Mf∣Δ∣2>LF(γ(s);Δ)\rho^*(\gamma(s))\\,M_f\\|\Delta\\|^2 > \mathcal{L}_F(\gamma(s);\Delta) for some ss. What matters is therefore the uncertainty along the path rather than at the endpoint, which is why a large endpoint ρ∗\rho^* does not prevent collapse.

The grid is consistent with this. Every collapsing cell is among the cells whose Fisher length falls, and the uncertainty at the barrier location falls with width in most collapsing cells and in none of the others (table below). Endpoint ρ∗\rho^*, in contrast, is nearly collinear with the regime on this grid (large under NTK-lazy, small in the feature-learning regimes), and within the feature-learning cells it explains none of αB\alpha_B. Accordingly, we no longer claim that controlled uncertainty is necessary.

Cellsnumberρ∗\rho^* at the barrier falls with widthmedian ρ∗\rho^* at the barrier, largest width
collapse25210.10
no collapse1100.77

Revision. The corrected conclusion will state that every collapsing cell reduces its Fisher length, and that small endpoint uncertainty accelerates collapse and makes the bound informative, but is not required.

Thank you for the detailed response and for running the additional ReLU and activation-matching experiments.

My specific questions are addressed, and the planned revisions---retracting the necessity claim for ρ∗\rho^*, qualifying the NTK-lazy growth and Standard-MLP rebound as L2L_2-matching effects, and adding the ReLU results and scope limitations---make the manuscript more accurate in my opinion.

That said, I will maintain my current score. While the empirical Length–barrier correlation holds across the expanded grid, some underlying limitations remain: (1) the evaluation is restricted to small residual-free MLPs and 3-layer CNNs; (2) activation-matching experiment shows that alignment algorithm choice can significantly alters barrier trajectories such that LFL_F alone doesnot capture; and (3) as also highlighted in the other discussion, theorem 4.1's bound is loose relative to the actual barrier, leaving the connection between Fisher length and barrier collapse primarily empirical.

Reviewer_2ZQmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates linear mode connectivity (LMC) in wide neural networks from the perspective of Fisher information geometry. The authors propose a Shape–Length decomposition, where Shape characterizes the local variation of the Fisher metric and the geometry of connecting paths, while Length measures the Fisher quadratic form along the displacement between independently trained solutions.

The theoretical contributions include bounds on the Christoffel symbols and sectional curvature under width-dependent metric flattening assumptions, as well as a Fisher-barrier inequality relating the interpolation barrier to Fisher energy, predictive uncertainty, and higher-order terms. The experiments cover multiple architectures, activation functions, and parameterization regimes. The main empirical observation is that Fisher metric flattening does not necessarily imply barrier collapse, whereas the full Fisher quadratic form is more strongly correlated with barrier behavior than the corresponding Rayleigh quotient.

Critical strengths
  1. Interesting perspective. The distinction between the geometric shape of the Fisher manifold and the Fisher energy of the displacement provides an interesting perspective on understanding linear mode connectivity.

  2. Theoretical analysis. The paper provides a mathematical framework connecting the Fisher information matrix, loss Hessian, geometric curvature, and interpolation barriers. The analysis is generally well organized, and the distinction between local geometric flattening and barrier reduction is conceptually useful.

Critical weaknesses
  1. The theoretical results do not fully support the main conclusion. Theorem 4.1 provides a one-sided upper bound on the interpolation barrier, which includes not only the Fisher quadratic form but also predictive uncertainty and higher-order remainder terms. The paper does not establish that these additional terms become negligible as network width increases. Therefore, it remains unclear under what conditions Fisher energy actually dominates the bound or guarantees barrier collapse. This limits the theoretical justification for the central claim that Fisher length governs barrier behavior.

  2. The geometric analysis relies on restrictive assumptions and limited validation. The width-dependent flattening result in Theorem 4.4 relies on Assumption 4.2, which is only theoretically justified for a frozen-softmax Fisher surrogate under the NTK regime. Its applicability to the true Fisher metric and other parameterization regimes is not fully established. Furthermore, the experimental geodesic–linear deviation is estimated using a first-order approximation rather than directly computing geodesics, and Appendix E.5 reports settings where this approximation becomes unreliable. These limitations weaken the generality of the proposed geometric interpretation.

Questions for authors
  1. Regarding the Fisher-barrier inequality (Theorem 4.1): The upper bound contains several terms beyond the Fisher quadratic form, including predictive uncertainty and a third-order remainder. Can the authors clarify under what conditions these additional terms become negligible as network width increases? In particular, can the authors provide empirical evidence showing that the Fisher quadratic form dominates the bound across different parameterization regimes?

  2. Regarding Assumption 4.2 and Theorem 4.4: The width-dependent flattening assumption is theoretically justified primarily for a frozen-softmax Fisher surrogate under the NTK regime. Can the authors clarify whether similar guarantees hold for the true Fisher matrix under other parameterization regimes? In particular, how is the additional softmax-dependent term controlled as network width increases, and how does the width-dependent damping parameter affect the curvature bounds?

  3. Regarding the geodesic–linear deviation experiments: The reported deviation is estimated using a first-order approximation rather than directly computing the Fisher geodesics. Moreover, Appendix E.5 reports settings where this approximation becomes unreliable. Can the authors clarify how these limitations affect the conclusions about geodesic straightening? Would it be possible to validate the approximation against directly computed geodesics for smaller networks?

5 comments

Thank you very much for your careful review and for your appreciation of our perspective on linear mode connectivity and our theoretical analysis. Below are our point-by-point responses to your questions.

[Q1] Regarding the Fisher-barrier inequality (Theorem 4.1): The upper bound contains several terms beyond the Fisher quadratic form, including predictive uncertainty and a third-order remainder. Can the authors clarify under what conditions these additional terms become negligible as network width increases? In particular, can the authors provide empirical evidence showing that the Fisher quadratic form dominates the bound across different parameterization regimes?

Response. Thank you for this insightful question. Theorem 4.1 identifies the quantities that can control the barrier, but which of them is largest is an empirical question. We therefore first derive the exact conditions under which the Fisher term dominates, and then measure all four terms.

Exact conditions. Dividing each term of R(w0)R(w_0) by T2=12LF=12∣Δ∣2RFT_2 = \tfrac12\mathcal{L}_F = \tfrac12\\|\Delta\\|^2\mathcal{R}_F gives T1T2=2∣∇L(w0)∣∣Δ∣,RF\frac{T_1}{T_2} = 2\frac{\\|\nabla L(w_0)\\|}{\\|\Delta\\|\\,\mathcal{R}_F}, T3T2=ρ∗(w0),MfRF\frac{T_3}{T_2} = \frac{\rho^*(w_0)\\,M_f}{\mathcal{R}_F} and T4T2=ML∣Δ∣3RF\frac{T_4}{T_2} = \frac{M_L\\|\Delta\\|}{3\mathcal{R}_F}. The Fisher term dominates if and only if these three ratios are small, and width alone does not guarantee this. Under NTK-lazy, ρ∗\rho^* does not decrease with width (Fig. 4b), so our theory gives no reason for T3T_3 to become small relative to T2T_2 in this regime. Under Standard and μ\muP, ρ∗(wA)\rho^*(w_A) is small at large width and far below its NTK-lazy values (Figs. 8–9). However, we have no width rate for MfM_f and MLM_L, so whether the Fisher term dominates in these two regimes is an empirical question.

Measurement of all four terms. We measured the four terms on the full MLP grid. MfM_f and MLM_L are evaluated along the segment in the direction Δ^\hat\Delta, which lower-bounds the suprema, so the reported BR^\frac{B}{\hat R} is an upper bound on the tightness ratio. The inequality holds on every pair, but it is loose. The bound exceeds the barrier by about two orders of magnitude, and the Fisher term accounts for only a small part of it (table below).

Which term carries the bound depends on the regime. Under NTK-lazy it is usually the uncertainty term, which is the gate we use to explain why the NTK barrier persists. Under Standard and μ\muP, where ρ∗\rho^* is small, it is the cubic remainder. The gradient term does not vanish with width either, because the trained endpoints are only approximate minima while ∣Δ∣\\|\Delta\\| grows. At the largest width it makes up almost half of R^\hat R under NTK-lazy and μ\muP.

Across the grid, BR^\frac{B}{\hat R} varies far less than the barrier itself, so the bound tracks the barrier up to a nearly constant factor. We stress, however, that this tracking does not come from the Fisher term. Our evidence that Fisher length governs barrier scaling is the exponent coupling in Fig. 5, rather than the size of T2T_2 in the bound.

Median share of each term in R^\hat R (T1T_1 gradient, T2T_2 Fisher, T3T_3 uncertainty, T4T_4 cubic):

RegimennT1T_1T2T_2T3T_3T4T_4
NTK-lazy6411.8%4.3%54.5%28.4%
NTK-lazy409646.9%3.5%27.6%22.6%
Standard640.9%1.0%8.5%88.5%
Standard40960.4%0.1%0.3%99.2%
μ\muP640.8%1.0%9.7%88.3%
μ\muP409647.0%1.9%14.9%37.4%

A bound without the gradient and cubic terms. Applying Lemma 4.5 to φ\varphi minus its linear interpolant yields, under C2C^2 only, B≤12∣LA−LB∣+18sup⁡s[ρ∗(γ(s)),Mf∣Δ∣2−LF(γ(s);Δ)]+B \le \tfrac12|L_A - L_B| + \tfrac18\sup_s\big[\rho^*(\gamma(s))\\,M_f\\|\Delta\\|^2 - \mathcal{L}_F(\gamma(s);\Delta)\big]_+, which removes both the gradient and the cubic term.

Revision. Contribution 3 becomes "LF\mathcal{L}_F is the only Fisher-geometric term in a valid bound and empirically tracks the barrier (Fig. 5)". We will also revise the sentence in §5.4 on ρ∗(w0)\rho^*(w_0) and term dominance, and the remark in App. D.2 on the endpoint gradient term, so that they report the measured values above.

[Q2] Regarding Assumption 4.2 and Theorem 4.4: The width-dependent flattening assumption is theoretically justified primarily for a frozen-softmax Fisher surrogate under the NTK regime. Can the authors clarify whether similar guarantees hold for the true Fisher matrix under other parameterization regimes? In particular, how is the additional softmax-dependent term controlled as network width increases, and how does the width-dependent damping parameter affect the curvature bounds?

Response. Thank you for this detailed question. We answer its three parts in turn.

(a) True Fisher and other regimes. Theorem 4.4 is conditional: it holds in any regime where Assumption 4.2 holds. We verify the assumption only for the frozen-softmax surrogate under NTK. This is stated in §3.3, and the abstract and introduction will say the same. For the true Fisher under NTK, Appendix C gives ∣∂F∣op≤c1′n+3CJ3ρS\\|\partial F\\|_{\mathrm{op}} \le \frac{c_1'}{\sqrt{n}} + 3C_J^3\rho_S. Under Standard and μ\muP, the assumption is supported by direct measurement rather than by a proof. ∣∂F∣op\\|\partial F\\|_{\mathrm{op}} decreases with width in every cell except one (Figs. 1b, 6; table below). The exception is the CNN with softplus under Standard, which shows no trend.

Width exponent α\alpha of ∣∂F∣op∼n−α\\|\partial F\\|_{\mathrm{op}} \sim n^{-\alpha}, as estimated in the paper (range over activations):

ArchitectureNTK-lazyStandardμ\muP
MLP0.86–1.160.60–1.021.19–1.93
Teacher–student1.08–1.371.32–1.681.53–2.55
CNN0.50–1.360.59–1.391.51–1.76

(b) The softmax-dependent term. This term is controlled by ρS≤(3+2),ρ∗\rho_S \le (3+\sqrt{2})\\,\rho^* instead of by width. Under Standard and μ\muP, ρ∗(wA)\rho^*(w_A) is at most 3×10−33\times 10^{-3} at the largest width in nearly all cells (Figs. 8–9), so the term is small in these regimes. Under NTK-lazy, ρ∗\rho^* does not shrink with width, and our theory thus gives no width control of this term in this regime.

(c) The damping parameter. Since ∂GF=∂F\partial G_F = \partial F, λ\lambda enters only the constants (KΓ∝1/λK_\Gamma \propto 1/\lambda, KSec∝(MF+λ)/λ3K_{\mathrm{Sec}} \propto (M_F+\lambda)/\lambda^3). With λ=10−2λmax⁡(F)\lambda = 10^{-2}\lambda_{\max}(F), the Christoffel bound of Theorem 4.4 reads ∣Γ(u,v)∣≤3β2λ,∣u∣∣v∣=150,βλmax⁡,∣u∣∣v∣,\\|\Gamma(u,v)\\| \le \frac{3\beta}{2\lambda}\\,\\|u\\|\\|v\\| = 150\\,\frac{\beta}{\lambda_{\max}}\\,\\|u\\|\\|v\\|, where β\beta bounds ∣∂F∣op\\|\partial F\\|_{\mathrm{op}}. Its width dependence is therefore set by ∣∂F∣λmax⁡(F)\frac{\\|\partial F\\|}{\lambda_{\max}(F)} and not by ∣∂F∣\\|\partial F\\| alone. Because λmax⁡(F)\lambda_{\max}(F) itself decreases with width, the damping can absorb part of the n−αn^{-\alpha} rate. This is why the paper reports width exponents rather than absolute curvature magnitudes (App. D.2), and why the curvature bound remains theory-only outside the NTK surrogate. A sweep of λ\lambda on the MLP shows this directly: under NTK-lazy, lowering λ\lambda by two orders of magnitude cuts the width exponent of DrelD_{\mathrm{rel}} almost in half.

[Q3] Regarding the geodesic–linear deviation experiments: The reported deviation is estimated using a first-order approximation rather than directly computing the Fisher geodesics. Moreover, Appendix E.5 reports settings where this approximation becomes unreliable. Can the authors clarify how these limitations affect the conclusions about geodesic straightening? Would it be possible to validate the approximation against directly computed geodesics for smaller networks?

Response. Thank you for this valuable suggestion. We first clarify how the paper uses DrelD_{\mathrm{rel}}, and then report the validation against directly computed geodesics.

Role of DrelD_{\mathrm{rel}} in the paper. The paper uses DrelD_{\mathrm{rel}} as a first-order diagnostic. It is the first-order residual of the linear path rather than a certified distance to the geodesic (App. D), and for the CNN under Standard/μ\muP it exceeds 1 at every width, so those curves are qualitative only. No central claim rests on DrelD_{\mathrm{rel}}: the Shape–barrier decoupling is established by comparing ∣∂F∣op\\|\partial F\\|_{\mathrm{op}} (Figs. 1b, 6) with the barrier.

Additional experiment. As the reviewer suggests, we validated the estimate against directly computed geodesics. On teacher–student networks with GELU and widths from 64 to 256, computed in double precision, we minimised the discrete path energy of GFG_F with fixed endpoints. Splitting the path into half as many segments changes the direct deviation by less than 10%.

Results. Under NTK-lazy, the first-order estimate is confirmed: it agrees closely with the direct deviation and decays with width at a similar rate. Under Standard and μ\muP, it is not confirmed. It overstates the direct deviation by up to a factor of two, and it also overstates how fast the deviation decays with width (table below). In every case, the geodesic stays within 8% of ∣Δ∣\\|\Delta\\| of the straight line. Under Standard and μ\muP, however, its energy is substantially lower than that of the line, whereas under NTK-lazy the two energies almost coincide (last column). At these widths, the straightening statement therefore holds for NTK-lazy only.

Median over 2 pairs:

RegimennDrelD_{\mathrm{rel}} first orderDrelD_{\mathrm{rel}} directdirect / first orderenergy below the line
NTK-lazy640.01760.01791.020.5%
NTK-lazy1280.01270.01311.030.3%
NTK-lazy2560.00900.00951.060.2%
Standard640.1260.0700.5653%
Standard1280.1370.0720.5290%
Standard2560.0740.0570.7778%
μ\muP640.1280.0700.5554%
μ\muP1280.1240.0650.5386%
μ\muP2560.0560.0430.7673%

Why the paper reports the first-order estimate. Computing a geodesic directly requires minimising the path energy over every interior node, with repeated Fisher-vector products at each step, and this cost grows with both the width and the number of segments. It is affordable for the small networks above, but too expensive for the full grid of the paper. For this reason, the paper reports the first-order estimate throughout and treats it as a diagnostic instead of a measurement of the geodesic.

Comments on the paper

1 comment

We thank all three reviewers for their careful and detailed reviews. We are glad that Reviewer 2ZQm found our perspective on linear mode connectivity interesting, that Reviewer 7Gy2 appreciated the conceptual clarity of the Shape-Length decomposition and our systematic empirical validation, and that Reviewer idLY found the central negative result well-supported. To answer their questions, we measured all four terms of the bound in Theorem 4.1, tested the Length-barrier relation on the ReLU networks, compared weight matching with activation matching, and checked our first-order estimate of the geodesic deviation against directly computed geodesics. Our replies under each review answer the questions in turn and describe the changes we will make to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.