acceptodds
Withdrawn from ICLR 2027

Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well”(Hochreiter and Schmidhuber, 1994; Keskar et al., 2017), where flatness can be measured by the trace of the Hessian of the empirical loss. However, Dinh et al. 2017) showed that, using symmetry of the network that can change flatness while keeping the population and empirical losses unchanged, any interpolator can be made sharper or flatter. This result makes the earlier heuristic statement vacuous. In this paper, we show that for learning an unknown multi-index model with -layer non-convex homogeneous neural networks, there is a connection between flatness and generalization, despite the existence of symmetries. This connection pertains to the “flattest” interpolators, i.e., the interpolators that have orderwise minimum flatness among all interpolators. First, we show that there exists a natural class of non-generalizing interpolators whose flatness cannot be made closer to the flattest possible, even using symmetries. Second, we show that for data generated by a sum of single-index models, if the approximation error and label noise are low, any flattest interpolator achieves small population loss, i.e., the flattest interpolators always generalize. This establishes a direct link between flatness and generalization which applies to a large class of activations and realistic data distributions.

Primary area:
learning theory
Keywords:
Flatness, generalization, multi-index models, homogeneity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZpL7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies width–depth limits during training in linear residual networks. It shows that ordinary gradient descent admits a joint limit in probability but can fail to converge in L2L^2. With gradient clipping, it establishes L2L^2 convergence with explicit width and depth rates.

Critical strengths

The comparison after optimizing over rescaling is useful, since arbitrary parameter scaling can otherwise obscure differences between interpolators. The extension beyond ReLU and quadratic activations is also of interest.

Critical weaknesses

I think the paper has two main shortcomings, outlined below.

  • Correctness: Proposition 3 is false as stated for ReLU. Taking B∼N(−2log⁡n,1)B\sim N(-\sqrt{2\log n},1) satisfies the restrictions on the Gaussian parameters in Assumption 2, but Var⁡[ReLU⁡(B)]→0\operatorname{Var}[\operatorname{ReLU}(B)]\to0, contradicting the required constant lower bound. The proof also contains substantive errors: the mean lower bound on page 42 does not account for cancellation between the two branches, and applying it to the derivative on page 43 is unjustified. The conditional variance bound is subsequently used in Lemmas 9–10 in the proof of Theorem 2. Thus, the claimed coverage of ReLU targets is not established. It is unclear whether the subsequent results can be recovered for these targets under the current assumptions.
  • Presentation: The theoretical analysis is very technical and difficult to follow. It would help to explain the main ideas and the scope of each result more clearly.
Questions for authors
  1. Can Proposition 3 be corrected to cover ReLU targets, and which downstream results remain valid after the correction? In particular, does the conditional variance argument require stronger assumptions?
  2. Does the lower bound on B(θ)B(\theta) in Eq. (1) hold uniformly over data-dependent directions? This seems necessary to apply the generalization result to all flattest interpolators.
Reviewer_K2RJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the relationship between flatness and generalization for two-layer homogeneous neural networks learning multi-index models. The authors characterize the minimum Hessian trace achievable through neuron rescaling, show that a particular class of poorly generalizing interpolators remains above the optimal sharpness scale, and establish generalization guarantees through a bound on the normalized output weights. My main concerns are with the interpretation and presentation of these results. The symmetry argument establishes that sharpness depends on the parameterization, but does not by itself invalidate a relationship between sharpness and generalization for solutions reached by a particular optimizer or architecture. Theorem 2 only excludes a restricted class of bad interpolators from being flattest, so its role in supporting the broader claim needs clarification. More importantly, the introduction appears to reverse the implication in Proposition 2: near-optimal sharpness implies bounded (\ell_1) norm under additional conditions, whereas bounded (\ell_1) norm does not generally imply near-optimal sharpness. The abstract should also explain what “closer to the flattest possible” means, Section 1.1 should define (\Upsilon^\star), and Theorem 1 should clarify the source of randomness and whether its lower bound holds uniformly over data-dependent hidden directions.

Critical strengths

The paper addresses an interesting question about the relationship between flatness and generalization in the presence of neural network symmetries. The characterization of minimum sharpness under rescaling and its connection to the norm of the output weights is a useful theoretical insight. The generalization result for sufficiently flat interpolators is also interesting, particularly because it applies to a relatively broad class of homogeneous activations, including ReLU and quadratic activations.

Critical weaknesses

Motivation and interpretation of symmetries: I disagree with the claim in the abstract that the existence of neural network symmetries makes the flatness-generalization heuristic vacuous. The fact that rescaling parameters can change sharpness without changing the represented function does not rule out the possibility that optimization procedures favoring flatter solutions lead to better generalization. In particular, different parameterizations or architectures, potentially exploiting symmetries in the data, can change optimization trajectories and convergence points. I would appreciate clarification of the authors' intended argument.

Unclear statements in the abstract: The phrase "cannot be made closer to the flattest possible, even using symmetries" is unclear. What exactly does it mean for an interpolator to become closer to the flattest possible? I suggest explaining this in terms of the multiplicative gap in sharpness.

Missing definition: The quantity (\Upsilon^\star) is used in Section 1.1 without being defined.

Theorem 1: It is unclear what randomness the high-probability statement is over and whether the lower bound holds uniformly over all interpolators or only for fixed hidden directions selected independently of the data. In particular, does the theorem leave open the possibility of data-dependent interpolators whose sharpness is smaller than the claimed order of (\Upsilon^\star)?

Limited implications of Theorem 2: Theorem 2 only shows that a particular class of bad interpolators cannot achieve minimum sharpness. However, this does not rule out other poorly generalizing interpolators with near-optimal sharpness. I find this result relatively weak in supporting the broader motivation of the paper, and I would like the authors to explain its significance more clearly.

Incorrect direction of implication: In Section 1.1, the authors appear to claim that interpolators with (|a|_1=\widetilde O(1)) achieve minimum sharpness. However, Proposition 2 establishes the reverse implication, under an additional condition. For (c>1), bounded (\ell_1) norm does not generally imply near-optimal sharpness. This also suggests that well-generalizing interpolators can have substantially suboptimal sharpness. While this does not contradict the stated sufficient condition for generalization, I think the distinction is important and the claims should be clarified.

Questions for authors

Why does the existence of rescaling symmetries make the flatness-generalization heuristic vacuous? Couldn't optimization toward flatter solutions still lead to better generalization?

In Theorem 1, what is the high-probability statement over? Does the lower bound also hold for data-dependent hidden directions?

Why does Section 1.1 suggest that bounded (\ell_1) norm implies minimum sharpness, when Proposition 2 only proves the reverse implication?

What is the significance of Theorem 2 if it only rules out a restricted class of bad interpolators from achieving minimum sharpness?

Reviewer_mh6SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the relationship between flatness and generalization when learning multi-index models with Gaussian inputs using two-layer homogeneous neural networks. It characterizes the minimum Hessian trace within each equivalence class induced by rescaling symmetry, derives lower bounds on the flatness of a specified class of bad interpolators, and establishes generalization guarantees under small label noise, low approximation error, and additional assumptions. The numerical experiments primarily fix the unit-norm inner-layer weights and optimize the outer-layer weights to examine the relationship between minimum flatness and population loss. For some ReLU settings, the paper also reports explicit interpolators obtained by repeating the true link function.

Critical strengths

The characterization of minimum flatness within an equivalence class under rescaling symmetry connects the Hessian trace to a weighted ℓpp\ell_p^p objective in the outer-layer weights, where p=2c/(c+1)p=2c/(c+1). This provides a useful tool for analyzing flatness under rescaling symmetry. Applying this analysis to the class of homogeneous activations considered in the paper has technical value.

Critical weaknesses
  1. Counterexample to the conditional mean bounds. Assumption 2 provides only an O(⋅)O(\cdot) upper bound on the mean norm, whereas Lemma 9 asserts a Θ(⋅)\Theta(\cdot) bound that fails for an admissible linear link function.
    Take c=1c=1, m⋆=1m^\star=1, and σ⋆(t)=t\sigma^\star(t)=t, which satisfies Assumption 2, with ReLU as the network activation, satisfying Assumption 1. For fixed inner-layer weights that are all orthogonal to the true direction, Gaussian independence gives E[yi⋆∣U^i]=0,μ~i=0,ς=1.\mathbb{E}[y_i^\star \mid \widehat{U}_i]=0, \widetilde{\mu}_i=0, \varsigma=1. Lemma 9 nevertheless requires the conditional mean norm to be Θ(1)\Theta(1). The corresponding quadratic-form lower bound in Lemma 10 also fails. Appendix E.4.4 invokes Assumption 2 without justifying this additional lower bound. The authors should correct these bounds and clarify whether Theorem 2 retains the claimed order of its flatness lower bound.

  2. Proposition 3 conflicts with the variance requirement for ReLU. Proposition 3 states that every activation satisfying Assumption 1 also satisfies Assumption 2. For ReLU, this claim conflicts with the uniform variance lower bound. Let Z∼N(0,1)Z\sim N(0,1) and consider the Gaussian input bd=Z−log⁡db_d=Z-\sqrt{\log d}. Its mean lies within the permitted O~(1)\widetilde O(1) range, but Var⁡[(bd)+]≤E[(Z−log⁡d)+2]→0.\operatorname{Var}[(b_d)_+] \leq \mathbb{E}[(Z-\sqrt{\log d})_+^2] \to 0. In contrast, when c=1c=1 and the input variance is 1, Assumption 2 requires an Ω(1)\Omega(1) lower bound on the output variance. The paper defines its asymptotic constants to be independent of m,n,dm,n,d, so the hidden constant cannot vary with this sequence of means.

  3. Limited scope of the experimental validation. The numerical optimization in Eq. (2) fixes the unit-norm inner-layer weights and optimizes only the outer-layer weights. It therefore computes the minimum flatness achievable with that fixed set of directions, rather than directly validating the flattest interpolator over the full parameter space. Moreover, the experiments control the correlation of each inner-layer weight with the subspace of true directions, whereas Definition 2 bounds the maximum correlation between unit vectors in the two subspaces; the former does not imply the latter. The experiments fix d=100,n=500,m=5000d=100,n=500,m=5000 and consider only c=1,2c=1,2. Thus, they do not test the predicted dependence on problem size or higher degrees of homogeneity.

  4. The claimed special cases require a precise correspondence. The paper describes Theorem 3.1 of Wen et al. and Theorem 7.1 of Ding et al. as special cases of Theorem 3, but does not explain how the cited results follow from that theorem. The cited theorem of Wen et al. concerns an XOR distribution on a discrete hypercube, whereas the paper assumes Gaussian inputs. The quadratic network in Ding et al. fixes the outer-layer weights to ±1\pm1 and takes the Hessian only with respect to the inner-layer weights, whereas the paper includes both layers. The authors should explain the correspondence between the input distributions, parameterizations, and flatness objectives underlying these special-case claims.

  5. Inconsistencies in formulas and notation. The exponent of ∣aj∣|a_j| in Lemma 1 should be 2c/(c+1)2c/(c+1), consistent with Lemma 3 and Eq. (5). The exact expression for B(θj)B(\theta_j) also omits the factor c−c/(c+1)+c1/(c+1)c^{-c/(c+1)}+c^{1/(c+1)}. The parameter lists in Definition 2 and Theorem 2 are inconsistent. On p. 9, line 437, the text refers to "Condition 4" of Definition 2, although the definition contains only three conditions.

Questions for authors
  1. Can the lower bound in Theorem 2 be established at the stated order without the claimed positive lower bounds involving the conditional mean? Please explain how Lemmas 9 and 10 should be revised and how these changes affect the subsequent proof.

  2. On p. 32, lines 1686-1699, the conditional mean term is omitted on the grounds that the correlation is small. How does the stated correlation condition control this term at the scale needed to establish ∥amin⁡,ℓ2∥∞∥amin⁡,ℓ2∥2=O~(m−1/2)\frac{\lVert a_{\min,\ell_2}\rVert_\infty}{\lVert a_{\min,\ell_2}\rVert_2}=\widetilde O(m^{-1/2})?

  3. Does Proposition 3 require a restriction on the allowable Gaussian means or a narrower class of link functions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.