acceptodds
Under review as a conference paper at ICLR 2027

Data Geometry and the Metric Entropy of Infinite-Width ReLU Networks

Abstract

The complexity of an infinite-width network is measured in , so it is a property of the data distribution and not of the architecture alone. For ReLU ridge features that dependence runs through a single quantity: the smoothness of the map from parameters to features, which is fixed by how much probability mass sits near moving activation boundaries. An anti-concentration, or margin, exponent for those boundaries therefore determines the metric entropy of the model class. We first show that this exponent can never exceed one, whatever the data, so a margin law imposed uniformly over the parameter space can only degrade the classical smoothness of ReLU features and never improve it. We then show that the degradation it predicts can be an artifact of that uniformity, and is so in the cases that motivate the question. Our main result is a stratified entropy law, in which the parameter smoothness is allowed to fall off on a set of positive codimension at a controlled rate; the resulting entropy exponent is governed by the better of two competing terms, and reduces to the smooth-dictionary bound of Siegel and Xu (2024) w when the bad set is everything. The motivating case is data concentrated on a curved low-dimensional manifold, where the margin exponent does drop, but only at parameters whose activation boundary is tangent to the data — a set of codimension one. There the stratified law returns the undegraded classical exponent, reconciling the margin mechanism with intrinsic-dimension theory, and we delimit when a margin exponent below one arises at all. The predicted local exponents, and the rate at which the smoothness degrades near the bad set, are confirmed numerically to four digits.

Primary area:
learning theory
Keywords:
learning theory, metric entropy, variation spaces, infinite-width neural networks, ReLU networks, approximation theory, anti-concentration, intrinsic dimension

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sioQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The paper bounds the metric entropy, in L2(PX)L^2(P_X), of the variation ball FR=R aco‾{ϕθ}\mathcal F_R = R\,\overline{\mathrm{aco}}\{\phi_\theta\} of infinite-width two-layer ReLU$^k$ networks. The data distribution enters through a margin (anti-concentration) exponent α\alpha: PX(∣θ⊤X~∣≤t)≲tαP_X(|\theta^\top\tilde X|\le t)\lesssim t^\alpha.

  • Mechanism. The parameter-to-feature map then has smoothness s=k+α/2s=k+\alpha/2 (Eq. 3), and the smooth-dictionary bound of Siegel & Xu (2024) gives em≲m−1/2−s/de_m\lesssim m^{-1/2-s/d}. Since a free bias forces α≤1\alpha\le1 (Prop. 7), a margin law imposed uniformly can only worsen the classical exponent 12+(k+12)/d\tfrac12+(k+\tfrac12)/d, e.g. for data on curves.
  • Main result (Thm. 3). It is a stratified version of the Siegel–Xu bound. The map may be only s0s_0-smooth near a bad set BB of codimension ν\nu, and s1s_1-smooth away from BB with a constant that blows up at rate κ\kappa. Under κ<νs1/p\kappa<\nu s_1/p, em≲m−1/2−γ⋆e_m\lesssim m^{-1/2-\gamma^\star} with γ⋆=min⁡{s1/p, s0/(p−ν)}\gamma^\star=\min\{s_1/p,\ s_0/(p-\nu)\}.
  • Applications. Thm. 10 turns a stratified margin law into these hypotheses. This recovers the classical exponent for arclength measure on a planar curve (Ex. 11) and for a density with a hyperplane singularity (Ex. 14). Prop. 12 delimits when α<1\alpha<1 arises.
  • Statistics and numerics. Cor. 16 and Thm. 17 translate entropy into learning rates, with a sharp translation-family example. Fig. 1 checks the local smoothness exponents numerically.
Critical strengths
  1. A clean and general localisation of the Siegel–Xu bound. Thm. 3 applies to Hilbert-valued parameter maps and uses a transparent shell decomposition around BB (App. A). It recovers Siegel–Xu at both extremes ν=0\nu=0 and ν=p\nu=p (Remark 5) and may be reusable beyond ReLU.
  2. A simple, illuminating mechanism. Eq. (3) isolates the only distributional input, the mass of a thin slab around the activation boundary. Prop. 12(i)–(ii) identify the order of contact between the data and hyperplanes as what governs α0\alpha_0.
  3. Careful positioning of the claims. Table 1 summarizes the setting. The paper states explicitly that the statistical route is standard (l. 152–154), discusses population vs. empirical entropy (Cor. 16, App. E), and provides a two-sided sharpness example (Thm. 17).
Critical weaknesses
  1. The degradation being repaired may be an artifact of the proof route.
    • For any PXP_X on a bounded domain, ∥f∥L2(PX)≤∥f∥∞\|f\|_{L^2(P_X)}\le\|f\|_\infty.
    • Siegel (arXiv 2307.15285, Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks, Thm. 3) gives nn-term ReLU$^k$ approximation in L∞(Bd)L^\infty(B^d) with ℓ1\ell^1-bounded coefficients at rate n−1/2−(2k+1)/(2d)n^{-1/2-(2k+1)/(2d)}. That is exactly the classical exponent.
    • Discretizing the parameters of such approximants would then give em(FR;L2(PX))≲(m/log⁡m)−1/2−(k+1/2)/de_m(\mathcal F_R;L^2(P_X))\lesssim(m/\log m)^{-1/2-(k+1/2)/d} for every such PXP_X and k≥1k\ge1. This would include the curves in d≥3d\ge3 that the paper leaves open.
    • If this is right, the stratified analysis gains at most a logarithmic factor in the paper's ReLU$^k$ applications. The claim that the margin exponent "determines the metric entropy" (abstract) would then need qualification.
    • This work is not cited. Yang & Zhou (2025), which is cited, may also cover general covariate laws; I have not checked their exact statement.
    • See Q1–Q2.
  2. The "reconciliation with intrinsic-dimension theory" appears overstated.
    • Ex. 11 recovers m−1/2−(k+1/2)/2m^{-1/2-(k+1/2)/2}, which is the exponent for the ambient dimension d=2d=2, not one governed by the curve's dimension 1.
    • So the result shows that curve data are no worse than full-dimensional data, whereas intrinsic-dimension results (e.g. Nakada & Imaizumi, 2020) lead one to expect better rates.
    • On the unit circle, each feature restricted to the curve seems to lie in a bounded set of BVk+1BV^{k+1}. That would suggest em≲m−(k+1)e_m\lesssim m^{-(k+1)}, i.e. m−2m^{-2} instead of m−5/4m^{-5/4} for k=1k=1. I have not verified this formally.
    • Also, two upper bounds cannot "conflict" (l. 61–63); the earlier bound was simply not tight.
    • See Q3.
  3. The applications are few, one of them appears vacuous, and the numerical check is weak evidence.
    • Only Ex. 11 and Ex. 14 instantiate the machinery. Curves in d≥3d\ge3 are left open (l. 470–471), and hypersurfaces need no repair (Prop. 12(ii)).
    • Prop. 12(iii) appears vacuous in codimension ≥2\ge2. At a point q∗q^\ast of MM farthest from the origin, HnH_n is definite for n=q∗/∣q∗∣n=q^\ast/|q^\ast|. This contradicts the constraint a=ba=b derived at l. 350–354. So the hypothesis cannot hold for any compact submanifold without boundary, not only for odd rr as stated (l. 353–354).
    • Fig. 1 covers only k=1k=1 on the circle and checks the smoothness exponents of Eq. (3) on one example. It does not test the entropy theorem itself, so "confirmed numerically to four digits" (abstract) is weaker evidence than the wording suggests.
    • See Q4–Q5.
Questions for authors
  1. Does the L∞L^\infty route via Siegel (arXiv 2307.15285, Thm. 3) already give your ReLU$^k$ rates, up to a logarithmic factor, for every PXP_X on a bounded domain? If so, in which settings do Thm. 3 / Thm. 10 give rates that are not available otherwise?
  2. How does Cor. 16 compare with Yang & Zhou (2025) and Parhi & Nowak (2023) under general covariate laws?
  3. On the unit circle, is F1\mathcal F_1 restricted to the curve contained in a bounded subset of BVk+1BV^{k+1}, so that em≲m−(k+1)e_m\lesssim m^{-(k+1)}? If so, how would Remark 15 and Problem 18 change?
  4. Does "compact submanifold" in Prop. 12(iii) exclude a boundary? If so, can you give an example in codimension ≥2\ge2 that satisfies its hypothesis, in view of the farthest-point argument?
  5. Is the fitted prefactor in Fig. 1 (≈ 0.388) available in closed form? Could you add k≥2k\ge2, other curves, or a direct numerical estimate of the entropy?
  6. Cor. 16 needs an εn\varepsilon_n-net in L2(PX)L^2(P_X), i.e. knowledge of PXP_X. Can the estimator be made data-driven, e.g. via empirical risk minimization?
  7. Minor: in Ex. 11 the bad set lies on the boundary of Θ\Theta when the bias range is [−1,1][-1,1]. How do the cut-offs of App. B handle this? Where is it shown that (†) fails for curves in d≥3d\ge3?
Reviewer_7MkPOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the complexity of infinite width ReLU networks with bounded norm under different input distributions. Prior work uses smoothness guarantee that holds across all neuron parameters. For curved input support, parameters defining tangent activation boundaries have worse smoothness, but occupy a lower-dimensional subset. The authors develop a bound that accounts for this subset’s dimension. Under some conditions, they obtain stronger complexity bounds than the prior work and corresponding statistical estimation guarantees.

Critical strengths

Based on my limited understanding of the paper and learning theory I am not sure if the results of this paper are significant contribution compare to the prior work.

Critical weaknesses

Authors have not shown the complexity bounds using the prior work on uniform circle dataset. It seems one can obtain similar or better bounds using prior work after reparameterization. And this will reduce the significance of proposed idea. Author should clarify this.

Questions for authors

See weaknesses.

Reviewer_G5rWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the metric entropy of infinite-width ReLU networks under the data-dependent norm L2(PX)L^2(P_X). The main motivation is that the smoothness of ReLU features with respect to their parameters depends on the probability mass near activation boundaries. For data supported on low-dimensional manifolds, this smoothness may deteriorate at certain parameters, for example when the activation boundary is tangent to the manifold.

The main theoretical contribution is a stratified entropy bound (Theorem 3), which allows the feature map to have different smoothness levels across the parameter space. The bound takes into account the codimension of the exceptional parameter set and the rate at which the smoothness constants deteriorate near it. The authors then connect this framework to ReLU features through stratified anti-concentration assumptions (Theorem 10). Two examples demonstrate that the resulting entropy bounds can recover the classical exponent even when a uniform smoothness analysis gives a worse bound. The paper also derives statistical learning rates and establishes matching entropy and minimax lower bounds for a separate family of translation features.

Critical strengths

1. A useful extension of existing entropy bounds. The paper extends the results of Siegel and Xu (2024) by considering how the output of a ReLU neuron changes as its weights and bias are varied. In particular, while this dependence is generally smooth, it can become less regular for certain parameter choices, such as when the activation boundary is tangent to the data distribution. Rather than using the worst-case smoothness across all possible parameters, the authors take into account that these problematic parameters may form only a small, lower-dimensional subset of the parameter space. This allows them to obtain improved entropy bounds in certain settings, which seems to be a useful theoretical contribution.

2. Interesting connection between data geometry and neural network complexity. The paper provides an interesting perspective on how the geometry of the data distribution affects the smoothness of ReLU features and, consequently, their metric entropy. The examples involving curved manifolds and singular densities illustrate cases where a uniform analysis can be overly pessimistic. I also appreciate that the authors explicitly discuss the limitations of their results.

Critical weaknesses

1. Unclear extent of technical novelty relative to existing work. Although Theorem 3 provides a stronger result under nonuniform smoothness assumptions, much of its proof appears closely related to the multiscale approximation and entropy techniques of Siegel and Xu (2024). From my understanding, the main additional ingredient is the decomposition into shells around the exceptional set, followed by corresponding counting and summability arguments. I am somewhat unsure whether this constitutes a substantial technical advance or a relatively direct extension of the existing framework.

2. Limited understanding of the applicability and robustness of the data-dependent results. The distributional examples are informative, but they rely on relatively specific geometric structures. It remains unclear how broadly the assumptions can be verified for more realistic distributions, particularly data concentrated near manifolds with some noise, or distributions containing discrete components or mixtures of different geometries. Since data-dependent complexity is a central motivation of the paper, I would have liked a better understanding of the robustness and generality of the proposed framework.

Questions for authors

1. Technical novelty relative to Siegel and Xu (2024). Several parts of the proof of Theorem 3, particularly the multiscale dictionary estimates and entropy allocation arguments, appear closely related to the techniques of Siegel and Xu (2024). Could the proof be simplified by reusing more of their existing results? Alternatively, could the authors clarify the main additional technical difficulties introduced by the stratification and which steps require substantially different arguments? This would help me better assess the technical novelty.

2. Robustness to noisy manifold-supported data. The examples consider data supported exactly on low-dimensional manifolds. In practice, however, data is more likely to be concentrated near a manifold with some noise. How would the results change in such settings? In particular, would the stratified analysis still provide meaningful improvements, and how sensitive are the bounds to the noise level?

3. Applicability to mixtures of distributions. Could the proposed framework handle mixtures of distributions with different geometric properties, such as data supported on multiple manifolds or subspaces, or distributions containing a small discrete component? It would be useful to understand whether the stratified assumptions can still be verified in these settings and whether they yield improvements over the uniform smoothness analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.