- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether overparameterized neural networks can generalize without relying on the implicit bias of a specific optimization algorithm. The authors consider an idealized learning rule that samples uniformly from the parameters of a large ReLU network that approximately interpolates the training data. A key technical contribution is the "deep lifting" construction, which maps a smaller network to a set of parameters within a larger architecture that yields approximately equivalent behavior. By introducing the concept of "structural stability," the authors derive generalization bounds that depend on the complexity of a smaller "witness" network rather than the total number of parameters. The analysis also addresses scenarios involving corrupted labels and unrealizable targets, and includes applications to Hölder and Barron function classes. The results demonstrate that, even when interpolating solutions are sampled uniformly, the network parameterization itself inherently favors simpler functions.
Critical strengths
-
A meaningful theoretical extension of existing results. In the continuous setting, the parameters corresponding to exact copies of a smaller network generally occupy zero Lebesgue measure, so the counting arguments used in previous work no longer apply directly. The positive-volume lifting construction in Section 4, particularly Lemma 10, addresses this difficulty.
-
A general framework connecting network stability, approximation, and generalization. Theorem 3 establishes a generalization rate of approximately () when the target can be approximated by a smaller network. The structured-stability framework is particularly useful because it makes the dependence on the witness network explicit. The extensions to corrupted labels (Theorem 6) and unrealizable targets (Theorem 8) further strengthen the contribution. I also appreciate the Hölder and Barron examples, which illustrate how the general theory can produce concrete statistical rates.
Critical weaknesses
-
The interpretation depends on an idealized sampling rule and parameterization. Uniform sampling from the interpolating parameter set is computationally intractable in general. Different parameterizations can induce substantially different distributions over functions. Consequently, the results establish an architectural bias under a specific parameter-space measure, but do not necessarily explain the behavior of practical optimization algorithms.
-
The generalization guarantees depend on restrictive conditions, particularly in the noisy setting.Theorem 3 requires a uniformly accurate smaller witness network, and the bound depends on the witness depth and stability properties. It remains unclear how tight these dependencies are. The corrupted-label results also require additional assumptions on the input distribution, parameter bounds, and network capacity. In particular, Corollary 7 establishes a tempered-overfitting-type guarantee at selected sample sizes, rather than an asymptotic guarantee for a fixed corruption rate.
Questions for authors
-
How essential is the coordinate-wise uniform distribution to the deep lifting argument?
-
Corollary 7 provides a tempered-overfitting-type result at a particular sample size, while the authors note that the bound eventually deteriorates logarithmically. Is this deterioration inherent to the proposed analysis, or is there a specific obstacle preventing an asymptotic result?