- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies various properties of two-layer neural networks when trained on a multi-index Gaussian regression task. In particular, it studies:
- the case where the parameters of the model belong to a fixed compact set;
- the excess risk of the family of two-layer neural networks with parameters belonging to a compact set.
The results are:
- the infimum of the excess risk on the compact set of models is strictly greater than 0 if the model does not belong to this set (Proposition 3.1);
- if the target function can be represented by bounded parameters, then there exists an "equivalent" function with known regularity (Theorem 3.2);
- if belongs to the compact class of models, then there is a bound on the excess risk , where is the regularity of the activation function and is the dimension of the input data of the model (Theorem 4.1);
- without the compact hypothesis: if is regular enough, then there is a bound on the excess risk (Theorem 5.1).
Critical strengths
Clarity
The paper is easy to understand, and proofs are presented with a "proof idea" at the top.
Novelty
To my knowledge, these results on Gaussian multi-index regression are novel.
Critical weaknesses
Relation to prior work
Section 6 provides references to prior work, but is not specific. For instance, there is no comparison between the obtained bounds (e.g., Th. 5.1) and existing bounds, and no comparison of the setup. Thus, the novelty and relevance of the work is difficult to evaluate.
Motivation
The initial motivation of this paper is apparently that compact parameter restrictions can silently change the learning problem, but it does not cite any article where it is an issue.
Also, norm-based analyses of two-layer networks (for instance, [A] and [B]) already deal with unbounded parameters and provide rates of the same type.
[A] Approximation and estimation bounds for artificial neural networks, Barron, 1994.
[B] A priori estimates of the population risk for two-layer neural networks, E et al., 2019.
Undiscussed hypotheses and results
The results, conditions and parameters are not easy to interpret.
Theorem 3.2. This theorem is meant to deal with the "compact representability" of , but I think there is an issue with it. Assume that is representable by a 2-layer neural network. Then is "bounded" in some way (even if it is not bounded by ), and the result should not change. The fact that should not affect the result.
Theorem 5.1. The role played by the regularity parameter is very unclear:
- is the regularity of the target functions (see definition of );
- is assumed to be bounded by above: are then assumed to be "not too regular"… why? it is surprising…
- also appears in Eqn. (4): the excess risk is bounded by ; however, given Condition (2) on , one can easily bound the excess risk by , which depends neither on , nor on , only on the input dimension .
Questions for authors
Can you state Th. 5.1 with the rate (up to logs) directly?
Why do you need an upper bound on in Equation (2)?
About the results in a compact parameter space: which existing papers are affected by your result?
General Comment
We thank all reviewers for their careful reading and constructive feedback. Several reviews raised closely related questions about the novelty and statistical significance of our work, particularly its distinction from existing approximation theory and MFLD feature-learning analyses. Hence, we begin by clarifying our central contribution and its implications for statistical learning theory.
1. Feature learning theory with statistical guarantee under noncompact parameter. A main contribution of this work lies in being the first to connect approximation, statistical estimation, and feature learning in the long-time limit of noncompact MFLD for a standard nonparametric multi-index model setting. Rather than assuming representability, we construct a fixed infinite-dimensional family of rough multi-index targets and establish a polynomial prediction guarantee together with statistical feature learning and recovery of the unknown index subspace.
2. Fixed parameter bounds can change the statistical meaning of learning guarantees. Another main contribution is implications about fixed parameter constraints. Hard constraints introduced for the convenience of theoretical analysis are not necessarily statistically harmless. The examples in App. B.5 show that a generalization gap bound or an MFLD convergence guarantee can hold while the imposed fixed parameter constraints prevent consistent estimation. We demonstrate why optimization and generalization guarantees must be investigated and interpreted together with the approximation properties of the constrained function class.
3. Takeaways for ICLR readers.
Here, we highlight new insight ICLR readers would take away from the paper.
Main Revision.