acceptodds
Under review as a conference paper at ICLR 2027

When Parameter Bounds Limit Learning: Approximation and Estimation in Mean-Field Neural Networks

Abstract

Learning theory for neural networks often imposes hard parameter bounds for tractability, but these restrictions can change what a model represents rather than merely simplify proofs. We study this in Gaussian multi-index regression, where targets depend on an unknown low-dimensional subspace, using mean-field networks and mean-field Langevin dynamics (MFLD) under compact and noncompact parameterizations. For a fixed compact parameter domain, the representable predictors form a closed class, so targets outside it incur an irreducible approximation error unaffected by sample size or training time. Even exactly represented multi-index targets inherit smoothness from the activation, showing that bounded weights restrict the regularity of admissible link functions. For these represented targets, the long-time limit of MFLD satisfies a finite-sample prediction upper bound that depends on the ambient dimension. By contrast, on a fixed infinite-dimensional family of rough multi-index targets separated from every fixed compact class, the long-time limit of MFLD over a noncompact model with unbounded input weights and biases achieves polynomially decaying prediction error, accompanied by feature learning. Thus hard constraints and soft regularization can define different learning problems, making approximation the first step in interpreting statistical and convergence guarantees.

Primary area:
learning theory
Keywords:
statistical learning theory, mean-field neural networks, feature learning, multi-index regression

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_92hmOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies various properties of two-layer neural networks when trained on a multi-index Gaussian regression task. In particular, it studies:

  • the case where the parameters of the model belong to a fixed compact set;
  • the excess risk of the family of two-layer neural networks with parameters belonging to a compact set.

The results are:

  1. the infimum of the excess risk on the compact set of models is strictly greater than 0 if the model does not belong to this set (Proposition 3.1);
  2. if the target function f⋆f^{\star} can be represented by bounded parameters, then there exists an "equivalent" function with known regularity (Theorem 3.2);
  3. if f⋆f^{\star} belongs to the compact class of models, then there is a bound on the excess risk ∝N−2α/(2α+d)\propto N^{-2 \alpha / (2 \alpha + d)}, where 2α2 \alpha is the regularity of the activation function and dd is the dimension of the input data of the model (Theorem 4.1);
  4. without the compact hypothesis: if f⋆f^{\star} is regular enough, then there is a bound on the excess risk ∝N−2s/(2s+m∗)\propto N^{- 2 s / (2 s + m^*)} (Theorem 5.1).
Critical strengths

Clarity

The paper is easy to understand, and proofs are presented with a "proof idea" at the top.

Novelty

To my knowledge, these results on Gaussian multi-index regression are novel.

Critical weaknesses

Relation to prior work

Section 6 provides references to prior work, but is not specific. For instance, there is no comparison between the obtained bounds (e.g., Th. 5.1) and existing bounds, and no comparison of the setup. Thus, the novelty and relevance of the work is difficult to evaluate.

Motivation

The initial motivation of this paper is apparently that compact parameter restrictions can silently change the learning problem, but it does not cite any article where it is an issue.

Also, norm-based analyses of two-layer networks (for instance, [A] and [B]) already deal with unbounded parameters and provide rates of the same type.

[A] Approximation and estimation bounds for artificial neural networks, Barron, 1994.

[B] A priori estimates of the population risk for two-layer neural networks, E et al., 2019.

Undiscussed hypotheses and results

The results, conditions and parameters are not easy to interpret.

Theorem 3.2. This theorem is meant to deal with the "compact representability" of f⋆f^{\star}, but I think there is an issue with it. Assume that f⋆f^{\star} is representable by a 2-layer neural network. Then ∥w∥2\|w\|_2 is "bounded" in some way (even if it is not bounded by 11), and the result should not change. The fact that ∥w∥≤1\|w\| \leq 1 should not affect the result.

Theorem 5.1. The role played by the regularity parameter ss is very unclear:

  • ss is the regularity of the target functions hh (see definition of G\mathcal{G});
  • ss is assumed to be bounded by above: hh are then assumed to be "not too regular"… why? it is surprising…
  • ss also appears in Eqn. (4): the excess risk is bounded by CN−2s/(2s+m∗)C N^{-2 s / (2 s + m^*)}; however, given Condition (2) on ss, one can easily bound the excess risk by CN−(1+5/d)/(2+5/d)=CN−βdC N^{-(1+5/d) / (2+ 5/d)} = C N^{-\beta_d}, which depends neither on ss, nor on m∗m^*, only on the input dimension dd.
Questions for authors

Can you state Th. 5.1 with the rate N−(d+5)/(2d+5)N^{-(d+5)/(2d+5)} (up to logs) directly?

Why do you need an upper bound on ss in Equation (2)?

About the results in a compact parameter space: which existing papers are affected by your result?

Reviewer_BhBWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors prove that with a compact first-layer weights and biases, the space of functions two-layer neural networks can represent is compact and cannot approximate functions that are not within its weight class with arbitrarily small error. The authors also prove that in this compact space, if the activation function is sufficiently smooth, sample complexity can be close to parametric learning. On the other hand, noncompact weight spaces can approximate and learn functions that are not exactly representable with arbitrary accuracy. The authors also study feature learning and subspace recovery bounds for noncompact weight spaces.

Critical strengths

The paper is relatively well-written and clear, and provides a clear separation between the approximation capability of families of models with compact and noncompact weights.

Critical weaknesses
  • I don't think the results of the paper are particularly surprising or teach enough new insights to the reader at the level of the ICLR venue. The fact that weight norms should be allowed to grow for increasing accuracy in approximation with two-layer neural nets is relatively well-known and many researchers rely on it. If one bounds the weights of the neural net a priori, it is somewhat expected that it cannot approximate functions outside of its class with arbitrary accuracy.

  • Similarly, there is not much intuition about the specific dynamics or properties of the MFLD in this paper. I feel many of the results work directly with Empirical Risk Minimization (ERM), which is in fact the solution MFLD reaches in the long time limit. Further, the feature learning result might suffer from a large runtime, and it is not a priori clear what would be the benefit of MFLD compared to any other optimization algorithm.

Questions for authors
  • Why doesn't Theorem 4.1 take advantage of the multi-index structure to replace dd with m∗m^*? Also, it feels like if the function can be exactly represented by a smooth activation with large α\alpha, then it must admit a large ss as well, especially since minimax lower bounds of type n−2s/(2s+d)n^{-2s/(2s + d)} applies.

  • Certain positive gaps in the paper can be arbitrarily small, which may limit their significance for machine learning theorists, e.g.:

    • Prop 3.1 does not imply a universal lower bound on approximation error. The lower bound might be f∗f^* dependent and arbitrarily small.
    • In Theorem 5.1, the gap could be arbitrarily small in Rc,BcR_c,B_c.
Reviewer_R7MmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper uses mean-field two-layer neural networks for Gaussian multi-index regression as a setting to study the impact of soft vs. hard constraints on the parameter set of models. First they show that for any fixed compact parameter class, the induced class of prediction functions is compact and therefore closed in L^2. Hence, if a target lies outside this class, there is a strictly positive approximation error that cannot be removed through additional samples, increased width, weaker regularization, or longer training. They then show that exact representability under bounded input weights imposes regularity. The authors then construct a fixed infinite-dimensional family of rough multi-index targets that remains separated from every fixed compact parameter class, but which is learnable by a noncompact tanh model. They derive the rate at which this is learned. They additionally show that the noncompact estimator exhibits genuine feature learning. Finally, the entropy-regularized estimators are identified with the infinite-time limits of mean-field Langevin dynamics (MFLD).

Critical strengths
  • The paper constructs a family G such that every fixed compact class has a positive approximation gap, while the noncompact estimator consistently learns the same family. This is a non-trivial construction.
  • the paper makes the point that a learning theorem can have strong optimization and estimation guarantees while remaining statistically inconsistent because a supposedly technical modeling assumption introduces irreducible approximation error. While this is itself not a novel observation it is a common implicit assumption that is not always made explicit (that is, that a function is realizable by a class of models).
  • The main results are genuinely about feature learning, not a static kernel.
Critical weaknesses

At a high level, the fact that compact parameters can restrict the available functions is not surprising and follows from essentially a standard topological argument: compact parameter distributions are mapped continuously into a compact class of functions. The point then should be that this actually materially impacts learning. The main noncompact theorem relies on a highly specific setup and so the rigorous conclusion is really that there exists a mathematically controlled setting in which fixed hard parameter bounds qualitatively change statistical learnability which is not that surprising. It doesn't establish that fixed compactness assumptions generally are impactful for learning theory. Furthermore any implemented neural network has an effectively bounded parameter space because of finite precision, finite memory, numerical constraints, and finite training.

The important question is then more along the lines of which existing theoretical results actually make fixed compactness assumptions for which this phenomenon changes the scientific conclusion? If prominent mean-field or neural-network generalization results become inconsistent on natural target classes specifically because of this issue, then the paper becomes significantly more meaningful.

Questions for authors

Can the authors provide concrete examples of existing mean-field or neural-network learning results where a fixed compactness assumption is treated mainly as a technical convenience, but where the present analysis suggests that the assumption materially changes the target class or consistency conclusion?

This would help establish the broader significance of the work.

Reviewer_PHgcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies the effect of bounded parameter constraints on learning a multi-index model with a two-layer neural network in the mean-field regime. The author first show that, in the case where the parameters are bounded, and suppose the activation function is at a certain level of smoothness, in order for the target function to be able to represent by the neural network, the target link function also need to be at a certain level of smoothness. Notably, if the activation function is smooth, the target link function also need to be smooth. The authors also characterize the convergence rate of the test error in this case. Then, the authors let the activation function to be tanh⁡\tanh which is smooth, and construct a functional class mathcalGmathcal G of the link function such that the corresponding target functions can be represented by neural networks that allow unbound parameters, but have finite approximation error when using bounded-parameter neural networks. The authors also characterize the convergence rate, and show that the signals in the multi-index model can be recovered.

Critical strengths

I think this paer is technically solid, in particular, the techniques of proving the error rate and the way of constructing a functional class that separates neural networks with the bounded and unbounded-parameters are interesting.

Critical weaknesses
  1. While I appreciate the technical contributions, my biggest concern about this paper is whether the problem studied itself is interesting enough. On one hand, from my knowledge, it is uncommon to put hard bounded parameter constraints in practice when training neural networks, and it's more common to use soft regularization such as weight decay or gradient clipping. On the other hand, it seems to me that the bounded-parameter constraints is always manually added by theorists either to adapt certain proof techniques, or appear naturally by algorithms like projected GD in cases where the norm of the parameters is unimportant to the problem (e.g. some classification problem). Could the authors elaborate more on the significance of the results?

  2. The scope of the results on the statistical separation between the bounded and unbounded neural network is not clear enough. The construction is based on a specific case where the activation function is smooth, how does this separation generalize to cases where the activation is not smooth such as relu (or relu with a hard threshold for mean-field networks)? Also, this paper focus on two-layer networks, if we allow deep neural networks, in what case such separation still holds?

Questions for authors
  1. While the authors show a statistical separation between bounded- and unbouned- parameter networks, does the authors also expect a computational separation? In particular, would it be possible to construct cases where the computational hardness for obtaining the optimal estimators in the bounded- and unbounded- parameter function class are different?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment

We thank all reviewers for their careful reading and constructive feedback. Several reviews raised closely related questions about the novelty and statistical significance of our work, particularly its distinction from existing approximation theory and MFLD feature-learning analyses. Hence, we begin by clarifying our central contribution and its implications for statistical learning theory.


1. Feature learning theory with statistical guarantee under noncompact parameter. A main contribution of this work lies in being the first to connect approximation, statistical estimation, and feature learning in the long-time limit of noncompact MFLD for a standard nonparametric multi-index model setting. Rather than assuming representability, we construct a fixed infinite-dimensional family of rough multi-index targets and establish a polynomial prediction guarantee together with statistical feature learning and recovery of the unknown index subspace.

ItemPrior work (mainly on feature recovery and optimization)This work
Main focusRecovery of feature directions, learning dynamics, optimization, etc.Nonparametric estimation of rough multi-index targets, feature learning, index subspace recovery
Parameter spaceTypically, a sphere or a fixed compact setOutput coefficients are kept bounded, while input weights and biases are unbounded
Link functionTypically, link functions under specific structural, smoothness, or representability assumptions.We construct a single fixed infinite-dimensional family of rough functions and show both its approximability and its learnability
Statistical guaranteeFeature recovery or learning guarantees for specific target classesPrediction excess risk upper bound for the rough function family with feature learning
Role of MFLDAnalysis of optimization and feature recovery in specific settingsNonparametric prediction guarantees and statistical feature learning in the long-time limit of MFLD
Essential differenceAnalyzes learning under given model and target conditionsExplicitly accounts for function approximation ability and guarantees learning in a nonparametric setting

2. Fixed parameter bounds can change the statistical meaning of learning guarantees. Another main contribution is implications about fixed parameter constraints. Hard constraints introduced for the convenience of theoretical analysis are not necessarily statistically harmless. The examples in App. B.5 show that a generalization gap bound or an MFLD convergence guarantee can hold while the imposed fixed parameter constraints prevent consistent estimation. We demonstrate why optimization and generalization guarantees must be investigated and interpreted together with the approximation properties of the constrained function class.


3. Takeaways for ICLR readers.

Here, we highlight new insight ICLR readers would take away from the paper.

AspectDefault readingReading after this paperWhere
Reading fast ratesA fast rate under bounded parameters is evidence that learning works well.It may be a consequence of smoothness inherited from the activation, and hence cannot be read as evidence that a structure was learned!Thm. 3.2, Thm. 4.1
What feature learning isThe kernel moving away from initialization; allowing large weights.Where to place the steep neurons needed to build rough structure! A fixed feature distribution cannot reach the target even with unbounded support.Prop. 5.2, Cor. 5.3, Cor. F.3
Meaning of parameter boundsA large constant introduced for analytical convenience; taking it large enough approaches the original problem.A fixed hard bound changes the learning problem itself and is not a substitute for weight decay. Even targets representable with a bounded quadratic cost cannot be reached under any fixed bound, however large.Thm. 5.1, Prop. D.7, App. B.5
What must growAllowing the weights to grow enables approximation.What must grow is not the weight norm but the tails (high-order moments) of the feature distribution!Prop. D.7, Thm. 5.1, Cor. F.3

Main Revision.

  • We have revised the Introduction to emphasize these statistical contributions.
  • We added concrete examples of the implications of fixed parameter bounds in App. B.5 to further clarify why this study matters.
  • We made further modifications in response to the reviewers’ suggestions.
Area_Chair_aPdTOpenReview ↗

Discussion phase coming to an end

Dear reviewers BhBW, 92hm and R7Mm,

As we are approaching the deadline of the discussion phase, please read the authors’ responses and share your thoughts in the discussion as soon as possible. In particular, please clarify whether the responses have addressed your main concerns. Please also update your review and scores if appropriate.

Your timely input will help us reach a well-informed decision. Thank you for your time and effort!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.