When Parameter Bounds Limit Learning: Approximation and Estimation in Mean-Field Neural Networks

Learning theory for neural networks often imposes hard parameter bounds for tractability, but these restrictions can change what a model represents rather than merely simplify proofs. We study this in Gaussian multi-index regression, where targets depend on an unknown low-dimensional subspace, using mean-field networks and mean-field Langevin dynamics (MFLD) under compact and noncompact parameterizations. For a fixed compact parameter domain, the representable predictors form a closed class, so targets outside it incur an irreducible approximation error unaffected by sample size or training time. Even exactly represented multi-index targets inherit smoothness from the activation, showing that bounded weights restrict the regularity of admissible link functions. For these represented targets, the long-time limit of MFLD satisfies a finite-sample prediction upper bound that depends on the ambient dimension. By contrast, on a fixed infinite-dimensional family of rough multi-index targets separated from every fixed compact class, the long-time limit of MFLD over a noncompact model with unbounded input weights and biases achieves polynomially decaying prediction error, accompanied by feature learning. Thus hard constraints and soft regularization can define different learning problems, making approximation the first step in interpreting statistical and convergence guarantees.