acceptodds
Under review as a conference paper at ICLR 2027

The Statistical Cost of Redundant Width in Bayesian Quadratic Neural Networks

Abstract

How much prediction error do redundant directions contribute when a quadratic network learns a low-rank target, and when does regularization suppress that contribution? We study this question through the singular geometry of the factor map . For a global Gaussian factor prior, fixed dimensions, and width , we prove a noisy boundary law for expected Gibbs and posterior-mean prediction risks. At matched likelihood temperature, its unregularized mean risk is , where and : each additional neuron costs within the same PSD function class. An Euler–Stein identity establishes beneficial vanishing regularization; a PSD-estimator theorem calibrates the risk gap. The same boundary law resolves teacher modes of order . At zero added decay, every finite nonzero PSD local signal lowers mean risk below the zero-signal endpoint. At , the zero- and strong-signal endpoints coincide, yet the risk is strictly lower between them. The population quartic-to-quadratic crossover as a regularization gap opens explains the geometry; the empirical law retains the surviving random tilt. Finite-data experiments and limiting-law quadrature examine these distinct predictions. The broader local-prior empirical crossover remains conjectural. The proved law links the width cost to regularization and a weak-signal transition that fixed-rank coefficients alone cannot describe.

Primary area:
learning theory
Keywords:
Singular learning theory, Bayesian neural networks, Generalization theory, Bayesian asymptotics, Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_ptk3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of learning with a wide two layer network with Hermite-2 activations in a teacher-student setting by using a Bayes sampler from the posterior in the limit of large number of samples while keeping the dimension of the problem and the widths of both the teacher and the student finite. The analysis is for noisy teacher and a non-zero temperature for the student. The main result is the negative impact on generalization of increasing the width of the student while being at positive temperature.

Critical strengths

None

Critical weaknesses

The paper is horribly written. Most of the quantities are not even defined before they are used. I think the paper needs to go through a major rewriting to make even just the results of the paper understandable. For what one can understand, the significance of this paper is extremely limited, as its results are in an extremely data abundant regime.

Questions for authors

Could you state your contributions concisely, as well as stating precisely the relation of this work with the previous literature, stating what is its contribution?

3 comments

Thank you for the direct assessment. Your criticism of the writing is fair. We sincerely apologize for the confusion coming from the previous presentation. The revised PDF rewrites the abstract and the main text while all the mathematics is unchanged. We start by answering the questions, then addressing the concerns:

Could you state your contributions concisely?

(The setting is a quadratic network with m ≥ d+1 hidden units, a Gaussian prior on the weights, a fixed teacher of rank r < d, and n noisy samples. The dimension d and the width m are fixed and n grows.) We derive a formula for the limit of the posterior and quantify the exact cost of overparametrization for a Bayesian learner (as in contrast to a non-Bayesian setting, we know over-parametrization gives benign optimization landscape and it also costs nothing to a learner that minimizes training loss). As a consequent of the limit, we also show rigorously that a small weight decay and additional small eigenvalues of the teacher network lower the risk.

State precisely the relation of this work with the previous literature.

Thanks very much for letting us elaborate on this. We made the comparison sharper in the revised manuscript. There are two major lines to work to compare to:

  • To compare with singular learning theory, the coefficient in item 2 of our manuscript is the one that that singular learning theory gives through the learning coefficient for priors on a compact set. Without weight decay, the limiting posterior is known when the teacher has rank d−1 and when m = d+1. The limiting posterior for rank at most d−2 and width at least d+2 is new and how not been studied in any prior literature to our best knowledge. Also, in every case, the convergence of the expected risks, the closed formula for m > d+1, the risk as a function of weight decay, and the risk for weak teacher eigenvalues are also new in this work.
  • To compare to statistical physics of quadratic networks (Maillard et al.; Erba et al.; Martin et al.; Defilippis et al.), we study the same model in a different regime, here we fix d instead of letting d grow with n \prop d^2. The prior works in this line evaluate the Bayes-optimal learner and the minimizer. We treat a Bayesian learner that is wider than its teacher. The two regimes meet for the minimizer. The least-squares benchmark considered in Section 5 of our manuscript is the regularized ERM of Erba et al., up to the scaling of the penalty, and in both analyses it does not depend on the width. As n/d² grows, the equations of Martin et al. (2026) give the constant of our Theorem 4.1 (this is also explicitly stated in the revised manuscript).

The results are in an extremely data abundant regime.

We agree with this, and this is also addressed in section 7. In this work, we do not analyze n \prop d^2. What the regime we work in gives us is the posterior itself in explicit form, not only the error. That is what makes the weight decay of order n^(−1/2) and the weak eigenvalues math derivations tractable. In the numerical experiments section, we included some observations on small samples.

I thank the authors for their response. I appreciate the effort in improving the presentation. I think however that it amounts to a complete rewriting of the paper, and I do not think it's appropriate to present an entirely different paper during rebuttal (even if the mathematical results are the same).

I also would argue that claiming that the mathematics is unchanged is just false, as defining the mathematical objects clearly is integral part of the writing, and the rewriting changed the way the objects are introduced.

I am strongly convinced that such a massive change warrants a new review cycle.

Beyond this, I am still not convinced of the novelty of the work. Your analysis is in the limit n≫d2n \gg d^2, and you correctly point out that there is a line of works studying the regime n=αd2n = \alpha d^2 with α\alpha finite. In particular Martin et al. also studies finite temperature Langevin dynamics at large times, which I would expect to reach equilibrium with no particular issues for m>dm > d. How is your result not a special case of their analysis in the limit α→∞\alpha\to\infty?

Hi, thanks for the direct reply. We answer your two points in turn.

The size of the revision

Thank you for taking the time to read the revision during the tight rebuttal period. We understand your position. The large change reflects our willingness to respond to the reviewers’ concerns, and our aim to make the technical results clear and easier to read, so that they are not buried under the writing. By “unchanged” we meant that every theorem, corollary, proposition and lemma makes the same claim as in the submission. We agree that the way the objects are introduced has changed.

How is your result not a special case of their analysis in the limit α → ∞?

We are aware of Martin et al. (2026). The paper is cited and discussed explicitly in our manuscript. In short, the two analyses meet only in the formula for the minimizer: their equations give the excess risk σ²[q + h(h+1)/4]/n to leading order as n/d² grows, with h = d − r, and this is the value in our Theorem 4.1. We point to their paper for this comparison in Section 1, where we position this work, and in more detail in Appendix A. The subject of our paper is the Bayesian learner, and for it Martin et al. give no formula of which this limit could be taken.

What their paper covers:

  • For Langevin training at finite temperature they derive the stationary measure (their Claim 5) by dynamical mean-field theory. We agree with you that for m > d this is the equilibrium measure. After the change of variables from W to WWᵀ it has the form of our limiting law.
  • They reduce this measure to scalar equations in two cases: zero temperature, and the Bayes-optimal point, where student and teacher have the same width.

What it does not cover:

  • The cost of width is zero in both of these cases. At zero temperature the learner is the minimizer, whose risk does not depend on the width for m ≥ d. At the Bayes-optimal point the student has no extra width. This is the case we treat, and it is the question that inspired us to work on this paper, and most of the technical results are done to answer this.
  • The cost appears only at finite temperature with a student wider than its teacher. They leave this case open (end of their Section 3.2.7). This is the case we treat.
  • A comparison for the Bayesian learner would need an evaluation of their Claim 5 in this case, and an exchange of the limits d → ∞ and n/d² → ∞. Neither has been done, by them or by us.
Reviewer_58KaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies quadratic neural networks under a Bayesian learning setting. In particular, one considers a neural network which consists of an inner product of centered quadratic features X(x)X(x) and quadratic weights A=WW⊤A = W W^\top. Furthermore, data is assumed to follow the same functional form with a set of unknown weight A∗A_* (=W∗W∗⊤= W_* W_*^\top) with observation noise. This corresponds to a student-teacher learning setting. As we are in the Bayesian learning setting, the weights WW follow a distribution. The primary concern of the paper is to examine the asymptotic behavior of the squared error between EA\mathbb{E} A and A∗A_* (squared Frobenius norm). The paper presents a variety of results concerning this quantity, including, analysis of weight decay strength and different (matrix) rank structures for the teacher.

Critical strengths

The analysis on RmeanR_\text{mean} appears to be interesting, especially with its dependence on uu and its weak teacher variant. I only have a broad familiarity with the topic area, so further input would be needed to access the novelty, e.g.

Critical weaknesses

The writing of the paper. There is a littering of notational inconsistencies and the writing is very opaque. The ordering of how concepts and (incomplete) definition are written in a way which makes reading the paper very difficult. Furthermore, the paper does not seem to be coherent when restricted to its main-text (which from I understand is a requirement / heavily desired). The following are more specific complaints:

  • There is poor motivation for why the model and data setting matter. As someone who does not work directly on this problem, the only motivation I could easily identify is that there is some prior work which connects to the paper.
  • Intro is confusing. There initial introduction of Eq (1) does not work in my opinion. It refers to various terms such as "matched likelihood temperature" and technical concepts without establishing the data / model. The introduction doesn't make much sense without reading the paper and coming back to it (multiple times). Its also confusing if Eq (1) is known in the literature. You seem to give a sketch of deriving it, but no actual proof (cross ref) nor reference. Definitions feel very opaquely stated in general. For example, I don't understand why the prior and likelihood are not just stated in equations in Sec 2.
  • Notation and terminology is not consistent. "matched temperature / matched likelihood temperature / matched likelihood" for instance should be made consistent. I don't understand why Q(D)Q(D) is defined in the main text when it only appears to be useful in the appendix. You reference the "boundary law" but it is only implicitly specified (I have assumed you are referring to Theorem 3.1). Some notation is defined, but then used inconsistently, e.g., subscript in Eq (7) vs Eq (10) and the ⟨⟩\langle \rangle conditional expectation notation in Eq (8, 9). In the main text, you only mention the "Euler--Stein identity" in the abstract and intro, but never call the actual result by that name. "endpoint" and its usage is never explicitly defined.
  • Key results referenced in the abstract seem to be confined to the appendix. I do not think that the short explanation in the main text ("Mechanism and proof") is enough to coherently describe the result.

In general, there are major clarity/presentation issues and I believe that the paper requires major rewrites.

Questions for authors

What exactly is the local prior referenced by Equation (33) (used in Prop 3.3)? I could not find what exactly χ\chi and φ\varphi are. Are they arbitrary?

2 comments

Thank you for the specific list and constructive feedback. They are helpful for the revision. We answer the questions first:

What exactly is the local prior? What are χ and φ? Are they arbitrary?

Yes, χ and φ are arbitrary with the condition that φ is any fixed smooth density that is positive everywhere on the weight space (e.g. a Gaussian prior), and χ is any smooth nonnegative function that is positive on the set of weights that represent the teacher and zero outside a small neighbourhood of that set. Our proof for Proposition 3.3 depends on only the shape of the loss near the teacher so it holds for every such choice.

Why this model and this data setting?

The model considered here is the simplest network in which extra hidden units do not change what the network can represent. So an effect of further width on prediction cannot come from the function class. Such width is known to help optimization, and it is free for a learner that minimizes the loss. Our question is what it costs a Bayesian learner. This question has not been answered rigorously in any model to the best of our knowledge. The inputs are Gaussian because the excess risk is then exactly the squared distance ‖A − A*‖² between the learned matrix and the teacher, which keeps every risk explicit.

Is the risk formula of Section 1 (Equation (1) in the submission, now Equation (2)) known, and where is it proved?

Yes, in Proposition 3.3, we see the risk formula and its value is the one that singular learning theory predicts for priors on a compact set. For the Gaussian prior on the whole weight space, this follows from Theorem 3.1 (the main theorem) in our manuscript. We included an outline of the key steps of the proof in the main text and the full proof is in Appendix C.

The other points on order of the introduction, terminology and notation.

Thank you very much for specifying these potential confusions for reading. We agree and have taken this constructive feedback to revise manuscript. We would very much appreciate feedback if the revised version made it more clear to follow. Thanks a lot.

Thank for the response.

Similar to the other reviewer's response, I can't help but feel that the change the manuscript are out-of-scope for a rebuttal and requires a new review cycle.

Reviewer_7TvJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies a teacher-student Bayesian regression setting. The teacher generates noisy labels through a fixed positive semidefinite teacher matrix A∗A_∗ of rank r < d. The student tries to fit the dataset by a matrix A parametrized in terms of some weights W∈Rd×mW ∈ R^{d×m} via A=WWTA = WW^T. The student is therefore a quadratic network with fixed input dimension d and width m. The posterior distribution is given in terms of an empirical squared-loss likelihood, and two weight decay terms. Although increasing the width beyond d does not enlarge the represented function class, it changes the prior induced on the matrix and therefore can affect learning. The authors derive a limiting posterior law and obtain asymptotic expressions for the Gibbs and posterior-mean prediction error.

Critical strengths

I think the analysis at fixed d and m is potentially interesting (if sufficiently supported by clear technical derivations).

Critical weaknesses

The main weakness of the paper is that is not clear, well structured and written. It is really hard to understand the paper from start to end without jumping to another section of the paper. The presentation makes the main result unnecessarily difficult to understand and assess. The introduction directly jumps in medias res without introducing first the problem (e.g. Equation (1) precedes the definition of RmeanR_{mean}). A reader cannot understand what the authors want to say in the introduction without reading the rest of the paper first. Once the model is introduced a substantial discussion of prior works and settings related is lacking. As a result, it was not immediately clear what are the main contributions of the work. Moreover I do not think the setting explored in this paper is sufficiently well motivated (for example: why is this particular fixed dimensional correction interesting enough, and what does it teach us beyond existing asymptotic theory?). A discussion of the principal implications and simple analytical information that can be extracted from Theorem 3.1 is also lacking. Finally the paper uses a terminology that actually confuses the reader and sometimes it does not provide any information. Terms as ”endpoint coefficient” or ”boundary law” are examples. The second main weakness (which is also connected to the first) crucially affects the verifiability of the paper. The paper does not provide to the reader a clear and organized pathway to understand the proof of their main result, i.e. Theorem 3.1. The ”Mechanism and proof” was completely incomprehensible to me even after several readings. Appendix C does not help in this respect either as it is not well structured and not well referenced in the main text. As such it was currently impossible to me to evaluate the correctness of the results. The novelty (even at the technical level) and significance also remain insufficiently established.

Questions for authors

• Could the authors clarify where the dimensional assumptions 0 ≤ r < d and m ≥ d + 1 enter the proof of Theorem 3.1? In particular, what prevents the argument from covering r = d, and what additional difficulties arise for m < d + 1? In particular what specifically fails in the proof for m < d + 1 and r = d? These clarifications would make the scope and imitations of the theorem easier to assess. • The role of γ and u looks the same in equation 11 (they both control the regularization strength of the weights), the only difference being their scaling with n. In the large n limit (which, as I can tell, is the main focus of the paper), clearly γ completely leaves the generalization unaffected at the leading order. Could the authors clarify beforehand why their choice to use γ as well, and how it enters in any of their results? Is it only relevant when u = 0? And if so, how? • The connection between the model the authors study in the paper and the case of one hidden neural networks with quadratic activation function should be made more explicit in the model definition. Also the distinction of the scaling regime the authors pick with respect to previous works should also be made more explicit. • The authors study the large sample limit (n → ∞) at fixed input dimension d and width m. Related work in machine learning and statistical physics studies both large input dimension d and width m regimes; in particular, Maillard et al. (2024), cited in the manuscript, consider m/d and n/d2n/d^2 fixed. Could the authors make the comparison with these results explicit at the level of the risk formulas? After accounting for differences in priors, normalizations, and prediction rules, is there an overlapping regime in which the large-sample expansion of those formulas agrees with the the ones obtained here? If a direct comparison is not possible, please identify the incompatible assumptions or order of limits issues. This would help clarify the contribution relative to existing results.

2 comments

Thank you very much for the detailed reading and the concrete questions. The revision follows your comments on the structure, with the model before the results, prior work after the model, terms defined before use. We start with addressing your general questions:

Why is this fixed-dimensional correction interesting, and what does it teach us beyond existing asymptotic theory?

Existing theory gives one number for this model (the learning coefficient). In this paper, we give the whole limiting distribution, and quantify the effect of weight decay which existing theory doesn't explain yet.

  • Classical asymptotics says that the posterior is Gaussian and that the prior does not affect the 1/n term of the risk. Here neither holds. The teacher lies on the boundary of the function class, the limit is not Gaussian, and the prior changes the risk at order 1/n.
  • Singular learning theory covers such models through the learning coefficient, where the model fixes the risk of the posterior mean at zero decay. Our results agree with this line of literature on zero decay and analyzes a more general case.
  • We also give precise answer to a simple question in this paper. For this network, width beyond d is free for a learner that minimizes the loss, and it has an exact price for a learner that averages over the weights (the answer was not clear without technical analysis).

A discussion of the principal implications and simple analytical information that can be extracted from Theorem 3.1 is lacking.

Thanks for the opportunity to explain this. We can read off quite a few implication from the formula in Theorem 3.1:

  • Under the Gaussian prior, without weight decay and with the true noise level in the likelihood, each unit beyond d+1 costs σ²(d−r)/(2n). For priors on a compact set the same increment holds for every unit beyond the teacher's rank.
  • At width d+1 and without weight decay, knowing that the matrix is positive semidefinite does not help the posterior mean. Its risk is p\signma^2/n, which is same as for unconstrained least squares.
  • For r ≥ 1, a small weight decay of order n^(−1/2) lowers the risk, and we obtain a best positive value in Corollary 3.2.
  • A teacher with additional small eigenvalues, of order n^(−1/2), has a lower zero-decay risk than a teacher of exact rank r.

Terminology and the route into the proof.

Thanks for brining this up. In the revised manuscript, the two endpoints of Section 5 are defined where they are first used. Section 3 gives a five-step outline of the proof of Theorem 3.1, each step points to the part of the appendix it uses and we also have a separate paragraph discussing where each assumption enters.

Following the previous discussion, we continue to address the questions:

Where do r < d and m ≥ d+1 enter the proof? What fails otherwise?

  • r < d gives the teacher at least one zero eigenvalue, which puts it on the boundary of the cone. For r = d the teacher is an interior point and the classical Gaussian analysis applies. Nothing fails there, expect there is no boundary to consider (so this case would be covered by classical analysis already, and there shall be no additional statistical cost).
  • m ≥ d+1 is used in the bound on posterior moments (Lemma C.1 in our manuscript), which turns convergence of the posterior into convergence of expected risks. It is exactly the range in which the prior induced on A is log-concave. All other steps work for every m ≥ d.
  • At m = d is the only case that this bound fails. We have results from various numerical experiments that indicate the conclusion shall be the same but we have no rigorous proof of this case.
  • For m < d the matrix WWᵀ has rank below d and has no density on the symmetric matrices, so the description by a law on the cone does not apply.

Why use γ as well as u?

Our learner uses the whole weight space, with no restriction to a neighbourhood of the teacher, so it needs a proper prior, and γ provides one when u = 0. In the proof, we also use γ > 0 to control the posterior on rare, badly conditioned datasets. The decay u is on the scale n^(−1/2) because that is the only scale at which it changes the 1/n term.

The connection with one-hidden-layer networks and the scaling regime should be more explicit.

Thanks for bringing up this potential confusion, and we agree. We addressed this in the revised manuscript, and section 2 now states the model clearly and section 1 states the regime. Section 7 emphasizes on the limitation again that the effect needs positive output weights.

Could the authors make the comparison with Maillard et al. (2024) explicit at the level of the risk formulas?

The two works had different assumptions and worked in different regimes while both working with quadratic networks.

  • To start with, the quantities considered in the two works are different, here we consider a coefficient of 1/n at fixed d, while in Maillard et al. (2024), the authors considered an error of order one at n \prop d^2.
  • The cost of width needs two things at once: a teacher of rank below d, and a student wider than the teacher. In the Bayes-optimal setting the student has the teacher's architecture, so the two never occur together. At fixed d such a student, seen through the matrix A, is a regular model at the teacher: the classical parameter count applies and there is no cost of width. This would be the large-sample tail of the formula in Maillard et al. (2024), while in this work, we focus on a case where the model is not regular.
Reviewer_4BPxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates the statistical cost of redundant width in Bayesian quadratic neural networks. It derives asymptotic prediction-risk formulas, characterizes the effect of regularization, and studies the transition associated with weak teacher eigenvalues.

Critical strengths

The research question is interesting, particularly the statistical cost of overparameterization.

The claimed theoretical results are strong, with precise characterization of expected prediction risks.

The numerical experiments provide convincing support for the theoretical predictions within the considered setting.

Critical weaknesses

Mathematical notations. Several important mathematical objects are introduced without adequate motivation or precise definitions. For instance, the transverse block (D) is used before its geometric construction is explained, while quantities such as the singular fluctuation (\nu_{\mathrm{SLT}}) are not defined in the main text. This makes the theoretical development unnecessarily difficult to follow and assess.

Limited generality. The main results concern a very restricted setting. Although, this could be acceptable for a theory-oriented paper, I think the paper should include clear discussions about what this implies and does not imply for more general neural networks. The results are primarily asymptotic, without explicit bounds. This could also be fine but requires expliciting the limitations, and a rationale for why the finite corrections could be negligible.

Questions for authors

Please elaborate on the significance of the result. What do the results tell us ? To what extent should the result persists beyond the specific Gaussian factor-prior family and fixed-dimensional setting?

1 comment

Thank you very much for the balanced review, and also for noting the two concerns precisely. We answer the questions first.

What do the results tell us?

We show three things:

  • In the setting that we consider, we write out the limiting posterior which tells us under the Gaussian prior, without weight decay and with the true noise level in the likelihood, each unit beyond d+1 still raises the prediction error of the Bayesian posterior mean, by σ²(d−r)/(2n) at leading order. (As in contrast to non-Bayesian approaches where over-parametrization is known to be beneficial for optimization at no additional statistical cost.)
  • We show the reason is the prior, not the model. Gaussian weights induce a prior on functions that puts little mass near a low-rank teacher, and less the wider the network is. A learner that minimizes the loss pays nothing for the same width.
  • The usual summary of such a model, the learning coefficient, gives the risk only without weight decay and for a teacher of exact rank. We identify additional information in this paper and show that for a weight decay of order n^(−1/2), or teacher eigenvalues of that order, one needs the limit of the whole posterior.

To what extent should the result persist beyond the Gaussian prior?

As far as results in this papers go, we have one part proved and one conjectured. In Proposition 3.3, the risk without weight decay is proved for every smooth positive prior on a compact set, at every width above the teacher's rank. The limiting posterior and the weight-decay curve are proved for the Gaussian prior only. We expect them to hold for other smooth positive priors on the weights, because every such prior induces the same vanishing factor near the teacher. We have included the numerical results that support this, and so far this more general statement is formulated as a conjecture in A.4.

... and beyond the fixed-dimensional setting?

This is beyond the scope of this paper as we do not analyze n \prop d^2 here. There is one point of contact with that literature: for the learner that minimizes the loss, our constant agrees with the large-sample limit of the equations of Martin et al. (2026) (in Appendix A). For the Bayesian learner the same comparison is open.

What this implies for more general networks.

The effect needs the teacher to lie on the boundary of the function class. Here that comes from the positive output weights. With output weights of both signs and at least d units, the network represents all symmetric matrices, the teacher is an interior point, and we expect no cost beyond the classical value. We make no claim for other activations. Section 7 has a paragraph that elaborates on what carries over and what does not.

A rationale for why the finite corrections could be negligible.

For this, we have no technical bound. We can offer measurements and a heuristic.

  • Measurements. At n = 64, with p = 6, the cost per unit is 41% and 45% above its limit. At n = 4096 it is 0.987 ± 0.019 and 1.003 ± 0.018, against 1 (± is twice the standard error). We also have more numerical details in Appendix N.
  • Heuristic. The limit drops three terms, each of relative size n^(−1/2): the randomness of the inputs, the coupling between the Gaussian coordinates and the cone coordinates, and the variation of the prior factor. So the corrections are small when n is large compared with p and when the nonzero eigenvalues of the teacher are large compared with σ/√n. The experiment with a small teacher eigenvalue shows the larger residual that this predicts. A proved bound is open as far as we know of.

Comments on the paper

1 comment

We thank the reviewers for their time. The main criticism on the presentation of the manuscript is fair. The results we included in the submission are technical and the manuscript is quite math-heavy. The authors would like to sincerely apologize if the previous presentation added additional pressure during the previous reviewing process. We have integrated the constructive feedback and uploaded a revised PDF in which the main text is restructured while all math results remains unchanged.

Starting with what the paper shows: For quadratic network with at least d hidden units, every function in its class is represented, so further units add nothing to what it can represent. Such extra width is known to help optimization, and it costs nothing to a learner that minimizes the training loss. We ask what it costs a Bayesian learner, which averages over the weights instead.

  • We show that extra width has a cost for Bayesian learner only by changing the prior that the Gaussian weights induce on functions, and we preciously quantify the cost of an additional neuron.
  • We find the limit of the posterior near the teacher as n grows, with the dimension and the width fixed. It is not Gaussian. In the directions where the teacher is zero, it is a Gaussian restricted to the cone of positive semidefinite matrices and multiplied by a factor that comes from the prior.
  • From this limit we obtain the prediction error exactly at order 1/n. Without weight decay, and with the true noise level in the likelihood, each unit beyond d+1 adds σ²(d−r)/(2n) to the excess risk of the posterior-mean predictor, where r is the rank of the teacher. The same limit gives the effect of a weight decay of order n^(−1/2) and the risk when the teacher has eigenvalues of order n^(−1/2).

To clarify what changed in the pdf:

  • The text is rewritten. The introduction now has the order of the abstract (the question, the prior that the weights induce, the limit of the posterior, the cost of width) and then says what was known before.
  • No technical result has changed. Every theorem, corollary, proposition and lemma makes the same claim as in the submission and keeps its number.

Also to be clear with the limitation and what we does not cover in this paper:

  • All results are limits as n grows at fixed dimension and width. There is no finite-sample bound, and the regime n \prop d² is not analyzed.
  • The effect needs positive output weights, which put the teacher on the boundary of the function class.
  • The limiting posterior is proved for the Gaussian prior only.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.