- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the problem of learning with a wide two layer network with Hermite-2 activations in a teacher-student setting by using a Bayes sampler from the posterior in the limit of large number of samples while keeping the dimension of the problem and the widths of both the teacher and the student finite. The analysis is for noisy teacher and a non-zero temperature for the student. The main result is the negative impact on generalization of increasing the width of the student while being at positive temperature.
Critical strengths
None
Critical weaknesses
The paper is horribly written. Most of the quantities are not even defined before they are used. I think the paper needs to go through a major rewriting to make even just the results of the paper understandable. For what one can understand, the significance of this paper is extremely limited, as its results are in an extremely data abundant regime.
Questions for authors
Could you state your contributions concisely, as well as stating precisely the relation of this work with the previous literature, stating what is its contribution?
3 comments
I thank the authors for their response. I appreciate the effort in improving the presentation. I think however that it amounts to a complete rewriting of the paper, and I do not think it's appropriate to present an entirely different paper during rebuttal (even if the mathematical results are the same).
I also would argue that claiming that the mathematics is unchanged is just false, as defining the mathematical objects clearly is integral part of the writing, and the rewriting changed the way the objects are introduced.
I am strongly convinced that such a massive change warrants a new review cycle.
Beyond this, I am still not convinced of the novelty of the work. Your analysis is in the limit , and you correctly point out that there is a line of works studying the regime with finite. In particular Martin et al. also studies finite temperature Langevin dynamics at large times, which I would expect to reach equilibrium with no particular issues for . How is your result not a special case of their analysis in the limit ?
Hi, thanks for the direct reply. We answer your two points in turn.
The size of the revision
Thank you for taking the time to read the revision during the tight rebuttal period. We understand your position. The large change reflects our willingness to respond to the reviewers’ concerns, and our aim to make the technical results clear and easier to read, so that they are not buried under the writing. By “unchanged” we meant that every theorem, corollary, proposition and lemma makes the same claim as in the submission. We agree that the way the objects are introduced has changed.
How is your result not a special case of their analysis in the limit α → ∞?
We are aware of Martin et al. (2026). The paper is cited and discussed explicitly in our manuscript. In short, the two analyses meet only in the formula for the minimizer: their equations give the excess risk σ²[q + h(h+1)/4]/n to leading order as n/d² grows, with h = d − r, and this is the value in our Theorem 4.1. We point to their paper for this comparison in Section 1, where we position this work, and in more detail in Appendix A. The subject of our paper is the Bayesian learner, and for it Martin et al. give no formula of which this limit could be taken.
What their paper covers:
- For Langevin training at finite temperature they derive the stationary measure (their Claim 5) by dynamical mean-field theory. We agree with you that for m > d this is the equilibrium measure. After the change of variables from W to WWᵀ it has the form of our limiting law.
- They reduce this measure to scalar equations in two cases: zero temperature, and the Bayes-optimal point, where student and teacher have the same width.
What it does not cover:
- The cost of width is zero in both of these cases. At zero temperature the learner is the minimizer, whose risk does not depend on the width for m ≥ d. At the Bayes-optimal point the student has no extra width. This is the case we treat, and it is the question that inspired us to work on this paper, and most of the technical results are done to answer this.
- The cost appears only at finite temperature with a student wider than its teacher. They leave this case open (end of their Section 3.2.7). This is the case we treat.
- A comparison for the Bayesian learner would need an evaluation of their Claim 5 in this case, and an exchange of the limits d → ∞ and n/d² → ∞. Neither has been done, by them or by us.
Thank you for the direct assessment. Your criticism of the writing is fair. We sincerely apologize for the confusion coming from the previous presentation. The revised PDF rewrites the abstract and the main text while all the mathematics is unchanged. We start by answering the questions, then addressing the concerns:
(The setting is a quadratic network with m ≥ d+1 hidden units, a Gaussian prior on the weights, a fixed teacher of rank r < d, and n noisy samples. The dimension d and the width m are fixed and n grows.) We derive a formula for the limit of the posterior and quantify the exact cost of overparametrization for a Bayesian learner (as in contrast to a non-Bayesian setting, we know over-parametrization gives benign optimization landscape and it also costs nothing to a learner that minimizes training loss). As a consequent of the limit, we also show rigorously that a small weight decay and additional small eigenvalues of the teacher network lower the risk.
Thanks very much for letting us elaborate on this. We made the comparison sharper in the revised manuscript. There are two major lines to work to compare to:
We agree with this, and this is also addressed in section 7. In this work, we do not analyze n \prop d^2. What the regime we work in gives us is the posterior itself in explicit form, not only the error. That is what makes the weight decay of order n^(−1/2) and the weak eigenvalues math derivations tractable. In the numerical experiments section, we included some observations on small samples.