- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when a singular model has similar generalization error under Bayesian prediction and maximum likelihood estimation. Its main condition is that the likelihood depends on the parameters through a regular function-space representation, and that the induced prior is continuous and positive near the truth. Under these conditions, both estimators achieve expected excess loss (effective d)/n. The authors establish this property for several linear and quadratic networks and empirically examine whether gradient descent reaches the corresponding maximum-likelihood solution.
Critical strengths
-
The paper identifies a useful distinction between redundant parameters and singular statistical behavior.
-
The role of the induced prior is informative.
-
The saturation arguments and the negative-moment analysis for quadratic networks address the nontrivial task of establishing regularity of the induced prior.
Critical weaknesses
-
There is an unresolved gap between the stated assumptions and the models analyzed. The appendix acknowledges that the cited singular-learning-theory results assume compact parameter spaces, whereas the examples use unbounded parameters with Gaussian priors. The extension is described as widely believed, without a proof.
-
The novelty of the general principle needs sharper positioning.
-
The nonlinear extension has a restricted scope. Although quadratic activation makes predictions nonlinear in the input, the model reduces to linear regression over symmetric-matrix coordinates, with a possible rank constraint.
Questions for authors
-
Can the Gaussian-prior examples be justified without relying on the unproved extension discussed in the appendix?
-
Which conclusions are new relative to existing regular estimation, quasi-regularity, and reduced-rank regression results?
-
What happens to the quadratic-network risk when unsuccessful optimization runs are included?
3 comments
Thank you for the clarifications which address my concerns about network description and unsuccessful optimization runs.
However, due to making the noncompact extension as an explicit assumption actually makes the Gaussian-prior generalization results a conditional result. This weakness still remains. Furtermore, the numerical results also do not provide the missing theoritical justification.
I also remain uncertian over the extension beyond regular asymptotic theory. I therefore retain my score.
Thank you. Every finite computation stays inside some ball of radius . If the Gaussian prior is restricted to such a ball, the parameter set is compact, Watanabe's conditions hold, and Assumption 1 is not needed. The proofs go through with the restricted prior, so for any realistic computation the results are unconditional.
Thank you for the review.
W1, Q1. Partly.
W2, Q2. Regular estimation, the λ values for reduced-rank regression and the ML rates are known, and we now cite them. What we see as new:
Quasi-regular models (Yamada and Watanabe 2012) have multiplicity at least 2 when singular, while effectively regular models have multiplicity 1.
W3. Agreed. We now describe the quadratic network as a symmetric reduced-rank regression (Section 7.3), and we removed the claim that it is genuinely nonlinear.
Q3. With all runs included, a single gradient-descent run does not reach the rate. We redid the experiment on two well-conditioned truths (App. E.6). About 30% of runs end at a spurious local minimum at every n, so n times the average excess risk grows linearly in n. Restarting gradient descent R times on the same training set and keeping the run with the lowest training loss recovers the rate if R grows slowly with n; R=16 matched the exact MLE up to n=5⋅104. This concerns only the optimizer: the exact MLE stays near λ, as the theory predicts.