acceptodds
Under review as a conference paper at ICLR 2027

Effective Regularity: When Point Estimates Generalize Like the Bayesian Posterior in Singular Models

Abstract

Singular learning theory explains how overparameterized models can generalize well despite having many parameters. Its main guarantees, however, concern Bayesian prediction, whereas neural networks are typically trained to a single solution using gradient descent. In singular models, these two approaches need not have the same generalization behavior. We identify a class of model-data regimes, which we call effectively regular, in which they do. Effective regularity arises when the likelihood depends on the parameters only through a lower-dimensional function class that is locally well behaved around the true function. We prove that Bayesian prediction and maximum likelihood then have the same leading generalization error, , where is the dimension of the network's function class near the truth rather than its number of parameters. We establish effective regularity for several shallow and deep linear networks and for a shallow nonlinear network. Finally, we examine empirically when gradient descent reaches the corresponding maximum-likelihood solution and therefore achieves the same rate.

Primary area:
learning theory
Keywords:
deep learning, singular learning theory, generalization theory, Bayesian learning, maximum likelihood estimation, overparameterized neural networks, real log-canonical threshold, deep linear networks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UqYuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when a singular model has similar generalization error under Bayesian prediction and maximum likelihood estimation. Its main condition is that the likelihood depends on the parameters through a regular function-space representation, and that the induced prior is continuous and positive near the truth. Under these conditions, both estimators achieve expected excess loss (effective d)/n. The authors establish this property for several linear and quadratic networks and empirically examine whether gradient descent reaches the corresponding maximum-likelihood solution.

Critical strengths
  1. The paper identifies a useful distinction between redundant parameters and singular statistical behavior.

  2. The role of the induced prior is informative.

  3. The saturation arguments and the negative-moment analysis for quadratic networks address the nontrivial task of establishing regularity of the induced prior.

Critical weaknesses
  1. There is an unresolved gap between the stated assumptions and the models analyzed. The appendix acknowledges that the cited singular-learning-theory results assume compact parameter spaces, whereas the examples use unbounded parameters with Gaussian priors. The extension is described as widely believed, without a proof.

  2. The novelty of the general principle needs sharper positioning.

  3. The nonlinear extension has a restricted scope. Although quadratic activation makes predictions nonlinear in the input, the model reduces to linear regression over symmetric-matrix coordinates, with a possible rank constraint.

Questions for authors
  1. Can the Gaussian-prior examples be justified without relying on the unproved extension discussed in the appendix?

  2. Which conclusions are new relative to existing regular estimation, quasi-regularity, and reduced-rank regression results?

  3. What happens to the quadratic-network risk when unsuccessful optimization runs are included?

3 comments

Thank you for the review.

W1, Q1. Partly.

  • λ\lambda and its multiplicity do not need compactness.
  • For the generalization error and ν\nu, we now state the extension as an explicit assumption (Assumption 1) instead of calling it widely believed.
  • The assumption holds for the prior truncated to any large ball.
  • Our Bayesian experiments on effectively regular models use the untruncated Gaussian prior, and their results agree with the assumption: E[Vn]=11.96±0.03\mathbb E[V_n]=11.96\pm0.03 against deff=12d_{\rm eff}=12 in the running example, and 1.005±0.0071.005\pm0.007 against 11 in a Regime A check (App. F).

W2, Q2. Regular estimation, the λ\lambda values for reduced-rank regression and the ML rates are known, and we now cite them. What we see as new:

  • a condition under which Bayes and maximum likelihood have the same rate in a singular model, with ν=λ=deff/2\nu=\lambda=d_{\rm eff}/2;
  • the role of the induced prior in deciding whether they agree;
  • the saturated regimes;
  • condition (iii) for the quadratic network.

Quasi-regular models (Yamada and Watanabe 2012) have multiplicity at least 2 when singular, while effectively regular models have multiplicity 1.

W3. Agreed. We now describe the quadratic network as a symmetric reduced-rank regression (Section 7.3), and we removed the claim that it is genuinely nonlinear.

Q3. With all runs included, a single gradient-descent run does not reach the rate. We redid the experiment on two well-conditioned truths (App. E.6). About 30% of runs end at a spurious local minimum at every nn, so nn times the average excess risk grows linearly in nn. Restarting gradient descent RR times on the same training set and keeping the run with the lowest training loss recovers the rate if RR grows slowly with nn; R=16R=16 matched the exact MLE up to n=5⋅104n=5\cdot10^4. This concerns only the optimizer: the exact MLE stays near λ\lambda, as the theory predicts.

Thank you for the clarifications which address my concerns about network description and unsuccessful optimization runs.

However, due to making the noncompact extension as an explicit assumption actually makes the Gaussian-prior generalization results a conditional result. This weakness still remains. Furtermore, the numerical results also do not provide the missing theoritical justification.

I also remain uncertian over the extension beyond regular asymptotic theory. I therefore retain my score.

Thank you. Every finite computation stays inside some ball of radius RR. If the Gaussian prior is restricted to such a ball, the parameter set is compact, Watanabe's conditions hold, and Assumption 1 is not needed. The proofs go through with the restricted prior, so for any realistic computation the results are unconditional.

Reviewer_TewuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

There is often a gap between the insights of SLT and their practical relevance in explaining the success of DL methods. While many SLT results characterize the generalization behavior in Bayesian learning, networks in practice are trained as point estimates using ML maximization. This paper identifies a singular regime (called effectively regular) in which Bayesian Learning and maximum-likelihood estimation have, up to leading order, the same expected excess generalization loss. They study several simple neural network architectures (linear, shallow quadratic) when the effective regularity assumptions hold. The paper furthermore adds experiments supporting the theoretically proven asymptotics.

Critical strengths

The biggest strength of the paper is that it gives a theoretical bridge between Bayesian SLT prediction and MLE methods: For effectively regular models, Bayesian prediction and MLE have the same leading expected excess loss coefficient: deff/2=λ=νd_{eff} /2 = \lambda = \nu. In general, this does not hold for singular models, and this paper gives explicit conditions under which we can relate both approaches.

Second, to show that their setting covers genuinely singular and overparameterized models, they prove that specific linear and quadratic models satisfy effective regularity. I believe their methods for proving, in particular, the quadratic case, are interesting and also give some genuine insight into understanding the effectively regular regime. The saturated case, for example, demonstrates that sufficient width makes the density of the induced prior on effective predictors continuous and bounded, a crucial condition of effective regularity. This illustrates how overparameterization can lead to more regularity in an effective model.

Critical weaknesses

For regular models, the equality d/2=λ=νd/2 = \lambda = \nu is clear. Their assumptions for effectively regular models identify specific parameterizations that allow to restrict to a regular model with a regular induced prior on an effective parameter space. Thus, the main contribution lies in understanding which interesting singular cases satisfy these assumptions, which is nontrivial. I think the deep linear and quadratic cases give a good starting point, but including more general models would have made the paper significantly stronger. In particular, they mainly study models in which adding redundant parameters does not increase the leading asymptotic generalization coefficient. This does not explain how highly expressive networks can lead to lower effective complexity solutions in practice. The same holds for the experiments; here, it would be particularly interesting to have empirical analysis of those cases not covered by the theory, to see if the theoretical insights extend to practically relevant settings. In particular, the opening sentence in the abstract, “SLT explains how overparameterized models can generalize well despite having many parameters,” is too general. As seen in the paper itself, Bayesian SLT generalization coefficients can be used for MLE methods only in specific cases.

Second, there are a few unclear steps in their proofs. First, in the saturation argument in Appendix B.4, they condition on (BT,AL)(B_T, A_L) to bound the density of F. However, for condition (iii), they need a bound for the joint density ρ(C,D,L,F)\rho(C, D, L, F). They claim this follows by dominated convergence, which is not clear to me.

Furthermore, for the submersion-based density arguments, they generally only use the local estimates for bounding the density. This requires an additional argument to extend this to a bound that includes contributions from the entire pre-image of a neighborhood of the true effective parameter, which they do not provide. As proving condition (iii) is the crucial task in verifying the assumptions of Theorem 1 in the examples, this should be fixed / explained better.

Questions for authors
  1. Can you explain how dominated convergence leads to a bound on the joint density? Which dominating function is used? Furthermore, why are the local estimates in the saturation bounds enough without accounting for the entire preimage?

  2. Can you add more experiments for more realistic models, for example, with ReLU activation? Even if they are not covered by your theory.

2 comments

Thank you for the careful reading. Both points were right.

W2, W3, Q1. The B.4 argument conditioned on blocks that are correlated, so it did not bound the joint density. Local submersion estimates also give only lower bounds. We replaced these proofs.

For the shallow network, W=BAW=BA is Gaussian given AA, so the pushforward density is exactly φˉ(W)=EA[N(W;0,A⊤A⊗I)].\bar\varphi(W)=\mathbb E_A[N(W;0,A^\top A\otimes I)]. Near W0W_0 the integrand is bounded by Cdet⁡(A2⊤A2)−N/2C\det(A_2^\top A_2)^{-N/2}, which is integrable exactly when H≥N+M−rH\ge N+M-r. Dominated convergence then gives continuity (Proposition 6, App. B.4). The other regimes and networks use similar bounds over the whole fibre (App. B to D).

W1. We rewrote the opening of the abstract and added a sentence on scope to the discussion.

Q2. We added a ReLU experiment (App. G) with a two-unit teacher in input dimension 3.

  • Student as wide as the teacher. ML and Bayes both give a coefficient near 3, which is half the functional dimension, and ν^≈3\hat\nu\approx3 as well. Half the number of weights would be 4.
  • Student twice as wide. The two separate: Bayes gives λ^≈3.7\hat\lambda\approx3.7 and ML about 5.9.

Thank you for the revisions. My proof concerns have been adequately addressed, and I appreciate the additional ReLU experiments. I am raising my score to 3.

I would not raise my score to 4. The main idea appears to be largly present in the literature (see reviewer b2xo). I appreciate the rigorous proofs of the framework’s assumptions for the specific models studied, but would have liked a clearer explanation of what these results suggest about generalization in realistic networks.

Reviewer_b2xoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces effective regularity: a singular model whose likelihood factors through a map onto a lower-dimensional function space, where the target is regular and the pushforward prior is well behaved. Under these conditions, the authors show that the learning coefficient and singular fluctuation both equal d_eff/2, and that maximum likelihood achieves the same leading generalization error, d_eff/(2n). They prove effective regularity in several regimes of shallow and deep linear networks and a quadratic network, identify two mechanisms (matched rank and saturation), discuss a regime where the condition fails, and empirically examine whether gradient descent reaches the maximum-likelihood solution in these models.

Critical strengths
  • Proofs are careful and detailed.
  • The ν measurement via an exact blocked Gibbs sampler, with a validation run against a known target, is well designed.
  • The authors are candid about GD trapping in the quadratic Regime D and about the conditioning bias.
  • The saturated-regime boundedness lemmas (Lemma 1, the deep Bartlett argument, the Veronese–Gram bound) are genuine technical work.
Critical weaknesses

First, the two main theorems are effectively statements of known results and ideas, and the paper does not really cite the relevant work here. The general term is generic vs. non-generic degeneracies (see here and here). Theorem 1 reduces to reparametrization invariance: eq. (12) is a change of variables, and Step 1 (eq. 10) is the law of total variance and in the submersive cases (Regimes A and D, and the quadratic Regime D analog), this is the known minimally singular / Morse–Bott setting. Then theorem 2 is immediate once the function-space model is regular, since the MLE depends on w only through the parameter function map.

The other formal results after this are all model specific, which is not strictly a weakness, but all of the results depend on an unproven fact that extends SLT to non-compact domains. While there is precedent for taking this as an assumption in the SLT literature in specific situations, it is not universal. Furthermore, the results of Ayogi on linear networks also use the compactness assumption. The qualitative Regime B conclusion survives, but the specific value λ\lambda = 7.5 and the claimed equality at H = N+M−r−1 are not established. Please justify the transfer or state it as an assumption.

The relationship with gradient descent is overstated and weak. the abstract and introduction set up "three notions of learning," and Figure 1 is captioned "The three notions of learning coincide." Section 6 and contribution 3 are actually modest ("outside the scope") and there are no strong theory results. Furthermore, there is literature looking explicitly at the relationship between gradient optimizers vs. Bayes and even explicitly with singular structure. Furthermore, the paper does not really engage meaningfully with the singular learning theory literature, where their notion of effectively regular is very closely related to notions of "minimally singular".

Questions for authors

Can you better describe the relationship with previous art in singular learning theory? Can you give a general characterization of when the pushforward prior is unbounded? When should we expect models to not be effectively regular? How generic is this condition?

1 comment

Thank you for the review and for the links.

W1, Q1. We added the missing references:

  • eq. (12) is Watanabe's volume formula (App. A, Step 3);
  • Section 9 now says that Regimes A and D are minimally singular in the sense of Wei et al. (2023);
  • the λ\lambda values are due to Aoyagi and Watanabe.

Theorem 2 is now a proposition.

Two points do not follow from these results.

  • Minimal singularity fixes λ\lambda but not ν\nu or the ML rate. The model Y∼N(ξ(cos⁡γ,sin⁡γ),I2)Y\sim N(\xi(\cos\gamma,\sin\gamma),I_2) is minimally singular with λ=1/2\lambda=1/2, but ν≈0.70\nu\approx0.70 and its ML coefficient is 1 (App. A). It has the same KK and prior as Y∼N(ξ,1)Y\sim N(\xi,1) with an unused parameter γ\gamma, for which ν=λ=1/2\nu=\lambda=1/2 and the ML coefficient is 1/21/2. So ν\nu and the ML rate are not determined by KK and the prior. Condition (iii) is what rules this out.
  • In Regime C, at low-rank truths, the model is effectively regular but not minimally singular.

W2.

  • Compactness. As you suggest, we now state the extension to non-compact parameter spaces as an explicit assumption (Assumption 1, at the start of the appendix). λ\lambda and its multiplicity do not need it, because the change of variables and Laplace's method on the target hold without compactness. For a prior truncated to a large ball, Watanabe's conditions hold.
  • λ=7.5\lambda=7.5 and the boundary. In Regime B, λ=7.5\lambda=7.5 is the Aoyagi-Watanabe value, derived for compact parameter sets, and its transfer to the untruncated prior is now part of the same stated assumption. At H=N+M−r−1H=N+M-r-1 the multiplicity is 2, so Bayes and ML share only the leading constant. We corrected the text (Section 8, App. B.1).

Q2 to Q4. The induced prior can be unbounded only at critical points of Φ\Phi on the fibre, or if prior mass escapes to infinity along it. In the linear networks this reduces to a Wishart negative moment, which is finite exactly when H≥N+M−rH\ge N+M-r (App. B.4, Proposition 6).

For any model of this form, λ=deff/2\lambda=d_{\rm eff}/2 at almost every truth. But the truths that matter for overparameterized networks are the low-rank ones. There, effective regularity needs saturation (Regime C) and fails otherwise (Regimes B and E).

W3. We agree. The abstract, introduction and Figure 1 no longer present gradient descent as a third notion of learning; the paper is about Bayes and maximum likelihood. Section 6 only notes that global minimizers of the training loss are ML estimators, and checks empirically whether gradient descent reaches one. We now cite Mingard et al. (2021) and Hennick and De Baerdemacker (2026).

Comments on the paper

1 comment

We thank the reviewers. A revised PDF is uploaded, and the supplementary material contains a version with the changes highlighted. The main result is unchanged: in effectively regular models, Bayes and maximum likelihood have the same leading generalization error deff/(2n)d_{\rm eff}/(2n), and ν=λ=deff/2\nu=\lambda=d_{\rm eff}/2. The main changes are:

  • the extension of SLT to non-compact parameter spaces is now stated as an explicit assumption.
  • the proofs of condition (iii) now bound the density over the whole fibre.
  • a minimally singular example where Bayes and ML differ shows what condition (iii) adds.
  • a small ReLU experiment checks whether ML and Bayes still share the rate set by the effective dimension.
  • gradient descent is now a side question: the abstract and introduction no longer present it as a third notion of learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.