acceptodds
Under review as a conference paper at ICLR 2027

Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding

Abstract

Posterior-seeking training need not produce the best predictive distribution in its family under model misspecification. For a last-layer weight belief, we locate the restriction responsible for this gap and quantify its cost. At the last layer the negative ELBO decomposes exactly into the belief’s predictive log scores, the prior KL, and a binding: a sum of per-observation Jensen gaps that constrains predictive dispersion through posterior covariance. Removing the binding gives the established KL-regularised form of predictive-score training; a parameterisation-dependent factor-normaliser form performs similarly at the last layer. At fixed predictive mean, the leading restricted direction is the variance at each input, and its local value is the classical variance-component score evaluated at the plug-in predictor. This yields both a diagnostic and a price: in the flat-leverage limit, the Gaussian cost is the Jensen gap of a representable residual-variance profile, whereas a likelihood without a dispersion parameter pays the full expressible improvement over the plug-in. Controlled experiments find typically tenths of a nat for Gaussian regression but units to tens of nats for overdispersed Poisson counts. In an exploratory, prospectively specified validation on 28 count sets, 27 unused elsewhere, a screen computed from the point-prediction head alone agreed with the measured outcome on 24 of 25 decided sets (an always-open rule: 18), and its price proxy, once fixed, was within a factor of three of the predictive-score gain of the fitted free profile on all 10 open sets tested afterwards. Score training changes the inferential target: it yields classical per-input overdispersed heads whose covariance represents residual dispersion, not posterior or epistemic uncertainty.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
variational inference, ELBO, Bayesian last layer, predictive log score, proper scoring rules, model misspecification, overdispersion, heteroscedastic regression, count regression, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1q17OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a Bayesian last layer: a one-hidden-layer network whose final linear weights have a Gaussian belief. It shows that the negative ELBO splits into the predictive log score, the prior KL, and a "binding" term (per-point Jensen gaps). The binding forces the posterior covariance to depend only on features and curvature, not on residuals, so it cannot capture heteroscedastic noise or overdispersion. The binding price is the test-NLL gap between the ELBO belief and the score-trained belief at the same features and mean. It is reported as tenths of a nat for Gaussian regression and units to tens of nats for overdispersed Poisson counts. The paper also proposes a dispersion screen, computed from the point-prediction head, to predict whether the price will be paid.

Critical strengths

1.The ELBO decomposition is clean and explains why the posterior covariance ignores the residuals. 2.The main experiment is well controlled: features and mean are fixed and only the last-layer covariance differs, so the gap can be attributed to the binding. 3.The practical advice, adding a dispersion parameter first for overdispersed counts, is simple and useful.

Critical weaknesses
  1. The scope seems narrow to me: only the last layer is Bayesian, the data are exchangeable, and the theory holds the mean and features fixed. Framing the problem as one restricted direction with a price is useful, but on the method side the free belief mostly recovers classical heteroscedastic and Poisson-lognormal heads.
  2. The experiments are small. Every trained backbone has a single hidden layer, and most datasets are UCI-scale or small count sets. The only strong models are frozen encoders like: ResNet-50, used as fixed feature extractors. It is unclear whether the results carry over to strong models trained end to end.
  3. Classification is not evaluated. For a Bernoulli or softmax head the label variance is determined by the mean, so I would expect the overdispersion channel to be closed and the price to be near zero. If so, the effect matters mainly for heteroscedastic regression and count data, which limits its relevance for most modern models.
  4. The practical relevance of the binding price is unclear. The clean gap requires both beliefs to share the same features and mean. When the models are trained separately, the gap also absorbs differences in features, optimisation, and early stopping. Given this narrow setting, I am not sure how much the binding matters for practitioners.
Questions for authors

See weakness

2 comments

Thank you for the review, and for the classification question, which improved the paper. We agree that the strongest substantive and controlled results are last-layer results at fixed latent mean and features. We have clarified this scope, while distinguishing it from the factorwise decomposition and the local variance expansion, which apply more generally (Section 2; Section 3 opens with the scope).

Below, we provide the answers to the questions/comments from the review. See also "Common response to reviews"

Recovering classical heads. We agree that without the binding the belief is a classical heteroscedastic or Poisson-lognormal head; the paper now says this and does not propose it as a method.

Size of the experiments; strong models trained end to end. For regression and counts this is unchanged: the trained backbones of the last-layer studies have one hidden layer, and the strong encoders are frozen. We added one deep network trained end to end, a ResNet-18 on CIFAR-10 (Appendix G.1, Table 30), which is the classification case below. A deep network trained end to end on a regression or count task is not included. Corollary 2 applies to a Gaussian last layer on any deterministic feature map, including features produced by a deep network, but it does not predict the size of the gap when features and mean adapt end to end, so we do not extrapolate the frozen-feature measurements to that setting.

Classification. Your expectation is what the theory gives, and it is now a statement: for a Bernoulli head with a logit or probit link at a matched mean the price is zero, and the same holds per output for one-vs-all heads (Corollary 3); a remark covers softmax informally. The CIFAR-10 experiment provides an empirical near-null consistent with this result; it is not a controlled closure test, because the latent mean is not shown to be matched. This limits the matched-mean dispersion mechanism in classification; predictive differences can still arise when the mean is misspecified.

Practical relevance when the models are trained separately. Table 2 now separates the two. The "frozen, bound" columns are the control at shared features and latent mean. The "ELBO" and "Laplace" columns are separately trained posterior-seeking models, and the text reads them out: an early-stopped ELBO belief pays less than the control, 0.04-0.11 nats for Gaussian regression, 0.6-2.1 on four of the five overdispersed count sets and nothing on the fifth; on the generators and news this is because early stopping keeps it from its optimum; the Laplace fit pays most (Section 5, "What does a trained ELBO belief pay?"). The exploratory 28-set screen study, with results on 27 valid data sets, gives the end-to-end comparison on further count sets (Appendix E, Table 12)

Reviewer_1q17OpenReview ↗

Thank you for the rebuttal.

Thank you for the response. The authors addressed my questions. Reviewer cjnP makes a fair point that a negative binomial likelihood removes most of the cost in practice. My positive rating rests on the paper’s explanatory contribution. The decomposition explains why the ELBO pays this price and predicts when it does, and I find that explanation useful. I will keep my rating.

Reviewer_cJnPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3(+1)
Read review
Paper summary

This paper studies last-layer Bayesian neural networks trained with the ELBO when the data are noisier in some regions than others. The authors show that the ELBO cannot correctly represent uncertainty in this setting, since its uncertainty only reflects how much data the model has seen. They propose training the predictions directly instead, without requiring the weight distribution to resemble a posterior. They also quantify how much this choice affects performance: the cost is small for Gaussian regression but large for count data, and a simple overdispersion test can predict it in advance.

Critical strengths
  1. The paper provides proof of ELBO splits exactly into how well the predictions fit + prior term + the binding (Theorem 1). This theoretically pins the gap between bayesian training and good prediction to one term.

  2. The experiments clearly back up the theoretical findings, which makes them more concrete and interesting (e.g., the fixed-feature controls that isolate the binding, the HMC check, and the zero cost on correctly specified datasets.).

Critical weaknesses
  1. The writing is very difficult to follow. It relies on coined terms, compressed sentences, and a main text that relies too heavily on the appendix.

  2. As the authors acknowledge, the paper components are known and the main takeaway is more of a practical guidance than a new method or insight. “If your counts are overdispersed, use negative binomial instead of Poisson” and “test for overdispersion first” are standard in count regression. Therefore, while the unification is interesting and well-done, the practical relevance is limited as the stated problem already has a known workaround.

Questions for authors
  1. How large is the binding cost when the ELBO uses a well-chosen likelihood (NB for counts), across all datasets rather than just three?

  2. What does the score-trained belief offer over simply using a negative binomial or mean–variance head?

3 comments

Thank you for the review; your summary helped us see how the submitted framing read. We have clarified that the paper does not propose training the predictions directly; that objective is known, and the abstract and the introduction now say so.

Below, we provide the answers to the questions/comments from the review. See also "Common response to reviews"

Writing: coined terms, compressed sentences, reliance on the appendix. See the common response. The named terms that remain (binding, bound and free belief, price) are each defined at first use, the statements needed to follow the argument are in the main text, and Appendix B opens with a list of the appendix statements the main text cites and what each is used for.

Components are known; the takeaway is practical guidance that is standard. We agree that the components are known (Table 6 lists the provenance of each) and that "use a negative binomial for overdispersed counts" is standard. Our contribution is not that recommendation, but the connection between the ELBO's Jensen-gap remainder (Theorem 1), the residual-blind Gaussian variational covariance under canonical likelihoods at fixed latent mean and features (Corollary 2), and the predictive improvement available through expressible latent variance (Corollaries 3-5), together with the measurement that the posterior estimated by HMC shows a similar loss on the count generators (Table 26).

Q1, the binding cost under a negative-binomial ELBO, on all data sets. Added: column "frozen, bound (NB)" of Table 2, every count row (details in Table 23). The gap is 0.01-0.09 nats on the overdispersed random-split rows, approximately zero on the closure and near-closure rows, and 0.10 and 0.40 on the two chronological rows, where the Poisson-bound belief pays 2-48. In this control the global dispersion is held at the fitted head's value and the features are that head's own; it is not a fully optimised negative-binomial ELBO model (Section 5; Appendix D).

Q2, what the score-trained belief offers over a negative-binomial or mean-variance head. In test NLL, nothing that we claim. It is within 0.02 nats of a directly trained head of the same family on the count sets (0.04 on one) and shows no systematic gain for the Gaussian; against heads of other families the differences go both ways (Table 3; Section 5, "Is there an advantage in optimising F_SC?")

Thank you for your clear response and explanation about the summary. The new negative-binomial control answers Q1, and I have raised my soundness score accordingly. However, the result (0.01–0.09 nats under NB versus 2–48 under Poisson) and the answer to Q2 confirm that the standard remedy removes nearly all of the cost and that the score-trained belief offers no predictive advantage. The contribution is therefore an explanation of a known gap with a known fix, and the presentation remains dense. I keep my rating.

Thank you for the follow-up.

We agree on the facts: with a negative-binomial likelihood the cost nearly vanishes, and the score-trained belief does not beat the best matched head. If the paper is primarily ranked by empirical gain over the best known method, there is indeed none, and we do not claim one.

One clarification of the explanation itself: the free belief under a Poisson likelihood is an overdispersed model (Poisson-lognormal), so the small negative-binomial numbers are what the decomposition predicts rather than a separate fix. We would adjust "an explanation of a known gap with a known fix" in three respects:

  • The paper gives a test on the likelihood that says in advance whether an ELBO-trained last layer pays (Section 4): nothing for Bernoulli (Corollary 3), typically tenths of a nat for Gaussian (Corollary 4), units to tens for Poisson (Corollary 5).
  • It measures that cost directly, at fixed features and latent mean (Section 5, Table 2, column "frozen, bound"; the negative-binomial control is the adjacent column).
  • Section 2 is not specific to the last layer: the decomposition (Theorem 1), the reading of the binding as a constraint that ties the belief to the posterior (Corollary 1), and its first-order penalty on predictive variance, which cancels part of the predictive score's reward (Proposition 1), all hold on any directed factor graph. The likelihood remedy answers the cost along the variance direction at fixed mean, which at the last layer is the whole of it, since there the binding only restricts the covariance. Below the last layer the constraint also acts on directions that move the mean, and whether a change of likelihood removes that part is not known. A method for that regime is outside this paper, but Section 2 says where to look.

On presentation: we acknowledge the paper builds on prerequisites that are not common knowledge, the Bethe free energy among them, and a reader can be lost before reaching our own results. If you could point to where that happened for you (e.g., motivation, the decomposition, Table 1, ...), it would be valuable feedback for our future work on this.

Reviewer_gYNhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper provides a quantitative analysis of the ELBO loss under the model miss-specification. Under a misspecified likelihood, the predictive role may benefit from a distribution that remains spread out, because that spread supplies variability missing from the likelihood. Posterior-oriented training tends to determine that spread through parameter-estimation uncertainty instead. The paper rewrites the difference between the objectives as a sum of Jensen gaps, studies its effect on last-layer variance, and estimates the predictive cost in particular regimes.

While the idea of using Bethe Ansatz is interesting and promising, the paper is very difficult to follow and requires major revision above the scope of the camera-ready edits.

Critical strengths
  • Paper addresses interesting scientific question.

  • Theorem 1 isolates the predictive log score, prior contribution, and observation-dependent Jensen gaps. This is clarifies the difference between the ELBO training and a predictive-score training.

  • Gaussian example (Def 4, p4) and experiment (F.2) is good motivational example to demonstrate the usefulness of the method: freezing features and the predictive mean, then refitting the variance under free and posterior-constrained parameterisations. e.g. p 5 Collorary 3 is

Critical weaknesses
  • The text is hard to follow and the information in the text is too compressed making it hard to parse. While the appendices are well written, the main article in the text should be clearly explain each term before using them for the time or cite them or provide reference to the appendix where the terms may be explained, e.g Bethe expression p.1

  • Theorem 2 is very hard to follow: "Theorem 2 (The price of the binding). In the setting of (21), with Ξ compact, under the regularity listed in its proof (Appendix C.2), in the flat-leverage limit (supx,ξ vbind ξ (x) → 0): " ... The theorem should be clearly stated with all its assumption, claims that conditions are stated in the proof is example of AI slop or poor writing style

Questions for authors
  1. What are authors aiming to define in Definition 6?

  2. What independently justifies the choice of prior regularizer? Removing the likelihood binding term from the ELBO yields predictive log loss plus KL(q∣p)\mathrm{KL}(q|p). The proposed objective makes an additional change by replacing this KL term with −log⁡∫p(w)q(w)dw-\log\int p(w)q(w)dw. Since the exact decomposition does not uniquely imply this replacement, what theoretical or statistical argument supports the overlap penalty, and under what conditions should it be preferred to retaining the KL regularizer?

  3. Which of these two changes accounts for the reported improvements? Changing the likelihood objective and changing the prior regularizer are distinct interventions. What do the reported prior-term ablations establish about their separate contributions, and do they demonstrate an additional benefit from the overlap penalty over predictive log loss with a KL regularizer when the model, optimization, and hyperparameter tuning procedures are comparable?

2 comments

Thank you for the careful reading, and for the specific places where the text failed. Besides the Gaussian case you describe, the revision brings forward the count case, where the same mechanism costs units to tens of nats under a Poisson likelihood (Corollary 5; Table 2).

Below, we provide the answers to the questions/comments from the review. See also "Common response to reviews"

Terms used before they are explained ("Bethe expression", p. 1). The phrase is gone. The Bethe free energy is introduced in one sentence where it first appears (Introduction, second paragraph) and is used only as the local view that motivates scoring each factor against the belief's own marginal (Section 2.1). Section 2 now starts from the three-term identity (Eq. 1), and each later term is defined at first use. We made a few additional framing/notation adjustments to ensure that terms are defined properly.

Theorem 2. It is now stated with all its assumptions inside the statement (Appendix B.3), followed by a paragraph saying what each part means. The main text gives the limit, the definition of the price, and its main conditions in words (Section 4), and Corollaries 3-5 each state the conditions they need.

Q1, submitted Definition 6. Definition 6 in the submitted version, now Definition 8, distinguishes three quantities: a screen for whether the plug-in leaves an available overdispersion direction, a proxy for the possible predictive gain, and a certificate obtained by fitting a free variance profile at a fixed latent mean. The screen and proxy are computed before training the belief; the certificate is measured afterward. We have added an explanatory paragraph and moved the definition to Appendix B.3 because the main argument does not depend on these diagnostics.

Q2, what justifies the prior term. The decomposition does not imply that the overlap penalty should replace the prior KL. Our motivation is the factor-local construction: scoring the prior factor by its normaliser, as for the other factors (Definition 1). We do not establish a statistical advantage for this choice, and we state its dependence on the parameterisation. Predictive log loss with the prior KL remains the established PPGPR / direct-loss alternative (Section 3, paragraph after Definition 4).

Q3, which of the two changes accounts for the differences. They are reported separately. The column "free, KL" of Table 2 is the prior swap: about 0.01 nats or less on every count row and in the median over the UCI sets, with single UCI sets differing from -0.18 to +0.13 (Table 18; for the UCI row the KL arm is also selected over a prior-precision grid). The binding is measured by a control that freezes the features and the latent mean and refits the covariance, free or bound; for the Gaussian rows both refits minimise the same objective, so the prior term does not change. For counts, the frozen comparison also changes the prior regulariser. The separate prior-swap ablation has a much smaller effect, about 0.01 nats on the random-split rows and up to 0.05 on the chronological rows, which supports, but does not prove exact isolation of, the reading that the large count gaps are predominantly due to posterior-seeking covariance fitting. We claim no benefit for the prior term (Section 5, first paragraph).

I would like to thank authors for their clarifications. I appreciate all the effort that authors put into reworking the manuscript. The change is too substantial that the paper should be re-submitted and reviewed again. I would like to encourage authors to carefully review the AI traces. Presented idea and study is interesting and if made more succinct and sharpened, it could be interesting submission in its future version. I will keep my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.