- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a Bayesian last layer: a one-hidden-layer network whose final linear weights have a Gaussian belief. It shows that the negative ELBO splits into the predictive log score, the prior KL, and a "binding" term (per-point Jensen gaps). The binding forces the posterior covariance to depend only on features and curvature, not on residuals, so it cannot capture heteroscedastic noise or overdispersion. The binding price is the test-NLL gap between the ELBO belief and the score-trained belief at the same features and mean. It is reported as tenths of a nat for Gaussian regression and units to tens of nats for overdispersed Poisson counts. The paper also proposes a dispersion screen, computed from the point-prediction head, to predict whether the price will be paid.
Critical strengths
1.The ELBO decomposition is clean and explains why the posterior covariance ignores the residuals. 2.The main experiment is well controlled: features and mean are fixed and only the last-layer covariance differs, so the gap can be attributed to the binding. 3.The practical advice, adding a dispersion parameter first for overdispersed counts, is simple and useful.
Critical weaknesses
- The scope seems narrow to me: only the last layer is Bayesian, the data are exchangeable, and the theory holds the mean and features fixed. Framing the problem as one restricted direction with a price is useful, but on the method side the free belief mostly recovers classical heteroscedastic and Poisson-lognormal heads.
- The experiments are small. Every trained backbone has a single hidden layer, and most datasets are UCI-scale or small count sets. The only strong models are frozen encoders like: ResNet-50, used as fixed feature extractors. It is unclear whether the results carry over to strong models trained end to end.
- Classification is not evaluated. For a Bernoulli or softmax head the label variance is determined by the mean, so I would expect the overdispersion channel to be closed and the price to be near zero. If so, the effect matters mainly for heteroscedastic regression and count data, which limits its relevance for most modern models.
- The practical relevance of the binding price is unclear. The clean gap requires both beliefs to share the same features and mean. When the models are trained separately, the gap also absorbs differences in features, optimisation, and early stopping. Given this narrow setting, I am not sure how much the binding matters for practitioners.
Questions for authors
See weakness
2 comments
Thank you for the rebuttal.
Thank you for the response. The authors addressed my questions. Reviewer cjnP makes a fair point that a negative binomial likelihood removes most of the cost in practice. My positive rating rests on the paper’s explanatory contribution. The decomposition explains why the ELBO pays this price and predicts when it does, and I find that explanation useful. I will keep my rating.
Thank you for the review, and for the classification question, which improved the paper. We agree that the strongest substantive and controlled results are last-layer results at fixed latent mean and features. We have clarified this scope, while distinguishing it from the factorwise decomposition and the local variance expansion, which apply more generally (Section 2; Section 3 opens with the scope).
Below, we provide the answers to the questions/comments from the review. See also "Common response to reviews"
Recovering classical heads. We agree that without the binding the belief is a classical heteroscedastic or Poisson-lognormal head; the paper now says this and does not propose it as a method.
Size of the experiments; strong models trained end to end. For regression and counts this is unchanged: the trained backbones of the last-layer studies have one hidden layer, and the strong encoders are frozen. We added one deep network trained end to end, a ResNet-18 on CIFAR-10 (Appendix G.1, Table 30), which is the classification case below. A deep network trained end to end on a regression or count task is not included. Corollary 2 applies to a Gaussian last layer on any deterministic feature map, including features produced by a deep network, but it does not predict the size of the gap when features and mean adapt end to end, so we do not extrapolate the frozen-feature measurements to that setting.
Classification. Your expectation is what the theory gives, and it is now a statement: for a Bernoulli head with a logit or probit link at a matched mean the price is zero, and the same holds per output for one-vs-all heads (Corollary 3); a remark covers softmax informally. The CIFAR-10 experiment provides an empirical near-null consistent with this result; it is not a controlled closure test, because the latent mean is not shown to be matched. This limits the matched-mean dispersion mechanism in classification; predictive differences can still arise when the mean is misspecified.
Practical relevance when the models are trained separately. Table 2 now separates the two. The "frozen, bound" columns are the control at shared features and latent mean. The "ELBO" and "Laplace" columns are separately trained posterior-seeking models, and the text reads them out: an early-stopped ELBO belief pays less than the control, 0.04-0.11 nats for Gaussian regression, 0.6-2.1 on four of the five overdispersed count sets and nothing on the fifth; on the generators and news this is because early stopping keeps it from its optimum; the Laplace fit pays most (Section 5, "What does a trained ELBO belief pay?"). The exploratory 28-set screen study, with results on 27 valid data sets, gives the end-to-end comparison on further count sets (Appendix E, Table 12)