- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes BayesNDE, a neural density estimation pipeline based on a latent-variable generative model. The model is fitted using an encoding generative modeling (EGM) warm start followed by alternating updates of observation-specific latent codes and shared network parameters. With the fitted parameters fixed, the method treats each observation’s marginal density as the normalizing constant of its latent posterior. It runs HMC, fits a defensive Student-t mixture proposal to a subset of the posterior samples, and combines proposal samples with the remaining posterior samples through bridge sampling. The paper presents consistency and overlap-based asymptotic efficiency results conditional on the fitted model, and evaluates the pipeline on synthetic and real-data density estimation, anomaly detection, and MNIST classification.
Critical strengths
Adapting the proposal to the latent posterior is a reasonable way to concentrate integration effort in relevant regions. The paper also includes a useful bridge-versus-importance-sampling ablation that holds the fitted model and proposal construction fixed.
Critical weaknesses
-
(Limited methodological novelty and insufficient differentiation from existing approaches.) The central recipe of obtaining posterior samples, fitting a proposal from those samples, and applying optimal bridge sampling is closely related to existing work, particularly Jia and Seljak (2020) [1]. The contribution appears primarily to be its observation-wise application with a particular mixture proposal. The manuscript acknowledges related work but does not convincingly establish what new technical obstacle this adaptation resolves. Wu et al. (2017) [2] also studied likelihood evaluation for decoder-based generative models using AIS. Direct comparisons with these closely related integration approaches on the same fitted models are missing, leaving the incremental contribution insufficiently demonstrated.
-
(The training procedure lacks a convincing statistical justification.) Ignoring the additional regularizers, Eqs. (31)–(32) take alternating optimization steps for the complete-data criterion , rather than the marginal log-likelihood . What statistical target does the full regularized procedure define, and how do its stationary points relate to marginal-likelihood estimation? The paper does not adequately address this distinction or justify choosing this training recipe over variational/ELBO-based training with a matched decoder. My concern is not that every density estimator must be an MLE, but that the particular surrogate and its additional penalties lack sufficient justification and controlled empirical validation. Section 4 conditions on an already fitted model and therefore does not resolve this issue.
-
(The experiments do not directly establish the accuracy of the proposed density computation.) The main comparisons evaluate different end-to-end pipelines and therefore combine model-fitting quality with integration accuracy. Appendix A.4 S6 partially addresses this through a fixed-model bridge-versus-IS ablation, which I appreciate. However, it still measures Spearman correlation with the data-generating density, rather than error relative to the fitted model’s density. These are different targets, and rank correlation cannot detect multiplicative normalization errors. A direct evaluation against reliable reference log-densities for fixed models, including Monte Carlo variability and computational cost, is needed to substantiate the central numerical claim.
-
(Important experimental comparisons are confounded, and the synthetic evidence is limited.) In Appendix A.5, BayesNDE shares parameters across labels in a single conditional model, whereas the baselines fit separate unconditional models for each label. This introduces a parameter-sharing difference that can affect the reported conditional-density improvements. In anomaly detection, Table 4 additionally uses different latent dimensions for the closest comparator: Roundtrip uses 3, 5, and 5, whereas BayesNDE uses 9, 21, and 16. These comparisons do not isolate the proposed methodological changes. Furthermore, the high-dimensional synthetic benchmark is a product of independent one-dimensional mixtures. Its exponentially many modes do not by themselves demonstrate difficulty in modeling high-dimensional dependence; a factorized density baseline is missing, while the single-Gaussian conditionals used by the MADE baseline are structurally misspecified for this benchmark.
-
(The computational tradeoff and posterior-sampling reliability are insufficiently established.) Each density query requires HMC, proposal fitting, and additional Monte Carlo evaluation. Yet the BANK runtime benchmark uses posterior draws, while the main UCI evaluations use 1,600. The runtime budget is selected by rank agreement with each method’s own high-budget estimate, not by a common log-density accuracy requirement. Thus, the reported timing does not establish the cost of achieving the main results or an advantage over related estimators at matched accuracy. Moreover, Appendix A.4 S5 computes the quantity called after pooling and randomly permuting chain samples. The authors acknowledge that this is not a conventional Markov-chain ESS. With all chains initialized close to the encoder output, actual mixing diagnostics and pointwise Monte Carlo error assessments are particularly important but are not provided. Numerical convergence of the bridge iteration does not establish adequate posterior exploration.
-
(The presentation obscures the scope and justification of the contribution.) The training description is fragmented across Sections 3.2 and 3.5 and Appendix A.4, making it unnecessarily difficult to reconstruct the procedure and distinguish inherited components from the proposed contribution. The emphasis on “Bayesian generative modeling” also requires more precise qualification: the network parameters are point estimates, and the posterior computation concerns latent variables. The manuscript should more clearly separate model fitting from density evaluation and explain which claims concern each stage. Reporting inconsistencies further hinder interpretation; for example, Table 1 reports test log-likelihoods but has a caption describing anomaly-detection precision@k.
Overall, I do not find the methodological increment or its practical advantages sufficiently established. The unresolved training rationale and the limitations of the empirical validation are substantive issues, rather than merely requests for additional exposition. I therefore recommend clear rejection.
References
[1] He Jia and Uros Seljak. Normalizing Constant Estimation with Gaussianized Bridge Sampling. AABI, PMLR 118:1–14, 2020.
[2] Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse. On the Quantitative Analysis of Decoder-Based Generative Models. ICLR, 2017. arXiv:1611.04273.
Questions for authors
Please see the weaknesses.
response w2 (cont.)
To evaluate the choice empirically, we compare MAP and ELBO training [2] with the same deterministic five-layer width-256 Gaussian decoder, two-dimensional latent space, EGM initialization, data, and 200-epoch schedule. Each objective is run with and without the additional θ-block penalties; results below average training seeds 0, 1, and 2 on the same 128 held-out points. Density evaluation uses quadrature of each fitted decoder, independently of its training encoder or latent codes. Let p∗ denote the data-generating density. For these training controls, Etrue is the mean absolute difference between each fitted decoder's quadrature log-density and logp∗; NLL is the negative mean of that quadrature log-density. Table 1 summarizes the four training controls.
Table 1. Matched-decoder training controls: means over training seeds 0, 1, and 2 on the same 128 test points.
All entries are in nats; smaller values are better. Density evaluation uses quadrature of each fitted decoder.
The penalty arms perform an additional correlation update and therefore use two θ updates per minibatch, versus one otherwise. These controls favor ELBO on Involute and show a benefit from the added penalties for MAP on GMM2; they do not establish universal superiority of one training objective.