acceptodds
Under review as a conference paper at ICLR 2027

BayesNDE: Bayesian Generative Modeling for Neural Density Estimation

Abstract

Density estimation is a fundamental problem in statistics and machine learning. In this work, we introduce BayesNDE, a neural density estimator based on Bayesian generative modeling. BayesNDE learns a Bayesian generative model and evaluates its density without requiring invertible networks or Jacobian-determinant computation. For each observation, it infers a sample-specific latent posterior to construct an adaptive proposal that focuses computation on regions contributing most to its density. Bridge sampling then combines samples from this proposal with separate posterior samples to estimate the density. Experiments on nonlinear and multimodal synthetic datasets show improved estimation of density values and better recovery of the density structure compared to the state-of-the-art neural density estimators. Applications to real-world datasets further demonstrate improved anomaly detection. Together, these results highlight BayesNDE as a flexible and effective neural density estimator, demonstrating how posterior inference can turn generative models into tools for density estimation. The code and tutorials are available at https://github.com/anonymous.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
density estimation, bridge sampling, bayesian generative modeling, anomaly detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ELoBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes BayesNDE, a neural density estimation pipeline based on a latent-variable generative model. The model is fitted using an encoding generative modeling (EGM) warm start followed by alternating updates of observation-specific latent codes and shared network parameters. With the fitted parameters fixed, the method treats each observation’s marginal density as the normalizing constant of its latent posterior. It runs HMC, fits a defensive Student-t mixture proposal to a subset of the posterior samples, and combines proposal samples with the remaining posterior samples through bridge sampling. The paper presents consistency and overlap-based asymptotic efficiency results conditional on the fitted model, and evaluates the pipeline on synthetic and real-data density estimation, anomaly detection, and MNIST classification.

Critical strengths

Adapting the proposal to the latent posterior is a reasonable way to concentrate integration effort in relevant regions. The paper also includes a useful bridge-versus-importance-sampling ablation that holds the fitted model and proposal construction fixed.

Critical weaknesses
  1. (Limited methodological novelty and insufficient differentiation from existing approaches.) The central recipe of obtaining posterior samples, fitting a proposal from those samples, and applying optimal bridge sampling is closely related to existing work, particularly Jia and Seljak (2020) [1]. The contribution appears primarily to be its observation-wise application with a particular mixture proposal. The manuscript acknowledges related work but does not convincingly establish what new technical obstacle this adaptation resolves. Wu et al. (2017) [2] also studied likelihood evaluation for decoder-based generative models using AIS. Direct comparisons with these closely related integration approaches on the same fitted models are missing, leaving the incremental contribution insufficiently demonstrated.

  2. (The training procedure lacks a convincing statistical justification.) Ignoring the additional regularizers, Eqs. (31)–(32) take alternating optimization steps for the complete-data criterion ∑ilog⁡pθ(xi,zi)\sum_i \log p_\theta(x_i,z_i), rather than the marginal log-likelihood ∑ilog⁡∫pθ(xi,z) dz\sum_i \log\int p_\theta(x_i,z)\,dz. What statistical target does the full regularized procedure define, and how do its stationary points relate to marginal-likelihood estimation? The paper does not adequately address this distinction or justify choosing this training recipe over variational/ELBO-based training with a matched decoder. My concern is not that every density estimator must be an MLE, but that the particular surrogate and its additional penalties lack sufficient justification and controlled empirical validation. Section 4 conditions on an already fitted model and therefore does not resolve this issue.

  3. (The experiments do not directly establish the accuracy of the proposed density computation.) The main comparisons evaluate different end-to-end pipelines and therefore combine model-fitting quality with integration accuracy. Appendix A.4 S6 partially addresses this through a fixed-model bridge-versus-IS ablation, which I appreciate. However, it still measures Spearman correlation with the data-generating density, rather than error relative to the fitted model’s density. These are different targets, and rank correlation cannot detect multiplicative normalization errors. A direct evaluation against reliable reference log-densities for fixed models, including Monte Carlo variability and computational cost, is needed to substantiate the central numerical claim.

  4. (Important experimental comparisons are confounded, and the synthetic evidence is limited.) In Appendix A.5, BayesNDE shares parameters across labels in a single conditional model, whereas the baselines fit separate unconditional models for each label. This introduces a parameter-sharing difference that can affect the reported conditional-density improvements. In anomaly detection, Table 4 additionally uses different latent dimensions for the closest comparator: Roundtrip uses 3, 5, and 5, whereas BayesNDE uses 9, 21, and 16. These comparisons do not isolate the proposed methodological changes. Furthermore, the high-dimensional synthetic benchmark is a product of independent one-dimensional mixtures. Its exponentially many modes do not by themselves demonstrate difficulty in modeling high-dimensional dependence; a factorized density baseline is missing, while the single-Gaussian conditionals used by the MADE baseline are structurally misspecified for this benchmark.

  5. (The computational tradeoff and posterior-sampling reliability are insufficiently established.) Each density query requires HMC, proposal fitting, and additional Monte Carlo evaluation. Yet the BANK runtime benchmark uses M=100M=100 posterior draws, while the main UCI evaluations use 1,600. The runtime budget is selected by rank agreement with each method’s own high-budget estimate, not by a common log-density accuracy requirement. Thus, the reported timing does not establish the cost of achieving the main results or an advantage over related estimators at matched accuracy. Moreover, Appendix A.4 S5 computes the quantity called MeffM_{\mathrm{eff}} after pooling and randomly permuting chain samples. The authors acknowledge that this is not a conventional Markov-chain ESS. With all chains initialized close to the encoder output, actual mixing diagnostics and pointwise Monte Carlo error assessments are particularly important but are not provided. Numerical convergence of the bridge iteration does not establish adequate posterior exploration.

  6. (The presentation obscures the scope and justification of the contribution.) The training description is fragmented across Sections 3.2 and 3.5 and Appendix A.4, making it unnecessarily difficult to reconstruct the procedure and distinguish inherited components from the proposed contribution. The emphasis on “Bayesian generative modeling” also requires more precise qualification: the network parameters are point estimates, and the posterior computation concerns latent variables. The manuscript should more clearly separate model fitting from density evaluation and explain which claims concern each stage. Reporting inconsistencies further hinder interpretation; for example, Table 1 reports test log-likelihoods but has a caption describing anomaly-detection precision@k.

Overall, I do not find the methodological increment or its practical advantages sufficiently established. The unresolved training rationale and the limitations of the empirical validation are substantive issues, rather than merely requests for additional exposition. I therefore recommend clear rejection.

References

[1] He Jia and Uros Seljak. Normalizing Constant Estimation with Gaussianized Bridge Sampling. AABI, PMLR 118:1–14, 2020.

[2] Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse. On the Quantitative Analysis of Decoder-Based Generative Models. ICLR, 2017. arXiv:1611.04273.

Questions for authors

Please see the weaknesses.

1 comment
AuthorsOpenReview ↗

response w2 (cont.)

To evaluate the choice empirically, we compare MAP and ELBO training [2] with the same deterministic five-layer width-256 Gaussian decoder, two-dimensional latent space, EGM initialization, data, and 200-epoch schedule. Each objective is run with and without the additional θ\theta-block penalties; results below average training seeds 0, 1, and 2 on the same 128 held-out points. Density evaluation uses quadrature of each fitted decoder, independently of its training encoder or latent codes. Let p∗p* denote the data-generating density. For these training controls, EtrueE_{\mathrm{true}} is the mean absolute difference between each fitted decoder's quadrature log-density and log⁡p∗\log p^*; NLL is the negative mean of that quadrature log-density. Table 1 summarizes the four training controls.

Table 1. Matched-decoder training controls: means over training seeds 0, 1, and 2 on the same 128 test points.

TrainingInvolute NLLInvolute EtrueE_{\mathrm{true}}GMM2 NLLGMM2 EtrueE_{\mathrm{true}}
MAP4.13690.26400.51950.2406
MAP + penalties4.16500.28540.46360.1305
ELBO4.07510.16210.47220.1307
ELBO + penalties4.12860.24300.48710.1660

All entries are in nats; smaller values are better. Density evaluation uses quadrature of each fitted decoder.

The penalty arms perform an additional correlation update and therefore use two θ\theta updates per minibatch, versus one otherwise. These controls favor ELBO on Involute and show a benefit from the added penalties for MAP on GMM2; they do not establish universal superiority of one training objective.

Reviewer_rSiYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BayesNDE, a novel approach to density estimation. The proposed method first trains a decoder p(x∣z)p(x \mid z), then uses Hamiltonian Monte Carlo (HMC) to sample from the posterior distribution p(z∣x)p(z \mid x). It subsequently constructs a proposal distribution in the latent space using a mixture of Student-t distributions and finally estimates the marginal likelihood through bridge sampling. Unlike normalizing flows, BayesNDE does not require invertible network architectures or Jacobian computations. Furthermore, unlike the ELBO used in VAEs, it enables direct estimation of the marginal likelihood. The theoretical analysis establishes the consistency of adaptive bridge estimation and characterizes the relationship between posterior–proposal overlap and estimation efficiency. Experiments on synthetic and real-world datasets, including MNIST, demonstrate that BayesNDE outperforms the baseline methods.

Critical strengths

The effectiveness of the proposed method is empirically demonstrated. Specifically, BayesNDE accurately estimates densities on synthetic datasets with various distributional structures and achieves strong performance on real-world datasets. The paper also provides theoretical guarantees for the proposed method.

Critical weaknesses

I believe the main advantages of the proposed method are that it can estimate the marginal likelihood directly, rather than relying on the ELBO as in VAEs, and that it does not require invertible network architectures or Jacobian computations, unlike flow-based models. However, I am not convinced of its practical advantages.

Although standard VAEs may not achieve particularly strong performance, they allow much faster inference than the proposed method. Moreover, by modifying the prior distribution or introducing hierarchical structures, as in diffusion models, they can capture complex real-world data distributions, such as high-resolution images. In other words, I believe that relying on the ELBO is not necessarily a major limitation for generative modeling. Furthermore, methods such as Importance Weighted Autoencoders (IWAE) [1] can also estimate the marginal likelihood using importance sampling.

On the other hand, flow-based models enable exact likelihood evaluation, unlike standard VAEs and GANs. When flow-based models were first proposed, I believe they achieved better generative performance than VAEs and GANs. However, constraints on network architectures and computational costs have become bottlenecks, and diffusion models are now more widely used.

To demonstrate the value of the proposed method, I believe experiments in more practical settings are necessary. First, I suggest comparing BayesNDE with VAE-based methods. At a minimum, IWAE [1] should be included as a baseline. I expect the proposed method would likely outperform IWAE, but an empirical comparison would be valuable. If possible, comparisons with other VAE variants would also be beneficial. In addition, comparisons with standard diffusion models or VAEs with improved prior distributions would help establish the value of the proposed method as a generative model.

Alternatively, if the primary focus is density estimation, I believe the paper should more clearly demonstrate its practical value. Table 2 reports anomaly detection performance, which I consider an important application of density estimation. However, the dedicated anomaly detection baselines are limited to OC-SVM and Isolation Forest, and no neural network-based anomaly detection methods are included. Comparisons with neural network-based anomaly detection methods, preferably including more recent approaches, would help demonstrate the practical advantages of the proposed method.

Minor comments: I believe Table 1 reports log-likelihood results, but its caption appears to describe anomaly detection performance. Is this a mistake?

[1] Burda, Yuri, Roger Grosse, and Ruslan Salakhutdinov. "Importance weighted autoencoders." arXiv preprint arXiv:1509.00519 (2015).

Questions for authors

Please see Critical Weaknesses.

Reviewer_vygoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper offers a fresh and well-motivated perspective on neural density estimation by introducing BayesNDE, a framework that turns latent-variable generative models ((p_\theta(x \mid z) = \mathcal{N}(x; \mu_\theta(z), \Sigma_\theta(z)))) into explicit pointwise density estimators without requiring invertible architectures or Jacobian determinant computations. After an Encoding Generative Modeling (EGM) warm start and alternating stochastic gradient updates of sample-specific latents ({z_i}{i=1}^N) and generator parameters (\theta), BayesNDE evaluates the density of a new observation (x) by sampling the unnormalized latent posterior (\pi_x(z) = p{\hat{\theta}}(x \mid z)p_Z(z)) via Hamiltonian Monte Carlo (HMC), fitting an observation-adaptive defensive mixture-of-Student-(t) proposal (\tilde{q}x(z)), and applying Meng and Wong's optimal bridge-sampling fixed-point iteration to estimate the normalizing constant (Z_x = p{\hat{\theta}}(x)). The theoretical results establish asymptotic consistency and variance bounds conditional on a fitted generator (\hat{\theta}). They are appropriately specialized to explain variance behavior under posterior–proposal overlap, and the empirical results on synthetic structure recovery, UCI log-likelihoods, ODDS anomaly detection, and MNIST classification are encouraging. However, the contribution is currently tempered by test-time MCMC computational cost, a random-permutation bug in the HMC sample-splitting implementation, confounded conditional baseline protocols, and missing comparisons against AIS-based marginal likelihood estimators and modern spline flows.

Critical strengths
  1. Estimating Zx=∫pθ^(x∣z)pZ(z) dzZ_x = \int p_{\hat{\theta}}(x\vert{}z)p_Z(z)\,dz via HMC and bridge sampling effectively mitigates the high weight variance of one-sided encoder-centered importance sampling in Roundtrip.
  2. Combining a KK-component Student-tt mixture (ν=3\nu=3) with a prior defensive component (ϵ=0.05\epsilon=0.05) bounds bridge integrands and guarantees finite asymptotic variance (Theorem 2).
  3. BayesNDE accurately captures the 2D Involute manifold and up to 3253^{25} modes in the Independent GMM benchmark while outperforming plain importance sampling across HMC budgets.
Critical weaknesses
  1. Pooling and randomly permuting HMC draws before splitting into Dfit\mathcal{D}_{\text{fit}} and Dbridge\mathcal{D}_{\text{bridge}} and computing MeffM_{\text{eff}} (Appendix A.4 S5) places adjacent Markov chain steps into opposite splits, violating conditional independence (Theorems 1–2) and destroying the temporal autocorrelation required for valid ESS calculation.
  2. Evaluating synthetic benchmarks solely via Spearman rank correlation (Sections 5.2, 5.5) only tests relative ordering and is blind to multiplicative bias in the normalizing constant Z^x\hat{Z}_x.
  3. Training updates deterministic point estimates ziz_i via MAP gradient ascent (Eq. 31) rather than integrating over pθ(zi∣xi)p_\theta(z_i \mid x_i), requiring an ad-hoc penalty Llogvar\mathcal{L}_{\text{logvar}} tethering σj2(z)\sigma_j^2(z) to 0.010.01 (Eq. 30) to prevent manifold variance collapse.
  4. Conditional baselines are trained on CC separate class subsets (1/C1/C of the data, leaving ∼171\sim 171 samples/class on Vehicle) whereas BayesNDE shares weights across all classes; ODDS sets Dim(z)=Dim(x)\text{Dim}(z)=\text{Dim}(x) for BayesNDE but restricts Roundtrip to d∈{3,5,5}d \in \{3,5,5\}; and key comparators (AIS, IWAE, Neural Spline Flows, MDN, classical GMM/KDE) are omitted.
  5. Limited discussion of alternative normalizing-constant estimators used in generative modeling, such as AIS (used for evaluating decoder-based models; e.g., Wu et al., 2017; Ding & Freedman, 2019) and recent improvements (e.g., Monte Carlo Diffusion for better AIS proposals).
Questions for authors
  1. If you split Dfit\mathcal{D}_{\text{fit}} and Dbridge\mathcal{D}_{\text{bridge}} by independent HMC chains (without random permutation) and compute unshuffled multi-chain R^\hat{R} and ESS (Vehtari et al., 2021), what are the resulting R^\hat{R}/ESS values, and how do the UCI log-likelihoods in Table 1 change?
  2. What are the Mean Absolute Error (MAE) of log⁡p^θ^(x)−log⁡p∗(x)\log \hat{p}_{\hat{\theta}}(x) - \log p^*(x) and test NLL on the Involute and Independent GMM benchmarks, and how does bridge sampling compare in accuracy and wall-clock time against Annealed Importance Sampling (AIS; Wu et al., 2017) on the same fitted generator θ^\hat{\theta}?
  3. How do MAF, RealNVP, and CFM perform on Pendigits10, Vehicle, and EEGEye when conditioned on one-hot yy on the full training set, and how does Roundtrip perform on ODDS (with standard deviations and AUROC/AUPRC) when given Dim(z)=Dim(x)\text{Dim}(z) = \text{Dim}(x)?
  4. How sensitive are Table 1 log-likelihoods to the target variance constant 0.010.01 in Llogvar\mathcal{L}_{\text{logvar}} (Eq. 30) and proposal hyperparameters (K,ν,ϵK, \nu, \epsilon), and what exact log-density formula (including any logit-normal Jacobian and dequantization) was evaluated on MNIST?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.