- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors develop DensityFM, a general-purpose amortized neural density estimator across datasets. This paper is mainly motivated by two lines of work. First is the prior fitted net model that is pretrained on synthetic data (e.g., TabPFN). The second line of work is autoregressive neural density estimators (can't generalize to new distributions without refitting). Conceptually, this is an autoregressive neural density estimator that conditions on the training dataset as the "context". Then it is a conditional autoregressive neural density estimator that can generalize to new density without refitting. Athough the idea of amortized inference by pretraining on synthetic data is not new. The incorporation of that type of idea in density estimation is new to me. The reported empirical results showed that the proposed method is competitive, but many neural density estimator baselines were missed.
Critical strengths
- This is a useful architecture for zero-shot density estimation without any model refitting or fine-tuning when it comes to a new dataset.
- Table 3 gives the highest mean likelihood across all native datasets.
- The proposed method has a substantial advantage in model inference.
- The conditional results in reconstruction and SBI are also competitive with other density estimators
Critical weaknesses
- Vetter et al. (NeurIPS 2025) already demonstrate frozen, context-based unconditional density estimation using autoregressive TabPFN (see appendices B.4/D.9). DensityFM’s “first” claim of using in-context zero-shot density estimation may not stand. A direct comparison with autoregressive TabPFN for unconditional density estimation and NPE-PFN for SBI is then needed.
- benefit of marginal-copula separation is not clearly demonstrated.What is the advantage of doing such affine preprocessing? Even in Appendix C, it reports that exploratory Gaussianization did not outperform whitening. What incremental benefit does KDE-based Gaussianization provide over affine whitening?
- The authors also provide optional Gaussianization path that will clip the CDF without a corresponding Jacobian correction. This will lead to unnormalized density (integral does not equal to 1).
- Simple averaging of Table 12 gives DensityFM/NSF gaps of approximately 0.271/0.146 at d=32 and 0.260/0.175 at d=64, inconsistent with the claimed lowest mean error at every dimension.
Questions for authors
- Many neural density estimators were missed and not compared. The most relevant one is autoregressive TabPFN for unconditional density estimation and NPE-PFN for SBI (see Vetter et al. NeurIPS 2025). Other neural density estimators, such as MADE, Resflow Roundtrip, CFMs are also missed. 2.The five real datasets were exactly used by MAF. But metric and setting (e.g., dimension of the dataset) is very different from the original MAF paper. The test log likelihood is divided by dimension, but even though, the metrics are very different from MAF paper. Please clarify
- In SBI experiments, there are many methods for SBI, the authors should at least compare to the widely used SNL and NRE methods.
- The supplementary code provides no checkpoint for the pretrained model, which makes it impossible to reproduce any results in the paper. Please provide a checkpoint file.
- Which reported results actually use KDE-based Gaussianization rather than affine whitening? A more detailed ablation study is needed.
- As I previously mentioned, simple averaging of Table 12 gives DensityFM/NSF gaps of approximately 0.271/0.146 at d=32 and 0.260/0.175 at d=64, inconsistent with the claimed lowest mean error at every dimension from Figure 4a’s claim. Please clarify.
- Does “causal” refer solely to masked attention, without assuming a causal ordering of variables? Should be careful to use "causal". How sensitive are density and conditional-inference results to feature permutations?
4 comments
Questions
Q1(a). Comparisons with additional neural density estimators
Thank you for suggesting these additional baselines. We provide a comprehensive comparison against NPE-PFN, MADE, Residual Flows, Roundtrip and CFM below.
Table 4. Comparison of DensityFM (updated checkpoint) against additional neural density estimation baselines on synthetic benchmarks (averaged across 22 distribution families) and real-world benchmarks (averaged across four 64-dimensional datasets).
| Method (64D) | Synthetic gap/dim ↓ | Time (s) ↓ | Real NLL/dim ↓ | Time (s) ↓ |
|---|---|---|---|---|
| DensityFM (1–128D) | 0.0509 | 0.261 | 1.0097 | 1.543 |
| NPE-PFN (autoregressive, TabPFN3) | 0.0976 | 16.117 | 0.9399 | 193.897 |
| MADE | 0.1529 | 13.094 | 1.1267 | 107.769 |
| Residual Flows | 0.2681 | 546.612 | 1.3074 | 1077.426 |
| Roundtrip (IS) | 0.9693 | 293.889 | 1.7000 | 818.030 |
| CFM | 0.1755 | 233.194 | 1.2746 | 3949.540 |
Q1(b). Datasets, dimensions and likelihoods relative to MAF
Thank you for pointing out these differences. The discrepancies with the original MAF paper arise from two factors:
-
Dataset setup difference, as explained below.
Dataset MAF dimension Our dimension Explanation POWER 6 5 We use CCPP instead of household electricity consumption GAS 8 8 Same setup BSDS300 63 63 Same setup HEPMASS 21 22 MAF removed some features MiniBooNE 43 50 MAF removed some features -
Training set size. We use 1,024 training samples under a controlled evaluation protocol, whereas MAF uses over 10,000 samples, contributing to differences in the reported NLL values.
We will clarify these differences in the revised manuscript.
Q2. Comparisons with NPE-PFN, SNL and NRE on SBI
New comparison to SBI baselines. Thank you for suggesting these SBI baselines. We include a comparison with NPE-PFN (TabPFN3), SNL and NRE on five SBI tasks in Table 5. Currently, our method’s accuracy is still below SNL (probably due to no use of adaptive inference), but is comparable to NRE while being substantially faster.
| Method | Mean C2ST | Time (s) |
|---|---|---|
| DensityFM only | 0.6005 | 1.126 |
| MAF only | 0.6322 | 26.774 |
| NPE-PFN only | 0.5379 | 4.577 |
| SNL (w/ MAF) | 0.5494 | 293.011 |
| NRE | 0.6178 | 173.858 |
Q3. Pretrained checkpoints and reproducibility
The pretrained checkpoint files exceed the supplementary upload size limit. We commit to releasing the pretrained checkpoints on Hugging Face upon acceptance.
Q4. Preprocessing used and ablation study
Please see our response to W2 for the marginal–copula ablation.
Q5. Inconsistency between Figure 4a and Table 12
Please see our response to W4 above for the explanation of the discrepancy and the updated results.
Q6. Clarifying the meaning of “causal” and the sensitivity to feature permutations
-
Meaning of “causal”. It refers only to the decoder’s autoregressive attention mask, as in language models.
-
Feature permutation sensitivity. We examine feature-order sensitivity by permuting the features of the data. The results below show that our method is not very sensitive to feature permutations.
Distribution (64D) Original gap/dim (average) ↓ Permuted gap/dim (average) ↓ Gaussian (correlated) 0.01526 0.01508 Student-t (ν = 5) 0.02163 0.02161 Laplace 0.01490 0.01491
I appreciate the efforts and time from the authors for the additional experiments, especially implementing the most important missed baseline NPE-PFN (Vetter et al., NeurIPS 2025). The results basically reaffirmed my concern.
Without NPE-PFN, this would be a very innovative work for sure. But the fundemental novelty of pretrained foundation model + density esitmation was established in NPE-PFN. Now the new components, such as marginal-copula decomposition and shared context encoding, are relatively incremental.
More importantly, the new experiments did not demonstrate a clear empirical advantage over NPE-PFN. DensityFM is worse on real-world NLL/dim (1.0097 vs. 0.9399) and performs worse on most of the SBI tasks.
Considering both novelty and empirical advantage, I'm sorry I can not be more positive, and will keep my current rating.
Many thanks for your thoughtful follow-up! Below, we continue our discussion regarding DensityFM vs. NPE-PFN, not to challenge your evaluation, but to share perspectives.
1. DensityFM's focus is on the accuracy–efficiency trade-off
While slightly behind NPE-PFN on real-data, DensityFM achieves competitive accuracy with ~100× faster estimation. A few fine-tuning steps quickly eliminate this accuracy gap while retaining substantial speed advantages (3x faster).
| Method | Synthetic gap ↓ | Time (s) ↓ | Real NLL/dim ↓ | Time (s) ↓ |
|---|---|---|---|---|
| DensityFM | 0.05457 | 0.160 | 1.0097 | 1.563 |
| DensityFM (w/ fine-tuning) | 0.05206 | 5.372 | 0.9263 | 34.427 |
| NPE-PFN | 0.0986 | 16.018 | 0.9399 | 181.064 |
2. Architectural distinction from NPE-PFN
NPE-PFN does not natively support unconditional density estimation: for 64D data, it requires 64 separate conditional regression tasks using TabPFN, whereas DensityFM estimates all 64 conditional densities in a single forward pass via a masked autoencoder-like architecture. Together with our shared-context design, this enables a better accuracy–efficiency trade-off.
Summary. DensityFM is designed to achieve a favorable accuracy–efficiency trade-off through its architecture tailored for native density estimation, rather than to be universally superior in accuracy. The two approaches are complementary: NPE-PFN offers stronger accuracy, while DensityFM achieves comparable accuracy with substantially greater efficiency.
Weaknesses
We sincerely thank Reviewer ZALg for recognizing the practical value of DensityFM’s zero-shot capability, inference efficiency, and competitive performance. Below, we provide additional comparisons with the suggested baselines, ablate the benefits of marginal–copula separation, resolve inconsistencies, and clarify questions regarding evaluation and technical details.
W1. DensityFM’s “first” claim of using in-context zero-shot density estimation may not stand given NPE-PFN.
Sharpening the claim. We thank the reviewer for highlighting this work. We have refined our claim from “first in-context zero-shot density estimation” to “first pretrained architecture that natively supports in-context density estimation.”
Comparison with NPE-PFN. While NPE-PFN can be adapted for joint density estimation via autoregression, DensityFM performs it natively in a single forward pass, offering substantially faster inference; see the comparison below.
Table 1. Comparison of DensityFM (new checkpoint) and NPE-PFN (using TabPFN3) on 64D data, averaged across 22 synthetic families and four real-world datasets using full training set from the PCA64 panel of Table 2.
W2. Benefit of marginal–copula separation
We conducted an additional ablation study to isolate the benefit of marginal Gaussianization over affine whitening only. The results demonstrate substantial improvements of marginal Gaussianization.
Table 2. Ablation study evaluating the effectiveness of marginal Gaussianization compared with affine whitening.
Results are averaged over 4 real-world datasets and 22 synthetic distributions at 64D.
W3. CDF clipping, Jacobians and normalization
We thank the reviewer for pointing out this issue. We also notice this issue and in our newest checkpoint, we have replaced hard CDF clipping with stable log-CDF transformation, with the corresponding correction terms included. This ensures that the density is now correctly normalized.
W4. Inconsistency between Figure 4a and Table 12
Thank you for pointing out this issue. We have updated the results and corresponding claims to ensure consistency, using our latest model checkpoint developed after submission. The updated results confirm that DensityFM achieves lower mean errors than NSF at both dimensions:
Table 3. Updated comparison of DensityFM and NSF on the synthetic datasets from original Table 12 (now Table 18)