acceptodds
Under review as a conference paper at ICLR 2027

DensityFM: Foundation-Style Pretraining for Density Estimation

Abstract

Estimating probability densities is at the heart of many data science and machine learning pipelines. However, existing estimators face a fundamental accuracy–efficiency trade-off: classic non-parametric methods are fast but struggle in high dimensions, while more expressive flow-based methods require costly training and tuning for each new dataset. In this work, we present DensityFM, a pretrained, foundation model-style density estimator achieving the best of both worlds. Pretrained once on large-scale and diverse distributions, DensityFM performs zero-shot density estimation on unseen datasets with frozen parameters at test time. Experiments across diverse synthetic distributions and multiple 64-dimensional real-world datasets demonstrate that DensityFM processes each dataset in 0.5 seconds, delivering a speedup up to over the fastest neural baseline, while achieving comparable or even superior accuracy, establishing a new foundation for scalable data science and knowledge discovery.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Density estimation, In-context learning, Tabular foundation models, Copula modeling, Meta-learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZALgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors develop DensityFM, a general-purpose amortized neural density estimator across datasets. This paper is mainly motivated by two lines of work. First is the prior fitted net model that is pretrained on synthetic data (e.g., TabPFN). The second line of work is autoregressive neural density estimators (can't generalize to new distributions without refitting). Conceptually, this is an autoregressive neural density estimator that conditions on the training dataset as the "context". Then it is a conditional autoregressive neural density estimator that can generalize to new density without refitting. Athough the idea of amortized inference by pretraining on synthetic data is not new. The incorporation of that type of idea in density estimation is new to me. The reported empirical results showed that the proposed method is competitive, but many neural density estimator baselines were missed.

Critical strengths
  1. This is a useful architecture for zero-shot density estimation without any model refitting or fine-tuning when it comes to a new dataset.
  2. Table 3 gives the highest mean likelihood across all native datasets.
  3. The proposed method has a substantial advantage in model inference.
  4. The conditional results in reconstruction and SBI are also competitive with other density estimators
Critical weaknesses
  1. Vetter et al. (NeurIPS 2025) already demonstrate frozen, context-based unconditional density estimation using autoregressive TabPFN (see appendices B.4/D.9). DensityFM’s “first” claim of using in-context zero-shot density estimation may not stand. A direct comparison with autoregressive TabPFN for unconditional density estimation and NPE-PFN for SBI is then needed.
  2. benefit of marginal-copula separation is not clearly demonstrated.What is the advantage of doing such affine preprocessing? Even in Appendix C, it reports that exploratory Gaussianization did not outperform whitening. What incremental benefit does KDE-based Gaussianization provide over affine whitening?
  3. The authors also provide optional Gaussianization path that will clip the CDF without a corresponding Jacobian correction. This will lead to unnormalized density (integral does not equal to 1).
  4. Simple averaging of Table 12 gives DensityFM/NSF gaps of approximately 0.271/0.146 at d=32 and 0.260/0.175 at d=64, inconsistent with the claimed lowest mean error at every dimension.
Questions for authors
  1. Many neural density estimators were missed and not compared. The most relevant one is autoregressive TabPFN for unconditional density estimation and NPE-PFN for SBI (see Vetter et al. NeurIPS 2025). Other neural density estimators, such as MADE, Resflow Roundtrip, CFMs are also missed. 2.The five real datasets were exactly used by MAF. But metric and setting (e.g., dimension of the dataset) is very different from the original MAF paper. The test log likelihood is divided by dimension, but even though, the metrics are very different from MAF paper. Please clarify
  2. In SBI experiments, there are many methods for SBI, the authors should at least compare to the widely used SNL and NRE methods.
  3. The supplementary code provides no checkpoint for the pretrained model, which makes it impossible to reproduce any results in the paper. Please provide a checkpoint file.
  4. Which reported results actually use KDE-based Gaussianization rather than affine whitening? A more detailed ablation study is needed.
  5. As I previously mentioned, simple averaging of Table 12 gives DensityFM/NSF gaps of approximately 0.271/0.146 at d=32 and 0.260/0.175 at d=64, inconsistent with the claimed lowest mean error at every dimension from Figure 4a’s claim. Please clarify.
  6. Does “causal” refer solely to masked attention, without assuming a causal ordering of variables? Should be careful to use "causal". How sensitive are density and conditional-inference results to feature permutations?
4 comments
AuthorsOpenReview ↗

Weaknesses

We sincerely thank Reviewer ZALg for recognizing the practical value of DensityFM’s zero-shot capability, inference efficiency, and competitive performance. Below, we provide additional comparisons with the suggested baselines, ablate the benefits of marginal–copula separation, resolve inconsistencies, and clarify questions regarding evaluation and technical details.

W1. DensityFM’s “first” claim of using in-context zero-shot density estimation may not stand given NPE-PFN.

  1. Sharpening the claim. We thank the reviewer for highlighting this work. We have refined our claim from “first in-context zero-shot density estimation” to “first pretrained architecture that natively supports in-context density estimation.”

  2. Comparison with NPE-PFN. While NPE-PFN can be adapted for joint density estimation via autoregression, DensityFM performs it natively in a single forward pass, offering substantially faster inference; see the comparison below.

    Table 1. Comparison of DensityFM (new checkpoint) and NPE-PFN (using TabPFN3) on 64D data, averaged across 22 synthetic families and four real-world datasets using full training set from the PCA64 panel of Table 2.

    Synthetic gap/dim ↓Time (s) ↓Real NLL/dim ↓Time (s) ↓
    DensityFM0.054570.2611.00971.543
    NPE-PFN (autoregressive, TabPFN3)0.0986416.1170.9399193.897

W2. Benefit of marginal–copula separation

We conducted an additional ablation study to isolate the benefit of marginal Gaussianization over affine whitening only. The results demonstrate substantial improvements of marginal Gaussianization.

Table 2. Ablation study evaluating the effectiveness of marginal Gaussianization compared with affine whitening.

PreprocessingReal NLL/dim ↓Synthetic gap/dim ↓
Marginal Gaussianization + whitening1.046890.04826
Whitening only1.144530.08963
No marginal Gaussianization / whitening1.160090.10376

Results are averaged over 4 real-world datasets and 22 synthetic distributions at 64D.

W3. CDF clipping, Jacobians and normalization

We thank the reviewer for pointing out this issue. We also notice this issue and in our newest checkpoint, we have replaced hard CDF clipping with stable log-CDF transformation, with the corresponding correction terms included. This ensures that the density is now correctly normalized.

W4. Inconsistency between Figure 4a and Table 12

Thank you for pointing out this issue. We have updated the results and corresponding claims to ensure consistency, using our latest model checkpoint developed after submission. The updated results confirm that DensityFM achieves lower mean errors than NSF at both dimensions:

Table 3. Updated comparison of DensityFM and NSF on the synthetic datasets from original Table 12 (now Table 18)

DimensionDensityFM (new checkpoint) gap ↓NSF gap ↓
320.030250.146
640.054570.175

Q1(a). Comparisons with additional neural density estimators

Thank you for suggesting these additional baselines. We provide a comprehensive comparison against NPE-PFN, MADE, Residual Flows, Roundtrip and CFM below.

Table 4. Comparison of DensityFM (updated checkpoint) against additional neural density estimation baselines on synthetic benchmarks (averaged across 22 distribution families) and real-world benchmarks (averaged across four 64-dimensional datasets).

Method (64D)Synthetic gap/dim ↓Time (s) ↓Real NLL/dim ↓Time (s) ↓
DensityFM (1–128D)0.05090.2611.00971.543
NPE-PFN (autoregressive, TabPFN3)0.097616.1170.9399193.897
MADE0.152913.0941.1267107.769
Residual Flows0.2681546.6121.30741077.426
Roundtrip (IS)0.9693293.8891.7000818.030
CFM0.1755233.1941.27463949.540

Q1(b). Datasets, dimensions and likelihoods relative to MAF

Thank you for pointing out these differences. The discrepancies with the original MAF paper arise from two factors:

  1. Dataset setup difference, as explained below.

    DatasetMAF dimensionOur dimensionExplanation
    POWER65We use CCPP instead of household electricity consumption
    GAS88Same setup
    BSDS3006363Same setup
    HEPMASS2122MAF removed some features
    MiniBooNE4350MAF removed some features
  2. Training set size. We use 1,024 training samples under a controlled evaluation protocol, whereas MAF uses over 10,000 samples, contributing to differences in the reported NLL values.

We will clarify these differences in the revised manuscript.

Q2. Comparisons with NPE-PFN, SNL and NRE on SBI

New comparison to SBI baselines. Thank you for suggesting these SBI baselines. We include a comparison with NPE-PFN (TabPFN3), SNL and NRE on five SBI tasks in Table 5. Currently, our method’s accuracy is still below SNL (probably due to no use of adaptive inference), but is comparable to NRE while being substantially faster.

MethodMean C2STTime (s)
DensityFM only0.60051.126
MAF only0.632226.774
NPE-PFN only0.53794.577
SNL (w/ MAF)0.5494293.011
NRE0.6178173.858

Q3. Pretrained checkpoints and reproducibility

The pretrained checkpoint files exceed the supplementary upload size limit. We commit to releasing the pretrained checkpoints on Hugging Face upon acceptance.

Q4. Preprocessing used and ablation study

Please see our response to W2 for the marginal–copula ablation.

Q5. Inconsistency between Figure 4a and Table 12

Please see our response to W4 above for the explanation of the discrepancy and the updated results.

Q6. Clarifying the meaning of “causal” and the sensitivity to feature permutations

  1. Meaning of “causal”. It refers only to the decoder’s autoregressive attention mask, as in language models.

  2. Feature permutation sensitivity. We examine feature-order sensitivity by permuting the features of the data. The results below show that our method is not very sensitive to feature permutations.

    Distribution (64D)Original gap/dim (average) ↓Permuted gap/dim (average) ↓
    Gaussian (correlated)0.015260.01508
    Student-t (ν = 5)0.021630.02161
    Laplace0.014900.01491

I appreciate the efforts and time from the authors for the additional experiments, especially implementing the most important missed baseline NPE-PFN (Vetter et al., NeurIPS 2025). The results basically reaffirmed my concern.

Without NPE-PFN, this would be a very innovative work for sure. But the fundemental novelty of pretrained foundation model + density esitmation was established in NPE-PFN. Now the new components, such as marginal-copula decomposition and shared context encoding, are relatively incremental.

More importantly, the new experiments did not demonstrate a clear empirical advantage over NPE-PFN. DensityFM is worse on real-world NLL/dim (1.0097 vs. 0.9399) and performs worse on most of the SBI tasks.

Considering both novelty and empirical advantage, I'm sorry I can not be more positive, and will keep my current rating.

Many thanks for your thoughtful follow-up! Below, we continue our discussion regarding DensityFM vs. NPE-PFN, not to challenge your evaluation, but to share perspectives.

1. DensityFM's focus is on the accuracy–efficiency trade-off

While slightly behind NPE-PFN on real-data, DensityFM achieves competitive accuracy with ~100× faster estimation. A few fine-tuning steps quickly eliminate this accuracy gap while retaining substantial speed advantages (3x faster).

MethodSynthetic gap ↓Time (s) ↓Real NLL/dim ↓Time (s) ↓
DensityFM0.054570.1601.00971.563
DensityFM (w/ fine-tuning)0.052065.3720.926334.427
NPE-PFN0.098616.0180.9399181.064

2. Architectural distinction from NPE-PFN

NPE-PFN does not natively support unconditional density estimation: for 64D data, it requires 64 separate conditional regression tasks using TabPFN, whereas DensityFM estimates all 64 conditional densities in a single forward pass via a masked autoencoder-like architecture. Together with our shared-context design, this enables a better accuracy–efficiency trade-off.

 

Summary. DensityFM is designed to achieve a favorable accuracy–efficiency trade-off through its architecture tailored for native density estimation, rather than to be universally superior in accuracy. The two approaches are complementary: NPE-PFN offers stronger accuracy, while DensityFM achieves comparable accuracy with substantially greater efficiency.

Reviewer_5KqxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DensityFM, a pretrained neural framework for zero-shot multivariate density estimation. The main idea is to separate marginal density estimation from dependence modeling: univariate marginals are estimated independently, while a pretrained autoregressive model learns the dependence structure in copula-transformed coordinates. This allows the model to estimate densities for previously unseen datasets without target-specific neural network optimization. The framework is pretrained on a diverse collection of synthetically generated distributions and supports dimensionalities up to 64. The authors evaluate DensityFM on synthetic benchmarks, real-world datasets, and downstream conditional inference tasks, comparing its performance and efficiency with classical density estimators and other neural approaches. The reported results indicate competitive or superior density estimation accuracy in diverse settings, with substantially lower inference costs than neural methods which require dataset-specific training.

Critical strengths

The main strength of DensityFM is its ability to perform zero-shot neural density estimation without target-specific optimization. The combination of marginal estimation and pretrained dependence modeling provides an interesting approach which reduces the computational cost when working on previously unseen datasets. The experimental results, both in accuracy and temporally, are promising, not only on synthetic and real-world benchmarks, but also on downstream tasks.

Critical weaknesses

The proposed framework combines established components, such as marginal density estimation, copula-based factorization, and autoregressive density modeling. While their integration may be practically valuable, the manuscript does not convincingly demonstrate which components are responsible for the reported improvements. A closely related concern is the experimental comparison with existing pretrained density estimators. Although the authors discuss DiScoFormer and NPE-PFN in the related work, neither is included in the empirical evaluation. In my opinions, the stated architectural differences do not fully justify this omission. Finally, the evidence for generalization remains somewhat limited by the relationship between the synthetic pretraining distribution and the evaluation families. Although the real-world results are promising, performance is less consistently superior on the higher-dimensional benchmarks and some downstream tasks, making it difficult to assess how broadly the claimed advantages extend beyond the distributions seen in training.

Questions for authors

Question 1. Could the authors clarify whether the pretrained autoregressive estimator satisfies the uniform marginal constraints required of a copula density in practice? While Appendix A establishes that the architecture can approximate a target copula arbitrarily well, it does not guarantee that the pretrained model preserves uniform marginals on unseen datasets. Question 2. Could the authors provide an ablation study to clarify the contribution of the copula–marginal decomposition? How does DensityFM compare with a similarly pretrained autoregressive model that estimates the joint density directly? Question 3. Could the authors provide an empirical comparison with existing pretrained density estimators, i.e., DiScoFormer and NPE-PFN? Although the manuscript discusses their architectural differences, a feasible comparison on the dimensionalities supported by the competing methods would help clarify the practical advantages of DensityFM over alternative zero-shot approaches. Question 4. Since several distribution families appear in both pretraining and evaluation settings, could the authors investigate generalization to a family explicitly excluded from pretraining? Question 5. Could the authors investigate the sensitivity of DensityFM to the number of context observations, particularly in higher dimensions? While the appendix reports inference times for different context sizes, also investigating the corresponding effects on estimation accuracy would be meaningful. Minor comment: the authors could consider moving the Related Work section before the Problem Formulation section, as this would provide a more natural progression and coherence to the manuscript.

3 comments

We appreciate Reviewer 5Kqx’s thoughtful review and recognition of DensityFM’s generalization capability and its design separating marginal estimation from dependence modeling. Our responses below clarify the formulation and provide the requested ablations, comparisons with pretrained estimators, and analyses of generalization and context-size sensitivity.

W1/Q2. Which components contribute to improvement / Ablation of marginal–copula separation

Thank you for this suggestion. We conducted an ablation study to assess the contribution of marginal–copula separation.

Real NLL/dim ↓Synthetic gap/dim ↓
Marginal–copula separation1.046890.04826
Direct joint modeling1.160090.10376

Results are averaged over four real-world datasets and 22 synthetic distributions at 64D, using the same backbone and training budget.

W2/Q3. Comparison with NPE-PFN and DiScoFormer

  1. NPE-PFN (TabPFN3). We provide an additional comparison with NPE-PFN below. DensityFM shares a context encoding across dimensions, whereas autoregressive NPE-PFN estimates one conditional density at a time. Further analysis is provided in Section 2.2 (Neural Copula Estimation) of the revised paper.

    MethodSynthetic gap ↓Time (s) ↓Real NLL/dim ↓Time (s) ↓
    DensityFM0.054570.1601.00971.563
    NPE-PFN (TabPFN3)0.098616.0180.9399181.064
  2. DiScoFormer. We could not find publicly released 100D checkpoints, so we trained a 100D model using the authors’ code and compared it with our frozen model across five distribution families.

    MethodMean joint log-density MSE ↓Time (s) ↓
    DensityFM38.92480.875
    DiScoFormer (reproduction)3440.85250.008

W3/Q4. Generalization beyond the pretraining distributions

Thank you for raising this question. We assess generalization from two perspectives:

  1. Real-world data: All evaluated real-world datasets are out-of-distribution relative to pretraining. DensityFM achieves competitive performance, demonstrating its ability to generalize to unseen distributions.

  2. Unseen synthetic distributions: We additionally evaluate on three distributions excluded from pretraining, using NSF as a baseline. The results further demonstrate DensityFM's generalization ability.

    Method (64D)Inverse GaussianBurr XIIGumbel
    DensityFM gap ↓0.55040.27920.1066
    NSF gap ↓0.67030.38050.1880

Q1. Does the pretrained estimator preserve uniform copula marginals on unseen datasets?

  1. Our architecture does not explicitly enforce uniform copula marginals. However, this does not appear to be a significant issue in practice, as evidenced by our strong empirical results.

  2. The primary motivation for pretraining in copula space is to simplify density modeling and pretraining.

Q5. How does context size affect estimation accuracy and latency, especially in higher dimensions?

Thank you for the suggestion. We vary context size on three 64D real-world datasets with fixed query sets, evaluated on a single H200 GPU.

  1. Accuracy: NLL consistently improves as N increases, with N ≥ 1,024 typically yielding good enough results.

  2. Latency: Remains below 5 seconds even with up to 60000 context points.

Context size N1285121,0244,096Full*
Mean NLL/dim ↓1.48261.26251.19331.08980.9865
Mean latency (s) ↓0.2460.2550.2730.3921.857

*Full context sizes are 60,000 (MNIST), 7,352 (HAR) and 6,238 (ISOLET).

Thank you very much for the clarifications. I'm satisfied with the additional information provided so I will update the score accordingly.

We are grateful for your positive assessment. Thank you once again for the insightful comments, which have significantly improved this work.

Reviewer_e2ZYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper produces a foundation model pretrained on many diverse distributions (synthetic data), that can do zero-shot density estimation on unseen datasets. It is faster at computing one particular density than traditional neural density estimators while achieving comparable estimation accuracy. The method copula-transforms observations, processes them with a Transformer encoder, and then applies a autoregressive spline-based decoder to output the joint density in transformed coordinates. The authors pretrain many such models for data in different dimensions.

Evaluation covers 9 real datasets up to 64 dimensions (held-out likelihood), likelihoods on various synthetic data with known distributions, and downstream applications such as conditional reconstruction (estimating masked part of input), simulation-based inference (posterior distribution quality measured against a known reference).

Critical strengths

Interesting approach to pretrain a foundation model for density estimation, which can save compute costs if one were doing a vast number of density estimation tasks.

Done with a reasonable architecture that greatly reduces costs of density estimation on new data vs. traditional neural density estimators.

Critical weaknesses

DensityFM incurs a big one-time pretraining cost, but requires no training on new datasets unlike traditional density estimators. So it provides computational benefits if that pretraining cost gets amortized across many subsequent density-estimation tasks. I am not sure this is particularly practical, as I do not think one would have a massive volume of density estimation tasks / datasets at test-time.

The model only works for data up to 64 dimensions.

Also the authors need to separately pretrain many models for different dimensionalities, which greatly increases the computational cost.

The authors should compare their single 64-D model checkpoint (with appropriate padding) against dimension-specific model checkpoints across all evaluated dataset dimensions, reporting both accuracy and inference cost.

The authors write that DiScoFormer only supports 10 dimensions, but Table 5 in DiScoFormer paper has d = 100. The DensityFM authors provide no empirical comparison against DiScoFormer.

The authors also provide no empirical comparison against NPE-PFN.

There are not very clear ablation experiments establishing the value of each of the components/steps of DensityFM.

The synthetic data evaluations should test for out-of-distribution generalization by testing datasets that look more different than those encountered in pretraining.

Missing several related works including:

TraDE: Transformers for Density Estimation (Fakoor et al)

Data-driven deep density estimation (Puchert et al)

Copula & Marginal Flows: Disentangling the Marginal from its Joint (Wiese et al)

Copula Flows for Synthetic Data Generation (Kamthe et al)

Copula Density Neural Estimation (Letizia et al)

Questions for authors

Why did the authors chose to report results on the datasets listed in Table 3? Why these particular datasets and not others? How different can we expect these datasets are vs. the pretraining data?

How do you arrive at the 156x speedup figure? How do you think about amortizing the costs pretraining against density estimation on a new dataset? In what applications would density estimation on new datasets be the bigger compute cost?

3 comments
AuthorsOpenReview ↗

Weakness (1/2)

We sincerely thank Reviewer e2ZY for recognizing the potential of pretrained foundation models for density estimation and their lower estimation cost. Below, we address concerns about pretraining cost, demonstrate scalability to 128D with a new checkpoint, provide the requested comparisons with NPE-PFN, and present further analyses of OOD generalization.

W1. Due to the high pretraining cost, DensityFM is only beneficial when there is massive volume of density estimation tasks at test-time.

We clarify that the pretraining cost is fully covered by the authors, that DensityFM is designed as a publicly available foundation model. Once pretrained by the authors, it will be shared through public checkpoints (upon acceptance), analogous to LLMs such as Qwen. Users can directly enjoy fast, in-context density estimation without any training.

W2. DensityFM only supports data up to 64D

Since submission, we have improved DensityFM to support 1–128 dimensions with a single checkpoint, enabled by improvements in architecture and pretraining. We have updated the experimental results accordingly; please see the updated manuscript.

W3. Ablation study: comparing dimension-specific checkpoints and a single 64D unified checkpoint

We have provided this comparison in Appendix A, Table 5.

W4. The authors write that DiScoFormer only supports 10 dimensions, but Table 5 in DiScoFormer paper has d = 100. No empirical comparison against DiScoFormer.

Thank you for pointing this out. While DiScoFormer evaluates 100D two-component GMMs, the corresponding checkpoint is not publicly available, making direct comparison difficult. We therefore reproduced a 100D DiScoFormer model using its official code and provide a empirical comparison below. The results demonstrate the clear advantages of DensityFM.

MethodMean joint log-density MSE* ↓Time (s) ↓
DensityFM38.92480.875
DiScoFormer (reproduction)3440.85250.008
NSF (reference)65.377041.726

*Results are averaged across five 100D synthetic distribution families.

AuthorsOpenReview ↗

Weakness (2/2)

W5. Missing empirical comparison with NPE-PFN

We have included a new empirical comparison with NPE-PFN below and in the revised manuscript, demonstrating DensityFM's advantages in both accuracy and efficiency. Further analysis is provided in Section 2.2 (Neural Copula Estimation) of the revised manuscript.

ModelSynthetic Gap/dim ↓Time (s) ↓Real NLL/dim ↓Time (s) ↓
DensityFM (1-128D)0.054570.2611.00971.543
NPE-PFN (autoregressive, TabPFN3)0.0986416.1170.9399193.897

W6. Ablation on marginal–copula separation

Thank you for the suggestion. We conducted an ablation study to evaluate the contribution of marginal–copula separation. The results below demonstrate substantial accuracy improvements on both synthetic and real-world datasets.

Real NLL/dim ↓Synthetic gap/dim ↓
Marginal–copula separation1.046890.04826
Direct joint modeling1.160090.10376

Results are averaged over four real-world datasets and 22 synthetic distributions at 64D, using the same backbone and training budget.

W7. How well does DensityFM generalize beyond the distributions encountered during pretraining?

Thank you for raising this question. We answer this from two aspects:

  1. Real data generalization. All real-world datasets in our evaluation are out-of-distribution relative to pretraining. DensityFM's strong performance demonstrates its ability to generalize to unseen data distributions.

  2. Unseen synthetic distributions: We additionally evaluate on three distributions entirely excluded from pretraining, using NSF as a baseline. The results show our method’s strong generalization on these unseen cases.

    Method (64D)Inverse GaussianBurr XIIGumbel
    DensityFM gap ↓0.55040.27920.1066
    NSF gap ↓0.67030.38050.1880

W8. Missing related work

Thank you for highlighting these relevant works. We have incorporated and discussed all the suggested references in the revised manuscript.

Q1. Why were the Table 3 datasets selected, and how different are they from the pretraining data?

These are standard real-world density-estimation benchmarks from the widely cited MAF paper, and were entirely unseen during pretraining.

Q2. How is the 156× speedup calculated, and what applications benefit from DensityFM?

  • The speedup is measured against MAF, a widely-used neural density estimator.
  • As explained above, DensityFM is intended as a publicly available foundation model, so users can perform in-context density estimation on datasets of interest in seconds.

Comments on the paper

2 comments
Area_Chair_graoOpenReview ↗

Author-Reviewer Discussion ends in 1 day, 12 hours

Dear authors and reviewers,

Thank you to the authors for their detailed responses of Nov 15 and the updated manuscript. The discussion period closes in about a day and a half, so this is the last chance for a direct exchange before the reviewer-AC discussion starts on Nov 19.

Reviewers: I haven’t yet seen a reply from any of you. Please read the authors’ responses and the revision, which address several of your points directly. Please let us know whether your concerns have been addressed, and update your recommendation if your assessment has changed. Even a short acknowledgment helps.

Authors: please be ready to answer any follow-up questions in the time left, focusing on the points most likely to affect the decision.

Thank you all for keeping the discussion constructive and timely.

Your AC

AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the AC for taking care of our submission and the reviewers for their constructive and insightful feedback. Below, we provide a summary of the rebuttal.

Strengths highlighted

  • Zero-shot density estimation on unseen datasets without task-specific training (e2ZY, 5Kqx, ZALg), enabled by our architecture and large-scale pretraining.
  • Substantially lower inference cost (e2ZY, 5Kqx, ZALg) with comparable or even superior accuracy in certain settings, particularly for cases with massive density estimation tasks (e2ZY).
  • Interesting marginal–copula disentanglement, combining marginal estimation with pretrained dependence modeling (5Kqx).
  • Competitive performance in density estimation and conditional inference (5Kqx, ZALg).

These assessments highlight DensityFM’s potential as a foundation for real-time, large-scale density estimation across applications.

Main concerns addressed

  • Pretraining cost and intended use (e2ZY). We clarify that DensityFM is designed as a publicly available foundation model, requiring no user-side (pre)training for downstream applications.
  • Comparison with NPE-PFN and other methods (e2ZY, 5Kqx, ZALg). We sharpened our positioning and added comparisons with NPE-PFN, along with DiScoFormer, Residual Flows, and CFM.
  • Higher-dimensional scalability (e2ZY). Our new checkpoint supports 1–128 dimensions without dimension-specific pretraining, with updated results included in the manuscript.
  • Marginal–copula disentanglement ablation (e2ZY, 5Kqx, ZALg). Ablation studies demonstrate the advantages of such disentanglement over joint modeling.
  • Generalization (e2ZY, ZALg). Experiments on real-world datasets and additional synthetic distribution families excluded from pretraining demonstrate strong OOD generalization.
  • Inconsistency issues (ZALg). We clarified dataset differences relative to the MAF paper and corrected inconsistencies across tables and figures.

Other concerns, including context-size scaling, feature-order sensitivity, clipping-induced unnormalized densities are fixed. We commit to releasing the pretrained checkpoints on Hugging Face.

We again thank the AC and reviewers for their time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.