- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces PRODiGI, a pretrained model that infers executable generative programs directly from empirical data. It supports GMMs, SCMs, and copulas through template prediction and non-autoregressive parameter decoding. The authors also propose program-space fine-tuning, which optimizes the inferred program's parameters while keeping the pretrained model fixed. They evaluate the approach on data generation, density estimation, and score estimation tasks.
Critical strengths
-
Interesting approach: Predicting a reusable generative program directly from data is a useful idea. The resulting program can be inspected and executed without needing the pretrained model again.
-
Good experimental evaluation: The paper compares against several relevant baselines and includes useful ablations. PRODiGI achieves very fast zero-shot generation, while program-space fine-tuning significantly improves generation quality and density estimation accuracy.
Critical weaknesses
-
Dependence on fine-tuning: The strongest accuracy results come from PRODiGI-FT rather than the one-pass model. For example, density MAE improves from 7.58 to 2.19 after fine-tuning, while generation time increases from 0.06 to 12.90 seconds. It is unclear how much benefit comes from the learned program initialization compared with conventional parameter optimization. A comparison using the same predicted program structures with alternative initializations would help clarify this.
-
Limited program recovery and generalization: The method is restricted to predefined GMM, SCM, and copula templates. Its performance also drops on real-world datasets. Moreover, matching the observed distribution does not necessarily mean recovering the true generating program. This is particularly important for SCMs, where different causal structures may produce similar observations. The paper provides only qualitative examples of SCM recovery, and SCM templates are disabled for real-world density and score evaluation.
Questions for authors
-
How does PRODiGI-FT compare with fitting the same program structures from random or classical initializations under a matched computational budget?
-
Could the authors provide quantitative results on program recovery, particularly SCM graph structure and parameter accuracy?
-
Could the authors report fine-tuning results separately for GMMs, SCMs, and copulas? Since SCMs use MMD while the other families use NLL, a family-wise comparison would help clarify the effectiveness of the refinement procedure.
-
How well does the approach generalize to distributions outside the supported program families, given the performance drop on real-world datasets?
5 comments
Q2, Quantitative program recovery
We thank the reviewer for this suggestion. We have added quantitative program reconstruction results in Figure 4 (left) and Appendix F.2 (Figure 12), evaluating both structural choices and numerical parameters. Overall, PRODiGI correctly predicts the program family in 743/744 cases (99.9%), the Copula marginal function in 98.8% of cases, and the number of GMM components in 75% of cases.
SCM graph recovery: We additionally evaluate the recovery of the underlying causal graph on 32 synthetic SCM datasets. All variables are observed, and the ground-truth graphs contain 36.1 edges on average. As a baseline, we generate random graphs that preserve each node's true number of parents, but select those parents randomly.
| Metric | PRODiGI | Random graph |
|---|---|---|
| Directed edge F1 ↑ | 0.59 ± 0.05 | 0.39 |
| Precision ↑ | 0.67 ± 0.04 | 0.39 |
| Recall ↑ | 0.54 ± 0.05 | 0.39 |
| Skeleton F1 ↑ | 0.63 ± 0.05 | 0.44 |
| SHD (directed) ↓ | 32.3 ± 6.6 | 61.5 |
PRODiGI achieves substantially better graph recovery than the random baseline, reducing structural Hamming distance by 47%, despite the baseline receiving the ground-truth in-degrees.
SCM parameter reconstruction: Beyond graph recovery, PRODiGI predicts the correct activation function in 68.1% of cases, the maximum in-degree in 51.6% (MAE 0.58), and parent indices in 67.5%. Continuous parameter reconstruction yields an MAE of 0.187 for noise parameters and 1.39 for structural factors.
These results demonstrate partial recovery of the generating SCM structures, substantially exceeding a random graph baseline. Importantly, our objective is not to recover the exact causal structure, but to infer an executable program that accurately reproduces the observed distribution. Since different SCMs can yield similar observational distributions, exact structural recovery is neither guaranteed nor necessary for our primary tasks of generation, density estimation, and score estimation. Nevertheless, the reconstruction results show that PRODiGI learns meaningful structural information beyond merely approximating the data distribution.
Revision. We add a comparison of PRODiGI’s program parameter recovery performance to Section 4.1, Figure 4 (left), and Appendix F.2.
Q3, Family-wise fine-tuning effectiveness
We thank the reviewer for this suggestion. While detailed family-wise results are available in Appendix G, we additionally conducted experiments directly comparing refinement effects across the three program families. We also corrected an inconsistency in our original methodology, highlighted by another reviewer: our current SCM prior permits analytical density and score evaluation. Consequently, we now use NLL-based fine-tuning for all three families. To examine this choice, we additionally compare NLL- and MMD-based refinement for SCMs.
In-distribution results (32 datasets per family):
| Family | FT loss | MMD ×10³ ↓ | Spearman ↑ | Score cosine ↑ |
|---|---|---|---|---|
| GMM | NLL | 12.94 → 0.39 | 0.63 → 0.95 | 0.62 → 0.92 |
| Copula | NLL | 2.14 → 0.93 | 0.75 → 0.84 | 0.42 → 0.59 |
| SCM | MMD | 1.28 → 0.27 | 0.78 → 0.76 | 0.73 → 0.71 |
| SCM | NLL | 1.28 → 0.36 | 0.78 → 0.89 | 0.73 → 0.84 |
| (Values show PRODiGI → PRODiGI-FT.) |
Fine-tuning significantly improves generation MMD across all families (all p ≤ 2 × 10⁻⁴). Importantly, the SCM comparison demonstrates why we switched to NLL optimization: while MMD refinement slightly outperforms NLL refinement on generation MMD, it does not improve density ranking or score direction. In contrast, NLL refinement improves Spearman correlation from 0.78 to 0.89 and score cosine from 0.73 to 0.84 (both p ≤ 4 × 10⁻⁵).
Real-world results (grouped by predicted program family):
| Predicted family | FT loss | MMD ×10³ ↓ |
|---|---|---|
| GMM (46) | NLL | 87.92 → 32.79 |
| Copula (36) | NLL | 50.23 → 39.00 |
| SCM (14) | MMD | 51.89 → 42.17 |
| SCM (14) | NLL | 51.89 → 47.69 |
Refinement also improves mean MMD across all three predicted families on real-world data, although the improvement for SCMs under NLL refinement is not statistically significant (p = 0.06). The improvement is most pronounced for GMMs. This is consistent with their greater amenability to gradient-based refinement: for a fixed number of components, their continuous mixture parameters can all be optimized, whereas SCMs and Copulas also involve discrete structural choices (e.g., causal graphs, activation functions, or marginal families) that remain fixed during refinement. Overall, these experiments demonstrate that program-space refinement benefits all three families, while its effectiveness depends on the program's parameterization.
Revision. We show family-wise fine-tuning results in Appendix G, but have updated these with new SCM results.
Q4, Out-of-distribution generalization
We thank the reviewer for raising this important limitation. To quantify generalization beyond our training priors, we conducted additional experiments on 96 synthetic OOD datasets (32 per family), generated using mechanisms not encountered during pretraining:
GMM: Increase the number of mixture components from 1–10 to 11–20.
SCM: Replace the training activation functions with five previously unseen functions.
Copula: Replace the five training marginal distributions with five unseen marginals.
While these experiments do not cover entirely unrelated generative families, they provide controlled tests of generalization to distributions that cannot generally be represented exactly by our learned program vocabulary.
OOD results (Appendix F.5, Table 9):
| Method | MMD ×10³ ↓ | Spearman ↑ | Score cosine ↑ |
|---|---|---|---|
| PRODiGI | 8.70 ± 1.67 | 0.56 ± 0.03 | 0.32 ± 0.02 |
| PRODiGI-FT | 1.82 ± 0.45 | 0.71 ± 0.03 | 0.58 ± 0.03 |
| NNKDE-FT | 7.98 ± 1.38 | 0.80 ± 0.01 | 0.40 ± 0.02 |
| KDE | 3.22 ± 0.74 | 0.82 ± 0.01 | 0.58 ± 0.03 |
| GMM (BIC) | 1.35 ± 0.56 | 0.84 ± 0.01 | 0.67 ± 0.03 |
| TabPFN | 0.04 ± 0.02 | 0.86 ± 0.01 | 0.60 ± 0.02 |
OOD degradation: PRODiGI is more strongly affected by the distribution shift than classical baselines. For PRODiGI-FT, mean MMD increases from 0.56 to 1.82 ×10⁻³, density Spearman decreases from 0.90 to 0.71, and score cosine decreases from 0.78 to 0.58. This confirms that the restricted program vocabulary introduces a meaningful limitation on generalization.
Remaining performance: Despite this shift, PRODiGI-FT retains competitive generation quality, achieving lower MMD than KDE (3.22) and NNKDE-FT (7.98), which outperforms PRODiGI-FT on real-world data, although fitted GMMs and TabPFN perform better (1.35, 0.04). Fine-tuning reduces PRODiGI's initial OOD MMD from 8.70 to 1.82, demonstrating that program-space optimization remains effective even when the underlying distribution cannot be represented exactly.
Computational efficiency: The OOD experiments use the same dimensionalities and input-size settings as our in-distribution evaluation, so we expect comparable inference costs. For reference, measured in-distribution generation times (including fitting and generating 1,000 samples) are 0.06 s for PRODiGI, 12.90 s for PRODiGI-FT, and 37.24 s for GMM. Overall, these experiments highlight both the potential and limitations of our approach: PRODiGI can approximate distributions outside its training prior, but its expressiveness is constrained by the supported program vocabulary. Expanding this vocabulary to include more diverse generating mechanisms is therefore an important direction for future work.
Revision. We include the experiments on OOD generalization in Section 4.1 and Appendix F.5 (Figures 18–20, Table 10).
We thank the reviewer for their constructive assessment and suggestions, which motivated substantial new experiments on initialization, SCM structure recovery, family-wise refinement, and out-of-distribution generalization. We believe these results directly address the questions raised in the review and significantly strengthen the empirical evaluation. We would be grateful if you could reconsider your rating in light of this additional evidence. We welcome any remaining questions or concerns during the discussion period.
We sincerely thank the reviewer for their positive assessment and for recognizing the value of directly inferring reusable generative programs, the comprehensive experimental evaluation, and the effectiveness of zero-shot inference and program-space fine-tuning. We also appreciate the requests for comparisons against alternative initializations, quantitative program reconstruction results, family-wise refinement analyses, and out-of-distribution evaluation. We address each point below, with corresponding additions highlighted in blue in the updated manuscript uploaded to OpenReview.
Q1, Does fine-tuning require PRODiGI initialization?
We thank the reviewer for this suggestion. To isolate the contribution of PRODiGI's learned initialization, we compare it against three increasingly unconstrained random initializations, while keeping the fine-tuning procedure and optimization budget identical:
Same template: Retain PRODiGI's predicted template but randomly initialize its parameters following our program prior.
Same family: Randomly select a template within PRODiGI's predicted family (GMM, SCM, or Copula), with random parameters.
Random family: Randomly select both the program family and template, with random parameters. We evaluate all three settings on 96 in-distribution (ID) and 96 real-world (RW) datasets.
PRODiGI initialization significantly outperforms all three random baselines in both settings (two-sided sign tests, all p ≤ 9 × 10⁻⁶). Crucially, even when the exact same template is used, PRODiGI's learned parameters reduce mean MMD from 3.07 to 0.56 ×10⁻³ on in-distribution data and from 48.11 to 37.29 ×10⁻³ on real-world data. Full per-dataset comparisons are included in Appendix F.4 (Figures 15 and 16).
Classical initialization: We additionally evaluated a classical GMM baseline initialized with k-means, followed by the same 300 NLL optimization steps as PRODiGI-FT.
On in-distribution GMM datasets (32), a fixed 10-component GMM performs extremely well, outperforming PRODiGI-FT in generation MMD (0.01 vs. 0.39 ×10⁻³) and density Spearman (0.976 vs. 0.954), while achieving identical score cosine (0.917). This demonstrates that specialized classical initialization can outperform learned parameter initialization for individual program families.
However, PRODiGI's predicted program structure still provides a measurable benefit. When k-means uses PRODiGI's predicted number of components instead of a fixed 10, density Spearman increases from 0.976 to 0.985 and score cosine from 0.917 to 0.957, with improvements on 27 datasets versus 2 for each metric. The fixed 10-component GMM retains lower MMD (0.01 vs. 0.06 ×10⁻³), illustrating that better sample matching does not necessarily imply more accurate density or score estimation. Across all 96 in-distribution datasets, using a fixed 10-component GMM also achieves lower generation MMD than PRODiGI-FT (0.25 vs. 0.56 ×10⁻³), but substantially worse density Spearman (0.820 vs. 0.895) and score cosine (0.512 vs. 0.784). These results clarify the complementary roles of learned program inference and classical optimization: even with the same k-means initialization and refinement procedure, PRODiGI's structural predictions improve density and score estimation. Moreover, k-means is specific to GMMs, whereas PRODiGI provides a unified mechanism for inferring structures and parameters across GMMs, SCMs, and Copulas.
Revision. The revised manuscript contains a more detailed version of these experiments in Appendix F.4.