acceptodds
Under review as a conference paper at ICLR 2027

From Data to Program: Fast & Direct Generative Program Inference from Empirical Data

Abstract

Estimating probability densities from a finite set of samples typically requires dataset-specific model fitting. We introduce PRODiGI, a pretrained data-to-program model that infers an explicit, executable generative program in a single forward pass. Pretrained on synthetic datasets paired with their ground-truth programs, PRODiGI accommodates diverse generative families and data dimensionalities through template prediction and non-autoregressive program parameter decoding. Its inferred programs support direct sampling, density and score evaluation, and inspection independently of the pretrained model. We further introduce program-space fine-tuning, which refines differentiable program parameters by matching generated and empirical samples while keeping model parameters intact. Experiments show that PRODiGI achieves lower average density and score MAE than existing pretrained models, while offering multi-fold speedups over its closest competitors. Program-space fine-tuning further reduces generation MMD by 84%. By turning empirical data into explicit, reusable programs, PRODiGI introduces a new direction for fast, interpretable tabular generative modeling.

Primary area:
generative models
Keywords:
Tabular Learning, Data Generation, Density Estimation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_iRRjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PRODiGI, a pretrained model that infers executable generative programs directly from empirical data. It supports GMMs, SCMs, and copulas through template prediction and non-autoregressive parameter decoding. The authors also propose program-space fine-tuning, which optimizes the inferred program's parameters while keeping the pretrained model fixed. They evaluate the approach on data generation, density estimation, and score estimation tasks.

Critical strengths
  1. Interesting approach: Predicting a reusable generative program directly from data is a useful idea. The resulting program can be inspected and executed without needing the pretrained model again.

  2. Good experimental evaluation: The paper compares against several relevant baselines and includes useful ablations. PRODiGI achieves very fast zero-shot generation, while program-space fine-tuning significantly improves generation quality and density estimation accuracy.

Critical weaknesses
  1. Dependence on fine-tuning: The strongest accuracy results come from PRODiGI-FT rather than the one-pass model. For example, density MAE improves from 7.58 to 2.19 after fine-tuning, while generation time increases from 0.06 to 12.90 seconds. It is unclear how much benefit comes from the learned program initialization compared with conventional parameter optimization. A comparison using the same predicted program structures with alternative initializations would help clarify this.

  2. Limited program recovery and generalization: The method is restricted to predefined GMM, SCM, and copula templates. Its performance also drops on real-world datasets. Moreover, matching the observed distribution does not necessarily mean recovering the true generating program. This is particularly important for SCMs, where different causal structures may produce similar observations. The paper provides only qualitative examples of SCM recovery, and SCM templates are disabled for real-world density and score evaluation.

Questions for authors
  1. How does PRODiGI-FT compare with fitting the same program structures from random or classical initializations under a matched computational budget?

  2. Could the authors provide quantitative results on program recovery, particularly SCM graph structure and parameter accuracy?

  3. Could the authors report fine-tuning results separately for GMMs, SCMs, and copulas? Since SCMs use MMD while the other families use NLL, a family-wise comparison would help clarify the effectiveness of the refinement procedure.

  4. How well does the approach generalize to distributions outside the supported program families, given the performance drop on real-world datasets?

5 comments

We sincerely thank the reviewer for their positive assessment and for recognizing the value of directly inferring reusable generative programs, the comprehensive experimental evaluation, and the effectiveness of zero-shot inference and program-space fine-tuning. We also appreciate the requests for comparisons against alternative initializations, quantitative program reconstruction results, family-wise refinement analyses, and out-of-distribution evaluation. We address each point below, with corresponding additions highlighted in blue in the updated manuscript uploaded to OpenReview.

Q1, Does fine-tuning require PRODiGI initialization?

We thank the reviewer for this suggestion. To isolate the contribution of PRODiGI's learned initialization, we compare it against three increasingly unconstrained random initializations, while keeping the fine-tuning procedure and optimization budget identical:

Same template: Retain PRODiGI's predicted template but randomly initialize its parameters following our program prior.

Same family: Randomly select a template within PRODiGI's predicted family (GMM, SCM, or Copula), with random parameters.

Random family: Randomly select both the program family and template, with random parameters. We evaluate all three settings on 96 in-distribution (ID) and 96 real-world (RW) datasets.

InitializationID MMD ×10³ ↓ID winsRW MMD ×10³ ↓RW wins
PRODiGI0.56–37.29–
Random parameters, same template3.0770/9648.1172/96
Random template, same family6.3579/9653.4777/96
Entirely random program11.9281/9652.9574/96
Wins indicate datasets where PRODiGI initialization achieves lower MMD than the respective random initialization, after identical fine-tuning.

PRODiGI initialization significantly outperforms all three random baselines in both settings (two-sided sign tests, all p ≤ 9 × 10⁻⁶). Crucially, even when the exact same template is used, PRODiGI's learned parameters reduce mean MMD from 3.07 to 0.56 ×10⁻³ on in-distribution data and from 48.11 to 37.29 ×10⁻³ on real-world data. Full per-dataset comparisons are included in Appendix F.4 (Figures 15 and 16).

Classical initialization: We additionally evaluated a classical GMM baseline initialized with k-means, followed by the same 300 NLL optimization steps as PRODiGI-FT.

On in-distribution GMM datasets (32), a fixed 10-component GMM performs extremely well, outperforming PRODiGI-FT in generation MMD (0.01 vs. 0.39 ×10⁻³) and density Spearman (0.976 vs. 0.954), while achieving identical score cosine (0.917). This demonstrates that specialized classical initialization can outperform learned parameter initialization for individual program families.

However, PRODiGI's predicted program structure still provides a measurable benefit. When k-means uses PRODiGI's predicted number of components instead of a fixed 10, density Spearman increases from 0.976 to 0.985 and score cosine from 0.917 to 0.957, with improvements on 27 datasets versus 2 for each metric. The fixed 10-component GMM retains lower MMD (0.01 vs. 0.06 ×10⁻³), illustrating that better sample matching does not necessarily imply more accurate density or score estimation. Across all 96 in-distribution datasets, using a fixed 10-component GMM also achieves lower generation MMD than PRODiGI-FT (0.25 vs. 0.56 ×10⁻³), but substantially worse density Spearman (0.820 vs. 0.895) and score cosine (0.512 vs. 0.784). These results clarify the complementary roles of learned program inference and classical optimization: even with the same k-means initialization and refinement procedure, PRODiGI's structural predictions improve density and score estimation. Moreover, k-means is specific to GMMs, whereas PRODiGI provides a unified mechanism for inferring structures and parameters across GMMs, SCMs, and Copulas.

Revision. The revised manuscript contains a more detailed version of these experiments in Appendix F.4.

Q2, Quantitative program recovery

We thank the reviewer for this suggestion. We have added quantitative program reconstruction results in Figure 4 (left) and Appendix F.2 (Figure 12), evaluating both structural choices and numerical parameters. Overall, PRODiGI correctly predicts the program family in 743/744 cases (99.9%), the Copula marginal function in 98.8% of cases, and the number of GMM components in 75% of cases.

SCM graph recovery: We additionally evaluate the recovery of the underlying causal graph on 32 synthetic SCM datasets. All variables are observed, and the ground-truth graphs contain 36.1 edges on average. As a baseline, we generate random graphs that preserve each node's true number of parents, but select those parents randomly.

MetricPRODiGIRandom graph
Directed edge F1 ↑0.59 ± 0.050.39
Precision ↑0.67 ± 0.040.39
Recall ↑0.54 ± 0.050.39
Skeleton F1 ↑0.63 ± 0.050.44
SHD (directed) ↓32.3 ± 6.661.5

PRODiGI achieves substantially better graph recovery than the random baseline, reducing structural Hamming distance by 47%, despite the baseline receiving the ground-truth in-degrees.

SCM parameter reconstruction: Beyond graph recovery, PRODiGI predicts the correct activation function in 68.1% of cases, the maximum in-degree in 51.6% (MAE 0.58), and parent indices in 67.5%. Continuous parameter reconstruction yields an MAE of 0.187 for noise parameters and 1.39 for structural factors.

These results demonstrate partial recovery of the generating SCM structures, substantially exceeding a random graph baseline. Importantly, our objective is not to recover the exact causal structure, but to infer an executable program that accurately reproduces the observed distribution. Since different SCMs can yield similar observational distributions, exact structural recovery is neither guaranteed nor necessary for our primary tasks of generation, density estimation, and score estimation. Nevertheless, the reconstruction results show that PRODiGI learns meaningful structural information beyond merely approximating the data distribution.

Revision. We add a comparison of PRODiGI’s program parameter recovery performance to Section 4.1, Figure 4 (left), and Appendix F.2.

Q3, Family-wise fine-tuning effectiveness

We thank the reviewer for this suggestion. While detailed family-wise results are available in Appendix G, we additionally conducted experiments directly comparing refinement effects across the three program families. We also corrected an inconsistency in our original methodology, highlighted by another reviewer: our current SCM prior permits analytical density and score evaluation. Consequently, we now use NLL-based fine-tuning for all three families. To examine this choice, we additionally compare NLL- and MMD-based refinement for SCMs.

In-distribution results (32 datasets per family):

FamilyFT lossMMD ×10³ ↓Spearman ↑Score cosine ↑
GMMNLL12.94 → 0.390.63 → 0.950.62 → 0.92
CopulaNLL2.14 → 0.930.75 → 0.840.42 → 0.59
SCMMMD1.28 → 0.270.78 → 0.760.73 → 0.71
SCMNLL1.28 → 0.360.78 → 0.890.73 → 0.84
(Values show PRODiGI → PRODiGI-FT.)

Fine-tuning significantly improves generation MMD across all families (all p ≤ 2 × 10⁻⁴). Importantly, the SCM comparison demonstrates why we switched to NLL optimization: while MMD refinement slightly outperforms NLL refinement on generation MMD, it does not improve density ranking or score direction. In contrast, NLL refinement improves Spearman correlation from 0.78 to 0.89 and score cosine from 0.73 to 0.84 (both p ≤ 4 × 10⁻⁵).

Real-world results (grouped by predicted program family):

Predicted familyFT lossMMD ×10³ ↓
GMM (46)NLL87.92 → 32.79
Copula (36)NLL50.23 → 39.00
SCM (14)MMD51.89 → 42.17
SCM (14)NLL51.89 → 47.69

Refinement also improves mean MMD across all three predicted families on real-world data, although the improvement for SCMs under NLL refinement is not statistically significant (p = 0.06). The improvement is most pronounced for GMMs. This is consistent with their greater amenability to gradient-based refinement: for a fixed number of components, their continuous mixture parameters can all be optimized, whereas SCMs and Copulas also involve discrete structural choices (e.g., causal graphs, activation functions, or marginal families) that remain fixed during refinement. Overall, these experiments demonstrate that program-space refinement benefits all three families, while its effectiveness depends on the program's parameterization.

Revision. We show family-wise fine-tuning results in Appendix G, but have updated these with new SCM results.

Q4, Out-of-distribution generalization

We thank the reviewer for raising this important limitation. To quantify generalization beyond our training priors, we conducted additional experiments on 96 synthetic OOD datasets (32 per family), generated using mechanisms not encountered during pretraining:

GMM: Increase the number of mixture components from 1–10 to 11–20.

SCM: Replace the training activation functions with five previously unseen functions.

Copula: Replace the five training marginal distributions with five unseen marginals.

While these experiments do not cover entirely unrelated generative families, they provide controlled tests of generalization to distributions that cannot generally be represented exactly by our learned program vocabulary.

OOD results (Appendix F.5, Table 9):

MethodMMD ×10³ ↓Spearman ↑Score cosine ↑
PRODiGI8.70 ± 1.670.56 ± 0.030.32 ± 0.02
PRODiGI-FT1.82 ± 0.450.71 ± 0.030.58 ± 0.03
NNKDE-FT7.98 ± 1.380.80 ± 0.010.40 ± 0.02
KDE3.22 ± 0.740.82 ± 0.010.58 ± 0.03
GMM (BIC)1.35 ± 0.560.84 ± 0.010.67 ± 0.03
TabPFN0.04 ± 0.020.86 ± 0.010.60 ± 0.02

OOD degradation: PRODiGI is more strongly affected by the distribution shift than classical baselines. For PRODiGI-FT, mean MMD increases from 0.56 to 1.82 ×10⁻³, density Spearman decreases from 0.90 to 0.71, and score cosine decreases from 0.78 to 0.58. This confirms that the restricted program vocabulary introduces a meaningful limitation on generalization.

Remaining performance: Despite this shift, PRODiGI-FT retains competitive generation quality, achieving lower MMD than KDE (3.22) and NNKDE-FT (7.98), which outperforms PRODiGI-FT on real-world data, although fitted GMMs and TabPFN perform better (1.35, 0.04). Fine-tuning reduces PRODiGI's initial OOD MMD from 8.70 to 1.82, demonstrating that program-space optimization remains effective even when the underlying distribution cannot be represented exactly.

Computational efficiency: The OOD experiments use the same dimensionalities and input-size settings as our in-distribution evaluation, so we expect comparable inference costs. For reference, measured in-distribution generation times (including fitting and generating 1,000 samples) are 0.06 s for PRODiGI, 12.90 s for PRODiGI-FT, and 37.24 s for GMM. Overall, these experiments highlight both the potential and limitations of our approach: PRODiGI can approximate distributions outside its training prior, but its expressiveness is constrained by the supported program vocabulary. Expanding this vocabulary to include more diverse generating mechanisms is therefore an important direction for future work.

Revision. We include the experiments on OOD generalization in Section 4.1 and Appendix F.5 (Figures 18–20, Table 10).

We thank the reviewer for their constructive assessment and suggestions, which motivated substantial new experiments on initialization, SCM structure recovery, family-wise refinement, and out-of-distribution generalization. We believe these results directly address the questions raised in the review and significantly strengthen the empirical evaluation. We would be grateful if you could reconsider your rating in light of this additional evidence. We welcome any remaining questions or concerns during the discussion period.

Reviewer_tREQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PRODiGI, a pretrained model that takes a tabular dataset of continuous variables as input and infers, in a single forward pass, an explicit and executable program (GMM / SCM / Copula) that could plausibly have generated the data. The core idea is to set the programs used for generating tables as the reconstruction target, and amortize per-dataset fitting into pretraining. The model consists of a Set Transformer encoder, a classification head that selects one of 786 templates, and a NAR decoder that fills the template’s slots in parallel. The program itself is normalized and canonically serialized. After inference, the paper proposes program-space fine-tuning, which adjusts only the parameters of the inferred program. Across the three tasks of generation, density, and score, PRODiGI offers fast and accurate density estimation.

Critical strengths
  • Proposed framework is useful. Inverting the usual pipeline so that the generative programs behind synthetic data become the reconstruction target is a simple and effective idea.
  • The design choices are well motivated. Separating templates from values fixes the number and type of slots, which makes non-autoregressive parallel decoding possible. It makes sense and is validated empirically by 100% program validity and 44× faster inference. Normalization and serialization are also well motivated.
  • Program-space fine-tuning is natural for the unsupervised setting. When labels model fine-tuning is impossible, one can adjust only the small parameter space of the program with no backpropagation through the model.
Critical weaknesses

There is no "different proposer + same tuner" baseline. For example, there exist approaches in which an LLM inspects some specification(possibly table rows), writes a program, and then tunes that program's parameters [1]. Such methods require no pretraining and impose no template constraints, yet the paper provides no comparison against this line of work. It is not shown where PRODiGI holds an advantage over an LLM proposer. Two conceivable advantages are that, unlike an LLM, PRODiGI directly observes the entire sample set, and that its inference cost is low. However, neither has been verified experimentally. The real-world results indicate that structure proposal is the bottleneck. When the initial structure is wrong, PRODiGI cannot recover from it, whereas LLM-based methods, which can draw on priors learned from vast amounts of data and re-propose after diagnosis, are favored precisely in this regime. A performance comparison against LLM-based methods on real-world data is needed.

[1] LLM-SR: Scientific Equation Discovery via Programming with Large Language Models, ICLR 2025

Questions for authors

See weaknesses

2 comments

We sincerely thank the reviewer for their constructive assessment and for recognizing the usefulness of learning generative programs directly from data, the motivation behind our template-based non-autoregressive decoder, and the benefits of program-space fine-tuning. We also appreciate the suggestion to compare against LLM-based program inference, particularly regarding generation accuracy, computational cost, and access to the input dataset. We address these concerns through a new experiment below, with corresponding revisions highlighted in blue in the updated manuscript uploaded to OpenReview.

W1/Q1, Comparison with LLM-based program proposal

We thank the reviewer for suggesting a comparison with LLM-based program inference and for highlighting two potential limitations: computational cost and restricted access to the input dataset.

We implemented an LLM-based baseline using Claude Haiku 5.5. (We chose Haiku for cost efficiency across 192 datasets and up to 576 LLM calls; stronger models may yield different results). The LLM receives 50 randomly sampled observations and feature names (where available) and generates an executable Python program for sampling and density estimation (we calculate scores from the density estimates later). Unlike restricting the LLM to our predefined templates, this allows it to propose arbitrary model structures, transformations, and parameter-fitting procedures.

The generated program is fitted using 75% of the input data, and its NLL on the remaining 25% is returned to the LLM for up to two additional proposal rounds. We select the program with the lowest validation NLL and evaluate it on an independent test set. We chose Claude Haiku 5.5 to keep the comparison computationally feasible across 192 datasets (96 synthetic and 96 real-world), requiring up to 576 LLM proposal calls followed by program fitting and evaluation. Using a larger Claude model would substantially increase API costs (estimated at hundreds of dollars), so we regard this experiment as a comparison with a computationally accessible LLM baseline rather than an upper bound on LLM-based program inference.

Results (96 datasets per setting):

MethodID MMD ×10³ ↓ID Spearman ↑ID score cosine ↑RW MMD ×10³ ↓
Haiku, 1 round0.29 ± 0.150.832 ± 0.0150.587 ± 0.03418.79 ± 3.84
Haiku, 3 rounds0.41 ± 0.300.900 ± 0.0110.652 ± 0.03419.00 ± 3.92
PRODiGI-FT0.56 ± 0.200.895 ± 0.0140.780 ± 0.02737.29 ± 4.58
GMM (BIC)0.78 ± 0.300.817 ± 0.0160.590 ± 0.03516.25 ± 3.21

Accuracy: LLM-generated programs achieve strong generation quality, outperforming PRODiGI-FT in mean MMD on both in-distribution and real-world datasets. On in-distribution data, Haiku and PRODiGI-FT achieve nearly identical density-ranking accuracy (0.900 vs. 0.895), while PRODiGI-FT achieves higher score-direction accuracy (0.780 vs. 0.652). Effect of additional LLM calls: Additional proposals improve Haiku's density-ranking accuracy (0.832 → 0.900) and score cosine (0.587 → 0.652), but do not consistently improve generation MMD (0.29 → 0.41 in-distribution; 18.79 → 19.00 real-world). This illustrates that improvements in validation NLL and density estimation do not necessarily translate into better distribution matching.

Computational cost: On in-distribution datasets, the three-round LLM pipeline takes 162 s per dataset (47 s API latency and 113 s executing the generated code), compared with 12.90 s for PRODiGI-FT and 0.06 s for zero-shot PRODiGI. On real-world datasets, the corresponding runtimes are 120 s, 9.64 s, and 0.06 s. Thus, the LLM pipeline is approximately 12.5× slower than PRODiGI-FT and 2,000–2,700× slower than zero-shot PRODiGI. Additionally, the LLM incurs approximately $0.007-$0.008 in API charges per dataset, excluding execution costs.

Access to the input data: Providing the entire dataset directly to the LLM would be computationally impractical and costly. Instead, we provide 50 randomly sampled observations and ask the LLM to generate an executable fitting procedure, rather than a fitted model. This procedure is subsequently executed on the full training partition, allowing it to estimate its parameters using all available data. Thus, the 50-sample restriction applies only to the program proposal stage, not to parameter estimation or fitting.

Program structures: Despite being free to propose arbitrary executable programs, Haiku selects GMM-based approaches on 89/96 real-world datasets (57 plain GMMs, 32 with additional transformations) and 94/96 in-distribution datasets (38 plain GMMs, 56 with transformations).

Overall, these results establish LLM-based program inference as a strong alternative in terms of generation accuracy. However, its substantial runtime and API costs, together with limited direct access to the data during program proposal, highlight the advantages of amortized inference. PRODiGI predicts an explicit generative program directly from the input dataset in a single forward pass, with substantially better score accuracy in our in-distribution comparison.

We believe the new results directly address your central concern and provide a clearer understanding of PRODiGI's advantages and limitations relative to LLM-based program inference. We would be grateful if you would consider revisiting your rating in light of this new evidence. We would also welcome any further questions or suggestions that could strengthen the paper.

Revision. We add a comparison of PRODiGI to this LLM-based approach, as well as the prompt used in Appendix F.6.

Reviewer_qjccOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PRODiGI maps empirical datasets to explicit generative programs spanning Gaussian mixtures, structural causal models, and Gaussian copulas. It combines a Set Transformer encoder, template prediction, and non-autoregressive parameter decoding, trained on normalized and canonically serialized synthetic programs. Optional refinement updates program parameters while keeping network weights fixed. Experiments report fast generation and substantial synthetic-data refinement gains. The synthetic comparisons against analytical ground-truth densities and scores are restricted to GMM and copula priors.

Critical strengths
  1. A single model handles several distribution families and varying dimensionalities, returning explicit structures and parameters. These can be retained independently of the network and observations. This is useful when an inferred distribution must be inspected, stored, or sampled repeatedly, and the evaluated tractable families also permit statistical queries through program execution.
  2. Figure 5 tests the main decoding choices directly. In the 20-dimensional ablation, non-autoregressive decoding produces syntactically valid programs and is reported as 44 times faster than autoregressive decoding. Normalization reduces real-world generation MMD by 68%, giving concrete support for handling scale variation during training and inference.
  3. Table 2 shows a useful trade-off after refinement: MMD falls from 5.46×10−35.46\times10^{-3} to 0.89×10−30.89\times10^{-3}, close to the fitted GMM’s 0.78×10−30.78\times10^{-3}. Reported generation-task runtimes are 12.90 and 37.24 seconds, respectively. This supports the refinement strategy, separately from the unrefined model’s much shorter 0.06-second runtime.
Critical weaknesses
  1. The treatment of SCM likelihoods needs clarification. Equation 11 and Table 5 appear to describe a fully observed DAG with additive Gaussian noise. If disturbances are mutually independent, the joint density factorizes into tractable Gaussian conditionals, and scores are computable. Yet Section 3.5 invokes likelihood intractability to motivate MMD refinement, and the synthetic comparisons against analytical ground-truth densities and scores omit SCMs. An unreported modeling assumption could explain this; the current specification does not.
  2. Table 20’s NLL aggregates do not match its dimensional breakdowns. Every PRODiGI-FT group has positive NLL, ranging from 0.58 to 31.75, while the overall value is −0.22, also reported in Table 2. The overall rank of 1.67 is better than every group’s average rank. These cannot be ordinary weighted averages of the displayed means. The cause may be a reporting or aggregation mismatch, but the NLL comparison needs reconciliation before it can be interpreted.
  3. The comparison with earlier amortized parameter inference is incomplete. Lee et al.’s Set Transformer directly predicts GMM parameters from datasets and evaluates subsequent EM refinement in Section 5.3 and Table 3. PRODiGI extends this direction through multiple families, variable templates, and a unified representation. A matched GMM parameter-prediction baseline would help establish the benefit of the proposed formulation beyond direct parameter inference followed by refinement.
Questions for authors
  1. What prevents exact likelihood and score evaluation for Equation 11? Are any variables latent or disturbances dependent in the implementation?
  2. How are Table 20’s overall NLL and rank computed, and which comparisons would change after reconciling them with the dimensional breakdowns?
  3. On the GMM subset, how does PRODiGI compare with direct Set Transformer parameter prediction using matched training data and refinement budgets?
3 comments

We sincerely thank the reviewer for their careful assessment and for recognizing the value of reusable, explicit generative programs, the effectiveness of our architectural choices, and the accuracy--runtime tradeoff achieved through refinement. We particularly appreciate the observations regarding analytical SCM likelihoods, inconsistencies in the reported NLL results, and the need for a direct comparison with Set Transformer. We address each point below, with corresponding corrections and additions highlighted in blue in the updated manuscript uploaded to OpenReview.

W1/Q1, SCM densities and scores

We thank the reviewer for identifying this inconsistency. The reviewer is correct: our current SCM formulation permits analytical density and score evaluation. All variables are observed, and the disturbances are independent. Earlier versions of our SCM prior included latent variables and more general transformations, preventing straightforward analytical evaluation. However, we subsequently simplified the prior without updating the corresponding evaluation and fine-tuning procedures. We have now corrected this oversight: SCM densities and scores are computed analytically, SCM datasets are included in the corresponding evaluations, and program-space fine-tuning uses NLL for all three prior families. The updated experiments further improve our in-distribution results, with PRODiGI-FT now achieving the lowest rank across all methods and performance metrics on in-distribution datasets.

Revision. We have updated the manuscript accordingly by adding SCM-based density and score evaluations to each experiment (e.g., Table 2, Figure 3, and the Tables in Appendix G), and by simplifying our fine-tuning procedure to always use an NLL-based loss (Section 3.5).

W2/Q2, NLL aggregation inconsistency

We thank the reviewer for identifying this error. The discrepancy was caused by an incorrect absolute-value operation in our reporting script, affecting the dimensional NLL breakdowns and their average ranks. The overall NLL values and interpretations in the original submission remain correct, but the dimensionality results were reported incorrectly. We have corrected the reporting script and updated the corresponding tables in the revised manuscript. Additionally, following the reviewer's observation in W1/Q1, we now include analytical density and score evaluations for SCM datasets, which also changes the reported aggregate results. The original overall comparisons were unaffected by the reporting error, and the corrected evaluation does not change our main conclusions.

Revision. We have updated tables 27-30 (previously 20-22) on pages 50-53 with correct per-dimension numbers.

W3/Q3, Comparison to Set Transformer

We agree that Set Transformer (Lee et al., 2019) is an important predecessor for amortized GMM parameter inference, and we will clarify this connection in the revised paper. While our architecture also employs a Set Transformer encoder, our main contribution is the template-based non-autoregressive decoder, enabling variable structures and multiple generative families. In contrast, the original Set Transformer predicts GMM parameters only for a fixed dimensionality and fixed component count. We have added a reference to Set Transformer to our related work.

To directly address the requested comparison, we implemented the Set Transformer GMM parameter predictor following Lee et al., trained on our 2D GMM prior with a fixed 10-component output. We compare it against PRODiGI and a 2D-specialized version, both before and after refinement.

Without fine-tuning (zero-shot):

DataMetricPRODiGIPRODiGI-2DSet Transformer
GMM (32)MMD x10^3 ↓21.4512.4242.62
Spearman ↑0.460.590.50
Score cosine ↑0.300.460.40
Copula (16)MMD x10^3 ↓5.520.3427.34
Spearman ↑0.750.990.54
Score cosine ↑0.690.960.44
SCM (16)MMD x10^3 ↓25.6311.7752.29
Spearman ↑0.790.840.30
Score cosine ↑0.740.780.03

With refinement:

DataMetricPRODiGI-2D-FTSet Transformer + EM
GMM (32)MMD x10^3 ↓0.740.30
Spearman ↑0.950.95
Score cosine ↑0.870.89
Copula (16)MMD x10^3 ↓0.720.66
Spearman ↑0.990.86
Score cosine ↑0.970.68
SCM (16)MMD x10^3 ↓0.820.58
Spearman ↑0.980.80
Score cosine ↑0.960.28

Refinement budgets: Set Transformer + EM is optimized until convergence (scikit-learn default tolerance of (10^{-3})), whereas PRODiGI-2D-FT uses 300 Adam steps with a learning rate of 0.01. EM is substantially faster than PRODiGI's program-space refinement. Thus, the comparison uses the respective refinement procedures rather than matched computational budgets.

On GMM data, PRODiGI-2D achieves 71% lower zero-shot MMD than Set Transformer, alongside improved density ranking and score direction. Following refinement, Set Transformer + EM achieves lower MMD (0.30 vs. 0.74 ×10⁻³) and converges substantially faster, while density ranking and score accuracy are comparable. Beyond GMMs, PRODiGI-2D-FT achieves substantially higher density-ranking and score-direction accuracy on Copula and SCM datasets, although Set Transformer + EM achieves slightly lower MMD.

These results support our contribution of extending amortized parameter inference to heterogeneous, variable-structure generative programs, while also demonstrating improved zero-shot performance in the original GMM setting. We appreciate the reviewer’s careful assessment, which helped us identify and correct two important issues in the original submission and further improve our results. We believe the revised methodology, corrected results, and direct Set Transformer comparison now address the central concerns raised in the review. We would be grateful if you would reconsider your initial rating in light of these changes, and we welcome any further questions or remaining concerns.

Revision. We have added a qualitative comparison to Set-Transformer in the related work and a quantitative one in Appendix F.7.

Dear Reviewer,

We wanted to kindly follow up on our rebuttal. We would be happy to hear your thoughts and clarify anything that may still be unclear.

Thank you again for your time and feedback!

Reviewer_YMvpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a Prodigl approach for predicting generative programs from a numerical dataset. Their approach trains a neural network to predict templates from the GMM, SCM, and copula families, along with their parameters. They also refine these predicted parameters

Critical strengths
  • Their method shows fast zero generation. The refinement steps show good improvement
  • The paper mixes synthetic and real-world evaluations, with a good breakdown of results and extensive ablations
  • Their method unifies multiple model families within a single architecture.
Critical weaknesses
  • Not sure about novelty. The Set Transformer paper already predicts the GMM parameters from the dataset and subsequent refinement. The method seems to have more overlap than discussed in the paper
  • NNKDE-FT achieves lower mean generation MMD than PRODiGI-FT while being approximately 8.6× faster
  • Comparisons combine different model-selection and inference procedures, while PRODiGI’s refinement budget and the meaning of reported uncertainty are insufficiently specified.
  • Real-world density/score evaluation uses GMM-derived proxy ground truth and excludes SCM outputs. Refinement improves synthetic score-error rank but worsens mean score MAE.
Questions for authors
  • As mentioned in the weaknesses section Set Transformer already does similar GMM parameter prediction and refinement; what do the proposed program representation and decoder add? Also, did the authors compare against it on the GMM task?
  • Given that NNKDE-FT has better mean MMD and lower runtime, and GMM has substantially better MMD, which datasets or operating conditions favor PRODiGI? Can complementary distribution metrics and per-dataset results clarify this tradeoff?
  • Could you specify the refinement hyperparameters, timing boundaries, and held-out evaluation splits?
  • How should the density/score claims be qualified given the GMM proxy ground truth and exclusion of SCM templates, and what explains the deterioration in mean score MAE after refinement?
8 comments

We sincerely thank the reviewer for their constructive feedback and for recognizing PRODiGI's fast zero-shot generation, substantial improvements through refinement, unified treatment of multiple generative families, and extensive experimental evaluation. We also appreciate the questions regarding the connection to Set Transformer, favorable operating conditions, evaluation protocols, and the interpretation of density and score results. We address each point below, with corresponding revisions highlighted in blue in the updated manuscript uploaded to OpenReview.

W1/Q1, Set Transformer: We thank the reviewer for highlighting the connection to Set Transformer (Lee et al., 2019). While we also employ a Set Transformer encoder, our main contribution is the template-based non-autoregressive decoder, enabling a single model to predict GMM, SCM, and Copula programs across varying dimensionalities and structures. In contrast, the original Set Transformer predicts GMM parameters only for a fixed dimensionality and fixed component count. We added this reference to our related work.

New experiments: We implemented a Set Transformer GMM predictor following Lee et al., trained on our 2D GMM prior with a fixed 10-component output. We compare against PRODiGI and PRODiGI-2D (trained exclusively on 1D and 2D data), both before and after refinement.

Without fine-tuning (zero-shot):

DataMetricPRODiGIPRODiGI-2DSet Transformer
GMM (32)MMD x10^3 ↓21.4512.4242.62
Spearman ↑0.460.590.50
Score cosine ↑0.300.460.40
Copula (16)MMD x10^3 ↓5.520.3427.34
Spearman ↑0.750.990.54
Score cosine ↑0.690.960.44
SCM (16)MMD x10^3 ↓25.6311.7752.29
Spearman ↑0.790.840.30
Score cosine ↑0.740.780.03

With refinement:

DataMetricPRODiGI-2D-FTSet Transformer + EM
GMM (32)MMD x10^3 ↓0.740.30
Spearman ↑0.950.95
Score cosine ↑0.870.89
Copula (16)MMD x10^3 ↓0.720.66
Spearman ↑0.990.86
Score cosine ↑0.970.68
SCM (16)MMD x10^3 ↓0.820.58
Spearman ↑0.980.80
Score cosine ↑0.960.28

EM is run until convergence (scikit-learn default tolerance of 1e-3) and is substantially faster than PRODiGI's program-space refinement. PRODiGI-2D achieves 71% lower zero-shot MMD on GMM data than Set Transformer, alongside improved density ranking and score direction. After refinement, Set Transformer + EM achieves lower MMD, while density and score accuracy are comparable on GMMs. On Copula and SCM datasets, PRODiGI achieves substantially higher density-ranking and score-direction accuracy, although EM yields slightly lower MMD after refinement.

Revision. These changes are now reflected in Appendix F.7.

W2/Q2, Operating conditions favoring PRODiGI We agree that NNKDE-FT achieves lower mean MMD and runtime than PRODiGI-FT on real-world datasets. However, our additional analyses reveal substantial differences across operating conditions, particularly between in-distribution, out-of-distribution, and real-world data.

In-distribution GMM data: On data generated from our GMM prior, PRODiGI-FT achieves substantially lower MMD than conventional GMM fitting (0.39 vs. 1.60 ×10⁻³) while being 5.6× faster (5.82 vs. 32.77 seconds). NNKDE-FT obtains a higher MMD of 13.43 ×10⁻³. Notably, PRODiGI uses only diagonal covariance matrices, whereas the fitted GMM baseline uses more expressive full covariance matrices (Appendix E.3). This shows that learned initialization followed by program-space refinement can be highly effective even relative to a more expressive, conventionally fitted model. We do not claim that this advantage necessarily extends to arbitrary GMM datasets or fitting procedures.

Out-of-distribution data: We additionally evaluate 96 synthetic OOD datasets with unseen generating mechanisms (Appendix F.5, Table 10). PRODiGI-FT substantially outperforms NNKDE-FT in MMD (1.82 vs. 7.98 ×10⁻³), average generation rank (3.31 vs. 5.59), and two-sample rejection rate (34.7% vs. 75.8%). Fitted GMMs achieve lower MMD (1.35 ×10⁻³). These results demonstrate that PRODiGI remains competitive under controlled departures from its training prior, despite its limited program vocabulary.

Real-world operating conditions: To investigate where PRODiGI-FT becomes more competitive, we conduct an exploratory analysis of the 96 real-world datasets. We identify the quartile with the highest fitted-GMM MMD within each dimensionality group, avoiding a partition driven solely by dimensionality.

MetricPoor GMM fit (24)Other datasets (72)
GMM MMD ×10³ ↓46.76.1
PRODiGI-FT MMD ×10³ ↓62.328.1
NNKDE-FT MMD ×10³ ↓73.417.0
PRODiGI-FT wins vs. NNKDE-FT13/24 (54%)20/72 (28%)

In the poorly fitted subset, PRODiGI-FT achieves 15% lower mean MMD than NNKDE-FT, whereas NNKDE-FT performs considerably better on the remaining datasets. PRODiGI-FT's per-dataset win rate also increases from 28% to 54%. Although this is an exploratory stratification rather than evidence of a universally favorable regime, it suggests that PRODiGI becomes relatively more competitive when conventional GMMs fit the data poorly. Importantly, fitted GMMs still achieve lower mean MMD in both subsets. Their full-covariance representation provides greater flexibility than PRODiGI's GMM templates, making representational restrictions one plausible contributor to the real-world performance gap. However, these results do not isolate expressiveness as its sole cause.

Runtime tradeoff: Without refinement, PRODiGI is both faster (0.06 vs. 0.10 seconds) and more accurate (68.53 vs. 79.31 MMD ×10⁻³) than NNKDE on real-world datasets. Compared with NNKDE-FT, zero-shot PRODiGI is approximately 19× faster, although less accurate. With refinement, PRODiGI-FT is slower than NNKDE-FT (9.64 vs. 1.12 seconds) but remains faster than fitted GMMs (17.48 seconds).

Overall, these results identify complementary strengths rather than uniform dominance: PRODiGI excels at extremely fast zero-shot generation and accurate inference on its supported priors, while PRODiGI-FT remains competitive on the evaluated synthetic OOD distributions. On real-world datasets, its relative performance improves in the subset poorly approximated by GMMs, but important accuracy limitations remain. Expanding the supported program vocabulary is therefore a promising direction for improving real-world performance.

Our work provides an initial proof of concept, demonstrating the promise of amortized program inference and opening the door for future work to improve its expressiveness and broaden its applicability.

Revision. The revised manuscript contains the additional OOD experiments in Section 4.1 and Appendix F.5.

W3/Q3: Thank you for the suggestion. We clarify the experimental protocol below.

Refinement: PRODiGI uses 300 Adam steps (learning rate 0.01), minimizing the NLL of each input observation once per step.

Evaluation: Refinement uses only training data; separate fixed test sets evaluate generation MMD, densities, and scores. For real-world data, we use OddBench's train/test splits. Uncertainties are standard errors of the mean (SEM) across datasets.

Runtime: Reported runtimes include preprocessing, model inference, refinement (if applicable), and generation or evaluation of 1,000 samples. GPU execution is synchronized for accurate timing.

Refinement budget: We evaluate different refinement budgets on 96 synthetic datasets. PRODiGI improves with additional optimization but exhibits diminishing returns: increasing from 300 to 1,000 steps reduces MMD from 0.56 to 0.45 ×10⁻³ (20%) while tripling runtime (12.90 to 38.67 seconds). We therefore choose 300 steps. NNKDE largely converges by 100 steps, while additional DiScoFormer refinement does not consistently improve generation quality. DiScoFormer uses epochs, whereas PRODiGI and NNKDE use optimization steps.

MethodRefinement budgetMMD ×10³ ↓Runtime (s) ↓
PRODiGI10 steps2.680.61
PRODiGI100 steps0.804.47
PRODiGI300 steps0.5612.90
PRODiGI1000 steps0.4538.67
NNKDE100 steps11.060.24
NNKDE1000 steps10.981.20
DiScoFormer3 epochs39.2979.39
DiScoFormer10 epochs67.5979.44

Revision. Section 4.1, Figure 4 (right), now analyzes refinement budgets.

W4/Q4, Real-world ground truth and score MAE deterioration

GMM proxy ground truth: Since real-world ground-truth densities and scores are unavailable, we approximate them using fitted GMMs. To limit approximation error, we retain only 37 datasets where a two-sample test does not reject equality between GMM-generated and real samples. These remain imperfect proxies, potentially favoring GMM-based estimators (GMM, KDE, NNKDE) and GMM-trained models (DiScoFormer, NNKDE). We therefore report real-world density/score results only in the appendix and base our main conclusions on synthetic datasets with analytical ground truth. As an alternative, Appendix F.5 evaluates synthetic OOD distributions with mechanisms outside our training vocabulary. PRODiGI's performance deteriorates but remains competitive.

SCM templates were previously excluded from real-world density/score evaluation due to presumed analytical intractability. Following Reviewer qjcc's observation, we corrected this oversight and now support analytical SCM densities and scores. The real-world proxy results remain unchanged because none of the 37 selected datasets receive SCM predictions.

Score MAE deterioration: On 96 synthetic datasets, refinement improves score MAE on 72 datasets, compared with 17 losses (7 ties). Median MAE decreases from 14.98 to 4.59, and average rank improves from 4.89 to 2.32.

The increase in mean MAE (94.80 → 232.00) is driven by three extreme datasets, contributing 158.9 to the overall mean difference of 137.2. Excluding them, mean MAE decreases from 68.68 to 46.25.

Highly concentrated distributions can produce extreme score magnitudes, making absolute errors sensitive to small parameter deviations across program families. Cosine similarity, which measures directional agreement independently of magnitude, improves from 0.589 to 0.784.

On the 37 real-world proxy datasets, mean score MAE is nearly unchanged (5,810 vs. 5,831), while average rank improves (5.68 → 4.68) and refinement wins on 21 versus 14 datasets (2 ties).

Thus, the higher synthetic mean MAE reflects a few extreme cases rather than a general deterioration. Median MAE, ranks, per-dataset comparisons, and cosine similarity indicate improved typical performance.

Revision. The revised manuscript includes updated results and additional OOD experiments (Appendix F.5).

Given these additional experiments and revisions, we hope you might reconsider your rating. We would welcome the opportunity to address any remaining concerns during the discussion period.

Comments on the paper

2 comments

We sincerely thank all four reviewers and the Area Chair for their time and constructive feedback. We have made substantial revisions and conducted extensive additional experiments to address the reviewers' concerns. Reviewer YMvp raised their score from 2 to 3, stating that our clarifications addressed their remaining questions. Reviewer iRRj expressed appreciation for our substantial additional experiments, particularly the helpful new comparisons and analyses addressing their concerns.

TL;DR. PRODiGI introduces a new direction for tabular generative modeling: instead of fitting a separate density estimator to each dataset, a pretrained model directly infers an explicit, executable generative program. A single model supports GMMs, SCMs, and Copulas across different dimensionalities and structures. The inferred programs can be inspected, stored, sampled, and evaluated independently of the neural network. Our experiments establish the effectiveness of fast amortized program inference, while program-space refinement provides additional accuracy when needed.

Strengths highlighted by the reviewers.

  • A useful and novel direction: Reviewer tREQ described learning generative programs as reconstruction targets as "a simple and effective idea." Reviewer iRRj called the approach "interesting" and "a useful idea."
  • Explicit and reusable programs: Reviewer qjcc highlighted that the inferred structures and parameters can be retained independently of the network and observations, making them useful for inspection, storage, repeated sampling, and statistical queries. Reviewer YMvp appreciates the unified treatment of multiple generative families.
  • Well-motivated architecture: Reviewer tREQ found our design choices "well motivated," highlighting the separation of program templates and parameters. Reviewer qjcc recognized the 100% syntactic validity and 44× faster non-autoregressive decoding of our NAR decoding approach.
  • Strong empirical evaluation: Reviewer iRRj highlighted our "Good experimental evaluation" and "very fast zero-shot generation," noting that refinement "significantly improves generation quality and density estimation accuracy." Reviewer YMvp similarly recognized the extensive experiments, fast generation, and effectiveness of refinement.

Key additions during the discussion.

We substantially expanded the evaluation with new Set Transformer and LLM-based baselines, controlled initialization comparisons, quantitative program recovery, 96 additional OOD datasets, and further refinement analyses. We also corrected methodological and reporting issues identified by the reviewers. All changes are highlighted in blue in the revised 10-page manuscript, with detailed results in the appendix.

List of reviewer requests and our new results:

ReviewerRequestNew results and revisionsDetails
qjcc, YMvpAnalytical SCM densities and scores; evaluation correctnessWe expanded the SCM evaluation, increasing the synthetic density/score benchmark from 64 to 96 datasets. We now use analytical SCM likelihoods and NLL refinement across all three families. We also corrected dimensional NLL reporting inconsistencies identified by qjcc.Revised Section 3.5, Table 2, Figure 3, Appendix G, Tables 27–30; responses to qjcc
qjcc, YMvpDirect comparison with Set TransformerWe implemented the original Set Transformer GMM predictor with EM refinement. PRODiGI-2D achieves 71% lower zero-shot GMM MMD (12.42 vs. 42.62 ×10⁻³), while Set Transformer + EM achieves better refined GMM MMD (0.30 vs. 0.74 ×10⁻³). This further shows the distinction between specialized GMM fitting and our heterogeneous, variable-structure program inference.Revised Related Work, Appendix F.7; responses to qjcc and YMvp
tREQComparison against LLM-based program inferenceWe introduced an LLM-based program-proposal baseline with iterative validation, evaluated on 192 synthetic and real-world datasets. LLM-generated programs achieve better generation MMD, but mainly reproduce GMM results and require approximately 12.5× more runtime than PRODiGI-FT and 2,000–2,700× more than zero-shot PRODiGI in addition to token cost. PRODiGI-FT also achieves higher in-distribution score cosine (0.780 vs. 0.652).Revised Appendix F.6; response to tREQ

CONTINUED IN THE NEXT POST

CONTINUED

ReviewerRequestNew results and revisionsDetails
iRRjIs a learned initialization better than random or classical initialization?We added three random initialization baselines under identical refinement budgets. Even with the same predicted template, PRODiGI initialization reduces in-distribution MMD from 3.07 to 0.56 ×10⁻³, winning on 70/96 datasets. We additionally compare with k-means initialization, identifying both the strengths of classical GMM fitting and the value of PRODiGI's structural predictions.Revised Appendix F.4; response to iRRj, Q1
iRRjQuantitative program and SCM graph recoveryWe added quantitative reconstruction experiments. PRODiGI predicts the correct program family in 743/744 cases (99.9%). For SCMs, directed edge F1 improves from 0.39 to 0.59 over a random-graph baseline, with 47% lower structural Hamming distance. We additionally report activation, structural, and continuous parameter recovery.Revised Figure 4 (left), Appendix F.2; response to iRRj, Q2
iRRjFamily-wise effectiveness of refinementWe added separate GMM, SCM, and Copula analyses and compared NLL and MMD refinement for SCMs. While MMD gives slightly better SCM generation MMD, NLL improves density Spearman from 0.78 to 0.89 and score cosine from 0.73 to 0.84.
iRRj, YMvpOut-of-distribution generalizationWe introduced 96 OOD datasets with unseen mixture component counts, SCM activation functions, and Copula marginals. PRODiGI-FT achieves lower OOD generation MMD than NNKDE-FT (1.82 vs. 7.98 ×10⁻³) and KDE (1.82 vs. 3.22 ×10⁻³), while fitted GMMs and TabPFN perform better with more compute. Our OOD results show the limitations of our program vocabulary and invite future work.Revised Section 4.1, Appendix F.5; responses to iRRj and YMvp
YMvpRefinement budgets, runtime, and real-world limitationsWe added refinement-budget ablations (10–1,000 steps), clarified timing and evaluation protocols, investigated outliers in score MAE, and analyzed on which subsets of real-world datasets where PRODiGI performs best.Revised Figure 4 (right), Appendix F.5; responses to YMvp

Why PRODiGI?

  • A different approach to generative modeling. Rather than predicting a density implicitly or fitting a specialized model from scratch, PRODiGI amortizes the inference of an entire executable generative program. This combines learned inference with explicit and inspectable representations.
  • Generality across structures and tasks. A unified model handles three distinct generative families, varying dimensionalities, and variable program structures. The resulting programs support generation, density estimation, and score estimation, rather than just one task.
  • Fast inference without dataset-specific optimization. Zero-shot PRODiGI generates 1,000 samples in 0.06 seconds on average, while optional program-space refinement improves accuracy. Unlike LLM-based program proposal, inference requires neither repeated language-model calls nor execution of newly generated fitting procedures.
  • Learned initialization and structure prediction both matter. The new random-initialization experiments show that PRODiGI's predictions contribute beyond the subsequent optimization procedure. Classical initialization can be particularly effective for GMMs, but our model provides a unified mechanism for predicting programs across different families.

Discussion outcome.

Following these revisions, Reviewer YMvp increased their rating from 2 (weak rejection) to 3 (weak acceptance), and Reviewer iRRj acknowledged the value of our additional experiments and clarifications.

Reviewers qjcc and tREQ have not yet engaged in the discussion, despite our detailed responses and follow-up. In particular, we directly addressed qjcc's concerns through corrected SCM likelihoods, reconciled NLL reporting, and a Set Transformer comparison, and tREQ's central concern through a new LLM-based program inference baseline. We therefore have not had the opportunity to learn whether these additions resolve their remaining concerns.

We would be grateful if the Area Chair could take these substantial additional experiments and corrections into account in the final assessment, including those addressing concerns for which we have not yet received reviewer feedback.

We sincerely thank the reviewers again for their constructive input, which has helped us substantially strengthen both the paper and its empirical evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.