acceptodds
Withdrawn from ICLR 2027

Fast In-Context Tabular Diffusion via Synthetic Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models for tabular data generation require training on each new dataset, making their use on new datasets computationally expensive. We introduce DiffPFN, an in-context diffusion model that generates tabular data without dataset-specific parameter updates. Pretrained on synthetic datasets drawn from a structural causal model prior, DiffPFN conditions its denoising process on observed training examples to generate samples with numerical and categorical features. Its transformer architecture jointly denoises all features, enabling a single pretrained model to operate across datasets with varying numbers and types of features. Experiments on eight datasets show competitive data fidelity, with a – speedup over diffusion baselines in total training and sampling time for new datasets. When fine-tuned, the pretrained model achieves higher downstream performance than a model trained from scratch at every budget up to 1,000 steps.

Primary area:
generative models
Keywords:
Tabular diffusion, Prior-fitted networks, Foundation models, Synthetic tabular data

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_64pxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a diffusion-based method (DiffPFN), an in-context method for tabular data generation. Similar to prior works, it trains on synthetic datasets generated by structural causal models. It generates synthetic tables (query rows) in a zero shot manner by conditioning on context rows (observed table cells). The architecture uses a continuous diffusion backbone and an axial transformer to handle contextual information and varying row and column dimensions on the fly. The empirical results are strong compared to baseline, especially showing huge speedups over methods that train a model per dataset. Nevertheless, I have several key concerns regarding the clarity, design choices, novelty, and analysis of results, prompting me to give this work a weak reject.

Critical strengths
  1. The approach provides zero-shot adaptability, which is very useful in practice.

  2. The idea of extending diffusion models to foundation-scale capabilities is ambitious and relevant.

  3. The empirical results are promising.

Critical weaknesses
  1. The core of the work rests on the architecture, i.e., how the design is able to handle varying feature and row sizes, and how it handles contextual information on the fly. The axial transformer is a key contributor towards this objective. However, the technical description in Sec 3.5 and Figure 2 are not detailed enough. There is a lack of precise mathematical formulations (and conceptual illustrations) for the feature-wise and row-wise attention operations, including the tensor dimension transformations, and formal definition of the attention mask.

  2. The proposed work seems to heavily borrow from existing works (e.g. TabPFN, TabICL, DiffICL), but the novelty over these methods is not very well described. The work appears to mostly be an amalgamation of existing works. For example, the idea of using diffusion with in-context learning is already in DiffICL. Using PFN and axial transformers is also borrowed from TabPFN.

  3. There are several unjustified design choices. For example, why use a continuous backbone instead of well-established mixed tabular diffusion backbones (TabDDPM, TabDiff) for the foundation model architecture? Why circular encoding and not standard one-hot encoding or simple ordinal (integer) encoding? To me, it appears that these design choices are implicitly made due to the type of synthetic data the model was pre-trained on. However, this is unclear.

  4. In the results, we see that scaling the context sizes does not really improve performance. I find this surprising, yet it is not very well analysed in the manuscript. It is unclear if the lack of performance gains is due to the method's inability to leverage larger row contexts, or something else.

  5. Appendix A1 (Table 2) lists the details of the prior sampling parameters. It is unclear if the proposed method can handle feature cardinalities that are greater than what it is trained on in a zero-shot manner. I say this because Table 4 in Appendix B1 shows that all the tested datasets have a cardinality that falls within the range the network was pretrained on (2-200). For example, the Churn dataset [1], has 2000+ categories and I'm not sure if the method can generalise to that.

  6. In addition to the point above, Appendix A1 lacks more details on the types of functions and value distributions used in the SCM graph. It is unclear if these graphs use continuous representations and whether this is what forces the proposed design to use a continuous diffusion architecture.

  7. The results focus on the training time costs on new datasets. However, the proposed method incurs hidden training costs because of training on the synthetic prior. It is not clear what these runtime costs amount to since they are not listed.

  8. The background on prior fitted networks and in-context learning needs to be explained in more detail

[1] Churn modelling dataset: https://www.kaggle.com/datasets/shrutimechlearn/churn-modelling

Questions for authors

Please see the weaknesses W1-W7

Reviewer_dYJDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

DiffPFN is an in-context diffusion model for tabular data generation that eliminates per-dataset training. DiffPFN is pretrained once on synthetic datasets sampled from a SCM prior, and at inference it conditions the denoising process on the observed training examples without any parameter updates. It unifies categorical and continuous features through a circular representation, adopts velocity parameterization, and employs an axial transformer that jointly denoises all features, so a single pretrained model can handle datasets with varying numbers and types of features. The authors evaluate DiffPFN on eight datasets in terms of data fidelity, downstream task performance, and privacy. Compared with seven baseline methods, the proposed method achieves competitive fidelity while reducing total training and sampling time by 39 to 104 times relative to per-dataset trained generators.

Critical strengths
  1. DiffPFN enables per-dataset training-free generation with a diffusion model, by conditioning the denoising process on the observed training examples of a new dataset.
  2. Compared with per-dataset trained generators, DiffPFN substantially reduces the total training and inference time.
Critical weaknesses
  1. This work is an incremental combination of PFN-style in-context learning, e.g., TabPFN and TabICL, and existing tabular diffusion models, e.g., TabRep. In particular, the main architecture (axial transformer) of this model is adopted from TabPFN-v2, and the table representation is from TabRep.
  2. Not only TabPFN but many other table foundation models, such as TabICL, can also be used for generation through ad-hoc schemes. A comparison with these TFMs would make this paper more convincing.
  3. TabPFN outperforms DiffPFN on downstream utility, which is considered the most important ability for table generative models, across all datasets (Fig. 5 d), making it unclear why TabPFN should not simply be used when downstream performance is the goal.
  4. Unlike in-context models, generators that require training per dataset depend heavily on hyperparameter search. TabArena, one of the most widely used table benchmark recently, also compares per dataset methods after tuning according to each model's specification. This paper states that each baseline was trained once, so it is unclear whether the baselines are compared at their actual performance. This undermines the validity of the comparisons to baselines.
Questions for authors
  1. Which version of TabPFN was used in the experiments (e.g., TabPFN-2, 2.5, or 3)? Since the performance and the capability of TabPFN differ substantially across versions, please specify it.
  2. TabPFN's unsupervised generation produces columns autoregressively in a fixed order, which imposes an artificial ordering on the table and may prevent it from faithfully capturing the joint distribution. This limitation can be alleviated by using multiple column permutations, and the generated data are known to get closer to the real table as the number of permutations increases. In contrast, DiffPFN denoises all columns jointly without assuming any ordering. Does this order-free generation give DiffPFN a real advantage over TabPFN in terms of data fidelity, and does the advantage persist when TabPFN uses more than three permutations? A comparison between DiffPFN and TabPFN with various numbers of permutations would clarify this.
  3. How does DiffPFN compare with TabPFGen and TabEBM, which leverage TabPFN for generation?
Reviewer_TP63OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Summary: The paper's goal is to adapt the tabular foundation model paradigm to the generation of tabular data, through their proposed model DiffPFN. This is to work towards removing the need for per-dataset training of individual models. They use the synthetic dataset prior from TabICLv2 [1] to pre-train a transformer parameterizing the velocities in a diffusion model. (Potentially heterogeneous) datasets are jointly embedded in a continuous space via the transformations from TabRep [2]. The results show that DiffPFN is the fastest generator, but falls slightly behind per-dataset SOTA in terms of their evaluation metrics.

Contributions: The main contribution is in the architecture and use of the TabICLv2 prior for pre-training such a model.

Validation: They provide a baseline of 8 datasets of varying sizes and 7 reasonable baselines, one of which is TabPFN (this is the most comparable to DiffPFN as a pre-trained foundation model).

[1] TabICLv2: A better, faster, scalable, and open tabular foundation model

[2] TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation

Critical strengths

DiffPFN provides a fast, pre-trained method for generating synthetic tabular data, beating all baselines in terms of time taken to generate the first dataset (after incorporating training times for per-dataset methods).

Critical weaknesses

There are issues with synthetic dataset evaluation, specifically in terms of the metrics used for evaluation.

Firstly, the authors compute C2ST with logistic regression. This metric has been recently criticized for providing little discriminative power, able to be fooled by a simple model [3]. This implies that we can't draw strong conclusions from results of this metric. Moreover, the metric is rather saturated on some datasets (e.g. see the C2ST column of Wine Quality Red in Table 8). This also affects the conclusions which can be drawn from method rankings, since values can be very close. Instead, C2ST should be computed with a stronger classifier like XGBoost, similar to what is done in [3] and [4]. This would make the evaluation of the method much clearer and stronger.

Secondly, the Trend metric (named "PCC" here) has been shown to have issues, as has also been explored by the same recent work [3]. It was shown to be sensitive to quality of univariate marginals, meaning that as a measure of pairwise dependency modeling, it can have failure modes. These limitations should be addressed in the metric descriptions. Using the wNMIS metric proposed in [3] could also help to strengthen the argument around DiffPFN's performance in terms of pairwise dependency modeling.

However, even with these concerns about the metrics, the model performance remains less than convincing, even with finetuning. For example, in Table 6, DiffPFN's rankings are not incredibly convincing, falling somewhat behind current per-dataset SOTA.

As another issue, the paper is missing a key reference and comparison to the very closely related work TabFORGE [5] - TabFORGE also implements a foundation model for generative modeling of tabular data. In comparison to DiffPFN's use of the TabICLv2 prior, TabFORGE reuses pre-trained encoder layers from Real-TabPFN-2.5. Adding a comparison to this method, as well as a section to make it clear how the two different approaches to the problem differ in terms of both architecture and performance, would make the paper much stronger.

[3] A Sobering Look at Tabular Data Generation via Probabilistic Circuits

[4] Continuous Diffusion for Mixed-Type Tabular Data

[5] TabFORGE: A Tabular Foundation Model for Generative Modelling

Questions for authors
  1. What is the justification of selecting a maximum of 1000 rows as context during inference? The main paper has an ablation including 256, 1024, and 4096 rows, but this design choice does not seem well-justified in the text to me.

  2. How is missing data handled in dataset pre-processing? Details are provided in Section B.1 for Beijing, but not for other datasets.

  3. Which version of TabPFN are you using: TabPFN-3 / TabPFN-3.5, or something else? Relatedly, what version of tabpfn-extensions are you using? It is not clear to me from reading the paper, and this could have a significant impact on the performance listed for TabPFN. Listing the version of the extensions package would also help with reproducibility.

  4. What conclusions should we draw from the probing of query representations in Section 4.3? The paper lists some numbers for held-out R^2 without much discussion of the implications. Some prior works have shown that intermediate pre-trained FM encoder layers can encode some structure of the dataset (e.g. [6]) - what does this tell us about DiffPFN in question?

  5. Are you certain that your description of alpha-precision and beta-recall in the appendix are correct? To me, the characterization doesn't seem to agree with the description of these metrics in the original paper [7].

[6] Does TabPFN Understand Causal Structures?

[7] How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

Reviewer_s6R5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiffPFN is pre‑trained on synthetic tabular tasks. With designs including axial Transformer, FiLM time conditioning and reusable context cache, it keeps network weights frozen during inference and only fits pre‑processing mappings. It claims tens to over a hundred‑fold speed‑ups on new tasks compared with baselines. Fine‑tuning can improve certain metrics yet incurs additional overhead.

Critical strengths

1.The work pursues a practically‑valuable goal of eliminating generator retraining for new tables to reduce deployment costs. Its model architecture aligns well with this goal: reusable cached context states enable one set of parameters to adapt to diverse tables. The combination of purely synthetic pre‑training and context‑conditioned diffusion is research‑worthy. 2.The reported speed‑up figures are numerically consistent. Comparisons between pre‑trained training and training from scratch are informative. Negative findings such as flaws of the frozen model and unstable fine‑tuning performance are also openly reported.

Critical weaknesses

1.The 39‑104× speed‑ups depend on specific training setups without proof of advantages under equal generation quality. Frozen DiffPFN underperforms TabPFN on downstream tasks and fine‑tuning brings extra cost. Poor documentation of baseline hardware and timing scope makes speed‑ups hard to verify. 2.Context validation is limited to synthetic tasks without real‑table controls to disentangle component effects. Direct comparison against TabRep is missing, and empirical differences from peer methods are insufficient. 3.Most experiments are trained only once; repetitions come merely from sampling, ignoring fluctuations from training seeds and data splits. Some results originate from one single training run instead of independent repeats. 4.Evaluations are limited to small‑scale tables, lacking tests on high‑dimensional or distribution‑shifted cases. High statistical fidelity coexists with poor downstream performance, yet failure modes are not well analysed.

Questions for authors
  1. Clarify the applicable scenarios of DiffPFN under equal‑time or equal‑quality constraints, and specify experimental hardware settings as well as the scope of timing statistics.
  2. Disentangle speed and performance gains brought by preprocessing, context modeling and individual model modules through real‑table control experiments and ablation studies.
  3. Verify the stability of core results against varying training seeds and dataset splits; explain the discrepancy between high statistical fidelity and poor downstream performance; and illustrate performance‑resource trade‑offs and generalization boundaries as table scale grows.

Comments on the paper

1 comment

We have decided to withdraw our submission to allow for further development and evaluation of the work. We appreciate the reviewers' time and thoughtful feedback, which will help guide our revisions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.