- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a diffusion-based method (DiffPFN), an in-context method for tabular data generation. Similar to prior works, it trains on synthetic datasets generated by structural causal models. It generates synthetic tables (query rows) in a zero shot manner by conditioning on context rows (observed table cells). The architecture uses a continuous diffusion backbone and an axial transformer to handle contextual information and varying row and column dimensions on the fly. The empirical results are strong compared to baseline, especially showing huge speedups over methods that train a model per dataset. Nevertheless, I have several key concerns regarding the clarity, design choices, novelty, and analysis of results, prompting me to give this work a weak reject.
Critical strengths
-
The approach provides zero-shot adaptability, which is very useful in practice.
-
The idea of extending diffusion models to foundation-scale capabilities is ambitious and relevant.
-
The empirical results are promising.
Critical weaknesses
-
The core of the work rests on the architecture, i.e., how the design is able to handle varying feature and row sizes, and how it handles contextual information on the fly. The axial transformer is a key contributor towards this objective. However, the technical description in Sec 3.5 and Figure 2 are not detailed enough. There is a lack of precise mathematical formulations (and conceptual illustrations) for the feature-wise and row-wise attention operations, including the tensor dimension transformations, and formal definition of the attention mask.
-
The proposed work seems to heavily borrow from existing works (e.g. TabPFN, TabICL, DiffICL), but the novelty over these methods is not very well described. The work appears to mostly be an amalgamation of existing works. For example, the idea of using diffusion with in-context learning is already in DiffICL. Using PFN and axial transformers is also borrowed from TabPFN.
-
There are several unjustified design choices. For example, why use a continuous backbone instead of well-established mixed tabular diffusion backbones (TabDDPM, TabDiff) for the foundation model architecture? Why circular encoding and not standard one-hot encoding or simple ordinal (integer) encoding? To me, it appears that these design choices are implicitly made due to the type of synthetic data the model was pre-trained on. However, this is unclear.
-
In the results, we see that scaling the context sizes does not really improve performance. I find this surprising, yet it is not very well analysed in the manuscript. It is unclear if the lack of performance gains is due to the method's inability to leverage larger row contexts, or something else.
-
Appendix A1 (Table 2) lists the details of the prior sampling parameters. It is unclear if the proposed method can handle feature cardinalities that are greater than what it is trained on in a zero-shot manner. I say this because Table 4 in Appendix B1 shows that all the tested datasets have a cardinality that falls within the range the network was pretrained on (2-200). For example, the Churn dataset [1], has 2000+ categories and I'm not sure if the method can generalise to that.
-
In addition to the point above, Appendix A1 lacks more details on the types of functions and value distributions used in the SCM graph. It is unclear if these graphs use continuous representations and whether this is what forces the proposed design to use a continuous diffusion architecture.
-
The results focus on the training time costs on new datasets. However, the proposed method incurs hidden training costs because of training on the synthetic prior. It is not clear what these runtime costs amount to since they are not listed.
-
The background on prior fitted networks and in-context learning needs to be explained in more detail
[1] Churn modelling dataset: https://www.kaggle.com/datasets/shrutimechlearn/churn-modelling
Questions for authors
Please see the weaknesses W1-W7
We have decided to withdraw our submission to allow for further development and evaluation of the work. We appreciate the reviewers' time and thoughtful feedback, which will help guide our revisions