- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper demonstrates a systematic gap in descriptive statistics of graphs from real world data generating processes from those sampled from SCM priors used in training tabular foundation models. To improve performance of TFMs on real world graphs, a new synthetic data generator is proposed, replacing SCM generation via BN from pruning MLPs to a mixture of other classes of graphs (e.g. Erdos-Renyi or scale-free graphs). Proposal is evaluated empirically on held out real world datasets.
Critical strengths
The motivation and idea are sound. I appreciate the need for a more systematic study (in public) of synthetic data generation for the purpose of training TFMs.
Critical weaknesses
The paper feels currently quite underdeveloped. Specifically:
- Insufficient theoretical depth
- Graph G (or scm S) induces observed distribution P which is the actual input into a TFM. This should be elaborated upon and investigated (e.g. different complexity S can induce similar if not same observed distribution)
- The theoretical insight as to why matching these graph properties to real world data should improved performance is not developed in detail (it does intuitively make sense)
- representativeness of the reference graphs is not well established
- Empirical investigation is shallow
- Mixture of five graph families is given without ablation or analysis of individual component contribution. In other words the potential of the paper's own method is not systematically investigated
- The performance improvement seems barely statistically significant.
- Anonymized code repo supplement would be highly appreciated (while not required by conference, I see it as an established good scientific standard) for reproducibility and checkability of work.
- Presentation is excessively long in details, particularly Sections 7-8. Stronger prioritization of key findings and overall maintaining prose with a narrative thread throughout the paper would make it more readable. More thorough treatment of related work would also be appropriate.
In summary, the work in its current form does not do justice to an interesting research direction that it proposes. Better theoretical treatment and better accompanying empirical evaluation are necessary before I can see this is a valuable contribution. Considering the idea is interesting I am open to increasing my score but conditional on addressing the current significant insufficiencies of the manuscript.
Questions for authors
Can the authors:
- establish deeper theory, including but not limited to treatment of the observed distribution P?
- provide better experiments including ablations, possibly more seeds (than the current one) and show a more clearly superior result following their method?
- elaborate on the robustness of their results?
- provide an anonymized code repository?