- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper describes new intervention methods (basically, overwriting in the synthetic data generation process) to improve training of prior-data fitted networks (PFN).
PFNs are the state of the art in tabular data modeling and improvements are highly relevant.
While the results are probably very good and highly important, the paper is not accessible to the readers in its present form. It reads like an internal note accessible only to those very closely working in the area. As a reader with only general knowledge on PFNs I had to query term by term all notions in the Abstract and the Introduction, and I would still not be able to reproduce the algorithm of Section 3. I will expand in the Questions.
The reproducibility statement and Appendix H refers to a reviewer source code package but I find neither supplementary material in the submission nor a link to an anonymous repository in the paper.
Overall, I find the results probably very valuable, but the paper not yet ready for publication.
Critical strengths
PFNs are the state-of-the-art in tabular data modeling, outperforming all competitors in almost all aspects. Hence improved PFN training is a very important contribution.
Critical weaknesses
The paper reads like a highly technical internal note to a developer team. As a reviewer with general knowledge only, I had to search term by term and still cannot reproduce the intervention algorithm for myself. In its current form, the paper is not accessible for the conference audience.
Questions for authors
Please include definitions of
- Structural equation models - just a sentence that this is the process of generating the synthetic training data
- What is an intervention, how does it modify training the foundation model
- Instead of four-arm, more accessible to say four ways of intervening/overwirting(?)
- Say at least once KL is Kullback-Leibler (I hope so)
- Signed, Fresh, without explanation are just arbitrary names of Q+R and Q+A+R. And in the Introduction, we don't yet know what is the definition of q_Z. The meaning of the sentence "Signed supplies an experimental counterfactual" still remains unclear to me.
Please revise Section 3, clearly listing the notation and the algorithm. When reading H=(D,x) starting with "20 complete 3-variable ..." is it a general definition or an illustration with parameters 20 and 3? Is it a 2-dimensional synthetic data plus a label? Why 2-dimensional and why 20 observations? Later in my understanding, q, Q are used as general definitions.
Please also give a clear single-chart summary on what is the baseline training procedure, I assume using synthetic data generated by broadly described in Table 4 but I would like to see a more precise statement.