- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a method for synthetic tabular data generation using GANs, and demonstrates that combining this synthetic data with real data results in an improvement in performance on a particular clinical application (detecting dental implant failures). For instance, it raises the AUC of their baseline method from 0.81 to 0.89 with 50% synthetic examples.
Critical strengths
The prose is genuinely well-written and the paper is easy to follow, in contrast to many of the AI-slop papers in my reviewing batch. Kudos to the authors for writing their paper the good-old-fashioned way, or at least making it appear that way :)
Overall, the paper presents a clearly-explained approach to the problem, and conducts a reasonably thorough evaluation of their approach (with plenty of well-motivated baselines) on this particular dataset. I appreciated the care that went into designing the synthetic data generation approach to avoid potential leakage (via cross-validation, which seems computationally expensive, but at least principled), and Figure 2 presents a rare sight: An ROC curve that shows different trade-offs at different levels (e.g., for a very low FPR, the baseline might be a better choice, whereas their method works better in the high-recall regime).
Critical weaknesses
The obvious limitations of this paper are the narrow focus and scope, showing improvement in performance on a specific clinical dataset. While the method is fairly general, it is unclear how we should expect the method to generalize to other tabular prediction tasks. The authors may want to consider using TabArena or other compilations of tabular prediction tasks to see if their method works more generally, and to try and derive some insights into why it seems to work.
Moreover, insofar as the goal of the paper is to explore synthetic data generation, there do not appear to be any other baselines that perform synthetic data generation: The baselines are largely either (a) classical classification methods, and (b) the FFNNet baseline + SMOTE, which is "a reference for conventional imbalance handling". I raise this point in part because I was surprised to see the use of GANs as opposed to diffusion models or other synthetic data-generation approaches, given the difficulties and instabilities typically encountered while training GANs. Perhaps the GAN training is simpler in this case, given the fact that all features modeled by the GAN are first tested for being continuous and Gaussian? But in that case, would it work equally well to just directly fit via MLE a multi-variate Gaussian to all of the Gaussian features, as opposed to using a GAN?
Finally, I did have some minor concerns related to the restrictions imposed by the synthetic data generation model: First, the testing of continuous features for Gaussianity seems somewhat arbitrary (few variables will be exactly Gaussian), and I wonder how sensitive the results are to that design choice. Moreover, I read Equation (3) as saying that the approach requires fitting a MLE estimate to the Gaussian distribution for each level of every non-Gaussian feature, which seems like it could be quite computationally intensive if there are a lot of non-Gaussian features. Finally, my understanding of Equation (3) is that the non-Gaussian features as modeled as conditionally independent given the Gaussian features: For instance, if all features were discrete, would this approach reduce to treating all features as independent? That seems like an unrealistic restriction.
Questions for authors
Can you answer my questions below to clarify how the model works?
- Does Equation (3) fit the conditional distribution separately for each level of each non-Gaussian feature? E.g., if I have three binary non-Gaussian features x_1, x_2, x_3, do I have six conditional fits (one for x_1 = 0, one for x_1 = 1, etc)?
- Am I right to read Equation (4) as modelling each non-Gaussian feature as conditionally independent given the Gaussian features? If so, am I correct to infer that if there are no Gaussian features, then each non-Gaussian feature is sampled independently?