- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits tree-based autoregressive models for mixed-type tabular data synthesis and proposes Synthpop+, an improved version of the classical Synthpop approach. Synthpop+ ensembles models trained with randomized feature orders and uses out-of-bag observations for honest leaf-value estimation, aiming to improve fidelity while reducing overfitting and membership disclosure risk. The method is evaluated on 16 datasets against 11 recent tabular generative baselines, including tree-based, GAN, VAE, autoregressive, and diffusion-based approaches. The results show that Synthpop+ achieves strong synthetic-data fidelity and competitive utility while substantially reducing training costs compared with recent deep generative models.
Critical strengths
-
The paper addresses an important question for tabular data synthesis. It provides a timely examination of whether increasingly complex deep generative models are necessary when simpler classical approaches may offer competitive or better performance at substantially lower training cost.
-
The paper is generally well written and clearly presents both the motivation and the proposed improvements. The connections among feature-order sensitivity, tree regularization, fidelity, and memorization are explained in an accessible and logically structured manner.
-
The experiments are thorough within the considered benchmark setting. The paper compares Synthpop+ against a broad range of recent baselines and evaluates multiple dimensions, including fidelity, utility, overfitting, membership disclosure risk, and computational cost. The component-wise ablations and sensitivity analyses also provide clear evidence for the effects of the proposed design choices.
Critical weaknesses
-
Synthpop+ primarily combines established techniques within the existing Synthpop framework. The proposed method ensembles CART-based autoregressive models over randomized feature orders, trains each model on a data subsample, and uses out-of-bag observations for honest leaf-value estimation. Although this combination produces substantial empirical gains, the paper could more clearly explain which methodological insights are specific to tabular synthesis rather than direct adaptations of established bagging and honest-estimation techniques.
-
The experimental scope may not fully support the paper’s broad conclusions about the necessity of deep generative models. The evaluation covers 16 static i.i.d. tabular datasets with at most 86 features. It remains unclear whether the conclusions extend to settings that may more strongly motivate expressive deep models, such as substantially higher-dimensional tables, complex continuous distributions, or data with structural and domain-specific constraints.
-
The computational advantage is concentrated primarily in training rather than generation. Synthpop+ requires autoregressive sampling, and the reported average sampling time is higher than that of several deep generative baselines. The paper also shows that sampling time and memory usage increase with the number of features and ensemble members. Therefore, the claimed computational advantage may not hold in applications requiring repeated or large-scale synthetic-data generation.
-
The handling of baselines that fail to complete within the resource limit may affect the aggregate comparison. UnmaskingTrees and ForestFlow fail on several datasets, and the paper assigns them the worst observed metric values and the maximum two-hour training time. Although their failures are operationally relevant, this imputation strategy may amplify their disadvantage in the averaged results. Reporting aggregate comparisons separately over datasets on which all methods complete would make the comparison more transparent.
Questions for authors
-
What aspects of Synthpop+ provide methodological insights specific to tabular synthesis beyond combining random-order ensembling, subsampling, and honest estimation?
-
How well does Synthpop+ generalize to substantially higher-dimensional or structurally constrained tabular datasets?
-
How does Synthpop+ compare with deep generative models when large volumes of synthetic data must be generated repeatedly?
-
Do the main conclusions remain unchanged when the comparison is restricted to datasets on which all baseline methods complete successfully?
6 comments
Thank you for the detailed clarifications. While the arguments regarding the fidelity-generalization tradeoff (via test-set metrics and DCR/MIA) and scalable CPU-based parallelization are helpful, the core concern regarding the marginal improvement in training-set C2ST relative to the original Synthpop remains only partially addressed.
To help finalize this evaluation, could you briefly clarify whether a direct ablation study (or concrete metrics) was conducted to isolate the exact performance gain contributed specifically by the out-of-bag honest estimation versus random-order ensembling alone?
Thank you for your reply. Yes, the paper provides a direct comparison for:
- solely using random order ensembling
- gains from also including out-of-bag honest estimation
Specifically, Table 3 provides this. Note that this table provides additive model improvements, such that the model in each row also includes the improvements from the rows above.
Here you can compare the following rows:
- +subsample data: this contains both the random order ensemble (+ensemble feature order) and the subsampling of the training data. This should be considered as the full random order ensembling, since the subsampling is used to further decorrelate the ensemble members, as is typical.
- +honest trees: this contains the full random order ensembling, and out-of-bag honest estimation. This therefore indicates the exact performance gain contributed specifically by the out-of-bag honest estimation versus random-order ensembling alone, as you requested.
As you can see from the results, out-of-bag honest estimation substantially improves on DCR Share, and therefore reduces overfitting. Please consider for each of the metrics the scale of the original error when evaluating the scale of the improvement.
Finally, note line 408 in the paper: the various improvements we supply are ``pick and choose". If you're less worried about overfitting and membership disclosure risk you may omit out-of-bag honest estimation and/or subsampling of the training data. If you do this, our improvements actually do provide a substantial improvement on training-set C2ST (see Table 3)!
Thank you again for your reply. If you feel like we've substantially addressed some of your concerns, we humbly request to consider reflecting this in your score, as per the conference guidelines.
If you have any further questions regarding our work we're happy to help.
Thank you for your thorough review. We will address your mentioned weaknesses and questions point-by-point below.
Critical Weaknesses
1
The applications of these existing ideas are naturally specific to tabular synthesis. Namely, bagging over random feature orders only makes sense if we're autoregresively factorizing the joint distribution. For discriminative models there is no ordering to the features.
Also, it is very typical for ML research to reuse existing ideas. The important point is whether how it is implemented is a good application considering the problem, theoretically justified, and whether it substantially improves the results. We have shown all in the paper.
2
We provide in-depth discussion in the Limitations section regarding this. We do not claim that our method is the best choice for any and all dataset; this would never be possible.
However, we can say that our benchmarking is extremely intensive compared to other works published in this venue: 16 mixed-type tabular datasets, with a big range of number of rows, features, and cardinality. Asking us to extend this already very intensive benchmark even further seems harsh. We have shown great performance of our method on an already great variety of tabular datasets.
Regarding data with structural- or domain-specific constraints: see our answer to Reviewer v8cD (Question 1). In short, such constraints can be easily enforced through pre- and postprocessing schemes in any generative model, and their ability to directly model such constraints is not as practically relevant as it may seem.
3
Yes, sampling time is a widely discussed limitation in the paper. However, note that our method still outperforms 4/11 baselines, on average. Also, consider that sampling time is measured in seconds, whereas training times are measured in minutes. The practical implications in terms of absolute runtimes are therefore smaller than they might seem.
Also, note that our method can easily be scaled up in terms of sampling time and memory by simply using more CPU cores and RAM. Since we parallelize over the random feature orders at each feature index, we can scale up roughly linearly as long as we have less cores than random orders. This is in contrast to GPU-based methods which would require much more expensive GPUs (either extra cards or more capable cards) to scale up.
Finally, consider again that we've already considered much more varied, large, and complex datasets than most similar works, and that we've shown that the runtime and memory consumption of our method are low considering the quite restrictive hardware we used.
4
If we restrict the results to datasets on which UTrees and ForestFlow complete, we can only include very small datasets. Then, the whole evaluation loses its value. The interesting part is whether our method can indeed scale to large datasets; it can!
If you do wish to evaluate performance on individual datasets, Appendix B.6 provides this. Here we see that our method greatly outperforms both baselines. Therefore your suggestion does not change the conclusions, but it would harm the information on the scaling abilities of our method.
Finally, note that this imputation scheme is typical in this field. See for example the CDTD and TabCascade paper.
Questions
1
Random-order ensembling is specific to tabular synthesis, as it is only relevant when factorizing the joint distribution, and not for, e.g., discriminative models. Also, we've shown that these methodological improvements substantially improve generalization of the generative distribution. The work should be about evaluating whether the improvements are sound, sensible for the problem at hand, not used in this manner before, and provide good results. They do all of that.
2
As mentioned above, structurally constrained datasets are not necessarily interesting from a practical perspective, as structural constraints can be easily enforced in any generative model through pre- and postprocessing schemes.
Scaling performance to higher-dimensional datasets can be directly investigated from Table 4 (dataset characteristics) and Appendix B.6 (dataset-wise results). We find that our method generalizes well to higher-dimensional datasets, also considering high-cardinality datasets.
3
We expect you mean in terms of sampling times. You can inspect the dataset characteristics (Table 4) and the dataset-wise sampling times (Table 24). This way you can find how our method compares to other baselines for the larger datasets. Notably, this already considers repeated sampling.
4
Yes, the conclusions remain the same. This can be seen directly from the dataset-wise results in Appendix B.6.
Thank you again for your thorough review. If you have further questions, please let us know.