- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes FCU, an evaluation metric for assessing the synthetic tabular data. Unlike traditional utility-based evaluation methods that mainly focus on specific downstream tasks, FCU aims to evaluate if synthetic data preserves the dependencies among multiple features via a sparse factorized chain. Overall, the proposed idea is intuitive and theoretically motivated to me, which is more comprehensive than TSTR while offering a clearer probabilistic interpretation than AllMLU. The authors validate FCU through extensive experiments and demonstrate its robustness and discriminative ability across various synthetic data generators. However, I still have some concerns regarding the effectiveness of FCU under challenging settings, as well as its practical applicability (please see my questions).
Critical strengths
-
The paper addresses an important and under-explored problem in synthetic tabular data evaluation. Existing evaluation protocols focus more on whether the generated data can solve a specific downstream task. In contrast, FCU asks whether the generated data preserve relationships among features. I think this perspective is intuitive and potentially useful for the XAI community.
-
The experimental evaluation is relatively comprehensive. The authors validate FCU on various representative tab generators and show its robustness, calibration ability, and better discriminative power compared with other metrics.
Critical weaknesses
-
The paper motivates FCU as a comprehensive evaluation beyond downstream utility. However, to my best knowledge, in many practical tabular synthesis scenarios, synthetic data is primarily generated to support a specific downstream application. The paper would benefit from a deeper discussion of whether preserving global dependency structures should be prioritized over task-specific utility, and how FCU should complement rather than replace existing evaluation metrics.
-
The construction of the dependency chain relies on mi estimation. I am concerned that mi may be less reliable in small-sample or high-dimensional datasets, where estimated dependencies can be affected by spurious correlations. Moreover, mi is a relatively standard dependency measure, and some existing methods [1] that already optimize or rely on similar dependency structures may get an unfair advantage under FCU evaluation.
[1] SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation
Questions for authors
-
Although the experiments cover several representative methods, I am curious about the generality of FCU in more challenging cases. Have you tried to evaluate FCU with tabular foundation models, such as tabpfn and tabicl? Furthermore, when FCU is applied to small datasets such as Iris, can FCU still reliably identify meaningful feature dependencies?
-
The construction of the dependency chain requires estimating pairwise dependencies among all features. Therefore, I suppose when the number of features becomes extremely large, this process may become computationally expensive. How should FCU be applied in high-dimensional tables?
-
FCU uses the real-trained predictor as a reference. However, in some cases, synthetic data may achieve better predictive performance than real data due to shortcut learning or distribution artifacts. How does FCU handle them? Can synthetic data receive an overly optimistic FCU score despite failing to faithfully represent the underlying data distribution?
2 comments
I would like to thank the authors for their response. I have updated my score.
We thank the reviewer for their thoughtful comments and questions on this work.
”1. The paper motivates FCU as a comprehensive evaluation beyond downstream utility ...”
In our view, generative modeling of tabular data serves a much broader range of applications than the single-target prediction tasks used in machine-learning leaderboards. A synthetic hospital dataset may support research on mortality, readmission, treatment patterns, and length of stay, as well as exploratory analyses whose questions are not specified in advance. No single target column, or utility score for predicting it, adequately represents these uses.
Even when a specific prediction target is given, predictive accuracy alone does not capture every relevant use. Understanding what a model has learned can require preserving dependencies among its input variables. Conditional feature importance methods, for example, assess a feature’s contribution by resampling it conditional on other predictors, explicitly relying on relationships within (X), beyond prediction of (Y). Of course, when a single task-specific prediction task is of interest, we agree that this utility should be privileged. The purpose of FCU is not to replace such task-specific utility, but to complement it: often the focus on a single prediction task is only due to convenience, and success on one prediction task provides limited evidence about the usefulness of an entire synthetic dataset.
”2.The construction of the dependency chain relies on mi estimation. I am concerned that mi may be less reliable in small-sample or high-dimensional datasets, where estimated dependencies can be affected by spurious correlations...”
Thank you for this question. Indeed, MI estimation may sometimes be unreliable. Note however that we use MI only for examining pairwise estimation, so it should be relatively stable on small samples. Furthermore, under our revised pruning policy, the MI results are only used to order the columns. In the pruning step, the decision to keep a parent column depends on whether they improve prediction on held-out real rows, so a spurious correlation that does not help prediction is dropped. In our experiments, replacing MI with another dependence measure, or with random orderings, leaves the generator rankings unchanged (ρ ≥ 0.99; Appendix, ablation table). We therefore do not think that the local reliance on MI gives a substantial advantage for MI methods. When this is a specific concern, it is easy to adapt FCU using an alternative to MI, as we show in these ablation experiments.
Answers For Reviewer
Q1.
To answer both questions, we generated a small-n version of the controlled simulated-data experiment described in the general comment. We see some small-data costs for FCU, but these reverse at relatively modest dataset sizes. In the experiment, we restricted each dataset to n = 150 or 300 rows in the training set, and only looked at the 12-column datasets. (The original results are for n=6000). We find that for n=150, FCU has more ranking errors that allMLU (23.4% against 17.2%), but this relation reverses already at n=300 (12.6% against 13.9%).
Furthermore, we replaced LightGBM with TabPFN as FCU and allMLU’s learning algorithm on the small-dataset version of the experiment. On the n=150 sized datasets, this replacement reduces FCU's wrong rankings from 22.6% to 12.4% , which is similar to allMLU with TabPFN (12.9%). At n=300 FCU with TabPFN does better than allMLU with TabPFN (8.5% against 11.2%). (*) These are in the up-to-10 parents condition
Q2.
We are not sure that practically this is the main issue with large models. For p columns, under allMLU we are expected to fit p models with p-1 predictors. Estimating easily parallelizable p^2 summaries is relatively cheap in comparison. Also, screening operations on these relations are simple to implement. The main question is not clear how tabular generators themselves should work for extremely large numbers of features without some types of reductions or structure, and we expect that these reductions would also be used to externally constraint the estimated structure.
Q3.
Shortcuts that exist only in the synthetic data are penalized: both models are scored on held-out real rows, where such a pattern makes predictions worse. A synthetic model can still beat the real one, for example through smoothing; FCU caps the total at 1, and the per-relationship scores show where.
FCU can be too optimistic in three known ways, each with a check. Generators that replicate training rows can get perfect scores by construction; we therefore report two memorization diagnostics beside it (Appendix D.8). As we show in the general response, our revised default handles the “Matching the chain's relationships” and “imitating FCU's own learner” concerns.
We would be glad to clarify further.