- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TabQueryBench, a query-centric benchmark for evaluating the quality of synthetic tabular data. The method constructs 44 query templates from 12 public SQL sources, covering five categories of analytical tasks, and measures Query Fidelity by comparing the results of SQL queries run on real data with those run on synthetic data. The authors conducted experiments on 49 datasets and 11 generative models and found that traditional statistical similarity metrics fail to adequately reflect query fidelity; existing models struggle in particular to preserve high-cardinality attributes, local conditional structures, and tail distributions.
Critical strengths
-
The paper addresses a meaningful limitation of existing synthetic data evaluation approaches.
-
The proposed benchmark provides a relatively comprehensive and automated evaluation framework, and the experiments are extensive and provide useful insights into existing generative models.
Critical weaknesses
-
The validity of Query Fidelity still requires further validation. This paper proposes evaluating generation quality by comparing the results of SQL queries on real data and synthetic data, which is a reasonable research motivation. However, even if the generative model has correctly learned the distribution of real data, randomly generated data may still produce different query results. Therefore, it remains unclear to what extent these differences reflect shortcomings in the generative model. I recommend that the authors include a “Real-vs-Real” baseline and further validate whether Query Fidelity can effectively reflect the performance of synthetic data in practical analytical tasks.
-
Some experimental results and scoring methods require further explanation. First, Section 3.3 mentions that each dataset contains more than 100 SQL queries, but Section 4.6 reports a total of only 2,763 queries, and Appendix Table 17 also shows that some datasets have fewer than 100 queries. I recommend that the authors explain why these numbers do not match. Second, the authors evaluate the generative models by calculating the average score across all SQL queries. However, since the number of queries may vary by type, query types with higher counts may have a greater impact on the final score. I recommend that the authors test whether the final model rankings would change if queries of different types were assigned equal weights. I am willing to increase my score if my concerns are addressed, especially the second one.
Questions for authors
-
Could the authors include a Real-vs-Real comparison experiment? Specifically, randomly select two subsets of the same size with no overlap from the same real dataset, execute the same SQL query on both subsets, and observe how much variation there is in the query results themselves. This would help determine whether the differences in query results between synthetic and real data stem from poor performance of the generative model or are simply due to variations that would naturally occur with random sampling.
-
Could the authors explain the inconsistencies in the number of SQL queries reported in the paper? Section 3.3 mentions that over 100 executable SQL queries were generated for each dataset, but Section 4.6 reports a total of only 2,763 queries, and Appendix Table 17 also shows some datasets with fewer than 100 queries. Do these figures stem from different experimental setups or counting methods?
-
Have the authors considered the impact of the number of different query types on the final score? For example, if one type of query is very common while another is rare, the more common type may have a greater influence on the final score. Could the authors try assigning equal weights to all five query types and examine whether the final rankings of different generative models would change?
2 comments
Follow-up on Our Rebuttal and Additional Results
Since the rebuttal period is approaching the end, we’d like to kindly ask whether you’ve gone through our new results and whether they’ve addressed your previous concerns. We understand that you might be busy with other reviewing tasks and your own submissions, but we sincerely appreciate your latest comments on our rebuttal contents and your willingness of updating the score. Thank you so much!
Rebuttal to Reviewer GFAk
We thank you for finding our paper clear and recognizing the breadth and practical insights of our automated SQL benchmark. Below, we address your questions on Real-vs-Real sampling validation, query-count reconciliation, and equal query-family weighting, all updated in our revised PDF.
W1/Q1: Real-vs-Real Split-Half and Resampling Validation Following your exact recommendation, we evaluated three Real-vs-Real sampling baselines across all 49 datasets—including partitioning each real table into two non-overlapping 50% halves and running the identical SQL suite on both (full calibrated table in our reply to Reviewer NUsw, W1):
Even at half the sample size with zero row overlap (0.829), real data subsets outperform every non-memorizing generator trained at full ntrain scale (0.443–0.636 raw; 0.480–0.695 calibrated against the 0.906 bootstrap oracle). This confirms that the reported deficits reflect genuine generative modeling limitations rather than natural random-sampling variance.
W2/Q2: Reconciling SQL Query Counts (Section 3.3, Section 4.6, Appendix Table 17) Thank you for catching both discrepancies, which stem from an unedited prose reference to an early prototype and multi-run API cost logging:
W3/Q3: Equal Weighting Across the Five Query Families We implemented equal weighting across active query families within each dataset (added in revised Appendix R.4):
The top five generators remain completely unchanged, confirming our conclusions are robust to query-family frequency. TabPFGen rises from 9th to 6th (0.529 to 0.608) because its per-family performance is balanced whereas cardinality—its weakest family—contributes the most queries under default averaging. Because many real datasets lack
NULLs or high-cardinality columns (making only 3 or 4 families active and shifting family weights between 33.3%, 25.0%, and 20.0%), we report both query-equal and family-equal scores side-by-side in our revised leaderboard.