acceptodds
Withdrawn from ICLR 2027

Contemporary Tabular Data Generators Do Not Preserve Conceptual Structure

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Synthetic tabular data is usually judged by agreement with low-order marginals, by the loss of a downstream predictor, or by the failure of membership-inference attacks. Each is computed from a low-dimensional projection of the joint distribution, so none can tell whether a generator preserved the dependence structure among features. We measure this conceptual structure by treating a binary table as a formal context. We score 15 generators over multiple seeds, 13 learned and two structure-free baselines, on 63 real-world contexts. Each generated table is compared with its ground truth at matched density on the number of closed attribute sets, the size of the canonical implication base, and two bounded rise rates of the concept lattice. All 13 learned generators produce synthetic data with fewer closed sets and a smaller canonical base than their ground truth, and on these counts none is measurably better than a structure-free baseline. The two rise rates order the generators differently, and on them some learned generators are more faithful than one of the two baselines, so the second result holds in full for the counts only.

Primary area:
generative models
Keywords:
synthetic tabular data, evaluation of generative models, structural fidelity, formal concept analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BTMeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether synthetic tabular data preserves the combinations of attributes present in the original data. It introduces an evaluation based on Formal Concept Analysis and compares 13 learned generators with two structure-free baselines across 63 binary datasets. The authors find that learned generators tend to produce fewer closed attribute sets and smaller implication bases, without a measurable advantage over the baselines on these counts.

Critical strengths

The motivation is convincing for me. It is easy to see how a synthetic table could match column marginals or give a decent average classifier score while still dropping specific attribute combinations that matter for subgroup analysis or database queries.

The empirical comparison in Figure 2 is also quite striking. Seeing 13 widely used generators (across GANs, diffusion, and tabular foundation models) fail to beat trivial coin-toss and row-sum baselines on concept counts is a interesting result.

Critical weaknesses

(1)I have some concern about the dataset. Section 3.2 states that every dataset is clarified and reduced before fitting the generators, which removes all duplicate rows and columns. For statistical generative models (GANs, VAEs), duplicate rows are how empirical frequencies are represented. Removing them before training alters the data distribution and reduces datasets to tiny matrices (mostly 10–60 rows) where every pattern appears only once by construction. This makes it hard to tell whether the structural loss discussed in Section 5 is an inherent flaw of the generators' training objectives on rare patterns, or an artifact of training distribution-based models on deduplicated formal contexts.

(2) Lacking application scenarios: The datasets are very small in tabular synthesis applications, and a substantial fraction come from related wiki-edit networks. While Sections 1 and 2 frame the work around tabular synthesis applications (such as subgroup prediction and database testing), Appendix A reveals that 48 of the 63 contexts are network adjacency or incidence matrices from KONECT. It is unclear how well findings on tiny graph adjacency matrices reflect typical tabular data settings where columns represent distinct semantic features. I understand why exact computation requires this setting. However, it leaves open whether the observed failures mainly reflect a difficult small-data regime or a broader limitation of current generators.

Questions for authors
  1. Can you show even one concrete example from your datasets where this loss of closed sets actually breaks a query or hurts subgroup prediction, even though standard marginal/utility metrics look fine?

  2. Due to the over-representation of WIKI-edit contexts, can you show the robustness of your result across diverse subsets. Does the main conclusion remain similar when the WIKI-edit datasets are excluded? It remains unclear whether the observed failure is a general limitation of tabular generators or mostly an artifact of fitting models on tiny, deduplicated graph matrices.

Reviewer_gqZ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Currently synthetic tabular data generators are mainly evaluated based on 1) agreement with low-order marginal distributions, 2) performance of downstream predictors, and 3) failure of membership-inference attacks. The paper argues that these metrics only evaluate low-dimensional projections of the joint distribution and do not test whether the dependence structure among features is preserved.

To address this gap, the paper uses Formal Concept Analysis (FCA), where binary tables are treated as formal contexts. The authors evaluate 1313 learned generators against two structure-free baselines using the number of closed attribute sets, the size of the canonical implication base, and two rise rates measures of the concept lattice. The paper finds that all 1313 learned generators produce fewer closed sets and a smaller canonical implication base than their source tables. On these two count based measures, none of the learned generators is measurably better than a structure-free baseline. However, the rise-rate measures give a different ordering, and some learned generators perform better than one baseline. The authors recommend adding structural measures and simple baselines to synthetic-data evaluations.

Critical strengths
  1. The work identifies an underexplored evaluation dimension in tabular synthetic data generation that could be important for measuring the quality of these generators.

  2. The authors evaluate a broad range of models against two simple and interpretable baselines. The results show that the tested models have a clear gap in preserving conceptual structure, especially on the count-based measures.

  3. The experimental methodology is relatively clear, including the matched-density comparisons and statistical tests. The authors clearly acknowledge the limitations of their work and carefully explain the missing measurements and incomplete model coverage.

Critical weaknesses
  1. The experiments are only done on very small binary datasets, ranging from 3×33 \times 3 to 68×6268 \times 62. Most tabular benchmarks and real-world datasets are much larger and contain a mixture of continuous and categorical features. Therefore, it is not clear whether the same findings hold in more realistic settings.

  2. By nature, deep neural network based models need larger data to learn accurate joint distributions, and fitting them on small tables with few rows gives too little signal to capture higher-order dependencies (This criticism does not apply equally to every evaluated models). Notable strong models such as TabDiff and TabSyn are also missing from the evaluation. The paper cannot separate genuine failure of learned generators from the effect of evaluating data-hungry models in a data-scarce regime.

  3. In small tables, random combinatorial patterns are more likely to appear as observed structure. This may explain why the two structure free baselines outperform the learned generators. They produce random closed sets whose total count, on small tables, happens to be comparable to that of the source, but this does not mean the baselines are recovering the same closed sets or the same structure..

  4. The evaluation relies on exactly enumerating some combinatorial patterns such as closed attribute sets which can be expensive in general. Even within this small corpus, 461461 measurements time out, and the largest lattice's rise rates exceed the computation budget entirely. Real-world tabular data routinely has hundreds or thousands of columns, making exact enumeration infeasible.

Questions for authors
  1. How the scalability issue mentioned in weakness 4 can be addressed? Is it possible to use some approximation scheme or sampling based estimator for scaling these measures to larger tables?

  2. Can the authors train one strong diffusion-based model like TabDiff on a larger table (selected from their paper) and select a few closed sets, then count the number of occurrences of those closed sets in the training and synthetic data?

Reviewer_Ej9NOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether tabular generators reproduce selected Formal Concept Analysis (FCA) statistics on small binary tables. It compares 13 learned generators and two random baselines on 63 contexts, measuring closed-set counts, canonical implication-base size, and two lattice-shape statistics. The learned generators generally produce smaller counts and do not outperform the baselines on those counts, although the shape statistics give different results.

Critical strengths
  • FCA provides an interesting way to examine attribute combinations beyond individual column statistics. Comparing several generator families against simple random baselines is useful within this setting.
  • The paper reports repeated samples, paired comparisons, and multiple-testing correction. It also reports where the shape measures disagree with the count measures, rather than presenting one consistent ranking.
  • The authors disclose substantial AI assistance in protocol design, implementation, and writing. The appendices also identify missing measurements and acknowledge that some omissions favor the baselines.
Critical weaknesses
  • The paper manuscript does not follow the ICLR writing style.
  • The title and conclusions are much broader than the experiments. This is a focused study of reduced binary contexts ranging from 3×3 to 68×62, with generators largely run at library defaults. Moreover, 21 of the 63 contexts are similar wiki-edit networks. This does not establish a general failure of contemporary tabular generators. Evaluating every model on the same corpus does not prevent its composition from favoring some models.
  • The novelty is applying FCA diagnostics to these generators. Dependency failures were already studied by Umesh et al. (2025) and Long et al. (2025). The paper does not compare conventional fidelity or downstream utility measures on these same outputs, so it has not shown that those evaluations would overlook the observed failures. Its conclusions should remain specific to the measured FCA statistics until that comparison is made.
  • The density and missing-measurement controls leave the baseline comparison unresolved. Table 8 reports median absolute density gaps of .096 for TabDDPM and .061 for TVAE, despite the repeated “matched density” description. Cointoss also matches overall density, not individual column marginals. Appendix C further states that expensive, unfavorable baseline measurements time out and are omitted. Use a column-marginal baseline, quantify sensitivity to density, and compare paired complete measurements before retaining C3.
Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.