acceptodds
Under review as a conference paper at ICLR 2027

Factorized Chain Utility: A Calibrated Divergence Diagnostic for Synthetic Tabular Data

Abstract

Synthetic tabular data helps share information while protecting commercial or personal interests, so its value depends on how well it replicates the joint distribution of the real data. Machine-learning utility measures whether a model trained on synthetic data predicts a target variable on real data. Because this score depends only on a single conditional distribution, a synthetic table can score well on this measure while failing to preserve important relationships among the other variables. We propose Factorized Chain Utility (FCU), a diagnostic for synthetic tables based on a sparse chain of conditional prediction tasks. FCU uses proper log scores to quantify distortion in selected conditional relationships. The selected conditionals multiply to a distribution, unlike the leave-one-out conditionals used by prior all-column metrics. This gives FCU's aggregate score a well-defined divergence interpretation for the selected chain. We report a pruning deficit that quantifies the dependence information omitted by the sparse factorization, and per-factor scores identify the relationships responsible for degraded utility. Controlled validation shows that FCU responds to dependency loss even when marginal distributions are preserved. Across diverse datasets and generators, FCU further separates synthetic datasets that appear comparable under single-task utility or discriminator-based evaluations, providing a principled generator ranking with actionable diagnostics.

Primary area:
generative models
Keywords:
synthetic tabular data, synthetic data evaluation, machine-learning utility, conditional distribution shift, distributional fidelity, proper scoring rules, KL divergence, probabilistic graphical models, mutual information, tabular generative models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iEkyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FCU, an evaluation metric for assessing the synthetic tabular data. Unlike traditional utility-based evaluation methods that mainly focus on specific downstream tasks, FCU aims to evaluate if synthetic data preserves the dependencies among multiple features via a sparse factorized chain. Overall, the proposed idea is intuitive and theoretically motivated to me, which is more comprehensive than TSTR while offering a clearer probabilistic interpretation than AllMLU. The authors validate FCU through extensive experiments and demonstrate its robustness and discriminative ability across various synthetic data generators. However, I still have some concerns regarding the effectiveness of FCU under challenging settings, as well as its practical applicability (please see my questions).

Critical strengths
  1. The paper addresses an important and under-explored problem in synthetic tabular data evaluation. Existing evaluation protocols focus more on whether the generated data can solve a specific downstream task. In contrast, FCU asks whether the generated data preserve relationships among features. I think this perspective is intuitive and potentially useful for the XAI community.

  2. The experimental evaluation is relatively comprehensive. The authors validate FCU on various representative tab generators and show its robustness, calibration ability, and better discriminative power compared with other metrics.

Critical weaknesses
  1. The paper motivates FCU as a comprehensive evaluation beyond downstream utility. However, to my best knowledge, in many practical tabular synthesis scenarios, synthetic data is primarily generated to support a specific downstream application. The paper would benefit from a deeper discussion of whether preserving global dependency structures should be prioritized over task-specific utility, and how FCU should complement rather than replace existing evaluation metrics.

  2. The construction of the dependency chain relies on mi estimation. I am concerned that mi may be less reliable in small-sample or high-dimensional datasets, where estimated dependencies can be affected by spurious correlations. Moreover, mi is a relatively standard dependency measure, and some existing methods [1] that already optimize or rely on similar dependency structures may get an unfair advantage under FCU evaluation.

[1] SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation

Questions for authors
  1. Although the experiments cover several representative methods, I am curious about the generality of FCU in more challenging cases. Have you tried to evaluate FCU with tabular foundation models, such as tabpfn and tabicl? Furthermore, when FCU is applied to small datasets such as Iris, can FCU still reliably identify meaningful feature dependencies?

  2. The construction of the dependency chain requires estimating pairwise dependencies among all features. Therefore, I suppose when the number of features becomes extremely large, this process may become computationally expensive. How should FCU be applied in high-dimensional tables?

  3. FCU uses the real-trained predictor as a reference. However, in some cases, synthetic data may achieve better predictive performance than real data due to shortcut learning or distribution artifacts. How does FCU handle them? Can synthetic data receive an overly optimistic FCU score despite failing to faithfully represent the underlying data distribution?

2 comments

We thank the reviewer for their thoughtful comments and questions on this work.

”1. The paper motivates FCU as a comprehensive evaluation beyond downstream utility ...”

In our view, generative modeling of tabular data serves a much broader range of applications than the single-target prediction tasks used in machine-learning leaderboards. A synthetic hospital dataset may support research on mortality, readmission, treatment patterns, and length of stay, as well as exploratory analyses whose questions are not specified in advance. No single target column, or utility score for predicting it, adequately represents these uses.

Even when a specific prediction target is given, predictive accuracy alone does not capture every relevant use. Understanding what a model has learned can require preserving dependencies among its input variables. Conditional feature importance methods, for example, assess a feature’s contribution by resampling it conditional on other predictors, explicitly relying on relationships within (X), beyond prediction of (Y). Of course, when a single task-specific prediction task is of interest, we agree that this utility should be privileged. The purpose of FCU is not to replace such task-specific utility, but to complement it: often the focus on a single prediction task is only due to convenience, and success on one prediction task provides limited evidence about the usefulness of an entire synthetic dataset.

”2.The construction of the dependency chain relies on mi estimation. I am concerned that mi may be less reliable in small-sample or high-dimensional datasets, where estimated dependencies can be affected by spurious correlations...”

Thank you for this question. Indeed, MI estimation may sometimes be unreliable. Note however that we use MI only for examining pairwise estimation, so it should be relatively stable on small samples. Furthermore, under our revised pruning policy, the MI results are only used to order the columns. In the pruning step, the decision to keep a parent column depends on whether they improve prediction on held-out real rows, so a spurious correlation that does not help prediction is dropped. In our experiments, replacing MI with another dependence measure, or with random orderings, leaves the generator rankings unchanged (ρ ≥ 0.99; Appendix, ablation table). We therefore do not think that the local reliance on MI gives a substantial advantage for MI methods. When this is a specific concern, it is easy to adapt FCU using an alternative to MI, as we show in these ablation experiments.

Answers For Reviewer

Q1.

To answer both questions, we generated a small-n version of the controlled simulated-data experiment described in the general comment. We see some small-data costs for FCU, but these reverse at relatively modest dataset sizes. In the experiment, we restricted each dataset to n = 150 or 300 rows in the training set, and only looked at the 12-column datasets. (The original results are for n=6000). We find that for n=150, FCU has more ranking errors that allMLU (23.4% against 17.2%), but this relation reverses already at n=300 (12.6% against 13.9%).

Furthermore, we replaced LightGBM with TabPFN as FCU and allMLU’s learning algorithm on the small-dataset version of the experiment. On the n=150 sized datasets, this replacement reduces FCU's wrong rankings from 22.6% to 12.4% , which is similar to allMLU with TabPFN (12.9%). At n=300 FCU with TabPFN does better than allMLU with TabPFN (8.5% against 11.2%). (*) These are in the up-to-10 parents condition

Q2.

We are not sure that practically this is the main issue with large models. For p columns, under allMLU we are expected to fit p models with p-1 predictors. Estimating easily parallelizable p^2 summaries is relatively cheap in comparison. Also, screening operations on these relations are simple to implement. The main question is not clear how tabular generators themselves should work for extremely large numbers of features without some types of reductions or structure, and we expect that these reductions would also be used to externally constraint the estimated structure.

Q3.

Shortcuts that exist only in the synthetic data are penalized: both models are scored on held-out real rows, where such a pattern makes predictions worse. A synthetic model can still beat the real one, for example through smoothing; FCU caps the total at 1, and the per-relationship scores show where.

FCU can be too optimistic in three known ways, each with a check. Generators that replicate training rows can get perfect scores by construction; we therefore report two memorization diagnostics beside it (Appendix D.8). As we show in the general response, our revised default handles the “Matching the chain's relationships” and “imitating FCU's own learner” concerns.

We would be glad to clarify further.

I would like to thank the authors for their response. I have updated my score.

Reviewer_LLYUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FCU, a utility score for synthetic tabular data built from a sparse chain of conditional prediction tasks. The motivation is that leave-one-out conditionals do not multiply, so existing all-column scores have no divergence interpretation. The construction is clean, the calibration experiments are careful, and the paper is honest about its limits.

Critical strengths

S1. The paper is unusually disciplined about what its score does and does not measure — it separates marginal fidelity, reports per-factor scores, and keeps a memorization diagnostic beside every number, which makes the metric usable.

S2. The calibration is convincing. On families where the divergence is known, the summed excess tracks its target with a rank correlation between -0.96 and -0.99, and the single clip on the sum is a better design than clipping each factor.

Critical weaknesses

W1. I could not find a decision that FCU changes. On the four tables where allMLU was run, the two metrics agree at Spearman 0.976 and pick the same leading generator, and allMLU predicts utility on the held-out half better. The paper says this holds on every table, but allMLU was run on four of the twelve. The remaining advantage is cost and interpretation, and the saving is per-model width, not the number of models fitted. For a paper whose abstract promises a principled ranking, that is the central claim and it is not established.

W2. The auditability claim does not survive real data. The pruning deficit needs the true distribution, and what is reported on real tables is a diagnostic gain, not an estimate of the deficit, so a reader cannot tell whether a high FCU means a good generator or a narrow chain. The parity example in Appendix C.4 is the clearest case: a generator that destroys a pure three-way interaction can score as well as a faithful sample.

Questions for authors

Q1. Can you point to a concrete evaluation decision where FCU and allMLU disagree on the preferred generator, and argue that FCU is right? Table 13 shows the same leader on all four tables compared.

Q2. What should a practitioner report on a new table to bound what the chain omits? Has the full-versus-pruned gain been validated against the true deficit on any family where both are computable?

Q3. The 0.07 tolerance appears to be a ladder mean. Table 23 has per-rung errors up to 0.32 and -0.64 at forty columns, and the mixed-type family has no target at all. Which quantity should a reader trust, and can the cell-level error be reported with the score?

Q4. TabStruct also defines a structural fidelity metric against a supplied causal graph, and the comparison uses only its global utility. Did you compare against that structural metric on the synthetic families where the graph is known, and does FCU add anything there?

4 comments

We thank the reviewer for these thoughtful comments.

”W1. I could not find a decision that FCU changes. On the four tables where allMLU was run, the two metrics agree at Spearman 0.976 and pick the same leading generator, and allMLU predicts utility on the held-out half better. [...]. The remaining advantage is cost and interpretation, and the saving is per-model width, not the number of models fitted. For a paper whose abstract promises a principled ranking, that is the central claim and it is not established.”

Please see our general response, where we discuss our additional experiments targeting this question. For the real data, we indeed do not find any change in ranking over the real-tables for which allMLU was run. To investigate accuracy in ranking, we design a controlled scenario with pair-wise comparisons of generators where the true better generator is known. We show the gains in accuracy of FCU based methods. We further show substantial cost differences when more than one generator is used. We think that these do establish that the ranking is principled, with an interpretable measure of divergence between generator and true distribution.

”W2. The auditability claim does not survive real data. The pruning deficit needs the true distribution, and what is reported on real tables is a diagnostic gain, not an estimate of the deficit, so a reader cannot tell whether a high FCU means a good generator or a narrow chain..”

You are correct that this is a real limitation of the original (now updated) pruning method in our submission. Following your comments, in order to limit the potential effects of pruning based on pair-wise information, we develop a new method that is based on predictive variable gain on the real data (see general comments). We can see a substantial difference in the pruning deficit: With the original default, the pruning left out 0.39 nats per table. The revised default leaves out 0.02 nats, about 17 times less. This helps us bound the pruning deficit using the real data, while still making sure that only sets that are relevant for predicting the real data are kept. In particular, as we will discuss in the new version of the paper, the revised pruning algorithm does capture the three-way interaction model from Appendix C.4, in contrast to the previous algorithm. (This comment in particular made us develop the new pruning method, which attempts to bound the (prediction) information lost on the real data by pruning).

Answers For Reviewer:

”Q1. Can you point to a concrete evaluation decision where FCU and allMLU disagree on the preferred generator, and argue that FCU is right? Table 13 shows the same leader on all four tables compared.”

As discussed, we designed an experiment that shows substantial differences between allMLU and FCU in terms of preferred generators, with the results displayed in the general comments. On the real-data tables, FCU and allMLU give identical results.

”Q2. What should a practitioner report on a new table to bound what the chain omits? Has the full-versus-pruned gain been validated against the true deficit on any family where both are computable?”

Thank you for this comment. Our new pruning algorithm allows us to measure, on the real data, the prediction-error deficit from using smaller parent sets. This can be reported, and can provide a lower estimate of the information left out.

”Q3. The 0.07 tolerance appears to be a ladder mean...”

FCU always scores a more damaged dataset as worse, and its worst error is 3 to 4% of the KL on discrete networks, 22% of the 40-column numeric networks, and 46% on the mixed-type family network, where we compare with the full KL (0.06 nats on a KL of 0.14).

On a real table the error cannot be computed because the true KL is unknown. Instead, each FCU score is reported with its bootstrap interval, and the per-family errors above show how large the error was where the truth was known.

”Q4. TabStruct's structural fidelity metric against a supplied causal graph?”

Thank you for this suggestion. We do not think the algorithms are directly comparable. TabStruct tries to identify graphical relations, whereas FCU and allMLU focus on measuring the magnitudes of recovering predictability. To measure these effects, we implemented TabStruct (according to Appendix B1 in their paper) on the new synthetic dataset discussed in the general comments. Here we append the table with the TabStruct results. We can see that TabStruct results are substantially worse than the other algorithms.

MetricParents per columnRanking errors
allMLU, published scoring1117.1% ± 1.5
FCU, original default2.015.0% ± 1.9
FCU, up to 10 parents5.96.8% ± 1.0
FCU, no pruning6.06.5% ± 0.9
FCU, revised default2.65.0% ± 0.9
TabStruct-32.9% ± 2.4%*

We would be glad to clarify further.

Thank the authors for their response. While the response above is sincere, I still have concerns about this manuscript, including the following points:

  1. The proposed improvements for W2 are not included in the main text of the current submission;

  2. Furthermore, the authors have already acknowledged in their response to W1/Q1 that the rankings for FCU and allMLU are identical when evaluated on real-world datasets; therefore, the final review conclusion remains unchanged.

Taking these points and the responses to the questions into account, I maintain my initial "weak reject" based on the current version of the submission. The proposed revisions remain merely at the stage of a "revision plan," and the changes would be too extensive.

We thank the reviewer for the follow-up. We have now uploaded the revised paper, so the changes below are in the uploaded PDF

W2 is addressed in the main text.

  • Section 3.2: the new default parent rule. It ranks candidate parents by split gain and keeps them by a full-path rule, so it keeps parents that help only jointly.
  • Section 5.2: on the parity study of Appendix C.4, the chain now finds the interaction in all 27 settings. A generator that ignores it scores about 0, while a faithful sample scores at least 0.99. This is the case the reviewer raised.
  • Section 5.3: a generator that reproduces only the conditionals the chain scores must already be closer to the truth than 259 of 290 single-edge generators (Appendix C.6).
  • Section 3.3: where the true deficit is known, the full-versus-pruned gain falls short of it in every cell. We now describe it as a lower estimate, and Section 8 states the remaining limitation.

On real tables, the rankings are close but not identical. Our earlier reply referred to the four tables compared in the original submission, where the two metrics picked the same leader. With the new rule, we have now compared FCU and allMLU on eleven benchmark tables (Section 6.2; Appendix D.5):

  • On ten tables the two rank generators alike (Spearman 0.93 to 1.00) and pick the same leader.
  • On cdc_diabetes, allMLU cannot separate its top two (0.913 against 0.911; the order flips across runs). FCU ranks TabDiff ahead in all 9 runs.

Agreement on real tables does not make the two metrics equivalent. On a real table the correct order is unknown, so agreement only shows that both are reasonable there. Where the correct order is known (Section 5.3), FCU's ranking error rate is 5.0%, compared with 17.1% for allMLU. It is better on 13 of the 14 datasets, and worse on none. The result holds on 8 larger datasets. FCU also costs less to score each further generator, has a stated divergence target, and shows which relationships a generator loses.

Thank you for the revised paper. The two points I raised are partially addressed and one part of W1/Q1 remains open. I asked for an evaluation decision FCU changes on real data. The known-answer experiment answers this on simulated data, where the correct order is easy to construct.

I will raise my score to 3, and I hope the authors can fill the above gap before the camera-ready.

Reviewer_cbj8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Factorized Chain Utility (FCU), a metric for evaluating synthetic tabular data. It uses a sparse chain of conditional prediction tasks to assess how well synthetic data preserve dependencies among variables. The method provides a divergence-based interpretation and identifies distorted conditional relationships. Experiments on controlled and real-world datasets demonstrate its effectiveness.

Critical strengths
  1. The paper addresses an important limitation of existing synthetic data evaluation metrics. The proposed framework provides a principled way to assess conditional dependencies beyond single-task utility.

  2. The theoretical analysis is well supported by comprehensive experiments. The results demonstrate the effectiveness of FCU in detecting dependency distortions and providing interpretable diagnostics.

Critical weaknesses
  1. The practical advantages of FCU over existing all-column utility metrics could be better established. FCU and allMLU produce highly consistent generator rankings in the experiments. While FCU offers a clearer divergence interpretation and uses fewer conditioning variables, its additional benefits in practical evaluation remain less clear. A direct comparison of computational costs would further strengthen the paper.
Questions for authors
  1. Could the authors further clarify the practical benefits of FCU over allMLU, beyond its theoretical interpretation? In particular, how much computational efficiency does the sparse factorization provide?

  2. Could the authors briefly discuss how the numeric scoring approximation and pruning deficit affect the interpretation of FCU on real-world datasets?

1 comment

We thank the reviewer for the positive assessment and for two precise questions.

"The practical advantages of FCU over existing all-column utility metrics could be better established. FCU and allMLU produce highly consistent generator rankings in the experiments. While FCU offers a clearer divergence interpretation and uses fewer conditioning variables, its additional benefits in practical evaluation remain less clear. A direct comparison of computational costs would further strengthen the paper."

Please see our general response for additional experiments and discussion of the accuracy and cost structure. We show accuracy gains under controlled settings, and show large computational savings when multiple generators are compared.

Answers For Reviewer:

"Could the authors further clarify the practical benefits of FCU over allMLU, beyond its theoretical interpretation? In particular, how much computational efficiency does the sparse factorization provide?"

The costs of FCU are split into the setup phase (which includes the chain construction and the parent set selection), and a training phase (when prediction models are trained from the parent sets). The setup phase is only run for the first generator, whereas modeling is run for each generator separately. We show in our new experiments (see the general response) that the computational gains from the sparse factorizations are between 60% when the factorization is dense (parent sets are still smaller than all-variable sets), and 20-25% under sparse factorizations.

"Could the authors briefly discuss how the numeric scoring approximation and pruning deficit affect the interpretation of FCU on real-world datasets?"

We did not find a strong sensitivity of real-world datasets to the numeric approximations used for the scoring. A utility method is best used for prediction, and using the mean-squared error ensures a relatively stable result even for small test samples.

In contrast, the pruning does matter for the interpretation of real-world data. FCU checks only the relationships that the estimated parent-set keeps. If the pruning algorithm is too trigger-happy on dropping parents, there is no penalty for generators that ignore these relations as well.

We have therefore decided to change the default pruning algorithm, into an algorithm that is more careful with retaining the important relations. See the discussion in the general comments. When we measure the effect of this change on the real tables, we can see a substantial difference in the pruning deficit: With the original default, the pruning left out 0.39 nats per table. The revised default leaves out 0.02 nats, about 17 times less.

We hope these additions answer the reviewer's questions and would be glad to clarify further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading, and are glad they found that FCU fills an important gap: checking whether relationships among features are preserved, beyond a single downstream task. We were alerted by the repeated concerns regarding the selection of parents, the benefits to accurate ranking of models, and the relevant costs.

One critique that stood out was that FCU measures depend on the choice of parents, and that a stringent parent set may allow problematic generators to look good (W2, LLYU). Furthermore, we used an aggressive default parent set in the original submission, limiting the number of parents for each variable to 2. We do not think that the cost perspective is the most important one; rather, identifying a useful parent set allows us to better identify important differences in the generator.

We therefore built a synthetic-data / synthetic generator experiment where each test generator ignores exactly one column link in the data-generating dependency graph. These generator errors can be ranked by information loss, so we get ground-truth labeling on every generator pair. We can then compare variants of FCU and allMLU on their accuracy in generator ranking, and on cost.

Accuracy:

MetricParents per columnRanking errors
allMLU1117.1% ± 1.5
FCU, original default2.015.0% ± 1.9
FCU, up to 10 parents5.96.8% ± 1.0
FCU, no pruning6.06.5% ± 0.9
FCU, revised default2.65.0% ± 0.9

(12 columns, 6,000 rows, 5 seeds)

Computation time*:

SettingFirst generatorEach further generator
allMLU9.3 s9.3 s
FCU, original default2.6 s (0.29×)1.4 s (0.18×)
FCU, up to 10 parents6.0 s (0.59×)6.0 s (0.59×)
FCU, no pruning6.0 s (0.60×)5.9 s (0.59×)
FCU, revised default10.7 s (1.13×)2.4 s (0.26×)
  • Median of 28 runs on one machine, 8 CPU cores, 64GB.

We see that our default method (15% errors) is slightly better than allMLU (17% errors) in accuracy, with very aggressive parent-pruning and therefore computation time savings. However, we agree that this may not be best. With additional parent allowance (up to 10, or no pruning), we get substantially less ranking errors (6.8%-6.5%), with some computational savings (60% in run-time).

(As a reminder, allMLU costs are identical per generator, but for FCU there is an initial cost for generating the parent sets in the first run and subsequent generators use the same parent sets).

Based on these findings, and seeking to find a parent set that captures relevant relations in the data, we devised a second adaptive pruning algorithm (a revised default). The algorithm starts with a target column and potential parent set:

  • Fit a boosting tree prediction model on real data from the parent set to the target column.
  • Estimate variable-added information gain for each variable on the full model.
  • Rank the variables.
  • For each prefix (variable ranked 1 through k), estimate a boosting tree prediction model. Stop when there is no information gain by adding the next variable to the parent set.

The new strategy is (a) the most accurate among all methods (5% errors), (b) results in relatively sparse parent sets (average 2.6 parents), and (c) substantially faster than allMLU when using more than one generator (113% run time on the first generator, but 26% on subsequent ones). On the 12 real tables of the paper's cost study, each further generator costs less than with allMLU, at 0.02 to 0.91 times its time.

We will revise the paper in response to your review questions and the revised adaptive pruning method, so it reflects the additional experiments and results.

AuthorsOpenReview ↗

Summary of changes in the revised paper

We thank the reviewers again. We have uploaded a revised version of the paper. The main changes are listed below, with the reviewers who raised each point and where to find it.

1. A new default parent pruning rule (all reviewers). Section 3.2; Appendix B.1. Parents are now ranked by the split gain of a boosted model and kept with a full-path, up to k parents. Unlike the earlier pairwise rule, it does not prune parents based on pairwise interaction, but looks at the joint distribution. The earlier rule remains as the fast rule, and every result that uses it is labeled.

2. New experiment: ranking where the answer is known (LLYU W1/Q1, cbj8, iEky). New Section 5.3 and Table 1; Appendix C.6. We look at pair-wise rankings of generators on 14 simulated datasets and 3,059 generator pairs. FCU's ranking error rate is 5.0%, compared with 17.1% for allMLU.. It is better on 13 of the 14 datasets, and worse on none. The result holds on 4 additional datasets of 20 columns and 4 of 40 columns. Appendix C.6 adds:

  • TabStruct's structural metric, which makes 32.9% ranking errors.
  • A test against a “cheating” generator that reproduces the FCU chain and parents.
  • Experiments with small sample sizes (150 and 300 rows, including TabPFN as FCU's learner).

3. Computational cost measurements (cbj8, iEky Q2). Section 5.3 and Table 1; Appendix D.5. We measured FCU and allMLU on the same rows with the same learner: Our method has a setup cost and both methods have per-generator costs. On the simulated table data, the first generator is 1.13 times longer for FCU due to startup costs, and every additional generator is 0.26 times compared to allMLU.
We also state the complexity of each metric.

4. allMLU on the real tables (LLYU W1/Q1). Section 6.2; Appendix D.5. We rescored eleven benchmark tables with the new rule:

  • FCU and allMLU rank generators alike (Spearman 0.93 to 1.00) and pick the same leader on ten tables.
  • On the eleventh table (cdc_diabetes), allMLU cannot separate its top two (0.913 against 0.911; the order flips across runs). FCU separates them clearly, ranking TabDiff ahead in all 9 runs.
  • We also explain where the fast rule disagreed with allMLU.

5. Pruning deficit and parity (LLYU W2/Q2, cbj8). Sections 3.3 and 5.2; Appendices C.4, C.5 and D.4.

  • Parity: under the new rule, the chain finds the parity interaction in all 27 settings.
  • Edge-drop study: FCU's divergence reads 0.96 to 1.01 times the true divergence.
  • Ĝ: calibrated against the true deficit, it falls short in every cell, so the paper now calls it a lower estimate.

6. Calibration error per rung (LLYU Q3). Section 5.2; Appendix C.3. The single "within 0.07" statement is replaced by the error per family and per rung.

7. Corrections (LLYU).

  • allMLU targets a pseudo-likelihood divergence. We no longer say it "has no target".
  • "The same leader on every table" is replaced by the measured agreement.
  • The abstract drops "principled generator ranking" and reports the known-answer result instead.
  • The rule used by each experiment is stated. The 12-table benchmark tables and the held-out-column experiment use the fast rule (Sections 4 and 6.2).

8. Discussion and limitations (iEky W1, W2, Q3). Introduction; Section 8; Appendix E.

  • Introduction: why many prediction tasks matter, and how FCU complements task-specific utility.
  • Limitations: the role of the association measure; when FCU can be too optimistic (copying, a generator that matches the chain, a generator tuned to FCU's learner); small samples; and interactions among columns outside the top ten candidates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.