acceptodds
Under review as a conference paper at ICLR 2027

FAITH: Typed Cell-Level Tabular Generation with a Frozen Language Backbone

10 views

Abstract

Synthetic tabular data must reproduce heterogeneous column domains, their dependencies, and the predictive structure carried by complete rows. Pretrained causal language models offer a reusable mechanism for composing contextual dependencies, but existing language-model generators usually retain a textual interface: a cell expands into several tokens, prediction is performed over a general vocabulary, and numerical values are emitted as strings. We introduce FAITH, which keeps a frozen causal backbone while replacing its language interface with typed cell encoders and schema-local output distributions. Every cell occupies one causal position; categorical columns are predicted on their observed support, and numerical columns combine quantile-bin selection with a continuous conditional Beta density. This construction separates the sequence model that shares context across columns from the distributions that define individual columns. Across ten datasets and five baselines, FAITH obtains the highest primary synthetic-train-to-real-test utility on four datasets and its strongest joint quality on the dependency-intensive Allstate and HBA1C cases. On these datasets, FAITH trains - and samples - faster than the token-level GReaT baseline. The evaluation separately reports fidelity, utility, runtime, and empirical disclosure behavior.

Primary area:
generative models
Keywords:
tabular data, synthetic data, small language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y9asOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FAITH, a method designed to address the challenges of generating tables using language models—specifically, the issues of excessively long text sequences and unnatural numerical representations. FAITH encodes each table cell as a vector; it utilizes a frozen pre-trained language backbone to process previously generated fields, while employing column-specific output heads to predict subsequent fields. For categorical fields, it uses local classification distributions, whereas for numerical fields, it first selects an appropriate interval and then generates a specific value within that range using a Beta distribution. The core concept is to retain the contextual reasoning capabilities of pre-trained language models while replacing their conventional text-based input and output mechanisms. Experiments conducted across ten tabular datasets demonstrate superior generation performance on several of them, alongside faster training and generation speeds compared to baseline models that rely on text-serialized table representations.

Critical strengths

In my view, the most valuable aspect of this paper is that it provides a clear and comprehensive solution to a practical representation challenge in table generation. By ensuring that each cell occupies only a single position and constraining the output to the specific type and value space of the corresponding field, the approach reduces the overhead associated with text serialization and eliminates the need for a language-based vocabulary when generating numerical values. Another positive feature is that the paper goes beyond merely presenting a concept; it evaluates generation quality, downstream utility, and runtime across multiple datasets, reporting competitive results on datasets such as Allstate and HBA1C. These findings suggest that this interface design warrants further research and application.

Critical weaknesses

I have two primary concerns. First, the paper fails to clearly justify the choice to completely freeze the backbone. While the authors explain that freezing reduces parameter gradients and optimizer states—a valid engineering rationale—this does not explain why full freezing is preferable to further adaptation, nor does it demonstrate why a pre-trained backbone is superior to a randomly initialized one with the same architecture. Given that multiple components—cell encoding, column-specific output heads, numerical distribution handling, and the backbone adaptation strategy—are altered simultaneously, it is difficult to isolate the source of the performance gains. Furthermore, the results on the Allstate and HBA1C datasets are insufficient to attribute these advantages specifically to a pre-trained model’s capacity to capture complex dependencies. Second, the incremental contribution of this work relative to the most closely related studies remains unclear. Existing methods like DiffLM already employ frozen LLMs with trainable interfaces, and TabDLM has previously integrated specialized numerical representations into pre-trained language models. While FAITH differs in its use of a fully frozen backbone, single-position representations for all fields, and an autoregressive output mechanism, there is a lack of direct evidence quantifying the specific benefits provided by these design choices. Moreover, the primary experiments fail to include TabDiff and TabNAT—models that the authors themselves discuss. Consequently, while I acknowledge that this combination of techniques is functional, it remains unclear why this specific design is optimal, or what significant value it adds beyond existing approaches.

Questions for authors

I would like the authors to elaborate on the experimental rationale—beyond the reduction of training parameters and optimizer states—for choosing to keep the backbone entirely frozen. Have the authors conducted comparisons—using identical input/output interfaces and controlled semantic initialization of field names—between a pre-trained frozen backbone, a randomly initialized frozen backbone, and LoRA-based adaptation? Such results are crucial for my assessment of the contributions of pre-training and the chosen freezing strategy. Furthermore, what do the authors consider to be the most significant new capability of this model compared to DiffLM, TabDLM, or TabNAT, and which specific experiment serves to validate this? Finally, please verify the results for "Tabby" on the "House" dataset as presented in Table 1 and Table 8: the former reports an R2R^2 of 0.66, while the latter reports an RMSE of 25,314.01. Were these figures derived using the same target scaling, test set, and prediction outputs? This discrepancy requires clarification; otherwise, it undermines my confidence in the consistency of the reported results.

4 comments

Dear Reviewer Y9as,

Thank you for your evaluation of our work! Below, we respond to your concerns and comments.

I would like the authors to elaborate on the experimental rationale—beyond the reduction of training parameters and optimizer states—for choosing to keep the backbone entirely frozen. Have the authors conducted comparisons—using identical input/output interfaces and controlled semantic initialization of field names—between a pre-trained frozen backbone, a randomly initialized frozen backbone, and LoRA-based adaptation?

We compare FAITH, FAITH with a randomly initialized frozen backbone, and FAITH+LoRA. All three variants retain identical cell interfaces, semantic initialization, schemas, and splits. The variant with a randomly initialized frozen backbone uses a fresh LFM2.5-230M with the same configuration: attention, feed-forward, and short-convolution blocks are reinitialized using the model's native rules (linear and convolutional weights N(0,0.022)\mathcal{N}(0, 0.02^2), zero biases where present, and unit normalization scales). All backbone parameters remain frozen. Column and code embeddings and the beginning-of-row vector are initialized from pretrained token embeddings before replacement and remain trainable alongside the numerical position maps and typed heads. LoRA adapts all backbone linear layers (rank 16, alpha 32, dropout 0.05; adapter/interface learning rates 1e-5/3e-4).

Dataset / metricFAITHRandom frozen backboneFAITH+LoRA
Adult / F10.7979 ± 0.00480.46760.8023 ± 0.0018
Allstate / R²0.5234 ± 0.0071-1.4314e+490.5269 ± 0.0113
Beijing / R²0.5577 ± 0.00420.25460.5672 ± 0.0144
Default / F10.6685 ± 0.00390.56470.6742 ± 0.0340
Diabetes / F10.5867 ± 0.09820.47380.5763 ± 0.1224
HBA1C / F10.4086 ± 0.00300.260.3969 ± 0.0059
House / R²0.7313 ± 0.0105-23819.590.7425 ± 0.0149
Magic / F10.8233 ± 0.00720.58830.8210 ± 0.0044
News / R²-0.0736 ± 0.0067-0.0576-0.0408 ± 0.0188
Shoppers / F10.7924 ± 0.00530.55050.7879 ± 0.0378

In this table, FAITH and FAITH+LoRA report means accompanied by 95% Student’s t confidence interval half-widths across 3 seeds. The table shows that FAITH with a pretrained backbone outperforms FAITH with a randomly initialized frozen backbone.

LoRA has a higher mean on 6/10 datasets. However, the improvements are modest, and FAITH+LoRA consumes more GPU memory. One of our initial goals was to keep the model memory-efficient. Below, we report peak GPU memory usage during training for FAITH and FAITH+LoRA on the most memory-intensive dataset, Allstate (batch_size=32 for both runs):

DatasetFAITHFAITH+LoRA
Allstate3812MiB5184MiB

This reduction in memory usage may be important when GPU memory is limited.

Furthermore, what do the authors consider to be the most significant new capability of this model compared to DiffLM, TabDLM, or TabNAT, and which specific experiment serves to validate this?

We want to clarify our main goals. Our objective is efficient LLM-based generation with quality competitive with diffusion models; this is why our main competitors are LLM-based methods such as GReaT and Tabby. We measure efficiency in terms of training and sampling times and the GPU memory required. FAITH is much faster to train and sample from than GReaT or Tabby, and it also requires much less GPU memory. Our hypothesis is that using a pretrained LLM backbone allows the model to better understand complex relationships in large datasets.

We tried to evaluate DiffLM, but it did not yield satisfactory output quality. In fact, the model’s output was gibberish; we suspect a bug in its implementation.

We evaluated TabNAT on every dataset reported in the paper, including two additional large datasets: Ecom Offers (160,057 rows, 119 predictors; repeat-buyer classification) and NYC Green Taxi 2016 (150,000-trip subset, 16 predictors; tip-amount regression). Below, we compare MLE results.

Dataset / metricFAITHTabNAT
Adult / macro-F10.79790.7830
Allstate / R²0.52340.0402
Beijing / R²0.55770.5613
Default / macro-F10.66850.6830
Diabetes / macro-F10.58670.4363
HBA1C / macro-F10.40860.3814
House / R²0.73130.7746
Magic / macro-F10.82330.8392
News / R²-0.0736-0.4246
Shoppers / macro-F10.79240.7845
Ecom / macro-F10.60310.5462
Taxi / R²0.50150.4805

FAITH leads on most datasets, including the largest ones: Allstate (a large improvement over TabNAT), HBA1C, Ecom (also a major improvement), and Taxi. This supports our hypothesis about large datasets.

Below, we compare training and sampling times (in seconds). It is important to mention that we added validation-based early stopping to TabNAT and set patience to 100. This was done to make a fair comparison with FAITH which also used early stopping.

DatasetFAITHTabNAT
Adult535/3031759/212
Allstate14697/56987005/31393
Beijing636/240924/59
Default882/449514/45
Diabetes21/5.620/1.3
HBA1C3961/15392733/890
House340/145684/7.7
Magic459/172538/19
News1280/11221668/52
Shoppers277/127967/61
Ecom35116/78736997/546
Taxi4905/14707010/170

FAITH trains faster than TabNAT on 7/12 datasets.

We will include TabNAT as well as TabDLM as competitors in the paper.

Moreover, the primary experiments fail to include TabDiff and TabNAT—models that the authors themselves discuss.

Although TabDiff is not our main competitor because it is solely diffusion-based and we are interested in LLM-based generation specifically, we have now included its results. We added validation-based early stopping to TabDiff and set patience to 100. Below, we report an MLE comparison table.

Dataset / metricFAITHTabDiff
Adult / F10.79790.60
Allstate / R²0.52340.5233
Beijing / R²0.55770.5931
Default / F10.66850.68
Diabetes / F10.58670.73
HBA1C / F10.40860.39
House / R²0.73130.8101
Magic / F10.82330.84
News / R²-0.0736-0.0444
Shoppers / F10.79240.80
Ecom / F10.60310.58
Taxi / R²0.5015-0.1779

In this table, FAITH reports averages over three runs with different seeds. We lead on Adult, HBA1C, Ecom, and, by a large margin, Taxi (Allstate is effectively tied with TabDiff). These are the largest datasets in our paper, and this supports our hypothesis that a pretrained backbone allows our model to better understand large, complex datasets.

Pairwise fidelity is higher for FAITH as well:

DatasetFAITHTabDiff
Allstate0.93530.8718
HBA1C0.93360.9318
Ecom0.90610.6397

The following table compares the training and sampling times of FAITH and TabDiff:

DatasetFAITHTabDiff
Adult535/3031242/559
Allstate14697/569825479/27623
Beijing636/2402203/222
Default882/4491858/196
Diabetes21/5.6225/1.9
HBA1C3961/15398155/36103
House340/145811/6.3
Magic459/172920/14
News1280/11221203/43
Shoppers277/1271472/70
Ecom35116/787314580/568
Taxi4905/14702851/485

On most datasets, FAITH trains faster than TabDiff.

We also recorded peak GPU memory usage during training for FAITH and TabDiff on the most memory-intensive dataset, Allstate, and obtained the following results (batch_size=32 for FAITH and batch_size=4096 for TabDiff):

DatasetFAITHTabDiff
Allstate3812MiB7276MiB

FAITH requires almost half as much GPU memory, which may be important when GPU memory is limited.

Finally, please verify the results for "Tabby" on the "House" dataset as presented in Table 1 and Table 8: the former reports an of 0.66, while the latter reports an RMSE of 25,314.01. Were these figures derived using the same target scaling, test set, and prediction outputs?

Thank you for pointing this out. The quoted Tabby RMSE comes from a legacy evaluation. We will update it in the paper.

Thank you for the detailed response and additional experiments. I appreciate the new controlled comparisons, which address several of my concerns. However, my main concern regarding methodological novelty remains unresolved. FAITH samples faster than TabNAT on only 1/12 datasets and TabDiff on 3/12 datasets, despite efficiency being a central contribution. It remains unclear what specific advantage the proposed architecture offers beyond existing approaches. Additionally, LoRA outperforms freezing on 6/10 datasets, the extremely negative R² values suggest possible instability, and the GPU memory comparisons use different batch sizes.

Let us elaborate further on the results. Our primary goal was to outperform LLM-based methods such as GReaT and Tabby in terms of MLE, training and sampling times, and peak GPU memory consumption. FAITH indeed outperforms these methods across all of these criteria. Here’s a GPU training memory comparison of these methods on Allstate at batch size 32:

MethodGPU memory (GiB, rounded)
FAITH2.5
GReaT34
Tabby133

We also aimed to be on par with the MLE performance of diffusion-based methods, at least on large datasets such as Allstate, HBA1C, Ecom, and Taxi. Indeed, FAITH achieves comparable performance and even outperforms these methods on some of the large datasets.

Dataset / metricFAITHTabNATTabDiffTabSynTabDDPM
Adult / macro-F10.79790.78300.60440.790.79
Allstate / R²0.52340.04020.52330.49460.4266
Beijing / R²0.55770.56130.59310.48730.5326
Default / macro-F10.66850.68300.67730.650.68
Diabetes / macro-F10.58670.43630.73030.730.75
HBA1C / macro-F10.40860.38140.38670.380.32
House / R²0.73130.77460.81010.79200.8002
Magic / macro-F10.82330.83920.83770.860.86
News / R²-0.0736-0.4246-0.0444-0.2108-108.8045
Shoppers / macro-F10.79240.78450.80440.800.74
Ecom / macro-F10.60310.54620.57710.520.5301
Taxi / R²0.50150.4805-0.17790.51500.4732

FAITH is also generally faster during training. Its sampling is mostly slower than that of diffusion-based methods, but outperforming them in sampling speed was never our aim.

Regarding FAITH+LoRA, its advantage is generally modest. Here is a table comparing 95% CIs for the mean difference (FAITH+LoRA - FAITH) of MLE scores across 3 seeds:

Dataset / metricMean difference95% CI
Adult / F1+0.0043[−0.0001, +0.0087]
Allstate / R²+0.0035[−0.0145, +0.0215]
Beijing / R²+0.0095[−0.0007, +0.0196]
Default / F1+0.0058[−0.0243, +0.0358]
Diabetes / F1−0.0103[−0.1228, +0.1022]
HBA1C / F1−0.0117[−0.0205, −0.0029]
House / R²+0.0112[−0.0117, +0.0341]
Magic / F1−0.0023[−0.0098, +0.0051]
News / R²+0.0328[+0.0194, +0.0462]
Shoppers / F1−0.0045[−0.0382, +0.0292]

The intervals provide limited evidence of a consistent advantage over frozen FAITH.

Also, FAITH+LoRA obviously uses more GPU memory and has longer training times. Here is a comparison of training times on two large datasets (in seconds, with a batch size of 32 for both variants):

DatasetFAITHFAITH + LoRA
Allstate14,69726,714
HBA1C3,9616,739

Regarding the GPU memory comparison with TabDiff, we should have explained the choice of batch sizes in more detail. A batch size of 4,096 was reported in the TabDiff paper, and using it helped keep training time on the same order of magnitude as FAITH’s. We prioritized training time. Using smaller batch sizes for TabDiff leads to slower training and even instability. Below are the results of a small experiment on the Allstate dataset:

batch sizeR²
4,0960.5233
2,0480.3441

And here is one for Shoppers:

batch sizemacro-F1Training time (s)
4,0960.80441,472
2,0480.79783,889
320.44285,658
Reviewer_dzQqOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces FAITH, a tabular generative model based on a pretrained (and frozen) language model (LM), with column-specific encoding and decoding modules. In contrast to prior LM-based methods, FAITH does not generate in the textual token space but rather directly in the heterogeneous tabular space.

Critical strengths

None. (The method is straightforward, and the technical contributions are very limited)

Critical weaknesses
  • The experimental setup and results are questionable. State-of-the-art diffusion baselines like TabDiff [1] or CDTD [2] are missing; Table 2 only shows the two datasets out of 10 datasets where FAITH achieves the best performance, while the remaining datasets are deferred to the appendix, despite sufficient space in the main paper; the empirical performance is overall very modest.
  • The presentation of the paper is very poor. Some details are missing (see questions), figures are not referenced in the text, and the paper is overall hard to read.
Questions for authors
  1. How is the generation order chosen?
  2. What is the motivation for the 0.5 coefficient in Eq. (4)?

References

[1] Shi, Juntong, et al. "Tabdiff: a mixed-type diffusion model for tabular data generation." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Mueller, Markus, Kathrin Gruber, and Dennis Fok. "Continuous diffusion for mixed-type tabular data." International Conference on Learning Representations. Vol. 2025. 2025.

4 comments

Dear Reviewer dzQq,

Thank you for your time and evaluation of our work! Below, we respond to your concerns.

The experimental setup and results are questionable. State-of-the-art diffusion baselines like TabDiff [1] or CDTD [2] are missing; Table 2 only shows the two datasets out of 10 datasets where FAITH achieves the best performance, while the remaining datasets are deferred to the appendix, despite sufficient space in the main paper; the empirical performance is overall very modest.

We want to clarify our main goals. Our objective is efficient LLM-based generation with quality competitive with diffusion models; this is why our main competitors are LLM-based methods such as GReaT and Tabby. We measure efficiency in terms of training and sampling times and the GPU memory required. FAITH is much faster to train and sample from than GReaT or Tabby, and it also requires much less GPU memory. Our hypothesis is that using a pretrained LLM backbone allows the model to better understand complex relationships in large datasets.

To address your concerns about diffusion baselines, we evaluated TabDiff on all reported datasets, including two additional large datasets: Ecom Offers (160,057 rows, 119 predictors; repeat-buyer classification) and NYC Green Taxi 2016 (150,000-trip subset, 16 predictors; tip-amount regression). We added validation-based early stopping to TabDiff and set patience to 100. Below, we compare the MLE performance of FAITH and TabDiff.

Dataset / metricFAITHTabDiff
Adult / F10.79790.60
Allstate / R²0.52340.5233
Beijing / R²0.55770.5931
Default / F10.66850.68
Diabetes / F10.58670.73
HBA1C / F10.40860.39
House / R²0.73130.8101
Magic / F10.82330.84
News / R²-0.0736-0.0444
Shoppers / F10.79240.80
Ecom / F10.60310.58
Taxi / R²0.5015-0.1779

In this table, FAITH reports averages over three runs with different seeds. We lead on Adult, HBA1C, Ecom, and, by a large margin, Taxi (Allstate is effectively tied). These are the largest datasets in our paper, and this supports our hypothesis.

Pairwise fidelity is higher for FAITH as well:

DatasetFAITHTabDiff
Allstate0.93530.8718
HBA1C0.93360.9318
Ecom0.90610.6397

The following table compares the training and sampling times (in seconds) of FAITH and TabDiff:

DatasetFAITHTabDiff
Adult535/3031242/559
Allstate14697/569825479/27623
Beijing636/2402203/222
Default882/4491858/196
Diabetes21/5.6225/1.9
HBA1C3961/15398155/36103
House340/145811/6.3
Magic459/172920/14
News1280/11221203/43
Shoppers277/1271472/70
Ecom35116/787314580/568
Taxi4905/14702851/485

On most datasets, FAITH trains faster than TabDiff.

We also recorded peak GPU memory usage during training for FAITH and TabDiff on the most memory-intensive dataset, Allstate, and obtained the following results (batch_size=32 for FAITH and batch_size=4096 for TabDiff):

DatasetFAITHTabDiff
Allstate3812MiB7276MiB

FAITH requires almost half as much GPU memory, which may be important when GPU memory is limited.

How is the generation order chosen?

We use the fixed column order in the prepared training table, including the target as another column. The same order is used for teacher-forced training and generation, without order search or random permutations. The reported runs use autoregressive training.

What is the motivation for the 0.5 coefficient in Eq. (4)?

The encoder averages the column vector eje_j and the value representation vj,cj+Ijgj(uj−12)v_{j, c_j} + I_j g_j (u_j-\frac{1}{2}). The factor 0.5 gives equal weight to the column identity and the value representation. By the triangle inequality, the composite norm cannot exceed the average component norm; averaging therefore controls the scale introduced by combining the two representations before the frozen backbone. We tested several coefficients and 0.5 was the best. We will clarify this averaging interpretation in the text.

The presentation of the paper is very poor. Some details are missing (see questions), figures are not referenced in the text, and the paper is overall hard to read.

Thank you for your honest feedback. We will add explicit figure references alongside the corresponding method descriptions and include implementation details for our method and the competitors, as well as details of the data splits and ablations, in the appendix.

Thank you for the response.

We also recorded peak GPU memory usage during training for FAITH and TabDiff on the most memory-intensive dataset, Allstate, and obtained the following results (batch_size=32 for FAITH and batch_size=4096 for TabDiff):

What is the rationale behind comparing memory when TabDiff uses a 128x larger batch size?

Yes, we should have elaborated a bit more! A batch size of 4096 was reported in the TabDiff paper, and using it helped to maintain a training time on the same magnitude as FAITH's. We prioritized training time. Smaller batch sizes for TabDiff led to longer training times and even instability. Below is a small test for Allstate:

TabDiff batch sizeR²
4,0960.5233
2,0480.3441

And here is one for Shoppers:

TabDiff batch sizemacro-F1Training time (s)
4,0960.80441,472
2,0480.79783,889
320.44285,658

Thanks for the clarification! Given the initial rebuttal, I will raise my score to a weak reject.

Reviewer_FjPFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FAITH, a tabular generator that replaces text serialization with one learned vector per cell while keeping a pretrained causal language backbone frozen. Categorical columns use column-specific output distributions, while numerical columns combine quantile bins with a conditional Beta density. Experiments on ten datasets report competitive synthetic-to-real predictive utility, particularly on Allstate and HBA1C, and substantial training and sampling speedups over GReaT.

Critical strengths
  • The cell-level interface addresses a concrete inefficiency in language-model generators: a numerical value or category need not occupy several text tokens. Restricting categorical outputs to each column’s support also avoids invalid category strings.
  • The numerical head retains variation within bins instead of reducing every value to a bin representative. Treating missing values and singleton bins separately makes the handling of these cases explicit.
  • The evaluation separates predictive utility, fidelity, runtime, and disclosure measurements. The authors also acknowledge that TabSyn and CTGAN remain faster samplers, so the efficiency claim is specifically relative to text-based generation.
Critical weaknesses
  • The main claim is that pretrained language computation helps preserve difficult tabular dependencies, but the experiments do not isolate this contribution. There is no comparison using the same cell interface with a randomly initialized frozen backbone or a backbone trained from scratch. The reported gains could therefore come from the typed representation and output heads.
  • Tables 1 and 8 contain an unresolved inconsistency. Tabby’s House result is R²=0.66 with RMSE=25,314.01, while FAITH reports R²=0.71 with RMSE=0.63. These values cannot describe the stated common test targets on the same scale. This needs to be resolved before the utility comparison can be assessed.
  • Important evaluation details are missing, including the downstream predictor, generator repetitions, and complete training configurations. HBA1C, one of the two datasets supporting the main argument, lacks source metadata and a description of its target construction. The confidence intervals cover test-row resampling only, leaving uncertainty across generator fits unmeasured.
Questions for authors

What evidence separates the benefit of pretraining from the benefit of the cell interface? What exactly is the HBA1C task?

2 comments

Dear Reviewer FjPF,

Thank you for your time and assessment of our work! Below, we respond to your concerns and comments.

What evidence separates the benefit of pretraining from the benefit of the cell interface?

Below, we compare the MLE results of FAITH with a pretrained frozen backbone, as presented in the paper, and FAITH with a randomly initialized frozen backbone. Both variants retain identical cell interfaces, semantic initialization, schemas, and splits. The variant with a randomly initialized frozen backbone uses a fresh LFM2.5-230M with the same configuration: attention, feed-forward, and short-convolution blocks are reinitialized using the model's native rules (linear and convolutional weights N(0,0.022)\mathcal{N}(0, 0.02^2), zero biases where present, and unit normalization scales). All backbone parameters remain frozen. Column and code embeddings and the beginning-of-row vector are initialized from pretrained token embeddings before replacement and remain trainable alongside the numerical position maps and typed heads. In addition to the datasets initially reported in the paper, we include two additional large datasets: Ecom Offers (160,057 rows, 119 predictors; repeat-buyer classification) and NYC Green Taxi 2016 (150,000-trip subset, 16 predictors; tip-amount regression).

Dataset / metricPretrained backboneRandomly initialized backbone
Adult / F10.79790.4676
Allstate / R²0.5234-1.4314e+49
Beijing / R²0.55770.2546
Default / F10.66850.5647
Diabetes / F10.58670.4738
HBA1C / F10.40860.26
House / R²0.7313-23819.59
Magic / F10.82330.5883
News / R²-0.0736-0.0576
Shoppers / F10.79240.5505
Ecom / F10.60310.56
Taxi / R²0.5015-21.9294

The “Pretrained backbone” column reports means across 3 seeds. The table shows that a pretrained backbone is crucial for the architecture to achieve good MLE scores, outperforming a randomly initialized backbone by a large margin on almost every dataset.

Tables 1 and 8 contain an unresolved inconsistency. Tabby’s House result is R²=0.66 with RMSE=25,314.01, while FAITH reports R²=0.71 with RMSE=0.63. These values cannot describe the stated common test targets on the same scale. This needs to be resolved before the utility comparison can be assessed.

Thank you for pointing this out! The quoted Tabby RMSE comes from a legacy evaluation. We will update it in the paper.

What exactly is the HBA1C task?

HBA1C is our prepared copy of the UCI Diabetes 130-US Hospitals dataset, containing 101,766 inpatient encounters from 1999–2008. The task is three-class, single-label classification: predict the recorded outcome (target readmitted) after discharge. The possible values are readmission within 30 days (<30), after 30 days (>30), or no recorded readmission (NO). After removing encounter and patient IDs, there are 47 predictors: 8 numerical and 39 categorical. They describe demographics, diagnoses, treatment, previous healthcare use, and laboratory results. The training, validation, and test splits contain 76,324/7,632/17,810 encounters, respectively. The splits are at the encounter level and do not guarantee patient-disjoint partitions. We will make this task definition and provenance explicit in the dataset description.

Important evaluation details are missing, including the downstream predictor, generator repetitions, and complete training configurations.

All methods use the same prepared splits (approximately 75%/7.5%/17.5%, with seed 42), with categorical support preserved in training. Generator transformations are fitted on training rows; validation data are used to select checkpoints. Each generator produces as many rows as the training split contains.

For MLE, XGBoost is trained on synthetic rows, with hyperparameters selected on real validation rows using macro-F1/R². It is evaluated on the untouched real test split. The 36-setting grid covers the number of estimators {10, 50, 100}, maximum depth {5, 10, 20}, minimum child weight {1, 10}, and gamma {0, 1}. Categorical encodings are fitted on the downstream training table.

The training configurations are as follows:

MethodTraining batch; optimizer/LRBudget; early stoppingSampling
FAITH32; AdamW 3e-450 epochs; patience 5batch 32; cache; block 1
CTGAN500; Adam 2e-4300 epochs; check every 5, patience 20 checksnative generator
TabDDPM4096; AdamW 1e-320k updates; check every 1k, patience 5 checks1000 steps
TabSyn4096; Adam 1e-3VAE 4k/diffusion 10k epochs; patience 100 after minimum KL / 50050 steps
TabDiff4096; AdamW 1e-38k epochs; patience 10050 steps; best EMA
GReaT8; AdamW 5e-550 epochs; patience 5batch 64; temperature 0.7

FAITH uses a frozen LFM2.5-230M-Base backbone, up to 128 quantile bins, a conditional Beta head, weight decay of 0.01, a gradient-clipping threshold of 1, a 5% warm-up followed by cosine decay, and bf16 arithmetic. GReaT uses the same backbone.

We will include all these details in the appendix.

The confidence intervals cover test-row resampling only, leaving uncertainty across generator fits unmeasured.

The new FAITH results are computed from runs with 3 different seeds. We report their means.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.