- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FAITH, a method designed to address the challenges of generating tables using language models—specifically, the issues of excessively long text sequences and unnatural numerical representations. FAITH encodes each table cell as a vector; it utilizes a frozen pre-trained language backbone to process previously generated fields, while employing column-specific output heads to predict subsequent fields. For categorical fields, it uses local classification distributions, whereas for numerical fields, it first selects an appropriate interval and then generates a specific value within that range using a Beta distribution. The core concept is to retain the contextual reasoning capabilities of pre-trained language models while replacing their conventional text-based input and output mechanisms. Experiments conducted across ten tabular datasets demonstrate superior generation performance on several of them, alongside faster training and generation speeds compared to baseline models that rely on text-serialized table representations.
Critical strengths
In my view, the most valuable aspect of this paper is that it provides a clear and comprehensive solution to a practical representation challenge in table generation. By ensuring that each cell occupies only a single position and constraining the output to the specific type and value space of the corresponding field, the approach reduces the overhead associated with text serialization and eliminates the need for a language-based vocabulary when generating numerical values. Another positive feature is that the paper goes beyond merely presenting a concept; it evaluates generation quality, downstream utility, and runtime across multiple datasets, reporting competitive results on datasets such as Allstate and HBA1C. These findings suggest that this interface design warrants further research and application.
Critical weaknesses
I have two primary concerns. First, the paper fails to clearly justify the choice to completely freeze the backbone. While the authors explain that freezing reduces parameter gradients and optimizer states—a valid engineering rationale—this does not explain why full freezing is preferable to further adaptation, nor does it demonstrate why a pre-trained backbone is superior to a randomly initialized one with the same architecture. Given that multiple components—cell encoding, column-specific output heads, numerical distribution handling, and the backbone adaptation strategy—are altered simultaneously, it is difficult to isolate the source of the performance gains. Furthermore, the results on the Allstate and HBA1C datasets are insufficient to attribute these advantages specifically to a pre-trained model’s capacity to capture complex dependencies. Second, the incremental contribution of this work relative to the most closely related studies remains unclear. Existing methods like DiffLM already employ frozen LLMs with trainable interfaces, and TabDLM has previously integrated specialized numerical representations into pre-trained language models. While FAITH differs in its use of a fully frozen backbone, single-position representations for all fields, and an autoregressive output mechanism, there is a lack of direct evidence quantifying the specific benefits provided by these design choices. Moreover, the primary experiments fail to include TabDiff and TabNAT—models that the authors themselves discuss. Consequently, while I acknowledge that this combination of techniques is functional, it remains unclear why this specific design is optimal, or what significant value it adds beyond existing approaches.
Questions for authors
I would like the authors to elaborate on the experimental rationale—beyond the reduction of training parameters and optimizer states—for choosing to keep the backbone entirely frozen. Have the authors conducted comparisons—using identical input/output interfaces and controlled semantic initialization of field names—between a pre-trained frozen backbone, a randomly initialized frozen backbone, and LoRA-based adaptation? Such results are crucial for my assessment of the contributions of pre-training and the chosen freezing strategy. Furthermore, what do the authors consider to be the most significant new capability of this model compared to DiffLM, TabDLM, or TabNAT, and which specific experiment serves to validate this? Finally, please verify the results for "Tabby" on the "House" dataset as presented in Table 1 and Table 8: the former reports an of 0.66, while the latter reports an RMSE of 25,314.01. Were these figures derived using the same target scaling, test set, and prediction outputs? This discrepancy requires clarification; otherwise, it undermines my confidence in the consistency of the reported results.
4 comments
Below, we compare training and sampling times (in seconds). It is important to mention that we added validation-based early stopping to TabNAT and set patience to 100. This was done to make a fair comparison with FAITH which also used early stopping.
| Dataset | FAITH | TabNAT |
|---|---|---|
| Adult | 535/303 | 1759/212 |
| Allstate | 14697/5698 | 7005/31393 |
| Beijing | 636/240 | 924/59 |
| Default | 882/449 | 514/45 |
| Diabetes | 21/5.6 | 20/1.3 |
| HBA1C | 3961/1539 | 2733/890 |
| House | 340/145 | 684/7.7 |
| Magic | 459/172 | 538/19 |
| News | 1280/1122 | 1668/52 |
| Shoppers | 277/127 | 967/61 |
| Ecom | 35116/7873 | 6997/546 |
| Taxi | 4905/1470 | 7010/170 |
FAITH trains faster than TabNAT on 7/12 datasets.
We will include TabNAT as well as TabDLM as competitors in the paper.
Moreover, the primary experiments fail to include TabDiff and TabNAT—models that the authors themselves discuss.
Although TabDiff is not our main competitor because it is solely diffusion-based and we are interested in LLM-based generation specifically, we have now included its results. We added validation-based early stopping to TabDiff and set patience to 100. Below, we report an MLE comparison table.
| Dataset / metric | FAITH | TabDiff |
|---|---|---|
| Adult / F1 | 0.7979 | 0.60 |
| Allstate / R² | 0.5234 | 0.5233 |
| Beijing / R² | 0.5577 | 0.5931 |
| Default / F1 | 0.6685 | 0.68 |
| Diabetes / F1 | 0.5867 | 0.73 |
| HBA1C / F1 | 0.4086 | 0.39 |
| House / R² | 0.7313 | 0.8101 |
| Magic / F1 | 0.8233 | 0.84 |
| News / R² | -0.0736 | -0.0444 |
| Shoppers / F1 | 0.7924 | 0.80 |
| Ecom / F1 | 0.6031 | 0.58 |
| Taxi / R² | 0.5015 | -0.1779 |
In this table, FAITH reports averages over three runs with different seeds. We lead on Adult, HBA1C, Ecom, and, by a large margin, Taxi (Allstate is effectively tied with TabDiff). These are the largest datasets in our paper, and this supports our hypothesis that a pretrained backbone allows our model to better understand large, complex datasets.
Pairwise fidelity is higher for FAITH as well:
| Dataset | FAITH | TabDiff |
|---|---|---|
| Allstate | 0.9353 | 0.8718 |
| HBA1C | 0.9336 | 0.9318 |
| Ecom | 0.9061 | 0.6397 |
The following table compares the training and sampling times of FAITH and TabDiff:
| Dataset | FAITH | TabDiff |
|---|---|---|
| Adult | 535/303 | 1242/559 |
| Allstate | 14697/5698 | 25479/27623 |
| Beijing | 636/240 | 2203/222 |
| Default | 882/449 | 1858/196 |
| Diabetes | 21/5.6 | 225/1.9 |
| HBA1C | 3961/1539 | 8155/36103 |
| House | 340/145 | 811/6.3 |
| Magic | 459/172 | 920/14 |
| News | 1280/1122 | 1203/43 |
| Shoppers | 277/127 | 1472/70 |
| Ecom | 35116/7873 | 14580/568 |
| Taxi | 4905/1470 | 2851/485 |
On most datasets, FAITH trains faster than TabDiff.
We also recorded peak GPU memory usage during training for FAITH and TabDiff on the most memory-intensive dataset, Allstate, and obtained the following results (batch_size=32 for FAITH and batch_size=4096 for TabDiff):
| Dataset | FAITH | TabDiff |
|---|---|---|
| Allstate | 3812MiB | 7276MiB |
FAITH requires almost half as much GPU memory, which may be important when GPU memory is limited.
Finally, please verify the results for "Tabby" on the "House" dataset as presented in Table 1 and Table 8: the former reports an of 0.66, while the latter reports an RMSE of 25,314.01. Were these figures derived using the same target scaling, test set, and prediction outputs?
Thank you for pointing this out. The quoted Tabby RMSE comes from a legacy evaluation. We will update it in the paper.
Thank you for the detailed response and additional experiments. I appreciate the new controlled comparisons, which address several of my concerns. However, my main concern regarding methodological novelty remains unresolved. FAITH samples faster than TabNAT on only 1/12 datasets and TabDiff on 3/12 datasets, despite efficiency being a central contribution. It remains unclear what specific advantage the proposed architecture offers beyond existing approaches. Additionally, LoRA outperforms freezing on 6/10 datasets, the extremely negative R² values suggest possible instability, and the GPU memory comparisons use different batch sizes.
Let us elaborate further on the results. Our primary goal was to outperform LLM-based methods such as GReaT and Tabby in terms of MLE, training and sampling times, and peak GPU memory consumption. FAITH indeed outperforms these methods across all of these criteria. Here’s a GPU training memory comparison of these methods on Allstate at batch size 32:
| Method | GPU memory (GiB, rounded) |
|---|---|
| FAITH | 2.5 |
| GReaT | 34 |
| Tabby | 133 |
We also aimed to be on par with the MLE performance of diffusion-based methods, at least on large datasets such as Allstate, HBA1C, Ecom, and Taxi. Indeed, FAITH achieves comparable performance and even outperforms these methods on some of the large datasets.
| Dataset / metric | FAITH | TabNAT | TabDiff | TabSyn | TabDDPM |
|---|---|---|---|---|---|
| Adult / macro-F1 | 0.7979 | 0.7830 | 0.6044 | 0.79 | 0.79 |
| Allstate / R² | 0.5234 | 0.0402 | 0.5233 | 0.4946 | 0.4266 |
| Beijing / R² | 0.5577 | 0.5613 | 0.5931 | 0.4873 | 0.5326 |
| Default / macro-F1 | 0.6685 | 0.6830 | 0.6773 | 0.65 | 0.68 |
| Diabetes / macro-F1 | 0.5867 | 0.4363 | 0.7303 | 0.73 | 0.75 |
| HBA1C / macro-F1 | 0.4086 | 0.3814 | 0.3867 | 0.38 | 0.32 |
| House / R² | 0.7313 | 0.7746 | 0.8101 | 0.7920 | 0.8002 |
| Magic / macro-F1 | 0.8233 | 0.8392 | 0.8377 | 0.86 | 0.86 |
| News / R² | -0.0736 | -0.4246 | -0.0444 | -0.2108 | -108.8045 |
| Shoppers / macro-F1 | 0.7924 | 0.7845 | 0.8044 | 0.80 | 0.74 |
| Ecom / macro-F1 | 0.6031 | 0.5462 | 0.5771 | 0.52 | 0.5301 |
| Taxi / R² | 0.5015 | 0.4805 | -0.1779 | 0.5150 | 0.4732 |
FAITH is also generally faster during training. Its sampling is mostly slower than that of diffusion-based methods, but outperforming them in sampling speed was never our aim.
Regarding FAITH+LoRA, its advantage is generally modest. Here is a table comparing 95% CIs for the mean difference (FAITH+LoRA - FAITH) of MLE scores across 3 seeds:
| Dataset / metric | Mean difference | 95% CI |
|---|---|---|
| Adult / F1 | +0.0043 | [−0.0001, +0.0087] |
| Allstate / R² | +0.0035 | [−0.0145, +0.0215] |
| Beijing / R² | +0.0095 | [−0.0007, +0.0196] |
| Default / F1 | +0.0058 | [−0.0243, +0.0358] |
| Diabetes / F1 | −0.0103 | [−0.1228, +0.1022] |
| HBA1C / F1 | −0.0117 | [−0.0205, −0.0029] |
| House / R² | +0.0112 | [−0.0117, +0.0341] |
| Magic / F1 | −0.0023 | [−0.0098, +0.0051] |
| News / R² | +0.0328 | [+0.0194, +0.0462] |
| Shoppers / F1 | −0.0045 | [−0.0382, +0.0292] |
The intervals provide limited evidence of a consistent advantage over frozen FAITH.
Also, FAITH+LoRA obviously uses more GPU memory and has longer training times. Here is a comparison of training times on two large datasets (in seconds, with a batch size of 32 for both variants):
| Dataset | FAITH | FAITH + LoRA |
|---|---|---|
| Allstate | 14,697 | 26,714 |
| HBA1C | 3,961 | 6,739 |
Regarding the GPU memory comparison with TabDiff, we should have explained the choice of batch sizes in more detail. A batch size of 4,096 was reported in the TabDiff paper, and using it helped keep training time on the same order of magnitude as FAITH’s. We prioritized training time. Using smaller batch sizes for TabDiff leads to slower training and even instability. Below are the results of a small experiment on the Allstate dataset:
| batch size | R² |
|---|---|
| 4,096 | 0.5233 |
| 2,048 | 0.3441 |
And here is one for Shoppers:
| batch size | macro-F1 | Training time (s) |
|---|---|---|
| 4,096 | 0.8044 | 1,472 |
| 2,048 | 0.7978 | 3,889 |
| 32 | 0.4428 | 5,658 |
Dear Reviewer Y9as,
Thank you for your evaluation of our work! Below, we respond to your concerns and comments.
We compare FAITH, FAITH with a randomly initialized frozen backbone, and FAITH+LoRA. All three variants retain identical cell interfaces, semantic initialization, schemas, and splits. The variant with a randomly initialized frozen backbone uses a fresh LFM2.5-230M with the same configuration: attention, feed-forward, and short-convolution blocks are reinitialized using the model's native rules (linear and convolutional weights N(0,0.022), zero biases where present, and unit normalization scales). All backbone parameters remain frozen. Column and code embeddings and the beginning-of-row vector are initialized from pretrained token embeddings before replacement and remain trainable alongside the numerical position maps and typed heads. LoRA adapts all backbone linear layers (rank 16, alpha 32, dropout 0.05; adapter/interface learning rates 1e-5/3e-4).
In this table, FAITH and FAITH+LoRA report means accompanied by 95% Student’s t confidence interval half-widths across 3 seeds. The table shows that FAITH with a pretrained backbone outperforms FAITH with a randomly initialized frozen backbone.
LoRA has a higher mean on 6/10 datasets. However, the improvements are modest, and FAITH+LoRA consumes more GPU memory. One of our initial goals was to keep the model memory-efficient. Below, we report peak GPU memory usage during training for FAITH and FAITH+LoRA on the most memory-intensive dataset, Allstate (
batch_size=32for both runs):This reduction in memory usage may be important when GPU memory is limited.
We want to clarify our main goals. Our objective is efficient LLM-based generation with quality competitive with diffusion models; this is why our main competitors are LLM-based methods such as GReaT and Tabby. We measure efficiency in terms of training and sampling times and the GPU memory required. FAITH is much faster to train and sample from than GReaT or Tabby, and it also requires much less GPU memory. Our hypothesis is that using a pretrained LLM backbone allows the model to better understand complex relationships in large datasets.
We tried to evaluate DiffLM, but it did not yield satisfactory output quality. In fact, the model’s output was gibberish; we suspect a bug in its implementation.
We evaluated TabNAT on every dataset reported in the paper, including two additional large datasets: Ecom Offers (160,057 rows, 119 predictors; repeat-buyer classification) and NYC Green Taxi 2016 (150,000-trip subset, 16 predictors; tip-amount regression). Below, we compare MLE results.
FAITH leads on most datasets, including the largest ones: Allstate (a large improvement over TabNAT), HBA1C, Ecom (also a major improvement), and Taxi. This supports our hypothesis about large datasets.