acceptodds
Under review as a conference paper at ICLR 2027

LATTE: Bridging LLMs and Tabular Foundation Models for Tabular Learning with Strings

Abstract

Real-world tables often mix numerical features with free-form strings. Table foundation models (TFMs) excel on numerical tables but require strings to be encoded numerically, while large language models (LLMs) hold the world knowledge those strings refer to. Most existing bridges use the LLM as a plug-in encoder that takes an isolated cell in, and gives a mean-pooled last-layer vector out. We show that this *interface* between LLM and table matters as much as the LLM itself. Read one cell at a time, a larger LLM does not give better features: on average, Qwen3-0.6B even beats Qwen3-8B. With our interface, built from three ingredients—*row serialization* (giving every cell its row-level context), *intermediate-layer extraction* (avoiding layers specialized for next-token prediction) and *columnar pooling* (removing mean pooling's length bias)—the same LLMs improve steadily with size. With *by-pass features* routing numbers and exact cell identity past the LLM, these ingredients form LATTE (LLM-Augmented Table Text Embeddings). Paired with TFMs, LATTE sets a new state of the art for tables with strings. Across 100 diverse tables, it beats the strongest baseline on 89% of splits, with gains across 7 LLMs and 10 tabular learners, and ranks first on the 108-dataset STRABLE benchmark.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Learning, Large Language Models, Table Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zuGJOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes LATTE for prediction on tables with numerical and string columns. It serializes each row with its column names, extracts representations from an intermediate layer of a frozen LLM, and pools them separately for each string column. After standardization and per-column PCA, these embeddings are combined with the original numerical features, low-cardinality categorical features, and cross-fitted target encodings for a tabular learner. This lets the LLM supply contextual features while the learner retains direct access to numerical values and category information. Experiments on 100 datasets from TextTabBench, CARTE, and WikiDBs show gains over cell-wise encoders and other baselines across several training sizes, seven LLM backbones, and ten learners. Ablations support the use of inter-column context, intermediate layers, and the numerical bypass.

Critical strengths
  1. The performance gains hold across datasets and models. LATTE improves over strong baselines with multiple LLM backbones and tabular learners, winning about 89% of splits against the strongest Elo baseline on the 100-dataset benchmark. These results support its practical value.

  2. The design is practical, and the ablations offer useful guidance. LATTE combines LLM representations with tabular learners without task-specific LLM fine-tuning. Ablations support the roles of row context and direct numerical inputs, providing concrete guidance for applying LLMs to tables with strings.

Critical weaknesses
  1. The contribution of columnar pooling remains unclear. The comparison also increases representation dimensionality, making it difficult to separate the benefit of column structure from that of additional capacity. The authors acknowledge this limitation, but a dimension-matched comparison is still needed to support this central design choice.

  2. Independent validation needs clarification. Key settings were selected on the main benchmark, while STRABLE results are not reported separately for non-overlapping datasets and the choice of 55 baselines is unexplained. These gaps limit confidence in generalization to new tasks and the scope of the best-performance claims.

Questions for authors
  1. With the same extraction layer, final total feature dimensionality, numerical/categorical bypasses, and TFM, does columnar pooling still outperform global mean or last-token pooling? Could you provide this comparison within the complete pipeline to establish how much of the advantage in Figure F.3 is independent of representation dimensionality?

  2. Which training/validation/test splits were used to choose the extraction layer, feature-capacity window, and cardinality threshold? What are the full configurations, selection criteria, and score sources for the 55 baselines in Figure 3? Do they use exactly the same STRABLE splits and training-sample budgets as LATTE, and do they cover the strongest post-processing configurations in the original benchmark?

  3. With the current settings fixed and the shared sources listed in A.6 removed, do LATTE's gains and ranking relative to the strongest baseline persist? Could you report paired differences and dataset-level uncertainty on this non-overlapping subset, and define the existing error bars in Figure 3? This would directly strengthen the external validation without requiring a redesigned benchmark.

2 comments

We thank the reviewer for noting that LATTE's gains hold across datasets, LLM backbones and learners, and that its design is practical and its ablations useful.

Q1 / W1: columnar pooling at matched dimensionality

The table below compares pooling strategies at the same layer (14), same dimensionality, same by-passes and same TFM (TabPFN-2.5), on all 100 tables x 5 seeds. Raw scores x100, 95% bootstrap CI over tables:

PoolingR² (65 tables)AUC (35 tables)Δ R² with columnar,Δ AUC with columnar
Columnar (LATTE)70.889.5––
Global mean70.989.1-0.1 [-0.5, +0.3]+0.36 [+0.12, +0.67]
Last token67.388.1+3.4 [+2.6, +4.4]+1.41 [+0.96, +1.90]

At matched dimension, columnar pooling is on par with mean pooling on regression, and better on classification (p = 0.008), and is far ahead of last-token pooling on both, with no loss on any table. Section 5.2 now states this (in blue). The new Appendix F.5 crosses pooling with row context, layer and the categorical by-pass in a full factorial ablation. Averaged over these settings, columnar pooling gains +0.43 R² [+0.05, +0.83] and +0.65 AUC [+0.36, +0.99], and it is the largest contributor to LATTE's classification gain.

Q2a: splits used for selection

LATTE's three settings were chosen on the same evaluation splits. The layer was chosen on 27 of the 100 tables, and the feature-capacity window and the cardinality threshold on all 100. We therefore checked the 73 tables never used for the layer:

73 held-out tablesLATTE −- strongest baselineWins / ties / losses
Regression (48)+6.2 R² [+4.2, +8.5]42 / 3 / 3
Classification (25)+0.7 AUC [+0.4, +1.0]19 / 6 / 0

The Elo gap is +340 on these tables, against +333 on all 100. Re-selecting all three settings on the 27 tables alone gives the same layer (14) and values on the same plateaus, at a cost of at most 0.17 normalized points on the 73 tables.

Q2b: the STRABLE pipelines of Figure 3

Figure 3 shows 51 baselines and 4 LATTE variants, listed with their raw scores in the new Table H.5 (p 40). The baselines include STRABLE's own selected encoders and learners, including the strongest post-processing configurations of the original benchmark. The scores of 50 of them come from the STRABLE authors' result files. We ran only TF-IDF + Causilo, and LATTE, with STRABLE's 3-fold splits (same seed) protocol code.

Q3: STRABLE without the shared sources

We report results on STRABLE without datasets sharing sources with our benchmark, with paired differences of LATTE + Causilo against the strongest baseline (TF-IDF + Causilo). Normalized score: 1 = best baseline of each table. Details are in the new Appendix F.8 (p 34).

STRABLE tablesRegression: normalized score (position)Δ R² [95% CI], W/T/LClassification: normalized score (position)Δ AUC [95% CI], W/T/L
All 1081.043 (1st)+2.4 [+1.0, +4.1], 55/4/170.977 (1st)+0.04 [-0.87, +0.83], 22/1/9
Without shared (88)1.050 (1st)+2.3 [+0.8, +4.3], 46/3/150.966 (1st)-0.33 [-1.49, +0.66], 14/1/9

On the non-overlapping tables, for regression, LATTE is significantly ahead of all STRABLE's baselines, tuned learners, and every post-processing variant included. On classification, LATTE keeps the first place in mean normalized score, but the paired difference with the strongest baseline is not significant. Conversely, no pipeline of the leaderboard is significantly ahead of it.

The error bars of Figure 3 are 95% normal intervals of the mean normalized score over the 108 tables (±1.96 SD/√108), each table being averaged over its 3 folds.

Thank the authors for the additional experiments and clarifications. I think my main concerns have been addressed and have adjusted my scores accordingly.

Reviewer_rLpsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

LATTE puts a frozen decoder LLM in front of a table foundation model. Each row is serialized with the TabLLM template; the LLM is truncated at an intermediate layer; token states inside each string column's span (header + value) are mean-pooled and PCA-reduced per column; raw numerics, native categoricals and out-of-fold target encodings are concatenated by-passing the LLM. Claim: the LLM–table interface, not the LLM, decides whether scale helps.

Critical strengths
  • Simple, learner-agnostic, uses only public frozen components; gains hold across seven backbones and ten learners, including GBDTs.

  • Evaluation is thorough — 100 datasets, shared splits, three training sizes, per-dataset raw tables, bootstrap CIs on the Elo gap, explicit STRABLE overlap audit, and honest disclosure of caveats.

Critical weaknesses
  • Attribution is incomplete and columnar pooling is never ablated end to end.

  • Min-max normalization uses a pool containing three LATTE variants; Elo counts wins on five near-saturated WikiDBs classification tasks equally. Raw classification gain over the best baseline is ~1.1 AUC.

Questions for authors

How much survives with a small bidirectional encoder instead of a decoder LLM?

2 comments

We thank the reviewer for noting LATTE's easy applicability (frozen public components, gains across seven backbones and ten learners) and the thoroughness and transparency of the evaluation.

1. Attribution, with columnar pooling ablated end to end

We ran the full factorial ablation over row context (full vs. inter-column attention masked), layer (intermediate vs. last), pooling (columnar vs. a global mean reduced to the same feature count) and the categorical by-pass, with TabPFN-2.5 and everything else fixed, on all 100 tables x 5 seeds.

The table below shows the effect of removing individual components from LATTE (raw scores x100, 95% bootstrap CI); the total gain is from the configuration with all four components off to LATTE. In the new Appendix F.5 (pp 31-32), the results are given for all 16 configurations (Table F.1, Figure F.4).

ComponentLoss if removed from LATTE: R²AUCShapley share of the total gain: R² (+4.6)AUC (+1.5)
Row-level context2.3 [1.8, 2.9]0.45 [0.24, 0.69]2.10.40
Intermediate layer1.5 [1.0, 2.0]0.04 [-0.25, 0.30]1.60.19
Columnar pooling-0.1 [-0.5, 0.3]0.36 [0.12, 0.67]0.40.65
Categorical by-pass0.19 [0.05, 0.33]0.09 [-0.04, 0.25]0.60.30

On regression, row context is the most important component (45% of the gain), followed by the intermediate layer (34%). On classification, the largest gains come from using columnar pooling instead of mean-pooling (42% of the gain). Columnar pooling is also far ahead of last-token pooling (+3.4 R² [2.6, 4.4], +1.4 AUC [1.0, 1.9], no loss on any table).

2. Normalization pool, saturated tasks, and the raw AUC gain

Re-analysis of the submission's results, against the strongest of the 17 pool baselines:

RegressionClassification
Normalized score, paper pool: LATTE vs. best baseline90.2 vs. 74.188.9 vs. 80.6
Same, pool of the 17 baselines only (LATTE excluded, unclipped)99.3 vs. 80.696.3 vs. 86.0
Raw gain over the strongest baseline+6.8 R² [+5.1, +8.6]+1.0 AUC [+0.7, +1.5]
Wins / ties / losses over datasets56 / 5 / 428 / 7 / 0
Share of the remaining error removed18.9% [14.5, 23.6]9.0% [6.3, 12.8]

Taking LATTE out of the pool widens its lead, and the order of the top methods is unchanged. A score of 100 means matching an oracle that picks the best baseline per dataset.

The Elo gap between the best LATTE and the best other method is +333 on all tasks, +341 without the five near-saturated WikiDBs tasks, and +348 without all 17 WikiDBs classification tasks. All six LATTE learners stay above every other method. Removing the saturated tasks raises the gap.

The gain in raw AUC is smaller than R² because AUC saturates, but it is consistent. LATTE loses on no classification dataset and removes 9% of the remaining error.

3. Small bidirectional encoders

We tried LATTE with four encoders (same row serialization, layer selected by the same probe, columnar pooling, by-passes and TabPFN-2.5). The results are in the new Appendix F.7 (p 33, Table F.3).

Encoder in the LATTE interfaceR²AUCRow interface −- same encoder cell-wise (normalized)Share of LATTE's gain over TF-IDF
TF-IDF (reference)63.987.3–0%
MiniLM-L6 (23M)65.287.5-0.5 [-1.7, +0.8]13%
ModernBERT-base (149M)64.988.0+1.7 [+0.9, +2.6]17%
ModernBERT-large (395M)64.888.0+1.8 [+0.8, +2.7]16%
E5-large-v2 (335M)66.688.4+0.5 [-0.4, +1.6]41%
Llama-3.1-8B70.889.5+12.2 [+10.1, +14.3]100%

Between 13% and 41% of LATTE's gain over TF-IDF survives. However, the main difference with large decoder-only models is that small encoders gain almost nothing from reading the row context instead of just the cell. We hypothesize that model capacity is the limiting factor, as larger models benefit more from row context (Figure 4). All four encoders are smaller than Qwen3-0.6B, the smallest decoder-only model used in our main study. Their limited gain is thus likely not a matter of bidirectionality, but of current encoders being too small.

Reviewer_rLpsOpenReview ↗

Final Recommendation

Thank you for the detailed rebuttal. Most of my concerns have been addressed. I confirm my original score.

Reviewer_KRq6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LATTE, a framework that integrates LLM representations with table foundation models for tabular learning with string features. LATTE combines row serialization, intermediate-layer extraction, and columnar pooling to obtain contextualized, column-aligned embeddings, while using bypass features to preserve numerical and categorical information. Experiments on 100 datasets and the external STRABLE benchmark demonstrate consistent improvements over existing string encoders and tabular learning baselines. Ablation studies further validate the effectiveness of the proposed components.

Critical strengths

S1. Effective integration of LLMs and tabular learners. LATTE provides a practical interface that incorporates row-level semantic context while preserving column structure. The finding that larger LLMs benefit more from contextualized representations than isolated cell embeddings is particularly interesting.

S2. Strong experimental validation. The evaluation covers 100 datasets, seven LLM backbones, and ten tabular learners, with additional validation on STRABLE. The consistent improvements and extensive ablations provide convincing evidence of the method's effectiveness.

Critical weaknesses

W1. Limited isolation of individual contributions. Although LATTE provides extensive ablations, its overall performance depends on several interacting design choices, including row-level contextualization, intermediate-layer extraction, pooling, and bypass features. The large impact of numerical bypass features makes it important to further clarify the relative contribution of the proposed interface under more tightly controlled comparisons.

W2. Sensitivity to column ordering and computational overhead. The method relies on causal LLM representations, making its performance sensitive to column order, as shown in Table 2. Additionally, the extra LLM inference cost may limit scalability compared with lightweight string encoders.

Questions for authors

Q1. Could the authors further disentangle the contributions of row-level contextualization, intermediate-layer extraction, and columnar pooling under a unified experimental setting?

Q2. How robust is LATTE to random column permutations beyond the reversed-order experiment in Table 2?

Q3. Could the authors further clarify the accuracy-efficiency trade-off between LATTE and lightweight alternatives such as TF-IDF combined with recent TFMs?

2 comments

We thank the reviewer for highlighting LATTE's practical integration of LLMs with tabular learners, the finding that larger LLMs benefit from row context, and the breadth of the validation. We updated the paper (changes in blue), and ran three new experiments:

Q1 / W1: contributions of isolated components

We ran the full factorial ablation over row context (full vs. inter-column attention masked), layer (intermediate vs. last), pooling (columnar vs. a global mean reduced to the same feature count) and the categorical by-pass, with TabPFN-2.5 and everything else fixed, on all 100 tables x 5 seeds.

The table below shows the effect of removing single components from LATTE (raw scores x100, 95% bootstrap CI); the total gain is from the configuration with all four components off to LATTE. In the new Appendix F.5 (pp 31-32), the results are given for all 16 configurations (Table F.1, Figure F.4).

ComponentLoss if removed from LATTE: R²AUCShapley share of the total gain: R² (+4.6)AUC (+1.5)
Row-level context2.3 [1.8, 2.9]0.45 [0.24, 0.69]2.10.40
Intermediate layer1.5 [1.0, 2.0]0.04 [-0.25, 0.30]1.60.19
Columnar pooling-0.1 [-0.5, 0.3]0.36 [0.12, 0.67]0.40.65
Categorical by-pass0.19 [0.05, 0.33]0.09 [-0.04, 0.25]0.60.30

On regression, row context is the most important component (45% of the gain), followed by the intermediate layer (34%). On classification, the largest gains come from columnar pooling (42% of the gain). The numerical by-pass is also used by the baselines and thus does not explain LATTE's lead.

Q2 / W2: random column permutations

We ran LATTE under three random column permutations, on the same splits. Only the order of the columns the LLM reads changes. The results are in the new Appendix F.6 (p 33, Table F.2).

Regression (R²)Classification (AUC)
LATTE, table order / random orders 1, 2, 370.8 / 70.5, 70.4, 70.789.46 / 89.44, 89.38, 89.35
SD of one run across orders vs. across seeds1.1 vs. 3.10.4 vs. 0.8
Worst random order −- strongest baseline+6.5 [+4.8, +8.4]+0.81 [+0.50, +1.16]

On average a random order costs only half the cost of the full reversal of Table 2. Indeed, reversal is often a worst case, since tables often put identifying columns first. Nevertheless, under all four column orders, LATTE beats the strongest baseline with a CI above zero.

Q3 / W2: accuracy-efficiency trade-off

Median wall-clock time of one run (1,024 train + up to 1,024 test rows; LLM extraction, on a V100 GPU, included):

MethodR²AUCSeconds per run
TF-IDF + TabPFN-2.563.987.34.4
TF-IDF + TabPFN-3.565.487.94.7
Llama-3.1-8B, cell-wise + TabPFN-2.564.688.416
LATTE (TabPFN-2.5)70.889.528

LATTE costs roughly 6 times a TF-IDF + TFM run, but moves far along the accuracy axis. Its improvability (error relative to the best method per table, lower is better) is 3.7% on regression and 11.5% on classification, against 18.4% and 30.3% for the best TF-IDF + TFM.

TF-IDF with a recent TFM is thus the right choice when compute is tight, and LATTE when accuracy matters.

Reviewer_KRq6OpenReview ↗

Final Recommendation

Thank you for the detailed rebuttal. My main concerns have been addressed. I confirm the original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussions

We thank all the reviewers for their constructive comments and are glad they positively assessed our contribution.

During the rebuttal, we ran new experiments to address all the questions that were raised. They confirm or strengthen the paper's claims. The results are in new appendices (in blue), with two sentences also added to the main text, and we provide a summary below:

  • Every component of LATTE contributes to the gains (KRq6, rLps, zuGJ). In a full 2⁴ factorial ablation (App. F.5), we show that row context yields the largest share of the gain in regression (45%) and columnar pooling in classification (42%), at a matched feature count.
  • LATTE’s lead on STRABLE holds when we remove the overlap with our benchmark (zuGJ). Without the tables that share a source with our benchmark, LATTE keeps first place in both regression and classification (App. F.8).
  • LATTE costs about 6 times a TF-IDF + TFM run, but using LATTE instead of TF-IDF reduces the improvability from 18.4% to 3.7% in regression, and from 30.3% to 11.5% in classification (KRq6).
  • LATTE is robust to column permutation (KRq6). Under its least favorable random column order, LATTE still leads the best baseline by +6.5 R² [95% CI 4.8, 8.4] and +0.81 AUC [+0.50, +1.16] (App. F.6).
  • On the 73 tables never used to choose LATTE’s settings, the Elo gap between LATTE and the strongest baseline reaches +340, against +333 on the full 100 tables (zuGJ).
  • Dropping the 5 near-saturated tasks increases LATTE's lead from +333 to +341 Elo (rLps).
  • Row context pays off with a large LLM (rLps). Small bidirectional encoders gain little from reading the whole row, consistent with model capacity being the limit (App. F.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.