- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes LATTE for prediction on tables with numerical and string columns. It serializes each row with its column names, extracts representations from an intermediate layer of a frozen LLM, and pools them separately for each string column. After standardization and per-column PCA, these embeddings are combined with the original numerical features, low-cardinality categorical features, and cross-fitted target encodings for a tabular learner. This lets the LLM supply contextual features while the learner retains direct access to numerical values and category information. Experiments on 100 datasets from TextTabBench, CARTE, and WikiDBs show gains over cell-wise encoders and other baselines across several training sizes, seven LLM backbones, and ten learners. Ablations support the use of inter-column context, intermediate layers, and the numerical bypass.
Critical strengths
-
The performance gains hold across datasets and models. LATTE improves over strong baselines with multiple LLM backbones and tabular learners, winning about 89% of splits against the strongest Elo baseline on the 100-dataset benchmark. These results support its practical value.
-
The design is practical, and the ablations offer useful guidance. LATTE combines LLM representations with tabular learners without task-specific LLM fine-tuning. Ablations support the roles of row context and direct numerical inputs, providing concrete guidance for applying LLMs to tables with strings.
Critical weaknesses
-
The contribution of columnar pooling remains unclear. The comparison also increases representation dimensionality, making it difficult to separate the benefit of column structure from that of additional capacity. The authors acknowledge this limitation, but a dimension-matched comparison is still needed to support this central design choice.
-
Independent validation needs clarification. Key settings were selected on the main benchmark, while STRABLE results are not reported separately for non-overlapping datasets and the choice of 55 baselines is unexplained. These gaps limit confidence in generalization to new tasks and the scope of the best-performance claims.
Questions for authors
-
With the same extraction layer, final total feature dimensionality, numerical/categorical bypasses, and TFM, does columnar pooling still outperform global mean or last-token pooling? Could you provide this comparison within the complete pipeline to establish how much of the advantage in Figure F.3 is independent of representation dimensionality?
-
Which training/validation/test splits were used to choose the extraction layer, feature-capacity window, and cardinality threshold? What are the full configurations, selection criteria, and score sources for the 55 baselines in Figure 3? Do they use exactly the same STRABLE splits and training-sample budgets as LATTE, and do they cover the strongest post-processing configurations in the original benchmark?
-
With the current settings fixed and the shared sources listed in A.6 removed, do LATTE's gains and ranking relative to the strongest baseline persist? Could you report paired differences and dataset-level uncertainty on this non-overlapping subset, and define the existing error bars in Figure 3? This would directly strengthen the external validation without requiring a redesigned benchmark.
2 comments
Thank the authors for the additional experiments and clarifications. I think my main concerns have been addressed and have adjusted my scores accordingly.
We thank the reviewer for noting that LATTE's gains hold across datasets, LLM backbones and learners, and that its design is practical and its ablations useful.
Q1 / W1: columnar pooling at matched dimensionality
The table below compares pooling strategies at the same layer (14), same dimensionality, same by-passes and same TFM (TabPFN-2.5), on all 100 tables x 5 seeds. Raw scores x100, 95% bootstrap CI over tables:
At matched dimension, columnar pooling is on par with mean pooling on regression, and better on classification (p = 0.008), and is far ahead of last-token pooling on both, with no loss on any table. Section 5.2 now states this (in blue). The new Appendix F.5 crosses pooling with row context, layer and the categorical by-pass in a full factorial ablation. Averaged over these settings, columnar pooling gains +0.43 R² [+0.05, +0.83] and +0.65 AUC [+0.36, +0.99], and it is the largest contributor to LATTE's classification gain.
Q2a: splits used for selection
LATTE's three settings were chosen on the same evaluation splits. The layer was chosen on 27 of the 100 tables, and the feature-capacity window and the cardinality threshold on all 100. We therefore checked the 73 tables never used for the layer:
The Elo gap is +340 on these tables, against +333 on all 100. Re-selecting all three settings on the 27 tables alone gives the same layer (14) and values on the same plateaus, at a cost of at most 0.17 normalized points on the 73 tables.
Q2b: the STRABLE pipelines of Figure 3
Figure 3 shows 51 baselines and 4 LATTE variants, listed with their raw scores in the new Table H.5 (p 40). The baselines include STRABLE's own selected encoders and learners, including the strongest post-processing configurations of the original benchmark. The scores of 50 of them come from the STRABLE authors' result files. We ran only TF-IDF + Causilo, and LATTE, with STRABLE's 3-fold splits (same seed) protocol code.
Q3: STRABLE without the shared sources
We report results on STRABLE without datasets sharing sources with our benchmark, with paired differences of LATTE + Causilo against the strongest baseline (TF-IDF + Causilo). Normalized score: 1 = best baseline of each table. Details are in the new Appendix F.8 (p 34).
On the non-overlapping tables, for regression, LATTE is significantly ahead of all STRABLE's baselines, tuned learners, and every post-processing variant included. On classification, LATTE keeps the first place in mean normalized score, but the paired difference with the strongest baseline is not significant. Conversely, no pipeline of the leaderboard is significantly ahead of it.
The error bars of Figure 3 are 95% normal intervals of the mean normalized score over the 108 tables (±1.96 SD/√108), each table being averaged over its 3 folds.