- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces KnowsTFM, a knowledge-informed fine-tuning framework for tabular foundation models (TFMs) that incorporates external relational knowledge into feature-level attention. The proposed method constructs feature interaction graphs from curated knowledge bases such as PrimeKG and Wikidata, or directly from LLM-generated relationships, and injects these structural priors into transformer attention through hard masks or learnable soft biases, combined with parameter-efficient adaptation (LoRA). The goal is to improve domain adaptation in data-scarce settings while preserving pretrained in-context learning capabilities.
The authors evaluate their approach across four NanoTabPFN checkpoints and TabPFN-v3 using 42 biomedical gene-expression datasets (CUMIDA-42) and 13 heterogeneous cross-domain datasets. Additional experiments examine feature-selection regimes, graph sources, random graph controls, ground-truth causal graphs, and transfer to NanoTabICL. Results show improvements over standard fine-tuning, particularly for smaller models on biomedical tasks, while gains on the frontier model are more modest and cross-domain improvements are less consistent. Overall, the study suggests that structural attention priors can benefit TFM adaptation, although their effectiveness depends on model capacity, adaptation strategy, and the available feature structure.
Critical strengths
-
A well-motivated and practical framework for incorporating external knowledge into tabular foundation models. The paper addresses an important challenge in adapting pretrained TFMs to data-scarce domains where external relational knowledge is available but difficult to incorporate effectively. Injecting graph-derived structural priors directly into feature attention through hard masks or learnable soft biases is conceptually simple, modular, and compatible with parameter-efficient fine-tuning. The applicability to both curated knowledge graphs and LLM-generated feature relationships further increases the potential utility of the approach.
-
Extensive empirical evaluation with informative ablations across model scales and adaptation regimes. The evaluation spans 42 biomedical datasets, 13 cross-domain datasets, multiple NanoTabPFN checkpoints, and the frontier TabPFN-v3 model, with additional transfer experiments on NanoTabICL.
Critical weaknesses
-
Insufficient evidence that the improvements arise from meaningful external knowledge rather than generic structural regularization. The central claim of the paper is that knowledge-grounded feature relationships improve TFM adaptation. However, the random-graph controls do not systematically preserve graph density or topology, making it difficult to isolate the contribution of semantically meaningful edges from the regularizing effects of constrained attention. Furthermore, experiments using ground-truth causal graphs do not demonstrate a consistent advantage over alternative graph structures. This is consequential because the semantic value of external knowledge is the primary motivation and distinguishing contribution of the proposed framework, yet the experiments do not convincingly establish that accurate knowledge relationships are necessary for the observed improvements.
-
Limited improvements over strong adaptation baselines, particularly for the frontier model. Although KnowsTFM shows meaningful gains for smaller NanoTabPFN checkpoints, its improvements on TabPFN-v3 are marginal. On CUMIDA-42, KnowsTFM achieves 86.3% balanced accuracy compared with 86.0% for graph-free LoRA and 86.2% for zero-shot inference. On the cross-domain benchmark, graph-free LoRA achieves comparable or slightly better average performance. Moreover, the reported statistical significance primarily concerns comparisons against vanilla full fine-tuning, which itself degrades the pretrained frontier model. Consequently, the experiments provide limited evidence that knowledge injection offers a substantial advantage over parameter-efficient adaptation alone. This weakens the practical significance and generality of the proposed method, especially for strongly pretrained TFMs.
Questions for authors
-
Comparison with Relational Foundation Models (RFMs). How does KnowsTFM compare with recent Relational Foundation Models which are explicitly designed to learn and exploit relational structure?
-
The paper reports improvements with knowledge-grounded attention, but it remains unclear whether these gains arise from semantically meaningful relationships or generic structural regularization.
-
The paper motivates knowledge-informed adaptation primarily through data scarcity, yet the reported analysis does not establish a clear relationship between training-set size and the observed gains. Do the authors have evidence that KnowsTFM becomes more beneficial as the number of labelled examples decreases?
3 comments
W2. Limited improvements over strong adaptation baselines on the frontier model.
[New] Graph-free LoRA on the same three blocks as KnowsTFM, so that the only difference is the gate.
Table R2. TabPFN-v3: KnowsTFM minus each baseline (percentage points, Wilcoxon p).
| Setting | vs zero-shot | vs LoRA (all blocks) | vs LoRA (same 3 blocks) | vs density-matched random |
|---|---|---|---|---|
| CUMIDA-42 rnd100 | +0.18, p=0.13 | +0.35, p=0.008 | +0.29, p=0.003 | +0.34, p=0.001 |
| xdomain13 | +0.31, p=0.024 | −0.05, p=0.83 | +0.08, p=0.91 | +0.05, p=0.91 |
On CUMIDA-42 rnd100, the curated graph improves on parameter-efficient adaptation, also with LoRA in the same blocks, whereas the gate with a random graph does not (85.99 vs 86.04). On xdomain13, graph-free LoRA is equally good.
Implication. Under a matched adaptation protocol, the frontier-model contribution is specific: the gate with the curated graph adds 0.29 points over LoRA in the same blocks, and the same gate with a random graph adds nothing. We present this as a result for rnd100 and do not claim a general advantage over parameter-efficient adaptation.
In the revision: Q1 and Q2 paragraphs of Section 5; Appendix O (Table 29).
Q1. Comparison with Relational Foundation Models?
RFMs such as KumoRFM and Griffin are pretrained on multi-table relational databases, with rows as nodes and key links as edges. Our setting is a single table without a relational schema, so there is no structure for an RFM to exploit; our graph is over columns and comes from an external source.
In the revision: added to the related work (Section 2).
Q3. Does KnowsTFM become more beneficial as labels decrease?
[New] Label-budget sweep at ten budgets (4 to 20 labelled rows per class), all five models.
- Setup: all five models, same seeds, folds and learning-rate grid as in the submission. A dataset enters at budget K only if every fold has K rows in every class, so the set of eligible datasets shrinks as K increases. No CUMIDA-42 dataset supports K=50, so 20 is the largest budget we can report.
Table R3. CUMIDA-42 rnd100, K labelled rows per class, differences in percentage points; * p < 0.05; † too few non-tied datasets for a test.
| K per class | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 12 | 15 | 20 |
|---|---|---|---|---|---|---|---|---|---|---|
| NanoTabPFN 1.1M: KnowsTFM − vanilla FT | +1.31 | +2.82* | −1.94 | +2.03* | +3.63* | +3.57* | +6.03* | +0.11 | +4.16* | +3.89* |
| TabPFN-v3: KnowsTFM − vanilla FT | +0.16 | +0.10 | +0.17 | +0.13 | +0.02 | +0.14 | −0.06 | +0.46* | +0.11 | +0.86† |
| TabPFN-v3: KnowsTFM − zero-shot | +0.01 | +0.14 | −0.01 | +0.10 | −0.02 | +0.12* | +0.05 | −0.16 | +0.01 | +0.04 |
The benefit does not grow as labels decrease.
Implication. On the 1.1M small-prior model, KnowsTFM outperforms vanilla fine-tuning at nine of ten budgets, significantly at seven, by up to 6.03 points; TabPFN-v3 with KnowsTFM remains close to zero-shot throughout. The motivation is therefore adaptation with limited labels in the p >> n regime, without a claim that the benefit increases at smaller budgets. Because the eligible datasets change with K, these results establish neither a general label-budget trend nor a threshold.
In the revision: Appendix P (Tables 30 to 32) and one sentence in the Q1 paragraph of Section 5.
Presentation.
In the revision, we added a Q3 paragraph on knowledge versus structure to Section 5, replaced the random-graph figure with matched controls (Figures 3 and 4), added a TabPFN-v3 figure (Figure 2), and moved the new controls, paired tests, label budget, relation policies, and cost into Appendices M to R, each referenced from the main text.
Follow-up on W2: KnowsTFM on 52 further datasets (TabArena, BeyondArena)
Following up on W2, we have extended the frontier-model evaluation to 65 datasets: xdomain13 and two standard benchmarks, TabArena (20) and BeyondArena (32), with their official splits (3 repeats x 3 folds) and metrics.
W2: "its improvements on TabPFN-v3 are marginal [...] the experiments provide limited evidence that knowledge injection offers a substantial advantage over parameter-efficient adaptation alone."
We respectfully disagree. Over these 65 datasets, KnowsTFM (Wikidata) beats vanilla fine-tuning on 55 and zero-shot on 37 (26 losses), and its pooled improvability is 0.51%, vs 1.29% for zero-shot and 4.90% for vanilla FT (Table G).
On a strong pretrained model, small differences are the scale at which the best methods differ. On the official TabArena results for our 20 datasets, neighbouring methods of the top 10 differ by a median of 0.0001 AUC; over all 30 binary TabArena datasets, the top-10 methods (LimiX-2, Kumo, Seldon, Chakra-Tab, TabPFN-3.5, TabFM and others) differ by a median of 0.0002 AUC on the same 9 splits (Appendix S). We therefore also report TabArena's own summary measure, improvability: the mean relative error gap to the best method on each dataset (lower is better).
Table F. Improvability (%) on CUMIDA-42, TabPFN-v3, balanced accuracy. Best = best of the four methods on each dataset.
| Method | rnd100 | top25rnd75 | top50rnd50 | top100 |
|---|---|---|---|---|
| zero-shot | 1.52 | 2.24 | 1.13 | 1.28 |
| vanilla FT | 4.09 | 1.98 | 1.17 | 2.12 |
| graph-free LoRA | 3.00 | 1.46 | 1.02 | 1.52 |
| KnowsTFM (PrimeKG) | 0.85 | 1.37 | 0.63 | 0.26 |
Table G. Improvability (%) on the cross-domain and standard benchmarks, TabPFN-v3. xdomain13: balanced accuracy; TabArena and BeyondArena: official metrics. All 65 pools the per-dataset gaps.
| Method | xdomain13 | TabArena (20) | BeyondArena (32) | All 52 | All 65 (pooled) |
|---|---|---|---|---|---|
| zero-shot | 1.52 | 0.92 | 1.43 | 1.23 | 1.29 |
| vanilla FT | 6.26 | 3.26 | 5.38 | 4.57 | 4.90 |
| KnowsTFM (Wikidata) | 0.34 | 0.41 | 0.65 | 0.56 | 0.51 |
| KnowsTFM (LLM graph) | 0.67 | 0.92 | 0.83 | 0.87 | 0.83 |
Graph-free LoRA (LoRA on KnowsTFM's three column-attention blocks, no graph) was run on xdomain13 only. With it as a fifth method there, improvability is 0.36% for KnowsTFM (Wikidata), 0.61% for graph-free LoRA, 0.70% for KnowsTFM (LLM graph), 1.55% for zero-shot and 6.28% for vanilla FT.
- KnowsTFM is closest to the best method in all 9 settings, including all 52 standard-benchmark datasets, where it beats vanilla FT on 45 (p<0.001) and needs no curated knowledge graph.
- Against graph-free LoRA, where the graph is the only difference, KnowsTFM has lower improvability in all four CUMIDA regimes (Table F) and on xdomain13. Its accuracy is significantly higher on rnd100 (+0.29, p=0.003, 12/1) and never significantly lower. We therefore respectfully disagree that knowledge injection offers no advantage over parameter-efficient adaptation alone: at the same layers, the graph brings the model consistently closer to the best method.
In the revision: Tables F–G will be added to Appendix S, and all 104 graphs (52 Wikidata, 52 LLM-generated) will be released.
We hope these results, on 65 further datasets, address your concern about the practical benefit on the frontier model, and we would be grateful if you would consider them in your final assessment.
Thank you for the review. We uploaded a revised PDF in which all changes are in blue; [New] marks experiments run for this rebuttal.
Indeed, the original random control did not preserve density. We have now extended this analysis. [New] Matched graph controls: a random graph with the curated graph's edge count (density), a degree-preserving rewiring (topology: every feature keeps its number of neighbours), and graphs from LLM and KG embeddings. Every control has exactly as many edges as the curated graph on each panel: 9.4 on average, i.e. 0.19% of the 4,950 feature pairs of a 100-gene panel. This tests which properties of the curated structure are sufficient to reproduce its performance.
Table R1. CUMIDA-42 rnd100, balanced accuracy (%); bold: best per model.
At the nanoscale, no control performs significantly worse than the curated graph, which supports generic sparsity as an important factor. On TabPFN-v3, the curated graph adds a small effect that random and embedding graphs do not reproduce (+0.26 to +0.34, p ≤ 0.042); the rewiring matches it, so the effect is not established for exact edges.
[New] Density sweep (Figure 8, Table 27). Sparse hard masks help the 1.1M model by 3.3 to 5.1 points, and the curated graph (0.19%) lies in that range (Figure 8a,b). On TabPFN-v3, a sparse hard mask hurts (Table 27), which is why KnowsTFM uses a soft gate; with the soft gate, random graphs at all 37 densities stay close to zero-shot and below the curated graph (Figure 2a).
Implication. The role of the knowledge graph differs across model scales. For NanoTabPFN, the density of the prior is an important factor, and the knowledge graph supplies a useful one: the random control of the submission, whose density was drawn without the knowledge graph, is lower on all four checkpoints (by up to 2.89 points, p = 0.004). On TabPFN-v3, the curated graph provides a small advantage over alternative graphs with the same number of edges, which degree-preserving rewiring retains; graph properties beyond density matter at this scale, and the experiments do not isolate which preserved property explains the effect.
In the revision: the central claim is restated in the abstract, contribution 4, and the conclusion; a new Q3 paragraph in Section 5; the random-graph figure is replaced by matched controls (Figures 3 and 4), and a new TabPFN-v3 figure (Figure 2); Appendices M and N.