acceptodds
Desk rejected at ICLR 2027

KnowsTFM: Towards Knowledge-Grounded Tabular Foundation Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Tabular foundation models have advanced deep learning for tabular data by achieving strong performance across many tasks. However, adapting these methods to data-scarce domains is challenging. Many such domains provide curated relational knowledge in the form of knowledge graphs and bases, but how to use this knowledge to improve and steer specialist tabular foundation models remains unclear. We address this problem through Knowledge-informed fine-tuning of Tabular Foundation Models (\modelname), which incorporates relational knowledge directly into feature attention. We evaluate the approach across controlled nanoscale TabPFN checkpoints and the frontier TabPFN-v3 model to study when external structure improves downstream adaptation. During fine-tuning, we use structural priors derived from knowledge graphs to guide feature-to-feature attention, encouraging interactions between features linked in the graph. Our results on 42 biomedical and 13 cross-domain datasets show that this approach yields meaningful gains over standard fine-tuning in domain-specific tasks, while maintaining competitive performance on general-domain tasks.Source code is available at https://anonymous.4open.science/r/kg_tuned_tfms-DBC5.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
tabular foundational models, fine-tuning, knowledge adaptation, knowledge graphs

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_p7hROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces KnowsTFM, a knowledge-informed fine-tuning framework for tabular foundation models (TFMs) that incorporates external relational knowledge into feature-level attention. The proposed method constructs feature interaction graphs from curated knowledge bases such as PrimeKG and Wikidata, or directly from LLM-generated relationships, and injects these structural priors into transformer attention through hard masks or learnable soft biases, combined with parameter-efficient adaptation (LoRA). The goal is to improve domain adaptation in data-scarce settings while preserving pretrained in-context learning capabilities.

The authors evaluate their approach across four NanoTabPFN checkpoints and TabPFN-v3 using 42 biomedical gene-expression datasets (CUMIDA-42) and 13 heterogeneous cross-domain datasets. Additional experiments examine feature-selection regimes, graph sources, random graph controls, ground-truth causal graphs, and transfer to NanoTabICL. Results show improvements over standard fine-tuning, particularly for smaller models on biomedical tasks, while gains on the frontier model are more modest and cross-domain improvements are less consistent. Overall, the study suggests that structural attention priors can benefit TFM adaptation, although their effectiveness depends on model capacity, adaptation strategy, and the available feature structure.

Critical strengths
  1. A well-motivated and practical framework for incorporating external knowledge into tabular foundation models. The paper addresses an important challenge in adapting pretrained TFMs to data-scarce domains where external relational knowledge is available but difficult to incorporate effectively. Injecting graph-derived structural priors directly into feature attention through hard masks or learnable soft biases is conceptually simple, modular, and compatible with parameter-efficient fine-tuning. The applicability to both curated knowledge graphs and LLM-generated feature relationships further increases the potential utility of the approach.

  2. Extensive empirical evaluation with informative ablations across model scales and adaptation regimes. The evaluation spans 42 biomedical datasets, 13 cross-domain datasets, multiple NanoTabPFN checkpoints, and the frontier TabPFN-v3 model, with additional transfer experiments on NanoTabICL.

Critical weaknesses
  1. Insufficient evidence that the improvements arise from meaningful external knowledge rather than generic structural regularization. The central claim of the paper is that knowledge-grounded feature relationships improve TFM adaptation. However, the random-graph controls do not systematically preserve graph density or topology, making it difficult to isolate the contribution of semantically meaningful edges from the regularizing effects of constrained attention. Furthermore, experiments using ground-truth causal graphs do not demonstrate a consistent advantage over alternative graph structures. This is consequential because the semantic value of external knowledge is the primary motivation and distinguishing contribution of the proposed framework, yet the experiments do not convincingly establish that accurate knowledge relationships are necessary for the observed improvements.

  2. Limited improvements over strong adaptation baselines, particularly for the frontier model. Although KnowsTFM shows meaningful gains for smaller NanoTabPFN checkpoints, its improvements on TabPFN-v3 are marginal. On CUMIDA-42, KnowsTFM achieves 86.3% balanced accuracy compared with 86.0% for graph-free LoRA and 86.2% for zero-shot inference. On the cross-domain benchmark, graph-free LoRA achieves comparable or slightly better average performance. Moreover, the reported statistical significance primarily concerns comparisons against vanilla full fine-tuning, which itself degrades the pretrained frontier model. Consequently, the experiments provide limited evidence that knowledge injection offers a substantial advantage over parameter-efficient adaptation alone. This weakens the practical significance and generality of the proposed method, especially for strongly pretrained TFMs.

Questions for authors
  1. Comparison with Relational Foundation Models (RFMs). How does KnowsTFM compare with recent Relational Foundation Models which are explicitly designed to learn and exploit relational structure?

  2. The paper reports improvements with knowledge-grounded attention, but it remains unclear whether these gains arise from semantically meaningful relationships or generic structural regularization.

  3. The paper motivates knowledge-informed adaptation primarily through data scarcity, yet the reported analysis does not establish a clear relationship between training-set size and the observed gains. Do the authors have evidence that KnowsTFM becomes more beneficial as the number of labelled examples decreases?

3 comments

Thank you for the review. We uploaded a revised PDF in which all changes are in blue; [New] marks experiments run for this rebuttal.

W1 / Q2. Knowledge or generic structural regularisation? The random-graph controls do not preserve density or topology.

Indeed, the original random control did not preserve density. We have now extended this analysis. [New] Matched graph controls: a random graph with the curated graph's edge count (density), a degree-preserving rewiring (topology: every feature keeps its number of neighbours), and graphs from LLM and KG embeddings. Every control has exactly as many edges as the curated graph on each panel: 9.4 on average, i.e. 0.19% of the 4,950 feature pairs of a 100-gene panel. This tests which properties of the curated structure are sufficient to reproduce its performance.

Table R1. CUMIDA-42 rnd100, balanced accuracy (%); bold: best per model.

ModelCuratedDensity-matched randomRewiredLLM-emb.KG-emb.
NanoTabPFN 1.1M, small prior64.2064.9364.2264.3464.90
NanoTabPFN 3.7M, small prior63.2162.7762.3262.3462.78
NanoTabPFN 1.1M, big prior62.9064.1363.7162.9764.11
NanoTabPFN 3.7M, big prior62.6463.2264.1163.0763.44
TabPFN-v3 53M86.3385.9986.3186.0786.07

At the nanoscale, no control performs significantly worse than the curated graph, which supports generic sparsity as an important factor. On TabPFN-v3, the curated graph adds a small effect that random and embedding graphs do not reproduce (+0.26 to +0.34, p ≤ 0.042); the rewiring matches it, so the effect is not established for exact edges.

[New] Density sweep (Figure 8, Table 27). Sparse hard masks help the 1.1M model by 3.3 to 5.1 points, and the curated graph (0.19%) lies in that range (Figure 8a,b). On TabPFN-v3, a sparse hard mask hurts (Table 27), which is why KnowsTFM uses a soft gate; with the soft gate, random graphs at all 37 densities stay close to zero-shot and below the curated graph (Figure 2a).

Implication. The role of the knowledge graph differs across model scales. For NanoTabPFN, the density of the prior is an important factor, and the knowledge graph supplies a useful one: the random control of the submission, whose density was drawn without the knowledge graph, is lower on all four checkpoints (by up to 2.89 points, p = 0.004). On TabPFN-v3, the curated graph provides a small advantage over alternative graphs with the same number of edges, which degree-preserving rewiring retains; graph properties beyond density matter at this scale, and the experiments do not isolate which preserved property explains the effect.

In the revision: the central claim is restated in the abstract, contribution 4, and the conclusion; a new Q3 paragraph in Section 5; the random-graph figure is replaced by matched controls (Figures 3 and 4), and a new TabPFN-v3 figure (Figure 2); Appendices M and N.

W2. Limited improvements over strong adaptation baselines on the frontier model.

[New] Graph-free LoRA on the same three blocks as KnowsTFM, so that the only difference is the gate.

Table R2. TabPFN-v3: KnowsTFM minus each baseline (percentage points, Wilcoxon p).

Settingvs zero-shotvs LoRA (all blocks)vs LoRA (same 3 blocks)vs density-matched random
CUMIDA-42 rnd100+0.18, p=0.13+0.35, p=0.008+0.29, p=0.003+0.34, p=0.001
xdomain13+0.31, p=0.024−0.05, p=0.83+0.08, p=0.91+0.05, p=0.91

On CUMIDA-42 rnd100, the curated graph improves on parameter-efficient adaptation, also with LoRA in the same blocks, whereas the gate with a random graph does not (85.99 vs 86.04). On xdomain13, graph-free LoRA is equally good.

Implication. Under a matched adaptation protocol, the frontier-model contribution is specific: the gate with the curated graph adds 0.29 points over LoRA in the same blocks, and the same gate with a random graph adds nothing. We present this as a result for rnd100 and do not claim a general advantage over parameter-efficient adaptation.

In the revision: Q1 and Q2 paragraphs of Section 5; Appendix O (Table 29).

Q1. Comparison with Relational Foundation Models?

RFMs such as KumoRFM and Griffin are pretrained on multi-table relational databases, with rows as nodes and key links as edges. Our setting is a single table without a relational schema, so there is no structure for an RFM to exploit; our graph is over columns and comes from an external source.

In the revision: added to the related work (Section 2).

Q3. Does KnowsTFM become more beneficial as labels decrease?

[New] Label-budget sweep at ten budgets (4 to 20 labelled rows per class), all five models.

  • Setup: all five models, same seeds, folds and learning-rate grid as in the submission. A dataset enters at budget K only if every fold has K rows in every class, so the set of eligible datasets shrinks as K increases. No CUMIDA-42 dataset supports K=50, so 20 is the largest budget we can report.

Table R3. CUMIDA-42 rnd100, K labelled rows per class, differences in percentage points; * p < 0.05; † too few non-tied datasets for a test.

K per class45678910121520
NanoTabPFN 1.1M: KnowsTFM − vanilla FT+1.31+2.82*−1.94+2.03*+3.63*+3.57*+6.03*+0.11+4.16*+3.89*
TabPFN-v3: KnowsTFM − vanilla FT+0.16+0.10+0.17+0.13+0.02+0.14−0.06+0.46*+0.11+0.86†
TabPFN-v3: KnowsTFM − zero-shot+0.01+0.14−0.01+0.10−0.02+0.12*+0.05−0.16+0.01+0.04

The benefit does not grow as labels decrease.

Implication. On the 1.1M small-prior model, KnowsTFM outperforms vanilla fine-tuning at nine of ten budgets, significantly at seven, by up to 6.03 points; TabPFN-v3 with KnowsTFM remains close to zero-shot throughout. The motivation is therefore adaptation with limited labels in the p >> n regime, without a claim that the benefit increases at smaller budgets. Because the eligible datasets change with K, these results establish neither a general label-budget trend nor a threshold.

In the revision: Appendix P (Tables 30 to 32) and one sentence in the Q1 paragraph of Section 5.

Presentation.

In the revision, we added a Q3 paragraph on knowledge versus structure to Section 5, replaced the random-graph figure with matched controls (Figures 3 and 4), added a TabPFN-v3 figure (Figure 2), and moved the new controls, paired tests, label budget, relation policies, and cost into Appendices M to R, each referenced from the main text.

AuthorsOpenReview ↗

Follow-up on W2: KnowsTFM on 52 further datasets (TabArena, BeyondArena)

Following up on W2, we have extended the frontier-model evaluation to 65 datasets: xdomain13 and two standard benchmarks, TabArena (20) and BeyondArena (32), with their official splits (3 repeats x 3 folds) and metrics.

W2: "its improvements on TabPFN-v3 are marginal [...] the experiments provide limited evidence that knowledge injection offers a substantial advantage over parameter-efficient adaptation alone."

We respectfully disagree. Over these 65 datasets, KnowsTFM (Wikidata) beats vanilla fine-tuning on 55 and zero-shot on 37 (26 losses), and its pooled improvability is 0.51%, vs 1.29% for zero-shot and 4.90% for vanilla FT (Table G).

On a strong pretrained model, small differences are the scale at which the best methods differ. On the official TabArena results for our 20 datasets, neighbouring methods of the top 10 differ by a median of 0.0001 AUC; over all 30 binary TabArena datasets, the top-10 methods (LimiX-2, Kumo, Seldon, Chakra-Tab, TabPFN-3.5, TabFM and others) differ by a median of 0.0002 AUC on the same 9 splits (Appendix S). We therefore also report TabArena's own summary measure, improvability: the mean relative error gap to the best method on each dataset (lower is better).

Table F. Improvability (%) on CUMIDA-42, TabPFN-v3, balanced accuracy. Best = best of the four methods on each dataset.

Methodrnd100top25rnd75top50rnd50top100
zero-shot1.522.241.131.28
vanilla FT4.091.981.172.12
graph-free LoRA3.001.461.021.52
KnowsTFM (PrimeKG)0.851.370.630.26

Table G. Improvability (%) on the cross-domain and standard benchmarks, TabPFN-v3. xdomain13: balanced accuracy; TabArena and BeyondArena: official metrics. All 65 pools the per-dataset gaps.

Methodxdomain13TabArena (20)BeyondArena (32)All 52All 65 (pooled)
zero-shot1.520.921.431.231.29
vanilla FT6.263.265.384.574.90
KnowsTFM (Wikidata)0.340.410.650.560.51
KnowsTFM (LLM graph)0.670.920.830.870.83

Graph-free LoRA (LoRA on KnowsTFM's three column-attention blocks, no graph) was run on xdomain13 only. With it as a fifth method there, improvability is 0.36% for KnowsTFM (Wikidata), 0.61% for graph-free LoRA, 0.70% for KnowsTFM (LLM graph), 1.55% for zero-shot and 6.28% for vanilla FT.

  • KnowsTFM is closest to the best method in all 9 settings, including all 52 standard-benchmark datasets, where it beats vanilla FT on 45 (p<0.001) and needs no curated knowledge graph.
  • Against graph-free LoRA, where the graph is the only difference, KnowsTFM has lower improvability in all four CUMIDA regimes (Table F) and on xdomain13. Its accuracy is significantly higher on rnd100 (+0.29, p=0.003, 12/1) and never significantly lower. We therefore respectfully disagree that knowledge injection offers no advantage over parameter-efficient adaptation alone: at the same layers, the graph brings the model consistently closer to the best method.

In the revision: Tables F–G will be added to Appendix S, and all 104 graphs (52 Wikidata, 52 LLM-generated) will be released.

We hope these results, on 65 further datasets, address your concern about the practical benefit on the frontier model, and we would be grateful if you would consider them in your final assessment.

Reviewer_6NT8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes KnowsTFM, a fine-tuning method for tabular foundation models that injects external relational knowledge into feature-to-feature attention. Table columns are mapped to knowledge-graph entities with the help of an LLM to build a feature adjacency matrix. This adjacency matrix enters the pre-softmax attention logits, either as a hard mask or as a learnable per-head soft bias. It is combined with LoRA or full fine-tuning. The paper also explores using an LLM to fully construct the adjacency matrix when entity linking to an external knowledge graph is not available. The results show that on CUMIDA-42 (biomedical data), KnowsTFM improves over vanilla fine-tuning on the small NanoTabPFN models, and, on the larger TabPFN-3 model, recovers the degradation caused by vanilla fine-tuning to match zero-shot performance.

Critical strengths
  1. The method is novel and well-motivated. Using curated knowledge bases as an inductive bias on feature attention is an appealing way to adapt TFMs in label-scarce regimes. The LLM/Wikidata pipeline with abstention also extends it beyond domains with curated KGs.

  2. The evaluation is controlled and transparent. The paper provides per-dataset results for every checkpoint and regime, comprehensive ablations on the mechanisms and regimes, and a random-graph, LLM-graph and oracle-DAG controls. It also contains an honest discussion of negative results (no gain on xdomain13 for nano models, unresolved role of exact edges).

Critical weaknesses

W1. The practical significance of the gains is unclear, particularly relative to zero-shot TabPFN-3, and costs are not reported.

  • The headline improvements are achieved on toy NanoTabPFN models, trained on only 12k synthetic tasks (under 11 minutes on one GPU, Table 4), which seem to be very weak baselines. Their zero-shot balanced accuracy on CUMIDA-42 is about 0.44. Many binary datasets sit at exactly 50.0, i.e. constant predictions. Even after KnowsTFM (0.632), they remain 23 points below zero-shot TabPFN-3 (0.862). Only one of four nano checkpoints shows a significant gain over vanilla fine-tuning (Table 1).
  • For TabPFN-3, KnowsTFM brings no significant gains over zero-shot. KnowsTFM reaches 0.863 vs. 0.862 zero-shot on CUMIDA-42 (rnd100). It ties zero-shot under the other three feature regimes (89.5/89.3, 89.6/89.6, 90.2/90.2). On xdomain13, KnowsTFM gives 76.1 vs. 75.8 zero-shot, while graph-free LoRA gives 76.2 (Table 19). The reported significance (p < 0.01) is against vanilla fine-tuning, which itself falls below zero-shot. The method therefore recovers a loss that practitioners can avoid by not fine-tuning at all.
  • Costs are not reported, which makes the method’s value difficult to assess. Zero-shot TabPFN-3 needs a single forward pass, while KnowsTFM requires several LLM calls and Wikidata/SPARQL queries per dataset, and per-dataset fine-tuning.

W2. The scope of applicability is unclear, and the evaluation outside genomics is weakly motivated.

  • The empirical setting is artificial. The method is most convincing on CUMIDA-42 with 100 randomly sampled genes out of 7k–21k. Nobody would use that regime in practice. Once features are relevance-selected (top100), the advantage disappears for every checkpoint (Table 2; Table 9), and for NanoTabICL it turns negative on the base model (Table 22). It remains unclear whether the method helps in realistic p >> n pipelines, for example TabPFN-3 with standard feature selection or with more features. It is also unclear whether it helps outside the "many features, few samples" regime at all.
  • The use of the ad-hoc xdomain13 benchmark is not properly motivated. Why not use an established benchmark with semantically meaningful column names (e.g., the CARTE benchmark suite) or a curated subset of TabArena? It would have made the scope claims more credible.
  • The role of the knowledge itself is unresolved, as conceded by the authors. Several results suggest the gains may come from a generic sparsity/regularization effect rather than from domain knowledge (for instance the Wikidata and LLM graphs overlap by only 12.8% yet perform the same). This weakens the "knowledge-grounded" framing, which is the paper's central claim.
Questions for authors
  1. Can you report paired tests of KnowsTFM against zero-shot TabPFN-3 and against graph-free LoRA, on both CUMIDA-42 and xdomain13?
  2. What is the end-to-end cost of KnowsTFM compared with zero-shot inference: graph construction (LLM and API calls), fine-tuning including the learning-rate grid, GPU memory, and inference latency?
  3. Does the method improve TabPFN-3 in a realistic setting, for example an established benchmark with semantically meaningful column names (e.g., CARTE) or genomic data with standard feature selection?
  4. Can you add a density-matched (and degree-matched) random-graph control with significance tests? Can you explain why graphs with almost no overlap (LLM vs. Wikidata) yield similar results?
5 comments

Thank you for the review. We uploaded a revised PDF in which all changes are in blue; [New] marks experiments run for this rebuttal.

W1 (first two points) / Q1. […] paired tests of KnowsTFM against zero-shot TabPFN-3 and against graph-free LoRA, on both CUMIDA-42 and xdomain13?

[New analysis] Paired tests on TabPFN-v3, including [New] graph-free LoRA on the same three blocks as KnowsTFM, so that the only difference is the gate.

Table R1. TabPFN-v3: KnowsTFM minus each baseline (percentage points, Wilcoxon p, wins/losses).

Settingvs zero-shotvs LoRA (all blocks)vs LoRA (same 3 blocks)
CUMIDA-42 rnd100+0.18, p=0.13, 8/4+0.35, p=0.008, 13/5+0.29, p=0.003, 12/1
CUMIDA-42 top100+0.01, p=0.69, 4/3+0.10, p=0.31, 5/3+0.09, p=0.22, 5/2
xdomain13+0.31, p=0.024, 9/2−0.05, p=0.83, 7/4+0.08, p=0.91, 5/4

Against zero-shot, KnowsTFM does not differ significantly on CUMIDA-42 and performs significantly better on xdomain13. Against graph-free LoRA, it performs significantly better on rnd100, including when LoRA is applied to the same blocks, and shows no significant difference elsewhere.

Implication. With a median of 34 training rows, the practical question for a strongly pretrained model is whether it can be fine-tuned without loss. KnowsTFM stays at or above zero-shot in every setting we tested, whereas vanilla fine-tuning falls 0.57 points below zero-shot on rnd100 and 1.28 points below on xdomain13. This is in line with Rubachev et al. (2025), who find that fine-tuning gains for TabPFNv2 on small datasets are often not significant.

For the nanoscale models, we agree: they are a controlled testbed for the scale dependence of the prior, not competitive predictors.

In the revision: the introduction states this and reports the gap to zero-shot TabPFN-v3; the Q1 and Q2 paragraphs of Section 5 report the comparisons with zero-shot and graph-free LoRA; Appendix O (Table 29, all four CUMIDA regimes).

W1 (third point) / Q2. End-to-end cost?

[New] Cost measurement on one A100 (medians over the 42 CUMIDA datasets).

Table R2. TabPFN-v3.

Zero-shotVanilla FTLoRA (same 3 blocks)KnowsTFM
Parameters adaptednone53.15M86,01686,016 + 48 gate
One fit / full protocol (4 lr × 9 splits)—5.6 s / 7.2 min4.9 s / 6.4 min5.0 s / 6.5 min
Peak GPU memory0.3 GB1.1 GB0.5 GB0.5 GB
Prediction latency per call0.38 s0.37 s0.38 s0.39 s

Graph construction is a one-off cost per dataset: about 3 s for PrimeKG (a lookup) and about 9 min for Wikidata (2 LLM calls and a median of 535 cached requests).

Implication. The gate adds 48 parameters, 0.1 s per fit, and no additional memory compared with LoRA in the same blocks. One fit is also cheaper than vanilla fine-tuning. The extra cost relative to zero-shot is the one-off graph construction and the fine-tuning itself.

In the revision: Appendix R (Table 36); the limitations paragraph states the upfront cost relative to zero-shot.

W2 (first point) / Q3. Realistic setting: standard feature selection, or an established benchmark such as CARTE?

With standard feature selection (CUMIDA-42 top100), TabPFN-v3 with KnowsTFM performs comparably to zero-shot (+0.01, Table R1). The benefit is largest when features are not relevance-selected and on small models; we use rnd100 as a diagnostic regime, not as a recommended pipeline. We did not evaluate on CARTE: 40 of its 51 tasks are regression, and its semantic content is mostly in string cell values, whereas our prior is a graph over column names.

The classification tasks of CARTE, with a text encoder for the string columns, are a natural follow-up evaluation.

In the revision: Section 5 (feature-selection regimes, after Table 3) describes rnd100 as a diagnostic regime; CARTE is named as future work in the experimental setup.

W2 (second point). xdomain13 not motivated.

xdomain13 tests the case without a curated knowledge graph. A dataset had to provide feature names or descriptions, have at most 50 features, and have at least 150 samples with five or more in the minority class (Appendix C.2). Columns were not screened for name quality or Wikidata coverage, so the benchmark includes realistic cases such as uninformative column names (F1 to F22 in SPECT); the linker abstains on 77 of the 261 columns.

In the revision: this selection procedure is now in the experimental setup of the main text.

W2 (third point) / Q4. Density-matched and degree-matched random controls with significance tests? Why do graphs with almost no overlap perform alike?

[New] Matched graph controls. Every control has exactly as many edges as the curated graph on each panel: 9.4 on average, i.e. 0.19% of the 4,950 feature pairs of a 100-gene panel. The density-matched controls inherit this edge count from the curated graph, and the rewiring additionally preserves each feature's degree. This tests which properties of the curated structure are sufficient to reproduce its performance.

Table R3. CUMIDA-42 rnd100, balanced accuracy (%); bold: best per model.

ModelCuratedDensity-matched randomDegree-preserving rewiringLLM-emb.KG-emb.
NanoTabPFN 1.1M, small prior64.2064.9364.2264.3464.90
NanoTabPFN 3.7M, small prior63.2162.7762.3262.3462.78
NanoTabPFN 1.1M, big prior62.9064.1363.7162.9764.11
NanoTabPFN 3.7M, big prior62.6463.2264.1163.0763.44
TabPFN-v3 53M86.3385.9986.3186.0786.07

At the nanoscale, no control performs significantly worse than the curated graph (curated − random: −1.23 to +0.43, p ≥ 0.14), which supports sparsity as an important factor. On TabPFN-v3, the curated graph outperforms the random graph (+0.34, p = 0.001) and both embedding graphs (+0.26, p ≤ 0.042), while the rewiring performs comparably.

Why low-overlap graphs perform alike. At the nanoscale, density matters more than which pairs are connected (Figure 8a,b): the Wikidata and LLM graphs share 12.8% of their edges but have similar density, and an LLM-embedding graph that shares an average of 0.02 of the 9.4 curated edges scores 64.34 against 64.20.

Implication. The role of the knowledge graph differs across model scales. For NanoTabPFN, the density of the prior is an important factor, and the knowledge graph supplies a useful one: the random control of the submission, whose density was drawn without the knowledge graph, is lower on all four checkpoints. On TabPFN-v3, the curated graph provides a small advantage over alternative graphs with the same number of edges, which degree-preserving rewiring retains; graph properties beyond density matter at this scale, and the experiments do not isolate which preserved property explains the effect.

In the revision: Q3 paragraph in Section 5, Figures 3 and 4, Appendices M and N.

I thank the authors for their responses and the additional experiments. Overall I like the idea of the paper, I think it is novel, reasonable, and quite elegant. However given the empirical results my main concerns remain:

1. Performance gains for applications of interest

From what I see of the results, the performance gains over zero-shot are quite marginal. Most importantly, they are concentrated in settings that, in my opinion, lack realism.

For CUMIDA-42, the realistic setting is to use the top-100 relevant features rather than random ones (because it gives better results). In this setting zero-shot does as good as KnowsTFM.

For xdomain13, my point was that there already exist standard benchmarks for tabular learning (TabArena [1], BeyondTabArena [2], Grinsztajn [3], and more), coming with datasets that don't have an associated curated knowledge graph, and that are supposed to be representative of some real-world applications. So why not use them and introduce xdomain13 instead? Seeing KnowsTFM evaluated on standard benchmarks would make me trust much more that it can have real practical value.

[1] Erickson, N., Purucker, L., Tschalzev, A., Holzmüller, D., Desai, P., Salinas, D., & Hutter, F. Tabarena: A living benchmark for machine learning on tabular data. NeurIPS 2025.

[2] Purucker, L., Tschalzev, A., Erickson, N., Blayer, G., Holzmüller, D., Arazi, A., ... & Hutter, F. Beyond IID: How general are tabular foundation models, really?. NeurIPS 2026.

[3] Grinsztajn, L., Oyallon, E., & Varoquaux, G. Why do tree-based models still outperform deep learning on typical tabular data?. NeurIPS 2022.

2. Role of knowledge

The curated KG working no better than a random graph is a very disappointing result, and in my opinion it completely undermines the paper's narrative. This suggests that the gains over vanilla fine-tuning are probably due to some form of regularization of the attention mechanism, and not due to any kind of external knowledge. These gains are well demonstrated in the paper and interesting by themselves, however I don't buy the knowledge-integration narrative after seeing these results.

Therefore I am sorry but I will maintain my current rating.

Reviewer_2xL5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper proposes KnowsTFM, a knowledge-informed fine-tuning approach for tabular foundation models (TFMs) that incorporates external knowledge graphs (KGs) into feature-level attention. The method first constructs a feature interaction graph by mapping tabular columns to entities in curated KGs or Wikidata, with LLM assistance for general-domain datasets. It then injects the resulting graph structure into transformer attention through hard masks or learnable soft biases, combined with parameter-efficient fine-tuning using LoRA. The authors evaluate KnowsTFM on multiple NanoTabPFN checkpoints and TabPFN-v3 across 42 biomedical and 13 cross-domain datasets. The results show that knowledge-informed adaptation can improve downstream performance in certain settings.

Critical strengths

Strengths

  1. Interesting and relevant research problem. The paper investigates how external knowledge graphs can improve the adaptation of tabular foundation models, particularly in data-scarce domains. Incorporating domain knowledge into pretrained TFMs is a practically relevant direction that deserves further investigation.

  2. Simple and flexible methodology. The proposed approach injects graph-derived structural priors into feature-level attention through hard masks or learnable soft biases. The method is relatively straightforward and can be integrated into existing transformer-based tabular models without fundamentally changing their architectures.

  3. Extensive experimental evaluation. The authors evaluate the approach across multiple TabPFN checkpoints, 42 biomedical datasets, and 13 cross-domain datasets. The experiments also include comparisons of graph sources, feature-selection regimes, and adaptation mechanisms, providing useful insights into when knowledge-informed adaptation may be beneficial.

Critical weaknesses

Weaknesses

  1. The construction of knowledge graphs relies on manually designed relation types that require further justification. Section 3.1 constructs feature-level knowledge graphs using three predefined relation policies: direct, 1-hop, and ancestor. However, the selection of these relation types appears to involve substantial feature engineering, and the paper does not sufficiently justify why these particular relations are appropriate for modeling feature interactions across different domains. Appendix J further shows that none of the 13 cross-domain datasets contains direct feature-to-feature Wikidata edges, and the resulting graphs rely heavily on shared ancestors. Some constructed graphs are extremely dense, raising questions about whether they capture meaningful domain knowledge or merely introduce generic attention biases. A more systematic justification and evaluation of these relation-selection choices would strengthen the proposed approach.

  2. The empirical improvements in Tables 1 and 3 are relatively limited and their practical significance is unclear. In Table 1, the improvements over vanilla fine-tuning are modest for two of the four NanoTabPFN checkpoints. Notably, the two checkpoints with the largest improvements also exhibit larger standard deviations than their corresponding vanilla fine-tuning baselines, raising concerns about the consistency of these gains. For TabPFN-v3, the pretrained model already achieves 0.862 balanced accuracy, while KnowsTFM only improves it to 0.863. A similar pattern appears in Table 3, where the pretrained TabPFN-v3 achieves 0.758 and KnowsTFM reaches 0.761. Furthermore, Table 19 in Appendix H shows that graph-free LoRA achieves 0.762 on the cross-domain datasets, slightly outperforming KnowsTFM (0.761). These results suggest that the proposed approach provides limited additional benefits for strongly pretrained TFMs, and its practical effectiveness remains unclear.

Questions for authors

Questions

  1. How are the relation types for knowledge graph construction selected and justified? Section 3.1 introduces three relation policies (direct, 1-hop, and ancestor), but it is unclear how the underlying Wikidata predicates are selected and why these particular relations are appropriate for capturing feature interactions across different domains. Could the authors provide more details on the relation-selection criteria and investigate how different choices of relation types affect downstream performance?

  2. How does the quality of the constructed knowledge graphs affect model performance? Appendix J shows that the general-domain graphs rely heavily on shared ancestors, with some graphs being extremely dense. Could the authors investigate whether these relations capture meaningful feature dependencies? In particular, would the performance remain similar when using density-matched random graphs or alternative relation-selection strategies?

  3. How consistent are the improvements reported in Tables 1 and 3? Given the relatively small average gains and substantial across-dataset variability, could the authors provide a more detailed analysis of paired performance differences, including confidence intervals and the number of datasets on which KnowsTFM outperforms vanilla fine-tuning? This would help assess whether the improvements are consistent or driven by a small subset of datasets.

  4. What is the additional benefit of KnowsTFM for strongly pretrained models such as TabPFN-v3? In Tables 1 and 3, KnowsTFM provides only marginal improvements over the pretrained TabPFN-v3 model. Furthermore, Table 19 shows that graph-free LoRA slightly outperforms KnowsTFM on the cross-domain benchmark. Could the authors clarify under which conditions knowledge-informed adaptation provides meaningful improvements for strongly pretrained models such as TabPFN-v3?

4 comments

Thank you for the review. We uploaded a revised PDF in which all changes are in blue; [New] marks experiments run for this rebuttal.

W1 / Q1. How are the relation types […] selected and justified? […] investigate how different choices of relation types affect downstream performance?

The policies are the generic ways in which two entities can be related: a direct link, a shared neighbour (1-hop), or a shared ancestor. No predicate list is tuned per dataset or domain. CUMIDA-42 uses the direct PrimeKG policy, and xdomain13 uses the union of the three Wikidata policies.

[New] PrimeKG relation policies on CUMIDA-42. We rebuilt the biomedical graph under every PrimeKG policy, including the full union written in Section 3.1, and trained KnowsTFM on TabPFN-v3 unchanged. Each policy is also compared with its own density-matched random graph.

Table R1. TabPFN-v3, CUMIDA-42 rnd100: graph minus baseline (percentage points, Wilcoxon p).

PolicyEdges / panelBal. acc.− vanilla FT− graph-free LoRA− own density-matched random
Direct (main results)9.486.33+0.75, p=0.002+0.35, p=0.008+0.34, p=0.001
Pathway13.986.25+0.68, p=0.006+0.27, p=0.019+0.20, p=0.041
Disease68.886.23+0.65, p=0.003+0.25, p=0.009+0.14, p=0.11
Direct + pathway22.686.27+0.69, p=0.004+0.29, p=0.013+0.25, p=0.013
Direct + disease77.886.22+0.64, p=0.006+0.23, p=0.027+0.15, p=0.067
Direct + disease + pathway90.486.21+0.63, p=0.006+0.23, p=0.017+0.13, p=0.16

All six policies lie within 0.12 points of each other, and all improve significantly on vanilla fine-tuning and graph-free LoRA. Where a ground-truth DAG exists (asia, sachs), all Wikidata policies are within 0.89 points of the DAG (Table 34).

Implication. The choice of relation policy is not a sensitive design decision: on CUMIDA-42, the policies change the number of edges almost tenfold while accuracy changes by at most 0.12 points, so the method does not rely on hand-picked predicates. The margin over each policy's own random graph is largest for the sparsest policies, in line with the role of density discussed under Q2.

In the revision: Section 3.1 now presents the relations as policies and states which one each experiment uses; Appendix Q (Tables 33 to 35, Figures 9 and 10); per-cancer-type gains are also shown in Figure 2b.

Q2. […] would the performance remain similar when using density-matched random graphs or alternative relation-selection strategies?

[New] Density-matched random graph. On every panel, it has exactly as many edges as the curated graph: 9.4 on average, i.e. 0.19% of the 4,950 feature pairs of a 100-gene panel. It inherits this edge count from the curated graph, but not which pairs are connected.

Table R2. CUMIDA-42 rnd100, balanced accuracy (%); bold: best per model.

ModelCuratedDensity-matched randomDiff.p
NanoTabPFN 1.1M, small prior64.2064.93−0.740.21
NanoTabPFN 3.7M, small prior63.2162.77+0.430.62
NanoTabPFN 1.1M, big prior62.9064.13−1.230.14
NanoTabPFN 3.7M, big prior62.6463.22−0.580.49
TabPFN-v3 53M86.3385.99+0.340.001

At the nanoscale, performance remains similar, which supports density as an important factor. On TabPFN-v3, the curated graph is significantly better. For the dense ancestor graphs you point to, the density sweep (Figure 8a,b) shows that small models lose the benefit of a hard mask at high density, consistent with the absent nanoscale gains on xdomain13. Alternative relation strategies are presented in Table R1 above.

Implication. The density at which the prior helps small models is the one the knowledge graph supplies: the random control of the submission, whose density was drawn without the knowledge graph, is lower on all four NanoTabPFN checkpoints (by up to 2.89 points, p = 0.004 on the 1.1M small-prior model).

In the revision: Q3 paragraph in Section 5, Figures 3 and 4, Appendices M and N.

W2 / Q3. How consistent are the improvements […]? Paired differences, confidence intervals, number of datasets where KnowsTFM wins?

[New analysis] Paired comparisons with bootstrap 95% intervals and win/loss counts.

Table R3. KnowsTFM − vanilla FT (percentage points), CUMIDA-42 rnd100 and xdomain13.

Modelrnd100xdomain13
NanoTabPFN 1.1M, small prior+3.11 [+0.39, +5.96], p=0.045, 28/14+0.44, p=0.64, 7/6
NanoTabPFN 3.7M, small prior+0.18 [−1.56, +1.94], p=0.75, 22/19−0.80, p=0.080, 2/11
NanoTabPFN 1.1M, big prior+2.40 [+0.07, +4.86], p=0.071, 24/17+0.54, p=0.31, 9/4
NanoTabPFN 3.7M, big prior+0.54 [−1.53, +2.59], p=0.64, 22/19−0.49, p=0.11, 3/10
TabPFN-v3+0.75 [+0.31, +1.27], p=0.002, 15/4+1.59 [+0.47, +2.92], p=0.003, 10/1

The gains are not driven by a few datasets: on the 1.1M models, KnowsTFM wins on about two-thirds of them (23 to 31 of 42 across the four CUMIDA regimes). The larger standard deviations you note reflect differences in difficulty between datasets (11 to 16 points for every arm); the paired analysis compares the methods on the same datasets.

Implication. The paired view shows where the gains are consistent: on the 1.1M models, the bootstrap interval excludes zero in seven of the eight CUMIDA cells, and on TabPFN-v3, in three of five settings. The 3.7M models show no consistent gain, which matches the scale dependence of the useful density (Figure 2a,b).

In the revision: Appendix O (Table 28, all four CUMIDA regimes), referenced from Tables 1 and 2.

Q4. What is the additional benefit of KnowsTFM for strongly pretrained models such as TabPFN-v3? […]

[New] Graph-free LoRA on the same three blocks as KnowsTFM, so that the only difference is the gate.

Table R4. TabPFN-v3: KnowsTFM minus each baseline (percentage points, Wilcoxon p).

Settingvs zero-shotvs vanilla FTvs LoRA (all blocks)vs LoRA (same 3 blocks)
CUMIDA-42 rnd100+0.18, p=0.13+0.75, p=0.002+0.35, p=0.008+0.29, p=0.003
xdomain13+0.31, p=0.024+1.59, p=0.003−0.05, p=0.83+0.08, p=0.91

On CUMIDA-42 rnd100, the curated graph adds a benefit over parameter-efficient adaptation, also when LoRA sits in the same blocks. On xdomain13, KnowsTFM is significantly above zero-shot, and graph-free LoRA is equally good.

Implication. On the strongly pretrained model, the conditions are: when the features are not relevance-selected (rnd100), the curated prior adds a benefit beyond parameter-efficient adaptation; in all five settings we tested, KnowsTFM stays at or above zero-shot, whereas vanilla fine-tuning falls below it on rnd100 and xdomain13.

In the revision: Q1 and Q2 paragraphs of Section 5; Appendix O (Table 29).

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.

  1. W1/Q1: I am still not fully convinced that sharing a common ancestor is a natural way to construct meaningful feature relationships in every domain. However, if these strategies work effectively on both CUMIDA-42 and xdomain13, I do not consider this a major drawback.

  2. Q2: The additional experiments with density-matched random graphs have addressed my concern about graph density.

  3. W2/Q3: Thank you for providing the confidence intervals and paired comparisons, which address some of my concerns about statistical reliability. However, Table R3 shows that 7 out of 10 comparisons have absolute performance differences of less than 1 percentage point, with some even showing negative improvements. I think the improvement is limited.

  4. Q4: I am particularly concerned about the limited improvements on the strongly pretrained TabPFN-v3 model. Table R4 shows that KnowsTFM improves over zero-shot inference by only 0.18 and 0.31 percentage points on CUMIDA-42 and xdomain13, respectively. More importantly, the additional improvements over matched graph-free LoRA are only 0.29 and 0.08 percentage points, with the latter being statistically insignificant. These results do not convincingly demonstrate the practical benefit of introducing external knowledge graphs for strongly pretrained models.

Overall, while the rebuttal has addressed some of my concerns, I still find the empirical improvements relatively limited. Therefore, I would like to maintain my rating.

AuthorsOpenReview ↗

Response to Comment by Reviewer 2xL5

Thank you for your follow-up. We address the remaining points with our new experiments on 52 TabArena and BeyondArena datasets.

W1/Q1: "I am still not fully convinced that sharing a common ancestor is a natural way to construct meaningful feature relationships in every domain."

We agree that a shared ancestor is not the most natural relation in every domain, and our results show where it is and is not needed.

  • With a curated knowledge graph, the most specific relation works best. On CUMIDA-42 rnd100 (TabPFN-v3), direct gene–gene edges give the highest accuracy of all PrimeKG policies (86.33, vs 86.21–86.27 for pathway, disease and their unions). Direct edges also give the largest margin over vanilla FT, graph-free LoRA and the density-matched random graph (Appendix Q). Which relation helps most also varies between cancer types (Figure 3b).
  • Without curated relations, we propose the LLM graph instead of ancestor pooling. The LLM proposes direct feature–feature relations, with no entity linking and no ancestors, and its graphs are much sparser: median density 10.8% vs 60.7% for the Wikidata graphs over the 52 new datasets. It performs on par or better: 76.1 vs 76.1 (bal. acc.) on xdomain13, and over the 52 datasets it is the top-ranked method (LLM Elo 1042 vs 1030 for Wikidata and 1000 for zero-shot; mean rank 2.06).
  • Even with ancestor pooling, the Wikidata graph works. By improvability (below), it shows the largest gains of all methods on xdomain13 and on the 52 new datasets: 0.34% and 0.56% vs 1.52% and 1.23% for zero-shot (Table G).

W2/Q3: "7 out of 10 comparisons have absolute performance differences of less than 1 percentage point [...] I think the improvement is limited." Q4: "KnowsTFM improves over zero-shot inference by only 0.18 and 0.31 percentage points [...] the additional improvements over matched graph-free LoRA are only 0.29 and 0.08 percentage points."

We respectfully disagree that the improvement is of limited practical value. Over 65 datasets (13 xdomain13, 20 TabArena and 32 BeyondArena), KnowsTFM (Wikidata) beats vanilla fine-tuning on 55 and zero-shot on 37 (26 losses), and its pooled improvability is 0.51%, vs 1.29% for zero-shot and 4.90% for vanilla FT (Table G).

On a strong pretrained model, differences below 1 point are the scale at which the best methods differ. On the official TabArena results for our 20 datasets, neighbouring methods of the top 10 differ by a median of 0.0001 AUC (Appendix S). Over all 30 binary TabArena datasets, the top-10 methods (LimiX-2, Kumo, Seldon, Chakra-Tab, TabPFN-3.5, TabFM and others) differ by a median of 0.0002 AUC on the same 9 splits. We therefore also report TabArena's own summary measure, improvability: the mean relative error gap to the best method on each dataset (lower is better).

Table F. Improvability (%) on CUMIDA-42, TabPFN-v3, balanced accuracy. Best = best of the four methods on each dataset.

Methodrnd100top25rnd75top50rnd50top100
zero-shot1.522.241.131.28
vanilla FT4.091.981.172.12
graph-free LoRA3.001.461.021.52
KnowsTFM (PrimeKG)0.851.370.630.26

Table G. Improvability (%) on the cross-domain and standard benchmarks, TabPFN-v3. xdomain13: balanced accuracy; TabArena and BeyondArena: official metrics. All 65 pools the per-dataset gaps.

Methodxdomain13TabArena (20)BeyondArena (32)All 52All 65 (pooled)
zero-shot1.520.921.431.231.29
vanilla FT6.263.265.384.574.90
KnowsTFM (Wikidata)0.340.410.650.560.51
KnowsTFM (LLM graph)0.670.920.830.870.83

Graph-free LoRA (LoRA on KnowsTFM's three column-attention blocks, no graph) was run on xdomain13 only. With it as a fifth method there, improvability is 0.36% for KnowsTFM (Wikidata), 0.61% for graph-free LoRA, 0.70% for KnowsTFM (LLM graph), 1.55% for zero-shot and 6.28% for vanilla FT.

  • KnowsTFM is closest to the best method in all 9 settings. (in terms of improvability)
  • Against graph-free LoRA, where the graph is the only difference, KnowsTFM has lower improvability in all four CUMIDA regimes (Table F). Its accuracy is significantly higher on rnd100 (+0.29, p=0.003, 12/1) and never significantly lower, including on xdomain13. We therefore respectfully disagree: at the same layers and cost, the graph brings the model consistently closer to the best method.

Cost. KnowsTFM costs the same as graph-free LoRA and less than LoRA on all blocks (Table R12, response to 6NT8; Appendix R): it adapts 86,016 parameters + 48 gates (0.16%) vs 2.84M (5.34%), 33 times fewer, with a faster fit (5.0 vs 6.2 s) and lower prediction latency (0.39 vs 0.48 s).

In the revision: Tables F–G will be added to Appendix S, and all 104 graphs will be released.

Reviewer_2aLkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether curated relational knowledge can steer fine-tuning of tabular foundation models when labels are scarce. Columns are mapped to knowledge-graph entities — PrimeKG after HGNC normalisation, Wikidata via an LLM-assisted mapper with abstention — giving a feature adjacency, which is added to the pre-softmax logits of the feature-attention layers under a per-block hard/soft schedule, with the graph frozen and the backbone adapted by LoRA. Evaluation covers CUMIDA-42 and 13 cross-domain datasets, four self-pretrained NanoTabPFN checkpoints, TabPFN-v3 and a NanoTabICL transfer, with random, oracle-DAG and LLM-graph controls.

Critical strengths

(1) The intervention is cleanly isolated. Only the pre-softmax feature-attention logits are touched, the adjacency is frozen, and the injection is crossed with both LoRA and full fine-tuning (Appendix G). This lets the negative results carry weight: graph-free LoRA falls below vanilla, so the gain is attributable to the prior rather than to low-rank regularisation.

(2) The finding on strongly pretrained models is the most useful result and is falsifiable. Full fine-tuning of TabPFN-v3 drops below its own zero-shot (0.862 → 0.856) and the prior returns it to 0.863 (p < 0.01). Framing the prior as an anchor against drift, rather than a channel for knowledge, is a sharper and more transferable claim.

Critical weaknesses

(1) The title's claim is broader than the controls support. The curated graph is compared against randomly wired, oracle-DAG and LLM-generated graphs, and the paper's own conclusion concedes that the benefit of specific semantic edges remains unresolved; cross-domain, the ordering between the curated and random graphs is not consistent across checkpoints. A degree-preserving rewiring of the curated graph may help separate structure from sparsity. The cross-domain evidence is also weaker than the abstract suggests: the nanoscale effects fall below the paper's own stated detection limit, and the one significant result on the frontier model mainly recovers the loss introduced by fine-tuning, differing only marginally from that model's zero-shot performance. "No measurable difference" would be more precise than "maintaining competitive performance".

(2) One close paper is undiscussed. BOLERO (Le et al., arXiv:2512.12405) adds graph priors to a frozen tabular transformer. A sentence of positioning would help readers place the contribution.

Questions for authors

Q1. The label budget is fixed throughout while only features vary. Could the authors add a support-set sweep at fixed features (e.g. 5 / 10 / 20 / 50 per class)? I would expect the benefit to shrink as labels arrive; a decaying curve would turn the premise into a measurement and indicate when the method is worth reaching for.

Q2. An LLM enters both as entity linker and as graph generator, and the two sources come out equivalent. Separating the roles (LLM as predictor, as graph source, as parametric knowledge base) would clarify the claim. Relatedly, since the graph must be externalised into a K×K matrix, why is that preferable to using LLM representations directly as column encoders?

4 comments

Thank you for the careful reading and feedback. We uploaded a revised PDF in which all changes are in blue; [New] marks experiments run for this rebuttal. We adopt your "anchor against drift" reading of the frontier-model result.

W1 (first part). […] A degree-preserving rewiring of the curated graph may help separate structure from sparsity.

[New] Matched graph controls. Besides the rewiring you suggested, we built graphs from LLM text embeddings (BGE-M3) and from pretrained PrimeKG node embeddings. Every control has exactly as many edges as the curated graph on each panel: 9.4 on average, i.e. 0.19% of the 4,950 feature pairs of a 100-gene panel. The density-matched controls inherit this edge count from the curated graph, and the rewiring additionally preserves each feature's degree. This tests which properties of the curated structure are sufficient to reproduce its performance.

Table R1. CUMIDA-42 rnd100, balanced accuracy (%); bold: best per model.

ModelCuratedRewiredRandomLLM-emb.KG-emb.
NanoTabPFN 1.1M, small prior64.2064.2264.9364.3464.90
NanoTabPFN 3.7M, small prior63.2162.3262.7762.3462.78
NanoTabPFN 1.1M, big prior62.9063.7164.1362.9764.11
NanoTabPFN 3.7M, big prior62.6464.1163.2263.0763.44
TabPFN-v3 53M86.3386.3185.9986.0786.07

At nanoscale no control is significantly below the curated graph, which supports sparsity as an important factor. On TabPFN-v3 the curated graph beats the random (+0.34, p = 0.001), LLM-embedding (+0.26, p = 0.042) and KG-embedding graphs (+0.26, p = 0.027), while the rewiring matches it; the effect is therefore not established for exact edges.

Implication. The role of the knowledge graph differs across model scales. For NanoTabPFN, the density of the prior is an important factor: the density-matched random graph performs comparably to the curated graph, while the random-density baseline of the submission is lower on all four checkpoints (by up to 2.89 points, p = 0.004 on the 1.1M small-prior model). On TabPFN-v3, the curated graph provides a small advantage over alternative graphs with the same number of edges, which degree-preserving rewiring retains. This suggests that graph properties beyond density matter at the frontier scale; the experiments do not isolate whether the degree distribution, the retained edges or other preserved properties explain the effect.

In the revision: new Q3 paragraph in Section 5 ("Sparsity explains the nanoscale gains…"), Figures 3 and 4, contribution 4, Appendix M (Tables 23 to 26).

W1 (second part). The cross-domain evidence is also weaker than the abstract suggests. […]

On xdomain13 no NanoTabPFN checkpoint differs significantly from vanilla fine-tuning (Table 28). On TabPFN-v3, KnowsTFM is significantly above zero-shot (+0.31, p = 0.024, 9 wins / 2 losses), not only above vanilla fine-tuning (Table 29).

Implication. On xdomain13, KnowsTFM avoids the degradation observed under vanilla full fine-tuning (1.28 points below zero-shot) and ends 0.31 points above zero-shot. This is consistent with your anchor-against-drift reading, measured here as predictive performance. Note that TabPFN-v3 may also be saturated, as it was pretrained on a billion-scale pretraining base, potentially accumulating many of these structures.

In the revision: the abstract now states that the domain-specific gains depend on model scale; the Q2 paragraph of Section 5 reports the comparison with zero-shot.

W2. BOLERO […]

Thank you for the pointer; we have added the positioning to Section 2. The two graphs differ in kind: BOLERO's is built from the table and relates rows, ours comes from outside the data and relates features.

  • BOLERO (Le et al., 2025) keeps a pretrained RoBERTa-Tab backbone frozen and adds a static bipartite graph that links each row to feature-value anchor nodes, refined by a small GNN.
  • KnowsTFM injects an external feature graph into the pre-softmax feature-attention logits of an in-context TFM, adapts the backbone with LoRA, and studies how the effect of the prior depends on model scale.
  • A direct comparison is not possible because, to our knowledge, the model has not been publicly released.

Since the two approaches operate on different relational structures, their combination is a possible direction for future work.

Q1. […] a support-set sweep at fixed features […] I would expect the benefit to shrink as labels arrive.

[New] Label-budget sweep at ten budgets, all five models.

  • Setup: all five models, same seeds, folds and learning-rate grid as in the submission. A dataset enters at budget K only if every fold has K rows in every class, so the set of eligible datasets shrinks as K increases. No CUMIDA-42 dataset supports K=50, so 20 is the largest budget we can report.

Table R2. CUMIDA-42 rnd100, K labelled rows per class, differences in percentage points; * p < 0.05; † too few non-tied datasets for a test.

K per class45678910121520
Datasets4234312522211915129
NanoTabPFN 1.1M: KnowsTFM − vanilla FT+1.31+2.82*−1.94+2.03*+3.63*+3.57*+6.03*+0.11+4.16*+3.89*
TabPFN-v3: KnowsTFM − vanilla FT+0.16+0.10+0.17+0.13+0.02+0.14−0.06+0.46*+0.11+0.86†
TabPFN-v3: KnowsTFM − zero-shot+0.01+0.14−0.01+0.10−0.02+0.12*+0.05−0.16+0.01+0.04
TabPFN-v3: vanilla FT − zero-shot−0.16+0.05−0.17−0.04−0.04−0.03+0.10−0.62−0.10−0.82

There is no decaying curve: the benefit over vanilla fine-tuning does not grow as labels become scarcer. On TabPFN-v3, KnowsTFM stays within 0.16 points of zero-shot at every budget, while vanilla fine-tuning drops up to 0.82 below it.

Implication. The sweep does not support the hypothesis that the benefit of the prior increases as labels become scarcer. On the 1.1M small-prior model, KnowsTFM outperforms vanilla fine-tuning at nine of ten budgets, significantly at seven, by up to 6.03 points. TabPFN-v3 with KnowsTFM remains close to zero-shot throughout, while vanilla full fine-tuning degrades more at some budgets. Because the eligible datasets change with K, these results establish neither a general label-budget trend nor a threshold.

In the revision: Appendix P (Tables 30 to 32, all five models) and one sentence in the Q1 paragraph of Section 5.

Q2. […] Why is [an adjacency] preferable to using LLM representations directly as column encoders?

Thank you for this suggestion. Following it, we have extended KnowsTFM so that the feature graph can also be built from representations: from LLM text embeddings (BGE-M3) or from pretrained knowledge-graph node embeddings, by connecting the most similar feature pairs ([New], Table R1). With the same number of edges, these graphs tie the curated graph at nanoscale and are 0.26 points below it on TabPFN-v3.

In our pipeline the LLM is an entity linker (Wikidata arm) or a graph source (LLM-graph and embedding arms), never a predictor. The adjacency adds no parameters beyond the per-head gate, is frozen and inspectable, and can be swapped for matched controls. Using LLM representations directly as column encoders changes the backbone's input, which is a different intervention. Since the prior acts on attention and column encoders act on the input, combining the two is a possible direction for future work.

In the revision: Appendix M (Table 25) and the Ablations paragraph of Section 4.

AuthorsOpenReview ↗

Cross-domain evidence on 65 datasets TabArena (20), BeyondArena(32) and xdomain13 (13)

Thank you again for the constructive review. Following up on W1 (cross-domain evidence) and on your "anchor against drift" reading, we have extended the frontier-model evaluation to two standard benchmarks: TabArena (20 classification datasets) and BeyondArena (32 further datasets), with their official splits (3 repeats × 3 folds) and metrics (AUC binary, log-loss multiclass). Neither benchmark provides a curated knowledge graph, so KnowsTFM uses Wikidata-grounded or LLM-generated graphs.

W1: "the one significant result on the frontier model mainly recovers the loss introduced by fine-tuning, differing only marginally from that model's zero-shot performance."

The new benchmarks confirm the anchor-against-drift reading at scale, now on 65 datasets (xdomain13 + TabArena + BeyondArena):

Table S1. TabPFN-v3 on the 52 standard-benchmark datasets. Elo with zero-shot fixed at 1000; paired comparisons: wins/losses, Wilcoxon p.

MethodElo [95% CI]Mean rankvs vanilla FTvs zero-shot
KnowsTFM (LLM graph)1042 [965, 1126]2.0645/7, p<0.00129/23, p=0.53
KnowsTFM (Wikidata)1030 [935, 1120]2.1245/7, p<0.00128/24, p=0.75
zero-shot10002.27
vanilla FT719 [577, 823]3.56
  • Drift is systematic: vanilla fine-tuning falls below zero-shot on 43 of 52 datasets (p<0.001).
  • The prior anchors the model: KnowsTFM beats vanilla FT on 45 of 52 and stays at or slightly above zero-shot (both variants rank above it).

On how marginal the difference to zero-shot is: on the official TabArena results for our 20 datasets, neighbouring methods of the top 10 differ by a median of 0.0001 AUC; over all 30 binary TabArena datasets, the top-10 methods (LimiX-2, Kumo, Seldon, Chakra-Tab, TabPFN-3.5, TabFM and others) differ by a median of 0.0002 AUC. KnowsTFM's difference from zero-shot (≤ 0.0004 AUC) is of this size, whereas vanilla FT loses 0.0016 AUC, more than ten times the median gap. We therefore also report TabArena's improvability, the mean relative error gap to the best method on each dataset (lower is better):

Table S2. Improvability (%), TabPFN-v3. xdomain13: balanced accuracy; TabArena and BeyondArena: official metrics. All 65 pools the per-dataset gaps.

Methodxdomain13TabArena (20)BeyondArena (32)All 65 (pooled)
zero-shot1.520.921.431.29
vanilla FT6.263.265.384.90
KnowsTFM (Wikidata)0.340.410.650.51
KnowsTFM (LLM graph)0.670.920.830.83

KnowsTFM (Wikidata) is closest to the best method on every benchmark, with less than half of zero-shot's gap over the 65 datasets (0.51% vs 1.29%). This speaks to your Q2 as well: the LLM-generated graphs, which need no entity linking, perform on par with the Wikidata graphs and rank first by Elo on the 52 datasets.

In the revision: new Appendix S (protocol, per-benchmark tables, per-dataset results), Figure 4b (improvability), and the release of all 104 feature graphs (52 Wikidata, 52 LLM-generated).

We hope these results address your remaining concern on the cross-domain evidence, and we would be grateful if you would take them into account in your final assessment. Thank you again for the suggestions that shaped this revision.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: revised manuscript

Dear Reviewers, ACs, and SACs,

We thank the reviewers again for their constructive feedback. We are encouraged that the reviewers found the problem relevant and well motivated [2xL5, 6NT8, p7hR], the intervention simple, modular, and cleanly isolated [2aLk, 2xL5, p7hR], the evaluation extensive, controlled, and transparent [2xL5, 6NT8, p7hR], the reporting of negative results honest [2aLk, 6NT8], the frontier-model finding useful and falsifiable [2aLk], and the approach applicable beyond domains with curated knowledge graphs [6NT8, p7hR].

We have addressed all concerns raised by the reviewers and revised the manuscript accordingly. The revised manuscript has been uploaded, with all changes marked in blue. In total, we added 8 new experiments (more than 800,000 fine-tuning runs, about 3,500 GPU-hours) across all five models and both benchmarks, following the unchanged experimental protocol, with paired Wilcoxon tests, bootstrap 95% confidence intervals, and win/loss counts.

New experiments and where to find them

#ExperimentReviewersIn the revision
1Density-matched random graph2aLk, 2xL5, 6NT8, p7hRFig. 4a, App. M
2Degree-preserving rewiring2aLk, 6NT8, p7hRFig. 4a, App. M
3LLM- and KG-embedding graphs2aLk, 6NT8, p7hRFig. 4a, App. M
4Density sweep, 37 densities2aLk, 2xL5, 6NT8, p7hRFig. 2a, App. N
5Graph-free LoRA on the same three blocks2xL5, 6NT8, p7hRTable 1, App. O
6Label budget, 4–20 rows per class2aLk, p7hRApp. P
7PrimeKG relation policies2xL5Sec. 3.1, Fig. 2b, App. Q
8Computational cost6NT8Limitations, App. R

Main findings

  • Nanoscale: The gains are largely explained by a sparse structural prior; density-matched random, rewired, and embedding-based graphs perform comparably to the curated graph.
  • TabPFN-v3: The curated graph outperforms random graphs across all 37 tested densities and LLM- and KG-embedding graphs, while degree-preserving rewiring performs comparably. It also outperforms graph-free LoRA applied to the same three blocks (+0.29 points, p = 0.003). Improvements over graph-free LoRA are observed across all six PrimeKG relation policies.

Main changes to the manuscript

  • Claims: Revised the abstract, contributions, Q3 discussion, and conclusion to reflect the scale-dependent findings: sparsity explains the nanoscale gains, while graph properties beyond density matter on TabPFN-v3.
  • Figures: Added Figure 2 (TabPFN-v3 density sweep and relation policies) and Figure 4a (matched graph controls); added confidence intervals to Figure 3; and redrew the Bayesian-network panel.
  • Methods and related work: Clarified the relation policies in Section 3.1 and specified that the main experiments use direct PrimeKG edges. Added BOLERO and relational foundation models to the related work, and computational cost to the Limitations section.
  • Appendices: Added Appendices M–R covering the new experiments and analyses.

We believe these additional experiments strengthen the evidence for our findings and substantially improve the manuscript. We would be happy to address any further questions.

AuthorsOpenReview ↗

General Comment to Reviewers

Dear Reviewers,

As the discussion period draws to a close, we would be grateful if all reviewers could acknowledge the updates and consider revisiting their scores in light of the additional experiments (the eight listed below, as well as the extended evaluation across 52 datasets from TabArena and BeyondTabArena; see Figure 4b and Appendix S, where KnowsTFM achieves the best improvability score) and the clarifications provided throughout the discussion.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.