acceptodds
Under review as a conference paper at ICLR 2027

Why Tabular Foundation Models Excel on Relational Data They Have Never Seen

Abstract

Given a relational database (RDB) storing heterogeneous tabular information, how can we predict missing (or future) values in some target column of interest? As candidate targets are numerous in enterprise settings, it is preferable to avoid learning a new model from scratch each time there is a new prediction task. Frozen foundation models (FMs) based on in-context learning (ICL) and RDB-specific pre-training provide a viable solution, but the ideal design still remains largely unresolved. Perhaps unexpectedly, preliminary evidence has recently shown that existing single-table FMs, when combined with suitable parameter-free relational encoders, can be quite competitive in practice despite zero exposure to RDB pre-training or relational data of any kind. In this work we first empirically confirm this phenomena with a more diverse array of experiments. We then turn to our main contribution, namely, surfacing multiple complementary explanations for why tabular FMs behave so effectively. These explanations ultimately hinge on innate synergies whereby parameter-free encoders map relational effects to fixed-length embeddings that loosely mimic distributions for which tabular FMs were originally designed; in so doing, desirable single-table properties transfer to relational settings (e.g., convergent posterior predictive distributions). Collectively, our analysis forges new pathways for improving RDB FMs, particularly when we prioritize interpretability and inevitable integration within agentic systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation models, relational database, in-context learning, tabular data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UsdoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why tabular foundation models, combined with parameter-free relational encoders, can perform well on relational databases without RDB-specific pretraining. Through extensive benchmark comparisons, the authors show that TFM-based approaches such as RDBLearn and TabPFN-Rel are competitive with specialized relational foundation models. The paper provides several theoretical and empirical explanations, including the continuum between tabular and relational data, locality and approximate IID assumptions in in-context learning, the expressivity of parameter-free encoders, and their compatibility with agent-based feature engineering.

Critical strengths
  1. The paper investigates why tabular foundation models can generalize effectively to relational data without RDB-specific pretraining, which is a meaningful research direction.
  2. The authors evaluate multiple approaches across diverse relational benchmarks, providing strong empirical evidence for the competitiveness of TFM-based methods.
  3. The paper offers several perspectives on locality, in-context learning, encoder expressivity, and the relationship between tabular and relational data.
Critical weaknesses
  1. Sections 5–7 are difficult to follow. The main arguments are buried in lengthy discussions, and the connection between the theoretical claims and experiments is often unclear. Figure 3 is particularly hard to read. The paper would benefit from a more concise and structured presentation.
  2. Section 3 largely revisits comparisons already explored in prior work, including RelAgent and KumoRFM2. While the broader benchmark coverage is useful, it is unclear what new insights these experiments provide beyond confirming existing observations. The authors should better distinguish this evaluation from prior studies and clarify differences in experimental protocols, especially since results from different sources and model-selection procedures are combined.
  3. In Section 4, relational databases are constructed by splitting categorical columns of single-table datasets into lookup tables while keeping the original labels. This is quite different from real temporal RDB prediction tasks, such as rel-f1, where predictions depend on historical events, their ordering, and evolving entity states. The experiment mainly tests whether models retain single-table predictive performance after schema transformation, rather than their ability to exploit genuine relational dependencies.
  4. The arguments favoring uniform sampling and parameter-free encoders are stronger than what the experiments establish. For example, RT-J demonstrates substantially better performance in low-label regimes and shows that graph-aware retrieval can outperform uniform sampling. This suggests that relational pretraining may improve context efficiency, an important dimension largely overlooked here. More controlled experiments varying the number of support examples, retrieval strategy, and encoder design are needed before drawing broader conclusions about relationally pretrained models.
  5. The analysis in Section 5 relies heavily on approximate IID sampling and sufficient relational representations, neither of which is empirically examined. In particular, the proof of Proposition 5.2 appears to infer conditional independence between labels from the sufficiency of individual representations. This does not generally follow when samples share entities or latent factors. The authors should clarify the required assumptions and the scope of the theoretical result.
Questions for authors

Please check the questions in the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Usdo

We thank the reviewer for listing several strengths of our work, as well as providing a number of clearly-stated concerns. We address each of the latter below.

Comment: Sections 5–7 are difficult to follow. The main arguments are buried in lengthy discussions, and the connection between the theoretical claims and experiments is often unclear. Figure 3 is particularly hard to read. The paper would benefit from a more concise and structured presentation.

Response: We apologize for any lack of clarity, and remark that we have uploaded a new draft to openreview that may help a bit in this regard (although further improvements are still possible via reviewer feedback). We also mention that the lengthy discussion in places is a reflection of the considerable nuance and complexity of the underlying topic. With respect to Section 5–7 experiments and connections to theory, we offer the following clarification of Figure 3 (which distills all such experiments). This figure is meant to convey three main points:

  • Consistent with the analysis from Section 6, TFM-based RDBLearn can actually handle prediction tasks involving explicit interaction terms competitively with other RDB FMs, despite relying on a parameter-free encoder (Figure 3, left and right). This counters prior work suggesting that TFM-based models like RDBLearn/TabPFN-Rel should be incapable in such regimes.
  • When a hypothetical agent explicitly provides an interaction term as an additional input feature, TFM-based RDBLearn is better able to utilize this knowledge (Figure 3, right), consistent with the analysis in Section 7.
  • RT-J based on random-walk sampling struggles with the tasks in Figure 3 (left and right), in part because random walks cannot collect necessary context over disconnected relational graphs. This directly supports the analysis from Section 5.

Comment: Section 3 largely revisits comparisons already explored in prior work, including RelAgent and KumoRFM2. While the broader benchmark coverage is useful, it is unclear what new insights these experiments provide beyond confirming existing observations. The authors should better distinguish this evaluation from prior studies and clarify differences in experimental protocols, especially since results from different sources and model-selection procedures are combined.

Response: Neither the RelAgent nor the KumoRFM-2 paper cover the full range of comparisons we consider in Figure 1. For example, the KumoRFM-2 paper does not evaluate against any TFM-based models (our focus) outside of RelBench-v1 testing. Meanwhile, the RelAgent paper does not compare on any of the 8 SALT benchmark tasks, nor the Mimic task in RelBench-v2. Additionally, this paper does not compare against the strongest TFM-based approaches like TabPFN-Rel or the latest RDBLearn, nor several other strong RDB FMs such as RT-J, PFN-Rel, and RandGNN. Hence the coverage of RDB FMs in particular is not comprehensive in the RelAgent paper (this is somewhat expected, since RelAgent is not an FM-oriented work).

Consequently, our submission represents the first comparison specifically targeting TFM-based models (that see no RDB data during pre-training) versus non-TFM-based RDB FMs (which do see RDB data) across a broad range of benchmarking tasks. This is critical for establishing our starting premise that TFM-based approaches do indeed excel on RDB data they have never seen.

AuthorsOpenReview ↗

Response to Reviewer Usdo Cont.

Comment: In Section 4, relational databases are constructed by splitting categorical columns of single-table datasets into lookup tables while keeping the original labels. This is quite different from real temporal RDB prediction tasks, such as rel-f1, where predictions depend on historical events, their ordering, and evolving entity states. The experiment mainly tests whether models retain single-table predictive performance after schema transformation, rather than their ability to exploit genuine relational dependencies.

Response: There are many diverse RDB forms, and we concede that the construction in Section 4 is only one of them. That being said, we believe that Section 4 still motivates a valid testing regime, given that single tables, as well as multi-table expansions thereof, represent viable RDB special cases not covered by current benchmarks. In this regard, the extraction of auxiliary tables (steps 1--3) introduces new cross-row relational dependencies that may be challenging or impossible (depending on setting details) for a standard TFM to capture without a relational encoder gθg_\theta or gpfg_{\tiny pf}. We also emphasize that the train/test splits we adopt in Figure 2 are explicitly temporal (non-IID), such that predictions do in fact depend on historical context as in the rel-f1 case (even if some other details may not be the same).

Regardless, we are sympathetic to the reviewer's overall point, and recognize that more diverse testing has merit. In this regard, please see our general response to all reviewers above, whereby we introduce a revision to Figure 2 as well as new experiments varying τ\tau. The latter allows for interpolating relational effects from small to large.

Comment: The arguments favoring uniform sampling and parameter-free encoders are stronger than what the experiments establish. For example, RT-J demonstrates substantially better performance in low-label regimes and shows that graph-aware retrieval can outperform uniform sampling. This suggests that relational pretraining may improve context efficiency, an important dimension largely overlooked here. More controlled experiments varying the number of support examples, retrieval strategy, and encoder design are needed before drawing broader conclusions about relationally pretrained models.

Response: The reviewer's point is well noted. Please see our general response to all reviewers above, whereby we demonstrate that RDBLearn is actually simultaneously capable of achieving greater sample efficiency (on a per-task basis) and accuracy relative to RT-J. Furthermore, we show that this achievement relies on uniformly sampling a fixed context, as analyzed in Section 5, as opposed to random-walk sampling, conclusions based on new testing of these variants head-to-head for the first time in a controlled setting. And lastly, our new experiments also include varying context sizes in low-sample regimes, whereby RDBLearn still maintains a consistent advantage when granted uniform sampling. We sincerely appreciate the feedback and believe these results considerably improve our draft.

Comment: The analysis in Section 5 relies heavily on approximate IID sampling and sufficient relational representations, neither of which is empirically examined. In particular, the proof of Proposition 5.2 appears to infer conditional independence between labels from the sufficiency of individual representations. This does not generally follow when samples share entities or latent factors. The authors should clarify the required assumptions and the scope of the theoretical result.

Response: ICL samples can share entities and/or latent factors and still have conditionally independent labels provided the conditioning variables forming the context are sufficiently rich. The assumption of Proposition 5.2 is merely that our encoder has distilled available relational information about the label, such that remaining factors of variation are largely exogenous. We agree that this is an approximation, but it is one often adopted in analysis settings, including within the graph neural network literature involving analogous relational context (e.g., please see Remark 1 in Ma et al., "Subgroup Generalization and Fairness of Graph Neural Networks," NeurIPS 2021). Overall, some form of assumption like this is needed to make theoretical progress (which is mostly lacking thus far in the RDB FM space). We have included this reference in our updated draft and can try to clarify further if the reviewer feels that would be helpful. Thanks for the suggestions here.

Thanks for the response, the new experiment has solved most of my concerns and i will raise the score to 3.

AuthorsOpenReview ↗

Follow-up to reviewer Usdo

We put significant time into preparing the rebuttal. Thanks so much for looking into it and increasing the score, while acknowledging that the new content addressed concerns.

Reviewer_5G81OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why tabular foundation models can perform well on relational databases without relational-specific pretraining. The authors compare existing approaches, including RDBLearn and TabPFN-Rel, with specialized relational foundation models such as KumoRFM. Figure 1 shows that tabular foundation model approaches achieve competitive or better performance across six benchmark groups.

The paper mainly tries to explain why these models work well. The authors suggest that relational databases are not always very different from single tables, that randomly selecting labeled examples can be useful, and that simple methods can capture information from related tables without additional training. The authors support their arguments with theoretical analysis and experiments on real and synthetic datasets, including tests involving interactions between relational features.

Critical strengths
  1. Broad experimental comparison. The paper compares many relational database prediction methods across six benchmark groups. Figure 1 shows that approaches based on tabular foundation models, such as RDBLearn and TabPFN-Rel, often outperform specialized relational foundation models, including KumoRFM. These results are valuable because they show that relational-specific pretraining is not always necessary to achieve strong performance.

  2. Useful insights into why simple relational encoders work. The paper offers several explanations for why tabular foundation models can handle relational data. In particular, Section 6 examines whether simple encoders can capture interactions between features. Figure 3 shows that RDBLearn remains competitive with more complex relational models, even on tasks involving feature interactions. This provides useful insights into the strengths of simpler approaches.

Critical weaknesses
  1. Limited experimental validation of the proposed continuum between single-table and relational tasks. In Section 4, the authors introduce a parameter τ\tau that controls how much the target label depends on information from related tables. However, in Figure 2, they convert existing single-table datasets into relational databases while keeping the original labels. They do not test different values of τ\tau. A useful experiment would compare RDBLearn and relational foundation models, such as KumoRFM-2, as τ\tau increases from 0 to 1. This would show whether RDBLearn remains competitive when predictions depend more strongly on relational information. Without this experiment, one of the paper's main explanations is not directly tested.

  2. Limited experimental support for the proposed explanations of model performance. In Section 5, the authors argue that uniform sampling of labeled examples helps tabular foundation models perform well on relational databases. However, they do not directly compare uniform sampling with graph-based sampling while keeping the model, encoder, and number of examples fixed.

The theoretical analysis in Section 5 assumes that predictions become more accurate and stable as the number of labeled examples increases. The authors do not test whether this behavior occurs for RDBLearn or TabPFN-Rel. An experiment measuring prediction error and variability across different context sizes would help connect the theoretical analysis to the observed performance.

Since the main contribution of the paper is to explain why existing tabular foundation models perform well on relational databases, these missing validations limit the strength of its conclusions.

Questions for authors
  1. Evidence for the continuum between single-table and relational tasks. In Section 4, the authors introduce the parameter τ\tau to control how much the target depends on relational information. However, Figure 2 keeps the original labels and does not vary τ\tau. Could the authors clarify what evidence supports their claim that tabular foundation models perform well across different levels of relational dependence? Results with increasing values of τ\tau would help determine whether RDBLearn remains competitive when relational information becomes essential for prediction.

  2. Evidence for the benefits of uniform sampling. In Section 5, the authors suggest that uniform sampling helps tabular foundation models generalize to relational databases. However, the comparisons with models such as RT-J involve differences in both architecture and sampling strategy. Could the authors clarify what evidence shows that uniform sampling itself contributes to the better performance? A controlled experiment comparing uniform and graph-based sampling with the same model, encoder, and number of labeled examples would help support this explanation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5G81

We appreciate the reviewer acknowledging that our work provides useful insights into why simple relational encoders work, particularly w.r.t. our Section 6 analysis of feature interactions, and empirical validation thereof. We also note that all reviewer concerns boil down to requests for further experiments, particularly:

  1. Varying τ\tau from Section 4,
  2. Comparing uniform vs random-walk sampling with relevance to Section 5, and
  3. Testing variation over context sizes, also related to Section 5.

As other reviewers have requested similar experiments, we consolidate all new testing to general responses above, which explicitly cover conditions 1–3 (see also new Appendix C.1 in our revised draft for related details). Thanks for these recommendations, which together serve to improve our work considerably.

AuthorsOpenReview ↗

Final Engagement Request

As the discussion period will soon come to an end, we politely draw attention to our extensive rebuttal material that we believe positively addresses each initial reviewer comment. Does the reviewer feel that these responses adequate? If not, please let us know so that we still have a chance before the discussion deadline to provide any further details as might be helpful for clarifying our work and interpretations thereof.

Reviewer_ovsFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks why single-table tabular foundation models used on flattened RDB neighborhoods perform so well on relational prediction tasks despite never being pre-trained on relational data. Relational foundation models pre-trained on RDB data perform less well. The paper compares two TFM-based pipelines (RDBLearn and TabPFN-Rel) against open- and closed-source RDB FMs and supervised baselines on six benchmarks: RelBench v1, RelBench v2 (classification and regression), 4DBInfer and SALT. The paper argues that: 1) single tables and RDBs lie on a continuum; 2) a necessary condition for asymptotically unbiased ICL predictions is that the predictor relies only on context samples close to the query in encoder space, and uniform context sampling plus column-wise encoding approximately recreates the IID setting in which TFM posterior predictive distributions converge, while random-walk context sampling does not; 3) parameter-free encoders are not fundamentally disadvantaged, since any fixed lossy encoder admits counter-examples; 4) column-wise encoders combine naturally with LLM agents that can add "golden features".

Critical strengths

S1. The evaluation is done across many different benchmarks, which gives robust empirical observations that lay the ground for the analysis that follows.

S2. Evaluating relational foundation models on RDBs induced from single tables is a nice benchmarking proposal, making the continuum perspective argument an evaluation principle.

Critical weaknesses

W1. The explanations are largely disconnected from the experiments. The empirical results only show that TFM pipelines are competitive, which is already known from Xu et al. 2026 and Hayler et al. 2026, but do not isolate why they are.

  • The continuum argument is not tested empirically. The generator of Eq. (1) is never used experimentally. Fig. 2 keeps the original labels, so the induced relations carry no information by construction.
  • The locality and posterior predictive distributions argument (Sec. 5) hinges on uniform vs. random-walk context sampling and on growing context size n, but neither is varied. A direct test would be to run RDBLearn with random-walk context, or RT-J with uniform context, and to vary n.

W2. The explanations are of limited novelty or remain opinions and give no actionable guidance for improving RDB FMs. The paper gives no concrete direction for RDB FM design beyond “behave like a TFM on single tables”. RDB FMs are strictly more expressive and see in-domain pre-training data, so the interesting question is what currently prevents them from exploiting this: the pre-training prior, context selection, encoder design, label usage? The paper does not answer it.

Questions for authors
  1. Can you provide ablations that test each explanation directly? For instance:
  • vary τ\tau in the Eq. (1) generator rather than keeping the original labels;
  • swap context-sampling strategies within the same model (uniform vs. random-walk);
  • compare column-wise vs. column-mixing encoders in otherwise identical pipelines.
  1. If you append increasing numbers of uninformative columns to the auxiliary tables, distances in z grow and neighbourhoods become non-local. Does TFM performance degrade as Prop. 5.2’s reasoning would suggest? If not, what does the locality argument explain at practical context sizes? Does the argument distinguish TFMs from RFMs that also sample context uniformly?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ovsF

We thank the reviewer for pointing out that our manuscript is clear and easy to follow, and for recognizing that the continuum perspective leads to a useful evaluation principle. We address each reviewer concern in what follows.

Comment: The explanations are largely disconnected from the experiments. The empirical results only show that TFM pipelines are competitive ... but do not isolate why they are.

Response: In Section 6 explanations we rigorously argue for why TFMs should be able to handle interaction terms comparatively well, despite prior work claiming that this should not be possible. Figure 3 experimental results then directly corroborate this analysis. Likewise for the agentic arguments from Section 7 with validation in Figure 3 (right). But we agree with the reviewer that other experimental corroboration is warranted, which we have now provided in our general response to reviewers above.

Comment: The continuum argument is not tested empirically. The generator of Eq. (1) is never used experimentally. Fig. 2 keeps the original labels, so the induced relations carry no information by construction.

Response: Please see our general response to all reviewers above, where we directly address this concern, including new experiments based directly on Eq. (1).

Comment: The locality and posterior predictive distributions argument (Sec. 5) hinges on uniform vs. random-walk context sampling and on growing context size n, but neither is varied. A direct test would be to run RDBLearn with random-walk context, or RT-J with uniform context, and to vary n.

Response: We provide new comparisons between random-walk and uniform sampling in our general response above to all reviewers. These results demonstrate that RDBLearn with uniform sampling can achieve higher accuracy, even with fewer labeled samples per task, relative to RT-J with random-walk sampling. We also conduct new experiments as the context size nn is varied, whereby RDBLearn outperforms RT-J at each available per-task context budget.

Comment: The explanations are of limited novelty or remain opinions and give no actionable guidance for improving RDB FMs. The paper gives no concrete direction for RDB FM design beyond “behave like a TFM on single tables”.

Response: Our consideration of single tables, and relational extensions thereof, is merely to emphasize that RDBs include such regimes as a special case. It then naturally follows that RDB FMs should natively handle single tables or single-table adjacent scenarios, which motivates inclusion of complementary RDB benchmarking tasks like what we have introduced in Section 4 (which does not appear in prior work). In other words, if an RDB FM does not perform well on single tables or similar, that indicates a notable deficit that prior benchmarks have thus far not considered. See below for additional responses related to the explanatory value of our contributions.

AuthorsOpenReview ↗

Response to Reviewer ovsF Cont.

Comment: RDB FMs are strictly more expressive and see in-domain pre-training data, so the interesting question is what currently prevents them from exploiting this: the pre-training prior, context selection, encoder design, label usage?

Response: These questions are addressed in Sections 5 and 6, but we apologize if our intended message was not sufficiently clear (and concede that our draft can still be further improved). We clarify a few key high-level points as follows:

  • Among other things, in Section 5 we argue that adopting RDB synthetic priors for pre-training may be less effective than analogous pre-training in single-table regimes. This is largely because of two inter-related effects: (i) ICL samples need no longer factorize, which can compromise desirable asymptotic effects that help overcome inevitable prior misspecifications; and (ii) Prior misspecifications can be more likely to occur in RDBs than single-table regimes, simply because RDBs are much more complex and fewer real-world examples are available as templates. Hence the end result is that RDB synthetic priors can potentially have greater misspecification (relative to single tables), and yet there is reduced ability to compensate via the posterior predictive distribution.
  • Meanwhile Section 6 examines the expressiveness issue, arguing, perhaps counter-intuitively, that more parameters in a pre-trained encoder does not actually resolve interaction terms. This is because encoder parameters must remain fixed for all RDBs (unlike supervised regimes where they can be tuned on a per-task basis), so they cannot adapt to select different combinations of candidate interactions which may vary across tasks. For resolving the latter, a key aspect is the compression rate of the encoder (which determines how many interactions can even be preserved), whereby parameterized and parameter-free encoders are on an equal footing. And empirically (Figure 3) we show that the TFM-based approach is capable of handling interaction terms as well or better than the other RDB FMs, supporting our analysis. (Note that if we allow for per-task fine-tuning, this would surely change, since these other RDB FMs could adjust their encoder parameters to select the needed interactions through direct supervision. But of course then we are no longer in the FM regime.)

Comment: Compare column-wise vs. column-mixing encoders in otherwise identical pipelines.

Response: This is a good suggestion; however, it has actually been done in prior work. For example, please see Figure 8 in reference (Xu et al., 2026). Additionally, we remark that the RandGNN baseline in Figure 1 of our submission combines such cross-column mixing with a TFM, and serves as a form of ablation to RDBLearn/TabPFN-Rel which strictly rely on column-wise encoders.

Comment: If you append increasing numbers of uninformative columns to the auxiliary tables, distances in z grow and neighbourhoods become non-local. Does TFM performance degrade as Prop. 5.2’s reasoning would suggest?

Response: This is a great question. And yes, generally speaking increasing the number of informative columns while fixing the number of ICL samples can in principle degrade performance as suggested. But there are two other important factors to weigh here:

  • Firstly, Prop 5.2 considers fixing the number of columns and then varying the number of ICL samples; so even when uninformative columns are present, increasing the ICL samples can provide mitigation.
  • And secondly, while Prop 5.2 provides a useful asymptotic perspective, examining what happens outside of strict convergence is still relevant in practice. For example, the prior specification can steer the posterior predictive distribution towards areas that largely ignore uninformative dimensions, even if we cannot hope to exactly achieve Eq. (2) within a given ICL sample budget. And indeed, TFMs are explicitly pre-trained with this in mind by including diverse sets of uninformative columns within synthetic datasets.

Comment: Does the argument distinguish TFMs from RFMs that also sample context uniformly?

Response: Also a good question. Prop 5.2 does not specify how the RDB FM was obtained, and so by itself does not distinguish a TFM-based from a non-TFM-based approach. However, later discussion in Sections 5.2 and 5.3 does address potential differences between these two options. Specifically, we point out that when pre-training with synthetic RDBs drawn from a prior that does not generally factorize over ICL samples, then Eq. (6) need not be satisfied. And so reverting to a simple IID approximation as obtained from a TFM may be sufficient. Even so, the difference between the TFM-based model and other RDB FMs that sample context uniformly is likely to be less per our theory, which is also born out by our empirical results (e.g., Figures 1,2,3), whereby KumoRFM-2 generally outperforms RT-J.

AuthorsOpenReview ↗

Final Engagement Request

As the discussion period will soon come to an end, we politely draw attention to our extensive rebuttal material that we believe positively addresses each initial reviewer comment. Does the reviewer feel that these responses adequate? If not, please let us know so that we still have a chance before the discussion deadline to provide any further details as might be helpful for clarifying our work and interpretations thereof.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response and New Experiments for All Reviewers

We appreciate the constructive comments provided for our paper. As multiple reviewers had significant overlapping questions, we address these upfront here. We emphasize that the new experiments and analysis inspired by reviewer suggestions significantly improve our work. We have already uploaded a new draft of our paper to openreview that includes some of these changes.

Clarification of Section 4 Continuum Perspective

Our original intention for the continuum perspective of Section 4 was not to introduce a new synthetic benchmark, whereby we artificially vary relational effects through the use of τ\tau. Instead, we introduced Eq. (1) and the surrounding text as a representative illustration of the fact that there is generally no clear line between single tables and RDBs. This then motivates testing RDB FMs on single-table-like tasks with reduced relational effects (small τ\tau), as stronger relational effects (larger τ\tau) are already implicitly explored in the existing RDB benchmarks from Figure 1. Along these lines, the Figure 2 experiments in Section 4 were our first attempt at testing the small τ\tau regime, where we intentionally chose to use original labels for realism, allowing for the fact that the adopted benchmarks may inherently have some minor degree of cross-row dependency over the official non-IID temporal train/test splits. Hence we continue to believe that this original setup represents a viable complementary benchmark for RDB FM testing.

That being said, after reading all reviewer comments, it has now also become clear that explicitly testing the continuum by varying τ\tau from 0 to 1 represents another worthwhile means of validation. We conduct such experiments below; thanks to all reviewers who each expressed some version of this recommendation.

Bug Fix in Original Figure 2 Experiment

In our original experiments behind Figure 2, we recently found a bug in the code, whereby RDBLearn was inadvertently run without a temporal cut-off (this bug did not impact RT-J or KumoRFM-2). Once corrected, on regression tasks RDBLearn average performance is actually a bit better than TabPFN (see updated Section 4 and Table 7 in revised draft; see also new table with τ=0\tau =0 below). This plausibly indicates the presence of modest relational effects along the continuum within at least some of these datasets.

New Experiments Varying τ\tau

Per reviewer suggestions, we have conducted new experiments following the prescription from Section 4. To generate new labels over varying τ\tau, we implement Eq. (1) with f1f_1 equal to original/real data set labels and Ψ\Psi as a leaky-ReLU function. We then form f2f_2 by collecting 2-hop neighbors over the induced graph, compute column skewness features for each neighborhood, concatenate, and pass through a randomized MLP. In this way, τ=0\tau = 0 is equivalent to the original data label subject to an additional non-linearity, while τ=1\tau = 1 is independent of original labels, and entirely determined by induced relational effects.

We execute this process across all regression tasks from Figure 2, and repeat for different values of τ\tau. The table below shows mean R2R^2 values (higher = better) at each τ\tau; top two model results per row are highlighted in bold. From these results, we observe that RDBLearn achieves the best overall performance across the tested spectrum, while the second best flips from TabPFN (w/ smaller τ\tau) to KumoRFM-2 (w/ larger τ\tau). As expected, TabPFN performance falls off for large τ\tau (becoming worse than all RDB FMs), as we do not grant this model any relational features.

τTabPFNRDBLearnRT-JKumoRFM
0.000.3510.3680.0650.269
0.250.3000.4150.0880.222
0.500.0360.4300.0610.203
0.750.0270.4310.0400.187
1.00−0.2640.5070.0310.252
AuthorsOpenReview ↗

General Response and New Experiments for All Reviewers Cont.

New Comparisons with Random-Walk vs Uniform Sampling

Per multiple reviewer suggestions, in the new table below we compare random walk (RW) sampling with uniform sampling averaged across all RelBench-v1 classification tasks (approximate numbers are estimated from RT-J paper Figure 3; please see new Appendix C.1 in our revised draft which contains further details). We also include the number of unique labeled target rows used for context, averaged across tasks (we adopt geometric means since scales may differ over tasks). From these results, we observe that RDBLearn w/ uniform sampling actually achieves higher accuracy with a fewer number of labeled samples per task relative to RT-J (or RDBLearn w/ RW sampling).

We emphasize that the label usage averages in the RT-J paper are computed in a completely different way. Specifically, the RT-J paper only averages the number of labels used for a given test point, without accounting for the fact that each RW-chosen label set changes from test point to test point, i.e., each test point requires its own unique set of ICL samples. This method of accounting substantially penalizes RDBLearn with uniform sampling, which relies on a larger but fixed ICL sample set that is shared across all test points for a given task. Hence for a given task, the total number of unique labeled samples required by RDBLearn will generally be much less than RT-J, as shown in the RelBench-v1 classification result table below.

ModelSamplingRW Context SizeAvg. AUROC (↑)\left(\uparrow \right)Avg Unique Labels Used per Task (↓)\left(\downarrow \right)
RT-JRW256∼\sim 6922197
RT-JRW8K∼\sim 7383020
RT-J (w/ ensembling)RW8K74.4595079
RDBLearn (w/ TabICL-v2)RW256∼\sim 6522197
RDBLearn (w/ TabICL-v2)RW8K∼\sim 7283020
RDBLearn (w/ TabICL-v2)UniformN/A75.07816

In the table above, for all RW models, we obtained the last column values by counting up the total number of unique labeled target rows across all test points used for a given task (i.e., if a labeled context row is used as context at multiple test points, it is not counted twice). Meanwhile, for the Uniform model, the total number per task is simply given by min(10K, train_set_size), the default RDBLearn context size. Additionally, we do not consider RT-J with a uniformly sampled context since the model was not pretrained for this scenario, and so performance will understandably degrade significantly as alredy shown in Figure 6 of the RT-J paper.

AuthorsOpenReview ↗

General Response and New Experiments for All Reviewers Cont.

New Experiments Varying Context Size

We also conducted experiments whereby we vary a shared number of context labels (or equivalently ICL samples) available for both RT-J and RDBLearn. To explore the low-sample regime per reviewer comments, we chose the rel-trial dataset (from RelBench-v1) which has a small test set. In this way, the number of labels used by RT-J stays relatively small as we vary the number of context cells (e.g., between 256 to 8K as in the RT-J paper); we bound RDBLearn by the same RT-J label count so obtained. This range keeps the matched label count used by each model below 10K, the default RDBLearn context size when available. Hence we can compare models head-to-head with equal label budgets in lower sample regimes. The tables below display the performance on the study-outcome (classification) and study-adverse (regression) tasks. Overall, RDBLearn (w/ uniform sampling) shows stronger performance, and both models largely saturate once enough ICL samples are available.

RT-J Context (# Cells)Total Labels UsedRT-J AUROC (↑)\left(\uparrow \right)RDBLearn AUROC (↑)\left(\uparrow \right)
2564680.4960.637
5128400.5380.670
1K14740.5540.691
2K27130.5700.710
4K49800.5740.712
8K83350.5530.719
RT-J Context (# Cells)Total Labels UsedRT-J MAE (↓)\left(\downarrow \right)RDBLearn MAE (↓)\left(\downarrow \right)
256169858.5047.81
512267257.2143.99
1K355357.1744.26
2K523556.5043.62
4K958755.7943.77

In both tables above, we sweep RT-J context size (first column), then run the RT-J random walk to compute the total number of allowed labeled target rows for the task (second column), restricted to be 10K or less. We next run both RT-J and RDBLearn (w/ uniform sampling) both bound by the same shared label limit. (Note that, because of large test sets, for most other RelBench-v1 tasks, even the smallest RT-J context size of 256 leads to total labels much greater than 10K.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.