- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RELICL, a framework for relational data with tabular foundation models. The paper identifies two challenges within the relational data framework, and proposes RELICL that replaces column-wise aggregation with step-by-step row embedding propagation and fusion using an inference Directed Acyclic Graph (DAG). The claims of the paper shown empirically, and results show competitive performance of RELICL.
Critical strengths
In general, the paper is organized and easy to follow. The problems that the paper identifies (which has been around in the literature, but not well formulated) are key to improving tabular models for relational data. Moreover, the performances of the proposed method is competitive.
Critical weaknesses
- The scope of the datasets used to prove the claims is limited. Only a few datasets from OpenML, which are probably mostly numerical (ignoring non-numerical values) could weaken the argument. Moreoever, most of the analysis on RelBench, which authors do mention as the limitation, could be critical.
- It would be good to have some comparisons and analysis in computational perspectives.
Questions for authors
- Since the evaluation is rather limited on RelBench and the results show mixed results (for instance compared to TabPFN-Rel), what are some characteristics from the data that guarantees strong performances or that might hurt RELICL?
2 comments
W2: Missing comparison and analysis of computational perspectives
We added run time results to App. E.7 (single GPU). RelICL's run time grew sublinearly with respect to the number of validation/test rows but it did not grow with the number of rows in the database (due to sampling). Most of RelICL's run time stemmed from the TFM. We did not optimize our code for faster run time, and a direct run time comparison with other methods would require rerunning all baselines on the same hardware, which is infeasible for us.
Also note that RelICL needs neither training nor per-task tuning (App. D.3 and D.4) and that the number of features added to the task table does not explode with the schema size (unlike DFS-based methods, Sec. 2.3).
W2: Details
It would be good to have some comparisons and analysis in computational perspectives.
Compared to many GNN-based methods, e.g., GraphSAGE or RelGNN, RelICL needs no training time. RelICL uses a common set of hyperparameters for all databases and tasks (App. D.3), whereas other methods tune a configuration per task (e.g., RDBLearn and TabPFN-Rel).
Compared to DFS-based methods, RelICL does not require any per-task tuning. Further, the number of features added to the task table for RelICL depends only on its direct neighbors (Sec. 3), while for DFS-based methods it depends on the number of meta paths, which may grow exponentially with respect to the number of tables in the database (Lam et al., 2017; Sec. 2.2 and 2.3).
The added run time results (App. E.7) were produced on a shared cluster on a single GPU; we did not optimize our code towards better run time (no parallelization, single thread, no caching of embeddings or rewrites, etc.).
In short, RelICL's prediction time is dominated by the TFM used to compute row embeddings and make the final predictions (more than half of the total run time, based on the logs). The run time per task is correlated with the number of validation/test rows per task; more of those lead to more batches, in turn leading to a longer prediction time (App. E.7, Fig. 7, left panel). Run time does not grow with the database size (right panel) because RelICL samples rows per context table (App. D.3, Tab. 9b).
Q1: Data characteristics that influence RelICL's performance
Since the evaluation is rather limited on RelBench and the results show mixed results (for instance compared to TabPFN-Rel), what are some characteristics from the data that guarantees strong performances or that might hurt RELICL?
Thank you for this question. We added an exploration of data characteristics (number of tables, columns, and rows) to App. E.5, Fig. 5. We found no evidence for a relationship between these data characteristics and the performance of RelICL; they do not seem to capture task difficulty. In comparison to TabPFN-Rel, RelICL does not tune per task.
Q1: Details
The panels in Fig. 5 (App. E.5) plot the rank of RelICL against the number of (i) tables, (ii) columns, and (iii) rows of its database. The rank is among the models we compared against in the main section (Tab. 5a and 5b). None of (i)--(iii) showed a visible correlation with the rank, i.e., we found no evidence that these data characteristics influence RelICL's performance.
Thank you for your detailed, constructive feedback and support of our work.
We share our thoughts on the weak points you raised as well as the questions below. We revised the paper accordingly; changes are highlighted in blue.
We'd like to briefly position our paper first: The goal of RelICL is to explore whether strong tabular foundation models (TFM) can be leveraged directly for zero-shot relational learning, without being limited by feature explosion and interaction blindness and without task-specific training, fine-tuning or tuning. Our results indicate that RelICL, albeit simple, is competitive with SOTA methods (and it was the best-performing method that did not use per-task training/fine-tuning/tuning).
W1: Limited scope of datasets & focus on RelBench
We agree that (i) the experiments exploring Assumption 1 focused only on a few classification datasets from OpenML and one TFM, and (ii) our main empirical evaluation was limited to RelBench.
To address (i), we extended our experiments (Sec. 3.1, App. C.2) to include twice the number of datasets (including regression datasets and with different feature types) and also added a different TFM (TabPFN). In short, we found that Assumption 1 is reasonable in that encoding and dimensionality reduction typically retain TFM performance (small drops, if any, often not statistically significant). There are exceptions, but as RelICL is based on Assumption 1, our main experimental results provide further justification for the assumption. (Also note that the DFS-based approaches inherently assume that feature interactions do not carry predictive signal, an assumption that RelICL does not make.)
(ii) is more difficult to address. The goal of our experimental study was to compare RelICL to prior work, and prior work was predominantly evaluated on RelBench (a diverse benchmark in itself: 21 tasks, small and large, regression and classification). We added results on the RelArena benchmark (Sec. 4, App. E.6) to the paper, but note that RelArena is based on RelBench and that RelArena results for relevant related work (Tab. 4) are often unavailable. We found that RelICL was competitive on RelArena as well: It was within critical distance of the best-performing method, without using per-task training/fine-tuning/tuning (in contrast to TabPFN-Rel or RDBLearn, which use per-task tuning).
W1: Details
The following table summarizes our experimental results around Assumption 1. Performance loss when using embeddings (4th vs. 3rd column) and dimensionality reduction (5th vs. 4th col) was small and often not statistically significant (see standard errors). Dataset statistics and detailed results per dataset are available in App. C.2.
Thanks! Half of the now 20 datasets also have categorical features, and two of them are purely categorical (kr-vs-kp and mercedes). We added feature types to the dataset statistics table in the appendix.