- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
Every database is mapped into one fixed "meta-schema" with everything (tables, columns, references, cells) stored as data -- reification, essentially -- and a standard heterogeneous GNN is then trained across databases and applied to unseen ones. The claim is that foundation-model capabilities (zero-shot transfer, label-efficient fine-tuning) emerge from this representation alone, with unchanged GNNs matching or beating the custom relational foundation models on RelBench, following RT’s leave-one-database-out protocol (the model gets the entity’s past labels, “self-labels”, as RT does, plus a one-sentence task description that RT doesn’t get).
This is a very straightforward and somewhat provocative idea - a bit like saying "just turn RDL into knowledge base completion over triplets and it works better (so why bother with RDL?)" - and I’d guess many people had it and dismissed it, since it breaks the core structural character/bias of the data-model match that the whole heterogeneous-GNN story of RDL is built on. And here it is presented very nonchalantly, in the standard method-experiments-ablations form, as if nothing happened... so I’m somewhat suspicious.
- I’m not surprised it doesn’t fall to zero (even propositionalization itself is pretty strong on RelBench), but beating SotA like this seems either like some data leakage, or an artifact of the benchmark/evaluation itself (the likely option, see below).
Critical strengths
- The idea is clean and easy to understand, nothing gets invented architecturally, and code is linked (not checked). If the result holds, any existing RDL model becomes transferable for free, which sounds promising (I'd say).
- The limitations are stated honestly (single seed, data versions, the task text RT doesn’t get), so the concerns below are still answerable in a rebuttal (with ablations).
Critical weaknesses
-
What really transfers is likely not the representation but the self-labels + task text, and that rather tells more about RelBench (a classic weakness, also reported in previous works) than about the representation. The Entity Mean/Median baseline, i.e. just the entity’s own label history, is on par with you on regression and close on classification on average :) - and RT is apparently just bad on regression in general (below the baseline in Tab. 1), so "surpassing RT" there means little, I think. Your ablation (Fig. 3) then shows the task text has by far the largest effect, while the schema names have none at all and the statistics little on average - i.e. the model doesn’t seem to read any schema semantics, it uses the schema tuples merely as bare connectivity - so "schema variation moved into the data" -- technically yes, but the model mostly ignores it, which kind of breaks the main point of the paper... This could be resolved with an ablation removing the self-labels (and both self-labels and task text together), or vice versa - keep only those two without the graph and see if it drops. Plus the original-schema baseline in the fine-tuning comparison "reads neither the task text nor the self-labels" doesn't look like a fair comparison, no?
-
Overclaiming -- with a single seed (and the ~1 AUROC run-to-run variation you mention), the results are “comparable to RT” on classification, not “match or surpass” (on regression you do beat it, but see point 1 - RT is below the trivial baseline there; Griffin is clearly behind, fair). Similarly, the “contrast” with the complex architectures is overstated - relational transformers are schema-agnostic by design in the same reification sense (everything embedded at the cell level), so the two seem closer than the text suggests, and the fixed meta-vocabulary idea has prior work (ULTRA for KGs, afaik, before the cited “concurrent” one).
Questions for authors
- The self-labels -- are they filtered by the tuple time T, or by the end of the label window T+w? (a label for [T, T+w) is only known at T+w -- i.e., leakage for training rows if the former)
- Can you do the zero-shot without self-labels and without the task text (and/or the no-graph version)? -- this decides it for me (either way).
- And the original-schema fine-tuning baseline with self-labels added...
Suggestion: the actually interesting finding here is that plain message passing over such reified graphs doesn’t lose on RelBench - I’d say that openly from the start, with what it implies about the benchmark...
3 comments
Response to Reviewer a5QY (2/2)
W1 (schema semantics, Fig. 3). For RelGT, we agree with your reading. We ran the same ablation on GraphSAGE, and there the schema carries more. Masking the task text costs less (1.8 AUROC, against 4.3 for RelGT). Masking the schema names costs 0.9 on average, and dropping the schema tuples costs 2.2 on average. We will add this ablation next to Fig. 3.
Q3 (fine-tuning baseline). Agreed, the baseline lacked both inputs. We ran the one you asked for, and the conclusion holds: with matched inputs, the pretrained model still learns faster than training from scratch, on either schema. On the original schema with self-labels and the task text added (RelGT), these inputs help the baseline, mostly on regression, but the pretrained model stays ahead: 73.8 vs. 70.4 AUROC at 128 steps and 75.0 vs. 72.9 at 256 steps. On the same universal schema and inputs, pretrained vs. random initialization gives +6.4 AUROC at 128 steps for RelGT (73.8 vs. 67.4), +8.0 for HGT and +1.6 for GraphSAGE. At fixed numbers of distinct labels the picture is the same: with 256 labels the pretrained model reaches 73.7 mean AUROC against 69.6 and 69.5 for the two from-scratch baselines (see our reply to Reviewer UMzU, Q3).
Q1 (self-label time). There is no leakage here: a self-label is used only if its timestamp is strictly before the prediction time, and because RelBench spaces all of a task’s timestamps at least one label window apart, every such label’s window has already closed at prediction time (see experiments/data/self_labels.py:191 in our repository and relbench/base/task_base.py:107-144 in RelBench).
W2 (overclaiming). Accepted. A second seed for all three models moves the means by at most 0.4, so we will write "comparable to RT", not "match or surpass". On the contrast with RT: both are schema-agnostic at the cell level, but we encode this in the data rather than the architecture, so a standard GNN can be used. We will tone the contrast down and cite ULTRA as earlier work on a fixed vocabulary for knowledge graphs.
Suggestion. We will do this. The introduction will say that plain message passing over the reified graph does not lose on RelBench, and that under RT's protocol the label history alone matches the full model on average. RT's own ablation points the same way. With the experiments above in the paper, the setting without self-labels becomes the stronger test of transfer.
I appreciate the detailed replies and results, and I believe they largely support my interpretation that most of the transfer is the label history (only when it's weak, the representation adds a bit, granted). The experiments without self-labels are very nice (likely the only real test of transfer), thanks. It's also good to see the benefits of fine-tuning given matched input info. Also the self-label timing concerns are gone, so I'm increasing my rating of soundness. Overall, I see this as a nice, extensive experimental analysis of what actually transfers on RelBench (and of its weaknesses), rather than some breakthrough representation idea though, which I find somewhat less striking/impactful than what the paper framing suggests.
Response to Reviewer a5QY (1/2)
Thank you for a highly useful review. We ran the experiments you asked for. They helped us sharpen the claims.
Q2 / W1 (what transfers). We removed the database: the model sees only the entity's self-labels and the task text, with the same model, tasks, and budget. On average, the self-labels alone seem to match the full model. Per task, the picture splits in two. Where the label history is highly predictive, the database adds nothing and can even lower the score. Where it is weak, the database raises the score, sometimes by a lot: +5.8 (RelGT) and +10.6 (GraphSAGE) on Trial study-outcome, and +7.5 and +3.4 on Avito user-clicks. So the experiment says something about RelBench, and also about the universal schema: where the label history is not enough, a standard GNN trained on other databases makes use of a database whose schema it never saw.
Mean AUROC over the 12 held-out tasks:
Task text. The task text matters less than the ablation suggests. Fig. 3 masks it at inference, in a model trained with it. Trained without it, GraphSAGE reaches 70.7 instead of 71.6 and RelGT 68.4 instead of 70.8.
Relational transfer with no self-labels at all. This is your second suggestion: train and test without self-labels and without task text. Without both, the model has no way of knowing what to predict: a task and its negation would get exactly the same input. So the model must be told the task in some other way, and it must be one that reveals nothing about the entity being predicted, so that the original concern does not return. We use 1,024 labeled examples of the task, taken from other entities, and the same examples for every prediction. Each example is embedded by the same GNN, and the prediction attends to these examples and their labels. Because they are identical for every entity, they cannot tell two entities apart. Any score above 50 AUROC must therefore come from the entity's own data in the mapped database. To keep RelBench out of training entirely, we trained on 22 databases from The Join, none from a RelBench source, and tested on all of RelBench. Following Reviewer UMzU's concern, the checkpoint was also chosen on other Join databases that are never evaluated, so no RelBench label is used for training or for selection.
Mean AUROC is 64.2 with per-task checkpoint selection and 63.7 with this target-blind selection. The 14 points above 50 come from reading a mapped database the model never saw. Outside F1 and Event, the two databases where label history dominates, this model reaches 66.9, level with the self-label models (67.1) and above Entity Mean (65.4), with no label history of the entity at all.
With your agreement, we would be glad to add these experiments to the paper, and we would welcome any other control you consider informative.