- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes HGraphFM, a prior-data fitted network for node classification and regression on heterogeneous graphs and relational databases. It has three components.
- A synthetic heterogeneous graph prior. It extends the prior of GraphPFN: a homogeneous graph is generated, its nodes are split into tables, and each table gets new features and labels.
- A model that adds graph attention layers to TabICL. It is trained in two stages: first on the new prior, then on the relational prior of RDB-PFN.
- DFS-BP, a feature augmentation step for relational data at test time. It builds a graph over the rows of the task table, filters the features over this graph, and adds the filtered columns that score best on the training labels.
The evaluation covers node classification with 10, 20 and 40 labeled examples on ACM, DBLP and IMDB (stage-1 model), and classification and regression with 1024 labeled examples on ten RelBench tasks (stage-2 model). HGraphFM is best or close to best in most settings, and DFS-BP gives further gains on RelBench classification.
Critical strengths
The paper has a relevant goal: one model that is pretrained only on synthetic data and handles heterogeneous graphs, including relational databases. On RelBench, without DFS-BP, it reaches an average AUROC of 0.694, against 0.680 for RDB-PFN, the strongest PFN baseline. However, the weaknesses below limit what we can conclude from it.
Critical weaknesses
1. The paper is hard to follow, and key parts of the method are missing from the main text.
Much of the main text consists of general statements that do not explain what the method does, such as "a unified architecture with both in-context learning and structure learning capabilities, enabling unified modeling and fast adaptation across diverse heterogeneous graph tasks" (lines 68-69). At the same time, important details are missing:
- Section 3.1 describes the prior three times (as a list, in Figure 1, and as a paragraph), each with its own notation. For example, the number of tables is in the list and in the paragraph, and denotes both the latent matrix and the number of layers. Moreover, the details of the graph generation, the graph size and the number of tables are given only in Appendix D and the GraphPFN paper.
- The architecture is described only by a short paragraph in Section 3.2 and by Figure 2. Neither explains how graph attention works across tables with different columns, whether the model uses edge types, or how the subgraph for each query is sampled.
- The paper never explains why its prior should transfer to the target data. The prior has a single type of edges, both within and across tables, and nodes are assigned to tables at random. In contrast, heterogeneous graphs have several edge types, almost all between different node types, while relational databases have only foreign-key edges between tables.
- Homogeneous graph foundation models are applied to heterogeneous graphs using only PCA and zero-padding of node features (Appendix C). However, the paper does not say which graph these models receive.
2. The paper combines existing components without analysis, and the evidence is limited.
The stage-1 prior extends GraphPFN, the stage-2 prior comes from RDB-PFN, the model is TabICL with graph attention, and DFS-BP combines deep feature synthesis with existing spectral filters. Combining existing parts can be a valid contribution, but then the paper needs to show which parts matter and why. Here, the only ablation turns DFS-BP on and off.
- No ablation tests the stage-1 prior, the graph attention layers or the two-stage training. Therefore, the claim that stage-1 pretraining helps relational tasks has no support.
- The model is called unified, but each stage is evaluated only in its own setting: stage 1 on heterogeneous graphs, stage 2 on relational data.
- DFS-BP confounds the comparison with the baselines. It selects features by their score against the labels of the full training set. In contrast, the baselines see only 1024 labeled rows, and DFS-BP is never applied to them. Moreover, Figure 4 shows that on two Avito tasks, a single selected feature already scores close to or above RDB-PFN. Therefore, the reported gain may come from the label information in DFS-BP alone.
- Besides, DFS-BP is not connected to the main idea of the paper, which is pretraining on a synthetic prior. It runs outside the model, only on relational data, and only at test time. Thus, it looks like one more trick added to raise the numbers.
- Finally, the heterogeneous graph evaluation uses three small, old datasets. The paper does not explain how the ten reported RelBench tasks were chosen, and it reports no standard deviations, although many gains are small.
3. The released code differs from the paper in ways that affect the reported results.
- DFS-BP can use data from after the test date. For the test split, the database is loaded with all records, including those after the test timestamp (
src/tabicl/eval/relbench_official_eval.py:527). The rest of the pipeline removes these records by time, but the DFS-BP step that links rows of the task table does not (src/tabicl/eval/binary_pseudo_label.py:261-262). In churn tasks, the future activity of a user is exactly what the label measures, so the DFS-BP features can carry information about the test labels. This contradicts Section 3.3, which states that timestamps are used to prevent such leakage. - All RelBench runs use a feature that the paper does not mention. The function
dense_label_history(src/tabicl/eval/relbench_official_eval.py:281-320, enabled by default) adds the past labels of each entity, for example, whether a user churned in earlier quarters. It uses only labels known at prediction time, so it is valid in principle. However, it is a strong feature, and it is unclear whether the baselines get it. - The repository has no code for any baseline, so the baseline numbers seem to come from other papers. Meanwhile, HGraphFM is evaluated on a random subset of at most 10,000 test rows (
scripts/run_extra_tasks.sh:11,47). As a result, HGraphFM and the baselines may be evaluated on different test rows. - On ACM, DBLP and IMDB, the training script computes test scores every 5000 steps (
scripts/train_multitab_mixed_stage1.sh:83-89), and ACM and DBLP have no validation split. Moreover, another script compares stage-1 checkpoints from 50k to 142k steps by test macro-F1 (scripts/eval_hgb_fullctx_ckpts.py:22-27,132-143), while the paper reports 200k training steps. The paper does not say which checkpoint produced Table 1 or how it was chosen. - Some settings differ per task, although the RelBench pipeline has no validation split to choose them. For example, the DFS-BP sampling size is 50 for some tasks and 20 for others (
scripts/run_extra_tasks.sh:31-43), and the labeled examples are sampled uniformly for rel-hm user-churn but with equal numbers per class for all other tasks (scripts/run_extra_tasks.sh:21,110-112).
There are also smaller differences:
- The model does not use timestamps (
src/tabicl/_model/learning.py:19), although Figure 2 shows a "temporal-aware" graph component. - Edge types and directions are dropped (
src/tabicl/eval/hgb_eval.py:356,src/tabicl/_model/gnn/graph_attn.py:15-27). - The stage-2 script can train on either RDB-PFN or PluRel data, and the scripts disagree on which is the default (
scripts/train_multitab_mixed_stage2.sh:24,scripts/rdbpfn_pool_pregen.sh:78). - The RelBench seed script uses three seeds instead of five (
scripts/run_clf_seeds.sh:5). - The code can also label test rows by thresholding a DFS-BP feature and add them to the labeled examples (
src/tabicl/eval/binary_pseudo_label.py:446), yet the released scripts do not enable this option, and the paper does not mention it.
Therefore, I find it difficult to rely on the RelBench results, and on the DFS-BP gains in particular, until the authors clarify these issues.
Questions for authors
Clarifications:
- How were the homogeneous graph foundation models applied to ACM, DBLP and IMDB? Were all node types merged into one graph, and were edge types dropped?
- How were the ten reported RelBench tasks chosen? The evaluation script also configures rel-amazon item-churn, rel-stack user-badge, rel-amazon user-ltv, rel-amazon item-ltv and rel-hm item-sales (
scripts/run_extra_tasks.sh:65-81). Were these tasks evaluated, and if so, why are they not reported? - How were the DFS-BP hyperparameters chosen, for example, selected features, filter order , and hops (
scripts/run_extra_tasks.sh:26-43)? Is DFS-BP used in the finetuning experiments?
Experiments that would affect the assessment:
- Results of one checkpoint in both settings: stage 2 on ACM, DBLP and IMDB, and stage 1 on RelBench.
- Ablations of the main components: training on the relational prior only (without stage 1), and removing the graph attention layers.
- DFS-BP applied to RDB-PFN and TabPFN, to check whether it helps these models as much as it helps HGraphFM.