acceptodds
Under review as a conference paper at ICLR 2027

Heterogeneous Graph Prior-Data Fitted Foundation Model

Abstract

Heterogeneous graphs are ubiquitous in real-world applications, yet their schema heterogeneity, relational diversity, and complex attribute distributions make it highly challenging to build foundation models with cross-task generalization. Existing graph foundation models are mostly designed for a single setting, such as homogeneous graphs or relational data, or rely on large-scale real-world pretraining data, and therefore face data scarcity and limited task adaptability in heterogeneous scenarios. To address this, we propose HGraphFM, a foundation model framework for heterogeneous graph node-level tasks, which enables unified modeling of diverse heterogeneous graph tasks through synthetic-data-driven pretraining. In a nutshell, we first design a general heterogeneous graph prior, which, together with the relational-data prior, provides large-scale, diverse pretraining data for the foundation model. HGraphFM extends a single-table model by introducing graph attention layers, which in turn enhance in-context learning attention, enabling a transition from processing single-table data to handling heterogeneous data. It combines both in-context learning and structure learning capabilities, and can jointly model heterogeneous attributes and relations that may be subject to temporal constraints. Furthermore, we introduce a feature augmentation mechanism, termed Deep Feature Synthesis Based on Graph Band-Pass Filtering (DFS-BP), which effectively alleviates insufficient context utilization and the need for manual feature selection. The mechanism, reminiscent of “lazy learning,” produces synthesized features that directly match or even surpass the performance of some existing pretrained models in classification tasks, and provides substantial gains in model generalization. After a two-stage training procedure based on the two types of priors, experiments on multiple real-world heterogeneous graph datasets show that our method outperforms existing approaches, including homogeneous-graph and relational foundation models, in both few-shot and fine-tuned settings, validating the effectiveness of the proposed heterogeneous graph foundation model paradigm.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Foundation Model, Heterogeneous Graph, Prior-Data Fitted Networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8jUbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes HGraphFM, a prior-data fitted network for node classification and regression on heterogeneous graphs and relational databases. It has three components.

  1. A synthetic heterogeneous graph prior. It extends the prior of GraphPFN: a homogeneous graph is generated, its nodes are split into tables, and each table gets new features and labels.
  2. A model that adds graph attention layers to TabICL. It is trained in two stages: first on the new prior, then on the relational prior of RDB-PFN.
  3. DFS-BP, a feature augmentation step for relational data at test time. It builds a graph over the rows of the task table, filters the features over this graph, and adds the filtered columns that score best on the training labels.

The evaluation covers node classification with 10, 20 and 40 labeled examples on ACM, DBLP and IMDB (stage-1 model), and classification and regression with 1024 labeled examples on ten RelBench tasks (stage-2 model). HGraphFM is best or close to best in most settings, and DFS-BP gives further gains on RelBench classification.

Critical strengths

The paper has a relevant goal: one model that is pretrained only on synthetic data and handles heterogeneous graphs, including relational databases. On RelBench, without DFS-BP, it reaches an average AUROC of 0.694, against 0.680 for RDB-PFN, the strongest PFN baseline. However, the weaknesses below limit what we can conclude from it.

Critical weaknesses

1. The paper is hard to follow, and key parts of the method are missing from the main text.

Much of the main text consists of general statements that do not explain what the method does, such as "a unified architecture with both in-context learning and structure learning capabilities, enabling unified modeling and fast adaptation across diverse heterogeneous graph tasks" (lines 68-69). At the same time, important details are missing:

  • Section 3.1 describes the prior three times (as a list, in Figure 1, and as a paragraph), each with its own notation. For example, the number of tables is tt in the list and TT in the paragraph, and LL denotes both the latent matrix and the number of layers. Moreover, the details of the graph generation, the graph size and the number of tables are given only in Appendix D and the GraphPFN paper.
  • The architecture is described only by a short paragraph in Section 3.2 and by Figure 2. Neither explains how graph attention works across tables with different columns, whether the model uses edge types, or how the subgraph for each query is sampled.
  • The paper never explains why its prior should transfer to the target data. The prior has a single type of edges, both within and across tables, and nodes are assigned to tables at random. In contrast, heterogeneous graphs have several edge types, almost all between different node types, while relational databases have only foreign-key edges between tables.
  • Homogeneous graph foundation models are applied to heterogeneous graphs using only PCA and zero-padding of node features (Appendix C). However, the paper does not say which graph these models receive.

2. The paper combines existing components without analysis, and the evidence is limited.

The stage-1 prior extends GraphPFN, the stage-2 prior comes from RDB-PFN, the model is TabICL with graph attention, and DFS-BP combines deep feature synthesis with existing spectral filters. Combining existing parts can be a valid contribution, but then the paper needs to show which parts matter and why. Here, the only ablation turns DFS-BP on and off.

  • No ablation tests the stage-1 prior, the graph attention layers or the two-stage training. Therefore, the claim that stage-1 pretraining helps relational tasks has no support.
  • The model is called unified, but each stage is evaluated only in its own setting: stage 1 on heterogeneous graphs, stage 2 on relational data.
  • DFS-BP confounds the comparison with the baselines. It selects features by their score against the labels of the full training set. In contrast, the baselines see only 1024 labeled rows, and DFS-BP is never applied to them. Moreover, Figure 4 shows that on two Avito tasks, a single selected feature already scores close to or above RDB-PFN. Therefore, the reported gain may come from the label information in DFS-BP alone.
  • Besides, DFS-BP is not connected to the main idea of the paper, which is pretraining on a synthetic prior. It runs outside the model, only on relational data, and only at test time. Thus, it looks like one more trick added to raise the numbers.
  • Finally, the heterogeneous graph evaluation uses three small, old datasets. The paper does not explain how the ten reported RelBench tasks were chosen, and it reports no standard deviations, although many gains are small.

3. The released code differs from the paper in ways that affect the reported results.

  • DFS-BP can use data from after the test date. For the test split, the database is loaded with all records, including those after the test timestamp (src/tabicl/eval/relbench_official_eval.py:527). The rest of the pipeline removes these records by time, but the DFS-BP step that links rows of the task table does not (src/tabicl/eval/binary_pseudo_label.py:261-262). In churn tasks, the future activity of a user is exactly what the label measures, so the DFS-BP features can carry information about the test labels. This contradicts Section 3.3, which states that timestamps are used to prevent such leakage.
  • All RelBench runs use a feature that the paper does not mention. The function dense_label_history (src/tabicl/eval/relbench_official_eval.py:281-320, enabled by default) adds the past labels of each entity, for example, whether a user churned in earlier quarters. It uses only labels known at prediction time, so it is valid in principle. However, it is a strong feature, and it is unclear whether the baselines get it.
  • The repository has no code for any baseline, so the baseline numbers seem to come from other papers. Meanwhile, HGraphFM is evaluated on a random subset of at most 10,000 test rows (scripts/run_extra_tasks.sh:11,47). As a result, HGraphFM and the baselines may be evaluated on different test rows.
  • On ACM, DBLP and IMDB, the training script computes test scores every 5000 steps (scripts/train_multitab_mixed_stage1.sh:83-89), and ACM and DBLP have no validation split. Moreover, another script compares stage-1 checkpoints from 50k to 142k steps by test macro-F1 (scripts/eval_hgb_fullctx_ckpts.py:22-27,132-143), while the paper reports 200k training steps. The paper does not say which checkpoint produced Table 1 or how it was chosen.
  • Some settings differ per task, although the RelBench pipeline has no validation split to choose them. For example, the DFS-BP sampling size is 50 for some tasks and 20 for others (scripts/run_extra_tasks.sh:31-43), and the labeled examples are sampled uniformly for rel-hm user-churn but with equal numbers per class for all other tasks (scripts/run_extra_tasks.sh:21,110-112).

There are also smaller differences:

  • The model does not use timestamps (src/tabicl/_model/learning.py:19), although Figure 2 shows a "temporal-aware" graph component.
  • Edge types and directions are dropped (src/tabicl/eval/hgb_eval.py:356, src/tabicl/_model/gnn/graph_attn.py:15-27).
  • The stage-2 script can train on either RDB-PFN or PluRel data, and the scripts disagree on which is the default (scripts/train_multitab_mixed_stage2.sh:24, scripts/rdbpfn_pool_pregen.sh:78).
  • The RelBench seed script uses three seeds instead of five (scripts/run_clf_seeds.sh:5).
  • The code can also label test rows by thresholding a DFS-BP feature and add them to the labeled examples (src/tabicl/eval/binary_pseudo_label.py:446), yet the released scripts do not enable this option, and the paper does not mention it.

Therefore, I find it difficult to rely on the RelBench results, and on the DFS-BP gains in particular, until the authors clarify these issues.

Questions for authors

Clarifications:

  1. How were the homogeneous graph foundation models applied to ACM, DBLP and IMDB? Were all node types merged into one graph, and were edge types dropped?
  2. How were the ten reported RelBench tasks chosen? The evaluation script also configures rel-amazon item-churn, rel-stack user-badge, rel-amazon user-ltv, rel-amazon item-ltv and rel-hm item-sales (scripts/run_extra_tasks.sh:65-81). Were these tasks evaluated, and if so, why are they not reported?
  3. How were the DFS-BP hyperparameters chosen, for example, K=30K = 30 selected features, filter order C=50C = 50, and h=6h = 6 hops (scripts/run_extra_tasks.sh:26-43)? Is DFS-BP used in the finetuning experiments?

Experiments that would affect the assessment:

  1. Results of one checkpoint in both settings: stage 2 on ACM, DBLP and IMDB, and stage 1 on RelBench.
  2. Ablations of the main components: training on the relational prior only (without stage 1), and removing the graph attention layers.
  3. DFS-BP applied to RDB-PFN and TabPFN, to check whether it helps these models as much as it helps HGraphFM.
Reviewer_pTgcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HGraphFM, a prior-data fitted foundation model for heterogeneous graph node prediction. It introduces a synthetic heterogeneous graph prior built upon GNN-SCM, extends TabICL with graph attention, and employs two-stage pretraining on heterogeneous graph and relational data priors. Additionally, it proposes DFS-BP for inference-time feature augmentation. Experiments on heterogeneous graph benchmarks and RelBench demonstrate competitive few-shot and fine-tuning performance.

Critical strengths
  • The paper addresses cross-dataset generalization for heterogeneous graphs, which combines heterogeneous feature processing, graph structure learning, and in-context learning to support both general heterogeneous graphs and relational data.
  • The method achieves competitive performance against graph and relational foundation models across classification and regression tasks.
Critical weaknesses
  • The overall framework closely follows existing PFN-based approaches, particularly GraphPFN and RDB-PFN. The heterogeneous prior primarily partitions homogeneous graphs and regenerates type-specific features, while the architecture extends TabICL with graph attention. The paper does not sufficiently establish what fundamentally new heterogeneous graph properties these modifications capture.
  • The paper lacks controlled experiments isolating the contributions of the proposed heterogeneous prior, graph attention, and two-stage pretraining. Consequently, it remains unclear which components account for the observed improvements over existing approaches.
  • The paper evaluates general heterogeneous graphs using the Stage-1 checkpoint and relational data using the Stage-2 checkpoint. It therefore does not establish whether a single final checkpoint can generalize effectively across both settings.
Questions for authors

Please refer to weaknesses.

Reviewer_VbSVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HGraphFM, a foundation model for node-level tasks on heterogeneous graphs, adopting the Prior-Data Fitted Networks (PFN) paradigm and using synthetic-data pretraining to circumvent the scarcity of real heterogeneous-graph data. The contributions are threefold: (i) a synthetic prior for general heterogeneous graphs — a progressive blueprint from single-table SCM → homogeneous graph → heterogeneous graph, partitioning the GNN-SCM latent L by Dirichlet and resampling features/labels with a novel MLP-SCM; (ii) introducing multi-head graph attention on top of TabICL to obtain a unified architecture (column attention + ICL row attention + graph attention) supporting joint structural and contextual learning; (iii) DFS-BP feature augmentation — at inference, building a graph within the task table, generating new features via a graph band-pass filter, and selecting top-k by training-set scoring. Pretraining proceeds in two stages (heterogeneous prior → relational prior). Experiments compare against classical HGNNs, graph foundation models, and relational foundation models on ACM/DBLP/IMDB (general heterogeneous graphs) and RelBench (relational data), claiming superiority under both few-shot and fine-tuning settings.

Critical strengths

The topic hits a real gap: heterogeneous-graph foundation models are indeed an under-explored setting within PFN/graph foundation models, with strong practical relevance. The experimental lineup is extensive: multiple families — HGNNs (GraphPFN/NodePFN/HiGPT…), tabular PFNs (TabICL/TabPFN…), relational FMs (Griffin/RDBPFN/PluRel…) — with 5 seeds.

Critical weaknesses

General heterogeneous graphs" (small networks like ACM/DBLP/IMDB) and "relational data" (RelBench) are essentially different; stage 1/stage 2 process the two data types separately, so the claimed "unified heterogeneous-graph foundation model" is more narrative than methodological unification. RelBench is a relational database, not a heterogeneous graph in the GNN sense. The paper only ablates DFS-BP and never proves the graph-attention layer does substantial work. General HG only reports stage-1; RelBench only reports stage-2; few-shot = 10/20/40 on small datasets yields weak generalization evidence for a foundation model.

Questions for authors

Can you add an ablation that removes only the graph-attention layer while keeping everything else, to prove the graph attention contributes on stage-1 (and stage-2)? How are "general heterogeneous graphs" and "relational data" truly unified methodologically? Is it just two-stage separate adaptation? Please clarify to avoid overclaiming. Can DFS-BP also be tested on ACM/DBLP/IMDB to prove it is equally effective on the heterogeneous-graph side?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.