acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Progress in Graph Foundation Models: A Comprehensive Benchmark and New Insights

Abstract

Graph foundation models (GFM) aim to acquire transferable knowledge by pre-training on diverse graphs, which can be adapted to various downstream tasks. However, domain shift in graphs is inherently two-dimensional: graphs differ not only in what they describe (topic domains) but also in how they are represented (format domains). Most existing GFM benchmarks vary only topic domains, thereby obscuring how knowledge transfers across both dimensions. We present a new benchmark that jointly evaluates topic and format gaps across the full GFM pipeline, including multi-domain pre-training and few-shot downstream adaptation, and provides a timely evaluation of recent GFMs in the rapidly evolving landscape. Our protocol enables controlled assessment in four settings, covering seen and unseen datasets as well as cross-topic and cross-format evaluation, designed to disentangle topic generalization from robustness to format shifts. We conduct extensive evaluations of eight state-of-the-art GFMs on 33 datasets spanning seven topic domains and six format domains, surfacing new empirical results and insights for future research. Codes and data are available at https://anonymous.4open.science/r/GFMBenchmark-7EA0.

Primary area:
datasets and benchmarks
Keywords:
Graph foundation models, benchmarking, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_738POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The author's propose a benchmarking protocol for evaluating the current progress of Graph Foundation Models within the multi-domain pre-training to cross-domain few-shot adaptation problem. Specifically, the author's introduce their benchmark to address the gap that has formed from GFMs only being benchmarked on single dimensions of domain diversity. In this capacity, the author's devise that GFM benchmarking can be uplifted by treating graph heterogeneity as separate topic and format domains, which exist along varying semantic and structural axes. This is then decomposed into four pre-train then adapt settings which the author's evaluated extensively. This involves experiments across 33 different graph datasets and numerous graph foundation models.

Critical strengths

The scope of the benchmarking protocol included within this work is ambitious and extensive, the potential for building a unified protocol for GFM evaluation is a promising contribution for the maturation of GFM technology on it's own. In particular, the four standardized yet separate problem setting formulations provide a solid point of comparison that makes it possible for practitioners and researchers to define a ceiling from which to estimate progress of their models versus other GFMs. This standardization is helpful in it's own right and likely to prevent incoming researchers and practitioners from falling into pitfalls. Additionally, what is particularly surprising to me is the relatively strong results of baseline methods like GCN or GraphPrompt out-performing GFMs in one-shot or unseen scenarios, indicating capacity for GFM pre-training to boost baseline performance.

Critical weaknesses

One critical weakness for this review is that the proposed work represents a benchmarking protocol of which existing GFM works already cover. In essence, the current GFM paradigm is to pre-train and then apply few-/one-shot adaptation on target (usually unseen) domains or datasets. As presented, this work's contribution is centered exclusively around a benchmarking protocol that already exists in a significant portion of how it is presented in the paper.

The second critical weakness is on the strength of framing the results for GFMs relative to specific pre-training methods, which may indicate an issue with the training pipeline in general. For example, Lines 293-296 describe how "On the dynamic graph Reddit, several GFMs (MDGFM, SAMGPT, and GFT) attain better or comparable performance than pre-training methods specifically designed for dynamic graphs (DDGCL and TGN)". Given the best GFM from Table 3 on Reddit, with a mean Accuracy of 55.73±30.46 relative to DDGCL (54.12±18.92) across five random seeds. Even for GFMs on the one-shot setting the reported level of standard deviation spread is quite drastic [1]. Additionally, the work claims a comprehensive scope but misses citations entirely on works such as [1].

[1] Yuan H, Sun Q, Shi J, Fu X, Hooi B, Li J, Yu PS. How much can transfer? BRIDGE: Bounded multi-domain graph foundation model with generalization guarantees. InForty-second International Conference on Machine Learning 2025 Jul 13.

Questions for authors
  1. This is to address my first concern around the pre-train then adapt paradigm already existing for GFMs. How is it that analysis from this proposed benchmarking protocol can extend knowledge beyond what is already proposed for existing GFMs? Are there insights that are unique only if GFM benchmarking is framed in the proposed topic and format domains? I ask this question largely because GFM research on the pre-train then adapt paradigm implicitly follows (and often explicitly [1]) the topic and format domain structure. That, and my interpretation is that the central justification of this work is explicit evaluation of topic and format domains, yet current analysis is heavily-centered around GFM performance and less on what the separation between topic and format domains mean, Figure 5 and Appendix C.4 are certainly a good-start, but still seem shallow relative to the breadth of experimentation within this work.

  2. How is it that the datasets within this work reach the extent of separate "topological patterns, temporal dependencies, and attribute or relational schemas, which can fundamentally affect both pre-training and adaptation"? In particular, why do we see such large standard deviation spreads for results within these datasets? My thought would be that it partially due to hyperparameter count but GraphCLIP with 12 layers and the largest dimensions displays a similar spread relative other models.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 738P (1/2)

W1 & Q1: Contribution and new insights

Our contribution is not this pipeline itself, but a two-dimensional view of graph domains, distinguishing topic semantics from graph formats, and unified evaluation settings that characterize cross-topic and cross-format generalization on both seen and unseen downstream datasets. As discussed in Sec. 2, existing benchmarks typically vary topic domains while fixing or tightly constraining graph formats, or focus on a narrow task or application area. Our benchmark explicitly accounts for both topic and format domains in pre-training and downstream adaptation, while covering representative GFMs across major modeling regimes. This enables systematic analysis of different generalization behaviors, rather than simply comparing performance on more datasets. In particular, Sec. 5.2 discusses the following insights:

(1) Semantic topic proximity is not a sufficient indicator of transferability. Setting III, together with Fig. 5 and Appendix C.4, shows why topic-matched pre-training need not provide the strongest specialization benefit: datasets within the same topic can differ substantially in structural and textual properties, while molecular downstream tasks additionally differ in granularity from predominantly node-level pre-training objectives. The implication is that useful domain-specific priors cannot be identified through topic labels alone; pre-training strategies must also account for dataset-level diversity and task granularity.

(2) Different format gaps call for different forms of format awareness. Setting IV reveals a distinction between heterophilic/text-attributed targets, which benefit more from broad multi-format pre-training, and heterogeneous/relational/dynamic targets, where broader format mixing yields limited gains over base-format pre-training (Fig. 2(b)). As discussed in Sec. 5.2, this suggests two complementary design needs: exposure to meaningful variation in heterophily and textual signals, and explicit type-, relation-, and time-aware mechanisms where exposure alone remains insufficient. Thus, format diversity should not be treated as a single interchangeable source of training data.

(3) Format awareness also depends on the interaction between the backbone and the pre-training mechanism. Table 6 shows that the same switch to FAGCN substantially improves some GFMs but not others. This indicates that a backbone suited to the target structure is not, by itself, a general solution to format transfer. Format-aware GFM design must consider how structural inductive biases interact with pre-training objectives and learned representations.

Regarding BRIDGE [1] (Yuan et al., ICML 2025), thank you for identifying the missing reference; we will cite and discuss it. BRIDGE explicitly distinguishes cross-dataset transfer within semantic domains from transfer to an unseen semantic domain. We acknowledge this existing treatment of topic-domain differences. However, its evaluation uses six datasets from three semantic domains to assess its proposed model and generalization mechanisms. It does not systematically compare single-topic versus multi-topic pre-training and base-format versus multi-format pre-training across multiple GFMs. Heterogeneous and dynamic graphs are discussed as challenges in its limitations. Our benchmark complements BRIDGE by providing these comparisons and empirical evidence about the generalization behaviors and limitations of current GFMs.

AuthorsOpenReview ↗

Response to Reviewer 738P (2/2)

W2 & Q2: Performance variation and format-specific evaluation

The reported standard deviations are not calculated over five random seeds alone. As specified in Appendix B.2 and the Reproducibility Statement, we construct 50 few-shot episodes and evaluate each with five random seeds, yielding 250 runs per dataset and shot setting. The standard deviations include variation across support sets as well as random seeds. In one-shot adaptation, changing the single labeled instance per class changes the supervision available to the model, contributing to substantial variation across episodes regardless of model size.

The results remain informative because the mean summarizes performance across repeated few-shot episodes, while the standard deviation describes the variability of individual adaptation outcomes. A large standard deviation does not itself make the mean comparison meaningless; it additionally reveals sensitivity to limited supervision, which is relevant to evaluating GFM reliability. For Reddit, the appropriate conclusion is that the evaluated GFMs attain mean performance comparable to the dynamic-specific baselines under our protocol. Our broader analysis also examines behavior across datasets, pre-training settings, and both one- and five-shot adaptation, rather than relying on this individual comparison.

Regarding format differences, Table 1 identifies the dataset properties, and Appendix B.1 specifies how they enter the evaluation. Heterophilic graphs retain their connectivity and node features; heterogeneous graphs retain connections across node types and type metadata in the unified object; relational graphs retain edge direction and relation metadata; and text-attributed graphs provide textual fields. Each method uses the attributes supported by its implementation, so the evaluation captures both its transfer performance and its ability to exploit the available format-specific information. For dynamic graphs, we aggregate historical interactions into a static snapshot and use each node’s latest observed label. This retains accumulated interaction structure but does not test explicit temporal-dependency modeling. As already discussed in Sec. 5.1, this snapshot treatment may help explain why static GFMs remain competitive on Reddit.

Reviewer_738POpenReview ↗

Rebuttal Response

Thank you to the authors for their clarification on the performance variation and the format-specific evaluation, this has addressed my concerns on the presented performance within the submitted manuscript. I appreciate the inclusion of the cited Yuan et. al paper and agree that this work can be treated as an extension to the BRIDGE work, of course on a significantly larger scope.

For my remaining concern, although I trust the insights provided within (1)-(3) as a response to my W1/Q1 questions, I am still unsure how it is grounded in specific measurements, especially given the scope of this benchmarking work. Are there specific metric values beyond downstream performance and overperformance which can support the insights made on: semantic topic proximity not indicating transferability, different format gaps requiring different awareness, and format awareness depending on interaction between the backbone and pre-training?

  1. Are there distance calculations which support the semantic topic proximity point? Do we see inconsistent gains for downstream performance within semantic domains relative to between semantic domains to justify this point?

  2. Is there some statistical value to indicate meaningful variation for heterophily and textual signals versus mechanism where exposure fails? The sheer number of viable mechanisms, even with the discussion in Section 5.2, keeps it difficult to ground interpretations of the downstream results in anything but model mechanisms instead of analysis that delineates what is 'meaningful variation'.

  3. What are the exact numbers that produce this gap between format-aware GFM, structural inductive bias, pre-training, and learned representations? Is there some correlation within the model's internal embedding where FAGCN damages existing representations in some models to degrade performance but successfully boost's performance in other models given alignment with that target model's pre-training strategy?

All three points are certainly interesting, I also understand that the current paper includes these insights. However, from an outside reader's perspective, all of the claimed insights remain difficult to substantiate without measured evidence to support said claims. I am not demanding comprehensive coverage of all three points on every tested model and dataset, but the current insights are abstruse enough that downstream performance should be supported by positive (and/or negative) examples. That way, claims on what is 'meaningful variation' or the gap across multiple separate GFM concepts at-once can be grounded in observations that are measurable via some form of: distance metric, statistical analysis, or further visualization to enable direct correlation. Figure 5 is certainly a good start, but the figure is so dense (and located in the Appendix) that it appears difficult to support the author's insights without direct metric comparison. I ground this reasoning behind my response in the philosophy that benchmarks should inspire further investigation into a given problem, if the authors can provide some evidence beyond performance to support the provided insights then my concern will be addressed.

Reviewer_Eeo3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper compares the performance of a number of recent graph foundation models (GFMs) across a range of datasets under different settings. The key observation of the paper is that graph dataset domains can differ not only in topic but also in graph format (e.g., homogeneous vs heterogeneous graphs, static vs temporal graphs). Thus, the paper varies dataset domains both across topics and across formats. The paper considers 4 settings: (1) pretraining on different topics and formats and adapting to unseen datasets, (2) pretraining on different topics and formats and adapting to seen datasets, (3) pretraining on a single topic and adapting to unseen topics, (4) pretraining on a single format and adapting to new formats. For each setting, the paper provides a number of insights.

Critical strengths

Providing a unified fair reevaluation of GFMs is important for the field, as a lot of GFMs have recently been proposed and most of them are originally evaluated under vastly different settings.

Critical weaknesses

I believe the paper's main premise of conducting experiments on graphs with different formats is fundamentally flawed. The reason for this is that the vast majority of the models considered in this paper were originally designed to work only with one graph format (specifically, homogeneous static graphs). They were not designed to be applied to heterogeneous or temporal graphs and do not originally support them. The paper circumvents this by designing a unified representation format for graphs with different original formats and retraining all the considered GFMs on graphs represented in this format. As a result of these, the comparison provided in the paper is NOT a comparison of the original GFMs that the paper claims to compare. It instead compares models trained by the authors from scratch (it is not clear if pretraining was performed as well as in the case of the original models) on datasets in a new author-designed format that was not used by the original models (it is not clear that this format works equally well for the different considered GFMs). Thus, the comparison is not of the original GFMs, and it is strongly confounded by specific decisions about dataset formats used. As a consequence, the conclusions made cannot be relied upon to make inferences about the original models. Further, the decision to retrain all the GFMs excludes the recent PFN-based GFMs (G2T [1], TAG [2], GraphPFN [3], NodePFN [4]) which are expensive to pretrain and generally follow a very different pretrainign strategy, but have been shown to achieve the best results among currently available GFMs [5]. The paper mentions [5] but claims that this work focuses specifically on PFNs - this is not actually the case, since [5] reevaluates a range of different GFMs including many of those used in this paper, it just comes to the conclusion that PFNs a by far the best current GFMs. Thus, omitting PFNs from the present comparison is a critical weakness.

Further, the dataset choice used for the experiments is poor. Many of these datasets have been criticised many times for being unrealistic, buggy, and unreliable. Most notably: cora, pubmed, ogbn-arxiv contain lots of mislabeled nodes [6]; chameleon contains duplicated nodes which leads to train-test data leakage [7]; texas, cornell, wisconsin are extremely small and do not have enough class representation (texas even has a class containing a single node) [7], ogbn-proteins (a protein-protein interaction network) does not have node features despite proteins having lots of ways to extract them. In general, there has recently been a lot of concern about graph machine learning papers using flawed, unreliable, and unrealistic datasets [8, 9] and better benchmarks have been proposed [9, 10]. Datasets from these benchmarks should be used instead of outdated and unreliable datasets used in this paper (which make it difficult to trust the conclusions obtained from experiments on such datasets).

Some smaller issues:

Why are homophilous and heterophilous graphs considered different graph formats? The property of homophily describes the relation between node labels and edges in the graph, it has nothing to do with the graph format. Simply relabeling nodes in the graph can change its homophily without changing anything about its format.

Table 1 provides wrong references for almost all the considered datasets. Examples: cora and pubmed datasets have been actively used for more than 20 years, they definitely were not introduced in 2024, texas, cornell and wisconsin were prepared by the Geom-GCN paper (and raw data was available long before that), they were definitely not introduced in 2025, photo and computers datasets are from [11], not from the referenced papers, PCBA and HIV are part of the MoleculeNet benchmark (and were later made part of the OGB benchmark) and are based on even earlier datasets, again not from the referenced paper from 2024.

Why are molecular graph datasets like PCBA and HIV considered homogeneous when each node (atom) and edge (chemical bond) in them actually has a discrete type?

Why are datasets from the proteins topic (Proteins, Enzymes, and ogbn-proteins) considered to have the same format, when Proteins and Enzymes represent a single atom as a node, while ogbn-proteins represents a whole protein as a node? These are very different objects.

Overall, these issues highlight the problem of poor dataset selection.

[1] Turning Tabular Foundation Models into Graph Foundation Models (2025)

[2] Bringing Graphs to the Table: Zero-shot Node Classification via Tabular Foundation Models (2025)

[3] GraphPFN: A Prior-Data Fitted Graph Foundation Model (ICML 2026)

[4] Learning Posterior Predictive Distributions for Node Classification from Synthetic Graph Priors (ICLR 2026)

[5] A Fair Evaluation of Graph Foundation Models for Node Property Prediction (2026)

[6] GraphCleaner: Detecting Mislabelled Samples in Popular Graph Learning Benchmarks (ICML 2023)

[7] A critical look at the evaluation of GNNs under heterophily: Are we really making progress? (ICLR 2023)

[8] Graph Learning Will Lose Relevance Due To Poor Benchmarks (ICML 2025)

[9] GraphLand: Evaluating Graph Machine Learning Models on Diverse Industrial Data (NeurIPS 2025)

[10] GraphBench: Next-generation graph learning benchmarking (2025)

[11] Pitfalls of Graph Neural Network Evaluation (2018)

Questions for authors

Can the authors address the issues raised in weaknesses?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Eeo3 (1/2)

The ICLR review form asks reviewers to identify one or two critical weaknesses that support rejection and explain why each is consequential to the overall assessment. The review raises several distinct concerns without numbering them individually. To facilitate cross-reference, we assign labels W1–W7 in the order in which the concerns appear.

W1: Cross-format evaluation of existing GFMs

First, we respectfully disagree that evaluating GFMs across graph formats is fundamentally flawed. A central goal of graph foundation models is to handle graphs across as many topic and format domains as possible. The ability to adapt a single model to diverse domains and tasks is a major reason for the success of large language models as foundation models. Our benchmark evaluates how far current GFMs have progressed toward comparable breadth on graph data, where both what a graph represents and how it is structured can vary. The limited native format support highlighted in the review makes cross-format evaluation particularly important. It lets us measure how far existing methods can transfer when given inputs their implementations can process.

Second, the methods are the objects of comparison, and the pre-training corpus is an experimental condition. The evaluated papers introduce GFM methods through their model designs, pre-training objectives, and adaptation procedures. We retain these defining components in our main comparisons and train each method on the same pre-training corpus within each benchmark setting. Appendix B documents the implementation adjustments, and Table 6 reports deliberate backbone substitutions separately.

Using a common corpus makes the methods directly comparable under the same source data conditions. Using each method’s released checkpoint would also introduce differences in the pre-training data, making it harder to interpret differences in transfer performance. Settings III and IV go one step further by testing the same method across different corpus conditions. If a method’s transfer performance depends strongly on a particular corpus, that dependence is itself relevant to assessing its generality. Our results characterize the existing GFM methods under the documented benchmark conditions.

W2: Scope of the comparison with PFN-based GFMs

The cited PFN-based GFMs use a different pre-training paradigm. They construct pre-training tasks from synthetic graph priors, and the design of these priors is central to their approach. Our experimental settings vary the topic and format composition of real graph corpora to examine downstream transfer. Examining how its performance changes with pre-training data would involve varying its synthetic priors, which is a separate experimental question.

AuthorsOpenReview ↗

Response to Reviewer Eeo3 (2/2)

W3: Dataset selection and reliability of the findings

We disagree with the blanket assessment that the benchmark uses a poor choice of datasets. The suitability of a dataset depends on the comparison it enables. The reliability of a result depends on whether a specific dataset defect affects how the data are used in that experiment. Our benchmark selects source and target graphs to support comparisons across topic and format domains, and Table 1 specifies the role of each dataset. Applying these criteria to the examples raised in the review leads to different conclusions:

  • ogbn-arxiv contributes a large citation graph to the pre-training corpus. We use it exclusively for unlabeled pre-training, so the cited node-label errors affect neither the training supervision nor the evaluation metrics in our experiments.
  • Texas contributes a heterophilic web graph to the pre-training corpus. It is used exclusively for unlabeled pre-training, so the single-node class cited in the review places no constraint on our K-shot episodes.
  • Actor provides a separate heterophilic downstream target for the backbone comparison in Table 6. On Actor alone, switching to FAGCN improves transfer for some GFMs and worsens it for others. Thus, even if the duplicate-node issue affects the Chameleon results, the finding that a heterophily-aware backbone does not uniformly improve transfer is also supported by Actor.

GraphLand and GraphBench are useful candidates for future extensions. The current dataset suite already supports the comparisons across topic domains, graph formats, and prediction tasks that motivate our benchmark. The availability of additional datasets does not, by itself, constitute a weakness of the current benchmark.

W4: Homophily and heterophily as a format axis

We use “graph format” to cover structural regimes that affect how models process graphs. For node classification, homophily and heterophily describe how labels relate to connectivity. In homophilous graphs, adjacent nodes tend to share a label. In heterophilous graphs, adjacent nodes more often have different labels. This distinction has direct consequences for model design. Aggregating neighboring features can reinforce class-relevant signals under homophily, while under heterophily it can mix signals from different classes. This motivates different aggregation strategies, including the heterophily-aware backbone evaluated in Table 6. Relabeling can change homophily without changing the graph schema, but it also changes how informative neighboring nodes are for predicting the target label. This is why we include homophily as a task-dependent structural regime in our format evaluation.

W5: Dataset references in Table 1

The citations in Table 1 identify the sources of the dataset versions used in our experiments. For each dataset, we used the version provided by the cited work.

W6: Molecular graph representation

In PCBA and HIV, we represent all atoms in a single node set and all bonds in a single edge set, with atom and bond categories encoded as attributes. These categories can also be used to define an explicitly typed graph schema.

W7: Format classification of protein graphs

Table 1 classifies graph formats according to the representation criteria defined in the paper. The difference highlighted in the review concerns the biological meaning of the nodes and does not contradict their classification under those criteria.

I appreciate the rebuttal, however, I do not think it substantially addresses the issues raised in the review. Thus, I maintain my score.

Further, regarding the reply about homophily-heterophily. The authors claim that homophilous and heterophilous datasets might require different model designs because "aggregating neighboring features can reinforce class-relevant signals under homophily". I would like to note that homophily describes a relationship between node labels and the graph connectivity, not between node features and the graph connectivity. Node features can contain information way beyond node labels that can be useful under both homophily and heterophily. It has, in fact, been repeatedly shown that standard message passing GNNs perform perfectly fine under heterophily, outperforming heterophily-specific models [1-6], and thus different model designs for these settings are not neccessary.

[1] Is homophily a necessity for graph neural networks? (ICLR 2022)

[2] Revisiting heterophily for graph neural networks (NeurIPS 2022)

[3] A critical look at the evaluation of GNNs under heterophily: Are we really making progress? (ICLR 2023)

[4] Characterizing Graph Datasets for Node Classification: Homophily–Heterophily Dichotomy and Beyond (NeurIPS 2023)

[5] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification (NeurIPS 2024)

[6] Oversmoothing, Oversquashing, Heterophily, Long-Range, and more: Demystifying Common Beliefs in Graph Machine Learning (ICLR 2026)

Reviewer_brZWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitation that existing benchmarks for graph foundation models (GFMs) fail to effectively characterize the dual domain shift of "topic" and "format" (structure) in graph data, and proposes a new benchmark that covers the complete pipeline of pretraining and few-shot downstream adaptation. The benchmark systematically evaluates eight state-of-the-art GFMs on 33 datasets under four controllable settings. Experimental results demonstrate that although current GFMs possess a certain degree of cross-domain transfer capability, their performance remains uneven, with notable deficiencies persisting in heterophily and textual formats. This benchmark provides potential directions for future model design.

Critical strengths

It is the first to explicitly decompose domain shift in graph data into two dimensions, namely "topic" and "format," and to construct a benchmark covering the complete pipeline of pretraining and few-shot adaptation.

Critical weaknesses

1.The claims regarding Topic Shift and Format Shift lack sufficiently rigorous experimental support. The different experimental settings not only alter the topic or format but also simultaneously change the scale, composition, and structural distribution of the pretraining data, making it difficult to attribute performance differences unambiguously to specific transfer factors.

2.The flattening of heterogeneous graphs and the conversion of dynamic graphs into static snapshots may attenuate key characteristics of the original graph formats, thereby compromising the validity of the cross-format generalization evaluation.

Questions for authors

1.How does the benchmark disentangle topic shifts from format shifts when the pre-training datasets used in Settings III and IV also differ in graph size, number of datasets, structural properties, and topic composition?

2.Since heterogeneous graphs are flattened and dynamic graphs are converted into static snapshots, to what extent does the benchmark evaluate genuine cross-format generalization rather than adaptation to simplified homogeneous and static representations?

3.Is the performance difference between Sets I and IV primarily due to format diversity, or can it be attributed to differences in the scale and topic composition of the pre-training data?

2 comments
AuthorsOpenReview ↗

Response to Reviewer brZW

W1 & Q1 & Q3: Topic/format comparisons

Thank you for your questions. Settings III and IV evaluate pre-training corpus selection strategies defined by topic and format. Setting III asks whether selecting source graphs from the target topic provides better transfer than selecting other topics or using a broad corpus. Setting IV asks whether restricting pre-training to the base format is sufficient for transfer to other formats. Different corpus sizes and compositions naturally arise from these choices; they do not invalidate comparisons of the resulting strategies on the same downstream targets.

Our protocol makes the source topics and formats explicit and uses common downstream datasets and adaptation procedures to evaluate their transfer performance. The conclusions concern these evaluated configurations: topic-matched pre-training does not consistently yield the strongest transfer, and broad pre-training does not uniformly improve performance across target formats. Neither conclusion requires topic or format to be the sole determinant of performance.

We appreciate your suggestion to examine corpus size, composition, and structural statistics more closely. These factors can help explain why a particular configuration performs better. Our existing analysis in Appendix C.4 already highlights that topic labels alone do not fully characterize dataset similarity. We will clarify that the settings separate topic- and format-oriented evaluation questions, while explanations of individual performance differences may involve multiple corpus properties.

W2 & Q2: Format handling

There is a trade-off between preserving format-specific information and enabling a unified comparison of current GFMs. Most evaluated methods do not natively model heterogeneous schemas or temporal event streams. Our preprocessing makes these datasets accessible through the representations supported by each method, allowing us to evaluate transfer under their existing modeling constraints.

For heterogeneous graphs, homogenization does not remove all heterogeneity: node and edge types are retained in the unified data object, alongside available relation descriptions, and each method uses the attributes supported by its implementation (Appendix B.1). For dynamic graphs, we aggregate historical interactions into a static snapshot and use each node’s latest observed label. This retains accumulated interaction structure, although it does not preserve event order as a modeling input.

We acknowledge that these adaptations can weaken format-specific signals, including explicit type-dependent semantics and temporal dependencies. However, this trade-off also reflects an important capability boundary of current GFMs: broader pre-training does not automatically equip a model to explicitly process types or time. Evaluating these models through their supported representations provides a practical assessment of their applicability across graph formats.

Our results should therefore be interpreted as cross-format transfer under the practical constraints of the evaluated GFMs. This scope does not invalidate the comparisons, but determines what they establish. In particular, the substantial gap to HeCo on ACM (Table 3) highlights heterogeneous graphs as a remaining challenge and motivates stronger format-aware GFM designs. We will clarify this scope and explicitly distinguish snapshot-based classification from temporal modeling.

Thank you for the detailed response. I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.