- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The author's propose a benchmarking protocol for evaluating the current progress of Graph Foundation Models within the multi-domain pre-training to cross-domain few-shot adaptation problem. Specifically, the author's introduce their benchmark to address the gap that has formed from GFMs only being benchmarked on single dimensions of domain diversity. In this capacity, the author's devise that GFM benchmarking can be uplifted by treating graph heterogeneity as separate topic and format domains, which exist along varying semantic and structural axes. This is then decomposed into four pre-train then adapt settings which the author's evaluated extensively. This involves experiments across 33 different graph datasets and numerous graph foundation models.
Critical strengths
The scope of the benchmarking protocol included within this work is ambitious and extensive, the potential for building a unified protocol for GFM evaluation is a promising contribution for the maturation of GFM technology on it's own. In particular, the four standardized yet separate problem setting formulations provide a solid point of comparison that makes it possible for practitioners and researchers to define a ceiling from which to estimate progress of their models versus other GFMs. This standardization is helpful in it's own right and likely to prevent incoming researchers and practitioners from falling into pitfalls. Additionally, what is particularly surprising to me is the relatively strong results of baseline methods like GCN or GraphPrompt out-performing GFMs in one-shot or unseen scenarios, indicating capacity for GFM pre-training to boost baseline performance.
Critical weaknesses
One critical weakness for this review is that the proposed work represents a benchmarking protocol of which existing GFM works already cover. In essence, the current GFM paradigm is to pre-train and then apply few-/one-shot adaptation on target (usually unseen) domains or datasets. As presented, this work's contribution is centered exclusively around a benchmarking protocol that already exists in a significant portion of how it is presented in the paper.
The second critical weakness is on the strength of framing the results for GFMs relative to specific pre-training methods, which may indicate an issue with the training pipeline in general. For example, Lines 293-296 describe how "On the dynamic graph Reddit, several GFMs (MDGFM, SAMGPT, and GFT) attain better or comparable performance than pre-training methods specifically designed for dynamic graphs (DDGCL and TGN)". Given the best GFM from Table 3 on Reddit, with a mean Accuracy of 55.73±30.46 relative to DDGCL (54.12±18.92) across five random seeds. Even for GFMs on the one-shot setting the reported level of standard deviation spread is quite drastic [1]. Additionally, the work claims a comprehensive scope but misses citations entirely on works such as [1].
[1] Yuan H, Sun Q, Shi J, Fu X, Hooi B, Li J, Yu PS. How much can transfer? BRIDGE: Bounded multi-domain graph foundation model with generalization guarantees. InForty-second International Conference on Machine Learning 2025 Jul 13.
Questions for authors
-
This is to address my first concern around the pre-train then adapt paradigm already existing for GFMs. How is it that analysis from this proposed benchmarking protocol can extend knowledge beyond what is already proposed for existing GFMs? Are there insights that are unique only if GFM benchmarking is framed in the proposed topic and format domains? I ask this question largely because GFM research on the pre-train then adapt paradigm implicitly follows (and often explicitly [1]) the topic and format domain structure. That, and my interpretation is that the central justification of this work is explicit evaluation of topic and format domains, yet current analysis is heavily-centered around GFM performance and less on what the separation between topic and format domains mean, Figure 5 and Appendix C.4 are certainly a good-start, but still seem shallow relative to the breadth of experimentation within this work.
-
How is it that the datasets within this work reach the extent of separate "topological patterns, temporal dependencies, and attribute or relational schemas, which can fundamentally affect both pre-training and adaptation"? In particular, why do we see such large standard deviation spreads for results within these datasets? My thought would be that it partially due to hyperparameter count but GraphCLIP with 12 layers and the largest dimensions displays a similar spread relative other models.
7 comments
Response to Reviewer 738P (2/2)
W2 & Q2: Performance variation and format-specific evaluation
The reported standard deviations are not calculated over five random seeds alone. As specified in Appendix B.2 and the Reproducibility Statement, we construct 50 few-shot episodes and evaluate each with five random seeds, yielding 250 runs per dataset and shot setting. The standard deviations include variation across support sets as well as random seeds. In one-shot adaptation, changing the single labeled instance per class changes the supervision available to the model, contributing to substantial variation across episodes regardless of model size.
The results remain informative because the mean summarizes performance across repeated few-shot episodes, while the standard deviation describes the variability of individual adaptation outcomes. A large standard deviation does not itself make the mean comparison meaningless; it additionally reveals sensitivity to limited supervision, which is relevant to evaluating GFM reliability. For Reddit, the appropriate conclusion is that the evaluated GFMs attain mean performance comparable to the dynamic-specific baselines under our protocol. Our broader analysis also examines behavior across datasets, pre-training settings, and both one- and five-shot adaptation, rather than relying on this individual comparison.
Regarding format differences, Table 1 identifies the dataset properties, and Appendix B.1 specifies how they enter the evaluation. Heterophilic graphs retain their connectivity and node features; heterogeneous graphs retain connections across node types and type metadata in the unified object; relational graphs retain edge direction and relation metadata; and text-attributed graphs provide textual fields. Each method uses the attributes supported by its implementation, so the evaluation captures both its transfer performance and its ability to exploit the available format-specific information. For dynamic graphs, we aggregate historical interactions into a static snapshot and use each node’s latest observed label. This retains accumulated interaction structure but does not test explicit temporal-dependency modeling. As already discussed in Sec. 5.1, this snapshot treatment may help explain why static GFMs remain competitive on Reddit.
Rebuttal Response
Thank you to the authors for their clarification on the performance variation and the format-specific evaluation, this has addressed my concerns on the presented performance within the submitted manuscript. I appreciate the inclusion of the cited Yuan et. al paper and agree that this work can be treated as an extension to the BRIDGE work, of course on a significantly larger scope.
For my remaining concern, although I trust the insights provided within (1)-(3) as a response to my W1/Q1 questions, I am still unsure how it is grounded in specific measurements, especially given the scope of this benchmarking work. Are there specific metric values beyond downstream performance and overperformance which can support the insights made on: semantic topic proximity not indicating transferability, different format gaps requiring different awareness, and format awareness depending on interaction between the backbone and pre-training?
-
Are there distance calculations which support the semantic topic proximity point? Do we see inconsistent gains for downstream performance within semantic domains relative to between semantic domains to justify this point?
-
Is there some statistical value to indicate meaningful variation for heterophily and textual signals versus mechanism where exposure fails? The sheer number of viable mechanisms, even with the discussion in Section 5.2, keeps it difficult to ground interpretations of the downstream results in anything but model mechanisms instead of analysis that delineates what is 'meaningful variation'.
-
What are the exact numbers that produce this gap between format-aware GFM, structural inductive bias, pre-training, and learned representations? Is there some correlation within the model's internal embedding where FAGCN damages existing representations in some models to degrade performance but successfully boost's performance in other models given alignment with that target model's pre-training strategy?
All three points are certainly interesting, I also understand that the current paper includes these insights. However, from an outside reader's perspective, all of the claimed insights remain difficult to substantiate without measured evidence to support said claims. I am not demanding comprehensive coverage of all three points on every tested model and dataset, but the current insights are abstruse enough that downstream performance should be supported by positive (and/or negative) examples. That way, claims on what is 'meaningful variation' or the gap across multiple separate GFM concepts at-once can be grounded in observations that are measurable via some form of: distance metric, statistical analysis, or further visualization to enable direct correlation. Figure 5 is certainly a good start, but the figure is so dense (and located in the Appendix) that it appears difficult to support the author's insights without direct metric comparison. I ground this reasoning behind my response in the philosophy that benchmarks should inspire further investigation into a given problem, if the authors can provide some evidence beyond performance to support the provided insights then my concern will be addressed.
Response to Reviewer 738P (1/2)
W1 & Q1: Contribution and new insights
Our contribution is not this pipeline itself, but a two-dimensional view of graph domains, distinguishing topic semantics from graph formats, and unified evaluation settings that characterize cross-topic and cross-format generalization on both seen and unseen downstream datasets. As discussed in Sec. 2, existing benchmarks typically vary topic domains while fixing or tightly constraining graph formats, or focus on a narrow task or application area. Our benchmark explicitly accounts for both topic and format domains in pre-training and downstream adaptation, while covering representative GFMs across major modeling regimes. This enables systematic analysis of different generalization behaviors, rather than simply comparing performance on more datasets. In particular, Sec. 5.2 discusses the following insights:
(1) Semantic topic proximity is not a sufficient indicator of transferability. Setting III, together with Fig. 5 and Appendix C.4, shows why topic-matched pre-training need not provide the strongest specialization benefit: datasets within the same topic can differ substantially in structural and textual properties, while molecular downstream tasks additionally differ in granularity from predominantly node-level pre-training objectives. The implication is that useful domain-specific priors cannot be identified through topic labels alone; pre-training strategies must also account for dataset-level diversity and task granularity.
(2) Different format gaps call for different forms of format awareness. Setting IV reveals a distinction between heterophilic/text-attributed targets, which benefit more from broad multi-format pre-training, and heterogeneous/relational/dynamic targets, where broader format mixing yields limited gains over base-format pre-training (Fig. 2(b)). As discussed in Sec. 5.2, this suggests two complementary design needs: exposure to meaningful variation in heterophily and textual signals, and explicit type-, relation-, and time-aware mechanisms where exposure alone remains insufficient. Thus, format diversity should not be treated as a single interchangeable source of training data.
(3) Format awareness also depends on the interaction between the backbone and the pre-training mechanism. Table 6 shows that the same switch to FAGCN substantially improves some GFMs but not others. This indicates that a backbone suited to the target structure is not, by itself, a general solution to format transfer. Format-aware GFM design must consider how structural inductive biases interact with pre-training objectives and learned representations.
Regarding BRIDGE [1] (Yuan et al., ICML 2025), thank you for identifying the missing reference; we will cite and discuss it. BRIDGE explicitly distinguishes cross-dataset transfer within semantic domains from transfer to an unseen semantic domain. We acknowledge this existing treatment of topic-domain differences. However, its evaluation uses six datasets from three semantic domains to assess its proposed model and generalization mechanisms. It does not systematically compare single-topic versus multi-topic pre-training and base-format versus multi-format pre-training across multiple GFMs. Heterogeneous and dynamic graphs are discussed as challenges in its limitations. Our benchmark complements BRIDGE by providing these comparisons and empirical evidence about the generalization behaviors and limitations of current GFMs.