acceptodds
Under review as a conference paper at ICLR 2027

Pitfalls of Evaluating Graph Foundation Models for Node Property Prediction

Abstract

Due to the wide use of graph-structured data in different fields of industry and science, the development of Graph Foundation Models (GFMs) has recently attracted a lot of attention. While many different types of models get called GFMs, particular interest has been paid to GFMs designed for node property prediction tasks, which is one of the most popular settings in Graph Machine Learning with lots of real-world applications from fraud detection in financial and social networks to recommendation systems for e-commerce and user-generated content platforms. While a number of GFMs for node property prediction have been recently proposed, the field has not converged to a unified evaluation setting, and different works evaluate their models in widely different ways. We argue that the currently used evaluation settings often have significant limitations, preventing reliable assessment of GFM quality and comparison of GFMs with each other and to other types of models. In this work, we aim to highlight common pitfalls in the evaluation of GFMs for node property prediction, discuss potential solutions, and draw attention to open questions. We discuss such aspects as dataset and evaluation metric selection, aggregation of results across different datasets, proper tuning of baselines, and computational cost considerations, supporting each issue with illustrative examples from recent works in the field. We conclude with a fair reevaluation of 9 GFMs from the literature, showing that only the most recent GFMs based on the Prior-data Fitted Networks paradigm can compete with (and often outperform) properly tuned classic Graph Neural Networks in predictive performance, although at a much higher inference cost.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural network, gnn, graph foundation model, gfm, prior-data fitted network, pfn, evaluation, benchmarking, pitfalls, node property prediction, methodology

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YXEBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on benchmarking graph foundation models for node property prediction. The authors identify 5 main pitfalls that bedevil current comparison and benchmarking, including: dataset selection, metric selection, result aggregation, baseline implementation, and computational cost. The authors consider testing a variety of GFMs on the GraphLand dataset. They further include better implementations and tuning of classic GNNs, multiple ways of aggregating the results, and the training, tuning, and inference times.

Critical strengths
  1. Pitfall 3 is a very good point. As of now, it's unclear how to aggregate results. I appreciate that the authors use multiple ways in their paper.

  2. Pitfall 4 is also good. Papers often use outdated baseline implementation which artifically improve the results. So I really like how the authors made sure that the results for basic GNNs were as a good as possible.

  3. I like the analysis of time in pitfall 5. For models to work in the real world, we need to be able to actually run them efficiently. So it's nice to see the authors including this.

Critical weaknesses
  1. For pitfalls 1 & 2, I find the contribution of the paper is nonexistant. Prior work [5, 6] ave already test various methods on the the GraphLand datasets with the given metrics. Not even that, but for the GFMs, the authors used the exact same ones already present in the GraphPFN paper. So it's obvious that for most of the datasets/models, the authors simply reuse the results from [5, 6]. While yes, the authors included improved classical GNN baselines plus 2 more datasets, the fact that most of the results are just taken from another paper makes it very difficult for me to see how big of a contribution this is. I mean, this is just a slight extension of the results in [5, 6] with some more analysis.

  2. For pitfall 1, it's not clear to me why Finkelshtein et al. (2025) is not appropriate. This is for a few reasons. One, the authors say that they use too many citation datasets? However, what's considered too many? It seems arbitrary to me. Furthermore, while some issues they note are fair (leakage or incorrect labels), it's unclear to me why having no node features is bad. Some real world graphs don't necessarily have features. I'd argue that it's actually good to have a few datasets like this.

  3. Continuing on weakness 2, the authors seem to choose GraphLand for the datasets. Again, this seems arbitrary. Why Graphland and not GraphBench? For example, one can argue that GraphLand doesn't are too biased towards industrial data. Why not use both? The authors need to be more convincing here.

  4. The authors are missing many baseline methods [1,2,3,4]. It's imperative to include all relevant baselines for a proper benchmarking.

  5. So much of the paper details are in the appendix. In and of itself, this is fine. However, it makes it practically impossible to read this paper purely based on the main text (as is required of us reviewers). For example, pitfalls 1-3 are accompanied by a referenced section in the appendix that explain the problems in more details. The actually background on the differnet models are in the appendix as well (C & D) along with the experimental setup. In isolation, any of these is generally okay. However, all together, it reads like a paper that is not self-contained in the main text. I don't want to make a big deal about this, but it does make the paper feel very disjointed when reading.

[1] Wang, Zehong, et al. "Gft: Graph foundation model with transferable tree vocabulary." NeurIPS 2024.
[2] Wang, Zehong, et al. "Towards Graph Foundation Models: Learning Generalities Across Graphs via Task-Trees." ICML 2025.
[3] Zhao, Jianan, et al. "Fully-inductive node classification on arbitrary graphs." ICLR 2025.
[4] Lee, Dooho, and Jaemin Yoo. "Node4all: Learning node representation beyond datasets." KDD 2026.
[5] Eremeev, Dmitry, et al. "Turning tabular foundation models into graph foundation models." arXiv 2025.
[6] Eremeev, Dmitry, et al. "GraphPFN: A Prior-Data Fitted Graph Foundation Model." ICLR 2026.

Questions for authors
  1. Please explain the critical contributions of this paper beyond prior ones (see weakness 1)?
  2. Why is GraphLand chosen over other benchmarks (see weakness 2 and 3)?
  3. Can you please include results for more GFMs (see weakness 4)?
3 comments

Dear reviewer, thank you for your feedback and constructive comments. We address your concerns below.

W1 & Q1 Indeed, our setting is similar to that of [5, 6] since these papers avoided many of the pitfalls discussed in our study. In addition to using more datasets and baselines, we also analyze computational efficiency, which is an important part of our analysis missing from [5, 6]. We also provide a more detailed model comparison by using several aggregation measures.. Additionally, the examples provided in our paper demonstrate that most works in the field fall victim to pitfalls 1 & 2, and thus we believe it is important to highlight them. Overall, our aim is to encourage the adoption of more reliable evaluation practices in the field.

W2 First, many of the datasets used there are flawed/buggy, as demonstrated in prior works and mentioned in our work. As for citation networks, the problem with them is that they do not represent any real-world applications, yet they are extensively being used in graph machine learning due to ease of obtaining them. Having many citation network datasets leads to lack of diversity in the benchmark. Further, as [7] shows, citation network datasets tend to have lots of label errors. Finally, citation network datasets are a natural example of text-attributed graphs (TAGs) which can likely be more effectively processed by LLM-based methods, while most papers using these datasets for GFM evaluation rely on extremely outdated bag-of-words features, making the setup particularly unrealistic.

W3 & Q2 In our work, we specifically focus on node property prediction as one of the most popular and practically relevant graph machine learning settings. As we discuss in lines 154-161, GraphLand is specifically a benchmark for node property prediction, while GraphBench does not provide classic node property prediction datasets (it instead focuses on other applications of graph machine learning, which is very useful for highlighting different real-world use-cases, but moves this benchmark beyond the scope of our study), which is why we use GraphLand.

W4 & Q3 Thank you for pointing out these works, they are indeed relevant to our study. However, to the best of our knowledge, none of them provide pretrained models, which is one of the criterions we used for selecting GFMs for our study.

W5 Thank you for pointing this out, we will do our best to improve the structure of the paper in its future revisions.

[7] GraphCleaner: Detecting Mislabelled Samples in Popular Graph Learning Benchmarks (ICML 2023)

We also provide a more detailed model comparison by using several aggregation measures.. Additionally, the examples provided in our paper demonstrate that most works in the field fall victim to pitfalls 1 & 2

I understand, however I think this is a fairly minimal contribution. Thus my concern still stands.

Citation Networks

I think that's fair. I think if you give a more detailed explanation in the paper (maybe the Appendix) that would be better. Because right now it's unclear.

while GraphBench does not provide classic node property prediction datasets

Yes they do? They include 5 node regressions datasets and 1 for node classification (see Table 6 here).

Other models

I don't think no provided checkpoint is a valid answer. They each provide the code to conduct pre-training. For example, I quickly look at the Github for [1, 2]. They provide the data and detailed instruction for pre-training. I think it's perfectly reasonable to take a week to pre-train some some baselines.

Due to these concerns, I will keep my score.

Yes they do? They include 5 node regressions datasets and 1 for node classification (see Table 6 here).

We would like to clarify that we specifically meant the classic transductive node property prediction in a single graph setting, where a large graph (e.g. a social network) is given and node labels are available for some of the nodes but not for others. This is arguably the most popular setting in contemporary graph machine learning which is also very wide-spread in industrial applications, and our paper focuses on it. GraphBench does not provide datasets for this setting. While it has node property prediction datasets, they use very different settings, mostly node property prediction in a large number of relatively small graphs (which can appear, e.g., in combinatorial optimization tasks). While these settings are also important, they are beyond the scope of our work.

Reviewer_qPMfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides common pitfalls of Graph Foundational Models and reevaluates them in different datasets in the application of node classification, focusing mainly on the classification and running-time performance. While GFMs are performing better than GNNs, they are computationally more intensive during inference.

Critical strengths
  • The paper is well written and easy to follow.
  • The paper considers multiple datasets and methods in its experiments, providing good initial overview of GFMs and GNNs.
  • The paper shares the code and implementations.
Critical weaknesses
  1. Lack of enough contribution.
  2. Missing details and weak organization overall.

Details:

  • The paper mentions very basic pitfalls that can be applicable to any data structure other than graphs. There is no uniqueness in this study.
  • Proposed solutions to the pitfalls are very weak; they don’t include any big innovations. It is a good study but very surface level and there are no real conclusions from the methods beyond basic ones. It doesn't provide ICLR-standard quality study.
  • The paper discusses the pitfalls very long in its main body while very few experiments and their conclusions are very basic. As a survey paper, it requires more deep-dive in the methods and pitfalls. The organization of the paper is weak.
  • The paper says that node property prediction is the most impactful application in industrial settings. I think this comparison is unnecessary. The paper shouldn't compare different applications. It should focus on what it provides. One can argue that link prediction is another valuable application which is used in recommendation systems. However, the paper shouldn't focus on this argument.
  • The paper mentions Accuracy is okay to use when there is a cost difference between FN and FP, however this is not true. 3% positive sample example actually proves that and this is not related to the cost difference. Also note that ROC-AUC is also a weak metric to use for imbalanced datasets, which can show the method better than it actually is.
  • The paper mentions that it improves GCN, GAT architecturally, but there are no details. Running-time experiments don't provide mean and std. Experimental settings are missing details.
Questions for authors

Feel free to provide your answers to the weaknesses section.

2 comments

Dear reviewer, thank you for your feedback. We address your concerns below.

W1 Indeed, reliable evaluation practices are important in all fields of machine learning. However, we would like to note that the evaluation of GFMs in particular very frequently suffers from poor evaluation settings, which we highlight with multiple examples from recent works. Further, note that our discussion of dataset limitations is specific to graph datasets and our discussion of weak baselines is specific to GNNs.

W2 While our proposed solutions are indeed quite simple, our work provides many examples of prior papers that do not follow them, thus we believe emphasizing these solutions has significant importance for the field.

W3 Could you please clarify your suggestion regarding the paper structure? Namely, which discussions/results are missing from the main text? We did our best to thoroughly discuss every pitfall in the main text but we are open to suggestions which will help us to improve the presentation.

W4 Please note that we do not compare different applications, we just provide examples of impactful applications where node property prediction is used.

W5 Please note that we explicitly claim that using accuracy is not justified when FN and FP costs are different (lines 179-182).

W6 As we discuss in Section 4, we use improved GNN implementations from [1], which augment the models with normalizations, skip connections, and dropout.

[1] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification (NeurIPS 2024)

Reviewer_qPMfOpenReview ↗

Thank you. I maintain my score.

I thank the authors for their responses. Regarding W3, I would like to see more of the experiments and explanations in the main body rather than mostly in the appendix. For that, my suggestion would be condensation of the pitfalls section to make room for them.

I maintain my score.

Reviewer_UEutOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper points out five pitfalls when evaluating graph foundation models for node property prediction, namely, dataset selection, evaluation metric selection, results aggregation across datasets, baseline evaluation, and computational cost considerations. The authors also provide corresponding solutions to each of the pitfall, and conduct a comprehensive experiment based on them. Final evaluation includes 15 baselines and 10 datasets from GraphLand.

Critical strengths
  1. This paper is well-written and easy to follow. For example, it explains each pitfall with problem, example and potential solutions.
  2. Final evaluation is rigorous, with the settings proposed by the authors.
Critical weaknesses
  1. The justification for each pitfall is anecdotal. Each pitfall is supported by one or two cherry-picked examples rather than a systematic analysis across the literature or datasets. For example, in Sec. 3.1 (dataset selection), the paper never measures the diversity of previous benchmarks or of GraphLand; readers are simply told to use GraphLand. If the pitfalls are the main selling point of this paper, they need a systematic analysis showing how common each pitfall is and quantifying its effect on conclusions. Without it, the proposed solutions are not well justified.
  2. No graph statistics are reported. The paper gives no node/edge counts, feature dimensions, class balance, or homophily for any dataset, even though Sec. 3.1 lists these as required dimensions of diversity.
  3. The topic is not specific to graph foundation models. The pitfalls are general requirements for any graph machine learning experiment, so the GFM framing is not justified.
Questions for authors
  1. Can the authors provide a systematic analysis for each pitfall, i.e., how many recent GFM papers exhibit it and how much it changes their conclusions? For example, in particular for Sec. 3.1, please quantify the diversity of previous benchmarks versus GraphLand rather than recommending GraphLand without comparison.
  2. What are the statistics of the graphs used in Sec. 4?
  3. Which of the five pitfalls are specific to graph foundation models rather than graph machine learning in general?
2 comments

Dear reviewer, thank you for your feedback. We address your concerns below.

W1 & Q1 We would like to note that we purposefully avoided a systematic analysis of the prevalence of the discussed pitfalls in recent works because our aim is not to diminish or criticise particular prior works but to encourage the adoption of more reliable evaluation practices in the field. The few examples are provided only to show that the problems being discussed indeed exist and are quite wide-spread in the literature. Rather than criticise prior works, we would instead like to highlight works that aim to propose solutions to these pitfalls like GraphLand and GraphBench curating more reliable datasets or [1] proposing strong GNN baselines.

As for investigating whether fixing these pitfalls changes conclusions, this is why we conducted our own experimental study in a more reliable setting in Section 4. While all the considered prior works proposing GFMs claim that they outperform classic GNNs, our study shows that only PFN-based GFMs outperform well-tuned GNNs, which is a very notable difference in conclusions.

Regarding why we recommend the GraphLand benchmark over prior datasets, we extensively discuss the problems with popular prior datasets in Section 3.1 and Appendix B.1. Please note the GraphLand was specifically constructed to avoid such problems, as discussed in its original paper [2].

W2 & Q2 Please note that a table describing the characteristics of GraphLand datasets is already provided in the original paper (Table 1 in [2]), demonstrating this benchmark’s diversity in terms of node/edge counts, feature dimensions, homophily, clustering and other characteristics. We will explicitly refer to this table when discussing the diversity of GraphLand. Further, GraphLand datasets represent different real-world applications of graph machine learning, while prior benchmarks mostly focus on a single domain – citation networks.

W3 & Q3 While reliable evaluation settings are indeed important for all of graph machine learning, our examples demonstrate that the pitfalls we highlight are particularly prevalent when evaluating new GFMs and thus distort conclusions being made about these models, which is why we specifically focus on GFMs. Further, as we discuss in Section 4 (lines 280-285), there has already been a number of works providing unified reevaluations of classic GNNs (but we note that insights from these reevaluations are typically not used to create proper strong baselines for newly proposed GFMs), while there have been no such works for GFMs, which is a gap we are aiming to fill with our work.

[1] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification (NeurIPS 2024)

[2] GraphLand: Evaluating Graph Machine Learning Models on Diverse Industrial Data (NeurIPS 2025)

I thank the authors for the response, but my concerns are not addressed. I will keep my score as it is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.