- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on benchmarking graph foundation models for node property prediction. The authors identify 5 main pitfalls that bedevil current comparison and benchmarking, including: dataset selection, metric selection, result aggregation, baseline implementation, and computational cost. The authors consider testing a variety of GFMs on the GraphLand dataset. They further include better implementations and tuning of classic GNNs, multiple ways of aggregating the results, and the training, tuning, and inference times.
Critical strengths
-
Pitfall 3 is a very good point. As of now, it's unclear how to aggregate results. I appreciate that the authors use multiple ways in their paper.
-
Pitfall 4 is also good. Papers often use outdated baseline implementation which artifically improve the results. So I really like how the authors made sure that the results for basic GNNs were as a good as possible.
-
I like the analysis of time in pitfall 5. For models to work in the real world, we need to be able to actually run them efficiently. So it's nice to see the authors including this.
Critical weaknesses
-
For pitfalls 1 & 2, I find the contribution of the paper is nonexistant. Prior work [5, 6] ave already test various methods on the the GraphLand datasets with the given metrics. Not even that, but for the GFMs, the authors used the exact same ones already present in the GraphPFN paper. So it's obvious that for most of the datasets/models, the authors simply reuse the results from [5, 6]. While yes, the authors included improved classical GNN baselines plus 2 more datasets, the fact that most of the results are just taken from another paper makes it very difficult for me to see how big of a contribution this is. I mean, this is just a slight extension of the results in [5, 6] with some more analysis.
-
For pitfall 1, it's not clear to me why Finkelshtein et al. (2025) is not appropriate. This is for a few reasons. One, the authors say that they use too many citation datasets? However, what's considered too many? It seems arbitrary to me. Furthermore, while some issues they note are fair (leakage or incorrect labels), it's unclear to me why having no node features is bad. Some real world graphs don't necessarily have features. I'd argue that it's actually good to have a few datasets like this.
-
Continuing on weakness 2, the authors seem to choose GraphLand for the datasets. Again, this seems arbitrary. Why Graphland and not GraphBench? For example, one can argue that GraphLand doesn't are too biased towards industrial data. Why not use both? The authors need to be more convincing here.
-
The authors are missing many baseline methods [1,2,3,4]. It's imperative to include all relevant baselines for a proper benchmarking.
-
So much of the paper details are in the appendix. In and of itself, this is fine. However, it makes it practically impossible to read this paper purely based on the main text (as is required of us reviewers). For example, pitfalls 1-3 are accompanied by a referenced section in the appendix that explain the problems in more details. The actually background on the differnet models are in the appendix as well (C & D) along with the experimental setup. In isolation, any of these is generally okay. However, all together, it reads like a paper that is not self-contained in the main text. I don't want to make a big deal about this, but it does make the paper feel very disjointed when reading.
[1] Wang, Zehong, et al. "Gft: Graph foundation model with transferable tree vocabulary." NeurIPS 2024.
[2] Wang, Zehong, et al. "Towards Graph Foundation Models: Learning Generalities Across Graphs via Task-Trees." ICML 2025.
[3] Zhao, Jianan, et al. "Fully-inductive node classification on arbitrary graphs." ICLR 2025.
[4] Lee, Dooho, and Jaemin Yoo. "Node4all: Learning node representation beyond datasets." KDD 2026.
[5] Eremeev, Dmitry, et al. "Turning tabular foundation models into graph foundation models." arXiv 2025.
[6] Eremeev, Dmitry, et al. "GraphPFN: A Prior-Data Fitted Graph Foundation Model." ICLR 2026.
Questions for authors
- Please explain the critical contributions of this paper beyond prior ones (see weakness 1)?
- Why is GraphLand chosen over other benchmarks (see weakness 2 and 3)?
- Can you please include results for more GFMs (see weakness 4)?
3 comments
We also provide a more detailed model comparison by using several aggregation measures.. Additionally, the examples provided in our paper demonstrate that most works in the field fall victim to pitfalls 1 & 2
I understand, however I think this is a fairly minimal contribution. Thus my concern still stands.
Citation Networks
I think that's fair. I think if you give a more detailed explanation in the paper (maybe the Appendix) that would be better. Because right now it's unclear.
while GraphBench does not provide classic node property prediction datasets
Yes they do? They include 5 node regressions datasets and 1 for node classification (see Table 6 here).
Other models
I don't think no provided checkpoint is a valid answer. They each provide the code to conduct pre-training. For example, I quickly look at the Github for [1, 2]. They provide the data and detailed instruction for pre-training. I think it's perfectly reasonable to take a week to pre-train some some baselines.
Due to these concerns, I will keep my score.
Yes they do? They include 5 node regressions datasets and 1 for node classification (see Table 6 here).
We would like to clarify that we specifically meant the classic transductive node property prediction in a single graph setting, where a large graph (e.g. a social network) is given and node labels are available for some of the nodes but not for others. This is arguably the most popular setting in contemporary graph machine learning which is also very wide-spread in industrial applications, and our paper focuses on it. GraphBench does not provide datasets for this setting. While it has node property prediction datasets, they use very different settings, mostly node property prediction in a large number of relatively small graphs (which can appear, e.g., in combinatorial optimization tasks). While these settings are also important, they are beyond the scope of our work.
Dear reviewer, thank you for your feedback and constructive comments. We address your concerns below.
W1 & Q1 Indeed, our setting is similar to that of [5, 6] since these papers avoided many of the pitfalls discussed in our study. In addition to using more datasets and baselines, we also analyze computational efficiency, which is an important part of our analysis missing from [5, 6]. We also provide a more detailed model comparison by using several aggregation measures.. Additionally, the examples provided in our paper demonstrate that most works in the field fall victim to pitfalls 1 & 2, and thus we believe it is important to highlight them. Overall, our aim is to encourage the adoption of more reliable evaluation practices in the field.
W2 First, many of the datasets used there are flawed/buggy, as demonstrated in prior works and mentioned in our work. As for citation networks, the problem with them is that they do not represent any real-world applications, yet they are extensively being used in graph machine learning due to ease of obtaining them. Having many citation network datasets leads to lack of diversity in the benchmark. Further, as [7] shows, citation network datasets tend to have lots of label errors. Finally, citation network datasets are a natural example of text-attributed graphs (TAGs) which can likely be more effectively processed by LLM-based methods, while most papers using these datasets for GFM evaluation rely on extremely outdated bag-of-words features, making the setup particularly unrealistic.
W3 & Q2 In our work, we specifically focus on node property prediction as one of the most popular and practically relevant graph machine learning settings. As we discuss in lines 154-161, GraphLand is specifically a benchmark for node property prediction, while GraphBench does not provide classic node property prediction datasets (it instead focuses on other applications of graph machine learning, which is very useful for highlighting different real-world use-cases, but moves this benchmark beyond the scope of our study), which is why we use GraphLand.
W4 & Q3 Thank you for pointing out these works, they are indeed relevant to our study. However, to the best of our knowledge, none of them provide pretrained models, which is one of the criterions we used for selecting GFMs for our study.
W5 Thank you for pointing this out, we will do our best to improve the structure of the paper in its future revisions.
[7] GraphCleaner: Detecting Mislabelled Samples in Popular Graph Learning Benchmarks (ICML 2023)