- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a benchmark for evaluating graph foundation models (GFMs) against GNNs. The benchmark comprises 51 real-world datasets together with evaluation metrics and their implementations, and the paper reports an evaluation and analysis of GFMs and GNNs. Its coverage is substantially broader than that of prior benchmarks, and this benchmark is likely to be widely used in future work.
Critical strengths
S1. Many existing papers that propose GFMs report that GFMs outperform GNNs, but their experimental settings are often insufficient. This paper evaluates GNNs in two configurations, default and tuned supervised, and shows that tuned GNNs outperform GFMs in most cases. Clarifying the performance limits of current GFMs and the open challenges for future work is a clear contribution.
- The authors tune 15 GNNs and evaluate 6 GFMs on 51 datasets. They show that, overall, the tuned GNNs (particularly GCNII, GPRGNN, and FAGCN) and GraphPFN, a GFM, achieve the highest performance.
- The fair comparison protocol is excellent, as it allows GFMs and GNNs to be compared at their best attainable performance.
Critical weaknesses
W1. Although the results are comprehensive and of high quality, they offer few surprises. Figure 5 contains the new findings described in S1. Figures 6-8, however, essentially present the results of Figure 5 from different angles and add relatively little new insight.
W2. The paper should better explain the motivation behind the choice of graph properties for evaluation.
- Properties such as #nodes, average degree, and density need no justification. However, the purpose of including nonzero fraction, label imbalance, and label entropy should be explained.
- Figure 4 visualizes pairs of properties, but the paper does not sufficiently explain why these particular combinations were chosen.
Questions for authors
Q1. One would normally expect GNNs to outperform GFMs in the split with a larger training ratio. However, in Figure 8, this conjecture does not appear to be the case, particularly for the tuned models. Could you explain why GraphPFN performs better than tuned GNNs for 50/25/25 split than 10/10/80 split?
3 comments
Q1. GraphPFN’s relative improvement with a larger label budget.
GraphPFN builds on LimiX, an ICL-based tabular foundation model [1,2], and uses labeled nodes as inference-time context while keeping its pretrained parameters fixed. Context-size ablations in related tabular foundation models show that additional labeled context can improve performance. TabDPT reports reduced performance when context is restricted to 256 examples [3], while ConTextTab reports improvements with longer contexts, particularly on larger datasets [4].
Therefore, richer labeled context offers a plausible explanation for GraphPFN’s stronger relative performance under 50/25/25. On Shanghai, for example, GraphPFN’s accuracy rises from 43.81% to 65.18% (+21.37 percentage points), compared with 46.60% to 47.55% for tuned GCNII (+0.95 points).
Such differences in improvement help explain the relative performance changes in Figure 8. Mean Improvability (MI) measures the average normalized gap to the best evaluated method on each dataset within each split. GraphPFN’s MI remains nearly unchanged, whereas the MI of GCNII and GPRGNN increases, indicating that they fall further behind the dataset-specific leaders. Thus, GraphPFN becomes more competitive relative to these baselines without implying that their absolute performance deteriorates.
References
[1] Eremeev et al. GraphPFN: A Prior-Data Fitted Graph Foundation Model. 2025. Paper.
[2] LimiX Team. LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence. 2025. Paper.
[3] Ma et al. TabDPT: Scaling Tabular Foundation Models on Real Data. NeurIPS, 2025. Paper.
[4] Spinaci et al. ConTextTab: A Semantics-Aware Tabular In-Context Learner. NeurIPS, 2025. Paper.
Thank you for the clarification.
We appreciate your thoughtful feedback and address your concerns below.
W1. Insights beyond the aggregate results.
Figure 6 reveals where the overall ordering reverses. Although tuned GNNs lead in Figure 5, GraphPFN achieves the best mean rank on binary tasks under both label splits, while tuned GNNs remain stronger on multiclass tasks. This identifies a task type where GraphPFN outperforms the tuned baselines on average despite its lower overall position.
Figure 7 separates overall improvement from persistent weaknesses. GraphPFN’s improved overall position at the higher label budget does not mean its weaknesses disappear. It continues to trail GCNII and GPRGNN on high-dimensional and highly imbalanced datasets. This identifies specific limitations that additional labels do not resolve, which the aggregate ranking cannot locate.
Figure 8 reveals competitive choices overlooked by ranking predictive performance alone. GVT and GraphPFN lie on the frontier against default and fully tuned supervised methods despite their lower predictive rankings. GVT offers lower workflow cost, while GraphPFN provides stronger predictions at greater cost. Other GFMs are dominated on both dimensions, showing which pretrained models offer useful computational tradeoffs and which do not.
We will revise Sections 5.1–5.3 to make these additional findings explicit.
W2a. Motivation for nonzero fraction, label imbalance, and label entropy.
These properties describe feature representations and label distributions beyond graph size and connectivity. They help assess whether the collection covers the different feature and label distributions encountered by the evaluated models.
Nonzero fraction distinguishes sparse from dense attributes, which feature dimension alone cannot capture. Together, these properties characterize both the size and sparsity of node representations.
Label imbalance, measured by the largest-to-smallest class-size ratio, captures minority-class scarcity. Even under stratified splits, the same training fraction can yield very different numbers of labeled examples across classes.
Normalized label entropy captures how probability mass is distributed across all classes, complementing the imbalance ratio’s focus on the largest and smallest classes. For example, a small minority class can substantially change the imbalance ratio without greatly changing the distribution of most probability mass.
W2b. Motivation for the property pairs in Figure 4.
We selected these pairs to distinguish complementary aspects of graph structure, feature representation, and label alignment. Each pair combines related properties that cannot substitute for one another, allowing us to examine whether the collection covers different combinations within these aspects.
Graph size × average degree separates graph scale from local connectivity, because a larger graph does not necessarily provide more neighbors per node.
Median nonzero degree × hub concentration separates typical connectivity from the degree distribution’s upper tail, because similar median degrees can coexist with very different hub concentrations.
Feature dimension × nonzero fraction separates the number of available attributes from their sparsity, because equally high-dimensional representations can be sparse or dense.
Feature-based label separation × adjusted homophily separates label alignment in node features from label alignment along edges, because one can be strong while the other is weak.
These pairings therefore assess coverage of distinct combinations that the individual property ranges in Figure 3 cannot establish. We will clarify this selection rationale in Section 3.2 and the Figure 4 caption.