- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a new approach to graph foundation models that is based on sampling random walks from a graph and applying to them a neural network that is based on prior-data fitted networks (PFNs) paradigm and is pretrained on synthetically-generated data. The proposed model (WANDER) can be applied to node classification, link prediction, and typed link prediction (in knowledge graphs) in the in-context learning regime, and is the first model that can perform all these tasks with a single checkpoint. The paper proves some theoretical properties of the proposed approach and provides extensive empirical evaluation.
Critical strengths
The general approach proposed in the paper is interesting and might be potentially useful.
Critical weaknesses
The paper is very difficult to follow, which is exacerbated by a lot of important information being hidden in the appendix. Most notably, Section 5 barely describes how the model actually works - while some basic information flow is described, it is not described what is an intra-node processor, how an in-context learning update is performed, what random walk algorithm is used, what is a recording protocol and a consensus protocol. Providing this information would have been much more useful than spending space on Figure 1 which does not add anything to the text. Another issue are the mathematical results on which the paper spends a lot of space. Almost all of them are conceptually quite straightforward, but following the provided derivations is very difficult due to convoluted notation that tries to unify node classification and link prediction. Overall, I believe the paper needs to be substantially rewritten to properly describe the method being proposed in the main text.
Another weakness is the experimental section. The experiments cover a vast range of datasets, however, most of the node classification and link prediction datasets used are not reliable and have been criticized many times [1, 2, 3]. The paper even cites some of these works, but still uses flawed datasets. For link prediction, it would have been much better to use datasets from OGB [4], and for node property prediction, it would have been much better to use datasets from GraphLand [3] (the paper uses 2 datasets from GraphLand instead of the full benchmark, and one of them is strongly downsampled, which raises the question about the scalability of the method, while on the second one WANDER is very strongly outperformed by a simple GCN, which is also a suspicious signal that is never investigated in the paper).
Finally, Appendix D2 reveals that the pretraining pipeline is very complex and involves multiple stages with different parts of the model being trained. This likely indicates that the model is unstable to train, but the training pipeline is not ablated.
Overall, I believe the paper needs much better presentation, experiments on more reliable datasets, and proper architetcure and training ablations.
[1] A critical look at the evaluation of GNNs under heterophily: Are we really making progress? (ICLR 2023)
[2] Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks (ICML 2025)
[3] GraphLand: Evaluating Graph Machine Learning Models on Diverse Industrial Data (NeurIPS 2025)
[4] Open Graph Benchmark: Datasets for Machine Learning on Graphs (NeurIPS 2020)
Questions for authors
Can the model scale to large graphs?
Why were link prediction and node classification experiments conducted on outdated datasets?
Can the training pipeline be simplified to train the entire model in one stage?
There are also some issues with references in the paper. Examples: TS-Net was published at NeurIPS 2025, not NeurIPS 2026. GraphPFN was published at ICML 2026 and the bibliography reference is correct, but Table 1 claims the year 2025 for this model.
5 comments
Response to reviewers concerns and questions (Part 2/2)
Question 1: Can the model scale to large graphs?
Yes. Our additional experiments demonstrate that Wander can be evaluated on graphs with millions of nodes, including Web Topics (2.9 million nodes) and OGB-Citation2 (2.9 million nodes).
For link prediction, Wander processes queries independently, while for node classification it processes test nodes in batches and reuses cached context representations. Larger evaluation sets therefore naturally require more computation because they contain more prediction queries. The random-walk interface provides control over the structural computation allocated to each query or batch: for a fixed architecture, this computation scales linearly with the number and length of sampled walks, rather than the total number of graph edges.
Whether a fixed sampling budget maintains predictive accuracy on larger graphs is a separate empirical question. As discussed in Appendix D.10, walk budgets and test-time ensembling provide a tradeoff between computation and predictive performance. Our additional large-graph experiments illustrate this tradeoff:
| Graph | Nodes | Edges | Walk budget × length | Batch size | Inference samples | Seconds/batch* | Accuracy | Sampled Recall@20 |
|---|---|---|---|---|---|---|---|---|
| PPA | 576,289 | 60,652,546 | 256 × 64 | 4 | 2 | 6.5 | — | 91.2 |
| PPA | 576,289 | 60,652,546 | 512 × 64 | 4 | 2 | 7.3 | — | 93.4 |
| Web Topics | 2,890,331 | 24,754,822 | 256 × 128 | 2,048 | 4 | 82 | 39.9 | — |
| Web Topics | 2,890,331 | 24,754,822 | 256 × 128 | 256 | 4 | 79 | 42.1 | — |
*measured on single H100 GPU
On PPA, halving the walk budget reduces runtime from 7.3 to 6.5 seconds at a cost of 2.2 percentage points in sampled Recall@20. On Web Topics, increasing batch size eightfold improves throughput by 7.7×, with a preliminary accuracy decrease of 3.2 percentage points. These results illustrate Wander’s computation–accuracy tradeoff on large graphs.
Separately, random-walk sampling also has precedents in large-scale production recommendation systems: Pinterest’s Pixie [1] reports deployed query-time random-walk recommendations over a graph with 3 billion nodes and 17 billion edges, while Twitter’s GraphJet [2] reports serving real-time recommendations using random-walk algorithms. These systems provide evidence that random-walk graph access can be practical at industrial scale.
[1] Eksombatchai et al: Pixie: A system for recommending 3+ billion items to 200+ million users in real-time. WWW 2018.
[2] Sharma et al.: GraphJet: Real-time content recommendations at Twitter. VLDB 2016.
Question 3: Can the training pipeline be simplified to train the entire model in one stage?
Yes, the entire model can be trained in one stage from scratch. The motivation for initially hiding structure is that the model might otherwise learn to predict labels from neighboring labels without learning to relate node features to labeled examples through the ICL layer. This shortcut can work well on the pretraining graphs but transfer poorly to graphs with different relationships between structure and labels. Hiding structure forces the model to learn from features and in-context supervision first.
Following reviewers advice, we added a preliminary ablation in Appendix D.15 comparing three runs:
- Tabular from scratch: our first pretraining stage, which hides graph structure.
- Graph from tabular: our third stage, which introduces structure after 200 epochs of structure-free pretraining.
- Graph from scratch: the same third-stage setup trained from scratch, skipping structure-free pretraining.
On Roman Empire, after the same 48,000 node-classification training steps, Tabular from scratch achieves 56.59% accuracy, compared with 42.44% for Graph from scratch. Thus, hiding structure leads to better transfer for the same number of training steps, consistent with encouraging more effective use of features and in-context supervision. Graph from tabular maintains strong performance after structure is introduced (59.58%), suggesting that it retains these capabilities.
On Cora, where neighboring labels are strongly predictive and the ICL layer may be less essential, Graph from scratch approaches the performance of Graph from tabular.
These early results motivate our curriculum but do not establish that it is necessary. We are running a full comparison with single-stage training and will report the results once available.
I appreciate the authors' responses. However, I believe the evaluation setting being used is still very flawed, especially in the node classification experiments, thus I maintain my score.
The authors refer to "direct comparability with existing graph foundation models, including GraphAny, TS-Net, and NodePFN" as explanation for their evaluation setting used. However, I would like to note that comparing to these models is actually a problem in itself - as the GraphPFN paper shows, TS-Net and NodePFN models obtain very weak results on realistic datasets and are outperformed by simple GNNs. As for GraphAny - it achieves even weaker results because it is simply a weighted ensemble of linear regression models. Note that the GraphAny paper does not actually claim GraphAny to be a GFM (an earlier version of the paper did claim that, but the paper was only excepted after this claim was removed). A recent study [1] shows that that GraphPFN, TAG, and G2T are currently the strongest GFMs for node property prediction, and they are what new models should be compared to (especially as the proposed model is also based on PFNs). And using flawed and actively criticized datasets for compatibility with prior studies is also not a good reason. For example, GraphPFN and [1] reevaluated multiple models including TAG and TS-Net on more reliable datasets for their studies.
Finally, I appreciate the experiments that show that the model scales to million-node graphs, however, results on the web-topics dataset once again demonstrate that on realistic feature-rich datasets WANDER is outperformed by simple GNNs (see the GraphLand paper for GNN results on this dataset).
[1] A Fair Evaluation of Graph Foundation Models for Node Property Prediction
Response to reviewers concerns and questions (Part 1/2)
We thank the reviewer for their detailed feedback. Their suggestions have helped us improve the presentation and extend the experimental evaluation. Below, we address each concern and describe the corresponding revisions and additional results.
We thank the reviewer for this feedback. We have expanded the method description in Section 5, which now explicitly describes both the role and the implementation of each component.
We have also simplified Section 4 by removing the former Theorem 1 and its associated risk notation. Instead, we decided to simply explain the posterior-mixture interpretation directly to reduce notation and complexity. The section now makes the connection to our experimental questions more explicit: learning across tasks and diverse graph families, and combining capabilities acquired during pretraining.
Our original dataset selection was motivated by direct comparability with existing graph foundation models, including GraphAny, TS-Net, and NodePFN, which evaluate on overlapping benchmarks. The datasets proposed by the reviewer also presented practical challenges for baseline evaluation. GraphPFN encountered memory limitations, while NodePFN supports at most 20 classes and cannot be applied to all datasets. These constraints motivated our initial use of a downsampled version of Pokec Regions.
To address the reviewer's concern, we have substantially extended our evaluation across all three prediction tasks. Across these additional benchmarks, Wander outperforms all evaluated baselines on both node-classification datasets, two of three homogeneous link-prediction datasets, and on OGB-BioKG, demonstrating its competitiveness on larger and more recent benchmarks.
Node classification: We additionally evaluated the full Pokec Regions and Web Topics datasets from GraphLand, with results incorporated into Table 5.
NodePFN and GraphPFN could not be run on these datasets as they exceed their maximum number of supported classes. Using the official ECOC implementation of GraphPFN exceeded the memory limit of the used 80GB H100 GPU.
Homogeneous link prediction: We additionally evaluated OGB-Collab, OGB-PPA, and OGB-Citation2 using sampled Recall@20 (%) with 1,000 randomly sampled negatives. Results have been added to Appendix D.4 (Table 18). AnyGraph could not be evaluated on Citation2 due to a runtime error in its official implementation, while NBFNet exceeded the memory capacity of an 80GB H100 GPU.
Knowledge graph completion: We additionally evaluated OGB-BioKG, obtaining the following results:
Regarding HM Categories, we agree that the substantial performance gap to GCN merits attention. As analyzed in Appendix D.6, this dataset is unusually dense, potentially limiting the structural coverage of a fixed random-walk budget. Increasing the base walk count from 256 to 2,048 improves WANDER's accuracy from 48.7% to 52.1%. Although this does not close the gap to GCN, it suggests that insufficient walk coverage contributes to the performance difference and that walk budgets may need to account for graph density as well as graph size.