acceptodds
Under review as a conference paper at ICLR 2027

Scalable Graph In-Context Learning with Stronger Predictive Performance

Abstract

Applying graph neural networks (GNNs) typically requires training and hyperparameter tuning for each dataset, incurring repeated costs in time and effort. Graph in-context learning (graph ICL) offers an alternative: a pretrained model predicts missing node labels on unseen graphs through inference alone, using observed labels as context. However, existing graph ICL models rely on dense attention across nodes, limiting inference efficiency and scalability. *We ask whether dense attention is necessary for graph in-context learning.* We introduce Ephris, which realizes graph ICL through sparse message passing, scaling linearly with node-feature entries and graph edges. We pretrain Ephris entirely on synthetic tasks from structural causal models extended with diverse graph structures and relational dynamics, capturing varied dependencies among topology, features, and labels. Across 51 node-classification datasets, we compare Ephris against 15 extensively tuned GNNs and prior graph ICL models under high-label (50/25/25) and low-label (10/10/80) train/validation/test splits. Ephris ranks 1st across all four aggregate metrics, including Elo, improvability, average rank, and accuracy, in both regimes. This strong performance comes at substantially lower adaptation cost, with inference comparable to a single GNN training run. Together, Ephris advances the performance-runtime Pareto frontier, offering a practical alternative to per-dataset training and tuning. Code and model weights are available at https://anonymous.4open.science/r/ephris/.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph In-Context Learning, Graph Foundation Models, Prior-Data Fitted Networks, Tabular Foundation Models, Synthetic Pretraining, Structural Causal Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gZDaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether dense cross-node attention is necessary for graph in-context learning, and presents a graph ICL architecture based on sparse message passing: each scalar feature is tokenized and refined by per-feature and per-node induced attention blocks into a fixed-length node representation; message-passing blocks then perform ICL (each block contains local attention, attention scaled by neighborhood size, and global node and residual propagation); finally, an output head built on fixed class embeddings handles an arbitrary number of classes.

Critical strengths
  • The paper raises a clear question: whether graph in-context learning must rely on dense cross-node attention. Handling context and query nodes with sparse message passing on the observed graph is a clear methodological route.
  • The ablation study cross-compares architectures with pretraining priors, attempting to separate the respective roles of model structure and task-generating mechanism.
Critical weaknesses
  • The overall method is closer to a composition of existing modules. The conclusions are clear and there is engineering value, but the methodological novelty is moderate.
  • The proposition "whether dense attention is necessary" is, in the experiments, actually tested as "whether a sparse structure suffices under this one constructed synthetic prior and this one training budget". The two are not equivalent. Although this is constrained by the compute budget, I do not think the paper discusses the distinction clearly.
Questions for authors
  • Beyond combining existing components, what incremental methodological contribution does the paper make to graph in-context learning, or is the incremental contribution mainly an improvement in inference efficiency?
  • Under which task conditions would the method be expected to fail? Could the authors provide a clearer and more accessible account?
3 comments
AuthorsOpenReview ↗

Author Response (1/2): Methodological Contributions

Thank you for taking the time to review our work and for your helpful feedback. We appreciate your recognition of the clarity of our research problem and our efforts to disentangle the contributions of architecture and prior design. All supporting PDFs for our rebuttal are available at:

https://anonymous.4open.science/r/ephris-rebuttal-results-E7C5/

We also provide direct links to the relevant files in our responses below.


W1/Q1. The methodological novelty is moderate.

We agree that the individual components of Ephris build on existing techniques. However, we believe its methodological novelty lies in what integration of these components make possible, rather than in the novelty of each component alone. We highlight four aspects of Ephris that support this claim.

(1) What does Ephris make possible? To the best of our knowledge, Ephris is the first PFN-based graph ICL model to eliminate dense attention entirely. This enables 13× faster inference than GraphPFN in our main experiments and allows Ephris to scale to million-node graphs, where existing graph ICL methods run out of memory.

(2) Why is this non-trivial? Removing dense attention is straightforward, but maintaining competitive predictive performance without it is challenging. Recent attempts to develop linear-complexity tabular in-context learners [1, 2] have either experienced performance degradation or retained dense attention in parts of their architectures. Ephris addresses this challenge by carefully integrating sparse message-passing components, whose importance is demonstrated by our architectural ablations (Figure 8 in the Appendix).

[1] Schnurr et al., Adapting Linear-Time Architectures for Tabular In-Context Learning, arXiv 2026.

[2] Wang et al., SOMTab, arXiv 2026.

(3) What else distinguishes Ephris? Recent strong graph ICL methods rely on pretrained tabular foundation models (TFMs). G2T-FM [3] converts graph structure into tabular features for a pretrained TFM, while GraphPFN [4] adds graph-aware attention layers to a pretrained TFM and further trains it on synthetic graphs. Thus, both approaches remain dependent on existing TFMs for their predictive performance and computational efficiency. In contrast, Ephris is pretrained end-to-end from scratch using only synthetic graphs, yet achieves competitive predictive performance. This demonstrates that graph ICL can be learned independently of existing TFMs and highlights the needs of developing architectures and pretraining strategies specifically for graph ICL.

[3] Eremeev et al., G2T-FM, ICLR 2026.

[4] Eremeev et al., GraphPFN, ICML 2026.

(4) What additional property does Ephris provide? Recent high-performing tabular and graph in-context learners, including GraphPFN, mostly use feature grouping [5, 6], which make their predictions sensitive to feature ordering. In contrast, Ephris achieves competitive predictive performance without feature grouping, preserving feature permutation invariance, as demonstrated in R9. This ensures consistent predictions regardless of feature ordering, improving the model's reliability in practice.

[5] Qu et al., TabICLv2, ICML 2026.

[6] TabPFN-3.5, 2026.


AuthorsOpenReview ↗

Author Response (2/2): Dense Attention and Model Limitations

W2. Scope of the dense-attention claim.

Thank you for highlighting this point. We agree that our original phrasing, including "whether dense attention is necessary" and "strong performance without dense cross-node attention," may suggest that removing dense attention improves both predictive performance and efficiency, which goes beyond what our experiments establish.

We acknowledge that our experiments do not isolate the effect of dense attention through matched architectural comparisons. Adding dense attention under our pretraining configuration leads to out-of-memory issues, making such a comparison infeasible within our current computational setup. Nevertheless, we agree that we cannot rule out the possibility that dense attention could further improve predictive performance.

To better reflect the scope of our findings, we will revise the research question in the abstract to "We investigate whether competitive graph in-context learning is possible without all-to-all cross-node attention." We will also revise statements throughout the paper that could imply that removing dense attention directly improves predictive performance.


Q2. Under which task conditions would Ephris be expected to fail?

Thank you for raising this important question. Based on our experiments, we identify three conditions under which Ephris may exhibit weaker predictive performance.

Distribution shift. In the submitted Appendix F, we observe performance degradation under both covariate and concept shifts across several datasets. We believe this may partly stem from our pretraining setup, where context and query nodes are sampled i.i.d., exposing the model to limited distribution shift during training. More generally, Ephris may be vulnerable to distributional differences between context and query nodes, including shifts in node features, feature–label relationships, and graph connectivity.

Scarce labeled context. During the rebuttal, we additionally evaluated Ephris on a recent few-shot graph benchmark [7], using the official fixed test nodes and 50 support episodes at one and five shots per class (R7). Ephris generally underperforms graph foundation models specifically designed for few-shot learning, highlighting its limitations when labeled context is scarce. GraphPFN shows similar limitations, suggesting that this may be a broader challenge for current PFN-based graph in-context learners. We suspect this may partly stem from our pretraining setup, which does not explicitly expose the model to such extremely low-label settings.

[7] Yu et al., Evaluating Progress in Graph Foundation Models: A Comprehensive Benchmark and New Insights, arXiv preprint, 2026.

Large feature, class, and graph dimensions. As discussed in Section 5.2, Ephris's relative performance becomes weaker on datasets with more than 5,000 features or 10 classes. Our additional experiments on four million-scale graphs (R3) also show less consistent advantages over supervised GNNs on graphs exceeding 2 million nodes. These observations identify regimes where Ephris's predictive performance is less competitive.

Overall, these observations suggest that Ephris's generalization may be limited by the coverage of its pretraining distribution, particularly under distribution shifts, scarce labeled context, and extreme task dimensions. We will explicitly discuss these limitations and their potential connections to the pretraining setup in the limitations section of the revised manuscript.


Thank you again for your thoughtful feedback and the time you have taken to review our work. We will incorporate the additional experimental results and discussions into the revised manuscript. We sincerely hope that our responses and new experiments have helped address your concerns.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Manuscript Updates

Dear Reviewer,

We would like to follow up on our responses and let you know that we have updated the manuscript to reflect the points we promised to address in our rebuttal above:

  • Scope of the dense-attention claim (Abstract, main text, and §6). We revised the research question to ask whether competitive graph in-context learning is possible without all-to-all cross-node attention. We also clarified that our results demonstrate the feasibility of a scalable architecture, while leaving open whether dense attention could further improve predictive performance.
  • Generalization limitations and pretraining coverage (§6). We explicitly discussed the weaker performance under distribution shifts, scarce labeled context, and large feature, class, and graph dimensions, along with their possible connections to the synthetic pretraining distribution.
  • Additional experimental evidence (§5.2 and Appendices F.1, F.2, and K). We incorporated the million-node results, the few-shot evaluation, and the feature-column permutation analysis to document both the strengths and the limitations discussed in our responses.

All revisions are marked in blue. We would appreciate hearing whether our responses and these revisions have helped address your concerns, and whether any points need further clarification. Thank you again for your time.

Reviewer_Z6XoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ephris, a scalable graph in-context learning framework that replaces dense cross-node attention with sparse message passing and global virtual nodes. It further introduces a diverse synthetic graph prior for pretraining. Across 51 node-classification datasets, Ephris achieves strong predictive performance without dataset-specific parameter updates.

Critical strengths
  1. The paper addresses an important scalability limitation of existing graph ICL methods.
  2. The evaluation is extensive, covering 51 datasets, 15 tuned GNNs, and six graph foundation models.
  3. The architecture–prior factorial study and empirical runtime scaling provide useful evidence for the proposed design.
Critical weaknesses
  1. Major concern is the limited methodological novelty. The main technical novelty appears to lie in extending existing graph-aware SCM pretraining with more diverse relational dynamics. Most architectural components are established techniques, including inducing attention from Set Transformers, sparse graph attention, and learnable global nodes commonly used in Graph Transformers. Their integration is effective, but the paper offers limited evidence of a fundamentally new modeling mechanism.
  2. Unfair comparisons due to label propagation. Ephris explicitly incorporates context labels into node representations and propagates them through the graph, whereas the 15 supervised GNN baselines do not use labels as input features. Comparisons with label-input GNNs, Correct and Smooth (C&S), and conventional label propagation are needed to establish whether the improvements arise from the proposed method rather than additional label information.
  3. Insufficient evaluation on truly large-scale graphs. Despite emphasizing scalability, the largest evaluated real-world graph contains approximately 569K nodes. Larger benchmarks such as ogbn-products and ogbn-papers100M are absent. Moreover, the scaling experiment fixes the feature dimension at 32, leaving computational efficiency and memory consumption on simultaneously large and high-dimensional graphs insufficiently validated.
  4. Insufficient analysis under heterophily. Since Ephris propagates context-label information through graph neighborhoods, its effectiveness may depend on the underlying label-transition patterns. Although heterophilic datasets are included, controlled experiments varying homophily from strongly positive to strongly negative, while holding feature informativeness and context ratio constant, are needed to distinguish genuine in-context task adaptation from learned label propagation.
  5. Although the paper evaluates numerous extensively tuned GNNs and several scalable Graph Transformers, its architectural baselines do not comprehensively cover recent developments from 2024–2026. Comparisons with newer scalable GNNs and Graph Transformers, particularly under matched label-conditioning and ICL pretraining protocols, would provide stronger evidence for the architectural advantages of Ephris.
  6. Marginal improvements in the low-label regime. Under the 10/10/80 split, Ephris only marginally outperforms tuned GCNII (1381 vs. 1371 Elo; 74.23% vs. 73.59% average accuracy), with a pairwise win rate of 54%. Moreover, pretraining uses context fractions of 30–90% in Stage 1 and 80% in Stage 2, leaving the 10% context setting outside the training distribution. More extensive few-shot evaluations and statistical significance analyses are needed to support its claimed generalization capabilities.
Questions for authors

Please refer to weakness.

4 comments
AuthorsOpenReview ↗

Author Response (1/3): Methodological Contributions

Thank you for taking the time to review our work and for your thoughtful feedback. We address your concerns (W1–W6) in three separate responses below. All supporting PDFs for rebuttal are available at:

https://anonymous.4open.science/r/ephris-rebuttal-results-E7C5/

We link directly to the relevant files where we discuss them below.


W1. The methodological novelty is limited.

We agree that the individual components of Ephris build on existing techniques. However, we believe its methodological novelty lies in what integration of these components make possible, rather than in the novelty of each component alone. We highlight four aspects of Ephris that support this claim.

(1) What does Ephris make possible? To the best of our knowledge, Ephris is the first PFN-based graph ICL model to eliminate dense attention entirely. This enables 13× faster inference than GraphPFN in our main experiments and allows Ephris to scale to million-node graphs, where existing graph ICL methods run out of memory.

(2) Why is this non-trivial? Removing dense attention is straightforward, but maintaining competitive predictive performance without it is challenging. Recent attempts to develop linear-complexity tabular in-context learners [1, 2] have either experienced performance degradation or retained dense attention in parts of their architectures. Ephris addresses this challenge by carefully integrating sparse message-passing components, whose importance is demonstrated by our architectural ablations (Figure 8 in the Appendix).

[1] Schnurr et al., Adapting Linear-Time Architectures for Tabular In-Context Learning, arXiv 2026.

[2] Wang et al., SOMTab, arXiv 2026.

(3) What else distinguishes Ephris? Recent strong graph ICL methods rely on pretrained tabular foundation models (TFMs). G2T-FM [3] converts graph structure into tabular features for a pretrained TFM, while GraphPFN [4] adds graph-aware attention layers to a pretrained TFM and further trains it on synthetic graphs. Thus, both approaches remain dependent on existing TFMs for their predictive performance and computational efficiency. In contrast, Ephris is pretrained end-to-end from scratch using only synthetic graphs, yet achieves competitive predictive performance. This demonstrates that graph ICL can be learned independently of existing TFMs and highlights the needs of developing architectures and pretraining strategies specifically for graph ICL.

[3] Eremeev et al., G2T-FM, ICLR 2026.

[4] Eremeev et al., GraphPFN, ICML 2026.

(4) What additional property does Ephris provide? Recent high-performing tabular and graph in-context learners, including GraphPFN, mostly use feature grouping [5, 6], which make their predictions sensitive to feature ordering. In contrast, Ephris achieves competitive predictive performance without feature grouping, preserving feature permutation invariance, as demonstrated in R9. This ensures consistent predictions regardless of feature ordering, improving the model's reliability in practice.

[5] Qu et al., TabICLv2, ICML 2026.

[6] TabPFN-3.5, 2026.


AuthorsOpenReview ↗

Author Response (2/3): Label Use and Scalability

W2. The comparisons are unfair due to differences in label usage.

We believe our comparison between Ephris and supervised GNNs is fair for the following reasons.

First, both Ephris and supervised GNNs have access to the same training labels. Supervised GNNs use these labels to update their parameters, while Ephris uses them as inference-time context. This reflects a difference in how the methods use the available information, rather than an unfair advantage in information access. In fact, our evaluation protocol favors supervised GNNs, as they additionally use validation labels for early stopping and hyperparameter tuning, whereas Ephris does not use validation labels at all.

Second, we made substantial efforts to ensure that our supervised GNN baselines are competitive. We trained and tuned each GNN separately for every dataset–split pair, evaluating up to 200 hyperparameter configurations. Our search also included architectural choices such as residual connections and normalization, which have been shown to substantially improve node classification performance [7]. As a result, these tuned GNNs provide strong baselines, outperforming prior graph foundation models that had previously reported superior performance over supervised GNNs under comparable evaluation protocol.

[7] Luo et al., Classic GNNs are Strong Baselines, NeurIPS 2024.

Finally, to directly address the reviewer's concern about label usage, we additionally evaluated GCNII and GPRGNN, the two strongest supervised baselines in our main experiments, with label inputs or Correct & Smooth (C&S). These variants improved performance on some datasets, but did not consistently outperform the original baselines. Ephris still ranks first in three aggregate metrics (Elo, mean improvability, and average rank) under both label regimes, supporting the main observations of our original comparison. Full results are available in R5.


W3. Evaluation on large-scale graphs is insufficient.

We additionally evaluated Ephris on four million-scale datasets: pokec-regions, web-topics, fraud, and ogbn-products [8], using their official splits. Full results are available in R3.

Ephris successfully completes inference on all four datasets, whereas both NodePFN and GraphPFN encounter out-of-memory errors on every dataset, preventing direct performance comparisons. These results highlight Ephris's computational efficiency, enabling it to scale to million-node graphs that existing graph ICL methods cannot handle.

In terms of predictive performance, the results are mixed. Ephris outperforms all reported baselines across four splits on pokec-regions (1.6M nodes). On ogbn-products (2.45M nodes), it achieves 79.80% accuracy, outperforming GraphSAGE but falling behind Polynormer [9]. Its relative performance is weaker on web-topics and fraud (2.89M nodes), where it does not outperform published baselines.

These results suggest that, although Ephris scales efficiently to million-node graphs, its predictive advantage over supervised GNNs becomes less consistent on graphs with more than 2M nodes. We thank the reviewer for motivating this evaluation and will explicitly acknowledge this limitation of the current model in the revised paper.

[8] Hu et al., Open Graph Benchmark: Datasets for Machine Learning on Graphs, NeurIPS 2020.

[9] Deng et al., Polynormer: Polynomial-Expressive Graph Transformer in Linear Time, ICLR 2024.

To address the 32-feature limit of our original scaling experiment, we additionally varied the number of nodes and features jointly, up to 1 million nodes and 4,096 features (R4). Ephris successfully completed all 56 tested configurations across three seeds, whereas NodePFN was limited by its architecture's supported feature dimensionality and GraphPFN encountered out-of-memory errors. Furthermore, the measured inference times demonstrate approximately linear scaling with both the number of nodes and features, further supporting Ephris's scalability.


AuthorsOpenReview ↗

Author Response (3/3): Context Adaptation and Low-Label Performance

W4. Analysis distinguishing in-context task adaptation from learned label propagation is missing.

To better understand how Ephris uses context labels, we conduct three additional analyses (R8).

First, we examine performance across 51 real-world datasets grouped by label homophily. Ephris consistently outperforms label propagation across all homophily levels, suggesting that its predictive behavior goes beyond simple label propagation.

Second, we investigate how performance changes as more labeled context becomes available across graphs with different homophily levels. Ephris consistently benefits from additional context labels, showing improvements comparable to those of GNN baselines regardless of graph homophily. This demonstrates its ability to effectively leverage additional context across diverse homophily levels.

Third, we conduct controlled synthetic experiments to examine whether Ephris adapts to different labeling tasks using context labels. Given the same graph and features, Ephris successfully handles both homophilic and heterophilic label relationships without parameter updates. It also leverages node features when structural signals are weakened, showing that its predictions are not solely driven by label propagation.

Together, these results suggest that Ephris uses context labels to adapt its predictions to the given task, effectively drawing on both graph structure and node features.


W5. Comparisons with recent architectures are missing.

We would like to clarify that the primary goal of Ephris is to make graph in-context learning more scalable with strong predictive performance. Accordingly, our main comparisons focus on graph foundation models (GFMs), particularly prior graph in-context learners that address the same problem. All six GFM baselines were published between 2024 and 2026, and our experiments demonstrate Ephris's consistent advantages in both predictive performance and scalability over these recent methods.

We also note that adapting recent per-dataset graph architectures to graph ICL is non-trivial. Graph ICL requires a single parameterization to handle varying feature dimensions and numbers of classes, whereas most GNN architectures assume fixed input and output dimensions. Therefore, evaluating these architectures under the ICL protocol would require architectural modifications and a suitable pretraining strategy, rather than a straightforward comparison.


W6. Improvements in the low-label regime are marginal.

Thank you for raising this point. We conducted a paired dataset bootstrap analysis to quantify uncertainty in Ephris's performance gains over leading baselines under both label regimes (R6). In the high-label regime, all 95% confidence intervals lie entirely above zero, favoring Ephris. In the low-label regime, however, intervals for several strong GNN baselines, including GCNII and GPRGNN, include zero. We will clarify in the revised manuscript that Ephris's advantage over the strongest supervised GNNs is less conclusive under low labels.

We also evaluated Ephris and GraphPFN using frozen checkpoints on few-shot node classification benchmark [10], with one or five labeled examples per class (R7). Although Ephris and GraphPFN achieve competitive mean accuracy in several dataset–shot settings, published baselines outperform both models in eight of the 12 settings. These results indicate that both graph in-context learners generally fall behind methods specifically designed for few-shot graph learning. We will explicitly acknowledge this limitation of current graph in-context learners in the revised manuscript.

[10] Yu et al., Evaluating Progress in Graph Foundation Models: A Comprehensive Benchmark and New Insights, arXiv preprint, 2026.


Thank you again for your thoughtful feedback and the time you have taken to review our work. We will incorporate the additional experimental results and discussions into the revised manuscript. We sincerely hope that our responses and new experiments have helped address your concerns.

AuthorsOpenReview ↗

Follow-up on Our Responses and Manuscript Revision

Dear Reviewer,

We would like to follow up on our responses and let you know that we have updated the manuscript to reflect the points we promised to address in our rebuttal above:

  • Low-label performance (§5.1). We included the paired bootstrap analysis and clarified that, in the low-label regime, the confidence intervals against the strongest baselines, GCNII and GPRGNN, include zero, making Ephris’s advantage less conclusive.
  • Million-node graphs (§5.2 and §6). We added experimental results on four million-node datasets and explicitly stated as a limitation that Ephris’s predictive gains over supervised GNNs are less consistent across these datasets.
  • Few-shot limitations (§6 and Appendix F.2). We explicitly acknowledged that graph ICL methods, including Ephris, face limitations in few-shot learning and included the GFMBenchmark results in the appendix.

All revisions are marked in blue. We would appreciate hearing whether our responses and these revisions have helped address your concerns, and whether any points need further clarification. Thank you again for your time.

Reviewer_M8bcOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Ephris, a graph ICL method for node classification pretrained on synthetic data. The key idea of Ephris is to remove quadratic dense attention and instead use only message passing and a form of induced self-attention. This allows Ephris to achieve linear inference time complexity in the number of nodes, while preserving competitive downstream performance. In particular, on the considered collection of 51 datasets, Ephris achieves the best Elo score, improvability, and average rank, while having an inference time comparable to that of a single GNN training run.

Critical strengths
  1. The core idea and results are novel and should be interesting to the community.

While I believe that Ephris with dense attention would show even better results, showing that competitive downstream performance can be achieved even without dense attention is a solid contribution. It might not only offer better trade-offs for practitioners, but also help the community better understand how graph ICL models work.

Critical weaknesses
  1. Dataset selection and result reporting

First, the work mostly evaluates on classic graph datasets, which have been widely criticized for their lack of real-world applications and narrow coverage of domains and tasks [1, 2]. Examples include citation networks like cora, citeseer, and pubmed, where the task is likely to be better solved by NLP methods than by GraphML methods, or transport datasets like la, london, paris, and shanghai, where the target is computed by a deterministic algorithm, so there is no need to apply ML methods.

Second, while Ephris claims better scalability, the evaluation is limited to datasets with at most 568,795 nodes. Evaluating on graphs with millions of nodes could strengthen the work and better support the scalability claim.

Third, I have glanced through the appendices and the code repository and found no raw per-dataset results. Please correct me if I missed something, or provide these results (at least for Ephris, GraphPFN, and GCNII), since they are important for obtaining a better and more nuanced picture of a method's performance.

To address this concern, I encourage the authors to evaluate Ephris on all 7 GraphLand classification datasets (with the original splits and metrics), provide their raw per-dataset results, and compare them with the GNN and GraphPFN (on 4/7 datasets) results from the original papers. In my opinion, such a comparison would greatly strengthen the work, even if Ephris does not outperform GraphPFN or GNNs.

  1. Regression is not supported

Graph ICL methods for node-level tasks have historically focused on node classification due to a lack of regression datasets, but GraphLand recently introduced 7 real-world regression datasets, and more recent graph ICL methods like GraphPFN support node regression. Moreover, modern tabular foundation models like TabPFN-3.5, TabICLv2, or Kumo-Tabular support both classification and regression (though some of them have separate checkpoints for different tasks). Thus, I believe the lack of regression support is a substantial limitation of this work.

  1. Positioning requires further qualification

Specifically, the paper argues that (with similar claims also appearing in the abstract and introduction):

... strong ICL can be achieved through message passing over the observed graph without dense cross-node attention.

I believe that some readers might interpret this claim as "using message passing without dense attention improves both performance and inference time." However, the paper provides no controlled comparison of models with and without dense attention, with all other design choices fixed. Thus, I encourage the authors to either explicitly mention that dense attention may improve Ephris's performance even further or conduct an experiment that adds dense attention to Ephris and shows that it does not improve performance.

But generally, I like this work and would be happy to increase my score once my concerns are addressed.


[1] Bechler-Speicher et al., "Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks," ICML 2025.

[2] Bazhenov et al., "GraphLand: Evaluating Graph Machine Learning Models on Diverse Industrial Data," NeurIPS 2025.

Questions for authors

See weaknesses.

Minor comments and suggestions

These comments have no effect on my score, and I do not require the authors to address them.

  1. I hypothesize that the idea of replacing dense attention with message passing may have connections with classic graph learning methods like label propagation or the "labels as features" trick in graph neural networks. The authors might consider discussing this connection. This connection might also help us better understand why and how Ephris works.
5 comments
AuthorsOpenReview ↗

Author Response (1/2): Datasets, Scalability, and Raw Results

Thank you for your thoughtful review and detailed feedback. We are glad that you found our ideas and results novel and interesting. We address your concerns in two separate responses below.

All supporting PDFs and raw results are available at:

https://anonymous.4open.science/r/ephris-rebuttal-results-E7C5/

We link directly to the relevant files where we discuss them below.


W1a. The datasets have limited real-world relevance.

Our dataset selection was motivated by the importance of generalizing across graphs with diverse feature and structural properties. We therefore included datasets covering a wide range of graph characteristics, as shown in R1. While NLP-based methods or specialized algorithms may perform better on individual datasets, we believe this collection still provides a useful testbed for understanding where graph foundation models generalize well and where they struggle.

We also agree that evaluation on more application-oriented datasets would help better assess Ephris's performance. We therefore evaluated Ephris on all 26 classification dataset–split conditions in GraphLand [1] (results at R2). We used published baseline results and ran GraphPFN [2] on conditions not covered in its original paper.

Notably, Ephris successfully handles all 26 dataset–split conditions, whereas GraphPFN encounters out-of-memory errors on 12 of them, even on an H200 GPU (140 GB VRAM). Despite this difference in scalability, Ephris remains competitive, outperforming the strongest GraphLand baseline in 13 out of 26 conditions and GraphPFN in 6 out of the 14 conditions where its results are available. These results highlight Ephris's scalability and competitive performance on industrial datasets.

[1] Bazhenov et al., GraphLand, NeurIPS 2025.
[2] Eremeev et al., GraphPFN, ICML 2026.

W1b. Scalability has not been evaluated on million-node graphs.

To address this concern, we evaluated Ephris on four million-scale datasets: pokec-regions, web-topics, fraud, and ogbn-products [3] (results at R3). The first three are included in the GraphLand evaluation discussed in W1a, while ogbn-products was evaluated using its official split.

Ephris successfully handles all four million-scale datasets, whereas both NodePFN and GraphPFN encounter out-of-memory errors. This demonstrates that Ephris can scale to graphs that existing graph ICL methods cannot handle.

In terms of predictive performance, Ephris outperforms all reported baselines across four splits on pokec-regions (1.6M nodes). On ogbn-products (2.45M nodes), it achieves 79.80% accuracy, outperforming GraphSAGE but falling behind Polynormer [4]. Its relative performance becomes weaker on web-topics and fraud (2.89M nodes), where it does not outperform GCN. Overall, these results show that Ephris can handle million-scale graphs while remaining competitive on some of them, although improving predictive performance on larger graphs remains an important direction.

[3] Hu et al., Open Graph Benchmark, NeurIPS 2020.
[4] Deng et al., Polynormer, ICLR 2024.

W1c. Per-dataset results are missing.

We agree that transparent reporting is important. The raw results include the Table 13 summary and per-split results for Ephris and all 36 baseline configurations across 51 datasets. We also plan to package and release our benchmarking code as a standalone project after the rebuttal period, so that other researchers can reproduce our evaluations and benchmark new methods. We hope these resources will support future research on scalable and generalizable graph foundation models.


AuthorsOpenReview ↗

Author Response (2/2): Regression and Label-Based Learning

W2. Regression is not supported.

We focused on node classification in this work because we believe it provides a sufficient setting to demonstrate our main contributions: strong predictive performance with linear scalability and the feasibility of pretraining a graph in-context learner from scratch.

As the reviewer pointed out, tabular foundation models commonly use separate checkpoints for classification and regression. We expect that Ephris could be extended to regression in a similar way by changing the prediction head and training objective, for example, using the 999-quantile prediction head and pinball loss adopted in TabICLv2 [5].

However, training a separate regression checkpoint would require substantial computational resources. We therefore cannot commit to including this experiment in the revision, but will explicitly acknowledge the lack of regression checkpoint as a limitation of our work.

[5] Qu et al., TabICLv2, ICML 2026.


W3. Positioning requires further qualification.

Thank you for highlighting this point. We agree that our original phrasing, including "whether dense attention is necessary" and "strong performance without dense cross-node attention," may suggest that removing dense attention improves both predictive performance and efficiency, which goes beyond what our experiments establish.

We acknowledge that our experiments do not isolate the effect of dense attention through matched architectural comparisons. Adding dense attention under our pretraining configuration leads to out-of-memory issues, making such a comparison infeasible within our current computational setup. Nevertheless, we agree that we cannot rule out the possibility that dense attention could further improve predictive performance.

To better reflect the scope of our findings, we will revise the research question in the abstract to "We investigate whether competitive graph in-context learning is possible without all-to-all cross-node attention." We will also revise statements throughout the paper that could imply that removing dense attention directly improves predictive performance.


Q1. Connections to label-based graph learning.

Label-based graph learning methods, such as UniMP [6] and Correct & Smooth (C&S) [7], share a similar form with graph ICL: leveraging node features and observed labels together to predict missing labels. However, they differ in two key aspects. First, these methods follow the conventional supervised graph learning paradigm, aiming to improve performance on a given dataset through dataset-specific training and tuning. In contrast, graph ICL is pretrained across diverse graphs to make predictions on unseen datasets without additional training. Thus, they differ in both how they are trained and where they can be applied. Second, in terms of architecture, these methods typically rely on models with fixed feature dimensions and numbers of classes, whereas graph ICL architectures such as Ephris can handle varying feature dimensions and label spaces within a single parameterization. We will add this discussion to the related work section of the revised manuscript.

[6] Shi et al., UniMP, IJCAI 2021.
[7] Huang et al., Correct & Smooth, ICLR 2021.


Thank you again for your time and feedback. We will incorporate the additional results on GraphLand and ogbn-products, along with the discussion of label-based learning and the limitations in scalability and regression, into the revised manuscript. We hope our clarifications and experiments have addressed your concerns.

Dear Authors,

Thank you for your detailed response. Assuming that you will faithfully implement all of your commitments, I think my concerns are addressed, so I will increase my score.

Below are some minor comments that do not affect my decision and do not require an answer:

  1. I believe Ephris' results on GraphLand are very strong for a GFM pretrained from scratch and working without dense attention. However, as you already say, Ephris is competitive with GraphPFN rather than substantially outperforming it, unlike in your main experiments. I encourage you to include the GraphLand results in the main text, and not just in the appendix, to provide a more balanced and nuanced picture of Ephris' performance.

  2. Regarding dense attention, it's rather surprising that you hit OOM. I've compared the dataset sizes used in your Stage 1 pretraining and GraphPFN pretraining, and it seems that they are comparable. Did you use efficient attention kernels (like FlashAttention) to avoid quadratic memory consumption? Did you try reducing the batch size, perhaps combined with gradient accumulation? It would be very interesting to compare your current model against a matched model with dense attention on both GraphLand and your 51-dataset collection.

  3. Regarding label propagation, thank you for the clarification, but I believe that the distinctions between LP and graph ICL are rather obvious. In particular, there is no need to add a discussion of these distinctions to the revised paper just to address my question. I was more interested in their similarities: Ephris adds target embeddings to context nodes and then performs message passing. I believe this specific procedure is very similar to label propagation, and some form of label propagation may contribute to Ephris' performance and serve as a motivation. To be clear, I do not assume that Ephris is limited to label propagation, but I do hypothesize that label propagation can be part of Ephris' internals.

Best regards,

Reviewer M8bc

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.