- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether dense cross-node attention is necessary for graph in-context learning, and presents a graph ICL architecture based on sparse message passing: each scalar feature is tokenized and refined by per-feature and per-node induced attention blocks into a fixed-length node representation; message-passing blocks then perform ICL (each block contains local attention, attention scaled by neighborhood size, and global node and residual propagation); finally, an output head built on fixed class embeddings handles an arbitrary number of classes.
Critical strengths
- The paper raises a clear question: whether graph in-context learning must rely on dense cross-node attention. Handling context and query nodes with sparse message passing on the observed graph is a clear methodological route.
- The ablation study cross-compares architectures with pretraining priors, attempting to separate the respective roles of model structure and task-generating mechanism.
Critical weaknesses
- The overall method is closer to a composition of existing modules. The conclusions are clear and there is engineering value, but the methodological novelty is moderate.
- The proposition "whether dense attention is necessary" is, in the experiments, actually tested as "whether a sparse structure suffices under this one constructed synthetic prior and this one training budget". The two are not equivalent. Although this is constrained by the compute budget, I do not think the paper discusses the distinction clearly.
Questions for authors
- Beyond combining existing components, what incremental methodological contribution does the paper make to graph in-context learning, or is the incremental contribution mainly an improvement in inference efficiency?
- Under which task conditions would the method be expected to fail? Could the authors provide a clearer and more accessible account?
3 comments
Author Response (2/2): Dense Attention and Model Limitations
W2. Scope of the dense-attention claim.
Thank you for highlighting this point. We agree that our original phrasing, including "whether dense attention is necessary" and "strong performance without dense cross-node attention," may suggest that removing dense attention improves both predictive performance and efficiency, which goes beyond what our experiments establish.
We acknowledge that our experiments do not isolate the effect of dense attention through matched architectural comparisons. Adding dense attention under our pretraining configuration leads to out-of-memory issues, making such a comparison infeasible within our current computational setup. Nevertheless, we agree that we cannot rule out the possibility that dense attention could further improve predictive performance.
To better reflect the scope of our findings, we will revise the research question in the abstract to "We investigate whether competitive graph in-context learning is possible without all-to-all cross-node attention." We will also revise statements throughout the paper that could imply that removing dense attention directly improves predictive performance.
Q2. Under which task conditions would Ephris be expected to fail?
Thank you for raising this important question. Based on our experiments, we identify three conditions under which Ephris may exhibit weaker predictive performance.
Distribution shift. In the submitted Appendix F, we observe performance degradation under both covariate and concept shifts across several datasets. We believe this may partly stem from our pretraining setup, where context and query nodes are sampled i.i.d., exposing the model to limited distribution shift during training. More generally, Ephris may be vulnerable to distributional differences between context and query nodes, including shifts in node features, feature–label relationships, and graph connectivity.
Scarce labeled context. During the rebuttal, we additionally evaluated Ephris on a recent few-shot graph benchmark [7], using the official fixed test nodes and 50 support episodes at one and five shots per class (R7). Ephris generally underperforms graph foundation models specifically designed for few-shot learning, highlighting its limitations when labeled context is scarce. GraphPFN shows similar limitations, suggesting that this may be a broader challenge for current PFN-based graph in-context learners. We suspect this may partly stem from our pretraining setup, which does not explicitly expose the model to such extremely low-label settings.
[7] Yu et al., Evaluating Progress in Graph Foundation Models: A Comprehensive Benchmark and New Insights, arXiv preprint, 2026.
Large feature, class, and graph dimensions. As discussed in Section 5.2, Ephris's relative performance becomes weaker on datasets with more than 5,000 features or 10 classes. Our additional experiments on four million-scale graphs (R3) also show less consistent advantages over supervised GNNs on graphs exceeding 2 million nodes. These observations identify regimes where Ephris's predictive performance is less competitive.
Overall, these observations suggest that Ephris's generalization may be limited by the coverage of its pretraining distribution, particularly under distribution shifts, scarce labeled context, and extreme task dimensions. We will explicitly discuss these limitations and their potential connections to the pretraining setup in the limitations section of the revised manuscript.
Thank you again for your thoughtful feedback and the time you have taken to review our work. We will incorporate the additional experimental results and discussions into the revised manuscript. We sincerely hope that our responses and new experiments have helped address your concerns.
Follow-up on Our Rebuttal and Manuscript Updates
Dear Reviewer,
We would like to follow up on our responses and let you know that we have updated the manuscript to reflect the points we promised to address in our rebuttal above:
- Scope of the dense-attention claim (Abstract, main text, and §6). We revised the research question to ask whether competitive graph in-context learning is possible without all-to-all cross-node attention. We also clarified that our results demonstrate the feasibility of a scalable architecture, while leaving open whether dense attention could further improve predictive performance.
- Generalization limitations and pretraining coverage (§6). We explicitly discussed the weaker performance under distribution shifts, scarce labeled context, and large feature, class, and graph dimensions, along with their possible connections to the synthetic pretraining distribution.
- Additional experimental evidence (§5.2 and Appendices F.1, F.2, and K). We incorporated the million-node results, the few-shot evaluation, and the feature-column permutation analysis to document both the strengths and the limitations discussed in our responses.
All revisions are marked in blue. We would appreciate hearing whether our responses and these revisions have helped address your concerns, and whether any points need further clarification. Thank you again for your time.
Author Response (1/2): Methodological Contributions
Thank you for taking the time to review our work and for your helpful feedback. We appreciate your recognition of the clarity of our research problem and our efforts to disentangle the contributions of architecture and prior design. All supporting PDFs for our rebuttal are available at:
https://anonymous.4open.science/r/ephris-rebuttal-results-E7C5/
We also provide direct links to the relevant files in our responses below.
W1/Q1. The methodological novelty is moderate.
We agree that the individual components of Ephris build on existing techniques. However, we believe its methodological novelty lies in what integration of these components make possible, rather than in the novelty of each component alone. We highlight four aspects of Ephris that support this claim.
(1) What does Ephris make possible? To the best of our knowledge, Ephris is the first PFN-based graph ICL model to eliminate dense attention entirely. This enables 13× faster inference than GraphPFN in our main experiments and allows Ephris to scale to million-node graphs, where existing graph ICL methods run out of memory.
(2) Why is this non-trivial? Removing dense attention is straightforward, but maintaining competitive predictive performance without it is challenging. Recent attempts to develop linear-complexity tabular in-context learners [1, 2] have either experienced performance degradation or retained dense attention in parts of their architectures. Ephris addresses this challenge by carefully integrating sparse message-passing components, whose importance is demonstrated by our architectural ablations (Figure 8 in the Appendix).
[1] Schnurr et al., Adapting Linear-Time Architectures for Tabular In-Context Learning, arXiv 2026.
[2] Wang et al., SOMTab, arXiv 2026.
(3) What else distinguishes Ephris? Recent strong graph ICL methods rely on pretrained tabular foundation models (TFMs). G2T-FM [3] converts graph structure into tabular features for a pretrained TFM, while GraphPFN [4] adds graph-aware attention layers to a pretrained TFM and further trains it on synthetic graphs. Thus, both approaches remain dependent on existing TFMs for their predictive performance and computational efficiency. In contrast, Ephris is pretrained end-to-end from scratch using only synthetic graphs, yet achieves competitive predictive performance. This demonstrates that graph ICL can be learned independently of existing TFMs and highlights the needs of developing architectures and pretraining strategies specifically for graph ICL.
[3] Eremeev et al., G2T-FM, ICLR 2026.
[4] Eremeev et al., GraphPFN, ICML 2026.
(4) What additional property does Ephris provide? Recent high-performing tabular and graph in-context learners, including GraphPFN, mostly use feature grouping [5, 6], which make their predictions sensitive to feature ordering. In contrast, Ephris achieves competitive predictive performance without feature grouping, preserving feature permutation invariance, as demonstrated in R9. This ensures consistent predictions regardless of feature ordering, improving the model's reliability in practice.
[5] Qu et al., TabICLv2, ICML 2026.
[6] TabPFN-3.5, 2026.