acceptodds
Under review as a conference paper at ICLR 2027

Looping in Context: Node Classification with Looped Prior-Fitted Networks

Abstract

Prior-fitted networks (PFNs) for graphs classify nodes on arbitrary graphs in a single forward pass, yet how their predictions form across depth is unknown. For tabular foundation models, mechanistic work shows that depth is iterative *refinement*, so one transformer block, looped, recovers a full-depth model. We ask whether this holds for graphs, where every layer also performs one hop of *propagation*. Six mechanistic experiments on the 12-layer NodePFN over 23 benchmarks show that depth plays two roles. Attention transfers the labels in the first layer and is redundant and self-repairing in later layers. Message passing stays necessary on homophilous graphs, where the loss from a removed hop persists to the last layer. The depth at which predictions form therefore grows with homophily, in NodePFN and in a second PFN for graphs, GraphPFN, and a two-class analysis explains why no fixed depth can be optimal across homophily levels. We propose **Looped NodePFN**, a single block applied repeatedly, whose depth is selected per graph *in context* from held-out labeled nodes, with no gradient steps and no extra parameters. With a tenth of the parameters it reaches 66.3% mean accuracy against 66.6% for the 12-layer model, and a two-layer block reaches 66.5%. Once its depth is chosen, the one-layer block predicts up to 5.1 times faster. A PFN for graphs thus needs only one block, but how many times it is applied, and in how many of those iterations it passes messages, should follow the graph.

Primary area:
foundation or frontier models, including LLMs
Keywords:
prior-fitted networks, graph foundation models, node classification, in-context learning, looped transformers, adaptive depth, mechanistic interpretability, homophily

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8QhiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how depth is used in prior-fitted networks for node classification, mainly NodePFN (12 layers), with a repeat on GraphPFN. It borrows the mechanistic protocol of Balef et al. (2026) for tabular PFNs (CKA, probes, logit lens, layer skipping and self-repair) and adds branch-wise ablations, since each NodePFN layer has an attention branch and a message-passing branch. The main findings are:

  • the first layer's attention carries the labels to the queries, and later attention is redundant and self-repairing;
  • message passing matters mostly on homophilous graphs;
  • so the depth at which the prediction is "ready" grows with homophily.

A two-class cSBM analysis is used to argue that no fixed depth is optimal across homophily levels. Based on this, the authors train a single NodePFN block that is looped (Looped NodePFN). The number of iterations is chosen per graph by holding out part of the labeled context (ICDS). With about a tenth of the parameters, the looped model gets close to the 12-layer model on average and is faster at prediction time once the depth has been selected.

Critical strengths
  1. The mechanistic study is well designed. Zeroing one branch of one layer is a neat way to separate "refinement" from "propagation", and the self-repair experiment answers a question that plain ablation cannot.

  2. The finding that the tuned pre-smoothing in the official NodePFN pipeline already does part of the propagation, and therefore hides the depth/homophily relation, is a nice and practically relevant observation. People evaluating graph PFNs should know about it.

  3. The paper is very transparent. It has a threats-to-validity section, an explicit failure case for ICDS, oracle numbers shown separately and unranked, sensitivity tables for ICDS, released code and checkpoints, and a hypothesis stated before the experiments. This is good practice.

  4. The negative control (a 1-layer model trained without looping collapses when looped at inference) is convincing and makes the point that looping must be trained for.

Critical weaknesses

My major concerns are:

  • the method itself is quite simple (validation-based selection of one hyperparameter);
  • the headline "no loss in accuracy" hides a clear loss on heterophilous graphs and large per-graph swings;
  • the depth-follows-homophily story does not hold in the setting where the headline numbers are reported;
  • the theory overlaps with known oversmoothing results on CSBMs that are not cited;
  • the paper is not clearly positioned against existing low-cost GNN methods that already address homophily and over-squashing (e.g., DeltaGNN, spectral graph pruning, curvature- and community-based rewiring).
  • The paper is hard to read because the text is overloaded with numbers and implementation details.

A detailed feedback is included below:

  1. "No loss in accuracy" is only true on average.

    • The looped models are better on homophilous graphs and worse on heterophilous ones, and the two roughly cancel in the overall mean. On heterophilous graphs the gap is about two points under ICDS, and the paper itself says even the oracle depth does not close it.
    • The per-dataset table (Table 2) also shows very large swings in both directions.
    • I would like the abstract and contributions to state the heterophilous loss clearly. The paper should also explain these outliers, in particular the below-chance result on questions.
  2. The depth does not follow homophily in the headline setting.

    • The main claim is that the right depth is a property of the graph and that ICDS picks it up. But under the standard protocol, the chosen depth is unrelated to homophily (the correlation is even slightly negative for Loop-1L).
    • The relation only shows up without pre-smoothing, and there ICDS is below NodePFN-12L.
    • So in the setting where the method matches the baseline, the motivation does not apply. In the setting where the motivation applies, the method does not match the baseline. The paper acknowledges this in Appendix G, but the main text and abstract read as if both hold at once.
  3. ICDS is essentially validation-based model selection.

    • The authors themselves describe ICDS as cross-validating one hyperparameter within the forward pass. That is a reasonable thing to do, but it is not much of a methodological contribution.
    • Table 7 shows that a single global depth already recovers about half of ICDS's gain under the standard protocol. That global depth is chosen with test labels, so it is optimistic, but it shows how much of the gain comes simply from not using depth 12.
  4. The efficiency claims need more context: fewer parameters, but not less compute.

    • Training is not cheaper. The "10× fewer parameters" headline may suggest a cheaper model, but in floating-point operations pretraining Loop-1L costs practically the same as pretraining NodePFN-12L. The paper says so itself in Appendix E: a block applied twelve times costs as much computation as twelve layers. This needs to be clarified. What is the true gain here? This message should be consistent throughout the paper.
      • Training memory drops only by about a quarter, not by ten times.
      • The authors attribute the small remaining saving to updating fewer parameters, not to doing less computation.
    • The savings are in storage only. The parameter reduction does reduce checkpoint size (12 MB against 116 MB). The ethics statement says looping reduces the memory needed "to store and train" a PFN; that holds for storage, but only marginally for training.
    • Inference is not cheaper per iteration either. One looped iteration costs about the same as one layer. The prediction speed-up comes only from ICDS choosing fewer than twelve iterations. On the graphs where ICDS selects twelve or more iterations, there is no speed-up, and on two graphs the looped model is slower (Appendix E, "Exceptions").
    • ICDS cost is excluded. The "up to 5.1 times faster" figure leaves out ICDS, which runs 20 × Tmax block applications once per graph. For a single prediction, Looped NodePFN is slower than NodePFN-12L.
    • Suggested wording. The parameter reduction is real and useful for storage and deployment, but the abstract and contributions should say clearly that it does not reduce training or per-iteration compute. Any speed-up only applies when many predictions are made on the same graph.
  5. Proposition 1 is close to known results that are not cited.

    • The idea that on a CSBM there is a finite optimal number of propagation steps, which depends on homophily and degree, and is zero when the graph signal is too weak, has been studied before. Examples are Wu et al. (ICLR 2023, "A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks") and Keriven (NeurIPS 2022, "Not too little, not too much: a theoretical analysis of graph (over)smoothing"). Baranwal et al. is cited, but only for the one-step case.
    • Please position Proposition 1 against these works and say what is new.
    • The proof is labeled a "proof sketch" and relies on several approximation steps. A full proof, or at least a clear list of what is rigorous and what is heuristic, would help.
    • The result is also for two balanced classes, while most benchmarks have more. The multi-class case is only argued in a remark.
  6. Positioning against cheap methods that already handle homophily and over-squashing.

The paper's central message is that how much a model should propagate depends on the graph's homophily, and that this can be decided cheaply per graph. That message is not new on its own. Several recent works study the same tension between homophily, over-smoothing and over-squashing, and propose solutions with low computational cost. Most of them are not cited or are only mentioned in passing:

  • Spectral graph pruning (Jamadandi et al., NeurIPS 2024). Deleting edges can reduce over-squashing and over-smoothing at the same time, using a cheap spectral-gap criterion. It is evaluated on large heterophilous graphs.
  • ComFy (Rubio-Madrigal et al., ICLR 2025). Rewiring guided by communities and feature similarity. It shows that the usual over-squashing fix (maximizing the spectral gap) hurts on homophilous graphs, which is close in spirit to this paper's "the right amount of propagation depends on h".
  • DetlaGNN (e.g., Mancini et al., DeltaGNN TPAMI 2026). Oversmoothing & oversquashing detection and fixing across varied topologies.
  • AFRC rewiring (Fesser & Weber, LoG 2023). A linear-time curvature measure that characterizes both over-smoothing and over-squashing, with heuristics that avoid expensive hyperparameter searches.
  • Adaptive-depth and adaptive-propagation GNNs. Hevapathige et al. (AAAI 2026) select node-specific depths based on local homophily. GPR-GNN learns per-graph propagation weights. Both are cited, but only in one sentence, without discussion of how ICDS relates to them.

These methods differ from the paper because they are trained on the target graph or applied as preprocessing, whereas a PFN is not trained on the target graph. But that is exactly why they matter here. The paper itself shows that NodePFN's pre-smoothing already moves part of the propagation into preprocessing. A cheap, homophily-aware rewiring or edge-filtering step applied before NodePFN is therefore a natural alternative to looping plus depth selection, and should be discussed or tried as a baseline.

I would like the authors to state clearly what is unique to their work with respect to these methods. Is it:

  • the mechanistic finding about NodePFN's branches;
  • the observation that pre-smoothing hides the depth/homophily relation;
  • the parameter reduction from looping; or
  • the training-free depth selection from labeled context?

At the moment the paper mainly positions itself against tabular PFNs and looped transformers, and much less against the GNN literature on homophily and over-squashing, which is where most of the conceptual overlap lies.

  1. The generality claim is broader than the evidence.

    • The title and conclusion say a single block suffices for PFNs for graphs. But looping is only done for NodePFN, and GraphPFN is only analyzed, not looped.
    • All results are single-pass without ensembling, while both models are normally used with ensembles. It is not clear whether the conclusions (in particular the parity in accuracy) hold in the ensembled setting people actually use.
  2. Some reporting is confusing.

    • The per-dataset averages in Table 2 mix accuracy and ROC-AUC (for the four majority-level graphs), and they do not match the split-0 numbers in Table 7 and Appendix E. It took me a while to figure out which numbers belong to which setting.
    • Including four graphs where the model is at majority-class accuracy in the "All (23)" accuracy means also dilutes the comparison.
    • I suggest reporting all averages with one metric over the informative graphs, and stating the setting (protocol, split, metric) in every caption.
  3. Weight tying turns depth into repeated diffusion that the model cannot switch off.

In Looped NodePFN, each iteration is not pure refinement: it applies the same block, including its message-passing branch. Every extra iteration is therefore one more hop of diffusion with the same weights. The paper states this itself ("every iteration propagates", Section 5; "every iteration is also a hop of propagation", Appendix F).

This has two consequences that I think deserve more discussion:

(1) The looped model loses an ability the 12-layer model has. In NodePFN-12L, the later layers can effectively stop propagating. Their message-passing branches are nearly inactive (Section 3.3), and the Dirichlet energy rebounds after layer 2. In the looped model the energy falls at every iteration and never recovers (Fig. 14). Weight tying removes the freedom to refine without diffusing.

(2) I would like the authors to separate "how long to refine" from "how far to propagate" in the architecture itself, rather than only through test-time selection. Examples are a learned gate on the message-passing branch per iteration, or an initial-residual term as in APPNP/GCNII. They should also report whether this removes the heterophilous gap.

Questions for authors
  1. Can you report the main comparison with ensembling (as NodePFN and GraphPFN are normally used)? Does the parity in accuracy still hold?
  2. How does Proposition 1 differ from the CSBM oversmoothing analyses of Wu et al. (2023) and Keriven (2022)?
  3. Under the standard protocol, the chosen depth does not follow homophily. What is ICDS adapting to in that setting?
  4. Can you add training-seed variance for the looped models, at least for Loop-1L?
  5. What exactly is new with respect to spectral graph pruning, ComFy, DeltaGNN, AFRC rewiring and the adaptive-depth GNNs (Hevapathige et al., GPR-GNN)? Would a cheap homophily-aware rewiring or edge-filtering step before NodePFN give similar gains to looping plus ICDS?
  6. Since every looped iteration also propagates, can the block learn to switch off message passing in later iterations (e.g., a learned gate or an initial residual)? Does this close the gap with NodePFN-12L on heterophilous graphs?
  7. Could the branch schedule (message passing only in the first g iterations) be learned, or trained into the block, rather than selected at test time? The oracle gains suggest this is where the real improvement on heterophilous graphs is.
7 comments

Summary and map of this response

We thank the reviewer for the careful and detailed review. Any changes or additions in reponse to your review are highlighted in blue in the revised paper. The table below shows where we answer each major concern. After the table, we answer every point of the detailed feedback and every question.

Major concernOur answerWhere
The method is simpleOur analysis shows that each graph needs its own depth. Looping turns the depth into a choice at inference and ICDS makes this choice without test labelsW3
"No loss in accuracy" hides the heterophilous loss and large per-graph swingsWe have revised the claim. The branch schedule chosen in context removes the loss on average and part of the swings is split noiseW1
Depth does not follow homophily in the headline settingIn that setting the pre-smoothing tuned for each dataset performs part of the propagation. ICDS still follows the oracle depth of each graphW2 and Q3
The theory overlaps with known resultsWe now cite both works. Proposition 1 adds a closed-form boundary in homophily and degree that also covers heterophilous graphsW5 and Q2
No positioning against low-cost GNN methodsWe have added related work. No rewiring method improves NodePFN-12L when its strength is chosen in contextW6 and Q5
The text is overloaded with numbers and detailsWe have moved implementation details and secondary numbers to the appendixRevised paper

Response to W1 (a): "No loss in accuracy" is only true on average

We agree that "no loss in accuracy" in contribution 3 holds only on average. We have revised contribution 3 to "comparable performance on average" and stated the heterophilous loss in the abstract. The loss comes from questions and 3 small WebKB graphs. The branch schedule removes this loss on average. When ICDS also selects the schedule from the labelled context, Loop-1L reaches 59.7% and Loop-2L 59.3% on the 10 heterophilous graphs, against 59.5% for NodePFN-12L.

Heterophilous graphs, mean score (%)NodePFN-12LLoop-1LLoop-2L
All 10 graphs with ICDS as in Table 1 (loss)59.556.857.0
All 10 graphs with ICDS and the branch schedule59.559.759.3
3 WebKB graphs with ICDS over 10 splits and 3 seeds73.869.869.9
All 10 graphs with the official ensemble and the branch schedule per member59.561.461.5

Our analysis predicts this loss, as W9 explains. The large per-graph differences in Table 2 are partly split noise, since NodePFN-12L itself ranges from 62% to 87% over the 10 splits of texas. We have added the per-graph and per-seed results to Appendix H.1.


Response to W1 (b): The below-chance result on questions

Selecting the depth and the branch schedule with hold-outs scored by ROC-AUC fixes the below-chance result on questions. On this graph each iteration pulls the positive nodes, which have a high degree, toward their negative neighbours. The paper's ICDS was unable to detect this because it scored the hold-outs by accuracy.

ROC-AUCNodePFN-12LLoop-1LLoop-2L
questions with ICDS (paper)0.630.440.47
questions with AUC-scored ICDS and the branch schedule0.630.660.65
All 4 binary graphs with AUC-scored ICDS and the branch schedule0.660.680.68

We have added the diagnosis to Appendix H.2.

Response to W2&Q3: The depth does not follow homophily in the headline setting

We agree that without pre-smoothing ICDS is below NodePFN-12L, by 1.5% for Loop-1L and 1.4% for Loop-2L (Table 7). The branch schedule reduces this difference to 0.9% and 1.0%. In this setting the depth selected by ICDS follows homophily, as the analysis predicts (Figure 8b).

Under the standard protocol the depth does not follow homophily, because NodePFN tunes its pre-smoothing for each dataset and this step performs part of the propagation before the model (Figure 10). The motivation still applies in this setting, since the depth a graph needs still depends on the graph. For Loop-1L the best depth for each graph is 1.8% above the best single depth for all graphs (Table 7). To answer Q3, ICDS adapts to this per-graph depth. The depth it selects follows the per-graph oracle depth, with a rank correlation of 0.38 for Loop-1L and 0.36 for Loop-2L (Appendix E).

We have removed from contribution 3 the claim that the chosen depth follows homophily on the benchmarks.


Response to W3: ICDS is essentially validation-based model selection

We agree that ICDS is a simple validation rule and that much of its gain comes from not using depth 12. Our contribution is that looping makes the depth a choice for each graph, and our analysis shows that this choice is necessary.

Table 7, which the reviewer cites, shows this under the standard protocol. With test labels, the best single depth for all graphs improves Loop-1L from 64.6% at the training depth to 65.9%, and the best depth for each graph improves it to 67.7%. Thus, a choice for each graph adds 1.8% beyond any single depth. Without test labels the best single depth is unknown. ICDS uses only the labelled context and reaches 66.3%, above this single depth. For Loop-2L, ICDS reaches 66.9% against 66.4%. We have marked the test-label rows of Table 7.


Response to W4: Fewer parameters, but not less compute

We agree that looping does not reduce training or per-iteration compute, and that training memory drops by only about a quarter. The true gains are a 10 times smaller model, from 116 MB to 12 MB, and a depth chosen for each graph after pretraining. We have corrected the ethics statement, the only place that suggested cheaper training, and the abstract states the speed-up only "once its depth is selected".

The one-off cost of ICDS can be significantly reduced. A large graph needs few hold-outs, so we set their number by the size of the graph.

On questions, the largest benchmarkNodePFN-12LLoop-1L with ICDS as in the paperLoop-1L with hold-outs set by graph size
Mean score over the 23 benchmarks64.6%64.0%64.0%
ICDS once per graphnone87.0 s6.6 s
Time per prediction4.10 s1.04 s1.04 s

With hold-outs set by graph size, Loop-1L is faster in total than NodePFN-12L from the third prediction on questions. Later predictions are faster than with NodePFN-12L on 21 of the 23 benchmarks. On air-europe and air-brazil ICDS selects more iterations than the training depth. The training cost can also be lowered. Truncated backpropagation through the loop (Geiping et al., 2025), which keeps the first iteration and the last four, cuts the time of a training step by about 40% and its GPU memory by about half. In our first untuned attempt it also lowers accuracy, and we leave its tuning to future work. We have added these results to Appendix H.3.


Response to W5&Q2: Proposition 1 and known results

We agree that a finite optimal number of propagation steps on the CSBM is known, and we now cite both works in Appendix D. Proposition 1 adds two results. First, it gives in closed form the boundary in homophily and degree between graphs where no propagation is best and graphs where propagation helps. Wu et al. (2023) give bounds on the variance and on the optimal depth and assume p>qp > q, so they do not cover heterophilous graphs. Keriven (2022) studies latent-space random graphs rather than the CSBM. Second, Proposition 1 states the consequence for a PFN. No fixed depth is optimal over a prior of graphs, and this is its role in the paper.

In our revised paper, Appendix D lists which steps of the proof are rigorous and which are heuristic, and it marks the multi-class remark as an argument rather than a proof. Section 5 tests the multi-class case on syn-Cora with 7 classes, where the selected depth rises with homophily as predicted (Figure 8(d)).

Response to W6&Q5: Positioning against cheap methods

We ran the suggested baselines. We placed each method in front of NodePFN-12L and chose its strength from the labelled context, in the way ICDS chooses the depth. None of them improves NodePFN-12L.

Mean score (%) in front of NodePFN-12LStrength chosen in contextStrength chosen on test labels
None64.6
One of three edge filters per graph (feature similarity / AFRC / predicted labels)64.266.6
ComFy / FeaSt64.6 / 64.665.2 / 65.6
Spectral pruning (ProxyDelete max / min)64.6 / 63.665.6 / 65.2
DeltaGNN score without training64.566.0

Test labels would raise NodePFN-12L to between 65.2% and 66.6%, but hold-out selection in the context does not find these strengths. A rewiring step therefore does not give the gain of ICDS, which improves Loop-1L from 61.8% to 64.0% (Q5). What is new in our work is the explanation of how the two branches of a graph PFN use depth, and a depth selected inside a pretrained model from the labelled context, without changing the graph or training on it. Adaptive-depth GNNs such as GPR-GNN and the method of Hevapathige et al. learn the propagation by training on the target graph. We have added a related-work paragraph to Appendix F and these results to Appendix H.4.


Response to W7&Q1: The comparison with ensembling

The parity holds with the official ensembles. When ICDS selects the depth and the branch schedule for each member, the looped models are as accurate as NodePFN-12L on all 23 benchmarks and more accurate on the heterophilous graphs. Each member is a separate forward pass with its own permutation of features and classes and its own preprocessing, so ICDS can select a different depth for each member. In ourr paper, we use a single pass because each readout of the analysis must reflect one computation (Section 3).

Mean score (%) with the official ensembleParametersAll 23Heterophilous 10
NodePFN-12L29.0M67.559.5
Loop-1L / Loop-2L with one depth for all members (paper)2.97M / 5.33M66.6 / 67.059.5 / 59.2
Loop-1L / Loop-2L with ICDS per member and the branch schedule2.97M / 5.33M67.8 / 68.261.4 / 61.5

We have added all selection rules we ran to Appendix H.6.


Response to W7&W8: Scope and reporting

We agree that we loop only NodePFN, and we have limited the claim in the abstract to NodePFN. GraphPFN trains only graph adapters, 3.6M of its 20.1M parameters, on a frozen LimiX backbone, so looping it would mean pretraining that backbone itself. Thus, we use GraphPFN to test the mechanistic findings. It also cannot work without its first layer, and its prediction depth grows with homophily (Appendix C).

Table 1 follows NodePFN in reporting ROC-AUC on the four majority-level graphs. We have added a version to Appendix H.2 that reports ROC-AUC over these 4 graphs and accuracy over the other 19 graphs separately. In the revised manuscript, the averages of Table 2 are reported in this way. Table 7 and Appendix E use accuracy on all 23 graphs, which is why their numbers differ from Table 2. In the updated manuscript, eevery result table states the protocol, the split and the metric in its caption.

Reviewer_mH4oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies graph foundation model for in-context node classification. A shared block of looped nodePDN is used and the authors discover two distinctive operations using a group of benchmarks, i.e. the attention branch and the message passing branch. It explains why no fixed depth is optimal across homophily levels. To address fixed depth inefficiency, the paper introduces Looped NodePFN, which replaces 12 distinct layers with a single shared Transformer block applied iteratively T times.

Critical strengths

Some discovery of this work is insightful. For example, the authors separate how long to think from how far to look. They are two dimensions but both support GNN.

The work shows a strong parameter efficiency, significantly reducing the parameters from a 12-layer model to a single-layer model without much accuracy loss. This can be valuable for real-world use cases.

Gradient-Free In-Context Depth Selection (ICDS) is an interesting contribution, exploiting a property uniqueness and selects execution depth entirely at test time

Critical weaknesses

The overhead seems strong in experimental performance. More specifically, 87 seconds on large graphs like questions

According to the experiments, the ICDS can overestimats early-iteration accuracy and select sub-optimal shallow depths on graphs with very few labeled context nodes

While the proposed branch schedule restricts message passing to the first g iterations, selecting g via ICDS only recovers a fraction of the oracle gap;

Looped NodePFN consistently lags behind NodePFN-12L on heterophilous graphs, the investigation and how to mitigate that should be clearly discussed.

Questions for authors

n/a

4 comments

We sincerely thank Reviewer mH4o for the positive feedback and the clear list of concerns. Changes made in response to this review are highlighted in orange in the revised paper.


Response to W1: ICDS overhead

We have found a simple solution to reduce this overhead on Questions from 87.0 s to 6.6 s. The average score for 23 benchmarks remains unchanged at 64.0%. A large graph needs few hold-outs, because each hold-out on questions already holds out 4,892 labelled nodes, so we set the number of hold-outs by the size of the graph. After this one-off step, each prediction on Questions is 3.9 times faster than with NodePFN-12L, so Loop-1L is faster in total from the third prediction.

On questions (48,921 nodes)NodePFN-12LLoop-1L with ICDS as in the paperLoop-1L with 10 hold-outs and maximum depth 8Loop-1L with hold-outs set by graph size
Time of ICDS once per graphnone87.0 s21.8 s6.6 s
Time per prediction as measured in Table 54.10 s1.04 s1.04 s1.04 s
Mean score over the 23 benchmarks64.6%64.0%63.8%64.0%

The last column uses enough hold-outs for 10,000 held-out nodes in total, between 3 and 20, with a maximum depth of 8 when fewer than 20 are used. It changes ICDS only on the five graphs with the most labelled nodes and keeps the paper's ICDS on the other 18. Predicting the depth from the labelled context could completely remove the hold-out passes, which we leave to future work. The training cost is discussed in our response to Reviewer 8Qhi (W4). These results are included in Appendix H.3.


Response to W2: Few labelled nodes

ICDS does not choose shallow depth simply because the number of labels is small. With only 5 or 10 labels per class, 10 to 150 labelled nodes per graph, ICDS still improves on the training depth and keeps its mean depth close to the oracle depth.

Loop-1L on the 23 benchmarks (chance level about 25%)Training depthICDSDepth of ICDS / oracle
5 labels per class50.0%50.7%5.0 / 5.1
10 labels per class55.0%57.2%6.2 / 4.8

The case in Appendix E is a synthetic cSBM that combines very few labels with weak features, and the limitations in Appendix G state this condition. There each hold-out has only 10 held-out nodes, which is why the first iteration can look better than it is. A more conservative rule, the deepest iteration within one standard error of the best, recovers most of the loss in this case. On real graphs it costs up to 1.3%, so ICDS remains the better default. We have added these results to Appendix H.9 (Tables 20 and 21).


Response to W3: The branch schedule recovers only part of the oracle

Selecting g in context already removes the heterophilous loss, which is the purpose of the schedule (W4). Since the oracle chooses g with test labels,, the remaining difference to the oracle is expected. ICDS recovers 55% of the oracle's gain for g on the heterophilous graphs, a share similar to the 61% it recovers for the depth in Table 1. Training the schedule into the block does not show better performance than selecting the schedule.

Mean score (%) of Loop-1LAll 23Heterophilous 10
ICDS64.056.8
ICDS with the branch schedule chosen in context64.859.7
Schedule trained into the block and chosen in context64.258.8
Oracle schedule chosen with test labels67.462.1
NodePFN-12L64.659.5

The oracle also shows that the looped model can go beyond NodePFN-12L on heterophilous graphs (62.1% against 59.5%), and a gate conditioned on the labelled context is a natural next step. We have added these results to Appendix H.8.


Response to W4: Heterophilous graphs

We identified the cause of this loss and a solution that removes it on average. The cause is weight tying. In heterophilous graphs, additional propagation hurts. While NodePFN-12L almost stops message passing in later layers (Section 3.3), a single looped block propagates at every iteration (Section 5). The solution is a branch schedule, that blocks message passing after an iteration chosen in context.

Mean score (%) on heterophilous graphsNodePFN-12LLoop-1L with ICDSLoop-1L with ICDS and the branch schedule
questions (ROC-AUC)62.948.865.7
3 WebKB graphs over 10 splits and 3 seeds73.869.871.4
6 other heterophilous graphs51.452.653.4
All 10 heterophilous graphs59.556.859.7

With the schedule, a loss of about 2% remains only on the WebKB graphs, which have fewer than 260 nodes. We have stated the loss in the abstract and added this analysis to Appendix H.1 (Table 9).

Thank you for the detailed rebuttal and the additional experiments. They address several of my concerns such as ICDS overhead and Few labelled nodes. However, it may needs some additional efforts to address the other concerns such as a loss remains on small heterophilous graphs as the authors acknowledged. The paper's core method and headline results are still Loop-1L with the original ICDS, which still lags on heterophilous graphs. As my scores are already on the positive side, I would like to keep them as they are.

Thank you for the reply. We are glad that the concerns on the ICDS overhead and on few labelled nodes are addressed. We answer the two remaining points.

  1. The branch schedule is part of the method and of the main results. The submitted paper introduces it in Section 5, and Table 1 reports it next to ICDS. With it, Loop-1L reaches a mean score of 64.8% against 64.6% for NodePFN-12L, and 59.7% against 59.5% on the ten heterophilous graphs.

  2. The remaining loss is limited to three small graphs. These are the WebKB graphs, which have only 37 to 51 test nodes, and the loss is about 2%. On the other 7 heterophilous graphs, Loop-1L with the branch schedule is more accurate than NodePFN-12L on average (Appendix H.1).

AuthorsOpenReview ↗

Request for your feedback on our follow-up response

Dear Reviewer mH4o,

Thank you again for the positive evaluation and for the discussion. As the discussion period ends soon, we would like to check whether our follow-up response above resolves your remaining concerns, on the role of the branch schedule in the method and on the loss on small heterophilous graphs.

We would appreciate a short reply before the discussion period ends.

Reviewer_X3AEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work focuses on introspecting the role of depth in prior-fitted networks (PFNs) for node classification, helping to understand the necessity of deeper models. The authors argue that depth has two purposes for PFNs applied to graphs, including refining predictions and propagating information. Moreover, they highlight that the amount of hops needed for each graph can vary. Following work for tabular PFNs, the authors show that later attention layers can be redundant, with the importance of propagation depending on homophily. To address this, the authors proposed Looped NodePFN, which applies a block repeatedly and selects the number of iterations using a set of held-out labeled nodes without gradient updates. The authors show that this is able to achieve highly similar accuracy to the non-looped model, while using a small fraction of the total parameters. Overall, I think the paper investigates an interesting question, although the work is quite limited in scope.

Critical strengths
  1. I found the distinction between prediction refinement and graph propagation useful, particularly in explaining why different graphs may benefit from different depths for PFNs.
  2. The empirical results are impressive within the PFN setting, given the substantial reduction in parameter count with little change in average accuracy.
  3. The mechanistic analysis is relatively comprehensive, with several complementary experiments that help explain the behaviors observed in these graph PFNs.
Critical weaknesses
  1. One of my main concerns with the paper is that the practical efficiency benefit hinges on separating depth selection from prediction. While there are clearly fewer parameters, it is less obvious that the cost of searching for the depth is justified by the subsequent inference savings. The authors quantify and acknowledge this overhead, but it remains unclear when the selected depth would be reused enough times to offset it, given that a fixed graph and labeled context may only require one prediction pass. The performance gap on heterophilous graphs also raises a related question where improving it requires selecting both the total depth and the number of iterations that retain message passing. Can the authors better establish the practical accuracy-cost tradeoff of these selection methods and demonstrate a setting where the upfront computation is worthwhile in comparison to the standard node PFN setting?
  2. I think the paper could more clearly explain what its mechanistic findings add to our existing understanding of graph learning. The authors discuss oversmoothing and adaptive-depth GNNs, but several motivating observations, such as propagation becoming harmful at greater depths and the useful amount of propagation depending on graph structure, are already well established. What is less clear to me is which observed behaviors arise specifically from the PFN setting and which reflect properties of message passing more generally. For instance, how do the findings about early label transfer and recovery after removing later computations relate to existing explanations of depth in graph models? More clearly identifying what prior work would predict, and what these experiments reveal beyond those expectations, would help establish the significance of the mechanistic contribution.
Questions for authors

Please see weaknesses.

4 comments

We sincerely thank Reviewer X3AE for the thoughtful questions. Changes made in response to this review are highlighted in green in the revised paper.


Response to W1: When does the upfront ICDS cost pay off?

ICDS pays off whenever a graph is queried more than a few times, and on large graphs this happens quickly. A large graph needs few hold-outs, so we now set their number by the size of the graph. On Questions dataset, the largest benchmark, this cuts ICDS from 87.0 s to 6.6 s without changing the mean score, and each later prediction takes 1.04 s against 4.10 s for NodePFN-12L (Table 5). The table gives the accuracy-cost trade-off.

Loop-1L with a single passMean scoreICDS on questionsPredictions on questions until ICDS is recoveredMedian ICDS time over the 23 graphs
Training depth without ICDS61.8%nonenone
ICDS as in the paper64.0%87.0 s292.7 s
ICDS with hold-outs set by graph size64.0%6.6 s32.7 s
ICDS with hold-outs set by graph size and the branch schedule64.8%33 s11about 13 s
NodePFN-12L64.6%nonenone

(a) When the upfront cost is worthwhile. It is worthwhile in the standard NodePFN setting. NodePFN is normally used with an ensemble of up to 32 forward passes. In this setting, Loop-1L with ICDS is faster in total than NodePFN-12L from the second ensembled prediction on 13 of the 23 benchmarks. Even for a single prediction without ensembling, ICDS runs once per graph and takes less than 0.5 s on Cora and Citeseer and at most 7.0 s on any of the 23 benchmarks once the hold-outs are set by graph size (Table 5), in exchange for a model 10 times smaller.

(b) Selecting the branch schedule together with the depth. Selecting both the depth and the branch schedule in context removes the heterophilous loss and improves the average score to 64.8%, which is higher than 64.6% for NodePFN-12L (fourth row). Even with this joint selection, the total time of Loop-1L on the Questions dataset is below that of NodePFN-12L from the 11th prediction on. We have added this table to Appendix H.3.


Response to W2: What do the mechanistic findings add?

Each NodePFN layer has two branches, attention over all context and query nodes and message passing over the edges. Only the message-passing branch follows existing GNN theory. The two findings the reviewer asks about concern the attention branch and are not predicted by this theory.

Number of hops (general to message passing). In a GNN, each layer adds one hop (Xu et al., 2018), and the useful number of hops depends on the graph, as Wu et al. (2023) and Keriven (2022) show. In NodePFN each message-passing branch adds one hop, and without pre-smoothing of the features the useful number of hops grows with homophily.

Early label transfer (specific to PFNs). Label propagation predicts that labels far from a node need more layers to reach it (Wang & Leskovec, 2021). In NodePFN the attention branch of the first layer sends the context labels to every query node at once. When we remove it, later layers do not recover the accuracy. In NodePFN the labels therefore reach the query nodes in one layer, whatever their graph distance, and later layers are not needed for this. Skipping the first layer of GraphPFN also lowers its accuracy from 67.3% to 23.7% on homophilous graphs (Appendix C).

Recovery after removal (from attention, new for graph models). The receptive-field view predicts that removing a later layer loses information. In NodePFN, when we remove the attention branch of layer 2 or 3, the accuracy at layer 12 stays within 2% of the unchanged model. The attention branches of later layers therefore refine the prediction rather than add new information. Such recovery has been reported for language models and tabular PFNs but not for graph models.

These findings define our method. Later layers refine in a similar way, so one block applied repeatedly can replace them. The useful number of hops depends on the graph, so ICDS selects the number of iterations for each graph. We have added this discussion to Appendix H.5.

Reviewer_X3AEOpenReview ↗

Thank you

Thank you for your response. I appreciate your points on practical efficiency, and I now see that the upfront cost does seem justified in the context of the model. For the second, I am still a bit unsure. The point of my question was not to argue against these results for NodePFN, but rather to better understand their broader significance as novel insights. While I appreciate that these behaviors have not necessarily been demonstrated for graph PFNs, I am still unsure what new understanding they provide to the graph ML literature beyond confirming that established properties of graph learning and attention-based models hold. In particular, I would have liked to better understand what new properties or interactions emerge specifically from combining attention and graph propagation. With this in mind, I will maintain my original score.

Thank you for the follow-up. We are glad that the cost concern is resolved.

On the second point, the new understanding is that in a graph PFN, labels and graph structure travel through different branches, and the way the model uses depth follows from this separation. To our knowledge, the GNN literature and the literature on attention models do not predict this. Therefore, the findings describe new properties of the combined model, which the established properties of each branch alone do not explain.

The reason is that attention in a PFN is not the attention of a graph transformer or a language model. Query nodes attend only to the labelled context nodes, whose labels are part of the input (Section 2). Thus, attention transfers the labels, and message passing aggregates the features of neighbouring nodes. Three properties follow from combining the two.

  1. Labels need one layer, and structure needs a depth that depends on the graph. In label propagation, and in GNNs that take labels as input, the labels travel along the edges, one hop per layer. In NodePFN the attention of the first layer transfers the labels to every query node, and the number of layers that depends on the graph is the number that propagation needs (Section 3).

  2. Later attention cannot replace a missing propagation step. Later layers recover a removed attention branch of layer 2 or 3 to within 2%. They do not recover a removed message-passing step on homophilous graphs, where about half of the drop remains for layer 3 (Figure 7).

  3. Later layers partly recover from the early smoothing. In a message-passing network, more layers mean more smoothing. Without pre-smoothing, the Dirichlet energy of NodePFN-12L on heterophilous graphs falls from 0.68 to 0.32 and then rises to 0.42, because later layers propagate little and their attention and feed-forward blocks increase the differences between neighbouring nodes again (Appendix E and Figure 4b).

These properties have two implications for the design of such models. Properties 1 and 2 imply that one looped block can replace the layers, with a depth selected for each graph. Property 3 implies that propagation can stop after the first iterations. Section 5 tests both. We have added this summary to Appendix H.5.

AuthorsOpenReview ↗

Request for your feedback on our follow-up response

Dear Reviewer X3AE,

Thank you again for the discussion. As the discussion period ends soon, we would like to check whether our follow-up response above resolves your remaining concern on "what new properties or interactions emerge specifically from combining attention and graph propagation". The response describes three properties of the combined model, which are now in Appendix H.5.

We understand from your comment that you would have liked to better understand this point. We hope that our response provides this understanding, and we would appreciate a short reply before the discussion period ends.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Discussion for Reviewers and AC

Dear Reviewers and AC,

We sincerely thank all reviewers for their constructive feedback and for the recognition of our contributions.

Contributions recognized by the reviewers include

  • the separation of refinement from propagation, which explains why graphs need different depths (mH4o, X3AE),
  • a well-designed and comprehensive mechanistic study (8Qhi, X3AE), with a convincing negative control (8Qhi),
  • the finding that the tuned pre-smoothing of NodePFN hides the relation between depth and homophily (8Qhi),
  • strong parameter efficiency with little change in average accuracy (mH4o, X3AE),
  • gradient-free depth selection in context (mH4o).

Based on the reviews, we have made the following improvements in the revised paper. All changes are highlighted in blue, orange and green for Reviewers 8Qhi, mH4o and X3AE.

  1. Addressed the loss on heterophilous graphs. The branch schedule chosen in context improves Loop-1L from 56.8% to 59.7% on the ten heterophilous graphs, against 59.5% for NodePFN-12L (Appendix H.1).
  2. Reduced the cost of ICDS. Hold-outs set by graph size cut ICDS on the questions graph from 87.0 s to 6.6 s with the mean score unchanged, and we report the accuracy-cost trade-off (Appendix H.3).
  3. Extended the experiments. We added results with the official ensembles, with three training seeds, with few labelled nodes, with rewiring methods in front of NodePFN-12L and with gates and learned schedules (Appendices H.4 and H.6 to H.9).
  4. Clarified the claims and the analysis. We revised the accuracy and efficiency claims, limited the single-block claim to NodePFN, positioned Proposition 1 against Wu et al. (2023) and Keriven (2022), and described the properties that come from combining attention with message passing (Appendices D and H.5).

All three reviewers have responded to our rebuttal, and each of them confirmed that it resolves most of their concerns.

  • Reviewer mH4o stated that our responses "address several of my concerns such as ICDS overhead and Few labelled nodes", and maintained the positive evaluation of our submission. We have answered the remaining concern on small heterophilous graphs in a follow-up comment.
  • Reviewer X3AE stated, "I now see that the upfront cost does seem justified in the context of the model". We have answered the remaining question on the mechanistic contribution in a follow-up comment and in Appendix H.5.
  • Reviewer 8Qhi stated that "the rebuttal addresses most of my concerns", that the errors, inconsistencies, ethical concerns and unsubstantiated claims "have been explicitly acknowledged and addressed", and that "the quality of the paper has improved". We have answered the remaining reservation on methodological novelty in a follow-up comment.

We thank the ACs and all reviewers again for their efforts in reviewing our paper and for the engagement in the discussion. If any concern remains, please do not hesitate to let us know.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.