acceptodds
Under review as a conference paper at ICLR 2027

PairShift: A Controlled Benchmark for Graph Foundation Models under Density Shift

Abstract

Evaluating the structural robustness of graph foundation models (GFMs) often yields a paradox: poorly performing models can appear deceptively robust, whereas highly capable models often prove brittle. Existing cross-dataset evaluations fail to isolate this failure mode, as topology, features, labels, and sampling confound one another. In this paper, we introduce PAIRSHIFT, a benchmark of paired density views: nested edge sets that share nodes, features, labels, and edge randomness. Its context-by-query response surfaces and an exact decomposition distinguish ordinary density effects from a matched-density interaction. Across six GFMs, G2T-FM and GraphPFN incur 16–17 percentage points of mean transfer failure on the common grid, rising to 19 points for G2T-FM on an unseen universe; the other four systems show much smaller interactions, sometimes alongside weak accuracy or unstable predictions. Fixed-checkpoint interventions then show that GraphPFN’s graph-convolution adapters increase mean transfer failure by 11–44 points across controlled synthetic settings, with effects that depend on shift direction and adapter depth. Motivated by this pathway, we construct a class-compatibility neighbor residual with no learned parameters. It improves G2T-FM and SAMGPT across two evaluated universes (2–3 and 1–2 points, respectively) and GOBLIN by 4.4 points in a post-hoc extension, but harms models when additional neighbor evidence is redundant or unreliable. A local error-alignment analysis predicts the direction of its cross-entropy change in all 24 decisive tested conditions. PAIRSHIFT thus connects controlled structural evaluation to a causal failure pathway and a bounded, mechanism-inspired correction.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Foundation Models, Graph Learning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tMVTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the predictive performance of GFMs varies when context and query graphs have different edge densities. The authors introduce PairShift, a benchmark that constructs paired density views with nested edge sets while keeping nodes, features, labels, and edge randomness fixed. It compares model performance on the same query graph using context graphs with either matched or mismatched densities. The authors further investigate GraphPFN’s failure pathway through fixed-checkpoint interventions on its graph adapters and evaluate a class-compatibility neighbor residual that uses query pseudo-labels to adjust the base predictions.

Critical strengths

1.The paired context-by-query design is sound and yields results that are straightforward to interpret. The benchmark methodology and analysis are clearly presented. 2.The fixed-checkpoint interventions on GraphPFN’s graph adapters are well motivated.

Critical weaknesses

1.In Section 5.2, the authors interpret the reversal in residual gains across adapter depths as evidence that the graph adapters have already absorbed related neighbor evidence, making the residual redundant or harmful. However, according to Equations (4)–(6), the query pseudo-labels used to construct the residual come from the base model’s predictions, and the residual is scaled to match the base-logit scale. Both may therefore change with adapter depth. It is unclear whether the reversal is due to redundant neighbor evidence or changes in the residual itself. 2.For GraphPFN, the authors report a large adapter contribution to mean transfer failure in synthetic universes, but a much smaller contribution on real graphs with original labels. I wonder why would this happen and under what specific practical task conditions would this failure mode be expected to occur? I expect more specific analysis.

Questions for authors

See the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer tMVT — W1

Thank you for recognizing the paired evaluation and the motivation for the fixed-checkpoint interventions. We have added a fully fixed-message control and expanded the task and temporal analyses to address your two questions.

W1: do changing pseudo-labels and scale confound the depth reversal?

You are correct: the original experiment holds the residual update rule fixed, while its pseudo-labels and logit-based scale change with adapter depth. That comparison establishes a depth-dependent response but cannot isolate evidence absorption.

We added a cached counterfactual that freezes the complete depth-zero update vector, including pseudo-labels, transition estimates, centering, scale, and strength, and applies exactly that vector at depths 0/6/12 on the same query graphs. The fixed-message accuracy summaries are reported below alongside the primary loss contrast. Let Δd be corrected-minus-base NLL at depth d and Q = Δ12 − Δ0. The proposed reduction in NLL benefit at depth twelve predicts Q > 0.

SettingFixed-message accuracy gain, depth 0 → 12 (pp)Q in NLL units [95% CI]
U4+3.10 → +6.17−0.13089 [−0.14782, −0.11450]
U5+2.51 → +4.09−0.03751 [−0.04565, −0.02886]
U9+5.89 → +7.63−0.02829 [−0.03863, −0.01861]

These estimates average the two extreme directions, 2→32 and 32→2. Accuracy values are descriptive means from the same saved control; Q intervals resample query latent identities jointly across strata and directions. With the complete message fixed, the original positive-to-negative accuracy reversal does not reappear, and all three Q intervals lie below zero. The control therefore does not support the stronger absorption prediction. It does not determine whether pseudo-label changes, scaling, or their interaction dominates the original reversal.

We retain the observed depth dependence when each depth constructs its own residual, remove the internal-absorption inference, and rename §5.2 accordingly. The measured adapter-path contribution and the explicit residual's conditional loss explanation through error alignment and curvature remain supported; the latter uses evaluation labels.

Changes and evidence: §4.3, §5.2, §5.4 and §6; fully fixed-message protocol, primary NLL results, and descriptive accuracy summaries in Appendix C.2. Per-query outcomes from this cached experiment are in the supplement. Native 0/6/12 failure is reported separately in Appendix E.3, Table 27.

AuthorsOpenReview ↗

Response to Reviewer tMVT — W2

W2: why do synthetic and real effects differ, and when should practical failure be expected?

The strongest tested explanation concerns the relationship between the task and topology. Keeping real edges and blocks fixed, we replace both labels and features with the generator's task process. The GraphPFN adapter contribution then changes as follows; all values are percentage points, and ranges span planted settings rather than confidence intervals:

Fixed real edge setOriginal taskGenerator label/feature process
Penn94+0.8+11.2 to +17.1
Cornell5+0.6+5.0
JohnsHopkins55−0.1+10.8
Squirrel+2.0−0.1 to +2.3

Thus real topology can carry a large adapter-path effect under a different task process. The Squirrel exception, including larger-block controls, shows that the edge structure also matters. Because labels and features change together, this comparison does not identify labels alone as the cause. Feature degradation and larger blocks keep the original-task effects small, arguing against a simple accuracy-ceiling explanation. High homophily is also insufficient to explain the gap: it can increase failure within the controlled generator, where substantial effects persist at a fourfold density gap.

For practical conditions, these results suggest a candidate risk setting: a model relies on class-relevant neighborhood patterns transferred from context, and changing density alters how that neighborhood evidence is used. This is a hypothesis motivated by the controls, not a validated sufficient condition. Our results do not establish a reliable natural-task risk rule based on mean degree or homophily alone; the frozen real-graph context-selector test also remains failed.

We further tested naturally timed changes using HighSchool2013, including an earlier consistently sparse context, and ogbn-arxiv citation snapshots with their original subject labels. On identical late queries, GraphPFN's matched-late-minus-day-one contact differences are −11.11/−4.17 points across banks; its natural-2019-minus-2013 citation differences are −6.45/−11.39. G2T-FM's citation differences are +13.87/−7.62. These substantial but mixed context effects do not establish a consistent benefit from later or closer-density context. The citation density reference is explicitly approximate, and overlapping banks and coupled topology/task changes make these descriptive comparisons rather than isolated density effects.

The revised motivation explains why synthetic graphs provide repeatable control of these coupled factors. We have aligned the Abstract, Introduction and Discussion with the evidence: task–topology dependence is supported, while the prevalence and sufficient conditions of this failure on natural tasks remain open.

Changes and evidence: Task-process and ceiling controls in Appendix D.2–D.3, Tables 15–16; temporal contacts and citation growth in Appendix D.4, Tables 17–19; the original selector decision in Appendix D.5, Table 20. Motivation and scope are revised in the Abstract, §1 and §6.

The reply has addressed most of my concerns and I will maintain my rating of 3.

Reviewer_wzjoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies graph foundation models (GFMs) that predict node labels by looking at a few labeled example graphs, called the context. The question is simple. What happens when the context graphs have many more, or many fewer, edges than the graph being predicted on (the query)? To test this, the authors have built a synthetic benchmark called PairShift. Each graph is generated once and saved in five versions, with 2, 4, 8, 16 and 32 edges per node on average. The nodes, features and labels are the same in every version. Only edges are added or removed. The authors then measure accuracy on a query graph twice: once with a context of the same density, and once with a context of a different density. The drop between the two is called transfer failure. Of the six models tested, G2T-FM and GraphPFN lose 16 to 17 accuracy points on average. The other four lose much less. Inside GraphPFN, the authors show that the failure disappears when its graph adapter layers are switched off. They also propose a simple fix that nudges each node's prediction toward the predictions of its neighbors. It helps three models by 1 to 4 points and slightly hurts or does nothing for the other three. On real graphs (Appendix D), the effect is close to zero unless the real labels are replaced with synthetic ones

Critical strengths

What I liked most is how the comparison is set up. Most GFM evaluations compare results across datasets. When the dataset changes, the nodes, features, labels and edges all change at once, so it is hard to say what caused a drop. Here, the matched and mismatched runs use exactly the same query graph and test nodes (lines 192-195). Only the context changes. I haven't seen this done for GFMs before, and I think it is the right way to ask the question.

I also think it is important that accuracy is reported next to the failure number (Table 2). A model that ignores the edges can't be hurt by changes to them, so it can look robust even when it is weak. AnyGraph shows this well. Its accuracy is around 0.3 and its average failure is near zero, but Figure 2 shows that this zero comes from positive cells (up to +12) and negative cells (down to -10) cancelling out.

Finally, I appreciated how openly the authors report what didn't work. They say when an experiment failed a success test they had set in advance (lines 327-328, 800-803, 830-833). They say which models were added later (lines 662-663). They also kept the real-graph results, even though these weaken the main message. This made me trust the results that did pass.

Critical weaknesses

My main concern is that the effect is large on synthetic graphs but nearly absent on real ones. On synthetic data, GraphPFN's adapters add 11 to 44 points of failure (Table 7). On real graphs with their real labels, the same number is between -0.1 and +2 points (Table 11). The other models tested on real graphs also show almost no effect (lines 859-860). The effect only returns when the generator's labels are planted on the real edges (Table 12), and even then not on Squirrel. My reading is that the generator creates the labels and the edges from the same hidden groups. Because of this, the edges carry a lot of information about the labels. Real tasks don't seem to have this property nearly as strongly. For a benchmark, this is the biggest issue for me: if a model scores badly on PairShift, I can't tell whether it will struggle on real graphs. The authors acknowledge this in Section 6 (lines 467-468). However, the abstract and introduction make much stronger claims, such as a "catastrophic performance drop" (line 43) and "a critical yet obscured vulnerability" (lines 46-47). I think the framing should match Section 6. My second concern is that some of the mechanism results hold by construction. The adapters are the only part of GraphPFN that uses the edges. With all of them switched off, the model never sees the edges, so the failure has to be zero (the paper notes this in line 323). The finding that "disabling them removes the effect" (line 70) therefore could not have come out any other way. A more informative claim would be that stacking many adapters makes the problem worse than using the edges once. However, the experiment that comes closest to testing this failed the authors' own test (lines 327-328). The paper also runs GraphPFN with 6 of its 12 adapters on (line 322), but I couldn't find the failure number for that setting. The same applies, in part, to Eq. 7. Eq. 7 approximates how much the error changes when the fix is applied. It is computed with the true test labels and then compared with the real change on those same labels. For a change of this size, I would expect such an approximation to get the direction right in most cases. So the 24 out of 24 result mainly shows that the approximation is accurate. It does not test why the fix works, and I don't think it belongs in the abstract. Similarly, the "exact decomposition" in Appendix B.1 is correct, but as far as I can tell it writes the same average failure in a different form and adds little new information. I also don't think a simpler explanation has been ruled out. Table 1 describes GraphPFN's adapters as using the "raw adjacency". I read this as adding up messages from neighbors without dividing by the number of neighbors. G2T-FM turns graph structure into extra input columns. If these columns include counts such as node degree, then going from 2 to 32 neighbors makes them about 16 times larger. That alone could confuse a model that only saw small values in the context. The two baselines point the same way. In their standard form, GCN and GraphSAGE do divide by the number of neighbors, and they fail by only 2.6 and 5.5 points (Table 2). They are trained differently, so this is only a hint. The check in Appendix C.2 (lines 778-779) argues against my explanation for GraphPFN, but it is a correlation rather than a direct test. I also looked in Appendix A.2 for how G2T-FM builds and scales its graph columns, and couldn't find it. A few smaller points: The proposed fix is never tested on transfer failure itself (lines 366-367). So it isn't clear that it addresses the problem the benchmark measures. A standard method, label propagation, also does about as well: about a point worse for G2T-FM and slightly better for SAMGPT (Table 9). I am not sure density is the only thing that changes between versions. Homophily is how often connected nodes share a label. Each version is allowed to miss its homophily target by up to 0.15 (lines 632-634), which is half the gap between the two settings the paper compares. So the sparse and dense versions of a graph could differ in homophily as well as density. I couldn't find the actual values in Appendix A. The repeated "universes" share a random seed for labels and edges (lines 637-640). This makes the repeated results weaker evidence than they first appear. The code is only promised after acceptance (lines 504-505). This makes a benchmark paper hard to check. I found Sections 4 and 5 hard to follow. Many terms, such as "headroom gate", "evidence absorption" and "post-freeze", are introduced quickly, and I had to read Appendix C side by side with the main text. On the figures: Figure 1 is meant to show that the sparse graph is a subset of the dense one, but the drawings are too small to see this. It also uses densities (1.5 and 10) that are not in the benchmark. In Figure 2, the numbers in the cells are in accuracy points, but the color bar is in fractions. The caption doesn't say which of the two AnyGraph versions is shown. The grids are also averaged over four graph types. Section 5.1 shows that low and high homophily behave differently for the fix, so I suspect the failure grids differ too, and the average hides this. Figure 3 was the hardest for me to read. It shows three experiments on three different x-axes, with very small labels and colors that are not explained. In panel A, two of the models are evaluated on a different universe from the rest. Table 2 has no error bars, even though the paper says they are computed (lines 671-672).

Questions for authors

These are the questions whose answers could change my score: What is the actual homophily, and the actual number of edges per node, in each density version? How much transfer failure does GraphPFN show with 6 of its 12 adapters on? How does G2T-FM turn the graph into extra input columns, and are these columns rescaled for each graph? Does the fix reduce transfer failure itself? Since the predictions are already saved, this seems quick to check.

How many synthetic universes were generated and rejected before the reported ones?

I also think a few experiments would make the paper much stronger: 1)A real change in density. One option is a network that grows over time, such as a citation graph. An early snapshot could serve as the context and a later, denser one as the query, compared against a matched context. 2) Dividing by the number of neighbors inside GraphPFN's adapters, and rescaling G2T-FM's graph columns for each graph. This needs no retraining. If most of the failure went away, even at some cost in accuracy, that would be a useful finding in itself. 3) Giving G2T-FM and GraphPFN a context that mixes all five densities, to see whether this removes most of the problem. Keeping homophily fixed across the density versions, and testing smaller density gaps (2x or 4x). These seem more realistic to me than a 16x gap.

10 comments
AuthorsOpenReview ↗

Response to Reviewer wzjo — Q1-Q4

Thank you for the careful reading and concrete checks. We answer your Questions and experimental suggestions first (Q1–Q10), then the remaining concerns (Q11–Q19). Locations refer to the revised manuscript. Accuracy and failure differences are in percentage points unless stated otherwise.

Q1. What are the actual homophily and edges per node in each density view?

We now report measured values in Tables 22–24: means and graph-level ranges at all five densities and four structural strata, with 100 latent graphs per stratum, for the historical U2 grid and both new populations. Mean degree is 2|E|/N; exact edge counts, standard deviations, isolation rates, and community/class homophily are in the supplement.

For the U2 stratum with target homophily 0.2 and tail exponent 2.5:

Target degreeActual mean degreeActual class homophily
21.9570.2256
43.9070.2262
87.8250.2260
1615.6070.2257
3230.9120.2244

These are means, not intervals. The 0.15 tolerance is an acceptance threshold, not observed paired drift. Table 25 measures drift across all ten density pairs: U2's stratum-specific 95th percentiles are 0.0269–0.0397, with maximum 0.0642. Thus similar population means do not eliminate the mixing concern. Under the stricter control in Q9, maximum paired drift falls to 0.005958 while failure remains substantial in both models and populations.

Locations: Appendix E.1, Tables 22–25; Appendix C.1, Table 5; per-view/per-pair audits in the supplement.

Q2. How much failure occurs with six of GraphPFN's twelve adapters?

Six adapters already produce substantial failure. The saved-prediction estimates below average the extreme directions, 2→32 and 32→2, with 95% latent-cluster intervals; these are distinct from Table 2's complete-grid averages.

UniverseDepth 0Depth 6Depth 12
U40.00 [0.00, 0.00]19.79 [18.42, 21.24]23.05 [20.45, 25.66]
U50.00 [0.00, 0.00]12.32 [10.47, 14.21]11.39 [8.97, 14.04]
U90.00 [0.00, 0.00]18.61 [17.36, 20.01]20.86 [18.19, 23.58]

All depth-six intervals exclude zero. The results establish depth dependence, not monotone amplification: U5's point estimate is higher at six than twelve. They also do not directly compare one use of edges with repeated use. The failed factorial composite test remains reported; Q12 explains the adapter-path contribution.

Locations: §4.1; Appendix E.3, Table 27. Added summary of cached outcomes.

Q3. How does G2T-FM construct and scale its graph columns?

The groups are raw numeric features, neighborhood means, Laplacian features, log-degree, log-PageRank, and PEARL. Raw features and neighborhood means are standardized. Laplacian, log-degree, and log-PageRank use native quantile-normal transformations fitted jointly over visible context/query nodes; PEARL follows its native implementation.

Degree therefore enters after logarithmic and quantile-normal transformations, with joint rather than per-graph fits. Q7 tests changing only the degree/PageRank fits to per graph.

Locations: Appendix A.2; §4.1 and Appendix C.2.

Q4. Does the residual reduce transfer failure itself?

Not consistently. We recomputed failure with both corrected matched references and corrected mismatched predictions on the two complete independent-population grids. Positive base-minus-corrected failure denotes improvement:

ModelPopulation p1 reduction [95% CI]Population p2 reduction [95% CI]
GraphPFN−2.31 [−2.81, −1.84]−0.92 [−1.43, −0.38]
G2T-FM+1.01 [0.78, 1.28]−0.06 [−0.52, 0.36]

Failure increases for GraphPFN; G2T-FM improves in p1 and is unresolved in p2. A historical post-hoc GOBLIN grid gives 2.218→1.298. Legacy SAMGPT/MDGFM caches lack corrected matched cells, so accuracy gains cannot establish failure reduction. We retain model-specific benefits and the conditional loss explanation, and withdraw general failure repair.

Locations: §1, §5.4; Appendix C.2, Table 8. LP comparison: Q15.

AuthorsOpenReview ↗

Response to Reviewer wzjo — Q5-Q6

Q5. How many synthetic candidates were generated and rejected?

The main prospective U2 grid and both new populations each accepted their first specified candidate; we did not compare model outcomes across candidate universes. Across the broader studies with retained numerical QC records, 99 candidate-stage trials comprise 27 passes and 72 rejections. These count data-construction decisions across protocols, generator versions, and sizes, not successful/failed model experiments or unique independent environments.

Each trial screens a generator configuration and seed. Acceptance requires identical nodes, features, and labels across views; nested edges; relative target-degree error ≤0.25; absolute target-homophily error ≤0.15; and probability-clipping proportion ≤0.05. Study-specific checks, including degree-tail separation, also apply. These construction tolerances differ from Q1's measured drift and Q9's tighter controls.

Candidates were examined in the predeclared order within each formal stage, and accepted or rejected before target-GFM evaluation on that candidate. Accuracy and failure never entered these checks. Rejection advanced to the next candidate; exhausting the candidate budget stopped the stage as infeasible. Thus the 72 rejections are not filtering after unfavorable model results. Failed model-level tests and negative/mixed outcomes are retained separately.

Locations: Appendix A.1; Appendix E.2, Table 26, including earlier generator-revision stages, development and reuse; protocols, QC records, and source hashes in the supplement.

Q6. What happens under a naturally timed change in graph density?

We added your citation-snapshot comparison and an earlier, consistently sparser contact context. Both compare context policies on the same late query. Later or closer-density context is not consistently better.

Citation growth: ogbn-arxiv. Publication years and released edges define snapshots with the original 40 subject labels and text features. Each bank uses a fixed 1,536-node 2019 query; 2013/2015/2017/2019 contexts each contain 1,024 labels with identical class counts. GraphPFN uses ten ensemble members and G2T-FM five for its native 40-class encoding. The two banks' 2013-context/late-query mean degrees are 2.99/4.53 and 3.11/4.21.

We compare natural 2019 context and a closer-density late context selected by graph structure and class budget, without model outcomes or query labels. The latter remains 8.86%/4.69% below query degree, so it is an approximate-density reference. All 20 runs are retained. Relative to 2013 context:

ModelBankNatural 2019 − 2013 [95% CI]Approximate-density 2019 − 2013 [95% CI]
GraphPFN1−6.45 [−12.96, 0.07]−13.15 [−19.14, −7.16]
GraphPFN2−11.39 [−19.34, −4.43]−13.35 [−22.59, −5.08]
G2T-FM1+13.87 [3.06, 24.67]+10.03 [−1.89, 21.81]
G2T-FM2−7.62 [−16.02, −0.72]−6.77 [−16.02, 2.21]

The 2015/2017 comparisons also show no consistent benefit. A fixed text-only logistic-regression reference gives natural-2019-minus-2013 changes of +1.11/−2.02, illustrating the effect of changing context cohorts on the text-based task.

Contact accumulation: HighSchool2013. Twelve added runs compare day-one context with the existing late and exactly aggregate-density-matched references. Participants, labels, 32 first-day features, and label budgets stay fixed; features are available by the earliest context day's end. Day-one context is sparser than every day-3/4/5 query bank (query/context degree ratios 1.21–1.61). Averaging the three late dates within each query block:

ModelBank 1: matched − day one [95% CI]Bank 2: matched − day one [95% CI]
GraphPFN−11.11 [−22.22, −3.47]−4.17 [−8.33, −0.69]
G2T-FM−0.69 [−12.50, 9.72]−7.99 [−13.89, 0.00]

Positive values favor the later reference. Intervals jointly resample policies/dates over six contact or twelve citation query blocks per bank; they are descriptive, and overlapping banks are not independent datasets. Contacts also change mixing; citation cohorts change nodes/text, and publication-year snapshots with released embeddings are retrospective approximations. These accuracy contrasts do not isolate density causally and differ from synthetic-grid failure.

The tests show context sensitivity but no consistent density-matching benefit. They bound external extrapolation while retaining the controlled finding.

Locations: §6; Appendix D.4, Tables 17–19 retain the original day-two study and add first-day contacts/citation growth; framing in Abstract, §1, §§6–7.

Reference: Hu et al. (2020), Open Graph Benchmark, NeurIPS. ogbn-arxiv dataset.

AuthorsOpenReview ↗

Response to Reviewer wzjo — Q7-Q8

Q7. Can normalization or per-graph rescaling reduce the failure?

G2T-FM: fitting only log-degree/PageRank quantile transforms separately per component graph reduces failure by 1.88 [1.38, 2.38] and 3.23 [2.88, 3.59], leaving 8.14/8.68 points. Matched-accuracy changes are +0.01 [−0.06, 0.07] and −0.04 [−0.13, 0.06]; both populations pass the fixed two-point capability-loss bound. The checkpoint, other groups, column masks, and budgets stay fixed. This identifies a preprocessing contribution without explaining the full effect or testing every structural column.

GraphPFN: the native adapter already uses edge-softmax-normalized attention. We added your degree-division control: divide the aggregated neighbor message by max(in_degree, 1) before the original output projection in all 12 adapters. The checkpoint, projection bias, layer norm, skip connection, MLP branch, and ten-member ensemble stay fixed. All 200 new and 200 native reference cells are verified on identical inputs.

Native failure is 16.18/15.27. The intervention uses its own matched reference after degree division; paired 95% latent-cluster intervals give:

PopulationRemaining failureFailure reductionMatched accuracy changeCross-density accuracy change
p10.22 [0.10, 0.33]15.97 [14.92, 17.08]-10.14 [-12.16, -8.11]5.83 [4.09, 7.46]
p20.51 [0.38, 0.64]14.76 [13.54, 15.98]-7.79 [-9.37, -6.23]6.97 [5.91, 8.06]

Degree division removes most failure and improves cross-density accuracy, with the substantial matched-accuracy cost you anticipated. Because native attention was already normalized, this does not identify an unnormalized sum as the cause. The fixed-0.25 attenuation control tests message strength separately. GCN/GraphSAGE differ in training and architecture, so their smaller gaps cannot be attributed specifically to normalization.

Locations: Table 1, §4.1, §6, Appendix A.2; preprocessing, attenuation and degree division in Appendix C.2, Tables 6–7; source/outcomes/replay in the supplement.

Q8. Does a context mixing all five densities remove most of the problem?

Yes: mixed context recovers most of the average gap in both models and populations, without retraining or a larger budget. The same 60 context latent graphs contribute one view each: 12 at each density under three fixed assignment seeds. Features, labels, and node/label budgets stay fixed; all five query densities are evaluated.

The original gap compares matched and average mismatched single-density contexts on identical queries. Mixed context uses that same matched reference. Intervals are paired 97.5%; recovery percentages are descriptive point estimates.

Model, populationOriginal gapMixed-context gap [97.5% CI]Gap recovered
GraphPFN, p116.182.63 [1.40, 3.77]84%
GraphPFN, p215.27-0.17 [-1.95, 1.52]101%
G2T-FM, p110.021.59 [0.90, 2.30]84%
G2T-FM, p211.913.08 [2.45, 3.68]74%

All four improve significantly over mismatched context. Only GraphPFN p2 passes the stricter criterion ruling out >2 points of loss relative to matched context; its negative point gap has an interval spanning zero. Thus broad density coverage mitigates mismatch, while near-matched performance under that criterion is less general. With no unique context density, the mixed-context gap is distinct from the original single-density failure metric.

Locations: Appendix C.2; assignments, budget audits and original joint decisions in the supplement.

Reviewer_SDwSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PairShift, a benchmark for evaluating the robustness of graph foundation models under graph density shifts. By constructing paired graph views that share nodes, features, and labels while varying edge density, the benchmark isolates the effects of context-query density mismatch. The authors evaluate six GFMs and introduce response-surface analysis to distinguish density effects from transfer failures. Further experiments investigate the role of graph adapters in GraphPFN and propose a parameter-free neighbor residual correction. The results reveal substantial density sensitivity in some GFMs, while the effectiveness of the proposed correction depends on the model and graph structure.

Critical strengths

S1. The paired graph construction provides a systematic way to study density shifts while controlling other factors. The context-by-query response surfaces offer a more informative evaluation than conventional scalar transfer metrics.

S2. The experiments cover six GFMs with different topology-processing mechanisms, revealing important differences between predictive capability, structural robustness, and prediction stability.

S3. The fixed-checkpoint interventions on GraphPFN provide evidence that graph-convolution adapters contribute to density sensitivity. The additional analysis of neighbor residual corrections offers useful insights into when structural information improves or harms predictions.

Critical weaknesses

W1. The main findings rely heavily on synthetic graphs. Although real-graph experiments are included, the observed density-shift effects are substantially weaker than those in synthetic settings. This raises concerns about the practical significance and generalizability of the identified failure mechanism.

W2. The benchmark extends an existing synthetic graph generator, while the response-surface decomposition follows a standard additive-interaction formulation. Moreover, the proposed parameter-free correction is closely related to existing label propagation and prediction smoothing methods. The methodological contributions beyond these established techniques should be clarified.

W3. The neighbor residual does not consistently improve performance across GFMs and can even degrade accuracy and calibration. More importantly, the experiments mainly report accuracy improvements rather than directly demonstrating reductions in the proposed transfer-failure metric. This weakens the connection between the identified failure mechanism and the proposed correction.

W4. The causal investigation focuses primarily on GraphPFN, making it unclear whether the identified mechanism applies to other architectures. Moreover, removing graph adapters identifies their contribution to density sensitivity but does not fully explain the internal mechanism responsible for the observed failures.

W5. The benchmark primarily considers nested edge-density changes generated through a specific synthetic graph family. It remains unclear whether the findings extend to other structural changes, such as degree-distribution shifts, topology rewiring, or changes in class-compatibility patterns independent of density.

Questions for authors

See the above

5 comments
AuthorsOpenReview ↗

Response to Reviewer SDwS — W1

W1: synthetic dependence and practical significance

Natural graphs contain observational noise, and density changes often coincide with changes in node sampling, features, or the relationship between labels and topology. This makes it difficult to attribute a cross-dataset performance difference to density alone. Our synthetic graph benchmark makes a specific capability test repeatable: paired density views share nodes, features, labels, and edge randomness, and matched/mismatched contexts are compared on the same query. The crossed grid separates ordinary density effects from the additional mismatch penalty. Synthetic generation also lets us systematically vary target density, homophily, and degree-tail regimes while retaining the paired task.

To assess external validity, we added temporal-contact comparisons on HighSchool2013 and publication-year snapshots of ogbn-arxiv. The original 36-cell contact study is extended by 12 runs with earlier, consistently sparser context; the citation study adds 20 runs using the original 40 subject classes. On fixed late queries, contact matched-late-minus-day-one accuracy changes range from −11.11 to −0.69 points; citation natural-2019-minus-2013 changes range from −11.39 to +13.87. Later or closer-density context provides no consistent benefit. These are descriptive temporal contrasts with overlapping banks and coupled structural/task changes; Appendix D.4 states their assumptions and the citation reference's approximate density match.

The existing fixed-edge task controls also help interpret the synthetic–real gap. On Penn94, Cornell5, and JohnsHopkins55, replacing labels and features with the generator's process while retaining the same real edges restores GraphPFN adapter-path failure contrasts of 5.0–17.1 percentage points, compared with −0.1 to +0.8 under the original tasks. Squirrel does not show comparable restoration. Thus the pathway can arise on real topology under a different task process, but its occurrence depends on more than the edge set alone.

Separately, two newly sampled synthetic populations reproduce mean failure of 15.27–16.18 points for GraphPFN and 10.02–11.91 for G2T-FM, with all primary 97.5% intervals positive. A triggered strict-mixing sensitivity study on those same populations retains the effect. These tests strengthen the reliability of the controlled finding.

We therefore retain the benchmark's contribution as a repeatable test of density-transfer capability and the measured GraphPFN pathway contribution. We have aligned the practical framing with the evidence: the controlled failure is reproducible, while its prevalence and magnitude in naturally occurring tasks remain open.

Changes: Motivation and scope in the Abstract, §1 and §§6–7; independent-population and strict-mixing tests in Appendix C.1, Table 5; fixed-real-edge task controls in Appendix D.3, Table 16; temporal-contact and citation-growth studies and assumptions in Appendix D.4, Tables 17–19. Real-label thinning appears in Appendix D.2, Table 15.

AuthorsOpenReview ↗

Response to Reviewer SDwS — W2

W2: contribution beyond the generator, decomposition, and smoothing

Thank you for asking us to distinguish the established components from our contribution. The paper contributes a controlled evaluation and experimentally tested explanations of density-transfer behavior. The three components have distinct roles:

  1. From an existing generator to a paired benchmark. We build on GraphUniverse's synthetic graph model. Our extension renders the same latent task at nested densities, sharing nodes, features, labels, and potential-edge randomness. Disjoint context and query instances are then crossed over the density grid, providing a matched-context reference on the same query for every shifted cell. This design makes the effect of changing context density measurable without changing query difficulty or independently resampling query edges. Reporting matched accuracy, prediction stability, and the full response surface also distinguishes useful robustness from weak capability or cancellation across shift directions.

  2. Using a standard decomposition to answer a specific question. The additive-interaction decomposition and mean-failure identity are standard algebra. On the complete, equally weighted grid, main effects already cancel from mean failure, leaving an aggregate matching-interaction contrast. The full accuracy surface additionally retains context/query trends and direction-specific behavior. A sparse-to-dense accuracy difference alone cannot establish an additional mismatch penalty. The contribution is the paired experimental design and the model-dependent behaviors it reveals, including substantial, reproducible matched-density interactions for GraphPFN and G2T-FM. The fixed-checkpoint GraphPFN interventions then identify a contributing adapter pathway.

  3. A propagation-based residual with an explicit loss explanation. The residual belongs to the label-propagation/prediction-smoothing family. Its concrete construction uses class compatibility estimated from labeled context edges to convert query-neighbor pseudo-labels into degree-averaged evidence, matches that evidence to each node's logit scale, and adds it once with a fixed half-step. This requires no learned weights or model-specific tuning. The explicit message supports a testable account of its cross-entropy effect through error alignment and curvature. Using evaluation labels, the approximation matches the NLL-change sign in all 24 decisive conditions among 25 tested. Smoother controls establish model-specific benefits: pooled accuracy gains for the residual versus one-step LP are +2.7 versus +1.7 percentage points for G2T-FM, and +1.4 versus +1.8 for SAMGPT. Thus the evidence supports a conditional explanation and useful corrections for particular models, rather than general superiority over propagation methods.

Clarifications and evidence in the revised manuscript: Contribution statements in §1; paired design and generator attribution in §§3.1–3.2 and Appendix A.1; balanced-grid identity and full-surface interpretation in §3.4 and Appendix B.1; propagation relationship in §2; residual construction and explanation in §§4.2–4.3 and §5.4, with smoother/alignment results in Appendix C.5, Tables 13–14.

AuthorsOpenReview ↗

Response to Reviewer SDwS — W3

W3: does the residual reduce transfer failure, and what about calibration?

We agree that accuracy gains alone do not demonstrate reduced transfer failure. We added direct evaluations of the proposed metric and revised the correction claim accordingly.

Direct failure evaluation. On both newly sampled independent populations, we applied the fixed residual to complete density grids, correcting both the matched references and mismatched predictions. Defining improvement as base-minus-corrected failure, GraphPFN gives −2.31 [−2.81, −1.84] and −0.92 [−1.43, −0.38] percentage points: failure increases in both populations. G2T-FM gives +1.01 [0.78, 1.28] and −0.06 [−0.52, 0.36]: one population improves, while the other is unresolved. These are paired 95% intervals. The separate historical post-hoc GOBLIN grid supplies a positive example, 2.218→1.298 points; it is not pooled with the new populations. Legacy SAMGPT/MDGFM caches lack corrected matched cells, so their accuracy gains cannot establish failure reductions.

Accuracy and NLL. We report these outcomes separately. At GraphPFN depth six, accuracy gains of 1.6–4.9 percentage points accompany NLL increases of 0.13–0.36. The error-alignment/curvature analysis accounts for the tested loss responses, including deterioration, using evaluation labels; it does not guarantee improved accuracy or calibration. We retain the adverse outcomes alongside the model-specific benefits.

Separate preprocessing and message-strength controls. Fitting G2T-FM's degree/PageRank transformations per graph reduces failure by 1.88 [1.38, 2.38] and 3.23 [2.88, 3.59] points, with matched-accuracy changes of +0.01/−0.04; both populations pass the fixed two-point capability-loss bound. Additional receiver-degree division lowers GraphPFN failure from 16.18/15.27 to 0.22/0.51 points, improves cross-density accuracy by 5.83/6.97, and costs 10.14/7.79 matched points. Thus it supplies a strong mitigation with a capability tradeoff. These controls are separate from the neighbor residual.

We therefore describe the residual as a mechanism-inspired output-space probe and a conditional correction. The experiments do not establish that it reverses GraphPFN's internal failure pathway or generally reduces transfer failure. This narrows the repair claim while retaining the residual's tested loss explanation.

Changes: Residual contribution and interpretation in §1 and §§4.2–4.3; accuracy/NLL distinction in §5.2 and §5.4; direct residual failure in Appendix C.2, Table 8; preprocessing and message-strength controls in §4.1 and Appendix C.2, Tables 6–7.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions and additional experiments

We thank the reviewers for their constructive comments. We address reliability, practical relevance, mechanism, and novelty through new experiments and clearer claims. Locations refer to the revised manuscript.

  1. Reliability of the controlled finding. Two independently sampled synthetic populations, with new labels, features, assignments, and edge randomness, retain mean failure of 15.27–16.18 percentage points for GraphPFN and 10.02–11.91 for G2T-FM; all primary 97.5% intervals exclude zero. Stricter class-pair-mixing controls on these same populations retain the effect; they are sensitivity tests, not another independent replication. Locations: Appendix A.1; C.1, Table 5.

  2. Practical relevance. Natural graphs are noisy and undergo coupled changes; our synthetic benchmark holds nodes, features, and labels fixed across density views. We added HighSchool2013 contacts, including an earlier sparse context [1], and ogbn-arxiv snapshots [2]. On identical late queries, citation late-minus-early context changes span −11.39 to +13.87 points; contact matched-late-minus-day-one changes span −11.11 to −0.69. Density matching provides no consistent benefit. These mixed results and smaller real-label thinning/rewiring effects limit extrapolation without changing the controlled finding. Locations: Abstract, §1, §6; Appendix D.2–D.5, Tables 15–21.

  3. Intervention outcomes and explanation. Additional degree division lowers GraphPFN failure from 16.18/15.27 to 0.22/0.51 points, gaining 5.83/6.97 cross-density accuracy points while losing 10.14/7.79 matched points. G2T-FM per-graph degree/PageRank transforms reduce failure by 1.9–3.2 points with little matched-accuracy change. Mixed-density contexts recover 74–101% of the average gap across both models and populations (point estimates). The explicit residual message, error alignment, and curvature explain tested cross-entropy responses using evaluation labels. Inconsistent residual failure reductions and the failed fixed-message absorption test narrow repair and internal-mechanism interpretations; the controlled finding, adapter contribution, and conditional loss explanation remain. Locations: §§4–5; Appendix C.2, Tables 6–8; C.5, Table 14.

  4. Contribution and reproducibility. We clarify paired same-query evaluation, full response surfaces, and the explicit compatibility message; the decomposition and loss expansion are standard tools. We report competitive label-propagation results, actual graph statistics, screening history, 2×/4× gaps, stratified surfaces, Table-2 intervals, and revised figures. An anonymous source/cache supplement includes numerical replay commands. Locations: §§1–3, especially §3.4; §5.4; Table 2, Figures 1–3; Appendix B.1, C.5, E; Reproducibility Statement.

Claims retained and revised.

  • Controlled failure: strengthened by independent populations and mixing controls, alongside capability and stability checks. Locations: §3; Appendix C.1, Table 5.
  • Adapter contribution: retained. Fixed-checkpoint interventions identify a contributing pathway, not its exact hidden-state mediator. Locations: §4.1; Appendix C.2, C.4.
  • Residual loss explanation: retained, conditional on the tested message and evaluation labels. Locations: §§4.2–4.3, §5.4; Appendix C.5, Table 14.
  • Natural-task prevalence: not established by smaller or mixed real-graph effects. Locations: Abstract, §1, §§6–7; Appendix D.
  • Evidence absorption: inference withdrawn. The fixed-message control does not support the predicted reduction in benefit with depth. Locations: §5.2; Appendix C.2.
  • Repair: interventions distinguished. Residual failure reduction is inconsistent; separate controls mitigate mismatch with capability tradeoffs. Locations: §4.1, §5.4; Appendix C.2, Tables 6–8.

References

[1] Mastrandrea et al. (2015). Contact Patterns in a High School, PLoS ONE. HighSchool2013 data.

[2] Hu et al. (2020). Open Graph Benchmark, NeurIPS. ogbn-arxiv data.

AuthorsOpenReview ↗

Code and reproducibility materials

We have prepared an anonymous supplement with experiment source, versioned configurations, source/checkpoint hashes, per-graph outcomes, and CPU commands for recomputing the paired estimates and intervals. Its README maps descriptive study names to manuscript sections and outputs; a separate provenance index links archived source identifiers. It includes cached predictions for the original temporal-contact study, its first-day supplement, and ogbn-arxiv citation growth, with identity/label checks. Additional receiver-degree division is covered by source, all per-query metrics, and paired endpoint replay. A portable native GraphPFN/G2T launcher supports separately installed public model environments and weights, configurable ensemble counts, and the degree-division intervention.

The CPU replay checks independent-population/control endpoints and original decisions, strict mixing, the fixed-message test, residual alignment, temporal contrasts, and complete real-graph rewiring matrices. It also recomputes Table-2 intervals, realized graph statistics, paired drift, depth failures, density-ratio results, stratified surfaces, and candidate-stage counts. The candidate ledger validates retained QC decisions and explicitly marks protocol-only descriptions of deleted records. Failed tests, negative/mixed outcomes, and the retrospective status of the rewiring analysis are retained.

The lightweight supplement provides cached-result reproduction and a native inference entry point. Model weights, environments, and the full historical raw data store are not bundled; no fresh GPU inference or full historical retraining is claimed for the replay. The detailed coverage and commands are in the README.

Changes: Reproducibility Statement; Appendix A.4 and Appendix E; anonymous supplement README, evidence index, and native-inference guide.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.