- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how the predictive performance of GFMs varies when context and query graphs have different edge densities. The authors introduce PairShift, a benchmark that constructs paired density views with nested edge sets while keeping nodes, features, labels, and edge randomness fixed. It compares model performance on the same query graph using context graphs with either matched or mismatched densities. The authors further investigate GraphPFN’s failure pathway through fixed-checkpoint interventions on its graph adapters and evaluate a class-compatibility neighbor residual that uses query pseudo-labels to adjust the base predictions.
Critical strengths
1.The paired context-by-query design is sound and yields results that are straightforward to interpret. The benchmark methodology and analysis are clearly presented. 2.The fixed-checkpoint interventions on GraphPFN’s graph adapters are well motivated.
Critical weaknesses
1.In Section 5.2, the authors interpret the reversal in residual gains across adapter depths as evidence that the graph adapters have already absorbed related neighbor evidence, making the residual redundant or harmful. However, according to Equations (4)–(6), the query pseudo-labels used to construct the residual come from the base model’s predictions, and the residual is scaled to match the base-logit scale. Both may therefore change with adapter depth. It is unclear whether the reversal is due to redundant neighbor evidence or changes in the residual itself. 2.For GraphPFN, the authors report a large adapter contribution to mean transfer failure in synthetic universes, but a much smaller contribution on real graphs with original labels. I wonder why would this happen and under what specific practical task conditions would this failure mode be expected to occur? I expect more specific analysis.
Questions for authors
See the weaknesses.
3 comments
Response to Reviewer tMVT — W2
W2: why do synthetic and real effects differ, and when should practical failure be expected?
The strongest tested explanation concerns the relationship between the task and topology. Keeping real edges and blocks fixed, we replace both labels and features with the generator's task process. The GraphPFN adapter contribution then changes as follows; all values are percentage points, and ranges span planted settings rather than confidence intervals:
| Fixed real edge set | Original task | Generator label/feature process |
|---|---|---|
| Penn94 | +0.8 | +11.2 to +17.1 |
| Cornell5 | +0.6 | +5.0 |
| JohnsHopkins55 | −0.1 | +10.8 |
| Squirrel | +2.0 | −0.1 to +2.3 |
Thus real topology can carry a large adapter-path effect under a different task process. The Squirrel exception, including larger-block controls, shows that the edge structure also matters. Because labels and features change together, this comparison does not identify labels alone as the cause. Feature degradation and larger blocks keep the original-task effects small, arguing against a simple accuracy-ceiling explanation. High homophily is also insufficient to explain the gap: it can increase failure within the controlled generator, where substantial effects persist at a fourfold density gap.
For practical conditions, these results suggest a candidate risk setting: a model relies on class-relevant neighborhood patterns transferred from context, and changing density alters how that neighborhood evidence is used. This is a hypothesis motivated by the controls, not a validated sufficient condition. Our results do not establish a reliable natural-task risk rule based on mean degree or homophily alone; the frozen real-graph context-selector test also remains failed.
We further tested naturally timed changes using HighSchool2013, including an earlier consistently sparse context, and ogbn-arxiv citation snapshots with their original subject labels. On identical late queries, GraphPFN's matched-late-minus-day-one contact differences are −11.11/−4.17 points across banks; its natural-2019-minus-2013 citation differences are −6.45/−11.39. G2T-FM's citation differences are +13.87/−7.62. These substantial but mixed context effects do not establish a consistent benefit from later or closer-density context. The citation density reference is explicitly approximate, and overlapping banks and coupled topology/task changes make these descriptive comparisons rather than isolated density effects.
The revised motivation explains why synthetic graphs provide repeatable control of these coupled factors. We have aligned the Abstract, Introduction and Discussion with the evidence: task–topology dependence is supported, while the prevalence and sufficient conditions of this failure on natural tasks remain open.
Changes and evidence: Task-process and ceiling controls in Appendix D.2–D.3, Tables 15–16; temporal contacts and citation growth in Appendix D.4, Tables 17–19; the original selector decision in Appendix D.5, Table 20. Motivation and scope are revised in the Abstract, §1 and §6.
The reply has addressed most of my concerns and I will maintain my rating of 3.
Response to Reviewer tMVT — W1
Thank you for recognizing the paired evaluation and the motivation for the fixed-checkpoint interventions. We have added a fully fixed-message control and expanded the task and temporal analyses to address your two questions.
W1: do changing pseudo-labels and scale confound the depth reversal?
You are correct: the original experiment holds the residual update rule fixed, while its pseudo-labels and logit-based scale change with adapter depth. That comparison establishes a depth-dependent response but cannot isolate evidence absorption.
We added a cached counterfactual that freezes the complete depth-zero update vector, including pseudo-labels, transition estimates, centering, scale, and strength, and applies exactly that vector at depths 0/6/12 on the same query graphs. The fixed-message accuracy summaries are reported below alongside the primary loss contrast. Let Δd be corrected-minus-base NLL at depth d and Q = Δ12 − Δ0. The proposed reduction in NLL benefit at depth twelve predicts Q > 0.
These estimates average the two extreme directions, 2→32 and 32→2. Accuracy values are descriptive means from the same saved control; Q intervals resample query latent identities jointly across strata and directions. With the complete message fixed, the original positive-to-negative accuracy reversal does not reappear, and all three Q intervals lie below zero. The control therefore does not support the stronger absorption prediction. It does not determine whether pseudo-label changes, scaling, or their interaction dominates the original reversal.
We retain the observed depth dependence when each depth constructs its own residual, remove the internal-absorption inference, and rename §5.2 accordingly. The measured adapter-path contribution and the explicit residual's conditional loss explanation through error alignment and curvature remain supported; the latter uses evaluation labels.
Changes and evidence: §4.3, §5.2, §5.4 and §6; fully fixed-message protocol, primary NLL results, and descriptive accuracy summaries in Appendix C.2. Per-query outcomes from this cached experiment are in the supplement. Native 0/6/12 failure is reported separately in Appendix E.3, Table 27.