acceptodds
Under review as a conference paper at ICLR 2027

Why Graph Pretraining Fails to Transfer: Capability Ceiling and Accessibility Deficit

20 views

Abstract

Graph self-supervised learning (SSL) provides the foundation for reusable graph representations. However, cross-domain pretraining often fails to match target-domain SSL performance, leading to incomplete or negative transfer. We decompose this transfer gap into two distinct mechanisms: *capability loss*, where target-task information is not retained, and *finite-supervision accessibility loss*, where retained information is difficult to extract from limited target labels. To unify both mechanisms, we propose the Task–Geometry Profile (TGP), a theoretical framework that models representation-mode strength alongside target-task signal alignment. Our analysis reveals that positive transfer depends not only on how much task-relevant signal is retained, but also on where that signal is organized in representation geometry. Furthermore, we show how graph aggregation architectures distort this profile through distinct inductive biases: GCN's normalized diffusion induces mode-selective spectral attenuation, whereas GIN's unnormalized accumulation causes degree-driven modal reallocation. Across diverse domains, SSL objectives, backbones, and 120 transfer settings, TGP quantities show systematic correspondence with empirical capability and accessibility behavior. Mechanism-guided interventions further reveal that the two components need not improve together: degree-tempered aggregation reduces the trajectory-level transfer deficit by up to 56% on the classical benchmark suite, while large-scale validation shows substantial absolute utility gains.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Learning, Graph Self-supervised Learning, Cross-Domain Generalizaiton

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KPT4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why cross-domain graph SSL often underperforms target-matched pretraining. The authors decompose the transfer gap into capability loss and accessibility loss, and propose the task–geometry profile (TGP) to characterize both the amount of retained task signal and its placement in representation geometry. They develop a linear theory based on optimal predictors and a spectral ridge surrogate, link TGP distortion to GCN and GIN aggregation mechanisms, and validate the framework through extensive transfer experiments across domains, SSL objectives, backbones, and probes.

Critical strengths

The proposed TGP provides an intuitive framework linking transfer performance to both the amount and placement of task-relevant signal, and is supported by extensive empirical evaluation across diverse transfer settings.

Critical weaknesses

The paper’s central explanatory claims are not fully supported by the theory. The capability–accessibility decomposition is exact only within a constructed population-eigenmode ridge surrogate, while the experiments use finite-sample classification objectives and nonlinear probes whose connection to this surrogate remains empirical. Moreover, Section 3.5 establishes that graph aggregation changes feature covariance, but not that it relocates task signal, which additionally depends on target-label alignment. Consequently, the proposed framework is an informative diagnostic, but its ability to explain the observed transfer behavior and support architecture-level conclusions is less rigorous than claimed.

Questions for authors
  1. The theory is developed under a population-optimal linear predictor and a spectral ridge surrogate, whereas part of the empirical evaluation uses classification metrics and nonlinear MLP probes. Could the authors provide additional evidence (empirical or theoretical) that the proposed capability and accessibility quantities remain meaningful beyond the linear ridge setting?
  2. Section 3.5 argues that graph aggregation can relocate task signal toward weaker representation modes. However, the analysis primarily derives changes in covariance structure, while the task-signal component additionally depends on target-label projections. Could the authors clarify what assumptions are required to connect aggregation-induced covariance distortion to changes in the full TGP?
Reviewer_zvGnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the problem of understanding and mitigating the lack of positive transfer in cross-domain graph pretraining. It decomposes the transfer gap between cross-domain pretraining and target-domain self-supervised learning (SSL) into two components: capability loss (how much information is simply not accessible after pretraining) and finite-supervision accessibility loss (how much is simply represented in parts of the representation space geometry that are inccessible without more supervision). It proposes a quantity called the Task Geometry Profile (TGP) that explains how much linearly predictable target variance is explained by each direction of variation in the embeddings, and connect TGP distortion to canonical graph aggregation mechanisms. Across two backbone models, five standard node classifiation datasets, and six graph SSL objectives, a correlation is found between the TGP and the capability/accessory gaps it diagnoses, and graph aggregation mechanism changes are shown to improve transfer deficit.

The theory on this work makes for an interesting read and is overall well written. While the TGP is eventually connected to graph-specific aggregation mechanisms, the decomposition into capability and accessibility failure modes does not seem to be graph-specific and may overlap with similar risk decompositions proposed for other domains (e.g. [1], which similarly decomposes into errors due to finite training data and errors due to lack of encoder architecture capacity). The paper should discuss what is novel relative to such prior work or reframe the contributions to make clear what is background versus original analysis.

However, most datasets are small with the OGBN-ARXIV dataset being the largets (~1e5 nodes). More importantly, these datasets are well-labeled, but it is often desirable to pretrain on unlabeled data. When labels are scarce, estimating the feature-label cross-moments becomes more challenging, and this could potentially inhibit the usability of the proposed empirical audit in such settings. Similarly computational dependence on embedding dimension (a full O(d^3) eigendecomposition is required) and dataset size (computing a second target-matched pretraining run could be painful for truly large-scale pretraining.)

Several experimental findings are promising, but more rigorous support is needed for the claims that are drawn from them. For example, TGP predictions are shown to have appreciable correlation with measured learning-curve quantities in general, but providing little ranking information for accessibility in GIN in Fig. 3. Similarly, in Tab. 1, the change in measured accessibility and its proposed predictor often seem to move in opposite directions with GCN regularization. Finally, the interplay between capability and accessibility needs to be carefully disentangled, as at the largest (but still finite-budget) endpoint, there is still an accessibility effect. How good a proxy for pure capability effects a large finite budget affords has yet to be determined.

[1] Dubois, Y., Hashimoto, T. & Liang, P. (2023). Evaluating Self-Supervised Learning via Risk Decomposition. ICML 2023.

Critical strengths

S1: The proposed method is experimentally used to audit both dataset-level and model-level effects on transfer, across a range of dataset sizes and common graph learning paradigms (backbone / SSL objective). S2: Potential ability to diagnose effective dataset-specific GNN mechanism choices is a useful capability.

Critical weaknesses

W1: Method is evaluated in small to moderate-size (dataset and embedding dimension), well-labeled regimes. Unclear how usable it is for very large dataset size (a second full pretraining run is required as a reference), label scarcity (affects estimation of feature-label interaction), and possibly high embedding dimension (full eigendecomposition is required). W2: Needs to establish that the changes in TGP actually demonstrate a change in the capability / accessibility that they purportedly track. For instance, TGP predictions are shown to have appreciable correlation with measured learning-curve quantities in general, but providing little ranking information for accessibility in GIN in Fig. 3. Similarly, in Tab. 1, the change in measured accessibility and its proposed predictor often seem to move in opposite directions with GCN regularization.

Questions for authors

Is the decomposition into capability and accessibility novel, and if so, how does it compare to similar decompositions into finite-data regime and encoder lack of capability in related works for other domains e.g. [1]?

How do you establish distinct mechanisms established by the capability/accessibility decomposition? Capability and accessibility do seem to be entangled even at endpoints with high label budgets.

[1] Dubois, Y., Hashimoto, T. & Liang, P. (2023). Evaluating Self-Supervised Learning via Risk Decomposition. ICML 2023.

4 comments

Thank you again for your thoughtful and constructive feedback. As the public discussion period is approaching its deadline (November 18), we would like to politely follow up and let you know that we have uploaded our revised manuscript, with all changes highlighted in red. In particular, we have formally incorporated the comparison with Dubois et al. (ICML 2023), the finite-endpoint disentanglement results, the intervention-tracking analyses, and the scarce-label scalability.

Could you kindly let us know if our response and revised manuscript resolve your concerns, and whether you have any other questions we can further clarify before the discussion closes? Thank you very much for your time!

Thank you for these detailed updated results. The experimental evaluation is considerably more extensive, and my concerns about scalability and to a lesser extent reliability with scarce labels have been addressed. I do have some concerns about the ability to track accessibility or separate it from capability. For instance, in the endpoint analysis for GCN/GAE/Cora, the capability term is 0.0333, while the residual accessibility contributes 0.0593 (larger than the capability), so it seems that the endpoint is not primarily a capability measurement. In the detailed tracking measurements, it seems there are some opposing trends across architectures (e.g. GCN accessibility predictor falling over layers, while empirical accessibility with an MLP probe rises over layers on Computers), and cases where linear accessibility seems to be outright difficult to track at all meaningfully (e.g. linear accessibility sign agreement of 54.2% on frozen GIN interventions). Though I see promise in the current contributions, I think more work needs to be done to make sense of this large number of results and reconcile some messy empirical trends.

Thank you for engaging so thoughtfully with our new results and for confirming that your concerns regarding scalability and label scarcity are addressed.

Your three observations about endpoint coupling and tracking trends actually highlight a central takeaway of our work: raw empirical learning-curve summaries (ΔC,ΔA\Delta_C, \Delta_A) are inherently coupled at finite nmax⁡n_{\max} and across readout families, and reconciling them is precisely why an explicit representation-level model (TGP) is needed. Below, we briefly clarify how TGP reconciles all three cases:

1. Complete decoupling at a finite endpoint nmax⁡n_{\max} is impossible in raw curves—which is why Theorem 1 is needed (GCN/GAE/Cora):
We fully agree that the empirical endpoint ΔC=G(nmax⁡)\Delta_C = G(n_{\max}) is not a pure capability measurement when nmax⁡n_{\max} is moderate and weak modes carry substantial signal. Our paper distinguishes two levels:

  • Curve summaries (Sec. 2.2): ΔC=G(nmax⁡)\Delta_C = G(n_{\max}) and ΔA=AG−ΔC\Delta_A = A_G - \Delta_C summarize the observed finite-budget trajectory without a model.
  • Population vs. finite-sample separation (Theorem 1, Eq. 11): Rs,nmax⁡−Rt,nmax⁡=ΔΨ+[As(nmax⁡)−At(nmax⁡)]R_{s,n_{\max}} - R_{t,n_{\max}} = \Delta_\Psi + [\mathcal{A}_s(n_{\max}) - \mathcal{A}_t(n_{\max})] analytically separates true population capability loss (ΔΨ\Delta_\Psi) from the unresolved accessibility residual at nmax⁡n_{\max}.

GCN/GAE/Cora (nmax⁡=947n_{\max} = 947) is the clearest illustration of why this theoretical separation matters: GAE traps 11.2% of source task signal in the bottom-25% weakest modes (vs. 0.6% in target). Looking only at the empirical endpoint (ΔC=0.0895\Delta_C = 0.0895) would misattribute this gap to lost capability; Theorem 1 reveals that two-thirds of this finite-nmax⁡n_{\max} gap (0.05930.0593) is actually unresolved weak-mode signal rather than lost population capability (ΔΨ=0.0333\Delta_\Psi = 0.0333). On larger label pools (Amazon-Computers, nmax⁡≈4813n_{\max} \approx 4813), the median residual drops nearly 5×5\times (to 0.00490.0049), closer to the asymptotic regime of Corollary 1.

2. Reconciling GCN / Computers across layer:
Because TGP (ΔΨ,ΔATG\Delta_\Psi, \Delta_A^{\text{TG}}) is a linear spectral surrogate, on GCN / Computers (layer_1 →\to layer_3) it tracks the Linear probe's exact trajectory: as depth increases from ℓ=1→3\ell=1 \to 3, weak-mode signal is further attenuated into unrecoverable linear loss, so ΔΨ\Delta_\Psi rises (+0.0713→+0.1157+0.0713 \to +0.1157) alongside Linear ΔC\Delta_C (+0.0493→+0.0928+0.0493 \to +0.0928), while ΔATG\Delta_A^{\text{TG}} falls (+0.3012→+0.1548+0.3012 \to +0.1548) alongside Linear ΔA\Delta_A (+0.0178→+0.0033+0.0178 \to +0.0033).

Why does the MLP probe show rising ΔA\Delta_A (+0.0356→+0.0619+0.0356 \to +0.0619) and small ΔC\Delta_C (+0.0288→+0.0312+0.0288 \to +0.0312)? Examining the total trajectory deficit AG=ΔC+ΔAA_G = \Delta_C + \Delta_A reveals that Linear and MLP incur nearly identical total deficits at every layer:

  • layer_1: Linear AG=0.0671A_G = 0.0671 vs. MLP AG=0.0644A_G = 0.0644
  • layer_2: Linear AG=0.0786A_G = 0.0786 vs. MLP AG=0.0843A_G = 0.0843
  • layer_3: Linear AG=0.0961A_G = \mathbf{0.0961} vs. MLP AG=0.0931A_G = \mathbf{0.0931}

At the large Computers endpoint (nmax⁡≈4813n_{\max} \approx 4813), the nonlinear MLP recovers part of the linearly attenuated signal (reducing ΔC\Delta_C from 0.09280.0928 to 0.03120.0312), which converts what is a permanent endpoint loss (ΔC\Delta_C) for a linear readout into a finite-sample accessibility cost (ΔA=AG−ΔC=0.0619\Delta_A = A_G - \Delta_C = 0.0619) for a nonlinear readout.

3. Post-hoc Frozen-GIN (54.2% Linear ΔA\Delta_A sign agreement) vs. Pretrained DT-GIN (100%):
In GIN Frozen-$\beta$, encoder weights and BatchNorm statistics remain frozen from vanilla GIN (β=0\beta=0) pretraining, where degree amplification has already collapsed weak-mode signal to ∼10−8–10−5\sim 10^{-8}\text{–}10^{-5}. Changing β\beta only at inference time rescales the dominant modes (tracking ΔΨ\Delta_\Psi at ρ=0.540\rho = 0.540, 75% sign agreement), but cannot populate weak modes that were never learned during pretraining (weak-mode fraction remains <10−4<10^{-4} in Table 2a), leaving linear ΔA\Delta_A shifts near noise (54.2%). In contrast, when DT-GIN is applied during pretraining (β=0.8\beta=0.8), the encoder learns a redistributed spectrum, and TGP tracks both capability recovery and the opposing accessibility shift with 100% (Linear) and 83.3% (MLP) sign agreement.

Thanks again for your constructive comments. We hope these clarifications address your concerns.

Reviewer_zvGnOpenReview ↗

Improving, needs a little stronger results on predicting accessibility

Thank you for these additional experiments. At this time, I'm maintaining my score; I do think the paper has a lot of promising empirical contributions and some initial concerns I had about its applicability have certainly been resolved, and I wouldn't mount an objection to the paper's acceptance if others advocate for it. Mainly I still see a gap in how well accessibility can be estimated and am not clear on when the accessibility estimation can be trusted. For instance, accessibility prediction seems to be possible on, say, GCN but not on GIN; the authors show that under their stabilized audit, vanilla GIN has almost no estimated task signal in the bottom quarter of its spectrum, but this doesn't suffice to demonstrate that there is no meaningful accessibility variation left. The stabilization also changes the empirical procedure from the theory that characterizes it and may blur the distinction between capability and accessibility. I think the paper's contributions of a diagnosis of some kind of transfer-affecting deficit is supported; characterizing it as capability vs accessibility seems to still be not as quite strong as claimed.

Reviewer_3yF3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies an important gap in graph pretraining and transfer: why SSL methods work pretty well on individual graphs (the target graph) yet fail when transferring from a different source? Lack of a fundemantal understanding of this problem has blocked the progress of related fields like graph foundation models for several years. Topicwise, I think this is a good paper; However, I have serious concerns regarding the specific techniques and the presentation.

Critical strengths

The paper targets an interesting question and tackles it in a reasonable way. The decoupling of information retention and accessibility makes sense.

The experimental design largely demonstrates the usability of the proposed diagnostics.

Critical weaknesses
  1. The biggest issue is that the paper does not answer the question in its title. It claims to answer "why pretraining fails," but the experiments only provide indirect evidence showing correlations between the diagnostic metrics and the actual performance. The correlations are not consistently strong either. After reading the paper, it is still unclear what is actually blocking successful transfer.

  2. Also, the depth of the paper is very limited. It proposes two diagnostic metrics, which are based on standard tools in representation learning. It does not connect the observed phenomena, such as information loss, to the underlying mechanisms. E.g., what specific pretraining methods tend to preserve which information? what signals are needed for the downstream task? Although they used different SSLs in the experiment, no discussion is provided on the distinctions of these methods and how they lead to different transfer behaviours (if any).

  3. The scope of experiments is also too limited for a paper with such claims. They only tested on 5 datasets, far from covering the diverse properties of graphs. Architectures only include GCN and GIN, leaving the critical components like attention and context sampling out of discussion.

  4. The presentation also needs substantial improvement. The draft interleaves too many equations in the main text, which does not help understand the core points and contributions.

Questions for authors
  1. why not seperate the observations SSL-wise? It would be really interesting and greatly enhance the paper if discussions are provided on "what each SSL preserves" and "how good/bad they transfer".

  2. What is the point of seperating the Linear and MLP probe heads? Are the conclusions fundemantally different from their results?

1 comment

Thank you for your time and review. As the public discussion deadline (November 18) is approaching, we are writing a brief follow-up to let you know that we have uploaded the revised manuscript, with all changes highlighted in red. The revision incorporates our additional experiments on GAT and Sampled GraphSAGE, SSL-wise property recoverability, and the ΔΨ\Delta_\Psi-to-Macro-F1 bridge, along with clear pointers to our existing OGBN-Arxiv and SSL-wise results.

We would greatly appreciate it if you could let us know whether our response and revised manuscript have addressed your concerns. Please feel free to let us know if there are any remaining points you would like us to clarify before the deadline. Thank you very much for your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewer Feedback and Key Rebuttal Evidence

Dear Area Chair,

Thank you for overseeing the review of our submission. Since the rebuttal period ends soon and we have not yet heard back from Reviewer 3yF3, we would like to briefly summarize the reviewer feedback and highlight the key evidence we added in response to their review.

Reviewers KPT4 and zvGn both engaged with our responses and confirmed that our additional experiments resolved key concerns. Specifically, Reviewer KPT4 (Rating: 3, weak acceptance) confirmed that "all of my concerns have been addressed" and maintained their support for the paper. Reviewer zvGn (Rating: 2) confirmed across two rounds of discussion that our scalability and label-scarcity experiments addressed their applicability concerns, noted the paper's "promising empirical contributions," and explicitly stated that they "wouldn't mount an objection to the paper's acceptance if others advocate for it."

We have not yet received a follow-up from Reviewer 3yF3 (Rating: 1). Below, we briefly summarize how we addressed their main concerns:

1. Factual clarifications on existing content. Several core criticisms in Review 3yF3 overlooked results already present in the submission: the paper evaluates 6 datasets including large-scale OGBN-Arxiv (Sec. 4.4, Table 4), conducts controlled mechanistic interventions beyond correlations (DT-GIN reduces AGA_G by 49%–56% and boosts OGBN-Arxiv Macro-F1 by +0.291 to +0.322; RR closes 95% of GCN's OGBN-Arxiv capability gap; Secs. 4.3–4.4), and reports SSL-disaggregated results (Fig. 2, Appendix F Fig. 6, Appendix G Tables 5–7).

2. Bridging ΔΨ\Delta_\Psi to Macro-F1 and SSL-wise information recoverability. We added a multi-scale ridge audit on GIN/Cora bridging quadratic risk to held-out MSE (ρ=0.829\rho = 0.829) and Macro-F1, and evaluated attribute, clustering, and degree recoverability across all 6 SSL methods (24 settings), showing that preserving generic structure (e.g., degree R2∈[0.91,0.98]R^2 \in [0.91, 0.98] on GIN) does not guarantee downstream transfer utility.

3. Generalization to attention and neighborhood-sampled architectures. We extended TGP evaluation to 3-layer GAT and neighborhood-sampled GraphSAGE (fanout [15,10,5][15, 10, 5]), achieving 100% capability sign agreement (ρ=0.600–0.829\rho = 0.600\text{–}0.829) and strong accessibility correspondence on sampled GraphSAGE (ρ=0.829\rho = 0.829 Linear / 0.8860.886 MLP) and GAT/MLP (ρ=0.600\rho = 0.600).

We appreciate the reviewers' feedback and thank you for considering these results in your assessment.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.