acceptodds
Under review as a conference paper at ICLR 2027

More Domains, No Gains? FLO: Fast–Slow Reciprocal Learning for Multi-Domain Graph Pre-training

Abstract

Inspired by the great success of foundation models in the development of artificial general intelligence, Graph Foundation Models (GFMs) have recently attracted significant attention and have the potential to advance graph intelligence. Exhibiting such intelligence requires effectively learning knowledge from large-scale, multi-domain graph data during pre-training. Increasing domain diversity during pre-training is expected to yield richer and more generalizable knowledge. Most existing methods pursue this goal by designing domain alignment strategies that help a single graph model learn from multiple domains. However, we observe that existing methods struggle to benefit from increasing domain diversity. Through empirical and theoretical analysis, we reveal two key limitations. Firstly, pre-training in most GFMs is confined to a single message-passing paradigm. Since propagation flattens relations among neighboring nodes, the graph model can hardly perceive the knowledge beyond propagation, namely the attribute-predictable component of these relations. Secondly, domain alignment strategies often rely on randomly initialized parameters that are jointly updated with the graph model using immediate loss feedback, leaving early knowledge acquisition vulnerable to cross-domain interference. To address these issues, we propose FLO, a Fast–sLow reciprOcal approach for multi-domain graph pre-training. Specifically, a propagation-free attribute branch predicts the graph model’s relations without message passing and returns these predictions to refine the model. We further stabilize knowledge acquisition by pairing a fast process that adapts to the current domain with a slow process that integrates knowledge across domains. On 12 datasets, FLO benefits from increasing domain diversity and achieves the best average performance in cross-domain and cross-task transfer.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Neural Network, Graph Pre-training, Graph Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_tJkdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FLO, a fast–slow reciprocal approach for multi-domain graph pre-training. This method mainly contain two components: PARR and FSKA. The former uses an attribute branch to predict relations between nodes without message passing and feeds these predictions back to the graph model. The latter uses fast and slow learning processes to reduce the cross-domain interference. Exiperiments are conducted in 12 datasets. The results show that FLO improves transfer performance.

Critical strengths

S1. The paper studies a meaningful problem: increasing the number of source domains does not always improve model performance.

S2. The method is clear and easy to understand.

S3. The experiments are extensive. The paper evaluates several different downstream tasks and provides comprehensive ablation studies.

Critical weaknesses

W1. One of the most important conclusion of the work is that FLO can benefits from more source domain, although the experiment only use six source doamin and use the fixed rank. I believe do more experiment about unfixed rank and more different domain can be more convincing.

W2. For a graph foundation model, one pre-training seed is too few to demonstrate robustness. I believe that five seeds for pre-training would be a sweet spot for supporting the competitive performance considering the cost.

W3. The hyperparameter settings are not consistent. In Appendix B, "Singular components kept for the target graph" is 16 for Cora, 8 for Citeseer, 512 for Computers and Propagation steps are 5 for Cora and 1 for other datasets .This is my biggest concern. Although experiment shows a strong competitive performance, additional track might affect the results significantly. I am curious about how much of the improvement actually comes from FLO's pre-training method?

Questions for authors

See W1~3

7 comments
AuthorsOpenReview ↗

R1 for W1: More source graphs, source order, and rank

R1 for W1: More source graphs, source order, and rank

Thank you for this valuable comment. To directly address your question, we conducted two sets of additional experiments: adding more source graphs (Table 1) and varying the order of source graphs (Table 2). All results and curves are included in Appendix D.9 of the revised manuscript (Tables 16–18 and Figures 10–11).

1. Adding more source graphs. Starting from the five source graphs in main Table 1, we sequentially add DBLP, Physics, and DBLP-APA, and evaluate all models with the same downstream evaluation as in main Table 1.

Table 1: Node classification accuracy (%, mean ± std) as new source graphs are sequentially added.

Target5 sources+ DBLP+ Physics+ DBLP-APA
Cora85.33 ± 0.5785.57 ± 0.4785.25 ± 0.7686.06 ± 0.72
CiteSeer72.63 ± 1.0072.50 ± 0.4673.50 ± 0.6173.75 ± 0.26
PubMed86.74 ± 0.3586.81 ± 0.3186.79 ± 0.2686.85 ± 0.24
CS93.64 ± 0.0693.69 ± 0.0393.71 ± 0.0493.71 ± 0.03
Photo93.53 ± 0.1593.52 ± 0.1693.65 ± 0.2093.60 ± 0.15
Computers90.19 ± 0.0990.03 ± 0.3590.17 ± 0.1590.13 ± 0.19
Mean87.01 ± 0.1487.02 ± 0.1287.18 ± 0.2487.35 ± 0.20

From Table 1, we draw two conclusions:

  • (i) FLO benefits from more source domains. As DBLP, Physics, and DBLP-APA are added, the six-target mean accuracy rises at every step: 87.01% → 87.02% → 87.18% → 87.35% (+0.34 percentage points overall, whereas the four multi-domain baselines gain at most 0.11 percentage points over the same expansion in Appendix Figure 10).

  • (ii) Five of the six individual targets improve, while the remaining one is already near saturation. CiteSeer (+1.12 percentage points) and Cora (+0.73 percentage points) gain the most. Computers already reaches 90.19% with five source graphs and remains stable within one standard deviation (90.13%).

2. Varying the source-domain order. We test domain order at both levels: (a) reversing the order in which the five source graphs are visited within each epoch (Table 2; Appendix D.9, Table 18), and (b) adding source graphs from one to five along six different inclusion orders (Appendix D.9, Figure 11 and Table 17).

Table 2: Node classification accuracy (%, mean ± std) under the original and reversed source-domain orders.

TargetOriginal orderReverse orderDifference (Reverse − Original)
CS93.59 ± 0.0893.62 ± 0.03+0.03
Photo93.68 ± 0.2293.64 ± 0.16−0.04
Computers90.15 ± 0.0590.03 ± 0.07−0.12

From Table 2 and Appendix Figure 11, we draw two conclusions:

  • (i) FLO is insensitive to source-domain order. Reversing the training order changes accuracy by at most 0.12 percentage points across CS, Photo, and Computers.

  • (ii) Averaged across six different source-inclusion orders, accuracy rises at every step as more source graphs are added (from 1 to 5 source graphs: 91.16% → 93.62% on CS, 89.37% → 93.62% on Photo, and 82.86% → 90.13% on Computers; see Appendix Figure 11 and Table 17).

3. Fixed versus unfixed rank. FLO ranks first both with per-target ranks (Appendix D.4, Tables 11–13) and with one shared rank of 21 for all six targets (Appendix D.9, Table 22; see R3 for W3).

AuthorsOpenReview ↗

R2 for W2: Five pre-training seeds

R2 for W2: Five pre-training seeds

Thank you for this valuable comment. We re-ran pre-training across five random seeds for FLO and the baselines in Table 3 across all three transfer tasks, and compare the results with those of the original submission. Table 3 reports the mean performance across the target datasets of each task, and the per-dataset five-seed results have been updated in the paper:

  • Node Classification: mean over Cora, CiteSeer, PubMed, CS, Photo, and Computers, corresponding to main Table 1 (per-dataset five-seed table in Appendix D.9, Table 19).

  • Link Prediction: mean over Physics, WikiCS, and DBLP, corresponding to main Table 2 (left) (per-dataset five-seed table in Appendix D.9, Table 20).

  • Graph Classification: mean over COIL-RAG, ENZYMES, and PROTEINS, corresponding to main Table 2 (right) (per-dataset five-seed table in Appendix D.9, Table 21).

Table 3: Multi-task transfer performance (%): node classification (NC, Acc.), link prediction (LP, Hits@50), and graph classification (GC, Acc.). Original is the single pre-training run in the original submission; 5 seeds is the mean ± std across five random pre-training seeds.

MethodNC OriginalNC 5 seedsLP OriginalLP 5 seedsGC OriginalGC 5 seeds
DGI82.6082.79 ± 0.19—58.36 ± 0.34—63.19 ± 0.08
GRACE78.8378.68 ± 0.22—56.07 ± 0.83—63.69 ± 0.53
BGRL83.6983.88 ± 0.28—66.68 ± 0.58—63.77 ± 0.32
MDGPT86.0285.99 ± 0.1057.6958.17 ± 0.4560.9960.98 ± 0.21
SAMGPT85.7785.74 ± 0.1059.1858.31 ± 0.6060.7161.36 ± 0.59
FUG84.9685.60 ± 0.0562.1362.19 ± 0.4863.3463.21 ± 0.07
TIG86.6886.80 ± 0.0358.7559.60 ± 0.5260.6860.55 ± 0.22
MDGFM85.8785.92 ± 0.1959.0858.90 ± 0.4262.7163.27 ± 0.63
BRIDGE85.8085.78 ± 0.1458.6758.75 ± 0.2761.5561.50 ± 0.16
FLO87.0786.98 ± 0.0671.1371.84 ± 1.2464.6864.63 ± 0.20

"—": DGI, GRACE, and BGRL were added to the link prediction and graph classification comparison in the revision.

From Table 3, we draw two conclusions:

  • (i) FLO achieves the highest five-seed mean on all three tasks, outperforming the strongest baseline by 0.18 percentage points in node classification (TIG), 5.16 percentage points in link prediction (BGRL), and 0.86 percentage points in graph classification (BGRL).

  • (ii) The five-seed results agree with the original submission. FLO's mean changes by −0.09, +0.71, and −0.05 percentage points on the three tasks, and FLO ranks first on all three tasks in the original results, in the five-seed means, and on every individual seed. The cross-seed std of FLO is 0.06 percentage points in node classification.

AuthorsOpenReview ↗

R3 for W3: Shared few-shot configuration

R3 for W3: Shared few-shot configuration

Thank you for this valuable comment. To measure how much improvement comes purely from FLO's pre-training without per-dataset hyperparameter tuning, we re-evaluated FLO and the nine baselines in Table 4 under one unified configuration across all six target datasets:

  1. Single fixed SVD rank: FLO uses one shared SVD rank (21, chosen once on a separate validation subset) for every target graph, while baselines use their own native input adapters.
  2. Identical readout and propagation for all methods: Every method uses the same classifier and is evaluated both under Direct readout (0 propagation steps)—removing post-hoc graph propagation completely—and under 1 shared propagation step applied identically to all methods.

Table 4 reports the mean accuracy across the six target datasets (Appendix D.9, Table 22). The per-dataset results under the original setting of Figure 3 remain in Appendix D.4 (Tables 11–13).

Table 4: Six-target mean few-shot accuracy (%, mean ± std) under the shared configuration.

MethodDirect (0-step) 1-shotDirect (0-step) 3-shotDirect (0-step) 5-shotShared (1-step) 1-shotShared (1-step) 3-shotShared (1-step) 5-shot
DGI47.97 ± 2.9459.75 ± 2.1164.98 ± 1.5253.24 ± 3.4865.38 ± 2.2470.20 ± 1.54
GRACE37.18 ± 2.4648.14 ± 1.9953.73 ± 1.6643.18 ± 3.0854.33 ± 2.4659.49 ± 1.84
BGRL45.47 ± 2.9656.49 ± 2.3061.84 ± 1.6848.74 ± 3.1559.48 ± 2.4264.50 ± 1.79
MDGPT52.46 ± 3.2462.98 ± 2.2567.58 ± 1.7254.84 ± 3.4765.05 ± 2.3169.28 ± 1.74
SAMGPT52.62 ± 3.3563.13 ± 2.3567.54 ± 1.8055.29 ± 3.5865.39 ± 2.4269.35 ± 1.81
FUG54.35 ± 3.3964.80 ± 2.4968.83 ± 1.9857.23 ± 3.6466.93 ± 2.6070.45 ± 2.04
TIG53.84 ± 3.7859.96 ± 3.0864.42 ± 2.7354.62 ± 3.8360.77 ± 3.1565.21 ± 2.75
MDGFM52.89 ± 3.1564.28 ± 2.0969.00 ± 1.5055.38 ± 3.3266.41 ± 2.1270.74 ± 1.47
BRIDGE50.93 ± 2.8762.49 ± 2.0167.65 ± 1.5254.98 ± 3.2166.10 ± 2.0970.61 ± 1.51
FLO57.17 ± 3.3467.02 ± 2.2570.43 ± 1.6458.41 ± 3.4767.74 ± 2.3470.95 ± 1.69

From Table 4, we draw two conclusions:

  • (i) With direct readout (0 propagation steps) and a single SVD rank, FLO leads the strongest baseline by 2.82, 2.22, and 1.43 percentage points at 1, 3, and 5 shots, respectively.

  • (ii) With one shared propagation step applied equally to all methods, FLO still ranks first at every shot count (+1.18, +0.81, and +0.21 percentage points), showing that FLO's lead does not depend on per-dataset tuning.

Reviewer_38TZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to enable a GNN to learn transferable knowledge from data across different domains and perform well on new data that was not included in pretraining. The authors propose FLO, which primarily consists of two modules: PARR, which enables the GCN and MLP to learn the relationships between nodes from each other, and FSKA, which uses EMA to optimize the teacher network and mitigate the interference caused by alternating training across different domains. The model achieves competitive results on in-domain, few-shot, and cross-task tasks.

Critical strengths
  1. A relevant problem in multi-domain graph learning.
  2. Competitive empirical results on several transfer tasks.
  3. Efficient inference through training-only auxiliary branches.
Critical weaknesses
  1. I think the analysis of the experimental results in this paper is somewhat vague. For example, Section 4.2 explains that the improvement in classification performance is due to the model learning “knowledge beyond propagation,” but the results show an increase in accuracy rather than changes in knowledge or improvements in cross-domain stability. Section 4.4 further interprets the stability of the loss as filtering out cross-domain interference. These inferences are not self-evident and need more explaination.
  2. There are also some issues that need clarification. Please see the “questions”.
Questions for authors
  1. Why should the attribute-predictable component be more transferable than the residual R, especially when graph structure contains task-relevant information unavailable in node attributes?
  2. Each epoch takes one optimizer step per source domain, increasing the number of domains also increases the total training steps. Does FLO still benefit from more domains under a fixed optimizer-step budget?
  3. Could the author discuss how PARR performs on graphs with sparse node attribute information? For example, on datasets like ogbn-proteins and ogbn-mag, how does the model handle situations where node attributes are missing?
8 comments
AuthorsOpenReview ↗

R1 for W1: Direct evidence for PARR and FSKA

R1 for W1: Direct evidence for PARR and FSKA

Thank you for this valuable comment. We added direct empirical measurements of both components in Section 4.6 (Figures 5 and 6) and Appendix D.8 (Table 14).

1. PARR (Table 1). Under the leave-one-domain-out setting of main Table 1, we compare FLO against a Propagating partner control that replaces the propagation-free MLP branch with an equal-width GNN branch. An independent attribute-only linear classifier splits test nodes into A+ (classified correctly from attributes alone) and A− (misclassified from attributes alone). Table 1 reports the percentage of the control's errors corrected by FLO and the percentage of the control's correct predictions turned into errors. Results are reported separately for each of the six target datasets, with mean ± standard deviation over three splits for each dataset.

Table 1: Node prediction changes (%, mean ± std) of FLO relative to the propagating-partner control.

TargetCorrected on A+Corrected on A−New errors on A+New errors on A−
Cora59.60 ± 1.8227.32 ± 1.581.11 ± 0.438.29 ± 1.80
CiteSeer68.66 ± 3.3125.09 ± 0.782.93 ± 0.1920.37 ± 3.56
PubMed25.66 ± 0.769.05 ± 1.021.65 ± 0.1511.71 ± 1.22
CS49.36 ± 3.8823.84 ± 1.970.94 ± 0.0310.51 ± 0.37
Photo44.73 ± 2.7315.71 ± 4.030.79 ± 0.076.58 ± 1.61
Computers33.58 ± 4.3113.71 ± 2.241.13 ± 0.116.75 ± 0.64

From Table 1, we draw two conclusions:

  • (i) PARR mainly repairs nodes that attributes alone can classify but the propagating partner misses. On all six targets, FLO corrects a much higher share of errors on A+ (25.66%–68.66%) than on A− (9.05%–27.32%), with gaps of 43.57 percentage points on CiteSeer and 32.28 percentage points on Cora.

  • (ii) PARR rarely turns correct predictions into errors, introducing minimal new errors on A+ (0.79%–2.93%) while still correcting 9.05%–27.32% of errors on A−.

2. FSKA (Table 2). During five-source alternating pre-training, the source cycle repeats every 5 updates, whereas cross-domain evolution unfolds over longer periods (>50 updates). Using discrete Fourier analysis on fixed probes, Table 2 reports the percentage of the online relational target's amplitude retained by FSKA's slow target. Each row reports the measurements for one of the six held-out target datasets.

Table 2: Spectral amplitude retention ratio (%) of FSKA slow relational targets relative to online relational targets.

Held-out targetSource-switching frequency (period = 5 updates)Low-frequency evolution (periods > 50 updates)
Cora5.4346.59
CiteSeer3.2748.10
PubMed5.1949.92
CS5.5447.80
Photo3.9346.86
Computers1.4330.08

From Table 2, we draw two conclusions:

  • (i) FSKA removes over 94% of the fluctuations synchronized with source switching, retaining only 1.43%–5.54% of the online amplitude at the 5-update cycle.

  • (ii) FSKA selectively tracks slower multi-cycle evolution, retaining 30.08%–49.92% of the amplitude at periods longer than 50 updates.

AuthorsOpenReview ↗

R2 for Q1: Attribute-predictable component and residual R

R2 for Q1: Attribute-predictable component and residual R

Thank you for this valuable question. Eq. (9) splits the relational information of the graph branch into I(C;ξ)I(C;\xi), the part predictable from aligned attributes ξ\xi, and the conditional residual R=I(C;P∣ξ)R = I(C; P \mid \xi) (Section 3.4). We clarify two points:

  1. I(C;ξ)I(C;\xi) is not restricted to non-structural information. It also captures structural regularities that correlate with node attributes, while structural variation not predictable from attributes sits in RR. Theorem 3.3 shows that moving the graph branch's relations toward the attribute branch's target contracts the conditional residual RR by at least a fraction η\eta, and preserves I(C;ξ)I(C;\xi) when the target equals P‾=E[P∣ξ]\overline{P} = \mathbb{E}[P \mid \xi].
  2. Graph topology is used in both pre-training and inference. During pre-training, the shared GNN aggregates graph neighborhoods for masked feature reconstruction. At inference time, the MLP branch is discarded and only the GNN is deployed to aggregate target-graph edges (Figure 2).
AuthorsOpenReview ↗

R3 for Q2: Fixed optimizer-step budget

R3 for Q2: Fixed optimizer-step budget

Thank you for raising this point. In Figure 4(a) (Section 4.4), we use the same total budget of 15,000 optimizer updates across all source counts. With S∈{1,2,3,4,5}S \in \{1, 2, 3, 4, 5\} source graphs, training runs for 15,000/S15,000 / S epochs, with one update per source graph per epoch. Table 3 below provides the per-target results (Appendix D.9, Table 15).

Table 3: Node classification accuracy (%, mean ± std) from 1 to 5 source graphs under a fixed budget of 15,000 optimizer updates.

Target1 source (15,000 epochs)2 sources (7,500 epochs)3 sources (5,000 epochs)4 sources (3,750 epochs)5 sources (3,000 epochs)
Cora81.71 ± 0.7682.39 ± 0.8684.97 ± 0.5785.45 ± 0.5685.79 ± 0.69
CiteSeer69.93 ± 0.3170.15 ± 0.3071.09 ± 0.4471.71 ± 0.3572.06 ± 0.33
PubMed83.99 ± 0.2986.17 ± 0.1386.36 ± 0.2086.58 ± 0.3186.73 ± 0.36
CS92.29 ± 0.1893.17 ± 0.0393.50 ± 0.0693.57 ± 0.0993.66 ± 0.06
Photo89.64 ± 0.8292.07 ± 0.3693.14 ± 0.2593.49 ± 0.2793.58 ± 0.24
Computers83.80 ± 0.3187.43 ± 0.3688.82 ± 0.1689.84 ± 0.1590.11 ± 0.11
Mean83.56 ± 0.3385.23 ± 0.1886.31 ± 0.2286.77 ± 0.2086.99 ± 0.26

From Table 3, we draw two conclusions:

  • (i) At a fixed 15,000-step budget, accuracy rises monotonically with every added source graph on every target, lifting the six-target mean from 83.56% to 86.99% (+3.43 percentage points), with per-target gains from +1.37 percentage points (CS) to +6.31 percentage points (Computers).

  • (ii) Each source graph receives fewer passes as SS increases, confirming that the gain comes from domain diversity rather than more optimization steps.

Comments on the paper

6 comments
AuthorsOpenReview ↗

R1 for W1 & Q1: Relation learning and matched distillation controls

R1 for W1 & Q1: Relation learning and matched distillation controls

Thank you for this valuable comment. We clarified the conceptual distinction in Section 2 and Appendix C, and added the requested ReSSL-style and NOSMOG-RSD-style EMA baselines in Appendix D.9 (Tables 23–26, and Figures 12–13).

Conceptual distinction.

  • GLNN and NOSMOG distill a fixed single-graph GNN teacher into an MLP so that downstream inference requires no edges.

  • LSP and FreeKD transfer supervision only between propagating GNNs, without a propagation-free reference.

  • BGRL and ReSSL apply EMA teachers across augmented views of a single instance/graph.

  • FLO deploys the shared GNN to unseen target graphs and uses the propagation-free MLP only during pre-training: G→MG \to M calibrates the MLP to the GNN on aligned attributes, and M→GM \to G returns attribute-grounded slow targets across rotating source domains.

Controlled empirical comparison. Following main Table 1, we implement ReSSL-style + EMA and NOSMOG-RSD-style + EMA on FLO's exact GCN–MLP backbone, masked feature reconstruction, and EMA momentum (μ=0.99\mu=0.99), changing only the relational objective (Appendix D.9, Table 23).

Table 1: Leave-one-domain-out accuracy (%, mean ± std) comparing FLO with matched EMA distillation controls.

MethodCoraCiteSeerPubMedCSPhotoComputersMean
Reconstruction only83.03 ± 0.3666.73 ± 1.1586.64 ± 0.2393.11 ± 0.3193.47 ± 0.2789.85 ± 0.2685.47 ± 0.17
Propagating partner79.74 ± 0.1055.56 ± 0.2286.65 ± 0.3191.63 ± 0.4592.37 ± 0.3489.55 ± 0.2782.58 ± 0.13
ReSSL-style + EMA76.54 ± 0.8364.83 ± 1.2283.62 ± 0.2990.30 ± 0.1192.16 ± 0.1887.13 ± 0.1282.43 ± 0.39
NOSMOG-RSD-style + EMA79.37 ± 1.5866.02 ± 0.4984.68 ± 0.3091.45 ± 0.1692.31 ± 0.0786.83 ± 0.1583.44 ± 0.22
FLO85.83 ± 0.6572.50 ± 0.9186.74 ± 0.3593.62 ± 0.0493.62 ± 0.2390.13 ± 0.0787.07 ± 0.13

From Table 1 and Appendix D.9, we draw two conclusions:

  • (i) Both EMA distillation controls fall below Reconstruction only (82.43% and 83.44% vs. 85.47%), whereas FLO reaches 87.07% and leads on every target. When tuning the relational weight on source validation (Appendix Table 24), both controls select weight 00 in every fold.

  • (ii) FLO scales better from 1 to 5 source graphs. Across 1–5 source graphs (Appendix Tables 25–26 and Figure 13), FLO leads both controls in all 30 target-by-source-count cells (+4.84 and +4.13 percentage points at five sources). Over the last 5,000 updates, FLO also has the lowest reconstruction loss of the three methods, 0.0056 vs. 0.0074 and 0.0070 (Figure 12).

AuthorsOpenReview ↗

R2 for W2: What the theoretical analysis establishes

R2 for W2: What the theoretical analysis establishes

Thank you for this valuable comment. We agree that the proofs use standard tools: spectral bounds, the chain rule of KL divergence, and the frequency response of an exponential moving average. The contribution lies in what these tools are applied to. Each result states a property of multi-domain graph pre-training that the tools alone do not give, and each one determines a design choice of FLO. We revised Section 3.4 to state this explicitly.

1. Knowledge beyond propagation becomes a defined quantity (Eq. 9). We split the relational information of the graph branch as I(C;P)=I(C;ξ)+RI(C;P) = I(C;\xi) + R, where R=E KL(P ∥ P‾)R = \mathbb{E}\,\mathrm{KL}(P\,\|\,\overline{P}) and P‾=E[P∣ξ]\overline{P} = \mathbb{E}[P \mid \xi] is the part of the relations that propagation-free attributes predict. GNN-to-MLP distillation (GLNN, NOSMOG) matches relations without separating these two parts. This decomposition defines the target of PARR and makes it measurable.

2. Domain alignment cannot remove relational flattening (Theorem 3.2). Over-smoothing bounds are usually stated for node embeddings on a single graph. We bound the candidate relation distribution: after KK propagation steps, its distance to uniform decays at a rate set by the graph spectrum. Feature alignment does not change this rate. This explains why alignment-based multi-domain methods do not recover the lost relational information, and it requires the partner branch to be propagation-free. Empirically, the relation change caused by propagation on each graph correlates with the accuracy drop without relation learning (r=0.89r = 0.89, Appendix A.8). Replacing the partner with a propagating GNN gives no gain on any target (main Table 3).

3. Why both directions, in this order (Theorem 3.3). Mutual learning (DML, FreeKD) exchanges predictions symmetrically. Theorem 3.3 shows that the two directions play different roles:

  • G→MG \to M drives the attribute branch toward P‾\overline{P}, since E KL(P∥Q)=R+E KL(P‾∥Q)\mathbb{E}\,\mathrm{KL}(P\|Q) = R + \mathbb{E}\,\mathrm{KL}(\overline{P}\|Q).
  • M→GM \to G then contracts RR by at least a fraction η\eta and preserves I(C;ξ)I(C;\xi), which keeps the structural patterns that attributes predict.

Feedback without calibration therefore has no guarantee. At the same feedback weight, removing G→MG \to M lowers accuracy by 3.87 to 29.46 percentage points (Appendix D.9, Table 27 and Figure 14).

4. Source rotation places cross-domain interference at a known frequency (Proposition 3.4). Momentum teachers (Mean Teacher, MoCo, BYOL, BGRL) are analyzed as smoothed averages of one training stream. In multi-domain pre-training, rotating among SS source graphs adds an oscillation of period SS. Proposition 3.4 shows that slow parameters keep only ρ≈0.009\rho \approx 0.009 of this oscillation for μ=0.99\mu = 0.99 and S=5S = 5, while slower trends pass through. Section 4.6 (Figure 6) measures this selectivity in the relation targets: slow targets keep 1.43%–5.54% of the online amplitude at the source-cycle frequency and 30.08%–49.92% at periods longer than 50 updates.

Together, these results explain why combining relational distillation with an EMA teacher is not sufficient. Without a propagation-free partner, calibration before feedback, and filtering at the source-rotation frequency, the matched ReSSL-style and NOSMOG-RSD-style controls fall below reconstruction only (see R1 for W1 & Q1).

We would be very grateful if you could point out which of these statements you consider already established, and in which prior work. We will cite those results explicitly and state more precisely what our analysis adds beyond them.

AuthorsOpenReview ↗

R3 for Q2: Why attribute-to-graph feedback alone degrades

R3 for Q2: Why attribute-to-graph feedback alone degrades

Thank you for this valuable question. In FLO, G→MG \to M first calibrates the propagation-free MLP by training it to predict the GNN's candidate relations from attributes; M→GM \to G then returns these calibrated relations to regularize the GNN. Without G→MG \to M (M → G only), the MLP learns solely from node-wise reconstruction and drives the shared GNN with uncalibrated relational targets (Section 4.5).

Because main Table 3 doubles the unidirectional weight when removing G→MG \to M, we also test M → G only using the exact same M→GM \to G weight as full FLO (Appendix D.9, Table 27 and Figure 14).

Table 2: Node classification accuracy (%, mean ± std) comparing M → G only against FLO at the same M → G weight.

TargetM → G onlyFLO
Cora64.24 ± 1.1485.80 ± 0.70
CiteSeer42.98 ± 0.4672.44 ± 0.68
PubMed81.31 ± 0.2986.66 ± 0.24
CS83.70 ± 0.1593.62 ± 0.04
Photo89.75 ± 0.7493.62 ± 0.23
Computers84.93 ± 0.1590.13 ± 0.07

Table 2 and Appendix Figure 14 show two clear results:

  • (i) Removing G→MG \to M calibration at the same M→GM \to G weight lowers accuracy by 3.87 to 29.46 percentage points across all six targets (Figure 14(a)).

  • (ii) Without calibration, M→GM \to G gradients overwhelm feature reconstruction (Figure 14(b)). Cross-branch relational KL rises from 0.0000630.000063–0.0002490.000249 (FLO) to 0.4660.466–1.2581.258 (M → G only). On source probes for the Cora, CiteSeer, and CS folds, the weighted M → G feedback gradient norm reaches 19.12–83.62 times the reconstruction gradient norm (0.013–0.029 in FLO).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.