acceptodds
Under review as a conference paper at ICLR 2027

Distilling Interaction Spaces for Graph Transformers

Abstract

Graph Transformers (GTs), with sophisticated attention mechanisms, naturally support beneficial distant node interactions and have shown strong potential across graph applications. Yet, in semi-supervised node classification, their flexibility makes the choice of interaction structure particularly important: a single interaction graph can over-constrain the student, whereas unrestricted structure learning can be weakly constrained by scarce labels and co-adapt with attention. To bridge this gap, by a constrained bi-level optimization reformulation, we disentangle graph searching and attention learning to reduce detrimental structure–attention co-adaptation and improve generalizability. Furthermore, inspired by weak-to-strong distillation, we distill a task-informed interaction space from a structurally simpler GNN teacher and let the more expressive GT student adaptively search for a suitable candidate within it. This way not only flexibly encodes task-specific implicit constraints on graphs tailored for downstream tasks, but also incorporates students’ model-adaptiveness beyond hand-crafted explicit heuristics. Technically, we propose a two-stage framework: 1) the Disentangling Stage extracts several diverse and task-specific prototypical graphs as a graph basis that spans a reasonable interaction space, with their task relevance enhanced by our Multi-Aspect Cooperative Distillation; 2) the Fusing Stage conducts adaptive search in this space via a simple node-wise linear fuser trained with our Meta Self-Update mechanism. Our framework fully mines the value of teacher knowledge and better releases students’ potential. It empirically outperforms state-of-the-art counterparts while retaining generality and practical efficiency.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Transformers, Knowledge distillation, Semi-supervised node classification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K2noOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes interaction-space distillation for Graph Transformers. A GNN teacher builds task-informed prototype graphs, which a student combines across nodes and layers. Results on nine node-classification datasets are promising.

The central claim needs stronger support. Single-graph comparisons do not match edge counts or training budgets, so the benefit of searching an interaction space remains unclear. The paper also defines admissible graphs as convex combinations while allowing Max fusion, and does not explain whether the student is refitted after the final fuser update. Its pseudo-label monitor is not a direct measure of generalization.

These issues limit the mechanism claim and reproducibility. Matched-budget controls and a precise account of the implemented training procedure are needed.

Critical strengths

1.Clear problem formulation. The paper distinguishes between deciding which nodes may interact and assigning attention weights to those interactions. It identifies a concrete structure-selection challenge under limited supervision.

2.Thoughtful method design. The teacher supplies multiple task-informed prototype graphs, while the student can combine them differently across nodes and layers. Results on nine datasets and several ablations provide empirical support for the complete approach.

Critical weaknesses

1.The main contribution is not fully isolated. Multi-prototype fusion is compared with single graphs without matching candidate-edge counts or training budgets, so the gains cannot be attributed solely to interaction-space search.

2.The optimization procedure is unclear. Fusion coefficients depend on changing student representations, meaning that a fixed fuser may still produce a changing graph. The paper also does not specify whether the student is refitted after the final fuser update.

Questions for authors
  1. Definition of the interaction space. Which fusion operator, Sum or element-wise Max, is used in the reported experiments? Since the admissible graph family is defined through convex combinations of prototypes, please clarify how Max fusion fits that definition and which claims depend on the choice of operator.

  2. Interpretation of the alternating optimization. Are the fused graphs recomputed as student representations change during inner training, even when the fuser parameters are fixed? After the final fuser update, is the student refitted using the resulting graphs? These details affect whether the implemented procedure corresponds to the stated bilevel objective.

  3. Evidence for adaptive interaction-space search. Multi-prototype fusion may change the number of candidate edges, model capacity, and training cost relative to a single graph. Can the authors provide comparisons that match these factors and report whether adaptive fusion still improves test performance? This would clarify how much of the observed gain is attributable to searching the proposed interaction space.

8 comments

We sincerely thank you for your insightful and constructive comments. We have carefully considered your feedback and responded to each of your points below.

Q1. Fusion operator used in the experiments

Question description: Which fusion operator, Sum or element-wise Max, is used in the reported experiments?

Short answer and conclusion: All reported main experiments use element-wise Sum fusion. The complete operator comparison below also shows that Sum outperforms the tested Max and temperature-softmax alternatives on all nine datasets.

Detailed response and evidence: We thank the reviewer for asking us to clarify this important implementation choice. We use element-wise Sum in all main experiments. To examine the choice more closely, we also evaluate element-wise Max and a temperature-softmax alternative while keeping the teacher, prototype graphs, student architecture, splits, and checkpoint-selection protocol unchanged. Sum combines the weighted contributions of all prototypes. Max selects the largest weighted contribution for each edge. Temperature-softmax provides a smooth max-like alternative, with temperature τ=0.05\tau=0.05. We provide some comparative experiments with results reported below. Each entry is test accuracy (mean ± SD, %). DIFFormer is included only as a GT reference baseline, not as a fusion-operator variant.

MethodCoraCiteseerPubmedActorCornellTexasWisconsinArXivFlickr
Ours (Sum)84.88 ± 0.3874.05 ± 0.3582.58 ± 0.1838.74 ± 0.2081.08 ± 2.4283.24 ± 2.2684.71 ± 1.4757.28 ± 0.2343.81 ± 1.41
Ours (Max)83.47 ± 0.6973.20 ± 0.3381.17 ± 0.3135.15 ± 0.7177.48 ± 1.2780.18 ± 1.2781.05 ± 0.9256.03 ± 0.5436.42 ± 4.19
DIFFormer83.32 ± 0.5274.46 ± 0.4278.16 ± 0.3234.51 ± 0.7660.00 ± 1.2168.11 ± 2.2663.92 ± 1.0753.60 ± 0.6444.25 ± 0.86
Ours (Temp-softmax)83.40 ± 0.5072.77 ± 0.5981.03 ± 0.4835.64 ± 0.9077.48 ± 5.5580.18 ± 1.2780.39 ± 0.0056.26 ± 0.2333.52 ± 9.76

Sum outperforms Max on all nine datasets by 0.85–7.39 percentage points and temperature-softmax by 1.02–10.29 points. This gives us both a clear default operator and empirical support for that choice. We include the same experiment in Appendix T.6 (Table 39 and Figure 33).

Q2. Max fusion and the convex-combination definition

Question description: How does Max fusion fit the convex-combination definition of the admissible graph family?

Short answer and conclusion: The convex-combination interpretation applies only to Sum, whereas Max is a nonlinear alternative outside the simplex-defined family.

Detailed response and evidence: Thank you for pointing out this distinction. For an edge (u,v)(u,v), we define au,v,k=βu,kGk,u,va_{u,v,k}=\beta_{u,k}G_{k,u,v}. Under Sum, the fused edge weight is ∑kau,v,k\sum_k a_{u,v,k}, so the nonnegative coefficients βu,k\beta_{u,k}, which sum to one, give a convex combination of prototype edge weights. Under Max, the result is max⁡kau,v,k\max_k a_{u,v,k}, which is a nonlinear operation and does not have the same convex-combination interpretation. We also test temperature-softmax as a smooth max-like alternative, defined by:

Mτ(au,v,:)=∑k=1KSoftmax⁡(au,v,:τ)kau,v,k,τ=0.05M_{\tau}(a_{u,v,:}) = \sum_{k=1}^{K}\operatorname{Softmax} \left(\frac{a_{u,v,:}}{\tau}\right)_k a_{u,v,k}, \qquad \tau=0.05

For clarity, the convex-combination interpretation is specific to Sum. Max and temperature-softmax are nonlinear alternatives. We make this distinction explicit in Appendix T.6.

Q3. Operator-dependent methodological claims

Question description: Which theoretical or methodological claims depend on the choice between Sum and Max fusion?

Short answer and conclusion: The simplex, convex-hull, and convex-combination interpretation applies to Sum fusion. Max and temperature-softmax offer alternative ways to combine the same candidate structures, but they do not have that convex-combination interpretation.

Detailed response and evidence: We appreciate the request to distinguish the operator-independent idea from the claims that require a particular fusion rule. Under Sum, the nonnegative fusion coefficients add up to one, so each node's potential neighborhood is a convex combination of the prototype neighborhoods. This is the basis of our simplex and convex-hull interpretation. By contrast, Max selects the largest weighted value for each edge, and temperature-softmax gives a smooth max-like combination. These operators still use the teacher-derived prototype family and node-wise fuser, while changing the way an edge's prototype contributions are merged.

The full nine-dataset table is provided in Q1 above (also Appendix T.6, Table 39 and Figure 33). Sum performs better than both alternatives on all nine datasets, supporting our choice of Sum for the main method. Thus, our task-informed candidate-space and student-adaptive fusion framework remains meaningful across operators, while the specific convex-combination geometry belongs to Sum.

Q4. Dynamic graph recomputation during inner training

Question description: Are fused graphs recomputed as student representations change during inner training, even when fuser parameters remain fixed?

Short answer and conclusion: We construct each potential graph once at the beginning of an alternating round and keep it fixed throughout that round’s inner student-fitting phase. We do not recompute it at every inner forward pass.

Detailed response and evidence: We appreciate the request to clarify this point. We follow four steps. (1) At the start of a round, we use the current student representation and fuser to construct the potential graph. (2) We fit the student with this graph fixed throughout the inner loop. (3) We hold the fitted student fixed and update the fuser in the outer loop. (4) We use the updated fuser to construct the graph for the next round. Thus, the student representation may change during inner fitting, but the potential graph for that round remains fixed. The stage-level timing table in Q8 below (Appendix T.5, Table 33) reports 2–5 graph-fusion calls per dataset, matching the 2–5 outer-update calls, while the student inner loop runs 200–600 calls. This call-count pattern is consistent with constructing the fused graph at the round level and reusing it during that round’s inner fitting. Algorithm 5 describes the same update order.

Q5. Student refitting after the final fuser update

Question description: After the final fuser update, is the student refitted using the resulting graphs?

Detailed response and evidence: Thank you for asking us to state the final step explicitly. In each round, Algorithm 5 first fits the student and then updates the fuser. Once the final outer update is complete, the procedure returns the student from that round’s inner phase. It does not start an additional inner phase using the graph implied by the last fuser update. We have clarified this point in Appendix T.5.

Q6. Generalization validity of the pseudo-label monitor

Question description: What does agreement with the teacher-pseudo-labeled monitor establish about generalization, given that it is not ground-truth validation accuracy?

Short answer and conclusion: Agreement with the teacher-pseudo-labeled monitor is used to select historical-success checkpoints. It is not a direct estimate of test accuracy. Our offline oracle comparison also shows that a more accurate monitor does not necessarily lead to a better final student.

Detailed response and evidence: We appreciate the reviewer emphasizing this distinction. We agree that the monitor should not be described as a ground-truth validation set. In our method, the fixed monitor contains high-confidence teacher pseudo-labels and is used to decide whether an inner-loop checkpoint qualifies as a new historical success. When a checkpoint passes the current agreement threshold, we confidence-weight its predictions into YMY_M. The outer fuser objective then uses the updated meta-targets. Thus, monitor agreement controls checkpoint selection, while the outer loss is computed from YMY_M.

We also reproduce the full nine-dataset diagnostic here so this answer can be read independently. Ours (paper) is the submitted method’s reference test accuracy. Oracle-monitor student is the final student from the offline diagnostic that substitutes an all-node ground-truth monitor. Pseudo-monitor RR and Oracle-monitor RR are the confidence-weighted correct-target shares of the meta-target distributions under the two monitor choices. Specifically,

\qquad w_u = \bigl(\operatorname{INE}(Y_{M,u})\bigr)^{\eta}.$$ For each node $u$, $Y_{M,u}$ is the current meta-target distribution, $y_u$ is its ground-truth class for this offline diagnostic, and $w_u$ is the confidence weight used by the outer objective. Thus, $R$ is the weighted share of nodes whose meta-target argmax matches the ground-truth class. It measures meta-target correctness, not the monitor’s own agreement score. In the table, Ours (paper) is the submitted method’s test-accuracy reference, Oracle-monitor student is the offline oracle diagnostic’s final test accuracy, and the two $R$ columns give the corresponding confidence-weighted meta-target correctness shares. | Dataset | Ours (paper) | Oracle-monitor student | Pseudo-monitor $R$ | Oracle-monitor $R$ | |---|---:|---:|---:|---:| | Cora | 84.88 ± 0.38 | 83.58 ± 0.86 | 0.8509 | 0.8474 | | Citeseer | 74.05 ± 0.35 | 72.27 ± 0.59 | 0.7433 | 0.7271 | | Pubmed | 82.58 ± 0.18 | 81.23 ± 0.95 | 0.8155 | 0.8234 | | Actor | 38.74 ± 0.20 | 36.19 ± 0.42 | 0.8299 | 0.8300 | | Cornell | 81.08 ± 2.42 | 82.52 ± 1.08 | 0.8915 | 0.8982 | | Texas | 83.24 ± 2.26 | 80.18 ± 2.55 | 0.8930 | 0.9198 | | Wisconsin | 84.71 ± 1.47 | 81.54 ± 1.69 | 0.9026 | 0.9180 | | OGBN-ArXiv | 57.28 ± 0.23 | 54.59 ± 2.07 | 0.5659 | 0.5725 | | Flickr | 43.81 ± 1.41 | 42.90 ± 7.70 | 0.5340 | 0.5945 | | Dataset | Oracle-monitor student − Ours (paper) (pp) | Oracle-monitor $R$ − Pseudo-monitor $R$ | |---|---:|---:| | Cora | −1.30 | −0.0035 | | Citeseer | −1.78 | −0.0162 | | Pubmed | −1.35 | +0.0079 | In the first table, we compare the full-method reference, the oracle-monitor student result, and weighted meta-target correctness across nine datasets. Its first difference is final student accuracy, and its second is the change in $R$. The oracle raises $R$ on seven of nine datasets, but final accuracy does not improve consistently. On the three paired citation datasets, the accuracy changes are -1.30, −1.78, and −1.35 pp. One possible explanation is that ***small errors in the pseudo monitor may have a mild regularizing effect on checkpoint selection and structure learning.***. The oracle uses all-node ground truth only for this offline analysis. Our training procedure continues to use the pseudo-labeled monitor. We include the same diagnostic in Appendix T.4 (Table 30 and Figure 29).
Reviewer_izJiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies interaction-structure acquisition for Graph Transformers (GTs) in semi-supervised node classification. It proposes Interaction-Space Distillation, where a message-passing GNN teacher provides task-informed prototypical graphs and a GT student learns node- and layer-specific combinations. Multi-Aspect Cooperative Distillation (MACD) improves aspect predictions before graph construction, while Meta Self-Update guides the fuser through alternating optimization using teacher predictions and selected historical student predictions. Experiments cover nine node-classification benchmarks, with distillation comparisons and component ablations.

Critical strengths

S1. The paper offers an insightful perspective on how GTs can obtain task-relevant interactions under scarce supervision, identifying a useful middle ground between rigid fixed-graph learning and unconstrained structure search. Building on this perspective, the disentangling–fusing framework is an elegant design that combines task-informed prototypes with node- and layer-specific fusion, allowing the student to adapt the teacher’s structural guidance to its own representations.

S2. The experimental evaluation is extensive, covering nine node-classification benchmarks with strong overall results and informative distillation comparisons and ablation studies.

Critical weaknesses

W1. The paper motivates Interaction-Space Distillation through the risk of Cooperative Collapse under scarce labels. Figure 15 shows that the tested GGT with a jointly learned potential graph fits the training nodes but generalizes worse than the naive GGT. However, the paper lacks a study varying the number of labeled nodes per class, and the learned-graph analysis focuses on homophily rather than how the interactions differ from the input graph. Additional label-budget and structural analyses, together with comparisons against other ways of mitigating overfitting, would strengthen the motivation for constraining node interactions under scarce supervision and clarify which structural changes help in this setting.

Relatedly, Table 9 shows that adaptive search outperforms the best individual prototype, and the node-wise versus graph-wise fuser comparison supports node-dependent adaptation. However, a single prototype generally exposes fewer candidate edges than the prototype union, so the best-prototype comparison mixes adaptive weighting with candidate coverage, as Section C.7 acknowledges. Further ablations controlling candidate coverage would help establish the benefit of adaptive graph search beyond access to a larger set of interactions.

W2. Equation 3 introduces a constrained bi-level formulation, but the practical procedure uses a finitely trained student, student resets, outer updates with student parameters held fixed, and dynamically updated meta targets (Algorithm 5; Sections B.5, G.3, and L.2). Section G.3 further notes that fixing the fuser parameters does not fix the potential graph when student representations change, leaving the effectiveness of the intended separation between student fitting and structural selection insufficiently validated. Additional convergence analysis or empirical stability results across alternating rounds would strengthen the justification for this procedure.

Questions for authors

Q1. Removing MACD causes a large performance drop, e.g., from 84.88% to 53.70% on Cora in Table 3. However, Equations 8, 10, and 23 describe aspect prediction heads alongside direct supervision of the holistic prediction, while Section B.2 states that direct supervision of the aspect predictions is omitted. How are the aspect prediction heads trained in the variant without L^(Dis)? If they no longer receive a task-training signal, poor Average Aspect Accuracy and prediction-based graph construction would be expected even if the underlying aspect representations contain useful information.

Q2. How do the gains change with the number of labeled nodes per class, and do they persist when candidate coverage is controlled?

Q3. Could the authors provide convergence analysis or empirical stability results across alternating rounds?

9 comments

We are deeply grateful to you for the careful and insightful comments on our manuscript. Our detailed responses to your questions are outlined below.

Q1. Structural differences from the input graph

Question description: How do the learned interactions differ from the original input graph beyond their homophily ratios?

Short answer and conclusion: We find a clear peak-and-decline pattern in both average learned-neighbor counts and assigned interaction weight across original-graph hop distances. The peak varies across datasets from hop 2 to hop 6, showing that the learned interaction structure adapts its preferred interaction range to the graph, which may help information-insufficient nodes collect customized distant knowledge.

Detailed response and evidence: We thank the reviewer for suggesting a direct comparison beyond homophily. We group the learned interaction graph by shortest-path distance (hop) in the original graph. Each table entry is the average number of learned non-self neighbors per source node / the mean sum of assigned interaction weights at that hop (as a percentage of the row-normalized weight). Bold values mark the peak for each dataset. The Hop ≥8 column adds together the longer finite-hop tail, so it is not a single hop and is excluded when identifying the peak. Unreachable nodes and self-connections are not included in these finite-hop entries.

DatasetHop 1Hop 2Hop 3Hop 4Hop 5Hop 6Hop 7Hop ≥8 (tail sum)Peak hop
Cora0.66 / 2.08%2.49 / 7.13%4.63 / 13.73%5.71 / 16.75%5.07 / 14.71%4.15 / 11.57%2.83 / 7.74%3.87 / 10.55%4
Citeseer0.21 / 0.64%0.62 / 1.87%1.28 / 3.53%1.71 / 4.52%2.09 / 5.39%2.01 / 5.09%1.72 / 4.46%6.17 / 15.99%5
Pubmed0.009 / 0.04%0.09 / 0.38%0.38 / 1.51%2.01 / 8.46%3.54 / 15.27%6.32 / 27.51%4.51 / 19.53%6.06 / 27.28%6
Actor0.04 / 0.10%1.61 / 4.62%8.50 / 24.06%14.54 / 41.04%8.19 / 22.87%2.25 / 5.99%0.46 / 1.14%0.07 / 0.16%4
Cornell0.03 / 0.71%1.83 / 32.14%1.75 / 30.52%1.40 / 25.72%0.39 / 7.54%0.09 / 1.75%——2
Texas0.31 / 1.17%12.31 / 45.03%7.20 / 23.78%5.91 / 21.95%1.65 / 5.58%0.46 / 1.74%0.04 / 0.18%—2
Wisconsin0.43 / 1.27%15.21 / 39.41%9.78 / 26.41%8.04 / 21.78%2.94 / 8.17%0.60 / 1.37%0.13 / 0.25%0.01 / 0.04%2
OGBN-ArXiv0.10 / 0.37%2.67 / 9.21%8.01 / 27.51%5.96 / 20.51%4.33 / 14.83%3.22 / 11.05%2.13 / 7.22%2.61 / 8.87%3
Flickr0.003 / 0.06%0.02 / 0.20%0.36 / 6.12%2.55 / 46.05%2.94 / 46.84%0.12 / 0.73%——5

Across all nine datasets, both the average neighbor count and assigned interaction weight generally rise to a peak and then decline. Notably, the two peaks coincide within each dataset, but their positions differ across graphs: hop 2 on Cornell, Texas, and Wisconsin; hop 3 on OGBN-ArXiv; hop 4 on Cora and Actor; hop 5 on Citeseer and Flickr; and hop 6 on Pubmed. This shows that our learned interactions can prioritize different nonlocal distances according to the graph, instead of relying on a single predetermined hop range. We provide the full per-hop profiles and visualizations in Appendix T.3 (Table 25 and Figures 22–24), which may provide a better understanding for the properties of our framework.

Q2. Performance-relevant structural changes

Question description: Which structural changes in the learned interaction graphs contribute to improved performance?

Short answer and conclusion: All 21 step-wise same-label evidence gains are positive across the seven medium-sized datasets, and the local-scarcity gain is positive on all seven as well.

Detailed response and evidence: We thank the reviewer for asking which structural changes are associated with the performance gains. The table below summarizes two post-hoc diagnostics. Step 1, Step 2, and Step 3 gain measure the increase in same-label evidence relative to the original graph after the corresponding interaction-propagation step. These values are structural diagnostics, not percentage-point changes in test accuracy. Local-scarcity gain compares the learned-graph gain for nodes with weaker original local same-label evidence with the gain for nodes with stronger original evidence. A positive value means that the lower-evidence group receives a larger structural gain.

DatasetStep 1 gainStep 2 gainStep 3 gainLocal-scarcity gain
Cora0.22560.22130.1837+0.0792
Citeseer0.14880.12180.0744+0.1218
Pubmed0.25610.18190.1189+0.1782
Actor0.09120.07240.0553+0.2241
Cornell0.59490.44570.4514+0.0347
Texas0.51170.29930.3117+0.0113
Wisconsin0.61310.45940.4598+0.2662

We further connect these structural diagnostics to test performance with the matched candidate-edge-budget control. In this table, DD is the average number of non-self candidate edges per node in the union of the prototypes, and kwidek_{\mathrm{wide}} is its ceiling. Best Prototype is the strongest individual prototype at its original budget. Best-Wide is the same single prototype after widening its neighborhood to kwidek_{\mathrm{wide}}, without node-wise fusion. Ours (paper) is the submitted-paper reference, and the final column gives the Ours-minus-Best-Wide difference in percentage points.

DatasetDDkwidek_{\mathrm{wide}}Best PrototypeBest-WideOurs (paper)Ours − Best-Wide (pp)
Cora34.5317573582.50*82.53 ± 0.2184.88 ± 0.38+2.35
Citeseer38.5320133972.40*72.23 ± 0.6574.05 ± 0.35+1.82
Pubmed22.9272712382.00*81.77 ± 0.1782.58 ± 0.18+0.81
Actor35.6647343635.48 ± 1.0135.88 ± 0.2238.74 ± 0.20+2.86
Cornell5.562841671.17 ± 4.5974.77 ± 1.2781.08 ± 2.42+6.31
Texas28.0382502980.18 ± 1.2780.18 ± 1.2783.24 ± 2.26+3.06
Wisconsin37.4780883879.08 ± 0.9279.74 ± 1.8584.71 ± 1.47+4.97
OGBN-ArXiv29.0901093049.21 ± 0.9849.37 ± 1.2257.28 ± 0.23+7.91
Flickr5.995742641.86 ± 0.2240.73 ± 1.6943.81 ± 1.41+3.08

Across the seven datasets, all 21 step-wise same-label evidence gains and all seven local-scarcity gains are positive. The largest local-scarcity gains occur on Wisconsin (+0.2662), Actor (+0.2241), and Pubmed (+0.1782), indicating that nodes with weaker original local evidence benefit more. Importantly, after a single prototype is widened to the average edge budget of the prototype union, Ours still leads on all nine datasets by +0.81 to +7.91 pp. Together, these results show that our learned graph adds task-relevant same-label evidence across multiple interaction steps (hops), especially for locally under-supported nodes, and that the performance advantage persists after controlling candidate coverage. This supports our use of node-wise fusion over task-informed prototypes. The structural diagnostics are in Appendix T.3 (Table 27 and Figures 25–26), and the matched-budget comparison is in Appendix T.1 (Table 19 and Figure 19).

Q3. Alternative overfitting-mitigation methods

Question description: How does the interaction-space constraint compare with other approaches to mitigating overfitting?

Short answer and conclusion: Even the best-performing regularized Joint setting remains below Ours on all nine datasets, and our teacher-derived candidate family outperforms heuristic and random alternatives on all nine.

Detailed response and evidence: We appreciate the suggestion to compare our interaction-space constraint with other ways of controlling overfitting. We first vary dropout, attention dropout, weight decay, label smoothing, and early-stopping settings for joint learning. To help interpret the settings table, each JL-R row is one configuration. Dropout and Attention dropout are the rates applied to model representations and attention weights. Weight decay is the coefficient for parameter regularization. Label smoothing is the smoothing strength for class targets. Max epochs is the training cap, and Patience is the early-stopping patience, with zero indicating that patience-based early stopping is disabled. Loss identifies ordinary cross-entropy (CE) or cross-entropy with label smoothing. Together, these rows form a sensitivity sweep across increasingly strong regularization settings.

ModelDropoutAttention dropoutWeight decayLabel smoothingMax epochsPatienceLoss
JL-R00.000.000.0000.002000CE
JL-R10.200.150.0010.002000CE
JL-R20.500.400.00250.002000CE
JL-R30.800.700.0050.002000CE
JL-R40.820.720.0060.012000CE + smoothing
JL-R50.840.740.0080.0320040CE + smoothing
JL-R60.850.750.0100.0520030CE + smoothing
JL-R70.870.770.0200.0618025CE + smoothing
JL-R80.890.790.0350.0814022CE + smoothing
JL-R90.900.800.0500.1010020CE + smoothing

We report the full accuracy results for these joint-learning configurations below:

ModelCoraCiteseerPubmedOGBN-ArXivTexasWisconsinCornellActorFlickr
JL-R053.80 ± 0.5652.40 ± 1.3969.37 ± 1.6329.43 ± 0.2077.48 ± 1.5675.82 ± 1.1372.07 ± 5.6336.21 ± 0.3223.76 ± 6.68
JL-R155.50 ± 1.3953.10 ± 1.4070.70 ± 1.9329.64 ± 1.0576.58 ± 1.5677.78 ± 2.2673.87 ± 4.1336.23 ± 0.1023.32 ± 5.27
JL-R257.67 ± 0.2554.27 ± 0.0671.50 ± 0.7929.93 ± 0.3376.58 ± 4.1379.08 ± 2.2672.07 ± 3.1236.14 ± 1.4924.47 ± 2.34
JL-R358.83 ± 0.1257.07 ± 0.5572.33 ± 0.7430.92 ± 0.3177.48 ± 1.5679.74 ± 1.1372.97 ± 2.7036.45 ± 1.1520.09 ± 2.73
JL-R458.87 ± 0.3256.67 ± 0.9173.33 ± 0.2330.68 ± 0.4677.48 ± 1.5681.05 ± 1.1373.87 ± 4.1336.58 ± 0.1725.56 ± 8.99
JL-R558.90 ± 0.5056.43 ± 0.8656.47 ± 16.0030.49 ± 0.3078.38 ± 0.0080.39 ± 0.0072.97 ± 2.7036.34 ± 0.1023.48 ± 3.40
JL-R658.80 ± 0.6256.23 ± 0.7859.87 ± 13.1921.25 ± 14.7978.38 ± 0.0082.35 ± 0.0072.97 ± 2.7036.23 ± 0.3722.94 ± 3.63
JL-R759.07 ± 0.2157.40 ± 0.4645.33 ± 3.297.83 ± 4.9678.38 ± 0.0079.08 ± 1.1372.07 ± 1.5635.39 ± 0.3325.57 ± 13.62
JL-R859.47 ± 0.8557.43 ± 2.0643.73 ± 27.6112.42 ± 10.8068.47 ± 6.2479.74 ± 2.2673.87 ± 1.5635.75 ± 0.2019.70 ± 13.51
JL-R960.83 ± 1.0157.70 ± 1.2352.93 ± 16.0216.34 ± 2.9669.37 ± 7.8078.43 ± 1.9660.36 ± 17.1636.51 ± 0.6315.87 ± 1.87

Even after selecting the best-performing JL-R configuration separately for each dataset, its mean accuracy remains below Ours on all nine datasets, with gaps of 2.16–26.36 pp. These results show that regularization can partially alleviate the weakness of joint structure learning, but it does not close the gap to our method.

Reviewer_TYDjOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a Graph Transformer (GT) should obtain its candidate interaction structure for semi-supervised node classification when labels are scarce. The authors argue that a single fixed graph over-constrains attention, while learning the structure freely under the same scarce-label loss lets structure and attention co-adapt ("Cooperative Collapse"). Their proposal, Interaction-Space Distillation, is a "subspace distillation" idea: instead of transferring one fixed target, a structurally simpler message-passing GNN teacher defines a family of candidate graphs, and the more expressive GT student searches inside that family.

Concretely, the framework has two stages. In Stage 1 (Disentangling), a DisenGCN-style teacher is split into K aspect views. Multi-Aspect Cooperative Distillation (MACD) distils the teacher's holistic prediction back into each aspect head with a confidence-weighted (inverted normalized entropy) KL loss, raising the task relevance of each aspect. Each aspect's predictions are then turned into a sparse prototypical graph (top-K′ neighbours by prediction-similarity × endpoint confidence, retrieved via a MIPS/Faiss reduction, Prop. B.1), and the K prototypes form a "graph basis" spanning the interaction space. In Stage 2 (Fusing), a node-wise, layer-specific linear fuser (an MLP + softmax over the K prototypes) produces a student-specific potential graph per node and layer. The fuser is trained with a Meta Self-Update mechanism motivated by a constrained bi-level formulation (Eq. 3): the student is fitted in an inner loop on the labelled loss (plus a near-to-far entropy curriculum and a confidence-aware implicit-sparsity regulariser), while the fuser is updated in an outer loop on a surrogate "generalisation" loss L̂_GE whose targets are teacher pseudo-labels on all nodes, ensembled with historical student predictions that beat a threshold on a fixed, teacher-pseudo-labelled monitor set.

On nine node-classification benchmarks (Cora, Citeseer, Pubmed, Actor, Cornell, Texas, Wisconsin, OGBN-ArXiv, Flickr) the method reports average rank 1.22 against 16 GNN/GT baselines, leading on 7 of 9 datasets, with additional comparisons to CoBFormer, Polynormer, DUALFormer, GCFormer and M³Dphormer in the appendix.

Critical strengths
  1. A new and well-motivated distillation object: the interaction space itself. The paper reframes the problem from "which single graph should a Graph Transformer use?" to "which family of candidate graphs should it be allowed to search?" Instead of transferring a fixed prediction, representation, or single graph, a structurally simpler GNN teacher distils a task-informed set of prototypical graphs, and the student learns node- and layer-specific combinations within that space. This is a consequential distinction from prior adaptive-structure methods such as ANS-GT and M³Dphormer, whose candidate structures are hand-designed heuristics: here the candidate family itself is learned from the task. It occupies a principled middle ground between a rigid fixed graph, which attention cannot correct, and unrestricted structure learning, which scarce labels cannot supervise — the tension the paper identifies as "Cooperative Collapse."

  2. Strong, broad empirical results, with a comparison design that supports the central idea. Across nine benchmarks covering homophilous and heterophilous graphs plus two large graphs (OGBN-ArXiv, Flickr), the method achieves average rank 1.22 against 16 GNN/GT baselines and leads on 7 of 9 datasets. Crucially, the Tab. 2 distillation comparison separates teacher quality from transfer mode: with the improved teacher fixed, searching the interaction space beats using its single holistic graph on all nine datasets (I vs. H), and adaptive search also beats the best individual prototype (Tab. 9). The informative negative result — a vanilla multi-graph space performs far worse than a single graph (G vs. F, e.g. Cora 53.70 vs. 81.38) — shows the gain depends on the space being task-aligned via MACD, rather than on simply having more graphs, which makes the empirical case for the proposed mechanism substantially more credible.

Critical weaknesses
  1. The gains are not cleanly attributed to the interaction space. The result depends heavily on components other than the proposed space itself. Removing MACD collapses performance (Tab. 5: Cora 84.88 → 53.70, Citeseer 74.05 → 51.50, Pubmed 82.58 → 69.40), while the MACD-improved teacher alone already reaches 82.30 / 72.12 / 81.48 on the citation graphs (Tab. 2, row E) — within ~1–2.5 points of the full student. Yet there is no control for standard prediction-level distillation or self-training from that same improved teacher into the same student with no graph space, even though the outer loop is supervised by teacher pseudo-labels on all nodes. Two further confounds run in the same direction: the fused graph's support is the union over K prototypes, up to K× the edge budget of the single-graph baselines it is compared against, with no matched-support comparison (the authors note this in Sec. C.7); and hyperparameters are selected from 200 random searches per dataset (Sec. P.3), with selected values such as λ_Reg spanning 1e-5 to 10 across datasets (Tabs. 15–16), while the tuning budget given to the baselines is not stated. This is consequential because the paper's central claim is causal — that distilling and searching an interaction space is what helps — and as presented, a substantial part of the gain could come from a better self-distilled teacher, a larger candidate edge budget, or search.

  2. The label-scarcity premise, on which the whole motivation rests, is never directly tested. There is no label-rate experiment (e.g. 1 / 5 / 10 / 20 labels per class) showing that the advantage over baselines, or over jointly learning the structure, grows as labels become scarcer, as the "Cooperative Collapse" account predicts. The actual protocols are mixed, as the authors acknowledge (Sec. P.2): the citation graphs use the standard 20-per-class splits and ArXiv/Flickr use custom 20-per-class splits whose results are not comparable to standard-split results, whereas Texas, Wisconsin, Cornell and Actor have ~48% of nodes labelled — and the largest headline gains are on exactly those webpage graphs, whose test sets contain only 37, 37 and 51 nodes (Tab. 14), so that a single test node is worth 2.0–2.7 points and the reported gaps amount to a handful of nodes, with no split-level significance reported. This is consequential because a method motivated entirely by scarce supervision needs evidence in the scarce-label regime specifically, and the current evaluation does not establish that the benefit appears where the theory says it should.

Questions for authors

Prediction-level distillation control: What accuracy does the same student obtain when trained with standard prediction-level distillation (or self-training) from the same MACD-improved teacher, with no interaction space and no fuser? Please add this control to the Tab. 2 (A–I) design, since the MACD teacher alone is already within ~1–2.5 points of the full method on most datasets. Matched edge budget: The fused graph's support is the union over K prototypes, up to K× the edges per node of a single prototype. If the fused graph is pruned back to K′+1 edges per node — or the best single prototype is allowed the same total budget — how much of the Tab. 9 and (I vs. H) gap remains? Label-rate sweep: On Cora and OGBN-ArXiv, how do your method, the strongest baseline (e.g. DIFFormer / CoBFormer), and jointly learned structure behave at 1, 5, 10 and 20 labels per class? Does your advantage grow as labels become scarcer, as the Cooperative Collapse motivation predicts? Tuning budget and total cost: What hyperparameter search budget was used for each baseline, compared with the 200 random searches reported for your method (Sec. P.3)? And what is the end-to-end wall-clock time to the reported accuracy on ArXiv and Flickr — teacher training, graph construction and all alternating rounds included, not only per-epoch student time (Tab. 4)? Small webpage graphs: For Cornell, Texas and Wisconsin the test sets contain 37, 37 and 51 nodes. What are the results (mean ± std) across multiple standard splits rather than initialisations on a single split, and how many test nodes account for the reported gaps over the strongest baselines? Teacher failure mode: If the teacher is weakened or corrupted (e.g. trained with label noise, or with far fewer labels so its pseudo-labels are unreliable), how does the student degrade relative to training the student on its own? In particular, can the Meta Self-Update loop — whose monitor set is itself teacher-pseudo-labelled — amplify early teacher errors rather than correct them?

10 comments

We are extremely grateful for your detailed constructive feedback and positive recognition on our manuscript. Below, we address your questions point-by-point.

Q1. Prediction-level distillation control

Question description: What accuracy does the same student achieve with standard prediction-level distillation or self-training from the MACD-improved teacher, without the interaction space or fuser?

Short answer and conclusion: Both prediction-transfer controls fall below our interaction-space method on all nine datasets. This consistent gap suggests that transferring the teacher's class predictions alone does not recover the gains we obtain by constructing and searching a task-informed family of interaction structures.

Detailed response and evidence: We appreciate the reviewer’s suggestion to include this important control. To isolate the contribution of the interaction space, we trained the same student backbone using the same MACD-improved teacher predictions, data split, and original input graph, but without constructing the prototype family or using the node-wise fuser. We considered two standard forms of prediction transfer. Standard KD matches the teacher’s soft class probabilities, while hard pseudo-label training uses the teacher’s high-confidence argmax predictions as labels. The full results are reported below.

DatasetControl objectiveControl result (mean ± SD)Ours (paper)Ours − control (pp)
CoraStandard KD82.8667 ± 0.787484.88 ± 0.38+2.0133
CoraHard pseudo-label82.5333 ± 0.235784.88 ± 0.38+2.3467
CiteseerStandard KD72.1667 ± 0.330074.05 ± 0.35+1.8833
CiteseerHard pseudo-label71.9000 ± 0.081674.05 ± 0.35+2.1500
PubmedStandard KD81.2333 ± 0.094382.58 ± 0.18+1.3467
PubmedHard pseudo-label79.8667 ± 0.623682.58 ± 0.18+2.7133
ActorStandard KD35.9649 ± 0.591738.74 ± 0.20+2.7751
ActorHard pseudo-label34.9561 ± 0.589338.74 ± 0.20+3.7839
CornellStandard KD70.2703 ± 0.000081.08 ± 2.42+10.8097
CornellHard pseudo-label69.3694 ± 3.370981.08 ± 2.42+11.7106
TexasStandard KD77.4775 ± 2.548183.24 ± 2.26+5.7625
TexasHard pseudo-label76.5766 ± 1.274183.24 ± 2.26+6.6634
WisconsinStandard KD78.4314 ± 0.000084.71 ± 1.47+6.2786
WisconsinHard pseudo-label75.8170 ± 0.924384.71 ± 1.47+8.8930
FlickrStandard KD42.0966 ± 0.096643.81 ± 1.41+1.7134
FlickrHard pseudo-label42.1119 ± 0.397543.81 ± 1.41+1.6981
OGBN-ArXivStandard KD53.8091 ± 0.238257.28 ± 0.23+3.4709
OGBN-ArXivHard pseudo-label53.0349 ± 0.123557.28 ± 0.23+4.2451

We include the complete results and comparison plot in Appendix T.1 (Table 18 and Figure 18). Across all nine datasets, both prediction-only controls fall behind our method, supporting the central distinction in our approach: the teacher contributes not only class predictions, but also a task-informed family/space of candidate interactions that the student can adapt to its own customized representations.

Q2. Matched candidate-edge budget

Question description: How much of the adaptive-fusion gain remains when the fused graph is pruned to the same edge budget as a single prototype, or the best single prototype is given the same total edge budget?

Short answer and conclusion: Even after we widen the best single prototype to approximately match the average candidate-edge budget of the prototype union, our method remains ahead on all nine datasets. The gains range from 0.81 to 7.91 percentage points.

Detailed response and evidence: We appreciate the reviewer’s concern that a larger candidate set could explain part of the gain. To examine this directly, we added an average-budget control. We first measure DD, the average number of non-self candidate edges per node in the union of our prototypes, and set kwide=⌈D⌉k_{\mathrm{wide}}=\lceil D\rceil as the target neighborhood budget for the widened single-prototype control. Specifically, we widen these individual prototypes to that budget while keeping the student architecture unchanged, and select the best for the comparison with ours. The next table reports DD, its ceiling kwidek_{\mathrm{wide}}, the original Best Prototype result, the widened Best-Wide result, our method, and the difference between our reference and Best-Wide.

DatasetDDkwidek_{\mathrm{wide}}Best PrototypeBest-WideOurs (paper)Ours − Best-Wide (pp)
Cora34.5317573582.5082.53 ± 0.2184.88 ± 0.38+2.35
Citeseer38.5320133972.4072.23 ± 0.6574.05 ± 0.35+1.82
Pubmed22.9272712382.0081.77 ± 0.1782.58 ± 0.18+0.81
Actor35.6647343635.48 ± 1.0135.88 ± 0.2238.74 ± 0.20+2.86
Cornell5.562841671.17 ± 4.5974.77 ± 1.2781.08 ± 2.42+6.31
Texas28.0382502980.18 ± 1.2780.18 ± 1.2783.24 ± 2.26+3.06
Wisconsin37.4780883879.08 ± 0.9279.74 ± 1.8584.71 ± 1.47+4.97
OGBN-ArXiv29.0901093049.21 ± 0.9849.37 ± 1.2257.28 ± 0.23+7.91
Flickr5.995742641.86 ± 0.2240.73 ± 1.6943.81 ± 1.41+3.08

We report the complete comparison in Appendix T.1 (Table 19 and Figure 19). Widening a single prototype can improve its result on some datasets, but it does not close the gap to our adaptive combination. Our method remains ahead across all nine benchmarks.

Q3. Lower label-rate studies

Question description: How do Ours, a strong GT baseline, and jointly learned structures behave on Cora and OGBN-ArXiv with 1, 5, 10, and 20 labels per class?

Short answer and conclusion: At 1, 5, and 10 labels per class, our method outperforms the paired Joint control in all 18 comparisons across six datasets. For the standard 20-label setting, we report the submitted results for Ours and DIFFormer separately because these are standard-protocol references.

Detailed response and evidence: We thank the reviewer for asking for a controlled label-rate study. We evaluated our method and a Joint structure-learning control using the same teacher, labeled subset, and paired seeds. In Joint, the student and fuser are updated together with one optimizer, without the outer-loop monitor. We also include DIFFormer as a strong GT reference and JL-R3 as a separate regularized joint-learning reference (jointly learning a structure by node embedding dot products). We report the comparative results in the table below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.