- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes interaction-space distillation for Graph Transformers. A GNN teacher builds task-informed prototype graphs, which a student combines across nodes and layers. Results on nine node-classification datasets are promising.
The central claim needs stronger support. Single-graph comparisons do not match edge counts or training budgets, so the benefit of searching an interaction space remains unclear. The paper also defines admissible graphs as convex combinations while allowing Max fusion, and does not explain whether the student is refitted after the final fuser update. Its pseudo-label monitor is not a direct measure of generalization.
These issues limit the mechanism claim and reproducibility. Matched-budget controls and a precise account of the implemented training procedure are needed.
Critical strengths
1.Clear problem formulation. The paper distinguishes between deciding which nodes may interact and assigning attention weights to those interactions. It identifies a concrete structure-selection challenge under limited supervision.
2.Thoughtful method design. The teacher supplies multiple task-informed prototype graphs, while the student can combine them differently across nodes and layers. Results on nine datasets and several ablations provide empirical support for the complete approach.
Critical weaknesses
1.The main contribution is not fully isolated. Multi-prototype fusion is compared with single graphs without matching candidate-edge counts or training budgets, so the gains cannot be attributed solely to interaction-space search.
2.The optimization procedure is unclear. Fusion coefficients depend on changing student representations, meaning that a fixed fuser may still produce a changing graph. The paper also does not specify whether the student is refitted after the final fuser update.
Questions for authors
-
Definition of the interaction space. Which fusion operator, Sum or element-wise Max, is used in the reported experiments? Since the admissible graph family is defined through convex combinations of prototypes, please clarify how Max fusion fits that definition and which claims depend on the choice of operator.
-
Interpretation of the alternating optimization. Are the fused graphs recomputed as student representations change during inner training, even when the fuser parameters are fixed? After the final fuser update, is the student refitted using the resulting graphs? These details affect whether the implemented procedure corresponds to the stated bilevel objective.
-
Evidence for adaptive interaction-space search. Multi-prototype fusion may change the number of candidate edges, model capacity, and training cost relative to a single graph. Can the authors provide comparisons that match these factors and report whether adaptive fusion still improves test performance? This would clarify how much of the observed gain is attributable to searching the proposed interaction space.
8 comments
Q3. Operator-dependent methodological claims
Question description: Which theoretical or methodological claims depend on the choice between Sum and Max fusion?
Short answer and conclusion: The simplex, convex-hull, and convex-combination interpretation applies to Sum fusion. Max and temperature-softmax offer alternative ways to combine the same candidate structures, but they do not have that convex-combination interpretation.
Detailed response and evidence: We appreciate the request to distinguish the operator-independent idea from the claims that require a particular fusion rule. Under Sum, the nonnegative fusion coefficients add up to one, so each node's potential neighborhood is a convex combination of the prototype neighborhoods. This is the basis of our simplex and convex-hull interpretation. By contrast, Max selects the largest weighted value for each edge, and temperature-softmax gives a smooth max-like combination. These operators still use the teacher-derived prototype family and node-wise fuser, while changing the way an edge's prototype contributions are merged.
The full nine-dataset table is provided in Q1 above (also Appendix T.6, Table 39 and Figure 33). Sum performs better than both alternatives on all nine datasets, supporting our choice of Sum for the main method. Thus, our task-informed candidate-space and student-adaptive fusion framework remains meaningful across operators, while the specific convex-combination geometry belongs to Sum.
Q4. Dynamic graph recomputation during inner training
Question description: Are fused graphs recomputed as student representations change during inner training, even when fuser parameters remain fixed?
Short answer and conclusion: We construct each potential graph once at the beginning of an alternating round and keep it fixed throughout that round’s inner student-fitting phase. We do not recompute it at every inner forward pass.
Detailed response and evidence: We appreciate the request to clarify this point. We follow four steps. (1) At the start of a round, we use the current student representation and fuser to construct the potential graph. (2) We fit the student with this graph fixed throughout the inner loop. (3) We hold the fitted student fixed and update the fuser in the outer loop. (4) We use the updated fuser to construct the graph for the next round. Thus, the student representation may change during inner fitting, but the potential graph for that round remains fixed. The stage-level timing table in Q8 below (Appendix T.5, Table 33) reports 2–5 graph-fusion calls per dataset, matching the 2–5 outer-update calls, while the student inner loop runs 200–600 calls. This call-count pattern is consistent with constructing the fused graph at the round level and reusing it during that round’s inner fitting. Algorithm 5 describes the same update order.
Q5. Student refitting after the final fuser update
Question description: After the final fuser update, is the student refitted using the resulting graphs?
Detailed response and evidence: Thank you for asking us to state the final step explicitly. In each round, Algorithm 5 first fits the student and then updates the fuser. Once the final outer update is complete, the procedure returns the student from that round’s inner phase. It does not start an additional inner phase using the graph implied by the last fuser update. We have clarified this point in Appendix T.5.
Q6. Generalization validity of the pseudo-label monitor
Question description: What does agreement with the teacher-pseudo-labeled monitor establish about generalization, given that it is not ground-truth validation accuracy?
Short answer and conclusion: Agreement with the teacher-pseudo-labeled monitor is used to select historical-success checkpoints. It is not a direct estimate of test accuracy. Our offline oracle comparison also shows that a more accurate monitor does not necessarily lead to a better final student.
Detailed response and evidence: We appreciate the reviewer emphasizing this distinction. We agree that the monitor should not be described as a ground-truth validation set. In our method, the fixed monitor contains high-confidence teacher pseudo-labels and is used to decide whether an inner-loop checkpoint qualifies as a new historical success. When a checkpoint passes the current agreement threshold, we confidence-weight its predictions into . The outer fuser objective then uses the updated meta-targets. Thus, monitor agreement controls checkpoint selection, while the outer loss is computed from .
We also reproduce the full nine-dataset diagnostic here so this answer can be read independently. Ours (paper) is the submitted method’s reference test accuracy. Oracle-monitor student is the final student from the offline diagnostic that substitutes an all-node ground-truth monitor. Pseudo-monitor and Oracle-monitor are the confidence-weighted correct-target shares of the meta-target distributions under the two monitor choices. Specifically,
\qquad w_u = \bigl(\operatorname{INE}(Y_{M,u})\bigr)^{\eta}.$$ For each node $u$, $Y_{M,u}$ is the current meta-target distribution, $y_u$ is its ground-truth class for this offline diagnostic, and $w_u$ is the confidence weight used by the outer objective. Thus, $R$ is the weighted share of nodes whose meta-target argmax matches the ground-truth class. It measures meta-target correctness, not the monitor’s own agreement score. In the table, Ours (paper) is the submitted method’s test-accuracy reference, Oracle-monitor student is the offline oracle diagnostic’s final test accuracy, and the two $R$ columns give the corresponding confidence-weighted meta-target correctness shares. | Dataset | Ours (paper) | Oracle-monitor student | Pseudo-monitor $R$ | Oracle-monitor $R$ | |---|---:|---:|---:|---:| | Cora | 84.88 ± 0.38 | 83.58 ± 0.86 | 0.8509 | 0.8474 | | Citeseer | 74.05 ± 0.35 | 72.27 ± 0.59 | 0.7433 | 0.7271 | | Pubmed | 82.58 ± 0.18 | 81.23 ± 0.95 | 0.8155 | 0.8234 | | Actor | 38.74 ± 0.20 | 36.19 ± 0.42 | 0.8299 | 0.8300 | | Cornell | 81.08 ± 2.42 | 82.52 ± 1.08 | 0.8915 | 0.8982 | | Texas | 83.24 ± 2.26 | 80.18 ± 2.55 | 0.8930 | 0.9198 | | Wisconsin | 84.71 ± 1.47 | 81.54 ± 1.69 | 0.9026 | 0.9180 | | OGBN-ArXiv | 57.28 ± 0.23 | 54.59 ± 2.07 | 0.5659 | 0.5725 | | Flickr | 43.81 ± 1.41 | 42.90 ± 7.70 | 0.5340 | 0.5945 | | Dataset | Oracle-monitor student − Ours (paper) (pp) | Oracle-monitor $R$ − Pseudo-monitor $R$ | |---|---:|---:| | Cora | −1.30 | −0.0035 | | Citeseer | −1.78 | −0.0162 | | Pubmed | −1.35 | +0.0079 | In the first table, we compare the full-method reference, the oracle-monitor student result, and weighted meta-target correctness across nine datasets. Its first difference is final student accuracy, and its second is the change in $R$. The oracle raises $R$ on seven of nine datasets, but final accuracy does not improve consistently. On the three paired citation datasets, the accuracy changes are -1.30, −1.78, and −1.35 pp. One possible explanation is that ***small errors in the pseudo monitor may have a mild regularizing effect on checkpoint selection and structure learning.***. The oracle uses all-node ground truth only for this offline analysis. Our training procedure continues to use the pseudo-labeled monitor. We include the same diagnostic in Appendix T.4 (Table 30 and Figure 29).
We sincerely thank you for your insightful and constructive comments. We have carefully considered your feedback and responded to each of your points below.
Q1. Fusion operator used in the experiments
Question description: Which fusion operator, Sum or element-wise Max, is used in the reported experiments?
Short answer and conclusion: All reported main experiments use element-wise Sum fusion. The complete operator comparison below also shows that Sum outperforms the tested Max and temperature-softmax alternatives on all nine datasets.
Detailed response and evidence: We thank the reviewer for asking us to clarify this important implementation choice. We use element-wise Sum in all main experiments. To examine the choice more closely, we also evaluate element-wise Max and a temperature-softmax alternative while keeping the teacher, prototype graphs, student architecture, splits, and checkpoint-selection protocol unchanged. Sum combines the weighted contributions of all prototypes. Max selects the largest weighted contribution for each edge. Temperature-softmax provides a smooth max-like alternative, with temperature τ=0.05. We provide some comparative experiments with results reported below. Each entry is test accuracy (mean ± SD, %). DIFFormer is included only as a GT reference baseline, not as a fusion-operator variant.
Sum outperforms Max on all nine datasets by 0.85–7.39 percentage points and temperature-softmax by 1.02–10.29 points. This gives us both a clear default operator and empirical support for that choice. We include the same experiment in Appendix T.6 (Table 39 and Figure 33).
Q2. Max fusion and the convex-combination definition
Question description: How does Max fusion fit the convex-combination definition of the admissible graph family?
Short answer and conclusion: The convex-combination interpretation applies only to Sum, whereas Max is a nonlinear alternative outside the simplex-defined family.
Detailed response and evidence: Thank you for pointing out this distinction. For an edge (u,v), we define au,v,k=βu,kGk,u,v. Under Sum, the fused edge weight is ∑kau,v,k, so the nonnegative coefficients βu,k, which sum to one, give a convex combination of prototype edge weights. Under Max, the result is maxkau,v,k, which is a nonlinear operation and does not have the same convex-combination interpretation. We also test temperature-softmax as a smooth max-like alternative, defined by:
Mτ(au,v,:)=∑k=1KSoftmax(τau,v,:)kau,v,k,τ=0.05
For clarity, the convex-combination interpretation is specific to Sum. Max and temperature-softmax are nonlinear alternatives. We make this distinction explicit in Appendix T.6.