- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the benefits of intermediate-layer GNN distillation arise from learned teacher representations or auxiliary optimization effects. The authors compare a trained teacher with an untrained teacher, a teacher-free variance–covariance objective, and random gradient perturbations under matched gradient norms. Experiments show that the alternative objectives recover most of the trained teacher’s final accuracy gains. Additional analyses examine node correspondence, training dynamics, and generalization across datasets.
Critical strengths
-
The paper challenges the common assumption that improvements from intermediate-layer distillation necessarily depend on learned teacher representations. Through controlled experiments, the authors show that most of the observed performance gains can be recovered without using a trained teacher's intermediate representations under the studied training setting. This finding provides a valuable perspective on the role of teacher information in knowledge distillation.
-
The authors conduct a systematic empirical investigation using a diverse set of control experiments, including an untrained teacher, variance–covariance regularization, and random Gaussian gradient perturbations. Additional experiments examine node correspondence, per-dimension regression, gradient matching, and training dynamics. These analyses provide substantial empirical evidence supporting the paper's main findings.
Critical weaknesses
-
The intermediate-layer auxiliary objective is applied only during the first 150 of 3,000 training updates, rather than throughout student training. This raises concerns about whether the findings generalize to continuous intermediate-layer distillation. Notably, the authors' own results show that the differences between experimental conditions change substantially after auxiliary supervision is removed. Therefore, the observation that alternative auxiliary signals recover most of the trained teacher's final gains may depend on this specific training schedule.
-
The CT, GV, and RN controls rely on gradient norms recorded from the trained-teacher CC runs. Although they do not use learned teacher representations as targets, their gradient magnitudes still depend on the trained-teacher experiments. Moreover, matching gradient norms does not control gradient directions or optimizer updates. Thus, the experiments do not fully disentangle information transfer from optimization effects.
Questions for authors
See weaknesses
Reply to Reviewer wuSU
Thank you for the thorough review and precise concerns. We addressed all concerns and All changes are in the revised PDF in dark green ink.
W1. "The intermediate-layer auxiliary objective is applied only during the first 150 of 3,000 training updates, rather than throughout student training."
The review is right about scarce labels. On full-label arXiv, CT and GV recovered most of CC’s gain with 750 or 3,000 hint updates.
With all labels and the hint kept on for all 3,000 updates, the untrained teacher recovers 95% [86%, 102%] of CC’s gain and GV recovers 96% [88%, 104%]. The original recovery result therefore persists under continuous intermediate supervision. With 750 updates both recover 0.94.
With 1% of the labels, it does matter, as you suspected. With the hint on throughout, the teacher beats the twin by +0.45 pp [+0.20, +0.70] at the end of training (+0.19 pp [+0.04, +0.34] at each run’s best-validation checkpoint, post hoc). With the paper's 150-update window, the twin recovers 0.92, and CC’s final lead over CT is +0.07 pp [−0.21, +0.34]. CC still leads GV by +0.37 pp [+0.11, +0.63]. Three of the four preregistered predictions for that window failed. A lead over CT was visible when the hint switched off (+0.43 pp at update 150, descriptive only) and had narrowed to +0.09 pp [−0.04, +0.22] by update 300 (Figure 6). Please see our response to Reviewer c3JM, W2(Q1).
In the revised PDF, please check the new Section 4.7 with trajectories in Figure 6 for 1% supervision and Figure 7 for full supervision.
W2, first part. "The CT, GV, and RN controls rely on gradient norms recorded from the trained-teacher CC runs."
In the main arXiv setting, unscaled GV recovered 92% of CC’s gain [81%, 102%] without using the CC gradient-norm trace to set its strength.
We ran GV at a fixed strength, from 0.25X to 4X the teacher hint's average gradient size. That average is the only number taken from the teacher runs for the fixed-strength sweep. The reported CT and RN comparisons still use the per-update CC trace. We also ran GV with no rescaling at all, and the reverse case: the teacher's hint rescaled to GV's natural strength.
We found that the gain is flat over the 16X range: +0.95 to +1.03 pp (recovery 0.91–0.99), never significantly below the teacher's hint. The teacher's hint at GV's strength gains +0.92 pp [+0.66, +1.18]. All four predictions passed. Please check the revised PDF (One sentence in Section 3.2 and a dose–response in Figure 8).
W2, second part. “…matching gradient norms does not control gradient directions or optimizer updates.”
Correct, so we measured both encoder update norms and the cosine of the hint gradient with the task gradient. For optimizer updates, please see our response to Reviewer yfCa, W2(Q2).
Direction. The teacher's hint, GV, and the twin all pull against the task gradient to a similar degree (cosine −0.3 to −0.7). Plain noise sits at about 0. Noise built to match the teacher hint's angle to the task gradient recovers 0.74 [0.62, 0.85], close to plain noise (0.77) in the Adam rerun. The observed gap between noise and the structured hints persisted after cosine matching. In the revised PDF, an update-size table and an SGD table (Appendix A.18) are added. The main text states the two limits: GV under SGD and noise under SGD.
W2, third part: “Thus, the experiments do not fully disentangle information transfer from optimization effects.”
We will replace “disentangling” with “probing” to better describe the role of our controls. They test the additional benefit of learned intermediate targets under controlled training conditions, without claiming a unique decomposition of the gain into information transfer and optimization. In the revised PDF, please check the title, the abstract, and Section 1.