DETAIL: Disentangling Information Transfer and Optimization in Intermediate-layer GNN Distillation

Intermediate-layer distillation in graph neural networks (GNNs) is often said to pass useful hidden representations from a teacher to a student. But higher accuracy alone does not always show that teacher information caused the gain. The extra objective may help simply by changing how the student trains. Standard ablations cannot separate these effects because they also change the strength of the auxiliary gradient. We introduce DETAIL, a preregistered test that changes the auxiliary signal while keeping its gradient strength on the student encoder fixed. We replace the teacher's hidden states with a teacher-free variance and covariance objective or with the hidden states of an untrained network of the same architecture, and we also test random Gaussian noise in place of the auxiliary gradient. At every update, we scale each substitute's gradient on the student encoder to the same norm as the real teacher's. On ogbn-arxiv, the intermediate objective improves a GraphSAGE student with 16 hidden units per layer by +1.04 percentage points (pp; 95% confidence interval [0.77, 1.32]) over a baseline that already uses logit distillation. The teacher-free objective and the untrained network recover $0.98$ and $0.96$ of this gain, with a $+0.13$ pp upper confidence bound on the real teacher's added benefit over the teacher-free objective. Random noise of the same size recovers 0.76 of the gain on its own. A standard per-dimension objective and a student with 64 hidden units give the same picture. On ogbn-Mag, a rule fixed before any run with the objective predicted that it would help. It did (+0.83 pp [0.59, 1.07]), and the substitutes again recovered most of the gain (0.88 and 0.95). On ogbn-Products, the largest graph, the gain is $+0.13$ pp $[-0.03,0.29]$, even with the same exposure as on ogbn-arXiv. Within the tested settings, most of the gain can be reproduced without learned intermediate teacher representations.