acceptodds
Under review as a conference paper at ICLR 2027

DETAIL: Disentangling Information Transfer and Optimization in Intermediate-layer GNN Distillation

Abstract

Intermediate-layer distillation in graph neural networks (GNNs) is often said to pass useful hidden representations from a teacher to a student. But higher accuracy alone does not always show that teacher information caused the gain. The extra objective may help simply by changing how the student trains. Standard ablations cannot separate these effects because they also change the strength of the auxiliary gradient. We introduce DETAIL, a preregistered test that changes the auxiliary signal while keeping its gradient strength on the student encoder fixed. We replace the teacher's hidden states with a teacher-free variance and covariance objective or with the hidden states of an untrained network of the same architecture, and we also test random Gaussian noise in place of the auxiliary gradient. At every update, we scale each substitute's gradient on the student encoder to the same norm as the real teacher's. On ogbn-arxiv, the intermediate objective improves a GraphSAGE student with 16 hidden units per layer by +1.04 percentage points (pp; 95% confidence interval [0.77, 1.32]) over a baseline that already uses logit distillation. The teacher-free objective and the untrained network recover and of this gain, with a pp upper confidence bound on the real teacher's added benefit over the teacher-free objective. Random noise of the same size recovers 0.76 of the gain on its own. A standard per-dimension objective and a student with 64 hidden units give the same picture. On ogbn-Mag, a rule fixed before any run with the objective predicted that it would help. It did (+0.83 pp [0.59, 1.07]), and the substitutes again recovered most of the gain (0.88 and 0.95). On ogbn-Products, the largest graph, the gain is pp , even with the same exposure as on ogbn-arXiv. Within the tested settings, most of the gain can be reproduced without learned intermediate teacher representations.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph neural networks, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wuSUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the benefits of intermediate-layer GNN distillation arise from learned teacher representations or auxiliary optimization effects. The authors compare a trained teacher with an untrained teacher, a teacher-free variance–covariance objective, and random gradient perturbations under matched gradient norms. Experiments show that the alternative objectives recover most of the trained teacher’s final accuracy gains. Additional analyses examine node correspondence, training dynamics, and generalization across datasets.

Critical strengths
  1. The paper challenges the common assumption that improvements from intermediate-layer distillation necessarily depend on learned teacher representations. Through controlled experiments, the authors show that most of the observed performance gains can be recovered without using a trained teacher's intermediate representations under the studied training setting. This finding provides a valuable perspective on the role of teacher information in knowledge distillation.

  2. The authors conduct a systematic empirical investigation using a diverse set of control experiments, including an untrained teacher, variance–covariance regularization, and random Gaussian gradient perturbations. Additional experiments examine node correspondence, per-dimension regression, gradient matching, and training dynamics. These analyses provide substantial empirical evidence supporting the paper's main findings.

Critical weaknesses
  1. The intermediate-layer auxiliary objective is applied only during the first 150 of 3,000 training updates, rather than throughout student training. This raises concerns about whether the findings generalize to continuous intermediate-layer distillation. Notably, the authors' own results show that the differences between experimental conditions change substantially after auxiliary supervision is removed. Therefore, the observation that alternative auxiliary signals recover most of the trained teacher's final gains may depend on this specific training schedule.

  2. The CT, GV, and RN controls rely on gradient norms recorded from the trained-teacher CC runs. Although they do not use learned teacher representations as targets, their gradient magnitudes still depend on the trained-teacher experiments. Moreover, matching gradient norms does not control gradient directions or optimizer updates. Thus, the experiments do not fully disentangle information transfer from optimization effects.

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Reply to Reviewer wuSU

Thank you for the thorough review and precise concerns. We addressed all concerns and All changes are in the revised PDF in dark green ink.

W1. "The intermediate-layer auxiliary objective is applied only during the first 150 of 3,000 training updates, rather than throughout student training."

The review is right about scarce labels. On full-label arXiv, CT and GV recovered most of CC’s gain with 750 or 3,000 hint updates.

With all labels and the hint kept on for all 3,000 updates, the untrained teacher recovers 95% [86%, 102%] of CC’s gain and GV recovers 96% [88%, 104%]. The original recovery result therefore persists under continuous intermediate supervision. With 750 updates both recover 0.94.

With 1% of the labels, it does matter, as you suspected. With the hint on throughout, the teacher beats the twin by +0.45 pp [+0.20, +0.70] at the end of training (+0.19 pp [+0.04, +0.34] at each run’s best-validation checkpoint, post hoc). With the paper's 150-update window, the twin recovers 0.92, and CC’s final lead over CT is +0.07 pp [−0.21, +0.34]. CC still leads GV by +0.37 pp [+0.11, +0.63]. Three of the four preregistered predictions for that window failed. A lead over CT was visible when the hint switched off (+0.43 pp at update 150, descriptive only) and had narrowed to +0.09 pp [−0.04, +0.22] by update 300 (Figure 6). Please see our response to Reviewer c3JM, W2(Q1).

In the revised PDF, please check the new Section 4.7 with trajectories in Figure 6 for 1% supervision and Figure 7 for full supervision.

W2, first part. "The CT, GV, and RN controls rely on gradient norms recorded from the trained-teacher CC runs."

In the main arXiv setting, unscaled GV recovered 92% of CC’s gain [81%, 102%] without using the CC gradient-norm trace to set its strength.

We ran GV at a fixed strength, from 0.25X to 4X the teacher hint's average gradient size. That average is the only number taken from the teacher runs for the fixed-strength sweep. The reported CT and RN comparisons still use the per-update CC trace. We also ran GV with no rescaling at all, and the reverse case: the teacher's hint rescaled to GV's natural strength.

We found that the gain is flat over the 16X range: +0.95 to +1.03 pp (recovery 0.91–0.99), never significantly below the teacher's hint. The teacher's hint at GV's strength gains +0.92 pp [+0.66, +1.18]. All four predictions passed. Please check the revised PDF (One sentence in Section 3.2 and a dose–response in Figure 8).

W2, second part. “…matching gradient norms does not control gradient directions or optimizer updates.”

Correct, so we measured both encoder update norms and the cosine of the hint gradient with the task gradient. For optimizer updates, please see our response to Reviewer yfCa, W2(Q2).

Direction. The teacher's hint, GV, and the twin all pull against the task gradient to a similar degree (cosine −0.3 to −0.7). Plain noise sits at about 0. Noise built to match the teacher hint's angle to the task gradient recovers 0.74 [0.62, 0.85], close to plain noise (0.77) in the Adam rerun. The observed gap between noise and the structured hints persisted after cosine matching. In the revised PDF, an update-size table and an SGD table (Appendix A.18) are added. The main text states the two limits: GV under SGD and noise under SGD.

W2, third part: “Thus, the experiments do not fully disentangle information transfer from optimization effects.”

We will replace “disentangling” with “probing” to better describe the role of our controls. They test the additional benefit of learned intermediate targets under controlled training conditions, without claiming a unique decomposition of the gain into information transfer and optimization. In the revised PDF, please check the title, the abstract, and Section 1.

Reviewer_yfCaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks the following question: in inner distillation of a GNN student, is it that the student learns something from the teacher’s hidden layers, or is it just because the extra loss changes how the student trains? To test this, the authors take one existing hint loss, and replace the teacher’s hidden states with fake targets; then, at every training step, each fake signal is rescaled so that its gradient on the student has the same size as the real one. The hint is only active for the first steps

The findings are that the real hint gives about +1 pp on top of logit distillation; the teacher-free and untrained teacher versions recover about 96–98% of that, and noise recovers about 77%. The authors conclude that most of the benefit does not need what the teacher actually learned (so it is an optimization smoothing effect).

Critical strengths

Pros:

  • Asks a good question, with a simple control
  • Very careful experimental design and stats: 30 paired seeds, preregistered analyses, reporting of failed predictions, little to no overclaiming
  • The node-pairing experiment is clean, and the advance prediction is nice
Critical weaknesses

Cons:

  • The setting restricts the teacher's power: the teacher’s hidden layers carry almost nothing beyond logits; the student is tiny; the hint runs for only 150 steps. There is no "positive control" showing the method can detect teacher information when it exists.
  • Matching gradient’s size does not take into account Adam with clipping
  • The title and abstract are quite broad, beyond the limited evidence provided
  • The paper reads like a MSc thesis / lab notebook, and key results are buried in the appendix
Questions for authors

Q1. Can you show a case where DETAIL says “the teacher matters”? Q2. Could you report the actual update sizes after Adam and clipping, or repeat the main comparison with SGD? Q3. Isn't the gain mostly fixing a weak baseline? Q4. Is the untrained teacher really "information-free"? It still averages input features over neighbors, right? Q5. Why only 150 steps?

[THank you for the rebuttal response, my questions are addressed in large part.]

2 comments
AuthorsOpenReview ↗

Reply to Reviewer yfCa (part 1/2)

Thank you for the clarification question and the great questions. We answered all questions and concerns and revised the PDF accordingly. All changes in the revised PDF are shown in dark green.

W1(Q1). "Can you show a case where DETAIL says 'the teacher matters'?"

Yes. Please see our response to Reviewer c3JM, W2(Q1).

For the reviewer’s point that the teacher's hidden layers carry almost nothing beyond the logits, we tested a 6-layer teacher whose layers do carry extra, probe-detectable information (Appendix A.11 and Table 29 in Appenidx A.19 in the revised PDF ). The teacher’s hidden layer as the intermediate target has useful information beyond its logits, but using it gave no significant improvement over the untrained teacher.

Please also see our response to Reviewer c3JM, W4(Q3), and check the revised PDF for new Section 4.6 and a new Figure 2.

W2(Q2). "Could you report the actual update sizes after Adam and clipping, or repeat the main comparison with SGD?"

We measured the encoder parameter-update norms under Adam, recorded gradient-clipping events, and repeated the main comparison with SGD.

Clipping. It never activated during the 3,000 updates in any run, so clipping does not explain the differences.

Adam. Relative to CC, the measured encoder-update norms are 0.89–1.10 for GV, 0.97–1.01 for the untrained twin, and 0.64–0.77 for noise. GV and the twin therefore produce updates close in size to CC’s. Noise produces smaller updates, despite matching the gradient norm before Adam.

SGD. The twin recovers 0.98 [0.95, 1.02] of CC’s gain, so the near-complete recovery with untrained targets also holds under SGD. GV recovers 0.80 [0.75, 0.86], below our preregistered prediction of at least 0.90; CC retains a +0.46 pp advantage over GV. Noise recovers only 0.09, compared with 0.77 under Adam.

In the paper. The result for the untrained twin holds under both optimizers, but the amount recovered by GV and especially noise depends on the optimizer. We will include the update measurements and SGD results in appendix tables, and specify in the abstract that the noise result is under Adam.

Please check Appendix A.18 in the revised PDF.

Q3. "Isn't the gain mostly fixing a weak baseline?"

We checked whether the result was driven by the learning-rate choice. The paper's learning rate (0.002) is the best of {0.002, 0.005, 0.01} on validation.

The activation experiments suggest that the larger gain is partly related to how the narrow ReLU student trains. Without a hint, some units become inactive early; this is not observed with CC, CT, or GV. Replacing ReLU with LeakyReLU reduces CC’s gain from +1.04 to +0.37 pp [0.20, 0.54]. This supports early unit inactivity as a contributing factor, although the intervention does not isolate its contribution precisely. We discuss these experiments in our response to Reviewer c3JM, W3(Q2).

The benefit also remains in the wider student, though it is smaller. At width 64, the paper's 150-update window gave +0.36 pp. With the hint on throughout, the gain is +0.13 pp [+0.07, +0.20].

These results help explain why the gain varies with the student. They also reinforce the paper’s motivation: an improvement from intermediate distillation can reflect help with student training, and the accuracy gain alone does not establish that learned teacher representations were necessary.

The abstract and the main results in the revised PDF say this directly. Please check Section 4.3 and Appendix A.20.

Q4. "Is the untrained teacher really 'information-free'? It still averages input features over neighbors, right?"

Right. It is free of learned weights, not of information. We changed the twin’s feature or graph inputs one at a time. The twin still recovers the gain with shuffled features (0.93), with all edges removed (1.03) and on a rewired graph (0.94). We had expected these changes to matter, and 2 of our 3 predictions failed. A fixed random linear projection of the features, with no graph propagation, recovers 0.99 [0.89, 1.09].

Thus, the edge-free twin and random projection can recover most of the gain without graph propagation in constructing the intermediate targets. This does not make feature information irrelevant - the random projection still uses node features. Our conclusion concerns the need for learned intermediate targets; The GraphSAGE student and the trained teacher that supplies logits still use graph information. In the revised PDF, the twin is described as untrained, not as information-free. Please check Appendix A.8 in the revised PDF.

AuthorsOpenReview ↗

Reply to Reviewer yfCa (part 2/2)

W1(Q5). "Why only 150 steps?"

The paper removed the hint after 150 updates to measure what survives (Section 3.4), and a small earlier study had found no larger gain from longer hints (Appendix A.8, n = 5). That was too thin, so we ran longer hints with 30 seeds. Please see our response to Reviewer wuSU, W1.

In the revised PDF, please check the new Section 4.7, with the trajectories in Figure 6 for 1% supervision and Figure 7 for full supervision.

W3. "The title and abstract are quite broad, beyond the limited evidence provided."

Per your request, we revised the title (entangling) and abstract to show what the comparisons establish.

Revised title: “DETAIL: Probing Information Transfer and Optimization in Intermediate-Layer GNN Distillation.”

Revised abstract sentences: “With all training labels and intermediate supervision throughout training, untrained-teacher targets and a student variance–covariance objective recover 95% and 96% of the trained-target gain. When labels and logit supervision are restricted to 1% of training nodes, with intermediate supervision again on throughout training, the trained intermediate targets outperform both substitutes; the untrained targets recover 60% of the gain at the fixed final checkpoint. The size of the gain also depends on student width and activation.”

“Probing” better describes the contribution: our controls test the additional benefit of learned intermediate targets, without claiming a unique causal decomposition of the gain.

Also about the strength:"…the advance prediction is nice." The prediction held on Mag, but not on arxiv-year. There, the baseline had reached 92.8% of its final training accuracy by update 150, and the teacher led by only 1.88 pp. The rule predicted no gain, yet the hint improved accuracy by +1.95 pp [+1.66, +2.23].

We think the issue is that the rule's two numbers (the baseline's training progress at update 150 and the teacher's accuracy lead) miss part of what the hint does. Reaching 92.8% of final training accuracy does not mean training is 92.8% complete. Accuracy can change little while prediction margins and hidden representations are still changing. Similarly, the teacher’s accuracy lead measures the difference in their predictions; it does not measure how much an auxiliary objective could help the student train.

On arxiv-year, the untrained teacher and GV recover about 98% of the gain. This supports the possibility that useful changes to training are missed by our rule, although it does not identify the exact mechanism. We now report both outcomes and clarify that the rule is a heuristic from the original settings, not a general condition for intermediate supervision to help.

In the revised PDF, please check Section 4.5 for the results and Appendix A.19 for a possible explanation of the arxiv-year result.

W4. "The paper reads like a MSc thesis / lab notebook, and key results are buried in the appendix."

Respectfully, we believe our paper was in good shape and format at the time of submission. To address your concern, we made the main argument easier to follow by bringing the positive control, continuous-supervision comparison, and architecture–dataset scope table into the main text and put more detail in the appendix. We welcome any further suggestions on what to put in the main text and what to put in the appendix, and we will make sure we address them properly in the final version.

Reviewer_c3JMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Knowledge distillation aims to train a smaller model (student) out of supervision from a larger one (teacher). In intermediate-layer distillation, this process does not happen only at the output level, but also the student is encouraged to adopt aspects of the internal states of the teacher. Despite this might improve on accuracy, it is not clear if the intermediate layer information is the true underlying cause of the advantage. The authors propose a DETAIL, a test to discern the value of intermediate-layer distillation on Graph Neural Networks.

Critical strengths

-The manuscript shows a solid formulation with extensive complementary material. -The authors succeed in showing that information-layer distillation gains can be matched for specific experimental settings. -The authors are thoughtful in the execution of experiments, which are extensively described.

Critical weaknesses

-The experiments support the claim that most of the gain can be reproduced without learning intermediate-layer representations. However, this does not disentangle information transfer from optimization, as the title and introduction section suggest. The method does not discern if it is indeed the intermediate-layer information causing the advantage, instead, it proves that such advantage can be matched in a different way. These two claims are not equivalent.

For example, the gain is mostly matched by GV, which is a regularization on the student features. That regularization might be the teacher's additional knowledge for this particular experimental settings, which later transferred to the student on the reference CC.

-The manuscript lacks a positive control case. The paper’s title and framing suggest that DETAIL separates information transfer from optimization, but the framework is never validated in a case where useful intermediate-layer information is known to exist. Having a case where intermediate-layer information (CC) matters more than a regularization (GV) or untrained teacher (CT), would improve on the strength of the test.

-Showing that a gain similar to the intermediate-layer, can be obtained without intermediate-layer information is too narrow to justify a full publication. Identifying the possible cause of the intermediate-layer gain would strengthen the claims of the publication.

-The experiments include a narrow scope of teacher-student architectures. Considering deeper teachers or MLP students would help to improve confidence on the generalization of the method beyond these settings.

Questions for authors

As described in weakneses,

-Can you provide a positive control case where CC is not matched by GV or CT? -Do you have insights of what might be the actual cause behind the information-layer distillation? Is it indeed equivalent to a regularization of graph features? In that case, which regularization? Does it depend on the dataset or the architecture? -Can you provide more architectures for students (e.g. MLP on GV) and deeper teachers (e.g. GCNII, RevGCN-Deep...). -Can you provide experiments on a more heterophilic dataset (e.g. arxiv-year..)?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer c3JM (part 1/2)

Thank you for the careful review and valuable comments. We addressed all comments below and put the corresponding results and extensive explanations in the revised PDF.

W1. "…this does not disentangle information transfer from optimization, as the title and introduction section suggest."

DETAIL compares the trained teacher’s hint with other auxiliary signals of the same gradient norm. It asks whether these alternatives can recover the accuracy gain. It does not tell us exactly why the teacher’s hint works.

GV may provide the same kind of regularization as CC, as you suggest. Our results do not rule this out. With all labels, an untrained teacher with no edges recovers 103% of CC’s gain, and a random projection of node features recovers 99%. These controls achieve similar gains without learned intermediate targets. However, similar accuracy does not mean they work in the same way. All conditions still use the trained teacher’s logits.

We have changed “disentangling” to “probing” and clarified this point in the title, abstract, Sections 1 and 3.4, and Figure 1(a). Please see our response to Reviewer yfCa, W3. Please check the revised PDF for all these changes.

W2(Q1). "Can you provide a positive control case where CC is not matched by GV or CT?"

Yes. We added a positive control on arxiv with labels and logit supervision restricted to 1% of the training nodes, and the intermediate hint active for all 3,000 updates in training. In this setting, CC gains +1.11 pp [+0.84, +1.39] and significantly outperforms all three substitutes: the untrained teacher recovers 60%, GV 36%, and noise 21%. The teacher beats the twin (i.e., CT, the untrained teacher) by +0.45 pp [+0.20, +0.70] and GV by +0.72 pp [+0.42, +1.01]. All three preregistered predictions passed, as did the preregistered joint test (Holm-adjusted p = 0.002). So the test can detect an advantage for trained targets.

In a synthetic known-answer task, the teacher also beats both substitutes (Section 4.6). We compared different amounts of supervision under the same continuous-hint schedule. The new finding is that the teacher's share depends on the labels. The table below (with 3000 steps) shows this. ( Please check Table 19 with both 150 and 3000 steps and the left panel of Figure 2 in the revised PDF):

\hline \textbf{Labels} & \textbf{Teacher's lead over GV} & \textbf{Teacher's lead over the twin} \\\\ \hline 1\\% & \text{+0.72 pp [+0.42, +1.01]} & \text{+0.45 pp [+0.20, +0.70]} \\\\ \hline 10\\% & \text{+0.20 pp [+0.07, +0.33]} & \text{+0.27 pp [+0.18, +0.36]} \\\\ \hline 100\\% & \text{+0.06 pp [−0.06, +0.18]} & \text{+0.08 pp [−0.03, +0.18]} \\\\ \hline \end{array}$$ These are end-of-training values. Only the 1% row overfits: in the post hoc comparison at each run’s best-validation checkpoint, CC leads GV by +0.17 pp and CT by +0.19 pp, while the 10% and 100% results barely change. At full supervision, both fixed-final confidence intervals include zero. The total gain from the hint stays at +1.1 to +1.4 pp. What changes is the size of CC’s lead over GV and CT? So, the paper's full-label result describes that regime. It is not a blind spot of the test. (The 10% row is descriptive: it had no preregistered prediction.) We also ran a synthetic task with a known generating rule. In this task, CC also beats both CT and GV (Section 4.6). It gives weaker supporting evidence that CC gains +0.43 pp, below our prediction of more than +5 pp, and the student remains below the majority-class accuracy. We therefore base the positive-control conclusion primarily on the arxiv experiment. Please check the new Section 4.6 and Figure 2 in the revised PDF.
AuthorsOpenReview ↗

Reply to Reviewer c3JM (part 2/2)

W3(Q2). "…what might be the actual cause behind the information-layer distillation? … which regularization? Does it depend on the dataset or the architecture?"

Early ReLU unit inactivity is a possible contributor to the gain in the width-16 arXiv student.

What we saw (exploratory). Without a hint, the width-16 student has, on average, 2.93 of its 16 hidden ReLU units dead by update 150, and on average still 2.93 at update 3,000. Students hinted by the teacher, the twin or GV have none. Across seeds, the hint's gain grows with the number of units the baseline lost by update 150 (Spearman +0.71).

What we then tested (preregistered). We replaced ReLU with LeakyReLU, so that units can recover. The hint's measured gain fell from +1.04 pp to +0.37 pp [+0.20, +0.54], a reduction of about two-thirds. The LeakyReLU baseline still had 3.47 inactive units at update 150, and they recovered by update 3,000.

Which regularization? No single one. The teacher's hint, the twin, GV and a random projection produce similar gains in the main arXiv setting, and the random-projection target uses node features without graph propagation.

Dataset and architecture. The size of the gain depends on them. For example, at width 64 with the hint on throughout, it is +0.13 pp [+0.07, +0.20]. With all labels, whether the gain needs a trained teacher does not depend on them. Specifically, across the full-label Adam configurations in Table 5 with a detectable CC gain and a CT comparison, CT recovered most of the gain.

In the revised PDF, the gain is described as specific to narrow ReLU students. The dead-unit analysis and the LeakyReLU test are added (Appendix A.20).

W4(Q3), Q4. "Can you provide more architectures for students (e.g. MLP on GV) and deeper teachers (e.g. GCNII,...)"; "a more heterophilic dataset (e.g. arxiv-year..)".

We ran an MLP student, two deeper teachers (6-layer GATv2 and 16-layer GCNII), and arxiv-year. Each changes one thing from the main setup (GraphSAGE student, 3-layer GATv2 teacher, ogbn-arxiv). The untrained twin (CT) is a network with the same architecture as the teacher that was never trained. In all four settings, the twin and GV recover about as much of the gain as the trained teacher's hint gives. We found no detectable CC advantage over CT in these four settings.

\hline \textbf{Settings} & \begin{array}{c}\textbf{Baseline} \\\\ \textbf{(no hint)}\end{array} & \begin{array}{c}\textbf{Trained teacher’s} \\\\ \textbf{hint (CC)}\end{array} & \begin{array}{c}\textbf{Untrained twin’s} \\\\ \textbf{hint (CT)}\end{array} & \textbf{GV} & \begin{array}{c}\textbf{Gain from teacher's} \\\\ \textbf{hint (CC - baseline)}\end{array} \\\\ \hline \text{MLP student (never sees the graph)} & 46.38\\% & 46.94\\% & 46.88\\% & 46.97\\% & \text{+0.56 pp [+0.32, +0.80]} \\\\ \hline \text{6-layer GATv2 teacher} & 64.96\\% & 66.18\\% & 66.18\\% & 66.11\\% & \text{+1.22 pp [+0.88, +1.56]} \\\\ \hline \text{16-layer GCNII teacher} & 64.97\\% & 66.04\\% & 66.08\\% & 66.02\\% & \text{+1.07 pp [+0.75, +1.39]} \\\\ \hline \text{arxiv-year (heterophilic)} & 40.77\\% & 42.71\\% & 42.68\\% & 42.67\\% & \text{+1.95 pp [+1.66, +2.23]} \\\\ \hline \end{array}$$ All rows use all labels and the paper's 150-update window. The baseline is the student with logit distillation and no hint; accuracies are test accuracies. arxiv-year has 29 seeds after one preregistered exclusion. Including that seed gives a CC − B gain of +1.97 pp [+1.70, +2.25]. In the revised PDF, please check the scope table in Section 4.5 (Table 5), with full tables in Appendix A.19 (Table 29).

I appreciate the time and effort of the authors on the response. The new architectures and the heterophilic dataset address my scope concerns.

I have three major concerns that are still unaddressed.


Lack of Motivation. These are useful diagnostic experiments, but they do not provide an exhaustive or theoretically principled decomposition of intermediate-layer distillation effects. They lack motivation.

Understanding the principles behind the test selection would be relevant to understand what is exactly being tested and what's exactly being left out of the teacher's knowledge.


Lack of Underlying Cause. I appreciate the effort of the authors on the exploratory analysis of the cause behind the gain. However, the dead ReLU is still not enough to understand the true reason for the intermediate distillation effect. Dead units are already prevented in some transformer architectures using GELU that succeed at intermediate layer distillation (TinyBERT, 2020).

The lack of understanding of distillation causes is tied to the lack of test motivation: since the tests are empirical and ad hoc, they do not provide a better understanding of the intermediate distillation process.

Understanding the cause, rather than accounting for an empirical test, is valuable since it adds to our actual knowledge of how intermediate layer distillation works.


Insufficient Experiments. The results are not convincing. Despite the new experiments, the results of untrained intermediate layers make me unsure whether the method is being tested in the correct setup.

  • It is known that the effects of intermediate feature distillation may fail for lightweight students (On Representation Knowledge Distillation for Graph Neural Networks, 2023). The students here are even narrower than in that work, and the teacher–student gaps are small.
  • It is also known that in some cases vanilla KD might outperform intermediate layer distillation (Contrastive Representation Distillation, 2019). Moreover, the KD loss in Eq. 1 is rescaled in a way that considerably weakens it, bringing the KD baseline closer to plain cross-entropy loss. Can the authors justify this choice?
  • Reviewer yfCa pointed to a relevant fact: even learning untrained intermediate layers recovers most of the hint’s gain. I have to remain skeptical in this regard. Could the authors share the code?

Additionally, the positive case introduced was shallow since, as the authors recognize, it overfits for 1% of nodes. At the best-validation checkpoint, CC leads GV and CT only marginally.

Do you have an expressive setup where the model has proven to improve with intermediate layer distillation, where the test gives positive results?

Overall, I agree with Reviewer yfCa: the manuscript reads as a collection of experiments rather than a unified argument.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Strengths Highlighted by the Reviewers

We thank the reviewers for their thoughtful feedback. We are encouraged that they value our research question and simple control (Reviewer yfCa) and recognize that intermediate-layer distillation gains can be matched in specific experimental settings (Reviewer c3JM). We appreciate their recognition of the care taken in conducting the experiments (Reviewer c3JM), the range of controls and supporting evidence (Reviewer wuSU), and the careful experimental design and statistical analysis, including 30 paired seeds, preregistered analyses, and reporting of failed predictions (Reviewer yfCa). We also thank Reviewer yfCa for highlighting the clean node-pairing experiment and the advance prediction, and Reviewer c3JM for highlighting the paper’s solid formulation, extensive supplementary material, and detailed experimental descriptions.

Below, we respond to the reviewers’ questions and concerns and point to the changes and additional results in the revised PDF. Please make sure you check the newly uploaded PDF.

AuthorsOpenReview ↗

Summary of how we addressed all comments, and where the changes are in the revised PDF

We thank the reviewers and the AC again. We believe we have addressed all comments raised in the reviews and in the discussion. The table below summarizes each concern, how we resolved it, and where it is in the revised PDF. A concern shared by several reviewers appears once. First-round changes are in green and second-round changes are in blue.

To answer these comments, we ran extensive additional experiments, taking into account every single point mentioned by our reviewers.

CC is the trained teacher's hint, CT the untrained teacher, and GV the teacher-free objective.

Reviewer(s)ConcernHow we addressed itWhere
c3JM, yfCa, wuSUTitle and framing: the test does not fully "disentangle" information transfer from optimizationChanged "Disentangling" to "Probing". The paper now says the comparisons test whether the gain requires trained intermediate targets, and that they report bounds, not a full decompositionTitle, abstract, Sections 1 and 3.4, Figure 1(a)
c3JMMotivation of the tests: what is tested and what is left outStated the one rule behind the tests: remove one ingredient of the hint at a time and keep its strength fixedSection 3, Appendix A.22 (Table 34)
c3JM, yfCaNo positive control where the teacher mattersWith 1% of the labels and the hint on throughout, CC beats CT by +0.45 pp and GV by +0.72 pp. A synthetic known-answer task agreesSection 4.6, Figure 2, Appendix A.15
c3JMA positive result in an expressive setup; students are narrowRan the 256-unit student of Joshi et al. With 10% of the labels, CC beats CT by +0.21 pp and GV by +0.63 pp at the fixed final checkpoint (the lead over CT includes zero at best validation). With all labels the gain is +0.17 pp and CT recovers 0.80Section 4.6, Table 5, Appendix A.24
c3JM, yfCaUnderlying cause; is the gain fixing a weak baseline?Dead-unit analysis, then LeakyReLU and GELU students: the gain falls from +1.04 to +0.37 and +0.16 pp. The gain has three parts, each with its own test. Learning rate 0.002 is the best on validationSection 4.3, Appendix A.1.1, A.20, A.25, A.26 (Table 43)
c3JMEq. 1 weakens KD; vanilla KDReran Table 2 with the standard KD form, tuned and at the paper's values. The gain stays (+1.24 and +1.05 pp), CT and GV recover it, and the baseline is not strongerSection 2.1, Table 5, Appendix A.23
c3JM, yfCaNarrow scope; the teacher's hidden layers carry little beyond logitsAdded an MLP student, 6-layer GATv2 and 16-layer GCNII teachers, and arxiv-year. The 6-layer teacher has probe-detectable extra information. CT and GV recover the gain in all fourSection 4.5 (Table 5), Appendix A.11, A.19
yfCa, wuSUHint on for only 150 of 3,000 updatesKept the hint on for 750 and 3,000 updates. With all labels CT recovers 95% and GV 96%. With 1% of the labels a short hint can miss the teacher effect, and the paper says soSection 4.7, Table 5, Appendix A.16, Figures 6 and 7
yfCa, wuSUAdam, clipping, update sizes and gradient directionClipping never activated. We report update norms and cosines. Under SGD, CT recovers 0.98, GV 0.80 and noise 0.09, so the result for CT holds under both optimizersSection 4.7, Appendix A.18
wuSUControls rely on gradient norms from the CC runsRan GV at a fixed strength from 0.25X to 4X and with no rescaling. The gain is flat (+0.95 to +1.03 pp)Sections 3.2 and 4.7, Appendix A.17, Figure 8
yfCaIs the untrained teacher information-free?It is now described as untrained, not information-free. It still recovers the gain with shuffled features (0.93), with all edges removed (1.03) and as a random projection (0.99)Section 4.7, Appendix A.19 (Table 28)
c3JM, yfCaReads as a collection of experiments; key results in the appendixMoved the positive control, the continuous-hint comparison and the scope table into the main text. The paper states one rule and shows all cohorts side by sideSections 3 and 4.5 to 4.7, Conclusion, Table 43
c3JMSkeptical about untrained targets; codeShared a Python script with a README for this comparison. Trained targets on the wrong nodes cost 2.4 pp, so the student does react to target contentLink in our reply to c3JM, Section 4.4.1

We sincerely appreciate the time the reviewers and the AC have given to this paper. If these changes resolve your concerns, we would be grateful if you would consider updating your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.