acceptodds
Withdrawn from ICLR 2027

MIDAS: Multi-Teacher Knowledge Distillation with Adaptive Structure-Aware Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Graph Neural Networks achieve strong performance on graph-structured data but incur high inference latency due to iterative neighborhood aggregation. Knowledge distillation into lightweight multi-layer perceptrons (MLPs) addresses this issue. However, these methods are constrained by the choice of a single teacher, which can be sub-optimal in structurally diverse regions. We propose MIDAS, a multi-teacher distillation framework designed to obtain optimal supervision from GNN architectures across structurally diverse regions, with a student MLP that benefits from the locally optimal teacher rather than a globally best one. MIDAS realizes this through: (1) a dual-encoder contrastive alignment module that embeds structural identity into node representations without graph access at inference, (2) a density-based clustering and cluster-conditional teacher routing mechanism that assigns each node to its locally optimal teacher, and (3) a trunk-adapter student architecture with a two-phase training strategy that prevents gradient interference across divergent teacher signals. Experiments on seven benchmark datasets show that MIDAS outperforms 13 baselines across single-teacher and ensemble distillation settings, achieving up to 80 inference speedup

Primary area:
learning on graphs and other geometries & topologies
Keywords:
GNN to MLP Knowledge Distillation, Multi teacher Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bdYFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for multi-teacher knowledge distillation for graph neural networks. The primary motivation is to leverage on the knowledge of best-matched teacher for a structured region, in order to learn better the local structural context, making "graph free" inferences. Multi-teacher knowledge distillation, ie., distillation from multiple teachers into a single student learning model is not a novel idea (literature shows quite a few works on this topic). The paper argue that most GNN distillation work uses only a single teacher, hence the paper attempts to make contributions in the context of GNN, plus using local regional structural contextual information as a chosen teacher.

Critical strengths

Multi-teacher knowledge distillation is an interesting topic becoming increasingly popular in the area of knowledge distillation. It makes sense that ingesting multiple teachers' knowledge into a student model is likely to produce models that are more robust and generalize better, in some way similar to the ensemble methods vs single machine learning models. This paper advocates such a paradigm, and attempts to improve feature alignment and structural embeddings, so that through clustering the best teacher with the best regional structural knowledge can be identified.

The paper builds on existing literature in this steadily growing area of using multi-teacher knowledge distillation, so it is reasonably safe to extend the approach into GNN with some variations.

Critical weaknesses

For the experimental section, the 5 research questions are too vague and generic? These need to be more sharply defined.

Since multi-teacher knowledge distillation into single student model has already been well explored in literature, this work at the best only represents small incremental steps forward. The results are not surprising either, consistent with what one would expect from such an approach.

Another weakness for results presented in Table 1: although the data sets are more comprehensive, there are no statistical tests. For performance results reported on accuracy and ROC, which are the averages of 10 runs, each using a different random seed. From the table, it seems that MIDAS are slightly better, but these results are not statistically validated, eg., using t-test.

Questions for authors

Q1: What do you mean by "graph free"? and why it is important for at the time of inferencing?

Q2: Why do you need a partition into structural regions in a space that remains accessible without the graph? How do you decide the configuration, eg., the number of regions the parameter K? And what would be the implication of that?

Q3: How do you ensure your data sets are suitably chosen for answering your RQs in the experimental section, especially to determine teacher with regional structural context would be more effective than those without regional information? This is unclear to me.

Reviewer_9iQLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MIDAS, a multi-teacher knowledge distillation framework that transfers knowledge from multiple graph neural networks (GNNs) to a graph-free MLP student. The method combines feature-structure alignment, density-based clustering, region-based teacher assignment, and a trunk-adapter architecture with two-phase training. Experiments on seven benchmark datasets evaluate predictive performance, calibration, inference efficiency, and component contributions. However, inconsistencies in the methodological descriptions, insufficient evidence for several key claims, and substantial writing and presentation problems weaken the paper’s technical credibility and reproducibility.

Critical strengths
  • Relevant research problem. The paper addresses the limitations of single-teacher GNN-to-MLP distillation in structurally heterogeneous graphs.

  • Coherent methodological motivation. Combining structure-aware representations with region-specific teacher assignment is a reasonable direction for heterogeneous knowledge transfer.

  • Broad empirical evaluation. The experiments cover seven datasets and multiple baselines, with additional analyses of model components, calibration, and inference efficiency.

Critical weaknesses
  • Substantial writing and presentation problems. The manuscript is difficult to read smoothly due to awkward sentence construction, abrupt transitions, redundant statements, and repeated wording. For example, Section 5.5 introduces Figure 2 in consecutive sentences to make essentially the same point. Citation formatting errors are widespread, including duplicated author names, missing publication years, and inconsistent citation styles. There are also grammatical errors, punctuation problems, and unresolved cross-references in the appendix. Collectively, these issues substantially impair readability and make it harder to assess the technical contributions reliably.

  • Inconsistencies between the methodological description and the reported implementation. The structural descriptor is described in Section 4.1 as being learned from degree-aware random walks and Skip-Gram, whereas Algorithm 1 refers to Struc2Vec. Appendix A.3 also bases its complexity analysis on Struc2Vec, while Appendix A.15 explicitly distinguishes the proposed procedure from standard Struc2Vec. In addition, the training stages, loss terms, and node sets used for clustering are not consistently described across the main text and the algorithm. These discrepancies leave the actual implementation unclear and raise concerns about reproducibility.

  • Overstated theoretical and empirical claims. The theoretical analysis establishes limited properties of cluster-wise teacher selection but does not sufficiently justify the broader claim of optimality for the complete two-phase training strategy. Similarly, the claim of consistently outperforming all baselines is not supported across all datasets and evaluation settings. The reported inference speedup also depends on the comparison target: the approximately 20× speedup is primarily achieved relative to sequential multi-teacher inference, rather than a single teacher GNN. These claims should be more carefully qualified and aligned with the reported results.

Questions for authors

Check weaknesses above.

Reviewer_uL8mOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on GNN distillation, particularly the challenge that most distillation methods use a single teacher, despite evidence that GCN, GAT, and SAGE perform well on different local structures. Based on this observation, the authors devise a method for multi-teacher distillation.

Critical strengths
  1. The motivation is clear: some pretrained teachers might perform better on certain local structures.
  2. The overall presentation is clear, except for some typos.
Critical weaknesses
  1. The motivation for using multiple teachers could be better presented. For Tables 2 and 3, did you report mean accuracy scores or results from a single seed? Also, does the claim in lines 051–053 that “Pei et al. (2020) showed that GCN and GAT performance varies systematically with local homophily and node degree, suggesting that GNN performance is non-uniformly distributed across structurally distinct graph regions” still hold for more recent and advanced models such as BernNet?
  2. Does f_feat, trained on observed graph structures and features, generalize to different x_v values?
  3. Why use DBSCAN for partitioning when there are many alternative ways to cluster Z^{feat}?
  4. Within a cluster, are there still conflicts among different pretrained teachers, as in your motivation for clustering rather than using a single teacher? Similarly, might the same problem persist within each cluster? Do you assume that one teacher is assigned to each cluster, or do you have theoretical or empirical evidence that your clusters do not have conflicting issues?
  5. Please compare the total training and inference time costs with those of the baselines.
Questions for authors
  1. There are many formatting and typographical issues that make the paper difficult for me to understand:
    1. Check the reference format used in different contexts. For example, on line 053, there are two “Pei et al.” citations.
    2. There is a typo involving a colon on line 479.
    3. There is a typo involving a period on line 135.
    4. Lines 134–136: You first use vv for the pretrained teachers, which must also use graph structures. But then you say, “In the MIDAS setting, the model’s prediction for v depends on X_v alone,” which is confusing.
    5. Line 138: Recheck your definition of \pi and of \hat{\pi}.
    6. There are three dots below Table 3.
  2. In essence, what determines which teacher to use in a given cluster? Do you assume the trained model can generalize?
  3. Are there cases where using another teacher instead of the one assigned by your model yields better performance in a particular region? Have you observed any such cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.