- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for multi-teacher knowledge distillation for graph neural networks. The primary motivation is to leverage on the knowledge of best-matched teacher for a structured region, in order to learn better the local structural context, making "graph free" inferences. Multi-teacher knowledge distillation, ie., distillation from multiple teachers into a single student learning model is not a novel idea (literature shows quite a few works on this topic). The paper argue that most GNN distillation work uses only a single teacher, hence the paper attempts to make contributions in the context of GNN, plus using local regional structural contextual information as a chosen teacher.
Critical strengths
Multi-teacher knowledge distillation is an interesting topic becoming increasingly popular in the area of knowledge distillation. It makes sense that ingesting multiple teachers' knowledge into a student model is likely to produce models that are more robust and generalize better, in some way similar to the ensemble methods vs single machine learning models. This paper advocates such a paradigm, and attempts to improve feature alignment and structural embeddings, so that through clustering the best teacher with the best regional structural knowledge can be identified.
The paper builds on existing literature in this steadily growing area of using multi-teacher knowledge distillation, so it is reasonably safe to extend the approach into GNN with some variations.
Critical weaknesses
For the experimental section, the 5 research questions are too vague and generic? These need to be more sharply defined.
Since multi-teacher knowledge distillation into single student model has already been well explored in literature, this work at the best only represents small incremental steps forward. The results are not surprising either, consistent with what one would expect from such an approach.
Another weakness for results presented in Table 1: although the data sets are more comprehensive, there are no statistical tests. For performance results reported on accuracy and ROC, which are the averages of 10 runs, each using a different random seed. From the table, it seems that MIDAS are slightly better, but these results are not statistically validated, eg., using t-test.
Questions for authors
Q1: What do you mean by "graph free"? and why it is important for at the time of inferencing?
Q2: Why do you need a partition into structural regions in a space that remains accessible without the graph? How do you decide the configuration, eg., the number of regions the parameter K? And what would be the implication of that?
Q3: How do you ensure your data sets are suitably chosen for answering your RQs in the experimental section, especially to determine teacher with regional structural context would be more effective than those without regional information? This is unclear to me.