- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Model-to-Data (M2D) distillation, which transfers knowledge from complex GNN teachers into graph data rather than directly into student models. The method learns augmented node features and graph structures through teacher-guided distillation. The resulting graph can be reused to train lightweight models without further teacher access. Experiments investigate fairness transfer, predictive performance, and generalization across different GNN architectures.
Critical strengths
S1. Interesting research perspective. The paper introduces model-to-data distillation as an alternative to conventional knowledge distillation. Encoding teacher knowledge into reusable graph data is an interesting direction with potential applications.
S2. Broad evaluation of teacher models. The experiments consider fairness-aware GNNs, GATs, and Graph Transformers. This demonstrates the potential applicability of the framework to different teacher architectures and learning objectives.
S3. Cross-architecture transferability. The paper evaluates whether distilled graphs can benefit independently trained GCN, SGC, and MLP models. The results provide useful evidence that teacher knowledge can be transferred through graph data.
Critical weaknesses
W1. Limited novelty of the technical implementation.
Although model-to-data distillation provides an interesting perspective, the proposed implementation mainly combines existing graph augmentation and knowledge distillation techniques. Feature masking, graph structure learning, and KL-based distillation are well-established methods. The paper does not sufficiently distinguish its technical contributions from previous teacher-guided graph augmentation and graph condensation methods. A more detailed comparison with closely related methods is needed to establish the methodological novelty.
W2. Inconsistent improvements over standard knowledge distillation.
Table 1 shows that M2D does not consistently outperform conventional knowledge distillation. For example, on Cora with a GAT teacher, the standard distillation baseline achieves 79.27% accuracy, while the feature-based M2D variant achieves 82.52%. However, with a Graph Transformer teacher on Cornell, the best M2D result reaches only 58.79%, compared with 70.59% for the teacher. Furthermore, the adjacency-based variant frequently underperforms the feature-based variant. These results raise questions about the effectiveness of the proposed framework across different architectures.
W3. Limited evidence for model-agnostic knowledge transfer.
A central claim is that the distilled graph can transfer teacher knowledge to different downstream architectures. However, the main cross-architecture experiments in Tables 2 and 3 are limited to Cora and NBA. The evaluated downstream models are also restricted to GCN, SGC, and MLP. These experiments provide preliminary evidence but do not fully support the general model-agnostic claim. Additional evaluations across more datasets and substantially different architectures would strengthen this conclusion.
W4. Insufficient evaluation of computational efficiency and scalability.
The proposed method requires iterative optimization of node features and graph structure. The structure learner estimates pairwise relationships between nodes, which introduces quadratic complexity with respect to the number of nodes. Although the appendix discusses computational complexity, the paper lacks empirical comparisons of training time, memory consumption, and scalability against standard distillation methods. It remains unclear whether the benefits of reusing the augmented graph justify the additional computational cost.
W5. Limited validation of the theoretical claims.
The paper presents two theorems concerning fairness preservation and attention knowledge transfer. However, Theorem 1 depends on excess prediction losses that are not empirically quantified. Theorem 2 relies on local linearization and assumptions about the student solution. The experiments provide qualitative visualizations and correlations, but they do not directly evaluate the tightness of the theoretical bounds or the approximation errors. This weakens the connection between the theoretical analysis and the reported empirical findings.
Questions for authors
Q1. Novelty. What are the key technical differences between M2D and existing teacher-guided graph augmentation or graph condensation methods? Can the authors clarify which components are genuinely novel?
Q2. Knowledge Transfer. Table 1 shows that M2D does not consistently outperform standard knowledge distillation. Under what conditions does M2D provide clear advantages over conventional distillation?
Q3. Generalization. Tables 2 and 3 evaluate cross-architecture transfer mainly on Cora and NBA. Can the authors provide additional evidence that the distilled graphs generalize to substantially different student architectures and datasets?
Q4. Computational Efficiency. What are the training time and peak GPU memory costs of M2D compared with standard knowledge distillation? How does the method scale to large graphs given the quadratic complexity of pairwise structure learning?
Q5. Theoretical Validation. Can the authors quantify the approximation errors or assumptions involved in Theorems 1 and 2? How well do the theoretical predictions align with the empirical results?
2 comments
Rebuttal Follow up
We would like to follow up on our rebuttal and ask whether our responses have addressed your concerns. We would appreciate any additional feedback and would be happy to clarify any remaining questions. If you find our response satisfactory, we would be grateful if you could increase your score.
Response to reviewer bC4V
We thank the reviewer for recognizing M2D's contributions. We address all concerns below.
We agree that feature masking, structure learning, and KL-based distillation are established techniques. M2D's novelty lies in using the transformed graph as a reusable medium for transferring teacher knowledge. Existing methods combine graph augmentation and knowledge distillation [1–5], but none explicitly learns teacher-guided augmentation. Augmentation is generally performed independently of teacher predictions, and the transferred knowledge ultimately resides in the student model. We have clarified these distinctions in line 137.
[1] Knowledge distillation improves graph structure augmentation for graph neural networks. NeurIPS’22.
[2] T2-gnn: Graph neural networks for graphs with incomplete features and structure via teacher-student distillation. AAAI’23.
[3] Nosmog: Learning noise-robust and structure-aware mlps on graphs. ICLR’23.
[4] Graph augmentation guided federated knowledge distillation for multisite functional MRI analysis. Pattern Recognition.
[5] A cold-start aware group recommendation method based on an augmented graph and knowledge distillation. Applied Soft Computing.
The reviewer’s CORNELL example compares M2D with the teacher rather than KD. With a GT teacher, M2D achieves 58.79% accuracy versus 52.98% for KD. Similarly, on CORA, M2D achieves 82.52% versus 79.27% for KD (Table 1). These improvements demonstrate that M2D can transfer teacher knowledge more effectively than standard KD, even when a lightweight student does not match the teacher's performance. The effectiveness of feature- and adjacency-based distillation varies. Feature distillation often performs better for classification, joint transformations improve accuracy on CORNELL and TEXAS (Table 1), and structural transformations often improve fairness (Fig. 4).
Beyond CORA and NBA (Tables 2–3), the original Appendix Tables 8–10 evaluate independent GCN, SGC, and MLP across six additional datasets.
We further evaluate independently trained GAMLP [6] on graphs distilled using a FairGNN teacher and GCN student. GAMLP consistently improves teacher fidelity and reduces DP and EQOP across SIMULATION, GERMAN, and NBA, with accuracy reductions of up to 1.42 percentage points on SIMULATION and GERMAN, while improving accuracy on NBA.
Fairness and Accuracy of Independently Trained GAMLP
[6] Graph attention multi-layer perceptron. KDD’22.
We report runtime, peak GPU memory, and storage. M2D is more efficient than GT, FairVGNN, and NIFTY. We also demonstrate improved scalability without substantial accuracy loss through selective edge learning (see response to RzcJ (W1)).
Thm. 1 bounds the fairness difference between teacher and downstream models through distillation error and excess log losses based on an unknown optimal predictor. Although the unknown optimal predictor makes direct evaluation of bound tightness challenging, results in Sec. 4.3 and Appendix Tables 8–10 are consistent with this characterization.
Thm. 2 relates teacher attention and learned graph. It employs first-order approximation, an established technique for analyzing neural nets [7,8], while explicitly accounting for approximation residuals rather than assuming exact linearity. Fig. 6–7 support this relationship: feature-mask similarity is positively associated with teacher attention, while learned edge weights correlate with GAT attention on CORA. See Appendix for results on CORNELL and GT.
We have further clarified assumptions and experimental connections (lines 302, 312, and 422).
[7] Neural Tangent Kernel: Convergence and Generalization in Neural Networks. NeurIPS’18.
[8] Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. NeurIPS’19.