acceptodds
Withdrawn from ICLR 2027

DMSD: Validation-Guided Dual-Branch MLP Self-Distillation for Heterophilic Graph Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transferring graph structural knowledge to multilayer perceptrons (MLPs) offers a practical approach to efficient node classification. However, the effectiveness of topological supervision varies across graphs, particularly under heterophily, where local relations and broader propagation contexts provide different learning signals. Our analysis shows that neither route consistently dominates across the evaluated datasets, motivating an alternative to fixed topological supervision. We propose DMSD, a validation-guided, teacher-free dual-branch MLP self-distillation framework for heterophilic graph learning. DMSD independently trains local-relation and propagation-based candidates, each integrating topology-guided and attribute-guided self-distillation to learn complementary structural and feature representations without an external GNN teacher. Rather than uniformly averaging their predictions, DMSD computes a shared pair of fusion weights for each data split by applying a fixed-temperature sigmoid to the candidates’ validation accuracy difference. Their predictions are then combined in log-probability space, without using test labels or training an additional gating network. Experiments on ten node classification datasets show that DMSD achieves the highest mean accuracy among the compared methods on eight datasets, including all five heterophilic graphs, with an average gain of 3.97 percentage points over the strongest baseline on each heterophilic dataset. Ablation results further show that validation-guided soft fusion improves macro-averaged accuracy over uniform fusion by 2.98 percentage points.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural networks, knowledge distillation, self-distillation, graph representation learning, heterophilic graphs, node classification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pShPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

DMSD trains two MLP "candidates" for node classification. The local-relation candidate uses edges only as pair indices and regresses representation similarity to σ(cos(x_i, x_j)). The propagation candidate feeds Â^K X to an MLP and uses SSD's neighborhood-contrastive objective. Each candidate has a topology-guided (TSD) branch and an attribute-guided (ASD) branch with EMA targets and a cross-branch consistency loss, followed by a frozen-encoder linear probe. The two candidates' predictions are fused in log-probability space with a weight w_P = σ(Δ_val / 0.02), where Δ_val is their validation-accuracy gap. The paper reports the best mean accuracy on 8 of 10 datasets.

The local-relation objective is useful on the small WebKB graphs. However, the motivation for self-distillation is weak, and most of the gain comes from a generic validation-based fusion that is not specific to the method. The heterophilic evaluation relies on small, outdated benchmarks, the efficiency motivation is contradicted by the paper's own timing results, some ablation numbers are inconsistent with the fusion rule, and the code is claimed but not provided.

Critical strengths
  1. The paired validation/test analysis (Fig. 1a, Fig. C.1) shows clearly that the better topology route depends on the dataset.
  2. The local-relation candidate works well on WebKB. On its own it already beats all baselines on Cornell (54.55), Texas (70.54), and Wisconsin (71.14).
  3. The fusion is simple and training-free, and the protocol is documented in detail (fixed splits, seeds, LOGO selection of τ_g, edge-perturbation study).
Critical weaknesses
  1. The motivation and framing are unclear.
    • "Topological supervision" appears in the abstract before it is explained.
    • The paper does not explain why local relations vs. propagation is the right distinction for heterophily.
    • Self-distillation is not motivated. Why go teacher-free instead of using a better teacher? Why self-distillation rather than other self-supervised methods (e.g., GraphMAE, GRACE)? The "distillation" here is BYOL-style self-supervised pretraining plus linear probing, so no knowledge is transferred from a teacher, which makes the positioning against GLNN-type GNN-to-MLP distillation misleading.
    • The efficiency premise ("transferring structural knowledge to MLPs ... efficient node classification") is contradicted by Table 2. DMSD has the highest inference latency (18.85 ms vs. 3.08 ms for GCN and 7.03 ms for SSD), because it runs two candidates and still needs Â^K X preprocessing.
  2. The fusion is a generic plug-in, but it accounts for most of the gain.
    • Eqs. 7–9 use only validation accuracy and logits. Any two models can be combined this way, so the fusion reads more like a Kaggle-style trick than a core contribution.
    • Always-Propagation (66.42) is on par with SSD (66.77). Validation-based hard selection adds +4.95, of which about 3.7 pp comes from Texas alone. Soft fusion adds only another +0.91 over hard selection.
    • With τ_g = 0.02, 78/100 splits in Fig. 1(a) have |Δ_val| > 6 pp (w > 0.95). On 8/10 datasets, hard selection picks the same route in every split. The "controller" therefore acts as a per-dataset hyperparameter, and the within-dataset correlation is only 0.35.
    • Baselines would gain from the same trick. For example, the per-dataset best of {VQGraph, GPR-GNN} averages 69.43 (test-oracle upper bound), above every single baseline.
    • Requested: apply the fusion to the baselines (e.g., GCN+MLP, SSD-local/SSD-propagation); report DMSD-L and DMSD-P without fusion in Table 1; compare against simply choosing the route by homophily estimated from training labels.
  3. The necessity of the dual-branch, dual-candidate design is not shown.
    • Why are topology and attributes complementary, and can a single branch suffice? Table B.3 selects TSD-only on Chameleon and ASD-only on Citeseer.
    • "w/o TSD" (i.e., ASD only) falls far below a plain MLP (Computers 42.93 vs. 79.64; Photo 41.97 vs. 85.66; Citeseer 54.77 vs. 58.10). The likely reason is that ASD's online and target encoders receive the same input without augmentation, so the objective is nearly trivial and collapse is prevented only by the margin regularizer.
    • The local-relation target σ(cos) ∈ [0.27, 0.73] is always positive, so it still pulls neighbors together. This is at odds with the claim that it does not assume adjacent nodes share labels.
    • The paper should show how different the two branches and candidates are (e.g., CKA, prediction disagreement, error overlap).
  4. The heterophilic evaluation is weak.
    • The main gains come from Texas and Wisconsin (183 and 251 nodes). Their validation sets have only about 18–25 nodes, so a single node shifts accuracy by 4–5.6 pp, and the standard deviations are 4–6 pp.
    • Chameleon and Squirrel contain duplicated nodes (Platonov et al., 2023, which the paper itself cites).
    • The paper should use the newer heterophilic benchmarks (roman-empire, amazon-ratings, minesweeper, tolokers, questions) and large non-homophilous graphs (arxiv-year, snap-patents, pokec).
    • Heterophily-specific baselines (LINKX, FAGCN, ACM-GCN, GloGNN) are missing, even though LINKX and FAGCN are cited.
    • DMSD is a two-model ensemble compared against single models.
    • On Texas, the propagation candidate collapses to 33.99, below a plain MLP (61.28). The fusion masks this failure instead of explaining it.
  5. Some ablation numbers are inconsistent with Eq. 9.
    • Per-split points in Fig. 1(a) show Δ_val of 31.5–39.0 pp on Citeseer and 19.2–28.0 pp on Cora. With τ_g = 0.02, this gives w_L ≤ 1.4e-7 and w_L ≤ 6.9e-5, respectively. The fused argmax should then equal the propagation candidate's, yet Table 3 reports Soft > Hard = Always-P (71.11 vs. 70.78; 82.21 vs. 81.71).
    • The mean Pubmed gap in Fig. 1/C.1 (≈4.1 pp) differs from Table 3 (2.93 pp), and Cora differs as well (22.8 vs. 22.05). This suggests the figure and the ablation come from different runs, despite the caption's "same ablation experiment" claim.
  6. Code is not available. The reproducibility statement says anonymized code, configs, split masks, and scripts are provided as supplementary material, but I could not find them.

Minor Weaknesses

  • The abstract is too long, and quoting exact gains (3.97, 2.98 pp) is unusual.
  • Fig. 1(b) only plots Eq. 8 over ±12 pp, while 46/100 splits fall outside that range. Plotting the actual per-split weights would show that most are saturated.
  • Fig. 2 should be enlarged to full width, with the margins cropped.
  • Table 2: put the ↑/↓ arrows after the metric names rather than on a separate row. The accuracy–efficiency trade-off is better shown in the main text as a scatter plot (x: latency, y: accuracy).
  • Table 3 is too detailed for an ablation. Reporting 2–4 representative datasets with the relative change vs. the full model would be clearer.
  • Validation reuse: the readout α and λ_cross are chosen by mean validation accuracy over 10 random splits, but validation nodes of one split are test nodes of others. The same validation set is also used for checkpoint selection and for Δ_val.
  • λ_cross is said to be selected from {0, 1}, but it equals 1 on all datasets. Pretraining is a single run (seed 1024) reused across splits, so the reported std reflects only split variance.

References

  • Platonov et al., "A critical look at the evaluation of GNNs under heterophily," ICLR 2023: new heterophilic benchmarks and the duplicate-node issue in Chameleon/Squirrel.
  • Lim et al., "Large scale learning on non-homophilous graphs" (LINKX), NeurIPS 2021: large non-homophilous benchmarks and baseline.
  • Hu et al., "Graph-MLP," 2021: neighbor-based losses for message-passing-free MLPs, closely related to the local-relation objective.
  • Hou et al., "GraphMAE," KDD 2022: an alternative self-supervised representation learner.
Questions for authors
  1. What is the gain when the same validation fusion is applied to the baselines? What are DMSD-L and DMSD-P alone in Table 1?
  2. Please report per-split Δ_val, w_P, and the number of test predictions changed by fusion on Citeseer and Cora, and explain why Soft > Hard there.
  3. Why self-distillation instead of a stronger teacher or another SSL objective? Can a single branch match the dual-branch design?
  4. Why does the propagation candidate collapse on Texas?
  5. How does DMSD perform on the Platonov et al. (2023) and LINKX benchmarks?
  6. Where is the code?
Reviewer_CZErOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DMSD, a teacher-free MLP self-distillation framework for node classification, motivated by the observation that different forms of structural supervision perform differently across graphs. It trains two candidates with local-relation and propagation-based supervision, each combining topology-guided and attribute-guided learning through self-distillation and cross-branch consistency. Their validation accuracy difference determines fixed-temperature sigmoid weights for fusing predictions in log-probability space, without an external GNN teacher or a trainable gating network. Experiments on ten datasets report the highest mean accuracy among the compared methods on eight datasets, including all five heterophilic graphs. Component and fusion ablations examine the contributions of the learning objectives and validation-guided weighting. Additional analyses cover fusion-temperature sensitivity and learning under random edge perturbations. Efficiency measurements quantify the trade-off between improved accuracy and increased inference cost from retaining both candidates.

Critical strengths
  1. DMSD combines complementary local-relation and propagation-based supervision using validation evidence, without requiring an external GNN teacher. Its improvements over fixed candidates and uniform fusion support the usefulness of adapting structural supervision to different graphs.
  2. The method achieves the highest mean accuracy among the compared methods on eight of ten datasets, including all five heterophilic graphs. Component and fusion ablations provide evidence that both the representation-learning objectives and validation-guided weighting contribute to these gains.
Critical weaknesses
  1. The main methodological addition is validation-guided fusion of two candidates, but its benefit over straightforward model selection is modest: soft fusion improves macro-averaged accuracy over validation-based hard selection by only 0.91 percentage points. Without paired significance tests or comparisons with directly validation-tuned fusion weights, the evidence does not sufficiently establish the necessity of the proposed sigmoid weighting.
  2. The deployment benefits remain unclear. The propagation candidate requires graph-derived features, and Table 2 reports higher inference latency for DMSD than every compared method. Although the paper acknowledges this trade-off, it does not clearly establish when propagation can be precomputed or how updates are handled, weakening the connection between the efficient MLP motivation and the final system.
Questions for authors
  1. Does the 0.91-percentage-point gain over validation-based hard selection remain consistent across independent pretraining seeds and statistically significant under paired evaluation? How does the proposed sigmoid weighting compare with selecting fusion weights directly on the validation set?
  2. Under what deployment conditions can the propagation features be precomputed and reused? For new nodes or changes to features or graph structure, what graph access and recomputation are required, and how do these costs affect the claimed MLP inference benefits?
Reviewer_mqjtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DMSD, a validation-guided, teacher-free MLP self-distillation framework for node classification on heterophilic graphs. The main idea is to independently train two candidates with different structural inductive biases: a local-relation candidate that preserves feature-derived similarities between connected nodes, and a propagation-based candidate that learns from precomputed multi-hop graph features. Each candidate incorporates TSD, ASD, and cross-branch consistency.

Critical strengths
  1. Relevant problem. The paper addresses the practical challenge of transferring graph structural information into MLP-based predictors without relying on separately pretrained GNN teachers.

  2. Simple and interpretable fusion mechanism. The validation-guided controller is straightforward to implement. It provides an interpretable way to favor the candidate with higher validation accuracy while retaining contributions from both candidates when their performances are similar.

Critical weaknesses

1. The motivation is unclear.

The paper does not adequately justify two core design choices:

  • Why are two separately trained candidates necessary, rather than a single model that adaptively combines raw and propagated features?
  • Why is self-distillation necessary?

The main empirical motivation is that the better-performing route varies across datasets. However, this observation primarily motivates route selection rather than the proposed architecture.

2. The rationale for MLP-to-MLP distillation is unclear.

In conventional GNN-to-MLP distillation, a GNN teacher transfers structural knowledge to an MLP student, enabling graph-independent inference. In DMSD, both teacher and student are MLPs, and the teacher is simply an EMA copy of the student. Therefore, its specific benefit here is unclear. In particular, the ASD online and target encoders receive identical inputs without explicit augmentation. It remains unclear what additional information the teacher provides beyond enforcing representation consistency.

Cross-branch alignment may transfer structural information from TSD to ASD. However, the propagated-feature branch is still used in the final readout on nine of the ten datasets. Therefore, the graph-independent ASD encoder is generally not deployed on its own.

3. The necessity of two branches within the local candidate is unclear.

In the local-relation candidate, TSD and ASD receive exactly the same input features (XX) and use the same MLP architecture. They differ primarily in their training objectives, while the cross-branch consistency loss explicitly encourages their representations to align. It is unclear what additional information the second branch captures and why a single encoder trained with both objectives would be insufficient.

Moreover, removing TSD results in substantial performance degradation on Computers and Photo, even falling below a standard MLP. This raises concerns about the effectiveness of the remaining attribute-guided representation, although further analysis would be needed to establish whether representation collapse occurs.

4. The methodological novelty is limited.

The core contribution largely reduces to validation-based model selection and fusion, while its main components have clear precedents:

  • Combining raw and propagated features has been extensively studied. H2GCN separates ego and neighbor representations; GPR-GNN learns weights across propagation orders, including zero-hop features; LINKX processes attributes and structure separately; ACM-GCN adaptively combines different filtering channels. Node-MoE further explore node-level expert selection. HarmonyGNNs uses attention to adaptively balance raw and propagated features.
  • Applying an MLP to precomputed A^KX\hat A^KX follows the established decoupled propagation paradigm of SGC/SIGN.
  • Validation-accuracy-based ensembling is a standard technique.

5. The baseline comparisons are insufficient and outdated.

For a paper focused on heterophilic graph learning, several representative methods are missing:

  • Classical heterophily-oriented methods, including H2GCN, LINKX, and GloGNN. LINKX and FAGCN are discussed in the related work but are not evaluated.
  • Recent approaches such as HarmonyGNNs, which also employs an EMA teacher–student self-supervised framework to address homophily and heterophily.
  • Node-adaptive mixture-of-experts approaches, such as Node-MoE.

6. The heterophilic benchmarks have known data leakage problems.

All five heterophilic datasets come from the traditional WebKB and WikipediaNetwork collections. Platonov et al. (ICLR 2023), A Critical Look at the Evaluation of GNNs Under Heterophily, identified serious issues with these benchmarks, particularly duplicate nodes in Chameleon and Squirrel that can cause train–test leakage.

The manuscript uses the Geom-GCN-preprocessed versions of these datasets but does not describe any deduplication procedure. It is therefore unclear whether the reported improvements are affected by these known artifacts. Generating new random splits does not eliminate duplicate-node leakage.

7. The remaining heterophilic benchmarks are too small, while modern benchmarks are missing.

Texas, Cornell, and Wisconsin contain only 183, 183, and 251 nodes, respectively. Such small datasets provide limited evidence of generalization to realistic heterophilic graphs.

More importantly, the paper evaluates none of the modern heterophily benchmarks introduced by Platonov et al., including Roman-Empire, Amazon-Ratings, Minesweeper, Tolokers, and Questions.

Given that heterophilic graph learning is the primary focus, this omission substantially weakens the experimental evidence. The authors should evaluate DMSD on these benchmarks and address the duplicate-node issues in Chameleon and Squirrel.

8. The inference efficiency motivation is not supported by the results.

The abstract and introduction motivate DMSD by the inference overhead of message passing in latency-sensitive applications. However:

  • The propagation candidate requires A^3X\hat A^3X whenever its TSD representation contributes to the readout.
  • For a new node or an updated graph, computing these features may require accessing the corresponding multi-hop neighborhood. Although features can be cached on a fixed graph, the method does not eliminate graph dependency.
  • The experiments are transductive and do not establish efficient inductive inference.

Therefore, while DMSD avoids learned message passing, it does not achieve the graph-independent, low-latency inference typically motivating GNN-to-MLP distillation.

9. The proposed method lacks an explicit mechanism for modeling heterophily.

The local candidate operates on raw node features without feature propagation, while the propagation candidate relies on a fixed propagation operator A^3\hat A^3. Unlike methods such as FAGCN, ACM-GCN, and GPR-GNN, DMSD does not explicitly learn signed relationships, high-pass components, or adaptive propagation weights to handle heterophilic interactions. Thus, DMSD appears to handle heterophily primarily by selecting between non-propagation and propagation-based representations.

Consistent with this interpretation, edge homophily alone does not explain the preferred route. Further analysis is needed to establish how the proposed objectives capture heterophilic structural patterns.

Questions for authors

Please refer to weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.