- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DMSD trains two MLP "candidates" for node classification. The local-relation candidate uses edges only as pair indices and regresses representation similarity to σ(cos(x_i, x_j)). The propagation candidate feeds Â^K X to an MLP and uses SSD's neighborhood-contrastive objective. Each candidate has a topology-guided (TSD) branch and an attribute-guided (ASD) branch with EMA targets and a cross-branch consistency loss, followed by a frozen-encoder linear probe. The two candidates' predictions are fused in log-probability space with a weight w_P = σ(Δ_val / 0.02), where Δ_val is their validation-accuracy gap. The paper reports the best mean accuracy on 8 of 10 datasets.
The local-relation objective is useful on the small WebKB graphs. However, the motivation for self-distillation is weak, and most of the gain comes from a generic validation-based fusion that is not specific to the method. The heterophilic evaluation relies on small, outdated benchmarks, the efficiency motivation is contradicted by the paper's own timing results, some ablation numbers are inconsistent with the fusion rule, and the code is claimed but not provided.
Critical strengths
- The paired validation/test analysis (Fig. 1a, Fig. C.1) shows clearly that the better topology route depends on the dataset.
- The local-relation candidate works well on WebKB. On its own it already beats all baselines on Cornell (54.55), Texas (70.54), and Wisconsin (71.14).
- The fusion is simple and training-free, and the protocol is documented in detail (fixed splits, seeds, LOGO selection of τ_g, edge-perturbation study).
Critical weaknesses
- The motivation and framing are unclear.
- "Topological supervision" appears in the abstract before it is explained.
- The paper does not explain why local relations vs. propagation is the right distinction for heterophily.
- Self-distillation is not motivated. Why go teacher-free instead of using a better teacher? Why self-distillation rather than other self-supervised methods (e.g., GraphMAE, GRACE)? The "distillation" here is BYOL-style self-supervised pretraining plus linear probing, so no knowledge is transferred from a teacher, which makes the positioning against GLNN-type GNN-to-MLP distillation misleading.
- The efficiency premise ("transferring structural knowledge to MLPs ... efficient node classification") is contradicted by Table 2. DMSD has the highest inference latency (18.85 ms vs. 3.08 ms for GCN and 7.03 ms for SSD), because it runs two candidates and still needs Â^K X preprocessing.
- The fusion is a generic plug-in, but it accounts for most of the gain.
- Eqs. 7–9 use only validation accuracy and logits. Any two models can be combined this way, so the fusion reads more like a Kaggle-style trick than a core contribution.
- Always-Propagation (66.42) is on par with SSD (66.77). Validation-based hard selection adds +4.95, of which about 3.7 pp comes from Texas alone. Soft fusion adds only another +0.91 over hard selection.
- With τ_g = 0.02, 78/100 splits in Fig. 1(a) have |Δ_val| > 6 pp (w > 0.95). On 8/10 datasets, hard selection picks the same route in every split. The "controller" therefore acts as a per-dataset hyperparameter, and the within-dataset correlation is only 0.35.
- Baselines would gain from the same trick. For example, the per-dataset best of {VQGraph, GPR-GNN} averages 69.43 (test-oracle upper bound), above every single baseline.
- Requested: apply the fusion to the baselines (e.g., GCN+MLP, SSD-local/SSD-propagation); report DMSD-L and DMSD-P without fusion in Table 1; compare against simply choosing the route by homophily estimated from training labels.
- The necessity of the dual-branch, dual-candidate design is not shown.
- Why are topology and attributes complementary, and can a single branch suffice? Table B.3 selects TSD-only on Chameleon and ASD-only on Citeseer.
- "w/o TSD" (i.e., ASD only) falls far below a plain MLP (Computers 42.93 vs. 79.64; Photo 41.97 vs. 85.66; Citeseer 54.77 vs. 58.10). The likely reason is that ASD's online and target encoders receive the same input without augmentation, so the objective is nearly trivial and collapse is prevented only by the margin regularizer.
- The local-relation target σ(cos) ∈ [0.27, 0.73] is always positive, so it still pulls neighbors together. This is at odds with the claim that it does not assume adjacent nodes share labels.
- The paper should show how different the two branches and candidates are (e.g., CKA, prediction disagreement, error overlap).
- The heterophilic evaluation is weak.
- The main gains come from Texas and Wisconsin (183 and 251 nodes). Their validation sets have only about 18–25 nodes, so a single node shifts accuracy by 4–5.6 pp, and the standard deviations are 4–6 pp.
- Chameleon and Squirrel contain duplicated nodes (Platonov et al., 2023, which the paper itself cites).
- The paper should use the newer heterophilic benchmarks (roman-empire, amazon-ratings, minesweeper, tolokers, questions) and large non-homophilous graphs (arxiv-year, snap-patents, pokec).
- Heterophily-specific baselines (LINKX, FAGCN, ACM-GCN, GloGNN) are missing, even though LINKX and FAGCN are cited.
- DMSD is a two-model ensemble compared against single models.
- On Texas, the propagation candidate collapses to 33.99, below a plain MLP (61.28). The fusion masks this failure instead of explaining it.
- Some ablation numbers are inconsistent with Eq. 9.
- Per-split points in Fig. 1(a) show Δ_val of 31.5–39.0 pp on Citeseer and 19.2–28.0 pp on Cora. With τ_g = 0.02, this gives w_L ≤ 1.4e-7 and w_L ≤ 6.9e-5, respectively. The fused argmax should then equal the propagation candidate's, yet Table 3 reports Soft > Hard = Always-P (71.11 vs. 70.78; 82.21 vs. 81.71).
- The mean Pubmed gap in Fig. 1/C.1 (≈4.1 pp) differs from Table 3 (2.93 pp), and Cora differs as well (22.8 vs. 22.05). This suggests the figure and the ablation come from different runs, despite the caption's "same ablation experiment" claim.
- Code is not available. The reproducibility statement says anonymized code, configs, split masks, and scripts are provided as supplementary material, but I could not find them.
Minor Weaknesses
- The abstract is too long, and quoting exact gains (3.97, 2.98 pp) is unusual.
- Fig. 1(b) only plots Eq. 8 over ±12 pp, while 46/100 splits fall outside that range. Plotting the actual per-split weights would show that most are saturated.
- Fig. 2 should be enlarged to full width, with the margins cropped.
- Table 2: put the ↑/↓ arrows after the metric names rather than on a separate row. The accuracy–efficiency trade-off is better shown in the main text as a scatter plot (x: latency, y: accuracy).
- Table 3 is too detailed for an ablation. Reporting 2–4 representative datasets with the relative change vs. the full model would be clearer.
- Validation reuse: the readout α and λ_cross are chosen by mean validation accuracy over 10 random splits, but validation nodes of one split are test nodes of others. The same validation set is also used for checkpoint selection and for Δ_val.
- λ_cross is said to be selected from {0, 1}, but it equals 1 on all datasets. Pretraining is a single run (seed 1024) reused across splits, so the reported std reflects only split variance.
References
- Platonov et al., "A critical look at the evaluation of GNNs under heterophily," ICLR 2023: new heterophilic benchmarks and the duplicate-node issue in Chameleon/Squirrel.
- Lim et al., "Large scale learning on non-homophilous graphs" (LINKX), NeurIPS 2021: large non-homophilous benchmarks and baseline.
- Hu et al., "Graph-MLP," 2021: neighbor-based losses for message-passing-free MLPs, closely related to the local-relation objective.
- Hou et al., "GraphMAE," KDD 2022: an alternative self-supervised representation learner.
Questions for authors
- What is the gain when the same validation fusion is applied to the baselines? What are DMSD-L and DMSD-P alone in Table 1?
- Please report per-split Δ_val, w_P, and the number of test predictions changed by fusion on Citeseer and Cora, and explain why Soft > Hard there.
- Why self-distillation instead of a stronger teacher or another SSL objective? Can a single branch match the dual-branch design?
- Why does the propagation candidate collapse on Texas?
- How does DMSD perform on the Platonov et al. (2023) and LINKX benchmarks?
- Where is the code?