acceptodds
Under review as a conference paper at ICLR 2027

Separating Information Limits from Realization Gaps in Local Graph Distillation

Abstract

Graph distillation reports one teacher–student discrepancy, mixing unavailable teacher-relevant information with excess relative to the best predictor under the declared observation. We separate them using , where declares training-side information and is the radius- observation. This standard expected-KL decomposition yields the information term ; our contribution is to define and diagnose it under an explicit information set and compatible-world law. The observable pair does not determine that law, and one complete realization cannot uniformly recover a law-dependent without additional assumptions. We assign exact values to newly observed shells and characterize the information boundary through predictive Fisher curvature and conditional Green covariance. On Cora, PubMed, and Roman-empire, matched audits estimate all KL components under post-encoder graph-correlated completion laws. Conditioning on realized training-side logits changes radius-zero information shares from 12.5%, 17.9%, and 80.9% to 2.3%, 1.7%, and 23.7%, reversing the Roman-empire diagnosis. Predictive checks assess the adequacy of these model-conditional values, while qualitative conditioning-set diagnoses remain stable across retained laws. A contractive nonlinear teacher further tests the accounting beyond linear diffusion. The diagnosis is therefore relative to the declared information set and law, not intrinsic to one graph.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Distillation, Information Limit, Realization Gap, Conditional Mutual Information, Graph Locality

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_frtxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

When using a teacher model that can see the whole graph and a student model that has partial observability of the graph a loss in performance is often present. This paper essentially tries to separate the performance loss in two terms, one coming from the fact that some information is missing, and another one due to the student suboptimality.

Critical strengths
  1. The core idea is simple and straightforward to understand
Critical weaknesses
  1. The paper seems to be heavily AI generated. With excessive use of useless and bad / not defined jargon (compatible-world law, matched audits, information diagnostic framework, ...), AI captions remaining (Table 1: Quantities that must remain distinct in theory and experiment), copy-pasted text that lost original pointers (Ye et al. in line 120), and AI leftovers ("Code availability. The supplementary material includes the experiment, aggregation, plotting, and validation code; raw and aggregated results; run metadata; dependency versions; and the LaTeX source.").
  2. Some results do not seem novel at all. For example what is the novelty of Theorem 1?
Questions for authors
  1. Could you be more clear about the use of AI in this work? To me it does not seem it was limited to the tasks claimed in the AI use statement
  2. Where is the code claimed to be submitted? Would it be possible to see it?
  3. What is the real novelty of this paper?
Reviewer_Ek2HOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an information-theoretic framework for separating teacher–student distillation error into an irreducible information limit and a realization gap. It establishes identification limitations, characterizes the information value of graph neighborhoods through conditional mutual information and Fisher–Green analysis, and conducts empirical evaluations on several graph benchmarks.

Critical strengths
  • The paper is theoretically well-motivated and provides a systematic framework for analyzing graph distillation.

  • The paper includes empirical analyses across multiple graph benchmarks and examines the sensitivity to different conditioning settings.

Critical weaknesses
  1. The paper lacks convincing generative experiments that directly compare theoretical predictions with known ground-truth information limits. Most evaluations rely on estimates obtained under the same assumed distributions.

  2. The main finding that realization gaps dominate largely aligns with existing distillation research. It remains unclear what new actionable guidance the framework provides.

  3. Estimating information limits requires a user-specified completion law, and the predictive audits reveal substantial miscalibration. This raises concerns about the reliability and practical effectiveness of the proposed diagnosis.

Questions for authors

1.Can the authors provide generative experiments with known ground-truth information limits to quantitatively validate the theoretical predictions?

2.How does the proposed framework lead to better algorithmic decisions compared with existing distillation approaches?

3.Given the strong dependence on the completion law and the observed miscalibration, how can the proposed quantities be reliably estimated and used in practice?

Reviewer_sWcgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies why a student model fails to match a GNN teacher. Does it lack graph information, or does it fail to use the information available to it? It estimates these two sources of error by simulating changes to unseen nodes’ scores. The authors claim that larger neighborhoods and wider student networks can reduce different parts of the error.

Critical strengths

None

Critical weaknesses

The presentation is extremely bad. Very hard to follow even the research question and background.

About experiment setup. The student receives class scores produced by the teacher’s feature encoder, plus neighborhood summaries at larger radii. This studies how well an MLP can reproduce the teacher’s graph-propagation computation, NOT whether the student can learn a good representation from the original features. It differs from standard GNN-to-MLP distillation work, and the paper provided no justification for that.

The paper does not provide useful insights on how GNN distillation works/what it lacks beyond what is well-known; it also dose not give practical guidance and potential improvements.

Questions for authors
  1. what is the core research question of this paper?
  2. what is the core finding(s)/contribution(s) of the paper?
  3. why is the main experiment designed in the current form that deviates from standard GNN-to-MLP distillation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.