- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes IDG, a framework for the intrinsic evaluation and information-theoretic learning of dynamic graph neural networks. IDG introduces two diagnostic metrics: Structural Mutual Information (SMI), based on the spectral entropy of kernel representations, and Feature Mutual Information (FMI), which employs an NWJ-type variational estimator to quantify the dependency between node attributes and latent embeddings. These metrics are integrated into a "task-aware information bottleneck" objective function that incorporates a causal encoder. The authors evaluated IDG on link prediction, node classification, graph classification, and TGB node affinity ranking tasks, and conducted ablation experiments, correlation analysis, perturbation studies, distribution-shift experiments, temporal resolution tests, and runtime comparisons.
Critical strengths
- This paper explores an important practical problem: the lack of label-free diagnostic methods in the field of Dynamic Graph Embedding, which is a real limitation in the current evaluation of Dynamic Graph Neural Networks (DyGNN). The structural/feature decomposition approach proposed in the paper offers a sound framework for disentangling and examining this problem.
- The empirical verification in this paper is relatively comprehensive. This study covers 5 link prediction datasets, 4 node classification datasets, and 6 graph classification datasets, and incorporates the official TGB ranking results. In addition, it includes multi-random seed experiments, ablation studies, intrinsic control experiments, perturbation analysis, distribution shift experiments, and runtime evaluation. The availability of the code and implementation details further enhances the reproducibility of the research.
Critical weaknesses
- The paper describes FMI as mutual information; however, the appendix points out that for deterministic encoders with distinct empirical pairs, the empirical mutual information can saturate at a sample-size-dependent value, regardless of the encoder used. Therefore, an arbitrary injective encoder may still obtain a high empirical FMI score even if it cannot retain task-related information. Although the bounded critic may alleviate this problem to some extent in practical applications, the FMI score remains highly sensitive to the discriminator capacity, optimization process, kernel geometry, and sample size. While the correlation results among the 11 models presented in the paper provide encouraging evidence, they do not prove that SMI or FMI can reliably identify better models, nor do they prove that these metrics can distinguish between "useful representations" and "random, untrained, or task-irrelevant representations." Furthermore, "task information decomposition" itself is a standard identity, and this alone cannot establish a connection between SMI/FMI and downstream task performance.
- The appendix of this paper mentions that a 70:15:15 data partition ratio was used when performing link prediction and node prediction tasks on non-TGB datasets, but the paper does not clarify whether this partition is based on time order or random sampling. This distinction is critical, given the paper’s focus on issues such as "causal prefixes" and "future information leakage." Additionally, for graph-classification experiments, the authors utilized an experimental setup involving 100 bootstrap repetitions with an 80:10:10data split. Yet, they compare these results against kernel-based methods from the literature that were evaluated using repeated ten-fold cross-validation. Because these resampling schemes are inconsistent, it cannot be concluded that the proposed method is definitively superior to the others. Furthermore, the paper should specify the resources (budget) allocated for hyperparameter tuning for each baseline method, as well as the extent to which these baselines have been optimized.
Questions for authors
- For deterministic encoders, how do the authors ensure that the SMI/FMI metric reflects "the task-relevant information retained" rather than being affected solely by embedding dimension, sample size, or injectivity? Please provide evaluation scores for the random encoder (or untrained encoder) and the random latent space projection, and demonstrate that the proposed metrics score lower than the trained model for these cases.
- Could the authors provide a model selection experiment without labels? For example, when downstream task labels are unavailable, SMI/FMI can be used to filter from multiple candidate checkpoints or architectures and report their performance on downstream tasks, or to calculate the "selection regret" relative to the optimal candidate model. This directly supports the motivation in this paper regarding "achieving robust model selection".
- For link prediction and node prediction tasks on non-TGB datasets, is the 70:15:15 data partitioning method based on time order or on random partitioning based on events or snapshots? Please specify the exact splitting scheme for each dataset and confirm that no information from future interactions, labels, or samples that overlap in time or structure was used during training or validation.
- In graph classification tasks, why compare the results of 100 bootstrap repetitions with the results of kernel-based methods using repeated 10-fold cross-validation published in the literature? It is recommended to either re-evaluate these kernel-based baselines using the same resampling scheme or to present this comparison merely as a descriptive reference rather than as evidence of the model's superiority.
- How sensitive are the correlation results to the choice of "critic architecture," "kernel bandwidth," and the "number of paired records"? For the SMI/FMI correlation results, please provide uncertainty intervals or the distribution of outcomes obtained from multiple training runs of the critic, rather than relying solely on point estimates or ranges derived from a "leave-one-model-out" approach.
11 comments
Response to Reviewer QduF (2/7)
Supplementary experiment
-
To further address your concerns about dataset-specific exact splitting, we supplement a dataset splitting explanation. To avoid redundant presentation of experimental results, we provide the complete results in our response to your Question 3.
-
The original graph classification setting used 100 rounds of random sampling experiments, to evaluate the performance of the neural network under random initialization [1]. This does indeed raise concerns about the fairness of the experimental comparison. Therefore, we conducted a supplementary experiment on graph classification based on a unified standard. The seven kernel methods are evaluated via repeated 10-fold cross-validation [1,2]. Appendix F.1 documents that IDG ten-fold cross-validation resampling scheme to achieve fairness. To avoid redundant statements, we present full results in our response to your Question 4.
-
To further address your concerns about unified comparison conditions, we add documentation of baseline configuration sources and hyperparameter settings. The table QduF-1 lists our practically adopted configurations. After truncation, we do not conduct additional hyperparameter tuning, which ensures that the baseline can be set according to the optimal configuration as described in their original paper, thereby enabling a more fair comparison. Relevant run logs are available in the anonymous repository to resolve your questions regarding baseline configurations.
Table QduF-1: Implementation, Configuration and Tuning Records for Baselines.
| Method | Parameter Details |
|---|---|
| Jodie | Default values from the original authors’ implementation. Dynamic representation dimension 128, 50 training epochs; Adam optimizer, lr 1e-3, weight decay 1e-5. The specific modifications for our event-role adaptation are separately documented in F.3.2. |
| DyRep | Configurations from the implementation library and author-provided running commands. Flags: --use_memory --memory_updater rnn --dyrep --use_destination_embedding_in_message; batch size 200, lr 1e-4, neighbor count 10. |
| DySAT | Default values from the original authors’ TensorFlow implementation. lr 1e-3, batch size 512, 200 training epochs, structural/temporal dropout of 0.1/0.5 respectively, weight decay 5e-4. Both structural and temporal layers adopt width 128. |
| DyGNN | Default values and example scripts from the original authors’ implementation. batch size 200, lr 1e-3, weight decay 1e-3, dropout 0, attention and propagation enabled, log time decay. Default negative sample count is 5. |
| TGN | Default values in the original training script and reproduction commands. batch size 200, lr 1e-4, 1 layer, 2 attention heads, neighbor count 10, dropout 0.1, 50 training epochs, patience 5. The TGN-attn setting additionally enables --use_memory. |
| PINT | Per-dataset recommended configurations from the original authors. 2 layers, 50 epochs, memory enabled; memory dimension 172 and neighbor count 10 for both Wikipedia/Reddit, with beta set to 1e-4/1e-5 respectively. batch size 200, Adam optimizer and lr 1e-4. |
| DyGMamba | Default values and per-dataset configurations from the original authors’ implementation. Default batch size 200, Adam optimizer, lr 1e-4, 2 layers, 100 epochs, patience 20. When load_best_configs is enabled, maximum sequence lengths for Wikipedia/Reddit/MOOC are 32/64/256, with dropout 0.1/0.1/0.2; |
| RAG4DyG | Three-stage training configurations from the original authors. Taking wikiv2 as an example: 2 layers, 6 heads, representation dimension 768; learning rates are 1e-4/1e-6/1e-4, batch sizes are 32/128/32; topK retrieval =7 in the generation stage. |
| DyCIL | Default values and dataset-override configurations from the original authors’ implementation. Base default hidden dimension 32, lr 0.01, weight decay 0.005, dropout 0, up to 1000 epochs, patience 50, causal_ratio=0.2. use_cfg=1 enables per-dataset overrides by default. |
| LAMP | Per-dataset selected configurations provided by the original authors. AS-733: hidden dimension 32, 3 layers, batch size 64; Bitcoin-alpha: hidden dimension 128, 1 layer, batch size 32. Both use Adam with lr 0.01 and weight decay 1e-4. |
Thank you again for your suggestions. We have organized and summarized the above content and supplement it in the corresponding parts of Appendix F.
reference
[1] Micheli, Alessio, and Domenico Tortorella. "Discrete-time dynamic graph echo state networks." Neurocomputing 496 (2022): 85-95.
[2] Oettershagen, Lutz, et al. "Temporal graph kernels for classifying dissemination processes." Proceedings of the 2020 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2020.
[3] Rossi, Emanuele, et al. "Temporal graph networks for deep learning on dynamic graphs." arXiv preprint arXiv:2006.10637 (2020).
Response to Reviewer QduF (3/7)
Q1: Association between SMI/FMI and Task-related Information
We sincerely thank you for your insightful suggestions. As in Section 3.2, Appendix A.7 and E.3 of the original text, and based on the derivations in Table 10, we conducted a series of analyses regarding the role of SMI and FMI. Following your advice, we further broaden our verification by supplementary experiments using randomly initialized, untrained encoders to validate the association between SMI/FMI and task-related information.
Detailed explanation
- We test the relationship between SMI/FMI and representation quality with unified observation records, fixed estimation budgets, and controls over dimension, norm and effective rank. In Table 10, SMI/FMI show substantially stronger performance correlation across this model suite.
- Section 3.2, Appendix A.7 and E.3 jointly define the Detailed meaning and evaluation resolution of our metrics. SMI and FMI quantify input-representation dependence for structure and attributes, respectively. Appendix A.7 specifies shared records and bounded critic configurations, while E.3 links population-level dependence measures to input recovery and task log risk.
Supplementary experiment
To further address your concerns regarding randomness and injectivity, we add the following representation control experiments. All representations are kept frozen; embeddings share the same dimension for a given dataset. The min-max normalized SMI/FMI are computed under identical observation records and evaluation budgets. Our controls include trained IDG, untrained encoders, independent random representations, Gaussian attribute projections, and orthogonal rotations.
Table QduF-2: Comparison of Random, Untrained, Transformed and Learned Representations
| Comparison Item | Statistic | Mooc | GitHub | Wikipedia | Tgbl-Wiki | |
|---|---|---|---|---|---|---|
| Trained IDG | SMI | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| FMI | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | |
| AUC | 0.806$\pm$0.004 | 0.842$\pm$0.006 | 0.789$\pm$0.005 | 0.886$\pm$0.004 | 0.955$\pm$0.002 | |
| AP | 0.816$\pm$0.004 | 0.803$\pm$0.007 | 0.797$\pm$0.004 | 0.913$\pm$0.003 | 0.961$\pm$0.002 | |
| Untrained IDG Encoder | SMI | 0.221 | 0.198 | 0.243 | 0.176 | 0.152 |
| FMI | 0.237 | 0.214 | 0.258 | 0.191 | 0.168 | |
| AUC | 0.534$\pm$0.016 | 0.552$\pm$0.017 | 0.521$\pm$0.015 | 0.570$\pm$0.016 | 0.604$\pm$0.013 | |
| AP | 0.526$\pm$0.017 | 0.541$\pm$0.018 | 0.513$\pm$0.016 | 0.582$\pm$0.014 | 0.615$\pm$0.012 | |
| Independent Random Representations | SMI | 0.094 | 0.076 | 0.108 | 0.063 | 0.051 |
| FMI | 0.112 | 0.089 | 0.121 | 0.074 | 0.062 | |
| AUC | 0.523$\pm$0.018 | 0.516$\pm$0.020 | 0.534$\pm$0.017 | 0.509$\pm$0.019 | 0.527$\pm$0.016 | |
| AP | 0.517$\pm$0.021 | 0.508$\pm$0.019 | 0.526$\pm$0.018 | 0.504$\pm$0.022 | 0.532$\pm$0.017 | |
| Gaussian Projection of Input Attributes | SMI | 0.473 | 0.441 | 0.496 | 0.418 | 0.392 |
| FMI | 0.488 | 0.459 | 0.511 | 0.435 | 0.407 | |
| AUC | 0.651$\pm$0.010 | 0.674$\pm$0.011 | 0.638$\pm$0.012 | 0.702$\pm$0.009 | 0.768$\pm$0.008 | |
| AP | 0.659$\pm$0.011 | 0.662$\pm$0.012 | 0.631$\pm$0.010 | 0.715$\pm$0.008 | 0.779$\pm$0.007 | |
| Gaussian Projection of Learned Representations | SMI | 0.726 | 0.708 | 0.739 | 0.687 | 0.664 |
| FMI | 0.738 | 0.721 | 0.751 | 0.702 | 0.677 | |
| AUC | 0.728$\pm$0.007 | 0.759$\pm$0.008 | 0.711$\pm$0.009 | 0.796$\pm$0.007 | 0.871$\pm$0.005 | |
| AP | 0.735$\pm$0.008 | 0.727$\pm$0.009 | 0.719$\pm$0.008 | 0.822$\pm$0.006 | 0.886$\pm$0.004 | |
| Orthogonal Rotation of Learned Representations | SMI | 0.994 | 0.992 | 0.995 | 0.991 | 0.993 |
| FMI | 0.993 | 0.991 | 0.994 | 0.990 | 0.992 | |
| AUC | 0.803$\pm$0.005 | 0.839$\pm$0.007 | 0.786$\pm$0.006 | 0.882$\pm$0.005 | 0.952$\pm$0.003 | |
| AP | 0.812$\pm$0.005 | 0.799$\pm$0.008 | 0.793$\pm$0.005 | 0.908$\pm$0.004 | 0.958$\pm$0.003 |
Obs. R1. SMI/FMI distinguish learned representations from random and untrained baselines and stay stable under orthogonal coordinate transformations, largely excluding injective mapping artifacts. Under shared observation records, identical embedding dimensions and fixed estimation budgets, random/untrained representations cannot reach high scores comparable to learned ones. Across control cases from attribute Gaussian projections, representation Gaussian projections to fully trained IDG, SMI/FMI and AUC/AP rise consistently, matching downstream performance trends. Orthogonally rotated representations consistent with the invariance expected for information-preserving coordinate transforms. Collectively, these results directly verify SMI/FMI’s ability to discriminate representation quality.
Thank you again for your question. We have integrated these supplementary experiments and add them to the corresponding part of Appendix G.6.
Response to Reviewer QduF (1/7)
Dear QduF,
We sincerely appreciate you devoting significant time and effort to providing valuable insights and constructive comments on our work. To address your concerns thoroughly, we conduct rigorous supplementary experiments and detailed Detailed derivations to further clarify the technical details, validate the robustness of our SMI/FMI and strengthen the academic rigor of our IDG.
W1: The Role of Mutual Information Metrics
We greatly appreciate your concern regarding the exact role of SMI/FMI. The relevance between SMI/FMI and downstream task is jointly validated by population-level derivations and controlled experiments. Our original manuscript have supported the relevant conclusions from: unified estimation setup, finite-sample error analysis, known mutual-information experiments, and metric-risk derivation. To better address your concerns, we further strengthen our the viewpoints during the rebuttal via detailed explanations and supplement experiments as empirical evidence.
Detailed explanation
Supplementary experiment
To further address your concerns, we add control experiments with random, untrained, and transformed representations. To avoid redundant presentation of experimental results, we present the complete findings in our responses to your Questions 1, 2 and 5.
Thank you again for your comments. We organize the above discussions and supplement them in the corresponding parts of the manuscript: controls over random, untrained, transformed representations and entropy terms are added to Appendix G.6; sensitivity analyses of critics, bandwidth and sampling on real graph datasets are added to H.7; direct model-selection experiments are added to H.2.
W2: Data sampling and parameter tuning fairness
Thank you for your valuable comment. In our original paper, we provided causal temporal inputs, data split ratios, and sources for graph classification results in Appendices G.1, F.1/F.4 and H.1, respectively. To avoid redundant presentation, we elaborate on dataset preprocessing protocols in response to your Question 3. Additionally, we provide detailed baseline configuration specifications within this response to further ensure fairness of experimental comparisons.
Detailed explanation