acceptodds
Under review as a conference paper at ICLR 2027

IDG: Intrinsic Assessment and Information-Theoretic Learning for Dynamic Graph Neural Networks

Abstract

Dynamic graph neural networks (DyGNNs) aim to transform evolving graph structures into informative low-dimensional representations to support various downstream tasks, such as link prediction and node classification. However, assessing the quality of latent embeddings remains difficult because existing methods mainly rely on downstream task performance. Such end-to-end assessment lacks interpretable metrics for directly evaluating the latent embeddings themselves. This reliance leads to repeated tuning trials, limited diagnostic insight, and unstable model selection across tasks and datasets. To address this issue, we propose **IDG**, which introduces **I**ntrinsic Assessment and Information-Theoretic Learning for **D**ynamic **G**raph Neural Networks. Specifically, IDG contains two key components: (i) structure mutual information (SMI), which uses spectral entropy to measure structural information shared by inputs and latent embeddings, and (ii) feature mutual information (FMI), which uses -divergence to capture the dependence between latent embeddings and node attribute dynamics. To further demonstrate the practical utility of these metrics, we integrate them into a representation learning objective to guide embedding optimization. Comprehensive experiments show positive empirical associations between the proposed metrics and downstream performance across diverse benchmarks and model architectures, providing interpretable clues about model efficacy, failure modes, and robustness patterns across settings under the evaluated protocols. The code is available at https://anonymous.4open.science/r/IDG_ICLR-924F.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural networks, representation learning, mutual information, assessment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QduFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IDG, a framework for the intrinsic evaluation and information-theoretic learning of dynamic graph neural networks. IDG introduces two diagnostic metrics: Structural Mutual Information (SMI), based on the spectral entropy of kernel representations, and Feature Mutual Information (FMI), which employs an NWJ-type variational estimator to quantify the dependency between node attributes and latent embeddings. These metrics are integrated into a "task-aware information bottleneck" objective function that incorporates a causal encoder. The authors evaluated IDG on link prediction, node classification, graph classification, and TGB node affinity ranking tasks, and conducted ablation experiments, correlation analysis, perturbation studies, distribution-shift experiments, temporal resolution tests, and runtime comparisons.

Critical strengths
  1. This paper explores an important practical problem: the lack of label-free diagnostic methods in the field of Dynamic Graph Embedding, which is a real limitation in the current evaluation of Dynamic Graph Neural Networks (DyGNN). The structural/feature decomposition approach proposed in the paper offers a sound framework for disentangling and examining this problem.
  2. The empirical verification in this paper is relatively comprehensive. This study covers 5 link prediction datasets, 4 node classification datasets, and 6 graph classification datasets, and incorporates the official TGB ranking results. In addition, it includes multi-random seed experiments, ablation studies, intrinsic control experiments, perturbation analysis, distribution shift experiments, and runtime evaluation. The availability of the code and implementation details further enhances the reproducibility of the research.
Critical weaknesses
  1. The paper describes FMI as mutual information; however, the appendix points out that for deterministic encoders with distinct empirical pairs, the empirical mutual information can saturate at a sample-size-dependent value, regardless of the encoder used. Therefore, an arbitrary injective encoder may still obtain a high empirical FMI score even if it cannot retain task-related information. Although the bounded critic may alleviate this problem to some extent in practical applications, the FMI score remains highly sensitive to the discriminator capacity, optimization process, kernel geometry, and sample size. While the correlation results among the 11 models presented in the paper provide encouraging evidence, they do not prove that SMI or FMI can reliably identify better models, nor do they prove that these metrics can distinguish between "useful representations" and "random, untrained, or task-irrelevant representations." Furthermore, "task information decomposition" itself is a standard identity, and this alone cannot establish a connection between SMI/FMI and downstream task performance.
  2. The appendix of this paper mentions that a 70:15:15 data partition ratio was used when performing link prediction and node prediction tasks on non-TGB datasets, but the paper does not clarify whether this partition is based on time order or random sampling. This distinction is critical, given the paper’s focus on issues such as "causal prefixes" and "future information leakage." Additionally, for graph-classification experiments, the authors utilized an experimental setup involving 100 bootstrap repetitions with an 80:10:10data split. Yet, they compare these results against kernel-based methods from the literature that were evaluated using repeated ten-fold cross-validation. Because these resampling schemes are inconsistent, it cannot be concluded that the proposed method is definitively superior to the others. Furthermore, the paper should specify the resources (budget) allocated for hyperparameter tuning for each baseline method, as well as the extent to which these baselines have been optimized.
Questions for authors
  1. For deterministic encoders, how do the authors ensure that the SMI/FMI metric reflects "the task-relevant information retained" rather than being affected solely by embedding dimension, sample size, or injectivity? Please provide evaluation scores for the random encoder (or untrained encoder) and the random latent space projection, and demonstrate that the proposed metrics score lower than the trained model for these cases.
  2. Could the authors provide a model selection experiment without labels? For example, when downstream task labels are unavailable, SMI/FMI can be used to filter from multiple candidate checkpoints or architectures and report their performance on downstream tasks, or to calculate the "selection regret" relative to the optimal candidate model. This directly supports the motivation in this paper regarding "achieving robust model selection".
  3. For link prediction and node prediction tasks on non-TGB datasets, is the 70:15:15 data partitioning method based on time order or on random partitioning based on events or snapshots? Please specify the exact splitting scheme for each dataset and confirm that no information from future interactions, labels, or samples that overlap in time or structure was used during training or validation.
  4. In graph classification tasks, why compare the results of 100 bootstrap repetitions with the results of kernel-based methods using repeated 10-fold cross-validation published in the literature? It is recommended to either re-evaluate these kernel-based baselines using the same resampling scheme or to present this comparison merely as a descriptive reference rather than as evidence of the model's superiority.
  5. How sensitive are the correlation results to the choice of "critic architecture," "kernel bandwidth," and the "number of paired records"? For the SMI/FMI correlation results, please provide uncertainty intervals or the distribution of outcomes obtained from multiple training runs of the critic, rather than relying solely on point estimates or ranges derived from a "leave-one-model-out" approach.
11 comments
AuthorsOpenReview ↗

Response to Reviewer QduF (1/7)

Dear QduF,

We sincerely appreciate you devoting significant time and effort to providing valuable insights and constructive comments on our work. To address your concerns thoroughly, we conduct rigorous supplementary experiments and detailed Detailed derivations to further clarify the technical details, validate the robustness of our SMI/FMI and strengthen the academic rigor of our IDG.


W1: The Role of Mutual Information Metrics

We greatly appreciate your concern regarding the exact role of SMI/FMI. The relevance between SMI/FMI and downstream task is jointly validated by population-level derivations and controlled experiments. Our original manuscript have supported the relevant conclusions from: unified estimation setup, finite-sample error analysis, known mutual-information experiments, and metric-risk derivation. To better address your concerns, we further strengthen our the viewpoints during the rebuttal via detailed explanations and supplement experiments as empirical evidence.

Detailed explanation

  1. Practical FMI uses held-out-set evaluation with bounded critics. Appendix A.7 distinguishes the exact empirical MI from the computable NWJ score. Frozen encoders share the same function class, optimization budget and sampling protocol, enabling comparisons of extractable attribute dependence under consistent evaluation resolution.
  2. Estimation error and fitting performance are analyzed correspondingly. Appendix D.2 decomposes error into approximation, estimation and optimization components. G.7 tests sample size and critic capacity on synthetic Gaussian data with analytical MI = 1.0. Appendix A.7 also presents the log⁡Nrec\log N_{\text{rec}} result for empirical MI under distinct pairs and uniform sampling, clarifying definitions across different computational quantities.
  3. The Detailed connection is formally established by Proposition E.2 in Appendix E.3. These relations hold for finitely supported inputs, finite objectives, the Markov chain Y→D→ZY\rightarrow D\rightarrow Z, and expectations over finite critics under a common evaluation distribution. Finite-precision records constitute a realizable finitely supported setting.
  4. Existing controls directly examine the impacts of representation geometry and model ensemble. Table 10 compares dimension, norm, effective rank and HSIC; after excluding IDG in G.10, positive correlations are still observed across the ten baselines.

Supplementary experiment

To further address your concerns, we add control experiments with random, untrained, and transformed representations. To avoid redundant presentation of experimental results, we present the complete findings in our responses to your Questions 1, 2 and 5.

Thank you again for your comments. We organize the above discussions and supplement them in the corresponding parts of the manuscript: controls over random, untrained, transformed representations and entropy terms are added to Appendix G.6; sensitivity analyses of critics, bandwidth and sampling on real graph datasets are added to H.7; direct model-selection experiments are added to H.2.

W2: Data sampling and parameter tuning fairness

Thank you for your valuable comment. In our original paper, we provided causal temporal inputs, data split ratios, and sources for graph classification results in Appendices G.1, F.1/F.4 and H.1, respectively. To avoid redundant presentation, we elaborate on dataset preprocessing protocols in response to your Question 3. Additionally, we provide detailed baseline configuration specifications within this response to further ensure fairness of experimental comparisons.

Detailed explanation

  1. Temporal prediction adopts time ordering and observed prefix inputs. Appendix G.1 sorts events chronologically and constructs sequential windows; F.4 defines next-snapshot prediction based on observed history; F.1 specifies the 70:15:15 split and best validation checkpoints for non-TGB tasks. We explicitly state the time-based data splitting in the revision.
  2. Time-based splitting matches the forward prediction scenario. Earlier observations are used for training, and subsequent ones for validation and testing, ensuring each target lies after its observable history. TGN adopts a 70%/15%/15% temporal split, serving as a direct precedent for the ratios and ordering of our non-TGB tasks [3]. We implement this paradigm via snapshot windows in G.1 and next-snapshot objectives in F.4.
  3. Configuration sources, post-truncation retuning and tuning budgets are documented. The "model-specific default configurations" in Appendix F.3.1 link to respective baseline‘s implementation. Supplementary records explicitly note for each of the ten baselines and IDG: author-recommended configurations, library defaults, or validation-set selected hyperparameters in this work, alongside config files.
AuthorsOpenReview ↗

Response to Reviewer QduF (2/7)

Supplementary experiment

  1. To further address your concerns about dataset-specific exact splitting, we supplement a dataset splitting explanation. To avoid redundant presentation of experimental results, we provide the complete results in our response to your Question 3.

  2. The original graph classification setting used 100 rounds of random sampling experiments, to evaluate the performance of the neural network under random initialization [1]. This does indeed raise concerns about the fairness of the experimental comparison. Therefore, we conducted a supplementary experiment on graph classification based on a unified standard. The seven kernel methods are evaluated via repeated 10-fold cross-validation [1,2]. Appendix F.1 documents that IDG ten-fold cross-validation resampling scheme to achieve fairness. To avoid redundant statements, we present full results in our response to your Question 4.

  3. To further address your concerns about unified comparison conditions, we add documentation of baseline configuration sources and hyperparameter settings. The table QduF-1 lists our practically adopted configurations. After truncation, we do not conduct additional hyperparameter tuning, which ensures that the baseline can be set according to the optimal configuration as described in their original paper, thereby enabling a more fair comparison. Relevant run logs are available in the anonymous repository to resolve your questions regarding baseline configurations.

Table QduF-1: Implementation, Configuration and Tuning Records for Baselines.

MethodParameter Details
JodieDefault values from the original authors’ implementation. Dynamic representation dimension 128, 50 training epochs; Adam optimizer, lr 1e-3, weight decay 1e-5. The specific modifications for our event-role adaptation are separately documented in F.3.2.
DyRepConfigurations from the implementation library and author-provided running commands. Flags: --use_memory --memory_updater rnn --dyrep --use_destination_embedding_in_message; batch size 200, lr 1e-4, neighbor count 10.
DySATDefault values from the original authors’ TensorFlow implementation. lr 1e-3, batch size 512, 200 training epochs, structural/temporal dropout of 0.1/0.5 respectively, weight decay 5e-4. Both structural and temporal layers adopt width 128.
DyGNNDefault values and example scripts from the original authors’ implementation. batch size 200, lr 1e-3, weight decay 1e-3, dropout 0, attention and propagation enabled, log time decay. Default negative sample count is 5.
TGNDefault values in the original training script and reproduction commands. batch size 200, lr 1e-4, 1 layer, 2 attention heads, neighbor count 10, dropout 0.1, 50 training epochs, patience 5. The TGN-attn setting additionally enables --use_memory.
PINTPer-dataset recommended configurations from the original authors. 2 layers, 50 epochs, memory enabled; memory dimension 172 and neighbor count 10 for both Wikipedia/Reddit, with beta set to 1e-4/1e-5 respectively. batch size 200, Adam optimizer and lr 1e-4.
DyGMambaDefault values and per-dataset configurations from the original authors’ implementation. Default batch size 200, Adam optimizer, lr 1e-4, 2 layers, 100 epochs, patience 20. When load_best_configs is enabled, maximum sequence lengths for Wikipedia/Reddit/MOOC are 32/64/256, with dropout 0.1/0.1/0.2;
RAG4DyGThree-stage training configurations from the original authors. Taking wikiv2 as an example: 2 layers, 6 heads, representation dimension 768; learning rates are 1e-4/1e-6/1e-4, batch sizes are 32/128/32; topK retrieval =7 in the generation stage.
DyCILDefault values and dataset-override configurations from the original authors’ implementation. Base default hidden dimension 32, lr 0.01, weight decay 0.005, dropout 0, up to 1000 epochs, patience 50, causal_ratio=0.2. use_cfg=1 enables per-dataset overrides by default.
LAMPPer-dataset selected configurations provided by the original authors. AS-733: hidden dimension 32, 3 layers, batch size 64; Bitcoin-alpha: hidden dimension 128, 1 layer, batch size 32. Both use Adam with lr 0.01 and weight decay 1e-4.

Thank you again for your suggestions. We have organized and summarized the above content and supplement it in the corresponding parts of Appendix F.

reference

[1] Micheli, Alessio, and Domenico Tortorella. "Discrete-time dynamic graph echo state networks." Neurocomputing 496 (2022): 85-95.

[2] Oettershagen, Lutz, et al. "Temporal graph kernels for classifying dissemination processes." Proceedings of the 2020 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2020.

[3] Rossi, Emanuele, et al. "Temporal graph networks for deep learning on dynamic graphs." arXiv preprint arXiv:2006.10637 (2020).

AuthorsOpenReview ↗

Response to Reviewer QduF (3/7)

Q1: Association between SMI/FMI and Task-related Information

We sincerely thank you for your insightful suggestions. As in Section 3.2, Appendix A.7 and E.3 of the original text, and based on the derivations in Table 10, we conducted a series of analyses regarding the role of SMI and FMI. Following your advice, we further broaden our verification by supplementary experiments using randomly initialized, untrained encoders to validate the association between SMI/FMI and task-related information.

Detailed explanation

  1. We test the relationship between SMI/FMI and representation quality with unified observation records, fixed estimation budgets, and controls over dimension, norm and effective rank. In Table 10, SMI/FMI show substantially stronger performance correlation across this model suite.
  2. Section 3.2, Appendix A.7 and E.3 jointly define the Detailed meaning and evaluation resolution of our metrics. SMI and FMI quantify input-representation dependence for structure and attributes, respectively. Appendix A.7 specifies shared records and bounded critic configurations, while E.3 links population-level dependence measures to input recovery and task log risk.

Supplementary experiment

To further address your concerns regarding randomness and injectivity, we add the following representation control experiments. All representations are kept frozen; embeddings share the same dimension for a given dataset. The min-max normalized SMI/FMI are computed under identical observation records and evaluation budgets. Our controls include trained IDG, untrained encoders, independent random representations, Gaussian attribute projections, and orthogonal rotations.

Table QduF-2: Comparison of Random, Untrained, Transformed and Learned Representations

Comparison ItemStatisticRedditMoocGitHubWikipediaTgbl-Wiki
Trained IDGSMI1.0001.0001.0001.0001.000
FMI1.0001.0001.0001.0001.000
AUC0.806$\pm$0.0040.842$\pm$0.0060.789$\pm$0.0050.886$\pm$0.0040.955$\pm$0.002
AP0.816$\pm$0.0040.803$\pm$0.0070.797$\pm$0.0040.913$\pm$0.0030.961$\pm$0.002
Untrained IDG EncoderSMI0.2210.1980.2430.1760.152
FMI0.2370.2140.2580.1910.168
AUC0.534$\pm$0.0160.552$\pm$0.0170.521$\pm$0.0150.570$\pm$0.0160.604$\pm$0.013
AP0.526$\pm$0.0170.541$\pm$0.0180.513$\pm$0.0160.582$\pm$0.0140.615$\pm$0.012
Independent Random RepresentationsSMI0.0940.0760.1080.0630.051
FMI0.1120.0890.1210.0740.062
AUC0.523$\pm$0.0180.516$\pm$0.0200.534$\pm$0.0170.509$\pm$0.0190.527$\pm$0.016
AP0.517$\pm$0.0210.508$\pm$0.0190.526$\pm$0.0180.504$\pm$0.0220.532$\pm$0.017
Gaussian Projection of Input AttributesSMI0.4730.4410.4960.4180.392
FMI0.4880.4590.5110.4350.407
AUC0.651$\pm$0.0100.674$\pm$0.0110.638$\pm$0.0120.702$\pm$0.0090.768$\pm$0.008
AP0.659$\pm$0.0110.662$\pm$0.0120.631$\pm$0.0100.715$\pm$0.0080.779$\pm$0.007
Gaussian Projection of Learned RepresentationsSMI0.7260.7080.7390.6870.664
FMI0.7380.7210.7510.7020.677
AUC0.728$\pm$0.0070.759$\pm$0.0080.711$\pm$0.0090.796$\pm$0.0070.871$\pm$0.005
AP0.735$\pm$0.0080.727$\pm$0.0090.719$\pm$0.0080.822$\pm$0.0060.886$\pm$0.004
Orthogonal Rotation of Learned RepresentationsSMI0.9940.9920.9950.9910.993
FMI0.9930.9910.9940.9900.992
AUC0.803$\pm$0.0050.839$\pm$0.0070.786$\pm$0.0060.882$\pm$0.0050.952$\pm$0.003
AP0.812$\pm$0.0050.799$\pm$0.0080.793$\pm$0.0050.908$\pm$0.0040.958$\pm$0.003

Obs. R1. SMI/FMI distinguish learned representations from random and untrained baselines and stay stable under orthogonal coordinate transformations, largely excluding injective mapping artifacts. Under shared observation records, identical embedding dimensions and fixed estimation budgets, random/untrained representations cannot reach high scores comparable to learned ones. Across control cases from attribute Gaussian projections, representation Gaussian projections to fully trained IDG, SMI/FMI and AUC/AP rise consistently, matching downstream performance trends. Orthogonally rotated representations consistent with the invariance expected for information-preserving coordinate transforms. Collectively, these results directly verify SMI/FMI’s ability to discriminate representation quality.

Thank you again for your question. We have integrated these supplementary experiments and add them to the corresponding part of Appendix G.6.

Reviewer_Xme9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem that existing dynamic graph neural networks mainly rely on the performance of downstream tasks to evaluate representation quality and lack intrinsic evaluation metrics. It proposes an information theory-based dynamic graph representation learning framework, IDG. The framework introduces two metrics, Structural Mutual Information (SMI) and Feature Mutual Information (FMI), which quantify the extent to which latent representations of a dynamic graph preserve information regarding graph structure and node attributes, respectively. By incorporating these into a graph information bottleneck, a joint optimization objective is formulated. Experimental results across tasks such as link prediction, node classification, and graph classification demonstrate that this approach enhances downstream task performance, and that the proposed evaluation metrics exhibit a strong correlation with predictive performance.

Critical strengths
  1. The research problem is of significant interest. This paper focuses on evaluating the intrinsic quality of latent representations in dynamic graphs, aiming to overcome the limitations of traditional approaches that rely primarily on downstream task metrics; thus, it holds clear research value.

  2. The methodological design is systematic. The paper develops mutual information-based evaluation metrics that account for both graph structure and node features, and integrates these into a unified representation learning framework grounded in information bottleneck theory. The overall technical approach is well-defined.

  3. The experimental evaluation is comprehensive, and the results are competitive. Through a series of experiments—including comparative studies, ablation studies, correlation analysis, perturbation analysis, and efficiency evaluations—conducted on multiple public datasets, the paper effectively validates the proposed method and metrics.

Critical weaknesses
  1. The contribution regarding interpretability is insufficient. This paper functions more as a method for enhancing and evaluating the quality of dynamic graph representations rather than as a truly interpretable dynamic graph learning approach. While SMI and FMI can measure the structural and feature information preserved within latent representations, they primarily offer a global-level quality assessment; they fail to explicitly reveal which specific subgraphs, critical nodes, key features, or temporal dependencies have been captured by the model.

  2. The technical innovation of the proposed method requires further clarification. SMI is largely based on existing spectral entropy theory, FMI employs established variational mutual information estimation techniques, and the optimization component draws upon the graph information bottleneck framework. The overall innovation lies primarily in the combination of these existing techniques and their application to the context of dynamic graphs. The paper should more clearly articulate its fundamental differences from existing dynamic graph information bottleneck methods (such as the Dynamic Graph Information Bottleneck).

Questions for authors

Please see the weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Xme9 (1/3)

Dear Xme9,

We sincerely appreciate your positive assessment of our work and are grateful for your suggestions regarding contribution statement, which indeed highlight an aspect where our paper can be improved. Your feedback is highly valuable and has been instrumental in helping us further refine and strengthen the presentation of our work.


W1: Contribution on Interpretability

Thank you for your comment. We agree that it is necessary to clearly distinguish source information diagnosis from specific part of a certain graph. Our paper claims the former: by perturbing graph structure and node attributes separately, we observe the corresponding changes in SMI/FMI and task performance. SMI/FMI can identify whether representation degradation arises primarily from the loss of structural or attribute information, and can even quantify the correlation between such feature variations and downstream task performance. Specifically,

  1. We establish structure/attribute diagnosis through the correspondence of metric definition, input intervention and task response. Section 3.2 defines SMI/FMI from dependencies between input structure/attributes and latent representations. Table 21 in Appendix G.11 performs degree-preserving rewiring (R) and attribute shuffling (P) under frozen encoders, identical data, fixed evaluation budget and prediction target, associating metric responses with information categories.
  2. For example, on GitHub (Table 21, G.11), 20% degree-preserving rewiring reduces SMI/FMI by 49.70%/8.50% and AUC/AP by 0.0492/0.0753; 20% attribute shuffling reduces SMI/FMI by 1.70%/17.10% and AUC/AP by 0.0020/0.0115 (SMI/FMI: relative changes; AUC/AP: paired absolute changes). Across five datasets, structural rewiring primarily modifies SMI and attribute shuffling mainly affects FMI, with responses varying monotonically with intervention intensity. These results enable diagnosis of which category of information is impaired and are not equivalent to faithful node-level attribution. Tables 30-31 in Appendix H.5 use a separate node-feature shuffling protocol (10% rewiring, 30% node-feature shuffling) and observe consistent selective responses. On GitHub, structural perturbation drops SMI/FMI by 31.8%/4.8%, while node-feature shuffling yields a 2.8%/25.4% reduction.
  3. The node-window pairing interface in Appendix A.7 defines the observation unit for diagnosis. Each record includes node identity, observation cutoff, structural descriptor and attribute history, supporting controlled local variations over paired records. Still, we do not extend this interface to validated node-level causal attribution, limiting our interpretability granularity to diagnosis over structural and attribute information categories.

Thank you again for your suggestions. We hope our reply can address your concern.

AuthorsOpenReview ↗

Response to Reviewer Xme9 (2/3)

W2: Technical contributions and the differences from existing work

Thank you for your valuable comments. Our innovation does not lie in re-proposing spectral entropy or the NWJ estimator, but in assembling them into a unified framework for evaluation and learning of dynamic graph representations. As noted in your summary of key strengths, we develop mutual-information-based evaluation metrics that account for both graph structure and node attributes. Specifically, we disentangle structural and attribute information via a shared pairing interface and establish theoretical connections between information recovery and task risk, yielding clear research contributions. In detail:

  1. Cross-architecture paired evaluation. SMI/FMI pairs inputs with frozen representations on identical node-time window records, enabling comparisons across dynamic graph encoders with varying output dimensions without relying on internal channels or gating mechanisms of any specific model.

  2. Unified input representation interface. Sections 3.2.1 and D.1 pair normalized observed neighborhoods and log-degree coordinates with representations from the same node and window to compute SMI via joint and marginal kernel spectra; FMI quantifies dependence between attribute records and representations. Appendix A.7 applies this interface to all frozen encoders and accommodates varied output dimensions. In Appendix G.6 (Table 10), the average Pearson R2R^2 between SMI/FMI and AUC is 0.840/0.855. Even after excluding IDG from the ten baselines in G.10, the values remain 0.848/0.832. This demonstrates the correlation is not driven solely by IDG and supports general applicability.

  3. Distinct goals from existing information-bottleneck works. DGIB pursues robust representation learning via Minimal-Sufficient-Consensual conditions and minimal/sufficient, prediction-consistent channels [1]; IDG incorporates structural and attribute dependence rewards within its task-aware bottleneck objective in Eq. (10), while SMI/FMI are computed independently on frozen representations after training. The distinction is not whether information bottleneck principles are adopted, but differences in optimization objectives, input-representation interfaces, and intended outputs:

    Table Xme9-1: Modeling-level Comparison with DGIB.

    DimensionDGIB [1]IDG
    MotivationLearn robust representations that are minimal, sufficient and prediction-consistentPreserve and diagnose structural/attribute dependence based on a task-aware bottleneck
    Training SignalsMinimal/sufficient channels and prediction-consistency channelsSupervised prediction, compression term, and SMI+FMI rewards
    Evaluation InterfaceInternal representation-learning channels of the methodShared pairing interface over node-time-window inputs and frozen representations
    Output UsageRobust representation learningCross-architecture unsupervised diagnosis, also usable as training rewards
  4. We establish the connections between our evaluation quantities, input recovery and task log risk. Under a common evaluation distribution, Equations (45)-(47) in Appendix E.3 link the aggregate SMI/FMI functionals, conditional input entropy and optimal task log risk via local kernel state construction and the NWJ lower bound. The theory characterizes the roles of information fidelity and task supervision. Table 21 verifies selective intervention responses, and Figure 5 examines the training contribution of information rewards, forming mutually consistent evidence.

  5. Ablation studies validate the learning effect of the reward terms. All variants in H.3 retain identical encoder, decoder, prediction loss and data protocol. Removing both SMI and FMI rewards (λ=0\lambda=0) reduces AUC by 0.046-0.169 and AP by 0.052-0.228 across five datasets. Removing only SMI yields the largest AUC/AP drop on GitHub (0.122/0.193), while removing only FMI incurs the biggest degradation on Wikipedia (0.134/0.147). Therefore, our empirical claim is that the two reward terms offer complementary benefits across datasets, rather than introducing novel spectral entropy or NWJ base estimators.

In summary, the technical contributions of IDG lie in: a unified cross-architecture input-representation evaluation interface, diagnostic decomposition along structural and attribute dimensions, and integrating this decomposition into a task-aware learning objective. The interface, theoretical connections, selective interventions and objective ablations each provide supporting evidence.

Thank you again for your suggestions. We have integrated the above discussions and add the comparison between IDG and existing information-bottleneck works for dynamic graphs to Appendix B.3.

reference

[1] Yuan, Haonan, et al. "Dynamic graph information bottleneck." Proceedings of the ACM Web Conference 2024. 2024.

AuthorsOpenReview ↗

Response to Reviewer Xme9 (3/3)

Thank you again for your recognition of our work and for providing such thorough and insightful feedback. In the revised submission, we have carefully addressed all your questions and suggestions, with relevant changes clearly highlighted in yellow. We welcome any further comments, your feedback is crucial for refining our work.

Best wishes!

Reviewer_LKPHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes IDG, an end-to-end dynamic graph neural network framework. The model consists of an encoder and a predictor, the contribution lies in two intrinsic assessment metrics defined on that encoder, and in how they are incorporated into a representation-learning objective. SMI is a structural mutual information estimated with a Gaussian-kernel Gram construction and von Neumann (spectral) entropy over a Hadamard joint operator, measuring information shared between a node's structural descriptor (degree-normalized neighbor row of the symmetrized adjacency concatenated with a log-degree coordinate) and its latent embedding

Critical strengths
  1. The estimator is stress-tested against ground truth rather than assumed. Tables 7–8 evaluate NWJ on a correlated-Gaussian construction with analytic MI (=1.0): test FMI rises monotonically with sample count (0.014/0.365/0.569/0.625). App. D.2 gives a three-term error bound, and App. A.7 states the failure mode plainly (under uniform sampling, the empirical MI of any injective encoder equals log N_rec). This is the only place SMI/FMI are tested against ground truth, and it is why I give a weak rather than a clear rejection.

  2. A controlled perturbation test separates the two metrics. Table 14 perturbs the two information sources separately: at 20% edge rewiring, SMI drops 35.6–49.7% while FMI drops only 5.9–8.5%; at 20% attribute permutation, FMI drops 17.1–28.8% while SMI moves only 1.7–2.5%; every response is monotone in perturbation strength. This is the only result in which the two quantities move separately.

Critical weaknesses
  1. The two metrics are not shown to provide independent assessment information. SMI and FMI rank models almost identically (Spearman correlation 0.93–0.97 on all five datasets, from Table 16). The correlation study is descriptive: n=11, three non-standard datasets, cohort-defined min-max normalization, and leave-one-out sensitivity ranges rather than confidence intervals. Excluding IDG, the ten baselines retain R²=0.848/0.832 (App. G.10), which I credit. However, the SMI construction contains a directly rewarded entropy term while the Table 6 controls omit the natural nulls; FMI saturates at log N_rec for injective encoders, and on datasets whose features aggregate event-level attributes it may partly reflect that aggregation, which is not separated from attribute dynamics. Two distinct constructs are shown, but two complementary assessment axes are not—and that is the claim that matters for model selection.

  2. The empirical comparison cannot support the headline claims. Wikipedia and Reddit are truncated to their first 10,000 interactions and GitHub is replaced by a non-standard 19,664-interaction collection, so the numbers are not comparable with published full-benchmark results, and no link-prediction baseline is checked against its published behavior (even on full-size Tgbl-Wiki with its official split). The TGB node-affinity results are a partial counterweight but cover a different task. F.1/F.3.2 specify splits and replace native classification heads with a shared MLP readout and reformulate Jodie for unipartite event streams, but neither change is validated. The baseline configuration is underspecified ("model-specific default encoder configurations")—it is unclear whether these are the original recommended settings or generic defaults, or whether they were re-tuned after truncation; since IDG's λ, β, knbr are chosen via a cross-task sensitivity study, the baselines may be untuned by comparison.

Questions for authors
  1. On benchmark truncation. Mooc and Tgbl-Wiki are already full-size, so why are Wikipedia/Reddit cut to their first 10,000 interactions and GitHub replaced by a non-standard 19,664-interaction collection? Can you anchor at least one link-prediction baseline to its published number on a full-size dataset (ideally Tgbl-Wiki with its official split) to demonstrate the reimplementation is faithful? If the full instances don't fit, please state the projected memory/compute explicitly and justify the GitHub choice.

  2. On baseline treatment. What exactly were the ten baselines' configurations—original recommended settings or generic defaults—and were they re-tuned after truncation? The shared MLP readout and Jodie reformulation change the baselines' native interfaces; please report native-interface results alongside where applicable to separate protocol effects from representation quality. Please also clarify the status of the λ=0 ablation (IDG-No-SMI&FMI).

  3. On the two assessment coordinates. SMI and FMI rank the eleven models almost identically (Spearman 0.93–0.97 on all five datasets); excluding IDG, the baselines retain R²=0.848/0.832. Is there a concrete evaluation in which the two coordinates disagree and change the recommended model? If not, what decision does the second coordinate add beyond the first?

  4. On the SMI construction. The SMI formula contains a directly rewarded entropy term while the Table 6 controls omit the natural nulls. Please report these component/null correlations and the variant without the entropy term, to show that the SMI construction rather than a single entropy term drives the association.

8 comments
AuthorsOpenReview ↗

Response to Reviewer LKPH (1/7)

Dear LKPH,

We sincerely appreciate the time and effort you have devoted to identifying potential issues for our paper. We are also grateful for the opportunity to further clarify the design motivations and experiment details of our framework, which has allowed us to refine and improve the work accordingly.


W1: Complementary Information of SMI/FMI

Thank you for your valuable comments. Our core idea is to evaluate the latent representation via two information channels: Structure and Feature. We further examine their respective diagnostic roles through controlled interventions and cross-model comparisons. The original experiments and supplementary controls has been designed around this unified evaluation objective. Taking your insightful suggestions into account, we further broaden our verification by more targeted experiments to strengthen the argument for their independent contributions. Please refer to our responses to your Question 3 and Question 4 for more detailed explanations.

Detailed explanation

  1. We demonstrate via controlled interventions on structure and attributes that SMI and FMI respond selectively to different information sources. Specifically, Table 21 in Appendix G.11 shows: 20% degree-preserved rewiring reduces SMI by 35.60%-49.70%, and FMI by 5.90%-8.50%; 20% attribute shuffling decreases FMI by 17.10%-28.80%, and SMI by 1.70%-2.50%. These two sets of results correspond to the structural channel and attribute channel, respectively, supporting the complementary diagnostic roles defined in Section 3.2.
  2. Model rankings also reflect the distinct judgments offered by the two metrics. In Table 23, on Mooc dataset, DyCIL achieves a higher SMI than LAMP (0.870 vs. 0.822), while LAMP yields a higher FMI (0.867 vs. 0.810); on the GitHub dataset, the values are 0.785/0.844 and 0.729/0.881, showing the same ranking reversal. The above candidate pairs further illustrate the distinction between structural fidelity and attribute dependence. After excluding IDG in Appendix G.10, the average R2R^2 of the ten baselines remains 0.848/0.832, providing evidence for cross-architecture correlation independent of IDG data points.
  3. Both normalization and evaluation resolution have well-defined computational foundations. Appendix A.7 separates raw training scores from the min-max normalization used for visualization; positive affine transformations of non-constant columns preserve Pearson correlations and ranks. FMI denotes a dependence score under a given critic, budget, and sampling resolution. The meanings of empirical exact MI and fitted scores are separately defined in A.7. The complementarity of the two coordinates can be verified via intervention selectivity, ranking differences among candidates, and direct selection gains.

Supplementary experiment

To further address your concerns about the relationship between FMI and SMI, we add two experiments based on the existing comparisons in Table 10 to refine our argument. Under identical settings of frozen encoder, replay buffer, structural description, and kernel configuration for both SMI and FMI, we perform item-wise evaluations of input entropy, latent representation entropy, joint entropy, and paired shuffling scores. These experimental results strengthen our argument for the complementary diagnostic information. Cross-model correlations are computed against the AUC of the same task using raw scores prior to visualization normalization. Each cell reports the average Pearson R2R^2 / Spearman ρ\rho from 5 independent runs.

a) Whether the performance correlation of SMI mainly originates from the directly rewarded entropy term?

Table LKPH-1: Correlations between SMI entropy components, one-sided paired shuffling, and AUC.

Score ItemRedditMoocGitHubWikipediaTgbl-Wiki
Full SMI0.797/0.9840.909/0.9270.874/0.9860.816/0.9610.804/0.973
Input spectral entropy S(BA)S(B_A), cross-model constantN/AN/AN/AN/AN/A
Representation spectral entropy S(BZ)S(B_Z)0.502/0.5730.449/0.5510.361/0.4550.243/0.3040.417/0.414
Joint spectral entropy S(BAZ)S(B_{AZ})0.352/0.4990.404/0.5340.380/0.4470.313/0.4980.324/0.387
variant without the entropy (SMI − S(B_Z))0.458/0.5970.509/0.6320.484/0.5410.416/0.5990.421/0.492
Mean of one-sided shuffled SMI0.135/0.1580.090/0.0140.109/0.0450.005/0.0190.080/0.128

Note: The scores of input spectral entropy S(BA)S(B_A) are constant across models, so all corresponding correlation values are marked N/A.

AuthorsOpenReview ↗

Response to Reviewer LKPH (2/7)

Obs. R1. Across the five datasets (Table LKPH-1), the correlation between full SMI and AUC is consistently stronger than that of representation spectral entropy and joint spectral entropy alone. These results confirm full SMI outperforms standalone entropy terms in correlation with downstream performance for the tested models. The gap between valid pairing and shuffling controls indicates structure-representation pairing drives this strong correlation.

b) Practical Gain and Independence Control of FMI

For distinct and uniformly sampled one-to-one empirical pairs, the exact empirical MI of an injective encoder can reach log⁡Nrec\log N_{\text{rec}}. What is actually reported in A.7 is the NWJ dependence score under specified input normalization, bounded bilinear critic, fitting budget and sampling protocol; the bounded critic does not mathematically eliminate the aforementioned empirical saturation phenomenon. We explicitly distinguish the exact empirical MI from this operational score.

Table LKPH-2: Pair Usage and Independence Control of FMI.

DatasetTrue FMI vs. AUC: R2/ρR^2/\rhoTrue pair minus fixed-critic shuffling: ΔpairRefitted null scoreNull score vs. AUC: R2/ρR^2/\rho
Reddit0.918/0.9750.0870.7620.641/0.729
Mooc0.815/0.9450.1420.6580.573/0.682
GitHub0.808/0.9750.0930.7240.528/0.751
Wikipedia0.922/0.9090.1560.7410.684/0.617
Tgbl-Wiki0.813/1.0000.1210.6350.556/0.793

Obs. R2. True attribute-representation pairs supply FMI with evaluation signals correlated with model performance. Across the five datasets, score gaps of true pairs relative to the fixed-critic shuffling control are all positive, ranging from 0.087 to 0.156. This shows that the fitted critic exhibits a clear aggregate-level response to genuine correspondences. Meanwhile, the correlation between true FMI and AUC is higher than that of the refitted independence control on all datasets: Pearson R2R^2 increases by 0.238-0.280 and Spearman ρ\rho rises by 0.207-0.292. The consistent performance of pairing sensitivity and correlation advantage across the five datasets collectively supports that true attribute-representation correspondence constitutes one source of FMI’s evaluation signal. These results offer further evidence that FMI characterizes representation quality under the established evaluation protocol.

Taken together, combined with the existing selective responses, candidate ranking differences and cross-model correlations in the main paper that jointly support the evaluation capability of the two metrics, the above supplementary experiments further demonstrate their independent roles.

Thank you again for your valuable suggestions. We integrate the above supplementary experiments and analysis into Appendix G.6 / Tables 11-12.

W2: Clarification of Empirical Comparison Fairness

We greatly appreciate your insightful comment. As we have demonstrated in original paper, we have already provided baseline implementations (Appendix F.3.1 / Table 8), data descriptions (Appendix F.2), and parameter settings within the supplementary materials (Anonymous Repository README.md). To provide more direct evidence for empirical comparison fairness, we explicitly specify the relevant experimental setups and have added new experiments. To avoid redundant presentation, please refer to our responses to your Question 1 and Question 2 for the detailed supplementary content.

1. Explicit reproduction anchors for the link prediction pipeline. We add a complete benchmark with publicly verifiable results, where we align item-by-item the data version, task formulation, temporal split, input attributes, negative sampling and candidates, historical state updates, and evaluator. We compare reruns from official implementations and IDG under the same protocol. Detailed results are included in our response to your Question 1.

2. Public actual configurations and tuning. We verify configuration sources, concrete hyperparameters, code versions, and other information for the ten baselines and IDG on a per-dataset basis. Full records are available in our response to your Question 2.

Thank you very much for your suggestion. For detailed supplementary information, please refer to the following response.

AuthorsOpenReview ↗

Response to Reviewer LKPH (3/7)

Q1: Regarding Benchmark Truncation

Thank you for your valuable question. The preprocessing of Wikipedia, Reddit and GitHub aims to construct well-defined, reusable temporal observation units and enable unified representation evaluation across diverse data scales. Beyond further clarifying the benchmark setup, we have followed your suggestion to add Tgbl-Wiki with its official split as an additional dataset, on which we conduct supplementary experiments to further support experimental fidelity.

Detailed explanation

  1. Appendix F.2 and Table 6 have specified the retained scale and temporal processing scheme for each dataset. Mooc retains 411,749 interactions and Tgbl-Wiki retains 157,474; for Wikipedia/Reddit, we take the first 10,000 temporally ordered interactions. This processing combines scalability with unified snapshot comparison. All models share the retained events and prediction boundaries, enabling fair comparisons against all baselines.
  2. The adoption of continuous-time prefixes preserves the natural chronological order within the selected observation window and controls comparison overhead. The Reddit/Wikipedia datasets of JODIE are originally constructed using real activities within one month [1]. On this basis, we take the first 10,000 interactions while preserving their original chronological order, then aggregate them into 20 windows following the same rule; all models use this identical input. Following your suggestion, supplementary re-evaluation on full Tgbl-Wiki with its official split further examines the effect of varying coverage scales.
  3. Fixing the historical month for GitHub serves to define clearly reusable observation subjects. We select WatchEvents from January 1-31, 2015 to study account interactions and node attributes within this period. GitHub activities accumulate continuously, so fixed start and end dates clearly define the data extraction scope. Studies on event-stream collection and historical queries in GHTorrent provide literature support for temporally organizing GitHub activities [2]. This citation backs the practice of temporal log analysis; the exact month and snapshot definitions used in our work are presented in F.2.
  4. Full reproduction and post-processing comparison serve distinct validation goals. The former inspects implementation performance after aligning releases, splits, sampling strategies and evaluators; the latter compares representations under shared inputs and prediction boundaries. Prior evaluation studies have shown that negative sampling can alter method rankings, so we document these conditions item-by-item in our full-data re-evaluation [3].

Supplementary experiment

To further address your concerns regarding full data usage and reproducibility fidelity, we have followed your suggestion and supplemented tests of all baselines on Tgbl-Wiki (official split) to verify the reliability of our experimental results.

Table LKPH-3: Full-data Reproduced Link Prediction AUC under Tgbl-Wiki (Official Split)

MetricJodieDyRepDySATDyGNNTGNPINTDyGMambaRAG4DyGDyCILLAMPIDG
AUC (%)0.763$\pm$0.0120.692$\pm$0.0180.818$\pm$0.0090.780$\pm$0.0110.823$\pm$0.0070.838$\pm$0.0080.756$\pm$0.0130.887$\pm$0.0050.904$\pm$0.0040.871$\pm$0.0060.914$\pm$0.003
AP (%)0.748$\pm$0.0140.675$\pm$0.0200.824$\pm$0.0100.762$\pm$0.0120.807$\pm$0.0080.845$\pm$0.0090.739$\pm$0.0140.872$\pm$0.0060.888$\pm$0.0050.855$\pm$0.0070.899$\pm$0.004

Thank you again for your suggestions. We have revised the relevant descriptions in Appendix F.2 to improve the interpretability of the benchmark truncation. Descriptions regarding GitHub data extraction, filtering, and stepwise retention have also been added to Appendix F.2.

reference

[1] Kumar, Srijan, Xikun Zhang, and Jure Leskovec. "Predicting dynamic embedding trajectory in temporal interaction networks." Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019.

[2] Gousios, Georgios. "The GHTorent dataset and tool suite." 2013 10th Working conference on mining software repositories (MSR). IEEE, 2013.

[3] Poursafaei, Farimah, et al. "Towards better evaluation for dynamic link prediction." Advances in Neural Information Processing Systems 35 (2022): 32928-32941.

Comments on the paper

2 comments

Thank you again for your time and thoughtful review. With only three days remaining in the discussion period, we would greatly appreciate your feedback on whether our rebuttal has addressed your concerns. If any concerns remain, we would be happy to provide further clarification.

AuthorsOpenReview ↗

General Response

Dear Area Chair and all reviewers,

We sincerely thank you for the time and effort devoted to evaluating our paper. Although several reviewers have yet to participate in the discussion up to now, the constructive comments received thus far have helped us enhance the empirical validation and improve the presentation of our IDG method. To facilitate the final assessment, we summarize our work and the progress made during the rebuttal below.


Summary of Our Work

a. Broad Impact and Relevance

  • We propose IDG, a unified framework for intrinsic assessment and information-theoretic learning of dynamic graph representations. It addresses the need for label free assessment of representation quality, providing insights beyond downstream task performance.
  • IDG introduces Structure Mutual Information (SMI) and Feature Mutual Information (FMI) to assess structural and attribute information preserved in representations. A shared evaluation interface supports diagnosis across encoder architectures, while integrating these metrics into learning helps improve representations for downstream tasks.

b. Technical Merit

  • Theoretical grounding: We provide exhensive analyses connecting information preservation, input recovery, and downstream task risk, with explicit assumptions and a clear distinction between mutual information and its practical estimation.
  • Extensive empirical validation: Experiments across link prediction, node classification, and graph classification, together with ablations, controlled perturbations, metric validation, and model-selection studies, support the diagnostic value and practical utility of IDG.

Addressed Concerns

During the rebuttal, we conducted supplementary experiments, clarified the theoretical interpretation, and revised the manuscript with highlighted changes. The main progress is summarized below:

  • Reviewer LKPH: We addressed concerns about metric validity and complementarity, benchmark coverage, and baseline treatment (W1-W2, Q1-Q4). Additional controls, full-data evaluation, interface comparisons, and model-selection experiments strengthen the evidence for the metrics’ utility and the reliability of the empirical comparisons.
  • Reviewer Xme9: We addressed the minor suggestions about novelty and interpretability (W1-W2) by clarifying IDG’s differences from existing information-bottleneck methods and its structural/attribute diagnostic scope. These revisions make the methodological contribution and its supporting evidence more explicit.
  • Reviewer QduF: Under his recognition of our work, we addressed concerns about metric interpretation, label-free selection, temporal splitting, evaluation fairness, and sensitivity (W1-W2, Q1-Q5). Following the reviewer’s acknowledgment that most concerns had been addressed, we further expanded selection to all baselines and clarified FMI consistently as a variational lower-bound estimate, directly responding to the two minor remaining issues. We also incorporated the requested raw scores, configuration details, and broader sensitivity results.

We appreciate the positive progress in the discussion. Reviewer Xme9 stated that our reply had “addressed most of my concerns” and confirmed, “I maintain a positive rating.” Reviewer QduF explicitly acknowledged that most concerns had been addressed. We would greatly appreciate it if the reviewers could consider raising the ratings.

These revisions substantially strengthen the evidence for IDG’s contribution and practical value. We respectfully ask for a favorable reassessment based on the clear contributions and complete evidence during the rebuttal period. Thank you again for your careful consideration.

Sincerely, The Authors of Submission 2478

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.