- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TTGBench, a benchmark for evaluating temporal interaction dynamics in text-attributed temporal graphs. Motivated by the limited coverage of semantic evolution in existing temporal graph benchmarks, the authors construct six real-world review datasets spanning multiple domains, characterized by diverse semantic and structural evolution patterns, which they collectively refer to as Dual Volatility. TTGBench supports both temporal link prediction (TLP) and temporal node classification (TNC), including multi-class and multi-label settings. The authors evaluate 17 representative methods spanning temporal graph neural networks (TGNNs) and LLM-based approaches. Their experiments reveal a task-dependent performance asymmetry, where TGNN-based methods generally perform better on TLP while LLM-based predictors show stronger performance on TNC. The paper further investigates the limitations of TGNNs in semantic evolution tracking and the limitations of LLM-based predictors in structural evolution perception. Detailed computational efficiency and scalability analysis is also provided, highlighting the substantial computational overhead of LLM-based approaches and the performance–efficiency trade-offs among different model architectures.
Critical strengths
- The paper introduces the concept of dual volatility to characterize temporal interaction dynamics from both structural and semantic perspectives, representing temporal interaction dynamics with the evolving interaction distributions within the graph and the label drift behavior.
- The benchmark includes six real-world and text-attributed temporal graph datasets, exhibiting diverse structural and semantic characteristics, including high link novelty and varying patterns of semantic transitions. They provide useful resources for investigating existing temporal graph learning methods under more complex interaction dynamics.
- The experimental evaluation is relatively comprehensive, covering 17 representative methods across traditional TGNNs, LLM-enhanced TGNNs, and LLM-based predictors. Beyond the main predictive performance comparisons, the paper includes additional analyses of model behavior under different conditions, as well as computational efficiency and scalability. These experiments provide useful empirical observations regarding the capabilities and limitations of existing temporal graph learning approaches.
Critical weaknesses
-
Insufficient comparison with existing benchmarks in terms of dual volatility. Although the authors position dual volatility as a key distinguishing characteristic of TTGBench, the analysis primarily focuses on the proposed datasets without systematically comparing their structural and semantic dynamics against existing temporal graph benchmarks, especially DTGB. It remains unclear to what extent the proposed datasets exhibit fundamentally different or more challenging temporal dynamics than existing benchmarks. A more comprehensive comparison would help substantiate the distinctiveness and necessity of TTGBench.
-
Insufficient characterization of dual volatility. The proposed dual volatility concept provides an intuitive perspective on temporal interaction dynamics. However, for structural volatility, the analysis focuses on the changes of interaction distributions, without sufficiently examining the temporal dependencies and evolving interaction patterns underlying these observations, such as temporal locality and periodicity; for semantic volatility, the analysis focuses on label drifting behavior, providing limited insights for underlying semantic evolving patterns. Frequent changes in interaction patterns or semantic labels do not necessarily imply complex temporal dynamics that require sophisticated temporal modeling. Without more comprehensive characterization of evolution patterns, it is difficult to assess what specific modeling challenges dual volatility introduces.
-
Limited investigation of the interplay between structural and semantic dynamics. Although the paper emphasizes the importance of jointly characterizing structural and semantic evolution, the two dimensions are largely analyzed and evaluated independently. The benchmark primarily relies on TLP and TNC to assess structural and semantic modeling capabilities respectively. However, in real-world temporal graphs, structural and semantic evolution may be closely intertwined. Consequently, evaluating TLP and TNC separately may not fully reveal whether models can capture complex temporal interaction dynamics. A more systematic investigation of the interplay between these two dimensions would strengthen the benchmark's contribution.
Questions for authors
- Could authors provide a systematic comparison of the proposed benchmark with DTGB in terms of dual volatility? Such comparison would help clarify the distinctiveness of TTGBench.
- Could authors further clarify how the proposed volatility relate to specific temporal evolution patterns and modeling requirements? For instance, interaction density variation, hub activity instability, and frequent label transitions may arise from different temporal mechanisms, such as short-term dependencies, periodic behaviors, or long-term evolutionary trends. To what extent do the current dual volatility metrics distinguish these patterns?
- Could authors further investigate the role of temporal dependencies in TNC? It remains unclear whether label drifting has meaningful patterns, and whether the superior performance of LLM-Predictors in TNC stems from their ability to capture semantic evolving patterns instead of general semantic understanding capability.
7 comments
Response to Reviewer's Concerns - Part 2
We quantify semantic volatility using the label drift ratio. For each node , let denote its temporally ordered labels, where is the number of labeled interactions involving .
-
Multi-class datasets: Drift between consecutive interactions is defined as
where equals when the category changes and otherwise.
-
Multi-label datasets: Let denote the set of categories at time . We measure drift using the Jaccard distance:
For both settings, the dataset-level drift ratio is computed by averaging over all consecutive labeled-interaction pairs:
\frac{\sum_u\sum_{i=2}^{n_u}d_u(t_i)} {\sum_u(n_u-1)}.$$ **3. TTGBench's semantic prediction task benefits substantially from temporal graph history.** Table 3 compares semantic classification performance with and without historical graph information. On DTGB's Enron Edge Classification task, incorporating historical graph information does not improve the performance of JODIE or DyGFormer. In contrast, both methods achieve substantial gains on TTGBench's IMDB TNC task when historical graph information is available. These results demonstrate that TNC presents a meaningful challenge for temporal graph modeling and that historical interactions provide valuable information for predicting future semantic labels. **Table 3. Performance comparison with and without historical temporal graph information.** | Benchmark | Dataset / Task | Model | With Historical Graph (F1) | Without Historical Graph (F1) | Performance Gain | | --------- | --------------------------- | --------- | -------------------------- | ----------------------------- | ---------------- | | DTGB | Enron / Edge Classification | JODIE | 64.78 | 67.96 | −3.18 | | | | DyGFormer | 56.04 | 63.77 | −7.73 | | TTGBench | IMDB / TNC | JODIE | 12.65 | 0.76 | +11.89 | | | | DyGFormer | 20.05 | 0.93 | +19.12 | # Q2. Temporal mechanisms underlying dual volatility and their modeling requirements We extend the descriptive characterization of dual volatility in Figure 2 through a graph–temporal frequency analysis of structural and semantic dynamics. Specifically, we analyze structural and semantic dynamics within a unified graph–temporal frequency domain, treating them as different graph signals defined over the same underlying graph. Applying two-dimensional graph–temporal spectral analysis to these signals reveals their characteristic evolution patterns, ranging from slow, spatially smooth changes to rapid, spatially heterogeneous fluctuations. Tables 4 and 6 report the frequency-band energy distributions of structural and semantic dynamics across TTGBench datasets, respectively, while Tables 5 and 7 summarize their dominant patterns and interpretations. **The results reveal distinct, dataset-specific evolution patterns, highlighting the need for models to capture the dominant spatial and temporal scales of structural and semantic dynamics.** **Table 4. Graph–temporal frequency energy distribution of structural dynamics across TTGBench datasets.** | Temporal Frequency | Graph Frequency | FOOD | IMDB | Librarything | Beeradvocate | Amazon-Kindle | Ratebeer | | ------------------ | --------------- | ------- | ------- | ------------ | ------------ | ------------- | -------- | | Low | Low | 12% | 25% | 6% | 8% | **40%** | 9% | | | Medium | 8% | **29%** | 4% | 14% | 20% | 7% | | | High | 4% | 10% | 3% | **20%** | 6% | 6% | | Medium | Low | **28%** | 14% | 10% | 10% | 14% | 16% | | | Medium | 16% | 12% | 11% | 15% | 9% | 17% | | | High | 8% | 5% | 8% | 13% | 5% | **19%** | | High | Low | 12% | 2% | 12% | 5% | 3% | 7% | | | Medium | 7% | 2% | 17% | 7% | 2% | 9% | | | High | 5% | 1% | **29%** | 8% | 1% | 10% |Response to Reviewer's Concerns - Part 3
Table 5. Dominant structural frequency bands and their implications across TTGBench datasets.
| Dataset | Dominant Frequency Band | Interpretation and Modeling Implications |
|---|---|---|
| FOOD | Medium temporal × Low graph | Spatially smooth structural evolution at intermediate temporal scales, suggesting the importance of medium-range interaction history. |
| IMDB | Low temporal × Medium graph | Slowly evolving structural patterns with moderate spatial variation, requiring historical context beyond the most recent interactions. |
| Librarything | High temporal × High graph | Rapid fluctuations and strong spatial heterogeneity in new-link formation, suggesting the value of recent, fine-grained interaction history. |
| Beeradvocate | Low temporal × High graph | Slow structural evolution with pronounced node-level heterogeneity, highlighting the importance of persistent local structural patterns. |
| Amazon-Kindle | Low temporal × Low graph | Slowly evolving, spatially smooth structural patterns, emphasizing the value of long-term structural context. |
| Ratebeer | Medium temporal × High graph | Moderately paced but spatially heterogeneous structural evolution, contrasting with Amazon-Kindle's slow, smooth dynamics. |
Table 6. Graph–temporal frequency energy distribution of semantic dynamics across TTGBench datasets.
| Temporal Frequency | Graph Frequency | FOOD | IMDB | Beeradvocate | Amazon-Kindle | Ratebeer |
|---|---|---|---|---|---|---|
| Low | Low | 9% | 5% | 24% | 65% | 22% |
| Medium | 8% | 5% | 2% | 8% | 3% | |
| High | 5% | 5% | 9% | 7% | 14% | |
| Medium | Low | 23% | 12% | 34% | 11% | 36% |
| Medium | 17% | 10% | 24% | 4% | 19% | |
| High | 9% | 8% | 2% | 1% | 1% | |
| High | Low | 14% | 13% | 2% | 2% | 2% |
| Medium | 9% | 15% | 2% | 1% | 2% | |
| High | 6% | 27% | 1% | 1% | 1% |
Table 7. Dominant semantic frequency bands and their implications across TTGBench datasets.
| Dataset | Dominant Frequency Band | Interpretation |
|---|---|---|
| FOOD | Medium temporal × Low graph | Shared semantic category patterns among neighboring nodes evolve at intermediate temporal scales. |
| IMDB | High temporal × High graph | Rapid semantic evolution with pronounced node-level heterogeneity. |
| Beeradvocate | Medium temporal × Low–Medium graph | Semantic redistribution at intermediate temporal scales, combining shared group-level patterns with localized variation. |
| Amazon-Kindle | Low temporal × Low graph | Slowly evolving semantic patterns that remain relatively smooth across neighboring nodes. |
| Ratebeer | Medium temporal × Low–Medium graph | Semantic evolution at intermediate temporal scales, contrasting with Amazon-Kindle's concentration in low-frequency bands. |
Q3. How TLP and TNC reveal complex structural–semantic interplay.
We agree with the reviewer that reporting TLP and TNC scores alone is insufficient to characterize the complex interplay between structural and semantic dynamics. However, these tasks provide a basis for investigating such interplay through controlled experiments. Joint evaluation of TLP and TNC enables us to examine whether models capture predictive spatiotemporal patterns across structural and semantic dynamics, while controlled frequency-scale experiments reveal where such modeling succeeds or fails.
1.TLP and TNC guide models to learn task-relevant graph–temporal scales of structural–semantic dynamics.
To investigate whether task-specific training can identify informative frequency scales, we extend DyGFormer with nine learnable frequency-band scores for each task. These scores determine the weights assigned to historical frequency-band representations , producing a task-specific representation:
Response to Reviewer's Concerns - Part 1
We sincerely appreciate the reviewer's constructive feedback and insightful questions. We have carefully revised the manuscript to address the reviewer's concerns, with all revisions highlighted in blue, and provide detailed responses to each point below.
Q1. Distinctiveness of TTGBench’s Dual Volatility compared with DTGB
TTGBench's key distinction lies in its joint evaluation of structural evolution (TLP) and semantic evolution (TNC) arising from the same interactions under high structural novelty. DTGB primarily covers structural prediction, observed-edge classification, and text generation, without a task equivalent to TNC that predicts interaction-induced semantics strictly from historical information. TTGBench therefore evaluates structural and semantic volatility within a unified, temporally causal prediction setting and further investigates their underlying temporal evolution patterns.
1. DTGB lacks a dedicated task for evaluating semantic evolution. As shown in Table 1, DTGB supports structural prediction and text-related tasks but does not explicitly evaluate the predictability of semantic evolution. Although its Edge Classification task also involves categorical labels, the target edge is already known at prediction time, whereas TNC infers the semantic category exclusively from historical information. Similarly, Textual Relation Generation evaluates text generation capabilities rather than the predictability of evolving semantic categories. TTGBench uniquely quantifies the structural and semantic dynamics of the same interaction process through two complementary, temporally causal prediction tasks.
Table 1. Comparison of task objectives and dual-volatility coverage between DTGB and TTGBench.
2. TTGBench combines high structural volatility with measurable semantic volatility.
Table 2. Comparison of dual-volatility characteristics between DTGB and TTGBench.