acceptodds
Under review as a conference paper at ICLR 2027

TTGBench: Benchmarking Temporal Interaction Dynamics in Text-attributed Temporal Graphs

Abstract

Temporal graph learning models dynamic systems as sequences of timestamped interactions, where both structural and semantic dynamics evolve over time. However, existing benchmarks primarily emphasize structural dynamics through temporal link prediction (TLP), while support for semantic dynamics remains limited. Although temporal node classification (TNC) is sometimes included, it is often limited to simple binary settings. Moreover, commonly used datasets exhibit high link repetition, which can obscure whether models truly capture evolving interaction patterns. To address these limitations, we introduce TTGBench, a benchmark for temporal interaction dynamics in text-attributed temporal graphs. TTGBench comprises six real-world, text-rich datasets with high structural novelty and diverse semantic transition patterns, and jointly supports TLP as well as multi-class and multi-label TNC. We conduct a comprehensive evaluation of 17 state-of-the-art methods across Temporal Graph Neural Networks (TGNNs) and Large Language Model (LLM)-based paradigms. The results reveal a clear task-dependent performance asymmetry between the two paradigms: TGNN-based methods excel at structural prediction but fail at semantic dynamics, whereas LLM-based predictors show the opposite trend. Through in-depth analysis, we uncover their fundamental limitations and provide insights for developing more comprehensive temporal graph models. TTGBench datasets, leaderboards, and code are available at our project page https://ttgbench.netlify.app.

Primary area:
datasets and benchmarks
Keywords:
temporal graph, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5CNpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TTGBench, a benchmark for evaluating temporal interaction dynamics in text-attributed temporal graphs. Motivated by the limited coverage of semantic evolution in existing temporal graph benchmarks, the authors construct six real-world review datasets spanning multiple domains, characterized by diverse semantic and structural evolution patterns, which they collectively refer to as Dual Volatility. TTGBench supports both temporal link prediction (TLP) and temporal node classification (TNC), including multi-class and multi-label settings. The authors evaluate 17 representative methods spanning temporal graph neural networks (TGNNs) and LLM-based approaches. Their experiments reveal a task-dependent performance asymmetry, where TGNN-based methods generally perform better on TLP while LLM-based predictors show stronger performance on TNC. The paper further investigates the limitations of TGNNs in semantic evolution tracking and the limitations of LLM-based predictors in structural evolution perception. Detailed computational efficiency and scalability analysis is also provided, highlighting the substantial computational overhead of LLM-based approaches and the performance–efficiency trade-offs among different model architectures.

Critical strengths
  1. The paper introduces the concept of dual volatility to characterize temporal interaction dynamics from both structural and semantic perspectives, representing temporal interaction dynamics with the evolving interaction distributions within the graph and the label drift behavior.
  2. The benchmark includes six real-world and text-attributed temporal graph datasets, exhibiting diverse structural and semantic characteristics, including high link novelty and varying patterns of semantic transitions. They provide useful resources for investigating existing temporal graph learning methods under more complex interaction dynamics.
  3. The experimental evaluation is relatively comprehensive, covering 17 representative methods across traditional TGNNs, LLM-enhanced TGNNs, and LLM-based predictors. Beyond the main predictive performance comparisons, the paper includes additional analyses of model behavior under different conditions, as well as computational efficiency and scalability. These experiments provide useful empirical observations regarding the capabilities and limitations of existing temporal graph learning approaches.
Critical weaknesses
  1. Insufficient comparison with existing benchmarks in terms of dual volatility. Although the authors position dual volatility as a key distinguishing characteristic of TTGBench, the analysis primarily focuses on the proposed datasets without systematically comparing their structural and semantic dynamics against existing temporal graph benchmarks, especially DTGB. It remains unclear to what extent the proposed datasets exhibit fundamentally different or more challenging temporal dynamics than existing benchmarks. A more comprehensive comparison would help substantiate the distinctiveness and necessity of TTGBench.

  2. Insufficient characterization of dual volatility. The proposed dual volatility concept provides an intuitive perspective on temporal interaction dynamics. However, for structural volatility, the analysis focuses on the changes of interaction distributions, without sufficiently examining the temporal dependencies and evolving interaction patterns underlying these observations, such as temporal locality and periodicity; for semantic volatility, the analysis focuses on label drifting behavior, providing limited insights for underlying semantic evolving patterns. Frequent changes in interaction patterns or semantic labels do not necessarily imply complex temporal dynamics that require sophisticated temporal modeling. Without more comprehensive characterization of evolution patterns, it is difficult to assess what specific modeling challenges dual volatility introduces.

  3. Limited investigation of the interplay between structural and semantic dynamics. Although the paper emphasizes the importance of jointly characterizing structural and semantic evolution, the two dimensions are largely analyzed and evaluated independently. The benchmark primarily relies on TLP and TNC to assess structural and semantic modeling capabilities respectively. However, in real-world temporal graphs, structural and semantic evolution may be closely intertwined. Consequently, evaluating TLP and TNC separately may not fully reveal whether models can capture complex temporal interaction dynamics. A more systematic investigation of the interplay between these two dimensions would strengthen the benchmark's contribution.

Questions for authors
  1. Could authors provide a systematic comparison of the proposed benchmark with DTGB in terms of dual volatility? Such comparison would help clarify the distinctiveness of TTGBench.
  2. Could authors further clarify how the proposed volatility relate to specific temporal evolution patterns and modeling requirements? For instance, interaction density variation, hub activity instability, and frequent label transitions may arise from different temporal mechanisms, such as short-term dependencies, periodic behaviors, or long-term evolutionary trends. To what extent do the current dual volatility metrics distinguish these patterns?
  3. Could authors further investigate the role of temporal dependencies in TNC? It remains unclear whether label drifting has meaningful patterns, and whether the superior performance of LLM-Predictors in TNC stems from their ability to capture semantic evolving patterns instead of general semantic understanding capability.
7 comments
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 1

We sincerely appreciate the reviewer's constructive feedback and insightful questions. We have carefully revised the manuscript to address the reviewer's concerns, with all revisions highlighted in blue, and provide detailed responses to each point below.

Q1. Distinctiveness of TTGBench’s Dual Volatility compared with DTGB

TTGBench's key distinction lies in its joint evaluation of structural evolution (TLP) and semantic evolution (TNC) arising from the same interactions under high structural novelty. DTGB primarily covers structural prediction, observed-edge classification, and text generation, without a task equivalent to TNC that predicts interaction-induced semantics strictly from historical information. TTGBench therefore evaluates structural and semantic volatility within a unified, temporally causal prediction setting and further investigates their underlying temporal evolution patterns.

1. DTGB lacks a dedicated task for evaluating semantic evolution. As shown in Table 1, DTGB supports structural prediction and text-related tasks but does not explicitly evaluate the predictability of semantic evolution. Although its Edge Classification task also involves categorical labels, the target edge is already known at prediction time, whereas TNC infers the semantic category exclusively from historical information. Similarly, Textual Relation Generation evaluates text generation capabilities rather than the predictability of evolving semantic categories. TTGBench uniquely quantifies the structural and semantic dynamics of the same interaction process through two complementary, temporally causal prediction tasks.

Table 1. Comparison of task objectives and dual-volatility coverage between DTGB and TTGBench.

BenchmarkTaskPrediction Target / QueryRole in Dual-Volatility Evaluation
DTGBFuture Link PredictionGiven historical information and (u,v,t)(u,v,t), predict whether a future link will formStructural evolution: Future link formation
Destination Node RetrievalGiven historical information, a source node, and candidate destinations, retrieve the future destinationStructural evolution: Future interaction partners
Edge ClassificationGiven an observed target edge (u,v,t)(u,v,t) and its available attributes, predict its categoryedge classification: Does not directly evaluate whether future interaction semantics can be predicted from history
Textual Relation GenerationGiven nodes and historical relation texts, generate the target relation textRelation text generation: Evaluates a different objective from category-based prediction of semantic evolution
TTGBenchTLPGiven the historical graph and (u,v,t)(u,v,t), predict whether a future link will formStructural evolution: Future link formation
TNCGiven the historical graph and (u,t)(u,t), predict the future semantic category without access to its destination node or interaction textSemantic evolution: Future semantic labels and their temporal changes

2. TTGBench combines high structural volatility with measurable semantic volatility.

Table 2. Comparison of dual-volatility characteristics between DTGB and TTGBench.

BenchmarkLink NoveltySemantic Labels and Evaluation
DTGB0.23–0.41Provides relation/behavior categories, user–business ratings on a 1–5 scale, and binary event labels. Existing Edge Classification tasks predict categories given observed target edges, without explicitly evaluating changes in or predictability of future semantic categories.
TTGBench0.99–1.00Provides interaction-derived semantic labels exhibiting substantial temporal variation, with dataset-level label drift ratios ranging from 0.46 to 0.74.
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 2

We quantify semantic volatility using the label drift ratio. For each node uu, let yu(t1),yu(t2),…,yu(tnu)y_u(t_1),y_u(t_2),\ldots,y_u(t_{n_u}) denote its temporally ordered labels, where nun_u is the number of labeled interactions involving uu.

  • Multi-class datasets: Drift between consecutive interactions is defined as

    du(ti)=1[yu(ti)≠yu(ti−1)],d_u(t_i)=\mathbf{1}\left[y_u(t_i)\neq y_u(t_{i-1})\right],

    where 1[⋅]\mathbf{1}[\cdot] equals 11 when the category changes and 00 otherwise.

  • Multi-label datasets: Let Yu(ti)Y_u(t_i) denote the set of categories at time tit_i. We measure drift using the Jaccard distance:

    du(ti)=1−∣Yu(ti)∩Yu(ti−1)∣∣Yu(ti)∪Yu(ti−1)∣.d_u(t_i)=1-\frac{\left|Y_u(t_i)\cap Y_u(t_{i-1})\right|}{\left|Y_u(t_i)\cup Y_u(t_{i-1})\right|}.

For both settings, the dataset-level drift ratio is computed by averaging over all consecutive labeled-interaction pairs:

\frac{\sum_u\sum_{i=2}^{n_u}d_u(t_i)} {\sum_u(n_u-1)}.$$ **3. TTGBench's semantic prediction task benefits substantially from temporal graph history.** Table 3 compares semantic classification performance with and without historical graph information. On DTGB's Enron Edge Classification task, incorporating historical graph information does not improve the performance of JODIE or DyGFormer. In contrast, both methods achieve substantial gains on TTGBench's IMDB TNC task when historical graph information is available. These results demonstrate that TNC presents a meaningful challenge for temporal graph modeling and that historical interactions provide valuable information for predicting future semantic labels. **Table 3. Performance comparison with and without historical temporal graph information.** | Benchmark | Dataset / Task | Model | With Historical Graph (F1) | Without Historical Graph (F1) | Performance Gain | | --------- | --------------------------- | --------- | -------------------------- | ----------------------------- | ---------------- | | DTGB | Enron / Edge Classification | JODIE | 64.78 | 67.96 | −3.18 | | | | DyGFormer | 56.04 | 63.77 | −7.73 | | TTGBench | IMDB / TNC | JODIE | 12.65 | 0.76 | +11.89 | | | | DyGFormer | 20.05 | 0.93 | +19.12 | # Q2. Temporal mechanisms underlying dual volatility and their modeling requirements We extend the descriptive characterization of dual volatility in Figure 2 through a graph–temporal frequency analysis of structural and semantic dynamics. Specifically, we analyze structural and semantic dynamics within a unified graph–temporal frequency domain, treating them as different graph signals defined over the same underlying graph. Applying two-dimensional graph–temporal spectral analysis to these signals reveals their characteristic evolution patterns, ranging from slow, spatially smooth changes to rapid, spatially heterogeneous fluctuations. Tables 4 and 6 report the frequency-band energy distributions of structural and semantic dynamics across TTGBench datasets, respectively, while Tables 5 and 7 summarize their dominant patterns and interpretations. **The results reveal distinct, dataset-specific evolution patterns, highlighting the need for models to capture the dominant spatial and temporal scales of structural and semantic dynamics.** **Table 4. Graph–temporal frequency energy distribution of structural dynamics across TTGBench datasets.** | Temporal Frequency | Graph Frequency | FOOD | IMDB | Librarything | Beeradvocate | Amazon-Kindle | Ratebeer | | ------------------ | --------------- | ------- | ------- | ------------ | ------------ | ------------- | -------- | | Low | Low | 12% | 25% | 6% | 8% | **40%** | 9% | | | Medium | 8% | **29%** | 4% | 14% | 20% | 7% | | | High | 4% | 10% | 3% | **20%** | 6% | 6% | | Medium | Low | **28%** | 14% | 10% | 10% | 14% | 16% | | | Medium | 16% | 12% | 11% | 15% | 9% | 17% | | | High | 8% | 5% | 8% | 13% | 5% | **19%** | | High | Low | 12% | 2% | 12% | 5% | 3% | 7% | | | Medium | 7% | 2% | 17% | 7% | 2% | 9% | | | High | 5% | 1% | **29%** | 8% | 1% | 10% |
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 3

Table 5. Dominant structural frequency bands and their implications across TTGBench datasets.

DatasetDominant Frequency BandInterpretation and Modeling Implications
FOODMedium temporal × Low graphSpatially smooth structural evolution at intermediate temporal scales, suggesting the importance of medium-range interaction history.
IMDBLow temporal × Medium graphSlowly evolving structural patterns with moderate spatial variation, requiring historical context beyond the most recent interactions.
LibrarythingHigh temporal × High graphRapid fluctuations and strong spatial heterogeneity in new-link formation, suggesting the value of recent, fine-grained interaction history.
BeeradvocateLow temporal × High graphSlow structural evolution with pronounced node-level heterogeneity, highlighting the importance of persistent local structural patterns.
Amazon-KindleLow temporal × Low graphSlowly evolving, spatially smooth structural patterns, emphasizing the value of long-term structural context.
RatebeerMedium temporal × High graphModerately paced but spatially heterogeneous structural evolution, contrasting with Amazon-Kindle's slow, smooth dynamics.

Table 6. Graph–temporal frequency energy distribution of semantic dynamics across TTGBench datasets.

Temporal FrequencyGraph FrequencyFOODIMDBBeeradvocateAmazon-KindleRatebeer
LowLow9%5%24%65%22%
Medium8%5%2%8%3%
High5%5%9%7%14%
MediumLow23%12%34%11%36%
Medium17%10%24%4%19%
High9%8%2%1%1%
HighLow14%13%2%2%2%
Medium9%15%2%1%2%
High6%27%1%1%1%

Table 7. Dominant semantic frequency bands and their implications across TTGBench datasets.

DatasetDominant Frequency BandInterpretation
FOODMedium temporal × Low graphShared semantic category patterns among neighboring nodes evolve at intermediate temporal scales.
IMDBHigh temporal × High graphRapid semantic evolution with pronounced node-level heterogeneity.
BeeradvocateMedium temporal × Low–Medium graphSemantic redistribution at intermediate temporal scales, combining shared group-level patterns with localized variation.
Amazon-KindleLow temporal × Low graphSlowly evolving semantic patterns that remain relatively smooth across neighboring nodes.
RatebeerMedium temporal × Low–Medium graphSemantic evolution at intermediate temporal scales, contrasting with Amazon-Kindle's concentration in low-frequency bands.

Q3. How TLP and TNC reveal complex structural–semantic interplay.

We agree with the reviewer that reporting TLP and TNC scores alone is insufficient to characterize the complex interplay between structural and semantic dynamics. However, these tasks provide a basis for investigating such interplay through controlled experiments. Joint evaluation of TLP and TNC enables us to examine whether models capture predictive spatiotemporal patterns across structural and semantic dynamics, while controlled frequency-scale experiments reveal where such modeling succeeds or fails.

1.TLP and TNC guide models to learn task-relevant graph–temporal scales of structural–semantic dynamics.

To investigate whether task-specific training can identify informative frequency scales, we extend DyGFormer with nine learnable frequency-band scores aba_b for each task. These scores determine the weights assigned to historical frequency-band representations zb\mathbf z_b, producing a task-specific representation:

htask=∑b=19αbtaskzb,αbtask=eab∑j=19eaj.\mathbf h^{\mathrm{task}}=\sum_{b=1}^{9}\alpha_b^{\mathrm{task}}\mathbf z_b,\qquad\alpha_b^{\mathrm{task}}=\frac{e^{a_b}}{\sum_{j=1}^{9}e^{a_j}}.

Reviewer_PLpHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TTGBench comprises six text-attributed user–item interaction datasets, supporting temporal link prediction (TLP) on all six and temporal node classification (TNC) on five. For TNC, a user’s label at each interaction corresponds to the category or genre of the item reviewed. The paper benchmarks temporal GNNs, LLM-enhanced models, and adapted LLM predictors. Under the reported setup, GNNs generally outperform on link prediction, while LLM-based predictors excel at classification. The appendix further includes harder negative sampling settings and an efficiency comparison.

Critical strengths
  1. The benchmark provides a common testbed for both structure and text, featuring multi-label and multi-class targets, chronological splits, and up to several million interactions in the largest dataset.
  2. The evaluation is comprehensive, extending beyond simple link-prediction metrics to include 17 model variants, three random seeds, hard negative samplers, imbalance-aware classification metrics, and computational costs. These details should ensure the benchmark remains useful even if some broader claims are later refined.
Critical weaknesses
  1. The incremental contribution and TNC target definition require sharper justification. The paper cites DTGB [1], which already covers multiple tasks including textual relation generation, yet lacks a detailed comparison of dataset quality or task design. It also omits GDGB [2], which curates node and edge text specifically for dynamic graph generation. Against these existing benchmarks, TLP and TNC appear as fairly conventional prediction tasks. It remains unclear what new research questions TTGBench enables specifically because interactions carry text. The TNC formulation is particularly uncertain: Appendix A.1 derives user labels from the category of the item reviewed at each interaction. However, a change in item category does not necessarily imply a shift in user preference. Establishing what this task actually tests would require simple category-history or text-only baselines, or an independent measure of user state.
  2. Architecture-level conclusions are confounded by implementation asymmetries. TGNNs are tuned on TLP prior to TNC training, whereas adapted LLaGA and GraphGPT variants are trained for only few epoch, use a static graph without timestamps for grounding, and omit a key GraphGPT stage. Meanwhile, ICL methods receive mostly serialized structural triplets. These differences in training regimes and information budgets mean that Tables 2–3 reflect implementation choices rather than inherent capabilities. Consequently, the results cannot yet establish that TGNNs inherently miss semantics or that LLMs inherently miss structure. Similarly, given edge novelty rates of 0.99–1.00, EdgeBank’s near-chance performance is an expected artifact of the dataset regime rather than a meaningful failure.

Reference

[1] Zhang et al., DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphs (NeurIPS 2025)

[2] Peng et al., GDGB: A Benchmark for Generative Dynamic Text-Attributed Graph Learning (ICLR 2026)

Questions for authors
  1. Could you provide a direct comparison between TTGBench, DTGB, and GDGB regarding dataset provenance, overlap, node/edge text coverage and quality, temporal/structural properties, and evaluation protocols? Specifically, which concrete limitations of those benchmarks does each TTGBench dataset address?
  2. Why are TLP and TNC considered the most informative tasks for these text-attributed temporal graphs? Would a task requiring the prediction or generation of text, relations, or graph evolution more directly test the claimed interplay between semantic and structural dynamics? For the current TNC task, what evidence supports that item-category labels reflect an evolving user state rather than merely the category of the next chosen item? How do simple category-history and text-only baselines perform?
  3. For a given TNC query, which item descriptions, review texts, labels, and edges are visible prior to the target interaction? Can the static training graph include edges that occur after an individual training query?
10 comments
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 1

We sincerely appreciate the reviewer's constructive feedback and insightful questions. We have carefully revised the manuscript to address the reviewer's concerns, with all revisions highlighted in blue, and provide detailed responses to each point below.

Q1. Comparison of TTGBench with DTGB and GDGB

  1. Dataset comparison among TTGBench, DTGB, and GDGB
  • TTGBench provides substantially higher link novelty, richer real-world semantic labels, more informative textual attributes, and finer-grained temporal information (Table 1).
  • There is no dataset overlap among TTGBench, DTGB, and GDGB. GDGB’s IMDB is a movie-collaboration network, whereas TTGBench’s IMDB contains user–movie review interactions; we have verified that their underlying data sources do not overlap. DTGB’s Amazon Movies and TTGBench’s Amazon-Kindle both originate from the Amazon ecosystem, but differ in source releases, product domains, time spans, and filtering procedures, and therefore contain no dataset overlap.
  • TTGBench also provides higher-quality textual information. In Table 2, we follow GDGB and evaluate node/edge text quality using PPL and LLM Rating. TTGBench achieves better text-quality scores than the existing benchmarks. We further evaluate the performance gains from incorporating historical text. Compared with DTGB, TTGBench exhibits substantially larger improvements on both TLP and TNC, demonstrating the practical utility of its textual information.

Table 1. Basic dataset characteristics of the three benchmarks

DimensionDTGBGDGBTTGBench
Datasets and sourcesEnron, GDELT, ICEWS1819, Stack elec, Stack ubuntu, Googlemap CT, Amazon movies, YelpSephora, Dianping, WikiRevision, WikiLife, IMDB, WeiboTech, WeiboDaily, CoraFOOD, IMDB, Librarything, Beeradvocate, Amazon-Kindle, Ratebeer
Temporal informationMost datasets aggregate interactions into time intervalsMostly yearly/monthly/daily granularity or discrete time sequencesFine-grained timestamps for all datasets
Node text coverageMost source nodes have sparse or no text; most target nodes have rich text, while some have sparse or no textMost nodes have rich text; some have no text or only statistical/identifier informationAll target nodes have rich and informative text; some source nodes have no text
Edge text coverageOnly some edges contain text; several datasets have very short or low-quality edge textSome datasets contain empty or very short edge textAll edges contain rich and informative long-form text
Link novelty0.23–0.410.37–0.630.99–1.0
# semantic labelsMostly 1–5 rating levels or simple binary labels such as edited/not-edited or useful/not-usefulMostly 1–5 rating levels or simple binary labels such as comment/no-comment or repost/no-repostRich real-world semantic labels
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 2

Table 2. Data-quality comparison with DTGB and GDGB

Quality/Utility MetricDTGBGDGBTTGBench
Node-text PPL↓193.5875.9561.71
Edge-text PPL↓181.8288.1768.62
Node-text LLM Rating↑ (1–5)3.034.214.52
Edge-text LLM Rating↑ (1–5)3.554.174.48
Gain from historical text on TLP/TNCTLP: -2.71–2.54 Edge Classification: 0.08–2.81Generative tasksTLP: 0.68–16.06 TNC: 0.57–17.92
  1. Comparison of task design and evaluation protocols

Table 3 provides a detailed comparison of the task definitions and evaluation protocols of DTGB, GDGB, and TTGBench.

  • TTGBench’s TLP is directly comparable to DTGB’s Future Link Prediction.
  • TTGBench’s TNC is superficially related to DTGB’s Edge Classification but differs fundamentally in its setting. DTGB classifies a query edge (u,v,t)(u,v,t) and may use the textual attributes of both uu and vv, historical information, and the text associated with the current edge (u,v,t)(u,v,t). In contrast, TTGBench queries (u,t)(u,t) and predicts the future semantic label of uu using only information available before tt. The current node vv, its text, and the current edge text are unavailable. TNC is therefore a strictly causal forecasting task designed to evaluate whether models can infer temporal semantic labels from historical temporal-graph information.
  • Historical temporal-graph information is essential for TTGBench’s TNC but less important for DTGB’s Edge Classification. Table 4 compares each benchmark’s semantic classification task using historical graph information versus only current-query information. DTGB’s Edge Classification performs equally well or better from the current query alone, whereas TTGBench’s TNC suffers a substantial performance drop when historical graph information is removed.
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 3

Table 3. Comparison of task design and evaluation protocols among DTGB, GDGB, and TTGBench

BenchmarkTaskItemTask Design & Evaluation Protocols
DTGB1. Future Link PredictionInputG<t=V<t,E<t\mathcal{G}_{\lt t}=\\{\mathcal{V}_{\lt t},\mathcal{E}_{\lt t}\\}, containing all edges strictly before tt.
Query(u,v,t)(u,v,t); predict whether an edge (u,v)(u,v) occurs at tt.
OutputScore s(u,v)s(u,v) for binary classification.
2. Destination Node RetrievalInputG<t\mathcal{G}_{\lt t} and source node uu.
Query(u,V<t,t)(u,\mathcal{V}_{\lt t},t); rank candidate destination nodes.
OutputScore s(u,v)s(u,v) for each v∈Cv\in C and rank the ground-truth v⋆v^{\star}.
3. Edge ClassificationInputG<t\mathcal{G}_{\lt t}.
Query(u,v,t)(u,v,t); predict edge label lu,v∈Ll_{u,v}\in\mathcal{L}.
RepresentationConcatenate node representations: hu,v=[hu∥hv]\mathbf{h}_{u,v}=[\mathbf{h}_u\Vert\mathbf{h}_v].
Outputy^=softmax(MLP(hu,v))\hat{\mathbf{y}}=\mathrm{softmax}(\mathrm{MLP}(\mathbf{h}_{u,v})).
4. Textual Relation GenerationInputNode texts du,dvd_u,d_v and edge texts ru,⋅,r⋅,vr_{u,\cdot},r_{\cdot,v} from the most recent kk one-hop interactions of uu and vv, respectively.
Query(u,v,t)(u,v,t); generate future edge text r^u,v\hat r_{u,v}.
Outputr^u,v=LLM(Prompt(du,dv,ru,⋅,r⋅,v))\hat r_{u,v}=\mathrm{LLM}(\mathrm{Prompt}(d_u,d_v,\\{r_{u,\cdot}\\},\\{r_{\cdot,v}\\})).
GDGB1. Transductive Dynamic Graph Generation (TDGG)InputSeed graph G0=(N,E0)\mathcal{G}_0=(\mathcal{N},\mathcal{E}^{0}) and fixed source/target node sets Nsrc,Ndst\mathcal{N}_{\mathrm{src}},\mathcal{N}_{\mathrm{dst}}.
QueryAt each round k=1,…,Kk=1,\ldots,K, (u,Ndstk,t)(u,\mathcal{N}_{\mathrm{dst}}^{k},t); select a target and generate an edge (u,v,l^u,v,r^u,v,t^u,v)(u,v,\hat l_{u,v},\hat r_{u,v},\hat t_{u,v}).
2. Inductive Dynamic Graph Generation (IDGG)InputSeed graph G0=(N0,E0)\mathcal{G}_0=(\mathcal{N}^{0},\mathcal{E}^{0}) with dynamically expanding source/target node sets.
QueryAt each round kk, generate new nodes and then select destinations and generate edges.
TTGBench1. Temporal Link PredictionInputG<t=V<t,E<t\mathcal{G}_{\lt t}=\\{\mathcal{V}_{\lt t},\mathcal{E}_{\lt t}\\}, containing all edges strictly before tt.
Query(u,v,t)(u,v,t); predict whether an edge (u,v)(u,v) occurs at tt.
OutputP(e=(u,v,t)∣G<t)=σ(fθ(link)(u,v,G<t))P(e=(u,v,t)\mid\mathcal{G}_{\lt t})=\sigma\big(f_\theta^{(\mathrm{link})}(u,v,\mathcal{G}_{\lt t})\big).
2. Temporal Node ClassificationInputHistorical interaction graph G<t\mathcal{G}_{\lt t} and node uu.
Query(u,t)(u,t); predict the temporal semantic label expressed by uu at tt.
Output — Multi-classP(yu(t)=c∣u,G<t)=Softmax(fθ(node)(u,G<t))cP(y_u(t)=c\mid u,\mathcal{G}_{\lt t})=\mathrm{Softmax}\big(f_\theta^{(\mathrm{node})}(u,\mathcal{G}_{\lt t})\big)_c.
Output — Multi-labelP(Yu,c(t)=1∣u,G<t)=σ(fθ,c(node)(u,G<t))P(\mathbf{Y}_{u,c}(t)=1\mid u,\mathcal{G}_{\lt t})=\sigma\big(f_{\theta,c}^{(\mathrm{node})}(u,\mathcal{G}_{\lt t})\big).

Table 4. Historical temporal graph vs. current-query-only information for semantic classification in TTGBench and DTGB

BenchmarkTaskDatasetModelF1
DTGBEdge ClassificationEnronJODIE (w/ historical graph)64.78
JODIE (current query only)67.96
DyGFormer (w/ historical graph)56.04
DyGFormer (current query only)63.77
TTGBenchTNCIMDBJODIE (w/ historical graph)12.65
JODIE (current query only)0.76
DyGFormer (w/ historical graph)20.05
DyGFormer (current query only)0.93
Reviewer_UcewOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors claim that no benchmarks currently exist to measure the ability of methods to capture both structural and semantic dynamics of temporal graphs. The authors propose that TTGBench addresses this, comprising a comprehensive testbed for benchmarking models on predicting the structural and semantic dynamics of graphs. The authors then test this approach on existing methodologies (TGNN predictors and LLM predictors) to demonstrate that no single method effectively captures the semantics and structure of temporal graphs.

Critical strengths
  1. The authors present a benchmark that successfully addresses a gap in current benchmarks that reflects real-world problems
  2. The authors also benchmark a wide range of existing methods on the proposed benchmark, demonstrating fundamental weaknesses of existing approaches on this task
Critical weaknesses
  1. The writing is at times difficult to understand, with the authors arguing the meaning of results very briefly. Because of this, the value of the benchmarking work is hard to determine, leaving just the collection of datasets into one location. a. Additionally, some typos are present (Section 6.2, “Structural inertia…” paragraph and Section 1, third paragraph of page 2)
  2. No discussion or conclusions are given about the provided results. While the results are interesting, this makes it difficult to determine what the author’s main claims are about the results beyond the asymmetric poor performance of TGNNs and LLMs for this task.
Questions for authors

Why do the authors believe that LLM-as-enhancer architectures fail to recover semantics of dynamic networks, when LLM-predictors are able to do so much more effectively?

4 comments
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 1

We sincerely appreciate the reviewer's constructive feedback and insightful questions. We have carefully revised the manuscript to address the reviewer's concerns, with all revisions highlighted in blue, and provide detailed responses to each point below.

Q1. Why does LLM-as-Enhancer underperform LLM-as-Predictor on TNC?

Our experiments suggest that existing LLM-as-Enhancer methods underperform LLM-as-Predictor methods on TNC primarily because their architectures cannot effectively exploit semantic information. We identify three contributing factors:

  1. Existing LLM-as-Enhancer architectures are poorly suited to TNC. Table 1 compares the original linear-probing setting, where a TLP-pretrained backbone is frozen and only the TNC classification head is trained, with end-to-end TNC training of both the backbone and classification head. Neither method achieves substantial improvement with end-to-end training, suggesting that their limitations stem primarily from the inherited TGNN architectures and their difficulty in capturing temporal semantic dynamics.

    Table 1. TNC performance of LLM-as-Enhancer methods under linear probing and end-to-end training.

    MethodSettingIMDBBeeradvocate
    LKD4DyTAGLinear probing (original)8.516.14
    TNC end-to-end (new)8.667.70
    CROSSLinear probing (original)8.626.17
    TNC end-to-end (new)8.957.59
  2. Existing LLM-as-Enhancer methods make limited use of LLM-provided semantic information. Table 2 evaluates LKD4DyTAG after disabling its LLM-based teacher guidance, distillation, and input enhancement, and CROSS after removing its LLM-generated semantic summary branch. Removing these components yields only modest performance changes, indicating that the additional LLM-based semantic information provides limited benefits for TNC.

    Table 2. Effect of removing LLM-based semantic enhancement on TNC performance.

    MethodSettingIMDBBeeradvocate
    LKD4DyTAGFull (original)8.516.14
    w/o LLM guidance6.355.06
    CROSSFull (original)8.626.17
    w/o LLM summary8.446.10
  3. TGNN-based prediction limits the effective use of semantic information. Table 3 compares three prediction pathways under the same input information. A simple MLP slightly outperforms the original TGNN-based predictor, while direct LLM prediction achieves substantially higher performance. This contrast suggests that the historical aggregation and prediction mechanisms inherited from TGNNs may constrain the effectiveness of existing LLM-as-Enhancer methods on TNC.

    Table 3. TNC performance of different prediction pathways under identical input information.

    PathPredictorIMDBBeeradvocate
    CROSS (original)TGNN-based fusion and classification head8.626.17
    Semantic probeLightweight MLP8.786.20
    Direct LLMLLM-as-Predictor17.7812.05

These findings characterize the limitations of existing implementations of LLM-as-Enhancer on TNC, without ruling out the potential of this paradigm. They also motivate the development of new LLM-as-Enhancer architectures that can more effectively integrate semantic information into temporal graph modeling.

Q2. Clarifying the paper's narrative and deepening the experimental results analysis

We thank the reviewer for pointing out weaknesses in the paper's presentation and analysis. We have reorganized the paper and expanded our analysis of the experimental results to better articulate the motivation, distinctiveness, and research implications of TTGBench.

Core Research Question: Can models reliably predict the structural and semantic evolution arising from temporal interactions in text-attributed temporal graphs? How well do existing methods capture these dynamics, what mechanisms explain their limitations, and what insights can guide future model development?

Logical Flow: Temporal Interaction Dynamics → Coupled Structural and Observable Semantic Evolution → Dual Volatility → Learnable Temporal Dependencies and Cross-Dimensional Predictive Evidence → Modeling Challenges → TTGBench (TLP + TNC + Controlled Analyses) → Empirical Findings and Research Implications.

We elaborate on the key elements of this research narrative below.

AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 2

  1. Research Motivation: Existing benchmarks provide limited joint evaluation of high structural and semantics dynamics. This limitation may lead to an underestimation of the challenges in temporal graph modeling. Table 4 compares TTGBench with existing text-attributed temporal graph benchmarks. TTGBench introduces high link novelty, requiring models to learn temporal patterns beyond memorizing historical connections. It also provides a dedicated task for evaluating semantic evolution, enabling systematic investigation of the interplay between structural and semantic dynamics.

    Table 4. Evaluation gaps in existing benchmarks and the additional coverage provided by TTGBench.

    BenchmarkStructural Dynamics TasksSemantic Dynamics TaskLink NoveltyRemarks
    DTGBFuture Link Prediction, Destination Node RetrievalNo equivalent prediction protocol0.23–0.41Edge Classification involves semantic labels but does not directly evaluate semantic evolution.
    GDGBTemporal graph generation (TDGG / IDGG)No equivalent prediction protocol0.37–0.63Structural and textual generation are valuable but serve different evaluation objectives.
    TTGBenchTLPTNC0.99–1.00Discriminative tasks enable objective evaluation of structural and semantic evolution.
  2. Task Formulation: TLP and TNC provide complementary prediction objectives for interaction-driven structural and semantic evolution. Each interaction ei=(u,vi,ti,xi)e_i=(u,v_i,t_i,x_i) establishes an edge (u,vi)(u,v_i) and expresses an observable semantic category, thereby contributing to structural and semantic dynamics. TLP and TNC assess these two dimensions through well-defined discriminative tasks, enabling objective comparisons and systematic analysis of model limitations.

    Table 5. Complementary task formulations for evaluating temporal interaction dynamics.

    TaskQueryUnobserved InformationPrediction TargetMetricsEvaluated Capability
    TLP(u,v,t)(u,v,t)Whether the future edge occursP((u,v,t) occurs∣G<t)P((u,v,t)\text{ occurs}\mid\mathcal G_{<t})AP / AUROC / MRRPredicting temporal link formation from historical dynamics
    TNC(u,t)(u,t)Current destination vtv_t and its item/interaction textP(yu(t)∣u,G<t)P(y_u(t)\mid u,\mathcal G_{<t})Macro-F1 / bACCPredicting the interaction-induced temporal semantic label
  3. Benchmark Construction: Six datasets capture complementary dual-volatility conditions. As shown in Table 6, TTGBench combines high link novelty with varying degrees of semantic drift, dataset scales, and textual characteristics. Its TNC datasets support both multi-label and multi-class classification, providing diverse structural, temporal, and semantic evolution patterns.

    Table 6. Dual-volatility characteristics of TTGBench datasets.

    DatasetLink NoveltyLabel DriftSemantic Label Setting / Characteristics
    FOOD1.000.5155Multi-label; moderate drift
    IMDB1.000.6639Multi-label; relatively high drift
    Librarything1.00—High-novelty TLP with diverse textual information
    Beeradvocate0.990.7431Multi-class; high drift
    Amazon-Kindle1.000.4622Multi-class; class imbalance and large scale
    Ratebeer1.000.7373Multi-class; high drift
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 3

  1. Temporal Mechanisms: Dual volatility exhibits distinct graph–temporal evolution patterns and learnable temporal dependencies. We investigate the mechanisms underlying dual volatility through graph–temporal frequency analysis, category-transition baselines, and temporal ablations. Table 7 summarizes the dominant structural and semantic frequency bands, revealing dataset-specific evolution patterns that require models to capture different spatial and temporal scales. Table 8 further demonstrates that TNC labels contain learnable temporal dependencies.

    Table 7. Illustrative summary of dominant graph–temporal evolution scales.

    DatasetDominant Structural ScaleDominant Semantic ScaleModeling Implications
    IMDBLow temporal × Medium graph frequencyHigh temporal × High graph frequencyStructural dynamics require longer-term context, while semantic dynamics demand faster, more heterogeneous tracking.
    BeeradvocateLow temporal × High graph frequencyMedium temporal × Low/Medium graph frequencyStructural and semantic dynamics require different frequency-scale weights.
    Amazon-KindleLow temporal × Low graph frequencyLow temporal × Low graph frequencySlowly evolving, spatially smooth historical patterns may be particularly informative.
    LibrarythingHigh temporal × High graph frequency—TLP requires sensitivity to rapid, spatially heterogeneous structural changes.

    Table 8. Empirical evidence of temporal dependencies in TNC.

    MethodIMDBBeeradvocatePrediction Mechanism
    Global Frequency1.105.13Estimate p^(c)\hat p(c) from training data and predict arg⁡max⁡cp^(c)\arg\max_c\hat p(c).
    Global Markov2.366.60Estimate p^(c∣yu,i−1)\hat p(c\mid y_{u,i-1}) and predict the most probable category.
    User Frequency1.465.02Estimate p^(c∣u)\hat p(c\mid u) and predict arg⁡max⁡cp^(c∣u)\arg\max_c\hat p(c\mid u).
    User Markov3.025.16Estimate p^(c∣yu,i−1,u)\hat p(c\mid y_{u,i-1},u) and predict arg⁡max⁡cp^(c∣yu,i−1,u)\arg\max_c\hat p(c\mid y_{u,i-1},u).
    DyGFormer w/o time15.006.39Shuffle historical timestamps.
    DyGFormer Full20.0512.91Preserve the original temporal information.

    The Markov baselines consistently outperform their frequency-based counterparts, indicating that previous interaction categories provide predictive information beyond static category distributions. Moreover, DyGFormer's performance decreases substantially when historical timestamps are shuffled, confirming the predictive value of temporal information.

  2. Structural–Semantic Interplay: Cross-dimensional ablations quantify the predictive contributions of semantic, structural, and temporal information. We extend the analysis beyond independent TLP and TNC scores to investigate whether information from one dimension improves prediction in the other. As shown in Table 9, historical semantic information benefits structural prediction, cross-node graph context improves semantic prediction, and temporal information contributes to both tasks.

    Table 9. Cross-dimensional and temporal ablation results on IMDB.

    Research QuestionControlled ComparisonFullAblationGain
    Does historical semantic information improve structural prediction?TLP: Full vs. w/o semantics58.4252.96+5.46
    Does cross-node graph structure improve semantic prediction?TNC: Full vs. w/o cross-node graph context20.0514.57+5.48
    Does temporal information improve structural prediction?TLP: Full vs. w/o time58.4248.60+9.82
    Does temporal information improve semantic prediction?TNC: Full vs. w/o time20.0515.00+5.05
AuthorsOpenReview ↗

Response to Reviewer's Concerns - Part 4

  1. Main Findings: TGNN-Predictors and LLM-Predictors exhibit a consistent capability asymmetry across structural and semantic prediction. Table 10 demonstrates that TGNN-Predictors achieve stronger TLP performance, whereas LLM-Predictors perform better on TNC. Both tasks nevertheless remain challenging, revealing complementary limitations in the two paradigms.

    Table 10. Performance asymmetry between the best TGNN-Predictors and LLM-Predictors on TLP and TNC.

    DatasetBest TGNN-Predictor: TLPBest LLM-Predictor: TLPBest TGNN-Predictor: TNCBest LLM-Predictor: TNC
    IMDB69.7951.1620.0532.93
    Beeradvocate87.0152.6912.9118.10
  2. Mechanistic Explanation: The two paradigms exploit historical textual information differently, while existing LLM-as-Enhancer architectures struggle to translate semantic information into effective TNC predictions. Table 11 reveals task-dependent differences in how the two paradigms utilize historical interaction text. TGNN-Predictors obtain substantially greater textual benefits on TLP, whereas LLM-Predictors benefit more on TNC. Additional experiments (Tables 1–3) further suggest that existing LLM-as-Enhancer methods inherit limitations from TGNN-based temporal aggregation and prediction mechanisms, constraining their ability to exploit semantic information for TNC.

    Table 11. Performance gains from historical interaction text across paradigms and tasks.

    ParadigmTLP Gain from TextTNC Gain from TextKey Implication
    TGNN-Predictors+12.67+1.99Historical text contributes more strongly to structural prediction.
    LLM-Predictors+2.59+9.65Historical text contributes more strongly to semantic prediction.
  3. Research Implications: Future models should jointly exploit multi-scale temporal dependencies, cross-dimensional predictive information, and computationally efficient semantic representations. Based on the preceding analyses and Figure 4, we identify three overarching directions: unified structural–semantic modeling, adaptive temporal-scale selection, and effective yet efficient use of textual information. Table 12 summarizes the empirical evidence motivating these directions.

    Table 12. Research implications derived from the empirical findings.

    Empirical EvidenceIdentified Modeling RequirementFuture Research Direction
    Complementary TLP/TNC performance and cross-dimensional gains (Tables 9 and 10)Jointly model structural and semantic prediction objectives.Unified structural–semantic predictors
    Graph–temporal frequency patterns, Markov baselines, and temporal ablations (Tables 7 and 8)Capture dataset-specific temporal scales and predictive dependencies.Adaptive encoding of short- and long-term dependencies
    Paradigm-dependent textual gains (Table 11)Exploit textual information according to task-specific requirements.Task-aware semantic integration
    Limited gains from existing LLM-as-Enhancer architectures (Tables 1–3)Improve the effectiveness of semantic fusion and prediction mechanisms.Better integration of LLM guidance with temporal graph aggregation
    Dataset diversity in scale, drift, and novelty (Table 6)Adapt to heterogeneous data conditions while controlling computational costs.Transferable, efficient, and robust model designs

Taken together, these analyses clarify TTGBench's contribution as a benchmark for systematically evaluating the predictability, temporal mechanisms, and modeling challenges of coupled structural and semantic evolution. The resulting findings provide concrete empirical guidance for developing temporal graph models capable of addressing both dimensions effectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.