Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

Relational deep learning represents database rows and foreign-key links as a heterogeneous graph, enabling prediction from record attributes and relational context. Two forms of temporal signals play distinct roles in these graphs: record age changes as the prediction cutoff advances, whereas the interval between two observed records remains fixed. Prior work has explored temporal modeling and temporal pretraining for heterogeneous graphs, but typically treats time as a single source of information or focuses on either representation or supervision in isolation. We investigate how explicitly representing both temporal signals affects the benefits of temporal pretraining on downstream tasks. Our framework pairs two complementary temporal encodings: Multi-scale Time Encoding, which captures record age through learnable time scales and type-specific projections, and Rotary Time Encoding, which encodes signed time differences between linked records through rotary transformations during graph propagation. Rather than treating these encodings as architectural additions alone, we train them through three self-supervised objectives: recovering historical relations, forecasting horizon-dependent future relation activity, and contrasting temporally valid historical subgraphs. All model inputs are restricted to information available at their observation cutoffs. We further structure pretraining into two stages: learning neighborhood representations via subgraph contrastive learning, followed by refining these representations through either relation recovery or future activity prediction. We evaluate across five RelBench datasets and 11 classification and regression tasks using representative heterogeneous gnn and gt backbones. With both temporal encodings, the best evaluated staged schedules for the two backbones outperform direct supervised training using the same encodings by $3.02\%$ and $1.06\%$, respectively, and controls without pretraining or encodings by $3.24\%$ and $2.37\%$, respectively.