acceptodds
Under review as a conference paper at ICLR 2027

Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

Abstract

Relational deep learning represents database rows and foreign-key links as a heterogeneous graph, enabling prediction from record attributes and relational context. Two forms of temporal signals play distinct roles in these graphs: record age changes as the prediction cutoff advances, whereas the interval between two observed records remains fixed. Prior work has explored temporal modeling and temporal pretraining for heterogeneous graphs, but typically treats time as a single source of information or focuses on either representation or supervision in isolation. We investigate how explicitly representing both temporal signals affects the benefits of temporal pretraining on downstream tasks. Our framework pairs two complementary temporal encodings: Multi-scale Time Encoding, which captures record age through learnable time scales and type-specific projections, and Rotary Time Encoding, which encodes signed time differences between linked records through rotary transformations during graph propagation. Rather than treating these encodings as architectural additions alone, we train them through three self-supervised objectives: recovering historical relations, forecasting horizon-dependent future relation activity, and contrasting temporally valid historical subgraphs. All model inputs are restricted to information available at their observation cutoffs. We further structure pretraining into two stages: learning neighborhood representations via subgraph contrastive learning, followed by refining these representations through either relation recovery or future activity prediction. We evaluate across five RelBench datasets and 11 classification and regression tasks using representative heterogeneous gnn and gt backbones. With both temporal encodings, the best evaluated staged schedules for the two backbones outperform direct supervised training using the same encodings by and , respectively, and controls without pretraining or encodings by and , respectively.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Relational Deep Learning, Temporal Graph Learning, Self-Supervised Learning, Heterogeneous Graphs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tEiROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for pretraining on relational deep learning (RDL) graphs that combines temporal encoding with self supervised learning. Temporal encodings are used to encode two signals available as features in the relational database, and are complemented by three pre-training objectives that exploit label-free examples to learn patterns from these representations. The temporal encodings are TimeMix and TimeRoPE. They respectively encode record age with multi-resolution time scales, with type-specific projections for each table, and the interval between two records via a reformulation of rotary positional encodings (which encode their relative time differences). In addition, the framework uses three complementary pre-training objectives: temporal subgraph contrast (Temp-Sub), historical relation recovery (Rel-Hist), and horizon-aware future relation activity prediction (Rel-Future). These objectives are employed with a staged schedule.

The authors evaluate their approach on 11 tasks from five RelBench databases using two different backbones in their pre-training. Results report gains over supervised training and other nine pre-training baselines.

Critical strengths
  • The paper clearly separates two temporal signals in relational data (record age and inter-record interval), with a dedicated positional encoding for each. This is a positive point since the pretraining objectives are aware of this heterogeneity and cover complementary aspects (subgraph, histories, and future activity).
  • The experimental setting is well presented, and the authors validate their method on multiple databases and task types with two families of backbone models. They are also transparent about mixed results and limitations, and the supplementary material (code, implementation and methodological details) supports reproducibility.
Critical weaknesses
  • Staging and temporal design are not isolated empirically. There is no joint-loss baseline or reversed stage order, so the two-stage schedule is not compared against joint training or alternative ordering. The DayPE ablation also lacks a Ktime=1K_{time}=1 variant, and it is not shown that the temporal design of Rel-Hist and Rel-Future matters. To me this is important because many design choices (staging, multi-horizon sampling, multi-scale age) are only supported indirectly, so it is unclear which components drive the gains.
  • The presentation and analysis of when the method helps are limited. The experimental discussion and the methodological narrative could be easier to follow, and the gains are uneven. The mean improvements are mostly driven by regression tasks, while many classification gains are small (as can be inferred from Fig. 3), with no analysis of why. There is also no complexity or runtime analysis of the encodings and pre-training objectives.
Questions for authors

In addition, for each question below, please refer to the Weaknesses:

  • Can you report a joint-training baseline (Temp-Sub + Rel-Hist/Rel-Future with a summed loss) under the same settings, and the reversed stage order?
  • Can you report TimeMix when Ktime=1K_{time}=1 versus the chosen default value?
  • Can you discuss the method time and memory complexity, reporting empirical values as well?
2 comments

Thank you for your kind and comprehensive review! We address your questions below.

For easier access, we have now combined the main body and appendix into a single PDF and uploaded it. The main body remains unchanged in this revision; the only content additions are Appendix Sections F and G, which report additional experiments and analyses.

Additional Sections in this revisionContents
Appendix F, pp.25–26Joint pretraining and reverse-order pretraining schedule comparisons
Appendix G, pp.26–29Parameters and complexity; Table 12 (p.27), Table 13 (p.28), and Table 14 (p.29) report parameters, resource changes, and absolute database-level costs

1. Joint training and reversed stage order

Appendix F, Tables 10–11, now includes both requested comparisons. Joint-Hist and Joint-Future share the backbone and directly sum the complete Temp-Sub and Rel-Hist or Rel-Future losses, using independently sampled batches and one shared parameter update. The reversed schedules train Rel-Hist or Rel-Future before Temp-Sub (TS).

These additional comparisons retain the main study's three-layer graph models: hidden width 128, ResNet table encoders, fanouts (128,64,32), and eight attention heads for THGFM, on the same 11 tasks and chronological splits. The following results use TimeMix + TimeRoPE and report mean type gain over the corresponding supervised control B0:

BackboneRelation objective RR aloneJoint TS + RR → TSTS → R
HeteroGNNRel-Hist+1.14%+2.71%+0.65%+3.02%
HeteroGNNRel-Future+3.00%+0.81%+0.44%+2.83%
THGFMRel-Hist−0.69%−0.34%−1.29%+0.67%
THGFMRel-Future−0.52%−1.55%−0.29%+1.06%

Across all eight comparisons—two backbones × two encodings × two relation objectives—TS-first achieves higher mean type gain and mean regression gain than both joint training and the reversed order. This directly supports learning stable historical-neighborhood representations before refining them through relation prediction. On THGFM with both encodings, it also turns negative transfer from either relation objective alone into positive gains.

With both encodings, the mean type gain margins over joint training are 0.31, 2.02, 1.01, and 2.61 percentage points, respectively, in the table above. Holding the objective pair, backbone, and encoding fixed, these comparisons directly establish that the organization of supervision affects downstream representation quality.

TS-first exceeds the corresponding single relation objective in seven of eight comparisons. The HeteroGNN Rel-Future case (+3.00% alone versus +2.83% staged) identifies a configuration where the single objective is already effective. Our strongest consistent finding concerns the advantage over joint optimization and reversed ordering.

2. The K=1 reference and temporal encoding design

DayPE provides the single-scale (K=1) reference in our comparison: record ages are normalized by one fixed day scale, whereas default TimeMix uses four learnable scales. Here K counts temporal normalization scales. TimeMix initializes them at one second, minute, hour, and day, then learns their scales and mixture weights with type-specific projections.

Figure 4a compares DayPE + TimeRoPE against TimeMix + TimeRoPE without pretraining, holding interval encoding fixed. TimeMix improves mean type gain by 0.59 percentage points on HeteroGNN and 1.18 on THGFM. Holding TimeMix fixed, adding TimeRoPE contributes another 0.28 and 0.46 points, respectively. Their combination achieves the best aggregate encoding result on both backbones.

Appendix B.2 explains the data motivation: Avito contains variation at seconds-to-hours scales, Event and H&M predominantly span days to months, and Arxiv and F1 include years of history. The encoding comparison establishes the benefit of replacing the fixed single-scale representation with the adaptive multiscale representation.

The relation objectives also align supervision with time: Rel-Hist recovers hidden links from context available when the links were observed; Rel-Future predicts relation occurrence and counts over a specified future horizon from historical inputs. Appendix F tests these complete temporal objectives under different compositions and orders. Isolating individual horizon or count components is a finer-grained question than the objective-level effects established by these comparisons.

3. When the method helps

The most consistent improvements occur when historical relational context informs future quantities or rates. Our proposed schedules improve 8/11 HeteroGNN tasks and 7/11 THGFM tasks over A0, including all four regression tasks on both backbones; mean regression gains are +5.12% and +8.55%. In Appendix F, both TS-first schedules achieve +4.87%–+7.40% mean regression gains across the four backbone–encoding settings, exceeding their joint and reversed counterparts.

Regression tasks such as H&M item-sales and Avito ad-ctr predict future quantities or rates from the target entity's historical relational context. This aligns with learning that context through TS before refining it with relation-based supervision. For classification, Avito user-clicks, Avito user-visits, and H&M user-churn define labels through the occurrence or frequency of interactions within a future window, making temporal relation patterns directly relevant. With TimeMix + TimeRoPE, both schedules improve mean binary classification performance by +0.60%+0.60\%--+1.91%+1.91\% on HeteroGNN and +0.32%+0.32\%--+0.39%+0.39\% on THGFM over matched supervised controls. On HeteroGNN, both schedules also improve Arxiv author-category accuracy by +0.48%+0.48\%--+1.35%+1.35\%, giving the combined design positive gains across all three task types. Alongside type-balanced gains, Figure 3 reports task-balanced average ranks: our two staged schedules occupy the top two positions on each backbone with both encodings.

4. Parameters, complexity, and measured resources

Appendix G shows that our encodings reduce trainable parameters. Let T count timestamped node types, d be hidden width, K the number of scales, N_t the timestamped nodes in a batch, and E_ℓ the edges processed by rotation-enabled paths at layer ℓ.

OperatorTrainable parametersForward computationIntermediate activations
DayPE age branchT(d²+d)O(N_t d²)—
TimeMix2TKd+2KO(N_t Kd)O(N_t Kd)
TimeRoPE0O(d Σℓ E_ℓ)O(d Σℓ E_ℓ)

At d=128 and K=4, replacing DayPE with TimeMix removes 15,488T−8 parameters, or approximately 0.02M–0.09M per backbone across the five databases. TimeRoPE uses fixed frequency buffers and adds no trainable parameters or optimizer states.

The resource analysis in Appendix G uses the same three-layer graph-model configuration. All values cited below are three-layer results. Table 12 (p.27) reports parameters, Table 13 (p.28) summarizes paired resource changes, and Table 14 (p.29) gives absolute time and memory by database, model, and objective. Median paired overheads of TimeMix + TimeRoPE relative to DayPE are:

BackboneWall timeReserved GPU memory
HeteroGNN+15.29%+12.06%
THGFM+16.10%−0.04%

The resource comparison uses batch size 256, and 50,000 evaluations per objective: joint training performs 50,000 updates evaluating both objectives; sequential training performs 50,000 updates per stage. We report pretraining wall-clock time, including online sampling, target preparation, data transfer, forward and backward passes, and optimizer updates. Offline feature and graph-cache construction and downstream fine-tuning are excluded. Memory is the PyTorch reserved peak in GiB, summed across devices when multiple GPUs are used. Sequential costs sum the stage durations and take the larger stage memory footprint.

The updated Table 13 separates joint/sequential resource comparisons by backbone, encoding, and objective. For three-layer models with both encodings, the results are below.

ComparisonBackboneWall-time changeReserved GPU-memory change
Joint-Hist / Seq-Hist(TS->Rel-HIST)HeteroGNN+7.24%+6.49%
Joint-Hist / Seq-Hist(TS->Rel-HIST)THGFM−15.91%+8.92%
Joint-Future / Seq-Future(TS->Rel-FUTURE)HeteroGNN−27.64%+28.20%
Joint-Future / Seq-Future(TS->Rel-FUTURE)THGFM−22.78%+37.16%

Together with Appendix F, TS-first achieves higher mean type gain and lower reserved memory under the paired comparison in all four full-encoding settings. For HeteroGNN Hist, sequential training achieves +3.02% gain versus +2.71% joint, while joint training costs 7.24% more time and 6.49% more memory at the paired median. Joint-Future is faster, but its paired-median memory cost is 28.20% / 37.16% higher on the two backbones, with lower downstream gains than TS-first.

Sequential training retains one objective's computation graph at a time, explaining its memory advantage. Table 14 gives a concrete example: on three-layer F1 HeteroGNN with both encodings, Seq-Hist uses 57.74 h / 191.63 GiB versus Joint-Hist's 57.10 h / 322.68 GiB—40.6% less memory for 1.1% more time. The summary and database-level results make both performance and resource tradeoffs explicit.

Reviewer_MnzHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper adds two temporal encodings to relational deep learning via pre-training staging. TimeMix encodes a record's age at the prediction cutoff using learned time scales. TimeRoPE encodes the signed time gap between linked records. The models are pretrained in two stages: subgraph contrast (TS) then historical link recovery (Rel-Hist) or future-activity prediction (Rel-Future). This staging methodology beats supervised training with the same encodings.

Critical strengths
  • The paper provides a novel temporal encoding which combines two complementary types of temporal information. TimeMIX and TimeRoPE. They ensure no leakage in temporal pretraining and, through learning, allow time encodings to depend on different tables. Including this temporal encoding improves performance on RDL across many tasks.
  • Provides a staged pre-training recipe with Rel-Hist and Rel-Future that includes these encodings within the pre-training. Pretraining under this recipe provided the largest gains compared to the single-objective counterpart.
Critical weaknesses
  • Pre-training for transfer learning is a more relevant evaluation. It is perhaps unsurprising that providing a more in-depth pre-training recipe provides better performance; as such, it would a much stronger argument if such a recipe provided better transfer results. While this is noted as a limitation, a somewhat simple pre-training on their data and testing on OOD target databases would show these results.

NOTE: Illustrations are difficult to follow, in particular Figure 3 and Figure 4a).

Questions for authors

There are some references to the appendix but no appendix? This would be important for reproducibility.

1 comment

Thank you for recognizing the complementary temporal encodings and the staged pretraining recipe.

The original submission already included the appendix in the supplementary materials. For easier access, we have now combined the main body and appendix into a single PDF and uploaded it. The main body remains unchanged in this revision; the only content additions are Appendix Sections F and G, which provide additional objective-composition and stage-order experiments, and parameter, computational, and training-resource analyses, respectively.

1. Schema-agnostic methods and cross-database checkpoint transfer

Our temporal encodings and pretraining strategies are schema-agnostic in their formulation: they use record ages, inter-record intervals, historical neighborhoods, and relation activity, without requiring particular table names, columns, or aligned business semantics. TimeMix applies the same multiscale age representation; TimeRoPE applies the same interval-based rotation; TS, Rel-Hist, and Rel-Future construct supervision using the current database's types, relations, and timestamps. Evaluation on five databases tests their applicability across different schemas and tasks.

Transferring one pretrained checkpoint to an unseen database additionally requires the backbone to map parameters across schemas. The evaluated HeteroGNN and THGFM instantiate several schema-dependent modules:

ModuleParameter dependence
Table encoderA separate encoder for each node type/table
Column encodersColumn counts, types, vocabularies, and feature dimensions
Graph propagationModules instantiated by layer and edge type
Graph normalizationLayerNorm instantiated by layer and node type
Node-time mappingsProjections instantiated by node type

A new schema changes these module sets, input dimensions, and parameter meanings. Consequently, cross-database checkpoint transfer introduces a backbone adaptation problem, including parameter sharing or initialization for unseen types.

Our study establishes temporal representation and self-supervision methods applicable to different schemas, using database-level pretraining followed by task-specific fine-tuning within each database. A backbone supporting cross-schema parameter sharing can further adopt these methods for cross-database pretraining and transfer. This motivates our subsequent relational foundation-model architecture.

2. Appendix access and reproducibility

The revised PDF contains Appendices A–H, with the contents on page 13. The original experimental settings, implementation details, and training curves can now be consulted alongside the new evidence in F and G in the same document:

Location in the current complete PDFContents
Appendix A, p.14Databases, tasks, statistics, chronological splits
Appendix B, pp.15–20Backbone selection, encoding ablations, task-level results and standard deviations
Appendix C, pp.20–22Architecture, optimization, budgets, checkpoint rules
Appendix D, p.22 onwardTimeMix scales and TimeRoPE implementation
Appendix E, pp.23–25Sampling, decoders, and complete losses
Appendix F, pp.25–26Joint pretraining and reverse-order pretraining schedule comparisons
Appendix G, pp.26–29Parameters and complexity; Table 12 (p.27), Table 13 (p.28), and Table 14 (p.29) report parameters, resource changes, and absolute database-level costs
Appendix H, p.29 onwardPretraining and fine-tuning curves

These locations refer to the combined version now uploaded, making the settings and additional evidence directly accessible.

3. What Figures 3 and 4a show

Figure 3 shows the overall effects of temporal encoding and staged pretraining across backbones and downstream tasks. With both encodings, HeteroGNN TS → Rel-Hist and THGFM TS → Rel-Future achieve mean type gains of +3.24% and +2.37% over their DayPE supervised controls. Our two staged schedules occupy the top two task-balanced average-rank positions among the full-encoding configurations shown for each backbone. The representative schedules improve 8/11 and 7/11 tasks, including all four regression tasks on both backbones. Comparisons in Appendix - Section F further demonstrates the effectiveness of the schedule we propose.

Figure 4a shows that TimeMix and TimeRoPE are complementary: their combination achieves the best aggregate encoding result on both backbones. This ablation uses no pretraining, directly evaluating the contribution of temporal representation. Holding TimeRoPE fixed, replacing TimeMix with single-scale DayPE reduces mean type gain by 0.59 and 1.18 percentage points on HeteroGNN and THGFM, respectively. Holding TimeMix fixed, removing TimeRoPE reduces it by 0.28 and 0.46 points. These results support jointly representing record age and inter-record intervals.

Reviewer_o9rgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to use time when a relational database is modeled as a graph. It encodes two time signals: how old each record is at prediction time, and the time gap between two linked records. It also proposes three pretraining tasks: predict hidden past links, predict future link activity, and contrastive learning on corrupted past neighborhoods. On 11 RelBench tasks, the best schedule improves over a supervised baseline by 3.24% with a GNN and 2.37% with a graph transformer.

Critical strengths
  1. Separating record age from the time gap between linked records is well motivated.
  2. The baseline set is broad: nine pretraining methods, each also run with contrastive pretraining first.
Critical weaknesses
  1. Gains over strong baselines are modest. For example, on the GNN, the best schedule improves mean type gain over AttrMasking by only 0.28 percentage points (+3.24% vs. +2.96%).
  2. The best schedule differs by backbone, and the selection protocol is unclear. Moreover, Rel-Future alone slightly outperforms its staged variant on the GNN (+3.00% vs. +2.83%).
Questions for authors
  1. How are schedules selected?
  2. Does a supervised model trained for the same total number of steps as the pretrained models close the performance gap?
1 comment

Thank you for recognizing the motivation for the two temporal signals and the breadth of the pretraining baselines. We address schedule selection, the size of the gains, and training duration below.

The main body remains unchanged in this revision; the only content additions are Appendix Sections F and G, which report additional experiments and analyses.

Additional Sections in this revisionContents
Appendix F, pp.25–26Joint pretraining and reverse-order pretraining schedule comparisons
Appendix G, pp.26–29Parameters and complexity; Table 12 (p.27), Table 13 (p.28), and Table 14 (p.29) report parameters, resource changes, and absolute database-level costs

For easier access, we have now combined the main body and appendix into a single PDF and uploaded it.

1. Assessing gains over strong baselines and schedule selection

The HeteroGNN margin you identify is 0.28 percentage points: +3.24% versus AttrMasking's +2.96%. The broader evidence includes gains over the same-encoding supervised controls, results on the transformer family, and task-balanced ranks:

ComparisonHeteroGNNTHGFM
Our representative staged schedule versus DayPE supervised control A0+3.24%+2.37%
Strongest external pretraining baseline versus A0+2.96% (AttrMasking)+1.26% (TS → TVE)
Our representative staged schedule versus full-encoding supervised control B0+3.02%+1.06%

The B0 comparison holds the backbone and TimeMix + TimeRoPE fixed, isolating the additional value of self-supervised pretraining. The THGFM margin over the strongest external baseline is approximately 1.11 points.

The advantage also appears when all 11 tasks receive equal weight. In Figure 3's full-encoding setting, our two schedules occupy the top two average-rank positions on each backbone: 7.14 / 7.27 on HeteroGNN, versus 10.50 for AttrMasking, and 6.50 / 6.45 on THGFM. The representative schedules improve 8/11 and 7/11 tasks over A0, including all four regression tasks on both backbones.

Appendix F further tests whether the organization of supervision matters. These comparisons retain the main study's three-layer graph models, with hidden width 128, ResNet table encoders, fanouts (128,64,32), and eight attention heads for THGFM. With both encodings, the gains over matched supervised controls B0 are:

Backbone / relation objectiveJoint PretrainingReversed orderTS-first
HeteroGNN / Rel-Hist+2.71%+0.65%+3.02%
HeteroGNN / Rel-Future+0.81%+0.44%+2.83%
THGFM / Rel-Hist−0.34%−1.29%+0.67%
THGFM / Rel-Future−1.55%−0.29%+1.06%

The mean type gain margins of TS-first over joint training range from 0.31 to 2.61 percentage points. Holding the objective pair fixed, these comparisons support a concrete training principle: learn historical-neighborhood representations first, then refine them through relation prediction.

2. Why does Rel-Future alone slightly exceed its staged variant?

On HeteroGNN with both encodings, Rel-Future alone achieves +3.00% versus +2.83% staged, relative to B0. The single future objective is already effective in this configuration; the benefit of staging depends on the objective and backbone. Across both backbones, encodings, and relation objectives, TS-first improves over the single relation objective in seven of eight comparisons. HeteroGNN Rel-Hist increases from +1.14% to +3.02%, while THGFM Rel-Future increases from −0.52% to +1.06%. Together with the joint and reverse-order results, this establishes a more consistent organization of complementary supervision while retaining the observed configuration dependence.

3. Could the gains be explained by more training steps?

The main study matches updates across pretraining methods and uses the same downstream supervision and checkpoint-selection protocol for every model. Single-objective pretraining receives 100,000 updates; two-stage pretraining receives 50,000 per stage. All models, including direct supervised controls, receive the same 100,000-step SFT budget.

The training curves provide evidence that downstream undertraining is not the explanation for several of these comparisons. In the current PDF, the curves are in Appendix H. For example, the full-encoding HeteroGNN supervised control reaches its best validation performance at approximately epochs 6, 2, and 3 on paper-citation, user-ignore, and user-churn, respectively; continued training does not exceed these values. Each SFT epoch comprises 10,000 steps. These trajectories support convergence within the chosen supervised budget and the value of self-supervised initialization under the same downstream optimization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.