- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for pretraining on relational deep learning (RDL) graphs that combines temporal encoding with self supervised learning. Temporal encodings are used to encode two signals available as features in the relational database, and are complemented by three pre-training objectives that exploit label-free examples to learn patterns from these representations. The temporal encodings are TimeMix and TimeRoPE. They respectively encode record age with multi-resolution time scales, with type-specific projections for each table, and the interval between two records via a reformulation of rotary positional encodings (which encode their relative time differences). In addition, the framework uses three complementary pre-training objectives: temporal subgraph contrast (Temp-Sub), historical relation recovery (Rel-Hist), and horizon-aware future relation activity prediction (Rel-Future). These objectives are employed with a staged schedule.
The authors evaluate their approach on 11 tasks from five RelBench databases using two different backbones in their pre-training. Results report gains over supervised training and other nine pre-training baselines.
Critical strengths
- The paper clearly separates two temporal signals in relational data (record age and inter-record interval), with a dedicated positional encoding for each. This is a positive point since the pretraining objectives are aware of this heterogeneity and cover complementary aspects (subgraph, histories, and future activity).
- The experimental setting is well presented, and the authors validate their method on multiple databases and task types with two families of backbone models. They are also transparent about mixed results and limitations, and the supplementary material (code, implementation and methodological details) supports reproducibility.
Critical weaknesses
- Staging and temporal design are not isolated empirically. There is no joint-loss baseline or reversed stage order, so the two-stage schedule is not compared against joint training or alternative ordering. The DayPE ablation also lacks a variant, and it is not shown that the temporal design of Rel-Hist and Rel-Future matters. To me this is important because many design choices (staging, multi-horizon sampling, multi-scale age) are only supported indirectly, so it is unclear which components drive the gains.
- The presentation and analysis of when the method helps are limited. The experimental discussion and the methodological narrative could be easier to follow, and the gains are uneven. The mean improvements are mostly driven by regression tasks, while many classification gains are small (as can be inferred from Fig. 3), with no analysis of why. There is also no complexity or runtime analysis of the encodings and pre-training objectives.
Questions for authors
In addition, for each question below, please refer to the Weaknesses:
- Can you report a joint-training baseline (Temp-Sub + Rel-Hist/Rel-Future with a summed loss) under the same settings, and the reversed stage order?
- Can you report TimeMix when versus the chosen default value?
- Can you discuss the method time and memory complexity, reporting empirical values as well?
2 comments
3. When the method helps
The most consistent improvements occur when historical relational context informs future quantities or rates. Our proposed schedules improve 8/11 HeteroGNN tasks and 7/11 THGFM tasks over A0, including all four regression tasks on both backbones; mean regression gains are +5.12% and +8.55%. In Appendix F, both TS-first schedules achieve +4.87%–+7.40% mean regression gains across the four backbone–encoding settings, exceeding their joint and reversed counterparts.
Regression tasks such as H&M item-sales and Avito ad-ctr predict future quantities or rates from the target entity's historical relational context. This aligns with learning that context through TS before refining it with relation-based supervision. For classification, Avito user-clicks, Avito user-visits, and H&M user-churn define labels through the occurrence or frequency of interactions within a future window, making temporal relation patterns directly relevant. With TimeMix + TimeRoPE, both schedules improve mean binary classification performance by -- on HeteroGNN and -- on THGFM over matched supervised controls. On HeteroGNN, both schedules also improve Arxiv author-category accuracy by --, giving the combined design positive gains across all three task types. Alongside type-balanced gains, Figure 3 reports task-balanced average ranks: our two staged schedules occupy the top two positions on each backbone with both encodings.
4. Parameters, complexity, and measured resources
Appendix G shows that our encodings reduce trainable parameters. Let T count timestamped node types, d be hidden width, K the number of scales, N_t the timestamped nodes in a batch, and E_ℓ the edges processed by rotation-enabled paths at layer ℓ.
| Operator | Trainable parameters | Forward computation | Intermediate activations |
|---|---|---|---|
| DayPE age branch | T(d²+d) | O(N_t d²) | — |
| TimeMix | 2TKd+2K | O(N_t Kd) | O(N_t Kd) |
| TimeRoPE | 0 | O(d Σℓ E_ℓ) | O(d Σℓ E_ℓ) |
At d=128 and K=4, replacing DayPE with TimeMix removes 15,488T−8 parameters, or approximately 0.02M–0.09M per backbone across the five databases. TimeRoPE uses fixed frequency buffers and adds no trainable parameters or optimizer states.
The resource analysis in Appendix G uses the same three-layer graph-model configuration. All values cited below are three-layer results. Table 12 (p.27) reports parameters, Table 13 (p.28) summarizes paired resource changes, and Table 14 (p.29) gives absolute time and memory by database, model, and objective. Median paired overheads of TimeMix + TimeRoPE relative to DayPE are:
| Backbone | Wall time | Reserved GPU memory |
|---|---|---|
| HeteroGNN | +15.29% | +12.06% |
| THGFM | +16.10% | −0.04% |
The resource comparison uses batch size 256, and 50,000 evaluations per objective: joint training performs 50,000 updates evaluating both objectives; sequential training performs 50,000 updates per stage. We report pretraining wall-clock time, including online sampling, target preparation, data transfer, forward and backward passes, and optimizer updates. Offline feature and graph-cache construction and downstream fine-tuning are excluded. Memory is the PyTorch reserved peak in GiB, summed across devices when multiple GPUs are used. Sequential costs sum the stage durations and take the larger stage memory footprint.
The updated Table 13 separates joint/sequential resource comparisons by backbone, encoding, and objective. For three-layer models with both encodings, the results are below.
| Comparison | Backbone | Wall-time change | Reserved GPU-memory change |
|---|---|---|---|
| Joint-Hist / Seq-Hist(TS->Rel-HIST) | HeteroGNN | +7.24% | +6.49% |
| Joint-Hist / Seq-Hist(TS->Rel-HIST) | THGFM | −15.91% | +8.92% |
| Joint-Future / Seq-Future(TS->Rel-FUTURE) | HeteroGNN | −27.64% | +28.20% |
| Joint-Future / Seq-Future(TS->Rel-FUTURE) | THGFM | −22.78% | +37.16% |
Together with Appendix F, TS-first achieves higher mean type gain and lower reserved memory under the paired comparison in all four full-encoding settings. For HeteroGNN Hist, sequential training achieves +3.02% gain versus +2.71% joint, while joint training costs 7.24% more time and 6.49% more memory at the paired median. Joint-Future is faster, but its paired-median memory cost is 28.20% / 37.16% higher on the two backbones, with lower downstream gains than TS-first.
Sequential training retains one objective's computation graph at a time, explaining its memory advantage. Table 14 gives a concrete example: on three-layer F1 HeteroGNN with both encodings, Seq-Hist uses 57.74 h / 191.63 GiB versus Joint-Hist's 57.10 h / 322.68 GiB—40.6% less memory for 1.1% more time. The summary and database-level results make both performance and resource tradeoffs explicit.
Thank you for your kind and comprehensive review! We address your questions below.
For easier access, we have now combined the main body and appendix into a single PDF and uploaded it. The main body remains unchanged in this revision; the only content additions are Appendix Sections F and G, which report additional experiments and analyses.
1. Joint training and reversed stage order
Appendix F, Tables 10–11, now includes both requested comparisons. Joint-Hist and Joint-Future share the backbone and directly sum the complete Temp-Sub and Rel-Hist or Rel-Future losses, using independently sampled batches and one shared parameter update. The reversed schedules train Rel-Hist or Rel-Future before Temp-Sub (TS).
These additional comparisons retain the main study's three-layer graph models: hidden width 128, ResNet table encoders, fanouts (128,64,32), and eight attention heads for THGFM, on the same 11 tasks and chronological splits. The following results use TimeMix + TimeRoPE and report mean type gain over the corresponding supervised control B0:
Across all eight comparisons—two backbones × two encodings × two relation objectives—TS-first achieves higher mean type gain and mean regression gain than both joint training and the reversed order. This directly supports learning stable historical-neighborhood representations before refining them through relation prediction. On THGFM with both encodings, it also turns negative transfer from either relation objective alone into positive gains.
With both encodings, the mean type gain margins over joint training are 0.31, 2.02, 1.01, and 2.61 percentage points, respectively, in the table above. Holding the objective pair, backbone, and encoding fixed, these comparisons directly establish that the organization of supervision affects downstream representation quality.
TS-first exceeds the corresponding single relation objective in seven of eight comparisons. The HeteroGNN Rel-Future case (+3.00% alone versus +2.83% staged) identifies a configuration where the single objective is already effective. Our strongest consistent finding concerns the advantage over joint optimization and reversed ordering.
2. The K=1 reference and temporal encoding design
DayPE provides the single-scale (K=1) reference in our comparison: record ages are normalized by one fixed day scale, whereas default TimeMix uses four learnable scales. Here K counts temporal normalization scales. TimeMix initializes them at one second, minute, hour, and day, then learns their scales and mixture weights with type-specific projections.
Figure 4a compares DayPE + TimeRoPE against TimeMix + TimeRoPE without pretraining, holding interval encoding fixed. TimeMix improves mean type gain by 0.59 percentage points on HeteroGNN and 1.18 on THGFM. Holding TimeMix fixed, adding TimeRoPE contributes another 0.28 and 0.46 points, respectively. Their combination achieves the best aggregate encoding result on both backbones.
Appendix B.2 explains the data motivation: Avito contains variation at seconds-to-hours scales, Event and H&M predominantly span days to months, and Arxiv and F1 include years of history. The encoding comparison establishes the benefit of replacing the fixed single-scale representation with the adaptive multiscale representation.
The relation objectives also align supervision with time: Rel-Hist recovers hidden links from context available when the links were observed; Rel-Future predicts relation occurrence and counts over a specified future horizon from historical inputs. Appendix F tests these complete temporal objectives under different compositions and orders. Isolating individual horizon or count components is a finer-grained question than the objective-level effects established by these comparisons.