acceptodds
Under review as a conference paper at ICLR 2027

Learning Relations for Temporal Graph Foundation Models

10 views

Abstract

Temporal graph learning has traditionally relied on dataset-specific models, whereas recent advances suggest that large-scale pre-training can yield broadly transferable representations. Extending this foundation model paradigm to temporal graphs is difficult because structural and temporal information interact across heterogeneous data regimes: spatiotemporal graphs with evolving node signals and fixed topology, interaction networks with dynamic edges and static features, and fully dynamic graphs in which both change. We address this heterogeneity with a unified architecture that maps such diverse observation formats into a shared representation space. The model captures spatio-temporal patterns by integrating the observed topology with a temporal correlation-based message-passing operator. When pre-trained across multiple graphs within each domain, our model transfers zero-shot to unseen test graphs, achieving state-of-the-art performance. These results provide a concrete step toward temporal graph foundation models and highlight the benefits of treating relational structure as data-dependent.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural network, temporal graph

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GGcKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE combines a patched temporal Transformer with Gram-based and observed-topology propagation, gated residual updates, and layer aggregation. Separate models are pre-trained within traffic, transaction, and transportation domains, then evaluated without fine-tuning on held-out graphs for forecasting or graph classification.

Critical strengths
  1. The architecture offers a useful interface for featureless, static-topology, and fully dynamic graphs. Associative Gram multiplication avoids materializing a dense relation matrix, while the two branches combine observed and inferred dependencies.

  2. The reported transfer results are promising across three domains with compact models, including a MiNT mean AUC of 0.746 versus 0.721 for HTGN-zs. The paper explicitly distinguishes target-contaminated baselines and the transductive nine-dataset setting from zero-shot evaluation.

Critical weaknesses
  1. The TSFM baselines are recent, including FlowState (2026), but the directly relevant graph-transfer methods OpenCity [1] and UniFlow [2] are cited without experimental comparison. TSFMs receive no explicit topology and use different pre-training corpora. Compare graph-aware alternatives under matched data and inputs to establish the architectural contribution beyond combining established temporal and relational components.

  2. Table 4 changes the architecture while keeping the full model’s weights frozen, so degradation measures reliance on the trained configuration rather than the potential of alternative architectures. Pre-train each variant on the same source corpora, including temporal-only and standard-attention controls. Clarify whether ‘no propagation’ also disables the layer aggregator's node-wise attention [3]; otherwise it is not an independent per-series forecaster as claimed.

  3. Tables 8-9 report three separately trained domain models. These experiments establish a reusable architecture and within-domain graph transfer, but do not validate the conclusion’s single model pre-trained across heterogeneous domains. Narrow that claim or demonstrate a shared checkpoint trained on mixed-domain corpora, including genuinely varying feature dimensions.

  4. Only MiNT reports seed variability; forecasting tables lack uncertainty, despite a Seoul MAE margin of only 0.0010 over Chronos-2-uni. Report repeated runs and paired intervals. The corpus-size study also changes source identities, data volume, and potentially training compute together; repeated source subsets and controlled budgets are needed to isolate a diversity/scaling benefit.

  5. The Gram operator lacks node-count normalization: identical embeddings yield an all-ones matrix, so message magnitudes can grow with graph size. Test normalized kernels and size-shift robustness. Specify whether Gaussian R_t is shared across snapshots/windows and report sensitivity to random draws, since this choice can change temporal consistency.

  6. Associative Gram evaluation saves storage, but Section 3.4 still includes quadratic node attention through layer aggregation [3]. Its O(n(L+P)) memory bound omits attention maps unless a memory-efficient implementation is used. Clarify that implementation and the 4096-node cap at inference, and report end-to-end runtime and peak memory across graph sizes.

[1] Li, Z., et al. OpenCity: Open Spatio-Temporal Foundation Models for Traffic Prediction. TIST, 2025.

[2] Yuan, Y., et al. UniFlow: A Foundation Model for Unified Urban Spatio-Temporal Flow Prediction. TMC, 2026.

[3] Galron, Y., Sinai, H., Maron, H., and Eliasof, M. Learning from Historical Activations in Graph Neural Networks. ICLR, 2026.

Questions for authors
  1. Do the gains persist against OpenCity/UniFlow and separately pre-trained temporal-only, single-relation, and standard-attention variants? Which node-attention operations remain in the frozen ablations?

  2. Was any single checkpoint trained jointly across all three domains? If not, can the conclusions explicitly distinguish architectural reuse from cross-domain pre-training and transfer?

  3. How are Gaussian features reused over time, and how sensitive are results to random draws, Gram normalization, and changes in active-node count?

  4. Can the authors provide forecasting uncertainty, corpus-scaling results with controlled budgets, and measured runtime/memory, including the exact inference-time node-selection policy?

4 comments
AuthorsOpenReview ↗

Response to W1, W2 and W3

We thank you for your careful and detailed evaluation. We are encouraged by your recognition that The architecture offers a useful interface for featureless, static-topology, and fully dynamic graphs and that The reported transfer results are promising across three domains with compact models. We answer each point below.

In short: we include (i) OpenCity-Plus [1] on all four PeMS targets, (ii) frozen and retrained ablations that remove the node attention, (iii) three seeds for the MOBINS targets and for all four PeMS targets, (iv) the measured scaling of runtime and memory, and (v) a precise statement of the scope of the unification claim. Overall, during the rebuttal we ran 9 new experiments and 3 new analyses, all listed in our general comment. We hope that you find our responses satisfactory, and that you will consider revising your score.

Regarding Weaknesses

We kindly thank you for the feedback, which we feel helped us improve and clarify very relevant and important details of our paper.

W1 Thank you for this comment. We agree with the reviewer that including a graph-based baseline would position our contribution better. Accordingly, we included OpenCity-Plus [1], a graph-based spatiotemporal foundation model, to which we compare our performance. We report in the table below the MaskedMAE$\downarrow$ of OpenCity-Plus and of TRACE zero-shot averaged over three seeds (†\dagger means that the pre-training corpus of the model contains the dataset corresponding to the column, or traffic data from the same district). We have updated Table 1 of the revision to include OpenCity-Plus, and the three seed performances of TRACE in Appendix B.1.

ModelPeMS03PeMS04PeMS07PeMS08
OpenCity-Plus [1]16.1619.86$^\dagger$22.44$^\dagger$15.05$^\dagger$
TRACE zero-shot14.21 ±\pm 0.0619.23 ±\pm 0.0120.81 ±\pm 0.0814.86 ±\pm 0.15

The results show that TRACE zero-shot outperforms OpenCity-Plus on all four PeMS targets, even when the pre-training corpus of OpenCity-Plus contains the target dataset. Regarding UniFlow, the authors of the paper did not publicly release the pre-trained weights of their method, so this is not included as an additional baseline. Thank you.

W2 Thank you for this observation. We clarify the ablation definitions: no propagation removes both message passing methods considered in the paper, and with them also the node level attention, so this variant is graph-free; topology only, Gram only and Gram from X keep the attention across all nodes. We therefore extend Table 4 to remove node attention and report the results below:

VariantPeMS03PeMS04PeMS07PeMS08
TRACE-zs (full)14.1619.3121.0414.89
no node attention20.8128.6731.2321.83
topology only, no node attention19.8827.9830.0820.23
Gram only, no node attention19.4626.7428.7721.94

When pre-training on traffic flow forecasting and leaving out PeMS08 the performances are 14.59 without node attention, 14.61 without propagation and 14.66 with the Gram relation only, against 14.89 for the full model. Thus, with this ablation we show that those model components are not driving the gains for this pre-training. Thank you, we have updated the revision with these results and clarified the ablation details.

W3 Thank you for this observation. We agree with you that in the paper the evaluation of TRACE is in-domain and therefore does not establish cross-domain generalization. Our intention was to claim that TRACE is a single model definition that can be trained on any temporal graph domain, and that it can be pre-trained on multiple graphs within each domain. Still, we follow the reviewer’s comment and propose a new experiment where we jointly pre-train TRACE on traffic forecasting and transportation networks. In particular, we follow the settings we used for Table 1 and 2, and leave-out from pre-training PeMS04 and Daegu. We compare, for a single seed, the jointly pre-trained model with the single domain model MaskedMAE$\downarrow$ in the table below:

ModelPeMS04Daegu
TRACE-zs19.310.4119
TRACE-multi19.160.4107

The jointly pre-trained model is on par with, and slightly better than, the single-domain model. We have revised the paper to include this new experiment in Table 6 (Section 4.2, setting in Appendix B.4) and reframed our claims accordingly. Thank you.

AuthorsOpenReview ↗

Response to W4, W5, W6, Questions and References

W4 Thank you for pointing out this detail. Following your comment, we have revised the paper to include seed level variability for the forecasting experiments, in Appendix B.1 (Tables 12 and 13). In brief the standard deviation of our performances on traffic flow forecasting and MOBINS are on average 0.08 and 0.0036, which is small compared with most margins in Tables 1 and 2. On Seoul, the margin is within one standard deviation. Regarding the corpus-size study, we kindly redirect the reviewer to the dataset-level statistics presented in Table 7. We summarize them here for better clarity. All PeMS-D{5,6,10,11,12} datasets comprise 25045 timesteps, with a minimum of 150 nodes and a maximum of 820 nodes per corpus. These are of the same scale as the PeMS0{3,4,7,8} datasets, which have a similar amount of timesteps and nodes each. The settings and model size remained the same. Since each corpus contributes a fixed number of batches per epoch (see Appendix A.3 of the revision), the number of training steps grows with the number of corpora, so this study does not separate diversity from training budget, which we acknowledge as a limitation. The only change is the set of available datasets, which we used to pre-train TRACE (which is part of the study itself). Thank you.

W5 As correctly mentioned in your review, in case of identical temporal embeddings the Gram relation is a constant matrix, and the message passed to each node is the same. Following this observation, we ran a new experiment where we compare the Gram normalizations used on traffic forecasting. We compare between our employed normalization and the division by ndnd, where nn is the number of nodes and dd is the number of features. We report the results below, in zero-shot Masked MAE$\downarrow$, following the setting of Table 1:

Gram normalizationPeMS03PeMS04PeMS07PeMS08
Our employed normalization (1 run)14.1619.3121.0414.89
Division by ndnd (3 seeds)14.21 ±\pm 0.0619.23 ±\pm 0.0120.81 ±\pm 0.0814.86 ±\pm 0.15

The differences in performance are negligible compared to our employed normalization, with the new version yielding better performances on PeMS0{4,7,8} and slightly lower on PeMS03.

Regarding the Gaussian random walk features, we draw Rt\mathbf{R}_t of Equation (1) once per network prior to pre-training and we fix it. We follow the reviewer's suggestion and test the sensitivity of the zero-shot performance on the test set when drawing a different Rt\mathbf{R}_t for each network 10 times. We report the results below, in zero-shot AUC$\uparrow$, averaged over the 20 test networks:

Draw of Rt\mathbf{R}_tAUC$\uparrow$
Average (10 draws)0.747$\pm$0.001

The ten draws do not alter the performance, with low standard deviation. This result shows that our performance is not sensitive to the random Gaussian probe matrix, in the zero-shot setting. We thank you for this question, which we feel helped us improve an important part of our contributions.

W6 Thank you for asking about this detail. We agree with the reviewer and specify that although MHSA has O(n2)\mathcal{O}(n^2) computational complexity, its fused scaled dot-product attention kernels do not materialize the full n×nn \times n attention map [2, 3], which is the implementation we considered in our method. Regarding the 4096 node cap, it was originally introduced as a conservative memory safeguard for an earlier computational formulation that explicitly materialized this matrix. It was not a requirement of the proposed propagation mechanism and does not apply to the factorized implementation used in our method (also refer to our response to Q1 of reviewer 8aJz). In Appendix B.7 (Figure 5 and Table 19) of the paper revision we have included an explicit empirical complexity analysis that supports the theoretical expectations of Section 3.4. We thank you for the question again, which helped us improve an important ambiguity in the implementation details.

Regarding Questions

We thank the reviewer for all questions, whose responses are included in the Weaknesses section above.

Q1 We kindly redirect the reviewer to our responses W1 and W2.

Q2 We kindly redirect the reviewer to our response W3.

Q3 We kindly redirect the reviewer to our response W5.

Q4 We kindly redirect the reviewer to our responses W4 and W6.

In general, we have revised the paper to include experimental evidence and clarification to all issues raised by the reviewer. We thank you once again, and hope that our clarifications and additional experiments will lead to a reconsideration of your score.

References:

[1] Li et al. Open spatio-temporal foundation models for traffic prediction. ACM TIST, 2025.

[2] Rabe and Staats. Self-attention does not need O(n2)O(n^2) memory. arXiv, 2022.

[3] Dao et al. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. NeurIPS, 2022.

Thank you for the thorough response and clarifications. The authors have addressed my previous review comments and provided detailed explanations of many aspects of the implementation. These clarifications have strengthened my understanding of this work and helped me better appreciate its novelty. Based on this, I will raise my scores.

Dear Reviewer GGcK,

We are happy to read that you found our responses to address your comments, and that they helped you to better appreciate the novelty of our work. Thank you for raising your score.

We feel that your insightful and constructive feedback helped us to improve the quality of our paper --- thank you.

With kindest regards,
Authors.

Reviewer_fARmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE is a pre-trained encoder for discrete-time temporal graphs that is designed to handle different input regimes: traffic sensors on a fixed road graph, featureless transaction networks, and mobility graphs whose edges change over time. Each node's history is encoded by a temporal Transformer. The resulting node states are propagated over two relations: the observed graph, and a similarity ("Gram") graph computed from the temporal embeddings. Within each domain, the model is pre-trained on several graphs and evaluated zero-shot on held-out graphs from traffic (PeMS), mobility (MOBINS) and transaction (MiNT) data.

Critical strengths
  • The presented paper shows strong zero-shot transfer capability within each domain, using relatively small models
  • The handling of heterogeneous inputs is simple and reusable, and transfer improves as more source graphs are added
Critical weaknesses
  • Every reported model is trained separately for one domain, with different hyper-params. Yet the Conclusion claims that the results show a single model can be pre-trained across heterogeneous temporal-graph domains. No experiment supports this.
  • All ablations remove or change components of the already-trained model without retraining it. It does not show that a simpler model trained from scratch would transfer worse.
  • The paper cites graph-based zero-shot traffic models, but in traffic forecasting compares only against time-series foundation models that do not use the graph.
Questions for authors
  • Is any reported result obtained from a single checkpoint shared across domains? If not, could you train one model jointly on two domains and compare it with the per-domain models?
  • How does TRACE compare with OpenCity under the same leave-one-out protocol, with the target graph excluded from pre-training Please also report variance over runs for Tables 1 and 2, since some margins are very small
3 comments
AuthorsOpenReview ↗

Response to W1, Q1 and W2

We thank you for your careful evaluation. We are encouraged by your recognition that The presented paper shows strong zero-shot transfer capability within each domain, using relatively small models and that The handling of heterogeneous inputs is simple and reusable, and transfer improves as more source graphs are added. We answer each point below.

In short: As requested by the reviewer, we include: (i) an explicit discussion of the paper claims, (ii) a multi-domain pre-training and evaluation, (iii) further supervised ablation studies, (iv) OpenCity-Plus [1] a spatiotemporal foundation model to Table 1, and (v) the requested pre-training seeds for Table 1 and 2 (Appendix B.1). Overall, during the rebuttal we ran 9 new experiments and 3 new analyses, all listed in our general comment. We hope that you find our responses satisfactory, and that you will consider revising your score.

Regarding Weaknesses and Questions

W1 and Q1 Thank you for this observation. We agree with you that in the paper the evaluation of TRACE is in-domain and therefore does not establish cross-domain generalization. Our intention was to claim that TRACE is a single model definition that can be trained on any temporal graph domain, and that it can be pre-trained on multiple graphs within each domain. We did not intend to claim that a single checkpoint is shared across domains. Nevertheless, we follow the reviewer’s comment and propose a new experiment where we jointly pre-train TRACE on traffic forecasting and transportation networks. In particular, we follow the settings we used for Table 1 and 2, and leave-out from pre-training PeMS04 and Daegu. We compare, for a single seed, the jointly pre-trained model with the single domain model MaskedMAE$\downarrow$ in the table below:

ModelPeMS04Daegu
TRACE19.310.4119
TRACE-multi19.160.4107

The jointly pre-trained model is on par with, and slightly better than, the single-domain model. We have revised the paper to include this new experiment in Table 6 (Section 4.2, setting in Appendix B.4) and reframed our claims accordingly. Thank you.

W2 Thank you for pointing out this detail. We agree with the reviewer that our ablations target the components of the already pre-trained model. Accordingly, we perform a supervised experiment, where we ablate the components of TRACE on the PeMS04 network. We report in the table below the Masked MAE$\downarrow$ of the full TRACE model, averaged over three seeds, and of three ablations: (i) no propagation, (ii) no node attention, and (iii) Gram relation only.

ModelPeMS04
TRACE (full)18.46 ±\pm 0.04
no propagation18.75 ±\pm 0.06
no node attention18.48 ±\pm 0.07
Gram relation only18.47 ±\pm 0.04

The performance differences in this table are relatively contained, with the largest difference being 0.29 MAE (no propagation), which suggests that the component is needed for the considered dataset. We have added this table and the above statement to Table 5 and Section 4.3 of the revision, with the setting in Appendix B.3. Thank you.

We also pre-train the variants from scratch under the leave-one-out protocol of Section 4.1 and evaluate them zero-shot. We report the Masked MAE$\downarrow$ on PeMS08 (seed 0) and the MAE$\downarrow$ on the three MOBINS cities (three seeds):

ModelPeMS08SeoulBusanDaegu
TRACE (full)14.890.3338 ±\pm 0.00370.4792 ±\pm 0.00610.4143 ±\pm 0.0011
no propagation14.610.3231 ±\pm 0.00120.4620 ±\pm 0.00040.4063 ±\pm 0.0002
no node attention14.590.3257 ±\pm 0.00230.4709 ±\pm 0.00220.4134 ±\pm 0.0040
Gram relation only14.66---

Pre-trained from scratch, the variants reach an error comparable to, or lower than, the full model: those components are not driving the gains of pre-training, while the trained model relies on them (Table 4). The gains over the TSFMs come from pre-training across graphs: the variant without propagation also outperforms chronos-2-uni and tirex on all three cities of Table 2. We have added these results to Table 5 (PeMS08) and Appendix B.3 (Table 16, MOBINS) of the revision.

AuthorsOpenReview ↗

Response to W3, Q2 and References

W3 and Q2 Thank you for this question and suggestion. We agree with the reviewer that including a graph-based baseline would position our contribution better. Accordingly, we included OpenCity-Plus [1], a graph-based spatiotemporal foundation model, to which we compare our performance. We report in the table below the MaskedMAE$\downarrow$ of OpenCity-Plus and of TRACE zero-shot averaged over three seeds (†\dagger means that the pre-training corpus of the model contains the dataset corresponding to the column, or traffic data from the same district). We have updated Table 1 of the revision to include OpenCity-Plus, and the three seed performances of TRACE in Appendix B.1.

ModelPeMS03PeMS04PeMS07PeMS08
OpenCity-Plus [1]16.1619.86$^\dagger$22.44$^\dagger$15.05$^\dagger$
TRACE zero-shot, seed 014.2919.2320.7314.68
TRACE zero-shot, seed 114.1719.2320.8014.95
TRACE zero-shot, seed 214.1819.2220.9014.94
TRACE zero-shot14.21 ±\pm 0.0619.23 ±\pm 0.0120.81 ±\pm 0.0814.86 ±\pm 0.15

The results show that TRACE zero-shot outperforms OpenCity-Plus on all four PeMS targets, even when the pre-training corpus of OpenCity-Plus contains the target dataset, and every seed does so. Moreover, the standard deviation of the performance, at most 0.15 MAE, confirms that the obtained results are not due to lucky seeds.

We also include the seed spread of the results in Table 13 of the revised version, which we show in the table below for convenience as well:

Model (MAE$\downarrow$)SeoulBusanDaegu
chronos-2-uni0.33210.49110.4295
tirex0.33290.48630.4246
TRACE zero-shot, seed 00.32970.48600.4132
TRACE zero-shot, seed 10.33690.47740.4154
TRACE zero-shot, seed 20.33470.47420.4143
TRACE zero-shot0.3338 ±\pm 0.00370.4792 ±\pm 0.00610.4143 ±\pm 0.0011

On Busan and Daegu, all three seeds achieve a lower error than the strongest baselines, and on Seoul, TRACE lies within one standard deviation of chronos-2-uni and tirex. The standard deviation over the three considered seeds confirms that the obtained results are on par on Seoul and better on Busan and Daegu.

We thank you once again, and hope that our clarifications and additional experiments will lead to a reconsideration of your score.

References:

[1] Li et al. Open spatio-temporal foundation models for traffic prediction. ACM TIST, 2025.

AuthorsOpenReview ↗

Request for your feedback on our responses

Dear Reviewer fARm,

We hope you are doing well. We know this is a busy period, and we are grateful for the time you have already given our paper.

Your review helped us improve the paper. Following your suggestions, we pre-trained a single TRACE model jointly on traffic and transportation. It is on par with, or slightly better than, the single-domain models (Table 6), and we reframed the claim in the Conclusion accordingly. We also trained each ablated variant from scratch on PeMS04, over three seeds. We added OpenCity-Plus to Table 1, and zero-shot TRACE outperforms it on all four PeMS targets with 26 times fewer parameters. Finally, we reported the results of Tables 1 and 2 over three seeds (Appendix B.1).

Today is the last day of the discussion, so we would really appreciate hearing whether these changes address your concerns, and if you would consider revising your score. If anything is still unclear, we would be glad to clarify it.

Thank you again!

The Authors

Reviewer_8aJzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on transferable learning on temporal graphs and proposes TRACE-zs, which is trained on several domains of datasets and then is transferred on the same domain datasets, respectively. The key motivation is to address various kinds of graphs (evolving/fixed node signals and evolving/fixed topology) in a unified way.

Critical strengths

S1. The writing is clear and easy to follow. S2. The model design is reasonable and the experiment results demonstrate the superiority of the proposed method.

Critical weaknesses

W1. The paper motivates TRACE as a unified architecture for heterogeneous temporal graph regimes. However, the experiments train separate models within each domain and evaluate transfer only to held-out graphs from the same domain. These results demonstrate within-domain generalization, but do not establish whether the proposed unification enables knowledge transfer across graph regimes or offers an advantage over domain-specific architectures under matched training conditions.

W2. The practical value of the proposed transfer setting is not sufficiently quantified. Beyond predictive accuracy, the paper does not clearly establish the trade-off between source-domain training cost and the benefits of avoiding target-domain training. Comparisons with target-trained or lightly adapted baselines under matched compute or data budgets, together with runtime and peak-memory measurements, would help determine when the proposed approach provides a meaningful practical advantage.

W3. The forecasting experiments primarily compare TRACE against time-series foundation models, while graph-based supervised methods are represented only by a “Best supervised” result taken from prior work. Could the authors include graph-based baselines jointly trained on the same source datasets and evaluated under the same leave-one-dataset-out protocol? Such comparisons would help distinguish the benefit of the proposed architecture from that of access to explicit graph structure and multi-dataset training.

Questions for authors
  1. Does the implementation compute Gram-based propagation through the factorization H~(H~⊤X)\tilde H (\tilde H^\top X), without materializing the dense n×nn\times n matrix? If so, why is a 4,096-node cap still needed during training? Please clarify how nodes are selected, how excluded nodes are processed, whether the cap also applies at inference, and its impact on performance.

  2. Complexity of layer aggregation. Section 3.4 describes LayerAggregation as node-wise attention across layers, but assigns it an O(n2)O(n^2) computational cost. If attention operates only across the L+1L+1 layer representations of each node, why is the cost quadratic in the number of nodes? Please clarify the attention axis and provide a complexity analysis explicitly accounting for nn, LL, and hidden dimension dd.

7 comments
AuthorsOpenReview ↗

Response to W1 and W2

We thank you for your careful evaluation. We are encouraged by your recognition that The writing is clear and easy to follow and that The model design is reasonable and the experiment results demonstrate the superiority of the proposed method. We answer each point below.

In short: as requested by the reviewer we include (i) a revision of the proposed claims, (ii) new experiments testing transfer on traffic forecasting and MiNT, including fine-tuning, (iii) an empirical analysis of running time and memory, (iv) OpenCity-Plus [2] a spatiotemporal foundation model to Table 1, and (v) discussion about implementation details of the Gram operator and the layer aggregation. We hope that you find our responses satisfactory, and that you will consider revising your score.

Regarding Weaknesses

W1 Thank you for this observation. We agree with the reviewer that our method is trained and evaluated within domains. We propose an unification at the architectural level, which means that we need no architectural modifications to ingest graphs of different characteristics. Accordingly, we have modified the revision to state exactly this. Nevertheless, we follow the reviewer’s comment and propose a new experiment where we jointly pre-train TRACE on traffic forecasting and transportation networks. In particular, we follow the settings we used for Table 1 and 2, and leave-out from pre-training PeMS04 and Daegu. We compare, for a single seed, the jointly pre-trained model with the single domain model MaskedMAE$\downarrow$ in the table below:

ModelPeMS04Daegu
TRACE19.310.4119
TRACE-multi19.160.4107

The jointly pre-trained model is on par with, and slightly better than, the single-domain model.

W2 Thank you for this comment and cue. We agree with the reviewer that this analysis makes our contribution more sound. Therefore, we provide two new experiments. In the first, we follow the reference experimental settings of the paper to train TRACE on PeMS0{3,4,7,8} in a supervised fashion, and compare it with the paper's TRACE zero-shot leave-one-out checkpoints, and the fine-tuned version of each checkpoint. We report the MaskedMAE$\downarrow$ in the table below:

ModelPeMS03PeMS04PeMS07PeMS08
TRACE supervised14.6018.5319.1513.76
TRACE zero-shot14.1619.3121.0414.89
TRACE fine-tuned13.9518.1118.8613.49

The zero-shot version of TRACE outperforms the fully supervised counterpart only on PeMS03, while the fine-tuned variant outperforms all competitors on all networks. Regarding the computational burden, the fine-tuned version of TRACE requires 3.1 / 2.5 / 2.9 / 1.8 times less training time than training from scratch on PeMS03/04/07/08, respectively, with the same memory requirements.

To further strengthen the analysis, we measure positive and negative transfer on MiNT [1] as the second experiment. Specifically, we train TRACE with supervision on each target network separately and measure its gap in AUC$\uparrow$ with the zero-shot variant. We define transfer on the ii-th test network as Δi=AUCimulti−AUCisingle\Delta_i = \text{AUC}^{\text{multi}}_i - \text{AUC}^{\text{single}}_i, where Δi<0\Delta_i < 0 indicates negative transfer. We also compare it with the transfer of the two pre-trained baselines considered in our experiments, namely HTGN and GC-LSTM. We report in the table below the mean Δ\Delta over all 20 networks

Modelmean Δ\Delta
GC-LSTM+0.054
HTGN+0.042
TRACE+0.100

TRACE achieves the strongest positive transfer. Putting together the above-mentioned results on traffic forecasting and transaction networks, we can argue that TRACE shows benefits from joint pre-training with respect to fully supervised training. However, the trade-off between the computational cost of pre-training and the performance gains that any foundational architecture can measure stands at the motivations behind the development of foundation models [3]. Using a supervised model for each test network can be as prohibitive as pre-training an architecture from scratch on many networks. For instance, the cost can be measured in terms of the number of target networks, the employed hyperparameter searches for each of them, the memory requirements in saving a separate checkpoint for each, and also in terms of the patterns that a model can learn from a vaster amount of pre-training data, which supposedly expose more input distributional coverage.

In the revision, we have added the above-mentioned results to Section 4.1 (Tables 1 and 3, Figure 4) and Appendix B.2 (Tables 14 and 15). Furthermore, to support our complexity analysis, we have added a complete empirical analysis of the running time and memory requirements of TRACE, when varying the relevant dimensions of the input data. The outcome of the analysis is consistent with the theoretical expectations, which are now presented more clearly in the revision.

AuthorsOpenReview ↗

Response to W3, Q1, Q2 and References

W3 Thank you for pointing out this detail and for the question. The supervised baselines shown in Table 1 and 2 often comprise architectural components that encode node identities, as shown in a recent study [4]. These baselines have been methodologically defined for learning on single datasets and lack the inductive biases of architectures specifically tailored to zero-shot generalization. Therefore, we complement them with OpenCity-Plus [2], a recent graph-aware spatiotemporal foundation model specialized in traffic forecasting which publicly releases its pre-trained weights. We report in the table below the zero-shot MaskedMAE$\downarrow$ on the four traffic flow forecasting networks, where †^\dagger denotes that the pre-training corpus of the model contains the evaluation dataset, or traffic data from the same district:

ModelPeMS03PeMS04PeMS07PeMS08
OpenCity-Plus [2]16.1619.86$^\dagger$22.44$^\dagger$15.05$^\dagger$
TRACE zero-shot14.1619.3121.0414.89

TRACE outperforms OpenCity-Plus on all four datasets, including the three graphs OpenCity-Plus saw during pre-training. We have added this comparison to Table 1 of the revision. Thank you.

Regarding Questions

We kindly thank the reviewer for all the questions, which helped us improve and clarify very relevant and important details of our paper.

Q1 Thank you for this question. The Gram message is evaluated by associativity, H(H⊤X)H(H^\top X), without materializing the n×nn \times n Gram matrix. The 4,096-node cap was originally introduced as a conservative memory safeguard for an earlier computational formulation that explicitly materialized this matrix. It was not a requirement of the proposed propagation mechanism and does not apply to the factorized implementation used in our method. Consequently, the cap does not restrict node selection or processing during training or inference: all nodes are included in message passing, with no additional approximation or performance impact. We have removed this outdated restriction from the revised appendix to make the implementation description consistent with the actual method. We are grateful to the reviewer for pointing this out since it is a very important detail regarding the reproducibility of our paper.

Q2 Thank you for this question. We have added to Appendix A.6 the mathematical description of both attentions, and to Section 3.4 their complexity analysis in nn and LL. For convenience, we report here a summarized version of the analysis.

Layer aggregation follows HistoGraph [5] and consists of two sequential attention operations. First, layer-wise attention uses the last-layer representation of each node as a query over its L+1L+1 layer representations. The resulting attention scores are averaged across nodes to obtain one signed weight per layer, which is then used to aggregate the layer representations of each node. This operation costs O(Ln)O(Ln). Second, a multi-head self-attention is applied across the nn nodes, once after the final layer rather than at every layer. Its time complexity is O(n2)O(n^2) for each representation. The overall complexity is therefore O(Ln+n2)O(Ln + n^2). Since L≪nL \ll n, the quadratic term n2n^2 is dominant and comes entirely from the node-wise attention.

We thank you once again, and hope that our clarifications and additional experiments will lead to a reconsideration of your score.

References:

[1] Shamsi et al. MiNT: Multi-network transfer benchmark for temporal graph learning. NeurIPS Datasets and Benchmarks Track, 2026.

[2] Li et al. Open spatio-temporal foundation models for traffic prediction. ACM TIST, 2025.

[3] Bommasani et al. On the opportunities and risks of foundation models. arXiv, 2021.

[4] Chatterjee et al. Transfer learning for temporal link prediction. arXiv, 2025.

[5] Galron et al. Learning from historical activations in graph neural networks. ICLR, 2026.

Thank you for your detailed response and additional experiments. The clarifications address several of my concerns. However, two points remain. First, the total computational cost, e.g., total pre-training and fine-tuning times, remain unclear. The revision only provides the cost of per-epoch. Second, for the result of OpenCity-Plus, my original request concerns graph-based baselines under the same experimental settings, including datasets and evaluation protocal. Do you use the open weight of OpenCity-Plus to evaluate or rerun with the same settings?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision and of the new experiments

We thank all the reviewers for their careful evaluations and for their suggestions, which helped us improve and clarify very relevant and important details of our paper. We uploaded a revised PDF in which all changes are highlighted in blue. During the rebuttal, we ran 9 new experiments and 3 new analyses: they add 1 baseline, 10 new tables, 2 new figures and 8 new appendix sections (A.6 and B.1 to B.7), extend Tables 1 and 4, and move the per-network MiNT results to Appendix B.2. Below, each change is listed with its location and the reviewers who asked for it.

Experiments

  • OpenCity-Plus, a graph-aware spatiotemporal foundation model, is added as a zero-shot baseline on the four PeMS targets (Table 1; asked by 8aJz, fARm, GGcK)
  • TRACE trained on each PeMS target and the pre-trained checkpoint fine-tuned on each target, next to zero-shot TRACE (Table 1, Sec. 4.1, App. B.2; asked by 8aJz)
  • Positive and negative transfer on the 20 MiNT test networks, against GC-LSTM and HTGN (Table 3, Figure 4, Sec. 4.1, App. B.2; asked by 8aJz)
  • Three seeds for the four PeMS targets and the three MOBINS targets (App. B.1, Tables 12 and 13; asked by fARm, GGcK)
  • Supervised ablations of propagation, node attention and the Gram relation on PeMS04, over three seeds (Table 5, Sec. 4.3, App. B.3; asked by fARm)
  • Frozen ablations that remove the node attention (Table 4, Sec. 4.3; asked by GGcK)
  • Ablations retrained from scratch under leave-one-out on PeMS08 and on the three MOBINS cities (Table 5, Sec. 4.3, App. B.3 and Table 16; asked by fARm, GGcK)
  • Multi-domain pre-training on traffic and transportation networks, evaluated zero-shot on one held-out network per domain (Table 6, Sec. 4.2, App. B.4; asked by 8aJz, fARm, GGcK)
  • Comparison of Gram normalizations, including the division by ndnd (App. B.5, Table 17; asked by GGcK)

Analyses

  • Running time and peak memory of TRACE when varying the relevant dimensions of the input data (App. B.7, Figure 5, Table 19; asked by 8aJz, GGcK)
  • Sensitivity of the zero-shot AUC on MiNT to 10 draws of the Gaussian probe matrix Rt\mathbf{R}_t (App. B.6, Table 18; asked by GGcK)
  • Mathematical description and complexity of both attentions of layer aggregation in nn and LL (App. A.6, Sec. 3.4; asked by 8aJz)

Claims and clarifications

  • The unification claim is scoped to a single architecture with domain-wise checkpoints, and the domain-wise checkpoints of the main experiments are listed as a limitation (Sec. 1, Sec. 5; asked by 8aJz, fARm, GGcK)
  • The ablation definitions are clarified (Sec. 4.3; asked by GGcK)
  • The outdated 4,096-node cap is removed from the implementation description (App. A.3; asked by 8aJz, GGcK)
  • The node attention uses fused attention kernels that do not materialize the n×nn \times n attention map (Sec. 3.4; asked by GGcK)

We look forward to your feedback and hope that our responses help clarify our contributions and that you will consider revising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.