acceptodds
Under review as a conference paper at ICLR 2027

From Values to Interactions: Adaptive Graph Pre-training for Spatiotemporal Forecasting

Abstract

Spatiotemporal forecasting is inherently relational: future states depend not only on historical observations but also on how information is organized across interacting entities. Yet existing forecasting models typically learn such relations within task-specific predictors, while recent pre-training methods remain largely value-centric, emphasizing temporal reconstruction without explicitly encoding reusable cross-entity dependencies. We propose AGPST, an interaction-aware self-supervised framework that treats relational structure as reusable pre-training knowledge. AGPST combines data-adaptive graph aggregation with masked temporal modeling, enabling masked states to be reconstructed from both cross-entity evidence and long-range historical context. The resulting representations capture complementary relational and temporal information and can be transferred to heterogeneous downstream predictors through lightweight feature integration without modifying their internal architectures. Experiments on six traffic forecasting benchmarks show consistent improvements over representative task-specific and pre-training baselines across datasets, metrics, and prediction horizons. Ablation and cross-architecture studies further demonstrate the complementary benefits of adaptive interaction learning, graph propagation, and masked reconstruction, supporting the effectiveness and reusability of the learned representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Graph Learning, Graph Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_TyqZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an interaction-aware pre-training framework for spatiotemporal forecasting. AGPST first learns a data-adaptive graph from trainable node embeddings, applies graph aggregation to temporal patches, and then performs masked reconstruction with a Transformer encoder–decoder. The resulting representations are fused with downstream forecasting models through a lightweight integration module. Experiments on six traffic forecasting datasets, together with ablations and cross-architecture evaluations, suggest that AGPST can improve forecasting performance over several task-specific and pre-training baselines. The overall reasoning is sound. However, the paper tackles an overly ambitious problem and does not adequately address it. In fact, it is questionable whether the proposed approach meaningfully solves the stated problem.

Critical strengths
  1. The paper addresses a meaningful problem from a natural perspective. Explicitly incorporating cross-entity interactions into spatiotemporal pre-training is well motivated, and the combination of adaptive graph aggregation with masked temporal modeling is conceptually coherent.
  2. The method is relatively modular and easy to integrate with different downstream predictors. The paper provides extensive empirical comparisons, component ablations, sensitivity analyses, and cross-architecture experiments, which make the reported performance trends reasonably easy to examine.
Critical weaknesses
  1. The text and annotations in Fig 1 and 2 are quite small, which makes the architecture and masking procedure difficult to inspect. 2.The interaction matrix in Eq 3 is learned from global trainable node embeddings and is therefore essentially static for a given dataset. It does not depend on time, the current input, or the instantaneous system state. Thus, although the temporal features themselves vary over time, the interaction topology cannot represent cases in which the dependency between two entities is strong during one period and weak during another. This creates a mismatch between the paper’s motivation, which emphasizes time-varying spatial dependencies, and the proposed formulation, which mainly learns a global relational prior. 3.The claim that reconstructing graph-enhanced representations leads to reusable interaction-aware knowledge is not sufficiently justified. The pre-training loss reconstructs the latent representation Z, rather than the graph structure, interaction weights, or any explicitly defined relational quantity. Consequently, the model may learn a latent representation that is easy to reconstruct without necessarily learning transferable interaction knowledge. This concern is further complicated by the fact that Z itself is produced by the graph encoder, and the paper does not clearly specify whether the reconstruction target is detached. Finally, all experiments are conducted on traffic datasets, so the broader claims about general spatiotemporal forecasting are not fully supported.
Questions for authors
  1. Since the adjacency matrix is determined by global node embeddings, how does AGPST model interactions that vary across time or depend on the current system state? Is the intended contribution a static relational prior rather than dynamic interaction modeling?

  2. Why should masked reconstruction of Z be expected to learn reusable interaction knowledge rather than a representation that is simply convenient to reconstruct? Is the target representation detached, and how do the results change when reconstructing raw temporal patches or explicitly supervising the interaction structure?

  3. To what extent do the reported results generalize beyond traffic forecasting? Since all datasets come from the traffic domain, what supports the broader claims about general spatiotemporal pre-training?

Reviewer_LkpcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AGPST, an interaction-aware pre-training framework for spatiotemporal forecasting. The method combines a learnable adaptive adjacency matrix, graph-based spatial aggregation, patch-level masked temporal modeling, and a lightweight feature-fusion module for downstream predictors. Experiments are conducted on six traffic datasets, with additional ablations and cross-architecture evaluations.

The reported results are competitive, and the idea of transferring long-context relational representations to short-window forecasting models is potentially useful. However, the current manuscript contains several unresolved issues concerning the exact pre-training objective, the order of graph aggregation and masking, the novelty relative to existing spatiotemporal pre-training methods, and the fairness and reproducibility of the experiments. These issues affect the validity of the central claim that AGPST learns reusable interaction-aware representations.

Critical strengths
  1. The paper addresses an important problem. Many spatiotemporal forecasting models rely on short historical windows, and using longer contextual information through pre-training is a reasonable and practically relevant direction.

  2. The proposed framework is conceptually modular. The representation-level interface allows the same pre-trained module to be combined with several downstream predictors.

  3. The empirical results are promising. AGPST improves greatly over the reported baselines on the four PEMS flow datasets and on the METR-LA and PEMS-BAY speed datasets.

Critical weaknesses

1. The pre-training objective is ambiguous and may admit a trivial solution

The preliminary formulation in Eq. (1) describes reconstruction of the masked raw observations XLX^L. However, the actual methodology in Eqs. (6)--(13) appears to reconstruct the latent representation ZZ, where

Zp=AGNN⁡(Up;A^)+Pp.Z_p = \operatorname{AGNN}(U_p; \widehat{A}) + P_p.

The loss is then defined between Z^\widehat{Z} and ZZ, both of which are generated by trainable modules. The manuscript does not state whether the target representation is detached from the computation graph, generated by a frozen teacher, or otherwise regularized.

As written, the objective appears to admit a degenerate solution in which the encoder, positional embeddings, and decoder produce constant or zero representations, yielding a small reconstruction loss without learning meaningful temporal or relational information. This concern is especially important because the paper's central claim depends on the learned representation being informative.

The figure also contains a "GNN Decoder," but the decoder is not mathematically defined in the methodology. In contrast, Figure 4 shows reconstruction of the original time-series values. The relationship between the latent reconstruction objective in Eq. (13) and the value reconstruction shown in Figure 4 is therefore unclear.

2. The masking order may introduce information leakage

According to Eqs. (6)--(10) and Figure 1, the input is first passed through the graph encoder and only then masked at the patch level. If masks are sampled independently across nodes, the representation of a visible node can contain information from a masked node through graph aggregation.

For example, with one graph layer,

Sjp=Ujp+∑iA^jiUip.S_{jp} = U_{jp} + \sum_i \widehat{A}_{ji} U_{ip}.

If UipU_{ip} is masked but SjpS_{jp} remains visible, then SjpS_{jp} still depends directly on the original masked value UipU_{ip}. More explicitly,

∂Sjp∂Uip=A^ji.\frac{\partial S_{jp}}{\partial U_{ip}} = \widehat{A}_{ji}.

Consequently, the decoder may reconstruct a masked patch using information that has already been propagated from that patch before masking.

This issue would be avoided if masking were applied before graph aggregation, or if the same temporal mask were synchronously applied to all nodes. The paper does not specify which implementation is used. Therefore, the claim that the encoder cannot directly access the masked local observations is not established by the current description.

3. The novelty relative to existing work is insufficiently isolated

The adaptive adjacency parameterization

A^=SoftMax⁡(ReLU⁡(E1E2⊤))\widehat{A} = \operatorname{SoftMax}\left(\operatorname{ReLU}\left(E_1 E_2^{\top}\right)\right)

is the same basic parameterization used in Graph WaveNet. Thus, the graph construction itself is not new. The contribution must instead come from the particular integration of this graph mechanism with masked long-context pre-training and downstream transfer.

However, this distinction is not sufficiently demonstrated. Existing methods such as STEP, GPT-ST, and STD-MAE already study long-context masked pre-training, spatial dependency modeling, and representation transfer to heterogeneous downstream predictors. In particular, STD-MAE explicitly performs spatial and temporal masked reconstruction and transfers the learned representations to multiple forecasting backbones; GPT-ST also uses masked spatiotemporal pre-training together with learned spatial structures.

The paper does not provide a direct comparison with GPT-ST, nor does it present a controlled experiment showing that the proposed interaction-aware objective provides a benefit beyond simply adding a graph module to an existing masked pre-training model.

Furthermore, A^\widehat{A} is generated from global trainable node embeddings and is therefore static after training. This does not directly model time-varying interactions, despite the manuscript repeatedly motivating the method using dynamically changing dependencies. The terminology "dynamic adjacency matrix" should be clarified.

4. The ablations do not isolate the contribution of pre-training

The ablation study removes individual components, but it does not include the most important matched control: the same AGPST architecture and fusion module trained from random initialization using only the downstream forecasting objective.

Without this control, the reported improvements may result from increased model capacity, access to the longer historical input, the additional fusion network, different optimization schedules, or the pre-training objective itself.

It is also unclear whether the pre-trained encoder is frozen or fine-tuned during downstream forecasting. A fair comparison requires the corresponding random-initialization model to use the same training protocol.

The "w/o Masking" variant is not sufficiently specified either. Since Eq. (13) computes the loss only on masked positions, the paper should explain how the loss is defined when the masking ratio is zero.

Questions for authors
  1. What exactly is the reconstruction target in the implementation: raw values or latent representations?

  2. Is the target representation in Eq. (13) detached from the encoder computation graph? If not, what prevents a constant or zero representation from minimizing the loss?

  3. Is masking applied before or after graph aggregation? Are masks shared across all nodes for a given temporal patch?

  4. Is the pre-trained encoder frozen or fine-tuned during downstream forecasting?

  5. What is the performance of a randomly initialized AGPST model with the same architecture, input length, fusion module, and downstream training budget?

  6. How were the runtime values in Table 4 obtained, and why do they match the values reported in the STD-MAE paper despite the different hardware and timing descriptions?

  7. Why does Table 1 report an STD-MAE MAPE of 11.97 on PEMS04, whereas Table 6 reports 12.97 for the corresponding GWNet + STD-MAE configuration?

Minor Issues

  1. The text states that the pre-training length is varied from k=432k=432 to k=864k=864, while Figure 3 uses k∈{288,864,2016}k\in\{288,864,2016\}.

  2. The dataset statistics table appears to swap the meanings of the “time steps” and “time interval” columns.

  3. The GNN decoder shown in Figure 1 is not defined mathematically.

  4. The paper should distinguish more carefully between a globally learned static adjacency matrix and an input- or time-dependent dynamic graph.

  5. The paper should specify whether the reported baseline values are reproduced under the same codebase or copied from the original publications.

Reviewer_jdwpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study proposes AGPST, a self-supervised pre-training framework for spatiotemporal forecasting. AGSPT combines a data-adaptive interaction matrix, graph-based aggregation, and patch-level masked temporal modeling to learn representations that contain both cross-entity dependencies and long-range temporal information. The obtained representations are then integrated into downstream forecasting models through a lightweight fusion module without changing their internal architectures. Experiments on six traffic forecasting datasets show improvements over forecasting and pre-training baselines. This study also conducts ablation study on components, hyperparameter studies, runtime comparisons, and compatibility experiments with five downstream predictors.

Critical strengths
  1. Consistent improvements across datasets and downstream predictors: AGSPT achieves the best MAE, RMSE, and MAPE on all the six benchmark datasets. In addition, AGSPT shows that the learnt representations are useful across many downstream predictors.
  2. Practical separation between pre-training and downstream forecasting: AGSPT learns a long-context representations separately and introduces them into existing forecasting models through lightweight feature integration. This means that it is possible to use pre-training without redesigning or retraining each downstream forecasting. The ablation study on components show that adaptive graph learning, graph aggregation, and masking contribute to the performance.
Critical weaknesses
  1. Insufficient specification and inconsistency of the pre-training objective: The pre-training objective in Eq.(13) defines the reconstruction loss over latent representations Zip\\{ \mathbf{Z}_{ip} \\}. However, Zip\\{ \mathbf{Z}_{ip} \\} are produced by trainable embedding and graph aggregation layers in Eq.(8). It is not clear whether the authors stop gradients through the target representations or a separate target encoder is employed. I think that the objective produces degenerate solutions with collapsed representations if the reconstruction targets and predictions are jointly optimized.
  2. The validity of the masked pre-training is not clear due to potential information leakage: The masking procedure in Section4.2 might trigger information leakage, in particular, the graph aggregation in Eq.(4) transmits information of masked tokens in a node to another one, and then use it for prediction of the node. Specifically, each temporal patch is processed by the adaptive graph encoder, and then masking is applied to the obtained representations according to Eqs.(6)-(9). However, graph aggregation mixes information across nodes before the masked tokens are removed.
Questions for authors
  • How are reconstruction targets and their gradients handled? What prevents representation collapse? Why does the implementation use masked MAE and an additional sparsity loss instead of the objective in Eq. (13)?
  • How are masks sampled across spatial nodes? How is information leakage prevented when graph aggregation is performed before masking?
  • In implementation, how do "adaptive: False", "topK: 6", and "multiple patch sizes" correspond to the proposed formulation? How is "future_data" used during validation and testing?
Reviewer_kggvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents AGPST, a pre-training enhanced framework for spatiotemporal forecasting. The authors evaluated AGPST on six traffic forecasting benchmarks, claiming that the learned representations can improve performance and are transferable.

Critical strengths
  • S1. The AGPST framework is widely compatible with various prediction architectures.

  • S2. The author's writing is straightforward and easy to read.

Critical weaknesses
  • W1. This appears to be a paper from three or four years ago. Every component of the proposed framework consists of widely used, including learnable adjacency matrices, GNN-based spatial aggregation, and Transformer-based masked autoencoding. Although their integration is reasonable, these individual components have been thoroughly studied in the fields of spatiotemporal prediction and representation learning. Moreover, the paper fails to adequately clarify what fundamentally new learning mechanism AGPST introduces compared to existing graph-based pretraining methods (especially STEP, GPT-ST, and STD-MAE). Therefore, returning to the motivation emphasized by the authors, the mechanism of spatiotemporal interaction is not novel either; it has been tediously reiterated by at least 100+ papers in this field. (I sincerely apologize for my tone, but this is the truth.)

  • W2. Following W1, the authors actually compared against a bunch of outdated baselines; even the most recent one is only the 2023 STD-MAE. This is an ICLR 2027 paper?! It largely ignores recent works on spatiotemporal foundation models.

  • W3. There is a mismatch between the claimed adaptive interaction and the actual graph construction. A core motivation is that spatial dependencies vary over time and cannot be fully represented by a fixed topology. However, Equation (3) constructs the adjacency matrix solely from two trainable node embedding matrices, which makes the learned graph static relative to the input sequence and temporal context. Although this graph is learned from data during training, it cannot dynamically adapt to changing traffic conditions during inference. This distinction directly undermines the paper's core claim about modeling time-varying interactions. Furthermore, the experiments do not explicitly demonstrate that the learned graph captures evolving spatial dependencies or meaningful relational structures.

Questions for authors
  • Q1. Clarification on Adaptive Graph Modeling (W3). Equation (3) uses trainable node embeddings instead of input-dependent features to define the adjacency matrix. During inference, will this adjacency matrix change with different input sequences or time periods? If not, how does the proposed mechanism address the time-varying spatial dependencies emphasized in the motivation? Can the authors provide quantitative or qualitative evidence that the learned interactions reflect meaningful spatial relationships?

  • Q2. Clarification on Reconstruction Target. Equation (13) reconstructs graph-augmented patch representations rather than raw observations. Since the target representations are generated by a jointly trained graph encoder, could the authors clarify how training avoids degenerate or uninformative representations? Is the target encoder detached from gradient computation, or are additional constraints applied to preserve meaningful information?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.