acceptodds
Under review as a conference paper at ICLR 2027

VLM4ST: Repurposing Vision-Language Models for Spatiotemporal Forecasting

Abstract

Spatiotemporal (ST) forecasting requires modelling spatial dependencies and temporal dynamics in high-dimensional fields. Adapting pretrained large language models offers transferable sequence representations, but tokenising spatial fields into 1D sequences can obscure spatial topology. Recent vision-language model (VLM) approaches encode images of hand-crafted statistics alongside fixed textual templates, leaving the correspondence between VLM architecture and ST structure largely unexploited. In this work, we propose VLM4ST, a framework that aligns the dual-path architecture of VLMs with spatial and temporal modelling. A 2D visual encoder processes spatial fields, while a 1D sequential encoder captures temporal dynamics. Multi-layer cross-attention couples both encoders, preserving spatial topology while leveraging pretrained sequence representations. To bridge the gap between natural image–text pretraining and ST data, we introduce Prompt Forge, a hierarchical pattern-memory module that learns latent ST prototypes and generates task-adaptive continuous prompts for cross-modal fusion. These prompts encode temporal periodicity, spatial correlations, and cross-dimensional dependencies. Extensive experiments show that VLM4ST outperforms task-specific, LLM-based, and recent VLM-based predictors, reducing short-term MAE by up to 16.5% (12.3% on average) relative to the strongest baseline on each dataset, with substantial gains in few-shot and zero-shot transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatiotemporal Forecasting, Vision-Language Models, Urban computing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Mx2mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VLM4ST, a framework that repurposes pretrained vision-language models for grid-based urban spatiotemporal forecasting. The framework combines Disentangled Cross-Attention, a Dual-Path Format Transformer, and Prompt Forge, which generates continuous prompts through Hierarchical Pattern Memory. An ALBEF backbone adapted with LoRA integrates visual and sequential representations, and a Horizon Query Predictor produces future spatial fields. Experiments on bike flow, taxi flow, and traffic speed datasets report improvements over selected task-specific, LLM-based, and VLM-based baselines, supported by short-term, long-term, few-shot, zero-shot, and ablation evaluations.

Critical strengths

1.The framework provides a coherent approach to adapting a pretrained VLM to numerical spatiotemporal observations. The visual pathway processes spatially structured pseudo-images, while the text encoder receives continuous embeddings without natural-language input. Their interaction through cross-attention offers a useful design for transferring pretrained representations to urban forecasting.

  1. The supervised benchmark evaluation covers seven short-term forecasting tasks, four long-term forecasting tasks, and 17 baselines, including ST-VLM and ST-Vision-LLM. Tables 1 and 2 report the lowest MAE and RMSE across the evaluated tasks, with an average short-term MAE reduction of approximately 12.3% relative to the strongest baseline on each dataset. The breadth and consistency of these reported results provide meaningful empirical support for the complete framework.

  2. Appendix A includes useful controls examining the contribution of pretraining, including random initialization, shuffled pretrained weights, and comparisons with CLIP and BLIP. On BikeNYCin, pretrained ALBEF achieves an MAE of 1.018 compared with 1.451 for random initialization and 1.437 for shuffled weights. These results support the usefulness of pretrained representations under the evaluated adaptation procedure.

Critical weaknesses
  1. Important experimental details remain unclear, limiting confidence in the reproducibility and fairness of the reported comparisons. Section 3.1 and Appendix G state that methods use common datasets, splits, horizons, normalization, and metrics, but the manuscript does not clearly specify chronological splitting, the treatment of sequences crossing split boundaries, or which observations determine the normalization statistics. The baseline configuration also requires clarification: Section 3.1 identifies LLaMA-2 as the ST-LLM+ backbone, whereas Table 7 reports 83M total parameters without explaining this configuration. Exact configurations and training settings for the reimplemented VLM baselines are needed to assess the performance advantage.

  2. The information-preservation claims in Section 2.3 and Appendices D.2 and E are stronger than the analysis establishes. Flattening a grid does not itself remove information when the original dimensions and indexing are retained, although it can change how spatial relationships are represented. Similarly, having all historical steps contribute to Perceiver cross-attention does not guarantee complete temporal information preservation. The stated limit of three independent temporal statistics per spatial location also requires assumptions beyond the data processing inequality. These issues weaken the theoretical justification for the proposed representation design.

  3. The specific benefit of hierarchical pattern matching is not isolated by the reported ablations. Table 3 compares Prompt Forge with a zero prompt token, which supports the usefulness of the complete prompting module but does not establish the contribution of its coarse-grained and fine-grained prototype organization. A controlled comparison that changes the hierarchy while retaining the remaining prompting mechanism would help determine whether the proposed Hierarchical Pattern Memory offers a distinct advantage.

  4. The comparative evidence for few-shot and zero-shot forecasting is narrower than the supervised evaluation. Although Appendix F appropriately excludes the target dataset and its same-source counterpart from auxiliary training, Figure 4a compares methods only on BikeNYCin and does not include ST-VLM or ST-Vision-LLM. Figure 4b reports VLM4ST results on the remaining datasets without corresponding baseline comparisons. These results demonstrate transfer capability, but provide limited support for broad superiority over the closest VLM-based predictors under restricted target-data availability.

Questions for authors

• Could the authors specify the dataset versions, observation periods, and preprocessing procedures used for each benchmark? In particular, were the train, validation, and test sets separated chronologically, how were sequences crossing their boundaries handled, and which observations were used to determine the min-max normalization parameters?

• Which backbone and layers were used for ST-LLM+, and how should the LLaMA-2 description in Section 3.1 be reconciled with the 83M total parameters in Table 7? Please also provide the backbone checkpoints and training configurations used for the ST-VLM and ST-Vision-LLM reimplementations.

• In the few-shot experiments, do the reported percentages refer exclusively to the target training split, and were additional target validation labels used for model selection? For zero-shot evaluation, how were normalization and model selection performed, and did the compared methods use the same auxiliary datasets?

• Are results available comparing the two-level Hierarchical Pattern Memory with a version that uses a single level of prototypes under a comparable parameter budget? This would directly test the contribution of hierarchical matching beyond the improvement obtained by adding Prompt Forge.

• What assumptions justify the information-preservation statements in Appendices D.2 and E? Please distinguish the empirical benefit of combining the visual and sequence pathways from guarantees about preserving complete temporal information or recovering independent temporal statistics.

Reviewer_DZ1nOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VLM4ST adapts ALBEF to gridded spatiotemporal forecasting through disentangled attention, learned hierarchical prompts, and visual/sequential input transformations. It reports consistent gains on seven short-horizon and four long-horizon tasks, transfer settings, and component/backbone ablations. I find the empirical contribution sufficiently substantial for weak acceptance. The remaining issues concern the explanation of visual transfer, the strength of the information-preservation claims, and a precise account of the adaptation and evaluation stages.

Critical strengths
  1. The performance evidence is broad and internally coherent. Tables 1–2 (pp. 6–7) show gains across all evaluated datasets and both forecasting horizons. The comparison includes task-specific, language-model-based, and recent VLM-based predictors, while Table 3 tests the principal modules. This establishes a useful forecasting contribution beyond one favorable dataset or one baseline.

  2. The backbone controls materially support the value of pretraining. Appendix A (pp. 14–15) includes no-backbone, random-initialization, weight-shuffling, and alternative-VLM comparisons. These are consequential controls for separating useful pretrained representations from additional model capacity and surrounding forecasting components.

Critical weaknesses
  1. The visual-transfer explanation is less established than the forecasting gain. Appendix B (pp. 15–16) visualizes temporal-to-RGB weights and attention rollout on TaxiNYC-in, but attention to high-intensity regions does not establish that transferred spatial structure causes the improvement. A controlled spatial-structure perturbation or regional masking comparison with the other modules fixed would directly test that interpretation. The accompanying information argument is also too strong: Appendices D–E infer complete temporal preservation from all time steps participating in attention and characterize loss from output channel count. Neither participation nor channel count establishes recoverability after pooling, latent compression, and prior feature mixing. State the variables, assumptions, and recoverability criterion for any formal claim; the existing ablations establish predictive complementarity.

  2. The adaptation pipeline and cost claims need experiment-level specification. The supplied configuration identifies ALBEF-14M, LoRA rank 16, and a forecasting loss; no separate image–text continued-pretraining stage is described. Please distinguish imported VLM pretraining, auxiliary-dataset forecasting training, and task adaptation, with their objectives and trainable components. Appendix H/Table 7 (p. 21) already reports inference cost, but only on BikeNYC-in: VLM4ST uses 0.133 min and 2.015 GB versus UniST's 0.034 min and 0.280 GB. Training cost, size scaling, and cost comparisons with the VLM baselines are needed to substantiate the claimed practical tradeoff. Evaluation provenance also matters: the September 25 code fits extrema across all splits and selects transfer checkpoints on target validation loss; the manuscript was revised October 6. Identify the actual experiment revision and its normalization/source-only selection rules.

Questions for authors
  1. What evidence isolates useful pretrained spatial structure from intensity sensitivity and the surrounding ST modules, and which explicit assumptions justify the information-preservation statements?
  2. Please specify the training stages, whether any image–text CPT occurred, the experiment-matching configurations and transfer-selection rules, and the training/inference cost–accuracy tradeoff against the closest VLM baselines.
Reviewer_iYDaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adapts ALBEF for forecasting urban grid data, such as taxi demand and traffic speed. DiX-Attention extracts spatiotemporal features, which are converted into image and sequence inputs for ALBEF’s visual and text branches. Prompt Forge generates input-dependent prompts from a hierarchical memory. The main contribution is this adaptation pipeline and its consistent improvements across several forecasting tasks.

Critical strengths
  1. The two input paths offer a sensible way to combine spatial layout and temporal history without natural-language input.

  2. The method consistently outperforms recent baselines across both short-term and long-term forecasting tasks.

  3. The random-initialization and shuffled-weight comparisons provide useful evidence that pretraining helps.

Critical weaknesses
  1. The ablations do not show whether Prompt Forge and DiX-Attention outperform simpler prompt generators and spatiotemporal encoders.

  2. Transfer comparisons cover only BikeNYC_in and exclude the strongest VLM baselines, leaving the broader transfer claim poorly supported.

  3. The data splitting, zero-shot normalization, and training setup for randomly initialized backbones are not clearly specified.

Questions for authors

I have no more questions, other concerns please relate to my weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.