- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VLM4ST, a framework that repurposes pretrained vision-language models for grid-based urban spatiotemporal forecasting. The framework combines Disentangled Cross-Attention, a Dual-Path Format Transformer, and Prompt Forge, which generates continuous prompts through Hierarchical Pattern Memory. An ALBEF backbone adapted with LoRA integrates visual and sequential representations, and a Horizon Query Predictor produces future spatial fields. Experiments on bike flow, taxi flow, and traffic speed datasets report improvements over selected task-specific, LLM-based, and VLM-based baselines, supported by short-term, long-term, few-shot, zero-shot, and ablation evaluations.
Critical strengths
1.The framework provides a coherent approach to adapting a pretrained VLM to numerical spatiotemporal observations. The visual pathway processes spatially structured pseudo-images, while the text encoder receives continuous embeddings without natural-language input. Their interaction through cross-attention offers a useful design for transferring pretrained representations to urban forecasting.
-
The supervised benchmark evaluation covers seven short-term forecasting tasks, four long-term forecasting tasks, and 17 baselines, including ST-VLM and ST-Vision-LLM. Tables 1 and 2 report the lowest MAE and RMSE across the evaluated tasks, with an average short-term MAE reduction of approximately 12.3% relative to the strongest baseline on each dataset. The breadth and consistency of these reported results provide meaningful empirical support for the complete framework.
-
Appendix A includes useful controls examining the contribution of pretraining, including random initialization, shuffled pretrained weights, and comparisons with CLIP and BLIP. On BikeNYCin, pretrained ALBEF achieves an MAE of 1.018 compared with 1.451 for random initialization and 1.437 for shuffled weights. These results support the usefulness of pretrained representations under the evaluated adaptation procedure.
Critical weaknesses
-
Important experimental details remain unclear, limiting confidence in the reproducibility and fairness of the reported comparisons. Section 3.1 and Appendix G state that methods use common datasets, splits, horizons, normalization, and metrics, but the manuscript does not clearly specify chronological splitting, the treatment of sequences crossing split boundaries, or which observations determine the normalization statistics. The baseline configuration also requires clarification: Section 3.1 identifies LLaMA-2 as the ST-LLM+ backbone, whereas Table 7 reports 83M total parameters without explaining this configuration. Exact configurations and training settings for the reimplemented VLM baselines are needed to assess the performance advantage.
-
The information-preservation claims in Section 2.3 and Appendices D.2 and E are stronger than the analysis establishes. Flattening a grid does not itself remove information when the original dimensions and indexing are retained, although it can change how spatial relationships are represented. Similarly, having all historical steps contribute to Perceiver cross-attention does not guarantee complete temporal information preservation. The stated limit of three independent temporal statistics per spatial location also requires assumptions beyond the data processing inequality. These issues weaken the theoretical justification for the proposed representation design.
-
The specific benefit of hierarchical pattern matching is not isolated by the reported ablations. Table 3 compares Prompt Forge with a zero prompt token, which supports the usefulness of the complete prompting module but does not establish the contribution of its coarse-grained and fine-grained prototype organization. A controlled comparison that changes the hierarchy while retaining the remaining prompting mechanism would help determine whether the proposed Hierarchical Pattern Memory offers a distinct advantage.
-
The comparative evidence for few-shot and zero-shot forecasting is narrower than the supervised evaluation. Although Appendix F appropriately excludes the target dataset and its same-source counterpart from auxiliary training, Figure 4a compares methods only on BikeNYCin and does not include ST-VLM or ST-Vision-LLM. Figure 4b reports VLM4ST results on the remaining datasets without corresponding baseline comparisons. These results demonstrate transfer capability, but provide limited support for broad superiority over the closest VLM-based predictors under restricted target-data availability.
Questions for authors
• Could the authors specify the dataset versions, observation periods, and preprocessing procedures used for each benchmark? In particular, were the train, validation, and test sets separated chronologically, how were sequences crossing their boundaries handled, and which observations were used to determine the min-max normalization parameters?
• Which backbone and layers were used for ST-LLM+, and how should the LLaMA-2 description in Section 3.1 be reconciled with the 83M total parameters in Table 7? Please also provide the backbone checkpoints and training configurations used for the ST-VLM and ST-Vision-LLM reimplementations.
• In the few-shot experiments, do the reported percentages refer exclusively to the target training split, and were additional target validation labels used for model selection? For zero-shot evaluation, how were normalization and model selection performed, and did the compared methods use the same auxiliary datasets?
• Are results available comparing the two-level Hierarchical Pattern Memory with a version that uses a single level of prototypes under a comparable parameter budget? This would directly test the contribution of hierarchical matching beyond the improvement obtained by adding Prompt Forge.
• What assumptions justify the information-preservation statements in Appendices D.2 and E? Please distinguish the empirical benefit of combining the visual and sequence pathways from guarantees about preserving complete temporal information or recovering independent temporal statistics.