acceptodds
Withdrawn from ICLR 2027

Video Foundation Models Are Native Spatio-Temporal Field Forecasters

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Natural videos and urban spatio-temporal fields exhibit similar patterns of spatial evolution, suggesting that video pretraining may provide useful forecasting representations. However, the benefits of video pretraining and the feasibility of sharing a backbone across heterogeneous urban domains remain insufficiently explored. We introduce VFM4ST, a framework that renders numerical fields as pseudo-videos and jointly encodes spatial and temporal structure using a pretrained video foundation model. Domain-specific three-port decoders separate forecast baselines, field evolution, and output geometry. Direct access to native observations retains local information for forecasting on both dense grids and sparse stations. Across traffic flow, radar nowcasting, crime, and air quality, one shared 87M-parameter backbone achieves better mean performance than the strongest evaluated baseline in every domain. The shared model remains within 2.2% of independently specialised models in each domain. Under the same joint training protocol, video pretraining outperforms image pretraining and random initialisation across all four domains. Frozen-backbone and low-data evaluations further support the value of pretrained video representations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatio-temporal Forecasting, Video Foundation Models, Urban Computing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GszmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VFM4ST, which uses a domain encoder (not learnable), a Video ViT model (pre-trained), and a decoder (jointly optimized with the Video ViT model. The prediction for a domain is obtained by using a three-level processing representing Anchor, Dynamics, and Readout. The paper shows that this process leads to good results on four small-scale datasets, both when training for each downstream task independently and when training jointly for all tasks.

Critical strengths
  • The method is evaluated on four heterogeneous domains, including both dense grid-based prediction and sparse station-based forecasting.
  • The ablation comparing video-pretrained initialization with training from scratch shows consistent performance improvements.
  • The experiments include useful studies on pretraining modality, adaptation strategy, low-data performance, domain-specialized training, and shared multi-domain training.
Critical weaknesses
  • The work overlaps considerably with ST-VFM (Chen et al. 2025). ST-VFM is cited but not evaluated as a baseline.
  • ST-VFM uses image backbone and performs experiments with video backbones showing that these underperform. The speculated reason is that image datasets are much more diverse and large compared to video datasets, thus image backbones have better pre-training. This contradicts with the main narrative of this paper, and it is not proven otherwise by the paper.
  • The proposed “three-port decoder” is just expressing established mechanisms of the downstream tasks (persistence or climatological anchoring, warping/advection, residual correction, and geometry-specific readout).
  • Multi-domain training with domain embeddings and domain-specific heads is standard multi-task or multi-domain learning.
  • Experiments do not account for overall computational complexity.
  • It is not clear whether the performance improvements are due to the method or just the use of the specific backbone.
Questions for authors
  1. How does the method compare with ST-VFM?
  2. What is technically novel in the proposed “three-port decoder” beyond the established use of persistence or climatological anchors, warping or advection, residual correction, and task-specific readout mechanisms?
  3. What is the methodological contribution of the shared model, beyond standard multi-domain learning?
  4. What is the computational cost of the method and how does it compare with the rest methods in the comparisons?
Reviewer_csp8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VFM4ST transfers video-pretrained representations to heterogeneous numerical field forecasting. It renders historical observations as pseudo-videos, encodes them using VideoMAE V2, and predicts future values through domain-specific anchor, dynamics, and readout modules with access to native observations. Experiments cover traffic, radar nowcasting, complaint counts, and air quality. A shared backbone outperforms the strongest evaluated baseline in each domain, while initialisation comparisons support the usefulness of video pretraining.

Critical strengths
  1. The evaluation spans four domains with different temporal scales and observation geometries, demonstrating that one shared video backbone can support both dense grids and sparse stations.
  2. The matched joint-training comparison of video, image, and random initialisation provides direct evidence for video pretraining. Frozen-backbone and low-data experiments offer complementary support.
Critical weaknesses
  1. Limited novelty and incomplete positioning against closely related work. Prior studies have already repurposed video models for diverse downstream tasks. Relevant work such as Video Generation Models are General-Purpose Vision Learners (GenCeption) is not discussed in the manuscript. Although GenCeption addresses natural-video perception rather than numerical forecasting, this distinction alone does not establish VFM4ST’s methodological novelty. The paper should clarify its substantive advance over existing video-model transfer approaches and better isolate the contribution of video representations from its domain-specific decoders and native-history pathways.
  2. Several key comparisons do not control the factors needed for reliable attribution. Shared and specialised models differ in adaptation strategies and training budgets, while frozen-backbone comparisons use separate decoder-fitting schedules. Consequently, observed differences cannot be attributed cleanly to sharing or pretraining alone. The matched joint-initialisation comparison is useful, but it does not resolve these issues in the broader experimental claims.
Questions for authors
  1. How does VFM4ST differ methodologically from closely related video-model transfer approaches, including GenCeption, beyond the forecasting setting? Please clarify the incremental contribution and discuss the relevant literature.
  2. How do shared and specialised models compare under matched adaptation strategies and training budgets?
  3. Do all four shared-model results in Table 2 come from the same backbone checkpoint? How was that checkpoint selected given the different convergence schedules across domains?
Reviewer_54gEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VFM4ST asks whether video pretraining benefits urban spatio-temporal forecasting, and whether one backbone can be shared across heterogeneous urban domains. It renders numerical fields as pseudo-videos and jointly encodes spatial and temporal structure using a pretrained video foundation model, while domain-specific three-port decoders separate forecast baselines, field evolution, and output geometry and keep direct access to native observations. Across traffic flow, radar nowcasting, crime, and air quality, one shared backbone achieves better mean performance than the strongest evaluated baseline in every domain and remains close to independently specialized models. Under the same joint training protocol, video pretraining outperforms image pretraining and random initialization across all four domains, and frozen-backbone and low-data evaluations further support the value of pretrained video representations.

Critical strengths

S1. The study covers a broad range of domains and baselines. It covers four domains, and also compares against statistical baselines and learned baselines across five groups. The appendices specify data provenance, chronological splits, renderer mappings, adaptation schemes, training budgets, and computational costs for both the VFM4ST variants and representative baselines.

S2. The pretraining question is examined from several angles. Complementary comparisons cover initialization with the same architecture (video-pretrained, inflated image-pretrained, randomly initialized), a frozen backbone, frame-wise encoding, low-data evaluations, and alternative video checkpoints. The same initialization comparison is repeated under one identical joint training protocol.

S3. Null and negative results are reported. Adaptation on the other domains does not improve transfer to a held-out domain beyond the original video-pretrained representation, and one domain gets worse. Larger backbones bring no consistent gain. Sharing consolidates backbone weights while retaining near-specialized accuracy, but offers no training-cost advantage under these configurations. The paper also includes optimization-dynamics analyses (convergence schedules and gradient cosine similarities).

Critical weaknesses

W1. I think the title and framing overreach the evidence. “Native forecasters” suggests light adaptation, but the frozen backbone is below the strongest baseline on traffic and roughly tied on air quality. Other video foundation models (VideoMAE V1, TimeSformer, V-JEPA 2 ViT-L) all underperform VideoMAE V2 Base, so the finding concerns one checkpoint family with MAE-style pixel reconstruction, not VFMs in general. The air-quality canvas is a non-geographic station-name grid, so the “similar patterns of spatial evolution” motivation does not apply there, yet pretraining still helps. That points to a generic pretrained-Transformer effect rather than field-specific transfer.

W2. “Strongest evaluated baseline” excludes the strongest known baselines in at least two domains. To my knowledge, under Earthformer’s SEVIR protocol, the Earthformer paper reports CSI-M for both Earthformer and a ConvLSTM baseline above what is reported here. The strongest baseline evaluated here is a generative flow-matching model, and generative nowcasters typically trade CSI-M for realism. I think Earthformer and a ConvLSTM/PredRNN-class deterministic baseline must be run under this protocol. The argument that recent methods “have already been shown to outperform widely used classical models” holds on the datasets they were published on, not on UTD19-Zurich. Graph WaveNet (cited), AGCRN and STAEformer-class baselines are absent.

W3. Margins in several domains are within baseline seed noise, and no test is reported. On air quality the gap is smaller than the baseline’s own seed standard deviation. On crime and traffic the margins are also small relative to the baselines’ variation. With only a few seeds, “better mean performance in every domain” needs paired tests or a bootstrap over test windows, and the air and crime claims should be softened to match.

W4. The own-domain pretraining comparison is confounded by the adaptation scheme. Scratch and image-pretrained variants use full fine-tuning everywhere, while the video model uses validation-selected LoRA on traffic and air quality, and partial unfreezing on crime. Full fine-tuning of an 87M-parameter model on these small training sets will overfit, so part of the scratch gaps on traffic and crime may be adaptation rather than initialization. The joint-training comparison is the clean comparison and should carry the claim.

W5. Sharing delivers no measurable benefit, and the natural alternative is missing. Shared training offers no training-cost advantage under these configurations, is slightly worse than specialized in several domains, and leave-one-domain-out shows no positive transfer. The one argued benefit, a single backbone, is also provided by the PEFT variant (frozen base plus domain-specific LoRA sets) that the variant definitions and training budgets describe, yet PEFT accuracy is not reported. If PEFT is at or near specialized accuracy, it dominates the shared variant on storage, training cost and accuracy. The “effective cross-domain sharing” contribution then reduces to “sharing is possible at a small accuracy cost and higher training cost”.

Questions for authors

Q1. Were LoRA rank and learning rate re-tuned for V-JEPA, ViT-L, VideoMAE, and TimeSformer, or reused from V2 Base?

Q2. Why does video pretraining still help on air quality, where the canvas is a non-geographic station-name grid and the motivation of similar spatial evolution does not apply?

Q3. The PEFT variant (frozen base plus domain-specific LoRA sets) is defined in Table B1 and costed in Table B3, but its accuracy is not reported.

Q4. Please run Earthformer and a ConvLSTM/PredRNN-class deterministic baseline under the same SEVIR protocol, and report CSI at each threshold as well as CSI-M.

Q5. Could you also report the video-pretrained model with the same full fine-tuning recipe as the scratch and image-pretrained rows, in each domain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.