- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FieldDiff, a spatio-temporal foundation model (STFM) that addresses heterogeneous spatial structures through a two-stage latent neural field diffusion framework. In the first stage, the model learns a unified latent field representation by incorporating physical spatio-temporal coordinates and geometry-adaptive spatial tokens within a variational autoencoder. In the second stage, conditional residual latent diffusion is employed to recover unobserved field values from partial observations, enabling unified modeling of multiple downstream tasks, including forecasting, temporal interpolation, spatial extrapolation, and spatio-temporal imputation. Extensive experiments demonstrate competitive performance across diverse spatio-temporal datasets, including zero-shot and few-shot generalization settings.
Critical strengths
-
Unified framework for multiple downstream tasks: The paper provides a practical and unified framework for four spatio-temporal tasks: forecasting, temporal interpolation, spatial extrapolation, and spatio-temporal imputation. Formulating these tasks as conditional generation under different observation masks is appealing, as it avoids the need for separate task-specific architectures.
-
Extensive experimental evaluation: The experimental evaluation is comprehensive, covering diverse spatio-temporal domains and spatial scales. The authors pretrain the model on 20 datasets and evaluate its performance across multiple downstream tasks, including zero-shot and few-shot generalization on eight held-out datasets. FieldDiff demonstrates strong and relatively consistent performance improvements over existing baselines across different datasets and tasks.
Critical weaknesses
-
Insufficient empirical validation of the core contributions: Although the paper includes several ablation studies, some of its central claims remain insufficiently validated. In particular, the necessity of residual latent diffusion is unclear without a comparison against a deterministic predictor using the same latent field representation. Moreover, despite uncertainty modeling being a primary motivation for conditional diffusion, the evaluation relies exclusively on point prediction metrics (MAE and RMSE), without assessing the quality of the learned predictive distributions through metrics such as CRPS or calibration. These limitations make it difficult to determine the contribution of diffusion beyond the learned latent representation.
-
Limited theoretical justification for the latent representation: The theoretical analysis in Section 4.2 and Appendix C relies on strong assumptions, including a bi-Lipschitz encoder, which are not explicitly verified. Furthermore, the derived curvature bound does not guarantee lower score curvature in the latent space. The compactness claim also requires clarification: according to Table 6, the latent field contains 147,456 scalar elements (24 × 48 × 128), exceeding the raw input dimensionality (24N) for all reported pretraining datasets. Although the number of spatial tokens is reduced, the latent representation is not necessarily compact in terms of total dimensionality. Therefore, the theoretical and computational motivations for the proposed latent space require stronger justification.
Questions for authors
-
Clarification on the residual diffusion formulation: Equations (6) and (7) in Section 4.3 appear to be mathematically inconsistent. Given the definition of the standardized residual, transforming Eq. (6) back into the original latent space should introduce an additional factor of (\sigma_c) in the noise term of Eq. (7). Could the authors clarify whether this is a typographical error and confirm which formulation is used in the actual implementation?
-
Although the local queries are constructed through geometry-based assignments, Eq. (3) performs cross-attention over all spatial observations. How does the model ensure that the local tokens capture spatially localized information rather than learning global attention patterns?
6 comments
The Second Part of the Response to Reviewer PbeG
W2:Clarification of theoretical assumptions and latent representation compactness.
-
Interpretation of the theoretical result. Our theoretical analysis aims to establish geometric control rather than guarantee a reduction in score curvature. Specifically, Proposition A.1 shows that, under suitable regularity conditions, the latent score curvature is bounded by the data-space curvature and additional terms associated with the distortion of the representation mapping . This bound indicates that a sufficiently regular encoder can prevent uncontrolled amplification of score curvature during representation mapping.
-
Practical considerations of the theoretical assumptions. We further clarify how our architectural design supports the regularity assumptions.
-
Bi-Lipschitz assumption. The upper Lipschitz bound prevents excessive amplification of input perturbations. FieldDiff formulates discrete observations as samples of continuous coordinate-based fields. This formulation reduces dependence on discretization-specific variations. The lower bound prevents the encoder from collapsing distinct field states and controls the sensitivity of the inverse mapping. Spatial tokenization aggregates observations along the spatial dimension while retaining the temporal dimension, helping preserve informative field variations.
-
Smoothness assumption. FieldDiff employs continuous coordinate embeddings, differentiable projection and attention operations, and soft spatial assignments. The representation pathway does not involve hard assignments, Top-K selection, or discrete routing. These architectural choices favor a locally smooth representation mapping and avoid explicit discontinuities.
-
Beyond the theoretical analysis, Figure 7 in the original manuscript provides empirical evidence. We compare the local perturbation sensitivity of data-space and latent-space denoisers. The latent-space denoiser exhibits more concentrated and stable sensitivity, suggesting more stable local denoising behavior in the learned representation space.
-
-
Latent representation compactness. Our primary objective is to establish a unified latent topology for cross-domain spatio-temporal foundation modeling. The global and local token assignment enables domains with different spatial supports to share a common representation structure. As FieldDiff targets foundation modeling across diverse domains and spatial scales, we adopt a larger token budget () to provide sufficient representational capacity for capturing both rich intra-domain spatial patterns and transferable inter-domain characteristics. This increased capacity is particularly important for scaling pretraining across heterogeneous datasets, where spatial patterns can vary substantially within and across domains. For more restricted single-domain settings, a substantially smaller token budget can be adopted. To validate this, we conduct an additional experiment using only spatial tokens across four city-level bike datasets. We compare FieldDiff with UrbanDiT across all four downstream tasks, with results summarized below.
| Dataset | Task | FieldDiff MAE | FieldDiff RMSE | UrbanDiT MAE | UrbanDiT RMSE |
|---|---|---|---|---|---|
| NYCBike | Forecasting | 10.1639 | 18.1099 | 11.2600 | 20.7473 |
| Temporal Interpolation | 9.5922 | 17.8893 | 10.1260 | 19.8685 | |
| Spatial Extrapolation | 17.0229 | 27.0103 | 28.6315 | 44.9351 | |
| Spatio-Temporal Imputation | 18.2265 | 29.5867 | 19.9987 | 35.1855 | |
| CHIBike | Forecasting | 1.6175 | 4.2285 | 1.6421 | 3.8622 |
| Temporal Interpolation | 0.6869 | 1.6103 | 0.7410 | 1.5865 | |
| Spatial Extrapolation | 2.4646 | 5.7531 | 3.8181 | 8.0626 | |
| Spatio-Temporal Imputation | 1.3166 | 3.8213 | 3.2812 | 7.0795 | |
| TRTBike | Forecasting | 1.2367 | 4.1983 | 1.4083 | 4.9644 |
| Temporal Interpolation | 1.4723 | 4.1695 | 1.5085 | 4.1571 | |
| Spatial Extrapolation | 1.9100 | 4.2386 | 4.0021 | 9.6607 | |
| Spatio-Temporal Imputation | 1.9704 | 4.8351 | 2.9169 | 7.4619 | |
| DCBike | Forecasting | 1.2372 | 3.1541 | 1.2546 | 3.3806 |
| Temporal Interpolation | 0.5653 | 1.5222 | 0.5833 | 1.4161 | |
| Spatial Extrapolation | 2.6754 | 9.6189 | 3.6872 | 11.0590 | |
| Spatio-Temporal Imputation | 2.9459 | 10.5196 | 3.2251 | 9.5878 |
The Third Part of the Response to Reviewer PbeG
Q1: Correction of the residual diffusion formulation.
We sincerely thank the reviewer for identifying this inconsistency and apologize for the typographical error in Eq. (7). Transforming the standardized residual back to the original latent space requires an additional condition-dependent scale factor in the noise term. We have corrected this equation in the revised manuscript. We sincerely apologize for this oversight.
Q2: Clarification of domain-local tokens and spatial dependency modeling.
We would like to clarify that the term "local" refers to domain-specific representations. As described in the original manuscript, we use latent tokens. globally shared tokens capture transferable field patterns across domains, while local tokens are derived from the physical coordinates of each domain to capture domain-specific spatial characteristics. Therefore, the distinction between global and local tokens concerns cross-domain sharing versus domain-specific adaptation, rather than global versus neighborhood spatial attention. The spatial dependencies among the resulting latent tokens are further modeled through the subsequent spatial axial-attention blocks.
Thank you for your valuable feedback. Once again, we appreciate your guidance!
The First Part of the Response to Reviewer PbeG
Dear Reviewer PbeG,
We sincerely thank you for your hard work on our paper. We carefully considered each of your comments and dealt with them accordingly.