acceptodds
Under review as a conference paper at ICLR 2027

Latent Neural Field Diffusion for Spatio-Temporal Foundation Models

Abstract

Spatio-temporal foundation models (STFMs) seek to learn transferable patterns from heterogeneous data for reuse across domains. Unlike time-series foundation models, however, STFMs must additionally handle the spatial dimension, where real-world observations exhibit diverse spatial structures, resolutions, and scales. We formulate spatio-temporal foundation modeling from a field perspective, treating discrete observations as samples from continuous fields and enabling prediction at arbitrary spatio-temporal coordinates. This formulation introduces two key challenges: learning a shared field representation across heterogeneous spatial geometries and modeling the uncertainty of diverse tasks under partial observations. We propose FieldDiff, a two-stage latent neural field diffusion framework. FieldDiff learns a unified latent field by jointly encoding observed values and physical coordinates with globally shared and geometry-adaptive tokens, together with variational regularization. It further unifies forecasting, temporal interpolation, spatial extrapolation, and spatio-temporal imputation as conditional field generation, using residual latent diffusion to model uncertainty unresolved by the observations. Experiments across diverse spatio-temporal datasets demonstrate strong multi-task performance and robust zero-shot and few-shot generalization across domains and spatial scales.a

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatio-Temporal Data Mining, Spatio-Temporal Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PbeGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FieldDiff, a spatio-temporal foundation model (STFM) that addresses heterogeneous spatial structures through a two-stage latent neural field diffusion framework. In the first stage, the model learns a unified latent field representation by incorporating physical spatio-temporal coordinates and geometry-adaptive spatial tokens within a variational autoencoder. In the second stage, conditional residual latent diffusion is employed to recover unobserved field values from partial observations, enabling unified modeling of multiple downstream tasks, including forecasting, temporal interpolation, spatial extrapolation, and spatio-temporal imputation. Extensive experiments demonstrate competitive performance across diverse spatio-temporal datasets, including zero-shot and few-shot generalization settings.

Critical strengths
  1. Unified framework for multiple downstream tasks: The paper provides a practical and unified framework for four spatio-temporal tasks: forecasting, temporal interpolation, spatial extrapolation, and spatio-temporal imputation. Formulating these tasks as conditional generation under different observation masks is appealing, as it avoids the need for separate task-specific architectures.

  2. Extensive experimental evaluation: The experimental evaluation is comprehensive, covering diverse spatio-temporal domains and spatial scales. The authors pretrain the model on 20 datasets and evaluate its performance across multiple downstream tasks, including zero-shot and few-shot generalization on eight held-out datasets. FieldDiff demonstrates strong and relatively consistent performance improvements over existing baselines across different datasets and tasks.

Critical weaknesses
  1. Insufficient empirical validation of the core contributions: Although the paper includes several ablation studies, some of its central claims remain insufficiently validated. In particular, the necessity of residual latent diffusion is unclear without a comparison against a deterministic predictor using the same latent field representation. Moreover, despite uncertainty modeling being a primary motivation for conditional diffusion, the evaluation relies exclusively on point prediction metrics (MAE and RMSE), without assessing the quality of the learned predictive distributions through metrics such as CRPS or calibration. These limitations make it difficult to determine the contribution of diffusion beyond the learned latent representation.

  2. Limited theoretical justification for the latent representation: The theoretical analysis in Section 4.2 and Appendix C relies on strong assumptions, including a bi-Lipschitz encoder, which are not explicitly verified. Furthermore, the derived curvature bound does not guarantee lower score curvature in the latent space. The compactness claim also requires clarification: according to Table 6, the latent field contains 147,456 scalar elements (24 × 48 × 128), exceeding the raw input dimensionality (24N) for all reported pretraining datasets. Although the number of spatial tokens is reduced, the latent representation is not necessarily compact in terms of total dimensionality. Therefore, the theoretical and computational motivations for the proposed latent space require stronger justification.

Questions for authors
  1. Clarification on the residual diffusion formulation: Equations (6) and (7) in Section 4.3 appear to be mathematically inconsistent. Given the definition of the standardized residual, transforming Eq. (6) back into the original latent space should introduce an additional factor of (\sigma_c) in the noise term of Eq. (7). Could the authors clarify whether this is a typographical error and confirm which formulation is used in the actual implementation?

  2. Although the local queries are constructed through geometry-based assignments, Eq. (3) performs cross-attention over all spatial observations. How does the model ensure that the local tokens capture spatially localized information rather than learning global attention patterns?

6 comments
AuthorsOpenReview ↗

The First Part of the Response to Reviewer PbeG

Dear Reviewer PbeG,

We sincerely thank you for your hard work on our paper. We carefully considered each of your comments and dealt with them accordingly.


W1: Additional validation of diffusion and probabilistic Modeling.

  • Probabilistic Evaluation. We introduce the normalized Continuous Ranked Probability Score (nCRPS) as an additional evaluation metric. It is calculated by normalizing CRPS with the mean absolute value of the ground-truth observations. Lower nCRPS indicates better probabilistic prediction quality. We compare FieldDiff with two probabilistic models, UrbanDiT and CSDI, and further include FieldDiff w/o WarmStart to examine the contribution of condition-aligned residual diffusion. The results are summarized below. We have incorporated the nCRPS definition into the revised Appendix E.3 (Evaluation Metrics) and the corresponding probabilistic evaluation results into Section 5.3 (Model Analysis).
DatasetTaskFieldDiffw/o WarmStartUrbanDiTCSDI
NYCbikeTemopral Interpolation0.13600.13650.14620.1669
Spatio-Temporal Imputation0.13220.13410.13980.1406
CHIbikeTemopral Interpolation0.06400.06650.08120.1233
Spatio-Temporal Imputation0.06680.07310.07550.0965
TRTbikeTemopral Interpolation0.06120.06980.07100.0840
Spatio-Temporal Imputation0.05780.06420.06830.0763
  • Contribution of Diffusion. To directly investigate the contribution of diffusion beyond the learned latent field representation, we introduce an additional w/o Diff variant. Specifically, we retain the neural field representation learned in Stage I and replace the Stage II diffusion model with the deterministic processor adopted from OmniField. This processor consists of three Transformer modules with identical architectures but independent parameters.
PredictionTemporal InterpolationSpatial ExtrapolationSpatio-Temporal Imputation
DatasetMethodMAERMSEMAERMSEMAERMSEMAERMSE
NYCbikeOmniField18.7725.196.2110.356.1710.065.719.23
w/o Diff12.6118.175.9410.496.1410.016.0410.11
FieldDiff9.1515.455.509.025.839.625.338.48
NYCtaxiOmniField10.2817.206.2210.286.6211.076.0010.00
w/o Diff10.2117.535.8110.137.0711.715.639.69
FieldDiff10.0717.045.639.326.3110.915.539.12

AuthorsOpenReview ↗

The Second Part of the Response to Reviewer PbeG

W2:Clarification of theoretical assumptions and latent representation compactness.

  • Interpretation of the theoretical result. Our theoretical analysis aims to establish geometric control rather than guarantee a reduction in score curvature. Specifically, Proposition A.1 shows that, under suitable regularity conditions, the latent score curvature is bounded by the data-space curvature and additional terms associated with the distortion of the representation mapping κZ(d)(z)≤C1,dκX(d)(x)+C2,d\kappa_Z^{(d)}(z)\leq C_{1,d}\kappa_X^{(d)}(x)+C_{2,d}. This bound indicates that a sufficiently regular encoder can prevent uncontrolled amplification of score curvature during representation mapping.

  • Practical considerations of the theoretical assumptions. We further clarify how our architectural design supports the regularity assumptions.

    • Bi-Lipschitz assumption. The upper Lipschitz bound LdL_d prevents excessive amplification of input perturbations. FieldDiff formulates discrete observations as samples of continuous coordinate-based fields. This formulation reduces dependence on discretization-specific variations. The lower bound mdm_d prevents the encoder from collapsing distinct field states and controls the sensitivity of the inverse mapping. Spatial tokenization aggregates observations along the spatial dimension while retaining the temporal dimension, helping preserve informative field variations.

    • Smoothness assumption. FieldDiff employs continuous coordinate embeddings, differentiable projection and attention operations, and soft spatial assignments. The representation pathway does not involve hard assignments, Top-K selection, or discrete routing. These architectural choices favor a locally smooth representation mapping and avoid explicit discontinuities.

    • Beyond the theoretical analysis, Figure 7 in the original manuscript provides empirical evidence. We compare the local perturbation sensitivity of data-space and latent-space denoisers. The latent-space denoiser exhibits more concentrated and stable sensitivity, suggesting more stable local denoising behavior in the learned representation space.

  • Latent representation compactness. Our primary objective is to establish a unified latent topology for cross-domain spatio-temporal foundation modeling. The global and local token assignment enables domains with different spatial supports to share a common representation structure. As FieldDiff targets foundation modeling across diverse domains and spatial scales, we adopt a larger token budget (K=48K=48) to provide sufficient representational capacity for capturing both rich intra-domain spatial patterns and transferable inter-domain characteristics. This increased capacity is particularly important for scaling pretraining across heterogeneous datasets, where spatial patterns can vary substantially within and across domains. For more restricted single-domain settings, a substantially smaller token budget can be adopted. To validate this, we conduct an additional experiment using only K=8K=8 spatial tokens across four city-level bike datasets. We compare FieldDiff with UrbanDiT across all four downstream tasks, with results summarized below.

DatasetTaskFieldDiff MAEFieldDiff RMSEUrbanDiT MAEUrbanDiT RMSE
NYCBikeForecasting10.163918.109911.260020.7473
Temporal Interpolation9.592217.889310.126019.8685
Spatial Extrapolation17.022927.010328.631544.9351
Spatio-Temporal Imputation18.226529.586719.998735.1855
CHIBikeForecasting1.61754.22851.64213.8622
Temporal Interpolation0.68691.61030.74101.5865
Spatial Extrapolation2.46465.75313.81818.0626
Spatio-Temporal Imputation1.31663.82133.28127.0795
TRTBikeForecasting1.23674.19831.40834.9644
Temporal Interpolation1.47234.16951.50854.1571
Spatial Extrapolation1.91004.23864.00219.6607
Spatio-Temporal Imputation1.97044.83512.91697.4619
DCBikeForecasting1.23723.15411.25463.3806
Temporal Interpolation0.56531.52220.58331.4161
Spatial Extrapolation2.67549.61893.687211.0590
Spatio-Temporal Imputation2.945910.51963.22519.5878

AuthorsOpenReview ↗

The Third Part of the Response to Reviewer PbeG

Q1: Correction of the residual diffusion formulation.

We sincerely thank the reviewer for identifying this inconsistency and apologize for the typographical error in Eq. (7). Transforming the standardized residual back to the original latent space requires an additional condition-dependent scale factor in the noise term. We have corrected this equation in the revised manuscript. We sincerely apologize for this oversight.


Q2: Clarification of domain-local tokens and spatial dependency modeling.

We would like to clarify that the term "local" refers to domain-specific representations. As described in the original manuscript, we use K=Kg+KlK=K_g+K_l latent tokens. KgK_g globally shared tokens capture transferable field patterns across domains, while KlK_l local tokens are derived from the physical coordinates of each domain to capture domain-specific spatial characteristics. Therefore, the distinction between global and local tokens concerns cross-domain sharing versus domain-specific adaptation, rather than global versus neighborhood spatial attention. The spatial dependencies among the resulting latent tokens are further modeled through the subsequent spatial axial-attention blocks.


Thank you for your valuable feedback. Once again, we appreciate your guidance!

Reviewer_JZvyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents a method to learn representations of spatio-temporal data such that (i) heterogeneous spatial geometries are handled natively and (ii) multiple downstream tasks (e.g. forecasting, temporal interpolation) can be served by a single model. The authors achieve this by leveraging a neural field formulation, and unifying downstream tasks as querying field values at arbitrary spatio-temporal coordinates. The method is evaluated on several datasets against a large set of baselines.

Critical strengths
  • The neural field formualtion is appropriate for addressing the heterogeneous spatio-temporal data challenge, and the method behaves as expected
  • The experiments are extensive and include many comparisons; performance is, at a minimum, competitive with the baselines across the reported settings
  • Structure, exposition, and analysis are of good quality. The prose is clear and easy to follow, the comparisons are thorough, the ablations support the intended role of each component (on the two datasets where they are run), and an efficiency analysis is present
Critical weaknesses
  • There is a significant, missing portion of related work on neural field representation learning, which makes novelty an unresolved question. The related work section discusses spatio-temporal neural fields but omits the neural field representation learning literature, where the proposed two-stage field framework is mostly a standard construction. Closely related two-stage field frameworks include Functa [1], Spatial Functa[2], 3DShape2VecSet [3], NeuralField-LDM [4], ENF [5] , LH-NeF [6] , and AROMA [7]. The latter is precisely quite close, sharing lower level similarities, one of them being that notably, it is the only one in the previous list that also supports modeling signals over time. The review should cite these or similar, relevant works at a minimum, and contextualize the proposed architecture against the other particular instatiations of the 2-stage field framework.

  • The scope is narrower than the prose implies. The first pages suggest that heterogeneous geometry of any kind is in scope, but the method section makes clear that the formulation is restricted to a single geometry, S², since all coordinates are parameterized as lon/lat. The formulation also appears to accept only univariate field values, if I'm not mistaken. In general, the paper presents claims and narrative around "heterogeneous spatial geometries" but I believe the language here should be made more precise; perhaps multi-resolution, multi-density, and irregular sampling over a fixed domain are better descriptions of what is actually supported by the current formulation. For instance, and coming back to relevant literature, Functa, ENF and LH-NeF maintain general formulations over general geometries, which more accurately corresponds to "working across heterogeneous domains." Other related claims also remain unclear; see questions.

  • Crucial method details are missing, which prevents a low-level understanding of the approach. In Section 4.2, cdc_d and rdr_d are introduced as the spatial center and scale of domain d, but neither is defined: is cdc_d a Fréchet mean? over which elements? is it a Euclidean mean of the embedded coords? How is rdr_d computed? is it a radius? Similarly, the architecture of hsh_s is never specified (is it an MLP?). As written, the method section is not self-contained enough for a reader to reconstruct the approach without consulting the code.


References:

[1]Dupont, E., Kim, H., Eslami, S. M., Rezende, D., & Rosenbaum, D. (2022). From data to functa: Your data point is a function and you can treat it like one. arXiv preprint arXiv:2201.12204.

[2]Bauer, M., Dupont, E., Brock, A., Rosenbaum, D., Schwarz, J. R., & Kim, H. (2023). Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130.

[3] Zhang, B., Tang, J., Niessner, M., & Wonka, P. (2023). 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions On Graphics (TOG), 42(4), 1-16.

[4] Kim, S. W., Brown, B., Yin, K., Kreis, K., Schwarz, K., Li, D., ... & Fidler, S. (2023, June). Neuralfield-ldm: Scene generation with hierarchical latent diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 8496-8506). IEEE.7

[5] Wessels, D., Knigge, D., Valperga, R., Papa, S., Vadgama, S., Gavves, E., & Bekkers, E. (2025, May). Grounding continuous representations in geometry: Equivariant neural fields. In International Conference on Learning Representations (Vol. 2025, pp. 59774-59794).

[6] Urbano, A., Romero, D. W., Zimmer, M., & Pokutta, S. (2026). Neural Field Tokenizations with Hierarchy and Spatial Locality Priors. arXiv preprint arXiv:2606.08204.

[7] Serrano, L., Wang, T. X., Le Naour, E., Vittaut, J. N., & Gallinari, P. (2024). Aroma: Preserving spatial structure for latent pde modeling with local neural fields. Advances in Neural Information Processing Systems, 37, 13489-13521.

Questions for authors

1 - Why is a second condition encoder needed in Stage 2? Is it because the Stage 1 encoder is trained only on complete fields? Is the condition encoder trained from scratch or initialized from the Stage 1 encoder?

2 - What exactly is "w/o CondAlign" in the ablation? Is this the alignment loss appearing in Figure 2 as "geometry alignment"? It is never defined in any equation.

3 - There is no ablation that removes diffusion entirely. "w/o WarmStart" only replaces the condition-derived initialization/centering with pure noise, which leaves open whether the diffusion process itself contributes anything.

4 - Can the authors address the scope claims and the literature comparisons in Weaknesses 1 and 2? Some running example would perhaps make things clearer, since it is not clear to me whether certain scenarious are covered. For instance, given a dataset in which the same field (e.g., global temperature) is sampled at coordinates that differ across timesteps, with irregular and varying sampling regimes, can the current formulation learn from such a scenario?

5 - The efficiency comparison lacks detail. For the 50.68 s vs. 6.57 s comparison, please specify whether this measures throughput per batch, at what batch size, on what hardware, and disclose the full efficiency protocol.

In general, if these concerns are resolved positively, I am willing to raise my score.

4 comments
AuthorsOpenReview ↗

The First Part of the Response to Reviewer JZvy

Dear Reviewer JZvy,

We sincerely appreciate the time and effort you have spent providing insightful feedback on our paper. We have carefully considered each of your comments and have addressed them one by one.


W1: Discussion of neural field representation learning and existing two-stage neural field frameworks.

We sincerely thank the reviewer. This suggestion has helped us gain a more comprehensive understanding of neural fields beyond applications in spatio-temporal modeling. Following the reviewer's suggestions, we have expanded the Related Work section and organized the relevant studies into two categories.

(1) One-stage neural field representation learning: Functa, Spatial Functa, ENF, and LH-NeF. These methods progressively improve the geometric and hierarchical properties of neural field representations. However, their primary focus is spatial representation. STFMs face substantial cross-domain variations in spatial extent, resolution, sampling density, graph or grid structures, and domain semantics, which are not explicitly addressed by these representation designs.

(2) Neural fields with latent diffusion: 3DShape2VecSet, NeuralField-LDM, and AROMA. Both 3DShape2VecSet and NeuralField-LDM primarily focus on static spatial data for 3D shape and scene generation. AROMA is particularly relevant to our work, as it extends neural fields and diffusion to time-dependent PDE modeling. It predicts latent representations of future states through autoregressive rollouts. However, its autoregressive formulation is designed for only forecasting and does not naturally support diverse spatio-temporal tasks within a unified diffusion.

These comparisons also help clarify the novelty of FieldDiff. We do not claim that the two-stage combination of neural fields and latent diffusion is new in itself. Instead, FieldDiff specifically addresses the two key challenges discussed in Introduction of the manuscript. We have incorporated the above discussions and comparisons into the revised Related Work section.


W2: Clarification of data geometry.

After reviewing the related literature suggested in the previous comment, we agree that the term "heterogeneous spatial geometries" is overly broad and may imply support for more general geometric structures than our current framework considers.

  • The spatial heterogeneity we aim to address arises from variations in spatial extent, resolution, sampling density, and observation structures, including grid-based and graph-based data. We agree that this differs from the more general notion of geometry considered in neural field representation learning. Following the reviewer's suggestion, we have revised the corresponding description in Lines 51–53.

  • Univariate field modeling. In our formulation, different variables are treated as separate domains, each represented by a univariate field. By jointly pretraining on multiple such domains, FieldDiff learns transferable representations. This design enables unified modeling across different variables and datasets.

  • Definition of domains. We would also like to clarify that the term "domain" in our work follows a common usage in spatio-temporal data modeling. "Domains" refer to applications such as urban mobility, traffic, climate, and environmental monitoring, as introduced in Lines 33–38. This may differs from the notion of geometric domains in neural field representation learning, such as 2D or 3D spaces.


W3: Missing implementation details.

We apologize for the insufficient details in the original manuscript. The spatial center cdc_d is computed as the arithmetic mean of the coordinates of all spatial nodes in domain dd. The spatial scale rdr_d is defined as the root mean square distance of these nodes to cdc_d. The spatial projection network hsh_s is implemented as an MLP with two linear layers. We have added these details to Section 4.2.


AuthorsOpenReview ↗

The Second Part of the Response to Reviewer JZvy

Q1: Initialization of the condition encoder.

As described in the original manuscript, the condition encoder inherits the value projection, coordinate encoding, and spatial tokenizer from the Stage I encoder. Unobserved positions are represented by a learnable mask embedding while retaining their spatial and temporal coordinates. The cross-attention and spatio-temporal attention blocks are initialized with the corresponding Stage I weights and further fine-tuned during Stage II. Thus, the condition encoder builds upon the pretrained Stage I representations rather than being independently trained from scratch.


Q2: Clarification of the CondAlign ablation.

In the revised manuscript, we have added definitions of all ablation variants in the Ablation Study section. w/o ConAlign refers to training the Stage II condition encoder independently from scratch, without reusing the pretrained Stage I field encoder or inheriting its learned representations. Consequently, the condition encoder no longer preserves the latent-space geometry established by Stage I. We would also like to clarify that the term "geometry alignment" in Figure 2 refers to the alignment of latent-space representations.


Q3: Additional ablation study that removes diffusion entirely.

We have introduced an additional w/o Diff variant below. We retain the neural field representation learned in Stage I and replace the Stage II diffusion model with the deterministic processor adopted in OmniField. This processor consists of three Transformer modules with identical architectures but independent parameters. The results are summarized below.

PredictionTemporal InterpolationSpatial ExtrapolationSpatio-Temporal Imputation
DatasetMethodMAERMSEMAERMSEMAERMSEMAERMSE
NYCbikeOmniField18.7725.196.2110.356.1710.065.719.23
w/o Diff12.6118.175.9410.496.1410.016.0410.11
FieldDiff9.1515.455.509.025.839.625.338.48
NYCtaxiOmniField10.2817.206.2210.286.6211.076.0010.00
w/o Diff10.2117.535.8110.137.0711.715.639.69
FieldDiff10.0717.045.639.326.3110.915.539.12

Q4: Clarification of irregular spatio-temporal sampling scenario.

  • Irregular spatio-temporal sampling across different tasks. Our downstream tasks cover different forms of incomplete observations. Temporal interpolation handles discontinuous temporal observations. Spatial extrapolation recovers spatial locations that are not observed during training. Spatio-temporal imputation further considers incomplete observations across both spatial and temporal dimensions. As illustrated in the upper-right part of Figure 2, the observed coordinates can be irregularly distributed in both dimensions.

  • Cross-scale generalization. We further evaluate this capability through cross-scale zero-shot and few-shot transfer experiments. We pretrain FieldDiff on ERA5 temperature data, which provides global temperature at an hourly resolution. Then we transfer it to CMIP6clt, which contains climate data at a finer spatial granularity and a monthly temporal resolution. This experiment evaluates generalization across both spatial scales and temporal sampling resolutions.

We would be happy to further provide any additional scenarios the reviewer considers relevant, and discuss them further.


Q5: Efficiency study settings.

The reported inference times represent the total time required to process the entire test set. We conduct the evaluation on the forecasting task of NYCbike with a batch size of 1, measuring the total inference time across all test samples. All compared models are evaluated under the same hardware configuration to ensure a fair comparison. The GPU specifications are provided in Section E.2 Implementation Details of the original manuscript.


We hope the responses above address your concerns. Should you have any further suggestions, we are more than happy to discuss them and make necessary improvements to the paper.

Cheers,

All authors

Reviewer_JZvyOpenReview ↗

Rebuttal response

I thank the authors for the thorough response and the experiments ran. I believe the much-needed changes concerning missing ablations and missing literature address my concerns well. Overall, and with those concerns resolved, I do believe the empirical contributions of this paper are valuable, and have raised my score accordingly as a result.

AuthorsOpenReview ↗

Sincere Gratitude from Authors

We are delighted that our responses have successfully addressed your concerns. We would like to express our deepest gratitude for taking the time to review our paper and for providing such detailed and invaluable feedback.

Reviewer_CRBMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FieldDiff, a latent neural field diffusion framework for spatio-temporal foundation models. It combines adaptive geometry field representations with conditional residual diffusion to unify multi-tasks as conditional generation from partial observations from two stage training. Experiments demonstrate competitive performance across heterogeneous datasets, including zero-shot and few-shot generalization.

Critical strengths
  1. The paper addresses an important problem of unified spatio-temporal modeling across heterogeneous spatial structures and downstream tasks.
  2. The experimental evaluation is comprehensive. It covers multiple datasets and cross-domain generalization settings, including forecasting, interpolation, spatial extrapolation, and imputation. It also exhibits robust zero-shot and few-shot generalization capability.
Critical weaknesses
  1. The paper emphasizes modeling uncertainty through conditional latent diffusion, but the experiments rely exclusively on MAE and RMSE metrics. Without probabilistic metrics such as CRPS, or distributional fidelity, the claimed uncertainty modeling capability remains unsupported.
  2. The theoretical analysis relies on strong assumptions about the smoothness and invertibility of the learned encoder which are not verified in paper. Moreover, the derived curvature bound does not guarantee smoother latent distributions or improved diffusion performance, limiting the core theoretical justification.
Questions for authors
  1. Can the authors provide probabilistic evaluation metrics to validate the claimed uncertainty modeling capabilities?
  2. How are the theoretical assumptions on latent-space geometry satisfied in practice?
  3. How does the framework prevent degenerate estimation on condition mean and scale or ensure that the standardized residual meaningfully represents unresolved uncertainty?
  4. How sensitive is the model performance to KL weight, the number of global/local tokens, the initial diffusion noise level, and the number of sampling steps? The current sensitivity analysis to key hyperparameters is absent.
5 comments
AuthorsOpenReview ↗

The First Part of the Response to Reviewer CRBM

Dear Reviewer CRBM,

We greatly appreciate your hard work during the review process. We will address your concerns one by one to clarify any confusion. Specifically:


W1, Q1: Additional probabilistic evaluation of uncertainty modeling.

We introduce the normalized Continuous Ranked Probability Score (nCRPS) as an additional evaluation metric. It is calculated by dividing the CRPS by the mean absolute value of the ground-truth observations nCRPS=CRPS/(1M∑i=1M∣yi∣)\mathrm{nCRPS} = \mathrm{CRPS}/(\frac{1}{M}\sum_{i=1}^{M}|y_i|), where MM denotes the number of evaluated observations and yiy_i denotes the corresponding ground-truth value. Lower nCRPS indicates better probabilistic prediction quality. We compare FieldDiff with probabilistic models UrbanDiT and CSDI. We also include FieldDiff w/o WarmStart to evaluate the contribution of warm-start diffusion to probabilistic modeling.

DatasetTaskFieldDiffw/o WarmStartUrbanDiTCSDI
NYCbikeTemporal Interpolation0.13600.13650.14620.1669
Spatio-Temporal Imputation0.13220.13410.13980.1406
CHIbikeTemporal Interpolation0.06400.06650.08120.1233
Spatio-Temporal Imputation0.06680.07310.07550.0965
TRTbikeTemporal Interpolation0.06120.06980.07100.0840
Spatio-Temporal Imputation0.05780.06420.06830.0763

Compared with UrbanDiT and CSDI, FieldDiff consistently improves probabilistic prediction quality across different datasets and tasks. Moreover, removing warm-start diffusion increases nCRPS in all settings, further demonstrating the contribution of our condition-aligned residual diffusion to uncertainty modeling. We have incorporated the probabilistic evaluation metric in revised E.3 Evaluation Metrics section, and corresponding experimental results in revised 5.3 Model Analysis section.


W2, Q2: Justification of the theoretical assumptions and clarification of the latent score curvature bound.

  • Interpretation of the theoretical result. Our theoretical analysis aims to establish geometric control. Specifically, Proposition A.1 shows that, under suitable regularity, the latent score curvature is bounded by the data-space curvature and additional terms associated with the distortion of the representation mapping. This indicates that a sufficiently regular encoder can prevent uncontrolled amplification of score curvature.

  • Practical considerations of the theoretical assumptions.

    • The bi-Lipschitz assumption requires the encoder to preserve informative field variations while maintaining bounded sensitivity to input perturbations. (The upper Lipschitz bound LdL_d): FieldDiff formulates spatio-temporal observations as samples of underlying continuous coordinate-based fields. This formulation reduces dependence on discretization-specific variations in the input observations. (The lower Lipschitz bound mdm_d): FieldDiff compresses the spatial dimension through tokenization while retaining the temporal dimension, helping preserve temporal information during spatial aggregation.
    • The smoothness assumption is supported by our use of continuous coordinate embeddings, differentiable attention operations, and soft spatial token assignments. The representation pathway does not involve hard assignments, Top-K selection, or discrete routing operations. These choices favor a locally smooth mapping.
  • Empirical validation of latent-space smoothness. Beyond the theoretical analysis, we provide empirical evidence in Figure 7 of the original manuscript. We compare the finite-difference sensitivity of data-space and latent-space denoisers. The latent-space denoiser exhibits lower and more concentrated local sensitivity. These observations indicate more stable local denoising behavior in our learned latent space.

We have revised the theoretical discussion to clarify the theoretical geometric control, and explain the architectural motivation behind the assumptions.


AuthorsOpenReview ↗

The Second Part of the Response to Reviewer CRBM

Q3: How to prevent condition degeneration.

The condition mean and scale are not estimated by an independently trained prediction head with unconstrained parameters. Instead, our design anchors their estimation to the pretrained Stage I encoder.

  • The condition prediction head hch_c directly reuses the mean and log-variance prediction heads of the Stage I VAE. Their parameters remain frozen during Stage II training. These heads have already been optimized through the Stage I reconstruction and KL regularization objectives.

  • The condition encoder also inherits the value projection, coordinate encoding, and spatial tokenizer from Stage I, with all their parameters frozen. The cross-attention and spatio-temporal attention blocks are initialized with the corresponding Stage I weights and further fine-tuned during Stage II training. Only the mask embedding is initialized and learned from scratch. This design constrains the condition representation to remain aligned with the latent representation.


Q4: Additional sensitivity analysis of some hyperparameters.

We conduct additional sensitivity experiments covering four key hyperparameters in the two training stages.

  • Stage I hyperparameters. We investigate the KL regularization weight βKL\beta_{\mathrm{KL}} and the allocation of global and local tokens (Kg,Kl)(K_g,K_l). Although their contributions have already been examined through the w/o VAE and w/o AdaToken ablations, respectively, we further evaluate multiple configurations to assess their sensitivity.
ConfigurationSTP MAESTP RMSETI MAETI RMSESE MAESE RMSESTI MAESTI RMSE
w/o VAE9.5616.905.579.456.3410.795.368.55
βKLβ_{KL} = 1e-59.2316.335.529.296.019.975.338.50
βKLβ_{KL} = 1e-49.1515.455.509.025.839.625.338.48
βKLβ_{KL} = 1e-39.1515.505.519.005.849.655.338.48
Configuration (KgK_g/KlK_l)STP MAESTP RMSETI MAETI RMSESE MAESE RMSESTI MAESTI RMSE
w/o AdaToken9.5416.586.2310.035.8810.285.758.97
24/249.1615.725.759.645.819.625.378.70
16/329.1515.455.509.025.839.625.338.48
8/409.1515.805.639.115.919.725.388.48
  • Stage II hyperparameters. We evaluate the sensitivity to the initial diffusion noise level σstart\sigma_{\mathrm{start}} and the number of reverse diffusion sampling steps. Specifically, we vary σstart\sigma_{\mathrm{start}} from 0.5 to 4.0 and the sampling steps from 4 to 32, while keeping the remaining configurations unchanged.
σstartσ_{start}STP MAESTP RMSETI MAETI RMSESE MAESE RMSESTI MAESTI RMSE
0.59.1515.455.479.165.589.645.328.43
1.09.1715.465.469.055.849.615.368.52
2.09.1515.455.509.025.839.625.338.48
4.09.4715.835.569.065.949.795.418.58
Sampling StepsSTP MAESTP RMSETI MAETI RMSESE MAESE RMSESTI MAESTI RMSE
49.2015.555.619.145.959.675.448.55
89.1415.435.549.025.839.615.338.48
129.1515.455.509.025.839.625.338.48
209.1715.455.509.015.849.625.358.50
329.1615.455.499.025.839.625.338.49

All experiments are conducted on NYCbike across the four downstream tasks, including prediction (STP), temporal interpolation (TI), spatial extrapolation (SE), and spatio-temporal imputation (STI). Bold values indicate the hyperparameter configurations used in the original experiments. The results show that FieldDiff exhibite stable performance across a broad range of hyperparameter configurations. We have added the corresponding sensitivity analysis and discussion to the revised manuscript.


Thank you for your valuable feedback. Once again, we appreciate your guidance!

Thank you for the detailed rebuttal. These additions address some of my concerns. However, several important issues remain unresolved.

  • Although the additional experiments are appreciated, they cover only three closely related bike-sharing datasets and two tasks. The claimed general probabilistic modeling capability across heterogeneous domains and tasks remains insufficiently validated, particularly regarding uncertainty calibration and predictive interval coverage.

  • The authors clarify the interpretation of the curvature bound but do not establish that the learned encoder satisfies the required bi-Lipschitz assumptions. The finite-difference experiments measure denoiser sensitivity rather than directly verifying the theoretical curvature bound or demonstrating improved diffusion stability.

  • Reusing frozen VAE prediction heads does not necessarily guarantee accurate conditional mean and scale estimates under partially observed inputs. Moreover, the additional sensitivity studies are conducted only on NYCbike, leaving the generalizability across heterogeneous domains unclear.

Regarding the overall rebuttal, I maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.